[Europe] VUSION Cloud maintenance

Scheduled Maintenance Report for VUSION Cloud

Postmortem

Dear customers,

On the 21st of March 2023, during the release of VUSION Cloud, in Europe environment, our Cloud engineers encounter a major issue while deploying some transmitters' management services. As a result, a large subset of customers have experienced offline transmitters, for up to 1h30. During this period of time, most transmissions have failed for those stores.

Our Cloud engineers have identified that during the deployment of some resources, Microsoft Azure missed part of the software, while marked all steps as successful. Due to this, the software was unable to start properly and to establish connexions with transmitters. Around 16:40 UTC, our cloud engineers identified the impacted piece of software and the impacted units.

In order to improve the overall reliability and to avoid the occurrence of similar incidents in the future:

We will change our deployment method to control uploaded resources;
Our deployment process has been updated to add control steps before rolling out all units.

‌

On behalf of SES-imagotag, the whole Cloud team would like to apology for the impact it may have had. Please be assured that we are working continuously to improve the quality of the platform and the user experience.

Posted Apr 04, 2023 - 14:27 UTC

Completed

The scheduled maintenance has been completed.

Posted Mar 21, 2023 - 17:25 UTC

Verifying

Verification is currently underway for the maintenance items.

Posted Mar 21, 2023 - 17:08 UTC

Update

Dear customers,

the scheduled maintenance is still in progress. Our Cloud Engineers have encountered issues during deployment of changes. The root cause is a standard step of Azure that partially failed.

As consequence, the maintenance will take longer than scheduled and is expected to be done by 17:30 UTC.

Engineers do not expect any downtime in the last steps of the maintenance. For a subset of customer, transmitters may be offline during some minutes.

On behalf of SES-imagotag, the whole Cloud team would like to apologize for the impact and to thank you for your understanding.

Posted Mar 21, 2023 - 15:53 UTC

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Posted Mar 21, 2023 - 13:00 UTC

Scheduled

Dear customers,

As part of our regularly scheduled maintenance, SES-imagotag’s Cloud Operations will perform a maintenance on your system. During this maintenance, we will update the VUSION Cloud platform in order to improve your environment.

Clients are advised to reschedule any activities that could potentially generate transmission events, outside of this maintenance. This maintenance is planned for maximum three (3) hours, the expected service disruption is around ten (10) minutes. During this maintenance, you might experience minor disturbances on matchings and data integration, and delays in your transmissions.

This maintenance is implemented according to best practices, the window is given to allow time for SES-imagotag’s engineers to respond to any unforeseen issues that may occur with this change.

Maintenance summary:
- General updates on the environment (improvements and bug fixes)

Thank you for your understanding,
SES-imagotag team
Please check VUSION Docs for more information: https://ses-esl.atlassian.net/wiki/spaces/VRN

Posted Mar 13, 2023 - 10:47 UTC

This scheduled maintenance affected: Europe (VUSION Manager - Europe, VUSION Cloud API - Europe).