On November 6th, 2023, at 4:25 UTC, a subset of customers experienced a disruption to their Access Points (APs), which lost connection to the V:Cloud and could be seen as offline status on VUSION Manager.

Disruption was caused by an internal error on the AP management module which blocked the service from running properly. Additionally, when our engineers attempted to apply our recovery procedure on the affected endpoint via Microsoft Azure, the operation failed with an error pointing to another module. This prevented the team to apply curative action. The issue was escalated to Microsoft Azure. In parallel, Cloud engineers were able to identify an new workaround, and all APs went back online quickly after 8:40 UTC.

We have taken the following steps to reduce the likelihood and impact of similar incidents in the future:

• Scheduled a change to reduce the risk of such an internal error;
• Fine-tuned alerts to detect any similar issue earlier in order to avoid impacts for customers;
• Create alerts to detect such symptoms, as early as possible;
• Escalated the reboot issue to Microsoft Azure for root cause analysis
• Working on improving our internal processes, especially by training our engineers and support teams on how to react, communicate and fix this type of scenario.

Please note that all operations resumed successfully after the APs went back online and that no data or transmissions were lost, and no actions were required from customers.

On behalf of SES-imagotag, the entire Cloud team would like to apologize for any inconvenience caused. Please be assured that we are continually working to enhance the quality of our platform and improve the user experience.

Thank you for your understanding and continued support.