The incident is now resolved. Thanks to our redondant systems, an efficient work-around has been applied and operations have resumed. The load has been resolved.
Our engineers will keep monitoring and a post mortem analysis will be provided once we have the final RCA from Microsoft Azure.
On behalf of SES-imagotag, the whole Cloud team would like to apologize for the incident and to thank you for your understanding. Please be assured that we are committed to continuous improvement and that we strive to provide the best experience as possible.
Posted Dec 01, 2023 - 06:11 UTC
Update
Dear customers,
The workaround has shown efficient and operations are running as usual. Our monitoring is not showing any major issues since it has been applied.
The investigation with Microsoft Azure for the root cause is still on-going.
Our engineers keep monitoring to ensure that delayed messages are being processed as expected.
We thank you for your understanding.
Posted Dec 01, 2023 - 05:30 UTC
Monitoring
Our engineers have implemented our workaround capabilities and operations are resuming. Our metrics show that we do no get major errors since 04:25 UTC. The delayed actions are being processed. We have also contacted Azure to get a root cause and an estimate of complete recovery of their systems.
Posted Dec 01, 2023 - 04:48 UTC
Identified
The incident is indeed linked to an Azure outage on SQL server. Our engineers are working on applying workaround to resume standard operations as soon as possible.
Our monitoring shows that all incoming calls are successfully received and no data is lost. At this stage, no action is expected from customers.
We will update this page as soon as possible.
We thank you for your understanding.
Posted Dec 01, 2023 - 04:24 UTC
Investigating
Hello,
Our monitoring has detected slow data integration. It appears to be linked to an Azure service issue on databases in Europe.
Our engineers are investigating how to work around and will inform you as soon as possible.
On behalf of SES-imagotag, the whole Cloud team would like to apologize for the incident and to thank you for your understanding. Please be assured that we are committed to continuous improvement and that we strive to provide the best experience as possible.
Posted Dec 01, 2023 - 04:09 UTC
This incident affected: Europe (VUSION Manager - Europe).