We are experiencing elevated error rates affecting our ICSC and ICMC

Dear customers,

as part of our process to continuously improve our systems and processes, we have conducted a root cause analysis which you can find below.

We appreciate your patience during this disruption and apologize for any inconvenience.

Root Cause Analysis

During a planned maintenance window, some external IP addresses in on of our our Frankfurt data center became temporarily unreachable impacting our global database responsible for bucket and IAM data. Access to all of our data center infrastructure is ensured by redundancy of the internet access and an automatic failover mechanism. This failover mechanism had silently been disabled by an earlier misconfiguration. As a result, when the primary network gateway was taken offline for maintenance, traffic was not redirected to the backup as intended. The issue has been resolved and failover has been fixed and was rolled out to all sites. We have also introduced additional safeguards to catch similar misconfigurations earlier.

We see signs of improvement. Customers may see bucket data, users and iam data again. We are still working on the root cause and will monitor the situation closely.

Dear customers,

the issue has been resolved. Access to the service has been fully restored, and all operations are functioning normally.

We are now continuing to monitor the system closely to ensure continued stability.

We sincerely appreciate your patience during this disruption and apologize for any inconvenience caused.

Dear customers,

we are currently experiencing elevated error rates affecting our ICSC and ICMC. Customers can still login to the consoles, but may be unable to access their data. The S3 API is still functional.

This is a consequence of an unforeseen issue during re-configuration of one of our components to improve reliability and durability in our eu-central-2 region.

We sincerely apologize for the disruption and appreciate your patience while we resolve the issue.