Intermittent Outage of Object Storage in Region LPG

Management Summary
During a scheduled maintenance of our Ceph storage cluster in the LPG region, but unrelated to the actual maintenance work, the Object Storage in the LPG region faced multiple short downtimes on 2026-08-29 between 14:23 and 15:05 CEST.

Timeline
On 2026-08-28 in the RMA region and on 2026-08-29 in the LPG region, we updated the Ceph software in our storage clusters to fix a security vulnerability. Having tested the update in our lab environment in accordance with our internal change management process, we did not expect any relevant impact of both the new version and the update process itself on our customers’ services.

On 2026-08-29 at 14:06 CEST, while the update in our LPG region was in progress, our engineers were alerted by the internal monitoring system about memory running low and a service restart in a pair of redundant systems backing the Object Storage. Since the restarts happened one after the other, no outage of the Object Storage was caused at that time.

Starting from 14:23 CEST, however, a number of further service failures on both systems occurred, causing a short downtime of the Object Storage where both of the systems faced a failure at the same time. As a first mitigation, we started to proactively restart the involved services in a controlled manner before they failed, while investigating the root cause of the issue as a whole.

At 14:59 CEST, we identified that one of Ceph’s “OSD” daemons was slow to respond to some requests, and concluded that this could be the root cause for the increasing memory consumption and service failures we saw with the Object Storage.

At 15:05 CEST, after establishing that this OSD daemon’s behavior was not caused by or otherwise interfering with the software update which was still in progress, we restarted the daemon, which immediately stabilized the Object Storage operation again. Subsequent monitoring of the storage cluster and the Object Storage in particular did not reveal any further issues.

Next Steps
In our investigation, we found no hints of any causal link between the update process and the unresponsive OSD daemon, which in turn impacted some of the services backing the Object Storage.

Given that a single failing process is something which can happen at any time and in any context, we will evaluate improvements to our internal monitoring, allowing us to pinpoint and resolve an underlying issue like this one more quickly in future.

We sincerely apologize for the inconvenience this issue may have caused you and your customers.

Investigation of the incident showed that one single storage daemon used to serve Object Storage data was misbehaving starting from 14:05 until it was restarted at 15:04. This restart immediately resolved the issue.

We will publish a more detailed incident report within the next few days.

Please accept our apologies for the inconvenience this issue may have caused you and your customers.

Operation of the Object Storage in the LPG region has been stable again since about 15:05 CEST.

We keep watching the state and will update this incident ticket if necessary.

Please accept our apologies for the inconvenience this issue may have caused you and your customers.

The Object Storage in our LPG region is intermittently unreachable starting 2026-08-29 at about 14:23 CEST. Whether or not this is related to the current software upgrade is subject of our ongoing investigation.

We keep watching the state and will update this incident ticket if necessary.

Please accept our apologies for the inconvenience this issue may have caused you and your customers.