Emergency OS Update Caused Volume Operation Failures

Management Summary
We have patched various hosts of our cloud platform in the LPG (Lupfig) region against two security vulnerabilities in two consecutive maintenance windows, one in the evening of Thursday, 2026-09-03, and one on Friday, 2026-09-04.

One of the vulnerabilities affected Ceph, where we updated the relevant hosts and prepared a required key rotation during the first maintenance window. The second maintenance window was scheduled to live-migrate all cloud servers in the LPG1 zone to other compute hosts in order to both finalize the Ceph update as well as patch/reboot the compute hosts while no cloud servers were running there.

Between the two maintenance steps, and undetected in our tests performed in the lab environment beforehand, attaching volumes to existing virtual servers was not possible, and trying to do so caused subsequent failures in setups relying on this operation (such as Kubernetes clusters using our CSI driver). Once triggered, the issues persisted for the cloud server and volume involved until manually remediated (in part possible by the customers themselves, in part only by cloudscale engineers) which happened progressively until full recovery at about 16:00 CEST.

Customers and cloud servers that did not try to attach a volume during this time window, which were by far the majority, did not suffer from any impact on their cloud services.

Timeline
In the course of Thursday, 2026-09-03, we announced an emergency maintenance window on the same day as well as a follow-up maintenance window on Friday, 2026-09-04, to deploy two security updates. While we expected and announced minor impact on our control panel and the API during the first maintenance window, no relevant impact was expected from the live-migration of cloud servers (a well-established standard procedure) during the second maintenance window.

On 2026-09-03 at 22:31 CEST, the first maintenance work had been concluded as planned, and no unexpected impact on our customers had been observed. However, as it turned out, attaching volumes to existing cloud servers no longer worked, and a small number of our customers’ servers hit this issue during the night or early in the morning.

On 2026-09-04 at 06:36 CEST, one of our engineers noticed the errors in our logs, and started investigating. After checking back with the engineering team, it was established that live-migrating the affected cloud servers (which was planned anyway for that day) should resolve the issue.

By 07:20 CEST, the first affected cloud servers we were aware of had been migrated to different compute hosts, and the error rate in our logs diminished. However, other cloud servers kept hitting the same issue. Over the next couple of hours, we proceeded with the (planned) live-migrations in batches, prioritizing cloud servers where we got aware that they were actually affected by this issue. By live-migrating all cloud servers in turns, the issue was progressively reduced, and volumes could again be attached to already migrated cloud servers without triggering the issue.

From 07:35 CEST, we were in constant contact with a number of customers, assisting them in remediating affected setups while further investigating the underlying problems and incorporating the customers’ observations and insights.

At 09:00 CEST, it was found that in connection with the overall issue, inconsistencies regarding the status of individual volumes could prevent those volumes from being attached or detached too, regardless if the target server was affected by the issue or not. During further remediation efforts, cloudscale engineers fixed such inconsistencies once reported by customers or detected by our internal tools.

At 10:45 CEST (with a clarification at 11:51 CEST), we posted an update on our status page informing affected customers that they could also resolve the issue for their servers themselves by stopping and starting affected servers once. This intervention would start a new underlying virtualization process for the cloud server in question, which would then consistently use the new, rotated Ceph key for storage operations regardless of pre-existing connections to our storage cluster - with the same effect as a live-migration to a different compute host done from our side. Moreover, power-cycling a cloud server fixed things where a prior, failed volume attach operation had “blocked” the corresponding disk slot of the server (a condition which was preserved across live-migrating the server).

By about 16:00 CEST, the last batch of cloud servers in the LPG1 zone had been live-migrated at least once (and power-cycled where required), which means that attaching volumes had become possible again for all cloud servers.

Technical Background
Throughout the incident, cloud servers and volumes did or did not suffer from issues depending on a number of preconditions in different combinations:

The original issue was that new or existing volumes could not be attached to existing cloud servers between the first maintenance window on 2026-09-03 (concluded at 22:31 CEST) and the server’s next live-migration (scheduled for the maintenance window the next day) or power-cycle. Based on the testing performed in our lab environment, cloudscale had not been aware of this limitation, and has been working on developing a different maintenance procedure that prevents this issue before applying the same Ceph update to our RMA region.

Most customers and cloud servers did not try to attach a volume during this time window, and were therefore not affected by any of these issues in the first place.

If a volume attach operation was attempted and failed (to our knowledge in most cases in an automated way using Kubernetes/CSI), both the volume and the cloud server might have carried an effect from it, causing issues further down the road.

If an attach operation had failed, the affected volume might have suffered inconsistencies about its status (such as “attached” etc.) between different (database) components of OpenStack and the control panel / API. As a consequence, attaching the volume to a server might have failed again even if the server in question was not or no longer affected by the original issue.

In an affected cloud server, after trying to attach a volume, the mechanism connecting virtual servers and virtual hard disks may have been blocked, which in turn prevented attaching volumes even once the original issue had been mitigated by live-migrating the server. In this case, a power-cycle was needed to enable attaching volumes again. In cases where a server was deleted in such a state, volume status inconsistencies may have spread to other volumes that had been attached to this server.

Before the live-migration of all cloud servers could be completed, part of the servers were still susceptible to the original issue. This means that throughout the incident, customers deploying or shifting workloads may still have triggered or worsened the impact on themselves while we had already been working to mitigate the issue.

As a secondary effect, as clients such as the CSI driver retried failed volume operations, there were spikes in API usage and an error code may have been returned for a number of requests.

Next Steps
When we tested the updates in our lab environment, we had continuously been running comprehensive, automated tests from a customer’s perspective to establish that the impact of the process on actual customer workloads and (API) requests is minimal. We also used our Acceptance Test Suite in the lab environment before and after the updates to ensure that all features were still working as expected.

What had been missing in our test setup, however, were tests regarding changes to volume attachments on pre-existing cloud servers. For the upcoming updates in our RMA region, we have therefore been working on developing a different maintenance procedure that prevents similar issues.

We sincerely apologize for the inconvenience this issue may have caused you and your customers.

Volume update operations (such as attaching and detaching) for all cloud servers in zone LPG1 are working normally again.

We will publish a detailed incident report next week.

Please accept our apologies for the inconvenience this issue may have caused you and your customers.

Please note: Today’s failures that occurred during volume update operations were directly linked to yesterday’s and today’s scheduled emergency operating system updates, which were not expected to impact already running cloud servers. Therefore, we will only go ahead with the scheduled emergency operating system update in zone RMA1 next Monday and Tuesday, if our engineering team successfully tested a different maintenance procedure that prevents similar issues by then.

Update for clarification:

In order to mitigate failing volume attachment operations, power off and on affected servers instead of a mere reboot (so a new virtualization process is started on the underlying physical compute host).

Our engineers are continuing to work on fully resolving this issue. We will update this ticket as new information becomes available.

The underlying issue of failed volume attachments resulting from yesterday’s emergency OS update is complex and will take some time to fully resolve.

We’re in the progress of performing live migrations of affected servers to restore volume attachment functionality, but this will take significant time.

Customers currently affected by volume attachment issues can reboot their systems themselves, which should resolve the issue for that system.

We were able to reduce the number of servers affected; however, some servers are still affected by failing volume attachment operations.

Our engineers are investigating the issue and are working to fully restore our services. We will keep you posted.

After yesterday’s “Emergency Operating System Update in Region LPG” some volume update operations such as volume attachments were failing for a selected amount of customer systems.

The affected servers were live migrated to new compute hosts this morning at approximately 07:15 to remedy the issue.

Our engineers are monitoring the situation. We will keep you posted.