Control Plane Outage in RMA1

Major incident General Cloud Control Panel & API
2024-03-13 12:42 CEST · 7 hours

Updates

Resolved

The control plane is fully functional again. We are still closely monitoring its state, but we don’t expect any further issues at this point.

March 13, 2024 · 19:39 CEST
Monitoring

The recovery procedure was completed successfully.

We keep watching the state and will update this status ticket if necessary.

Please accept our apologies for the inconvenience this issue may have caused you and your customers.

March 13, 2024 · 17:47 CEST
Update

We are starting with the recovery procedure.

The recovery procedure will result in a downtime of the control plane of approximately 10 minutes.

March 13, 2024 · 17:13 CEST
Update

Our engineers were able to further pinpoint the issue while testing the initial recovery procedure in our lab environment. We are now preparing the final recovery procedure and will provide another update before applying it to production.

March 13, 2024 · 16:55 CEST
Update

Our engineers have come up with a recovery procedure and are currently testing it in our lab environment. The performance of our Cloud Control Panel and API is still degraded.

The recovery procedure will result in a downtime of the control plane of approximately 10 minutes. We expect to start recovery in half an hour and will provide another update before doing so.

March 13, 2024 · 16:07 CEST
Update

Our engineers were able to restore control plane services. The performance of our Cloud Control Panel and API is still degraded. Already running virtual servers are not affected by this issue.

We will continue investigating the issue and are working to fully restore our services.

March 13, 2024 · 14:18 CEST
Issue

During the scheduled upgrade of a control host, we started experiencing issues with our database cluster in the RMA region.

Access to our Cloud Control Panel and API is currently not possible. Already running virtual servers are not affected by this issue.

Our engineers are investigating the issue and are working to fully restore our services. We will keep you posted.

March 13, 2024 · 12:42 CEST

← Back