Failure of Certain Pre-Signed Uploads to Object Storage in Region RMA

Minor incident Region RMA (Rümlang, ZH, Switzerland) Object Storage Service (RMA)
2026-11-19 16:25 CET · 1 hour, 25 minutes

Updates

Retroactive

On 2026-11-19 at 16:25 CET, we made a configuration change in our object storage in the RMA (Rümlang) region, causing stricter AWS SigV4 signature checks. After we received the first customer report about failing uploads, the change was reverted at 17:50 CET, reportedly resolving the issues.

The configuration change was intended to deny PUT requests which are not allowed by the specification and would also be forbidden by AWS S3. Further investigation turned out, however, that this configuration triggered a bug in our object storage’s underlying Ceph version: signature checking was too strict and also rejected some requests that complied with the specification, affecting a number of client-side S3 implementations.

Unfortunately, we had not been aware of this bug, and it was not detected by the test suite we had used while testing this change in our lab environment. A new, fixed Ceph version is already under testing, and we will use an updated test suite to prevent this issue from recurring.

We apologize to all affected customers for any inconvenience this may have caused.

November 20, 2026 · 16:34 CET

← Back