Connectivity Issues

Minor incident Region RMA (Rümlang, ZH, Switzerland) Network Infrastructure (RMA)
2019-11-20 16:40 CEST · 5 hours, 57 minutes

Updates

Investigating

Unfortunately, the workaround provided by the vendor has only had a short-term effect. CPU utilization on the network devices is back at the (high) level it was before. We are continuing to investigate what is causing this and are also looking into other options, such as replacing the vendor or moving the routing to different equipment.

November 20, 2019 · 22:37 CEST
Monitoring

We have deployed the workaround to production without further impact to customer traffic. CPU utilization has decreased significantly and is continuing to decrease as we monitor the situation.

We will continue investigating this issue together with the vendor and keep you up to date on this status page.

Please accept our sincere apologies and rest assured that we will continue to treat this case with the utmost priority.

November 20, 2019 · 17:50 CEST
Update

We have received another potential workaround from the vendor that has been tested in their lab. Given the situation we are in right now, we have decided to deploy the workaround in an emergency maintenance window immediately.

In case we encounter any unexpected issues we should be able to roll back the changes in a matter of minutes.

November 20, 2019 · 17:21 CEST
Update

We were able to establish an emergency call with the vendor and are now discussing options. Unfortunately, further packet loss is expected until we have a workaround in place.

November 20, 2019 · 17:05 CEST
Issue

We are currently facing partial packet loss again and are trying to get the vendor on another emergency call. We will keep you posted.

November 20, 2019 · 16:40 CEST

← Back