The switches were successfully upgraded to a fixed version of software and have remained stable since.
Posted Dec 13, 2022 - 08:25 GMT
Update
The root cause of this outage was due to a software defect on one of our upstream Cloud switches. This software defect caused a loss of control-plane traffic between 21:08-21:20 BST before the internal diagnostics on the switch initiated a reload of the device. At the point of reload, customer traffic was re-routed to the secondary switch and services began to restore. Unfortunately, a knock on effect of the control-plane traffic issue was that some Cloud services were affected beyond this time. We have been liaising with several vendors that make up the individual components of the amatis Cloud platforms to ascertain the reason for the traffic failure and this is taking some time to gather all the data together.
We are working closely with these vendors to see if there are any improvements in design that can be implemented to avoid a corner-case scenario like we experienced.
In due course we will be issuing a planned maintenance window to patch against the software defect.
Posted Oct 18, 2022 - 15:10 BST
Monitoring
We have identified a switch that reloaded unexpectedly. We are collecting crashlogs and will raise to the vendor. All services are operating as expected, if you are experiencing issues please contact support through the normal channels. Apologies for any inconvenience caused.
Posted Sep 30, 2022 - 21:59 BST
Investigating
We are investigating alarms on the my amatis Cloud platform in Reading. An update will be posted shortly