The time of the outage coincided with a planned maintenance window for a specific customer (on dedicated hardware, but shared fabric), we were security patching their dedicated hardware. Upon reboot of one of their blades it caused the platform VXLAN VTEPS to reconfigure across all hosts in the VMware Data Centre.
Upon investigation with the vendor, it was caused by a specific version upgrade path which has caused this to sometimes happen.
Only services that use VXLAN VTEPS were affected. Mainly Edge > VM traffic.
We are progressing with a platform level upgrade project to migrate from VXLAN > GENEVE which is expected to be completed Q1 2024.
All maintenance will now be scheduled out of core working hours until this migration is complete, to avoid the possibility of a repeat outage.
Posted Nov 07, 2023 - 16:22 GMT
Investigating
We are currently investigating a momentary outage with some virtual machines hosted within the amatis Reading myAmatisCloud environment. Service has been restored, we are investigating the root cause
Posted Nov 02, 2023 - 10:07 GMT
This incident affected: Cloud Services (Virtual Datacentre (VDC)).