[RESOLVED] September 04, 2018 03:29 AM CDT - Hostway Azure Environment
At approximately 04:29 AM CDT, the Microsoft Azure environment experienced degraded connectivity issues for a subset of customers in the South Central U.S. Microsoft's Engineering Team are engaged and are working to determine the root cause. This also impacts the Azure Resource Manager as such customers may not be able to access their Azure Control Panel. Please see https://azure.microsoft.com/en-us/status/ for updates.
10:35 AM CDT - Microsoft is still working on remediating the event. We will post updates as we receive additional information.
UPDATE AS OF 6:00 AM CDT: Engineers have restored storage availability for the majority of impacted services, and most customers should now be seeing signs of recovery
CUSTOMER IMPACT: Starting at 09:29 UTC on 04 Sep 2018, customers with resources in South Central US may experience difficulties connecting to resources hosted in this region. A complete list of impacted services can be found below.
PRELIMINARY ROOT CAUSE: A severe weather event, including lightning strikes, occurred near one of the South Central US datacenters. This resulted in a power voltage increase that impacted cooling systems. Automated datacenter procedures to ensure data and hardware integrity went into effect and critical hardware entered a structured power down process.
ENGINEERING STATUS: Engineers have restored access to storage resources for the majority of services, and customers should be seeing signs of recovery. Engineers are continuing to work on any residual storage impact to fully mitigate this issue. The current mitigation workflow is outlined below:
1) Restore power to the South Central US datacenter (COMPLETED)
2) Recover software load balancers for Azure Storage scale units in South Central US (COMPLETED)
3) Recover impacted Azure Storage scale units in South Central US. (Mostly complete)
4) Recover the remaining Storage-dependent services in South Central US (Mostly complete)