[RESOLVED] May 29, 2019 8:15 AM MST - Denver Datacenter is on generator power
At 5:10 AM MST the utility power feeds went out in the Denver Datacenter due to severe weather conditions. The Power feeds fluctuated repeatedly all morning with our generators properly kicking in and providing power to our datacenter loads.
At 7:47 AM Time MST, power to the datacenter went offline for an unknown reason at this point. The generators are all functioning properly, but there appears to be an issue with the distribution to the datacenter. The Hostway|Hosting engineering teams have all mobilized and are working to identify and rectify the problem. The Hostway|Hosting staff is on standby to start working on customer devices once power is restored. We apologize for this issue, and are doing everything possible to restore service as quickly as possible.
At 8:15 AM MST The power to the datacenter loads were restored from our Generators at 8:15 AM MST. However, the utility power has still not yet been restored.
The updated reason for the failure is that we experienced a Programmable Logic Controller (PLC) failure that affected our C feed logic into the data center. A and B phase are active while C has been removed from the mix to eliminate the issue temporarily.
Our engineering teams are investigating this now. We will be on the generator until the utility power is fixed. We have 4000 gallons of diesel on premises with several thousand gallons on the way as well which should last until we rectify the problem. Most customer environments should be up by now. Support personnel continue to work on customer environments impacted by this issue to get them fully restored.