[RESOLVED] 6/20/2019 12:24am EDT Tampa Franklin Power Incident
At 6/20/2019 12:24am EDT - An power incident has occurred in our Tampa Franklin datacenter. We are currently gathering a scope of the impact and working on remediation steps as quickly as possible.
As of 6/20/2019 1:25am EDT - The Tampa Franklin datacenter was operating on generator power in preparation for semi-annual UPS maintenance. At 12:24am EDT the generator failed unexpectedly prior to the commencement of maintenance activities. One of the three UPS systems in the datacenter experienced a failure during this time, resulting in any server and network devices single-homed through that UPS to have been rebooted. We are actively working on restoring network connectivity while troubleshooting is commencing on the UPS. Network engineers and UPS service technicians are onsite for both issues.
As of 6/20/2019 1:48am EDT - All networking devices have been restored at this time. We are continuing to troubleshoot the UPS and generator issues.
At 6/20/2019 2:06am EDT - We are continuing to confirm all servers are online and functioning. The affected UPS has been successfully placed into maintenance bypass. The onsite engineers are continuing to troubleshoot that device. Our generator technician has been dispatched to the facility.
At 6/20/2019 2:36am EDT - We are continuing to confirm all servers are online and functioning normally. The UPS remains in maintenance bypass mode while onsite engineers continue to troubleshoot. Our generator technician ETA to be onsite is 4:00am EDT.
At 6/20/2019 3:00am EDT - Current facility status update: The Tampa Franklin datacenter is operating on utility power and all core network is functioning normally. We are continuing to troubleshoot individual server issues. The generator is currently offline while UPS2 is in a maintenance bypass state. The generator technician has arrived onsite and is beginning to troubleshoot.
At 6/20/2019 3:45a EDT - The affected UPS & generator have been restored to service at this time. The root cause of this event is still under investigation. In the coming days, we will be scheduling additional testing to ensure the datacenter's resiliency.
At 6/20/2019 4:00a EDT - This is the final update for the Tampa Power Incident. All remaining individual servers that were impacted have been restored to service. If you are continuing to experience any issues, please contact our support team at 1-800-308-1522.