Working Through a Very Rare Power Outage

Posted on: February 22, 2015

The University experienced a very rare power outage Tuesday morning, February 17, 2015. Fortunately, the primary and backup data centers and network operations centers, run and managed by Information Technology Services (ITS), all kept running. ITS has already taken steps to gather and analyze essential data from the event to understand and learn from what happened and take the necessary corrective actions. These findings will be shared with campus at a later date.

In the meantime, here are some preliminary topics of interest:

  • During the power outage, the primary University Data Center (UDC) and primary network operations center both automatically switched to generators as designed and no power loss within occurred. There was plenty of fuel for both generators. Thanks go out to the UDC team who maintain and test the generators multiple times each year. The 40+ units on campus taking advantage of UDC co-location services did not experience downtime, and the UDC team continues to exceed their service level commitments.
  • In the backup data center and network operations center, key equipment was powered by an Uninterruptable Power Supply (batteries). However, some email delivery was delayed due to a configuration issue in the back-up data center. This email configuration is being corrected to prevent future occurrences.
  • Special kudos also go out to the Networking and Building Security teams who worked tirelessly to restore Wi-Fi and Building Access services across the vast UT Austin campus. Its easy to forget that there are thousands of network and security devices that make Wi-Fi and security work. Networking, in one day, replaced 21 pieces of equipment (valued at $170K list), served 31 locations, and worked to get almost every building back on the air that day (an astonishing feat). Sadly, 90% of failed devices were past the recommended equipment life spans, and unfortunately those units had not invested in upgrades. We were lucky, since there are still many old network devices out there that could fail. One network device that failed, supporting three buildings, was installed in 1999! We encourage units to check their Network Building Report Cards to see how their network gear measures up to the minimum network standards endorsed by IT governance, and identify where investments are needed to mitigate future outages. Its a great planning tool.

ITS followed its emergency response and communication procedures as planned and practiced through the year. As soon as the widespread nature of the power outage become apparent, ITS senior staff jumped on a standing, off-site supported, conference bridge. Senior staff received updates as they happened and dispatched people to address issues as they developed. Continuous updates to the ITS Alerts page were posted, and a link was added to the UT Austin home page. This responsiveness wasn’t just luck, but reflects planning, practice, and hard work by all.

See Also: