Severe weather takes down Microsoft's Azure cloud services in Texas

Microsoft Corp.’s Azure public cloud has been bent out of shape by a real cloud, as the company blamed “severe weather” for an outage that took several services offline for users in the U.S. on Tuesday.

Microsoft said its South Central U.S. data center located in Texas was hit by a thunder and lightning storm that caused its cooling systems to fail, forcing the company to switch off many of its servers and systems to prevent further damage.

“A severe weather event, including lightning strikes, occurred near one of the South Central US data centers,” the company said in a statement on the Azure Status page. “This resulted in a power voltage increase that impacted cooling systems. Automated datacenter procedures to ensure data and hardware integrity went into effect and critical hardware entered a structured power down process.”

The cooling system is a vital component of modern data centers as it’s necessary to dissipate the intense heat generated by thousands of servers stacked tightly together in an enclosed area. If it fails, quite simply, everything will melt down.

Hence, companies such as Microsoft have procedures in place to automatically shut down their data center machinery should the temperature rise beyond a safe level. It’s an important safeguard for Microsoft’s data center investments, but also a big inconvenience for its cloud customers.

The severe weather Microsoft mentioned likely has something to do with Hurricane Gordon, a Category 1 storm that’s presently lurking just off the coast of Texas.

Microsoft said the outage has affected many of its Azure cloud services, including the Visual Studio Teams service. Other services that went offline include Azure Active Directory identity management service and Office 365, the cloud-based productivity suite.

“Customers with organizations outside of the South Central US region may also be experiencing impact with their CI/CD workflows, dashboards due to some internal infrastructure dependencies,” the Visual Studio Team Services group added.

Experts said the incident is a reminder for organizations using cloud services that only a fool would rely on a single provider when it comes to running critical workloads in the cloud.

“Today’s incident at Azure was another clear reminder for the need for organizations to build in their own redundancy rather than rely on a single vendor,” said Pete Banham, a cyberresilience expert at Mimecast Ltd.

Still, the event could also serve as an important lesson for Microsoft as it looks to avoid future incidents of this kind, said Holger Mueller, principal analyst and vice president of Constellation Research Inc.

“This event is a good reminder of how hard it is, even for IaaS vendors on the scale of Microsoft, to keep their data centers up and running,” Mueller said. “Lightning, flooding, wind, snow and rain can all affect data center availability. So the key question is, what did Microsoft learn from this and how can it avoid similar outages going forward? It’s an important lesson for the company as it looks to harden its cloud infrastructure.”

In an update, Microsoft said it’s working to get all of the affected services back online, though that work is apparently still ongoing at the time of writing.

“Engineers have successfully restored power to the datacenter. Additionally, engineers have recovered a majority of the impacted network devices,” Microsoft said. “While some services are starting to see signs of recovery, mitigation efforts are still ongoing.”

Image: FelixMittermeier/Pixabay