Microsoft outage: A cascading failure that started with an undetected bug

if ( !emtpy($headline_subheadline ) ) : ?>
The initial cause was an update in July that somehow didn’t cause a problem until last week.
endif; ?>
Credit: IB Photography – shutterstock.

[…Keep reading]

Microsoft outage: A cascading failure that started with an undetected bug

Microsoft outage: A cascading failure that started with an undetected bug

if ( !emtpy($headline_subheadline ) ) : ?>

The initial cause was an update in July that somehow didn’t cause a problem until last week.
endif; ?>

Credit: IB Photography – shutterstock.com

Following a roughly day-long global outage of many Microsoft services last week, Microsoft on Monday issued a preliminary post-incident report that identified multiple causes for the issues, including the admission that the environment used to test the underlying problematic software update did not match the actual infrastructure it would run on.

The report, a summary of which was posted on social media by Microsoft EVP Charles Lamanna, identified some of the factors that cascaded to cause the outage. 

“A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions,” the report said, adding, “this defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers.” Its deployment began in July, and, the report said, “for more than three months, the change was deployed worldwide without impact.”

Microsoft admitted that, although it did test the change before deployment, its test environment didn’t match reality.

“Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs,” it said. “The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores.”

A surge in traffic caused by unrelated issues also helped trigger the problem and delay it from being fixed, the company explained.

“Within Microsoft 365, we maintain two independent and heterogenous network gateway systems designed for isolation and high availability,” Microsoft said. “This allows many Microsoft 365 services to fail to/from them during outages or disruptions. Approximately fifteen hours before the outage, Teams failed over from the alternative networking gateway to the primary gateway that also supports Exchange Online. This increased the load on the networking gateway and changed its traffic patterns.”

To make things worse, the report said, “nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day.” Microsoft did not provide details of that unrelated issue. 

The combination of those elements led to the outage.

Cascading failure

“The increased load and the reduced server capacity caused the network gateway to encounter the latent bug from July,” Microsoft said. “This caused requests to build up until servers ultimately stopped responding.” And although, when servers lose connectivity, traffic management systems automatically route users to other healthy endpoints, the redirected traffic made the problem more likely to occur on the remaining functional endpoints.

“This caused a cascading failure from server to server as the same amount of load was spread across fewer servers,” the report said. “In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive.”

That cascading series of problems worked against Microsoft as it tried to resolve the issue, because as servers were brought back online, the deluge of waiting traffic triggered the bug and they became unresponsive again.

“Our normally automated recovery processes were not designed to restore service from this broad unresponsive state,” Microsoft said. “Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change.”

Explanations still needed

Consultants said that Microsoft overall did an admirable job of trying to get systems back into service, but added that the vendor needs to further explain why its testing environment did not match reality.

“I believe this is a risk inherent in the business model that can’t be mitigated with some clever workaround,” said David Shipley, CEO of Beauceron Security.

“Microsoft literally did everything I could ever think of, and it all failed in a spectacularly edge case way. I didn’t even have ‘Microsoft gets burned by Microsoft SKU complexity’ on my bingo card, but it makes a karmic kind of sense,” he said, stressing that the most concerning part of the report is that it gives customers no concrete reason to believe that a similar cascading series of errors might not happen at Microsoft next week.

“What’s terrifying about all of this is how easily it could happen again,” Shipley said. “What’s amazing is it took this long for this edge case to happen. This is the first time I can think of where bigger, better servers were part of the root cause of a bug like this.”

‘Latent defects become systemic failures’

Mike Wilkes, enterprise CISO at Aikido Security, agreed.

“The irony is that newer, more powerful hardware actually amplified the failure, a reminder that scalability and resilience are not synonymous,” he said. “This outage is a textbook example of how latent defects become systemic failures when engineering assumptions collide with production reality. Microsoft essentially admits that its testing environment failed to represent the hardware, concurrency and traffic characteristics of its own production infrastructure.”

Wilkes said that part of the report was delightfully ironic. “The most embarrassing revelation is that Microsoft’s status page was itself disrupted by the outage,” he noted. “How pathetic must your understanding of failure modes be when the very system intended to communicate that your infrastructure is broken depends on that same infrastructure working? That’s not merely a technical oversight. It’s an architectural failure of imagination and failure of applying experience that should have been gained 30+ years ago.”

Independent technology consultant Steven Eric Fisher said he, too, saw the testing disconnect as the biggest red flag.

“If your testing doesn’t adequately reproduce the hardware characteristics, concurrency, traffic patterns, and operational conditions of production, you have inherently accepted the risk that significant defects will escape detection,” Fisher said. “The particularly troubling aspect here is that Microsoft’s production hardware was more capable than its testing hardware, yet that additional capability made the defect worse.”

Frank Dickson, principal analyst at Dickson Research, added that another problem with the recovery effort was that automation quietly handled many isolated server failures, in effect delaying the full response because it hid the problem for so long. 

“There was also smoke long before the fire,” he pointed out. “Teams traffic shifted onto the shared gateway at 1:07 AM UTC, and isolated server failures cropped up in some regions over the next 15 hours. Automation handled each one quietly. The high-priority alert did not fire until 4:20 PM UTC. Automation that keeps putting out small fires can hide the fact that the building is getting hotter.”

Vendors and ProvidersProductivity SoftwareMicrosoft 365Office Suites

About Author

What do you feel about this?

Subscribe To InfoSec Today News

You have successfully subscribed to the newsletter

There was an error while trying to send your request. Please try again.

World Wide Crypto will use the information you provide on this form to be in touch with you and to provide updates and marketing.