Microsoft says three-month-old bug behind global M365 outage

Key points

A lock contention bug hidden in Microsoft’s networking gateway servers for more than three months caused last Friday’s global Microsoft 365 outage.

Microsoft says three-month-old bug behind global M365 outage

Microsoft says three-month-old bug behind global M365 outage

Key points

  • A lock contention bug hidden in Microsoft’s networking gateway servers for more than three months caused last Friday’s global Microsoft 365 outage.
  • A Teams failover onto the primary gateway serving Exchange Online, combined with nearly 14 per cent of gateway capacity already out of service, overloaded the remaining servers.
  • Recovery was slow because restored servers were immediately swamped by waiting traffic and the latent bug meant normal rollback and feature flags could not quickly revert the change.




Microsoft says three-month-old bug behind global M365 outage








Microsoft has blamed a bug that sat undetected in its networking gateway servers for more than three months for last Friday’s lengthy global Microsoft 365 outage.

The defect, said to be lock contention inside an internal process, made servers unresponsive under certain load conditions, Charles Lamanna, Microsoft executive vice-president of Copilot, agents and platform, said in a post on social media.

Lock contention occurs when many tasks queue for the same shared resource, meaning work stalls while they wait.

The bug was introduced by a fix Microsoft wrote specifically for the gateway servers.

That fix began rolling out in July, and had been running worldwide for more than three months without problems.

“Neither the underlying defect nor the fix applies to broader Windows deployments,” Lamanna said.

Load testing missed the problem because Microsoft’s labs concentrated on weaker hardware with fewer cores, while the contention was worse on newer machines with more processor cores.

Contributing to the issue, about 15 hours before the outage, Teams failed over from a separate gateway system onto the primary one that also serves Exchange Online, adding load and changing traffic patterns.

At the same time, nearly 14 per cent of gateway capacity was out of service because of an unrelated issue the previous day.

Servers began failing during peak United States traffic and as traffic management steered users away from them, the remaining machines were overloaded in turn.

In less than two hours, more than 80 per cent of the gateway’s servers became unresponsive.

Isolation between the gateway’s thousands of capacity units and its regions should have contained the damage, but did not “due to a separate issue”, which Lamanna did not describe.

Teams, OneDrive, SharePoint and Office apps were able to fail out of the faulty gateway and recovered earlier, but Exchange Online could not.

Recovery dragged on because restored servers were immediately swamped by waiting traffic.

Microsoft’s automated recovery processes were not built for such a broad failure.

“Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change,” Lamanna said.

Engineers instead built an update returning the code to its previous state, and applied it to large blocks of servers at a time, in some cases more than half a region.

A final post-incident review would provide more detail later this week, Lamanna said.



About Author

What do you feel about this?

Subscribe To InfoSec Today News

You have successfully subscribed to the newsletter

There was an error while trying to send your request. Please try again.

World Wide Crypto will use the information you provide on this form to be in touch with you and to provide updates and marketing.