Hidden defect left over 80% of affected gateway servers unresponsive

Dubai: Microsoft has revealed that a software bug introduced in July was a major factor behind last week’s global Microsoft 365 outage, which disrupted Outlook, Teams and other services for nearly 24 hours.
In a preliminary report released after the November 12-13 disruption, the company said the defect had gone undetected for more than three months before increased network traffic and reduced server capacity triggered widespread failures.
Get updated faster and for FREE: Download the Gulf News app now - simply click here.
At the height of the incident, more than 80 per cent of Microsoft’s affected networking gateway servers became unresponsive in less than two hours.
The disruption affected access to Outlook and Exchange Online, delaying emails and interrupting services used by businesses and organisations worldwide, including in the UAE.
Microsoft identified three contributing factors: a software defect dating back to July, increased traffic following a Microsoft Teams network failover, and an unrelated problem that had already taken nearly 14 per cent of the affected gateway’s server capacity offline.
The July update had been deployed globally without apparent problems. However, under certain traffic conditions, the defect caused servers to become overwhelmed and stop responding.
About 15 hours before the outage, Microsoft Teams switched to networking infrastructure also used by Exchange Online, increasing the load on those systems.
With part of the server capacity already unavailable, the additional traffic triggered the previously undetected defect.
As servers stopped responding, Microsoft’s systems automatically redirected traffic to other servers. Those servers then became overloaded, causing failures to spread across the network.
Microsoft said its testing had failed to reproduce the conditions that eventually triggered the problem.
The defect was more likely to occur on newer, more powerful servers with additional processor cores, while the company’s testing largely used less powerful hardware.
The combination of increased traffic and reduced capacity exposed the problem during peak US usage hours.
Microsoft said restarting affected servers did not immediately restore services because they were overwhelmed again by requests that had accumulated during the disruption.
The age of the defect also complicated recovery. Since the problematic code had been deployed months earlier, engineers could not simply reverse a recent update using the company’s normal rollback procedures.
Teams, OneDrive and SharePoint were able to switch to alternative infrastructure and recover earlier, while Exchange Online remained affected for longer.
Microsoft eventually developed and deployed a fix, restoring services by November 13.
Microsoft said it has increased server capacity, improved monitoring and introduced additional safeguards to reduce the risk of similar cascading failures.
The company has also confirmed that the outage was a technical incident, not a cyberattack or an incident involving AI agents.
Its investigation is continuing, with a final post-incident report expected later this week.