/cdn.vox-cdn.com/uploads/chorus_asset/file/23962442/acastro_STK067__03.jpg)
/cdn.vox-cdn.com/uploads/chorus_asset/file/24247717/lp_logo_3.0.jpg){kind=logo}
/cdn.vox-cdn.com/uploads/chorus_asset/file/24016885/STK093_Google_04.jpg)
/cdn.vox-cdn.com/uploads/chorus_asset/file/24808816/Starfield__The_Settled_Systems___Supra_Et_Ultra_____Starfield__The_Settled_Systems___Su)
[ad_1]
The Microsoft Azure DevOps outage in the South Brazil region, which lasted more than 10 hours, was caused by a typo in code that took out 17 production databases.
After apologizing to affected customers for the outage, Microsoft has now released a full post-mortem, detailing the investigation from when the outage was first observed at 12:10 UTC on May 24 to its treatment at 22:31 Details have been shared. UTC on the same day.
Eric Mattingly, Principal Software Engineering Manager, Microsoft shared Details of code base upgrades that will be part of Sprint 222. There was a hidden typo bug in the snapshot deletion task inside the pull request, which deleted Azure SQL Server instead of individual Azure SQL databases.
coding error
Mattingly explained: “When the job deleted Azure SQL Server, it also deleted all seventeen production databases for the scale unit,” confirming that no data was lost during the decommissioning process.
The outage was detected within 20 minutes, at which point the company’s on-call engineers got to work, although the root cause was identified at 16:04, nearly four hours after the outage began, according to the event log.
Microsoft blamed the fix time of more than ten hours on the fact that customers were unable to restore Azure SQL Server themselves, as well as backup redundancy complications and a complex set of issues with “(its) web servers.”
Having learned from its mistake, Microsoft has made no promises to roll out Azure Resource Manager locks for its key resources, in an effort to prevent accidental deletions in the future.
Despite a same-day rectification, customers in the area were left without access to some services for several hours, emphasizing how easy it is to get things wrong and to reduce reliance on single service providers. the importance of having backup plans for cloud storage and other off-premise infrastructure.
