And at 40$ per employee per month, with all those employees now without access to the company (auth is not working so internal apps are going to be dead too), 50% of that for one month covers less than 25 minutes of downtime if your employees work 38 hours per week for 5000$ on average (and I do hope they make at least 2x more, since this is before tax).
But no worries, getting the entire month of Office 365 for free justifies about 50 minutes of company downtime.
Besides: why worry? Claude code is down too, so your now cheaper techs can't do anything either.
The loss for a random due to MS software at this point is about 1400$ per seat (but probably at least double that) and rising.
Which is why every manager should now realize that their microsoft contract can never work as a valid disaster recovery option.
I tried to bring this out of our management staff when they were going all in on a SaaS that's 99% AWS backed. The SaaS was basically as integral is email is to many businesses to their line of work, and it was required to complete the business's b2b contracts which often had a turn around date. If the SaaS goes down, they're basically sitting idle and hitting contractual deadlines.
The response was basically, "we have a uptime garuntee and will pay out for downtime". But when it's clear _all_ of their clients have the same SaaS contract garuntee and a serious enough disruption would see _every_ client needing payment back, well, it's not much of a warranty is it.
Now I'm sure the SaaS has the redundancy in their system, but AI is the next deep integration that's going to hallow out a bunch of companies because some third party is just going to have a catastrophic loss somewhere down the line.
I’m extremely curious what actual veterans could even come up with to explain the situation as it currently appears.
I’ve seen the chats about cascading failures and it’s impossible to not expect AI is involved somehow (likely brain drain and what not), but what are the options?
It sure looks like “fired everyone who knew what they were doing” but I feel like at this point you’re breaking open the check book and calling them, and even if they did that I’m assuming this shouldn’t be a “normal” event?
Speculation is a fun game, but from experience, large outages tend to hinge on complex technical details not visible to the general public and I've never once seen even veterans of consummate skill and knowledge correctly guess what happened in the headline-making outages I've been privy to.
I've seen technical discussions on this stuff. Usually it's because you have a distributed system that's build up incrementally.
You bring one system online, then the next, the next; you load balance it carefully, etc.
But in these catastrophic scenarios, those involved (AI particularly), want to bring everything up all at once and that itself causes some DDoS like attacks on internal and external systems. And that's assuming you solved whatever "unhappy" system took it down in the first place.
The thing I've see with using AI for the past 6 month is it's really too dumb about how long things take. Humans have a normal sense of how long a process will take, whether a race condition can exist (eg, 1ns vs 10s), and other time based details. Whether a log is produced before something happens or after, etc.
I've been considering adding time stamps to mine plugins to track the time, but it's sorta like you need to track it both at the start of the request, at the end of it, at the bewginning of the generation, at the end of the generation.
Anyway, AI has to be shit to use in these scenarios because it can't tell that a system takes 5 minutes to bring up, and if you bring up 10 of the same systems at the same time, they have zero capabilities against race conditoins on shared resources etc.
Bringing up clusters that are getting smashed the moment they’re online because of the backed up traffic?.. is it just exchange or AD failures behind it?.. could be quite complex with the traffic waiting
That would be the status updates about isolating traffic. Still can be hard to do when half of the internet is retrying their failed requests on your servers.
I can think of several scenarios that could explain the outage:
A bad antivirus pattern file can cause enormous CPU spikes. E.g.: a very inefficient regular expression with too much backtracking. This tends to get pushed out every few hours to every server, irrespective of role. Hence, "rolling back" the servers to a "known good image" doesn't help, because within a few minutes they get the pattern update anyway and die with CPU overload. (IIRC this type of issue caused a Cloudflare global outage recently, and is similar to what happened with Crowdstrike.)
Similarly, someone here mentioned a very recent OpenSSL security update that also caused enormously high CPU load. These critical patches have been rolled out very quickly with minimal testing because without it, systems are very vulnerable to nation-states utilising AIs to rapidly exploit new vulnerabilities. Microsoft (probably) doesn't use OpenSSL much, but if they do, it would be everywhere because its use in layer-7 load balancers tends to be "pervasive" and hence a sweeping update can absolutely trash performance of everything in the data centre. (However, this category of issue would have been fixed by rolling back the server images)
It could also have been a simple underlying initial cause, but then the "backlog" of emails that accumulated throghout the day would overwhelm systems that come back online, especially if they can't start them all at once... which they've said they're not doing. So, you start 10% of your capacity, which tries to process 10x the normal email volume => splat.
Most other explanations wouldn't account for the global outage.
Active Directory, Sharepoint alongside Office, anyone involved in game development related to PC and consoles (all devkits use VS plugins) and Valve has failed to bring native games into SteamDeck, existing .NET enterprise applications, Typescript, Github, VSCode, node,...
It's not up to the IT department. Network effects means if you don't use MS you cannot interperate with other MS-based companies as well. So you need at least some MS integration and they are happy to sell you the rest of the stack bundled with it.
There is also another type of network effect: hiring candidates or contractors with knowledge of the MS stack is much easier than finding someone who can set up an alternative.
The decisions on which stack to use made by business people despite the objections from IT. I’ve seen this pattern so many times across various organizations
Looking forward to the postmortem on this one and how much AI contributed to the problem and troubleshooting.
I've been saying AI hasn't been tested properly. All the showcases I see aren't really pushing it far enough and they're always things you could in theory do by searching answers online. The real value is in all the problems that cannot be solved by online search.
I know this is only tenuously related, but after a week of them making changes (starting with DKIM) and blocking third party mailservers (and this week I got my first "your IP is blocked" but when you use their request to remove they say it isn't), I hope it goes down long enough to make more people migrate away
Really hoping it's a monumental fuckup that lasts through the weekend well into the next week.
The corrupted executives need to learn their lesson on consequences of their actions. Laying off the core of your system?. Say goodbye to triple-9 SLA.
Even if it lasted a month, that is not the lesson they would learn. Cut costs and grow revenue are as instinctive to them as breathing. There will be no apologies for that.
One can only hope for a damage to their reputation so significant, it encourages other companies to swtch providers, and the devs to stand up and develop competing solutions.
Since Musk fired most of Twitter engineers, it seems like the prevailing thought has been that you can just trim your organization of the unnecessary workers, only accelerated by AI development. I also hope that this proves out to be disastrous for MS and the thinking goes back to hiring top people and making sure they're happy.
> The corrupted executives need to learn their lesson on consequences of their actions. Laying off the core of your system?. Say goodbye to triple-9 SLA.
Do you think they actually care about their "triple-9 SLA"? Only in as much as they could use it as a marketing bullet point to sign contracts to increase revenues/profits.
> Engineering teams are cycling affected infrastructure through traffic
drain, restart, remediation, and staged return-to-service while monitoring
closely.
I heard a story, which you can't attribute to me of course, I just heard it -- we had a thing (literally yesterday) where a person from M$ came selling us some AI snake oil (we are very much an azure pilled org, but not super AI pilled).
The meeting was obviously in teems, because they sold us on that already and most of the presentation the screen sharing didn't work for half the people, which generated a continued stream of office-grade jokes for the whole duration of the presentation (30 minutes or so).
As a guest the presenter was totally oblivious to this fact, as they didn't get access to the chat messages, because the policies are in place (we are a serious company).
I guess we better get used to this new normal and pray your travel day or important appointment is not going to be on an outage day, whether that's MS, Crowdstrike or else.
hey Claude fix my production issue in production make no mistakes
investigating..
goofing..
finagling..
hey claude what the hell you deleted all my configs permanently!!
metastasizing.. You’re absolutely right. I went above and beyond what you had asked and started completely fresh. Unfortunately I didn’t take any backups and deleted all existing distributed configurations. Oopsie!
FYI, if your business uses Microsoft 365, you are eligible to having 50% of your monthly fee waived off as per MS online services SLA.
It's been 18 hours since the start of the outage. If service isn't restored in the next 18 hours (36hrs+ outage), you can get the entire fee waived.
Microsoft gives a 99.9% monthly uptime guarantee, which means maximum allowed downtime is 7hrs.
And at 40$ per employee per month, with all those employees now without access to the company (auth is not working so internal apps are going to be dead too), 50% of that for one month covers less than 25 minutes of downtime if your employees work 38 hours per week for 5000$ on average (and I do hope they make at least 2x more, since this is before tax).
But no worries, getting the entire month of Office 365 for free justifies about 50 minutes of company downtime.
Besides: why worry? Claude code is down too, so your now cheaper techs can't do anything either.
The loss for a random due to MS software at this point is about 1400$ per seat (but probably at least double that) and rising.
Which is why every manager should now realize that their microsoft contract can never work as a valid disaster recovery option.
I tried to bring this out of our management staff when they were going all in on a SaaS that's 99% AWS backed. The SaaS was basically as integral is email is to many businesses to their line of work, and it was required to complete the business's b2b contracts which often had a turn around date. If the SaaS goes down, they're basically sitting idle and hitting contractual deadlines.
The response was basically, "we have a uptime garuntee and will pay out for downtime". But when it's clear _all_ of their clients have the same SaaS contract garuntee and a serious enough disruption would see _every_ client needing payment back, well, it's not much of a warranty is it.
Now I'm sure the SaaS has the redundancy in their system, but AI is the next deep integration that's going to hallow out a bunch of companies because some third party is just going to have a catastrophic loss somewhere down the line.
I’m extremely curious what actual veterans could even come up with to explain the situation as it currently appears.
I’ve seen the chats about cascading failures and it’s impossible to not expect AI is involved somehow (likely brain drain and what not), but what are the options?
It sure looks like “fired everyone who knew what they were doing” but I feel like at this point you’re breaking open the check book and calling them, and even if they did that I’m assuming this shouldn’t be a “normal” event?
Speculation is a fun game, but from experience, large outages tend to hinge on complex technical details not visible to the general public and I've never once seen even veterans of consummate skill and knowledge correctly guess what happened in the headline-making outages I've been privy to.
all happy families are alike each unhappy family is unhappy in its own way
I've seen technical discussions on this stuff. Usually it's because you have a distributed system that's build up incrementally.
You bring one system online, then the next, the next; you load balance it carefully, etc.
But in these catastrophic scenarios, those involved (AI particularly), want to bring everything up all at once and that itself causes some DDoS like attacks on internal and external systems. And that's assuming you solved whatever "unhappy" system took it down in the first place.
The thing I've see with using AI for the past 6 month is it's really too dumb about how long things take. Humans have a normal sense of how long a process will take, whether a race condition can exist (eg, 1ns vs 10s), and other time based details. Whether a log is produced before something happens or after, etc.
I've been considering adding time stamps to mine plugins to track the time, but it's sorta like you need to track it both at the start of the request, at the end of it, at the bewginning of the generation, at the end of the generation.
Anyway, AI has to be shit to use in these scenarios because it can't tell that a system takes 5 minutes to bring up, and if you bring up 10 of the same systems at the same time, they have zero capabilities against race conditoins on shared resources etc.
Bringing up clusters that are getting smashed the moment they’re online because of the backed up traffic?.. is it just exchange or AD failures behind it?.. could be quite complex with the traffic waiting
I guess you must do that electric utility does to avoid blackout - drop a lot of consumers and add them slowly back as things stabilize.
That would be the status updates about isolating traffic. Still can be hard to do when half of the internet is retrying their failed requests on your servers.
I can think of several scenarios that could explain the outage:
A bad antivirus pattern file can cause enormous CPU spikes. E.g.: a very inefficient regular expression with too much backtracking. This tends to get pushed out every few hours to every server, irrespective of role. Hence, "rolling back" the servers to a "known good image" doesn't help, because within a few minutes they get the pattern update anyway and die with CPU overload. (IIRC this type of issue caused a Cloudflare global outage recently, and is similar to what happened with Crowdstrike.)
Similarly, someone here mentioned a very recent OpenSSL security update that also caused enormously high CPU load. These critical patches have been rolled out very quickly with minimal testing because without it, systems are very vulnerable to nation-states utilising AIs to rapidly exploit new vulnerabilities. Microsoft (probably) doesn't use OpenSSL much, but if they do, it would be everywhere because its use in layer-7 load balancers tends to be "pervasive" and hence a sweeping update can absolutely trash performance of everything in the data centre. (However, this category of issue would have been fixed by rolling back the server images)
It could also have been a simple underlying initial cause, but then the "backlog" of emails that accumulated throghout the day would overwhelm systems that come back online, especially if they can't start them all at once... which they've said they're not doing. So, you start 10% of your capacity, which tries to process 10x the normal email volume => splat.
Most other explanations wouldn't account for the global outage.
I am extremely curious how any IT department in 2026, still justifies using Microsoft. There is a difference between being cheap and being reckless.
Active Directory, Sharepoint alongside Office, anyone involved in game development related to PC and consoles (all devkits use VS plugins) and Valve has failed to bring native games into SteamDeck, existing .NET enterprise applications, Typescript, Github, VSCode, node,...
> Active Directory
This is big for medium/large fleets: fairly good central authentication, as well as being able to set policies for machine management (GPO).
I've also heard good things about PowerShell.
>> I've also heard good things about PowerShell.
That is despite Microsoft ...you do know they tried to fire the guy who created it?
It's not up to the IT department. Network effects means if you don't use MS you cannot interperate with other MS-based companies as well. So you need at least some MS integration and they are happy to sell you the rest of the stack bundled with it.
There is also another type of network effect: hiring candidates or contractors with knowledge of the MS stack is much easier than finding someone who can set up an alternative.
The decisions on which stack to use made by business people despite the objections from IT. I’ve seen this pattern so many times across various organizations
What's the alternative for large multi-national organisations?
Probably different open source projects + doing things in-house. Which--I would assume--would likely cause much more frequent and longer outages.
Cloud email as reached the utility status like electricity, it's not economically justified to run your own unless you have a very specific use case.
That's right. I don't think most people should
Most of the US government is on M365.
And the only reason they achieved a FedRAMP ATO is because most of the government was already on it. The platform is a piece of shit.
>It sure looks like “fired everyone who knew what they were doing"
This has two assumptions:
1. people who knew what they were doing were all fired. maybe they just lost touch with the system reality and outsourced too much to the shaitan;
2. if the people were to be called back they can still understand what the system become after shaitan was let loose in it
Looking forward to the postmortem on this one and how much AI contributed to the problem and troubleshooting.
I've been saying AI hasn't been tested properly. All the showcases I see aren't really pushing it far enough and they're always things you could in theory do by searching answers online. The real value is in all the problems that cannot be solved by online search.
I was thinking the same.
Btw, this is why no one should think AI-ops is a thing yet. All the promises of powerful, hands-off AI operations, preventive fixes, etc. are just BS.
Can't do AI-ops if you're the AI-ops provider that's got the outage :)
This is one for the history books. The status updates seem like AI nonsense, it doesn’t seem like they’ve even root-caused what’s driving this.
> This is one for the history books
It's amazing how frequently the world stops for these "historical events"
Historical doesn’t mean one off. It means written down in history and remembered.
I know this is only tenuously related, but after a week of them making changes (starting with DKIM) and blocking third party mailservers (and this week I got my first "your IP is blocked" but when you use their request to remove they say it isn't), I hope it goes down long enough to make more people migrate away
People won't migrate away. They'll double down on their contracts and spend more money. We saw that with Crowdstrike.
Unless it causes enough economic damage for governments to interfere, nothing will change other than MSSharePrice increasing.
Really hoping it's a monumental fuckup that lasts through the weekend well into the next week.
The corrupted executives need to learn their lesson on consequences of their actions. Laying off the core of your system?. Say goodbye to triple-9 SLA.
I have zero sympathy for big tech
Even if it lasted a month, that is not the lesson they would learn. Cut costs and grow revenue are as instinctive to them as breathing. There will be no apologies for that.
You sir or madam, unfortunately, are right...
One can only hope for a damage to their reputation so significant, it encourages other companies to swtch providers, and the devs to stand up and develop competing solutions.
Since Musk fired most of Twitter engineers, it seems like the prevailing thought has been that you can just trim your organization of the unnecessary workers, only accelerated by AI development. I also hope that this proves out to be disastrous for MS and the thinking goes back to hiring top people and making sure they're happy.
> The corrupted executives need to learn their lesson on consequences of their actions. Laying off the core of your system?. Say goodbye to triple-9 SLA.
Do you think they actually care about their "triple-9 SLA"? Only in as much as they could use it as a marketing bullet point to sign contracts to increase revenues/profits.
This was the event I needed to design and implement my email DR plan, by selfhosting Postfix again as a backup MX server.
downdetector says everything is recovering
but I still can't send an email in Outlook
the hell is going on?
Downdetector is a list of user reports. It's been broken so long that people have given up reporting that it's broken.
When you outsource all your coding, architecture and thinking to AI, you inevitably stop understanding what's actually going on.
That might then become a problem when things stop working.
[flagged]
https://status.cloud.microsoft/m365/referrer=serviceStatusRe...
> Engineering teams are cycling affected infrastructure through traffic drain, restart, remediation, and staged return-to-service while monitoring closely.
so they have no clue what is going on...
Yes. That's exactly what we do when we have no idea what is going on.
> Did you try turning if off and on again?
Hear that - that is the sound of every Engineer at Microsoft right now typing "We have a global incident - fix it" furiously into their chatbots!
Nah, most of us just went home because we couldn’t log into anything
You forgot the
> make no mistakes and do it fast
part.
Yeah, well, forgetting the part about making no mistakes is probably why we're here in the first place!
I heard a story, which you can't attribute to me of course, I just heard it -- we had a thing (literally yesterday) where a person from M$ came selling us some AI snake oil (we are very much an azure pilled org, but not super AI pilled).
The meeting was obviously in teems, because they sold us on that already and most of the presentation the screen sharing didn't work for half the people, which generated a continued stream of office-grade jokes for the whole duration of the presentation (30 minutes or so).
As a guest the presenter was totally oblivious to this fact, as they didn't get access to the chat messages, because the policies are in place (we are a serious company).
Wouldn’t it be funny if we had a Black Friday right about now and nobody could coordinate a response over email.
Reading every day about them, don't they think to let the superintelligences fix their computers?
This is what happens when you outsource your business to India and lay off your seniors.
Just use AI, duh?!
Call Sam Altman for some RAM first!
I guess we better get used to this new normal and pray your travel day or important appointment is not going to be on an outage day, whether that's MS, Crowdstrike or else.
What happened to five nines uptime? We don't want to hear excuses.
Interestingly, github is NOT down.
Different infrastructure.
...yet.
Maybe they morged[1] a bad commit somewhere
[1]: https://nvie.com/posts/15-years-later/
Why hasn't AI fixed it already?
You don't say.
Vibecoding is going well it seems
Get Copilot to fix it.
Cannot log in....
hey Claude fix my production issue in production make no mistakes
investigating..
goofing..
finagling..
hey claude what the hell you deleted all my configs permanently!!
metastasizing.. You’re absolutely right. I went above and beyond what you had asked and started completely fresh. Unfortunately I didn’t take any backups and deleted all existing distributed configurations. Oopsie!
[flagged]
[dead]
Gmail down for me also. China must be mad about the Nvidia dude.