Microsoft is unable to run a reliable hosted email service, given infinite resources. What people don't realise is that the former is in part a consequence of the latter. Without a forcing function, complexity multiplies, silent fragility accumulates, end-to-end understanding dwindles. There is no incentive to simplify.
Im not so sure about the infinite resources part, more like a self inflicted austerity of resources - its a technology problem emergent from a organizational problem, emergent from a certain late stage mindset
~220K staff, ~$330B revenue, ~$200B EBITDA. How they direct those resources (wisely or otherwise, timely or otherwise) is a management decision. I feel I should also point out that when talking about people, "resource" is not the same as "talent".
imho, especially in large tech orgs, every engineer is a 10x engineer for the right circumstances. But they're 1x engineers the rest of the time. But the whole reason to keep them around is you get the 10x performance when it's needed. Perhaps handy when a massive outage occurs
Any experienced dev at any org who knows how to deal with that particular pile of mess because it has been their job for at least half a decade is 10x developer.
I have a feeling it's like weight gain. If you run lean your whole life you are fine but if you let the pounds pile on then you are in for a whole lot of pain when you eventually try and lose it.
sometimes you're bulked up because it takes muscles to lift the extra 300lbs of load. now go on a sudden crash diet and see if you can lift the same weight without dropping it
Great insight. I would add that forcing function comes from the self, not from outside. The force is essentially the communication between the parts. When the communication breaks down, things fall apart, establishing a new kind of communication across the pieces to achieve equilibrium. The physical structures also have the same rule. They will crumble into a pile of lego bricks to achieve a new state of equilibrium and a simplified network of communication across the pile.
Does having 400M users and 1B emails per day factor into your forgiveness?
I’m no fan Outlook and what they’ve done for the world of email, but if they’re Rolls Royce and this outage is an engine failure then that’s an expected level of service?
Or they’re Boeing and this a 787 Max. I’m on the fence.
I heard someone threw half a billion dollars at an election and couldn't change people's minds. A company having a bunch of cash doesn't automatically get people to do the right thing. Especially when the priority at every turn is shareholders.
You're arguing that the governing structure democracy cannot accomplish things with money. That's true, but for completely different reasons. The analogy doesn't really work here because Microsoft is run by a CEO and is functionally a monarchy without any need to persuade anyone beyond their paycheck.
Someone else threw the same amount of money in the other direction. Your comparison would make sense if there were a second trillion-dollar company spending their entire budget on trying to make Outlook go down, which is not the case. But we do all know that your goal wasn't actually to make an apt analogy, and just to shove your political whining into an unrelated discussion.
>I heard someone threw half a billion dollars at an election and couldn't change people's minds.
hard to change the minds of a bunch of pissed off folks when your political campaign funding is pushing AI ads about transfolk and not grocery prices or datacenter rollouts.
Unlike other cloud systems, for email there doesn't seem to be any additional complexities, any dieconomies at scale, no? You could in theory run 100,000 domains as 100,000 separate email services. It's not like social media where everybody is looking at and interacting with everybody's posts. And the email algorithm is well established, it's definitely easier than, say, settling financial transactions.
I don't have very much forgiveness, perhaps only for those junior engineers tending the beast they grew, and which now has indigestion. No-doubt there are all the usual machinations of large corporations at work. Tech companies are no better (in some ways worse) than "normal" companies, because of the invisible complexity of software. Brooks, Ousterhout, and Stroustrup were not fools. But MS run a service, they get freedom to choose how it is engineered internally so long as it continues to present a service interface to the users/client apps/other internal and external services. This is an incredible level of freedom to choose what you engineer for. MS always had the ambition to serve the world with this service. It doesn't seem like they engineered accordingly. It's easy to knock them, but I think they must have made some significant wrong choices.
I'll spare readers the full anecdote, but I worked for the company running the largest mail service in the southern hemisphere, and at the same time a few of the large mail services in the northern hemisphere in the 1990s. We met with the Exchange team who wanted us to adopt their product. Where we could run a given service on 2-3 servers, Exchange sized by MS required 300 of them for just 1 of our countries. Fortunately our cash constraints prevented us going down that road. They offered the license for free and exerted pressure on the C-level, as you can imagine there were heated conversations and some choice language between mgmt and engineering on this topic.
> They offered the license for free and exerted pressure on the C-level, as you can imagine there were heated conversations and some choice language between mgmt and engineering on this topic.
Thanks to AI, heated conversations between management and engineering is mostly a thing of the past. Unless the engineer wields an outsized influence, they will be shown the door and replaced with a lower-priced AI-augmented engineer. Which might explain the increasing frequency of shitshows we are seeing lately.
No, and I've been discussing this with a friend today.
Internal software goes through rigorous change control, change windows, change freezes around key business dates, etc to minimize risk.
Then you sign up for some SaaS and they blow out email (which aside from security needs 0 updates lets be honest) midweek, mid business day without warning or reason.
When Microsoft has an outage it tends to be global. They don’t seem to learn institutionally how fault tolerance should work across a planet. All else- I wait for the postmortem to critiq
It's probably hacked together from bits of the on-prem version. We all know what these places are like on the inside. It's not the smooth running engine they like to present it as.
Many things change if the next generations will select another service because they have a bad impression (correct or not) about something.
Did you personally select anything IBM lately? At some point many years ago, people were saying "Getting rid of IBM in a corporate setting is close to impossible.
Actually it is about the perceived risk distribution within an org, not really about the quality of the product. "No one got fired for buying an IBM" directly tells the decision maker that their risk of getting questioned internally is low due to the brand reputation of IBM. It's a play on risk.
The main problem to get rid of Microsoft, is how they sold their services in the past more related to regulated environments. I worked at a place where we're using Google tools but with AWS as cloud provider, then one time we need to decide which tools we should continue use, the choices were Google and Microsoft. Microsoft won because they already had at the time the better auditable tools to track any single event across them which Google lack up to a certain point, we kept at least AWS though
And why would it? Most people using Microsoft 365 have bullshit jobs anyway. All they do is performative. They are not productive, with or without access to their e-mail and calendar. It doesn't make a real difference.
let me guess, the only real jobs are blue collar ones, and software developers of course, or whatever it is that you happen to do. May as well comment “I’m a bit green behind the ears” and leave it at that.
If anything, it's the opposite. Too many decades in corporate environments to not become cynical. Too many instances of having seen what the man behind the curtain is actually doing.
And I don't exclude myself here by the way. I know what I'm doing, I know what my colleagues are doing, I know what my managers are doing. We're all pretty busy, but our productivity is very low.
Seriously, who will change provider because of this? Our institution is so brainwashed and hidebound to Microsoft it would take a lot more than this. It seems like an abusive relationship: the bad times just increase fear generally, so make change less likely.
It's the same with AWS, Cloudflare. There is also this weird effect, often commented on here, where if half the Internet is down, everyone understands when you are down. If I use some random new start up for my infrastructure or services, and it goes down, customers and executives are furious. So it seems we are stuck here.
I think what it will mostly do is hardening existing customers anti-cloud stance.
There already was big reluctance by many enterprises to switch to Exchange Online and New Outlook due to lack of feature parity and control, to the degree that Exchange Server SE became a product (with support until at least 2035) and Classic Outlook is still the prescribed Outlook variant in many organizations (with support until at least 2029). I also know of some customers, which migrated back from Exchange Online to on-premise Exchange Server because they were so unsatisfied with it.
This will only make it harder for Microsoft to pull all customers to their cloud offerings. I think in general that's a significant blow to SaaS-ification of the field.
Apparently this single outage means Office 365 uptime this year drops to 3 nines (2 nines for this month). You can claim SLA credit worth 50% service fee by applying before December 31.
There's two parts, the actual downtime and the SLA agreement...
At my day job, we have a calendar integration that polls Microsoft's Graph API across hundreds of Microsoft 365 tenants. We saw our first metric spike in 5xx errors at 2026/11/12 17:33 UTC. As of writing it's 2026/11/13 09:44 UTC and we are still encountering a full outage, so 16 hours and 11 minutes.
With a SLA calculator [1], assuming no other downtime this month and entering 16h11m, we get:
* Yearly SLA: 99.82%
* Monthly SLA: 97.78%
Next, what is the actual agreement? I found an SLA document list here [2].
I downloaded Service Level Agreement for Microsoft Online Services (WW) English (2026/10/01). There's a heading for Exchange Online that says:
Uptime Percentage : Service Credit
< 99.9% : 25%
< 99% : 50%
< 95% : 100%
Which seems to agree with what the original commenter said for this month.
> 99% uptime for email means 3.5 days/ year no email
Depends on how it appears, if it was one or several large chunks that is much more disruptive than minutes per day. 3.5 days/year is equivalent to less than 15 minutes/day.
It sucks no matter how you slice it, but the severity/impact depends on more than the aggregate downtime.
We need to get beyond the arguments of 364 vs 365 and develop a new equilibrium in terms of availability that accounts for humans being equipped with these new found breaks as we spend time off of work.
I am certain this is the direction of thought at least for Europeans. IIRC France recently moved away from Microsoft. If right now they do not have this outage, I can only imagine how much it boosts confidence in such decisions
> I wonder whether this will help the already strengthening trend of some governments to unstick themselves from Microsoft (in particular their cloud).
It will help a bit, but it would need much more than 24h to have a real impact. If their service is still down in a week, now we're talking.
Yeah I mean a lot of people are nonchalant about it here, but email for business is like telephones. Basic infrastructure that has an assumed near-zero downtime and expectations of fault tolerance.
It's such bedrock infrastructure that I've seen it used as fallback in the event of electronic trading infrastructure outage, brokers would accept specifically formatted emails as orders.
At this point it would probably be less impact to have phones go out globally because people would fallback to walking across the floor, using zoom/slack for simultaneous communication or email for more async.
it happened on Thursday afternoon US time and it is still going on since then, Europe just joined this morning (unless someone worked yesterday late evening)
That's why most don't realize how big GMail is. Google has 1.8 billion GMail users, including a great many who are using their Google email professionally (no matter if it's a Google Workspace with their own domain or a redirection or just a plain GMail). Apple has 1 billion email users.
Microsoft 400 million and, granted, many businesses but it's not a fourth the number of email users GMail has.
And many, many, many people (real estate, delivery drivers, Uber and the likes) are using either Apple or GMail email from... Their phones.
It's a good thing it's only the "small" Microsoft email users base that is affected and not the 1.8 billion GMail users.
I would bet my lunch that the failing code is immediately attributable to AI, but the root cause is a failure of hiring, leadership, accountability, QA. If it was bad AI code, it was still deployed by a bad developer. The buck stops with us humans.
How many Microsoft employees are working on this? It's concerning that one of the biggest, most valuable companies on the planet, that runs services internationally, can have one of its primary services have gone down for this long.
6 hours ago they seemed to not exactly understand what the problem was, and that was 6 hours into the outage.
Not so long ago, I was assigned to a project that had to be built on Azure due to client requirements. Because we were underhanded and it was an MVP, we'd elected to use PaaS as much as possible: Postgres Flexible Server, Static Web Apps, Easy Auth, etc. On paper, these all looked fine. They had lots of documentation, example projects, GitHub repos, blogs, roadmaps, etc. Then we started building and running into issues, and we realised a bunch of these offerings were basically unmaintained. There were features that had been stuck in "preview" status for literally years. The blogs and roadmaps were no longer updated, and issues were piling up in GitHub.
I started digging into the historical blog posts, GitHub comments, etc. and I realised everybody at MS who'd been working on them, SWEs, PMs, architects, etc. had either left the company, or moved to other teams. And the one constant about the internal moves was that people were now working in some "AI" related project or product. Literally, I didn't find a single person who wasn't now working on AI.
Clearly, there has been a huge internal push to get into AI. Resources have been reallocated, and anything non-AI related has been starved. And since working on an AI project is probably considered the only route to advancing your career, anybody with any ambition has moved to a team doing something AI-related. The result is that huge numbers of products and teams doing vital but non-AI related work have been hollowed out, and now we're seeing the consequences.
This state of Azure saddens me. I learnt everything cloud with it. I've built many things (big and small) in it. It was the one part of Microsoft I thought they'd not neglect.
Lots of its features - App Services, DNS, managed DB, Front Door, Blobs, Durable Functions, Service Bus, App Insights - are very powerful. The frontend portal is mostly fine for management. The Az CLI makes it easy to provision things quickly. Coupled with .NET Core on MacOS, I've not needed to touch Windows locally for years.
I could learn AWS or GCP, but that's just going to cost more time when I'd rather use what I already know to build more things. With Azure, I'd hate to see the rot set in further.
That's related to something I realized which I think a lot of people might find amazing. They think of companies like Apple and Microsoft as trillion-dollar, globe-spanning companies, so it must be massive teams that work together on everything.
But no, read Hacker News or other niche places enough and eventually you realize a lot of the things we all know or use was a solo effort or a very small number of people, sometimes not even an assigned project but something they did on the side. Cue the xkcd about the lone open source project maintainer holding up the entire modern web but apply it to the tech giants and there you go, it happens there too.
Let me put this way, the Visual C++ standard library has officially one FTE working on it, everything else are open source contributions.
WinUI 3.0, WinAppSDK, and related CsWinRT, C++/WinRT are in a ridiculous position six years after Project Reunion announcement, due to ever meagre resources.
XBox app keeps under delivering user experience, and bugs on patch Tuesdays are getting a joke by now.
Maybe acting like jerk-offs got them to four trillion but there are consequences to burning all your social capital. A large camp of software developers have genuinely hostile feelings towards Microsoft, I would bet that this affects the quality of talent they can hire and these are downstream affects.
Lately it seems like more non-techies are souring on the company.
>A large camp of software developers have genuinely hostile feelings towards Microsoft
which, besides hatred, manifests in malice, like Nightmare Eclipse. I wouldn't be surprised if this outage was an engineer leaving a parting gift, using a bug they probably knew of for years but management didn't allow them to fix.
Tellingly, I will not go to MS for UI stuff.
When I have to anyway, I use their decades-old APIs.
It is a canary with regards to microsoft,
that they have no known champion for their UI stuff,
no trusted face for their UI stuff;
the company all just appears to coast along on
"I assume someone is handling it?"
The main microsoft employee I trust is Raymond Chen,
they have no current generations of Raymond Chen / that we know of.
they've still been hiring, just on an attritional basis - headcount has remained flat for a few yrs now. it is pretty clear though that they want to operate even leaner.
Enterprises have been failed time and time again by poor MS products, but techies will steadfastly refuse to move off it, so the business has no option.
> Enterprises have been failed time and time again by poor MS products, but techies will steadfastly refuse to move off it, so the business has no option.
My suspicion is that lots of the companies are so attached to all the MS stuff (and that's what their staff know), that it would be an incredibly large lift for them to move off it.
Like, I'm at a very windows/azure focused shop now (for the first time in my career), and literally everything is already on MS, or moving towards MS.
There are some good reasons for that, and some bad reasons, but I suspect that most of the sales process happens at the executive level, rather than the technical level (and Excel runs the world, so why not let MS do everything else?).
Jerk-offs can skim the profit of going big yesterday while the consequences affect everyone, but only in the future of yesterday. Who does such things, one might ask? Jerk-offs, off course. And PE too (likely not a disjoint set).
Was there ever a time when this was not true? M$ was always among top hated companies since 90s, arrogance, fuck-U to users all around, absolutely ignoring any standards larger than them, and famous 'Embrace, extend, and extinguish' which was pathetic to watch for any engineer worth 2 cents.
Top notch user experience peak was Windows XP and every subsequent release was massively larger, clunkier and slower while requiring way more resources, was spying on the users more and more. Office has similar story, peak of usability was maybe 2007 with ribbon introduction (which still a lot of power users hated with passion since it was far from clean design, more seemingly designed by committee of marketing guys).
This is what I don't get about big companies and their software; you'd think every application, every moving part would have a (small?) team of core maintainers, people that get paid well to know everything there is to know about something, ranging from (in microsoft's case) Paint to a particular configuration panel.
I get that this doesn't scale though - a lot of features are just "done" at some point and keeping one or more people whose sole job is to maintain this may lead to them bolting on features just to stay relevant / stop being bored. But big tech can easily afford to keep people like that on.
(I mean they keep (kept?) tons of middle management on, who knows what they even do? Not emailing right now that's for sure am I right?)
> every moving part would have a (small?) team of core maintainers, people that get paid well to know everything there is to know about something,
I work at a company with a few thousand employees in the Netherlands, and have been moved around a few teams in the last 7 years (not by choice, the team is nuked and I'm shuffled off elsewhere).
Technically, the product is supposed to have a "handover" to another team who can do the maintenance. What happens in practice is the person who worked on it gets saddled with maintaining it forever along with taking up the responsibilities of the newer work that comes along their way.
I've accumulated a string of such projects, including one that's used extensively by a few large companies, and the only person maintaining it is drumroll ... me, in whatever time I can spare for it.
Companies just don't want to compensate people to hang around and maintain a system, they want a consistent output of widgets that can be measured. Somehow our work culture is still stuck in factory worker mentality despite being well into the information revolution now.
Mythical man month, maybe throwing everybody in the company at it isn't the right thing to do. But they must be in a panic, and their own communications internally may well be hampered, could be MS everything is out for MS as well as everybody else.
Someone's going to have to write the Mythical AI Month after this.
If you lose 90% of developers to AI, but everything breaks, how much money have you saved?
The underlying moral issue is that the management and money people have contempt for people who don't manage and don't have money. Which includes customers and engineers.
It's not just an inhumane system - although it is, that, brutally - but it's also technologically dysfunctional and ultimately unstable.
I've seen the response to an emergency be "get every dev in a big room with some non-technical middle managers to direct them, and have everybody talk at once". My jaw was on the floor.
"When in danger, when in doubt, run in circles, scream and shout".
"Too expensive" as measured by bean counters that don't understand the actual risk profile - or do understand it but have such a short window of strategic concern it leads to this. It would be an interesting analysis to stack up costs to Msoft vs cost of labor cut as this outage evolves.
It may not be true now but 10 years down the line? This is disaster in the making yet while most people recognize that (when filtering out most fanatical pro-ai folks claiming everywhere how they get 10x or 100x work done) nobody is able to do anything. Sort of legacy technical debt being delivered but of different kind than usual, that train that will hit you since you stand on the rail line yet its moving 5kmh and you see it from 200m, but unable or unwilling to move away (Austin powers scene with steamroller does it better than me [1]).
Imagine if say BMW decided they don't want to manufacture their motors, suspensions, drivetrain etc. Outsourcing 100% of that knowledge to external suppliers, for all models, from now on forever. How do you think market would reacted to such move ie by valuation of stocks?
When you outsource everything, you’re not relevant anymore. I agree on that, anyone else can do it just as good as you.
But the biggest fear is how is ai going to improve then ? We’ve seen it can’t feed itself on data it produced, the model quality decrease. But if there is no more human to produce ever better quality code because they rely on llm to generate code, we’re going to reach a wall.
Microsoft can't fix Outlook because they don't have Claude, and Anthropic's employees can't fix Claude because they can no longer log in without Outlook?
Alas, Teams are working, or at least the chat and voice call portions of it. So managers are in a calling and chatting frenzy right now, pulling every body else in.
/s Their internal AI is full steam on it. The issue is just that their internal AI infrastructure is down too, so AI just need to fix itself first. Then it will focus on fixing the issue. Oh, wait...
Was he? I struggle to see it even in hindsight, and definitely his moves were bunch of FU to common users. Maybe in some specific area you care about, but not their core products like windows and office
yeah very effective, he fired 90% of our Windows Store team (reviewing the apps) in the first year to replace people with automation, I survived up to the last round of firing after which they kept ~20 employees out of original ~200
When your most important contacts use outlook, it does not matter what you do. Sending and receiving email when almost nobody else is sending and receiving is only small help.
Some problems are not solved by individual choice. If 30-40% of your important contacts use service that is down, your ability to work goes down dramatically.
I have currently important 3 daily contacts down they use MS services extensively contacted and said they can't do much of anything because their work process and the process of their contacts is down.
Wouldn't you then need to keep up-to-date backups but ensure the backups still stay secure? Learn all about and generate SPF, DKIM, and DMARC records but still regularly get sent to the spam folder or IP blacklists if your recipient has a big-provider email? Deal with self-rolled 2FA or passkeys and secure recovery methods. If you're self-hosting at your house, you're at the whims of nature, infrastructure, and creatures not disturbing the machine. You are the one patching security holes, or if you run someone else's containerized or easy-setup mail server, making the decision between daily updates that'll catch supply chain attacks and potentially introduce instability or new security holes; or monthly/annual/never updates that virtually ensure you have a few CVEs active at all times; or the lead maintainer quits/dies and you watch it deteriorate while users groupthink how to migrate their mail to a different container.
Sorry for the pessimism. I can't suggest a better alternative. I have one account on Gmail (for impersonal communication like shopping) and another on Fastmail and occasionally reflect on how annoying it will be to lose access to either.
>Learn all about and generate SPF, DKIM, and DMARC records but still regularly get sent to the spam folder or IP blacklists if your recipient has a big-provider email?
This is my biggest complaint when self hosting. You do everything right and they still send your mail to junk or just silently discard it while telling you 250-Ok for "reasons".
Although I recently had a legit Microsoft email also end up in the junk folder so I feel better that they can't even get it right themselves.
Microsoft have also recently kicked out self hosters from their "Smart Network Data Services" portal by only allowing the ASN owner (ie. your cloud provider) to access it.
There probably are some very good reasons not to self host your email server but apparently my own, self hosted, email server's uptime over the last decade is now better than that of outlook.com :-D
Quick lets collect all options people have to self host! in addition to docker-mailserver im aware of
I decided to self-host my mail a few weeks ago and ran through all of the big names I'd heard - Mailcow, mailinabox, Stalwart, and Dovecot+Postfix; honestly I didn't like any of them. Stalwart is very corpo, hard to configure (reading the docs[0] confused the shit out of me), and feels vibe-coded. Mailcow and miab push a huge stack onto you in a way that feels hard to modify yourself, e.g. expecting you to allocate ~2GB ram(?) to running ClamAV, and I never managed to get Dovecot+Postfix working properly.
In the end I paved over my Linux VM with OpenBSD and used Dovecot+OpenSMTPD which I found dead simple and so far faultless. I have nothing but kind words for it.
That's fair. It is perfectly reasonable to want customizability if control is what you are after. I remember the days fondly where I was running postfix and dovecot on bare metal.
Nowadays my interests have skewed towards being able to have groupware features without much maintenance so groupware in a box suits my needs just fine. The learning curve to doing serious customization with a large FOSS groupware project might be a bit longer but that is offset by other factors.
Yeah I definitely get less done on OpenBSD, but I have more fun doing it, and fortunately in my position where that works for me. If I were providing a service to others instead of just myself I'd have to rethink my stack.
I have always felt it's less about email redundancy and more about having a backup plan. If outlook goes down tomorrow, does everyone know how to keep work and client communication moving? that is something worth thinking about.
You joke, but during my time at Netflix there were at least two layers of backup communication systems: if slack went down, we would jump to google chat, and if google chat somehow went down too, there was an IRC server and a (hardware) phone bridge. I never touched any of it because I wasn't an SRE, but I did read through the training one day because I was bored.
There was no need to abandon perfectly working technology, with pain some people might learn this. Departmental servers, pesky lordly sysadmins, this was a working sustainable system that even scaled to the phone company and US government levels.
Just got up, watched BBC on treadmill. Everyone talks about how this is a phenomena of modern life, that the only other option is paper. Microsoft is winning. People cannot envision an alternative. But outlook is not email. Just use someone else, anyone else.
My email is still working, has been solid for moe than a decade. My calander app opened as normal this morning. My chat windows are all alive. My digital life is just normal today... with rather fewer emails in the inbox. DROP MIRCOSOFT. Pick up linux. Pay for a proper email service. There is a larger world out there!
This is going to cause absolute chaos. Basically every government organisation and business in the UK runs on outlook. The whole day is going to be a write off. It’s like a bank holiday!
Just the UK? Lol, you can add many, if not most government and private companies and educational institutions in the US and countless other countries to that list. The government and corporate world is grinding to a halt.
Government I have no idea, but loads of businesses don't use Outlook - alternatives do exist. I've personally not used Outlook for well over a decade now.
Quick google AI mode check says:
- MS 365/Outlook: ~40%
- Google Workspace: ~25%
- Zoho: ~2%
And then a bunch of others - Rackspace, Zimbra, IBM WebSphere Lotus Notes Blue For Enterprise XML Watson Edition et al
Depends what market you are in. In the UK I would estimate M365 usage is ~100% of local/national govt.
And in terms of classic 'corporate' sectors like law, finance, etc, it also close to 100%.
Whereas in "tech" it is far more likely to be Gmail.
I _still_ get people who are non technical that work in 'corporate' sectors that can't join a Google Meet because they think it is Teams. They literally haven't ever used a Google meeting before.
Every government globally with all parts and agencies is purely microsoft. Some EU states are trying to move away from it for obvious reasons (not this event, more like national security and sovereignty which you don't have with MS products).
Exchange and Active directory are the biggest obstacles by far, bureaucrats can learn how to work with open source office alternatives easily, and underlying OS can be anything as long as you have few icons to click on.
I believe nowadays the general solution is HCL Traveler (server side) wit HCL Verse as client. The latter offers email access over pretty nice looking mobile, desktop and web apps.
Having friends who work at Microsoft, this is not really surprising at all. The company has always had deep structural problems. The AI change has really just exposed it and is aggressively accelerating the inevitable implosion. It's truly in a free fall that is planned and orchestrated perfectly by their upper management. Their CEO and VPs are convinced you don't need engineering know-hows any more because of AI, and they got all the management layers to sincerely agree with them. Not in a malicious or subverting way, but in a "you can't get a man to understand something his salary depends on him not understanding it" way.
Microsoft has a moat, but their moat is having billions of dollars to spend on data centers. It's a failure everywhere else. It'll continue to rake in billions in revenue because people need compute now, but it's rotten at the top.
Outside of tech and disaster recovery, people _really_ don't understand that some redundancy--in this context--is a good thing.
The idea of paying for a backup system, or even having one on standby for situations like this is just not on people's minds, or if it is, it's simply not affordable because it would cost you too much compared to your competition.
Razor thin margins and generally getting away with it is what makes this work. Until it doesn't.
Or, they get the bad idea that they can just pay one supplier and that supplier will deal with the redundancy for them. When the redundancy needs to be at supplier level.
I don't see a reasonable alternative here. Pay for another email/office suite provider, and duplicate all mails and files? Twice the cost and exposure surface just to cover 0.01% of downtime.
Sure, it makes sense for mission critical companies (utilities, emergency services), but I don't think it does for an average company. Especially that you don't take much blame if everyone is down.
Email in a large part is just a tool for communication(and mostly low priority at that). Where I work, we have backups: Teams( same) provider, Cellphone ( different provider ) , Landline( different provider)
We had a situation recently where cellphones were down, we switched to landline.
You don't need a 1 to 1 mode backup for everything, just a failover plan.
Unless your company is infrastructure or medical, I doubt they have a backup/failover for loss of power. Or water.
The difference between a distributed system and a monolith is that in a monolith you have some idea what your single point of failure is, but in a distributed system you don't know your dozen single points of failure until they happen in production.
Example: in 2020, a 6-minute hiccup in a network switch caused asymmetric connectivity, breaking their distributed leader election algorithm, making that system read-only, breaking a different system's heartbeat mechanism causing that system to assume a failure, enact a recovery and degrade in performance, causing 25% of API requests to fail for 6 hours.
Google had a secret storage system back in the day, that required special encryption keys to be entered when it got re-initialised. The keys were in a safe. Guess where the combination to the safe was stored? ;)
Might be a coordinated attack against the network providers that MS uses?
MS: Our investigation determined that one of our network connections to a third-party ISP was not taking traffic. We've diverted traffic to alternate channels to expedite recovery. For more information please follow MO273940 in the admin center.
Other source: This is related to a BGP hijack of at least 87.128.0.0/10 belonging to Deutsche Telekom (AS3320) hijacked by AS212416 (Proximity, Bulgaria) passing through AS57344 (Telehouse EAD, Bulgaria) starting at 0633 UTC today.
Something I've been thinking about with the last few major outages is that as there is more and more consolidation (fewer large companies running a higher share of the world's infrastructure and critical services) is that we become increasingly exposed to massive disruption like this. Would some sort of mandated maximum occupancy per instance resolve this? So Outlook.com is allowed to support 100m accounts per instance. Then you are required to spin up Outlook2.com. You are not allowed to have any shared infrastructure. Yes it's wasteful and expensive, but these are companies with billions in revenue, so that's the price you pay. It just seems that keeping these huge, incredibly complex services running is turning in to an impossible task. Very much a hot take so interested in the opinions of peope who have worked at this scale.
They could (and should, and clearly don’t) architect the services like this internally. There will always be some single-points-of-failure, but minimizing those should be a top priority. Just from reading their status updates it’s clear most everything is sharing way too much in terms of infrastructure and configuration.
But even this requires someone high up in the organization to have the insight that reliability requires some level of “inefficiency” — ie retaining certain highly paid staff, duplication of effort, over provisioning of resources, etc.
The irony is that the execs who could make those decisions are convinced that LLM tools obviate those needs, and shift all spare resources in that direction. But without heavy constraints from experienced humans, LLM tools will only multiply the complexity at the root cause of these sorts of outages.
I am positive there are bare-metal, no shared tenancy, customers of MS already. Unclear if they are down too. While yes, these companies are massive, not being allowed to share infrastructure defeats the entire revenue-purpose of offering a cloud. The hardware in these servers is unbelievably expensive. You'd go back to everyone having an IT department and a few racks of servers for hardware configurations that meet their specific needs (which I'm totally okay with, and even rooting for!)
I'm an outsider who used to work at a different cloud provider, so take this with a grain of salt, but this doesn't smell like a limits and provisioning problem, per se. It's not necessarily like they just had 1 too many emails tip the scales for the entire world. Until today stuff was running okay. This was a monitoring and fail-over problem, at a comically tragic scale - a cascade failure. Architecturally they obviously did not design for a world where they could damage control failure at this scale. Failure should be expected and designed for at literally every layer. They cheaped-out on careful design in order to play catch-up building a public cloud offering and here we are.
Self-hosted email is trusted? Not in my experience -- maybe I had bad luck and a small sample size but I quickly abandoned it after a few contacts I needed to be able to e-mail didn't get my email. And generally it didn't even go to their spam box -- it was filtered from even getting that far by their provider.
I was using a dedicated VPS, maybe that was part of the issue, maybe self-hosting from a residential IP might work better. But it was easier for me to let iCloud handle my domains email. It's been flawless ever since even though they're not really a big player in e-mail. Used Proton for a bit, that was (almost) flawless as well, though even Proton occasionally gets blocked by stricter corporate rules.
did you do all the things? DMARC and what have you? Super easy in LLM land but wasn't hard even before it, just had to set aside 20 minutes to read and work through various guides, then maybe another 10 or 20 minutes in the subsequent weeks to confirm function.
Anthopic (unrelated to IPO announcement)... After 24 hours we directed our resources towards diagnosing the MS Office365 outage and 15min later informed them of a solution.
many universities bought into the cloud offering back when it was still live@edu, shown as an upgrade path for existing on-premise dovecot/exchange systems. back then a student would be lucky to get more than 100mb for their mailbox, and maybe a gig of personal file storage (which was only accessible on campus or via a vpn, the latter many would not have access to). microsoft just offered a _better deal_ at the time
frog boiling means that many never noticed how big (and therefore unspecialised) microsoft was until it was too deeply integrated
It's much better UX if everyone within the company uses the same office suite/email, it makes collaboration much nicer when everything is well integrated. And people get used to one system.
My guess is that having duplicate system would cost (in money for double subscriptions and lost productivity) much more than the benefit of being available 0.01% more. Especially for universities where almost nothing is really urgent and can't be delayed by a day.
Because it's not DNS related.. you can still resolve it.. and sometimes reach the frontend- but they can't route the requests to email servers because they are being "overloaded"
"Okay? Nothing. It’s just terrible. Every day is a nightmare. As recently as 1972, most buildings collapsed and rusty nails were found in cereal boxes all the time. Okay, so when you’re faced with statistics like this, it’s very difficult to be optimistic. So, hey, why not think about the statistics? I want to just put that out of your mind."
Care to explain the humour in the 'joke'? It is obvious that the statement is not earnest, but it is framing its criticism in potentially the most racist way possible. It wouldn't be particularly funny even without that.
What an excellent question. I am no comedian, but let me try and break it down as best I can.
What this joke is: Crude, absurdist satire of corporate DEI culture and cheap "offshoring" / importing of cheap labour.
Why it works: It packs quite a lot into so few words:
a) It hijacks a polite IT support cliche "Have you tried... (turning it off and on again)", setting the reader/listener up with a familiar rhythm.
b) It then seamlessly pivots into taboo-breaking; just like South Park (for one of a zillion examples, this stuff is literally older than Shakespeare), the "saying something you are not supposed to say" is what draws laughter.
c) It ends having made a good point: an aggressive focus on profit, rather than quality, has increasingly become one of Microsoft's quintessential characteristics. The outage is unsurprising in that light.
In their defense it was probably insanely fragile prior to the AI assisted coding boom.
Given the nesting of services at MS, the fractious org structure, and the length of service of some of these systems it was already a bit of a tinder-house.
AI just provided the accelerant, and while I don’t work at MS I can imagine they’ve lost some serious tenured talent over the last 5 years. I’d be surprised if anyone is left that actually understands how it’s all glued together.
Or rather, that might be the case before this outage. I suspect the org has relearned a fair bit over the last day.
No one knows if this is AI related, but it's easy to imagine that where changes written by AI fail in production, if AI can't find the solution (which will be common as AI access to production systems is typically reduced), then the devs won't either, or at least it will take them some time.
Perhaps the velocity is the problem, not that AI inherently produces worse code (you just need one bad or just inexperienced engineer have their PR slip through to have bad human written code in you code base). When the velocity is high, there's less time to battle test each change and new code piles up on the old one. When the velocity is low, shitty code fails in production, but people are aware of what was recently changed and can find and fix the problems.
I think this makes sense. If I ship 10 lines of code to production and something breaks, it's easier to reason about than if I shipped 1000 lines of code, even if I did read and understand them before shipping .
The bigger issue with having your systems written by AI, is that nobody knows how the system works anymore and can't pinpoint where the likely failure points are.
The status updates have a pretty strong "LLM-driven agent failing to debug a complex system" feel to them. I hope it's not the case, but I could imagine a scenario in which inexperienced engineers - or experienced ones who have lost touch with the codebase - are deferring to LLMs to drive the troubleshooting and fix here.
I do actually believe that LLMs and process around them will continue get better and become the accepted, but that there's going to be a 'fossil record' of a lot of problems having been seeded in these early years of AI that will take some time repair.
It will be interesting to know if the loss of codebase understanding due to agentic coding is the primary cause for the slowness of their response.
If it is indeed the case, this might be the first clear "win" for those who have been warning about the dangers having blind faith in AI and of pushing poorly understood code into production.
Alas, Teams is working, although parts of it which are integrated from other services are not, like calendar (exchange), or file sharing (share point).
This is why one signs a contract with Microsoft: no-one gets fired when they have a twelve-hour outage. You sign a contract with some unknown company and have a six-hour email outage, you had better start packing up your things.
Nothing could go wrong if you force/bind millions or billions of people to a single point of failure. Of course, I doubt Microsoft ever will have to pay a single cent for lost work hours/ money across the globe. They probably have a fine print against that.
Hopefully some of their "customers" learn a lesson and switch to a software without this side effect in worst case. There's plenty of alternatives (LibreOffice, OpenOffice, etc.)
Well then your one basket is your machine, or your backup system. And Office and Onedrive still work IIUC, so you're not really comparing apples to apples.
It's simple really, I've no cloud accounts, and even my Android phone is de-googled. I'd never trust any data to the cloud and least of all to MS.
What's happened here with MS clearly illustrates one of my reasons. I find it very strange that so many use Onedrive, Office 365, etc. and trust them even though they've not one iota of control over their operation.
As we've witnessed, what they're doing is strategically very risky.
Email is actually designed with intermittent connectivity in mind. The sending server buffers outgoing emails until they're accepted by the receiving server. Typical settings are to retry every 4 hours, with a permanent failure being counted after 3-7 days.
Their current engineers must have stored their KT notes from their laid off and leaving senior engineers in OneNote on their Mac, and might have been working with copilot to find the root cause /s
Microsoft is unable to run a reliable hosted email service, given infinite resources. What people don't realise is that the former is in part a consequence of the latter. Without a forcing function, complexity multiplies, silent fragility accumulates, end-to-end understanding dwindles. There is no incentive to simplify.
Im not so sure about the infinite resources part, more like a self inflicted austerity of resources - its a technology problem emergent from a organizational problem, emergent from a certain late stage mindset
~220K staff, ~$330B revenue, ~$200B EBITDA. How they direct those resources (wisely or otherwise, timely or otherwise) is a management decision. I feel I should also point out that when talking about people, "resource" is not the same as "talent".
Are those numbers accurate now that 60k were offered buyouts?
Anyone want to guess if any of those afflicted _could have_ caught an issue that may have prevented this disaster?
1) they haven’t left yet
2) they’re only expecting 10% or so to take the buyouts, so more like 6,000
Now what if those were mythical 10x engineers?
imho, especially in large tech orgs, every engineer is a 10x engineer for the right circumstances. But they're 1x engineers the rest of the time. But the whole reason to keep them around is you get the 10x performance when it's needed. Perhaps handy when a massive outage occurs
Any experienced dev at any org who knows how to deal with that particular pile of mess because it has been their job for at least half a decade is 10x developer.
They haven't left the company yet
Do you think those offered buyouts may have less incentive to focus as much?
I have a feeling it's like weight gain. If you run lean your whole life you are fine but if you let the pounds pile on then you are in for a whole lot of pain when you eventually try and lose it.
sometimes you're bulked up because it takes muscles to lift the extra 300lbs of load. now go on a sudden crash diet and see if you can lift the same weight without dropping it
Great insight. I would add that forcing function comes from the self, not from outside. The force is essentially the communication between the parts. When the communication breaks down, things fall apart, establishing a new kind of communication across the pieces to achieve equilibrium. The physical structures also have the same rule. They will crumble into a pile of lego bricks to achieve a new state of equilibrium and a simplified network of communication across the pile.
Does having 400M users and 1B emails per day factor into your forgiveness?
I’m no fan Outlook and what they’ve done for the world of email, but if they’re Rolls Royce and this outage is an engine failure then that’s an expected level of service?
Or they’re Boeing and this a 787 Max. I’m on the fence.
>does having 400M users and 1B emails per day factor into your forgiveness?
Trillion dollar companies shouldn't even be able to have excuses.
You have infinite money, stop fucking up.
I heard someone threw half a billion dollars at an election and couldn't change people's minds. A company having a bunch of cash doesn't automatically get people to do the right thing. Especially when the priority at every turn is shareholders.
Cash can't fix culture.
You're arguing that the governing structure democracy cannot accomplish things with money. That's true, but for completely different reasons. The analogy doesn't really work here because Microsoft is run by a CEO and is functionally a monarchy without any need to persuade anyone beyond their paycheck.
Someone else threw the same amount of money in the other direction. Your comparison would make sense if there were a second trillion-dollar company spending their entire budget on trying to make Outlook go down, which is not the case. But we do all know that your goal wasn't actually to make an apt analogy, and just to shove your political whining into an unrelated discussion.
>I heard someone threw half a billion dollars at an election and couldn't change people's minds.
hard to change the minds of a bunch of pissed off folks when your political campaign funding is pushing AI ads about transfolk and not grocery prices or datacenter rollouts.
Unlike other cloud systems, for email there doesn't seem to be any additional complexities, any dieconomies at scale, no? You could in theory run 100,000 domains as 100,000 separate email services. It's not like social media where everybody is looking at and interacting with everybody's posts. And the email algorithm is well established, it's definitely easier than, say, settling financial transactions.
>Does having 400M users and 1B emails per day factor into your forgiveness?
No.
Because of the infinite resources premise and because there exist email services of that scale that are reliable. Email is not an unsolved problem.
The rest of your comment does not make much sense to me ...
I don't have very much forgiveness, perhaps only for those junior engineers tending the beast they grew, and which now has indigestion. No-doubt there are all the usual machinations of large corporations at work. Tech companies are no better (in some ways worse) than "normal" companies, because of the invisible complexity of software. Brooks, Ousterhout, and Stroustrup were not fools. But MS run a service, they get freedom to choose how it is engineered internally so long as it continues to present a service interface to the users/client apps/other internal and external services. This is an incredible level of freedom to choose what you engineer for. MS always had the ambition to serve the world with this service. It doesn't seem like they engineered accordingly. It's easy to knock them, but I think they must have made some significant wrong choices.
I'll spare readers the full anecdote, but I worked for the company running the largest mail service in the southern hemisphere, and at the same time a few of the large mail services in the northern hemisphere in the 1990s. We met with the Exchange team who wanted us to adopt their product. Where we could run a given service on 2-3 servers, Exchange sized by MS required 300 of them for just 1 of our countries. Fortunately our cash constraints prevented us going down that road. They offered the license for free and exerted pressure on the C-level, as you can imagine there were heated conversations and some choice language between mgmt and engineering on this topic.
> They offered the license for free and exerted pressure on the C-level, as you can imagine there were heated conversations and some choice language between mgmt and engineering on this topic.
Thanks to AI, heated conversations between management and engineering is mostly a thing of the past. Unless the engineer wields an outsized influence, they will be shown the door and replaced with a lower-priced AI-augmented engineer. Which might explain the increasing frequency of shitshows we are seeing lately.
No, and I've been discussing this with a friend today.
Internal software goes through rigorous change control, change windows, change freezes around key business dates, etc to minimize risk.
Then you sign up for some SaaS and they blow out email (which aside from security needs 0 updates lets be honest) midweek, mid business day without warning or reason.
When Microsoft has an outage it tends to be global. They don’t seem to learn institutionally how fault tolerance should work across a planet. All else- I wait for the postmortem to critiq
>1B emails per day
Jesus, make sure you stay away from signal processing if this trickle of data scares you
Just for a sense of scale, on a single CPU core that is something like 1M instructions per email. ONE CPU CORE.
and it's all done automatically by a dedicated device, what's your point?
They could've split it into smaller e-mail services in stead of designing one monolith for the planet.
I mean it's e-mail.
It's probably hacked together from bits of the on-prem version. We all know what these places are like on the inside. It's not the smooth running engine they like to present it as.
Its not "email"
Its all connected
M365 is a suite. Make a 365 email and you also make an account in Entra, connections to onedrive, sharepoint, teams etc etc etc
Its not "just email"
I was digging through Teams API calls the other day and found some references to Skype in the responses. It’s all connected indeed
Skype for Business was previously Lync, which is what the teams voip components are based on. (Teams is essentially a lync/sharepoint client.)
They just one day decided to rename it Skype for Business while making it incompatible with Skype because brand synergy.
Its also a giant pile of shit.
Teams - absolute garbage OneDrive - clunky garbage Sharepoint - massive pile of shit
Every service becomes a monolith if it becomes popular enough.
Today is sounding like one hell of an incentive.
Why?
Do they lose any significant number of customers? Do anybody in leadership get fired?
Do reading the twitter status updates give you the impression of an organization that takes responsibility seriously?
Or do it sound more like an organization that is optimized for evaporating responsibility?
I think this will change nothing.
Well, at the very least there's going to be a lot of people pointing at their broken SLAs.
>Do reading the twitter status updates give you the impression of an organization that takes responsibility seriously?
Not implying otherwise, to be clear. Very far from defending MS.
Pointing at SLAs does nothing. Maybe they'll issue a discount or two. Getting rid of Microsoft in a corporate setting is close to impossible.
Many things change if the next generations will select another service because they have a bad impression (correct or not) about something.
Did you personally select anything IBM lately? At some point many years ago, people were saying "Getting rid of IBM in a corporate setting is close to impossible.
Actually it is about the perceived risk distribution within an org, not really about the quality of the product. "No one got fired for buying an IBM" directly tells the decision maker that their risk of getting questioned internally is low due to the brand reputation of IBM. It's a play on risk.
The main problem to get rid of Microsoft, is how they sold their services in the past more related to regulated environments. I worked at a place where we're using Google tools but with AWS as cloud provider, then one time we need to decide which tools we should continue use, the choices were Google and Microsoft. Microsoft won because they already had at the time the better auditable tools to track any single event across them which Google lack up to a certain point, we kept at least AWS though
I wonder if clients will take to court for damages/liability?
I imagine there is some kind of arbitration included in the service
No. That is contracted out.
And why would it? Most people using Microsoft 365 have bullshit jobs anyway. All they do is performative. They are not productive, with or without access to their e-mail and calendar. It doesn't make a real difference.
let me guess, the only real jobs are blue collar ones, and software developers of course, or whatever it is that you happen to do. May as well comment “I’m a bit green behind the ears” and leave it at that.
If anything, it's the opposite. Too many decades in corporate environments to not become cynical. Too many instances of having seen what the man behind the curtain is actually doing.
And I don't exclude myself here by the way. I know what I'm doing, I know what my colleagues are doing, I know what my managers are doing. We're all pretty busy, but our productivity is very low.
how do you figure?
Seriously, who will change provider because of this? Our institution is so brainwashed and hidebound to Microsoft it would take a lot more than this. It seems like an abusive relationship: the bad times just increase fear generally, so make change less likely.
It's the same with AWS, Cloudflare. There is also this weird effect, often commented on here, where if half the Internet is down, everyone understands when you are down. If I use some random new start up for my infrastructure or services, and it goes down, customers and executives are furious. So it seems we are stuck here.
Microsoft is worse. At least you can entertain the idea of switching between AWS and GCS but moving off outlook or Entra? Not a chance
What's wrong with GMail?
[dead]
It might not force a change away, but it definitely reduces the chance of people switching to them.
I think what it will mostly do is hardening existing customers anti-cloud stance. There already was big reluctance by many enterprises to switch to Exchange Online and New Outlook due to lack of feature parity and control, to the degree that Exchange Server SE became a product (with support until at least 2035) and Classic Outlook is still the prescribed Outlook variant in many organizations (with support until at least 2029). I also know of some customers, which migrated back from Exchange Online to on-premise Exchange Server because they were so unsatisfied with it.
This will only make it harder for Microsoft to pull all customers to their cloud offerings. I think in general that's a significant blow to SaaS-ification of the field.
Doubtful.
I suspect people will take it in their stride. What would happen after a month long outage is another question.
[dead]
tl;dr
https://en.wikipedia.org/wiki/XBill
Apparently this single outage means Office 365 uptime this year drops to 3 nines (2 nines for this month). You can claim SLA credit worth 50% service fee by applying before December 31.
Do you have official reference for this?
There's two parts, the actual downtime and the SLA agreement...
At my day job, we have a calendar integration that polls Microsoft's Graph API across hundreds of Microsoft 365 tenants. We saw our first metric spike in 5xx errors at 2026/11/12 17:33 UTC. As of writing it's 2026/11/13 09:44 UTC and we are still encountering a full outage, so 16 hours and 11 minutes.
With a SLA calculator [1], assuming no other downtime this month and entering 16h11m, we get:
* Yearly SLA: 99.82%
* Monthly SLA: 97.78%
Next, what is the actual agreement? I found an SLA document list here [2].
I downloaded Service Level Agreement for Microsoft Online Services (WW) English (2026/10/01). There's a heading for Exchange Online that says:
Uptime Percentage : Service Credit
< 99.9% : 25%
< 99% : 50%
< 95% : 100%
Which seems to agree with what the original commenter said for this month.
[1] https://uptime.is/reverse
[2] https://www.microsoft.com/licensing/docs/view/Service-Level-...
> Uptime Percentage : Service Credit > < 99.9% : 25% > < 99% : 50% > < 95% : 100%
These targets feel very very low nowadays.
99% uptime for email means 3.5 days/ year no email.
(although 99.9 = about 8 hours/year would seem bearable - but maybe not for a company like microsoft)
> 99% uptime for email means 3.5 days/ year no email
Depends on how it appears, if it was one or several large chunks that is much more disruptive than minutes per day. 3.5 days/year is equivalent to less than 15 minutes/day.
It sucks no matter how you slice it, but the severity/impact depends on more than the aggregate downtime.
https://old.reddit.com/r/sysadmin/comments/1xuqu5y/microsoft...
Thanks for the tip!
"Office 364" is apparently a trending term.
We need to get beyond the arguments of 364 vs 365 and develop a new equilibrium in terms of availability that accounts for humans being equipped with these new found breaks as we spend time off of work.
https://techcrunch.com/2026/01/05/microsofts-nadella-wants-u...
The running joke in my last shop a few years ago was "Office 265"
(shameless self plug) the microsoft 362 extension[0]:
replaces all occurances of microsoft/office 365 with microsoft/office 362 (not in links, just in the visible text)
[0] https://addons.mozilla.org/en-US/firefox/addon/microsoft-362...
Microsoft Outofluck
Office 360 at one point in history.
We just don’t know how good we’ve got it, that’s all!
I couldn't specifically say since my admin.microsoft.com fails pukes on a wildcard certificate for azureedge.net.
Also, multiple bankers I'm speaking to about Anthropic's IPO can't send or receive work e-mails. So that's a thing.
I just realized your name is the entire song / artist. Are you Jermaine Dupri?
Has someone got you at gunpoint, to be using Azure?
https://isolveproblems.substack.com/p/how-microsoft-vaporize...
https://www.binarysecurity.no/posts/2026/10/one-root-case
This is arguably one of the biggest outages in modern history. AWS region level outage was more visible, but email is a backbone of the modern world.
BBC is running live coverage of it at the level of government and ministers, feels surreal
I wonder whether this will help the already strengthening trend of some governments to unstick themselves from Microsoft (in particular their cloud).
I don't expect a fundamental shift but the boss having no access to his mails certainly seems like something I would bring up quite a bit.
I am certain this is the direction of thought at least for Europeans. IIRC France recently moved away from Microsoft. If right now they do not have this outage, I can only imagine how much it boosts confidence in such decisions
> I wonder whether this will help the already strengthening trend of some governments to unstick themselves from Microsoft (in particular their cloud).
It will help a bit, but it would need much more than 24h to have a real impact. If their service is still down in a week, now we're talking.
Now everyone's talking, because that will be the main option without email.
Yeah I mean a lot of people are nonchalant about it here, but email for business is like telephones. Basic infrastructure that has an assumed near-zero downtime and expectations of fault tolerance.
It's such bedrock infrastructure that I've seen it used as fallback in the event of electronic trading infrastructure outage, brokers would accept specifically formatted emails as orders.
At this point it would probably be less impact to have phones go out globally because people would fallback to walking across the floor, using zoom/slack for simultaneous communication or email for more async.
Felt bad for the engineers that it had to happen on a Friday but for a multi day outage I guess it's a blessing in disguise.
Barring corona & year end holidays, this is probably the closest we can get to a global work holiday.
Even consumer Outlook is down.
This means you cannot do anything that requires email auth (OTP, signup etc)
Email OTP is insecure anyway.
consumer outlook aka send your pw to M$ cloud outlook
it happened on Thursday afternoon US time and it is still going on since then, Europe just joined this morning (unless someone worked yesterday late evening)
Call it a long weekend, then :-)
My employer switched to Google this year. I’m sure someone is feeling proud.
> ... but email is a backbone of the modern world
That's why most don't realize how big GMail is. Google has 1.8 billion GMail users, including a great many who are using their Google email professionally (no matter if it's a Google Workspace with their own domain or a redirection or just a plain GMail). Apple has 1 billion email users.
Microsoft 400 million and, granted, many businesses but it's not a fourth the number of email users GMail has.
And many, many, many people (real estate, delivery drivers, Uber and the likes) are using either Apple or GMail email from... Their phones.
It's a good thing it's only the "small" Microsoft email users base that is affected and not the 1.8 billion GMail users.
Who wants to bet that this is ultimately due to AI, either an action taken by AI or just general AI code rot coupled with developer knowledge decay?
I would bet my lunch that the failing code is immediately attributable to AI, but the root cause is a failure of hiring, leadership, accountability, QA. If it was bad AI code, it was still deployed by a bad developer. The buck stops with us humans.
>The buck stops with us humans.
It should stop at the top of the organisation. If people are getting a directive from on high to go all in on AI then what are they going to do?
The problem is that AI can generate thousands lines of code....but I can read only few hundred a day with full focus.
It is much easier to read the code you wrote.
I know that we like to pretend otherwise but leadership are humans, too :-p
Or layoffs attributed to AI?
Or ai-agent-assisted attack, possibly directly using the hijacked infrastructure for inference.
They are just vibing a fix for this right now... any minute now!
[dead]
How many Microsoft employees are working on this? It's concerning that one of the biggest, most valuable companies on the planet, that runs services internationally, can have one of its primary services have gone down for this long.
6 hours ago they seemed to not exactly understand what the problem was, and that was 6 hours into the outage.
This is concerning.
Not so long ago, I was assigned to a project that had to be built on Azure due to client requirements. Because we were underhanded and it was an MVP, we'd elected to use PaaS as much as possible: Postgres Flexible Server, Static Web Apps, Easy Auth, etc. On paper, these all looked fine. They had lots of documentation, example projects, GitHub repos, blogs, roadmaps, etc. Then we started building and running into issues, and we realised a bunch of these offerings were basically unmaintained. There were features that had been stuck in "preview" status for literally years. The blogs and roadmaps were no longer updated, and issues were piling up in GitHub.
I started digging into the historical blog posts, GitHub comments, etc. and I realised everybody at MS who'd been working on them, SWEs, PMs, architects, etc. had either left the company, or moved to other teams. And the one constant about the internal moves was that people were now working in some "AI" related project or product. Literally, I didn't find a single person who wasn't now working on AI.
Clearly, there has been a huge internal push to get into AI. Resources have been reallocated, and anything non-AI related has been starved. And since working on an AI project is probably considered the only route to advancing your career, anybody with any ambition has moved to a team doing something AI-related. The result is that huge numbers of products and teams doing vital but non-AI related work have been hollowed out, and now we're seeing the consequences.
This state of Azure saddens me. I learnt everything cloud with it. I've built many things (big and small) in it. It was the one part of Microsoft I thought they'd not neglect.
Lots of its features - App Services, DNS, managed DB, Front Door, Blobs, Durable Functions, Service Bus, App Insights - are very powerful. The frontend portal is mostly fine for management. The Az CLI makes it easy to provision things quickly. Coupled with .NET Core on MacOS, I've not needed to touch Windows locally for years.
I could learn AWS or GCP, but that's just going to cost more time when I'd rather use what I already know to build more things. With Azure, I'd hate to see the rot set in further.
That's related to something I realized which I think a lot of people might find amazing. They think of companies like Apple and Microsoft as trillion-dollar, globe-spanning companies, so it must be massive teams that work together on everything.
But no, read Hacker News or other niche places enough and eventually you realize a lot of the things we all know or use was a solo effort or a very small number of people, sometimes not even an assigned project but something they did on the side. Cue the xkcd about the lone open source project maintainer holding up the entire modern web but apply it to the tech giants and there you go, it happens there too.
https://news.ycombinator.com/item?id=50250937
Let me put this way, the Visual C++ standard library has officially one FTE working on it, everything else are open source contributions.
WinUI 3.0, WinAppSDK, and related CsWinRT, C++/WinRT are in a ridiculous position six years after Project Reunion announcement, due to ever meagre resources.
XBox app keeps under delivering user experience, and bugs on patch Tuesdays are getting a joke by now.
All this in a four trillion valued company.
Maybe acting like jerk-offs got them to four trillion but there are consequences to burning all your social capital. A large camp of software developers have genuinely hostile feelings towards Microsoft, I would bet that this affects the quality of talent they can hire and these are downstream affects.
Lately it seems like more non-techies are souring on the company.
>A large camp of software developers have genuinely hostile feelings towards Microsoft
which, besides hatred, manifests in malice, like Nightmare Eclipse. I wouldn't be surprised if this outage was an engineer leaving a parting gift, using a bug they probably knew of for years but management didn't allow them to fix.
Wow Nightmare Eclipse is fascinating. It sounds like he is operating in a gray area and could be charged if Germany really wanted to?
Agree. I was having suspicions around that
Tellingly, I will not go to MS for UI stuff. When I have to anyway, I use their decades-old APIs. It is a canary with regards to microsoft, that they have no known champion for their UI stuff, no trusted face for their UI stuff; the company all just appears to coast along on "I assume someone is handling it?" The main microsoft employee I trust is Raymond Chen, they have no current generations of Raymond Chen / that we know of.
talent they can hire? large swaths of the company have been in a hiring freeze. they just sent 25% of the company voluntary severance offers.
they've still been hiring, just on an attritional basis - headcount has remained flat for a few yrs now. it is pretty clear though that they want to operate even leaner.
I've been hearing that since 2002, at least.
Enterprises have been failed time and time again by poor MS products, but techies will steadfastly refuse to move off it, so the business has no option.
> Enterprises have been failed time and time again by poor MS products, but techies will steadfastly refuse to move off it, so the business has no option.
My suspicion is that lots of the companies are so attached to all the MS stuff (and that's what their staff know), that it would be an incredibly large lift for them to move off it.
Like, I'm at a very windows/azure focused shop now (for the first time in my career), and literally everything is already on MS, or moving towards MS.
There are some good reasons for that, and some bad reasons, but I suspect that most of the sales process happens at the executive level, rather than the technical level (and Excel runs the world, so why not let MS do everything else?).
Jerk-offs can skim the profit of going big yesterday while the consequences affect everyone, but only in the future of yesterday. Who does such things, one might ask? Jerk-offs, off course. And PE too (likely not a disjoint set).
Was there ever a time when this was not true? M$ was always among top hated companies since 90s, arrogance, fuck-U to users all around, absolutely ignoring any standards larger than them, and famous 'Embrace, extend, and extinguish' which was pathetic to watch for any engineer worth 2 cents.
Top notch user experience peak was Windows XP and every subsequent release was massively larger, clunkier and slower while requiring way more resources, was spying on the users more and more. Office has similar story, peak of usability was maybe 2007 with ribbon introduction (which still a lot of power users hated with passion since it was far from clean design, more seemingly designed by committee of marketing guys).
>the Visual C++ standard library has officially one FTE working on it
One guy but it's Stephan T. Lavavej the guy who the "STL" was literally named after.
I think STL was developed before Stephan T. Lavavej was born.
This is what I don't get about big companies and their software; you'd think every application, every moving part would have a (small?) team of core maintainers, people that get paid well to know everything there is to know about something, ranging from (in microsoft's case) Paint to a particular configuration panel.
I get that this doesn't scale though - a lot of features are just "done" at some point and keeping one or more people whose sole job is to maintain this may lead to them bolting on features just to stay relevant / stop being bored. But big tech can easily afford to keep people like that on.
(I mean they keep (kept?) tons of middle management on, who knows what they even do? Not emailing right now that's for sure am I right?)
> every moving part would have a (small?) team of core maintainers, people that get paid well to know everything there is to know about something,
I work at a company with a few thousand employees in the Netherlands, and have been moved around a few teams in the last 7 years (not by choice, the team is nuked and I'm shuffled off elsewhere).
Technically, the product is supposed to have a "handover" to another team who can do the maintenance. What happens in practice is the person who worked on it gets saddled with maintaining it forever along with taking up the responsibilities of the newer work that comes along their way.
I've accumulated a string of such projects, including one that's used extensively by a few large companies, and the only person maintaining it is drumroll ... me, in whatever time I can spare for it.
Companies just don't want to compensate people to hang around and maintain a system, they want a consistent output of widgets that can be measured. Somehow our work culture is still stuck in factory worker mentality despite being well into the information revolution now.
Mythical man month, maybe throwing everybody in the company at it isn't the right thing to do. But they must be in a panic, and their own communications internally may well be hampered, could be MS everything is out for MS as well as everybody else.
Someone's going to have to write the Mythical AI Month after this.
If you lose 90% of developers to AI, but everything breaks, how much money have you saved?
The underlying moral issue is that the management and money people have contempt for people who don't manage and don't have money. Which includes customers and engineers.
It's not just an inhumane system - although it is, that, brutally - but it's also technologically dysfunctional and ultimately unstable.
"This is on the money!"
Sorry, had to.
I've seen the response to an emergency be "get every dev in a big room with some non-technical middle managers to direct them, and have everybody talk at once". My jaw was on the floor.
"When in danger, when in doubt, run in circles, scream and shout".
In our field, once you understand what the problem is, the problem is usually solved. Unless it's broken hardware, obviously
But Copilot is down too, so maybe they're screwed? :)
At least 1
The ones who actually knew how the system works were too expensive to retain, so now it's just some interns typing frantically into Claude Code.
"Too expensive" as measured by bean counters that don't understand the actual risk profile - or do understand it but have such a short window of strategic concern it leads to this. It would be an interesting analysis to stack up costs to Msoft vs cost of labor cut as this outage evolves.
Company policy is to use copilot but copilot is down so they are just hitting F5 and hoping
Haha this is ridiculous and yet there is some truth in it.
It may not be true now but 10 years down the line? This is disaster in the making yet while most people recognize that (when filtering out most fanatical pro-ai folks claiming everywhere how they get 10x or 100x work done) nobody is able to do anything. Sort of legacy technical debt being delivered but of different kind than usual, that train that will hit you since you stand on the rail line yet its moving 5kmh and you see it from 200m, but unable or unwilling to move away (Austin powers scene with steamroller does it better than me [1]).
Imagine if say BMW decided they don't want to manufacture their motors, suspensions, drivetrain etc. Outsourcing 100% of that knowledge to external suppliers, for all models, from now on forever. How do you think market would reacted to such move ie by valuation of stocks?
[1] https://www.youtube.com/watch?v=y_PrZ-J7D3k
When you outsource everything, you’re not relevant anymore. I agree on that, anyone else can do it just as good as you.
But the biggest fear is how is ai going to improve then ? We’ve seen it can’t feed itself on data it produced, the model quality decrease. But if there is no more human to produce ever better quality code because they rely on llm to generate code, we’re going to reach a wall.
Talking of car manufacturers:
https://en.wikipedia.org/wiki/Jaguar_Land_Rover_cyberattack
Now imagine the consequences if this was the main supplier of insulin.
Given how unreliable and poorly engineered modern BMWs are the market might react positively
This should be the top comment, my eyes are watering to this one
Oh shit: https://status.claude.com/incidents/nn11wk485zfn
Microsoft can't fix Outlook because they don't have Claude, and Anthropic's employees can't fix Claude because they can no longer log in without Outlook?
Ah, 'The Gift of the Magi', a classic Christmas tale.
> How many Microsoft employees are working on this?
all 3!
They're more effective than ever because with this outage, every manager above them can't email to ask for status every 5 min
Alas, Teams are working, or at least the chat and voice call portions of it. So managers are in a calling and chatting frenzy right now, pulling every body else in.
search in teams doesn't, very convenient for our Windows team to be unreachable for those of us still on WFH
Fortunately they can still ask on Teams!
This is HN. We should be asking why frontier models couldn't still dolve this! How many parallel agents does MS have working on this?
How may Microsoft employees are left anyway?
The lights are on but there's no one home
Your mailbox is not your own
The wheel is still spinning but the hamster is dead.
I guess the old light bulb jokes about MS will not be applying anymore
I wonder if it’s an AI death spiral: “the service is down. Diagnose the failure.” … Hello?
/s Their internal AI is full steam on it. The issue is just that their internal AI infrastructure is down too, so AI just need to fix itself first. Then it will focus on fixing the issue. Oh, wait...
And which made a load of staff redundant in the last month.
211 muted people on the Teams call, 2 guys trying to figure it out, an incident manager asking for an ETA and status update every so often.
a usual sev0 icm call on a friday evening.
the vibe coding group in charge of this product has decided to delete the production database and backups
I don't understand why they haven't asked copilot to fix it yet, make no mistakes.
Don’t be silly, they tried that but copilot is/was down!
That’s why it’s taken over 12 hours now, no one’s left that knows without copilot’s help.
[dead]
Maybe Nadella is still trying to figure out which of the 8000+ sacked employees knew how the system works /s
he was such a nice and effective CEO the first few years. he's really changed.
Was he? I struggle to see it even in hindsight, and definitely his moves were bunch of FU to common users. Maybe in some specific area you care about, but not their core products like windows and office
Not really. He killed the QA department immediately and consciously put MS onto the "slide into mediocrity" path.
Google killed its test engineers (or, rather, forcibly merged them into SWE) at around the same time without measurably denting uptime.
Can't have downtime if you kill everything you create before it can fail :)
Yet.
This happened in 2015.
Even with some of the Microsoft <3 Open Source, I do miss the Steve Balmer era in regards to Windows development experience.
Executive compensation corrupts, chief executive compensation corrupts absolutely.
yeah very effective, he fired 90% of our Windows Store team (reviewing the apps) in the first year to replace people with automation, I survived up to the last round of firing after which they kept ~20 employees out of original ~200
Is for this reason that my email is hosted on a cheap VPS:
https://gioorgi.com/2020/mail-server-on-docker/ https://github.com/docker-mailserver/docker-mailserver
It is very easy to set up (just a bit of time on average every 6 month to follow upgrades) and work out of the box.
Europe outlook users are experiencing problems too.
When your most important contacts use outlook, it does not matter what you do. Sending and receiving email when almost nobody else is sending and receiving is only small help.
Some problems are not solved by individual choice. If 30-40% of your important contacts use service that is down, your ability to work goes down dramatically.
I have currently important 3 daily contacts down they use MS services extensively contacted and said they can't do much of anything because their work process and the process of their contacts is down.
Enough of my contacts use Gmail. But both of these have more uptime than my self-hosted email.
If the person I'm contacting is using Outlook then I assume it's going to be unreliable anyway.
Personal email is almost exclusively for OTP, reminders/invoice.
I dunno...
Wouldn't you then need to keep up-to-date backups but ensure the backups still stay secure? Learn all about and generate SPF, DKIM, and DMARC records but still regularly get sent to the spam folder or IP blacklists if your recipient has a big-provider email? Deal with self-rolled 2FA or passkeys and secure recovery methods. If you're self-hosting at your house, you're at the whims of nature, infrastructure, and creatures not disturbing the machine. You are the one patching security holes, or if you run someone else's containerized or easy-setup mail server, making the decision between daily updates that'll catch supply chain attacks and potentially introduce instability or new security holes; or monthly/annual/never updates that virtually ensure you have a few CVEs active at all times; or the lead maintainer quits/dies and you watch it deteriorate while users groupthink how to migrate their mail to a different container.
Sorry for the pessimism. I can't suggest a better alternative. I have one account on Gmail (for impersonal communication like shopping) and another on Fastmail and occasionally reflect on how annoying it will be to lose access to either.
>Learn all about and generate SPF, DKIM, and DMARC records but still regularly get sent to the spam folder or IP blacklists if your recipient has a big-provider email?
This is my biggest complaint when self hosting. You do everything right and they still send your mail to junk or just silently discard it while telling you 250-Ok for "reasons".
Although I recently had a legit Microsoft email also end up in the junk folder so I feel better that they can't even get it right themselves.
Microsoft have also recently kicked out self hosters from their "Smart Network Data Services" portal by only allowing the ASN owner (ie. your cloud provider) to access it.
There probably are some very good reasons not to self host your email server but apparently my own, self hosted, email server's uptime over the last decade is now better than that of outlook.com :-D
Quick lets collect all options people have to self host! in addition to docker-mailserver im aware of
- https://github.com/stalwartlabs/stalwart
- https://github.com/mailcow/mailcow-dockerized
I decided to self-host my mail a few weeks ago and ran through all of the big names I'd heard - Mailcow, mailinabox, Stalwart, and Dovecot+Postfix; honestly I didn't like any of them. Stalwart is very corpo, hard to configure (reading the docs[0] confused the shit out of me), and feels vibe-coded. Mailcow and miab push a huge stack onto you in a way that feels hard to modify yourself, e.g. expecting you to allocate ~2GB ram(?) to running ClamAV, and I never managed to get Dovecot+Postfix working properly.
In the end I paved over my Linux VM with OpenBSD and used Dovecot+OpenSMTPD which I found dead simple and so far faultless. I have nothing but kind words for it.
[0] https://stalw.art/docs/configuration/
That's fair. It is perfectly reasonable to want customizability if control is what you are after. I remember the days fondly where I was running postfix and dovecot on bare metal.
Nowadays my interests have skewed towards being able to have groupware features without much maintenance so groupware in a box suits my needs just fine. The learning curve to doing serious customization with a large FOSS groupware project might be a bit longer but that is offset by other factors.
Yeah I definitely get less done on OpenBSD, but I have more fun doing it, and fortunately in my position where that works for me. If I were providing a service to others instead of just myself I'd have to rethink my stack.
Mailcow's README is garbage. It tells me nothing about the software while asking for $50-100/m.
> Mailcow's README is garbage.
Geez, that's right! :-D
It seems to be a fairly popular project though, so what does that say about the correlation of README.md quality to success?
I have heard good things.
It says that whether or not the README is good does not correlate with the popularity of the project.
I have always felt it's less about email redundancy and more about having a backup plan. If outlook goes down tomorrow, does everyone know how to keep work and client communication moving? that is something worth thinking about.
Who wants to bet that Microsoft is using slack to communicate now?
Fire up that old IRC server!
You joke, but during my time at Netflix there were at least two layers of backup communication systems: if slack went down, we would jump to google chat, and if google chat somehow went down too, there was an IRC server and a (hardware) phone bridge. I never touched any of it because I wasn't an SRE, but I did read through the training one day because I was bored.
[dead]
There was no need to abandon perfectly working technology, with pain some people might learn this. Departmental servers, pesky lordly sysadmins, this was a working sustainable system that even scaled to the phone company and US government levels.
For the amount of uneeded emails about future emails about other emails, probably no real work was lost.
Just got up, watched BBC on treadmill. Everyone talks about how this is a phenomena of modern life, that the only other option is paper. Microsoft is winning. People cannot envision an alternative. But outlook is not email. Just use someone else, anyone else.
My email is still working, has been solid for moe than a decade. My calander app opened as normal this morning. My chat windows are all alive. My digital life is just normal today... with rather fewer emails in the inbox. DROP MIRCOSOFT. Pick up linux. Pay for a proper email service. There is a larger world out there!
This is going to cause absolute chaos. Basically every government organisation and business in the UK runs on outlook. The whole day is going to be a write off. It’s like a bank holiday!
Same is Aus. Wife works for government, they got sent home early (typical gov that means 30 minutes before official clock off)
The biggest issue has been so many security and approval flows depend on email, so all processes have ground to a halt.
We're coming to the end of the Friday 13th business day in East Asia. Can confirm it caused chaos for the whole day.
Just the UK? Lol, you can add many, if not most government and private companies and educational institutions in the US and countless other countries to that list. The government and corporate world is grinding to a halt.
This observation applies to many other countries, too. Mmmm centralization is pretty fun!
Now's the time for those NHS fax machines to shine, I guess.
Citation needed.
Government I have no idea, but loads of businesses don't use Outlook - alternatives do exist. I've personally not used Outlook for well over a decade now.
Quick google AI mode check says:
- MS 365/Outlook: ~40%
- Google Workspace: ~25%
- Zoho: ~2%
And then a bunch of others - Rackspace, Zimbra, IBM WebSphere Lotus Notes Blue For Enterprise XML Watson Edition et al
Depends what market you are in. In the UK I would estimate M365 usage is ~100% of local/national govt.
And in terms of classic 'corporate' sectors like law, finance, etc, it also close to 100%.
Whereas in "tech" it is far more likely to be Gmail.
I _still_ get people who are non technical that work in 'corporate' sectors that can't join a Google Meet because they think it is Teams. They literally haven't ever used a Google meeting before.
Every government globally with all parts and agencies is purely microsoft. Some EU states are trying to move away from it for obvious reasons (not this event, more like national security and sovereignty which you don't have with MS products).
Exchange and Active directory are the biggest obstacles by far, bureaucrats can learn how to work with open source office alternatives easily, and underlying OS can be anything as long as you have few icons to click on.
> Lotus Notes Blue
I believe nowadays the general solution is HCL Traveler (server side) wit HCL Verse as client. The latter offers email access over pretty nice looking mobile, desktop and web apps.
Rackspace?
> Zimbra, IBM WebSphere Lotus Notes Blue For Enterprise XML Watson Edition et al
I see that you're a hater. Outlook sucks but your solution is to cure cancer (Outlook) with Alzheimer's (Zimbra, Lotus Notes)?
Good morning and happy Friday the 13th everyone
Having friends who work at Microsoft, this is not really surprising at all. The company has always had deep structural problems. The AI change has really just exposed it and is aggressively accelerating the inevitable implosion. It's truly in a free fall that is planned and orchestrated perfectly by their upper management. Their CEO and VPs are convinced you don't need engineering know-hows any more because of AI, and they got all the management layers to sincerely agree with them. Not in a malicious or subverting way, but in a "you can't get a man to understand something his salary depends on him not understanding it" way.
Microsoft has a moat, but their moat is having billions of dollars to spend on data centers. It's a failure everywhere else. It'll continue to rake in billions in revenue because people need compute now, but it's rotten at the top.
Why would anyone introduce a single point of failure out of their control into their company?
We're doing anything to avoid single points of failure in our products, but our IT is doing the opposite.
Outside of tech and disaster recovery, people _really_ don't understand that some redundancy--in this context--is a good thing.
The idea of paying for a backup system, or even having one on standby for situations like this is just not on people's minds, or if it is, it's simply not affordable because it would cost you too much compared to your competition.
Razor thin margins and generally getting away with it is what makes this work. Until it doesn't.
Or, they get the bad idea that they can just pay one supplier and that supplier will deal with the redundancy for them. When the redundancy needs to be at supplier level.
Efficiency is brittle.
I don't see a reasonable alternative here. Pay for another email/office suite provider, and duplicate all mails and files? Twice the cost and exposure surface just to cover 0.01% of downtime.
Sure, it makes sense for mission critical companies (utilities, emergency services), but I don't think it does for an average company. Especially that you don't take much blame if everyone is down.
Email in a large part is just a tool for communication(and mostly low priority at that). Where I work, we have backups: Teams( same) provider, Cellphone ( different provider ) , Landline( different provider)
We had a situation recently where cellphones were down, we switched to landline.
You don't need a 1 to 1 mode backup for everything, just a failover plan.
Unless your company is infrastructure or medical, I doubt they have a backup/failover for loss of power. Or water.
The difference between a distributed system and a monolith is that in a monolith you have some idea what your single point of failure is, but in a distributed system you don't know your dozen single points of failure until they happen in production.
Example: in 2020, a 6-minute hiccup in a network switch caused asymmetric connectivity, breaking their distributed leader election algorithm, making that system read-only, breaking a different system's heartbeat mechanism causing that system to assume a failure, enact a recovery and degrade in performance, causing 25% of API requests to fail for 6 hours.
https://blog.cloudflare.com/a-byzantine-failure-in-the-real-...
I love the dependency cycles that prevent things from being started up again from scratch.
Google had a secret storage system back in the day, that required special encryption keys to be entered when it got re-initialised. The keys were in a safe. Guess where the combination to the safe was stored? ;)
Yesterday I was attending an offline course that ended slightly before this outage started.
The final thing we did was photo a QR code with our phones that took us to an Office 365 form to evaluate the course.
I was surprised to see some fields on my form showing values that my course comrades had entered on their phone.
Could the course evaluation have been incorrectly developed and deployed using AI on Office365 by the course developer?
Or was some session state all messed up across parts of Office 365?
I was in a hurry so I decided to come back to check on this form later but now it is just stuck ”Loading…”.
All 3 of my outlook accounts are still dead.
I am looking at other current events and beginning to wonder if there is some kind of correlation.
Given the great options nowadays it might be smarter to self host email.
See: Stalwart Mail Server: https://stalw.art/ Bulwark Webmail/Groupware: https://bulwarkmail.org/
https://thenational.co.za/news/2026-11-13-cisa-confirms-hack...
This is not related to this outage, right?
16 hours and nothing...
Sudden blocked hotmail emails from sudden required DMARC DNS records and now this days later?
Jeez
Note: https://news.ycombinator.com/item?id=50416158
What I hate most, is that fucking piece of shit teams was still working, so I had to take meetings and calls.
Microsoft imposes all of this shit on the world. Poorly thought out products, buggy crap. Its finally come home to roost.
Might be a coordinated attack against the network providers that MS uses? MS: Our investigation determined that one of our network connections to a third-party ISP was not taking traffic. We've diverted traffic to alternate channels to expedite recovery. For more information please follow MO273940 in the admin center.
Other source: This is related to a BGP hijack of at least 87.128.0.0/10 belonging to Deutsche Telekom (AS3320) hijacked by AS212416 (Proximity, Bulgaria) passing through AS57344 (Telehouse EAD, Bulgaria) starting at 0633 UTC today.
Your other source quote is a direct copy of a tweet from 2021: https://x.com/rrbone_net/status/1420656486573805570
You don't think if it was a DDoS attack they would've identified that by now?
This is also not the normal type DDOS. They are not being spammed. Instead the connection to the ISP is being disrupted.
This is from MS themselves: https://x.com/MSFT365Status/status/2121079061774491735
That says nothing about DDos..
Something I've been thinking about with the last few major outages is that as there is more and more consolidation (fewer large companies running a higher share of the world's infrastructure and critical services) is that we become increasingly exposed to massive disruption like this. Would some sort of mandated maximum occupancy per instance resolve this? So Outlook.com is allowed to support 100m accounts per instance. Then you are required to spin up Outlook2.com. You are not allowed to have any shared infrastructure. Yes it's wasteful and expensive, but these are companies with billions in revenue, so that's the price you pay. It just seems that keeping these huge, incredibly complex services running is turning in to an impossible task. Very much a hot take so interested in the opinions of peope who have worked at this scale.
They could (and should, and clearly don’t) architect the services like this internally. There will always be some single-points-of-failure, but minimizing those should be a top priority. Just from reading their status updates it’s clear most everything is sharing way too much in terms of infrastructure and configuration.
But even this requires someone high up in the organization to have the insight that reliability requires some level of “inefficiency” — ie retaining certain highly paid staff, duplication of effort, over provisioning of resources, etc.
The irony is that the execs who could make those decisions are convinced that LLM tools obviate those needs, and shift all spare resources in that direction. But without heavy constraints from experienced humans, LLM tools will only multiply the complexity at the root cause of these sorts of outages.
I am positive there are bare-metal, no shared tenancy, customers of MS already. Unclear if they are down too. While yes, these companies are massive, not being allowed to share infrastructure defeats the entire revenue-purpose of offering a cloud. The hardware in these servers is unbelievably expensive. You'd go back to everyone having an IT department and a few racks of servers for hardware configurations that meet their specific needs (which I'm totally okay with, and even rooting for!)
I'm an outsider who used to work at a different cloud provider, so take this with a grain of salt, but this doesn't smell like a limits and provisioning problem, per se. It's not necessarily like they just had 1 too many emails tip the scales for the entire world. Until today stuff was running okay. This was a monitoring and fail-over problem, at a comically tragic scale - a cascade failure. Architecturally they obviously did not design for a world where they could damage control failure at this scale. Failure should be expected and designed for at literally every layer. They cheaped-out on careful design in order to play catch-up building a public cloud offering and here we are.
Self-host your email.
That would be a good idea, until you realize that everyone you send emails to (in a business context) also uses MS 365.
Most emails in a typical org are internal.
So what?
Good luck getting your self hosted email trusted by anybody.
It's trusted by everyone AFAICS - including outlook.com, if that's still a thing
Self-hosted email is trusted? Not in my experience -- maybe I had bad luck and a small sample size but I quickly abandoned it after a few contacts I needed to be able to e-mail didn't get my email. And generally it didn't even go to their spam box -- it was filtered from even getting that far by their provider.
I was using a dedicated VPS, maybe that was part of the issue, maybe self-hosting from a residential IP might work better. But it was easier for me to let iCloud handle my domains email. It's been flawless ever since even though they're not really a big player in e-mail. Used Proton for a bit, that was (almost) flawless as well, though even Proton occasionally gets blocked by stricter corporate rules.
did you do all the things? DMARC and what have you? Super easy in LLM land but wasn't hard even before it, just had to set aside 20 minutes to read and work through various guides, then maybe another 10 or 20 minutes in the subsequent weeks to confirm function.
Big tech are trying to take this option away from us.
I find it interesting that the stock has barely taken a hit from this outage.
Monopolies can provide a bad service and not lose customers.
Anthopic (unrelated to IPO announcement)... After 24 hours we directed our resources towards diagnosing the MS Office365 outage and 15min later informed them of a solution.
Instead I'm wondering if someone at OpenAI and Anthropic had a look to check what their agents were up to, just to be sure.
lol surely
..but why is it so centralized? It makes these events so final.
MY University's whole email system is Outlook/365, and that also seems kind of dumb to me.
many universities bought into the cloud offering back when it was still live@edu, shown as an upgrade path for existing on-premise dovecot/exchange systems. back then a student would be lucky to get more than 100mb for their mailbox, and maybe a gig of personal file storage (which was only accessible on campus or via a vpn, the latter many would not have access to). microsoft just offered a _better deal_ at the time
frog boiling means that many never noticed how big (and therefore unspecialised) microsoft was until it was too deeply integrated
It's much better UX if everyone within the company uses the same office suite/email, it makes collaboration much nicer when everything is well integrated. And people get used to one system.
My guess is that having duplicate system would cost (in money for double subscriptions and lost productivity) much more than the benefit of being available 0.01% more. Especially for universities where almost nothing is really urgent and can't be delayed by a day.
Except... Outlook also has some of the worst email UX I've ever experienced
Why is no-one mentioning DNS?
Because it's not DNS related.. you can still resolve it.. and sometimes reach the frontend- but they can't route the requests to email servers because they are being "overloaded"
Because MSFT is explaining it as a rogue process overrunning servers after start up.
[flagged]
[dead]
"Okay? Nothing. It’s just terrible. Every day is a nightmare. As recently as 1972, most buildings collapsed and rusty nails were found in cereal boxes all the time. Okay, so when you’re faced with statistics like this, it’s very difficult to be optimistic. So, hey, why not think about the statistics? I want to just put that out of your mind."
https://www.centennialsoftwaresolutions.com/help/transcript-...
https://xnotme.com/MSFT365Status/status/2121079061774491735
The amount of racist memes targeting Indians on the replies.
it's twitter, what do you expect?
[flagged]
Really? It's not very funny and mainly just white supremacy.
More like hate on white people.
They absolutely did hire anything but.
[flagged]
Care to explain the humour in the 'joke'? It is obvious that the statement is not earnest, but it is framing its criticism in potentially the most racist way possible. It wouldn't be particularly funny even without that.
What an excellent question. I am no comedian, but let me try and break it down as best I can.
What this joke is: Crude, absurdist satire of corporate DEI culture and cheap "offshoring" / importing of cheap labour.
Why it works: It packs quite a lot into so few words:
a) It hijacks a polite IT support cliche "Have you tried... (turning it off and on again)", setting the reader/listener up with a familiar rhythm.
b) It then seamlessly pivots into taboo-breaking; just like South Park (for one of a zillion examples, this stuff is literally older than Shakespeare), the "saying something you are not supposed to say" is what draws laughter.
c) It ends having made a good point: an aggressive focus on profit, rather than quality, has increasingly become one of Microsoft's quintessential characteristics. The outage is unsurprising in that light.
I don’t know why I need to explain this, but racism isn’t limited to white people.
They used AI to create something they didn't wholly understand to debug.
In their defense it was probably insanely fragile prior to the AI assisted coding boom.
Given the nesting of services at MS, the fractious org structure, and the length of service of some of these systems it was already a bit of a tinder-house.
AI just provided the accelerant, and while I don’t work at MS I can imagine they’ve lost some serious tenured talent over the last 5 years. I’d be surprised if anyone is left that actually understands how it’s all glued together.
Or rather, that might be the case before this outage. I suspect the org has relearned a fair bit over the last day.
26 Oct: Microsoft offers voluntary buyouts to 60,000 employees as AI leads to efficiency push
It's just a wild theory, but maybe there are still too much employees left, standing in the way of AI? I would try firing another 60,000 first.
No one knows if this is AI related, but it's easy to imagine that where changes written by AI fail in production, if AI can't find the solution (which will be common as AI access to production systems is typically reduced), then the devs won't either, or at least it will take them some time.
Perhaps the velocity is the problem, not that AI inherently produces worse code (you just need one bad or just inexperienced engineer have their PR slip through to have bad human written code in you code base). When the velocity is high, there's less time to battle test each change and new code piles up on the old one. When the velocity is low, shitty code fails in production, but people are aware of what was recently changed and can find and fix the problems.
Same thing, now, just faster: one bad engineer can now ship a million lines
Well, yes. My point was more that even if the quality was better, the speed of change is a problem. In particular in mature systems, I should add.
I think this makes sense. If I ship 10 lines of code to production and something breaks, it's easier to reason about than if I shipped 1000 lines of code, even if I did read and understand them before shipping .
For some reason people seem to think that just because AI exists now, code no longer needs testing thoroughly.
Yep, and having a ton of AI written unit tests pass is not the same as running incident free in production for a month.
The bigger issue with having your systems written by AI, is that nobody knows how the system works anymore and can't pinpoint where the likely failure points are.
Maybe the error itself isn't event AI related, but the engineers that knew how to fix it were let go
The status updates have a pretty strong "LLM-driven agent failing to debug a complex system" feel to them. I hope it's not the case, but I could imagine a scenario in which inexperienced engineers - or experienced ones who have lost touch with the codebase - are deferring to LLMs to drive the troubleshooting and fix here.
They sound exactly like frustrating conversations I'd have with a cheap model and an overextended context window.
Ugh. My sarcasm detector is down. Shouldn't have hosted on MS servers.
Who needs senior engineers. We are going to have LLMs troubleshoot billion-scale systems!
What could go wrong, said all of the hyperscalers. Enjoy your multi day downtimes.
I do actually believe that LLMs and process around them will continue get better and become the accepted, but that there's going to be a 'fossil record' of a lot of problems having been seeded in these early years of AI that will take some time repair.
> We're aggressively pursuing all pathways to reduce impact
I love this kind of wording. It makes everything suddenly more serious lol.
What a fancy way to say, "we highkey have no idea why it's broken"
It will be interesting to know if the loss of codebase understanding due to agentic coding is the primary cause for the slowness of their response.
If it is indeed the case, this might be the first clear "win" for those who have been warning about the dangers having blind faith in AI and of pushing poorly understood code into production.
At least Microsoft 360 still has a nice sound to it.
No Outlook, they should make this a public holiday!
Its microsoft 360 not 365 (days in year). So they have four more days.
If Teams is down I take that as a service upgrade
Alas, Teams is working, although parts of it which are integrated from other services are not, like calendar (exchange), or file sharing (share point).
wishful thinking from my part
This is very serious: I cannot access Viva Engage.
TIL that Yammer turned into that
What turned into MS Teams then?
Didn't they turn Skype for Business into Teams?
The repeated comments on their twitter feed are weird, is it an LLM writing it? why can't they just wait until its fully fixed?
Regular communication is an important part of outage/crisis communication, even if that communication is "nothing new to share".
It's actually nice to have some updates on what they're doing that go further than just "we're investigating."
[flagged]
This is why one signs a contract with Microsoft: no-one gets fired when they have a twelve-hour outage. You sign a contract with some unknown company and have a six-hour email outage, you had better start packing up your things.
Nothing could go wrong if you force/bind millions or billions of people to a single point of failure. Of course, I doubt Microsoft ever will have to pay a single cent for lost work hours/ money across the globe. They probably have a fine print against that.
Hopefully some of their "customers" learn a lesson and switch to a software without this side effect in worst case. There's plenty of alternatives (LibreOffice, OpenOffice, etc.)
Ho-hum, my LibreOffice on my local machine hasn't skipped a beat all day!
Isn't there an old saying about putting all one's eggs into one basket?
Well then your one basket is your machine, or your backup system. And Office and Onedrive still work IIUC, so you're not really comparing apples to apples.
" 'and' your backup system(s)." I've many.
It's simple really, I've no cloud accounts, and even my Android phone is de-googled. I'd never trust any data to the cloud and least of all to MS.
What's happened here with MS clearly illustrates one of my reasons. I find it very strange that so many use Onedrive, Office 365, etc. and trust them even though they've not one iota of control over their operation.
As we've witnessed, what they're doing is strategically very risky.
Correction, I think it is now Microsoft 364 ;)
Why did I scroll down to see the comments? I mean what was I expecting except a torrent of racism?
From where I'm sitting, everything is working but Exchange Online which is absolutely fine by me.
If there was just something like E-Mail or Matrix free from Microsoft dependency
Funny, my self hosted email has a better uptime than this pos
Email is actually designed with intermittent connectivity in mind. The sending server buffers outgoing emails until they're accepted by the receiving server. Typical settings are to retry every 4 hours, with a permanent failure being counted after 3-7 days.
Work's outbound email server for server/API/website emails (like password reset requests) has 127 emails in the queue, all to MS365 users.
Some of the big mail senders must currently have enormous queues of delayed messages.
I would take 99.0pct self hosted over having to use outlook!
I’d take carrier pigeons over using Outlook.
I hope it was not caused by AI generated code
I've heard enough, fire everyone
“Co-pilot, fix the outage. Make no mistakes.”
Welcome to Outlook/364
If only this year were a leap year Microsoft would be in the clear
Just by looking at the O365 UI changes they have been doing tells me there’s a massive competency crisis within that team.
It would not surprise me if the entire project is complete and utter (human-written) slop
Token maxxing can't help huh ?
Their current engineers must have stored their KT notes from their laid off and leaving senior engineers in OneNote on their Mac, and might have been working with copilot to find the root cause /s
[flagged]
[flagged]