It doesn’t seem like a cloud provider was at fault:
> “JPMorgan Chase has experienced a temporary issue related to a software bug in an on-premise hardware system — it is not cyber-related,” said Joe Evangelisti, spokesman for the bank.
It's wild that they pay like shit and attract 2nd rate programmers.
Eh, actually, I don't know if that's true about 2nd rate programmers, in a lot of ways they're probably better, but they do get paid less.
I realized this while working at a startup. The startup had a webpage that worked half the time--that's it, that's why we had funding. Well, that and the founder was going around telling investors we had fancy AI algorithms, which were actually mostly manual human work.
Meanwhile my friend working at the bank made a lot less.
I’ve always found it curious that FAANG/California-based employees receive comments of “hugs” from HN during periods of downtime, while those outside of the bubble receive only mild slander. Of course, no suggestion that you specifically have engaged in the former.
That will work for the first week to month of a disaster. Once people realize it's bigger then dollars will rapidly lose value compared to direct barter for useful things.
I don't agree with OP's point, but comparing the damage to the dollar that a localized hurricane and an extended national bank-outage would have is not an apples-to-apples comparison.
In a hurricane, the dollar keeps it's value, as others outside of the affected region are still using it to trade. This may not be true in the event of a national/global outage....not to mention the whole 'trust in the USD' aspect
If the “entire monetary system” stops working, then things are fucked enough already that gold is essentially the same in terms of utility as cash would be, in that its only immediate value is its scarcity.
Things like food, water, basic sanitary items, would be bartered for.
It’s funny how people just automatically go down “I have a gun and I’ll be good scenario”, meanwhile every natural disaster in modern history suggests that in crisis, people come together to help each other instead of threatening each other with guns.
People often come together initially, but it's a dramatic simplification to think that that implies that after the immediate issues are resolved (e.g. outright starvation) that competing groups don't often become extremely antagonistic.
You'd still not want to be a lone wolf type with a gun, though, you really want to have a group in all cases.
I realized this watching Gaza. Here I am watching a people that more than a few voices would tell me are sub human, but every time I see video of them dealing with the destruction of literally every single building within sight, they seem to be acting like regular humans, not savages.
No doubt there have been some crimes and things done in desperation, but humans don't just instantly devolve into roving gangs that kill on sight.
The people coming together helping each other can also wield guns far more effectively than some lone thief, who will be universally despised and viewed as a highly immoral agent.
> people come together to help each other instead of threatening each other with guns
People, collectively, come together. There's always outliers -- we call them assholes. It's nice that some folks are so clear on what side of line they intend to stand ahead of time -- pay attention to what your neighbors are saying, so you know which ones are the assholes.
Preppers think they'll be the local warlord, but warlords rely on some loyalty, not being the weirdo loner sitting on a weapons cache with zero friends to guard it while they sleep.
They’re talking about valuable goods to barter with in an apocalyptic scenario, they did not say they would go raid the preppers. At least that’s how I understood the comment.
Exactly my point, which is why having the addresses of 10 preppers would be more valuable than gold. So you can give the addresses to some badass willing to risk his life in exchange for goods.
They are all deadly, but out of any long gun besides .22, .223 is pretty much the weakest. The entire benefit to using .223 or 5.56 is that they are light so you can carry more ammo to enable suppression fire after traveling 20+ miles on foot.
If we were to judge people by the guns they carry, I would rank ar-15 carriers the least worrisome. They would be far less likely to be a decent shot on average, they are the least likely to know anything about actual firearm tactics and try some hollywood or COD:MW run and gun, and will likely be the first ones shot when they decide some 70 year old farmer or hunter is an easy target who suddenly blasts them from a dark window or tree stand.
What matters is that its hugely common and works in easy to repair weapons. In an actual scenario where you would need such things, having a fancy ass rifle with only a few parts to fix it is fucking useless. AR-15s and 9s aren't the best weapons out there but they are ubiquitous.
I honestly do wonder, has there ever been a time where two parties have been in a confrontation and one party pulls out a 9mm, and the other party reacts to it like John Candy's character in National Lampoon's Family Vacation reacts to the pellet gun?
There are countries where the monetary system basically is cash (lots of people don’t have bank accounts).
There are definitely issues with that, but it does work in the sense that people can buy and sell things and survive, provided the government is competent enough to provide good quality notes and coins and resists the temptation to print too many.
After all it’s how western societies worked throughout human history until relatively recently
Because credit-card systems ostensibly don't rely on debiting actual funds, they should be resilient and continue to work offline; when you use EMV (chip+PIN) it's the card that authenticates you and authorizes the transaction, not the bank down a phone-line (and who can forget credit-card imprinter machines?)
I don’t think cash is gonna help if the entire monetary system stops working - gold maybe?
Yeah, I'll just get out my pocket knife and carve a little chunk off that big bar of gold when I'm trying to barter for gas at the truck stop. That'll work.
I would happily trade people for gold because there is no way at all they would ever get even close to the value of their gold in goods. You could probably get thousands of dollars worth for some beans by time they decide to start trading away their gold.
Doesn't help if the world truly goes to shit and doesn't recover for decades, but if there is any kind of recovery then the gold would be worth something again.
You can't send a single cent of your physical cash over the internet into someone else's bank account who is also experiencing this outage. FedNow is likely not working either.
On the other hand stablecoins such as USDC can be sent over the internet on blockchains such as Tempo, Solana, XRP and Stellar worldwide, 24/7 in less than 5 mins and with extremely low fees.
To Downvoters: Come on. You cannot send your physical paper dollar notes to someone else in another state or around the world in less than 5 seconds.
I'd be annoyed if I was a BOA or Chase customer too. But the truth is that blockchains such as Tempo continue to work anyway worldwide 24/7 without fail.
When do people start demanding some sort of recompense for outages, I think Microsoft will decrease the cost of your licensing if they break an SLA. How about $100/hr of downtown into each users effected account?
My nontechnical co-workers see the Microsoft 365 outage, the FAA ground stop, and things like this as one interconnected problem. They aren't entirely wrong, but the general fear is hard to dismiss
Interestingly enough it took about four generations, a century, for someone to go back in and investigate the aqueducts properly. I think a lot of this is due to previously easy, simple things being ignored, in part because they are considered menial. The basics of maintenance are easily forgotten.
Also undervalued by society as a whole and politicians especially.
They'd rather allocate funding for a new bridge they can point at and say "I did that" than properly fund the unsexy work of making sure the existing bridges are maintained.
Now apply that across most western democracies, repeat for decades and we are where we are.
The countries doing their vast infrastructure build outs will likely repeat the pattern in a few decades.
The other issue though is that with 24/7 connectivity, any unplanned downtime can feel like a major outage. 50 years ago there were lots of outages, to, but people didn't care as much, they just went outside.
50 years ago, an outage in Seattle didn't affect me in St. Louis as a rule. I'd go to the bank, hand the teller a check, they'd eyeball it, maybe hand me some cash, and process all the things on paper.
When banks started moving to computerized systems, maybe a bank would be unable to process something... but not multiple banks. I can't recall any "outages" that affected banks nationwide for any period of time. Walk into your local bank and find out they can't do something because their big system at the mothership is down? Perhaps.
Consolidation likely plays a part in this as well. I can't speak to the FAA/ATC systems 50 years ago, I have no recollection of what they were like then, but the only nationwide system I can think of around that era was Ma Bell. And I don't honestly recall any nationwide outages back then. Regional, sure. You might have a problem with part of the country getting calls if there was some kind of emergency... (e.g., everybody trying to call their grandmas in Kansas after reports of a tornado) but I don't remember there being any nation-wide problems like that.
> And I don't honestly recall any nationwide outages back then
There was this one in 1990 [1], but it's notable because the system was otherwise very reliable. Also, this report says only 50% of call attempts failed, presumably most people would try their call a few times and likely get through.
I would also say that there are more single points of failure on the internet than there were 20 years ago.
In 2006, Amazon had just launched this new storage service called "S3." It would have struck people in 2006 as bizarre to think that AWS (and Cloudflare, and GCP, and friends) would one day be so crucial to the internet that a bug there could take down dozens or hundreds of companies.
And for a lot of companies (I know this is true for Capital One), the sole redundancy plan for the vast majority of services when US-East-1 fails is to fail everything you can over to a different AWS region.
We still had a banking system, we still had phone lines, etc. But yes, the general lack of an expectation of always-on connectivity meant that service interruptions were much less of a big deal.
One of the coolest things I've seen, system design wise, was working at libraries when the tech/power is out for a given period.
Every service except digital lending/materials continued to function, and there were fallback processes for recording everything that happened so the information could be added back into the digital system once the tech was fixed/power was restored. Specifically designed forms that were easy to use and easy to scan, an exact understanding of what needed to be recorded and why and where it would go later, etc. And a staff of people who were meticulous enough to actually follow through.
I originally missed OP's comment about "a decade ago", but I don't think our "ability to get big things done" has changed in the last 10 years, nor the number of major outages.
When I hear people talk about "we can't do big things anymore" in the US, they are usually referring to mid 20th century stuff a la https://patrickcollison.com/fast, not stuff that we did 10 years ago.
The topic of TFA and this thread is that these major outages appear to be happening more frequently this year - or in the last month - than we've seen in the past.
I do not believe the FAA has ever grounded the entire country due to a network upgrade going bad. So that is possibly an example of a hard thing we could do a decade ago, that we can't do now.
> The thing done today weren't even computationally possible a decade ago.
Not the things that are breaking, no. The FAA was doing ATC a decade ago. We had email and cloud servers a decade ago. We had bank websites a decade ago. The scale of those things today is perhaps larger than a decade ago, but not enormously larger; indeed, the increase in scale in those things is much smaller than the increase in computing power in the world from a decade ago. So these things should have gotten more reliable--but they haven't.
we made these systems simpler to maintain, to reduce the amount of skill we need to pay for to maintain them. And that does reduce the maintenance costs, at the cost of more compute. but who cares about compute costs where compute per unit of money doubles every 18 months, amirite?
unfortunately it also increases the competence required to change these systems without degrading their performance and usability.
since no one wants to pay that extra cost either, we let the systems degrade a little bit at every change instead, hoping they degrade slower than hardware improves.
that bet was bound to lose eventually, because of laws of physics, and because that degradation pattern affects hardware too.
and the double whammy is - with little demand for expertise - the supply is effectively gone too. so now that the walls have closed in, there is no way out.
I've been working in software for 30 years. This is the largest collection of systems to be down at the same time in that period. I have been literally experiencing a degradation in online services over the last year or two. Its either poor quality AI generated code, or your generation is just incompetent. Your call...
If I were you (or was working at somewhere with an outage), even if it wasn't AI, I would still be blaming AI. Because the alternative means you shouldn't have a job in tech.
None of these outages, as far as we know, have a common cause. They just happened by coincidence inside the same news cycle.
Nor is there any evidence to blame them on AI, or tech layoffs, or hackers from Iran. It's more likely something boring and technical that doesn't fit into anyone's narrative.
My guess is a total disrespect for actually qualified people. You know, the ones that specialized in one thing, for their whole life. Every company tries to see how few people they can have and still run fine. And move fast. And no trainings. And no recognition for the people that could actually train other people. I think all these come together at some point and you get multiple catastrophic events in multiple industries.
Nailed it. Downward pressure on salaries, layoffs, outsourcing all drive technical excellence out of an organization. Then, when companies DO hire it's very hard to find competent people (and I mean find them at all, not find them at the cutrate you want to hire them at) and the people you do hire are just seat warmers
1. Does there seem to be a larger concentration of outages recently? (I would say yes.)
2. Is there actually a larger concentration of outages recently, or does it only seem that way because of an increased focus on this? (I would say that there is really a larger concentration, but I'm less certain about this.)
3. If there is actually a larger concentration of outages recently, is that because of AI? (This is the point at which I become pretty unsure either way.)
4. If it is because of AI, how exactly is AI causing this? (I think more widespread vulnerability discovery / exploitation is more likely than "everyone is vibecoding everything now and that's screwing everything up", but I'm not entirely sure.)
5. Is this just transitional growing pains, or is this a new normal, or is this the beginning phase of a pattern that is only going to accelerate? (I don't know the answer to this one at all! I hope it's the first thing, but I wish I were more confident.)
You saying they *are* entirely wrong is as baseless as their claim. Sure, you can say they don't have a common cause, and I agree with that, but chances of these events with different causes occurring so close together is very low and the chances of a same root cause is probably more plausible.
Code freezes for the holidays were several weeks ago. The code would have been deployed weeks ago too. And if AI writes better code than you, that says more about you than AI.
No, it all comes down to the decision-makers at the top being bad at their jobs.
One way or another, they've tried to avoid addressing tech debt, avoid hiring and retaining good people, and have more or less been kicking the can down the road / running on the edge without knowing it.
Or else they did embrace AI-driven "move fast and break things" approaches, and so they broke things.
They flip-flop between maximizing profits and draining operational divisions of any resources, and hoping to sprinkle a bunch of IBM magic pixie dust all over their infrastructure to solve all their problems without needing to understand any of it. You rarely find one that strives to understand problems deeply and invests in long-term planning and maintenance before a crisis hits.
The result is old systems that eventually break, system upgrades that go bad because nobody understands them, security holes that someone might have hacked into, etc. All the same basic root cause at the top, which is that most of our leadership class is full of incompetent short-sighted narcissists. And it "works" because most of the time it pumps the stock due to lower costs, and we only see the fraction of times that it causes headline-grabbing catastrophes. On a long enough timeline, though, they'll happily drive our entire society off a cliff--and we're getting closer and closer to that (at least politically).
I am assuming you don't have a direct line to each of these outages, and you have no proof that there is no common cause. A common cause could also be something like "poor production deployment practices" or "reliance on AI built software", in which case...maybe they aren't entirely not wrong.
It's extremely funny how many of your replies are proving your point exactly. One reply says "Yours is easy to hold since it can't have evidence" when it is simply stating that _there is no evidence of a claim_. Imagine that, "There's no proof of X" is somehow a magically unfair argument because it can never be proven! Wow! Nobody has ever thought of this before. Guess we have to assume there's inherently merit to X, then, and don't have to worry about actually constructing a proof of it (that requires patience and knowledge and first-party sources and does not get you internet points).
The boring news is that laying off 100th of thousands of IT workers because of a newly promised, never-seen-before technology, does indeed cause trouble. Who'd thought that these people did in fact create value?
I'd want to read the post-mortem analysis from the team that diagnosed and resolved the fault.
They'd really be the only people with information about the cause of the issue; if they blame an AI-generated code change or the actions of an AI agent, that would be good evidence.
"As far as we know" just indicates a lack of data, not evidence that they're not related. This is especially relevant for a political administration whose response to COVID was to bury their head in the sand by hiding data. Also: A public comment of "AI is hacking all our shit" wouldn't exactly calm the public.
It's fine to say, "I'll wait until we know more", but in the meantime AI progress continues and the sheer volume of cyber vulnerabilities it's uncovering is undeniable. It's not surprising that some state-sponsored bad actors would be use this as an opportunity to infiltrate too...
You also don't have any evidence that they are related.
In the absence of data, you are assigning blame to things you already hate, like AI or the administration. Motivated reasoning is highly prone to error, and it's a form of stupid-brained thinking.
Well, digital centralisation and interdependency without fault hardening is a big problem. Just because it presents itself in slightly different ways across industries doesn't mean it is not a fundamental issue in our society.
I don't see much of a problem with depending on AWS or Google Cloud. It's Microsoft who are particularly flaky. And always will be, because IT people choose them regardless.
You are still "centralizing" on "someone else's computer". Just because those two have not, yet, had a MS365 event does not mean they won't have one at some point. And everyone "centralized" there will have a bad day when that happens.
AWS has had them, and is passed the growing pains. At this point when/if AWS goes down, it will still have a far higher uptime than others could manage, let alone those going it alone.
Microsoft on the other hand, has proven that it will never be reliable.
AWS quite literally lost an entire region in the Middle East and has said they will not be able to restore it.
You can argue that this is a result of something outside of their control, and you would not be wrong, but if the point here is about centralization rather than simply a particular company's operational procedures, then direct strikes on datacenters should be part of that conversation.
I was an Amazon red badge in the EC2 org, fwiw, and do think AWS has excellent operational culture, far beyond what I saw while working in the M365 org, and beyond what I've seen at other tech companies and hedge funds where I've worked. But that doesn't obviate issues with centralization.
Centralization is almost a good thing for extreme cases though.
If Bank of America's central datacenter (cloud or on-prem) gets hit by a cluster bomb, taking out banking for ~half the US, you can be damn sure that very smart people are going to be working 24/7 until it's resolved.
If you have all your savings stored with a local credit union, say Seattle Credit Union which has 50,000 customers, and they get hit with a ransomware attack... well, good luck. You (probably) won't lose any actual money thanks to FDIC rules, but it might be weeks or months before their services are back online.
Similar to self-hosting vs cloud, you get a bit of a paradox where it's overall better for society if there's no interdependencies so a SPoF outage doesn't impact everyone, but things are much worse for an individual in the event a localized outage does occur due to economies of scale, or lack thereof in this case.
It’s a tragedy that much of European industry and public sector are so deeply embedded with Microsoft that they barely even consider other options, even when everything Microsoft touches seems terrible
This is also true in Oceania (especially NZ) in my experience. Companies here are much more embedded with Microsoft than companies I worked with back in the US and are less willing to consider other options. My company actually does most stuff with on prem windows servers and a bit with Azure. One aspect for NZ in particular is that there was no NZ region for Azure or AWS until 2025 and parts of the country are 70+ms from Sydney, so for anything where latency is significant, on prem stuck around here a lot later.
I feel like on-prem Microsoft stack is a very well known and controllable beast.
Its pros/cons are known and gaps can be addressed with 3p software. You just keep on old outdated, but proven version of software and keep the perimeter locked tightly.
GP isn't talking about on-prem I believe. Small/medium businesses in Europe are much more dependent on MS than their US counterparts. Alternatives are mostly out-of-scope from their point of view.
Don't forget that MS has the advantage of non-English languages support that many alternatives lack.
> I feel like on-prem Microsoft stack is a very well known and controllable beast.
I feel like it is very unknown, there is no documentation and everyone relies on comments on random forums that are now 21 years old which tells to set undocumented registry keys to undocumented values.
They’re expensive and centralized. I always thought the general wisdom was to bootstrap your business with serverless infrastructure and then eventually work your way back to an on-prem-first philosophy. I guess a lot of companies skipped that last step to…save…something?
As a technical person, I tend to find it hard to dismiss as well. What's easier to explain. Someone said it in a comment, someone keeps pushing for the cheaper solution cloud infra was cheaper and now we see the cost of that.
I swear nothing works fine anymore, the world feels broken. Maybe those particular incidents are not interconnected directly but they are connected in the ether in the sense that everything is incredibly complicated and precarious.
It’s natural to turn to AI based development as one of the causes of the increase in large scale outages.
I prefer to ask a different question. Will all this AI development delivering us amazing new software? Are we just going to get 1000 free clones of Adobe Photoshop or can we expect some really exciting groundbreaking new tools?
Perhaps all the really clever use of AI development is in the pursuit of better AI and if that’s the case, why?
Yeah, I'd probably extend it to software too: which software actually delivers real value to society, and which is only created for software's own sake.
As to why, well economy must grow, no matter what.
It boils down to leadership. If all C-suite asks for is "more, faster and cheaper" then enshitiffication gets worse.
Unfortunately it will take many more outages for folks to realize that serious engineering talent has great value in the AI age. It's their judgement that helps speeding headfirst into building unreliable unmaintainable and difficult to extend systems.
I have a predication: In the next few years having any connection to the open Internet will come with massive risks. I.e. if you're not a government, military or $$$ company you won't be able to afford the steps it would take to harden said system against the onslaught from AI attacks.
Some other thoughts:
* Will the Internet become balkanised?
* These smaller networks will require a lot of personal information to even gain access.
It is an interconnected problem, but it's the problem we're all already aware of. All these businesses and the FAA have gutted their operations teams. There's too much to do, and AI doesn't take less time.
they're interconnected in that we're all getting really overloaded by the workload created by AI, and even though we can use AI to get through the workload as well, we humans still have to be aware of everything flying by, and I think things are falling through the cracks more often. These days I work on ten or twenty issues at a time. it's a little nuts
This misses that expectation versus reality is key to how a human experiences something. If it's 20 degrees F in November where I live (MI), I'm going to grumble and get about my life. If it's 20 degrees F in July, I'm going to be freaked out. The same situation (no access to banking outside of business hours) can have radically different context and communicate different things to onlookers. See the former USSR's reaction to seeing Swan Lake on TV in 1991.
The absence of something that once existed is not the same as that thing never having existed. A null entry and a non-existent entry aren't the same.
Yes, we should all stop complaining about unexplained technical interruptions in our infrastructure and go back to being grateful for every minute that my bank's website is open on the weekend.
The reason I say not entirely, is because there is kind of a shared cause. One way of looking at is is that the root cause is complexity, optimized for efficiency and speed rather than robustness. It's a feature of the economic superorganism we are operating as part of. See Joseph Tainter on Collapse of Complex Societies, the work of Nate Hagens, Schmachtenberger & Liv Boree talking about Moloch, etc.
Or put a bit more cynically, it is a symptom of a society-wide drift towards lower accountability and duty of care. It's a type of tragedy of the commons. Selfishness and externalized cost/risk means nobody optimizes for systemic reliability.
People "vibing" everything is just another component of this, not the root cause. Using AI tooling is just the new technique for what used to be called phoning it in, or going through the motions, or faking it until you make it.
Unfortunately, it's nearly always been an effective tactic for a selfish few. It's problematic when we don't have an equilibrium that keeps it in the minority...
Yes, that's how you take separate causes and transmorgify them into one omnicause, by muddying the waters until you can't see the differences anymore. Don't do that! It just adds confusion.
(The same thing happened with separate company's problems becoming "Big Tech." It didn't help.)
Once you give separate things a shared name they become the same thing. We could call those in the claim to be "race to the bottom effect" or "incentives against resiliance". The water is no longer muddled, its now a slushie.
Snarky, but there's a point: Abstraction leads to viewing different things as the same thing. They are - at one level of abstraction. And they aren't at other levels of abstraction. It's not a problem to have both viewpoints.
The trick is that, when someone blends things together, to not lose the fact that they are also separate (at other levels of abstraction).
So I have no problem saying that "optimizing for efficiency and speed" is the common cause of all of these problems, and saying that they all have different (immediate) causes. Both can be true.
Sometimes these abstractions are useful for design or analysis, but maybe not for explaining them to others? A library or programming language will probably be more usable if its users don't need to know any category theory.
I'm not sure "optimizing for speed and efficiency" can count as a common cause, though, because it's not a single event. It's more like a set of events that have something in common.
Maybe we're not looking far enough up the chain. Everyone used to use forums and other websites run by other people (eg Substack) to read and discuss about things they don't understand.
Now when people have a question, they ask AI, which is wrong a lot of the time.
That there has never been an incentive to build quality software. As a result we spent 50 years building our society on top of poor quality software. We're paying for it now and will continue paying for it until the incentives change and quality becomes a focus.
I suspect there are incentives for Iran not to be identified as the culprit right now whether they are or not. If it is indeed Iran that would show they have the capability to do this and maybe the administration doesn't want that to get widely circulated. (and if it is a state actor it doesn't have to only be Iran - could be a coordinated effort - NK just did the big Binance heist and Russia can do this kind of stuff too)
The main incentive for any nation state to avoid attribution to direct and material harm to many Americans is that they risk awakening American bloodlust. Americans are unhappy about the current Iran situation and wish it didn't start and that it would end already, but sufficient Iranian escalation would turn that around real quick and suddenly there are boots on the ground among other things.
From the other direction I could see that the administration wouldn't want to attribute incidents like this to Iran because it would just make Americans more unhappy that the administration decided to poke the hornet's nest when they really didn't need to. And like with not anticipating the closing of the SoH, they completely dropped the ball on cyberattacks as well.
Imagine a tree of relationships branching up to the very top labelled, "Because the Universe Exists" as being the root issue that relates everything.
Saying it's not related applies an assumption of what level of specificity you are thinking about. And that's how you quickly de-sync with other people, especially "lay people" who look at you funny when you claim there's just no way these things are related when it's hard to believe that based on experience alone.
As other comments suggest, it probably is related to an extent. Related by complexity of our Internet-based systems, related by age, related maybe even by enshittification of expertise, etc.
The further up the tree you find these relationships, the less specifically actionable they are (you cannot CVE enshittification), and the more they are about "what are the overall social/cultural/human issues we're grappling with in this era?"
Aviation got a whole lot safer when we started looking at the systems causes instead of the specific ways individual people, airlines, manufacturers screwed up. Ditto for anesthesiology, cars, etc.
Sure, focusing on individual small causes is helpful too, and it's a lot harder to do the higher and higher level system stuff. But, it can be done.
edit, P.S.: It's worth keeping a bare level of suspicions that there could be a common lower level cause, whether through inadvertent dependencies added or through hostile action.
That's the preliminary post mortem from Microsoft.
I don't think we'll ever get anything honest out of the administration for the FAA outage unless Congress really wants to dig into it and make the results public.
a) this is not known, there is no reliable data on either the FAA or new bank issue; and the information forthcoming from Microsoft may obscure primary causation
b) there is also a risk of sophistry and language games obscuring common sense truth
In the world where a state actor is engaging in active sabotage as part of a cyberwarfare effort, there is one shared cause.
Which exploits are responsible in each case may differ. Still one cause.
Do I think it's Iran?
No idea.
But the point is, none of us here have any idea at present.
It's true that we don't really know, and my first instincts lean toward hacking by a foreign government, but then I start thinking about media reports of shark attacks every summer. Shark attacks aren't unknown, but certain media channels like to pounce on them for ratings, and after a while to the average person it might seem like Sharknado. Stuff goes down all the time, but in this moment we're just extraordinary conscious of it.
not sure if vibecoding is a cause or consequence, but it’s related, it requires as an axiom to stop caring about internals. And it seems to require disregard for quality too.
Competency going down, timelines being rushed, project managers and CTOs forcing new tools at breakneck speeds.
i'm all for experimentation and adding new tools when they're shown to actually help, but it seems like large software companies (especially banks and e-commerce) just don't care about quality anymore.
I think quality of most products is seemingly going down. To me it is a likely consequence consequence of increased financialization and money being expensive.
I worry that this isn't the right metric for diagnosing whether there's a problem. I'd argue that we've had instances in the past where a massive increase in how accessible programming is ended up being a net gain in the long run for software even if the average quality went down (e.g., having computers move from giant room-sized mainframes that only companies and governments could afford to sitting in the living room on a table probably reduced the average quality because untrained amateurs could start write software even if they weren't skilled enough to be employed). I could potentially be convinced that this isn't the same and that it will end up being worse for software in the long run with a stronger argument, but "software quality is going down in general" isn't convincing enough evidence for me.
All of this is also downstream of what I think is even more important, which is what's better for society in general. Obviously low quality software is not better for society than high quality in a vaccuum, but I also don't think there's a single obvious objective answer for whether the influence of software on society is a net positive or negative today compared to, for example, when computers were all mainframes and all software was produced by people paid to write it. There are a lot of differences in society today compared to back then, and software certainly has some role in how we ended up here, but trying to attribute exactly what and how much is not an exact science.
This is a lot of words to basically say: if you're confident that this is a problem, I think it's important to state exactly why, since I think the interest parts of the discussion are in how you've concluded that.
That's more like the rise of Google (which may or may not be correlated to software quality), as "something went wrong" is an error message popularized by them.
Maybe node software was good initially, but its model was unsustainable, it relied on outsourcing to volunteer programmers, relying on such programmers means increased vulnerabilities, supply chain incidents, and low quality of software.
Vibecoding is an extension of the culture of overrelying on massive external dependency graphs. It's a preference for visual confirmation that something works and a disinterest in how it works. I think it's also related to the tendency of content feeds that are based on likes and swipes instead of searches, instead of specifying what we want, we are reduced to approving or dissaproving (until we are removed from the loop entirely)
considering how the experience with most SaaS products right now is insufferable with broken states and UI, I imagine the same broken behavior is leaking in every other area of product development, so I agree
Source = trust me bro. What’s that bias where things are so rare that when they happen it makes news, so you think they’re very common (eg plane crashes?).
Uptime and quality of software is about 10000x better than it was 10 years ago. Things just work all the time now.
I'd argue that just illustrates the point that it's impossible to have an interesting discussion without a stronger claim from the parent comment. If I say something that's too vague to falsify, and you can just as plausibly claim the opposite, you're not at fault for failing to turn my claim into a more rigorous one.
My dad is a doctor. He keeps asking me similar questions.
I told him it's just like how he keeps telling his patients to watch what they eat, drink and smoke less, and get some exercise, but they don't, and then they have a heart attack and wonder why.
If an unfriendly foreign power were attempting to impose on America the insecure and unstable vibe-coded digital infrastructure we’re building today, we’d probably view it as an act of war.
Everything you said has been true for a long time, vibe coded or not. How much of the Internet proved to be exposed to a supply chain attack on node? That wasn't AI, that was just how things worked out.
This is the obvious consequence of (failing to) replace software developers with AI slop. Despite the protests of the Antrhopic shareholders who dominate this site's comments, LLMs are hideously poor at designing software.
We (on HN) as generally technical people (despite the HNers who seem to think you can vibecode a NVidia competitor) that all of these incidents are unrelated with vastly different causes.
But I think there's an increasing consensus among "normies" like your coworkers who see these as related. They don't know why but assuring people they are unrelated fails in messaging terms when it feels related.
I think this will be an increasing problem where so much more code is not only written by AI but reviewed by AI. Things will slip through and companies won't be punished for these outages in terms of public perception (because it'll happen to everyone) or by the government. We just don't have serious consequences for corporate malfeasance or gross negligence. Like AFAICT the Sackler family should all die penniless in 6x8' grey prison cells.
Unfortunately we live in a country where the Sacklers get to stay billionaires while 1 year olds are shot dead at the behest of Walmart over a pack of diapers that weren't even stolen... with zero consequences [1].
It feels like this is just the next level of entshittification.
It is interconnected. Everyone bought the "cloud is up 24hrs" bullshit. My company had physical servers and we had 99% uptime because we had competent people. Profits are driving unsafe transitions, such is capitalism. Look at our crumbling old infrastructure and ask yourself why the "digital highway" would be any safer.
Honestly, I can't help but hope that ends up leading to enough popular support for more regulation for AI that something gets implemented. On the other hand, there's also ample historical precedent for hoping that a mass movement heading in the right direction ends up stopping at the point that feels reasonable and then being surprised when the momentum continues and people like me looking foolish for thinking that it wouldn't, so I might just be overly optimistic.
AI is not the cause of this kind of problem and it's regulation won't make it go away. Maybe it needs regulation, maybe it doesn't, but you're missing what's happening if you think that's what these widespread failures and vulnerabilities are about.
AI is acting as a sudden precipitator for problems that have been latent and growing for at least 15-20 years.
In pursuit of maximally rapid innovation and reduced capital expenditure, the software industry adopted a "move fast and break things" mentality and homogenized code and service infrastructure over a few winning cloud vendors, SaaS providers, and code solutions/frameworks/packages/etc.
The pitch was that you could rely on downstream experts to do better work than you could, that the biggest vendors and publishers had the best experts, that have them invest the capital would look better on your accounts, and that their scale would give you the best value for your investment.
The caveat is that effectively all technology products have serious faults lurking in them, even with the finest expertise working on them (not that most actually had that), and that when everybody homogenizes over the same providers and solution the industry shifts from a bubbling froth of localized bugs to a heavily correlated house of cards where widely correlated failures manifest in widespread collapse and catastrophe.
Whoops.
AI is acting as a breeze against the house of cards, but lots of things could have and would have and did, especially as it continued to inflate year over year.
If you're saying the parent comment I responded to wasn't talking about AI, what are you saying they were talking about? If you agree they were, why are you trying to correct me instead of them?
From experience, since the parent commenter didn't specify, I would take the "coworkers" as having a vague sense of dread and distrust with no real confident insight to the cause. They're likely to nod along to any explanation they hear from anyone they think is more technical or news-following than themselves, and then maybe repeat whatever last such they heard when chatting. With the occasional exception of politics, that's how most people approach technical topics outside of their expertise. I didn't guess what they might have in mind today at all, and didn't feel the need to.
But you did speak directly to AI, and specifically to advocate for regulation of it, which is why I replied to you. You were being specific and material in a way that the parent commenter was not, and that invited a different response.
I didn't specifically claim that it was the direct cause of this though, so you still decided to confront something I didn't claim over something the parent comment didn't claim
I don't really see how regulation for AI helps. Companies were on this crash course of trusting Microsoft (or pick your other dependency that cant be eliminated as it is a monopoly or the minimum) to provide more and more of surrounding infrastructure. Until they had AI to run simulations of outside.competence there wasn't a snowball's chance in hell they were competent enough at IT to run someone else's infrastructure.
Coming so soon after the Office 365 debacle and the meltdown in the FAA network for air traffic control which caused the ground stop, it does raise the possibility of a targeted cyber attack being the root cause.
Or, perhaps there was some change in the IT workflow a while back, which causes an accumulation of technical debt, that is starting to make itself felt?
If it were an attack, it would be something that were more than annoying. The crypto theft by North Korea, for example, was obviously a bad actor, with a clear motive/benefit.
FAA incident also came like 1-2 days after a new system was rolled out, which would be quite a coincidence if it were not related.
It's possible, but if you were a state actor, why would you want to give the US advanced warning about your capabilities and give them a chance to fix their issues?
Maybe you're just script kiddies? Or grey-hats trying to expose and create incentives to prioritize/fund repair before worse things (e.g. digital attacks as acts of war) start to happen
But I don't know, I'm just someone with some minor experience larping in security who is spitballing :)
i worked a pretty similar place to JPM, if qa was ever cut delivery times went up and people were okay with that for mission critical stuff. If you said 'this isn't tested well we're delaying' everyone would be okay with it.
on trading applications, and banking systems theres huge financial and legal consequences for cutting corners (look up knight capital). Pretty much the only way to get fired is to mess something like that up.
In some jurisdictions if you pushed a change that broke and it was flagged internally as not tested enough someone in the org would quite literally go to jail.
A few threads ago people are arguing with me about this.
Why have QA, every developer can simply QA and deploy, and talk to clients, and make sales calls.
Then again I don’t understand Capitalism. At a certain point these companies are so firmly entrenched they can ship buggy half working software. The consumer is effectively a hostage, all competitors are also delivering barely working products.
Btw, not trying to say you are wrong with not understanding capitalism, I find it helps to view it not as an imperative but rather as a local optima in game theory. Capitalism just works really well for its own utility functions that it self dictates upon us.
Unfortunately highly advanced tech people lack “selling skills”. Hopefully this changes with ai - prompt: summarize the work I’ve done and setup I’ve implemented and how it will help the company and why my role is extremely important.
West Coast, 3 out of 5 of these load for me no problem (I only use 3 of these services, cant check the others). Might be an east coast AWS thing? Can't seem to open a WireGuard tunnel into my home network on the east coast though, which is unusual. o_O
Having success with Chase on the East Coast as well at the moment. I'd like to know the actual scope of this as well. DownDetector as a source is kind of low value IMO, but the banks have released statements. Curious if these are actually a large issue since I don't have the opportunity to run into it myself.
Are you actually authenticating or just landing on main sites? The more public/marketing type pages are often hosted separately from the account platforms.
Why do people assume that this crash is the result of AI vibe coding (i.e. these companies rolling out buggy production code) versus AI-empowered vulnerability seekers? Seems more likely that a group is trying to swarm and exploit a possible vulnerability common to these banks' operations, than it is that Bank of America, Chase, and U.S. Bank all happened to roll out bad vibe code on Monday.
I know the leg I'm standing on is correlation but did we have such widespread outages pre-LLMs? The two that come to mind is that one time some intern pushed a config change that borked us-east-1 and Gitlab deleting their DB. And both had clear comms on the situation within acceptable time frames.
Also, I don't think it's that "AI code is inferior" but rather devs got sloppy and no longer review code. Not to say that everyone's lazy but me---a lot of systemic/environmental factors could be the reason.
But how do we know it is because of AI vibe coding though?
It could be because some critical infrastructure are due to upgrade after a long delay and this is inevitable at some point.
It could be that AI agentic hacking has pushed for some rearchitecture in faster deployment time, which may result in big outage for industry that has so much legacy and used to big bulk deployment only.
Or it could just be because we rely more on cloud infrastructure than previously.
With GitHub specifically, yes. But I remember AWS/GCP/Azure outages were like a yearly occurrence before the 2020s, and we kept having the same conversations back then about infrastructure resilience, multi-cloud, etc. that were too hard to address then and still too expensive to address now.
Maybe one major platform had a major outage annually but AWS certainly didn't have a major outage every year. More like every 3-4 years. I remember exactly two in the last 10 years that impacted the production application I run.
There have been many more than two in the past ten years. Maybe not for you, but depending on the region/AZ, especially us-east-1 as it’s aged, outages are impossible to avoid.
Much better than Azure, but that’s not a high bar.
It absolutely has. The "why" is a matter of first-party versus third-party (bad AI code making bad services, or good services unable to handle the load from everyone else pushing AI code, or a combination of the two)
> The "why" is a matter of first-party versus third-party (bad AI code making bad services, or good services unable to handle the load from everyone else pushing AI code, or a combination of the two)
Decades of cycles of layoffs and outsourcing, where the core business or organization no longer operates business-critical or organization-critical functions.
Failures aren't just a product of AI-written code in the infrastructure, but the infrastructure trying to bear the load of everyone else's AI-written code at massive scale.
But that’s the point around correlation =/= causation. What the parent comment is saying is that the cause could be that attackers have much more sophisticated tools to bring down services. The timing with AI adaption doesn’t prove on its own that lower code quality is the reason.
It's not that AI code is inherently inferior, it's that the bulk of the work engineering does now is code review and the scale of reviews has become exponential which means a lot of stuff just gets punted to agents and not carefully examined by anybody with actual subject matter expertise. Plus automation and platform work (like managing test suites) has also similarly become deprioritized and QA in general is being obsoleted as a field (for worse, really).
This is a systemic issue that's driven by AI but the responsibility lies with leadership who are still operating on the move fast and break things mentality. With everybody swinging for the fences and the median blast radius increasing and the checks/roadbumps eliminated you end up with a lot of incidents because the total number of merges and the size of the diff has increased by quite a bit per sprint.
Also most engineers have always been terrible at reviewing code. They could find most bugs when working through it line by line manually as they wrote the code and the tests, but most could never find bugs in their peer's code through inspection. Now that this - along with business and product understanding - are the primary skills they need, a lot of developers are under-skilled for their job requirements as they suddenly exist today.
Staff-level engineers can do this work because they always have, but if they are suddenly the only ones who can in an org then they are simply overwhelmed.
I don't know if it's as much as people just hoping that it fails, or the fact that in my opinion a programmer is far more likely to verify code that their reputation might depend on instead of just shoveling in whatever an agent spits out.
No the issue with AI code is not that it's inferior but that it solves the wrong problem or doesnt believe an edge case is necessary or other contextual issues unrelated to whether the code is actually correct as written.
Or maybe that cyber attacks are generally known after the fact that they are cyber attacks via things like extortion.
Like take the Binance hack for example. A naughty hacker, instead of stealing bitcoin, could have just caused mass outage and disruption without stealing anything, but didn't because bad actors usually want some financial benefit.
Like if we learn that PII was stolen, then it's be safe to say that this was a cyber attack.
While there may be people arguing for that, the more relevant point is that regardless of any possible positives of AI code, there are clear negatives that are being largely ignored. Specifically in such outage cases, the volume of code being put into production with little or no review or understanding.
AI code is inferior, but business, the media, and regular people don't care about that. They just want fast and cheap, so you get the slop code you pay for.
I find it fascinating that anyone thinks computer programs will be different than anything else.
Clothes. Food. Appliances. Tools. Electronics.
All of these things got worse, but got WAY cheaper. All of these industries had to shed excess labor like crazy in pursuit of optimization at the cost of quality.
>Because people want to believe that AI code is inferior,
This is armchair psychology and extremely bad one because the people who have been concerned with engineering quality have been vastly outnumbered by people who just want to be able to turn their brain off and "ship fast" for probably 15+ years, and that's been reflected in reliability long before LLMs came around.
The code these systems churn out at scale is absolute trash but the financial incentives are to produce out more of it because nobody is ever actually held liable for software breaking in our age
yes, I read some java today where I wished for the clankerd to have done it...
Think system(curl) and random-if-cascades which are never reached - they removed the system(python) in this iteration though...
At this point I think AI-code is just average and probably just fine for corporate environments/SaaS (especially given what thkse produce today), but I still think infrastructure should be a) understood by humans b) produced without installing a backdoor on the dev machine and c) a human should decide what to build and be able to reign in the machine.
it is though. at minimum qa/QC of ai code is way worse. Things are hitting prod right now in many sectors that should never have hit prod. Amazon on mobile (at least android + ff) is unusable ATM as the most recent example.
This gets to the implied argument: that humans are inherently better to machine-generated output. The truth is there are a ton of programmers getting paid well to generate crap. Anyone's who been a part of any team that isn't some highly curated A-squad knows this is true. Hell, even teams of 1 know it to be true, when you were rushed, didn't understand the domain or stack, under personal pressure etc.
That is all true, but those shouldn't be the stated reason that's reported to the public in a press-conference headed up by a US departmental head; that's how misconceptions poison the information-space for everyone.
I also wonder if Trump's administrations' spokespeople even realize they have a massive credibility problem, and that if they want people like me to believe anything then I have to hear it from a mid-level functionary, not a political appointee.
You say whatever makes your stocks go up and now down, and if you don't, you are stupid because it's normalized now, there are no repercussions, and if you don't someone else will.
If the mid-level functionary tacks on, "This is why we demand that Congress allocate $50 billion in new funding for modernization!" do you still believe them?
No, just no...Chernobyl was caused by the staff doing an experiment that to put it mildly was wildly risky. They disabled safety systems with wrenches (meaning they couldn't easily be reenabled). Challenger blew up because staff at NASA ignored warnings from the people who made the O-rings. Saying these were "straws" is completely irresponsible. Attitudes like this are what cause outrages. Also, unless you are being dishonest, this is the worst string of outrages in the history of tech. There is a cause. It can be found. And unless you think in these terms, you have no business working with a production system in any way.
You’re not in disagreement with op. That’s the point motionlessveloc is making, it’s a series of mistakes (straws that make up the hay on the camel’s back) that points to a broader systemic problem.
I think you missed their point a bit. I believe they are saying that the cause was the faulty wiring in the same way that Chernobyl was caused by a switch flip.
In other words, the wiring problem might be the wrong thing to focus on - but it probably happened, and the systems likely wouldn't have crashed that day without it.
It's the sort of thing that is common in all systems I've been inside- which is to say, something little, to do with configuration, hardware fiddling, etc.
While AI rollouts have had issues, it is far from safe to assume just because there is an issue, it is automatically AI-related or vibe code specifically. Things always are breaking. Lines go down, servers have power failures, etc.
The trend is your friend until it bites your rear end. To give a topical answer, the Outlook issues were blamed on a software bug from months ago. While AI is being embraced en masse and causing all manner of issues, we have AI issues as _well_ as traditional "it broke" moments. The main point is we should not jump to a conclusion without stated evidence. Just because AI is to blame sometimes, does not mean it always is.
U.S. Customs officials said Tuesday that they had traced the source of last weekend’s system outage that left 17,000 international passengers stranded in airplanes to a malfunctioning network interface card on a single desktop computer in the Tom Bradley International Terminal at LAX.
The card, which allows computers to connect to a local area network, experienced a partial failure that started about 12:50 p.m. Saturday, slowing down the system, said Jennifer Connors, a chief in the office of field operations for the Customs and Border Protection agency.
As data overloaded the system, a domino effect occurred with other computer network cards, eventually causing a total system failure a little after 2 p.m., Connors said.
I can really believe that they plugged in a misconfigured router and that a new router on a faster link was able to flood downstream legacy hardware with junk requests to the point where it started to fail.
… and no one wants to work anymore! Back in my day we’d hand assemble code on paper and then whistle it over the phone to a 300 baud teletype in the server room … and that shit never broke! Not once!
IMO the problem is not AI-generated code per se, but rather the utterly absurd expectations and resulting destruction of best-practices and discipline that developers all over are now dealing with.
My first thought for this week's outages has been that people are moving too fast and are, as the saying goes, breaking things.
Because there’s a lot of anti-ai sentiment going around right now. Blaming it on AI slop code supports pre-existing beliefs that AI is useless. If it’s AI hackers then that not only means AI is actually good at something, it’s potentially an argument for why we need increased spending and that there really is a race dynamic at play that we need to win.
Basically, blaming AI slop supports a lot of pre-existing beliefs, while blaming AI hackers does not.
I don't really think it's that. From the people I work with, it's more the push beyond the "capabilities" of AI to the point that we have this amazing tool (AI), which theoretically should allow us to ship much better and more resilient systems. But instead we are pushed to ship really cruddy systems without the proper amount of diligence, review, and testing due to the C-suite's desperation to ship as fast as possible - consequences be damned.
There was a thread contention bug in one if their backend services in an update that was pushed out in July and didn't happen until the system was under normal loads across all their services.
I avoid Zelle because Zelle shows my legal name rather than a nickname to any random stranger I transact with. At least with Venmo you can provide a pseudonym to the other person. The only use case I have involved buying / selling secondhand items on Craigslist and Facebook marketplace.
I view this as a positive. If someone sells me something that ends up being broken/not as described I would much rather have their full legal name vs. a nickname.
I also don't see how you would ever end up in this situation if transacting via fb marketplace as you're not even allowed to have accounts that are not you.
Not OP but isn't zelle yet another commercially motivated third party that will aggregate and mine user data for profit? If nothing else they're an additional point at which data could leak.
My bank used to let me send checks via online banking (either electronic or mailed out physically by the bank depending on the recipient) until they adopted zelle and discontinued that service. My credit union still offers the check thing for free. That's an easy choice at least for me.
It's only sort-of a third party; Zelle is owned by the banks
> Zelle (/zɛl/) is an American digital payments network run by a private financial services company owned by Bank of America, Truist, Capital One, JPMorgan Chase, PNC Bank, U.S. Bank, and Wells Fargo.
Not exactly. Zelle allows chargebacks against unauthorized origination, i.e. your account gets hacked.
Banks only allow a couple $k of transfers per day anyway. The company behind Zelle, Early Warning, is a bank-owned centralized fraud detection network as the name suggests.
The ability to send wires from your account is riskier... I wired the down payment on my house in one go.
"Banks draw a distinction between what they consider scams — when a customer is tricked into sending money to a thief — and fraud, when an unauthorized person gains access to a consumer’s account and siphons off cash."
If you're drunk at a bar and someone unlocks your phone with your face and sends themselves your money, that's gonna be authorized from their standpoint.
There's no consumer protection for any payment method that's not a credit/debit card or PayPal, Cash App, etc. which chain off credit/debit You pay for that in transaction fees; up to couple of percent.
Cash, check, wires, ACH, money orders are no different from Zelle in that respect.
You’re goal post moving. If you don’t want to use Zelle, fine. But don’t spread false info that it’s prone to drain your entire account relating the risks to crypto transfers.
I kinda expected things like this to happen the week of the Anthropic IP0. EA sees everything as a trolley problem anf tries to create them wherever they can. Thwy will mess with critical infra to get themselves the regulation they want.
Its a pet conspiracy, that I know has no evidence, but EA would love to get the public in agreement with them on AI so that the public consents to them being the arbiters of AI regulation.
I reckon, it would take 3 days of full outage for an average company to even consider the work needed to migrate from cloud providers or services. Obviously with some exceptions around risk to life systems, which should have an independent disaster recovery plan already.
M365 would have needed to be down for a full work week for job adverts to go out for an internal IT infrastructure team.
I doubt it, c-suite will never approve the spike capital expenditure for buying hardware now they're under a nice smooth subscription curve which doesn't harm their profits as dramatically.
Presumably a central provider takes down your dependent services with no local ability to navigate around the outage. Removing the dependency obviates the outage risk. It introduces its own downsides as well, but that's a different discussion.
Yes, it's sad these systems are going down all the time, but if we wanted them to stay up we'd have to pay the people who maintain them, which means less money for the stockholders. So you see, our hands are tied.
I love how trusting everyone is after three major outages on consecutive business days, hot on the heels of months of warnings and demonstrations of AI enabled attacks, that these aren’t attacks, solely because people incentivised to say that everything is normal are saying everything is normal.
They, too, didn't need programmers any longer and let AI take the wheel?
The thing is, even if true, they'll never admit it. People who make these kinds of mistakes (if this is what it is) are also not very prone to honest bouts of introspection either.
Honest bouts of introspection aren't necessary to keep the line going up.
So, your company shit the bed due to your plans. So what? You went to the right school and have some experience in the c-suite. You probably have enough money to go a decade without cleaning out your savings.
Break down the company for scrap, file an IPO for the good parts, whatever. You'll make the money back.
Their website isn't bad compared to some others I'm familiar with, they have locations everywhere, WF is banned from my current area (my previous bank), local and regional banks had terrible websites and security options, and they gave me $1k to open the account. Ultimately it's about convenience.
The US has no natsec currently due to its leadership. There has been an intentional purging of those with real expertise at the topmost layers. Mass layoffs of experienced folks. AI use rise and all that entails. Several billionaires with motive and resources to do sabotage or cause chaos. Now add the US attacks on Iran giving huge motives for the latter to push whatever buttons they can. One could argue we should be surprised 2026 hasn't been much worse than it has been. And there are reasons to expect it to worsen much more before it can ever get better.
what I refer to has been documented repeatedly and redundantly across US news orgs for over a year and a half into the current admin. Its no secret and not up for debate.
DownDetector's methodology is fine, it's just noisy. The way to use it is to look at the top-reported failing sites in the ranked list on their homepage, which has been the correct root cause for every major outage I've seen. Ignore the lower ranked ones.
By top-reported I mean the top two, maybe up to four. It's not high-fidelity.
I still think it's useful, especially since in 2026 the art of building a status page not dependent on the infrastructure it monitors has been lost.
This kind of rigid thinking is such an HN thing. The "cause" is obviously problems with payments from the bank outages, which, again, is clear from looking at the ranked list.
You can't source details that depend on technical knowledge from a non-technical audience, regardless of methodology. Downdetector is useful nonetheless.
But downdetector doesn’t track that financial transactions are failing. It tracks whether people are checking to see if a service is failing. It is only a charitable reading that people are only checking if they are experiencing a failure. A signal, user queries, is presented as another signal, service status, even though there is no correlation.
In much the same way, every time there’s a large conservative or republican gathering like a national convention in the USA, someone starts spreading fake headlines that “grindr” is down for that specific city. People go to down detector en masse which makes it display that grindr is down (it must be since so many people are checking), lending credit to the hoax. It was never down, and no app has city specific infra that would go down with a spike of a few hundred or even thousand people assuming the premise of the hoax was true.
From what I can gather from a quick search, Grindr hasn't gone down in response to GOP conventions, but there is commonly lots of traffic during places/times where GOP conventions happen, and it's plausible that if many people are trying to access a service from the same public wifi they could experience connectivity issues without anything being wrong on the service end.
Grindr CEO: “Yes, it was actually true that there was a significant spike in usage in Milwaukee during the [2024 GOP] convention.”
Just because someone, or a lot of people, mistakenly believe something doesn't make it true, and pointing out that mistaken belief is not "rigid thinking," it's literally just accepting reality.
>This kind of rigid thinking is such an HN thing. The "cause" is obviously problems with payments from the bank outages, which, again, is clear from looking at the ranked list.
Not to offend you with all this rigid accuracy, but what you describe is not a cause but an effect.
How do you know some of these aren't experiencing a partial outage? Responding instantly to partial outages that never get reported on status pages seems to be one of DownDetector's biggest strength.
I live close (as the fiber flies) to one of the more popular AWS datacenters so I have no doubt that there could be situations where I can access sites that others can't, or in the case of a proper AWS cockup, where I can't but many of you can.
And when it's not DNS (it's always DNS), it kind of depends on how load redistribution works. At some point I have to switch to load shedding, and so Idaho and Montana might just be shit out of luck.
If the NYC office is still able to do work, I don't want the Denver and Chicago office to overload their servers and now nobody can get any work done.
The article cites a Chase rep saying they are impacted by a bug of some kind.
The article writer then went to DownDetector and wrote a headline "Outages reported at Bank of America, Chase, U.S. Bank, Wells Fargo and Zelle" based on just that. As far as I can tell, there's zero genuine evidence for that.
Is it not true that Outages have been reported - via DownDetector?
DownDetector is a way for customers to report outages so the headline is not lying at all. It is not claiming that the root cause of these outages are these banks, or that the banks themselves have reported the outages.
I always think of the time the company that acquired us put our Wiki and (some of) production storage on the same drive array and when it failed we couldn't access our runbooks. Lots of anger lines coming off of people who are permanently grumpy the rest of the time. Like people too angry to trust themselves to talk, so one word answers.
After that our operations team got most of its autonomy back.
How do you make an app that watches everyone else's infrastructure but can't be taken out by a common cause?
I’m experiencing degraded AWS ECS service today so it’s atleast a bit helpful. AWS is probably never going to be down accross all regions and all services 100%.
Normal people Google "is Chase down?", find DownDetector, and click it. (Note that the article has a Chase rep quoted saying they do indeed have an outage, and none from other companies.)
Some of them then click into the DownDetector home page and browse around. Suddenly everything is spiking.
Every time I've seen any one major company with issues, DownDetector has spiked across the board. As it is now. Google and Amazon are apparently down.
I mean, if Visa/Amex transactions are not working because their bank is down, end users don't know the difference. Card says "Visa" on it, card is not working, end user blames Visa.
The degradation in software quality across the board (Spotify, Google Maps, GitHub, YouTube - to name a few) has been very noticeable for me, and I suspect AI has a lot to do with this.
The thing is - I also think AI can be used to build very high quality software. The problem, I think, is that the typical big tech/VC tech/MBA tech type company (who control most proprietary software at this point) have very little incentive to actually care about quality, and that was quite obvious even before AI.
I remember before I started using AI (I was pretty late to the game), I assumed all the bugs and instability were because people were writing software with AI. Now that I've seen how good the code that AI writes can be (CAN be), I'm moving more to the thinking that it's from AI being used to find vulnerabilities.
I've never had AI one-shot even a 10 line script that I didn't manually revise. To be fair, it usually produces something that works, but after seeing it I'll realize that my prompt was too vague or I left out some edge case. So it's the old problem with software "it does what was asked for but not what was needed."
So I speculate that a lot of problems with AI code are just that it is accepted uncritically because it builds and doesn't have any syntax errors.
There’s a pressure to deliver more and faster from management. So instead of using LLM to reduce the coding time and allocate more time for review, everything is scaled down and there’s even less time for review. And you’re responsible for more as teams are also scaled down.
I think majority of companies don't know how to build safely with AI. It's just kind of handed off to engineering and mostly used as one-off chat interaction to get stuff done. Very few are building any kind of harnesses around SDLC that focus on architecture, testing, security and etc. I think we'll get there at some point, but in my experience the smaller tech players are just now on-boarding to use AI in general in any kind of capacity. We're not talking about best practices yet.
Wondering if this has anything to do with the QA function being folded into the dev team in the last 15 years or so.
I always defended this, but it appears maybe we don’t have enough developers that take pride in the work they ship. LLM coding made this pretty obvious.
IMHO the only way to build good things is to pay attention to the details. And coding agents, while greatly increasing the appearance of productivity, they are causing people to pay less attention to what they're actually producing. This is the inevitable and obvious result.
Agentic coding could be a world-changing benefit, but the current way it's being sold and used as as a cost cutting measure -- produce software faster than ever before with fewer software engineers! Get rid of those pesky experts!
It is a C-level executive problem. With the FAA it is the agency directorship problem. Same issue.
These people, very literally, are trying to run the world with power point presentations and executive summaries drafted by sycophants. 70 years ago these people had actual experience doing actual work and trying to grow businesses and doing stuff at the ground level. Now they are given their positions through a combination of credentials, connections, and political maneuvering. None of which actually relates to how well they can or cannot run a organization.
When reality disagrees with what they get in their emails and presentations then it reality that is wrong. It is the messengers that are the problem, not their competency.
This is nothing new. But the failures in leadership compound over time, like interest to a unpaid debt. They start off with healthy growing companies and thanks to government protections and bailouts it takes decades to for these organizations to decay into total dysfunction. In the past these sorts of people would be removed and their assets reallocated to competent management through bankruptcy and competition. However that natural process is not allowed anymore.
All AI is doing is handing them more foot-guns.
They think they can do better/faster/stronger/cheaper with AI and if AI causes problems then it can just be fixed by paying for better AI. Which they assume is just right around the corner, always.
Very well put, and I think (open) AI will help competent people outcompete these clowns, which is what makes me optimistic. AI is a multiplier, but multiplying bullshit just creates more bullshit. These people are far more replaceable than competent engineers, even more so with AI.
Part of being sane is accepting that some tools, while being very powerful, are too easily misused to be widely distributed to those lacking proper training. This is why you cannot just go and buy 5 kilos of azithromycin
Ehh I don't really buy that. Who decides what tools are distributed to who and how?
Don't want to get into a political debate here but I'll say that I think that open AI models and access to them is a benefit to society, while closed, proprietary models, controlled by tech billionaires are probably net bad for society.
> some tools, while being very powerful, are too easily misused to be widely distributed to those lacking proper training.
Could you define 'proper training'?
I've endured plenty of fucked-up shit from so-called 'professionals/ experts/ trained people' over the last 30 years or so.
I'm currently assisting a couple of vibe-coders improve their 'too easily misused...[and] widely distributed' (available on their sites, or github) tools - I mean slop - that benefit me in ways traditional companies have no interest in.
And the updates are fast (daily, mostly) and accurate because, unlike jeremy with a pretty CS degree and a nice job, these people actually give a crap' and don't have a manager to say 'you can't do that'.
Also part of being sane is accepting that the cat is out of the bag and big $oftware/ tool manufacturer better get a move on.
Disclosure: my interests are not critical to anything; quite the opposite, but people are learning faster and faster with Ai learning faster and faster.
Who needs azithromycin if a series 1's and 0's can accomplish so much more?
For good or bad.
No, i cannot. If i could, i would be proposing concrete ideas. But I also understand why I cannot buy 3 kilos of plutonium or 5 kilos of antibiotics, and understand the modern ML models can do just as much damage while being more available.
That is just one large example, and Shopify supports stablecoin payments right now and grocery stores built on them can accept stablecoins at in their point of sale systems.
But with that attitude, no progress will ever be made and you will continue to professionally complain on HN.
But thankfully, companies using the Tempo blockchain such as Stripe (who built it), and others do not care and already have business interest with the use-case of payments.
- Cards not working.
- Physical branches closed or unresponsive.
- Online banking down.
This exact same thing happened in Europe with Revolut late last month. It was resolved in about 12 hours: https://www.revolut.com/blog/post/20octoutagetechnicalovervi...
In that case, it was a Google Cloud outage. Something similar (database outage) could be the culprit here.
Lots of banks have been down at various times, but its a lot more of a problem when multiple major banks are down at the same time.
article says it's mainly Zelle
Zelle has explicitly said they're not experiencing an outage. And it being 'mostly' Zelle would not affect in-person branches.
Its wild that even our banks are now reliant on cloud providers.
It doesn’t seem like a cloud provider was at fault:
> “JPMorgan Chase has experienced a temporary issue related to a software bug in an on-premise hardware system — it is not cyber-related,” said Joe Evangelisti, spokesman for the bank.
It's wild that they pay like shit and attract 2nd rate programmers.
Eh, actually, I don't know if that's true about 2nd rate programmers, in a lot of ways they're probably better, but they do get paid less.
I realized this while working at a startup. The startup had a webpage that worked half the time--that's it, that's why we had funding. Well, that and the founder was going around telling investors we had fancy AI algorithms, which were actually mostly manual human work.
Meanwhile my friend working at the bank made a lot less.
Looks like it was second-rate programmers:
“JPMorgan Chase has experienced a temporary issue related to a software bug in on-premise hardware systems,” the bank said in a statement
https://www.nytimes.com/2026/11/16/us/bank-outages-down.html
I’ve always found it curious that FAANG/California-based employees receive comments of “hugs” from HN during periods of downtime, while those outside of the bubble receive only mild slander. Of course, no suggestion that you specifically have engaged in the former.
Strangely enough for me (with Chase):
- Cards working
- Branches (the few we have left) open, or at least it was when I drove by at lunch
- Online banking has been working for me all day.
I've got my cash stash.
That will work for the first week to month of a disaster. Once people realize it's bigger then dollars will rapidly lose value compared to direct barter for useful things.
Guess how I know you've never been through a major hurricane.
I don't agree with OP's point, but comparing the damage to the dollar that a localized hurricane and an extended national bank-outage would have is not an apples-to-apples comparison.
In a hurricane, the dollar keeps it's value, as others outside of the affected region are still using it to trade. This may not be true in the event of a national/global outage....not to mention the whole 'trust in the USD' aspect
I don’t think cash is gonna help if the entire monetary system stops working - gold maybe?
If the “entire monetary system” stops working, then things are fucked enough already that gold is essentially the same in terms of utility as cash would be, in that its only immediate value is its scarcity.
Things like food, water, basic sanitary items, would be bartered for.
Diesel. And .223 and 9mm. And the addresses of the 10 nearest preppers.
It’s funny how people just automatically go down “I have a gun and I’ll be good scenario”, meanwhile every natural disaster in modern history suggests that in crisis, people come together to help each other instead of threatening each other with guns.
People often come together initially, but it's a dramatic simplification to think that that implies that after the immediate issues are resolved (e.g. outright starvation) that competing groups don't often become extremely antagonistic.
You'd still not want to be a lone wolf type with a gun, though, you really want to have a group in all cases.
I realized this watching Gaza. Here I am watching a people that more than a few voices would tell me are sub human, but every time I see video of them dealing with the destruction of literally every single building within sight, they seem to be acting like regular humans, not savages.
No doubt there have been some crimes and things done in desperation, but humans don't just instantly devolve into roving gangs that kill on sight.
The people coming together helping each other can also wield guns far more effectively than some lone thief, who will be universally despised and viewed as a highly immoral agent.
> people come together to help each other instead of threatening each other with guns
People, collectively, come together. There's always outliers -- we call them assholes. It's nice that some folks are so clear on what side of line they intend to stand ahead of time -- pay attention to what your neighbors are saying, so you know which ones are the assholes.
It’s definitely both, what books did you read?!
Careful, the preppers I know think your 9mm is adorable.
9mm is the most common sidearm cartridge on the market. Works fine for the guys in Delta and DEVGRU.
Preppers think sidearms are cute.
Preppers think they'll be the local warlord, but warlords rely on some loyalty, not being the weirdo loner sitting on a weapons cache with zero friends to guard it while they sleep.
Basically, these guys: https://www.nytimes.com/2025/11/22/us/texas-men-haiti-island...
That's all right. We'll let this .38 special do the talking.
They’re talking about valuable goods to barter with in an apocalyptic scenario, they did not say they would go raid the preppers. At least that’s how I understood the comment.
I don't read it that way.
Anyone known to be a prepper is a really great target in an apocalypse. One-stop shopping and typically doesn't work well with others!
Exactly my point, which is why having the addresses of 10 preppers would be more valuable than gold. So you can give the addresses to some badass willing to risk his life in exchange for goods.
Why would they give you the goods once they took them by force from someone else?
Those preppers couldn't even handle two weeks staying home during COVID.
All hat, no cattle.
I think you may have an inherent selection bias there
All firearms are deadly. Your prepper friends are not serious if they think certain kinds of bullets are too adorable to kill them.
.223, while we're on the subject, is used in the AR-15.
They are all deadly, but out of any long gun besides .22, .223 is pretty much the weakest. The entire benefit to using .223 or 5.56 is that they are light so you can carry more ammo to enable suppression fire after traveling 20+ miles on foot.
If we were to judge people by the guns they carry, I would rank ar-15 carriers the least worrisome. They would be far less likely to be a decent shot on average, they are the least likely to know anything about actual firearm tactics and try some hollywood or COD:MW run and gun, and will likely be the first ones shot when they decide some 70 year old farmer or hunter is an easy target who suddenly blasts them from a dark window or tree stand.
556 is used in real ar15s. 223 is for people who couldn’t afford the m4 bolt carrier group and upper. Aka: poseurs.
One kind of person who won't matter when SHTF are gear snobs.
"it matters not whether a bullet is 9mm or .50, as long as it hits enemy soft tissue"
-Deng Xiaoping (paraphrased)
What matters is that its hugely common and works in easy to repair weapons. In an actual scenario where you would need such things, having a fancy ass rifle with only a few parts to fix it is fucking useless. AR-15s and 9s aren't the best weapons out there but they are ubiquitous.
I honestly do wonder, has there ever been a time where two parties have been in a confrontation and one party pulls out a 9mm, and the other party reacts to it like John Candy's character in National Lampoon's Family Vacation reacts to the pellet gun?
There are countries where the monetary system basically is cash (lots of people don’t have bank accounts).
There are definitely issues with that, but it does work in the sense that people can buy and sell things and survive, provided the government is competent enough to provide good quality notes and coins and resists the temptation to print too many.
After all it’s how western societies worked throughout human history until relatively recently
It is pretty common to for credit card systems to go offline temporarily at grocery stores and restaurants. Always a good idea to have cash.
I haven't carried enough cash to pay for groceries or dinner at a restaurant in over 10 years.
You don't have to carry it with you to keep some available for emergencies.
If the restaurant card system goes down I’m leaving and telling them to bill me.
Chase still offers embossed cards that can be imprinted. I think amex business cards, too. They can still mail in the drafts.
Because credit-card systems ostensibly don't rely on debiting actual funds, they should be resilient and continue to work offline; when you use EMV (chip+PIN) it's the card that authenticates you and authorizes the transaction, not the bank down a phone-line (and who can forget credit-card imprinter machines?)
Bullets and shotgun shells.
lead maybe
I prefer ammo, canned goods, solar panels, and a well. And maybe a doctor or nurse practitioner friend.
Gold is good. You need a fairly large chunk to bludgeon the marauding hordes though.
I don’t think cash is gonna help if the entire monetary system stops working - gold maybe?
Yeah, I'll just get out my pocket knife and carve a little chunk off that big bar of gold when I'm trying to barter for gas at the truck stop. That'll work.
I would happily trade people for gold because there is no way at all they would ever get even close to the value of their gold in goods. You could probably get thousands of dollars worth for some beans by time they decide to start trading away their gold.
Doesn't help if the world truly goes to shit and doesn't recover for decades, but if there is any kind of recovery then the gold would be worth something again.
I only carry gold pressed latinum
can't eat gold
Actually, you can: https://barnabasgold.com/product-category/edible-gold-leaf/
You can also eat rocks. Neither will keep you starving to death.
You can't send a single cent of your physical cash over the internet into someone else's bank account who is also experiencing this outage. FedNow is likely not working either.
On the other hand stablecoins such as USDC can be sent over the internet on blockchains such as Tempo, Solana, XRP and Stellar worldwide, 24/7 in less than 5 mins and with extremely low fees.
To Downvoters: Come on. You cannot send your physical paper dollar notes to someone else in another state or around the world in less than 5 seconds.
I'd be annoyed if I was a BOA or Chase customer too. But the truth is that blockchains such as Tempo continue to work anyway worldwide 24/7 without fail.
I knew eventually a cryptobro would show up.
The U.S. government is very interested in keeping the banking system going and will throw all available resources at doing so.
When do people start demanding some sort of recompense for outages, I think Microsoft will decrease the cost of your licensing if they break an SLA. How about $100/hr of downtown into each users effected account?
My nontechnical co-workers see the Microsoft 365 outage, the FAA ground stop, and things like this as one interconnected problem. They aren't entirely wrong, but the general fear is hard to dismiss
They are at least interconnected in the sense that we are struggling societally to do big, hard things that we could do a decade ago.
Roman citizen wondering why the aqueduct isn’t working kind of thing.
It doesn't seem hard to use two angle of attack sensors if you pitch the nose of a plane down based on its output.
Interestingly enough it took about four generations, a century, for someone to go back in and investigate the aqueducts properly. I think a lot of this is due to previously easy, simple things being ignored, in part because they are considered menial. The basics of maintenance are easily forgotten.
> The basics of maintenance are easily forgotten.
Also undervalued by society as a whole and politicians especially.
They'd rather allocate funding for a new bridge they can point at and say "I did that" than properly fund the unsexy work of making sure the existing bridges are maintained.
Now apply that across most western democracies, repeat for decades and we are where we are.
The countries doing their vast infrastructure build outs will likely repeat the pattern in a few decades.
The other issue though is that with 24/7 connectivity, any unplanned downtime can feel like a major outage. 50 years ago there were lots of outages, to, but people didn't care as much, they just went outside.
50 years ago, an outage in Seattle didn't affect me in St. Louis as a rule. I'd go to the bank, hand the teller a check, they'd eyeball it, maybe hand me some cash, and process all the things on paper.
When banks started moving to computerized systems, maybe a bank would be unable to process something... but not multiple banks. I can't recall any "outages" that affected banks nationwide for any period of time. Walk into your local bank and find out they can't do something because their big system at the mothership is down? Perhaps.
Consolidation likely plays a part in this as well. I can't speak to the FAA/ATC systems 50 years ago, I have no recollection of what they were like then, but the only nationwide system I can think of around that era was Ma Bell. And I don't honestly recall any nationwide outages back then. Regional, sure. You might have a problem with part of the country getting calls if there was some kind of emergency... (e.g., everybody trying to call their grandmas in Kansas after reports of a tornado) but I don't remember there being any nation-wide problems like that.
We had to count on actual snow for snowdays. :-)
The manual processes don't exist anymore. If the IT is down, nothing can happen.
> And I don't honestly recall any nationwide outages back then
There was this one in 1990 [1], but it's notable because the system was otherwise very reliable. Also, this report says only 50% of call attempts failed, presumably most people would try their call a few times and likely get through.
[1] https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collap...
I would also say that there are more single points of failure on the internet than there were 20 years ago.
In 2006, Amazon had just launched this new storage service called "S3." It would have struck people in 2006 as bizarre to think that AWS (and Cloudflare, and GCP, and friends) would one day be so crucial to the internet that a bug there could take down dozens or hundreds of companies.
And for a lot of companies (I know this is true for Capital One), the sole redundancy plan for the vast majority of services when US-East-1 fails is to fail everything you can over to a different AWS region.
Anyone who hasn't embraced the 24/7 connectivity has been priced out.
50 years ago was 1976. Basically nobody had a computer at home. "Outages" were not a thing.
50 years ago my parents were getting power outages on the somewhat regular. I have had two in the last decade.
I had one last week. And multiple each winter. Its highly environment specific. Unless you're saying you still live in their house?
I had one this morning. I live in the middle of a metro area of half a million people.
We still had a banking system, we still had phone lines, etc. But yes, the general lack of an expectation of always-on connectivity meant that service interruptions were much less of a big deal.
Fifty years ago the store owner just wrote down your name and you could pay the bill next week.
One of the coolest things I've seen, system design wise, was working at libraries when the tech/power is out for a given period.
Every service except digital lending/materials continued to function, and there were fallback processes for recording everything that happened so the information could be added back into the digital system once the tech was fixed/power was restored. Specifically designed forms that were easy to use and easy to scan, an exact understanding of what needed to be recorded and why and where it would go later, etc. And a staff of people who were meticulous enough to actually follow through.
That isn’t something that has changed in the last 10 years.
I originally missed OP's comment about "a decade ago", but I don't think our "ability to get big things done" has changed in the last 10 years, nor the number of major outages.
When I hear people talk about "we can't do big things anymore" in the US, they are usually referring to mid 20th century stuff a la https://patrickcollison.com/fast, not stuff that we did 10 years ago.
The topic of TFA and this thread is that these major outages appear to be happening more frequently this year - or in the last month - than we've seen in the past.
I do not believe the FAA has ever grounded the entire country due to a network upgrade going bad. So that is possibly an example of a hard thing we could do a decade ago, that we can't do now.
The thing done today weren't even computationally possible a decade ago. The bar for competitive software is rising exponentially.
> The thing done today weren't even computationally possible a decade ago.
Not the things that are breaking, no. The FAA was doing ATC a decade ago. We had email and cloud servers a decade ago. We had bank websites a decade ago. The scale of those things today is perhaps larger than a decade ago, but not enormously larger; indeed, the increase in scale in those things is much smaller than the increase in computing power in the world from a decade ago. So these things should have gotten more reliable--but they haven't.
Perhaps we would be better off if businesses simply couldn't get this big.
we skill-squeezed ourselves as an industry.
we made these systems simpler to maintain, to reduce the amount of skill we need to pay for to maintain them. And that does reduce the maintenance costs, at the cost of more compute. but who cares about compute costs where compute per unit of money doubles every 18 months, amirite?
unfortunately it also increases the competence required to change these systems without degrading their performance and usability.
since no one wants to pay that extra cost either, we let the systems degrade a little bit at every change instead, hoping they degrade slower than hardware improves.
that bet was bound to lose eventually, because of laws of physics, and because that degradation pattern affects hardware too.
and the double whammy is - with little demand for expertise - the supply is effectively gone too. so now that the walls have closed in, there is no way out.
Important stuff is the same. Banking doesn’t need fancy stuff
Do you think someone could take a look at the things you do for work and accurately say whether it needs "fancy stuff?"
Both the FAA and the Microsoft services that failed have been around longer than a decade
I'm pretty sure I was able to do online banking a decade ago. USAA even had working 2FA at that time.
No we definitely had banking software in 2016.
We definitely did. And stuff like this happened back then too [1]. And even worse stuff before then [2].
[1]https://www.fintechfutures.com/baas/bt-fault-hits-uk-banks-d...
[2]https://abcnews.com/US/hackers-block-us-banks-online-service...
"Nothing existed before I was an adult" type vibes.
AI revisionism is out of control generally tbh lol
I've been working in software for 30 years. This is the largest collection of systems to be down at the same time in that period. I have been literally experiencing a degradation in online services over the last year or two. Its either poor quality AI generated code, or your generation is just incompetent. Your call...
If I were you (or was working at somewhere with an outage), even if it wasn't AI, I would still be blaming AI. Because the alternative means you shouldn't have a job in tech.
> The thing done today weren't even computationally possible a decade ago
I was able to log into my bank's website in the late 90s, FWIW. Yet today I can't seem to do that.
that's not really a good characterization. we have known this was a house of cards for decades. someone just turned on the fan.
>They aren't entirely wrong,
They are entirely wrong.
None of these outages, as far as we know, have a common cause. They just happened by coincidence inside the same news cycle.
Nor is there any evidence to blame them on AI, or tech layoffs, or hackers from Iran. It's more likely something boring and technical that doesn't fit into anyone's narrative.
My guess is a total disrespect for actually qualified people. You know, the ones that specialized in one thing, for their whole life. Every company tries to see how few people they can have and still run fine. And move fast. And no trainings. And no recognition for the people that could actually train other people. I think all these come together at some point and you get multiple catastrophic events in multiple industries.
Nailed it. Downward pressure on salaries, layoffs, outsourcing all drive technical excellence out of an organization. Then, when companies DO hire it's very hard to find competent people (and I mean find them at all, not find them at the cutrate you want to hire them at) and the people you do hire are just seat warmers
I would ask a few questions:
1. Does there seem to be a larger concentration of outages recently? (I would say yes.)
2. Is there actually a larger concentration of outages recently, or does it only seem that way because of an increased focus on this? (I would say that there is really a larger concentration, but I'm less certain about this.)
3. If there is actually a larger concentration of outages recently, is that because of AI? (This is the point at which I become pretty unsure either way.)
4. If it is because of AI, how exactly is AI causing this? (I think more widespread vulnerability discovery / exploitation is more likely than "everyone is vibecoding everything now and that's screwing everything up", but I'm not entirely sure.)
5. Is this just transitional growing pains, or is this a new normal, or is this the beginning phase of a pattern that is only going to accelerate? (I don't know the answer to this one at all! I hope it's the first thing, but I wish I were more confident.)
What evidence do you have that staff reduction is not at the root of some of this?
Sorry, are you asking them to prove a negative? They said there was no evidence it was.
Or maybe the more evidence there is that's unrelated to staff reduction, the less likely it is a big factor?
The thing is, we don't know.
You saying they *are* entirely wrong is as baseless as their claim. Sure, you can say they don't have a common cause, and I agree with that, but chances of these events with different causes occurring so close together is very low and the chances of a same root cause is probably more plausible.
It could be some people rushing to get code into prod before the holiday freeze, or just a coincidence.
Either way, now that AI is more capable, you can really mess stuff up at scale.
The messing stuff up at scale is more because LLMs aren't capable, and people treat them as though they are.
Is this how the "AI will lead to the end of the world" type stuff I've been hearing about starts?
Code freezes for the holidays were several weeks ago. The code would have been deployed weeks ago too. And if AI writes better code than you, that says more about you than AI.
As history has shown again and again, blaming something only requires political will, not evidence.
No, it all comes down to the decision-makers at the top being bad at their jobs.
One way or another, they've tried to avoid addressing tech debt, avoid hiring and retaining good people, and have more or less been kicking the can down the road / running on the edge without knowing it.
Or else they did embrace AI-driven "move fast and break things" approaches, and so they broke things.
They flip-flop between maximizing profits and draining operational divisions of any resources, and hoping to sprinkle a bunch of IBM magic pixie dust all over their infrastructure to solve all their problems without needing to understand any of it. You rarely find one that strives to understand problems deeply and invests in long-term planning and maintenance before a crisis hits.
The result is old systems that eventually break, system upgrades that go bad because nobody understands them, security holes that someone might have hacked into, etc. All the same basic root cause at the top, which is that most of our leadership class is full of incompetent short-sighted narcissists. And it "works" because most of the time it pumps the stock due to lower costs, and we only see the fraction of times that it causes headline-grabbing catastrophes. On a long enough timeline, though, they'll happily drive our entire society off a cliff--and we're getting closer and closer to that (at least politically).
Not trying to be that guy, but they do have a common cause: networked compute.
There are many more systems in the world outside of it, even if it is relatively ubiquitous at this point.
I am assuming you don't have a direct line to each of these outages, and you have no proof that there is no common cause. A common cause could also be something like "poor production deployment practices" or "reliance on AI built software", in which case...maybe they aren't entirely not wrong.
It's not my job to prove they aren't related. That's the default; most events are not related.
It's your job to provide evidence they are related, and you have nothing except speculation.
AI has been finding new critical 0 days at an increasing rate so it's actually not unreasonable to think these might be related.
Yes, but you have no proof while saying someone is for certain wrong. Seems like the burden is on you to back that up /shrug
It’s actually hilarious that you made the irrefutable claim, and then you are telling me to find proof.
It's extremely funny how many of your replies are proving your point exactly. One reply says "Yours is easy to hold since it can't have evidence" when it is simply stating that _there is no evidence of a claim_. Imagine that, "There's no proof of X" is somehow a magically unfair argument because it can never be proven! Wow! Nobody has ever thought of this before. Guess we have to assume there's inherently merit to X, then, and don't have to worry about actually constructing a proof of it (that requires patience and knowledge and first-party sources and does not get you internet points).
The boring news is that laying off 100th of thousands of IT workers because of a newly promised, never-seen-before technology, does indeed cause trouble. Who'd thought that these people did in fact create value?
Genuinely, what would it take for you to consider that it could, in fact, be the AI push?
I'd want to read the post-mortem analysis from the team that diagnosed and resolved the fault.
They'd really be the only people with information about the cause of the issue; if they blame an AI-generated code change or the actions of an AI agent, that would be good evidence.
Where there's smoke...
"As far as we know" just indicates a lack of data, not evidence that they're not related. This is especially relevant for a political administration whose response to COVID was to bury their head in the sand by hiding data. Also: A public comment of "AI is hacking all our shit" wouldn't exactly calm the public.
It's fine to say, "I'll wait until we know more", but in the meantime AI progress continues and the sheer volume of cyber vulnerabilities it's uncovering is undeniable. It's not surprising that some state-sponsored bad actors would be use this as an opportunity to infiltrate too...
You also don't have any evidence that they are related.
In the absence of data, you are assigning blame to things you already hate, like AI or the administration. Motivated reasoning is highly prone to error, and it's a form of stupid-brained thinking.
No blame or hate. Just hypotheses with conditional probabilities that will be updated in response to more data...
Seems to me that most state sponsored bad actors would want to infiltrate in a less noisy way.
Well, digital centralisation and interdependency without fault hardening is a big problem. Just because it presents itself in slightly different ways across industries doesn't mean it is not a fundamental issue in our society.
I don't see much of a problem with depending on AWS or Google Cloud. It's Microsoft who are particularly flaky. And always will be, because IT people choose them regardless.
You are still "centralizing" on "someone else's computer". Just because those two have not, yet, had a MS365 event does not mean they won't have one at some point. And everyone "centralized" there will have a bad day when that happens.
AWS has had them, and is passed the growing pains. At this point when/if AWS goes down, it will still have a far higher uptime than others could manage, let alone those going it alone.
Microsoft on the other hand, has proven that it will never be reliable.
AWS quite literally lost an entire region in the Middle East and has said they will not be able to restore it.
You can argue that this is a result of something outside of their control, and you would not be wrong, but if the point here is about centralization rather than simply a particular company's operational procedures, then direct strikes on datacenters should be part of that conversation.
I was an Amazon red badge in the EC2 org, fwiw, and do think AWS has excellent operational culture, far beyond what I saw while working in the M365 org, and beyond what I've seen at other tech companies and hedge funds where I've worked. But that doesn't obviate issues with centralization.
Centralization is almost a good thing for extreme cases though.
If Bank of America's central datacenter (cloud or on-prem) gets hit by a cluster bomb, taking out banking for ~half the US, you can be damn sure that very smart people are going to be working 24/7 until it's resolved.
If you have all your savings stored with a local credit union, say Seattle Credit Union which has 50,000 customers, and they get hit with a ransomware attack... well, good luck. You (probably) won't lose any actual money thanks to FDIC rules, but it might be weeks or months before their services are back online.
Similar to self-hosting vs cloud, you get a bit of a paradox where it's overall better for society if there's no interdependencies so a SPoF outage doesn't impact everyone, but things are much worse for an individual in the event a localized outage does occur due to economies of scale, or lack thereof in this case.
Anyone who says "someone else's computer" is just giving away that they don't understand the difference between some pet vm and a proper cloud infra.
One team won’t win the cup every year, the issue here is that everyone has picked their favorite while ignoring the risks of the league.
Have you tried using Azure? (I have, it is not good)
It’s a tragedy that much of European industry and public sector are so deeply embedded with Microsoft that they barely even consider other options, even when everything Microsoft touches seems terrible
This is also true in Oceania (especially NZ) in my experience. Companies here are much more embedded with Microsoft than companies I worked with back in the US and are less willing to consider other options. My company actually does most stuff with on prem windows servers and a bit with Azure. One aspect for NZ in particular is that there was no NZ region for Azure or AWS until 2025 and parts of the country are 70+ms from Sydney, so for anything where latency is significant, on prem stuck around here a lot later.
I feel like on-prem Microsoft stack is a very well known and controllable beast.
Its pros/cons are known and gaps can be addressed with 3p software. You just keep on old outdated, but proven version of software and keep the perimeter locked tightly.
its the azure cloud that is the wildcard.
GP isn't talking about on-prem I believe. Small/medium businesses in Europe are much more dependent on MS than their US counterparts. Alternatives are mostly out-of-scope from their point of view.
Don't forget that MS has the advantage of non-English languages support that many alternatives lack.
> I feel like on-prem Microsoft stack is a very well known and controllable beast.
I feel like it is very unknown, there is no documentation and everyone relies on comments on random forums that are now 21 years old which tells to set undocumented registry keys to undocumented values.
> It’s a tragedy that much of European industry and public sector are so deeply embedded with Microsoft
It's not just Europe! :(
How quickly we seem to have forgotten how often aws us east went down.
Yeah the FAA didn't see a problem with a single point of failure either look how that went. It's not flaky until it is.
They’re expensive and centralized. I always thought the general wisdom was to bootstrap your business with serverless infrastructure and then eventually work your way back to an on-prem-first philosophy. I guess a lot of companies skipped that last step to…save…something?
As a technical person, I tend to find it hard to dismiss as well. What's easier to explain. Someone said it in a comment, someone keeps pushing for the cheaper solution cloud infra was cheaper and now we see the cost of that.
I swear nothing works fine anymore, the world feels broken. Maybe those particular incidents are not interconnected directly but they are connected in the ether in the sense that everything is incredibly complicated and precarious.
It’s natural to turn to AI based development as one of the causes of the increase in large scale outages.
I prefer to ask a different question. Will all this AI development delivering us amazing new software? Are we just going to get 1000 free clones of Adobe Photoshop or can we expect some really exciting groundbreaking new tools? Perhaps all the really clever use of AI development is in the pursuit of better AI and if that’s the case, why?
Yeah, I'd probably extend it to software too: which software actually delivers real value to society, and which is only created for software's own sake.
As to why, well economy must grow, no matter what.
It boils down to leadership. If all C-suite asks for is "more, faster and cheaper" then enshitiffication gets worse.
Unfortunately it will take many more outages for folks to realize that serious engineering talent has great value in the AI age. It's their judgement that helps speeding headfirst into building unreliable unmaintainable and difficult to extend systems.
I'm as technical as most of us ... and I'm lightly suspicious ...
Not the same root cause of course, but there's something in the water
I have a predication: In the next few years having any connection to the open Internet will come with massive risks. I.e. if you're not a government, military or $$$ company you won't be able to afford the steps it would take to harden said system against the onslaught from AI attacks.
Some other thoughts:
* Will the Internet become balkanised?
* These smaller networks will require a lot of personal information to even gain access.
* The return of the Sneakernet?
It is an interconnected problem, but it's the problem we're all already aware of. All these businesses and the FAA have gutted their operations teams. There's too much to do, and AI doesn't take less time.
they're interconnected in that we're all getting really overloaded by the workload created by AI, and even though we can use AI to get through the workload as well, we humans still have to be aware of everything flying by, and I think things are falling through the cracks more often. These days I work on ten or twenty issues at a time. it's a little nuts
Internetworking
I think you mean internotworking.
Overall uptime is better than pre internet 9-5. It's so typical to become so used to technology that any disruption seems bad.
This misses that expectation versus reality is key to how a human experiences something. If it's 20 degrees F in November where I live (MI), I'm going to grumble and get about my life. If it's 20 degrees F in July, I'm going to be freaked out. The same situation (no access to banking outside of business hours) can have radically different context and communicate different things to onlookers. See the former USSR's reaction to seeing Swan Lake on TV in 1991.
The absence of something that once existed is not the same as that thing never having existed. A null entry and a non-existent entry aren't the same.
Yes, we should all stop complaining about unexplained technical interruptions in our infrastructure and go back to being grateful for every minute that my bank's website is open on the weekend.
/s
Any reason why it isn't the case?
Because there’s not one, shared cause between the outages. The lay person claims it must be AI, but that’s a bit meaningless.
The reason I say not entirely, is because there is kind of a shared cause. One way of looking at is is that the root cause is complexity, optimized for efficiency and speed rather than robustness. It's a feature of the economic superorganism we are operating as part of. See Joseph Tainter on Collapse of Complex Societies, the work of Nate Hagens, Schmachtenberger & Liv Boree talking about Moloch, etc.
Or put a bit more cynically, it is a symptom of a society-wide drift towards lower accountability and duty of care. It's a type of tragedy of the commons. Selfishness and externalized cost/risk means nobody optimizes for systemic reliability.
People "vibing" everything is just another component of this, not the root cause. Using AI tooling is just the new technique for what used to be called phoning it in, or going through the motions, or faking it until you make it.
Unfortunately, it's nearly always been an effective tactic for a selfish few. It's problematic when we don't have an equilibrium that keeps it in the minority...
Yes, that's how you take separate causes and transmorgify them into one omnicause, by muddying the waters until you can't see the differences anymore. Don't do that! It just adds confusion.
(The same thing happened with separate company's problems becoming "Big Tech." It didn't help.)
Once you give separate things a shared name they become the same thing. We could call those in the claim to be "race to the bottom effect" or "incentives against resiliance". The water is no longer muddled, its now a slushie.
You could call it "category theory" or something.
Snarky, but there's a point: Abstraction leads to viewing different things as the same thing. They are - at one level of abstraction. And they aren't at other levels of abstraction. It's not a problem to have both viewpoints.
The trick is that, when someone blends things together, to not lose the fact that they are also separate (at other levels of abstraction).
So I have no problem saying that "optimizing for efficiency and speed" is the common cause of all of these problems, and saying that they all have different (immediate) causes. Both can be true.
Sometimes these abstractions are useful for design or analysis, but maybe not for explaining them to others? A library or programming language will probably be more usable if its users don't need to know any category theory.
I'm not sure "optimizing for speed and efficiency" can count as a common cause, though, because it's not a single event. It's more like a set of events that have something in common.
Maybe we're not looking far enough up the chain. Everyone used to use forums and other websites run by other people (eg Substack) to read and discuss about things they don't understand.
Now when people have a question, they ask AI, which is wrong a lot of the time.
What does the "pro" person claim? That it's Iran?
That there has never been an incentive to build quality software. As a result we spent 50 years building our society on top of poor quality software. We're paying for it now and will continue paying for it until the incentives change and quality becomes a focus.
I suspect there are incentives for Iran not to be identified as the culprit right now whether they are or not. If it is indeed Iran that would show they have the capability to do this and maybe the administration doesn't want that to get widely circulated. (and if it is a state actor it doesn't have to only be Iran - could be a coordinated effort - NK just did the big Binance heist and Russia can do this kind of stuff too)
NK's heist was against Binance, not Coinbase.
The main incentive for any nation state to avoid attribution to direct and material harm to many Americans is that they risk awakening American bloodlust. Americans are unhappy about the current Iran situation and wish it didn't start and that it would end already, but sufficient Iranian escalation would turn that around real quick and suddenly there are boots on the ground among other things.
From the other direction I could see that the administration wouldn't want to attribute incidents like this to Iran because it would just make Americans more unhappy that the administration decided to poke the hornet's nest when they really didn't need to. And like with not anticipating the closing of the SoH, they completely dropped the ball on cyberattacks as well.
That it’s a nuanced mess of factors. Like reality always is.
Imagine a tree of relationships branching up to the very top labelled, "Because the Universe Exists" as being the root issue that relates everything.
Saying it's not related applies an assumption of what level of specificity you are thinking about. And that's how you quickly de-sync with other people, especially "lay people" who look at you funny when you claim there's just no way these things are related when it's hard to believe that based on experience alone.
As other comments suggest, it probably is related to an extent. Related by complexity of our Internet-based systems, related by age, related maybe even by enshittification of expertise, etc.
The further up the tree you find these relationships, the less specifically actionable they are (you cannot CVE enshittification), and the more they are about "what are the overall social/cultural/human issues we're grappling with in this era?"
Aviation got a whole lot safer when we started looking at the systems causes instead of the specific ways individual people, airlines, manufacturers screwed up. Ditto for anesthesiology, cars, etc.
Sure, focusing on individual small causes is helpful too, and it's a lot harder to do the higher and higher level system stuff. But, it can be done.
edit, P.S.: It's worth keeping a bare level of suspicions that there could be a common lower level cause, whether through inadvertent dependencies added or through hostile action.
To be fair, have there been post-mortems for any of them? Besides vague tweets or press statements?
https://www.365fanatic.com/wp-content/uploads/2026/11/PostIn...
That's the preliminary post mortem from Microsoft.
I don't think we'll ever get anything honest out of the administration for the FAA outage unless Congress really wants to dig into it and make the results public.
Oh wow. Time for a "latent bug" drinking game?
> The lay person claims it must be AI, but that’s a bit meaningless.
Must be the water then
Is there?
Apparently the FAA outage was caused by the pressure to do things quickly, fueled by the AI urgency.
It feels like all the IT industry is feeling this pressure.
a) this is not known, there is no reliable data on either the FAA or new bank issue; and the information forthcoming from Microsoft may obscure primary causation
b) there is also a risk of sophistry and language games obscuring common sense truth
In the world where a state actor is engaging in active sabotage as part of a cyberwarfare effort, there is one shared cause.
Which exploits are responsible in each case may differ. Still one cause.
Do I think it's Iran?
No idea.
But the point is, none of us here have any idea at present.
I can see why people think it. I don’t think I remember a 7-day stretch like this…. ever?
- Massive Microsoft outage - FAA ground stop - Now this outage
Definitely a black swan at least? It’s not even the number of failures, it’s the scale.
Also last week, a hack of Nat'l Weather services web site (but not affecting services) and a minor outage at some Amtrak services on the East Coast.
It's true that we don't really know, and my first instincts lean toward hacking by a foreign government, but then I start thinking about media reports of shark attacks every summer. Shark attacks aren't unknown, but certain media channels like to pounce on them for ratings, and after a while to the average person it might seem like Sharknado. Stuff goes down all the time, but in this moment we're just extraordinary conscious of it.
software quality is going down in general
not sure if vibecoding is a cause or consequence, but it’s related, it requires as an axiom to stop caring about internals. And it seems to require disregard for quality too.
Basic competency is going down. Everything is outsourced such that no one knows how their critical systems work.
Competency going down, timelines being rushed, project managers and CTOs forcing new tools at breakneck speeds.
i'm all for experimentation and adding new tools when they're shown to actually help, but it seems like large software companies (especially banks and e-commerce) just don't care about quality anymore.
I think quality of most products is seemingly going down. To me it is a likely consequence consequence of increased financialization and money being expensive.
I worry that this isn't the right metric for diagnosing whether there's a problem. I'd argue that we've had instances in the past where a massive increase in how accessible programming is ended up being a net gain in the long run for software even if the average quality went down (e.g., having computers move from giant room-sized mainframes that only companies and governments could afford to sitting in the living room on a table probably reduced the average quality because untrained amateurs could start write software even if they weren't skilled enough to be employed). I could potentially be convinced that this isn't the same and that it will end up being worse for software in the long run with a stronger argument, but "software quality is going down in general" isn't convincing enough evidence for me.
All of this is also downstream of what I think is even more important, which is what's better for society in general. Obviously low quality software is not better for society than high quality in a vaccuum, but I also don't think there's a single obvious objective answer for whether the influence of software on society is a net positive or negative today compared to, for example, when computers were all mainframes and all software was produced by people paid to write it. There are a lot of differences in society today compared to back then, and software certainly has some role in how we ended up here, but trying to attribute exactly what and how much is not an exact science.
This is a lot of words to basically say: if you're confident that this is a problem, I think it's important to state exactly why, since I think the interest parts of the discussion are in how you've concluded that.
Contrary to "vibecoding" (which is increasingly becoming how anyone programs) being a cause or consequence, it'll probably be part of the solution.
> Contrary to "vibecoding" (which is increasingly becoming how anyone programs)
Commonly asserted but not true. Most programmers are not just letting the model take the wheel.
> it'll probably be part of the solution.
LLMs can't solve the slop problem they themselves are creating.
Here's my “favorite” chart showing the steady decline in software quality over the last 25 years: https://trends.google.com/explore?q=%22something%20went%20wr...
That's more like the rise of Google (which may or may not be correlated to software quality), as "something went wrong" is an error message popularized by them.
Software quality peaked between XP SP2 and the victory in the market of node.
Maybe node software was good initially, but its model was unsustainable, it relied on outsourcing to volunteer programmers, relying on such programmers means increased vulnerabilities, supply chain incidents, and low quality of software.
Vibecoding is an extension of the culture of overrelying on massive external dependency graphs. It's a preference for visual confirmation that something works and a disinterest in how it works. I think it's also related to the tendency of content feeds that are based on likes and swipes instead of searches, instead of specifying what we want, we are reduced to approving or dissaproving (until we are removed from the loop entirely)
considering how the experience with most SaaS products right now is insufferable with broken states and UI, I imagine the same broken behavior is leaking in every other area of product development, so I agree
Source = trust me bro. What’s that bias where things are so rare that when they happen it makes news, so you think they’re very common (eg plane crashes?).
Uptime and quality of software is about 10000x better than it was 10 years ago. Things just work all the time now.
Also, source = trust me bro? How can you make the opposite claim without backing it up either
> Uptime and quality of software is about 10000x better than it was 10 years ago
Extraordinary claims require extraordinary evidence. You're being the 'trust me bro' in this situation.
I'd argue that just illustrates the point that it's impossible to have an interesting discussion without a stronger claim from the parent comment. If I say something that's too vague to falsify, and you can just as plausibly claim the opposite, you're not at fault for failing to turn my claim into a more rigorous one.
One might argue that lazy sloppy work and the generally pathetic state of IT maintenance is the common cause.
My dad is a doctor. He keeps asking me similar questions.
I told him it's just like how he keeps telling his patients to watch what they eat, drink and smoke less, and get some exercise, but they don't, and then they have a heart attack and wonder why.
If an unfriendly foreign power were attempting to impose on America the insecure and unstable vibe-coded digital infrastructure we’re building today, we’d probably view it as an act of war.
Everything you said has been true for a long time, vibe coded or not. How much of the Internet proved to be exposed to a supply chain attack on node? That wasn't AI, that was just how things worked out.
Or the H-1Bs
I’ve had people asking me if it’s Jensen Huang’s doing.
I guess for most “technology” is one big blurry mess.
Same with my parents, until I told them if FAA comms ran on teams and outlook we would have been boned a long time ago!
There does seem to be a correlation with the fall in use of Substack and the frequency of these fuckups.
Maybe Jensen was what was keeping all of the nation's infrastructure running
Not a bit occam razor's fan, eh ?
Haha ask them how much they'd pay for fewer problems
My nontechnical co-workers see the Microsoft 365 outage, the FAA ground stop, and things like this as one interconnected problem.
Wait till they hear about United Airlines.
Three massive outages in three business days in a row does seem like a pattern.
Or is this a period of every blip being watched more closely and becoming a headline after the recent two?
This is the obvious consequence of (failing to) replace software developers with AI slop. Despite the protests of the Antrhopic shareholders who dominate this site's comments, LLMs are hideously poor at designing software.
We (on HN) as generally technical people (despite the HNers who seem to think you can vibecode a NVidia competitor) that all of these incidents are unrelated with vastly different causes.
But I think there's an increasing consensus among "normies" like your coworkers who see these as related. They don't know why but assuring people they are unrelated fails in messaging terms when it feels related.
I think this will be an increasing problem where so much more code is not only written by AI but reviewed by AI. Things will slip through and companies won't be punished for these outages in terms of public perception (because it'll happen to everyone) or by the government. We just don't have serious consequences for corporate malfeasance or gross negligence. Like AFAICT the Sackler family should all die penniless in 6x8' grey prison cells.
Unfortunately we live in a country where the Sacklers get to stay billionaires while 1 year olds are shot dead at the behest of Walmart over a pack of diapers that weren't even stolen... with zero consequences [1].
It feels like this is just the next level of entshittification.
[1]: https://www.nbcnews.com/news/us-news/mississippi-ag-says-pol...
Because they are. Trump just said that deranged democrat/antifa was cutting cables everywhere.
It is interconnected. Everyone bought the "cloud is up 24hrs" bullshit. My company had physical servers and we had 99% uptime because we had competent people. Profits are driving unsafe transitions, such is capitalism. Look at our crumbling old infrastructure and ask yourself why the "digital highway" would be any safer.
Banks, the FAA, Microsoft all have significantly higher uptime than 99%
hahahahhahaha .. no
Honestly, I can't help but hope that ends up leading to enough popular support for more regulation for AI that something gets implemented. On the other hand, there's also ample historical precedent for hoping that a mass movement heading in the right direction ends up stopping at the point that feels reasonable and then being surprised when the momentum continues and people like me looking foolish for thinking that it wouldn't, so I might just be overly optimistic.
AI is not the cause of this kind of problem and it's regulation won't make it go away. Maybe it needs regulation, maybe it doesn't, but you're missing what's happening if you think that's what these widespread failures and vulnerabilities are about.
AI is acting as a sudden precipitator for problems that have been latent and growing for at least 15-20 years.
In pursuit of maximally rapid innovation and reduced capital expenditure, the software industry adopted a "move fast and break things" mentality and homogenized code and service infrastructure over a few winning cloud vendors, SaaS providers, and code solutions/frameworks/packages/etc.
The pitch was that you could rely on downstream experts to do better work than you could, that the biggest vendors and publishers had the best experts, that have them invest the capital would look better on your accounts, and that their scale would give you the best value for your investment.
The caveat is that effectively all technology products have serious faults lurking in them, even with the finest expertise working on them (not that most actually had that), and that when everybody homogenizes over the same providers and solution the industry shifts from a bubbling froth of localized bugs to a heavily correlated house of cards where widely correlated failures manifest in widespread collapse and catastrophe.
Whoops.
AI is acting as a breeze against the house of cards, but lots of things could have and would have and did, especially as it continued to inflate year over year.
If you're saying the parent comment I responded to wasn't talking about AI, what are you saying they were talking about? If you agree they were, why are you trying to correct me instead of them?
From experience, since the parent commenter didn't specify, I would take the "coworkers" as having a vague sense of dread and distrust with no real confident insight to the cause. They're likely to nod along to any explanation they hear from anyone they think is more technical or news-following than themselves, and then maybe repeat whatever last such they heard when chatting. With the occasional exception of politics, that's how most people approach technical topics outside of their expertise. I didn't guess what they might have in mind today at all, and didn't feel the need to.
But you did speak directly to AI, and specifically to advocate for regulation of it, which is why I replied to you. You were being specific and material in a way that the parent commenter was not, and that invited a different response.
I didn't specifically claim that it was the direct cause of this though, so you still decided to confront something I didn't claim over something the parent comment didn't claim
What's actually more likley are power grabs executed while the chaos is attracting all the attention. That's how we ended up with DHS.
I don't really see how regulation for AI helps. Companies were on this crash course of trusting Microsoft (or pick your other dependency that cant be eliminated as it is a monopoly or the minimum) to provide more and more of surrounding infrastructure. Until they had AI to run simulations of outside.competence there wasn't a snowball's chance in hell they were competent enough at IT to run someone else's infrastructure.
Coming so soon after the Office 365 debacle and the meltdown in the FAA network for air traffic control which caused the ground stop, it does raise the possibility of a targeted cyber attack being the root cause.
Or, perhaps there was some change in the IT workflow a while back, which causes an accumulation of technical debt, that is starting to make itself felt?
If it were an attack, it would be something that were more than annoying. The crypto theft by North Korea, for example, was obviously a bad actor, with a clear motive/benefit.
FAA incident also came like 1-2 days after a new system was rolled out, which would be quite a coincidence if it were not related.
It's possible, but if you were a state actor, why would you want to give the US advanced warning about your capabilities and give them a chance to fix their issues?
Maybe you're just script kiddies? Or grey-hats trying to expose and create incentives to prioritize/fund repair before worse things (e.g. digital attacks as acts of war) start to happen
But I don't know, I'm just someone with some minor experience larping in security who is spitballing :)
Only if you ignore all the information released by the FAA and Microsoft about those incidents.
Scuttlebutt on Twitter is that it's a compromised Optimizely script
Could it be related to https://news.ycombinator.com/item?id=50523300 ?
Across all of tech QA is the first thing you cut. When you’re doing your job right no one cares.
i worked a pretty similar place to JPM, if qa was ever cut delivery times went up and people were okay with that for mission critical stuff. If you said 'this isn't tested well we're delaying' everyone would be okay with it.
on trading applications, and banking systems theres huge financial and legal consequences for cutting corners (look up knight capital). Pretty much the only way to get fired is to mess something like that up.
In some jurisdictions if you pushed a change that broke and it was flagged internally as not tested enough someone in the org would quite literally go to jail.
Man. When I joined $DAYJOB quite a few moons ago, I was so happy to see that we had an actual QA department staffed with smart and active testers.
All of them got laid off at the next layoff. That was before AI.
A few threads ago people are arguing with me about this.
Why have QA, every developer can simply QA and deploy, and talk to clients, and make sales calls.
Then again I don’t understand Capitalism. At a certain point these companies are so firmly entrenched they can ship buggy half working software. The consumer is effectively a hostage, all competitors are also delivering barely working products.
Btw, not trying to say you are wrong with not understanding capitalism, I find it helps to view it not as an imperative but rather as a local optima in game theory. Capitalism just works really well for its own utility functions that it self dictates upon us.
Of course, they were done testing!
nobody ever gets credit for fixing problems that never happened
Unfortunately highly advanced tech people lack “selling skills”. Hopefully this changes with ai - prompt: summarize the work I’ve done and setup I’ve implemented and how it will help the company and why my role is extremely important.
That's 'loser thinking'! https://cybersecurityawards.com/journal/the-field/proton-los...
I have feelings foreign entities gave there hands on this
I wouldn't be suprised if it's Israel. They are known for their false flags.
West Coast, 3 out of 5 of these load for me no problem (I only use 3 of these services, cant check the others). Might be an east coast AWS thing? Can't seem to open a WireGuard tunnel into my home network on the east coast though, which is unusual. o_O
BOA and Chase both work for me on East Coast.
It's really hard for me to get a sense of how much outages are on the rise - versus every minor issue getting a lot of attention.
Having success with Chase on the East Coast as well at the moment. I'd like to know the actual scope of this as well. DownDetector as a source is kind of low value IMO, but the banks have released statements. Curious if these are actually a large issue since I don't have the opportunity to run into it myself.
I can tunnel into the east coast from south africa at least. Doesn't touch AWS though.
Are you actually authenticating or just landing on main sites? The more public/marketing type pages are often hosted separately from the account platforms.
Signed in via mobile app.
Why do people assume that this crash is the result of AI vibe coding (i.e. these companies rolling out buggy production code) versus AI-empowered vulnerability seekers? Seems more likely that a group is trying to swarm and exploit a possible vulnerability common to these banks' operations, than it is that Bank of America, Chase, and U.S. Bank all happened to roll out bad vibe code on Monday.
From the article: "...a technical issue at JPMorgan Chase caused ripple effects throughout the financial system."
Nobody is claiming multiple parallel failures. They are claiming one failure that impacted multiple organizations.
Because people want to believe that AI code is inferior, and that it's about to go poof with a return to the previous status quo.
I know the leg I'm standing on is correlation but did we have such widespread outages pre-LLMs? The two that come to mind is that one time some intern pushed a config change that borked us-east-1 and Gitlab deleting their DB. And both had clear comms on the situation within acceptable time frames.
Also, I don't think it's that "AI code is inferior" but rather devs got sloppy and no longer review code. Not to say that everyone's lazy but me---a lot of systemic/environmental factors could be the reason.
Large outages are not a new thing.
People were making memes about the GitHub unicorn a decade ago.
No one is saying they are, they’re asking whether the frequency and scale has changed.
But how do we know it is because of AI vibe coding though?
It could be because some critical infrastructure are due to upgrade after a long delay and this is inevitable at some point.
It could be that AI agentic hacking has pushed for some rearchitecture in faster deployment time, which may result in big outage for industry that has so much legacy and used to big bulk deployment only.
Or it could just be because we rely more on cloud infrastructure than previously.
With GitHub specifically, yes. But I remember AWS/GCP/Azure outages were like a yearly occurrence before the 2020s, and we kept having the same conversations back then about infrastructure resilience, multi-cloud, etc. that were too hard to address then and still too expensive to address now.
Maybe one major platform had a major outage annually but AWS certainly didn't have a major outage every year. More like every 3-4 years. I remember exactly two in the last 10 years that impacted the production application I run.
There have been many more than two in the past ten years. Maybe not for you, but depending on the region/AZ, especially us-east-1 as it’s aged, outages are impossible to avoid.
Much better than Azure, but that’s not a high bar.
It absolutely has. The "why" is a matter of first-party versus third-party (bad AI code making bad services, or good services unable to handle the load from everyone else pushing AI code, or a combination of the two)
> The "why" is a matter of first-party versus third-party (bad AI code making bad services, or good services unable to handle the load from everyone else pushing AI code, or a combination of the two)
Decades of cycles of layoffs and outsourcing, where the core business or organization no longer operates business-critical or organization-critical functions.
Or, the thing that has happened on a huge scale in much less time than decades.
Failures aren't just a product of AI-written code in the infrastructure, but the infrastructure trying to bear the load of everyone else's AI-written code at massive scale.
The crowdstrike outage was much much bigger, both in reach and in impact. It took days to weeks to recover from.
Remember when AWS messed up their BGP Routes?
edit: it was Facebook, not AWS
But that’s the point around correlation =/= causation. What the parent comment is saying is that the cause could be that attackers have much more sophisticated tools to bring down services. The timing with AI adaption doesn’t prove on its own that lower code quality is the reason.
It's not that AI code is inherently inferior, it's that the bulk of the work engineering does now is code review and the scale of reviews has become exponential which means a lot of stuff just gets punted to agents and not carefully examined by anybody with actual subject matter expertise. Plus automation and platform work (like managing test suites) has also similarly become deprioritized and QA in general is being obsoleted as a field (for worse, really).
This is a systemic issue that's driven by AI but the responsibility lies with leadership who are still operating on the move fast and break things mentality. With everybody swinging for the fences and the median blast radius increasing and the checks/roadbumps eliminated you end up with a lot of incidents because the total number of merges and the size of the diff has increased by quite a bit per sprint.
Also most engineers have always been terrible at reviewing code. They could find most bugs when working through it line by line manually as they wrote the code and the tests, but most could never find bugs in their peer's code through inspection. Now that this - along with business and product understanding - are the primary skills they need, a lot of developers are under-skilled for their job requirements as they suddenly exist today.
Staff-level engineers can do this work because they always have, but if they are suddenly the only ones who can in an org then they are simply overwhelmed.
if only...
I don't know if it's as much as people just hoping that it fails, or the fact that in my opinion a programmer is far more likely to verify code that their reputation might depend on instead of just shoveling in whatever an agent spits out.
No the issue with AI code is not that it's inferior but that it solves the wrong problem or doesnt believe an edge case is necessary or other contextual issues unrelated to whether the code is actually correct as written.
Or maybe that cyber attacks are generally known after the fact that they are cyber attacks via things like extortion.
Like take the Binance hack for example. A naughty hacker, instead of stealing bitcoin, could have just caused mass outage and disruption without stealing anything, but didn't because bad actors usually want some financial benefit.
Like if we learn that PII was stolen, then it's be safe to say that this was a cyber attack.
That's just a strawman argument.
While there may be people arguing for that, the more relevant point is that regardless of any possible positives of AI code, there are clear negatives that are being largely ignored. Specifically in such outage cases, the volume of code being put into production with little or no review or understanding.
blaming this problem to vibe coding != ai bad at code
shit programmers have always been a thing, these same shit programmers now putting output from claude straight to production
on top of this there are literally PMs getting involved and doing their own PRs, see vscode recently
AI code is inferior, but business, the media, and regular people don't care about that. They just want fast and cheap, so you get the slop code you pay for.
I find it fascinating that anyone thinks computer programs will be different than anything else.
Clothes. Food. Appliances. Tools. Electronics.
All of these things got worse, but got WAY cheaper. All of these industries had to shed excess labor like crazy in pursuit of optimization at the cost of quality.
Why would coding be different?
Because you write a program once and then you can sell copies without extra costs per unit you sell.
>Because people want to believe that AI code is inferior,
This is armchair psychology and extremely bad one because the people who have been concerned with engineering quality have been vastly outnumbered by people who just want to be able to turn their brain off and "ship fast" for probably 15+ years, and that's been reflected in reliability long before LLMs came around.
The code these systems churn out at scale is absolute trash but the financial incentives are to produce out more of it because nobody is ever actually held liable for software breaking in our age
yes, I read some java today where I wished for the clankerd to have done it...
Think system(curl) and random-if-cascades which are never reached - they removed the system(python) in this iteration though...
At this point I think AI-code is just average and probably just fine for corporate environments/SaaS (especially given what thkse produce today), but I still think infrastructure should be a) understood by humans b) produced without installing a backdoor on the dev machine and c) a human should decide what to build and be able to reign in the machine.
it is though. at minimum qa/QC of ai code is way worse. Things are hitting prod right now in many sectors that should never have hit prod. Amazon on mobile (at least android + ff) is unusable ATM as the most recent example.
Just like humans, it varies wildly. Stating anything other than that is speculation at best.
This gets to the implied argument: that humans are inherently better to machine-generated output. The truth is there are a ton of programmers getting paid well to generate crap. Anyone's who been a part of any team that isn't some highly curated A-squad knows this is true. Hell, even teams of 1 know it to be true, when you were rushed, didn't understand the domain or stack, under personal pressure etc.
Except human code is reviewable at a reasonable speed
Before laying off half of my company the last place I worked fired QA first
The initial postmortums are out already for the 2 previous event, and they appear to have nothing to do with vulnerability.
my sweet summer child
What a snarky, condescending thing to say.
I personally thought it was humorous but I can see the perception.
Do you really believe a guy doing some wiring caused an air traffic shutdown last week?
Yes, at least in a "straw that broke the camel's back" sense.
Chernobyl was caused by a guy flipping a switch, Challenger blew up because of a faulty O-ring, etc.
That is all true, but those shouldn't be the stated reason that's reported to the public in a press-conference headed up by a US departmental head; that's how misconceptions poison the information-space for everyone.
I also wonder if Trump's administrations' spokespeople even realize they have a massive credibility problem, and that if they want people like me to believe anything then I have to hear it from a mid-level functionary, not a political appointee.
Proper root cause analysis takes its time so the last obvious fail in the chain of events is what gets published quickly.
You are in a minority, and no-one cares anymore.
You say whatever makes your stocks go up and now down, and if you don't, you are stupid because it's normalized now, there are no repercussions, and if you don't someone else will.
If the mid-level functionary tacks on, "This is why we demand that Congress allocate $50 billion in new funding for modernization!" do you still believe them?
"Chernobyl was caused by a guy flipping a switch"
No, just no...Chernobyl was caused by the staff doing an experiment that to put it mildly was wildly risky. They disabled safety systems with wrenches (meaning they couldn't easily be reenabled). Challenger blew up because staff at NASA ignored warnings from the people who made the O-rings. Saying these were "straws" is completely irresponsible. Attitudes like this are what cause outrages. Also, unless you are being dishonest, this is the worst string of outrages in the history of tech. There is a cause. It can be found. And unless you think in these terms, you have no business working with a production system in any way.
You’re not in disagreement with op. That’s the point motionlessveloc is making, it’s a series of mistakes (straws that make up the hay on the camel’s back) that points to a broader systemic problem.
I think you missed their point a bit. I believe they are saying that the cause was the faulty wiring in the same way that Chernobyl was caused by a switch flip.
In other words, the wiring problem might be the wrong thing to focus on - but it probably happened, and the systems likely wouldn't have crashed that day without it.
Do you have any reason to believe it didn't?
It's the sort of thing that is common in all systems I've been inside- which is to say, something little, to do with configuration, hardware fiddling, etc.
While AI rollouts have had issues, it is far from safe to assume just because there is an issue, it is automatically AI-related or vibe code specifically. Things always are breaking. Lines go down, servers have power failures, etc.
With this level of frequency and breadth? The _only_ common factor is AI implementation.
The trend is your friend until it bites your rear end. To give a topical answer, the Outlook issues were blamed on a software bug from months ago. While AI is being embraced en masse and causing all manner of issues, we have AI issues as _well_ as traditional "it broke" moments. The main point is we should not jump to a conclusion without stated evidence. Just because AI is to blame sometimes, does not mean it always is.
I think it is the co-occurrence of many incidents across different industries that points to potentially common causality.
LAX outage is blamed on a single computer
https://www.latimes.com/travel/la-trw-laxdelays15aug15-story...
By Tami Abdollah Aug. 15, 2007 12 AM PT
U.S. Customs officials said Tuesday that they had traced the source of last weekend’s system outage that left 17,000 international passengers stranded in airplanes to a malfunctioning network interface card on a single desktop computer in the Tom Bradley International Terminal at LAX.
The card, which allows computers to connect to a local area network, experienced a partial failure that started about 12:50 p.m. Saturday, slowing down the system, said Jennifer Connors, a chief in the office of field operations for the Customs and Border Protection agency.
As data overloaded the system, a domino effect occurred with other computer network cards, eventually causing a total system failure a little after 2 p.m., Connors said.
I automatically trusted the accuracy/honesty of this explanation from 2007. Zero trust today. We have much to fear from fear itself.
Running Window 98 SE.
I can really believe that they plugged in a misconfigured router and that a new router on a faster link was able to flood downstream legacy hardware with junk requests to the point where it started to fail.
Are RIP and BGP still a thing?
BGP4 is only still a thing on The Internet.
Do I believe a change in networking caused a networking issue? Yes?
I don't know, do computers use wires ???
I bet he feels bad about it. He should have known that flipping over to the fiber link would take out the whole FAA. /s
I don't think the average person cares tbh.
It's caused my AI, it doesn't really matter who's fault it is
The companies in question have confirmed it wasn't due to cyber attacks. That's why.
What's it going to take for the AI pushers to see that maybe this isn't the road to go down.
> AI vibe coding
What's new? What has been operating fine until the AI, cloud, and centralized computing came onto the scene?
I'd assume that whatever happened affected a single centralized service that these banks all relied upon.
Like everything else it’s probably a combination of both (as well as other causes).
Everyone is vibe coding now. It's not just you. It's not an assumption.
Vibe coding, vibe reviewing, vibe fixing. No one has a clue how software works anymore.
… and no one wants to work anymore! Back in my day we’d hand assemble code on paper and then whistle it over the phone to a 300 baud teletype in the server room … and that shit never broke! Not once!
"AI cannot fail, you can only fail at AI".
Somehow I knew I would find something like this on this comment section.
Why can't it be both?
IMO the problem is not AI-generated code per se, but rather the utterly absurd expectations and resulting destruction of best-practices and discipline that developers all over are now dealing with.
My first thought for this week's outages has been that people are moving too fast and are, as the saying goes, breaking things.
Because there’s a lot of anti-ai sentiment going around right now. Blaming it on AI slop code supports pre-existing beliefs that AI is useless. If it’s AI hackers then that not only means AI is actually good at something, it’s potentially an argument for why we need increased spending and that there really is a race dynamic at play that we need to win.
Basically, blaming AI slop supports a lot of pre-existing beliefs, while blaming AI hackers does not.
I don't really think it's that. From the people I work with, it's more the push beyond the "capabilities" of AI to the point that we have this amazing tool (AI), which theoretically should allow us to ship much better and more resilient systems. But instead we are pushed to ship really cruddy systems without the proper amount of diligence, review, and testing due to the C-suite's desperation to ship as fast as possible - consequences be damned.
Hanlons razor. “Never attribute to malice that which is adequately explained by stupidity.”
I think we need a new razor: "Never attribute to AI incompetence that which is adequately explained by human incompetence."
If companies are forcing their employees to use AI, which human's incompetence is at play?
Probably multiple humans, I would say!
"There's three ways to do things. The right way, the wrong way, and the Max Power way!"
"Isn't that the wrong way?"
"Yeah, but faster!"
Presumably this must be the "best" time for phishing/fraud ?
You forgot to add "asking for a friend"
It's never a bad idea to carry a few dollars in case of a payment processing problem.
From Microsoft to the FAA to Chase.
The MTBF singularity is the one I'm concerned about.
Yeah, this is getting way past the point where coincidence seems a viable explaination.
Randomness is not evenly distributed.
neither is incompetence, unfortunately.
I thought you were a data geek?
What did MS say caused the Office 365 downtime?
There was a thread contention bug in one if their backend services in an update that was pushed out in July and didn't happen until the system was under normal loads across all their services.
How was the bug introduced?
And only on newer hardware with more cores that only exists in production
To see what's next, let's turn to prediction markets!
This appears it could to be related to the compromise of Optimizely, I see tags related to it on Wells Fargo's home page
takes me back to the 90's...
"on a long enough timeline the survival rate for everyone drops to zero"
Where is my mind?
Did anyone have Spamhaus put them on their spam list within the last week for not putting the proper HELO?
Ours was misconfigured as ourdomain.com instead of mailserver.ourdomain.com
Here's their announcement: https://www.spamhaus.org/resource-hub/deliverability/spamhau...
what does this have to do with a banking outage?
Logged into my account at all four of those banks just fine, even submitted payments.
Thank you for this information. Then it must be nothing.
Is Zelle the commonality for this outage? All of those banks work for me, but I have been careful to never enable Zelle on any accounts.
Why do you avoid Zelle? I often find it to be useful, so if it has some non obvious issues I prefer to know.
I avoid Zelle because Zelle shows my legal name rather than a nickname to any random stranger I transact with. At least with Venmo you can provide a pseudonym to the other person. The only use case I have involved buying / selling secondhand items on Craigslist and Facebook marketplace.
I view this as a positive. If someone sells me something that ends up being broken/not as described I would much rather have their full legal name vs. a nickname.
I also don't see how you would ever end up in this situation if transacting via fb marketplace as you're not even allowed to have accounts that are not you.
Facebook allows nicknames. Zelle does not.
What are you going to do if some rando sells you something that doesn’t work? Sue them in court? I’d rather not and just eat the loss. It’s not eBay.
Not OP but isn't zelle yet another commercially motivated third party that will aggregate and mine user data for profit? If nothing else they're an additional point at which data could leak.
My bank used to let me send checks via online banking (either electronic or mailed out physically by the bank depending on the recipient) until they adopted zelle and discontinued that service. My credit union still offers the check thing for free. That's an easy choice at least for me.
It's only sort-of a third party; Zelle is owned by the banks
> Zelle (/zɛl/) is an American digital payments network run by a private financial services company owned by Bank of America, Truist, Capital One, JPMorgan Chase, PNC Bank, U.S. Bank, and Wells Fargo.
https://en.wikipedia.org/wiki/Zelle
It's a bit like Bitcoin, in that money can disappear from your account permanently in seconds with little recourse.
Not exactly. Zelle allows chargebacks against unauthorized origination, i.e. your account gets hacked.
Banks only allow a couple $k of transfers per day anyway. The company behind Zelle, Early Warning, is a bank-owned centralized fraud detection network as the name suggests.
The ability to send wires from your account is riskier... I wired the down payment on my house in one go.
> Not exactly. Zelle allows chargebacks against unauthorized origination, i.e. your account gets hacked.
And your bank will always say it was authorized.
https://www.nytimes.com/2022/10/03/business/zelle-fraud-warr...
"Banks draw a distinction between what they consider scams — when a customer is tricked into sending money to a thief — and fraud, when an unauthorized person gains access to a consumer’s account and siphons off cash."
If you're drunk at a bar and someone unlocks your phone with your face and sends themselves your money, that's gonna be authorized from their standpoint.
There's no consumer protection for any payment method that's not a credit/debit card or PayPal, Cash App, etc. which chain off credit/debit You pay for that in transaction fees; up to couple of percent.
Cash, check, wires, ACH, money orders are no different from Zelle in that respect.
Every bank has a big warning before you use Zelle that says basically “you’re not getting this money back, are you sure?”
Paypal Purchase Protection covers eligible purchases. Otherwise, yeah, it's a great reason to use a credit card and then pay off your balance.
Yes. And that's why I carry a tiny bit of cash, a few credit cards, and none of the rest in my day-to-day life.
not quite? it's more like a bank wire. except bank wires can _sometimes_ be reversed.
So can cash if you pay someone and they walk away?
Yes, and most people don't carry their entire savings account worth of cash on them for that very reason.
There are strict limits on Zelle compared to wire transfers, or in this case your insistence on relating this to bitcoin.
You can’t drain your entire savings through Zelle in a single day (unless your entire savings is $3k).
That's a life-ruining amount to a whole bunch of people living paycheck to paycheck.
You’re goal post moving. If you don’t want to use Zelle, fine. But don’t spread false info that it’s prone to drain your entire account relating the risks to crypto transfers.
And most people don't store their life savings in a single checking account?
not sure what you're trying to say. I can be stupid with cash AND in my digital life, the choice is mine.
The question I responded to was "Why do you avoid Zelle?"
You don't have to follow my advice, for sure.
The median account of money people have in savings is quite low; $3k is a life-changing loss for many. https://money.com/average-american-savings-account/
Have been getting some core.finicity.shared.noAccountAvailable.instruction errors with Capital One when trying to connect back accounts
Im having GCP problems. Anyone else?
Hmm, Apple pay didn't work for me this morning, but Paypal (with a CC) did.
“Life can change in a blink.”
- Nathaniel Rothschild Nov 5, 2026
Your daily reminder that
- Your bank balance is not how much you have, but how much the bank owes you.
- Your portfolio is not how much stock you have, but how much the broker owes you.
People owing you is a good thing, if they pay you interest/dividends for it. Or if you like pithy quotes:
- Your lack of bank balance and portfolio is not how secure your money is, but how much work you owe the world to pay for food and shelter.
Your portfolio value is how much the current market is willing to pay for your portfolio.
America is melting
The businessmen and entrepreneurs have full control of the country. We're cooked.
Better empower the bartenders and career politicians. They'll surely save us.
I'd rather have a bartender in charge than a businessman.
When you're in a hole first stop digging.
Society was unironically functioning better before the “draining of the swamp”
To be expected from the 'melting pot', I suppose
I kinda expected things like this to happen the week of the Anthropic IP0. EA sees everything as a trolley problem anf tries to create them wherever they can. Thwy will mess with critical infra to get themselves the regulation they want.
How do you think this is being caused by EA?
Its a pet conspiracy, that I know has no evidence, but EA would love to get the public in agreement with them on AI so that the public consents to them being the arbiters of AI regulation.
Went into a branch and they couldn't help me!
companies relying on a single cloud (Azure) for mission critical workloads is a big problem
In past week i have heard about lot's of outages.
Just checked with my Chase account. It's fine. Was able to login via the mobile and web app.
There's going to be a big swing to get off the cloud methinks, hopefully it benefits my career
I reckon, it would take 3 days of full outage for an average company to even consider the work needed to migrate from cloud providers or services. Obviously with some exceptions around risk to life systems, which should have an independent disaster recovery plan already.
M365 would have needed to be down for a full work week for job adverts to go out for an internal IT infrastructure team.
I doubt it, c-suite will never approve the spike capital expenditure for buying hardware now they're under a nice smooth subscription curve which doesn't harm their profits as dramatically.
Hows the cloud responsible for the outages in a way that is mitigated by self hosting in recent outages and this one?
Not needing to run at big tech scale makes self hosting easier.
You also have full control of downtime and recovery rather the refreshing the MSFT status page.
Presumably a central provider takes down your dependent services with no local ability to navigate around the outage. Removing the dependency obviates the outage risk. It introduces its own downsides as well, but that's a different discussion.
USAA also appears to be down, at least for me.
I was able to login and check my account balance
We are under attack and the enemy is incompetence or actual terrorists.
They would/will never admit it is AI breaking everything either.
I asked Claude to log in to everyone's accounts and check on them. Don't worry, it had no trouble getting in and it says everything is fine.
Yes, it's sad these systems are going down all the time, but if we wanted them to stay up we'd have to pay the people who maintain them, which means less money for the stockholders. So you see, our hands are tied.
I love how trusting everyone is after three major outages on consecutive business days, hot on the heels of months of warnings and demonstrations of AI enabled attacks, that these aren’t attacks, solely because people incentivised to say that everything is normal are saying everything is normal.
I'm a little surprised I didn't learn about this outage from Flavor Flav [1]
[1] https://www.mediaite.com/media/weird/flavor-flav-more-than-o...
All of the project managers running large-scale production systems with Claude Code are having a rough go of it lately.
Whole lotta people in this thread blaming these outages on whatever their personal software industry bugbear is with zero evidence.
So glad that coding has been solved.
Outage at SEELE you say ? Does NERV know ?
If I had any money in the bank, I'd be worried. Fortunately, I'm tapped out now :|
They, too, didn't need programmers any longer and let AI take the wheel?
The thing is, even if true, they'll never admit it. People who make these kinds of mistakes (if this is what it is) are also not very prone to honest bouts of introspection either.
Honest bouts of introspection aren't necessary to keep the line going up.
So, your company shit the bed due to your plans. So what? You went to the right school and have some experience in the c-suite. You probably have enough money to go a decade without cleaning out your savings.
Break down the company for scrap, file an IPO for the good parts, whatever. You'll make the money back.
Sounds about right
Chase web services aren't down...
Wow, I wonder if there is an underlying cause.
Thursday: MS 365 Friday: FAA Monday: banks Wednesday: Anthropic IPO
AI fearmongeeing post to boost stocks along with comments to corroborate.
Aw man just begrudgingly switched to them
I'm curious the reason.
Their website isn't bad compared to some others I'm familiar with, they have locations everywhere, WF is banned from my current area (my previous bank), local and regional banks had terrible websites and security options, and they gave me $1k to open the account. Ultimately it's about convenience.
To who, banks?
They probably got tired of carrying coffers full of gold coins everywhere. Bad timing.
The US has no natsec currently due to its leadership. There has been an intentional purging of those with real expertise at the topmost layers. Mass layoffs of experienced folks. AI use rise and all that entails. Several billionaires with motive and resources to do sabotage or cause chaos. Now add the US attacks on Iran giving huge motives for the latter to push whatever buttons they can. One could argue we should be surprised 2026 hasn't been much worse than it has been. And there are reasons to expect it to worsen much more before it can ever get better.
> There has been an intentional purging of those with real expertise at the topmost layers.
People love to say this but can never point to who these experts are or what their expertise was.
what I refer to has been documented repeatedly and redundantly across US news orgs for over a year and a half into the current admin. Its no secret and not up for debate.
Thanks for confirming you can't point to any specifics beyond vague partisan accusations, but that it's "not up for debate".
Hello, Congress? Send up 8 more experts!
your 1st sentence there is rude and false. rude alone is the showstopper
"according to the website downdetector.com"
Ugh. DownDetector's methodology is… not great.
Steam isn't down: https://downdetector.com/status/steam/
Amex isn't down: https://downdetector.com/status/americanexpress/
Visa isn't down: https://downdetector.com/status/visa/
AWS isn't down: https://downdetector.com/status/aws-amazon-web-services/
But each has a huge corresponding spike there.
DownDetector's methodology is fine, it's just noisy. The way to use it is to look at the top-reported failing sites in the ranked list on their homepage, which has been the correct root cause for every major outage I've seen. Ignore the lower ranked ones.
Google's in the top 20 (#11) right now. Down?
"User reports show problems with Google"
https://downdetector.com/status/google/
By top-reported I mean the top two, maybe up to four. It's not high-fidelity. I still think it's useful, especially since in 2026 the art of building a status page not dependent on the infrastructure it monitors has been lost.
So if there ever is a widespread outage, how will you know? I mean besides "AWS us-east-1 is down" -> "everybody is down".
> By top-reported I mean the top two, maybe up to four.
As the Rick & Morty quote goes, "that just sounds like horrible methodology with extra steps".
There's an outage > you go to downdetector > you see the cause of the outage.
Seems straightforward. They could add some qualifiers about the lower ranks somewhere, but that's the base value proposition and it works fine.
There's not an outage > you go to downdetector > it shows an outage > you write a news article, apparently.
> you see the cause of the outage
What does DownDetector say the cause of today's apparent Amazon and Google outages are?
This kind of rigid thinking is such an HN thing. The "cause" is obviously problems with payments from the bank outages, which, again, is clear from looking at the ranked list.
You can't source details that depend on technical knowledge from a non-technical audience, regardless of methodology. Downdetector is useful nonetheless.
> The "cause" is obviously problems with payments from the bank outages, which, again, is clear from looking at the ranked list.
You told me only "the top two, maybe up to four" slots count. Five are named in the headline. Only Chase has confirmed any kind of issue, hours later.
(In fact, the article has now been updated to say that Bank of America doesn't have an issue, and suggests it's a Chase-specific outage.)
But downdetector doesn’t track that financial transactions are failing. It tracks whether people are checking to see if a service is failing. It is only a charitable reading that people are only checking if they are experiencing a failure. A signal, user queries, is presented as another signal, service status, even though there is no correlation.
In much the same way, every time there’s a large conservative or republican gathering like a national convention in the USA, someone starts spreading fake headlines that “grindr” is down for that specific city. People go to down detector en masse which makes it display that grindr is down (it must be since so many people are checking), lending credit to the hoax. It was never down, and no app has city specific infra that would go down with a spike of a few hundred or even thousand people assuming the premise of the hoax was true.
From what I can gather from a quick search, Grindr hasn't gone down in response to GOP conventions, but there is commonly lots of traffic during places/times where GOP conventions happen, and it's plausible that if many people are trying to access a service from the same public wifi they could experience connectivity issues without anything being wrong on the service end.
Grindr CEO: “Yes, it was actually true that there was a significant spike in usage in Milwaukee during the [2024 GOP] convention.”
Just because someone, or a lot of people, mistakenly believe something doesn't make it true, and pointing out that mistaken belief is not "rigid thinking," it's literally just accepting reality.
>This kind of rigid thinking is such an HN thing. The "cause" is obviously problems with payments from the bank outages, which, again, is clear from looking at the ranked list.
Not to offend you with all this rigid accuracy, but what you describe is not a cause but an effect.
Make a better one and we can use it instead!
We can start by not using a bad one.
False positives and false negatives can be worse than no reporting at all.
How do you know some of these aren't experiencing a partial outage? Responding instantly to partial outages that never get reported on status pages seems to be one of DownDetector's biggest strength.
> How do you know some of these aren't experiencing a partial outage?
Is there any other sign they are? That Google, AWS, Steam, Amex, Visa, etc. are all simultaneously having significant outages right now?
> Responding instantly to partial outages that never get reported on status pages seems to be one of DownDetector's biggest strength.
I've predicted all fifty of the last one recessions.
I live close (as the fiber flies) to one of the more popular AWS datacenters so I have no doubt that there could be situations where I can access sites that others can't, or in the case of a proper AWS cockup, where I can't but many of you can.
And when it's not DNS (it's always DNS), it kind of depends on how load redistribution works. At some point I have to switch to load shedding, and so Idaho and Montana might just be shit out of luck.
If the NYC office is still able to do work, I don't want the Denver and Chicago office to overload their servers and now nobody can get any work done.
Payment not working could be considered a partial outage.
i see this comment on every single "its down" post, it's not a useful comment- it is down, Chase is partially working.
The article cites a Chase rep saying they are impacted by a bug of some kind.
The article writer then went to DownDetector and wrote a headline "Outages reported at Bank of America, Chase, U.S. Bank, Wells Fargo and Zelle" based on just that. As far as I can tell, there's zero genuine evidence for that.
Is it not true that Outages have been reported - via DownDetector?
DownDetector is a way for customers to report outages so the headline is not lying at all. It is not claiming that the root cause of these outages are these banks, or that the banks themselves have reported the outages.
> Is it not true that Outages have been reported - via DownDetector?
Yes. That is basically always the case. Bring it up any day, at any time; DownDetector is always reporting spurious outages.
This story morphed into "AI took down a bunch of banks" already because of an irresponsible headline. "Bank outages" is the #5 trending on my Bluesky (https://bsky.app/profile/did:plc:qrz3lhbyuxbeilrc6nekdqme/fe...) right now, still.
Also happened last week during the Outlook outage. It showed the entire internet down for some reason.
I always think of the time the company that acquired us put our Wiki and (some of) production storage on the same drive array and when it failed we couldn't access our runbooks. Lots of anger lines coming off of people who are permanently grumpy the rest of the time. Like people too angry to trust themselves to talk, so one word answers.
After that our operations team got most of its autonomy back.
How do you make an app that watches everyone else's infrastructure but can't be taken out by a common cause?
I’m experiencing degraded AWS ECS service today so it’s atleast a bit helpful. AWS is probably never going to be down accross all regions and all services 100%.
Isn't it just based on user reports? What explains the spike in user reported issues in those services recently?
> Isn't it just based on user reports?
DownDetector considers someone looking up "is Visa down?" a report, even if it isn't.
Click my links and it contributes to those "reports".
One provider has a major issue, and people go there and click into non-down companies, which is seen as a spike.
Is there a problem with that? Do normal people google "is Visa down?" if they don't think Visa is down?
On the other side, does a user google "is Visa down?" going to a website and clicking "yes I think visa is down" any better?
> Do normal people google "is Visa down?"
Normal people Google "is Chase down?", find DownDetector, and click it. (Note that the article has a Chase rep quoted saying they do indeed have an outage, and none from other companies.)
Some of them then click into the DownDetector home page and browse around. Suddenly everything is spiking.
Every time I've seen any one major company with issues, DownDetector has spiked across the board. As it is now. Google and Amazon are apparently down.
> DownDetector considers someone looking up "is Visa down?" a report, even if it isn't. Click my links and it contributes to those "reports".
Really? It seems to me you have to click a specific aspect of the service to report as down once you land on that page to report an outage.
I assume their methodology weights active reports higher than the implicit ones.
https://downdetector.com/methodology/
> We process tens of millions of problem reports each month and cover over 36,000 services in 72 countries.
Ask a few normies if they know what it is, let alone have filed a report.
https://downdetector.com/about-us/
> Downdetector collects reports from users and monitors other online sources for service issues.
Or look at the comments on any of the pages. "My card declined"; "18yr femboy down for DM".
Has downdetector ever been down?!
Don't worry, check here: https://downdetectorsdowndetector.com/
Has this site ever been down? /s
https://downdetectorsdowndetectorsdowndetector.com/
It's down detectors all the way down.
https://news.ycombinator.com/item?id=50521819
I can't comment on the others, but Steam and Amex were both partially down earlier this morning.
I mean, if Visa/Amex transactions are not working because their bank is down, end users don't know the difference. Card says "Visa" on it, card is not working, end user blames Visa.
Steam/AWS, I got nothing.
“Life can change in a blink”
- Nathaniel, Rothschild Nov 5, 2026
Its so much easier to generate code with genAI though.
It's so fast. You can also put "make no mistakes" in the prompt.
And we can fire so many programmers now
The degradation in software quality across the board (Spotify, Google Maps, GitHub, YouTube - to name a few) has been very noticeable for me, and I suspect AI has a lot to do with this.
The thing is - I also think AI can be used to build very high quality software. The problem, I think, is that the typical big tech/VC tech/MBA tech type company (who control most proprietary software at this point) have very little incentive to actually care about quality, and that was quite obvious even before AI.
I remember before I started using AI (I was pretty late to the game), I assumed all the bugs and instability were because people were writing software with AI. Now that I've seen how good the code that AI writes can be (CAN be), I'm moving more to the thinking that it's from AI being used to find vulnerabilities.
I've never had AI one-shot even a 10 line script that I didn't manually revise. To be fair, it usually produces something that works, but after seeing it I'll realize that my prompt was too vague or I left out some edge case. So it's the old problem with software "it does what was asked for but not what was needed."
So I speculate that a lot of problems with AI code are just that it is accepted uncritically because it builds and doesn't have any syntax errors.
There’s a pressure to deliver more and faster from management. So instead of using LLM to reduce the coding time and allocate more time for review, everything is scaled down and there’s even less time for review. And you’re responsible for more as teams are also scaled down.
Not really a great recipe for good code.
It’s more that management pushes more output and with AI it’s very easy to achieve but that will create slop.
I think majority of companies don't know how to build safely with AI. It's just kind of handed off to engineering and mostly used as one-off chat interaction to get stuff done. Very few are building any kind of harnesses around SDLC that focus on architecture, testing, security and etc. I think we'll get there at some point, but in my experience the smaller tech players are just now on-boarding to use AI in general in any kind of capacity. We're not talking about best practices yet.
Wondering if this has anything to do with the QA function being folded into the dev team in the last 15 years or so.
I always defended this, but it appears maybe we don’t have enough developers that take pride in the work they ship. LLM coding made this pretty obvious.
It’s not lack of pride. It’s because they can’t say: “ok development is done, now I need a few days to thoroughly test this”.
> I also think AI can be used to build very high quality software.
I don't, and I think your comment is rhetorically useless because you've been bullied into making this ridiculous concession.
bro... what model did you use to write that? or is that how you actually talk? Lol
IMHO the only way to build good things is to pay attention to the details. And coding agents, while greatly increasing the appearance of productivity, they are causing people to pay less attention to what they're actually producing. This is the inevitable and obvious result.
Agentic coding could be a world-changing benefit, but the current way it's being sold and used as as a cost cutting measure -- produce software faster than ever before with fewer software engineers! Get rid of those pesky experts!
We're in the v1 machines screws stage where everything is worse but with a clear path to get better .... I hope
It isn't a software or tool problem.
It is a C-level executive problem. With the FAA it is the agency directorship problem. Same issue.
These people, very literally, are trying to run the world with power point presentations and executive summaries drafted by sycophants. 70 years ago these people had actual experience doing actual work and trying to grow businesses and doing stuff at the ground level. Now they are given their positions through a combination of credentials, connections, and political maneuvering. None of which actually relates to how well they can or cannot run a organization.
When reality disagrees with what they get in their emails and presentations then it reality that is wrong. It is the messengers that are the problem, not their competency.
This is nothing new. But the failures in leadership compound over time, like interest to a unpaid debt. They start off with healthy growing companies and thanks to government protections and bailouts it takes decades to for these organizations to decay into total dysfunction. In the past these sorts of people would be removed and their assets reallocated to competent management through bankruptcy and competition. However that natural process is not allowed anymore.
All AI is doing is handing them more foot-guns.
They think they can do better/faster/stronger/cheaper with AI and if AI causes problems then it can just be fixed by paying for better AI. Which they assume is just right around the corner, always.
Just wait until they start to realize that it is
Very well put, and I think (open) AI will help competent people outcompete these clowns, which is what makes me optimistic. AI is a multiplier, but multiplying bullshit just creates more bullshit. These people are far more replaceable than competent engineers, even more so with AI.
broken windows
Part of being sane is accepting that some tools, while being very powerful, are too easily misused to be widely distributed to those lacking proper training. This is why you cannot just go and buy 5 kilos of azithromycin
Ehh I don't really buy that. Who decides what tools are distributed to who and how?
Don't want to get into a political debate here but I'll say that I think that open AI models and access to them is a benefit to society, while closed, proprietary models, controlled by tech billionaires are probably net bad for society.
> Who decides what tools are distributed to who and how?
The same people we already delegate such responsibilities to: the government; ideally, a representative one.
> the government
Uhhhh... you mean the corrupt, baby-eating pedophiles who are owned by tech billionaires right?
Cool plan bro
> some tools, while being very powerful, are too easily misused to be widely distributed to those lacking proper training.
Could you define 'proper training'? I've endured plenty of fucked-up shit from so-called 'professionals/ experts/ trained people' over the last 30 years or so.
I'm currently assisting a couple of vibe-coders improve their 'too easily misused...[and] widely distributed' (available on their sites, or github) tools - I mean slop - that benefit me in ways traditional companies have no interest in. And the updates are fast (daily, mostly) and accurate because, unlike jeremy with a pretty CS degree and a nice job, these people actually give a crap' and don't have a manager to say 'you can't do that'. Also part of being sane is accepting that the cat is out of the bag and big $oftware/ tool manufacturer better get a move on. Disclosure: my interests are not critical to anything; quite the opposite, but people are learning faster and faster with Ai learning faster and faster. Who needs azithromycin if a series 1's and 0's can accomplish so much more? For good or bad.
No, i cannot. If i could, i would be proposing concrete ideas. But I also understand why I cannot buy 3 kilos of plutonium or 5 kilos of antibiotics, and understand the modern ML models can do just as much damage while being more available.
Software is solved!
Software is solved but not software engineering
Crypto blockchains such as Solana, Tempo and XRP are still up and running while the banks are down and you have no choice but to wait for them.
What a disaster.
Great, now where can I buy groceries with these garbage coins!
Doordash accepts stablecoins such as USDC so you can buy your groceries from there. [0] [1]
No excuses.
[0] https://cointelegraph.com/news/doordash-stablecoin-payments-...
[1] https://tempo.xyz/customer-stories/doordash/
"Plans to".
By this metric, I can hop on a supersonic flight on Boom.
That is just one large example, and Shopify supports stablecoin payments right now and grocery stores built on them can accept stablecoins at in their point of sale systems.
But with that attitude, no progress will ever be made and you will continue to professionally complain on HN.
But thankfully, companies using the Tempo blockchain such as Stripe (who built it), and others do not care and already have business interest with the use-case of payments.
Queue the "It's Biden's fault" excuse in 3,2,1 ...