Long time SRE/DevOps with 20 year of FinTech experience in banks/hedge funds/crypto and having seen many, many outages (some listed here [0]):
- There is almost always a standard operating procedure (SOP)
- Just b/c there is a SOP for releases etc, that doesn't mean that people always follow it
- This can be due to many reasons e.g. laziness, being distracted, pressure from higher ups ("just push it out there and I'll take the risk" ha!), tired from having a newborn etc etc
- Doing things outside the "green zone" deployment windows therefore always carries risk
- All of this being said, EVEN IF you ARE following SOP, things still break in all kinds of weird ways
- And the end user is not often the best observer nor do they have all of the context to determine what broke, how it broke and why. (There are exceptions to this though)
Most of us here at HN work (or have worked) at a place where if you screw up a SOP the company could lose lots of money. So we're careful.
That's different than working at a place where if you screw up, people die. It's a totally different environment (or at least it used to be, can't say what happened over the last ten years). There's a lot more 9's, and a lot more consideration of making changes during off periods just in case. I don't believe this was planned for that reason alone.
- The end user in this case has a better idea of what was going on than we do.
- I don't trust anyone in this administration.
- This administration has been acting like a wrecking ball; moving fast and breaking things.
- This administration has been contracting with incompetent private sector companies in blatant acts of corruption and bribery.
- Decision-makers are all drunk on AI these days.
I don't necessarily trust the specific explanation an end user would give for this outage, since they're probably wrong about the specifics. However, when it comes to Duffy, I certainly trust him less:
> Duffy offered a new detail on Saturday, saying that a system overload occurred when contractor L3Harris Technologies Inc. was replacing a section of the air traffic control system’s old copper wire with new fiber wire.
How does upgrading copper to fiber wind up overloading a system when you're adding more bandwidth? And why was this happening in the middle of the day, on a circuit that clearly critical ATC traffic went over? Apparently, the old copper circuits were so overloaded that the bandwidth limitation was critical to imposing request limits, via dropped packets, to prevent overloading the systems on the other end? And once there was enough bandwidth, the servers got effectively DoS'd? That doesn't make any sense.
I'm definitely going to believe the air traffic controllers when they tell me he has to be hiding some other kind of shitshow behind that supposed explanation.
yep. most IR peeps have plenty of stories where a single event is just a symptom of multiple failures, unfortunate accidents, and alike. most red teams will also regale chained attacks is where it is today. it is rarely a single piece of lock that breaks, it needs several other things to be in the "wrong" state.
Your third bullet is a bane of gov contractors. Not the sales people, but the people actually doing the work. Imagine if you had an FFP contract and you finished the job within 90% of the paid time. At $1mm that is sweet $100K that the big cheese keeps. (not you, you go back to the mines and since you demoed you can do it at 90%, next time you have to do it 80% time. and, no i am not sore ;) )
tl;dr My first reaction to Duffy's statement was "you have no clear idea yet, but this is your best guess". And, we may never know.
Asking air traffic controllers for their opinion on this is a bit odd — would they even know about networking equipment swaps? I would expect that to be handled by the engineers and technicians who work on the system, not the people who direct the planes.
They wouldn't necessarily know the specifics of upgrades, but they would likely know when upgrades typically happen because it's normal to send out emails about maintenance windows.
I imagine equipment is regularly updated and upgraded without them knowing.
I'm extremely skeptical of this administration's competence, but this story is 100% finding anyone saying what you want to hear, however unreliable the narration might be.
> I imagine equipment is regularly updated and upgraded without them knowing.
Part of the reason for this outage was that these systems are not regularly upgraded. That's why they're dealing with decades old network hardware and lines and marrying them up to modern fiber. It's expensive to do the upgrades, but instead of doing regular rolling upgrades to their network components they generally do big-bang upgrades like the one happening now (that apparently caused the outage). In 20-30 years there will be another big-bang upgrade replacing the Juniper routers and fiber lines they've been installing now, and it'll probably cause similar problems.
This also happens a lot in DOD (not surprising, same contractors doing the work and probably the same companies in the program offices for FAA as DOD giving the same bad advice), it's incredibly frustrating because it's very obvious that there are better ways to do things, but the funding isn't there until it's urgent. That's why they're still using 1980s computers with floppy disks in their critical path for ATC as well. Now that those systems are nearly impossible to maintain, there is enough noise to push funding a proper replacement effort, about 20-30 years after it should have been done.
Unless I’m just tired, the article, reporter and air traffic controllers are confused about hardware switches and routers having issues, and the act of switching out hardware. Duffy stated that the switches and routers were having issues and needed to be shut down. This seems to have been read as physical hardware switches, read ‘replacements’, were happening during the day.
I don't know what happened, but for me a red flag was the specific explanation given while the event was still ongoing: "overloaded router", followed by request for more money.
If the cause was so simple , then it was unclear why it wasn't fixed immediately, so something does not quite add up (at least till an independent investigation)
I don't believe anything that Sean Duffy or his ilk say. I would hope there is a detailed and accurate post-mortem published to explain exactly what happened. However, IMHO this is just another example of the destruction wrought by Trump/Musk/DOGE. I recommend reading this article by James Fallows: https://web.archive.org/web/20261114090809/https://www.nytim...
Why would you hope that? Who would be releasing said post-mortem? Would it not be from the "or his ilk" portion of the comment? Sure, some government employee or team might produce an honest and factual report, but there's no way this administration would release it. The only way for that to happen would be for it to be leaked
After 9/11, people preaching Death to America were at least half a world away. In 2025, people preaching Death to America took up residence in the White House.
"Refute" makes it sound like there's juicy insider info, but it's just skepticism that FAA scheduled upgrades on a Friday daytime.
This was a cascading multicast failure, which can take hours to build up. Earliest reports of disruptions came no later than 10:30 am, so it's quite possible the upgrades were performed in the overnight hours on Thursday.
In my observations, any time a politician issues some kind of technical excuse for why things screwed up, they are lying. Not because they know the difference, but because they can't understand the actual problem and managed to elicit some ELI5 explanation from people who actually do know.
"Explain it like I'm a politician!" (cf The Expanse.)
Even if MS stuff was not in any critical path it still could have played havoc with coordination of an update or a change, contributing to the failure.
Also FYI foil hats are actually resonant cavities and amplify the mind control waves. All true paranoids use Faraday mesh, preferably as a full head covering.
I’m not sure which version of Microsoft the FAA uses, but GCC High and the DoD version of Microsoft 365 have been working just fine through the commercial outage of Microsoft 365
I saw a post…I think on HN…the other day. It was a technical post about making a faraday cage out of a metal trash can. Afterwards I thought to myself let me check Amazon to see how saturated the faraday phone bag market is. Lo and behold I find out there’s faraday bags for all sorts of electronics now and head meshes. I’d be curious how well any of them work, but I wasn’t expecting to see it.
I guess it makes a little sense given a lot of electronics don’t have a removable battery or physical power switch nowadays.
Long time SRE/DevOps with 20 year of FinTech experience in banks/hedge funds/crypto and having seen many, many outages (some listed here [0]):
- There is almost always a standard operating procedure (SOP)
- Just b/c there is a SOP for releases etc, that doesn't mean that people always follow it
- This can be due to many reasons e.g. laziness, being distracted, pressure from higher ups ("just push it out there and I'll take the risk" ha!), tired from having a newborn etc etc
- Doing things outside the "green zone" deployment windows therefore always carries risk
- All of this being said, EVEN IF you ARE following SOP, things still break in all kinds of weird ways
- And the end user is not often the best observer nor do they have all of the context to determine what broke, how it broke and why. (There are exceptions to this though)
0 - https://x.com/alexpotato/status/1215876962809339904
> banks/hedge funds/crypto
Most of us here at HN work (or have worked) at a place where if you screw up a SOP the company could lose lots of money. So we're careful.
That's different than working at a place where if you screw up, people die. It's a totally different environment (or at least it used to be, can't say what happened over the last ten years). There's a lot more 9's, and a lot more consideration of making changes during off periods just in case. I don't believe this was planned for that reason alone.
If you calculate the imputed "uptime" from FAA ground stops since Jan 1, 2000 (there were 3 in the past ~quarter century), you get 99.976%.
In comparison, Stripe has been at 99.999% for many years now.
As a whole, yes, but the individual Stripe services have gone down plenty of times in the the last few months.
- The end user in this case has a better idea of what was going on than we do.
- I don't trust anyone in this administration.
- This administration has been acting like a wrecking ball; moving fast and breaking things.
- This administration has been contracting with incompetent private sector companies in blatant acts of corruption and bribery.
- Decision-makers are all drunk on AI these days.
I don't necessarily trust the specific explanation an end user would give for this outage, since they're probably wrong about the specifics. However, when it comes to Duffy, I certainly trust him less:
> Duffy offered a new detail on Saturday, saying that a system overload occurred when contractor L3Harris Technologies Inc. was replacing a section of the air traffic control system’s old copper wire with new fiber wire.
How does upgrading copper to fiber wind up overloading a system when you're adding more bandwidth? And why was this happening in the middle of the day, on a circuit that clearly critical ATC traffic went over? Apparently, the old copper circuits were so overloaded that the bandwidth limitation was critical to imposing request limits, via dropped packets, to prevent overloading the systems on the other end? And once there was enough bandwidth, the servers got effectively DoS'd? That doesn't make any sense.
I'm definitely going to believe the air traffic controllers when they tell me he has to be hiding some other kind of shitshow behind that supposed explanation.
"just push it out there and I'll take the risk": https://www.youtube.com/watch?v=U2qgTrqBS_I
(and SkyNet no less)
Twitter-free link: https://xxcancel.com/alexpotato/status/1215876962809339904
Duffy has been very loud about doing upgrades faster so this kind of result is predictable.
yep. most IR peeps have plenty of stories where a single event is just a symptom of multiple failures, unfortunate accidents, and alike. most red teams will also regale chained attacks is where it is today. it is rarely a single piece of lock that breaks, it needs several other things to be in the "wrong" state.
Your third bullet is a bane of gov contractors. Not the sales people, but the people actually doing the work. Imagine if you had an FFP contract and you finished the job within 90% of the paid time. At $1mm that is sweet $100K that the big cheese keeps. (not you, you go back to the mines and since you demoed you can do it at 90%, next time you have to do it 80% time. and, no i am not sore ;) )
tl;dr My first reaction to Duffy's statement was "you have no clear idea yet, but this is your best guess". And, we may never know.
Asking air traffic controllers for their opinion on this is a bit odd — would they even know about networking equipment swaps? I would expect that to be handled by the engineers and technicians who work on the system, not the people who direct the planes.
They wouldn't necessarily know the specifics of upgrades, but they would likely know when upgrades typically happen because it's normal to send out emails about maintenance windows.
they might know about how the last 10 equipment swaps went and why this one is different.
I imagine equipment is regularly updated and upgraded without them knowing.
I'm extremely skeptical of this administration's competence, but this story is 100% finding anyone saying what you want to hear, however unreliable the narration might be.
> I imagine equipment is regularly updated and upgraded without them knowing.
Part of the reason for this outage was that these systems are not regularly upgraded. That's why they're dealing with decades old network hardware and lines and marrying them up to modern fiber. It's expensive to do the upgrades, but instead of doing regular rolling upgrades to their network components they generally do big-bang upgrades like the one happening now (that apparently caused the outage). In 20-30 years there will be another big-bang upgrade replacing the Juniper routers and fiber lines they've been installing now, and it'll probably cause similar problems.
This also happens a lot in DOD (not surprising, same contractors doing the work and probably the same companies in the program offices for FAA as DOD giving the same bad advice), it's incredibly frustrating because it's very obvious that there are better ways to do things, but the funding isn't there until it's urgent. That's why they're still using 1980s computers with floppy disks in their critical path for ATC as well. Now that those systems are nearly impossible to maintain, there is enough noise to push funding a proper replacement effort, about 20-30 years after it should have been done.
> I imagine equipment is regularly updated and upgraded without them knowing
I highly doubt this.
FedGov loves some ITIL, even for the contractors. A change of this magnitude would have gone through CAB with stakeholder sign-off.
> I imagine equipment is regularly updated and upgraded without them knowing.
It's a good thing you imagined and did not assume. Equipment regularly updated? You mean like the floppies they are still using?
Unless I’m just tired, the article, reporter and air traffic controllers are confused about hardware switches and routers having issues, and the act of switching out hardware. Duffy stated that the switches and routers were having issues and needed to be shut down. This seems to have been read as physical hardware switches, read ‘replacements’, were happening during the day.
It's not just you. It reads like a confused mess to mee too.
TFA is a self-proclaimed progressive opinion site.
Its purpose is ragebait not accurate news reporting.
They may as well have linked to graffiti scrawled above a gloryhole.
The quote in the title is not from controllers as implied, but from a BlueSky post of a well-known social media provocateur.
I don't know what happened, but for me a red flag was the specific explanation given while the event was still ongoing: "overloaded router", followed by request for more money. If the cause was so simple , then it was unclear why it wasn't fixed immediately, so something does not quite add up (at least till an independent investigation)
I don't believe anything that Sean Duffy or his ilk say. I would hope there is a detailed and accurate post-mortem published to explain exactly what happened. However, IMHO this is just another example of the destruction wrought by Trump/Musk/DOGE. I recommend reading this article by James Fallows: https://web.archive.org/web/20261114090809/https://www.nytim...
Why would you hope that? Who would be releasing said post-mortem? Would it not be from the "or his ilk" portion of the comment? Sure, some government employee or team might produce an honest and factual report, but there's no way this administration would release it. The only way for that to happen would be for it to be leaked
> The only way for that to happen would be for it to be leaked
Which is very plausible considering the number of high level leaks and whistleblowing that's happened just in this term alone
Duffy is incompetent. This administration is incompetent.
I’ve more scared to fly since 2025 when they started cutting deep, than the months and the year after 9/11.
After 9/11, people preaching Death to America were at least half a world away. In 2025, people preaching Death to America took up residence in the White House.
Very good point on the timing. You do stuff like that at minimum load, not at maximum.
"Refute" makes it sound like there's juicy insider info, but it's just skepticism that FAA scheduled upgrades on a Friday daytime.
This was a cascading multicast failure, which can take hours to build up. Earliest reports of disruptions came no later than 10:30 am, so it's quite possible the upgrades were performed in the overnight hours on Thursday.
In my observations, any time a politician issues some kind of technical excuse for why things screwed up, they are lying. Not because they know the difference, but because they can't understand the actual problem and managed to elicit some ELI5 explanation from people who actually do know.
"Explain it like I'm a politician!" (cf The Expanse.)
Duffy is asserting everyone in this administration is permanently unaccountable and irresponsible.
https://bsky.app/profile/atrupar.com/post/3n2ebpzqe3s27
Tin foil hat on: Microsoft Cloud issues ? aka Office 365 over the same period :)
That's just a regular hat. No foil needed.
Even if MS stuff was not in any critical path it still could have played havoc with coordination of an update or a change, contributing to the failure.
Also FYI foil hats are actually resonant cavities and amplify the mind control waves. All true paranoids use Faraday mesh, preferably as a full head covering.
I’m not sure which version of Microsoft the FAA uses, but GCC High and the DoD version of Microsoft 365 have been working just fine through the commercial outage of Microsoft 365
I saw a post…I think on HN…the other day. It was a technical post about making a faraday cage out of a metal trash can. Afterwards I thought to myself let me check Amazon to see how saturated the faraday phone bag market is. Lo and behold I find out there’s faraday bags for all sorts of electronics now and head meshes. I’d be curious how well any of them work, but I wasn’t expecting to see it.
I guess it makes a little sense given a lot of electronics don’t have a removable battery or physical power switch nowadays.
Don't forget the foramen magnum!