The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
“The superhuman folks at the data center are hauling 55 gallon drums of diesel fuel up 17 flights of stairs. Assuming they're able to sustain a rate of transfer that exceeds the rate of consumption, the data center should run until electricity is restored. The latest estimate from Con Edison for lower Manhattan is 4 days. (The New York Stock Exchange will be open tomorrow, also running on generator power.)”
Yes. And as someone who worked at Fog Creek at the time, that was a completely avoidable situation that should never have happened, and I'm absolutely gobsmacked we nearly had a repeat of it over a decade later.
Context: in 2012, I literally quit Fog Creek, largely over the fact that we had no backup strategy and I was told that a backup strategy was not feasible. I was "convinced" (paid a lot) to stay on for another two years to see if Kiln would succeed. On my way out of that meeting, I commented to a coworker, "I mean, what's the chance a hurricane hits Manhattan in the next 24 months?"
100%. The chance was 100%. And the only reason people had to haul open containers of diesel up 17 flights of stairs was because no one wanted to admit that the chance wasn't 0%.
Anyway, not sure what was going on with Jane Street here, but holding up that Fog Creek/Trello/SquareSpace story as a positive always drives me somewhat nuts. It lionizes a failure.
> holding up that Fog Creek/Trello/SquareSpace story as a positive always drives me somewhat nuts. It lionizes a failure.
That's a common bias in management. Heroic fire fighting gets rewarded while silently preventing a crisis with proper planning isn't. Nobody ever gets credit for fixing problems that never happened.
We had a discussion about this at my last job. The War rooms were a three ring circus that my team stayed out of 4 times out of 5, substantially by actually planning ahead. Though as time went on we started doing a lot more to protect the services behind us instead of just ourselves, and then the total war room count dropped like a rock.
Someone hit on the notion that it’s Stockholm Syndrome. You bond with people you go through shit with. You don’t bond with people who prevent the shit. That was a super awkward conversation. The asshole in charge quit shortly after it, but 2 years later we still occasionally called each other out for making plans that accounted for a toxic chain of command that was no longer toxic.
I had a tech coworker who thought Y2K wasn't really a thing. I started working in tech in the middle of '99 and was dealing with Y2K stuff right out the gate. He started in the industry in 2001, so missed it.
Exactly because vaccines were so successful that people don't remember what it was like without them. Once enough people start dying in large scale measles outbreaks people will start getting vaccinated again.
That feels like Amazon's entire management philosophy. Always drop the most important thing to work on at a moment's notice because we need to put out the fires caused by not prioritizing the most important things to work on.
At one point we had 2 2pts in our group and the other manager’s performance for that year was based on operational excellence (is that no longer a thing)? We hadn’t grind ticket root causes for months, but on call went from being woken up multiple times a night to possibly never getting paged the whole week.
Unfortunately, that typically meant when you were paged, the problem was probably significant, hard, or it was another team and you were just being triaged in because you were affected or might be able to help root cause.
> in 2012, I literally quit Fog Creek, largely over the fact that we had no backup strategy and I was told that a backup strategy was not feasible
This anecdote hits differently for me. I grew up reading about Fog Creek and consuming all of their writings (and that one film) as an example of what a software development company should look like. It feels strange to have the curtain peeled back to reveal that something like backups, which would be considered basic requirements today, were not part of their early operations.
Fog Creek of 2012 was a pretty different company from the Fog Creek in the early 2000s. While Joel was still technically in charge he had mostly delegated responsibities and spent almost all of his time working on Stack Overflow.
But was "early Joel" FogBugz any good? I heard that it wasn't from somebody who tried it. That unsuccessful trial was also the only time I heard of anyone using it.
Waaay back in 2007ish I got Fogbugz for the small 3-man IT team that we were. As a small team hacking away on an internal ERP and minimal other apps, it was actually refreshing. In my prior role I'd used BugZilla and it was ok, but Fogbugz seemed pretty sweet for a small team. I've now been plodding away with Jira for many years and it's just feature-bloated slowness I'd go back to FogBugz in a jiffy.
Probability is a way of expressing uncertainty. This is the core insight of Bayesian methods. That uncertainty is not an objective universal fact, though. It is specific to your state of knowledge.
If I flip a fair coin and don't look at it as it sits in my hand, the probability of heads is 50%. Your estimate should be the same. Before or after the flip doesn't matter because our state of knowledge is the same.
If I look (and you don't), the probability that it is heads from my state of knowledge is 100% or 0%. Your estimate would be unchanged because you didn't get any more information.
If somebody looks at the coin and tells you what they say, you estimate should be somewhat less than 100% or more than 0% depending on how much you trust them.
All of this is the long way around the barn to say that after the hurricane, the probability that Sandy hit Manhattan is definitely 100%. That's past tense probability, however. It is still also true that the best estimate a year before it hit would be much, much lower. Using words like "probably that a hurricane would hit Manhattan" makes interpretation much harder, of course, because of the uncertain time frame.
I have never seen this definition used before. I was always taught that probability applies to future events. If I flip a coin the event has happened, me knowing or not knowing the outcome does not change it.
Now you can say after the coin is flipped and you still haven’t looked at the result that the probability of you guessing the correct outcome is 50%.
It seems to me you're conflating the probability of x happening in the future as assessed from some previous point in time versus the probability of x happening in the past as assessed from the present. Without clarification the wording is ambiguous but I think generally interpreted to mean the former. It would be quite odd to go with the latter as it's rather tautological.
I don’t mean to glorify the poor planning or risk management, but still have a ton of respect for the people who organized the response and who did that hard work, just because it was what people needed.
My company during Sandy was operated in both NY and SF, and our experience was a bunch of us on the west coast filling in for those who were not only unable to get to the office, but without power themselves in NYC. It was all unfamiliar stuff for us (different business functions were done there, but the NYC office’s work was time critical) but did build camaraderie.
"Everyone coming together to solve a problem by any means necessary" still loses something if the problem is sufficiently imaginary.
Like, carrying fuel up 17 flights to keep a children's hospital powered on, or a MUD that's been running continuously since '93? Sure, inspiring as hell.
"Our company's product will be offline for a few days and we'll theoretically lose money/marketshare, but all our customers will be super understanding because they were all watching the news and, holy shit, did you see that tunnel get flooded, and then everything will turn back on and be fine with no extraordinary effort on our part...?"
I think that's fair. I've been in situations where there are kind of "manufactured emergencies" which boil down to a sales conversation on a deadline which ends up getting rescheduled anyway. I have very little tolerance for that.
In the case of Fog Creek during Sandy, because the software we hosted provided planning and project management for a lot of people, going offline would have a pretty large ripple effect. Sure maybe 95% would've been fine, but I knew of customers who e.g. managed emergency services and crisis response, and so keeping them online was meaningful to me.
"Our company's product will be offline for a few days and we'll theoretically lose money"
This was a startup. The company getting fucked could mean our jobs going away. It was our equity too.
Not everybody is just punching a clock, while resenting the owners of the company. In our case, we wanted our company to do well because we liked working together.
> carrying fuel up 17 flights to keep a children's hospital powered on, or a MUD that's been running continuously since '93? Sure, inspiring as hell.
If MUD is the type of game you're falling into the same trap as the one you're signaling but worse. You're putting in the same sentence "saving kids' lives" and "the bragging rights for a game's uninterrupted session" on equal footing.
Before Jane Street I did more humble work helping gay men hook up as fast as possible via a proto-Grindr site. We were co-located downtown when 9/11 happened. They had diesel generators but they ran out of fuel after several days because the national guard wouldn't let the diesel supply trucks through.
When we went offline customers were super understanding!
Except one customer who was outraged and we felt totally comfortable banning their account.
It was sometime in the mid-2000s when I went from an admirer Joel's writings and what Fog Creek was doing to suddenly realizing it was all just very slick marketing on top of very mediocre stuff.
It really was a shocking wake up period for me in the realities of the real world vs. how we often perceive things.
IIRC it was the whole "Wasabi" thing that kind of opened up my eyes that there was a lot more smoke and mirrors involved than I had realized before.
Your comment brings back the times I've been on teams that worked double the hours a full time job should be, or spent 18 hours in a data center because a migration/data transfer was failing, and, and and.
So many times it's taken herculean effort to make things happen, usually because of poor planning and poor leadership.
Why these are viewed as "successes" or "heroic" baffles me as well.
Because it's cheaper to tell the team or company you're a hero and toss you a pizza and a day off than to hire enough people and competent leadership.
It's not a mystery, it's exploitation.
With all the business efficiencies we've made with computers as devs and IT, we should all be working 3 days weeks, instead a few people are so rich they can buy elections.
> With all the business efficiencies we've made with computers as devs and IT, we should all be working 3 days weeks, instead a few people are so rich they can buy elections.
It's never going to work like that, and you don't need any dastardly, sinister, or conspiratorial explanations for why. It's a natural side effect of the market being competitive.
When tools and technology make people more efficient, some people take that to mean they can work fewer hours and get the same job done. Which is true. But other people take that to mean they can work the same amount of hours, and get way more done. This is also true.
In a market in which two companies are competitors, and one takes the former approach and one takes the latter, which company do you think will still be around in ten or twenty years? Obviously the latter. And that pressures people in the former group to demand more output and stable working hours so they can remain competitive.
> In a market in which two companies are competitors, and one takes the former approach and one takes the latter, which company do you think will still be around in ten or twenty years?
That is an argument of faith (that companies where employees work longer survive more than companies that optimise differently), and I am not convinced it is supported by evidence.
One could argue that all other things being equal, companies with employees doing more hours are more productive (even though I seriously doubt that, there are diminishing returns very quickly). But then I would argue that if a company drives their employees harder, there are leadership and management issues, so other things are not equal. And this is associated with younger workers (adaptable: good, inexperienced: less good), high turn-over (so, long-term information retention issues) and lower worker satisfaction (so the whole thing is self-defeating in the long term).
Besides, there are many, many factors involved that have nothing to with with working time. Again, leadership, strategy, operations, marketing, personal connections, dumb luck…
Personally I think there is an optimum somewhere, and at some point you get more out of your workers by making them more invested and motivated than making them work longer.
The 40 hour work week was the result of strikes and legal maneuvering by unions that led to labor protection laws in the early 1900s. Otherwise we would likely still be working those 80 hour weeks (alongside our kids).
It’s not necessary that the longer hours actually translate into productivity, just that the boss feels like it should.
And the fact that the strikes had a chance was largely due to improving conditions including, among other things, a working legal system that made it harder to flat out kill strikers.
The answer is obviously that an overseer (i.e. the state) must enforce a floor to avoid a race to the bottom, because that argument implies then why not work 14 hours, 6 days a week. Like safety laws enforcing a common standard so companies are not out-competed by ditching safety measures, labour laws are also part of keeping the playing field fair in an competition economy.
Yet after 100 years of explosive technological growth we still have the labouring hours of the 1930s...
You can also curb greed with extremely high taxes at the top. We did it after the Great Depression, top marginal tax rate was something like 90% over 10 mm per annum.
The result was that business owners invested in their workers and long term health of their business so it would continue to give them 10 mm per year instead of trying to cash everything out every 1/4.
Been there, done that. 100% agree with your take.
The actual workers should be celebrated for the extra effort but the need of the extra effort should be seen as a failure. It always stops at the first part.
>Why these are viewed as "successes" or "heroic" baffles me as well.
Your source of confusion seems to be not recognizing that sometimes in life multiple things can be true at once. "Heroism" is generally about the actions of the individuals in response to a dangerous scenario, not how the scenario came about in the first place. "Success" is more overloaded, but can refer to both the overall system in place and the outcome of a given dangerous scenario, again separate from what caused that scenario to come about. This is near perfectly true in particular if, as is usually the case, the ones dealing with merde-hitting-spinning-blades on the ground are not the leadership. It may be idiotic, short sighted, lazy, inept leadership that lead to the problem (or it may simply be an unforeseen alignment of holes in the cheese), but if people on the ground go above and beyond to prevent the worst that's still heroic of them, personally. And if their efforts work that's a success. It'd be pretty strange to insist that the efforts of firefighters shouldn't be recognized just because somebody else did something stupid that resulted in a fire.
Yes, it's absolutely just as important what comes after. Ideally it's like the commercial aviation industry, where heroism on the parts of flight crews can be recognized and said heroism results in a top-to-bottom full systemic investigation and efforts to ensure that ideally nobody will ever be heroes in quite that way ever again. And have constant efforts to make sure heroism isn't ever required in the first place.
The opposite extreme of some organization using it as an excuse for inaction and cheapness (at least until the day arrives when heroic efforts are not enough and a true disaster really does occur) is abominable. Even worse, because in that sort of utterly failed leadership environment the people on the ground will probably get scapegoated too. And it's 100% worth acknowledging that there can be a "success trap" and Normalization Of Deviance situation here, where heroics keep averting the worst which lets bad leadership/management escape correction. But it's a trap precisely because each individual failure can still be bad, up to and including literally people dead, so the pressure to succeed is still there each time.
But if employees, be they civilian or military for that matter, perform far beyond the call of duty why would that ever not be heroic?
Volcanic eruptions have an unpleasant tendency to be episodic, and onset can be far faster than hurricanes or wildfires. Best to minimize how much of your infra has even a (supposed) 0.001% annual chance of being affected.
Anyway, I was the first Fog Creek employee on site at 55 broadway (one of our founders got their first, and two peer1 employees were also on site) and also part of the first wave of the "bucket brigade". Another dozen of us showed up over the coming hours as we assessed and fixed things.
The media coverage on this was never super accurate-- we were carrying (and later, relaying) 5 gallon covered plastic buckets (like the home depot buckets with sealed lids) up to the landing where the generator was to top up its reserve tank. At one point SquareSpace (who was also running exclusively out of this DC) hired a group of muscle to bring a 55 gallon drum up, but that wasn't us.
I don't view this as lionizing failure, though I can see your perspective. Across our products (then FogBugz, Kiln, Trello) we had different backup strategies in play, but we were definitely putting a lot of eggs in the local-datacenter-that-claimed-to-have-resilient-power basket. Kiln in particular had decided to use SANs for data resilience, and that proved particularly troublesome for trying to find a workaround once the storm hit-- FogBugz on the other hand was able to spin up an EC2 version and start a slow-but-would-have-worked-eventually migration (and we even had an employee hit the road for us-east-1 location with a bunch of encrypted backups so that we could get a fast data restore online), and Trello fast-tracked-and-completed its local-to-cloud migration. While it was true that the DC had a backup generator, the fuel tanks for the generator being in the basement and becoming unavailable was not part of their resiliency plan. I don't fault us for not catching that.
...although I did have to later write the dreaded "I'm sorry, we've been hacked" blog post as a direct consequence of us not properly deprovisioning a "temporary" AWS credential that we created during that sprint. I will fault us for not catching that.
Shortly after Sandy, I switched my role from core dev on FogBugz over to DevOps, and I built out the backup and restore resiliency we had been missing. It wouldn't have kept us online during Sandy, though.
It was dangerous, and a couple folks slipped and fell. That's not a good thing and I won't celebrate that. Many employees opted out of the bucket brigade for family, safety, or personal reasons. I threw out all the clothes I wore that week as they reeked of diesel... not great.
I wouldn't wish that experience on anyone, but having come through it with fortune and no slip-and-falls, I still think about:
- Walking around FiDi post storm and seeing the flooded tunnels, clogged with cars that had been washed into them from the street parking.
- Dropping off big armfuls of coffee and bagels in the lobby for the building work crews. The folks who were essentially dredging out the basement and getting the building back online did much harder work than us.
- Going shopping for aquarium supplies in order to best jerry-rig the pump that our founder has repurposed from the office fish tank in order to transfer diesel from the drums on the curb into our 5g buckets.
Something that kept me going was that I knew that one of our FogBugz customers was the emergency response service in Haiti, and they got absolutely walloped by the storm. They were using FogBugz to coordinate relief, so there wasn't any way I was going to let FogBugz go offline if I could help it. Turns out that meant climbing a lot of stairs.
> Across our products (then FogBugz, Kiln, Trello) we had different backup strategies in play, but we were definitely putting a lot of eggs in the local-datacenter-that-claimed-to-have-resilient-power basket. Kiln in particular had decided to use SANs for data resilience, and that proved particularly troublesome for trying to find a workaround once the storm hit-- FogBugz on the other hand was able to spin up an EC2 version and start a slow-but-would-have-worked-eventually migration
The issue that I remember was that we had so many SQL Server databases that it was impossible to keep anything vaguely approaching a sane backup strategy; SQL Server just isn't designed to handle as many as we had. (I in fact remember talking to a famous SQL Server consultant [edit: this was likely Brent Ozar] we were using who said "that won't work," and then I or someone else was tasked with talking to a SQL Server dev at Microsoft about it and they ALSO said that it couldn't work, and when I brought this up, I was told that Microsoft was wrong. IIRC, the implication was that it would take an extremely long time (many days) to restore, and that the snapshots would likely be days old. I don't think any of that contradicts your memory about the very a slow restore going on, but might add some color.
(Likewise, Kiln supported multiple backend instances and had admin commands to copy and move repos between them, so it the SAN had specifically been the issue, we had workarounds that would've been fine with whatever EBS was up to those days. But I remember the databases being the blocker.)
It was indeed Brent Ozar! Yes, that's all right, though a huge amount of the bottleneck was the off-the-shelf backup tooling we were using. In particular we couldn't move a database between servers without minutes-to-hours of downtime per one database, _and_ as our database count grew our time-to-take-a-backup grew _faster_ eventually meaning that by the time we did a full backup run, we'd be due to run another full backup run.
Our "One database per tenant" model was also the reason we were so slow to go to AWS-- the MSSQLServer licensing terms were cost-prohibitive.
However recognizing that around 90% of the FogBugz databases were free-tier and didn't get daily changes meant that teaching the backup scripts to skip creating redundant backups solved most of the issue.
Post-Sandy, I was able to solve most of this by:
1) Shipping full, diff, and incremental backups at different schedules
2) Scripting the ability to create warm standby servers powered by the latest fulls, then diffs as they come online, then the logs at the cutover time. (so you can provision a new server and seamlessly cut over to it)
3) Using our own scripts instead of the RedGate tool (which was _fine_ just not equipped to handle as many DBs as we had) to orchestrate the backups and restores. This in particular was needed for the "only back up modified dbs" fix.
This got us to most of what we wanted-- 0-downtown transitions of databases across servers, nightly offsite backups, fast backup runs. The final problem-- if our datacenter evaporated and we had to reprovision everything from scratch -- was left as an acceptable risk of "we'll fetch the nightlies from s3/Glacier while we do the slower part of rebuilding our servers."
But to your point, during Sandy, none of the above had been fixed yet. I think our sysadmin team was very strong, but they weren't programmers (and/or didn't have enough time to step back and program), and I was able to pick a lot of low hanging fruit in DevOps
I mean this with love, but dude, you put your health (short term and long term) at risk over a web app? I think you need to reassess your life priorities. You work so that you can live and enjoy life. You don’t live just to be a worker.
There is no reason that they couldn’t have had more than 1 DC when Sandy hit.
Life priorities are not static, when faced with once in a lifetime scenarios they will change very quickly. Rational thought during emergencies often goes out the door.
I hear you, and agree with your spirit. I never felt in danger and was choosing my risks throughout the day. I'd also see folks working much more dangerous jobs than me all throughout FiDi trying to handle cleanup.
I guess it helps that I grew up in a rural place and did construction for a while, so lugging gas to a generator wasn't a new thing for me.
Also I was... 25? I'd be more cautious today, I think.
> holding up that Fog Creek/Trello/SquareSpace story as a positive always drives me somewhat nuts. It lionizes a failure.
Firefighters are heroes who bravely enter burning buildings, beloved by children and adults alike, muscular figures who get pictured on sexy calendars and thanked by tearful families.
Officious paper-pushers who make a fuss about a few cardboard boxes stored under the fire escape stairs (like me!) seldom get pictured on sexy calendars.
A firefighter entering a burning building is literally risking his or her life at that moment. A paper pusher who works on preventing the fire in the first place is normally very far from danger.
To play devil's advocate, paying some dudes to haul diesel for a few days every decade might be a lot cheaper than maintaining a fully redundant failover site 24/7.
A just a single day of labor would pay for a fuel pump and line so nobody has to carry fuel drums up a staircase.
Im all for doing some dirty work when needed, but if they expect 100% uptime they should be ready for something as common as a storm flood without risking injuries hauling fuel drums by hand.
Yeah but if there is an emergency you can buy a stand alone 60 PSI fuel pump for like $120 plus a roll of fuel line. That would get you half the height by itself if you put an empty barrel in between. And by either moving the pump or buying a second pump you could transfer it up to the top again in either a fuel barrel or straight into the generator tank. Probably 12V pumps because its an emergency and you need what is on the shelf, but that isn't any real obstacle for a temporary setup.
60PSI will raise diesel about 50m. That's less than 17th floor, but it's worse than that because at the top of that 50m pipe, you get zero flow.
If you drop back to 30m of lift, you get still have 36PSI which is enough to get some flow. But that means that you need a holding tank (or lots of buckets) and another pump for another 30m of lift. That gets you to the 16th floor so you need another holding tank and a another pump.
That's a lot of complication. And getting those pumps to sit still can be a challenge.
Hahaha, we tried! Aquarium pumps were all that we were able to source. "Serious" resources were being deployed (appropriately) to hospitals, etc., so we had to make due with what we already had on hand. Thankfully we had an office aquarium, so...
They don't get paid extra. They're just expected to give up their free time and risk injury doing something they weren't hired for so bosses get a bigger bonus. (you also don't need a fully redundant site to maintain business continuity)
It's such a familiar engineering story. Heroic effort means someone, somewhere, screwed up.
I aim for boring competence so nobody has to plugs gaps in systems and processes with their literal body sometimes.
Without disagreeing, I was, at the time, the lead for Kiln. I was acutely aware that any major issue with our one and only data center would result in Kiln going offline and us suffering irreparable harm, and my name would be the one tied to that, despite me fighting really hard to solve the underlying problem. Yeah, it was a different time in some ways, but I'd make that same call in the same situation again in 2026, too.
You made the right call. I also worked at 55 wall.
My story [not as interesting] is that I was a emergency responder living in BPC and was relocated to the millennium hilton because it had a generator... BPC never lost power :/
I'm surprised they didn't put a window washing crane on the roof as DR to simplify the raising of the diesel. The strange ass shit I've seen implemented to make the failure mode easier instead of eliminating the failure has limited my faith in our system as a whole.
None of us felt comfortable operating the crane or MacGyvering it to lift a barrel of diesel into the sky, and the building staff that could've done so was focused on bailing out the basement. It was something we talked about though :-)
That's fair, Rigging a barrel is a trick. It's actually pretty neat to watch an experienced rigger belt strap some oddly shaped item.
You can do a barrel knot, though with a modern metal barrel, you can also generally just do a hook on top an bottom and lift horizontally.
My comment was more about the goofy shit people do to make the failure mode simpler next time instead of eliminating the failure mode all together. Some of the places I've worked would probably have made one of us get trained on the lift or something instead of make sure we have the fittings on had or a fuel pump and hoses with 17 floors of head pressure. That would be the temporary fix and then when they got the quote for the pump it would become the fix.
Seems to me it is more likely that someone did the calculations and figured it is cheaper to send a bunch of unlucky morons up the stairs with jerrycans in case of flooding.
Unlucky moron here... eh, it's like the 5th order effect of the planning. Somebody definitely calculated it wasn't worth the money to maintain a hot offsite data center, and for the most part that felt ok because we had a profit sharing component to our salaries. But I don't think anyone could have drawn a line from that to the fact that Peer1 wasn't resilient to flooding.
Startup bullshit. People at "scrappy companies" make cutting corners look like heroism and practical-mindedness. Meanwhile the little people suffer and prop everything up as the business almost implodes.
I can understand your frustration, but let’s consider:
The chance of a hurricane within 24 months was never 100%. Looking back, it did happen, but you can’t judge future-facing decisions on post-hoc information that you didn’t have at the time.
There is a difference between “no one wanted to admit it wasn’t 0%” and “no one else thought the risk-adjusted return was positive.” When people decline to take some preventative measure, that’s usually what they’re saying implicitly.
I’ve been where you are—frustrated that you predicted and told people exactly what would happen, they didn’t listen, and you were right—many times in my career. It’s infuriating. Part of learning from it is figuring out how to get people to change, not merely figuring out how to be right.
My guess is that this kind of black/white Strong Opinions approach has bitten you throughout your career.
I was working at a data center in the financial district during Sandy. That was quite the experience. For whatever reason, our building was one of the last to have power turned back on. We never had to move diesel by hand fortunately. We did have a fun moment early on where one of our generators needed to have most of its load reduced for a few minutes. There were a pair of external fuel filters that needed to be changed fairly frequently. They were meant to be redundant so you could change one at a time. By the time we were able to get a tech on site, they were so dirty that when we started switching to just one the generator came very close to stalling out. After some conversation we thought it was thought that if we could drop the load enough we should be able to get away with just one dirty filter for a short period of time. The only chance we had at that was to shut down all of our cooling units. The generator was on the roof, and I was coordinating the timing over radio. You could dedinitely hear the generator wasn’t happy, but it worked. As soon as the second filter was replaced I let everyone know it was time to turn all of the cooling units back on. No one responded, and after a pause I realized that meant that probably no one had heard me and things were getting hot fast. I sprinted back inside. On the way my radio dropped, so once inside I sprinted to all of the units to tell people to turn them back on. For some reason I didn’t think about grabbing someone else’s radio. That may have been a lucky mistake though, as I’m not sure what all of those compressors turning at at the same time would have done.
I had made sure to observe the filter replacement process in case we needed to do it again ourselves. We did, and the next time someone recorded me doing it so everyone could learn and we handled most I’d the rest of the swaps from then on.
There were a lot of memorable experiences that week or so.
We had the fuel filter issue at 75 Board street as well. After a few days we had to do a planned generator shutdown (the data center batteries could last a couple hours to give us coverage...) as a diesel mechanic flown in by Peer1 could service it. Props for handling that yourself!
Definitely a memorable week. I wonder just how many folks in Fidi were learning the finer points of emergency backup generators.
I would have thought that backup generators would cumulatively run for maybe a few dozen days over their lifetime. How did the fuel filters get dirty so fast?
The _fuel_ itself goes bad. Diesel will slowly accumulate dissolved water, and once it gets past a certain threshold, a lot of wonderful bacteria start to flourish in it.
So you get clumps of gunk in your fuel that rapidly clogs the filters.
Because a genset that runs once a month and has a fuel tank sized for days of wide open operations is a textbook recipe for asphaltane and other garbage build up. Which of course you won't notice until you actually run the thing for a day straight and the fuel filters start showing it.
Diesel fuel has to be periodically filtered ("polished") by passing it through cans of dessicant. It removes accumulated water and prevents bacterial growth. There are also biocide additives but they just _extend_ the periods between maintenance.
That's also why it's not a great idea to use diesel for your backup home generator. It will go bad sooner than you expect, and then you're left with drums of gunk that you can't dispose and can't use.
And gasoline won't help you either. It goes bad even faster.
Propane tanks are the best option they can sit in storage indefinitely.
Or pull an AT&T in Nashville and depend on natural gas for your backup generators.
Which doesn’t quite work so well when an explosion cuts your electrics and the fire department cuts the gas.
If you do still decide to depend on another energy grid for an energy grid failure, at least have roll-up generator hookups so you’re not spending a few days figuring out how to wire that up…
The IT person for McKinsey & Company said water came within feet of destroying all of the data for the firm and its clients. That led them to stand up a second DC.
Sandy was wild. you couldn't get computers for love or money, everything was jammed up. we ended up with whatever hardware configs had been returned and sitting in a dell warehouse just to have enough machines so for a long time after you'd have a rack of the standard build and then a random machine with a fibre channel card.
shocking how much engineering you can get done when the normal process becomes impossible and waiting isn't an option.
That one tested a lot of assumptions in planning for sure. I recall that we had a sound DR strategy, however it turns out, our main site was in Newyork and the DR site was in New Jersey to save costs and still be compliant. Good learning for sure.
Generators go on the 17th floor so they can vent out exhaust and heat. Pumps need to go in the basement because you can only suck a liquid up by one atmosphere, which is something like 30' for diesel, but you can push a liquid up basically as high as you want. (Also, stating the obvious, the diesel tanker trucks are gonna be street-level, so you'd need pumps there anyway if you put the actual tanks higher.) This does, are you're pointing out, kinda stink if the pump itself is flooded, but the design's common and does make sense
In firefighting it is quite common to take in surface water using a submersible pump which is hydraulically driven by an engine on land - like the fire truck or a dedicated diesel. See for example [0][1]. Doing it this way means you aren't limited by elevation and don't need to carry special suction hoses for water intake.
It makes me wonder if anyone is paranoid enough to do the same for highrise fuel pumps: sure, the basement may flood, but when the pump in there is designed to flood it's a lot less of a problem! Place the actual power source several floors up, where flooding is impossible outside a The Day After Tomorrow-esque civilization-ending catastrophe and you should be fine.
I see. I thought it was an emergency power generator they brought in for this incident. When you see that sort of thing for buildings, usually it's just a big trailer on the sidewalk hooked up to the building mains.
Probably on the 17th floor to put the generators closer to the loads and reduce transmission losses. Also reduces street level air and noise emissions (and avoids street level air intake)
With fuel storage on the same floor for a reasonable runtime, larger fuel storage in the basement and a pump (or two) to move fuel, you're well prepared for lots of things. As long as the fuel throughput on the stairs is better than the fuel use rate, then you're ok for as long as you've got people to move it. Which is ok for exceptional incidents.
More than likely that's where the available space was. Data centers many large generators and there may not always be space in the lower levels to fit them, especially if the data center was not one of the original tenants.
You put them up there so you get a voltage boost from the gravity pushing the electricity down. Put the generators on the ground floor, and by the time the electricity reached the top, there would be a voltage drop.
No joke - have you ever tried to siphon electricity? Can't do it more than 35 feet or so. You'll be boiling off your electrons into a plasma by the time you get to that height.
> The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
Awesome, story time! I was working at a not brand new but still relatively small tech company (~$100m yearly revenue) and we were in two major datacenters, one in Las Vegas, and another in Manhattan. We were essentially telephony in the cloud, back when that was still fairly novel. At the time, most of our revenue came from handling various 1-800 campaigns, like "Ronco Knives" if you remember that, and also big money raising telethons. Our platform was active/active between the two datacenters. Of course not for a given phone call, but we could pretty easily balance traffic between both datacenters, even fully away from one of them, though as we'll see later, at some peril.
Our main networking guy was also the main datacenter person, and he'd been in NY to help set us up there. I can't remember which datacenter it was, but we had like half of a floor well above the street.
I was watching Sandy carefully, and once the odds of it hitting Manhattan were substantial, I started to press for plans to move all traffic out of the NY datacenter no less than 24 hours before landfall.
The network guy disagreed; he said that he'd been there, on site. He saw the redundant generators. The redundant network ingress and egress. The site people joked that if somehow the first floor ever flooded, the the rest of the datacenter would continue operating normally.
I noted that while all that might be true, we don't have any such assurances for all of the other equipment that'll be underwater needed to support the network.
I ended up escalating to our boss, the head of operations. He was one cube over. :) We had a vigorous discussion, and the VP agreed to pull our traffic out of NY. So we did.
Probably 75% of our capacity at the time was in LV and 25% in NY. We were running pretty hot utilization wise, but LV could, under normal circumstances, handle the whole platform by itself.
Sandy came and flooded. Generators kicked on. The network stayed up. Then the DC/networking guy got an emergency call from the datacenter.
They had plenty of fuel on site, but it turns out that the fuel inlets for the generators were...under water and could not be reached. So everything was going to go down in a couple of hours.
And it surely did, for reasons I've forgotten, in a very crappy way. Our logs indicated that power hard flapped for all of our equipment quite a few times.
We lost quite a few PSUs due to that. Yes, we definitely should have shut our equipment down cleanly. Oops.
Man only on HN would someone would have the stones to argue “350lb+ drums up stairs [just] sucks”! My dude it may as well be superhuman, do we really need to split hairs here? I think some hyperbole is in order.
Probably an order of magnitude larger than the number who've moved enough 2-400lb objects on it and have a built up intuition for what techniques work and don't for various situations.
The "superhuman folks" above mentioned are the 75 Broad street bucket brigade crew, and uh... we made startup wages. We weren't quants, we made productivity software.
Though the "55 gallon" drum lift is greatly exaggerated, that was a crew of guys hired off the street who were walking around doing odd jobs in the disaster zone. Kinda wild. For us, we just carried the 5 gallon buckets.
I remembered going to our data center in Long Island City and labeling all drives and pulling them from the servers and putting them in a box and took them home.
One of the best decisions I made around that time was installing two backup Towerstream lines. When everyone else was out for weeks we were up and running, full operation.
> “They had their brigade of people carrying five-gallon buckets up 18 flights of stairs or whatever it was to get to their auxiliary generator. Quite a task by those guys,” Orchard said.
5 gallons of fuel at a time (hopefully not in a literal bucket) is a lot easier than managing a big drum on stairs. Unless you have a powered stair dolly, but powering your stair dolly up 17 flights of stairs seems like a challenge too. Team lifting a drum up one flight of stairs feels managable, doing it up 17 flights multiple times is a hard no.
Carrying 5 gallons of fuel up a dozen or more flights of stairs is hard work on its own.
I've done this job. I only had to do it a few times per day, but it sucked each and every time.
(When I showed up due to a report of an outage for our client's gear at the top of the building, the street was flooded. Water was flowing into a giant metal grating in the sidewalk like something from a cartoon. This grating lead to the basement.
And that basement was where the electrical switchgear was.
5 gallons is a standard size of literal buckets, they can have lids but I might prefer carrying a bucket of diesel up 18 flights of stairs with no lid over trying to get one of those goddamn lids off.
If you’re burning the diesel within 8 hours after it was in a bucket for day 1 hour it’s not an issue. You shouldn’t store solvents in buckets that aren’t rated for solvents but you could carry that shit up the stairs in a paper to go box and it would still burn.
I don't have much experience with diesel but the wrong plastic with gasoline will turn it to jelly in minutes. I wouldn't consider it worth the risk to use random plastic buckets to carry fuel.
lol, ok, well, it sounds like you don’t have much experience here, I wonder why you felt the need to weigh in?
Almost anything that’s not a single use plastic and has a shape to hold will be made out of hdpe or polypropylene, just like a gas can, or a bucket, or those black and yellow boxes that are everywhere, or believe it or not, gas tanks on most newer ice vehicles.
So, yeah, don’t use a trash bag, but that shouldn’t need to be said.
The main reason you shouldn’t use things for normal storage or transportation is because they don’t handle vapors, or pouring without spilling very well. But you can certainly carry gas up the stairs in a bucket, and you can store gas in a bucket as well, but temperature changes can be problematic over time.
I have a stack of clean buckets that I routinely use when I need to remove the fuel from a vehicle to work in the fuel system. Gas or diesel cans are simply to expensive to keep a dozen on hand to use for an afternoon or a week while replacing a fuel pump or the fuel lines.
And in the case of a 17 floor bucket brigade, also likely too expensive and probably hard to find in the quantities they’d need to make that work, if you assume one bucket each way for 17 floors.
You’re probably not going to walk into an Ace and find 34 5 gallon cans especially when anyone with power is running off a generator. But, you will find that many buckets because it’s also a paint store and they store easily and cost nothing.
So they were probably using buckets, even if it’s true that you shouldn’t put gasoline in a garbage bag.
Yep, somebody went to home depot or lowes and got a bunch of lidded buckets.
Some of us did the full round trip a handful of times, but that was mostly early morning of when the power first when out and we just had to keep the generator going. Once we got into the late morning, enough of us were there that we could have a person at each floor and they'd receive a bucket and relay it forward.
It is a federal crime in the US to have fuel in anything but approved containers. They probably won't look/care in this case (they probably already know and choose to look the other way) - but I still wouldn't admit to on the open internet like that.
I'm surprised they located their datacenter in NYC.
The NYSE, NASDAQ, and most other exchanges and brokerages run in datacenters in Cartaret, NJ and Mahwah, NJ. I can't think of a good reason why anyone would put a datacenter in Manhattan.
It was another time. Only big companies had dedicated data centers; the rest had an air-conditioned room in the office. That was before the "cattle, not pets" era, too: they would have a database server called "Debbie", a DNS server called "Denise", a file server called "Felix", etc.
100% this. It was almost weekly that we had to go to the DC to do something hands-on with the equipment, so having that be a 5 minute walk from our office was a huge benefit.
Apparently it was a 30-person 5 gallon bucket brigade of customers, contractors, and volunteers all lining up to help fill tanks because of interconnection issues between the supplier they got and where they needed to fuel the tanks up with.
"... neither we nor the fuel company had the proper coupling to tie into that existing infrastructure."
Sometimes "DR" in your plans stands for "didn't realize".
As a bonus though? They were able to in fact get ahead of the fuel burn, and the crew managed an eight hours' head back up on the tanks.
I had the dubious honor of rolling servers on office chairs and hundreds of drives from an EMC box in a sterilite to get it upstate to a satellite office.
There was about $5M of gear in my wife’s grossly overloaded minivan. The bean counters tried to deny my mileage because I was using a personal vehicle without prior auth. lol. My boss was ready to kill.
>The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
Sandy power disruption always reminds me of this photo - burned into my brain - https://archive.is/DoZsr
In the article he talks about living within a few minutes walk to work and telling too many people so they made him the emergency office guy. Same thing happened to me, foolishly told everyone hey I can see my apartment from here so I would get panicked calls to run in and shut down servers because the information security guy got a tip of an impending exploit about to be released on a holiday weekend with no way to do it remotely. Everyone else lived a ferry ride or commuter train away.
Not the person you asked, but I've learned that if you're not careful about what you tell your employers and/or coworkers that it will almost invariably come back to haunt you in the form of unpaid labor.
It took me an embarrassingly long time to realize that (my 30's), and now I keep my conversations with my coworkers very utilitarian and divulge minimal information about myself. I try not to be a dick or anything, I'm polite enough, but pretty much everything I talk about is either directly relevant to work and/or so surface level as to border on "vapid".
ETA:
Just as an example, I used to have really bad sleep issues in my 20's that would make it so I wouldn't be able to get to sleep until around 4am, if at all. I made the mistake of bringing that up at work to my manager. Two weeks later I got a phone call at 3am because there was something he wanted to look at and "he knew I would be up". I was up, but I didn't really have much desire to get my laptop out and do work. I liked that job at the time and I liked my manager, so I did it anyway, and it wasn't until years later did I think about that incident and how utterly weird it was that I was sort of being punished for my insomnia.
My sleep issues have gotten a lot better now, so if someone calls me at 3am I probably won't hear it because I'll be asleep, but I remember that incident now and have it constantly spinning in my brain when I chat with coworkers.
I'm fine sharing (some) things about myself with coworkers, but I'm also in a position where if someone says, "So you can be our on-call guy for on-prem servers?" I can say "Absolutely not without a really big fucking raise, and I am not kidding, if you call me at 3am to come in, I will hang up on you."
The problem here is not divulging info about yourself, its not being able to say no to work. I will happily draw a line where my responsiblities end and if anybody asks me to go over that I will ask for a role review. Too many people get walked over at work, stand up for yourself and say no!
Still, I do think there's merit in keeping your personal life to yourself and not telling your coworkers more than they need to know. I'm not saying you should be stoic or a dick or anything; by all means engage in conversations and be nice. I'm just saying that it's important to have boundaries.
I tend to be someone who overshares a lot (hence my enormous word count on Hacker News), so I just try and keep this in mind.
Fair response. I guess it also depends on the working environment, the type of person you are, and whether your coworkers are more or less like you as a person etc.
Exactly on point. Don't share anything that's not relevant to the job. In one of the previous jobs, I was coerced into doing a deployment to prod every 2 weeks at Friday 10 pm, since my boss knew I was single and not into partying or meeting with friends. Had I known better, I'd maybe even lie about my life and tell him to get lost.
Becoming a surface level/"vapid" work automaton would be a heavy price for to pay, perhaps not for you personally, but for me and I imagine many others. So, I'm not sure your example would lead me to the conclusion of your behavior. You mention being called at 3am as one incident that didn't mean much to you at the time and didn't become a pattern of exploitation, so what was really lost?
Maybe I'm lucky. I've so far not had to endure a job for which I was underpaid and expected to meet and exceed every contractual obligation placed on me.
I've learned that life is what you make of it. If you let yourself be used - others will use you. But if you showed your worth, but also showed others that you know your worth - things are different.
(otherwise where I work, I also avoid talking about politics for instance, but this is a different theme)
I don't really talk about politics at work either, but that's broadly because I live in a very progressive area where pretty much everyone has the same opinions that I do so it would be a pretty boring echo chamber.
No because I imagined a future where instead of me being asked for emergency help I'd be expected to perform a list of tasks every weekend as the defacto facilities manager. They were already paying me a ridiculous salary at the time. I thought about bringing it up if they ever said no to any requests I had and they never did so simply accepted it was my mistake broadcasting too much information at work which I never repeated.
I received many strange requests like to come in Friday night while drinking at a concert and ransack the offices looking for a laptop or possible phone left behind that they suspected was triggering alerts by repeatedly trying to connect to a foreign server. "You want me to drunkenly rob the offices at night and walk out the lobby carrying a stack of laptops? Ok"
If I would have 5min to work, I would probably be in the office most days. A 1 hour commute is something I want to avoid, not being in the office itself.
Odd that they didn't have at minimum 1) a physical Emergency Power Off (EPO) button inside the datacenter, such that one button could de-energize everything -- usually required as part of building a datacenter due to code variance for the "low voltage" (<4000V in this context...) wiring. 2) EPO interlink to the smoke alarms, such that it would de-energize equipment automatically (maybe with delay) if smoke detected.
You'd generally not want to have the sprinklers go off while things were energized, but there could be a mechanism to monitor flow, and only cut power if sprinklers were actually sending out water. But even if this was present, your first priority would be for the sprinklers to not go off as there would be a lot of clean-up / drying needed after the fact.
Unless you have a mist system (NFPA 750), sprinklers put out a lot of water.
I love this story and that they kept the wabi-sabi :-)
Missing NYC and this reminds me of NYI back in the day (still there perhaps) and operating datacenters in the heart of Manhattan before things changed.
I like that they're keeping the metal close to the traders actually and I'd be totally aligned with that decision. I am surprised however that they haven't shifted to Halon fire extinguishers or something that is safe(r) for a data center.
Interesting. The Sequence of Operations we write for the data halls I work on specifically do not shut down the CRACs on smoke alarm for this reason. The International Mechanical Code (IMC) specifically allows this where a air handler over 2000 cfm serves only one space because it's recognized that it will not spread smoke around a building. 2000 cfm is the threshold where you typically have to install smoke detectors, and typically they do shut down on smoke detection.
The CRAC won't feed a fire oxygen as it's just recirculating the same oxygen depleted air.
While CRACs won't recirculate oxygen, they can move air past sprinklers faster than they can handle which can delay operation, typically 5 fps. Although that's only if you are using sprinklers, clean agent is a different matter.
Yep, it is part of the design. The sprinklers are sometimes pre-action like described in the article, and sometimes clean agents. The only time we shutdown CRACs is if we EPO the whole room because otherwise some variant of this article happens.
He had to deal with so much absolute bullshit to get the diesel generator I mentioned in the story done, and then Hurricane Sandy destroyed it instantly a few years later. Felt bad for him.
Sprinkler heads on fire suppression systems go off locally - if there's a fire, then only the sprinkler heads in the vicinity will go off. The alternative is to let the fire spread and destroy the entire facility.
But in addition, this particular story takes place in a skyscraper in Manhattan, with the data center in question occupying a portion of one floor, with office space surrounding it. So letting a fire burn would create significant risk to human life as well.
Often only one sprinkler head would discharge, so the damage might be localized within that ~12' radius to some degree. It's not a deluge system like many movies make out sprinkler systems to be where every head discharges simultaneously.
The DC in question was in the middle of a 33+ story office building, so the the worst case is "entire building catches fire and people die", not "equipment destroyed".
I heard of an inert gas fire fighting equipment discharging without a fire in a data center. The pressure wave created by the initial discharge ruined all the hard disks and caused replacement of all the equipment even though there was no fire (or water) as the conditions meant no more warranty
> every minute, ask every server in each location directly for the temperature from its onboard sensors, take the average, and if it’s elevated, blast an alert out to everyone on the systems team to check for datacenter overheat.
I've once done the opposite: the office "server room" (a small room with three racks, two of them for the patch panels, one for the servers and switches) was cooled by a mini-split which was prone to locking up with the compressor enabled (the relay which controls the compressor got stuck) and freezing, which then stopped the air from circulating due to all the ice. I wrote a script which ran on one of the servers which read the ambient temperature through IPMI to detect when the room was too cold, email a warning, and power off (using ssh) the neighboring servers and then itself.
Oh, this reminds me a story from the time when I worked at a university. Poland, middle of harsh winter. The day before, there was a fire system inspection. On the day of event I was paged - our datacenter got flooded with water sprinklers (WTF why there are water sprinklers in DC!?), then the system opened all the doors and windows. Shit got frozen. The whole building was like an ice-skating rink. In the end only machines on top of the racks got damaged. Turned out that inspection was done by some relative of one of the professors and he fucked up something in fire system. It was a fun event haha
Great story, I love those in-the-trenches emergency stories. I wonder if it would have impacted Jane Street in any meaningful way, aside from a fat bill, if it ended with submarine servers. They could have diverged their compute to another datacenter surely.
We mostly hear about engineers from Jane Street here, and not much from their traders and finance people. I wish we could have some other perspective sometimes.
They publish a great podcast, Signal and Threads, I really recommend.
I wonder why the answer was to turn off the servers instead of turn their cooling back on. Would it not be able to stabilize the temperature fast enough?
The real answer is probably lost in the sands of time but even at 200 people we still had different teams with overlapping responsibilities. It's possible I was the only person reachable and close enough after hours, and while I didn't know where the CRAC unit controls were I did know how to rip power cords out.
That's how the server room at my school worked. They said to get out of there if the alarm went off because the room would fill with some inert gas and you'd suffocate.
The gas is used first, then water as a fallback. I'm not sure if sprinklers are required by code in all data centers, but they would be in an office building setup like Jane Street's.
Even though this happened in 2012, it's interesting that Jane Street is doubling down on compute and renting even more capacity from neoclouds. They are planning to expand 10x the amount of GPUs and are locking in tons of compute. On top of the $500M+ per MW of compute that they are paying Anthropic.
I specifically remember a TV commercial many years ago (15-20 maybe) that was trying to show the sophistication of the company with an anecdote of how when looking at some other company as a potential investment, considered if they used sprinklers in their data centers and determined that they did not. I could have sworn the company advertising was Jane Street.
Great post. I hope a big paycheck was given to the author for saving the DC and the company from disaster.
Many many years ago someone told me that your data is not safe if is not at least in 3 very different physical places. Today, I would probably add the requirement to be on different providers as well.
Yeah I didn't read the article in detail. It's the NYC that threw me off. And I didn't feel like it was that weird an concept, after all the real estate in NYC is super premium priced, not exactly prime datacenter country. I'm surprised there are many (like I said 33 Thomas St used to be just a phone exchange which used to be much bigger than they are now due to the mechanical tech involved).
Nitpick, but the headline is not confusing at all, even though a couple of people are apparently confused. I suspect that is the fault of a hasty read through, not the headline itself.
I think it looks cool. I used to work in buildings like this a lot. They're kinda cool. Dark, humming and clicking etc. I worked in telephony at a time when some (not many) of the old mechanical switches were still around and those buildings were especially spooky. It sounded like you were in the middle of a wild storm of crickets around 7pm when people finished dinner and started calling their friends. And all the little lights flashing all over (sometimes I used to keep the lights off for the cool effect if I could)
Some of them have half-floors for all the cable routing, where you can't really stand up.
I think this is why I like them. To me they're not just an ugly box but I always wonder what's inside.
First, the guy probably wouldn't have keycard access to whatever room contained such a switch. Second, it's highly unlikely that the system would be able to cool the room fast enough to avoid hitting the temperature limit.
Incidentally, we actually had a fire in the office building I worked in long ago. The HVAC didn't shut down, and smoke started to come out of random vents throughout the building. It was chaos. And yet, plenty of workers wanted to finish and send that email they were working on, commit code and push it to the remote server, etc. What sticks with me most is the one guy, a very senior manager, running around yelling at people to get off their computers and get out of the building, because no job is ever worth your life.
They'd be shutdown by a dry contact that may variously be coming from the VESDA (Very Early Smoke Detection Apparatus), or an internal smoke detector, or a duct mounted smoke detector. Not as simple as flipping an obvious switch on a front panel.
This would be the one time I get to hit the EPO button (which this DC seemed shockingly too ghetto to have) for our breakers and it not being a resume-generating event.
I’m surprised by the pulling cords by hand part. Did they really not have an emergency power off wired up to a shunt trip or some other device to efficiently cut the power? Isn’t this mandatory, even? Especially in a datacenter in an office building?
There's tons of substitutes, the ban on Halon wouldn't be why Jane St is using water.
Many people (like me) use "halon" when they are talking about general inert gas based fire suppression as there's not really a good short word for them otherwise.
Realizing that I'm spoiling some of it... but we're here, so some of it is inherently spoiled...
The fire alarm klaxon went off and the siren warning
of imminent halon gas release was screaming. We
started to panic but the data center manager shouted
over the din, "Don't worry, the halon system failed
its acceptance test last week. It's disabled and
nothing will happen."
He was half right, the primary halon system indeed
failed to discharge. But the secondary halon system
observed that the primary had conked and instantly
did its duty, which was to deal with Dire Disasters.
It had twice the capacity and six times the discharge
rate.
They don't have to blow that fast. It can be regulated.
But yes, I've seen low-O2 installations also. These have other issues though, like no protection when humans need to be in there for maintenance. And the need to be very very careful when reducing it again, making sure everyone is out. It happened to my colleague once that this went wrong and while it's not low enough that you'll die, he wasn't exactly feeling great for the next few hours.
We had it at work at a datacenter right beside offices and reception.
You just have to reinforce the ceiling panels and the windows so they don't get blown out, but it's not really an issue otherwise. It was all certified and regularly checked. Water is indeed a bit counterproductive as it's not really much better than fire for the equipment in there.
PS: Halon itself is of course gone but it's other less harmful gases these days.
What also happens is low-oxygen environments (they only raise the O2 when maintenance is being done).
The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
“The superhuman folks at the data center are hauling 55 gallon drums of diesel fuel up 17 flights of stairs. Assuming they're able to sustain a rate of transfer that exceeds the rate of consumption, the data center should run until electricity is restored. The latest estimate from Con Edison for lower Manhattan is 4 days. (The New York Stock Exchange will be open tomorrow, also running on generator power.)”
https://news.ycombinator.com/item?id=4720388 (https://web.archive.org/web/20121101194314/http://status.fog...)
https://arstechnica.com/information-technology/2012/11/how-o...
Yes. And as someone who worked at Fog Creek at the time, that was a completely avoidable situation that should never have happened, and I'm absolutely gobsmacked we nearly had a repeat of it over a decade later.
Context: in 2012, I literally quit Fog Creek, largely over the fact that we had no backup strategy and I was told that a backup strategy was not feasible. I was "convinced" (paid a lot) to stay on for another two years to see if Kiln would succeed. On my way out of that meeting, I commented to a coworker, "I mean, what's the chance a hurricane hits Manhattan in the next 24 months?"
100%. The chance was 100%. And the only reason people had to haul open containers of diesel up 17 flights of stairs was because no one wanted to admit that the chance wasn't 0%.
Anyway, not sure what was going on with Jane Street here, but holding up that Fog Creek/Trello/SquareSpace story as a positive always drives me somewhat nuts. It lionizes a failure.
> holding up that Fog Creek/Trello/SquareSpace story as a positive always drives me somewhat nuts. It lionizes a failure.
That's a common bias in management. Heroic fire fighting gets rewarded while silently preventing a crisis with proper planning isn't. Nobody ever gets credit for fixing problems that never happened.
We had a discussion about this at my last job. The War rooms were a three ring circus that my team stayed out of 4 times out of 5, substantially by actually planning ahead. Though as time went on we started doing a lot more to protect the services behind us instead of just ourselves, and then the total war room count dropped like a rock.
Someone hit on the notion that it’s Stockholm Syndrome. You bond with people you go through shit with. You don’t bond with people who prevent the shit. That was a super awkward conversation. The asshole in charge quit shortly after it, but 2 years later we still occasionally called each other out for making plans that accounted for a toxic chain of command that was no longer toxic.
> Nobody ever gets credit for fixing problems that never happened.
Exactly why the general public thinks Y2K was a "hoax" or "overblown".
I had a tech coworker who thought Y2K wasn't really a thing. I started working in tech in the middle of '99 and was dealing with Y2K stuff right out the gate. He started in the industry in 2001, so missed it.
See also: vaccines
And yet measles is making a comeback in the US
Exactly because vaccines were so successful that people don't remember what it was like without them. Once enough people start dying in large scale measles outbreaks people will start getting vaccinated again.
This is also a statistical fallacy biased towards dismissing vaccine injured people.
That feels like Amazon's entire management philosophy. Always drop the most important thing to work on at a moment's notice because we need to put out the fires caused by not prioritizing the most important things to work on.
At one point we had 2 2pts in our group and the other manager’s performance for that year was based on operational excellence (is that no longer a thing)? We hadn’t grind ticket root causes for months, but on call went from being woken up multiple times a night to possibly never getting paged the whole week.
Unfortunately, that typically meant when you were paged, the problem was probably significant, hard, or it was another team and you were just being triaged in because you were affected or might be able to help root cause.
> in 2012, I literally quit Fog Creek, largely over the fact that we had no backup strategy and I was told that a backup strategy was not feasible
This anecdote hits differently for me. I grew up reading about Fog Creek and consuming all of their writings (and that one film) as an example of what a software development company should look like. It feels strange to have the curtain peeled back to reveal that something like backups, which would be considered basic requirements today, were not part of their early operations.
Fog Creek of 2012 was a pretty different company from the Fog Creek in the early 2000s. While Joel was still technically in charge he had mostly delegated responsibities and spent almost all of his time working on Stack Overflow.
But was "early Joel" FogBugz any good? I heard that it wasn't from somebody who tried it. That unsuccessful trial was also the only time I heard of anyone using it.
Waaay back in 2007ish I got Fogbugz for the small 3-man IT team that we were. As a small team hacking away on an internal ERP and minimal other apps, it was actually refreshing. In my prior role I'd used BugZilla and it was ok, but Fogbugz seemed pretty sweet for a small team. I've now been plodding away with Jira for many years and it's just feature-bloated slowness I'd go back to FogBugz in a jiffy.
I religiously read Joel’s books - I think it made me a great Junior / Mid Engineer - sad to hear the failures.
Nit: the fact that an event occurred does not mean that the chance was 100% (although I guess this does depend on how one understands probability).
Probability is a way of expressing uncertainty. This is the core insight of Bayesian methods. That uncertainty is not an objective universal fact, though. It is specific to your state of knowledge.
If I flip a fair coin and don't look at it as it sits in my hand, the probability of heads is 50%. Your estimate should be the same. Before or after the flip doesn't matter because our state of knowledge is the same.
If I look (and you don't), the probability that it is heads from my state of knowledge is 100% or 0%. Your estimate would be unchanged because you didn't get any more information.
If somebody looks at the coin and tells you what they say, you estimate should be somewhat less than 100% or more than 0% depending on how much you trust them.
All of this is the long way around the barn to say that after the hurricane, the probability that Sandy hit Manhattan is definitely 100%. That's past tense probability, however. It is still also true that the best estimate a year before it hit would be much, much lower. Using words like "probably that a hurricane would hit Manhattan" makes interpretation much harder, of course, because of the uncertain time frame.
I have never seen this definition used before. I was always taught that probability applies to future events. If I flip a coin the event has happened, me knowing or not knowing the outcome does not change it.
Now you can say after the coin is flipped and you still haven’t looked at the result that the probability of you guessing the correct outcome is 50%.
It seems to me you're conflating the probability of x happening in the future as assessed from some previous point in time versus the probability of x happening in the past as assessed from the present. Without clarification the wording is ambiguous but I think generally interpreted to mean the former. It would be quite odd to go with the latter as it's rather tautological.
> All of this is the long way around the barn to say that after the hurricane, the probability that Sandy hit Manhattan is definitely 100%
Nobody talks like this in real life. I can't even believe someone bothered to write multiple sentences to try and justify it.
I don’t mean to glorify the poor planning or risk management, but still have a ton of respect for the people who organized the response and who did that hard work, just because it was what people needed.
My company during Sandy was operated in both NY and SF, and our experience was a bunch of us on the west coast filling in for those who were not only unable to get to the office, but without power themselves in NYC. It was all unfamiliar stuff for us (different business functions were done there, but the NYC office’s work was time critical) but did build camaraderie.
Ehhh.
"Everyone coming together to solve a problem by any means necessary" still loses something if the problem is sufficiently imaginary.
Like, carrying fuel up 17 flights to keep a children's hospital powered on, or a MUD that's been running continuously since '93? Sure, inspiring as hell.
"Our company's product will be offline for a few days and we'll theoretically lose money/marketshare, but all our customers will be super understanding because they were all watching the news and, holy shit, did you see that tunnel get flooded, and then everything will turn back on and be fine with no extraordinary effort on our part...?"
I think that's fair. I've been in situations where there are kind of "manufactured emergencies" which boil down to a sales conversation on a deadline which ends up getting rescheduled anyway. I have very little tolerance for that.
In the case of Fog Creek during Sandy, because the software we hosted provided planning and project management for a lot of people, going offline would have a pretty large ripple effect. Sure maybe 95% would've been fine, but I knew of customers who e.g. managed emergency services and crisis response, and so keeping them online was meaningful to me.
"Our company's product will be offline for a few days and we'll theoretically lose money"
This was a startup. The company getting fucked could mean our jobs going away. It was our equity too.
Not everybody is just punching a clock, while resenting the owners of the company. In our case, we wanted our company to do well because we liked working together.
> carrying fuel up 17 flights to keep a children's hospital powered on, or a MUD that's been running continuously since '93? Sure, inspiring as hell.
If MUD is the type of game you're falling into the same trap as the one you're signaling but worse. You're putting in the same sentence "saving kids' lives" and "the bragging rights for a game's uninterrupted session" on equal footing.
Humor isn't dead, despite HN doing its best to kill it.
Before Jane Street I did more humble work helping gay men hook up as fast as possible via a proto-Grindr site. We were co-located downtown when 9/11 happened. They had diesel generators but they ran out of fuel after several days because the national guard wouldn't let the diesel supply trucks through.
When we went offline customers were super understanding!
Except one customer who was outraged and we felt totally comfortable banning their account.
It was sometime in the mid-2000s when I went from an admirer Joel's writings and what Fog Creek was doing to suddenly realizing it was all just very slick marketing on top of very mediocre stuff.
It really was a shocking wake up period for me in the realities of the real world vs. how we often perceive things.
IIRC it was the whole "Wasabi" thing that kind of opened up my eyes that there was a lot more smoke and mirrors involved than I had realized before.
Speaking of Fog Creek, I didn’t realize that they renamed to Glitch and are totally out of business now.
It's hero culture and it's a curse people beg to keep.
Your comment brings back the times I've been on teams that worked double the hours a full time job should be, or spent 18 hours in a data center because a migration/data transfer was failing, and, and and.
So many times it's taken herculean effort to make things happen, usually because of poor planning and poor leadership.
Why these are viewed as "successes" or "heroic" baffles me as well.
Because it's cheaper to tell the team or company you're a hero and toss you a pizza and a day off than to hire enough people and competent leadership.
It's not a mystery, it's exploitation.
With all the business efficiencies we've made with computers as devs and IT, we should all be working 3 days weeks, instead a few people are so rich they can buy elections.
It is not even cheaper! Every time I have seen these, they burned immense amounts of unnecessary time and money.
> With all the business efficiencies we've made with computers as devs and IT, we should all be working 3 days weeks, instead a few people are so rich they can buy elections.
It's never going to work like that, and you don't need any dastardly, sinister, or conspiratorial explanations for why. It's a natural side effect of the market being competitive.
When tools and technology make people more efficient, some people take that to mean they can work fewer hours and get the same job done. Which is true. But other people take that to mean they can work the same amount of hours, and get way more done. This is also true.
In a market in which two companies are competitors, and one takes the former approach and one takes the latter, which company do you think will still be around in ten or twenty years? Obviously the latter. And that pressures people in the former group to demand more output and stable working hours so they can remain competitive.
> In a market in which two companies are competitors, and one takes the former approach and one takes the latter, which company do you think will still be around in ten or twenty years?
That is an argument of faith (that companies where employees work longer survive more than companies that optimise differently), and I am not convinced it is supported by evidence.
One could argue that all other things being equal, companies with employees doing more hours are more productive (even though I seriously doubt that, there are diminishing returns very quickly). But then I would argue that if a company drives their employees harder, there are leadership and management issues, so other things are not equal. And this is associated with younger workers (adaptable: good, inexperienced: less good), high turn-over (so, long-term information retention issues) and lower worker satisfaction (so the whole thing is self-defeating in the long term).
Besides, there are many, many factors involved that have nothing to with with working time. Again, leadership, strategy, operations, marketing, personal connections, dumb luck…
Personally I think there is an optimum somewhere, and at some point you get more out of your workers by making them more invested and motivated than making them work longer.
If we got rid of some of the greed and gave the workers a taste then maybe?
But everywhere I have seen people overworked has terrible morale and low productivity in spite of the hours.
This isn't entirely true, or we'd all still be working 80hrs like factory workers at the start of the industrial revolution.
I think it also remains to be proven that the company that works their employees the hardest will necessarily outcompete the others.
The 40 hour work week was the result of strikes and legal maneuvering by unions that led to labor protection laws in the early 1900s. Otherwise we would likely still be working those 80 hour weeks (alongside our kids).
It’s not necessary that the longer hours actually translate into productivity, just that the boss feels like it should.
And the fact that the strikes had a chance was largely due to improving conditions including, among other things, a working legal system that made it harder to flat out kill strikers.
Well, good organizing, too.
The answer is obviously that an overseer (i.e. the state) must enforce a floor to avoid a race to the bottom, because that argument implies then why not work 14 hours, 6 days a week. Like safety laws enforcing a common standard so companies are not out-competed by ditching safety measures, labour laws are also part of keeping the playing field fair in an competition economy.
Yet after 100 years of explosive technological growth we still have the labouring hours of the 1930s...
You can also curb greed with extremely high taxes at the top. We did it after the Great Depression, top marginal tax rate was something like 90% over 10 mm per annum.
The result was that business owners invested in their workers and long term health of their business so it would continue to give them 10 mm per year instead of trying to cash everything out every 1/4.
Been there, done that. 100% agree with your take. The actual workers should be celebrated for the extra effort but the need of the extra effort should be seen as a failure. It always stops at the first part.
>Why these are viewed as "successes" or "heroic" baffles me as well.
Your source of confusion seems to be not recognizing that sometimes in life multiple things can be true at once. "Heroism" is generally about the actions of the individuals in response to a dangerous scenario, not how the scenario came about in the first place. "Success" is more overloaded, but can refer to both the overall system in place and the outcome of a given dangerous scenario, again separate from what caused that scenario to come about. This is near perfectly true in particular if, as is usually the case, the ones dealing with merde-hitting-spinning-blades on the ground are not the leadership. It may be idiotic, short sighted, lazy, inept leadership that lead to the problem (or it may simply be an unforeseen alignment of holes in the cheese), but if people on the ground go above and beyond to prevent the worst that's still heroic of them, personally. And if their efforts work that's a success. It'd be pretty strange to insist that the efforts of firefighters shouldn't be recognized just because somebody else did something stupid that resulted in a fire.
Yes, it's absolutely just as important what comes after. Ideally it's like the commercial aviation industry, where heroism on the parts of flight crews can be recognized and said heroism results in a top-to-bottom full systemic investigation and efforts to ensure that ideally nobody will ever be heroes in quite that way ever again. And have constant efforts to make sure heroism isn't ever required in the first place.
The opposite extreme of some organization using it as an excuse for inaction and cheapness (at least until the day arrives when heroic efforts are not enough and a true disaster really does occur) is abominable. Even worse, because in that sort of utterly failed leadership environment the people on the ground will probably get scapegoated too. And it's 100% worth acknowledging that there can be a "success trap" and Normalization Of Deviance situation here, where heroics keep averting the worst which lets bad leadership/management escape correction. But it's a trap precisely because each individual failure can still be bad, up to and including literally people dead, so the pressure to succeed is still there each time.
But if employees, be they civilian or military for that matter, perform far beyond the call of duty why would that ever not be heroic?
The probability of a rare event does not go down after one happens. Especially with weather.
Most rare weather events take time for conditions to build up again. Earthquakes, volcano eruptions, wildfires...
Earthquakes and volcanoes are not normally considered to be weather events.
Heh. There's also cyclical weather patterns (El Niño). You get the point.
Volcanic eruptions have an unpleasant tendency to be episodic, and onset can be far faster than hurricanes or wildfires. Best to minimize how much of your infra has even a (supposed) 0.001% annual chance of being affected.
Oh, hi Gecko! Wild to see you in writing again.
Anyway, I was the first Fog Creek employee on site at 55 broadway (one of our founders got their first, and two peer1 employees were also on site) and also part of the first wave of the "bucket brigade". Another dozen of us showed up over the coming hours as we assessed and fixed things.
The media coverage on this was never super accurate-- we were carrying (and later, relaying) 5 gallon covered plastic buckets (like the home depot buckets with sealed lids) up to the landing where the generator was to top up its reserve tank. At one point SquareSpace (who was also running exclusively out of this DC) hired a group of muscle to bring a 55 gallon drum up, but that wasn't us.
I don't view this as lionizing failure, though I can see your perspective. Across our products (then FogBugz, Kiln, Trello) we had different backup strategies in play, but we were definitely putting a lot of eggs in the local-datacenter-that-claimed-to-have-resilient-power basket. Kiln in particular had decided to use SANs for data resilience, and that proved particularly troublesome for trying to find a workaround once the storm hit-- FogBugz on the other hand was able to spin up an EC2 version and start a slow-but-would-have-worked-eventually migration (and we even had an employee hit the road for us-east-1 location with a bunch of encrypted backups so that we could get a fast data restore online), and Trello fast-tracked-and-completed its local-to-cloud migration. While it was true that the DC had a backup generator, the fuel tanks for the generator being in the basement and becoming unavailable was not part of their resiliency plan. I don't fault us for not catching that.
...although I did have to later write the dreaded "I'm sorry, we've been hacked" blog post as a direct consequence of us not properly deprovisioning a "temporary" AWS credential that we created during that sprint. I will fault us for not catching that.
Shortly after Sandy, I switched my role from core dev on FogBugz over to DevOps, and I built out the backup and restore resiliency we had been missing. It wouldn't have kept us online during Sandy, though.
It was dangerous, and a couple folks slipped and fell. That's not a good thing and I won't celebrate that. Many employees opted out of the bucket brigade for family, safety, or personal reasons. I threw out all the clothes I wore that week as they reeked of diesel... not great.
I wouldn't wish that experience on anyone, but having come through it with fortune and no slip-and-falls, I still think about:
- Walking around FiDi post storm and seeing the flooded tunnels, clogged with cars that had been washed into them from the street parking.
- Dropping off big armfuls of coffee and bagels in the lobby for the building work crews. The folks who were essentially dredging out the basement and getting the building back online did much harder work than us.
- Going shopping for aquarium supplies in order to best jerry-rig the pump that our founder has repurposed from the office fish tank in order to transfer diesel from the drums on the curb into our 5g buckets.
Something that kept me going was that I knew that one of our FogBugz customers was the emergency response service in Haiti, and they got absolutely walloped by the storm. They were using FogBugz to coordinate relief, so there wasn't any way I was going to let FogBugz go offline if I could help it. Turns out that meant climbing a lot of stairs.
> Across our products (then FogBugz, Kiln, Trello) we had different backup strategies in play, but we were definitely putting a lot of eggs in the local-datacenter-that-claimed-to-have-resilient-power basket. Kiln in particular had decided to use SANs for data resilience, and that proved particularly troublesome for trying to find a workaround once the storm hit-- FogBugz on the other hand was able to spin up an EC2 version and start a slow-but-would-have-worked-eventually migration
The issue that I remember was that we had so many SQL Server databases that it was impossible to keep anything vaguely approaching a sane backup strategy; SQL Server just isn't designed to handle as many as we had. (I in fact remember talking to a famous SQL Server consultant [edit: this was likely Brent Ozar] we were using who said "that won't work," and then I or someone else was tasked with talking to a SQL Server dev at Microsoft about it and they ALSO said that it couldn't work, and when I brought this up, I was told that Microsoft was wrong. IIRC, the implication was that it would take an extremely long time (many days) to restore, and that the snapshots would likely be days old. I don't think any of that contradicts your memory about the very a slow restore going on, but might add some color.
(Likewise, Kiln supported multiple backend instances and had admin commands to copy and move repos between them, so it the SAN had specifically been the issue, we had workarounds that would've been fine with whatever EBS was up to those days. But I remember the databases being the blocker.)
It was indeed Brent Ozar! Yes, that's all right, though a huge amount of the bottleneck was the off-the-shelf backup tooling we were using. In particular we couldn't move a database between servers without minutes-to-hours of downtime per one database, _and_ as our database count grew our time-to-take-a-backup grew _faster_ eventually meaning that by the time we did a full backup run, we'd be due to run another full backup run.
Our "One database per tenant" model was also the reason we were so slow to go to AWS-- the MSSQLServer licensing terms were cost-prohibitive.
However recognizing that around 90% of the FogBugz databases were free-tier and didn't get daily changes meant that teaching the backup scripts to skip creating redundant backups solved most of the issue.
Post-Sandy, I was able to solve most of this by:
1) Shipping full, diff, and incremental backups at different schedules
2) Scripting the ability to create warm standby servers powered by the latest fulls, then diffs as they come online, then the logs at the cutover time. (so you can provision a new server and seamlessly cut over to it)
3) Using our own scripts instead of the RedGate tool (which was _fine_ just not equipped to handle as many DBs as we had) to orchestrate the backups and restores. This in particular was needed for the "only back up modified dbs" fix.
This got us to most of what we wanted-- 0-downtown transitions of databases across servers, nightly offsite backups, fast backup runs. The final problem-- if our datacenter evaporated and we had to reprovision everything from scratch -- was left as an acceptable risk of "we'll fetch the nightlies from s3/Glacier while we do the slower part of rebuilding our servers."
But to your point, during Sandy, none of the above had been fixed yet. I think our sysadmin team was very strong, but they weren't programmers (and/or didn't have enough time to step back and program), and I was able to pick a lot of low hanging fruit in DevOps
I mean this with love, but dude, you put your health (short term and long term) at risk over a web app? I think you need to reassess your life priorities. You work so that you can live and enjoy life. You don’t live just to be a worker.
There is no reason that they couldn’t have had more than 1 DC when Sandy hit.
Life priorities are not static, when faced with once in a lifetime scenarios they will change very quickly. Rational thought during emergencies often goes out the door.
Not everyone is built the same, thankfully.
I hear you, and agree with your spirit. I never felt in danger and was choosing my risks throughout the day. I'd also see folks working much more dangerous jobs than me all throughout FiDi trying to handle cleanup.
I guess it helps that I grew up in a rural place and did construction for a while, so lugging gas to a generator wasn't a new thing for me.
Also I was... 25? I'd be more cautious today, I think.
> holding up that Fog Creek/Trello/SquareSpace story as a positive always drives me somewhat nuts. It lionizes a failure.
Firefighters are heroes who bravely enter burning buildings, beloved by children and adults alike, muscular figures who get pictured on sexy calendars and thanked by tearful families.
Officious paper-pushers who make a fuss about a few cardboard boxes stored under the fire escape stairs (like me!) seldom get pictured on sexy calendars.
There's an element of risk involved, isn't there?
A firefighter entering a burning building is literally risking his or her life at that moment. A paper pusher who works on preventing the fire in the first place is normally very far from danger.
Large part of firefighters jobs is preventative - inspections and such.
To play devil's advocate, paying some dudes to haul diesel for a few days every decade might be a lot cheaper than maintaining a fully redundant failover site 24/7.
A just a single day of labor would pay for a fuel pump and line so nobody has to carry fuel drums up a staircase.
Im all for doing some dirty work when needed, but if they expect 100% uptime they should be ready for something as common as a storm flood without risking injuries hauling fuel drums by hand.
Fuel pump, oil line, and storage tank 17 stories high? In NYC? The is probably more then a few millions usd!
Yeah but if there is an emergency you can buy a stand alone 60 PSI fuel pump for like $120 plus a roll of fuel line. That would get you half the height by itself if you put an empty barrel in between. And by either moving the pump or buying a second pump you could transfer it up to the top again in either a fuel barrel or straight into the generator tank. Probably 12V pumps because its an emergency and you need what is on the shelf, but that isn't any real obstacle for a temporary setup.
60PSI will raise diesel about 50m. That's less than 17th floor, but it's worse than that because at the top of that 50m pipe, you get zero flow.
If you drop back to 30m of lift, you get still have 36PSI which is enough to get some flow. But that means that you need a holding tank (or lots of buckets) and another pump for another 30m of lift. That gets you to the 16th floor so you need another holding tank and a another pump.
That's a lot of complication. And getting those pumps to sit still can be a challenge.
Hahaha, we tried! Aquarium pumps were all that we were able to source. "Serious" resources were being deployed (appropriately) to hospitals, etc., so we had to make due with what we already had on hand. Thankfully we had an office aquarium, so...
They don't get paid extra. They're just expected to give up their free time and risk injury doing something they weren't hired for so bosses get a bigger bonus. (you also don't need a fully redundant site to maintain business continuity)
It would likely still be cheaper to pay people the going wage to lug barrels of diesel up the stairs.
Or you could label it Hyrox/Crossfit/$FITNESSTREND and charge people money to do it. #barrelpump #verticalmarathon #fitnessgoals #blessed
It's such a familiar engineering story. Heroic effort means someone, somewhere, screwed up. I aim for boring competence so nobody has to plugs gaps in systems and processes with their literal body sometimes.
> I literally quit Fog Creek, largely over the fact that we had no backup strategy and I was told that a backup strategy was not feasible.
Quitting a job because of this is so funny to me. 2012 was truly a different time.
Without disagreeing, I was, at the time, the lead for Kiln. I was acutely aware that any major issue with our one and only data center would result in Kiln going offline and us suffering irreparable harm, and my name would be the one tied to that, despite me fighting really hard to solve the underlying problem. Yeah, it was a different time in some ways, but I'd make that same call in the same situation again in 2026, too.
You made the right call. I also worked at 55 wall.
My story [not as interesting] is that I was a emergency responder living in BPC and was relocated to the millennium hilton because it had a generator... BPC never lost power :/
This is really interesting history. Thanks for sharing!
I'm surprised they didn't put a window washing crane on the roof as DR to simplify the raising of the diesel. The strange ass shit I've seen implemented to make the failure mode easier instead of eliminating the failure has limited my faith in our system as a whole.
Anyone here want to take bets on if NYFD would allow a crane to lift a barrel of diesel at 55 Wall?
For once , the name fits lol
I'm surprised they didn't put a window washing crane on the roof as DR to simplify the raising of the diesel.
If there was electricity to run the window washing crane, they wouldn't need to haul diesel.
If they are hauling diesel it’s to run a generator, a winch is nothing to a data center load.
None of us felt comfortable operating the crane or MacGyvering it to lift a barrel of diesel into the sky, and the building staff that could've done so was focused on bailing out the basement. It was something we talked about though :-)
That's fair, Rigging a barrel is a trick. It's actually pretty neat to watch an experienced rigger belt strap some oddly shaped item.
You can do a barrel knot, though with a modern metal barrel, you can also generally just do a hook on top an bottom and lift horizontally.
My comment was more about the goofy shit people do to make the failure mode simpler next time instead of eliminating the failure mode all together. Some of the places I've worked would probably have made one of us get trained on the lift or something instead of make sure we have the fittings on had or a fuel pump and hoses with 17 floors of head pressure. That would be the temporary fix and then when they got the quote for the pump it would become the fix.
Seems to me it is more likely that someone did the calculations and figured it is cheaper to send a bunch of unlucky morons up the stairs with jerrycans in case of flooding.
Unlucky moron here... eh, it's like the 5th order effect of the planning. Somebody definitely calculated it wasn't worth the money to maintain a hot offsite data center, and for the most part that felt ok because we had a profit sharing component to our salaries. But I don't think anyone could have drawn a line from that to the fact that Peer1 wasn't resilient to flooding.
Startup bullshit. People at "scrappy companies" make cutting corners look like heroism and practical-mindedness. Meanwhile the little people suffer and prop everything up as the business almost implodes.
I can understand your frustration, but let’s consider:
The chance of a hurricane within 24 months was never 100%. Looking back, it did happen, but you can’t judge future-facing decisions on post-hoc information that you didn’t have at the time.
There is a difference between “no one wanted to admit it wasn’t 0%” and “no one else thought the risk-adjusted return was positive.” When people decline to take some preventative measure, that’s usually what they’re saying implicitly.
I’ve been where you are—frustrated that you predicted and told people exactly what would happen, they didn’t listen, and you were right—many times in my career. It’s infuriating. Part of learning from it is figuring out how to get people to change, not merely figuring out how to be right.
My guess is that this kind of black/white Strong Opinions approach has bitten you throughout your career.
I was working at a data center in the financial district during Sandy. That was quite the experience. For whatever reason, our building was one of the last to have power turned back on. We never had to move diesel by hand fortunately. We did have a fun moment early on where one of our generators needed to have most of its load reduced for a few minutes. There were a pair of external fuel filters that needed to be changed fairly frequently. They were meant to be redundant so you could change one at a time. By the time we were able to get a tech on site, they were so dirty that when we started switching to just one the generator came very close to stalling out. After some conversation we thought it was thought that if we could drop the load enough we should be able to get away with just one dirty filter for a short period of time. The only chance we had at that was to shut down all of our cooling units. The generator was on the roof, and I was coordinating the timing over radio. You could dedinitely hear the generator wasn’t happy, but it worked. As soon as the second filter was replaced I let everyone know it was time to turn all of the cooling units back on. No one responded, and after a pause I realized that meant that probably no one had heard me and things were getting hot fast. I sprinted back inside. On the way my radio dropped, so once inside I sprinted to all of the units to tell people to turn them back on. For some reason I didn’t think about grabbing someone else’s radio. That may have been a lucky mistake though, as I’m not sure what all of those compressors turning at at the same time would have done.
I had made sure to observe the filter replacement process in case we needed to do it again ourselves. We did, and the next time someone recorded me doing it so everyone could learn and we handled most I’d the rest of the swaps from then on.
There were a lot of memorable experiences that week or so.
We had the fuel filter issue at 75 Board street as well. After a few days we had to do a planned generator shutdown (the data center batteries could last a couple hours to give us coverage...) as a diesel mechanic flown in by Peer1 could service it. Props for handling that yourself!
Definitely a memorable week. I wonder just how many folks in Fidi were learning the finer points of emergency backup generators.
GS? That building is known to have one of the best backup power systems in the area, spent time there during the 2003 blackout.
I would have thought that backup generators would cumulatively run for maybe a few dozen days over their lifetime. How did the fuel filters get dirty so fast?
The _fuel_ itself goes bad. Diesel will slowly accumulate dissolved water, and once it gets past a certain threshold, a lot of wonderful bacteria start to flourish in it.
So you get clumps of gunk in your fuel that rapidly clogs the filters.
I was thinking the same thing: fix the sediment problem in your fuel tanks, fuel filters shouldn’t get dirty as fast as this story indicates.
Because a genset that runs once a month and has a fuel tank sized for days of wide open operations is a textbook recipe for asphaltane and other garbage build up. Which of course you won't notice until you actually run the thing for a day straight and the fuel filters start showing it.
Bargain-basement and/or very old fuel? Backup generators are not known for having their diesel cycled on the regular
Diesel fuel has to be periodically filtered ("polished") by passing it through cans of dessicant. It removes accumulated water and prevents bacterial growth. There are also biocide additives but they just _extend_ the periods between maintenance.
That's also why it's not a great idea to use diesel for your backup home generator. It will go bad sooner than you expect, and then you're left with drums of gunk that you can't dispose and can't use.
And gasoline won't help you either. It goes bad even faster.
Propane tanks are the best option they can sit in storage indefinitely.
Or pull an AT&T in Nashville and depend on natural gas for your backup generators.
Which doesn’t quite work so well when an explosion cuts your electrics and the fire department cuts the gas.
If you do still decide to depend on another energy grid for an energy grid failure, at least have roll-up generator hookups so you’re not spending a few days figuring out how to wire that up…
Best info I found was on Reddit: https://www.reddit.com/r/sysadmin/comments/kk3j0m/nashville_...
This is why tri-fuel generators are a thing too, very cool tech for not a lot of money.
> 55 gallon drums of diesel fuel up 17 flights of stairs
Having moved 55 gallon drums around with a hand cart I don't think anyone is carrying them up stairs. They weigh 400 pounds!
The IT person for McKinsey & Company said water came within feet of destroying all of the data for the firm and its clients. That led them to stand up a second DC.
My colleague Brett has a Hurricane Sandy story about Jane Street as well
https://x.com/BrettHarrison/status/2117636364304892206
Sandy was wild. you couldn't get computers for love or money, everything was jammed up. we ended up with whatever hardware configs had been returned and sitting in a dell warehouse just to have enough machines so for a long time after you'd have a rack of the standard build and then a random machine with a fibre channel card.
shocking how much engineering you can get done when the normal process becomes impossible and waiting isn't an option.
"unwittingly reinvented datacenters" What? does he mean for JS or The Street?
This was in 2012, not 2002. Not having your DC in your office was well known by then to be a bad model especially for a DMM, NYSE itself knew this.
I worked for a MM, we were ready to open the next day because we were in actual datacenters.
Which in turn reminds me of the "Interdictor" blogger who did the same during Hurricane Katrina https://en.wikipedia.org/wiki/Interdictor_(blog)
That one tested a lot of assumptions in planning for sure. I recall that we had a sound DR strategy, however it turns out, our main site was in Newyork and the DR site was in New Jersey to save costs and still be compliant. Good learning for sure.
Why the hell did they park the generators on the 17th floor where people have to hand carry diesel fuel is the real question.
Generators go on the 17th floor so they can vent out exhaust and heat. Pumps need to go in the basement because you can only suck a liquid up by one atmosphere, which is something like 30' for diesel, but you can push a liquid up basically as high as you want. (Also, stating the obvious, the diesel tanker trucks are gonna be street-level, so you'd need pumps there anyway if you put the actual tanks higher.) This does, are you're pointing out, kinda stink if the pump itself is flooded, but the design's common and does make sense
In firefighting it is quite common to take in surface water using a submersible pump which is hydraulically driven by an engine on land - like the fire truck or a dedicated diesel. See for example [0][1]. Doing it this way means you aren't limited by elevation and don't need to carry special suction hoses for water intake.
It makes me wonder if anyone is paranoid enough to do the same for highrise fuel pumps: sure, the basement may flood, but when the pump in there is designed to flood it's a lot less of a problem! Place the actual power source several floors up, where flooding is impossible outside a The Day After Tomorrow-esque civilization-ending catastrophe and you should be fine.
[0]: https://hytrans.com/product/hydrosub-150/
[1]: https://www.youtube.com/watch?v=pLJT4On5naA
I see. I thought it was an emergency power generator they brought in for this incident. When you see that sort of thing for buildings, usually it's just a big trailer on the sidewalk hooked up to the building mains.
Probably on the 17th floor to put the generators closer to the loads and reduce transmission losses. Also reduces street level air and noise emissions (and avoids street level air intake)
With fuel storage on the same floor for a reasonable runtime, larger fuel storage in the basement and a pump (or two) to move fuel, you're well prepared for lots of things. As long as the fuel throughput on the stairs is better than the fuel use rate, then you're ok for as long as you've got people to move it. Which is ok for exceptional incidents.
More than likely that's where the available space was. Data centers many large generators and there may not always be space in the lower levels to fit them, especially if the data center was not one of the original tenants.
Ground floor risks flood damage presumably? But this doesn’t explain why the 17th was chosen.
You put them up there so you get a voltage boost from the gravity pushing the electricity down. Put the generators on the ground floor, and by the time the electricity reached the top, there would be a voltage drop.
Factoring in gravity isn't good physics or engineering. Gravity's effect is negligible in a generator placement in a skyscraper
I think it was a joke?
Something something sarcasm can neither be created nor destroyed
No joke - have you ever tried to siphon electricity? Can't do it more than 35 feet or so. You'll be boiling off your electrons into a plasma by the time you get to that height.
Carrying the diesel fuel up 18 flights of stairs also enhances its potential energy. About 47 Joules per foot.
You get out of it what you put into it. ;)
> The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
Awesome, story time! I was working at a not brand new but still relatively small tech company (~$100m yearly revenue) and we were in two major datacenters, one in Las Vegas, and another in Manhattan. We were essentially telephony in the cloud, back when that was still fairly novel. At the time, most of our revenue came from handling various 1-800 campaigns, like "Ronco Knives" if you remember that, and also big money raising telethons. Our platform was active/active between the two datacenters. Of course not for a given phone call, but we could pretty easily balance traffic between both datacenters, even fully away from one of them, though as we'll see later, at some peril.
Our main networking guy was also the main datacenter person, and he'd been in NY to help set us up there. I can't remember which datacenter it was, but we had like half of a floor well above the street.
I was watching Sandy carefully, and once the odds of it hitting Manhattan were substantial, I started to press for plans to move all traffic out of the NY datacenter no less than 24 hours before landfall.
The network guy disagreed; he said that he'd been there, on site. He saw the redundant generators. The redundant network ingress and egress. The site people joked that if somehow the first floor ever flooded, the the rest of the datacenter would continue operating normally.
I noted that while all that might be true, we don't have any such assurances for all of the other equipment that'll be underwater needed to support the network.
I ended up escalating to our boss, the head of operations. He was one cube over. :) We had a vigorous discussion, and the VP agreed to pull our traffic out of NY. So we did.
Probably 75% of our capacity at the time was in LV and 25% in NY. We were running pretty hot utilization wise, but LV could, under normal circumstances, handle the whole platform by itself.
Sandy came and flooded. Generators kicked on. The network stayed up. Then the DC/networking guy got an emergency call from the datacenter.
They had plenty of fuel on site, but it turns out that the fuel inlets for the generators were...under water and could not be reached. So everything was going to go down in a couple of hours.
And it surely did, for reasons I've forgotten, in a very crappy way. Our logs indicated that power hard flapped for all of our equipment quite a few times.
We lost quite a few PSUs due to that. Yes, we definitely should have shut our equipment down cleanly. Oops.
Diesel is ~7lbs/gallon. That is truly superhuman damn.
385 lbs, with a dolly, not too bad.
Not bad for who? A 385 lb deadlift is a lot even for very physically fit people. It is just asking for injuries or dropping barrels.
>17 flights of stairs
Assuming it’s all hands on deck, you rotate out the whole way. Shit sucks but it’s not superhuman unless you live your life in cans of Soylent.
Man only on HN would someone would have the stones to argue “350lb+ drums up stairs [just] sucks”! My dude it may as well be superhuman, do we really need to split hairs here? I think some hyperbole is in order.
I smiled when I saw this grayed out. I wonder what % of HN owns a proper hand truck.
The one we have wouldn't work on stairs.
Probably an order of magnitude larger than the number who've moved enough 2-400lb objects on it and have a built up intuition for what techniques work and don't for various situations.
I wonder how well these "superhuman folks" were paid
The "superhuman folks" above mentioned are the 75 Broad street bucket brigade crew, and uh... we made startup wages. We weren't quants, we made productivity software.
Though the "55 gallon" drum lift is greatly exaggerated, that was a crew of guys hired off the street who were walking around doing odd jobs in the disaster zone. Kinda wild. For us, we just carried the 5 gallon buckets.
It's Jane Street... they probably gave them a backpack of gold bullion for their trouble.
It was a bunch of quants so very well
I’d haul 55 gallon drums of diesel for measly 1.5m a year
You’ll need that money for the back surgery
I remembered going to our data center in Long Island City and labeling all drives and pulling them from the servers and putting them in a box and took them home.
One of the best decisions I made around that time was installing two backup Towerstream lines. When everyone else was out for weeks we were up and running, full operation.
The ars technica article says:
> “They had their brigade of people carrying five-gallon buckets up 18 flights of stairs or whatever it was to get to their auxiliary generator. Quite a task by those guys,” Orchard said.
5 gallons of fuel at a time (hopefully not in a literal bucket) is a lot easier than managing a big drum on stairs. Unless you have a powered stair dolly, but powering your stair dolly up 17 flights of stairs seems like a challenge too. Team lifting a drum up one flight of stairs feels managable, doing it up 17 flights multiple times is a hard no.
Carrying 5 gallons of fuel up a dozen or more flights of stairs is hard work on its own.
I've done this job. I only had to do it a few times per day, but it sucked each and every time.
(When I showed up due to a report of an outage for our client's gear at the top of the building, the street was flooded. Water was flowing into a giant metal grating in the sidewalk like something from a cartoon. This grating lead to the basement.
And that basement was where the electrical switchgear was.
It was bad. It stayed bad for a long time.)
5 gallons is a standard size of literal buckets, they can have lids but I might prefer carrying a bucket of diesel up 18 flights of stairs with no lid over trying to get one of those goddamn lids off.
Lots of gas cans are right around 5 gallons too; NATO 20L are 5.3 gallons, but widely available plastic ones tend to be 5 gallons as well.
Bucket lids are a PITA, but so is spilling diesel on the stairs.
I'm going to guess it was buckets. $2 each as opposed to 25-50.
Some plastics will dissolve in diesel. Which is why you should always use approved fuel containers and not a bucket.
If you’re burning the diesel within 8 hours after it was in a bucket for day 1 hour it’s not an issue. You shouldn’t store solvents in buckets that aren’t rated for solvents but you could carry that shit up the stairs in a paper to go box and it would still burn.
I don't have much experience with diesel but the wrong plastic with gasoline will turn it to jelly in minutes. I wouldn't consider it worth the risk to use random plastic buckets to carry fuel.
lol, ok, well, it sounds like you don’t have much experience here, I wonder why you felt the need to weigh in?
Almost anything that’s not a single use plastic and has a shape to hold will be made out of hdpe or polypropylene, just like a gas can, or a bucket, or those black and yellow boxes that are everywhere, or believe it or not, gas tanks on most newer ice vehicles.
So, yeah, don’t use a trash bag, but that shouldn’t need to be said.
The main reason you shouldn’t use things for normal storage or transportation is because they don’t handle vapors, or pouring without spilling very well. But you can certainly carry gas up the stairs in a bucket, and you can store gas in a bucket as well, but temperature changes can be problematic over time.
I have a stack of clean buckets that I routinely use when I need to remove the fuel from a vehicle to work in the fuel system. Gas or diesel cans are simply to expensive to keep a dozen on hand to use for an afternoon or a week while replacing a fuel pump or the fuel lines.
And in the case of a 17 floor bucket brigade, also likely too expensive and probably hard to find in the quantities they’d need to make that work, if you assume one bucket each way for 17 floors.
You’re probably not going to walk into an Ace and find 34 5 gallon cans especially when anyone with power is running off a generator. But, you will find that many buckets because it’s also a paint store and they store easily and cost nothing.
So they were probably using buckets, even if it’s true that you shouldn’t put gasoline in a garbage bag.
Yep, somebody went to home depot or lowes and got a bunch of lidded buckets.
Some of us did the full round trip a handful of times, but that was mostly early morning of when the power first when out and we just had to keep the generator going. Once we got into the late morning, enough of us were there that we could have a person at each floor and they'd receive a bucket and relay it forward.
It is a federal crime in the US to have fuel in anything but approved containers. They probably won't look/care in this case (they probably already know and choose to look the other way) - but I still wouldn't admit to on the open internet like that.
Podcast version - https://soundcloud.com/stack-exchange/stack-overflow-podcast...
I'm surprised they located their datacenter in NYC.
The NYSE, NASDAQ, and most other exchanges and brokerages run in datacenters in Cartaret, NJ and Mahwah, NJ. I can't think of a good reason why anyone would put a datacenter in Manhattan.
This is the actual NYSE datacenter https://maps.app.goo.gl/Da1E6Je9eJAGA1RK8
This is the actual NASDAQ datacenter https://maps.app.goo.gl/ZS9VJcZmf8VtvWEd6
It was another time. Only big companies had dedicated data centers; the rest had an air-conditioned room in the office. That was before the "cattle, not pets" era, too: they would have a database server called "Debbie", a DNS server called "Denise", a file server called "Felix", etc.
100% this. It was almost weekly that we had to go to the DC to do something hands-on with the equipment, so having that be a 5 minute walk from our office was a huge benefit.
yep
sandy was a wreck
the only time i remember executing on DR plans (which went well!)
(i became a consultant by the time covid hit, so i didn't get to do the whole covid scramble every sysadmin did)
https://web.archive.org/web/20121103050116/http://searchdata...
Apparently it was a 30-person 5 gallon bucket brigade of customers, contractors, and volunteers all lining up to help fill tanks because of interconnection issues between the supplier they got and where they needed to fuel the tanks up with.
"... neither we nor the fuel company had the proper coupling to tie into that existing infrastructure."
Sometimes "DR" in your plans stands for "didn't realize".
As a bonus though? They were able to in fact get ahead of the fuel burn, and the crew managed an eight hours' head back up on the tanks.
I had the dubious honor of rolling servers on office chairs and hundreds of drives from an EMC box in a sterilite to get it upstate to a satellite office.
There was about $5M of gear in my wife’s grossly overloaded minivan. The bean counters tried to deny my mileage because I was using a personal vehicle without prior auth. lol. My boss was ready to kill.
Not fun but educational.
Dude the bean counters. Geez. Sometimes they ought to get fired instead of overruled.
You should have billed as an emergency courier for $50k.
>The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
Sandy power disruption always reminds me of this photo - burned into my brain - https://archive.is/DoZsr
In the article he talks about living within a few minutes walk to work and telling too many people so they made him the emergency office guy. Same thing happened to me, foolishly told everyone hey I can see my apartment from here so I would get panicked calls to run in and shut down servers because the information security guy got a tip of an impending exploit about to be released on a holiday weekend with no way to do it remotely. Everyone else lived a ferry ride or commuter train away.
did you at least ask for pay increase?
Not the person you asked, but I've learned that if you're not careful about what you tell your employers and/or coworkers that it will almost invariably come back to haunt you in the form of unpaid labor.
It took me an embarrassingly long time to realize that (my 30's), and now I keep my conversations with my coworkers very utilitarian and divulge minimal information about myself. I try not to be a dick or anything, I'm polite enough, but pretty much everything I talk about is either directly relevant to work and/or so surface level as to border on "vapid".
ETA:
Just as an example, I used to have really bad sleep issues in my 20's that would make it so I wouldn't be able to get to sleep until around 4am, if at all. I made the mistake of bringing that up at work to my manager. Two weeks later I got a phone call at 3am because there was something he wanted to look at and "he knew I would be up". I was up, but I didn't really have much desire to get my laptop out and do work. I liked that job at the time and I liked my manager, so I did it anyway, and it wasn't until years later did I think about that incident and how utterly weird it was that I was sort of being punished for my insomnia.
My sleep issues have gotten a lot better now, so if someone calls me at 3am I probably won't hear it because I'll be asleep, but I remember that incident now and have it constantly spinning in my brain when I chat with coworkers.
I'm fine sharing (some) things about myself with coworkers, but I'm also in a position where if someone says, "So you can be our on-call guy for on-prem servers?" I can say "Absolutely not without a really big fucking raise, and I am not kidding, if you call me at 3am to come in, I will hang up on you."
If everyone had a spine like you we’d all be much better off for it
Our masters keep us desperate so we can't
There's only two answers to the question "can you be our on-call guy" which are:
* No
* Only if I get on-call pay
The problem here is not divulging info about yourself, its not being able to say no to work. I will happily draw a line where my responsiblities end and if anybody asks me to go over that I will ask for a role review. Too many people get walked over at work, stand up for yourself and say no!
I don't think I disagree with that at all.
Still, I do think there's merit in keeping your personal life to yourself and not telling your coworkers more than they need to know. I'm not saying you should be stoic or a dick or anything; by all means engage in conversations and be nice. I'm just saying that it's important to have boundaries.
I tend to be someone who overshares a lot (hence my enormous word count on Hacker News), so I just try and keep this in mind.
Fair response. I guess it also depends on the working environment, the type of person you are, and whether your coworkers are more or less like you as a person etc.
Exactly on point. Don't share anything that's not relevant to the job. In one of the previous jobs, I was coerced into doing a deployment to prod every 2 weeks at Friday 10 pm, since my boss knew I was single and not into partying or meeting with friends. Had I known better, I'd maybe even lie about my life and tell him to get lost.
Becoming a surface level/"vapid" work automaton would be a heavy price for to pay, perhaps not for you personally, but for me and I imagine many others. So, I'm not sure your example would lead me to the conclusion of your behavior. You mention being called at 3am as one incident that didn't mean much to you at the time and didn't become a pattern of exploitation, so what was really lost?
Maybe I'm lucky. I've so far not had to endure a job for which I was underpaid and expected to meet and exceed every contractual obligation placed on me.
I've learned that life is what you make of it. If you let yourself be used - others will use you. But if you showed your worth, but also showed others that you know your worth - things are different.
(otherwise where I work, I also avoid talking about politics for instance, but this is a different theme)
I don't really talk about politics at work either, but that's broadly because I live in a very progressive area where pretty much everyone has the same opinions that I do so it would be a pretty boring echo chamber.
Hehe, that got me laughing a lot.
A) because picturing at my current work place everybody learning of all my opinions at once and the resulting drama
B) I am in progressive spaces quite often, but they are far from an echo chamber with everyone having the same opinions either ..
No because I imagined a future where instead of me being asked for emergency help I'd be expected to perform a list of tasks every weekend as the defacto facilities manager. They were already paying me a ridiculous salary at the time. I thought about bringing it up if they ever said no to any requests I had and they never did so simply accepted it was my mistake broadcasting too much information at work which I never repeated.
I received many strange requests like to come in Friday night while drinking at a concert and ransack the offices looking for a laptop or possible phone left behind that they suspected was triggering alerts by repeatedly trying to connect to a foreign server. "You want me to drunkenly rob the offices at night and walk out the lobby carrying a stack of laptops? Ok"
Also a perfect way to ruin the ability to work remotely. If you live so close, why don't you come every day for 8 hours? /s
If I would have 5min to work, I would probably be in the office most days. A 1 hour commute is something I want to avoid, not being in the office itself.
Fair point, I prefer to avoid both.
Odd that they didn't have at minimum 1) a physical Emergency Power Off (EPO) button inside the datacenter, such that one button could de-energize everything -- usually required as part of building a datacenter due to code variance for the "low voltage" (<4000V in this context...) wiring. 2) EPO interlink to the smoke alarms, such that it would de-energize equipment automatically (maybe with delay) if smoke detected.
The 2020 US NEC seems to have provisions for an EPO, but I'm not sure when it was added if it was present during the time of OP's incident:
* https://www.f-t.com/post/emergency-power-off-in-data-centers
EPO disconnects for generators was around earlier than the above:
* https://www.mgiepss.com/blog/emergency-power-off-epo
You'd generally not want to have the sprinklers go off while things were energized, but there could be a mechanism to monitor flow, and only cut power if sprinklers were actually sending out water. But even if this was present, your first priority would be for the sprinklers to not go off as there would be a lot of clean-up / drying needed after the fact.
Unless you have a mist system (NFPA 750), sprinklers put out a lot of water.
I love this story and that they kept the wabi-sabi :-)
Missing NYC and this reminds me of NYI back in the day (still there perhaps) and operating datacenters in the heart of Manhattan before things changed.
I like that they're keeping the metal close to the traders actually and I'd be totally aligned with that decision. I am surprised however that they haven't shifted to Halon fire extinguishers or something that is safe(r) for a data center.
Interesting. The Sequence of Operations we write for the data halls I work on specifically do not shut down the CRACs on smoke alarm for this reason. The International Mechanical Code (IMC) specifically allows this where a air handler over 2000 cfm serves only one space because it's recognized that it will not spread smoke around a building. 2000 cfm is the threshold where you typically have to install smoke detectors, and typically they do shut down on smoke detection.
The CRAC won't feed a fire oxygen as it's just recirculating the same oxygen depleted air.
While CRACs won't recirculate oxygen, they can move air past sprinklers faster than they can handle which can delay operation, typically 5 fps. Although that's only if you are using sprinklers, clean agent is a different matter.
Yep, it is part of the design. The sprinklers are sometimes pre-action like described in the article, and sometimes clean agents. The only time we shutdown CRACs is if we EPO the whole room because otherwise some variant of this article happens.
The episode of their podcast where Ron explores the facilities side of their data centers and offices is worth a listen too
https://signalsandthreads.com/the-thermodynamics-of-trading/
Daniel's as solid a dude as they come.
He had to deal with so much absolute bullshit to get the diesel generator I mentioned in the story done, and then Hurricane Sandy destroyed it instantly a few years later. Felt bad for him.
Are there internet connected fire alarms that can be hacked to destroy the contents of some data centers? Like the European freezers?
I did not know that data centers with sprinklers exist. I guess this is a very stupid idea in the first place.
If there is a fire the decision is: all equipment might be lost in a fire OR all equipment will be lost by water damage.
Sprinkler heads on fire suppression systems go off locally - if there's a fire, then only the sprinkler heads in the vicinity will go off. The alternative is to let the fire spread and destroy the entire facility.
But in addition, this particular story takes place in a skyscraper in Manhattan, with the data center in question occupying a portion of one floor, with office space surrounding it. So letting a fire burn would create significant risk to human life as well.
Often only one sprinkler head would discharge, so the damage might be localized within that ~12' radius to some degree. It's not a deluge system like many movies make out sprinkler systems to be where every head discharges simultaneously.
The DC in question was in the middle of a 33+ story office building, so the the worst case is "entire building catches fire and people die", not "equipment destroyed".
I heard of an inert gas fire fighting equipment discharging without a fire in a data center. The pressure wave created by the initial discharge ruined all the hard disks and caused replacement of all the equipment even though there was no fire (or water) as the conditions meant no more warranty
> every minute, ask every server in each location directly for the temperature from its onboard sensors, take the average, and if it’s elevated, blast an alert out to everyone on the systems team to check for datacenter overheat.
I've once done the opposite: the office "server room" (a small room with three racks, two of them for the patch panels, one for the servers and switches) was cooled by a mini-split which was prone to locking up with the compressor enabled (the relay which controls the compressor got stuck) and freezing, which then stopped the air from circulating due to all the ice. I wrote a script which ran on one of the servers which read the ambient temperature through IPMI to detect when the room was too cold, email a warning, and power off (using ssh) the neighboring servers and then itself.
Oh, this reminds me a story from the time when I worked at a university. Poland, middle of harsh winter. The day before, there was a fire system inspection. On the day of event I was paged - our datacenter got flooded with water sprinklers (WTF why there are water sprinklers in DC!?), then the system opened all the doors and windows. Shit got frozen. The whole building was like an ice-skating rink. In the end only machines on top of the racks got damaged. Turned out that inspection was done by some relative of one of the professors and he fucked up something in fire system. It was a fun event haha
Great story, I love those in-the-trenches emergency stories. I wonder if it would have impacted Jane Street in any meaningful way, aside from a fat bill, if it ended with submarine servers. They could have diverged their compute to another datacenter surely.
We mostly hear about engineers from Jane Street here, and not much from their traders and finance people. I wish we could have some other perspective sometimes.
They publish a great podcast, Signal and Threads, I really recommend.
> They could have diverged their compute to another datacenter surely.
The latency impact would be huge
I wonder why the answer was to turn off the servers instead of turn their cooling back on. Would it not be able to stabilize the temperature fast enough?
The real answer is probably lost in the sands of time but even at 200 people we still had different teams with overlapping responsibilities. It's possible I was the only person reachable and close enough after hours, and while I didn't know where the CRAC unit controls were I did know how to rip power cords out.
I thought these data centers have a way to pump in a gas that prevents fires? Remember touring a data center at some point that has this ability.
That's how the server room at my school worked. They said to get out of there if the alarm went off because the room would fill with some inert gas and you'd suffocate.
I believe it's Halon, just like what's deployed for in-flight jet engine fires
It was a primary weapon for the BofH.
They have masks in the room. They should, at least.
They stopped using it many places because the risk to people was greater than the equipment.
The gas is used first, then water as a fallback. I'm not sure if sprinklers are required by code in all data centers, but they would be in an office building setup like Jane Street's.
Even though this happened in 2012, it's interesting that Jane Street is doubling down on compute and renting even more capacity from neoclouds. They are planning to expand 10x the amount of GPUs and are locking in tons of compute. On top of the $500M+ per MW of compute that they are paying Anthropic.
This is a trend in the wider algo trading market, not specific to Jane street
I specifically remember a TV commercial many years ago (15-20 maybe) that was trying to show the sophistication of the company with an anecdote of how when looking at some other company as a potential investment, considered if they used sprinklers in their data centers and determined that they did not. I could have sworn the company advertising was Jane Street.
Close, it was Janus Mutual Funds, 26 years ago.
https://youtu.be/EYJLeuUjMlU?is=lzmEC-Q_bJ93FTgG
TIL: wabi-sabi. I’ve been trying to find a way to express the beauty of imperfection in the age of AI, and this fits perfectly.
It’s extremely common and cliche for people to use that term for imperfect stuff these days.
God dangit Bobby
Great post. I hope a big paycheck was given to the author for saving the DC and the company from disaster.
Many many years ago someone told me that your data is not safe if is not at least in 3 very different physical places. Today, I would probably add the requirement to be on different providers as well.
good to see you make front mbac; glad you're still holding it down!!!!
remember sandy? that was crazy
How is it the only datacenter?
33 Thomas St is basically one huge datacenter, that's why it doesn't have any windows. https://en.wikipedia.org/wiki/33_Thomas_Street
It's Jane Street's only datacenter. not the only one in NYC. As they explain, they chose to specially locate their datacenter.
Yeah I didn't read the article in detail. It's the NYC that threw me off. And I didn't feel like it was that weird an concept, after all the real estate in NYC is super premium priced, not exactly prime datacenter country. I'm surprised there are many (like I said 33 Thomas St used to be just a phone exchange which used to be much bigger than they are now due to the mechanical tech involved).
The headline is confusing. It’s Jane Street’s only data center in NYC, not NYC’s only data center.
> The headline is confusing.
Nitpick, but the headline is not confusing at all, even though a couple of people are apparently confused. I suspect that is the fault of a hasty read through, not the headline itself.
Despite overall literacy increasing, reading comprehension has been falling for about a decade now.
This was pre-Hurricane Sandy, so before 2012. I'm guessing data centers have grown since then
Jane Street’s only NYC data center. Not the only data center in general.
Work nearby, it looks just as evil in person.
I think it looks cool. I used to work in buildings like this a lot. They're kinda cool. Dark, humming and clicking etc. I worked in telephony at a time when some (not many) of the old mechanical switches were still around and those buildings were especially spooky. It sounded like you were in the middle of a wild storm of crickets around 7pm when people finished dinner and started calling their friends. And all the little lights flashing all over (sometimes I used to keep the lights off for the cool effect if I could)
Some of them have half-floors for all the cable routing, where you can't really stand up.
I think this is why I like them. To me they're not just an ugly box but I always wonder what's inside.
> the smoke alarm is wired to shut down the CRAC units
Why not just turn the CRAC back on?
> you don’t trust the automation unless there’s a big red-painted, mechanically connected lever you can pull to turn it off
Seems like this is one of those cases where you want to override the automation.
First, the guy probably wouldn't have keycard access to whatever room contained such a switch. Second, it's highly unlikely that the system would be able to cool the room fast enough to avoid hitting the temperature limit.
Incidentally, we actually had a fire in the office building I worked in long ago. The HVAC didn't shut down, and smoke started to come out of random vents throughout the building. It was chaos. And yet, plenty of workers wanted to finish and send that email they were working on, commit code and push it to the remote server, etc. What sticks with me most is the one guy, a very senior manager, running around yelling at people to get off their computers and get out of the building, because no job is ever worth your life.
Building fires are no joke. Get the F out.
They'd be shutdown by a dry contact that may variously be coming from the VESDA (Very Early Smoke Detection Apparatus), or an internal smoke detector, or a duct mounted smoke detector. Not as simple as flipping an obvious switch on a front panel.
I did not know Jane Street had a datacenter in NYC. Must of had it a long time ago.
Wow crazy I didn't know they had such simple systems
This would be the one time I get to hit the EPO button (which this DC seemed shockingly too ghetto to have) for our breakers and it not being a resume-generating event.
Sounds like a good idea to hack in and blow up the infrastructure of the hedge funds. How you gonna hedge this suckers
Almost too close to 'Pool on the roof must have a leak' to be real ;)
The EPO is in the place I put that thing that time.
Ah the apocryphal VAX story/scenario is alive and well
I’m surprised by the pulling cords by hand part. Did they really not have an emergency power off wired up to a shunt trip or some other device to efficiently cut the power? Isn’t this mandatory, even? Especially in a datacenter in an office building?
I am shocked to hear they do not use foam.
Related :) - https://news.ycombinator.com/item?id=50434310
He earned his bonus that year.
Man, I wish MOC were still alive. I’d love to know his take on this story.
Surprised they even had sprinklers set up. Historically, things that avoid moisture like Halon Gas [1] have been used.
[1] - https://en.wikipedia.org/wiki/Bromotrifluoromethane
Likey codes or building management wouldn't let them make the necessary modifications to safely install a Halon system
Halon systems are banned, at the international and federal level, and have been since the 1990s; so, modern DCs don't and can't use them.
There are lots of 'clean agent' fire suppression systems that are not actually "Halon". I suspect people still use it as a generic term.
There's tons of substitutes, the ban on Halon wouldn't be why Jane St is using water.
Many people (like me) use "halon" when they are talking about general inert gas based fire suppression as there's not really a good short word for them otherwise.
If those go off, they end up destroying hard drives (the spinning kind), due to the sound pressure....
So fire suppression and data centers don't mix too well
Ahh... the Vax War (bit hard to search for it now... gotta search for the first line). VAXen, my children, just don't belong some places.
https://www.hack.org/~mc/texts/vaxen-joke.txt
Realizing that I'm spoiling some of it... but we're here, so some of it is inherently spoiled...
They don't have to blow that fast. It can be regulated.
But yes, I've seen low-O2 installations also. These have other issues though, like no protection when humans need to be in there for maintenance. And the need to be very very careful when reducing it again, making sure everyone is out. It happened to my colleague once that this went wrong and while it's not low enough that you'll die, he wasn't exactly feeling great for the next few hours.
Yeah, like the infamous Nasdaq Nordic outage in 2018 due to errant fire suppression gas release.
Other datacenters we built had them (which scared the shit out of me tbh). I don't remember why this one had to stay on sprinklers.
Maybe because it was in a normal office building? The gas extinguishers might be too hazardous there
Or CO2 but that's a life or death situation if there's any people still inside
Halon is not exactly fun to breathe, either.
Especially after it's been in contact with fire, the resulting products are even worse.
But it's only used in military and aviation these days anyway.
Probably not safe to use on a floor that has people on it.
We had it at work at a datacenter right beside offices and reception.
You just have to reinforce the ceiling panels and the windows so they don't get blown out, but it's not really an issue otherwise. It was all certified and regularly checked. Water is indeed a bit counterproductive as it's not really much better than fire for the equipment in there.
PS: Halon itself is of course gone but it's other less harmful gases these days.
What also happens is low-oxygen environments (they only raise the O2 when maintenance is being done).
Just curious was this on a ground level building? Seems unsafe in a high rise because of fewer egress points.
It was in a 3-level building. One DC was on the ground floor and one on the 3rd. But not exactly highrise no.
> Everyone’s heard of how awesome Jane Street is.
I guess I’m the last one to hear it.
Alarm firmware wasn't programmed in OCaml
They just need to hire some full stack people!
should be drowning tbh, less market manipulation
why don't they have a dry sprinkler system?
Datacenters on the coast never made sense to me.
Reminds me of DirectNic's datacenter flooding during Katrina (New Orleans), we all kept posting on Somethingawful until they finally shut it down.
> a false smoke alarm, a datacenter with no cooling, and a sprint to pull every power cord before the sprinklers went off
I smell AI