> 1. Assemble a small council of trusted senior engineers
> 2. Task them with creating a recommended list of default components for developers to use when building out new services. This will be your Golden Path, the path of convergence (and the path of least resistance).
> 3. Tell all your engineers that going forward, the Golden Path will be fully supported by the org. Upgrades, patches, security fixes; backups, monitoring, build pipeline; deploy tooling, artifact versioning, development environment, even tier 1 on call support. Pave the path with gold. Nobody HAS to use these components … but if they don’t, they’re on their own. They will have to support it themselves.
Step 3 is where I've seen my org completely fall on its face.
Yes, the golden path is "fully supported" yet after a year or two the company executive swoop in and say "why are we spending so much money on the golden path" and slowly, but surely, the support for is whittled away into nothing until the golden path is out of date and actively punishing anyone that chose to use it.
For example, one of our golden paths was a UX framework built on top of standard web tech for the time. The team maintaining that framework is no more and it's now very far out of date. Adopting it means you are pulling in Angular circa 2016 and that you'll be dealing with incompatibilities between that and any new web component you want to start using.
You can get screwed over by external (to your org, even more globally) requirements
For example I work for a Research University, mostly our software procurement is - while not always excellent - pretty good stuff. Maybe the supplier isn't as responsive as we'd like, maybe the software is buggier, maybe the documentation is worse, probably not all three.
However, the Government, responding to the usual anti-immigrant sentiment, decided it needs all Universities to check that people who are here on a restricted visa to get a degree attend classes.
The underlying sentiment is clearly racist, but OK in some cases you could imagine that's a real issue, a cheap course with foreign students who are registered but actually never attending because they're out delivering pizza or whatever. For a prestige research University though it doesn't make much sense - maybe you graduated top of your class in Taiwan, your parents pay an eyewatering sum so you can study here for an EE Masters to get that job back home with a team designing CPUs - then instead you skip classes to work as a taxi driver? No, absurd. But the government legally requires we solve this imaginary problem, and the only bidding supplier is garbage. So they're basically requiring us to procure garbage.
Because the supplier knows we're obligated, why should they support anything? Why care if it works, or is documented properly, or integrates with all the things they've told the government it can do? They know they're getting free money because of anti-immigrant sentiment, and they can take advantage of that until the winds change.
Their attendance tracking stuff could be useful. You can legitimately imagine having an early warning, OK, Sarah took a week for her mum's funeral, that's sad, but then she didn't attend any lectures at all for the next three weeks, we should have somebody go check Sarah is OK, or, well, she's clearly not OK, have them figure out what Sarah should do next. Take a year pause? Counselling on our dime? Right now, we aren't required to track Sarah unless she's on a restricted visa and the software is awful (for her, for us, for her teaching staff, everything) so we don't track her. So chances are nobody notices, especially if her class is big, and then in another month's time we find out Sarah was struggling after losing a parent, and we find that out because Police have to break her door open as she's been dead for long enough that neighbours noticed the smell.
I'd say yes. It seems like an unforced error to them to broadcast publicly how bad things might be, but that's because they want software to be an efficient factory that turns computer cycles into money.
You do need to be careful that you don't end up with a "build it and they will come" mindset when creating that Golden Path, though - it needs to have early and continuous input from actual users.
There's a tension between a theoretical Golden Path that leads someplace no-one actually wants to go, and simply paving every possible "desire line". Managing that is one of the trickiest parts of platform engineering.
"That's supported" has no universal interpretation beyond physically describing a tabletop atop legs.
Absent a clear/consistent definition, people interpret "support" in the most favorable way possible from the seats in which they sit. Then, all around sadness ensues.
This is really important - because "not supported" can range from "eh, we don't have to help you if it explodes" (which they often DO help with, but it's not official) to "this is literally against the license, the law, and the basics of human decency to even attempt".
> At some point, I realized that if I wrote a wiki page and documented the things that we were willing to support, I could wait about six months and then it would be like it had always been there. Enough people went through the revolving doors of that place such that six months' worth of employee turnover was sufficient to make it look like a whole other company. All I had to do was write it, wait a bit, then start citing it when needed.
Reading this I'm mostly shocked that groups had the power to throw boxes onto their outward-facing infrastructure and she had to handle that reactively. Like, it's not "requisition a public network server" but rather "you just jam it out there and we'll either baby it or boot it". That's crazy loose for a major corp.
Think "hardware groups with next generation experimental hardware". Facebook runs their own data centers, including designing their own servers. How do you verify that the servers actually perform well? One natural choice was to put them to work as web servers. Given its size, there was a lot of tooling there to be able to measure very precisely what the throughput of a web server was, so things like "How many of these new servers should we order" could be answered, in addition to the "does the rack catch fire" questions.
One example of a source of tension that such standards were trying to deal with here was in a group trying to run web servers on machines with SSDs that were way too small: obviously for the bean counter saving a bunch of money on the SSDs was nice, but for the team trying to make sure the disk can fit all the code and logs on it, it was less nice.
The funny thing with social networks is they all have both of these cohorts, so long as you go down the right rabbit-holes and engage with the right content creators
I interpreted it as the Rohingya / Myanmar genocide and Facebook’s role in it, and the CambridgeAnalytica scandal that (allegedly) enabled Trump to get elected.
As of 2023, it's all of them promoting a pro-genocide narrative, particularly Twitter these days is promoting outright pro-genocide accounts in my feed from people I don't follow, but I think the author is referring to the Rohingya genocide case in Myanmar.
It helps, but it’s important to have a name and a green check mark for each result so people understand WHY the magic doesn’t permit them to use the machine.
Yep, this is good. On one occasion, I saved myself a huge amount of back and forth by writing a dumb "test suite" for a software artifact to be shipped to a client; i.e. make sure it was compressed as a zipfile; it included an INSTALL.md; it was installable via [specific installer program]; it included [specific runtime libraries versions]; etc. etc.
Turns out human software engineers are really bad at fulfilling even well-specified requirements, so you will always save time by automating as much of the conformance testing as possible.
> I used to be on a team that was responsible for the care and feeding of a great many Linux boxes which together constituted the "web tier" for a giant social network. You know, the one with all of the cat pictures... and later the whole genocide thing and enabling fascism. Yeah, them.
I love how people are so willing to criticise companies that paid their salary but not their willingness to ignore the issues in the name of the fat paycheque.
This is clearly a reference to Meta, and in that sense the writing has been on the wall for years.
I wonder if the author feels or takes any responsibility in directly enabling that genocide and that fascism with their direct work.
Related, the concept of the "golden path" advocated by Charity Majors: https://charity.wtf/2018/12/02/software-sprawl-the-golden-pa...
> 1. Assemble a small council of trusted senior engineers
> 2. Task them with creating a recommended list of default components for developers to use when building out new services. This will be your Golden Path, the path of convergence (and the path of least resistance).
> 3. Tell all your engineers that going forward, the Golden Path will be fully supported by the org. Upgrades, patches, security fixes; backups, monitoring, build pipeline; deploy tooling, artifact versioning, development environment, even tier 1 on call support. Pave the path with gold. Nobody HAS to use these components … but if they don’t, they’re on their own. They will have to support it themselves.
Step 3 is where I've seen my org completely fall on its face.
Yes, the golden path is "fully supported" yet after a year or two the company executive swoop in and say "why are we spending so much money on the golden path" and slowly, but surely, the support for is whittled away into nothing until the golden path is out of date and actively punishing anyone that chose to use it.
For example, one of our golden paths was a UX framework built on top of standard web tech for the time. The team maintaining that framework is no more and it's now very far out of date. Adopting it means you are pulling in Angular circa 2016 and that you'll be dealing with incompatibilities between that and any new web component you want to start using.
You can get screwed over by external (to your org, even more globally) requirements
For example I work for a Research University, mostly our software procurement is - while not always excellent - pretty good stuff. Maybe the supplier isn't as responsive as we'd like, maybe the software is buggier, maybe the documentation is worse, probably not all three.
However, the Government, responding to the usual anti-immigrant sentiment, decided it needs all Universities to check that people who are here on a restricted visa to get a degree attend classes.
The underlying sentiment is clearly racist, but OK in some cases you could imagine that's a real issue, a cheap course with foreign students who are registered but actually never attending because they're out delivering pizza or whatever. For a prestige research University though it doesn't make much sense - maybe you graduated top of your class in Taiwan, your parents pay an eyewatering sum so you can study here for an EE Masters to get that job back home with a team designing CPUs - then instead you skip classes to work as a taxi driver? No, absurd. But the government legally requires we solve this imaginary problem, and the only bidding supplier is garbage. So they're basically requiring us to procure garbage.
Because the supplier knows we're obligated, why should they support anything? Why care if it works, or is documented properly, or integrates with all the things they've told the government it can do? They know they're getting free money because of anti-immigrant sentiment, and they can take advantage of that until the winds change.
Their attendance tracking stuff could be useful. You can legitimately imagine having an early warning, OK, Sarah took a week for her mum's funeral, that's sad, but then she didn't attend any lectures at all for the next three weeks, we should have somebody go check Sarah is OK, or, well, she's clearly not OK, have them figure out what Sarah should do next. Take a year pause? Counselling on our dime? Right now, we aren't required to track Sarah unless she's on a restricted visa and the software is awful (for her, for us, for her teaching staff, everything) so we don't track her. So chances are nobody notices, especially if her class is big, and then in another month's time we find out Sarah was struggling after losing a parent, and we find that out because Police have to break her door open as she's been dead for long enough that neighbours noticed the smell.
>or maybe it’s their executives who are afraid?
I'd say yes. It seems like an unforced error to them to broadcast publicly how bad things might be, but that's because they want software to be an efficient factory that turns computer cycles into money.
You do need to be careful that you don't end up with a "build it and they will come" mindset when creating that Golden Path, though - it needs to have early and continuous input from actual users.
There's a tension between a theoretical Golden Path that leads someplace no-one actually wants to go, and simply paving every possible "desire line". Managing that is one of the trickiest parts of platform engineering.
Also, document what "support" means.
"That's supported" has no universal interpretation beyond physically describing a tabletop atop legs.
Absent a clear/consistent definition, people interpret "support" in the most favorable way possible from the seats in which they sit. Then, all around sadness ensues.
This is really important - because "not supported" can range from "eh, we don't have to help you if it explodes" (which they often DO help with, but it's not official) to "this is literally against the license, the law, and the basics of human decency to even attempt".
> At some point, I realized that if I wrote a wiki page and documented the things that we were willing to support, I could wait about six months and then it would be like it had always been there. Enough people went through the revolving doors of that place such that six months' worth of employee turnover was sufficient to make it look like a whole other company. All I had to do was write it, wait a bit, then start citing it when needed.
Like!
Reading this I'm mostly shocked that groups had the power to throw boxes onto their outward-facing infrastructure and she had to handle that reactively. Like, it's not "requisition a public network server" but rather "you just jam it out there and we'll either baby it or boot it". That's crazy loose for a major corp.
Think "hardware groups with next generation experimental hardware". Facebook runs their own data centers, including designing their own servers. How do you verify that the servers actually perform well? One natural choice was to put them to work as web servers. Given its size, there was a lot of tooling there to be able to measure very precisely what the throughput of a web server was, so things like "How many of these new servers should we order" could be answered, in addition to the "does the rack catch fire" questions.
One example of a source of tension that such standards were trying to deal with here was in a group trying to run web servers on machines with SSDs that were way too small: obviously for the bean counter saving a bunch of money on the SSDs was nice, but for the team trying to make sure the disk can fit all the code and logs on it, it was less nice.
On a certain level I'm kind of impressed how long FB has celebrated that kind of hacker ethos, I just wish it wasn't for... y'know, evil.
She mentioned experiments. Facebook is indeed crazy loose with how they handle experiments on the site's user base.
https://journals.sagepub.com/doi/full/10.1177/17470161155995...
Move fast and break people, eh?
people, governments, political candidates, groups for old ladies trading knitting patterns...all of the above!
> a giant social network. You know, the one with all of the cat pictures
This really doesn't narrow it down.
> and later the whole genocide thing and enabling fascism.
Still not helping.
Rachel Kroll worked at Facebook. Her work history is not explicitly listed on her blog, but you can find it:
https://news.ycombinator.com/item?id=13400687
https://medium.com/wogrammer/rachel-kroll-7944eeb8c692
https://www.usenix.org/conference/srecon16/speaker-or-organi...
The funny thing with social networks is they all have both of these cohorts, so long as you go down the right rabbit-holes and engage with the right content creators
Rachel worked at Facebook for some time a while ago. She's been consistently mentioning the cat pictures to refer to it in her blog.
I suppose this way of referring to it is also meant to belittle it.
I assume the "the whole genocide thing and enabling fascism" part is more a dig than something to narrow down.
Yeah, might be Twitter, but might as well be Facebook. Though I'm leaning towards Twitter
I interpreted it as the Rohingya / Myanmar genocide and Facebook’s role in it, and the CambridgeAnalytica scandal that (allegedly) enabled Trump to get elected.
Maybe she didn’t really want to tell you which one it is :)
> genocide thing
I can only think of one social network when I hear that word. Are there others?
As of 2023, it's all of them promoting a pro-genocide narrative, particularly Twitter these days is promoting outright pro-genocide accounts in my feed from people I don't follow, but I think the author is referring to the Rohingya genocide case in Myanmar.
See https://www.amnesty.org/en/latest/news/2022/09/myanmar-faceb...
You could write a program that will check most of the system requirements, and say you must run this program and have it pass.
It helps, but it’s important to have a name and a green check mark for each result so people understand WHY the magic doesn’t permit them to use the machine.
Yep, this is good. On one occasion, I saved myself a huge amount of back and forth by writing a dumb "test suite" for a software artifact to be shipped to a client; i.e. make sure it was compressed as a zipfile; it included an INSTALL.md; it was installable via [specific installer program]; it included [specific runtime libraries versions]; etc. etc.
Turns out human software engineers are really bad at fulfilling even well-specified requirements, so you will always save time by automating as much of the conformance testing as possible.
[dead]
> I used to be on a team that was responsible for the care and feeding of a great many Linux boxes which together constituted the "web tier" for a giant social network. You know, the one with all of the cat pictures... and later the whole genocide thing and enabling fascism. Yeah, them.
I love how people are so willing to criticise companies that paid their salary but not their willingness to ignore the issues in the name of the fat paycheque.
This is clearly a reference to Meta, and in that sense the writing has been on the wall for years.
I wonder if the author feels or takes any responsibility in directly enabling that genocide and that fascism with their direct work.
But hey, a fat paycheque is a fat paycheque.
Maybe she left when it transitioned from cat pictures to fascism.
I would say you were less of a douchebag for taking those gigs once upon a time than now days.