The title reminds me of an interesting ancient Chinese anecdote. And it is also a bit ironic that Toyota has gotten itself into some scandals recently (https://www.bbc.com/news/articles/c1wwj1p2wdyo).
King Wen of Wei asked Bian Que:
“Of you three brothers, all physicians, who is the finest in the healing art?”
Bian Que replied:
“My eldest brother is the finest; my second brother comes next; I, Bian Que, am the least of the three.”
King Wen said:
“May I hear why?”
Bian Que answered:
“My eldest brother sees illness in the spirit, before it has taken shape, and removes it unseen; therefore his name is known only within our household.
My second brother treats illness when it is but a hair’s breadth from appearing; therefore his name does not travel beyond our village lane.
As for me, Bian Que: I pierce the blood vessels, administer strong medicines, and cut open the flesh. Thus, by such visible acts, my name has spread among the lords.”
I've been in those companies where "struggling departments" ended up getting all the praises and raise in budgets the following quarter because of the heroic saves they did, and raising awareness on how important they are... For stuff they totally caused on themselves.
Meanwhile, my perfectly purring department was struggling to keep the lights on.
It's a serious problem in this industry due to the disconnect between non-technical management (who understands how to double click) and engineering (who holds the company standing).
<insert IBM story about IT department cost cuts>
I'm not sure how we solve this, other than having management come from engineering.
By building pain into the system. If your hands dealt with injury directly without sending pain signals up to your brain, you'd never change the behaviour that led to that harm or reconsider your priorities. Like it or not, sometimes the best thing for an organisation isn't to just fix every problem and prevent it from bubbling up; it needs to be treated like a learning opportunity for org leadership, which means sending the pain signals upward before just repairing it.
Building the right incentives around that can be tricky, those incentives need to ensure the highest levels of management aren't themselves disincentivising their directs & their departments from surfacing pain & problems - but it's also pretty common for people to mask those signals purely out of a well-intentioned desire to help. It's important to coach people on the idea that in large group sizes, it's more efficient to let certain kinds of problems play out and not be so reactive to them.
Too many companies ground their performance incentives & processes around oversimplified ideas that don't match the reality of human behaviour
Often, 'leaders' make mistakes and people below suffer the consequences. It is important to let these leaders deal with the pain caused by their decisions from their cluelessness about how things work.
The problem is it's systemic. Ultimately, pain needs to come from outside. As long as society rewards incompetence, we'll have incompetent organizations.
No it does not need to come from the outside. If you're an underfunded IT department and your network has an issue twice a week, you will get that funding. If you're heroically obscuring the fact that things are falling apart you won't. That means even if you could somehow, heroically fix it, it isn't perceived as such if nobody ever felt the problem and saw you fix it.
This is a pain signal. Some IT dude saying things are crap in every meeting is not.
> This is a pain signal. Some IT dude saying things are crap in every meeting is not.
More often than not it is some IT dude observing network crap-out once a month, performing analysis, noticing an upward trend and then saying in every meeting that things are crap and there will be issues twice a week in some time.
> If you're an underfunded IT department and your network has an issue twice a week, you will get that funding.
More often than not, if the IT department is already neglected they will not get that funding. Things will be delayed until the crap outs eventually actually happen twice a week and then some external heroic consultants will be hired to fix the issue underfunded IT department "could not".
IT doesnt control the funding so at that point its not an issue of awareness but a management decision to live with this problem and focus funding elsewhere
more often than not, many things in the business are on fire and underfunded at the same time. you can get recognition for your work without the problem being permanently solved the right way, and it may not result in more funding but peopel will think of you for new opportunities that pop up later as someone who is reliable.
if you dont think the recognition will happen and youre just burning out solving these problems then stop solving them. new problem pops up thats outside your job description, its not your problem. generally though if youre working for someone like that anything you do is a lose-lose
How did you make the illogical leap to “could not”?
Repeatedly requesting time/budget to fix an ongoing issue is a requirement of any half-decent manager. If they’re reporting issues then just smiling blankly when asked “what can we do about it?” they’ve failed their basic job duty.
The problem is sometimes management knows who the heroes are and so by not fixing things they know you are not competent. Thus letting things bubble up isn't always a good plan. It is really hard when you are on the bottom to know which case things are.
The problem is that management witnesses the pain, but the response isn't to adjust behavior, it is then to punish the limb where the pain originated from. The reason that people pull heroics is also because the organization isn't healthy, and cannot reflect on its actions. Papering over organizational flaws is a symptom of a larger, often unseen problem. If it was healthy, someone would have already said, "hey, I think we need to work on this networking component" and it would have been looked at.
Pain propagation, to use the corpus metaphor isn't enough.
In the original DK experiment they asked students after they had taken a test that where would their score end up compared to their classmates (which they had no knowledge of). Obviously they picked scores around the middle*
Which resulted in top students 'undervaluating themselves' and bottom students 'overestimating themselves'. Or under/overvaluating a random future variable that they don't have knowledge of, at least.
The original DK paper actually shows a positive correlation between the guesses and the test results: students are generally aware how they are among their peers, and smarter students guessed higher than studetns with less time to study on their hand.
This being said, the 'DK effect' is something people talk about, and it might exist, and it might be perceived by people. It's just that the original DK paper does not support it.
* another lesser talked problem with the DK paper is that people don't actually believe the answers they give, because the question is nonsensical.
If someone just takes a test, they won't think that "I'm sure I'll end up the 24% this time". Even if they are forced to anwser this question, even then they won't believe it, because that's not how random and future work. People are generally aware of about where they will perform (with positive correlation, in fact the original DK paper shows it) but they are not aware of results of specific, random future events, and they are not claiming that they know the results of specific, random future events, or believe it in their hearts.
DK paper tries to frame them as they were actually believing this, but they are not.
Can't we just get to the pop culture version of the DK effect by deduction?
It seems reasonable to assume that for some group of intellects they are not smart enough to know how not smart they are. There is no definite boundary where this effect is either on or off, therefore there are probably some gradations to this awareness as you climb up the intelligence ladder.
Another way of putting it: if dumb people had more insight they would cease to be dumb.
Honestly? I’m just super direct with the exec team now after trying to do this dance. Obviously this is not allowed at every company, I’m just lucky to be at a place where the company culture allows for this.
I’ll ask for something preventative or that otherwise hardens our systems. They ask “is it a need?” and I’ll say something like “we can function without, but that means we have a 5-10% chance in the next 6mo of having a major failure and embarrassing ourselves in front of a live audience in the thousands as well as our client.” They then decide how much that risk is worth to them, and whatever they decide is kind of out of my hands at that point. If the thing I warned them of comes because they didn’t pay for it, I can point to the receipts (though I’ve never had to, we’re small enough people remember those conversations).
60% of the time they just get what I need maybe? But ultimately it’s about CYA. Tell them what’s up, tell them what the solution is, tell them what the consequences are if they don’t do the solution, and make them decide.
Again this obviously depends on company culture and structure, but I can’t imagine on the only person who can do this!
An example that not all companies are run by idiots. The job market is not a healthy market though, where its more important to know ppl then to be great at some skill. But if leadership sucks just leave if you can, that will fix the problem.
> By building pain into the system. If your hands dealt with injury directly without sending pain signals up to your brain, you'd never change the behaviour that led to that harm or reconsider your priorities.
I don't think that's it. Emergent problems require attention and action from leadership, who in turn can make the problem visible to higher ups. This creates signal, and positive feedback when the problem is fixed or mitigated.
If the problem doesn't exist to begin with, there is no signal. Managers don't get to show their fast-acting skills, and there are no heroics to speak of.
So ultimately poorly maintained and managed projects who deliver fixes for problems of their own doing create a perverse incentive, whereas no one is lauded or promoted for doing normal day-to-day things.
Well I think it is even more complex. If you're a plumber in a rotten system of pipes the whole company depends on, you can fix issues day in and day out, without speaking a word and they will notice everything is a bit unreliable and thus you do a bad job. You could do the exact same work, but make a big thing about every major fix, warn people a week ahead, give them the feeling the company depends on it and then do the exact same work and tell them how you fixed it. Suddenly you did a good job, despite you literally doing the exact same thing with your hands.
The difference is how it was communicated. Most non-Tech/non-infrastructure-people got no clue about these things. If they know you're battling the demons of plumbing on their behalf they will thank you, if you're the weird guy that has smeared dirt in the face and is seen once a week while the plumbing fails ever so often, guess what.
That means even if the problems and their fixes remain the same, the communication around them really matters. Tech people can be extremely bad with this. And if we're talking IT it is really the plumbing that holds the company together.
> If your hands dealt with injury directly without sending pain signals up to your brain, you'd never change the behaviour that led to that harm or reconsider your priorities.
At some point in one's early single-digit they learn that touching hot stuff hurts. They start to avoid stuff that they know is hot, but still come in contact with hot stuff accidentally. Later they learn techniques minimizing probability of touching hot stuff even by accident. By the time one reaches twenty or so, the only times a person burns themselves is really by being way too reckless.
> Like it or not, sometimes the best thing for an organization isn't to just fix every problem and prevent it from bubbling up; it needs to be treated like a learning opportunity for org leadership, which means sending the pain signals upward before just repairing it.
Should we accept that management as a whole is in general more clueless than your average teenager? The "learning opportunity" should, ideally, happen exactly once, realistically once in a very rare while.
> It's important to coach people on the idea that in large group sizes, it's more efficient to let certain kinds of problems play out and not be so reactive to them.
You are conflating two things here, I guess. Yes, some "problems" are not worth to be fixed proactively or at all, but that has very little to do with group sizes, it's a "simple" cost-benefit tradeoff. As groups grow the left hands tend to become increasingly unaware of what the right is doing and that is the primary reason why we have management class in the first place.
The problem OP raises is attention span of the metaphorical gold fish in the management layers. Even if a department does everything in their power to communicate impending problems, do risk weighed cost-benefit analyses, get proactive treatments pre-approved by higher management, the same higher management forgets the risks and costs savings once they have been mitigated, effectively incentivizing firefighting. Some teams gradually fall into eternal firefighting and burn out, others start manufacturing fires to get rewarded. The biggest problem is that it is nearly impossible to tell the two apart.
Unlike a teenage child, management has the unfortunate effect of being made up of people who can leave the company, forget past experience, etc. So you do kind of have to treat them like a child who needs continuous feedback and signals.
For a more broad example than IT cost center stuff, you can look at how some large companies go through cycles of arrogance with their customer bases, launch a product that fails, and then are humbled enough to try and pivot and earn good will back. Microsoft is always somewhere in this cycle for instance. The organization can never really learn this lesson permanently and will "regress" from time to time based on financial pressure or greed or some other impulse.
In my 35+ years in IT, the "hero attitude" was the one in the top three I most hated traits in a person working with or for me. And talking about traits, I considered crucial to always have in my teams a "saboteur" engineer - the one who thoght, found, come up with all the way we could break a design, service, infra components, app, etc., when all the others were designing or operating for perfect or normal conditions.
Genuine heroism - a willingness to step up when needed - isn't a bad thing in itself. But /needing/ a hero just to function means the system is fundamentally broken. Maybe it's a bad process, maybe it's understaffing, maybe it's neglected maintenance, maybe it's a lack of contingency planning. But there's no reason everyone can't go home at 5 PM every night and still get things done.
Love the "saboteur" approach. I honestly want to be one my own career in IT, but as you have rightfully conveyed, "hero attitude" is what gets you visibility!
At a previous job the CEO/owner had the idea that you'd get some percentage of any cost saving your could find as a bonus. Something like 20% of the savings for the first year.
My colleague in the IT department had one idea, replace our commercial certificates with Let's Encrypt and drop the EV requirement. In total he'd stand to get a bonus of a little over €2000. He never got the money, because things like that was part of his job apparently.
Even dumber, they've now got a disgruntled employee, and everyone around them knows they were cheated by the company.
If the policy is wrong and needs to be more specific, pay it out this time and change the policy. Don't just break your word.
The policy they think they've implemented is stupid. "Save money in someone else's department" is just going to create a ton of anger as people rush to step on each other's toes, and then those people have to constantly re-justify all the decisions they've made.
I think a good place to start is tracking all the proactive things being done and reporting them. At least then maybe someone will see why it’s quiet, because you’ve anticipated the problems and stopped them before they start.
When things come up with other teams, you’ll have a catalog of tasks that were done to show why you didn’t have the same issue. The work was done, just at a better time to avoid downtime.
> start is tracking all the proactive things being done and reporting them
Speaking from experience, this does nothing. If you're at a company that is okay with average performers, then absolutely, 100%, fix all the bugs in advance, make the system rock solid and stable, prevent downtime, be a good engineer.
If on the other hand if you're at a company where 10% of people must get stack ranked and PIP, or at a company where "meets expectations" actually means you're going to get the stick, and you're supposed to be "redefining" expectations every year ... then yeah, don't do anything preventative. The optics are better when you take the 3am on-call and fix the issue (that you secretly knew in the first place would happen some time in the future in your coworker's code, and already knew how to fix -- but don't actually fix it until it surfaces). Be the savior that the VPs praise in the next meeting, that's your insurance against the PIP.
They set the rules of the game, you just play the game. The rules were their choice. They could have chosen different rules.
True, stack ranking is a terrible management approach, and if you work at a company that does it, then playing the game is the only way. But frankly, I'd be looking to get out anyway. The best way to play thr stack ranking game is to be job hunting.
But I'm not sure the author of this thread works in such a place. In that case the game is different.
In the case where the "urgent midnight fix" is important, it's necessary to promote the visibility of your (just working) team. If visibility is the game, then be visible.
You know how test-driven-dev was always "write the test first"? In that environment a test is always written before any code.
Well in the "ticket closing" scenario it's important to open a ticket, regardless of how trivial, for every code action taken. For every meeting attended. For every scenario dodged. If tickets are the way to score then write tickets.
If "being a hero" is the valuable thing, then be a hero. Be prepared to champion your team every chance you get. Every time you interact with management stress the emergency you just fixed (before it became an emergency.) Tomorrow do it again with the next thing.
Management needs visibility. Be visible. I know, this seems stupid and beneath you. But that's why they call it a job, not playtime.
> I'm not sure the author of this thread works in such a place
I worked at Amazon, previously.
> Management needs visibility.
I know this very well, and this is a problem. The nature of jobs in any industry is that not all of them are equally visible. As a manager, you should be proactive in assessing the state of things rather than waiting for people to deliver visibility to you. People who deliver "visibility" in spades are often charlatans. People who deliver fixes, code, and improvements in spades usually do not have time to manage their own public relations for your visibility.
However, you have ALL the tools to proactively see what they've been upto. You can attend their standups and other regular meetings, you can set up an updates document, you can see what they've been posting in Slack, you can look at their PRs and commits, you can look at JIRA tickets, and in the age of AI you can have AI explain to you all of the parts of the above that you do not understand.
I refuse to play those games. If they want to fire me for avoiding problems instead of sacrificing my sleep, fine. I’ll go stock shelves at Walmart.
If someone is constantly playing the hero, I see that as incompetence. If the boss can’t see that, they are also incompetent. I have no respect for “leaders” who don’t know how to get out of the firefight.
I’ve made some high profile appearances, working 18 hour days on 4 day long outages, from vendor issues I was no part in causing. I figure that gives me some good will on playing hero without willingly creating problems for myself. I’m too old to manufacture stress for the optics.
For what it’s worth, with the right boss, I have had proper reporting work. Everything ran smooth and work was relaxed. My boss would regularly tell me I should take 3 months off because we were so far ahead of everyone. He would occasionally get bored and lob a grenade into the works to cause some chaos, but since everything else was running so smooth we were able to sort them out and keep going. People who couldn’t explain what they were doing were always getting yelled at and assumed to be doing nothing.
Yeah, but then I wouldn't have been able to pay for my healthcare. A certain toxic company's health insurance paid for my care, though. Prior to joining said toxic company I'd be racking up $6000+ in healthcare bills a year with shitty startup-sponsored insurance.
After 2 years, it was decided I didn't play the hero well enough though, and ended up having to leave. I work for a less toxic company now, but the next time I need a heart-related surgery (likely in ~5-10 years) I'll join a toxic company in the months leading up to pay for it.
The rules of the US, I guess.
> I’m too old to manufacture stress
My point was less about manufacturing artificial stress. I don't do that. But many times I see issues in coworkers' code. If the company will value and praise me for catching and fixing them early, then by all means I'll do that. But if fixing issues in the codebase early for prevention only gets me criticism of "you haven't met expectations, we expect you to exceed expectations every performance cycle" then hell, I don't feel like fixing anything proactively. In that world I'd rather be the hero that fixes it when it surfaces, that's more likely to nail the rating.
Health insurance does complicate things. I hope your heart is doing well now.
I will say my motivation for helping other people avoid issues has dropped. If they want to make problems for themselves, they can. Me helping them hasn’t worked so far, so maybe some sleepless nights will be a better teacher.
I had a former boss call me Brent after reading the Phoenix Project. That made me step back and stop helping so much. Everything seems worse, but whatever… if that’s what they want.
> They set the rules of the game, you just play the game.
Obviously the only winning move here is not to play. Things like stack ranking are a perversion and no amount of compensation would be worth working for a company like that. If you choose to play, you're complicit in the moral abomination.
Isn’t this a universal problem though, not just software industry? Even at home, if one kid just does his thing quietly but another kid is difficult, what do we say? “John has his problems but he is trying, we gotta encourage him”. While his brother gets no praise or attention for just doing his thing quietly without fuss.
When things run smoothly, very few people notice. When things break, everyone notices
Econometricians can solve it, bc we can create rigorous models that map causal inputs to output.
It’s extremely advanced technology, though, and most CEOs would rather rent seek / camp than give up some decision-making power (and very few are even aware it’s possible).
I'd generally point to econometrics and statistics applied to business. The key activity is causal inference and then the context determines the mix of econo vs. stats required to help the org make high-quality decisions to increase output or make it more lucrative or higher-quality.
> I'm not sure how we solve this, other than having management come from engineering.
I disagree with the implied idea here that "engineers are better managers". The solution is to have good management, not to assume that "engineers are better managers". I have seen good and bad managers, and in both groups there were engineers and non-engineers.
> I've been in those companies where "struggling departments" ended up getting all the praises and raise in budgets the following quarter because of the heroic saves they did, and raising awareness on how important they are... For stuff they totally caused on themselves.
This is a very game-able system, and I'd wager a decent amount that any senior engineers on those teams know exactly what they are doing. In a lot of (broken, but aren't they all) management structures, it's better to be seen to swoop in with the save than to quietly fix it ahead of time.
And if your management is structuring rewards like this, it leads to your seniors anticipating a bunch of these failures, lining up 90% of the fix before hand, so that they can jump on the oncall escalation with a 100% "Hail Mary" of a fix...
Ah, yes - the person who comes in at 7AM and gets shit done by 11 is a slacker, and the one in the office just doing nothing after 6PM is the hard worker. Same thing.
You can't fix this. Out of sight, out of mind. It is hard-wired into us. It's all about the optics, and will always be.
Managers will let you get away with anything if you time your reports correctly. They also don't want to sit in meetings where they are reminded of better outsourcing alternatives and they chose to dogfeed instead.
We've become too comfortable, since actual toil is no longer seen in the company: Manufacturing is overseas, customer support is overseas, logistics is an afterthought with established guarantees. Thus we want the mild weather and smooth meetings. If your engineering team is too smooth, maybe you should already branch out to help other related but "struggling" teams to get your hands dirty and noticed.
This thinking eventually results in The Scream Test. When the screams come as a system fails that is when they act on it.
Alas, for many parts of society there is a large amount of people that would rather be reactive than proactive. It means it is easier today but harder long term.
I guess the point of view is that if a department is well running, it means it is overressourced. So you reduce the ressources until it's breaking point, just enough for it to not fail. A jaded service manager told me it was part of its official training: if the clients was too satisfied that meant that human ressources were wasted on them, so he had to spin plates between clients. I guess it was economically optimal.
I run the media tech at an university solo. Needless to say I am underfunded. But more importantly, the infrastructure was underfunded too. I made it my first policy to also report near misses up the chain with their full implications, e.g. a list of events that we would not have been able to make.
E.g. that time a central media controls power supply broke down which would have made using one of the most prestigious rooms impossible. I fixed it myself by swapping in a spare power supply from a used unit, then went on to remind them twice a year that we are now living on borrowed time and I take no responsibilities if a fault I predict to happen and get no funds to fix will in fact happen. 4 years later I got the funds.
Having stuff costs money. Everybody wants to invest funds once, but nobody wants to keep paying for maintenance.
lol. I hate presentations. I like to run a tight ship. But that does not shine, so they made me do presentations every quarter. If you do some work, you must "take" credit. It is kinda a need when you manage people since you need to build their careers.
I finally moved on to be an IC. Same story, same pressure :) You need to present to directors not because they need to know, but because your managers have a quota of N presentations per quarter, and if you back out, someone else needs to step up.
Needless to say my productivity reduces by half and sometimes to almost zero during the week or fortnight of presentations every quarter.
The business defines it as "meetings, presentations, support, coding, whatever".
Your productivity remains at 100% when you are doing what they want.
I get that you thought you were hired as a coder, and thus measure your productivity by that. That's what I thought too. I ended up doing a lot of support (which is good, but that's another thread). Until I recalibrated my definition of productivity that frustrated me. When I realized that support was productivity I got much less frustrated.
I have been on the industry for 35 years. I have seen my share of technology evolutions and o have seen the work from a dozen different dimensions. If after all that time, I find the process painful, just trust me -- they can't change me, and I can't change them. You take the warts with the wins and move on. 2-3 bad weeks, 10 good weeks. Life moves on to next quarter. Complete CEO mindset :)
You heavily implied presentation preparation implies zero productivity. He tried to say this prep is also productive even if you personally don't or can't appreciate it.
I meant my other productivity drops because I am not a natural presenter so even though I am rehearsing / editing for 2 hours a day, the presentation consumes me / overwhelms me that I can't even focus for the remaining 4 hours or 2 hours. Just do the bare minimum email processing, just survive. Everyone knows it. But by being in that zone of paralysis, I can still deliver a presentation. Sometimes good sometimes ok.
I have this need for the presentation content to reside in my memory cache and other work disrupts the cache quite badly.
But that's not a way to live. The other work stalled for 3 weeks.
Which they should. I've been lucky enough to work at places that had great non-technical managers that promoted based on great execution, as well as highly technical managers that also promoted based on great execution.
Now I'm at the other kind of place and it sucks. They'll fire the performative engineers though during layoff season. It's almost like they like playing politics until it really matters.
I feel like this is a cultural symptom and something many people are hoping to solve in healthcare. Basically we treat solving problems as amazing rather than preventing problems. You get rewarded if you treat a sickness instead of keeping a healthy person healthy.
This is the same thing. We need to reward things never going wrong as a society since this is pervasive.
> something many people are hoping to solve in healthcare
Respectfully, the solution is don't smoke, exercise, eat well, sleep, avoid stressors... These aren't easy problems but their solution isn't at the individual patient level and is a simple question of capital and political will.
The 'hope' envisages a product to temporize the solution while extracting large payments.
Nope, I believe you are wrong: a path where we, for example, forbid smoking because the statistics point at it correlating with many health problems, is a world where we use the same statistical tool to prescribe human behavior to the last detail. It is not just about smoking, alcohol, late night dancing, switching sex partners, fast driving on a track, paragliding, skydiving, climbing, car driving, bicycle driving, motor biking, even staying late for astronomical observations (sleep patterns?)... all carry insignificant risk when looked at statistically.
> ...avoid stressors...
Most stress is caused by a conflict between our expectations/motivations and the reality (everyone else's).
> forbid smoking because the statistics point at it correlating with many health problems, is a world where we use the same statistical tool to prescribe human behavior to the last detail.
I had a really great egg for breakfast. This now means I will never eat anything else besides eggs.
Also, I realized that cars run better with oil changes every 3 months or 5,000 miles. Because shorter was better, we should all start changing oil daily.
The best player in the basketball game last week was over 7'4" tall. I guess I need to discourage anyone who isn't that tall from playing ever.
I believe it's a problem in most industries and even humanity in general. I don't believe it's a business problem at all.
Heroes are lauded even if they solve problems they themselves are the cause of - which is conveniently either forgotten or denied - or they are solving non-issues that are deemed important by the ignorami-class. Politics, for example, is utterly dominated by this dynamic.
It's the first instinct: let the expert run the show. However, one of the (many) ways to let a complex project fall apart completely is to hand over full control to engineers. I'm one myself, but I know what I'm good at and what not. Dunning-Kruger is often mentioned in these discussions, but don't forget it also applies to engineers that often lack any management or leadership experience of any appreciable kind. They vastly overestimate their ability to handle management and organization-wide issues and tend to not only miss the forest for the trees but actually miss the trees for the leaves.
"Unix: A History and a Memoir" by Brian Kernighan actually mentions how proper management was crucial to their success. It's a detail that's frequently conveniently forgotten by the engineers who think themselves better than the "suits". For the record, I don't claim engineers are the primary problem, but it's not just management's either. Quotes like "who holds the company standing" and "who understands how to double click" are enormous smells and IMO make quite clear what's happening here.
I don't have ready-made solutions unfortunately, but I do wish we would look further than "it's the suits". It's a systemic, human problem that I believe is a natural result of operating under informational constraints and, very human, cognitive biases on all sides.
Bell Labs is an outlier in basically every aspect. Mr Kernighan lists stability of the environment with regard to funding, structure, mission as well as technical competence of the management as main drivers of the culture. This is just not the reality in companies that look for financial results on a quarterly basis and where the executives are MBA types.
If one of the most successful engineering organizations in history attributes part of its success to capable management, that undermines simplistic narratives where management is inherently the problem and engineers would naturally thrive if left alone.
If anything, the Bell Labs example supports the idea that exceptional outcomes require both strong technical talent and strong management working together.
Not saying the "MBAs" are helping the situation, but the hero developers and their resume driven development practices aren't exactly angels either.
Capable is the loadbearing word, the directors were all PhDs in math, science and engineering fields.
I dont subscribe to the strawman argument that engineers would naturally strive on their own, but neither does simply any form of management automatically add value.
I agree also that hero type devs are an indicator of problems
>>I'm not sure how we solve this, other than having management come from engineering.
Given the whole point of management is to work to ensure their own survival and growth, it would in their interest to kill genuine competition when its coming up.
Who wants to raise their new competition and lose to them, no one!
Hikaru Nakamura had a good comment on famous chess moves like this: All the famous chess moves are famous sacrifices, genius saves, and last minute checkmates, but when you plug the games into stockfish you find out that these scenarios only happened cause the players made a series of awful moves early on in the game
My favorite is how elegant solutions often look simple in retrospect. So if you noodle on a problem for a while and then come up with a clever solution: once you explain it to someone they'll be like, "yeah, of course."
Meanwhile the guy next to you that overcomplicates the problem ends up getting kudos for building something so difficult :D
I am reminded of this course in university where there was a written assignment with a minimum page count. Even at that time, I remember thinking: "If I am able to express everything necessary in just one page, that should give me the absolute top grade!".
I feel like AI coding is accelerating everyone's work toward greater solution complexity and I think it's pushing people to build defenses and be more averse to someone else's complexity rather than being impressed by it. Bigco's are probably well behind the curve on this and are still impressed by complexity, but for people on the receiving end of AI stuff either directly via your own hand or indirectly via others, it seems like complexity is not as impressive as it once was.
Yeap. Managers perceive complexity by how personally confused they are. I'm late in my career, and I'm realizing I wasted so many man years trying to make code clean, user friendly, and maintainable, when that code was never read by another person again and forgotten 15 minutes after it was released, then used for years. This is why I think AI is coming for our jobs much sooner than many people think: clean code, separation of concerns, maintainability, etc, all the things we spend the most time on, have never actually been valued. "Good enough" is fine, and keeps management happy. And, if something does pop up, AI can patch it, even if with spaghetti, just like fucking that ass at work.
There was a thread recently on HN about Claude Shannon and how his papers were filled with clear descriptions and explanations. Then someone commented how they had found an elegant solution to a problem that either could be described shortly and beautifully so that a high schooler could understand or take the long and tedious path of a convoluted explanation.
The director then clearly advised that they should use the complicated way because that's how you get published: not because you're clever, but because your solutions sound complicated.
It resonates perfectly with your comment and it's an unfortunate reality that most people don't bother for beautiful solutions and praise complicated processes. That's how we neded up with bureaucracy, probably :D
On the other hand, I don't help people with their computer problems anymore because I've found that the more difficult the problem, the longer it takes me to rescue their data or whatever the less impressed they are. The more miraculous the save the more likely they are to tell me the story about their nephew who solved a trivial issue instantly as if to point out that I didn't.
The passage that comes to mind for me whenever this idea comes up, from the Brett version of the Holmes story "The Dancing Men":
H: So, Watson.
W: Hmm.
H: You do not propose to invest in South African securities?
W: How on earth do you know that?
H: Now, confess, you are utterly taken aback.
W: I am!
H: I should make you sign a paper to that effect.
W: Why?
H: Because in a few minutes you will say it is all so absurdly simple.
W: I should say nothing of the kind!
H: You see, my dear Watson, it is not really difficult to construct a series of inferences, each dependent upon its predecessor and each simple in itself. If, after doing so, one simply knocks out the central inferences and presents one's audience with the starting point and the conclusion, one may produce a startling, though possibly a meretricious, effect.
H: I can tell by an inspection of the groove between your left forefinger and thumb, that you have decided not to invest your small capital in the gold fields.
W: I can see no connection.
H: Very likely not; but I can quickly give you a close connection.
H: Here are the missing links in the very simple chain: You had chalk between your forefinger and thumb when you returned from the club last night. You put chalk there when you play billiards, to ease the cue. You never play billiards except with Thurston. Now, Thurston, you told me, four weeks ago, had an option on some South African security which expired in a month, and which he desired you to share with him. Your checkbook is locked in my drawer, and you have not asked for the key. So, you do not propose to invest your money in that manner.
W: How absurdly simple!
H: Quite so. Every problem is absurdly simple when it is explained to you.
Why on earth would you quote a TV version of a book, when the book is readily available to be cited?
⁂
“So, Watson,” said he, suddenly, “you do not propose to invest in South African securities?”
I gave a start of astonishment. Accustomed as I was to Holmes's curious faculties, this sudden intrusion into my most intimate thoughts was utterly inexplicable.
“How on earth do you know that?” I asked.
He wheeled round upon his stool, with a steaming test-tube in his hand and a gleam of amusement in his deep-set eyes.
“Now, Watson, confess yourself utterly taken aback,” said he.
“I am.”
“I ought to make you sign a paper to that effect.”
“Why?”
“Because in five minutes you will say that it is all so absurdly simple.”
“I am sure that I shall say nothing of the kind.”
“You see, my dear Watson”—he propped his test-tube in the rack and began to lecture with the air of a professor addressing his class—“it is not really difficult to construct a series of inferences, each dependent upon its predecessor and each simple in itself. If, after doing so, one simply knocks out all the central inferences and presents one's audience with the starting-point and the conclusion, one may produce a startling, though possibly a meretricious, effect. Now, it was not really difficult, by an inspection of the groove between your left forefinger and thumb, to feel sure that you did NOT propose to invest your small capital in the goldfields.”
“I see no connection.”
“Very likely not; but I can quickly show you a close connection. Here are the missing links of the very simple chain: 1. You had chalk between your left finger and thumb when you returned from the club last night. 2. You put chalk there when you play billiards to steady the cue. 3. You never play billiards except with Thurston. 4. You told me four weeks ago that Thurston had an option on some South African property which would expire in a month, and which he desired you to share with him. 5. Your cheque-book is locked in my drawer, and you have not asked for the key. 6. You do not propose to invest your money in this manner.”
“How absurdly simple!” I cried.
“Quite so!” said he, a little nettled. “Every problem becomes very childish when once it is explained to you. […]”
Every company where I've worked at as a SWE openly rewarded "engineering complexity" as a criteria for getting promoted, which I've always found to be absurd because complexity can always be manufactured (both of the problem and the solution).
One of those days, however, you come up with another of your elegant, simple solutions and it actually replaces a 150K LoC monstrosity with either a 1K script or, even better, with nothing as a simple shift in perspective or process obsoletes it completely.
In the long run, IME, you'll be recognized either by management or your peers if you keep doing that over and over again.
My favorite is how people will yell at you about how elegance doesn't matter, that they "just care that it works", and "keep it simple". I'm certain all the sayings repeated in industry are metastasized variants of actually good practices repeated by those who can't be bothered to understand what they mean.
And of course that's true. We push for speed, absent of direction, while praising velocity. To be honest, at this point I'm disappointed the engineers gave up and just started becoming business people.
I had this problem at a previous job. I spent almost all of my time taking care of the behind the scenes administrative work (scheduling meetings, making sure that people had the information they needed to come into the meetings prepared, etc.). However, when performance review came around I was told that the only thing that they cared about was that I hadn't completed many story points because I was too busy keeping things from falling apart.
So I stopped doing all the administrative work and focused on just completing story points. A week or two later my manager asks the team "how come all of our meetings are falling apart now? We get into a meeting and no one knows what's going on."
The headline is only half true. A more accurate rendition might be "Transactions don't happen because you fixed a problem that never occurred."
That illustrates the converse. People will absolutely try to avoid future problems if they are the ones that bear the consequences for them. Use birth control (or put your kid on it) so you don't have to raise another child for 20+ years. Don't hang out with that volatile "friend" who always seems to be having another crisis. Fix your roof so you don't lose the house. Don't go into debt because you're the one who will be paying interest on it.
But almost by definition, bearing the consequences of your own decisions implies that there's no transaction.
It's interesting and fitting that the article begins with a discussion of Toyota. People buy Toyotas because they want to avoid problems. The biggest selling point is reliability; a Toyota's value prop is that you can keep it for 20 years and you won't have unexpected things go wrong with it. Toyota has managed to turn this into a sales driver because they appeal to the self-interest of a buyer who will be living with the car for 10-20 years. There are thousands of individual decisions that Toyota makes to avoid problems with their cars, starting with a very conservative aversion to new technology or anything that is engineeringly risky. Each individual one is invisible to the customer, and often comes with significant costs in isolation. But because their sales driver is "be reliable at all costs" and they've ingrained that into the culture, they've built an organization that is willing to make these feature trade-offs for reliability.
Also, an interesting corollary is "Oftentimes, a good life is lived with few transactions."
I began migrating from network/hardware/IT work and into marketing after nearly 2 years of heavy lifting getting ready for Y2K. In the end, "nothing happened," so all that time and money was wasted, according to nearly every company I worked with. Even had one demand a full refund. I agreed as long as I could revert all the work that I had done. They agreed, and the next day after that their entire system collapsed.
I couldn't even get my own dad to pay for network support for his company since he would never pay my rate for anyone no matter what. After 2 other people failed to solve his problem I fixed it in 15 minutes and then he "really" didn't want to pay because it only took 15 minutes.
I was very good at what I did but got no appreciation for keeping things from breaking, only for fixing things after they broke. Marketing paid better, and I could point at real world numbers daily and justify my pay. I don't like it anywhere near as much, but at least it gets more respect than any other IT work I did.
If I move out a bit of my circle, where people do all kinds of work, I'd say that there's generally a stigma on the "IT" workers. Moreover the stigma is there even within the IT company/industry itself, where sales, marketing, non-engineering parts of management, and other similar types of supporting roles also look down on the engineering. And this unfortunately includes family members too. I learned that people are mostly envy but when you're surrounded by many it can become overwhelming - numerous times I heard phrases like "oh, you're bricklayers of a modern age" ... like wtf
"This is what my regular customers pay me. If I hired one of my friends or relatives I see it as my duty to pay them at least what they are worth, this is the way you raised me."
I believe this to be true btw. If someone is really your friend, you want them to do well and that means you pay what they usually get or you don't bother them and get someone else.
Idk I joined the field only like 5 years ago. Prior to knowing about programming I had a lot of respect for programmers and tech as it was this magic world to me. After having joined the field the magic is completely gone and I don't want to talk to programmers irl anymore because of how many insufferable people I've met so far.
The parts where I've got credit are the simple things, fix printer, fix computer issue A,B and C or small apps like an ad free Sudoku for android which I have built for my friends.
The parts I don't get credit for are the parts I get paid for. But I can see that in many industries, as soon as money is involved people are less thankful because they expect you to fulfill your part of the contract.
Generally people not knowing shit about tech think that devs sit in HO all day working only 30 minutes. AI didn't help that image.
Ian Rush said it best: "It's best being a striker. Miss five, score the winner, you're a hero. The goalkeeper plays a blinder, lets one in, and he's a villain."
Every place I've worked rewards the firefighter over the person who made sure nothing ever caught fire. And the worst part is the math is obvious to everyone except the people who set the incentives.
How would you set the incentives, though? Almost by definition, it's hard to reward things that aren't visible.
Note that there is also the flip side of the coin, people who spend all their time worrying about things that never happen, so it's not like you can just reward a defensive attitude things are more complicated than that.
The equivalent of 'days since last injury' bonuses is the first method that comes to mind, until you consider that this would mean people would be more likely to hide things going wrong.
So then many things are to rely on executive culture, and an executive who will walk the line and get their info from people at the bottom is like a unicorn. That won't scale, but it does work if you do have such an executive. Naturally they would need a basic understanding of how supply is created in their firm.
Yet there is something. Toyota Hiluxes and Honda Super Cubs got popular due to maintenance ease. AK-47s. Miele vacuums. Older Thinkpads.
What measures would make the human equivalent visible?
One basic example is not counting bugs as points in your ticket tracker. At my last job I had coworkers whose velocity was almost double everyone else’s but it was because they kept deploying and then fixing their own bugs.
I dont have an answer and you are mostly correct. I received some advice based on this that made sense which was to pick the roles in your career that naturally made it easier. Sales, PM, Dev etc and not support, Devops, escalation management, CSM etc.
This is how people get promoted at work. They break something, it escalates and gets visibility, emails sent to executives. Now, they 'fix' it, many thank yous from everyone for a job well done. Another version of this is delay the work you are supposed to do a long time back and let it gain visibility. Executives are blind, they can't see work being done by people who take ownership and get shit done before it becomes a problem. However, executive will remember the name of the person who breaks shit and 'saves' the day.
The sociopathic backhand. Someone breaks it, blames you, drags your name through the mud exacerbating the issue, goes to "fix it" yet makes it worse because of their incompetence. They continue to cuss you out for it all while licking the arse of the executive. "No, we better not let doublerabbit touch that", "I don't think they're a team player". All while it was my infrastructure.
My time in IT has really oscillated between these two extremes:
- "Everything around is working just fine. What are we paying IT for?"
- "Everything is broken. What are we even paying IT for?"
Personally, I strive for the former rather than the latter; I like to say "If I do my job right, you never know I'm here." But that's what got me let go.
(and for karma's sake, I keep in touch with folks at the old company; it's an absolute crapshow. So I got that going for me; which is nice)
We all learned this back in first grade. The kids that behaved in class and did their homework did not command most of the teacher's time and effort. It was the problem children who refused to follow the rules and needed constant praise for every bit of actual effort that they put into their studies; that got the teacher's attention.
This brings to mind Neil Rickert's "The Parable of the Two Programmers", which was published in the ACM SIG Software Engineering newsletter, January 1985.
Counterpoint: I absolutely give credit to Sonic for being a great ISP and recommend them to everyone. I got my parents to switch when Sonic finally rolled out to their neighborhood.
If online comments are anything to go by, I'm not alone.
If you're in the Bay Area and you can get a Sonic fiber connection, I would highly recommend them over AT&T/Comcast/etc.
This is where the industry made and continues to make mistakes wrt autonomous driving.
They should be able to quantitatively say how many crashes were reduced, avoided and spotted. The autonomous safety system should be running all the time and it should detect not only issues with primary vehicle but it should also catalog issues it sees in other vehicles in its vicinity.
We shouldn't have gotten AD before we got automated crash avoidance.
They stole that from the Tao Te Ching, chapter 17:
When the Master governs, the people
are hardly aware that he exists.
Next best is a leader who is loved.
Next, one who is feared.
The worst is one who is despised.
If you don’t trust the people,
you make them untrustworthy.
The Master doesn’t talk, he acts.
When his work is done,
the people say, “Amazing:
we did it, all by ourselves!”
You'll see capability traps everywhere once you learn about them.
Sterman, Repenning and other collaborators wrote several papers after this one. All fascinating and almost entirely depressing.
Especially since MIT's Sloan school, where system dynamics first became a discipline, is just around the bend from Harvard Business school, where system dynamics first became ignored.
One thing I don't get about the concept of capability traps is why is it expected that a company which is good at one thing would be capable at the new thing? What exactly makes a capability trap a trap?
The trap is that you can't get better without first getting worse. You can't get out of the destructive cycle of production pressure and decaying productivity without removing the pressure. Many managers expect, or at least behave as if they expect, improvement to be monotonic and costless.
When you are at capacity and in a degraded state, you have no additional headroom to get out of that state. Why wounds won't heal, or the poor stay poor.
I think because it’s a negative feedback cycle. So once it starts it’s hard to go back, the deeper you are the harder. A trap is something that’s easy to get into and hard to get out of.
I'm looking for some data -- if anyone has it -- on the fraction of companies that are led (CEO) by a technical person, over the years/decades. I have the (anecdotal) impression that this fraction has been falling (stories like Boeing), but it would be cool to support or refute this with hard data. Anyone know where to find/assemble something like this? Also, if this trend is true, then why?
I don't have the data, but I think a good case study is the company MITRE.
Originally it was engineers from the top down, but over the last 15-20 years those leaders with engineering backgrounds have retired and been replaced by non-engineer MBA's. And the more I look around, the more I see that as a common trope is the US.
There is something I saw on a reddit post of all places, about how every manager who doesn't predict a baseline of "3 annoying problems every month, 1 awful problem every 3 months" is essentially a bad manager. The reasoning being that, if your number of problems is under that threshold, then someone is doing a 'good job'.
My favorite example is the introduction of speed limit on some accident-ridden stretch of the Autobahn north of Berlin. After introducing the speed limit, the accident numbers went down dramatically. What did the local administration decide? Remove the speed limit again -- cause there were no accidents anymore!
When Covid started, our local government was very clear from the start in saying "If people think we have over reacted, then that means we have done a good job."
Article published in the Summer 2001 edition of California Management Review, yet it never mentioned Y2K, the first thing I thought of when I read the line "fixing problems that never happened". Perhaps it was actually written in 1999 and took a while to get published, because otherwise that seems a very strange omission. The Y2K problem was very much over-hyped by the American news media at the time (no, at no point would airplanes have been falling out of the sky — I literally heard someone say that would happen once — even if no effort had been put into fixing the bug).
But in recent years I have seen people (elsewhere, not on HN) claim that Y2K was a big nothingburger, and all the money spent on fixing the bug was wasted. No, that's not true either. All the money spent on fixing the bug was why it turned into a big nothingburger. Sure, some of that money was wasted, by executives who wanted an "official" Y2K-certified certificate, issued by a consulting firm that had nothing "official" about it except their own say-so. And so they spent $2 million learning what their own employees could have told them for $2,000. THAT money was wasted. But a lot of banks were running old COBOL code that used 2-digit years, and needed to be fixed. The fact that in January 2000, everyone's bank interest was still calculated correctly, and not calculated as if it was January 1900? THAT was entirely due to the vast amounts of money spent paying old COBOL coders to come out of retirement and fix the 2-digit years.
The lesson I learned from that is that it's possible for a problem to be overhyped, even massively overhyped, and yet still be a serious problem. The other lesson I should have learned is that people rarely get credit (I won't go so far as the article authors and say "nobody ever gets credit") for fixing problems that never happened.
The problem is that a lot of people have a very binary view on life. Either something is a complete success or a complete waste of money, rarely do we accept that most projects fall somewhere in the middle.
The binary view is mostly true, unless it's for events or problems they are themselves familiar with. There is a term for this, but can't for the life of me remember it: People think the problems they are dealing with are infinitely more nuanced, complex and unique than the problems other people are dealing with.
And even worse, they don't think probability is a thing. If something happens, it was certain to happen and we just failed to predict it correctly.
So when someone predicts something will happen with a 90% probability, and then the 10% chances happens and the predicted event does not happen, people will talk about what a bad prediction that was and how they were clearly wrong.
It's the same logic that causes people to say vaccines don't work because they don't stop a disease with 100% effectiveness, or that there is no point to wear a seatbelt because people still die while wearing one.
My issue with this version of explaining the lack of severity of Y2K is that there were lots of countries that were being derided for not taking the issue seriously but did not seem to suffer any ill effects.
It's also possible that in some places there were a few issues, but people looked at bills for 100 years of electrical service and said "Yeah right," and fixed the now-easier-to-find code that still used 2-digit dates. If that only happened a few times, the extra work involved in working out the January bill by hand (or waiting until February then billing for 2 months) wouldn't cause too many issues in the economy, and anyone looking in from outside wouldn't even realize there had been an issue. If it happened everywhere the economic impact would be more noticeable from outside.
>no, at no point would airplanes have been falling out of the sky
The assertion may have been unfounded, but I think it's just as unreasonable to assert the opposite. Bugs have cascading effects and in a sufficiently complex piece of software they can create chaos with unpredictable outcomes.
The one case I'm aware of where a software glitch did cause a plane crash, there was pilot error compounding the problem. Air France flight 447 was an Airbus A330 flying from France to Brazil, and while high over the Atlantic, the software recorded inconsistent data in its airspeed measurements. (The official crash analysis team concluded that the inconsistent data was likely due to ice crystals blocking the pitot tubes on the plane). The inconsistent data made the autopilot disengage. Pilot error then caused a stall. One pilot then tried the correct move to recover from a stall, pushing forward on the stick to nose down and regain speed. The other pilot was pulling up on the stick to stop the dive, not realizing that that's exactly the wrong thing to do in a stall (or more likely forgetting his training due to panic; he had a lot less experience). The flight software, receiving inconsistent inputs from both controls, averaged the inputs, resulting in zero change in pitch. (It also sounded the "Dual Input" alarm, but the pilots were too preoccupied with their own controls to figure out what that meant at first, and by the time they figured out what was going on it was too late to recover before the plane hit the water).
https://news.ycombinator.com/item?id=4224707 has some discussion of the events, including the fact that the control design (where each pilot has an independent stick) was part of the problem. On a design like Boeing uses where both sets of controls move together, the experienced pilot would have noticed the less-experienced pilot pulling up on the stick because his own stick would be moving, and he would have said "No, nose down." And if they had nosed down to recover speed while still high enough in the air, they almost certainly could have regained control of the plane and saved 228 lives (including their own).
So in retrospect, I think my first sentence was wrong. The software did not glitch, it did exactly what it was supposed to do. It was pilot error that caused the initial stall, and multiple pilot errors that caused the failure to recover from the stall.
There may be examples of software error that has caused planes to fall out of the sky, but I don't know of any. The only plane crashes whose cause I know were due to hardware failure or pilot error, usually a combination of the two.
I think your conclusion is upside down. Air safety is based on the "Swiss cheese" model. Multiple layers of safety nets are in place to compensate for issues in one layer. In particular, technical safeguards are there to prevent disasters if the human in the loop makes a mistake which will eventually happen. Any weakening of any technical safeguard makes the system less safe. No matter if the human ultimately made a mistake -- the technical system failing contributed to the accident just as much.
Y2K is especially interesting because the fact that the year 2000 would one day occur was entirely foreseeable, and no less probable in 1990 than in 1999. I can hardly think of anything with closer to 100% probability of happening.
To be fair, there was a non-zero chance that society could have ended (or your company, or the tech became obsolete) before 2000, which would be higher the earlier before 2000 you were.
The tech being obsolete is why Y2K was a smaller problem than it would have been otherwise. Most places were no longer running much COBOL code. But banks are famously slow to upgrade their tech, and for good reason much of the time, so most of the world's remaining COBOL code (and other code too, COBOL is just what I'm most familiar with, not that I'm all that familiar with it) was in banks and other financial institutions.
my first thought too. I've met a few people who assert that Y2K was a complete waste of money.
I earned my first house deposit helping the team fixing the water and gas company in Wales, UK. Their entire system was running off a set of COBOL programs on a mainframe, none of which had been properly documented over the years, and the whole thing used 2-digit dates. It would have caused actual deaths if not fixed; everything would have shut down, and no water and no heating in a British winter is potentially lethal. And then it would have sent everyone in Wales a bill for 100 years of water and gas.
They were bribing retired software devs to come out of retirement with huge stacks of money, because that was cheaper than training new COBOL devs and getting them familiar with the spaghetti system.
It worked, no-one died, life went on. So obviously it was all fake rolls eyes
I'm curious why things would have shut down when the system thought it was 1900. What part of the logic had the effect of "shut the system down if current date is less than (X date)?" (If you can remember the code 25+ years later, that is).
I recognise almost every aspect of this document - it's exactly what's so intractable about the software business. This is why I think you do need to do some programming every now and again no matter what your level is because otherwise you cannot see what's happening and you'll be tempted into the "lazy developer" attribution.
Its really hard to measure effectiveness, problem becomes even harder when a non-engineering person has the job to measure effectiveness of a engineering person.
On other hand. for software engineering some of the signals that can be used to measure such a management itself can be
1. On call requirement, outages and team burnout - A well written software should not require on-calls from the dev team
2. Ask them about the "concrete" roadmap for next 6 months to a year - Absence of concrete items is a bad sign
> Petrov underwent intense questioning by his superiors about his judgment. Initially, he was praised for his decision.[2] Colonel-general Yuri Votintsev, the then-commander of the Soviet Air Defense's Missile Defense Units, who was the first to hear Petrov's report of the incident (and the first to reveal it to the public in the 1990s), states that Petrov's "correct actions" were "duly noted".[2] Petrov himself states he was initially praised by Votintsev and promised a reward,[2][22] but recalls that he was also reprimanded for improper filing of paperwork because he had not described the incident in the war diary.[22][23]
> Petrov has said that he was neither rewarded nor punished for his actions.[24] According to Petrov, he received no reward because the incident and other bugs found in the missile detection system embarrassed his superiors and the scientists who were responsible for it, so that if he had been officially rewarded, they would have had to be punished.[2][24][22][23] He was reassigned to a less sensitive post,[23] took early retirement (although he emphasized that he was not "forced out" of the army),[22] and suffered a nervous breakdown.[23]
The same article points out that he received at least £26k in awards. It could be argued that the reward isn't proportional to the magnitude of his actions, but it exists.
I remember finding a comment in the first codebase I ever worked on professionally in my first ever job.
It read "This fixes a bug that hasn't happened yet".
It seemed really smart at first, but later I learned that the developer that added that code also had a pattern of appending spaces to the start and end of user input and comparing the length to 2 to determine whether the value was empty or not...
So I'm fairly sure "that hasn't happened yet" was probably more a case of "that I personally haven't introduced unnecessarily yet" :)
Pendatically speaking, people do get credit for fixing problems that never happened.
E.g. if the problems are quantifiable and there's a record, like dropping homicides from 100 per year to 20 per year in a city. Those extra homicides "didn't happen", but the improvement is understood.
For an one-off problem, it depends on how clear the path to the problem is. An electrician doing an inspection and noticing and fixing big electrical issues in the installation, would be appreciated, even if the accidents didn't happen.
> Those extra homicides "didn't happen", but the improvement is understood.
People are gonna criticize by saying "see? it was an overreaction to the problem, since there's not been many homicides at all!", when in fact the homicides were prevented by fixing the original problem. Same way with the electrician: "how much are you gonna charge again? And you're charging for a fix to a problem that didn't happen yet? Nah, I'll call you when the problem happens".
> An electrician doing an inspection and noticing and fixing big electrical issues in the installation, would be appreciated, even if the accidents didn't happen.
This is especially nice in the age of AI. I (the graybeard senior developer) does all the risky refactoring. I can take a performance issue and turn it into six regressions in half a day ($100). Then everyone is impressed when I let Opus fix these regressions in 20 minutes and $2 worth of AI.
No one notices when you cut 20% of some expensive process but cause no regressions.
Here is another take in s different context.I had a very bad manger who was credited earlier with causing several companies to go out of business while he was taking advantage of having worked for a FAANG company. He worked there for less than 6 months as a business development dude when the that big shot company was new in that market.After he joined us and when I was processing some payments I noticed that he was paying some people for 2 months ahead of their starting dates, it was some $20k.I notified him, and he corrected the date,then I was told by others in the department that my problem was that I should not 'interrupt the enemy when he is making a mistake'. Eventually, the company got rid of him but it was after a very serious damage.
In other words, it’s not just a tool problem, any more than it’s a human
resources problem or a leadership problem. Instead it is a systemic problem [...]
Shades of an LLMism, a bit padded, a quarter of a century ago. These days someone could easily give it a stink-eye. I'm sure that training has ingested this along with countless similar examples.
That should really be a cautionary tale for everybody accusing everyone of LLM manufacturing texts. Many people write like that. The self-censoring nowadays to try to avoid sounding like an LLM is really sth we need to grow out of.
Then we soon see non-technical people start leading the same, pushing for some people to be recognized for this every sprint. Meaningless recognition starts coming in. The process fades out in just a couple weeks.
The problem is there are these people in the mix, often leading, who do not understand.
I'd guess a lot of people here consider this "reality" at this point. Has anyone come up with a response—not a fix for the company or leaders behaving this way, but a response for their own path?
Did you change from a quiet diligent one to manipulating and playing the game (now that you know the game)? Did you go from quiet and diligent to quiet and not diligent (why do good work when meh work does the trick)? Another path?
I just do my job to the best of my ability. I have to change jobs every couple years anyway to get proper pay bumps, so I don't really care what the higher ups think of me. The people near me are who I'll use as references and they generally know I'm great at what I do.
Chernobyl instantly comes to mind. The giant problem with that particular RBMK design was known to a few people, but nobody fixed it, and if they had... we wouldn't even know, since it would be lost to history.
This is the real problem with performance reviews in companies, which then feeds into opportunities, promotions and compensation. It's just a popularity contest. And this is particularly harmful to people who are neurodivergent, particularly if they're on the autism spectrum, because neurotypical people, who end up making all these decisions, view such people negatively for literally no reason.
You could spin up a team of 6 engineers and have them go away and try some greenfield project. They could come up back in 6 months having shipped nothing. Which of these descriptions fits the facts?
1. The team learned a lot and ultimately decided there was no product-market fit and decided it was best to reallocate resources elsewhere. The learnings from that project will help a whole bunch of other projects across the division; and
2. They failed to ship and get subpar performance ratings for having no impact.
The answer is... both. Or either. How you are treated will depend on how you are viewed by your management chain and that's a social function. We've all encountered people who never shut up about how hard their job is. Often they end up solving problems that they created, often by not listening to anyone that those problems would occur. And they get credit for it.
You could say to people who anticipate problems to stop because it gets you nowhere. Let people fail. If only it worked that way. Instead you'll get blamed for not seeing a problem someone else created because you're viewed as competent but you aren't liked through no fault of your own.
Google seems to be the posterchild for a company that briefly solved this problem and then forgot what made them successful. I am referring to Project aristotle [1], which ultimately determined that psychological safety was the key ingredient in a team's success.
Now amplify all of this with constant rounds of layoffs where the environment isn't just for pay bumps and opportunities but where the cost of failing is losing your income. What you've created is an environment where office politics is everything.
- Arnold bought a fleet of mobile hospitals that would have been perfect for covid response, but the next governor didn’t want to pay 1% the fleet cost per year to maintain it, so he scrapped it.
- Under Obama, SARS v1 was stopped by US health workers that Trump fired because it was a “bad deal”. In the absence of that team, we got SARS v2, which was renamed to COVID 19.
There’s also the related category of “never blamed for fixing problems poorly, creating even bigger problems”.
Thanks to 9/11, plane cockpits can now be locked from the inside. Now, we have examples of commercial passenger airline pilots locking the doors and committing mass-murder-suicide by plane crash.
For some reason, these stories don’t make the news.
We know that the squeaky wheel is the one that gets oiled, and we don't want the wheels to come off, so we need the wheel to squeak loud enough to be heard ;)
Making critical decisions without oversight is just as bad, or maybe worse.
If you frame it this way in a meeting, you will get the attention you want. Don't say I didn't warn you because that comes with a lot of scrutiny you might not want.
The title reminds me of an interesting ancient Chinese anecdote. And it is also a bit ironic that Toyota has gotten itself into some scandals recently (https://www.bbc.com/news/articles/c1wwj1p2wdyo).
King Wen of Wei asked Bian Que:
“Of you three brothers, all physicians, who is the finest in the healing art?”
Bian Que replied:
“My eldest brother is the finest; my second brother comes next; I, Bian Que, am the least of the three.”
King Wen said:
“May I hear why?”
Bian Que answered:
“My eldest brother sees illness in the spirit, before it has taken shape, and removes it unseen; therefore his name is known only within our household.
My second brother treats illness when it is but a hair’s breadth from appearing; therefore his name does not travel beyond our village lane.
As for me, Bian Que: I pierce the blood vessels, administer strong medicines, and cut open the flesh. Thus, by such visible acts, my name has spread among the lords.”
Can you share the source of the anecdote? I tried to find it in Zhuangzi and was not successful.
Best I can tell, it's originally from Heguanzi: https://ctext.org/he-guan-zi/shi-xian
Beautiful text. By the way, I can't thank enough the maintainers at ctext.org. What a beautiful work they do.
An ounce of prevention is worth a pound of cure.
How many euros or kilograms is it?
Three Troy ergs.
or a simpler version: "A stitch in time saves nine...but I charge by the stitch"
literally one of the main reasons america spends so much on healthcare
I've been in those companies where "struggling departments" ended up getting all the praises and raise in budgets the following quarter because of the heroic saves they did, and raising awareness on how important they are... For stuff they totally caused on themselves.
Meanwhile, my perfectly purring department was struggling to keep the lights on.
It's a serious problem in this industry due to the disconnect between non-technical management (who understands how to double click) and engineering (who holds the company standing).
<insert IBM story about IT department cost cuts>
I'm not sure how we solve this, other than having management come from engineering.
By building pain into the system. If your hands dealt with injury directly without sending pain signals up to your brain, you'd never change the behaviour that led to that harm or reconsider your priorities. Like it or not, sometimes the best thing for an organisation isn't to just fix every problem and prevent it from bubbling up; it needs to be treated like a learning opportunity for org leadership, which means sending the pain signals upward before just repairing it.
Building the right incentives around that can be tricky, those incentives need to ensure the highest levels of management aren't themselves disincentivising their directs & their departments from surfacing pain & problems - but it's also pretty common for people to mask those signals purely out of a well-intentioned desire to help. It's important to coach people on the idea that in large group sizes, it's more efficient to let certain kinds of problems play out and not be so reactive to them.
Too many companies ground their performance incentives & processes around oversimplified ideas that don't match the reality of human behaviour
+10,000%
Often, 'leaders' make mistakes and people below suffer the consequences. It is important to let these leaders deal with the pain caused by their decisions from their cluelessness about how things work.
The problem is it's systemic. Ultimately, pain needs to come from outside. As long as society rewards incompetence, we'll have incompetent organizations.
No it does not need to come from the outside. If you're an underfunded IT department and your network has an issue twice a week, you will get that funding. If you're heroically obscuring the fact that things are falling apart you won't. That means even if you could somehow, heroically fix it, it isn't perceived as such if nobody ever felt the problem and saw you fix it.
This is a pain signal. Some IT dude saying things are crap in every meeting is not.
> This is a pain signal. Some IT dude saying things are crap in every meeting is not.
More often than not it is some IT dude observing network crap-out once a month, performing analysis, noticing an upward trend and then saying in every meeting that things are crap and there will be issues twice a week in some time.
> If you're an underfunded IT department and your network has an issue twice a week, you will get that funding.
More often than not, if the IT department is already neglected they will not get that funding. Things will be delayed until the crap outs eventually actually happen twice a week and then some external heroic consultants will be hired to fix the issue underfunded IT department "could not".
IT doesnt control the funding so at that point its not an issue of awareness but a management decision to live with this problem and focus funding elsewhere
more often than not, many things in the business are on fire and underfunded at the same time. you can get recognition for your work without the problem being permanently solved the right way, and it may not result in more funding but peopel will think of you for new opportunities that pop up later as someone who is reliable.
if you dont think the recognition will happen and youre just burning out solving these problems then stop solving them. new problem pops up thats outside your job description, its not your problem. generally though if youre working for someone like that anything you do is a lose-lose
How did you make the illogical leap to “could not”?
Repeatedly requesting time/budget to fix an ongoing issue is a requirement of any half-decent manager. If they’re reporting issues then just smiling blankly when asked “what can we do about it?” they’ve failed their basic job duty.
The problem is sometimes management knows who the heroes are and so by not fixing things they know you are not competent. Thus letting things bubble up isn't always a good plan. It is really hard when you are on the bottom to know which case things are.
The problem is that management witnesses the pain, but the response isn't to adjust behavior, it is then to punish the limb where the pain originated from. The reason that people pull heroics is also because the organization isn't healthy, and cannot reflect on its actions. Papering over organizational flaws is a symptom of a larger, often unseen problem. If it was healthy, someone would have already said, "hey, I think we need to work on this networking component" and it would have been looked at.
Pain propagation, to use the corpus metaphor isn't enough.
The Dunning Kruger effect suggests that the people who caused the pain are also those least likely to feel it.
That's why you change it to make the pain work. This does need CEO-level cooperation to implement, but i think it is possible.
CEO’s are generally the ones causing the pain
Dunning Kruger's paper didn't even show the Dunning Kruger effect everyone loves.
In the original DK experiment they asked students after they had taken a test that where would their score end up compared to their classmates (which they had no knowledge of). Obviously they picked scores around the middle*
Which resulted in top students 'undervaluating themselves' and bottom students 'overestimating themselves'. Or under/overvaluating a random future variable that they don't have knowledge of, at least.
The original DK paper actually shows a positive correlation between the guesses and the test results: students are generally aware how they are among their peers, and smarter students guessed higher than studetns with less time to study on their hand.
This being said, the 'DK effect' is something people talk about, and it might exist, and it might be perceived by people. It's just that the original DK paper does not support it.
* another lesser talked problem with the DK paper is that people don't actually believe the answers they give, because the question is nonsensical.
If someone just takes a test, they won't think that "I'm sure I'll end up the 24% this time". Even if they are forced to anwser this question, even then they won't believe it, because that's not how random and future work. People are generally aware of about where they will perform (with positive correlation, in fact the original DK paper shows it) but they are not aware of results of specific, random future events, and they are not claiming that they know the results of specific, random future events, or believe it in their hearts.
DK paper tries to frame them as they were actually believing this, but they are not.
More to read at: https://en.wikipedia.org/wiki/Dunning-Kruger_effect
Can't we just get to the pop culture version of the DK effect by deduction?
It seems reasonable to assume that for some group of intellects they are not smart enough to know how not smart they are. There is no definite boundary where this effect is either on or off, therefore there are probably some gradations to this awareness as you climb up the intelligence ladder.
Another way of putting it: if dumb people had more insight they would cease to be dumb.
Honestly? I’m just super direct with the exec team now after trying to do this dance. Obviously this is not allowed at every company, I’m just lucky to be at a place where the company culture allows for this.
I’ll ask for something preventative or that otherwise hardens our systems. They ask “is it a need?” and I’ll say something like “we can function without, but that means we have a 5-10% chance in the next 6mo of having a major failure and embarrassing ourselves in front of a live audience in the thousands as well as our client.” They then decide how much that risk is worth to them, and whatever they decide is kind of out of my hands at that point. If the thing I warned them of comes because they didn’t pay for it, I can point to the receipts (though I’ve never had to, we’re small enough people remember those conversations).
60% of the time they just get what I need maybe? But ultimately it’s about CYA. Tell them what’s up, tell them what the solution is, tell them what the consequences are if they don’t do the solution, and make them decide.
Again this obviously depends on company culture and structure, but I can’t imagine on the only person who can do this!
An example that not all companies are run by idiots. The job market is not a healthy market though, where its more important to know ppl then to be great at some skill. But if leadership sucks just leave if you can, that will fix the problem.
Totally. I am pretty lucky to be where I’m at.
> By building pain into the system. If your hands dealt with injury directly without sending pain signals up to your brain, you'd never change the behaviour that led to that harm or reconsider your priorities.
I don't think that's it. Emergent problems require attention and action from leadership, who in turn can make the problem visible to higher ups. This creates signal, and positive feedback when the problem is fixed or mitigated.
If the problem doesn't exist to begin with, there is no signal. Managers don't get to show their fast-acting skills, and there are no heroics to speak of.
So ultimately poorly maintained and managed projects who deliver fixes for problems of their own doing create a perverse incentive, whereas no one is lauded or promoted for doing normal day-to-day things.
Well I think it is even more complex. If you're a plumber in a rotten system of pipes the whole company depends on, you can fix issues day in and day out, without speaking a word and they will notice everything is a bit unreliable and thus you do a bad job. You could do the exact same work, but make a big thing about every major fix, warn people a week ahead, give them the feeling the company depends on it and then do the exact same work and tell them how you fixed it. Suddenly you did a good job, despite you literally doing the exact same thing with your hands.
The difference is how it was communicated. Most non-Tech/non-infrastructure-people got no clue about these things. If they know you're battling the demons of plumbing on their behalf they will thank you, if you're the weird guy that has smeared dirt in the face and is seen once a week while the plumbing fails ever so often, guess what.
That means even if the problems and their fixes remain the same, the communication around them really matters. Tech people can be extremely bad with this. And if we're talking IT it is really the plumbing that holds the company together.
> If your hands dealt with injury directly without sending pain signals up to your brain, you'd never change the behaviour that led to that harm or reconsider your priorities.
At some point in one's early single-digit they learn that touching hot stuff hurts. They start to avoid stuff that they know is hot, but still come in contact with hot stuff accidentally. Later they learn techniques minimizing probability of touching hot stuff even by accident. By the time one reaches twenty or so, the only times a person burns themselves is really by being way too reckless.
> Like it or not, sometimes the best thing for an organization isn't to just fix every problem and prevent it from bubbling up; it needs to be treated like a learning opportunity for org leadership, which means sending the pain signals upward before just repairing it.
Should we accept that management as a whole is in general more clueless than your average teenager? The "learning opportunity" should, ideally, happen exactly once, realistically once in a very rare while.
> It's important to coach people on the idea that in large group sizes, it's more efficient to let certain kinds of problems play out and not be so reactive to them.
You are conflating two things here, I guess. Yes, some "problems" are not worth to be fixed proactively or at all, but that has very little to do with group sizes, it's a "simple" cost-benefit tradeoff. As groups grow the left hands tend to become increasingly unaware of what the right is doing and that is the primary reason why we have management class in the first place.
The problem OP raises is attention span of the metaphorical gold fish in the management layers. Even if a department does everything in their power to communicate impending problems, do risk weighed cost-benefit analyses, get proactive treatments pre-approved by higher management, the same higher management forgets the risks and costs savings once they have been mitigated, effectively incentivizing firefighting. Some teams gradually fall into eternal firefighting and burn out, others start manufacturing fires to get rewarded. The biggest problem is that it is nearly impossible to tell the two apart.
Unlike a teenage child, management has the unfortunate effect of being made up of people who can leave the company, forget past experience, etc. So you do kind of have to treat them like a child who needs continuous feedback and signals.
For a more broad example than IT cost center stuff, you can look at how some large companies go through cycles of arrogance with their customer bases, launch a product that fails, and then are humbled enough to try and pivot and earn good will back. Microsoft is always somewhere in this cycle for instance. The organization can never really learn this lesson permanently and will "regress" from time to time based on financial pressure or greed or some other impulse.
In my 35+ years in IT, the "hero attitude" was the one in the top three I most hated traits in a person working with or for me. And talking about traits, I considered crucial to always have in my teams a "saboteur" engineer - the one who thoght, found, come up with all the way we could break a design, service, infra components, app, etc., when all the others were designing or operating for perfect or normal conditions.
Genuine heroism - a willingness to step up when needed - isn't a bad thing in itself. But /needing/ a hero just to function means the system is fundamentally broken. Maybe it's a bad process, maybe it's understaffing, maybe it's neglected maintenance, maybe it's a lack of contingency planning. But there's no reason everyone can't go home at 5 PM every night and still get things done.
Love the "saboteur" approach. I honestly want to be one my own career in IT, but as you have rightfully conveyed, "hero attitude" is what gets you visibility!
At a previous job the CEO/owner had the idea that you'd get some percentage of any cost saving your could find as a bonus. Something like 20% of the savings for the first year.
My colleague in the IT department had one idea, replace our commercial certificates with Let's Encrypt and drop the EV requirement. In total he'd stand to get a bonus of a little over €2000. He never got the money, because things like that was part of his job apparently.
Wow, that's pretty silly. 2000 Euros is almost nothing in the grand scheme of things, and it would have showed that the policy was sincere.
Even dumber, they've now got a disgruntled employee, and everyone around them knows they were cheated by the company.
If the policy is wrong and needs to be more specific, pay it out this time and change the policy. Don't just break your word.
The policy they think they've implemented is stupid. "Save money in someone else's department" is just going to create a ton of anger as people rush to step on each other's toes, and then those people have to constantly re-justify all the decisions they've made.
It's absolutely brain-dead.
Freefall has a discussion of that (mini-arc starting here: https://freefallmirror.com/ff4300/fv04289.htm).
> a disgruntled employee
With access to the SSL certificates.
I think a good place to start is tracking all the proactive things being done and reporting them. At least then maybe someone will see why it’s quiet, because you’ve anticipated the problems and stopped them before they start.
When things come up with other teams, you’ll have a catalog of tasks that were done to show why you didn’t have the same issue. The work was done, just at a better time to avoid downtime.
> start is tracking all the proactive things being done and reporting them
Speaking from experience, this does nothing. If you're at a company that is okay with average performers, then absolutely, 100%, fix all the bugs in advance, make the system rock solid and stable, prevent downtime, be a good engineer.
If on the other hand if you're at a company where 10% of people must get stack ranked and PIP, or at a company where "meets expectations" actually means you're going to get the stick, and you're supposed to be "redefining" expectations every year ... then yeah, don't do anything preventative. The optics are better when you take the 3am on-call and fix the issue (that you secretly knew in the first place would happen some time in the future in your coworker's code, and already knew how to fix -- but don't actually fix it until it surfaces). Be the savior that the VPs praise in the next meeting, that's your insurance against the PIP.
They set the rules of the game, you just play the game. The rules were their choice. They could have chosen different rules.
I'm sorry about your experience.
Personally, I only rehire people from projects that went smoothly, not ones where I had to make the urgent phone call.
Teams that "just work" are highly valued. They clear up my attention for other things.
Teams that just work can't exist in stack ranked companies. You can't keep the team as a whole, you always have to cut someone.
Which means that everyone is playing the game to not be cut.
True, stack ranking is a terrible management approach, and if you work at a company that does it, then playing the game is the only way. But frankly, I'd be looking to get out anyway. The best way to play thr stack ranking game is to be job hunting.
But I'm not sure the author of this thread works in such a place. In that case the game is different.
In the case where the "urgent midnight fix" is important, it's necessary to promote the visibility of your (just working) team. If visibility is the game, then be visible.
You know how test-driven-dev was always "write the test first"? In that environment a test is always written before any code.
Well in the "ticket closing" scenario it's important to open a ticket, regardless of how trivial, for every code action taken. For every meeting attended. For every scenario dodged. If tickets are the way to score then write tickets.
If "being a hero" is the valuable thing, then be a hero. Be prepared to champion your team every chance you get. Every time you interact with management stress the emergency you just fixed (before it became an emergency.) Tomorrow do it again with the next thing.
Management needs visibility. Be visible. I know, this seems stupid and beneath you. But that's why they call it a job, not playtime.
> I'm not sure the author of this thread works in such a place
I worked at Amazon, previously.
> Management needs visibility.
I know this very well, and this is a problem. The nature of jobs in any industry is that not all of them are equally visible. As a manager, you should be proactive in assessing the state of things rather than waiting for people to deliver visibility to you. People who deliver "visibility" in spades are often charlatans. People who deliver fixes, code, and improvements in spades usually do not have time to manage their own public relations for your visibility.
However, you have ALL the tools to proactively see what they've been upto. You can attend their standups and other regular meetings, you can set up an updates document, you can see what they've been posting in Slack, you can look at their PRs and commits, you can look at JIRA tickets, and in the age of AI you can have AI explain to you all of the parts of the above that you do not understand.
I don't disagree. However few managers are this proactive. If you have such a manager, then fantastic.
If not then making yourself more visible becomes necessary. Because you can be sure (at least some of) your co-workers are doing so.
Or, you know, stand on principle, then come here to complain about injustice as things work out badly. :)
I refuse to play those games. If they want to fire me for avoiding problems instead of sacrificing my sleep, fine. I’ll go stock shelves at Walmart.
If someone is constantly playing the hero, I see that as incompetence. If the boss can’t see that, they are also incompetent. I have no respect for “leaders” who don’t know how to get out of the firefight.
I’ve made some high profile appearances, working 18 hour days on 4 day long outages, from vendor issues I was no part in causing. I figure that gives me some good will on playing hero without willingly creating problems for myself. I’m too old to manufacture stress for the optics.
For what it’s worth, with the right boss, I have had proper reporting work. Everything ran smooth and work was relaxed. My boss would regularly tell me I should take 3 months off because we were so far ahead of everyone. He would occasionally get bored and lob a grenade into the works to cause some chaos, but since everything else was running so smooth we were able to sort them out and keep going. People who couldn’t explain what they were doing were always getting yelled at and assumed to be doing nothing.
> I’ll go stock shelves at Walmart
Yeah, but then I wouldn't have been able to pay for my healthcare. A certain toxic company's health insurance paid for my care, though. Prior to joining said toxic company I'd be racking up $6000+ in healthcare bills a year with shitty startup-sponsored insurance.
After 2 years, it was decided I didn't play the hero well enough though, and ended up having to leave. I work for a less toxic company now, but the next time I need a heart-related surgery (likely in ~5-10 years) I'll join a toxic company in the months leading up to pay for it.
The rules of the US, I guess.
> I’m too old to manufacture stress
My point was less about manufacturing artificial stress. I don't do that. But many times I see issues in coworkers' code. If the company will value and praise me for catching and fixing them early, then by all means I'll do that. But if fixing issues in the codebase early for prevention only gets me criticism of "you haven't met expectations, we expect you to exceed expectations every performance cycle" then hell, I don't feel like fixing anything proactively. In that world I'd rather be the hero that fixes it when it surfaces, that's more likely to nail the rating.
Health insurance does complicate things. I hope your heart is doing well now.
I will say my motivation for helping other people avoid issues has dropped. If they want to make problems for themselves, they can. Me helping them hasn’t worked so far, so maybe some sleepless nights will be a better teacher.
I had a former boss call me Brent after reading the Phoenix Project. That made me step back and stop helping so much. Everything seems worse, but whatever… if that’s what they want.
> They set the rules of the game, you just play the game.
Obviously the only winning move here is not to play. Things like stack ranking are a perversion and no amount of compensation would be worth working for a company like that. If you choose to play, you're complicit in the moral abomination.
Isn’t this a universal problem though, not just software industry? Even at home, if one kid just does his thing quietly but another kid is difficult, what do we say? “John has his problems but he is trying, we gotta encourage him”. While his brother gets no praise or attention for just doing his thing quietly without fuss.
When things run smoothly, very few people notice. When things break, everyone notices
Econometricians can solve it, bc we can create rigorous models that map causal inputs to output.
It’s extremely advanced technology, though, and most CEOs would rather rent seek / camp than give up some decision-making power (and very few are even aware it’s possible).
Do you have any good sources for this I'd be interested in learning more
I used AI to unpack it a bit here: https://statwonk.com/econometricians-can-build-decision-engi...
I'd generally point to econometrics and statistics applied to business. The key activity is causal inference and then the context determines the mix of econo vs. stats required to help the org make high-quality decisions to increase output or make it more lucrative or higher-quality.
> I'm not sure how we solve this, other than having management come from engineering.
I disagree with the implied idea here that "engineers are better managers". The solution is to have good management, not to assume that "engineers are better managers". I have seen good and bad managers, and in both groups there were engineers and non-engineers.
Engineers may not be better managers but it's not easy to really manage something you don't have any insight in.
The tragedy is that “nothing broke” looks like “nothing was done” to people far enough away from the system.
Things keep breaking - "What are we paying you for?"
Nothing ever breaks. - "What are we paying you for?"
Management can choose their burden.
> I've been in those companies where "struggling departments" ended up getting all the praises and raise in budgets the following quarter because of the heroic saves they did, and raising awareness on how important they are... For stuff they totally caused on themselves.
This is a very game-able system, and I'd wager a decent amount that any senior engineers on those teams know exactly what they are doing. In a lot of (broken, but aren't they all) management structures, it's better to be seen to swoop in with the save than to quietly fix it ahead of time.
And if your management is structuring rewards like this, it leads to your seniors anticipating a bunch of these failures, lining up 90% of the fix before hand, so that they can jump on the oncall escalation with a 100% "Hail Mary" of a fix...
itemize the problems you are preventing
"Accounted for X situation" "Added gaurdrails to protect against Y"
When working as a business analyst i have to do this sort of thing all hte time or else id get no credit for half my work
Ah, yes - the person who comes in at 7AM and gets shit done by 11 is a slacker, and the one in the office just doing nothing after 6PM is the hard worker. Same thing.
You can't fix this. Out of sight, out of mind. It is hard-wired into us. It's all about the optics, and will always be.
> It's a serious problem in this industry
s/in this industry//
Managers will let you get away with anything if you time your reports correctly. They also don't want to sit in meetings where they are reminded of better outsourcing alternatives and they chose to dogfeed instead.
We've become too comfortable, since actual toil is no longer seen in the company: Manufacturing is overseas, customer support is overseas, logistics is an afterthought with established guarantees. Thus we want the mild weather and smooth meetings. If your engineering team is too smooth, maybe you should already branch out to help other related but "struggling" teams to get your hands dirty and noticed.
This thinking eventually results in The Scream Test. When the screams come as a system fails that is when they act on it.
Alas, for many parts of society there is a large amount of people that would rather be reactive than proactive. It means it is easier today but harder long term.
> It's a serious problem in this industry
It’s not a problem in this industry, it’s a problem everywhere.
> I'm not sure how we solve this, other than having management come from engineering.
You mean the engineers who are causing the chaos you’re complaining about?
Engineers aren’t some magic group of people who know better than others - we’re just as fallible as other people.
I guess the point of view is that if a department is well running, it means it is overressourced. So you reduce the ressources until it's breaking point, just enough for it to not fail. A jaded service manager told me it was part of its official training: if the clients was too satisfied that meant that human ressources were wasted on them, so he had to spin plates between clients. I guess it was economically optimal.
This is a short-term view approach and can really hurt a company on the long term.
It's also why US car companies are a wreck.
I run the media tech at an university solo. Needless to say I am underfunded. But more importantly, the infrastructure was underfunded too. I made it my first policy to also report near misses up the chain with their full implications, e.g. a list of events that we would not have been able to make.
E.g. that time a central media controls power supply broke down which would have made using one of the most prestigious rooms impossible. I fixed it myself by swapping in a spare power supply from a used unit, then went on to remind them twice a year that we are now living on borrowed time and I take no responsibilities if a fault I predict to happen and get no funds to fix will in fact happen. 4 years later I got the funds.
Having stuff costs money. Everybody wants to invest funds once, but nobody wants to keep paying for maintenance.
Since it's entirely possible (extremely likely?) the "problem" would never materialize, this is quite reasonable.
SpaceX almost has a full grip on the planetary consciousness extinction problem.
lol. I hate presentations. I like to run a tight ship. But that does not shine, so they made me do presentations every quarter. If you do some work, you must "take" credit. It is kinda a need when you manage people since you need to build their careers.
I finally moved on to be an IC. Same story, same pressure :) You need to present to directors not because they need to know, but because your managers have a quota of N presentations per quarter, and if you back out, someone else needs to step up.
Needless to say my productivity reduces by half and sometimes to almost zero during the week or fortnight of presentations every quarter.
You define "productivity" as coding.
The business defines it as "meetings, presentations, support, coding, whatever".
Your productivity remains at 100% when you are doing what they want.
I get that you thought you were hired as a coder, and thus measure your productivity by that. That's what I thought too. I ended up doing a lot of support (which is good, but that's another thread). Until I recalibrated my definition of productivity that frustrated me. When I realized that support was productivity I got much less frustrated.
When did I say I code?
I have been on the industry for 35 years. I have seen my share of technology evolutions and o have seen the work from a dozen different dimensions. If after all that time, I find the process painful, just trust me -- they can't change me, and I can't change them. You take the warts with the wins and move on. 2-3 bad weeks, 10 good weeks. Life moves on to next quarter. Complete CEO mindset :)
You heavily implied presentation preparation implies zero productivity. He tried to say this prep is also productive even if you personally don't or can't appreciate it.
Last comment and I will see myself out.
I meant my other productivity drops because I am not a natural presenter so even though I am rehearsing / editing for 2 hours a day, the presentation consumes me / overwhelms me that I can't even focus for the remaining 4 hours or 2 hours. Just do the bare minimum email processing, just survive. Everyone knows it. But by being in that zone of paralysis, I can still deliver a presentation. Sometimes good sometimes ok.
I have this need for the presentation content to reside in my memory cache and other work disrupts the cache quite badly.
But that's not a way to live. The other work stalled for 3 weeks.
I feel that disconnect is everywhere, when the suits dont see anything and act on reports
Which they should. I've been lucky enough to work at places that had great non-technical managers that promoted based on great execution, as well as highly technical managers that also promoted based on great execution.
Now I'm at the other kind of place and it sucks. They'll fire the performative engineers though during layoff season. It's almost like they like playing politics until it really matters.
I feel like this is a cultural symptom and something many people are hoping to solve in healthcare. Basically we treat solving problems as amazing rather than preventing problems. You get rewarded if you treat a sickness instead of keeping a healthy person healthy.
This is the same thing. We need to reward things never going wrong as a society since this is pervasive.
> something many people are hoping to solve in healthcare
Respectfully, the solution is don't smoke, exercise, eat well, sleep, avoid stressors... These aren't easy problems but their solution isn't at the individual patient level and is a simple question of capital and political will.
The 'hope' envisages a product to temporize the solution while extracting large payments.
Nope, I believe you are wrong: a path where we, for example, forbid smoking because the statistics point at it correlating with many health problems, is a world where we use the same statistical tool to prescribe human behavior to the last detail. It is not just about smoking, alcohol, late night dancing, switching sex partners, fast driving on a track, paragliding, skydiving, climbing, car driving, bicycle driving, motor biking, even staying late for astronomical observations (sleep patterns?)... all carry insignificant risk when looked at statistically.
> ...avoid stressors...
Most stress is caused by a conflict between our expectations/motivations and the reality (everyone else's).
> forbid smoking because the statistics point at it correlating with many health problems, is a world where we use the same statistical tool to prescribe human behavior to the last detail.
I had a really great egg for breakfast. This now means I will never eat anything else besides eggs.
Also, I realized that cars run better with oil changes every 3 months or 5,000 miles. Because shorter was better, we should all start changing oil daily.
The best player in the basketball game last week was over 7'4" tall. I guess I need to discourage anyone who isn't that tall from playing ever.
Do you see why banning smoking is a good idea?
Car industry best practices
Track leading indicators, pricing them if possible.
could an AI product solve this?
I believe it's a problem in most industries and even humanity in general. I don't believe it's a business problem at all.
Heroes are lauded even if they solve problems they themselves are the cause of - which is conveniently either forgotten or denied - or they are solving non-issues that are deemed important by the ignorami-class. Politics, for example, is utterly dominated by this dynamic.
It's the first instinct: let the expert run the show. However, one of the (many) ways to let a complex project fall apart completely is to hand over full control to engineers. I'm one myself, but I know what I'm good at and what not. Dunning-Kruger is often mentioned in these discussions, but don't forget it also applies to engineers that often lack any management or leadership experience of any appreciable kind. They vastly overestimate their ability to handle management and organization-wide issues and tend to not only miss the forest for the trees but actually miss the trees for the leaves.
"Unix: A History and a Memoir" by Brian Kernighan actually mentions how proper management was crucial to their success. It's a detail that's frequently conveniently forgotten by the engineers who think themselves better than the "suits". For the record, I don't claim engineers are the primary problem, but it's not just management's either. Quotes like "who holds the company standing" and "who understands how to double click" are enormous smells and IMO make quite clear what's happening here.
I don't have ready-made solutions unfortunately, but I do wish we would look further than "it's the suits". It's a systemic, human problem that I believe is a natural result of operating under informational constraints and, very human, cognitive biases on all sides.
Bell Labs is an outlier in basically every aspect. Mr Kernighan lists stability of the environment with regard to funding, structure, mission as well as technical competence of the management as main drivers of the culture. This is just not the reality in companies that look for financial results on a quarterly basis and where the executives are MBA types.
If one of the most successful engineering organizations in history attributes part of its success to capable management, that undermines simplistic narratives where management is inherently the problem and engineers would naturally thrive if left alone.
If anything, the Bell Labs example supports the idea that exceptional outcomes require both strong technical talent and strong management working together.
Not saying the "MBAs" are helping the situation, but the hero developers and their resume driven development practices aren't exactly angels either.
Capable is the loadbearing word, the directors were all PhDs in math, science and engineering fields.
I dont subscribe to the strawman argument that engineers would naturally strive on their own, but neither does simply any form of management automatically add value.
I agree also that hero type devs are an indicator of problems
>>I'm not sure how we solve this, other than having management come from engineering.
Given the whole point of management is to work to ensure their own survival and growth, it would in their interest to kill genuine competition when its coming up.
Who wants to raise their new competition and lose to them, no one!
Hikaru Nakamura had a good comment on famous chess moves like this: All the famous chess moves are famous sacrifices, genius saves, and last minute checkmates, but when you plug the games into stockfish you find out that these scenarios only happened cause the players made a series of awful moves early on in the game
There are a lot of things like this.
My favorite is how elegant solutions often look simple in retrospect. So if you noodle on a problem for a while and then come up with a clever solution: once you explain it to someone they'll be like, "yeah, of course."
Meanwhile the guy next to you that overcomplicates the problem ends up getting kudos for building something so difficult :D
"Je n'ai fait celle-ci plus longue que parce que je n'ai pas eu le loisir de la faire plus courte."
("I have made this longer than usual, only because I have not had the time to make it shorter.")
Blaise Pascal
"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away."
— Antoine de Saint-Exupéry
I read this in Leonard Nimoy's voice, from narrating Civilization IV.
"A facility for quotation covers the absence of original thought."
― Dorothy Sayers
I am reminded of this course in university where there was a written assignment with a minimum page count. Even at that time, I remember thinking: "If I am able to express everything necessary in just one page, that should give me the absolute top grade!".
I feel like AI coding is accelerating everyone's work toward greater solution complexity and I think it's pushing people to build defenses and be more averse to someone else's complexity rather than being impressed by it. Bigco's are probably well behind the curve on this and are still impressed by complexity, but for people on the receiving end of AI stuff either directly via your own hand or indirectly via others, it seems like complexity is not as impressive as it once was.
Definitely. Previously, intent and effort were required to increase complexity. Now intent and effort are required to prevent complexity*!
But also, what a beautiful problem to have!
Yeap. Managers perceive complexity by how personally confused they are. I'm late in my career, and I'm realizing I wasted so many man years trying to make code clean, user friendly, and maintainable, when that code was never read by another person again and forgotten 15 minutes after it was released, then used for years. This is why I think AI is coming for our jobs much sooner than many people think: clean code, separation of concerns, maintainability, etc, all the things we spend the most time on, have never actually been valued. "Good enough" is fine, and keeps management happy. And, if something does pop up, AI can patch it, even if with spaghetti, just like fucking that ass at work.
There was a thread recently on HN about Claude Shannon and how his papers were filled with clear descriptions and explanations. Then someone commented how they had found an elegant solution to a problem that either could be described shortly and beautifully so that a high schooler could understand or take the long and tedious path of a convoluted explanation.
The director then clearly advised that they should use the complicated way because that's how you get published: not because you're clever, but because your solutions sound complicated.
It resonates perfectly with your comment and it's an unfortunate reality that most people don't bother for beautiful solutions and praise complicated processes. That's how we neded up with bureaucracy, probably :D
On the other hand, I don't help people with their computer problems anymore because I've found that the more difficult the problem, the longer it takes me to rescue their data or whatever the less impressed they are. The more miraculous the save the more likely they are to tell me the story about their nephew who solved a trivial issue instantly as if to point out that I didn't.
I have said it for decades - Basic is easy, Simple is hard.
But when someone comes up with something simple but effective, it always looks so obvious in retrospect.
The passage that comes to mind for me whenever this idea comes up, from the Brett version of the Holmes story "The Dancing Men":
Found the clip: https://www.youtube.com/watch?v=Zx6Dr1iJ4p8&t=3m13s
Meta: I found this video essay (?) "Jeremy Brett vs Basil Rathbone — Who Was the Real Sherlock Holmes?" interesting:
* https://www.youtube.com/watch?v=WaQFJcI_yfI
Real Sherlock was Vasily Livanov of course.
Liking it, but I think it's even better captured by the more lauded quote -
H: "How often have I said to you that when you have eliminated the impossible, whatever remains, however improbable, must be the truth?"
Why on earth would you quote a TV version of a book, when the book is readily available to be cited?
⁂
“So, Watson,” said he, suddenly, “you do not propose to invest in South African securities?”
I gave a start of astonishment. Accustomed as I was to Holmes's curious faculties, this sudden intrusion into my most intimate thoughts was utterly inexplicable.
“How on earth do you know that?” I asked.
He wheeled round upon his stool, with a steaming test-tube in his hand and a gleam of amusement in his deep-set eyes.
“Now, Watson, confess yourself utterly taken aback,” said he.
“I am.”
“I ought to make you sign a paper to that effect.”
“Why?”
“Because in five minutes you will say that it is all so absurdly simple.”
“I am sure that I shall say nothing of the kind.”
“You see, my dear Watson”—he propped his test-tube in the rack and began to lecture with the air of a professor addressing his class—“it is not really difficult to construct a series of inferences, each dependent upon its predecessor and each simple in itself. If, after doing so, one simply knocks out all the central inferences and presents one's audience with the starting-point and the conclusion, one may produce a startling, though possibly a meretricious, effect. Now, it was not really difficult, by an inspection of the groove between your left forefinger and thumb, to feel sure that you did NOT propose to invest your small capital in the goldfields.”
“I see no connection.”
“Very likely not; but I can quickly show you a close connection. Here are the missing links of the very simple chain: 1. You had chalk between your left finger and thumb when you returned from the club last night. 2. You put chalk there when you play billiards to steady the cue. 3. You never play billiards except with Thurston. 4. You told me four weeks ago that Thurston had an option on some South African property which would expire in a month, and which he desired you to share with him. 5. Your cheque-book is locked in my drawer, and you have not asked for the key. 6. You do not propose to invest your money in this manner.”
“How absurdly simple!” I cried.
“Quite so!” said he, a little nettled. “Every problem becomes very childish when once it is explained to you. […]”
— The Adventure of the Dancing Men, The Strand Magazine, Vol. 27, January 1904, The Return of Sherlock Holmes, by Arthur Conan Doyle <https://www.gutenberg.org/cache/epub/108/pg108-images.html#c...>
Every company where I've worked at as a SWE openly rewarded "engineering complexity" as a criteria for getting promoted, which I've always found to be absurd because complexity can always be manufactured (both of the problem and the solution).
One of those days, however, you come up with another of your elegant, simple solutions and it actually replaces a 150K LoC monstrosity with either a 1K script or, even better, with nothing as a simple shift in perspective or process obsoletes it completely.
In the long run, IME, you'll be recognized either by management or your peers if you keep doing that over and over again.
And of course that's true. We push for speed, absent of direction, while praising velocity. To be honest, at this point I'm disappointed the engineers gave up and just started becoming business people.
"An idiot admires complexity, a genius admires simplicity"
In academia that translates to, the more senior the faculty member, the easier the talk is to understand.
"elegance and speed go hand in hand" - d. mcilroy
I had this problem at a previous job. I spent almost all of my time taking care of the behind the scenes administrative work (scheduling meetings, making sure that people had the information they needed to come into the meetings prepared, etc.). However, when performance review came around I was told that the only thing that they cared about was that I hadn't completed many story points because I was too busy keeping things from falling apart.
So I stopped doing all the administrative work and focused on just completing story points. A week or two later my manager asks the team "how come all of our meetings are falling apart now? We get into a meeting and no one knows what's going on."
The headline is only half true. A more accurate rendition might be "Transactions don't happen because you fixed a problem that never occurred."
That illustrates the converse. People will absolutely try to avoid future problems if they are the ones that bear the consequences for them. Use birth control (or put your kid on it) so you don't have to raise another child for 20+ years. Don't hang out with that volatile "friend" who always seems to be having another crisis. Fix your roof so you don't lose the house. Don't go into debt because you're the one who will be paying interest on it.
But almost by definition, bearing the consequences of your own decisions implies that there's no transaction.
It's interesting and fitting that the article begins with a discussion of Toyota. People buy Toyotas because they want to avoid problems. The biggest selling point is reliability; a Toyota's value prop is that you can keep it for 20 years and you won't have unexpected things go wrong with it. Toyota has managed to turn this into a sales driver because they appeal to the self-interest of a buyer who will be living with the car for 10-20 years. There are thousands of individual decisions that Toyota makes to avoid problems with their cars, starting with a very conservative aversion to new technology or anything that is engineeringly risky. Each individual one is invisible to the customer, and often comes with significant costs in isolation. But because their sales driver is "be reliable at all costs" and they've ingrained that into the culture, they've built an organization that is willing to make these feature trade-offs for reliability.
Also, an interesting corollary is "Oftentimes, a good life is lived with few transactions."
I began migrating from network/hardware/IT work and into marketing after nearly 2 years of heavy lifting getting ready for Y2K. In the end, "nothing happened," so all that time and money was wasted, according to nearly every company I worked with. Even had one demand a full refund. I agreed as long as I could revert all the work that I had done. They agreed, and the next day after that their entire system collapsed.
I couldn't even get my own dad to pay for network support for his company since he would never pay my rate for anyone no matter what. After 2 other people failed to solve his problem I fixed it in 15 minutes and then he "really" didn't want to pay because it only took 15 minutes.
I was very good at what I did but got no appreciation for keeping things from breaking, only for fixing things after they broke. Marketing paid better, and I could point at real world numbers daily and justify my pay. I don't like it anywhere near as much, but at least it gets more respect than any other IT work I did.
I'm sorry your dad didn't respect your IT work...
If I move out a bit of my circle, where people do all kinds of work, I'd say that there's generally a stigma on the "IT" workers. Moreover the stigma is there even within the IT company/industry itself, where sales, marketing, non-engineering parts of management, and other similar types of supporting roles also look down on the engineering. And this unfortunately includes family members too. I learned that people are mostly envy but when you're surrounded by many it can become overwhelming - numerous times I heard phrases like "oh, you're bricklayers of a modern age" ... like wtf
I do think of all my computer work as predominantly janitorial.
On your dad:
"This is what my regular customers pay me. If I hired one of my friends or relatives I see it as my duty to pay them at least what they are worth, this is the way you raised me."
I believe this to be true btw. If someone is really your friend, you want them to do well and that means you pay what they usually get or you don't bother them and get someone else.
Idk I joined the field only like 5 years ago. Prior to knowing about programming I had a lot of respect for programmers and tech as it was this magic world to me. After having joined the field the magic is completely gone and I don't want to talk to programmers irl anymore because of how many insufferable people I've met so far. The parts where I've got credit are the simple things, fix printer, fix computer issue A,B and C or small apps like an ad free Sudoku for android which I have built for my friends. The parts I don't get credit for are the parts I get paid for. But I can see that in many industries, as soon as money is involved people are less thankful because they expect you to fulfill your part of the contract. Generally people not knowing shit about tech think that devs sit in HO all day working only 30 minutes. AI didn't help that image.
Ian Rush said it best: "It's best being a striker. Miss five, score the winner, you're a hero. The goalkeeper plays a blinder, lets one in, and he's a villain."
Every place I've worked rewards the firefighter over the person who made sure nothing ever caught fire. And the worst part is the math is obvious to everyone except the people who set the incentives.
How would you set the incentives, though? Almost by definition, it's hard to reward things that aren't visible.
Note that there is also the flip side of the coin, people who spend all their time worrying about things that never happen, so it's not like you can just reward a defensive attitude things are more complicated than that.
The equivalent of 'days since last injury' bonuses is the first method that comes to mind, until you consider that this would mean people would be more likely to hide things going wrong.
So then many things are to rely on executive culture, and an executive who will walk the line and get their info from people at the bottom is like a unicorn. That won't scale, but it does work if you do have such an executive. Naturally they would need a basic understanding of how supply is created in their firm.
Yet there is something. Toyota Hiluxes and Honda Super Cubs got popular due to maintenance ease. AK-47s. Miele vacuums. Older Thinkpads.
What measures would make the human equivalent visible?
One basic example is not counting bugs as points in your ticket tracker. At my last job I had coworkers whose velocity was almost double everyone else’s but it was because they kept deploying and then fixing their own bugs.
I dont have an answer and you are mostly correct. I received some advice based on this that made sense which was to pick the roles in your career that naturally made it easier. Sales, PM, Dev etc and not support, Devops, escalation management, CSM etc.
This is how people get promoted at work. They break something, it escalates and gets visibility, emails sent to executives. Now, they 'fix' it, many thank yous from everyone for a job well done. Another version of this is delay the work you are supposed to do a long time back and let it gain visibility. Executives are blind, they can't see work being done by people who take ownership and get shit done before it becomes a problem. However, executive will remember the name of the person who breaks shit and 'saves' the day.
The sociopathic backhand. Someone breaks it, blames you, drags your name through the mud exacerbating the issue, goes to "fix it" yet makes it worse because of their incompetence. They continue to cuss you out for it all while licking the arse of the executive. "No, we better not let doublerabbit touch that", "I don't think they're a team player". All while it was my infrastructure.
And people ask why I hate humans.
My time in IT has really oscillated between these two extremes:
- "Everything around is working just fine. What are we paying IT for?"
- "Everything is broken. What are we even paying IT for?"
Personally, I strive for the former rather than the latter; I like to say "If I do my job right, you never know I'm here." But that's what got me let go.
(and for karma's sake, I keep in touch with folks at the old company; it's an absolute crapshow. So I got that going for me; which is nice)
We all learned this back in first grade. The kids that behaved in class and did their homework did not command most of the teacher's time and effort. It was the problem children who refused to follow the rules and needed constant praise for every bit of actual effort that they put into their studies; that got the teacher's attention.
The squeaky wheel gets the grease, is how I always heard it phrased.
This brings to mind Neil Rickert's "The Parable of the Two Programmers", which was published in the ACM SIG Software Engineering newsletter, January 1985.
https://dl.acm.org/action/showFmPdf?doi=10.1145%2F1012443 for the original, or https://realmensch.org/2017/08/25/the-parable-of-the-two-pro... for a reprint.
I never gave credit to my electricity company for delivering electricity to me. I only get mad when there is an outage.
> I never gave credit to my electricity company
That's what the moneys for ...
Counterpoint: I absolutely give credit to Sonic for being a great ISP and recommend them to everyone. I got my parents to switch when Sonic finally rolled out to their neighborhood.
If online comments are anything to go by, I'm not alone.
If you're in the Bay Area and you can get a Sonic fiber connection, I would highly recommend them over AT&T/Comcast/etc.
Its either end of the spectrum-you do the best job(top 10%) or your everyone else.
If you only do middle of the pack(for one reason or another-cost, talent, etc) you become incentivized to cause problems then fix it.
Thus a net negative to society
*Also recommend sonic-their pricing and service is top tier
Just remember, the power grid fails in theory but works in practice.
Well in a way you do. They send you a fine bill every month and you do credit some of your allegedly hard earned bucks.
I think of electric vehicle fires and jet airliner crashes.
Also, telsa self-driving. yes, we know about the greatly publicized accidents, or the tweets of the founder, but the avoided incidents not so much.
This is where the industry made and continues to make mistakes wrt autonomous driving.
They should be able to quantitatively say how many crashes were reduced, avoided and spotted. The autonomous safety system should be running all the time and it should detect not only issues with primary vehicle but it should also catalog issues it sees in other vehicles in its vicinity.
We shouldn't have gotten AD before we got automated crash avoidance.
As Futurama said, when you do things right, people won't be sure you've done anything at all.
They stole that from the Tao Te Ching, chapter 17:
— <https://ttc.tasuki.org/section:17>You'll see capability traps everywhere once you learn about them.
Sterman, Repenning and other collaborators wrote several papers after this one. All fascinating and almost entirely depressing.
Especially since MIT's Sloan school, where system dynamics first became a discipline, is just around the bend from Harvard Business school, where system dynamics first became ignored.
One thing I don't get about the concept of capability traps is why is it expected that a company which is good at one thing would be capable at the new thing? What exactly makes a capability trap a trap?
The trap is that you can't get better without first getting worse. You can't get out of the destructive cycle of production pressure and decaying productivity without removing the pressure. Many managers expect, or at least behave as if they expect, improvement to be monotonic and costless.
When you are at capacity and in a degraded state, you have no additional headroom to get out of that state. Why wounds won't heal, or the poor stay poor.
I think because it’s a negative feedback cycle. So once it starts it’s hard to go back, the deeper you are the harder. A trap is something that’s easy to get into and hard to get out of.
I'm looking for some data -- if anyone has it -- on the fraction of companies that are led (CEO) by a technical person, over the years/decades. I have the (anecdotal) impression that this fraction has been falling (stories like Boeing), but it would be cool to support or refute this with hard data. Anyone know where to find/assemble something like this? Also, if this trend is true, then why?
I don't have the data, but I think a good case study is the company MITRE.
Originally it was engineers from the top down, but over the last 15-20 years those leaders with engineering backgrounds have retired and been replaced by non-engineer MBA's. And the more I look around, the more I see that as a common trope is the US.
If this has been true, perhaps at some point the pendulum will swing back the other way. BTW the current CEO of Boeing is a mechanical engineer.
https://www.boeing.com/company/bios/kelly-ortberg
After decades of infamous quality drops that have impacted their reputation and finally bottom line under non technical CEOs.
This is true but if you remove the need for credit then you can just get back to work and not have to create a category about it.
Two significant prior discussions:
https://news.ycombinator.com/item?id=8940820 - 24 Jan 2015, 50 comments
https://news.ycombinator.com/item?id=39472693 - 22 Feb 2024, 434 comments
There is something I saw on a reddit post of all places, about how every manager who doesn't predict a baseline of "3 annoying problems every month, 1 awful problem every 3 months" is essentially a bad manager. The reasoning being that, if your number of problems is under that threshold, then someone is doing a 'good job'.
Aka:
* https://en.wikipedia.org/wiki/Preparedness_paradox
My favorite example is the introduction of speed limit on some accident-ridden stretch of the Autobahn north of Berlin. After introducing the speed limit, the accident numbers went down dramatically. What did the local administration decide? Remove the speed limit again -- cause there were no accidents anymore!
This is why Apple never gets any credits.
This is exactly the problem with the nature prevention. When it's well done, it seems like nothing was done.
When Covid started, our local government was very clear from the start in saying "If people think we have over reacted, then that means we have done a good job."
Alas, that doesn't always fly with the populace.
Trouble is that it is not always true either. You can legitimately overreact and in hindsight it can be hard to distinguish between these two things.
Plus, even if you did overreact, that can still be the better side to have erred on, in moderation.
Article published in the Summer 2001 edition of California Management Review, yet it never mentioned Y2K, the first thing I thought of when I read the line "fixing problems that never happened". Perhaps it was actually written in 1999 and took a while to get published, because otherwise that seems a very strange omission. The Y2K problem was very much over-hyped by the American news media at the time (no, at no point would airplanes have been falling out of the sky — I literally heard someone say that would happen once — even if no effort had been put into fixing the bug).
But in recent years I have seen people (elsewhere, not on HN) claim that Y2K was a big nothingburger, and all the money spent on fixing the bug was wasted. No, that's not true either. All the money spent on fixing the bug was why it turned into a big nothingburger. Sure, some of that money was wasted, by executives who wanted an "official" Y2K-certified certificate, issued by a consulting firm that had nothing "official" about it except their own say-so. And so they spent $2 million learning what their own employees could have told them for $2,000. THAT money was wasted. But a lot of banks were running old COBOL code that used 2-digit years, and needed to be fixed. The fact that in January 2000, everyone's bank interest was still calculated correctly, and not calculated as if it was January 1900? THAT was entirely due to the vast amounts of money spent paying old COBOL coders to come out of retirement and fix the 2-digit years.
The lesson I learned from that is that it's possible for a problem to be overhyped, even massively overhyped, and yet still be a serious problem. The other lesson I should have learned is that people rarely get credit (I won't go so far as the article authors and say "nobody ever gets credit") for fixing problems that never happened.
The problem is that a lot of people have a very binary view on life. Either something is a complete success or a complete waste of money, rarely do we accept that most projects fall somewhere in the middle.
The binary view is mostly true, unless it's for events or problems they are themselves familiar with. There is a term for this, but can't for the life of me remember it: People think the problems they are dealing with are infinitely more nuanced, complex and unique than the problems other people are dealing with.
And even worse, they don't think probability is a thing. If something happens, it was certain to happen and we just failed to predict it correctly.
So when someone predicts something will happen with a 90% probability, and then the 10% chances happens and the predicted event does not happen, people will talk about what a bad prediction that was and how they were clearly wrong.
It's the same logic that causes people to say vaccines don't work because they don't stop a disease with 100% effectiveness, or that there is no point to wear a seatbelt because people still die while wearing one.
My issue with this version of explaining the lack of severity of Y2K is that there were lots of countries that were being derided for not taking the issue seriously but did not seem to suffer any ill effects.
This is interesting, do you have any links?
A couple of possible confounding factors I can think of:
1. Plenty of countries use software developed elsewhere.
2. I suspect that the more recently you computerised your economy, the less likely it would be to have code vulnerable to Y2K.
It's also possible that in some places there were a few issues, but people looked at bills for 100 years of electrical service and said "Yeah right," and fixed the now-easier-to-find code that still used 2-digit dates. If that only happened a few times, the extra work involved in working out the January bill by hand (or waiting until February then billing for 2 months) wouldn't cause too many issues in the economy, and anyone looking in from outside wouldn't even realize there had been an issue. If it happened everywhere the economic impact would be more noticeable from outside.
>no, at no point would airplanes have been falling out of the sky
The assertion may have been unfounded, but I think it's just as unreasonable to assert the opposite. Bugs have cascading effects and in a sufficiently complex piece of software they can create chaos with unpredictable outcomes.
The one case I'm aware of where a software glitch did cause a plane crash, there was pilot error compounding the problem. Air France flight 447 was an Airbus A330 flying from France to Brazil, and while high over the Atlantic, the software recorded inconsistent data in its airspeed measurements. (The official crash analysis team concluded that the inconsistent data was likely due to ice crystals blocking the pitot tubes on the plane). The inconsistent data made the autopilot disengage. Pilot error then caused a stall. One pilot then tried the correct move to recover from a stall, pushing forward on the stick to nose down and regain speed. The other pilot was pulling up on the stick to stop the dive, not realizing that that's exactly the wrong thing to do in a stall (or more likely forgetting his training due to panic; he had a lot less experience). The flight software, receiving inconsistent inputs from both controls, averaged the inputs, resulting in zero change in pitch. (It also sounded the "Dual Input" alarm, but the pilots were too preoccupied with their own controls to figure out what that meant at first, and by the time they figured out what was going on it was too late to recover before the plane hit the water).
https://news.ycombinator.com/item?id=4224707 has some discussion of the events, including the fact that the control design (where each pilot has an independent stick) was part of the problem. On a design like Boeing uses where both sets of controls move together, the experienced pilot would have noticed the less-experienced pilot pulling up on the stick because his own stick would be moving, and he would have said "No, nose down." And if they had nosed down to recover speed while still high enough in the air, they almost certainly could have regained control of the plane and saved 228 lives (including their own).
So in retrospect, I think my first sentence was wrong. The software did not glitch, it did exactly what it was supposed to do. It was pilot error that caused the initial stall, and multiple pilot errors that caused the failure to recover from the stall.
There may be examples of software error that has caused planes to fall out of the sky, but I don't know of any. The only plane crashes whose cause I know were due to hardware failure or pilot error, usually a combination of the two.
I think your conclusion is upside down. Air safety is based on the "Swiss cheese" model. Multiple layers of safety nets are in place to compensate for issues in one layer. In particular, technical safeguards are there to prevent disasters if the human in the loop makes a mistake which will eventually happen. Any weakening of any technical safeguard makes the system less safe. No matter if the human ultimately made a mistake -- the technical system failing contributed to the accident just as much.
Y2K is especially interesting because the fact that the year 2000 would one day occur was entirely foreseeable, and no less probable in 1990 than in 1999. I can hardly think of anything with closer to 100% probability of happening.
To be fair, there was a non-zero chance that society could have ended (or your company, or the tech became obsolete) before 2000, which would be higher the earlier before 2000 you were.
The tech being obsolete is why Y2K was a smaller problem than it would have been otherwise. Most places were no longer running much COBOL code. But banks are famously slow to upgrade their tech, and for good reason much of the time, so most of the world's remaining COBOL code (and other code too, COBOL is just what I'm most familiar with, not that I'm all that familiar with it) was in banks and other financial institutions.
Year 2038 says hi.
my first thought too. I've met a few people who assert that Y2K was a complete waste of money.
I earned my first house deposit helping the team fixing the water and gas company in Wales, UK. Their entire system was running off a set of COBOL programs on a mainframe, none of which had been properly documented over the years, and the whole thing used 2-digit dates. It would have caused actual deaths if not fixed; everything would have shut down, and no water and no heating in a British winter is potentially lethal. And then it would have sent everyone in Wales a bill for 100 years of water and gas.
They were bribing retired software devs to come out of retirement with huge stacks of money, because that was cheaper than training new COBOL devs and getting them familiar with the spaghetti system.
It worked, no-one died, life went on. So obviously it was all fake rolls eyes
I'm curious why things would have shut down when the system thought it was 1900. What part of the logic had the effect of "shut the system down if current date is less than (X date)?" (If you can remember the code 25+ years later, that is).
I recognise almost every aspect of this document - it's exactly what's so intractable about the software business. This is why I think you do need to do some programming every now and again no matter what your level is because otherwise you cannot see what's happening and you'll be tempted into the "lazy developer" attribution.
Its really hard to measure effectiveness, problem becomes even harder when a non-engineering person has the job to measure effectiveness of a engineering person.
On other hand. for software engineering some of the signals that can be used to measure such a management itself can be
1. On call requirement, outages and team burnout - A well written software should not require on-calls from the dev team
2. Ask them about the "concrete" roadmap for next 6 months to a year - Absence of concrete items is a bad sign
Apt comic: https://www.workchronicles.com/p/comic-prevention-vs-cure
I feel obliged to point out Stanslav Petrov, who absolutely got credit for fixing a problem that never happened. Granted it's a very extreme case.
Credit only in fame.
https://en.wikipedia.org/wiki/Stanislav_Petrov#Aftermath
> Petrov underwent intense questioning by his superiors about his judgment. Initially, he was praised for his decision.[2] Colonel-general Yuri Votintsev, the then-commander of the Soviet Air Defense's Missile Defense Units, who was the first to hear Petrov's report of the incident (and the first to reveal it to the public in the 1990s), states that Petrov's "correct actions" were "duly noted".[2] Petrov himself states he was initially praised by Votintsev and promised a reward,[2][22] but recalls that he was also reprimanded for improper filing of paperwork because he had not described the incident in the war diary.[22][23]
> Petrov has said that he was neither rewarded nor punished for his actions.[24] According to Petrov, he received no reward because the incident and other bugs found in the missile detection system embarrassed his superiors and the scientists who were responsible for it, so that if he had been officially rewarded, they would have had to be punished.[2][24][22][23] He was reassigned to a less sensitive post,[23] took early retirement (although he emphasized that he was not "forced out" of the army),[22] and suffered a nervous breakdown.[23]
His page links to https://en.wikipedia.org/wiki/Nuclear_close_calls which is a harrowing thing to read. We keep rolling the dice with no changes to the game.
The same article points out that he received at least £26k in awards. It could be argued that the reward isn't proportional to the magnitude of his actions, but it exists.
I remember finding a comment in the first codebase I ever worked on professionally in my first ever job.
It read "This fixes a bug that hasn't happened yet".
It seemed really smart at first, but later I learned that the developer that added that code also had a pattern of appending spaces to the start and end of user input and comparing the length to 2 to determine whether the value was empty or not...
So I'm fairly sure "that hasn't happened yet" was probably more a case of "that I personally haven't introduced unnecessarily yet" :)
Pendatically speaking, people do get credit for fixing problems that never happened.
E.g. if the problems are quantifiable and there's a record, like dropping homicides from 100 per year to 20 per year in a city. Those extra homicides "didn't happen", but the improvement is understood.
For an one-off problem, it depends on how clear the path to the problem is. An electrician doing an inspection and noticing and fixing big electrical issues in the installation, would be appreciated, even if the accidents didn't happen.
> Those extra homicides "didn't happen", but the improvement is understood.
People are gonna criticize by saying "see? it was an overreaction to the problem, since there's not been many homicides at all!", when in fact the homicides were prevented by fixing the original problem. Same way with the electrician: "how much are you gonna charge again? And you're charging for a fix to a problem that didn't happen yet? Nah, I'll call you when the problem happens".
Its maddening.
> An electrician doing an inspection and noticing and fixing big electrical issues in the installation, would be appreciated, even if the accidents didn't happen.
Not if nobody knew he'd fixed it.
Like nobody gets credit for avoiding problems or unnecessary things/complexity altogether. In fact the opposite may happen.
This is especially nice in the age of AI. I (the graybeard senior developer) does all the risky refactoring. I can take a performance issue and turn it into six regressions in half a day ($100). Then everyone is impressed when I let Opus fix these regressions in 20 minutes and $2 worth of AI.
No one notices when you cut 20% of some expensive process but cause no regressions.
Here is another take in s different context.I had a very bad manger who was credited earlier with causing several companies to go out of business while he was taking advantage of having worked for a FAANG company. He worked there for less than 6 months as a business development dude when the that big shot company was new in that market.After he joined us and when I was processing some payments I noticed that he was paying some people for 2 months ahead of their starting dates, it was some $20k.I notified him, and he corrected the date,then I was told by others in the department that my problem was that I should not 'interrupt the enemy when he is making a mistake'. Eventually, the company got rid of him but it was after a very serious damage.
And people often get credit for fixing issues they partially created.
Human groups work on shallow signalling and distributed confusion.
Couldn't help noticing:
In other words, it’s not just a tool problem, any more than it’s a human resources problem or a leadership problem. Instead it is a systemic problem [...]
Shades of an LLMism, a bit padded, a quarter of a century ago. These days someone could easily give it a stink-eye. I'm sure that training has ingested this along with countless similar examples.
That should really be a cautionary tale for everybody accusing everyone of LLM manufacturing texts. Many people write like that. The self-censoring nowadays to try to avoid sounding like an LLM is really sth we need to grow out of.
Since Covid nearly everyone in Germany knows this saying: "There is no glory in prevention."
This is why I'll never be a fire protection engineer
Depends on the business.
When everyone is technical to some degree, I find that credit for technical rescue is forthcoming.
Avengers get the glory, preventers get no story.
"Titanic effect" - No captain gets credit for preventing a disaster!
People do get credit for making things that "just work".
So let's create moments or days of observance to make people aware of preventative measures taken
I align as such.
Then we soon see non-technical people start leading the same, pushing for some people to be recognized for this every sprint. Meaningless recognition starts coming in. The process fades out in just a couple weeks.
The problem is there are these people in the mix, often leading, who do not understand.
A moment of silence to appreciate the silence the proactive measures gave us.
I'd guess a lot of people here consider this "reality" at this point. Has anyone come up with a response—not a fix for the company or leaders behaving this way, but a response for their own path?
Did you change from a quiet diligent one to manipulating and playing the game (now that you know the game)? Did you go from quiet and diligent to quiet and not diligent (why do good work when meh work does the trick)? Another path?
I just do my job to the best of my ability. I have to change jobs every couple years anyway to get proper pay bumps, so I don't really care what the higher ups think of me. The people near me are who I'll use as references and they generally know I'm great at what I do.
https://ribbonfarm.com/2009/10/07/the-gervais-principle-or-t...
People don’t get credit for fixing problems that do happen. Maybe possibly in a sales scenario where your fix unblocked the sale. Otherwise nada.
Prevention is hard to sustain because the success is invisible : nobody notice the defects delays or crises that never happened
Chernobyl instantly comes to mind. The giant problem with that particular RBMK design was known to a few people, but nobody fixed it, and if they had... we wouldn't even know, since it would be lost to history.
This is the real problem with performance reviews in companies, which then feeds into opportunities, promotions and compensation. It's just a popularity contest. And this is particularly harmful to people who are neurodivergent, particularly if they're on the autism spectrum, because neurotypical people, who end up making all these decisions, view such people negatively for literally no reason.
You could spin up a team of 6 engineers and have them go away and try some greenfield project. They could come up back in 6 months having shipped nothing. Which of these descriptions fits the facts?
1. The team learned a lot and ultimately decided there was no product-market fit and decided it was best to reallocate resources elsewhere. The learnings from that project will help a whole bunch of other projects across the division; and
2. They failed to ship and get subpar performance ratings for having no impact.
The answer is... both. Or either. How you are treated will depend on how you are viewed by your management chain and that's a social function. We've all encountered people who never shut up about how hard their job is. Often they end up solving problems that they created, often by not listening to anyone that those problems would occur. And they get credit for it.
You could say to people who anticipate problems to stop because it gets you nowhere. Let people fail. If only it worked that way. Instead you'll get blamed for not seeing a problem someone else created because you're viewed as competent but you aren't liked through no fault of your own.
Google seems to be the posterchild for a company that briefly solved this problem and then forgot what made them successful. I am referring to Project aristotle [1], which ultimately determined that psychological safety was the key ingredient in a team's success.
Now amplify all of this with constant rounds of layoffs where the environment isn't just for pay bumps and opportunities but where the cost of failing is losing your income. What you've created is an environment where office politics is everything.
[1]: https://psychsafety.com/googles-project-aristotle/
Human civilization runs on personal sacrifices but money bags will never care about that.
Good thing we’re converting human civilization into money bags at maximum speed, then. Solves all problems elegantly.
This is very true and applies for everything in life.
Very true! Along with it comes with peace/quietness at work so it’s not too bad.
The best hackers are the ones who never get caught.
Two counter-examples:
- Arnold bought a fleet of mobile hospitals that would have been perfect for covid response, but the next governor didn’t want to pay 1% the fleet cost per year to maintain it, so he scrapped it.
- Under Obama, SARS v1 was stopped by US health workers that Trump fired because it was a “bad deal”. In the absence of that team, we got SARS v2, which was renamed to COVID 19.
There’s also the related category of “never blamed for fixing problems poorly, creating even bigger problems”.
Thanks to 9/11, plane cockpits can now be locked from the inside. Now, we have examples of commercial passenger airline pilots locking the doors and committing mass-murder-suicide by plane crash.
For some reason, these stories don’t make the news.
You might want to double check your dates on that SARS claim. Are you talking about the swine flu outbreak?
Its called doing your job.
That depends. It might be going above-and-beyond your job.
We know that the squeaky wheel is the one that gets oiled, and we don't want the wheels to come off, so we need the wheel to squeak loud enough to be heard ;)
Boring is Better
But the same person can get paid for it. So there is an incentive to create, or at least pretend problems.
The jokes around Y2K being a nothing burger always annoy me. Nothing happened because a lot of talented people worked their asses off fixing it.
sounds like my day to day job experience.
Making critical decisions without oversight is just as bad, or maybe worse.
If you frame it this way in a meeting, you will get the attention you want. Don't say I didn't warn you because that comes with a lot of scrutiny you might not want.
> The combined expenditure of U.S. companies on management consul- tants and training in 1997 was over $100 billion
erhm, if this figure is close to true i can see what market ai companies is after.
AI slots quite neatly into the capability trap model, actually.
Which loop it belongs to in the model is left as an exercise for the reader.
Related. Others?
Nobody ever gets credit for fixing problems that never happened (2001) [pdf] - https://news.ycombinator.com/item?id=39472693 - Feb 2024 (424 comments)
Nobody Ever Gets Credit for Fixing Problems That Never Happened (2001) [pdf] - https://news.ycombinator.com/item?id=8940820 - Jan 2015 (50 comments)