The Lindy effect in software The longer a tech has been around, the more robust it is seen as compared to more recent ones, we often talk about a technology’s maturity The C language SQL has been around for a while, https://antonz.org/fancy-ql/ JS libraries seem to come and goes
Complete with rewriting everything in the trendy memory safe quasi-portable language that is faster than (poorly written) C in your personal microbenchmarks.
I work in enterprise, and java still reigns supreme. You see some (very limited) cracks coming from other jvm languages, but that's all. Nobody talks about Rust, rarely about C.
Sitecore example you can easily check, Sitecore XP/XM is still .NET Framework, all new products usually use other programming stacks as extension SDK, mainly Next.js or plain JS/TS.
People have been rewriting software in better languages ever since there was more than one programming language. Eventually people will be rewriting Rust programs in GoombaLang or whatever. Isn't this what we want?
I don't think this is what we want. We want people to maintain and incrementally improve existing software and tooling and not rewrite and change things all the time.
More often the past mistake is rewriting software in a newer language.
Like I've worked places where the business ran off a layer cake of '80s tech, '90s tech that partially replaced the '80s tech, and '00s tech that partially replaced the '90s tech, and were now on their way to launch a big project to replace all that with '10s tech, a project doomed to run out of steam half way through (because legacy code got hands), inevitably leading to a codebase that consists of three failed attempts to rewrite the '80s codebase, and the '80s codebase.
As the functionality of the code was business critical, and no shift in behavior could be tolerated, they're never getting out of this mess, and would have been better off staying on '80s tech.
Sorry, I think the idea that rewrites are good way to achieve more maintenable software is basically always based on a delusion. It is a very common and well understood delusion for programmers who always see the existing code as crap and imagine a beautiful world when they could only rewrite it.
A well understood mistake: https://www.joelonsoftware.com/2000/04/06/things-you-should-...
In the context of free software it is usually also a method to be able to sideline part of the existing user or developer community, which you can not easily justify when making changes the existing project but can be achieved with a rewrite. But the former leads to a honest view of the trade-offs and consequences where the rewrite is a toxic power move.
The issue with not rewriting or rebuilding things is that you miss out on the opportunity for conceptual changes and new approaches to solve problems in more robust and elegant ways. For example, Unreal Engine 6 will depreciate and eventually remove it's existing Actor based scenegraph with an entity component system. This is essentially a rewrite of the core of the engine, and it's a massive conceptual change away from the old style of inheritance-based OOP game programming to something which is much simpler to reason about and provides optimisations which are not possible with Actors. They aren't doing this out of delusion, it's that there are fundamental limitations to their current approach which are not solved by simply refactoring - it requires an entirely different conceptual approach which turns out to be simpler and better in pretty much every way.
The problem with never rewriting is that you miss the chance to discover better ways of doing things. No system is ever built perfectly, if all you ever do is iterate on a bad foundation you're limited by that. Approaching a problem in a different way is only possible when creating something new.
Of course you shouldn't rewrite something just to do it the same way, and you shouldn't rewrite something just because you don't understand it, but I also don't understand the perspective that you should never build something new when there is an existing solution, because it's often only through building out one or more bad solutions that you arrive at a good one. And if you follow such a strict rule, you never end up building anything good.
In Unreal's case, they built the Actor system and have used it for almost two decades. In that time we have discovered better ways of doing things, and it would be a shame to be stuck with an inferior design because Joel said to never rewrite things almost three decades ago...
Ok, you are right. There are situation where something should be re-designed from scratch. I am also not against building something new or better alternatives. But I would still argue that far more often than not rewrites are misguided.
The point of the article is pretty much that you can pick stable technology instead of joining the treadmill of rewriting in whatever the new thing is. When people are rewriting rust programs in GoombaLang, C will still be around. When people rewrite GoombaLang into SmurfLang, C will still be around, etc
If we had followed this more seriously in the past, we would have still stuck on to writing C for enterprise applications and had way too many memory bugs. Aren’t we glad there was a demographic who said no to C and brought the revolutionary idea to use Java instead?
Couldn’t a Lindy enthusiast have gone “umm but isn’t Java too new and shouldn’t we just stick to C which is well trodden and understood??”
It’s easy to write sloganeering articles. But it doesn’t tell me anything specific.
Invoking Lindy is just bias to status quo. I prefer bias to progress but respecting chestertons fence.
I think the Lindy effect is less about making strong arguments about which tool to use in debates, and more about calling out and explaining a real life phenomenon.
I've invoked it in my job mostly to explain to younger developers why learning vim keybindings+terminal git usage while they have the most plasticity is most likely going to be a good bet for the remainder of their career, as editors, operating systems and associated keybindings & UI will change around them much more often than those fundamentals.
It's not a guarantee, and i wouldn't bet my entire business on the Lindy effect, but it is worth reflecting on it as an explanation of something that is paradoxical or not obvious.
> I've invoked it in my job mostly to explain to younger developers why learning vim keybindings+terminal git usage
Interesting - I see it as the opposite: learning the git CLI is pointless. It’s slow, clunky and it doesn’t teach you any of the very interesting inner-workings of Git.
There are much better things to spend time learning, especially if your editor has a native git integration.
You should learn internals as well, it's not mutually exclusive :)
But at some point you have to teach your fingers some movements that will ideally make you fast on as many machines, as many operating systems, as many editors and as many keyboards for as many years of your career as possible.
> Couldn’t a Lindy enthusiast have gone “umm but isn’t Java too new and shouldn’t we just stick to C which is well trodden and understood??”
I think people often misrepresent history because they view it with the benefit of hindsight. People who chose to use Java didn't do it to embrace a revolutionary idea, they were embracing an evolutionary one. It seemed like a natural optimization step, rather than a clean slate.
Most Jave devs were proficient C devs who found the idea of a platform independent C quite appealing.
That's the Lindy effect. C (the essence of it) survives in Java and right up to JS/python/go.
In contrast consider Pascal, Fotran, Perl, COBOL, (and dare I say ... Lisp ?)
The Lindy effect is ultimately a kind of momentum. If that's the case, it seems like it's not the language itself, but rather the 'contracts,' 'interfaces,' and 'standards' that survive longer than the specific implementations a language provides.
Looking at the examples of Lindy that the OP mentioned, they're mostly at the infrastructure level. That's probably because many systems have been built on top of them, and the cost of replacing them is high.
On the flip side, things with weak Lindy effects are likely frontend frameworks or specific libraries. CSS methodologies are a good example of that.
In other words, the deeper something is, the harder it is to change, and as long as that deep language and its ecosystem aren't replaced, it will persist. As a counterexample, Fortran comes to mind—it's still being used today. Fortran has also evolved to exist beneath NumPy and Julia.
Ultimately, I think the core isn't the Lindy effect itself, but rather how many people you can attract commercially, and how many jobs you can create based on that.
In that sense, I think the next-generation language will succeed when it's used to build new infrastructure, and when the cost of refactoring becomes exponentially high. Right now, something that's growing similarly strong is CUDA. Personally, I'm always waiting to see what that language will be.
Terrific writing. Just terrific. Copied verbatim:
Still better than AI, it has personality and not paragraphs of empty prose
you might be confused as to what terrific means
I somehow missed the sarcasm and went to read the article. Was a bit confused for a minute there..
I'd say rust is the new Java, not Go.
Complete with rewriting everything in the trendy memory safe quasi-portable language that is faster than (poorly written) C in your personal microbenchmarks.
I work in enterprise, and java still reigns supreme. You see some (very limited) cracks coming from other jvm languages, but that's all. Nobody talks about Rust, rarely about C.
Which enterprise?
All the job postings I see are for C++ (Annoyingly. Fortran is better). Or Python obviously.
can confirm, some do C++ (more than C, I meant C as a language family in my original post)
Enterprise I've seen, all europe, deliberately vague: Banking, Telecoms, Trains, Insurance
Isn't .NET kinda on quite a rise in enterprise?
Not really, going open seems to only helped Microsoft shops to migrate to Linux for deployments, thus saving on server licences.
I work in a polyglot agency, and the RFPs asking for .NET have gone down, even key enterprise products like Sitecore, have moved away from .NET.
Without answering anything privately. Are you in Asia, Americas or Europe or Oceania?
Germany.
Sitecore example you can easily check, Sitecore XP/XM is still .NET Framework, all new products usually use other programming stacks as extension SDK, mainly Next.js or plain JS/TS.
None of them are nowhere near the Java tooling and ecosystem.
We only touch Go due to containers tooling, and Rust only due to the RIR stuff from Python and JavaScript.
Enterprise consulting is staying with Java, .NET, JavaScript/Typescript, Python, Powershell, SQL, and co.
Naturally Swift and Kotlin if doing mobile without Cordova, React Native and friends.
C or C++ for native libraries, as those are what SDKs support out of the box without additional tooling.
Boring technology for the most part, and usually a few versions behind stuck in some LTS.
People have been rewriting software in better languages ever since there was more than one programming language. Eventually people will be rewriting Rust programs in GoombaLang or whatever. Isn't this what we want?
I don't think this is what we want. We want people to maintain and incrementally improve existing software and tooling and not rewrite and change things all the time.
Case study: Turso[1]
Rewrite the open source SQLite in Rust under an MIT license[2].
Of course, once you complete the core product, there is the extended ecosystem to consider.
[1] https://turso.tech/blog/introducing-limbo-a-complete-rewrite...
[2] https://github.com/tursodatabase/turso
Exactly. Nothing which should exist.
A rewrite doesn't need to actually change the public API or experience using the software.
But by rewriting software, even in the same language we can learn from past mistakes and experiences and create better and more maintainable software.
More often the past mistake is rewriting software in a newer language.
Like I've worked places where the business ran off a layer cake of '80s tech, '90s tech that partially replaced the '80s tech, and '00s tech that partially replaced the '90s tech, and were now on their way to launch a big project to replace all that with '10s tech, a project doomed to run out of steam half way through (because legacy code got hands), inevitably leading to a codebase that consists of three failed attempts to rewrite the '80s codebase, and the '80s codebase.
As the functionality of the code was business critical, and no shift in behavior could be tolerated, they're never getting out of this mess, and would have been better off staying on '80s tech.
It's good for job security that's for certain.
Sorry, I think the idea that rewrites are good way to achieve more maintenable software is basically always based on a delusion. It is a very common and well understood delusion for programmers who always see the existing code as crap and imagine a beautiful world when they could only rewrite it. A well understood mistake: https://www.joelonsoftware.com/2000/04/06/things-you-should-...
In the context of free software it is usually also a method to be able to sideline part of the existing user or developer community, which you can not easily justify when making changes the existing project but can be achieved with a rewrite. But the former leads to a honest view of the trade-offs and consequences where the rewrite is a toxic power move.
The issue with not rewriting or rebuilding things is that you miss out on the opportunity for conceptual changes and new approaches to solve problems in more robust and elegant ways. For example, Unreal Engine 6 will depreciate and eventually remove it's existing Actor based scenegraph with an entity component system. This is essentially a rewrite of the core of the engine, and it's a massive conceptual change away from the old style of inheritance-based OOP game programming to something which is much simpler to reason about and provides optimisations which are not possible with Actors. They aren't doing this out of delusion, it's that there are fundamental limitations to their current approach which are not solved by simply refactoring - it requires an entirely different conceptual approach which turns out to be simpler and better in pretty much every way.
The problem with never rewriting is that you miss the chance to discover better ways of doing things. No system is ever built perfectly, if all you ever do is iterate on a bad foundation you're limited by that. Approaching a problem in a different way is only possible when creating something new.
Of course you shouldn't rewrite something just to do it the same way, and you shouldn't rewrite something just because you don't understand it, but I also don't understand the perspective that you should never build something new when there is an existing solution, because it's often only through building out one or more bad solutions that you arrive at a good one. And if you follow such a strict rule, you never end up building anything good.
In Unreal's case, they built the Actor system and have used it for almost two decades. In that time we have discovered better ways of doing things, and it would be a shame to be stuck with an inferior design because Joel said to never rewrite things almost three decades ago...
Ok, you are right. There are situation where something should be re-designed from scratch. I am also not against building something new or better alternatives. But I would still argue that far more often than not rewrites are misguided.
The point of the article is pretty much that you can pick stable technology instead of joining the treadmill of rewriting in whatever the new thing is. When people are rewriting rust programs in GoombaLang, C will still be around. When people rewrite GoombaLang into SmurfLang, C will still be around, etc
You can pick C, but that doesn't mean it's a good option to write something in it
Huh? Rust is nothing like Java. Java is a dynamic type-erased garbage-collected managed abomination with FizzBuzzEnterpriseEdition standard library.
If we had followed this more seriously in the past, we would have still stuck on to writing C for enterprise applications and had way too many memory bugs. Aren’t we glad there was a demographic who said no to C and brought the revolutionary idea to use Java instead?
Couldn’t a Lindy enthusiast have gone “umm but isn’t Java too new and shouldn’t we just stick to C which is well trodden and understood??”
It’s easy to write sloganeering articles. But it doesn’t tell me anything specific.
Invoking Lindy is just bias to status quo. I prefer bias to progress but respecting chestertons fence.
I think the Lindy effect is less about making strong arguments about which tool to use in debates, and more about calling out and explaining a real life phenomenon.
I've invoked it in my job mostly to explain to younger developers why learning vim keybindings+terminal git usage while they have the most plasticity is most likely going to be a good bet for the remainder of their career, as editors, operating systems and associated keybindings & UI will change around them much more often than those fundamentals.
It's not a guarantee, and i wouldn't bet my entire business on the Lindy effect, but it is worth reflecting on it as an explanation of something that is paradoxical or not obvious.
> I've invoked it in my job mostly to explain to younger developers why learning vim keybindings+terminal git usage
Interesting - I see it as the opposite: learning the git CLI is pointless. It’s slow, clunky and it doesn’t teach you any of the very interesting inner-workings of Git.
There are much better things to spend time learning, especially if your editor has a native git integration.
You should learn internals as well, it's not mutually exclusive :)
But at some point you have to teach your fingers some movements that will ideally make you fast on as many machines, as many operating systems, as many editors and as many keyboards for as many years of your career as possible.
This article from yesterday -
Good Tools Are Invisible
https://news.ycombinator.com/item?id=48858121
In my opinion, software tools are different from software libraries.
Knowing RDBMS concepts and SQL is one thing, vim and git are something else altogether.
I agree and the article uses Lindy not in descriptive but prescriptive level.
For prescriptive, I would use Chesterton's Fence https://fs.blog/chestertons-fence/
I really like the concept of novelty budget: keep most parts boring and use new stuff only where you can gain an advantage from it.
A lot of the article is extremely useful for people creating software libraries, IMHO.
Under the section -
"Applying the Lindy effect to software engineering"
The suggestions are -
- Prudent Adoption
- Stick to Proven Foundation
- Plan for Longevity
- Embrace Evolution, Not Revolution
This is just conservatism from first principles. Why doesn’t it apply to sociology? LGBTQ, equality rights and so on? Lot more at stake.
> Couldn’t a Lindy enthusiast have gone “umm but isn’t Java too new and shouldn’t we just stick to C which is well trodden and understood??”
I think people often misrepresent history because they view it with the benefit of hindsight. People who chose to use Java didn't do it to embrace a revolutionary idea, they were embracing an evolutionary one. It seemed like a natural optimization step, rather than a clean slate.
Most Jave devs were proficient C devs who found the idea of a platform independent C quite appealing.
That's the Lindy effect. C (the essence of it) survives in Java and right up to JS/python/go.
In contrast consider Pascal, Fotran, Perl, COBOL, (and dare I say ... Lisp ?)
The Lindy effect is ultimately a kind of momentum. If that's the case, it seems like it's not the language itself, but rather the 'contracts,' 'interfaces,' and 'standards' that survive longer than the specific implementations a language provides.
Looking at the examples of Lindy that the OP mentioned, they're mostly at the infrastructure level. That's probably because many systems have been built on top of them, and the cost of replacing them is high.
On the flip side, things with weak Lindy effects are likely frontend frameworks or specific libraries. CSS methodologies are a good example of that.
In other words, the deeper something is, the harder it is to change, and as long as that deep language and its ecosystem aren't replaced, it will persist. As a counterexample, Fortran comes to mind—it's still being used today. Fortran has also evolved to exist beneath NumPy and Julia.
Ultimately, I think the core isn't the Lindy effect itself, but rather how many people you can attract commercially, and how many jobs you can create based on that.
In that sense, I think the next-generation language will succeed when it's used to build new infrastructure, and when the cost of refactoring becomes exponentially high. Right now, something that's growing similarly strong is CUDA. Personally, I'm always waiting to see what that language will be.
Two words: Visual Basic.
(2023 - hopefully the post lasts another 100 years!)
[flagged]