Author points to TigerBeetle as an example, which I wasn't familiar with, so I went down a rabbit hole.
Wow. Yeah. If you're writing a financial ledger transactions database from scratch in Zig, where safety and performance are your top two goals, so that you can credibly handle a million transactions per second on a single CPU core, well, damn right you can't afford to take any risks on introducing any dependencies. Makes absolute perfect sense to me (no sarcasm).
But the average developer, writing most line-of-business CRUD systems, well, quite honestly they're not that strong to begin with. Most dependencies may be bug-infested garbage, but they're still higher quality compared to what most developers are capable of. Most companies are bottlenecked by the caliber of talent they can realistically recruit and retain; and internal development standards reflect the average caliber on the payroll.
So like most advice you read on the internet, remember, everybody is in different circumstances; and polar opposite pieces of advice can both be correct in different circumstances.
+100, context is the key thing, both TigerBeetle and rust-analyzer have strong culture of how the things are done, but the _specific_ culture is quite different, because they solve very different problems.
That being said, I _think_ you might be pattern-matching a bit against the usual dependencies good/dependencies bad argument, and the TFA is on a different level. Note the examples of dependencies used: POSIX, ECMA-48, the web platform. These are not libraries, these are systems interfaces!
Dependency on a library is not a big deal --- if it starts to bite, you can just rewrite the code! But dependency on that underlying thing that the library does is usually something that can't be changed, or extremely costly to change. Answering "No" to "is doing X in scope for my software?" is powerful
To refer to the sibling comment, if there's a team whose core business is writing matrix multiplication code, they _can_ use random library #24 for something else, but often, if you apply some software design, you might realize that the product surrounding matrix multiplication _shouldn't_ handle concern #24. It doesn't mean that concern #24 isn't handled by anything at all, rather that we try to find _better_ factoring of the overall system to compartmentalize essential dependencies.
The problem I have with this line of thinking is that it only works when you're talking about bottom of the barrel developers.
The tech world has this tendency to dumb down every advice so it caters to the lowest common denominator, but I honestly have never worked in such an environment where developers couldn't learn to duplicate the functionality of shitty dependencies while at the same time fixing the problem.
And as much as people love to mock CRUD, bad abstractions can be a horrible drain on teams doing CRUD work. And even the popular stuff can often suck and be a timesink unless you're doing absolutely basic stuff that's just repeating the tutorial.
I've seen a few people from the bottom of the barrel outside on the real world...
The funny thing is that no advice is dumbed-down enough for them. They will take anything you can say and implement wrongly.
That idea that if you dumb-down everything, it will be helpful is just plain bullshit. People that can't duplicate a dependency just can't choose the right dependency either, and if you force them to use some good one, they will use it wrong.
> but I honestly have never worked in such an environment where developers couldn't learn to duplicate the functionality of shitty dependencies while at the same time fixing the problem.
I have absolutely seen this. It’s not a matter of them being bad developers, but no one is an expert in all things and gaining expertise takes time which costs money. All dependencies have a lot of trade-offs baked in that must be re-evaluated if you decide to go the NIH route, and that’s usually where expertise is needed. If you lack the expertise you will still be making trade-offs, but you will lack the knowledge to foresee the long term consequences of these decisions.
In my own opinion core business differentiators are where engineering resources should be used. That’s not to say a dependency that is more generic than the problem your business is trying to solve should never be pulled in house, but the decision to do that shouldn’t be taken lightly as it will increase the burden of learning, maintaining, documenting, testing, etc for a team that is being judged on the success of business objectives. To put it another way, the business will be hesitant to give you time to work on something that is only tangentially related to their objectives and quality can and usually will suffer which will put you in a worse position.
I think software development has some unique characteristics that makes NIH a problem. If you’re an electrical engineer you will never be given the resources to make a custom microcontroller or a new way of manufacturing a PCB unless that is what your company is selling; the upfront costs would be staggering and that would not be allowed. The limitations of the technology available is simply that, a constraint on what the business can do given current tech. Perhaps there is an opportunity for something better, but that would be best served as a new company focused on providing that solution if it can be done in an economically viable way. Software doesn’t have these massive upfront manufacturing costs that are obvious like the example above, but they still exist.
It doesn't even have to be related to quality of developers. Whatever tool chain or product you use, you're using someone else's dependencies. In some places more than others, but most people aren't implementing their own matrix multiplication code, and the ones that are, aren't implementing random library #24 which isn't in their core business. So this whole discussion happens on black and white terms when in fact people only care if they have regulatory obligations, or they have a particular personal interest in implementing something, or they got attached to that idea for a specific library. But nobody is applying this fully or they'd still be at the beach collecting sand.
We apply this pretty much fully at TigerBeetle [1].
Reason being that TigerStyle is not only about shipping safer software, but shipping it in less time [2], factoring in the Total Cost of Ownership to end users. In other words, not only optimizing our development time, but also all the time spent in production with subsequent fixes.
Vasco! In a past life I've implemented (and tuned) things like Cauchy by hand, but we don't use any of that in TigerBeetle. Nevertheless, the experiences and learnings around that were valuable for me personally, and I don't think I'd have been able to do TigerBeetle without putting in those hours. Wish you all the best, man!
> But nobody is applying this fully or they'd still be at the beach collecting sand.
To this point I posit almost all of the "this good, that bad" type of posts are just "at my point on the abstraction continuum, this thing is better than that thing" without taking into account any context.
For most assembly is too low level for their context/use case, but for some, the C programming language is too HIGH level, as it was originally designed to be (but not for me).
> But the average developer, writing most line-of-business CRUD systems, well, quite honestly they're not that strong to begin with.
This is a wild take. Developers don't usually get to pick the system they work on, resource constraints exist during development time. We make tradeoff decisions every day and pulling in a "cheap" dependency will often make the difference between a shipped, working piece of software and plain "nothing".
You seem to be fairly little involved in actual software development, considering how generic and wrong this take is.
The problem, IMO, is that the trade-offs seem to always land on the side of “what ships faster.” That may well be the fault of PMs rather than SWEs, but either way, it doesn’t usually make for a sustainable product.
Isn't the industry still growing pretty quickly? If five years is enough to count as "senior", most won't be that good just because they haven't had time to get good yet. And any segment that's big enough or low enough margin won't be able to escape that average.
When looking at CVs and interviewing actively, it's honestly very easy to believe the quoted claim. Of course I work with some very skilled folks on line-of-business CRUD too so who knows.
I also don't quite like TigerBeetle as an example, because it is very new startup. They are pre-1.0 and just a bit more than a year since production release. It's way too early to tell whether their approach actually paid off.
While the company is 3 years old, TigerBeetle as an open source DBMS project actually started in 2020.
What we mean by "production" is also typically a bit different to what is typically meant.
For example, TigerBeetle is one of the first DBMS's to withstand an explicit storage fault model (i.e. disk corruption across all replicas). So the tolerances are a bit tighter for TigerBeetle, also with things like static memory allocation (cf. NASA's Power of Ten Rules for Safety-Critical Code).
Beyond this, when TigerBeetle was designed, it was (along with FoundationDB) one of few DBMS's to invest in a Deterministic Simulator in-house. Most DBMS's before 2020 were typically not designed from the ground up for Deterministic Simulation Testing (DST) and so now can only use Antithesis if they want to get into DST.
However, TB invests heavily in both, not only an external DST audit function, but more importantly, also our own internal DST audit function, which we designed and built entirely inhouse, and which can do considerably more damage to TigerBeetle because it's protocol-aware and understands how to push TB's consensus and storage to the breaking limit. In other words, not one DST simulator, but two, with “the VOPR” (the name for TB's own simulator) tailored to our fault models, and able to reach into any part of the code, across any machine in the simulation, and verify expectations. For example, do things like verify cache coherency between user space page cache and simulated storage engine.
Finally, if you're interested in the results of whether this rigid TigerStyle approach pays off, Kyle Kingsbury's Jepsen audit of TigerBeetle was one of the longest Jepsen engagements to date, and these were the findings: https://jepsen.io/analyses/tigerbeetle-0.16.11
I work at a medium company that is in a comparable sector implementing comparatively performant and externally constrained things pretty much wholly in C and pretty much wholly with our own dependencies with external dependencies being mostly located in ops, testing/qa and other stuff not in the critical path of what the people who pay us want the products to do. For us it's not about "built here" it's about control and uniformity. The company is a decade old and very profitable for its headcount though not subject to wild delusions about future hockey sticks because of what we build and for who.
So the approach clearly can work. It won't kill you. But it's also unclear the degree to which this approach contributed to vs detracted from our success since it's not like there's a control company that did it the other way you can compare to. It takes constant effort to do things this way, but so does managing dependencies. While unilateral control, and the responsibility for that, that comes with writing your own dependencies is a key aspect of how we do what we do with the headcount we do it's not like it's part of our marketing. It's just a decision that was made that we're sticking to because it seems to work for us. It's not some silver bullet, it takes effort to do well, like anything else done well.
I have worked at huge companies where they were biased toward using existing stuff where it existed and tweaking it just the bare minimum needed and that worked very well for them too and definitely had its pros too.
I pointed to tigerbeetle because they're one of the few companies I know of who 1) have a philosophy about dependencies and 2) talk about it openly in an easy to quote document.
I'm not sure if I succeeded, but I wanted to get across that I mean "dependency" in a very broad sense. Not just libraries, or services you interact with, but also what you need to build and deploy.
It wasn't a call to just not use dependencies. It was a call to think about them and be clear what you're depending on. Tigerbeetle have done that and given reasoning.
From the beginning, TigerStyle was intended not only for TigerBeetle as infrastructure, but for software (up and down the stack) in general.
Nobody wants buggier, slower software. And so TigerStyle as a methodology was codified primarily to address this problem: How to write “higher quality software in less time”? [1]
Before designing TigerBeetle, I was finding the principles more and more valuable, regardless of whether I was doing frontend or backend work, and so brought them into TigerBeetle.
Of course, software higher up is likely to use some dependencies, but there's always a cost, and this principle (“do more with less”), at least in my experience, tends to prove valuable when applied.
NIH is amazing as long as you are realistic about what you are taking ownership of.
For example, the cost of maintaining a bespoke web frontend "framework" that is specific to your problem domain is probably justifiable in many cases.
The same cannot be said for databases, game engines, web servers, cryptographic primitives, etc. If you have a problem so hard that no existing RDBMS or engine can support, you should seriously question the practicality of solving your problem at all. There's probably some theoretical constraint or view of the problem space you haven't discovered yet. Reorganizing the problem is a lot cheaper than reinventing the entire SQLite test suite from zero.
> The same cannot be said for databases, game engines, web servers, cryptographic primitives, etc.
It absolutely can in many cases.
- Many applications don't need much more of a database than files plus perhaps a runtime index. Spawning a database server or even embedding SQLite can be overkill in many cases.
- Most games would be better off with a bespoke game engine IMO, especially those made by small teams. The only real advantage to using an established engine is the familiar asset pipeline for replaceable artists but you pay for that with a huge overhead. Remember that custom engines or at least heavily modified ones maintained by the game developer used to be the norm - "just use Unity^WUnreal" is a relatively recent trend.
- There is no real difference in complexity between a simple web server and FastCGI application.
- Not all uses of cryptographic primitives are a security issues. If you calculate a hash to check against corruption you can implement that yourself (or copy an existing implementation) instead of pulling in a huge library. If your use cases is decrypting existing data offline then again you don't really care about cryptographic security only about getting the correctly decrypted output.
And there are many more cases like these. Learned helplessness when it comes to "hard" subjects helps no one.
Just because some existing solution "solves" your problem doesn't mean its the best or most efficient way to do it.
The latter already adds a lot of complexity, especially if the application has to scale up or out. If scaling up, the index will increase in complexity and re-implement some clever algorithms that have already been solved by the other. If scaling out, you'll need some locking mechanism if multiple processes can write. It quickly becomes logical to switch to an established solution then.
But the choice becomes more logical then; if one picks a scalable database or whatever without knowing they're going to need it, that's a lot of wasted effort.
> - Most games would be better off with a bespoke game engine IMO, especially those made by small teams.
I disagree; writing a game engine, even for simpler "indie" graphics, requires specialized knowledge and a dedicated engineer, whereas with off-the-shelf engines, non-programmer / creatives can build a good game without in-depth knowledge. A small team needs to be especially careful with what they choose to spend their time and money on, and solving a solved problem like building a game engine is a huge investment. Even big studios struggle with this, see also the dramatic release of Cyberpunk 20something by long-time own-engine-building veterans CDPR. They're switching to Unreal for Witcher 4.
Any mature game engine also comes with mature developer tooling. Think profiling, debugging, assets management, map editing etc. That’s a lot of initial hurdles to overcome before you can even start working on your game ideas. For indie / solo developers who tend to be, you know, driven by one’s creative urges, this is not just a huge time sink but also a good way to ruin your motivation by never moving to the part where you get to actually create the game. When your time and energy is constrained you need to choose your battle and for indie developers this often means forgoing the technical part (unless you really need to) and focus more on the game part.
I have seen many people say “there are more games engines written in Rust than games written in Rust” and I wonder if what happened is that the software developers fell for the allure of building their whole stack from scratch and ended up finding out it sucks to actually develop a game for an engine with no tooling and “making an entity inspector that lets you click on an enemy and edit its attributes” isn’t exactly the sexy part of game development.
The trap of "building a game engine" vs "making a game" has existed long before Rust.
When games were really simple, so simple that "making a game" vs "making an engine" had no real difference (say, building a game for 8 bit home computers) then it wasn't a big deal. The concept of "games engine" didn't really exist back then, or was uncommon at least.
But nowadays? A small indie team, or a single dev, are likely to go down the rabbit hole of making their own game engine, forced to solve a lot of problems that are already solved in existing engines, rediscover the wheel, and get frustrated because the step of making the game never actually comes.
I think it makes more sense to use a pre-made engine, unless the game really is so different this isn't an option, or the devs really enjoy making engines (which is valid, and you learn a ton of cool things, but is also a hurdle in the way of actually making a game).
Every games programmer enthusiast I know has a half-made, abandoned, badly performing engine with which they've made a couple of demos or a barely functional unfinished game... (I'm also guilty of this, but given my age, mine was written in C++!).
In which part is Noel disagreeing? He didn't write his own engine either, he just didn't use one (but didn't write everything from scratch either, note he uses lots of game/graphics frameworks, which in a sense is like using a "meta engine").
He's also making a 2D game, which traditionally is closer to game making of old (think 8 bit home computers).
Finally, he's not arguing against engines like Godot or Unreal, he says those are acceptable but he just didn't need them. Much like he doesn't need Windows and finds coding with Linux more comfortable.
PS: out of curiosity, which one is the popular game you mentioned? Looking through his portfolio I see lots of interesting games I'd like to play :)
> He didn't write his own engine either, he just didn't use one
It's hard for me to take in good faith an interpretation of that article that determines Noel "didn't write his own engine," but in case a couple direct quotes would help:
"I don't have much to say about assets, because when you're rolling your own engine you just load up what files you want, when you need them, and move on." ... "my City of None engine"
Noel literally describes his workflow as "rolling your own engine" and has a name for the engine he wrote.
> He's also making a 2D game, which traditionally is closer to game making of old
Please read the context of the thread you're replying to, which is arguing against writing your own engine even for games with "simpler 'indie' graphics," specifically called out.
> Finally, he's not arguing against engines like Godot or Unreal
He's not arguing against anything. He's advocating for writing your own engine.
Ok, ok, it's indeed an engine of sorts. And I am arguing in good faith.
I'm not familiar with Celeste and I cannot be sure it's one of the most popular indie games of the last decade (I read that it was a hit, good for Noel because it seems like a cool game), but regardless, it's a very specific kind of game! Mostly of the kind I've argued "writing the game" and "writing the engine" is almost the same thing. Nobody is making the argument you must always use an existing engine; only that it's what makes sense as the default option.
> Please read the context of the thread
If you've read my other comments, you know I'm aware of the context of the thread.
> He's not arguing against anything. He's advocating for writing your own engine.
Oh please. Advocating for a thing and arguing against not doing the thing are two sides of the same coin.
> - Most games would be better off with a bespoke game engine IMO, especially those made by small teams. The only real advantage to using an established engine is the familiar asset pipeline for replaceable artists but you pay for that with a huge overhead. Remember that custom engines or at least heavily modified ones maintained by the game developer used to be the norm - "just use Unity^WUnreal" is a relatively recent trend.
I think we can take Square-Enix as a case study: Final Fantasy XV and Final Fantasy XVI both used their own engines (though XVI used a modification of XIV's engine, from what I hear) and both took a long fucking time to make. Granted, XV had many issues outside of just building a new engine, but it definitely added one more problem onto the pile. My understanding is that Final Fantasy XVI's development cycle was pretty smooth, it still took 7 years from the launch of Final Fantasy XV!
Final Fantasy VII: Remake and Final Fantasy VII: Rebirth both used Unreal Engine 4. While Remake had a long development cycle, Rebirth was done in 3 years and contained a much bigger world!
I do totally get that part of the argument is "reuse the same damn engine and tooling," and I totally agree with that, too. It's just a bit easier overall when you can reach out to a community with your problems instead of learning every problem for yourself.
Most games that use an existing engine get like 60% on metacritic and ship like less than 1000 copies and are a generic flop. “Shipping no matter what” is a lot less valuable in games.
You think the only game you will lose are generic Unity shovelwares when in reality you will also lose good games made by people with good sense in game making but likely not the resources or technical expertise to make their own engines. Think Cities Skylines (Unity, the studio had only a dozen people in total when it released), Dusk (Unity), Undertale (GameMaker), Spelunky (GameMaker)..
Another comparison is game modding which is essentially using the base game as the engine of your modded game. Do you think Team Fortress or Counter Strike or DOTA would have been made if their creators thought they have to build their own game engine?
Lowering the friction to make a game means more games get made. Yes there will be a flood of bad ones, but we get good one too.
The adventure of writing a custom DBMS, game engine, or other complex and advanced software classes can be justified by 1) knowing that no existing solution is adequate but 2) it is possible to write something suitable and 3) it will be useful enough to justify the cost.
For example, to insist on the example of Tigerbeetle, 1) no current popular or specialized DBMS is particularly optimized for ledger use, 2) but suitable techniques are well understood; I'm not sure how often 3) it will be preferred to a more general purpose DBMS.
> - Most games would be better off with a bespoke game engine IMO
You have it backwards.
Most games won't benefit from a bespoke game engine at all. Most games should use an existing engine (not necessarily one of the big names, just one that works) if their goal is to actually deliver a game.
Some games are so unique they won't benefit from an existing engine, and only then the dev team must take the extra step (and eat the delay) of building their own engine.
I mean commercial games of course. Experimental/hobby devs are free to do whatever, since in that case it's the journey and not the destination that matters.
Why are there so many different database engines then? Is each instance of a different database out their an instance of NIH?
The answer is of course that every computer system of any complexity has to make trade-offs - do you need constraints, do you need scalability, do you need concurrency, security? Is your data of a specific form that benefits from a particular compressible storage form, is it optimised for write once read many or read/write etc etc..
In terns of the thrust of the main article - which is about the cost of dependencies - I find that I'm typically much happier to take a dependency if that dependency is very keen on minimising it's own dependencies! - ie I don't want to take on a whole forest of stuff.
In my view many automatic dependency management systems have helped create the mess that they claim to try and solve, and using careful manual dependency management can help cut down the burden.
If you are running into a problem that is bringing your site down and it's truely a problem/bug with a commercial database then you are in trouble and unlikely to see a quick fix as these products are complicated and while the problem might be existential to you, it's not to the bigger vendor.
However, it's much more likely it's a configuration issue - ie you didn't understand the product properly and have configured it incorrectly - and here support can really help.
The point I'm making above is not that you should be writing fully fledged all singing and dancing databases - but that you might be able to get away with something very simple, more reliable and faster in your specific use case.
If on the other hand your use case is complex and a good fit for something pre-existing - then sure you shouldn't be writing it yourself.
i can think of two reasons for using a third-party dependency
1) a dependency on a third-party service provider that publishes the dependency. So long as that service provider is current, the dependency should be maintained
2) short-cut to code i don't want to write
I have no arguments with (1), there's a business reason and the lifecycles should match. However, I should still expect major version breaking changes in order to keep my application running. For (2), the wins are less clear, more dependenent on the perceived complexity of what I can avoid writing.
Taking on any kind of dependency means that someone else can dictate when I need to spend time updating and testing changes that don't add anything to my product. Taking on a third-party dependency is always taking on a responsibility to maintain a codebase or the risk of not doing so.
I would argue that 2 is the much more important reason. A dependency is worth it if it saves you a lot of time building the same thing. For example, if you wanted to ship a general computing device, you should probably run Linux on it rather than building your own OS from scratch, because you'll save literal years of work. Even if Linux had stopped being maintained, this would still be the right call compared to building your own (of course, it would be even better to choose some other OS still being actively maintained, if any is available).
I wish we could have this discussion without going into exaggerated extremes like "Linux". The OS rarely gets into discussions of rewriting dependencies.
I recently had to rewrite a table component and it took about an hour with extra performance and it became faster-to-use and tailored to my company's needs. One hour, give or take! With measurable performance improvements, and it's simpler to use for our use cases, and it doesn't prevent anyone from using the old one.
Comparing one-hour or even one-day work with rewriting Linux is ridiculous fear-mongering.
But this is exactly my point. Some dependencies don't actually save you any time, they're trivial to write yourself, so there is no point in getting them (perhaps the most absurd and infamous example of this being left-pad). Other dependencies are extremely hard to write, so finding a version someone else built, even if they are not maintaining it anymore, is worth it.
Theres a third one, when it comes to compliance and security tools, you don't want to build it even if you can because.
1. It is a liability
2. There is trust deficit during audit and other events. If audits are internal only sure you can build it but when it is 3rd party audited, auditors often know the product and familiar with the features.
> auditors often know the product and familiar with the features.
or what if you chose a dependency for which this auditor is unfamiliar with, and so it takes even longer (where as if you NIH, you'd have the full source and thus can give the auditors the materials to audit).
I am not sure most auditors work on that level of detail. If it is a library they don't consider audited yet, they might just call it a day and make a statement about your code excluding the dependencies they are not familiar with. Otherwise you would have to pay for all third party dependency audits, which no one else paid for yet or the auditor is not aware of someone else having audited already.
I have also seen this narrative. If things go south in the org. btw we were using product A for doing our workflows, that replaced if with product B for the same, now everything is going to better.
This is where using boring languages like Go and Java come into their own: less frequent breaking changes.
There are some languages where communities aim for libraries to be stable for several months, and there’s others that think in terms of a decade or longer.
There is another solution which is to just retain the source for your dependencies and bring them in-house if needed. You still may need to abide by licenses, and many things shouldn’t need to rely on external deps, but I’ve seen a lot of wasted time that continues because we came up with our own way of doing something that others do better, and that can be difficult to get rid of because of unwarranted team pride or familiarity in the homegrown code or not wanting to trust the lone dev that says try this instead.
1) even though reality has proven us wrong time and time again, we can just not look at the dependency too closely and just act as if it's written and maintained by competent, caring people and is of highest quality. No worries!
2) in case shit hits the fan, let's assume worst case and there is a vuln in the dep and you get hacked... It's somebody else's fault! \o/
> 2) in case shit hits the fan... It's somebody else's fault!
A contrived example but, good luck explaining to the lawyers that openssl had this bug that caused all your customer data to leak and your company is being sued. If your motto for dependencies are “we can just not look at the dependency too closely and just act as if it's written and maintained by competent…” I’m reasonably sure someone is getting fired in that situation if it doesn’t sink the entire company.
Move fast and break things isn’t exactly a viable strategy in all industries.
Now as I said openssl was a contrived example but what if instead it was your ORM that didn’t use templated queries but rather just did string interpolation and there was an SQL injection attack. Considering this is still one if the most popular vulnerabilities on the web someone is messing stuff up somewhere and you are blindly like hoping it isn’t in the 10k lines of ORM library you pulled in instead.
You fairly quickly run into problems a RDBMS can't solve. They're above all incredibly hamstringed by being built around supporting concurrent mutation of the data set and mutable datasets in general.
If you don't need that, even a fairly naive index implementation will get orders of magnitude more performance out of a computer than you will with a RDBMS if you can assume the data is immutable.
I'd tend to argue the opposite. Unless you have a problem that very explicitly is a poor match for an RDBMS, they are the easiest, most reliable, and most flexible way to manage data, separate your data from your application, and adapt or add functionality as new requirements come along.
There are plenty of RDBMS here (wikipedia lists some 100+ of them), there are plenty of problems most of them can not solve, but some of them do solve.
These people considered practicality of their solution and went forward doing the implementation.
Absolutely. A bunch of them also predated the www, or started just when the www was gaining popularity, meaning that information on possible products might be more limited than it is today. Some have narrow use-cases, and some might have started as really narrow in terms of scope, but then ballooned after gaining popularity, sales (and feature requests), and some probably started because the alternative was deemed too expensive compared to just making it in-house.
I think the main point bob1029 was trying to make is that it can be worthwhile doing somehting in-house if the alternatives doesn't match the use-case, are too expensive or whatever else - but that you seriously need to consider if your architecture is the best way to solve the problem before going down that route.
> as long as you are realistic about what you are taking ownership of
Including training and onboarding. There's a reason "popular" libraries, languages, patterns, etc. are popular. One "win" is the network effect when looking for talent - they can hit the ground at least jogging.
Of course these questions are hard and if you are asking them you may need a general db.
The whole industry is wrapped up in these bizarre ideas that for the slightest requirement they are going to add just one more abstraction layer and it will do everything and the kitchen sink. But sometimes the requirement is simple and doesn't deserve solutions to anticipated problems and other times it does and is worth looking for an existing solution. Rarely the problem is complex and deserves a custom solution and then one should probably make a new startup for the one yak shave, etc..
The technology is only part of the problem. If you aren't happy with any existing database, throwing even more technology at it is definitely not going to help - statistically speaking. You're up against millions of engineering hours in this space.
At some point, we need to get up from the computer and go talk to the business and customer about their problem in terms that don't involve anything more sophisticated than excel spreadsheets. There is definitely some aspect you missed if you think you need to build a fully custom technology vertical to get them to pay for a solution.
Of course, all of this advice is pointless if what you are doing is for amusement or as a hobby. I think a lot of arguments about technology would be diffused if we prefaced with our intended applications.
> You're up against millions of engineering hours in this space.
Tangential, but the fact that many orgs choose to not use native DB features like foreign key constraints, saying that “they’ll handle it in application code,” has always seemed like the pinnacle of hubris to me. You’re going to ignore some of the most battle-tested pieces of technology, and recreate it poorly instead? Quite the decision.
While there are valid reasons to not want to use FKCs, IME much of the time when you peel apart the layers of why someone chose not to, you find that they made several erroneous assumptions that led them to that conclusion. Other times, it’s that they simply didn’t know something existed, because they never read the docs. Windowing functions are a good example: RDBMS can do quite a bit of manipulation to data, rather than just sending huge result sets and having you parse them. There are trade-offs, of course, but you should at least know that it’s even possible before choosing to ignore it.
I thought we were talking about NIH, H is for here. If you can't scope your requirements then obviously you need to import something a bit larger than the whole world to solve the whole world's problems.
Dependencies introduce risks, but not using them at all puts you at a competitive disadvantage against those who are using them and thus achieve faster development time and time to market.
What you need is a process to manage dependencies:
1) Only consider open-source dependencies.
2) Introducing new dependencies requires a review. Not just a code review on the pull request introducing it, but checking the license, estimating how much work it would be to rip out, auditing it for history of security vulnerabilities or bugs, whether it is live and still gets updates, how vibrant the community is, and so on.
3) If possible, review your dependencies for possible security issues. Doing this at scale is expensive and the economics of this are still an unsolved problem (I have my ideas: https://blog.majid.info/supply-chain-vetting/).
4) Do not adopt a dependency you are not willing and able to take over the maintenance of, or fork if necessary. At a minimum, it means you have built it from source at least once, not just used binary packages maintained by someone else.
5) Preemptively fork all your dependencies. People can and do delete their repos out of pique, see left-pad.
Both 4) and 5) are very important, but often forgotten.
Even for my own personal (small) projects, i've gotten hit with problems when i take an extended leave of absence and then return to a project, only to find my dependencies have become either completely outdated and unusable, or the repo was entirely deleted (with only a remote copy to build with).
I've since adopted the "fork" method; the dependency's source is forked (privately), and get the dependency built; this is recursively done with their dependencies (stopping at the language level libraries), just to get familiar with their build chain. Only then, will i feel good enough to add the dependency to my project (and use their publicly released artefacts from their publicly hosted library repository). It's a bit of work, but i think this effort is spent upfront, and removes future effort if the project lives long enough to see the ecosystem move/change directions (and you dont want to follow).
Sometimes i do find that source libraries (rather than pre-built binaries) to be better in this sense. I merely need to fork and link the sources into the project, rather than have to learn anything about build chains of dependencies...
For #4, while I agree sometimes, I also recognize that SQLite (and other similar types of dependencies) are 100% going to outlive the product I'm building. So it's a bit arrogant to say my product is going to outlive them. I'm not going to build the Linux kernel just because it's a dependency.
I think it still applies if consider you can build SQLite from a fork of the source and it probably needs no maintenance. In some ways, you could consider the SQLite package that ships with a given distro/OS a mini-fork
Still, surprises happen. Key contributor(s) could become ill, license changes can happen, ownership/stewardship change. Although super popular projects you can _probably_ rely on someone else forking
Iirc one of the MariaDB engines was maintained by some singular dude that would go offline for months at a time and maintained it as a hobby.
4 and 5. Not at least once, all of your code should be at the very least buildable with a network cable yoinked out, preferably without any binary artifacts, but that is not always possible.
With 5, forking may be a bit excessive, as keeping your fork up to date adds a big maintenance burden. Vendoring dependencies (just pushing them into git, git LFS, or operating a caching proxy) is valid though, especially for long-lived software projects. Maybe less so for NodeJS based ones as their dependencies use many small files, but there's Yarn and maybe PNPM that will keep dependencies in more convenient archive files instead.
Being in the energy sector dependencies is something we intentionally avoid because we'd actually have to go through and review changes. What has helped this immensely is AI code assistance. One of the primary uses is to use it to write CLI tools for code and config generation in the tool chain. All of it is around making your life easier, without pulling external dependencies.
An example is that we create openapi docs with LLM's. Then we serve them with Go's net/http + FileServer, meaning that we never leave the standard library. Of course the LLM itself is a third party dependency, but when you use it to write CLI tools that then do the code generation, it never actually sees the code. That also means that the quality of those CLI tools are less important, because it is their output that matters.
It only goes so long of course. I'm not sure any of our front-end engineers would want to live in a world where they weren't allowed to use React, but then, their products are treated as though they are external anyway.
Anyway, it's a lot easier to make engineers stop using "quality of life" dependencies when you give them something that still makes their lives easy.
Doesn't the LLM spit out the code of the dependency it has been trained on?
How is that any different from just forking the dependency and treating it like your own?
One advantage might be that you would only implement the (small) part of the library that you actually need, potentially avoiding bugs such as Log4Shell.
The risk of code duplication is a serious one though, and it would be nice if AI could automatically detect and clean this.
I guess we are in an annoying "in-between" state of things, with a pretty unsure future.
The benefit of using a library directly is your 3rd party library checks will warn you when a CVE is found in the version you are using. If an LLM creates the same functionality from copying a version of a library, you might be getting a version that already has known vulnerabilities, and you probably won't be pulling in any fixes/improvements in future until you find them.
Fork the dependency and use that, to have a stable non-changing base which you use. And additionally, make the original project a dependency but don't actually use it. This way you'll get CVE information from your tooling.
If you fork a dependency and change features, the CVE information on original depenency is now no longer valid for your code. Your additions or removals can induce new CVEs, or render CVE for original lib a moot point.
You would (probably) be avoiding commonly known exploits while introducing subtle AI-induced bugs like printing logs out of order or not locking/ordering resources properly.
Not necessarily, you can task the AI Agent with writing your tool without the use of external dependencies. Often the tool itself will be absolutely horrible, but this doesn't matter if the output of the tool is fine. As in the case of OpenAPI documentation, you basically get the LLM to write the equivalent of C#'s Swashbuckle (or whatever it's called these days) for your language. What it produces is horrible in comparison to the third party dependencies you would have pulled 5 years ago, but the output is the same.
You can also let it use outside dependencies, but then you're right, then it would make little difference in regards to security.
We figured this out because Go didn't have a "Swashbuckle", and nobody wanted to write the documentation or use codegens to go in the opposite direction. When it turned out to be so easy to use LLM's for these things, it grew into it's own thing in regards to replacing a lot of dependencies. Obviously the LLM is still the dependency, but you can replace all other external dependencies for a lot of things like this. Which means you won't acidentially pull something malicious.
I imagine we're going to see more and more of these in-house dependency replacement tools. Coming from Go, I obviously use SQLC (which is run inside it's own isolated container that is only available for requests from developers and is only updated by hand). If we couldn't run SQLC in total isolation then I imagine we would have had to build our own "SQLC". I haven't tried but I'm pretty confident that you could use LLM's to build a similar (but much worse in quality) tool. In an ideal world, we would have been allocated the resources we needed, but in reality, it would have just made us slower as "IT" is still viewed as a cost center on par with HR except that we aren't even "IT".
It’s easier to incrementally review the generated, specific code of an AI agent than it is to review a generic, overly featured long-lived library— and to keep on top of (and review) all changes to that library over time.
As well, paid dependencies usually only have one source of support, and when the company goes under or stops supporting the product you are in rough seas.
Given very few companies last forever, you have to assess if the trajectory of your project would be impacted by being coupled to their ability to support you.
For sure, this goes into the terrain of acquisition though, for which there are long-running procedures and assessment criteria. Long-term support / company longevity is one of them.
But likewise, these companies have the incentive to look like they have long-running success and can be relied upon for years / decades to come.
I've experienced some bad paid dependencies forced on us by a non-engineering team. I've had a few good experiences with "open core" kinds of dependencies that are widely used by the community, e.g. sidekiq, and therefore less likely to suddenly vanish one day as they would almost certainly get forked and maintained by others in the same boat.
The upside of paying for something is that, assuming the owner or company is stable, I don't have to worry about some unpaid solo maintainer burning out and never logging back in.
> My two cents. If a dependency is paid, than it is usually bad. Because the company providing that dependency has an incentive to lock you in.
Vendor lock-in is a risk for both purchased components and FOSS ones where the organization is unwilling to assume maintenance. The onus is on the team incorporating third-party component(s) to manage their risk, identify alternatives as appropriate, and modularize their solutions.
If a dependency is paid and it is bad, then maybe you just aren't paying enough for it.
If my code has a dependency then I want there to be people whose feel it is their job to support it.
Either there have to be enough people who are paid to support it, or there have to be enough people whose self-worth and identity is so wrapped up in the code that they take it as a point of honor to support it.
I don't need a company that's likely to abandon a code product and leave me hanging. I also don't need a free software author who says "bugs are not my problem - you have the source, go fix it yourself." If those are my choices, I'd rather write it myself.
> I want there to be people whose feel it is their job to support it.
their "feeling" will not change reality, which might conflict. For example, a specialized database vendor would prefer that you cannot move away, and even if they feel like they want to support you, there are other economic factors to consider which could override this.
I think the idea is that if you are paying, the dependency needs to implement some sort of open standard/interface to which there are at least one other implementation you could move to. The vendor cannot lock you in with this requirement, since you always would have the option to move (despite it being a bit expensive, it's just there as a threat).
Well, as a battle-scarred old war horse, I can say that "It Depends™."
I have found that "It Depends™" is a mantra for almost everything that I do, and experience has given me the ability to choose a fork in the road, when an "It Depends™" moment happens.
When I was younger, my world was governed by "hard and fast" rules. There could be no exceptions, and some of the worst code I wrote, consisted of the gymnastics required to shoehorn an unsuitable library/paradigm into my projects.
I tend to use a lot of self-developed dependencies. They are really distillations of things that I do frequently, so I factor them into standalone packages. That way, I don't have to reinvent the wheel, and I'm confident of the Quality and provenance.
But if there's functionality that is required, and beyond me, for whatever reason, I am always open to including dependencies, as long as I can reconcile myself to their quality and other issues.
People love to claim this, especially on this site, but in my experience it's the opposite. Many people like writing new code and will do it even when it's detrimental to the business, but 9 times out of 10 even using a "bad" dependency is far more effective than writing in-house.
Most vocal people work on "disposable" end of software. It's cheaper for software giants to just throw engineer-hours at rewriting piece of code that has fallen into organizational obscurity than to maintain (hehe) maintainability. There is usually no sense for small branding/webshop factories to churn out high quality, maintainable code.
However, I suggest you revisit the reason why the dreaded "enterprise patterns" exist in the first place. The main reason to use these architectures is so that five years down the line, when documentation is badly outdated, there is no organizational memory left behind that component, original developers have transitioned to either different teams/departments or left the company altogether, the component is still well isolated, analyzable and possible to work on.
Introduction of external dependency(-ies) carry two inherent business risks: either support for dependency will be dropped, meaning you will have to either absorb maintenance burden yourself or switch dependencies, or it will introduce breaking changes, meaning you have to stick to unmaintained version or update your product code. Both situations will eventually impact your feature flow, whatever it is.
Compromise between trunk and leaf (push of dependencies vs pull of deps) is intrinsic to modularization and is always there, however with internal components this compromise is internal, rather external.
> Many people like writing new code and will do it even when it's detrimental to the business, but 9 times out of 10 even using a "bad" dependency is far more effective than writing in-house.
If you are a SaaS company - most probably yes as it is the short-term outcome that is determinate of business success. However, if you work in any industry with safety and support requirements on software or put the burden of long term support on yourself, long-term horizon is more indicative of business success.
Remember, writing new code is almost never the bottleneck in any mature-ish organization.
> five years down the line, when documentation is badly outdated, there is no organizational memory left behind that component, original developers have transitioned to either different teams/departments or left the company altogether, the component is still well isolated, analyzable and possible to work on.
This will be far more true for an external dependency - even one that's no longer actively developed - than for an internally developed component, IME. Just at the most basic level an external dependency has to have some level of documentation and at least be usable by someone other than the original author to even get picked up.
> Introduction of external dependency(-ies) carry two inherent business risks: either support for dependency will be dropped, meaning you will have to either absorb maintenance burden yourself or switch dependencies, or it will introduce breaking changes, meaning you have to stick to unmaintained version or update your product code. Both situations will eventually impact your feature flow, whatever it is.
Sure, you need stay up to date, potentially even take over maintenance yourself, or accept the risk of not doing so, and none of that is free. But writing an internal implementation basically puts you in the worst-case scenario by default - you have to maintain the code yourself, and it's probably less maintainable than an external codebase.
> But writing an internal implementation basically puts you in the worst-case scenario by default - you have to maintain the code yourself, and it's probably less maintainable than an external codebase
This very much depends on the type of dependency we are talking about. if it's externally-facing, sure, you'll have to maintain a suitable security posture and keep up with spec/requirement changes coming in from the outside world.
If it's an internal-facing dependency, it's reasonably likely that you never have to touch it again once the problem is solved. When I worked at Amazon we had internal libraries that hadn't really changed in the past decade, because they were designed to operate inside controlled environments that insulated them from the changing world outside.
External, unfortunately. A library that only wrote log files would be internal, but log4j is one of those open-source solutions that has fallen prey to the kitchen-sink fallacy - bundling network transport and service discovery into your logging library creates a massive attack surface that isn't strictly related to the library's stated function.
> > five years down the line, when documentation is badly outdated, there is no organizational memory left behind that component, original developers have transitioned to either different teams/departments or left the company altogether, the component is still well isolated, analyzable and possible to work on.
> This will be far more true for an external dependency - even one that's no longer actively developed - than for an internally developed component, IME. Just at the most basic level an external dependency has to have some level of documentation and at least be usable by someone other than the original author to even get picked up.
Not sure if I can agree with the sentiment, especially with the "at least be usable by someone other than the original author to even get picked up." part. Components are easily usable either because they sort of adhere to the UNIX philosophy "do one thing and do it well", which makes them inherently well isolated, or they are easy to integrate, falling on the kitchen-sink end of the spectrum, making them inherently hard to isolate. I think it has more to do with the functional domain of the component, rather than source.
> Sure, you need stay up to date, potentially even take over maintenance yourself, or accept the risk of not doing so, and none of that is free.
The more of a kitchen-sink the component is, the more maintenance effort there is and the higher the risks. However, in my previous comment I wanted to highlight integration risks: components on the kitchen-sink end of the spectrum are more likely to break interfaces due to feature work, nudging organization towards trunk based development. It's not inherently worse or better than leaf based development, it's a compromise, however my key point remains that larger external components shift the compromise externally, which is impossible to plan. In my experience, internally-facing external components that diverge too much simply get code frozen instead of getting required support, whereas internal components get maintenance effort allocated.
Can I ask how seriously your company takes security vulnerabilities and licensing? I used to have a pretty lax attitude toward dependencies, but that changed significantly when I joined a company that takes those things very seriously.
I've worked with many companies over the years. License management should be automatic or mostly-automatic if you're taking dependency management seriously (particularly these days where so many projects use well-known open-source licenses and everything is machine-readable metadata), and I've never seen a single in-house codebase (even at e.g. F500 financial institutions) that took security more seriously than your average open-source library.
I think just about every experienced developer I know - most of whom are not from NZ - would agree with this article. We've all been burned by using the wrong dependency for the job at some point, or the pain of having to deal breaking changes when upgrading a library that really wasn't necessary to use in the first place.
To push back on this article's thrust, those are actual issues that can occur with dependencies. It's just that they usually don't, and most dependencies are harmless, small, and transparent, so articles like this and people complaining about dependencies are almost always overstating their case.
And therefore, adopting a "zero dependencies" policy is absolutely an overreaction.
I'd argue harmless, small and transparent dependencies are the easiest to avoid. The extreme of it "is_even", but any library that could fit in 200 lines should probably be owned code.
Where the article hits is critical libraries that you heavily rely on and that your code is slowly formatted around. I'm thinking specific data parsers, fancy query brokers, time management libraries etc.
I don't think the author is actually arguing for zero dependencies here. While they do quote a "zero dependency policy" in one open source project as being "instructive", they also propose a "framework for evaluating dependencies" - which of course anyone advocating for zero dependencies would have no need for.
There's also the distinction between a library - a tool that does one particular thing and does it well - and a framework - which dictates how you structure your entire application. And this is a mindset as well.
I mainly know the original quote in the context of Go, where the community as a whole (or at least the part that is in the Slack server) eschews big frameworks in favor of the standard library, lightweight architectures, and specialized libraries. A lot of people coming from other languages come into the communities and ask about whether there's anything like Spring for Go. Another common request is help with a major framework or abstraction like Gin (web API framework) or GORM (database ORM). But Gin is no longer really necessary as the SDK has good support for the common web API use cases, and GORM quickly breaks down as soon as you need anything more advanced than querying a single table, not to mention it's big and complicated but has only one lead developer (I believe), no major organization behind it.
But Gin and GORM dictate how you structure at least part of your application; especially the latter adds a layer of indirection so you no longer have to ask "how to I join two tables in SQL" but "how do I join two tables in GORM", the latter of which becomes very specialized very fast.
Anyway. Using the standard library is preferable, using small and focused libraries for specific problems that aren't solved by the standard library is also fine. Copy / pasting only the functions you need is also fine.
C programmers don't install dependencies. That would be insane. Better to "vendor" them into the same tree as everything else and build them at the same time.
If dependency == library, it is hard in C or Python, but it isn’t in Rust or Java.
You could easily run a Rust program on bare Linux, but using Docker might simplify the version management story.
But even if you use a language with a sane dependency management story, you probably depend on PostgreSQL, and running it in Docker can be easier than installing it on the system (since distros really like to mess with the PostgreSQL configuration).
The ubiquity criteria also informs scaling - for example, if a tooling or backend dependency is being production deployed by a company at a scale of 10^2 or 10^3 times your use case, you're much less likely to hit insurmountable performance issues until you get to similar scale.
They're also much more likely to find/fix bugs affecting that scale earlier than you do, and many companies are motivated to upstream those fixes.
Their libraries sometimes don’t even work for low scale though.
The protocol buffer compiler for Swift actually at one point crashed on unexpected fields. Defeating the entire point of protos. The issue happens when only it tries to deserialize from JSON, which I guess none of them actually use due to large scale.
To clarify, I'm not thinking of code/libraries written by a huge company, more about open source code that has been scaled far beyond your deployment size by someone/anyone else.
Also, if you're using some feature that isn't regularly exercised (like your Swift protobuf example), it's probably doesn't have the variety of use to be covered by Hyrum's Law (see https://www.hyrumslaw.com ), which is definitely a different aspect of the Ubiquity criteria.
Some of the worst bugs I've hit have been in libraries written by very large companies, supposedly "the best and brightest" (Meta, Google, Microsoft, in that order) but it takes forever for them to respond to issues.
Some issues go on for years. I've spent months working in issue trackers discussing PRs and whether or not we can convince some rules-lawyer it doesn't warrant a spec change (HINT: you never convince him), chasing that "it's faster/cheaper/easier to use a 3rd party package" dragon, only to eventually give up, write my own solution, fix the core issue, and do it in less time than I've already wasted. And probably improve overall performance while I'm at it.
I think a lot of it depends on the exact field you're working in. If you're working in anything sniffing close to consulting, work is a constant deluge of cockamamie requests from clients who don't understand they aren't paying you enough to throw together a PhD research thesis in a month with a revolving crew of junior developers you can't grow and keep because the consulting firm won't hire enough people with any amount of experience to give the small handful of senior developers they keep dragging into every God damned meeting in the building so we can have a chance to come up for air every once in a while.
I'm at a point where I have enough confidence in my skills as a software developer that I know pretty much for certain whether I can develop a given solution. There are very few I can't. I'm not going to try to train an AI model on my own. I won't try to make my own browser. A relational database with all the ACID trimmings, no.
But I'll definitely bang out my own agentic system running off of local inference engines. I'll for sure implement an offline HTML rendering engine for the specific reports I'm trying export to an image. I'll build a fugging graph database from scratch because apparently nobody can make one that I can convince anyone to pay for (budget: $0) that doesn't shit the bed once a week.
Most of the time, the clients say they want innovation, but what they really want is risk reduction. They wouldn't hire a consultant if it wasn't about risk, they'd put together their own team and drive forward. Being broadly capable and well-studied, while I may not be quite as fast at building that graph database example as an expert in Neo4j or whatever, we're also not getting that person and have no idea when they are showing up. If they even exist in the company, they're busy on other projects in a completely different business unit (probably not even doing that, probably stuck in meetings).
But I know I can get it done in a way that fits the schedule. Spending time reading the worst documentation known to mankind (Google's) because some drive-by said they did this once and used a Google product to do it is probably going to end in wasting you a lot of time only to realize that said drive-by didn't spend long enough actually listening to the meeting to understand the nuance of the problem. Time that you could have spent building your own and hitting your schedule with certainty.
Sorry, it's late and I'm tired from a full quarter of 12 hour days trying to rescue a project that the previous team did nothing on for the previous quarter because... IDK why. No adults in the room.
Don't go into consulting. If you do, it's impossible to get out. No product-oriented companies will ever hire you. Wish someone told me that 20 years ago.
This echoes the Joel On Software Classic, "In Defense of Not-Invented-Here-Syndrome", which gives some of the reasons Microsoft Excel won the spreadsheet wars. Of particular note:
=-=-=
“The Excel development team will never accept it,” he said. “You know their motto? ‘Find the dependencies — and eliminate them.’ They’ll never go for something with so many dependencies.”
NIH can also be great if you only need a subset of a "mature" dependency but you need to really nail it.
Since it is a solved problem you'll likely find ample documentation, algorithms and code examples so there is much less that can go wrong than in a greenfield project.
At my last job we rewrote InfluxDB as it as was too resource-hungry for our embedded device and the result was fantastic. 10x the time series storage at a fraction of the memory and CPU utilization, much more robust and flash-friendly.
"One technique for making software more robust is to minimize what your software depends on – the less that can go wrong, the less that will go wrong. Minimizing dependencies is more nuanced than just not depending on System X or System Y, but also includes minimizing dependencies on features of systems you are using."
My American brain, of course, immediately went to "National Institute of Health" and thought this was a software developer opining on the current administrations distaste for federal funding of anything.
Database drivers, crypto etc im always "installing", but for 97% of other stuff i tend to roll my own. And when i dont want to reinvent the wheel, i take time to vet the depency.
I take neither side in a dependencies vs NIH war. I like using SQLite as a database, and I use Tk (of Tcl/Tk) as my user interface toolkit, for example. Neither of those is going anywhere, they are both maintained, and I build them myself.
I would argue that the right way to think about dependencies, as some have alluded to, is to do a risk/benefit analysis. On the benefit side, how much work would it take me to duplicate this component, how well will it suit my needs, and so on. On the risk side, everything from continued maintenance to security and licence issues (for both free/libre/open-source and proprietary components). Then risk/benefit asks for ways of ameliorating risk, such as by forking code, or paying for support.
The right way to think of adding a dependency is to ask and answer questions like this prior to adding it. This is not the way that JavaScript developers used left_pad, they just said “it's in npm, so it's good.”
I do technical diligence on companies getting purchased or invested in, largely by private equity. Previous to that I was a developer for 15 years (still am, but open source now). One big one not touched on in the article is the cost of security updates. I see this all the time... companies that are in a bad position because they have too many dependencies and have wound up unable (for business reasons) to dedicate adequate resources to keeping them all current, and then end up with past EOL deps with published vulnerabilities. This – literally – costs them a lot at acquisition time. The purchaser (who is typically very risk averse) factors in all the cost to get things up to date and takes it off the price. This can be very significant. (looking at you, React et. al!)
On the other hand, I've seen companies where they used as little in deps as they could get away with, and they wind up in very strong positions. There really is a big labour cost in chasing the latest versions of underlying frameworks when those move too fast.
Either way you wrap each thing that acts as a dependency, even if it's internal. I treat dependencies another team in my company delivers the same as any other third party dependency. Never use the classes but always just wrap around them in a service or component.
When my 'task' is to publish things on a Kafka bus I create a publishing service that takes in an object that I control, only inside that service is there actual talk about the dependency and preferably even that's a bit wrapped. It's easy to go too far with this but a little bit of wrapping keeps you safe.
> Sometimes it's easier to write it yourself than install the dependency
This is definitely true, but only really relevant if you're either a solo dev or in for the long haul, so to speak. And it'll work way better for your use case too.
The problem is working with others. Others coming in are more apt to try to learn a known or semi-known system than "Dave who used to work here's crazy thoughts." Especially in this market where longevity is rare, and company specific knowledge is useless as resume fodder.
So from a dev standpoint it absolutely makes sense. From a business standpoint, probably markedly less so.
Short for Not Invented Here syndrome... when developers ignore existing libraries or solutions and implement their own version. Usually this is meant negatively, i.e. developers thinking they -need- to implement their own version, at great expense, when they could have just used an off-the-shelf library. However this article is positing that writing your own code is a good thing in some cases.
It is a good thing in some cases for a couple of reasons - first, off the shelf libraries often do a lot more things than are needed for the project at hand which can have the same effect that adding extra lanes to a highway does on traffic - more effort in the same space that could have been avoided. Second, OTS libraries can be opinionated, which winds up bending the existing codebase in ways that aren't optimal or aligned with the other code in a given ecosystem.
I find both the article and the title a bit too simplistic for the topic. This is the classic "build vs buy" problem and I think it's one of the hardest problems in softare development and maybe engineering in general.
I've been bitten by both sides of the decision many times and apart from trivial cases (like `is-even`), it's hard to come up with universal rule of thumb to decide this, especially for cases where it matters.
I can't say I am that impressed with ECMA-48. Back when it was VT-100s connected to a VAX or logging into a BBS over a 9600 baud modem with a PC I felt satisfied. Even when there was a bit of line noise.
Ever since 1990 or so when it has been using "telnet" or the equivalent on an xterm or CMD.EXE or some other GUI application that emulates I've never trusted it to work 100% right. Maybe 99.5%, but I expect the cursor to be a column off where it is supposed to be, for there to just be a little misalignment here and there. I mean, other people don't complain, but people don't complain when the center channel is burned out at the cinema or when the Linux desktop has a 179px wide label in a 59px wide space and such.
It's one reason I gave up on emacs and just use the 20% of vi functionality I know (hit 'i' and pretend that it's Notepad, as least ESC + :wq doesn't confound my muscle memory for ^S being search the way ^S and ^X^S do) when I am logged into a busted Linux install to get the package manager running again but really use IntelliJ IDEA as much as I can.
The tolerance for the amount of time it takes to build a new thing is a lot looser than the tolerance for the amount of time it takes to fix an old thing in production with PMs screaming about down time.
Since none of the two alternatives guarantee no downtime bugs in production, you're missing some other factor that would make NIH preferable, like, maybe, testing (which the new thing would have but adding the old one wouldn't because ... tolerance)?
NIH is cool and all for greenfield projects but please remember that most of industrial work today is compatibility with old standards and behaviors, dealing with their bugs and edge cases. In that case, a third-party dependency that has been existing for 20 years will be better.
If you have a third-party dependency that survived for 20 years. But what if you are trying to choose what to rely on today, and to decide if it will even exist in 20 years? Certainly none of the fashionable JavaScript frameworks will.
A good question I ask myself is: Can I vendor it in? If I cannot, that’s usually something standard, but also a complex domain (crypto, network protocol, platform framework,…). Anything else, I assume it’s only a short term gain, so I make sure to explore the library and get a good understanding of its internals.
I am failing to find it, but Simon Willison had a post where he vendored some small (<1000 line) blob of code instead of adding it as a dependency. The trick was that he had some Github action(?) automation which would automatically check if the vendored version ever fell out of sync with upstream. Get the benefits of vendoring while minimizing your actual support burden.
Only a realistic strategy for small bits of code, but I have considered do the same for a few utility libraries which are a focused niche, but subtle enough that you would rather outsource the complexity.
Vendoring is somewhat an answer, but not always.
I have some python code, vitrualenv is an obvious solution. However, older modules depend on older python3.9 behavior. And python 3.9 EOL is October 2025.
For me the "should I use a library for this" comes down to if I want to be able to dip under the abstraction.
For instance: terminal color codes. I'm building for linux, but would like my code to be reasonably portable to other platforms. By inserting my own \033[m... Codes, I take responsibility for doing colors on other platforms too, whereas a dependency absolves me of that.
On the other hand, if I need a graph algorithm like Bellman-Ford, I don't want to reach for a library, because that immediately forces me to use their structs for every graph node. I might also want to tune BF specifically to my graph's shapes, etc.
Interestingly I've only ever ran into this problem (as being a problem) in Advent of Code - you need a certain (more exotic) graph algorithm and you only find an example that goes contrarian to your current model.
At work I don't seem to need these weird algorithms, or they are in the language's standard library/default math or graph package and it's not worth reimplementing anyway.
Except it doesn't. Users don't care if the reason something is broken on their platform is your own code or a dependency.
IME platform abstractions often tend to be a trap because they can only abstract 90% without making compromised whereas if you do it yourself you can pierce the abstraction for the remaining 10% and get the best result on all platforms.
Note that the ECMA-48 escape sequences themselves are the good dependency, not abstractions that hide them, like your tput command or curses or what have you.
I think the OP's article is saying that they are both good and bad depending on the context. You need to evaluate these things on a case by case basis. Writing a little TUI for yourself or your team? Sure, go nuts with escape codes. Making a well supported tool that runs on any developer's machine? Maybe consider curses, there are probably more edge cases than you think.
Trying to find product market fit as a startup? Who gives a duck, find some customers before you care too much about the tech =P cattle not pets etc
I feel this post. I maintain software in Android, iOS, Python, embedded C, and of late adding Elixir to the mix.
Some in the community will wine about lack of Elixir ecosystem. But often, I’m fine just putting together what I need. It varies. I don’t want to do my own bandit sever or Phoenix live view stuff. But MQTT, no probs.
Often I find that the need for libraries can be an indictment of complexity against the problem space. Take Blootooth as an example. What a huge ball. Though at the end of the day, I’ve done straight to HCI python implementations that for some things are better than the “libraries” available. Don’t get me started on the “hold your hand” gpio libraries for the raspberry pi.
One type of dependency, that I kind of miss is the “copy this code to your project” dependency. You can take complete ownership right away. Whereas with more crafted dependencies, the dependency surface is more than just an algorithm writ large in code, but a whole philosophy/metaphor that may not be an exact fit for your own project.
You should absolutely use dependencies, and your should be able to tear them out as soon as you don't like them anymore.
Including the dependency can even be part of your NIH if you're so inclined. Instantiate two copies of your code (dep/NIH) and test that they behave the same.
Recently I had a good time avoiding a few dependencies. I am developing a project using Django and it go to a size, where I felt I should add testing to it.
But to test things, I would need instances of model classes. However, I didn't want to have to specify all the required fields every time I needed an instance for a test. The fields might be mandatory, but not always relevant for each test. Such a field needs to be filled, but not necessarily by me in the test code. I knew from earlier projects, that Faker generates random data and in the past I have thought it shouldn't be difficult to write a thing that generates random strings and numbers and ids for fields, and recursively other required entities. So I decided to write it myself [1], rather than introducing factory boy and faker as dependencies.
Sure those libraries will have a lot more features, but my code is simple and does what I need and I gained additional understanding of the complications with Django's model handling.
I also suspect, that factory boy has to hack around some issues, that I avoided by always creating entities in the database, which most of my code is doing anyway. I am not sure how they solved creating instances of models, that have many to many fields without persisting the entities in the database for example, because Django raises an error when one tries to do set a value for such a many to many field, without storing the entity in the database first, so that it gets an id. Which makes sense. So I suspect there is quite some magic in that dependency, that I don't need for now.
What's more is, that my factory functions are just as easy to use, if not easier. I simply do `factory_for_model(MyModel)(kwargs)`. It is also easy to add more functionality like `gen_multiple(factory, count)` for example. Uniqueness is more difficult, due to how Django's fields and models are compared. But as far as I understand neither does Faker do uniqueness out of the box. I may be wrong about that though.
I also developed a decorator for pytest functions, that helps me avoiding long test names and instead puts a test description in the decorator arguments. Those sentence long test names have previously annoyed me and always caused linting to attack me for it.
Test-only dependencies should be maintained separately from functional dependencies.
For standardized things like testing, I'm not sure if DIY makes sense. Even if you use 10% of its capabilities, sticking with an established test framework makes your codebase more maintainable and easier for others to contribute to.
Awesome! A literal actual debate between absurd humongous dependencies and the known dumbness of "Not Invented Here". Pick a wrong side folks. (No, "respect for reality" doesn't count as a side. Try again.)
One huge overlooked advantage of using dependencies is knowledge transfer. Software projects (for the moment) require people to write them and people move between jobs. There is a lot of common requirements between different software applications, it would be terrible if everyone just relied on what the standard lib offers.
PS: I assume this would be akin to soviet communism, where the state gives you "everything" and the rest you fashion at home. (being extremely loose here, do forgive me)
Everybody loves NIH until what gets invented is dogshit. Seriously, it's easy to romanticise the idea but wait until you have to work on something that evolved over a decade without any refactoring or documentation, and worse, the author is a lifer who couldn't ever imagine that their code could use improvement. I'll migrate to whatever silly API react router has cooked up this year over living that again, thanks.
“Find the dependencies — and eliminate them.” When you're working on a really, really good team with great programmers, everybody else's code, frankly, is bug-infested garbage, and nobody else knows how to ship on time.
[..] We didn't think that everyone else was producing garbage but, we also didn't assume that we couldn't produce something comparable to what we could buy for a tenth of the cost.
After many years using libraries without any sort of vetting process, I completely agree in principle that most code out there is bug infested garbage. One dependency had a concurrency bug so severe it almost cost our company our biggest customer early in our journey. We forked and completely rewrote the library as after looking at the source it was clear they didn’t have nearly as much care with their code as we did. This was the worst case but we faced many bugs in widely used libraries. We can’t replace them all as time is short but if we could we would probably replace most of them.
I wish. As far as I can tell the Venn diagram of people building piles of shit with NPM and people building piles of shit with LLMs seems pretty close to a circle.
From me using Claude Code, without a proper system prompt, Claude generates code rather than using a library, this week e.g. command line parameter and flag parsing. The difficulty is where the tipping point is to use a library, it can't be https://www.npmjs.com/package/is-even
> One of the biggest fallacies in coding is that dependencies have zero downsides.
Am I the only one who hates when articles start with a strawman like this? Or does the author of TFA really know people who think dependencies have zero downsides?
Just the word "dependency" brings to mind infantile helplessness. I cannot believe anyone is out there opining a lack of downsides.
>Instructive here is Tigerbeetle, a financial database written entirely with Vanilla Zig:
How does TigerBeetle's existence support the argument that rolling your own dependencies is better than using off-the-shelf tools?
TigerBeetle has existed as a company for less than three years, they haven't reached a 1.0 release, and they have a single-digit number of engineers. As far as I can tell, they haven't even reached the point of self-sufficiency, as they raised another round of VC funding a year ago.
The pain of rolling your own tools happens as you grow and your needs change. Rolling your own static site generator[0] is fine when you only have a handful of engineers, but I suspect over time, they'll burn more time on their homegrown HTML templating solution[1] than if they had just used a battle-tested, off-the-shelf solution like Hugo.
The only dependency TigerBeetle accepted is Zig, which also hasn't reached a stable 1.0 release and frequently forces developers to rewrite their code to deal with breaking changes, including a major one coming in Zig 0.15.0.[2]
I like TigerBeetle, but they're not proof of anything yet. For every TigerBeetle, there are a hundred profitable companies whose entire business depends on WordPress with 50 plugins they picked at random.
Aside from TigerBeetle, I find most of the arguments in this article weak. They sound like rationalization for a developer who enjoys rolling their own tools and seem not to consider actual business value.
The article also leaves out what I think is the most important consideration in evaluating dependencies: lock-in. How difficult is it to swap out this dependency later? Because if the answer is, "very easy" most of the other considerations don't matter.
There's no use doing an exhaustive survey of image processing libraries when you can swap out your choice with a few hours of dev work. But choosing something like a storage backend is very difficult to change later, so you want to put more thought into it.
> How does TigerBeetle's existence support the argument that rolling your own dependencies is better than using off-the-shelf tools?
At least from my own experience, without TigerStyle [0], there's no way we could have shipped TigerBeetle from 0 to production in 3.5 years, with the same quality, and to the same tolerances: https://jepsen.io/analyses/tigerbeetle-0.16.11
Whereas, for example, if we'd used off-the-shelf dependencies such as Raft or RocksDB, it would have been a nightmare, and taken longer, just to fix them up to solve our storage fault model and implement Protocol-Aware Recovery correctly. Even then, we'd have been stuck with a systems language of the last 30 years (C/C++), when we could have invested in a language for the next 30 and with a more powerful toolchain. Of course, we'd also have been without Deterministic Simulation Testing, because most software dependencies are typically not designed for DST.
But finally (and to your point about sustainability), without TigerStyle, we wouldn't have had the company, or the investment, team, and customers we have today, or be as far along financially. Not saying that you need to share our engineering philosophy. But it works for us.
>Often they're so large and complex that simply writing the functionality yourself is faster than learning. Sometimes it's easier to write it yourself than install the dependency...
Sometimes for good reason. Re-implementing something like HTTP with TLS is non-trivial and surely not easier to rewrite yourself...
Author points to TigerBeetle as an example, which I wasn't familiar with, so I went down a rabbit hole.
Wow. Yeah. If you're writing a financial ledger transactions database from scratch in Zig, where safety and performance are your top two goals, so that you can credibly handle a million transactions per second on a single CPU core, well, damn right you can't afford to take any risks on introducing any dependencies. Makes absolute perfect sense to me (no sarcasm).
But the average developer, writing most line-of-business CRUD systems, well, quite honestly they're not that strong to begin with. Most dependencies may be bug-infested garbage, but they're still higher quality compared to what most developers are capable of. Most companies are bottlenecked by the caliber of talent they can realistically recruit and retain; and internal development standards reflect the average caliber on the payroll.
So like most advice you read on the internet, remember, everybody is in different circumstances; and polar opposite pieces of advice can both be correct in different circumstances.
(I work on TigerBeetle)
+100, context is the key thing, both TigerBeetle and rust-analyzer have strong culture of how the things are done, but the _specific_ culture is quite different, because they solve very different problems.
That being said, I _think_ you might be pattern-matching a bit against the usual dependencies good/dependencies bad argument, and the TFA is on a different level. Note the examples of dependencies used: POSIX, ECMA-48, the web platform. These are not libraries, these are systems interfaces!
Dependency on a library is not a big deal --- if it starts to bite, you can just rewrite the code! But dependency on that underlying thing that the library does is usually something that can't be changed, or extremely costly to change. Answering "No" to "is doing X in scope for my software?" is powerful
To refer to the sibling comment, if there's a team whose core business is writing matrix multiplication code, they _can_ use random library #24 for something else, but often, if you apply some software design, you might realize that the product surrounding matrix multiplication _shouldn't_ handle concern #24. It doesn't mean that concern #24 isn't handled by anything at all, rather that we try to find _better_ factoring of the overall system to compartmentalize essential dependencies.
The problem I have with this line of thinking is that it only works when you're talking about bottom of the barrel developers.
The tech world has this tendency to dumb down every advice so it caters to the lowest common denominator, but I honestly have never worked in such an environment where developers couldn't learn to duplicate the functionality of shitty dependencies while at the same time fixing the problem.
And as much as people love to mock CRUD, bad abstractions can be a horrible drain on teams doing CRUD work. And even the popular stuff can often suck and be a timesink unless you're doing absolutely basic stuff that's just repeating the tutorial.
I've seen a few people from the bottom of the barrel outside on the real world...
The funny thing is that no advice is dumbed-down enough for them. They will take anything you can say and implement wrongly.
That idea that if you dumb-down everything, it will be helpful is just plain bullshit. People that can't duplicate a dependency just can't choose the right dependency either, and if you force them to use some good one, they will use it wrong.
Yeah, that's a great argument.
Not only it's harming people of average capability, it's also not even benefiting the people on the bottom.
The Overton-window of supply-chain has shifted so much to one side that it no longer benefit anyone but supply-chain providers.
> but I honestly have never worked in such an environment where developers couldn't learn to duplicate the functionality of shitty dependencies while at the same time fixing the problem.
I have absolutely seen this. It’s not a matter of them being bad developers, but no one is an expert in all things and gaining expertise takes time which costs money. All dependencies have a lot of trade-offs baked in that must be re-evaluated if you decide to go the NIH route, and that’s usually where expertise is needed. If you lack the expertise you will still be making trade-offs, but you will lack the knowledge to foresee the long term consequences of these decisions.
In my own opinion core business differentiators are where engineering resources should be used. That’s not to say a dependency that is more generic than the problem your business is trying to solve should never be pulled in house, but the decision to do that shouldn’t be taken lightly as it will increase the burden of learning, maintaining, documenting, testing, etc for a team that is being judged on the success of business objectives. To put it another way, the business will be hesitant to give you time to work on something that is only tangentially related to their objectives and quality can and usually will suffer which will put you in a worse position.
I think software development has some unique characteristics that makes NIH a problem. If you’re an electrical engineer you will never be given the resources to make a custom microcontroller or a new way of manufacturing a PCB unless that is what your company is selling; the upfront costs would be staggering and that would not be allowed. The limitations of the technology available is simply that, a constraint on what the business can do given current tech. Perhaps there is an opportunity for something better, but that would be best served as a new company focused on providing that solution if it can be done in an economically viable way. Software doesn’t have these massive upfront manufacturing costs that are obvious like the example above, but they still exist.
AI may change this calculus.
I am not saying it doesn’t exist, just that I have been around the block for quite a while but even then didn’t personally work in a place like this.
Places with average developers are more than equipped to do tasks that HN claims to be impossible to anyone but Carmack or Torvalds.
And average doesn’t mean rock bottom!
I don’t understand why those conversations only focus on the extremes. We should stop being so prescriptive of other people’s work.
Most professionals are average, by definition, because skill follows a Gauss curve.
HN is a bubble, the world runs on "meh".
Average doesn't mean rock bottom.
I've seen "Meh" do the things HN claims nobody can. Plenty of times.
It doesn't even have to be related to quality of developers. Whatever tool chain or product you use, you're using someone else's dependencies. In some places more than others, but most people aren't implementing their own matrix multiplication code, and the ones that are, aren't implementing random library #24 which isn't in their core business. So this whole discussion happens on black and white terms when in fact people only care if they have regulatory obligations, or they have a particular personal interest in implementing something, or they got attached to that idea for a specific library. But nobody is applying this fully or they'd still be at the beach collecting sand.
We apply this pretty much fully at TigerBeetle [1].
Reason being that TigerStyle is not only about shipping safer software, but shipping it in less time [2], factoring in the Total Cost of Ownership to end users. In other words, not only optimizing our development time, but also all the time spent in production with subsequent fixes.
[1] https://tigerbeetle.com/blog/2025-02-27-why-we-designed-tige...
[2] https://www.youtube.com/watch?v=w3WYdYyjek4
Where's your custom matrix multiplication implemented in your stack?
Vasco! In a past life I've implemented (and tuned) things like Cauchy by hand, but we don't use any of that in TigerBeetle. Nevertheless, the experiences and learnings around that were valuable for me personally, and I don't think I'd have been able to do TigerBeetle without putting in those hours. Wish you all the best, man!
> But nobody is applying this fully or they'd still be at the beach collecting sand.
To this point I posit almost all of the "this good, that bad" type of posts are just "at my point on the abstraction continuum, this thing is better than that thing" without taking into account any context.
For most assembly is too low level for their context/use case, but for some, the C programming language is too HIGH level, as it was originally designed to be (but not for me).
> But the average developer, writing most line-of-business CRUD systems, well, quite honestly they're not that strong to begin with.
This is a wild take. Developers don't usually get to pick the system they work on, resource constraints exist during development time. We make tradeoff decisions every day and pulling in a "cheap" dependency will often make the difference between a shipped, working piece of software and plain "nothing". You seem to be fairly little involved in actual software development, considering how generic and wrong this take is.
The problem, IMO, is that the trade-offs seem to always land on the side of “what ships faster.” That may well be the fault of PMs rather than SWEs, but either way, it doesn’t usually make for a sustainable product.
Isn't the industry still growing pretty quickly? If five years is enough to count as "senior", most won't be that good just because they haven't had time to get good yet. And any segment that's big enough or low enough margin won't be able to escape that average.
When looking at CVs and interviewing actively, it's honestly very easy to believe the quoted claim. Of course I work with some very skilled folks on line-of-business CRUD too so who knows.
I also don't quite like TigerBeetle as an example, because it is very new startup. They are pre-1.0 and just a bit more than a year since production release. It's way too early to tell whether their approach actually paid off.
Joran from TigerBeetle here!
While the company is 3 years old, TigerBeetle as an open source DBMS project actually started in 2020.
What we mean by "production" is also typically a bit different to what is typically meant.
For example, TigerBeetle is one of the first DBMS's to withstand an explicit storage fault model (i.e. disk corruption across all replicas). So the tolerances are a bit tighter for TigerBeetle, also with things like static memory allocation (cf. NASA's Power of Ten Rules for Safety-Critical Code).
Beyond this, when TigerBeetle was designed, it was (along with FoundationDB) one of few DBMS's to invest in a Deterministic Simulator in-house. Most DBMS's before 2020 were typically not designed from the ground up for Deterministic Simulation Testing (DST) and so now can only use Antithesis if they want to get into DST.
However, TB invests heavily in both, not only an external DST audit function, but more importantly, also our own internal DST audit function, which we designed and built entirely inhouse, and which can do considerably more damage to TigerBeetle because it's protocol-aware and understands how to push TB's consensus and storage to the breaking limit. In other words, not one DST simulator, but two, with “the VOPR” (the name for TB's own simulator) tailored to our fault models, and able to reach into any part of the code, across any machine in the simulation, and verify expectations. For example, do things like verify cache coherency between user space page cache and simulated storage engine.
Finally, if you're interested in the results of whether this rigid TigerStyle approach pays off, Kyle Kingsbury's Jepsen audit of TigerBeetle was one of the longest Jepsen engagements to date, and these were the findings: https://jepsen.io/analyses/tigerbeetle-0.16.11
I work at a medium company that is in a comparable sector implementing comparatively performant and externally constrained things pretty much wholly in C and pretty much wholly with our own dependencies with external dependencies being mostly located in ops, testing/qa and other stuff not in the critical path of what the people who pay us want the products to do. For us it's not about "built here" it's about control and uniformity. The company is a decade old and very profitable for its headcount though not subject to wild delusions about future hockey sticks because of what we build and for who.
So the approach clearly can work. It won't kill you. But it's also unclear the degree to which this approach contributed to vs detracted from our success since it's not like there's a control company that did it the other way you can compare to. It takes constant effort to do things this way, but so does managing dependencies. While unilateral control, and the responsibility for that, that comes with writing your own dependencies is a key aspect of how we do what we do with the headcount we do it's not like it's part of our marketing. It's just a decision that was made that we're sticking to because it seems to work for us. It's not some silver bullet, it takes effort to do well, like anything else done well.
I have worked at huge companies where they were biased toward using existing stuff where it existed and tweaking it just the bare minimum needed and that worked very well for them too and definitely had its pros too.
Author here!
I pointed to tigerbeetle because they're one of the few companies I know of who 1) have a philosophy about dependencies and 2) talk about it openly in an easy to quote document.
I'm not sure if I succeeded, but I wanted to get across that I mean "dependency" in a very broad sense. Not just libraries, or services you interact with, but also what you need to build and deploy.
It wasn't a call to just not use dependencies. It was a call to think about them and be clear what you're depending on. Tigerbeetle have done that and given reasoning.
Appreciated your post, Lewis!
From the beginning, TigerStyle was intended not only for TigerBeetle as infrastructure, but for software (up and down the stack) in general.
Nobody wants buggier, slower software. And so TigerStyle as a methodology was codified primarily to address this problem: How to write “higher quality software in less time”? [1]
Before designing TigerBeetle, I was finding the principles more and more valuable, regardless of whether I was doing frontend or backend work, and so brought them into TigerBeetle.
Of course, software higher up is likely to use some dependencies, but there's always a cost, and this principle (“do more with less”), at least in my experience, tends to prove valuable when applied.
[1] Some of the background motivation behind TigerStyle: https://www.youtube.com/watch?v=w3WYdYyjek4
NIH is amazing as long as you are realistic about what you are taking ownership of.
For example, the cost of maintaining a bespoke web frontend "framework" that is specific to your problem domain is probably justifiable in many cases.
The same cannot be said for databases, game engines, web servers, cryptographic primitives, etc. If you have a problem so hard that no existing RDBMS or engine can support, you should seriously question the practicality of solving your problem at all. There's probably some theoretical constraint or view of the problem space you haven't discovered yet. Reorganizing the problem is a lot cheaper than reinventing the entire SQLite test suite from zero.
> The same cannot be said for databases, game engines, web servers, cryptographic primitives, etc.
It absolutely can in many cases.
- Many applications don't need much more of a database than files plus perhaps a runtime index. Spawning a database server or even embedding SQLite can be overkill in many cases.
- Most games would be better off with a bespoke game engine IMO, especially those made by small teams. The only real advantage to using an established engine is the familiar asset pipeline for replaceable artists but you pay for that with a huge overhead. Remember that custom engines or at least heavily modified ones maintained by the game developer used to be the norm - "just use Unity^WUnreal" is a relatively recent trend.
- There is no real difference in complexity between a simple web server and FastCGI application.
- Not all uses of cryptographic primitives are a security issues. If you calculate a hash to check against corruption you can implement that yourself (or copy an existing implementation) instead of pulling in a huge library. If your use cases is decrypting existing data offline then again you don't really care about cryptographic security only about getting the correctly decrypted output.
And there are many more cases like these. Learned helplessness when it comes to "hard" subjects helps no one.
Just because some existing solution "solves" your problem doesn't mean its the best or most efficient way to do it.
> files plus perhaps a runtime index.
The latter already adds a lot of complexity, especially if the application has to scale up or out. If scaling up, the index will increase in complexity and re-implement some clever algorithms that have already been solved by the other. If scaling out, you'll need some locking mechanism if multiple processes can write. It quickly becomes logical to switch to an established solution then.
But the choice becomes more logical then; if one picks a scalable database or whatever without knowing they're going to need it, that's a lot of wasted effort.
> - Most games would be better off with a bespoke game engine IMO, especially those made by small teams.
I disagree; writing a game engine, even for simpler "indie" graphics, requires specialized knowledge and a dedicated engineer, whereas with off-the-shelf engines, non-programmer / creatives can build a good game without in-depth knowledge. A small team needs to be especially careful with what they choose to spend their time and money on, and solving a solved problem like building a game engine is a huge investment. Even big studios struggle with this, see also the dramatic release of Cyberpunk 20something by long-time own-engine-building veterans CDPR. They're switching to Unreal for Witcher 4.
Any mature game engine also comes with mature developer tooling. Think profiling, debugging, assets management, map editing etc. That’s a lot of initial hurdles to overcome before you can even start working on your game ideas. For indie / solo developers who tend to be, you know, driven by one’s creative urges, this is not just a huge time sink but also a good way to ruin your motivation by never moving to the part where you get to actually create the game. When your time and energy is constrained you need to choose your battle and for indie developers this often means forgoing the technical part (unless you really need to) and focus more on the game part.
I have seen many people say “there are more games engines written in Rust than games written in Rust” and I wonder if what happened is that the software developers fell for the allure of building their whole stack from scratch and ended up finding out it sucks to actually develop a game for an engine with no tooling and “making an entity inspector that lets you click on an enemy and edit its attributes” isn’t exactly the sexy part of game development.
The trap of "building a game engine" vs "making a game" has existed long before Rust.
When games were really simple, so simple that "making a game" vs "making an engine" had no real difference (say, building a game for 8 bit home computers) then it wasn't a big deal. The concept of "games engine" didn't really exist back then, or was uncommon at least.
But nowadays? A small indie team, or a single dev, are likely to go down the rabbit hole of making their own game engine, forced to solve a lot of problems that are already solved in existing engines, rediscover the wheel, and get frustrated because the step of making the game never actually comes.
I think it makes more sense to use a pre-made engine, unless the game really is so different this isn't an option, or the devs really enjoy making engines (which is valid, and you learn a ton of cool things, but is also a hurdle in the way of actually making a game).
Every games programmer enthusiast I know has a half-made, abandoned, badly performing engine with which they've made a couple of demos or a barely functional unfinished game... (I'm also guilty of this, but given my age, mine was written in C++!).
The author of one of the most popular games of the last decade disagrees with you: https://noelberry.ca/posts/making_games_in_2025/
In which part is Noel disagreeing? He didn't write his own engine either, he just didn't use one (but didn't write everything from scratch either, note he uses lots of game/graphics frameworks, which in a sense is like using a "meta engine").
He's also making a 2D game, which traditionally is closer to game making of old (think 8 bit home computers).
Finally, he's not arguing against engines like Godot or Unreal, he says those are acceptable but he just didn't need them. Much like he doesn't need Windows and finds coding with Linux more comfortable.
PS: out of curiosity, which one is the popular game you mentioned? Looking through his portfolio I see lots of interesting games I'd like to play :)
> He didn't write his own engine either, he just didn't use one
It's hard for me to take in good faith an interpretation of that article that determines Noel "didn't write his own engine," but in case a couple direct quotes would help:
"I don't have much to say about assets, because when you're rolling your own engine you just load up what files you want, when you need them, and move on." ... "my City of None engine"
Noel literally describes his workflow as "rolling your own engine" and has a name for the engine he wrote.
> He's also making a 2D game, which traditionally is closer to game making of old
Please read the context of the thread you're replying to, which is arguing against writing your own engine even for games with "simpler 'indie' graphics," specifically called out.
> Finally, he's not arguing against engines like Godot or Unreal
He's not arguing against anything. He's advocating for writing your own engine.
Noel made the indie hit Celeste: https://en.wikipedia.org/wiki/Celeste_(video_game)
Ok, ok, it's indeed an engine of sorts. And I am arguing in good faith.
I'm not familiar with Celeste and I cannot be sure it's one of the most popular indie games of the last decade (I read that it was a hit, good for Noel because it seems like a cool game), but regardless, it's a very specific kind of game! Mostly of the kind I've argued "writing the game" and "writing the engine" is almost the same thing. Nobody is making the argument you must always use an existing engine; only that it's what makes sense as the default option.
> Please read the context of the thread
If you've read my other comments, you know I'm aware of the context of the thread.
> He's not arguing against anything. He's advocating for writing your own engine.
Oh please. Advocating for a thing and arguing against not doing the thing are two sides of the same coin.
> - Most games would be better off with a bespoke game engine IMO, especially those made by small teams. The only real advantage to using an established engine is the familiar asset pipeline for replaceable artists but you pay for that with a huge overhead. Remember that custom engines or at least heavily modified ones maintained by the game developer used to be the norm - "just use Unity^WUnreal" is a relatively recent trend.
I think we can take Square-Enix as a case study: Final Fantasy XV and Final Fantasy XVI both used their own engines (though XVI used a modification of XIV's engine, from what I hear) and both took a long fucking time to make. Granted, XV had many issues outside of just building a new engine, but it definitely added one more problem onto the pile. My understanding is that Final Fantasy XVI's development cycle was pretty smooth, it still took 7 years from the launch of Final Fantasy XV!
Final Fantasy VII: Remake and Final Fantasy VII: Rebirth both used Unreal Engine 4. While Remake had a long development cycle, Rebirth was done in 3 years and contained a much bigger world!
I do totally get that part of the argument is "reuse the same damn engine and tooling," and I totally agree with that, too. It's just a bit easier overall when you can reach out to a community with your problems instead of learning every problem for yourself.
> Most games would be better off with a bespoke game engine IMO
Perhaps, but that would mean most of those games would never get shipped.
> If you calculate a hash ... instead of pulling in a huge library.
This is a bit of a strawman; what hash calcs require "a huge library"?
Most games that use an existing engine get like 60% on metacritic and ship like less than 1000 copies and are a generic flop. “Shipping no matter what” is a lot less valuable in games.
You think the only game you will lose are generic Unity shovelwares when in reality you will also lose good games made by people with good sense in game making but likely not the resources or technical expertise to make their own engines. Think Cities Skylines (Unity, the studio had only a dozen people in total when it released), Dusk (Unity), Undertale (GameMaker), Spelunky (GameMaker)..
Another comparison is game modding which is essentially using the base game as the engine of your modded game. Do you think Team Fortress or Counter Strike or DOTA would have been made if their creators thought they have to build their own game engine?
Lowering the friction to make a game means more games get made. Yes there will be a flood of bad ones, but we get good one too.
I don't think the engine has much to do with the creativity or originality of a game.
If anything, not having to build an engine frees up more time to get creative with the actual game.
Most games are bad, of course, but Sturgeon's Law applies.
Survivorship bias. Most games are indie games. Indie game devs who deside do use custom engine don't make it to sales.
[dead]
The adventure of writing a custom DBMS, game engine, or other complex and advanced software classes can be justified by 1) knowing that no existing solution is adequate but 2) it is possible to write something suitable and 3) it will be useful enough to justify the cost.
For example, to insist on the example of Tigerbeetle, 1) no current popular or specialized DBMS is particularly optimized for ledger use, 2) but suitable techniques are well understood; I'm not sure how often 3) it will be preferred to a more general purpose DBMS.
> - Most games would be better off with a bespoke game engine IMO
You have it backwards.
Most games won't benefit from a bespoke game engine at all. Most games should use an existing engine (not necessarily one of the big names, just one that works) if their goal is to actually deliver a game.
Some games are so unique they won't benefit from an existing engine, and only then the dev team must take the extra step (and eat the delay) of building their own engine.
I mean commercial games of course. Experimental/hobby devs are free to do whatever, since in that case it's the journey and not the destination that matters.
Why are there so many different database engines then? Is each instance of a different database out their an instance of NIH?
The answer is of course that every computer system of any complexity has to make trade-offs - do you need constraints, do you need scalability, do you need concurrency, security? Is your data of a specific form that benefits from a particular compressible storage form, is it optimised for write once read many or read/write etc etc..
In terns of the thrust of the main article - which is about the cost of dependencies - I find that I'm typically much happier to take a dependency if that dependency is very keen on minimising it's own dependencies! - ie I don't want to take on a whole forest of stuff.
In my view many automatic dependency management systems have helped create the mess that they claim to try and solve, and using careful manual dependency management can help cut down the burden.
> do you need constraints, do you need scalability, do you need concurrency, security
Don't forget, do you need someone you can call for support (or throw under the bus) when problems come up?
This is the thing.
If the database is essential to your business - as in if it goes down your revenue drops to zero - do you want to be the one debugging it at 3am?
Or would it be more cost effective to have a well-known database or even a managed DB
If you are running into a problem that is bringing your site down and it's truely a problem/bug with a commercial database then you are in trouble and unlikely to see a quick fix as these products are complicated and while the problem might be existential to you, it's not to the bigger vendor.
However, it's much more likely it's a configuration issue - ie you didn't understand the product properly and have configured it incorrectly - and here support can really help.
The point I'm making above is not that you should be writing fully fledged all singing and dancing databases - but that you might be able to get away with something very simple, more reliable and faster in your specific use case.
If on the other hand your use case is complex and a good fit for something pre-existing - then sure you shouldn't be writing it yourself.
i can think of two reasons for using a third-party dependency
1) a dependency on a third-party service provider that publishes the dependency. So long as that service provider is current, the dependency should be maintained 2) short-cut to code i don't want to write
I have no arguments with (1), there's a business reason and the lifecycles should match. However, I should still expect major version breaking changes in order to keep my application running. For (2), the wins are less clear, more dependenent on the perceived complexity of what I can avoid writing.
Taking on any kind of dependency means that someone else can dictate when I need to spend time updating and testing changes that don't add anything to my product. Taking on a third-party dependency is always taking on a responsibility to maintain a codebase or the risk of not doing so.
I would argue that 2 is the much more important reason. A dependency is worth it if it saves you a lot of time building the same thing. For example, if you wanted to ship a general computing device, you should probably run Linux on it rather than building your own OS from scratch, because you'll save literal years of work. Even if Linux had stopped being maintained, this would still be the right call compared to building your own (of course, it would be even better to choose some other OS still being actively maintained, if any is available).
I wish we could have this discussion without going into exaggerated extremes like "Linux". The OS rarely gets into discussions of rewriting dependencies.
I recently had to rewrite a table component and it took about an hour with extra performance and it became faster-to-use and tailored to my company's needs. One hour, give or take! With measurable performance improvements, and it's simpler to use for our use cases, and it doesn't prevent anyone from using the old one.
Comparing one-hour or even one-day work with rewriting Linux is ridiculous fear-mongering.
But this is exactly my point. Some dependencies don't actually save you any time, they're trivial to write yourself, so there is no point in getting them (perhaps the most absurd and infamous example of this being left-pad). Other dependencies are extremely hard to write, so finding a version someone else built, even if they are not maintaining it anymore, is worth it.
Theres a third one, when it comes to compliance and security tools, you don't want to build it even if you can because.
1. It is a liability
2. There is trust deficit during audit and other events. If audits are internal only sure you can build it but when it is 3rd party audited, auditors often know the product and familiar with the features.
> auditors often know the product and familiar with the features.
or what if you chose a dependency for which this auditor is unfamiliar with, and so it takes even longer (where as if you NIH, you'd have the full source and thus can give the auditors the materials to audit).
I am not sure most auditors work on that level of detail. If it is a library they don't consider audited yet, they might just call it a day and make a statement about your code excluding the dependencies they are not familiar with. Otherwise you would have to pay for all third party dependency audits, which no one else paid for yet or the auditor is not aware of someone else having audited already.
You are right. Thats not the level they often go to.
I have also seen this narrative. If things go south in the org. btw we were using product A for doing our workflows, that replaced if with product B for the same, now everything is going to better.
This is where using boring languages like Go and Java come into their own: less frequent breaking changes.
There are some languages where communities aim for libraries to be stable for several months, and there’s others that think in terms of a decade or longer.
There is another solution which is to just retain the source for your dependencies and bring them in-house if needed. You still may need to abide by licenses, and many things shouldn’t need to rely on external deps, but I’ve seen a lot of wasted time that continues because we came up with our own way of doing something that others do better, and that can be difficult to get rid of because of unwarranted team pride or familiarity in the homegrown code or not wanting to trust the lone dev that says try this instead.
Most important reasons imo:
1) even though reality has proven us wrong time and time again, we can just not look at the dependency too closely and just act as if it's written and maintained by competent, caring people and is of highest quality. No worries!
2) in case shit hits the fan, let's assume worst case and there is a vuln in the dep and you get hacked... It's somebody else's fault! \o/
> 2) in case shit hits the fan... It's somebody else's fault!
A contrived example but, good luck explaining to the lawyers that openssl had this bug that caused all your customer data to leak and your company is being sued. If your motto for dependencies are “we can just not look at the dependency too closely and just act as if it's written and maintained by competent…” I’m reasonably sure someone is getting fired in that situation if it doesn’t sink the entire company.
Move fast and break things isn’t exactly a viable strategy in all industries.
Now as I said openssl was a contrived example but what if instead it was your ORM that didn’t use templated queries but rather just did string interpolation and there was an SQL injection attack. Considering this is still one if the most popular vulnerabilities on the web someone is messing stuff up somewhere and you are blindly like hoping it isn’t in the 10k lines of ORM library you pulled in instead.
You fairly quickly run into problems a RDBMS can't solve. They're above all incredibly hamstringed by being built around supporting concurrent mutation of the data set and mutable datasets in general.
If you don't need that, even a fairly naive index implementation will get orders of magnitude more performance out of a computer than you will with a RDBMS if you can assume the data is immutable.
I'd tend to argue the opposite. Unless you have a problem that very explicitly is a poor match for an RDBMS, they are the easiest, most reliable, and most flexible way to manage data, separate your data from your application, and adapt or add functionality as new requirements come along.
That RDBMS example of yours is fascinating.
There are plenty of RDBMS here (wikipedia lists some 100+ of them), there are plenty of problems most of them can not solve, but some of them do solve.
These people considered practicality of their solution and went forward doing the implementation.
Absolutely. A bunch of them also predated the www, or started just when the www was gaining popularity, meaning that information on possible products might be more limited than it is today. Some have narrow use-cases, and some might have started as really narrow in terms of scope, but then ballooned after gaining popularity, sales (and feature requests), and some probably started because the alternative was deemed too expensive compared to just making it in-house.
I think the main point bob1029 was trying to make is that it can be worthwhile doing somehting in-house if the alternatives doesn't match the use-case, are too expensive or whatever else - but that you seriously need to consider if your architecture is the best way to solve the problem before going down that route.
What is NIH? Skimmed the article. Still don't understand, after doing so twice. Google says national institute of health.
Not Invented Here; the desire to always build your own rather than buy / borrow existing tools.
> as long as you are realistic about what you are taking ownership of
Including training and onboarding. There's a reason "popular" libraries, languages, patterns, etc. are popular. One "win" is the network effect when looking for talent - they can hit the ground at least jogging.
> Reorganizing the problem is a lot cheaper than reinventing the entire SQLite test suite from zero.
Sure, but if you aren't happy with existing DBs you are probably wrong in thinking you need a general DB instead of a file layout on a filesystem.
It seems easy, but actually managing files as a database safely is actually very hard. Especially ensuring atomicity.
Portability is a problem as well.
You do not need an SQL database however, there are others available.
Of course these questions are hard and if you are asking them you may need a general db.
The whole industry is wrapped up in these bizarre ideas that for the slightest requirement they are going to add just one more abstraction layer and it will do everything and the kitchen sink. But sometimes the requirement is simple and doesn't deserve solutions to anticipated problems and other times it does and is worth looking for an existing solution. Rarely the problem is complex and deserves a custom solution and then one should probably make a new startup for the one yak shave, etc..
The technology is only part of the problem. If you aren't happy with any existing database, throwing even more technology at it is definitely not going to help - statistically speaking. You're up against millions of engineering hours in this space.
At some point, we need to get up from the computer and go talk to the business and customer about their problem in terms that don't involve anything more sophisticated than excel spreadsheets. There is definitely some aspect you missed if you think you need to build a fully custom technology vertical to get them to pay for a solution.
Of course, all of this advice is pointless if what you are doing is for amusement or as a hobby. I think a lot of arguments about technology would be diffused if we prefaced with our intended applications.
> You're up against millions of engineering hours in this space.
Tangential, but the fact that many orgs choose to not use native DB features like foreign key constraints, saying that “they’ll handle it in application code,” has always seemed like the pinnacle of hubris to me. You’re going to ignore some of the most battle-tested pieces of technology, and recreate it poorly instead? Quite the decision.
While there are valid reasons to not want to use FKCs, IME much of the time when you peel apart the layers of why someone chose not to, you find that they made several erroneous assumptions that led them to that conclusion. Other times, it’s that they simply didn’t know something existed, because they never read the docs. Windowing functions are a good example: RDBMS can do quite a bit of manipulation to data, rather than just sending huge result sets and having you parse them. There are trade-offs, of course, but you should at least know that it’s even possible before choosing to ignore it.
What I want is sqlite but in native Java.
Using files instead of SQLite very much is NIH. Good luck not corrupting or losing your data.
The Gang Learns About Fsync.
If only that’s all it took.
It’s certainly something you can invent yourself in a cross-platform way, sure.
I thought we were talking about NIH, H is for here. If you can't scope your requirements then obviously you need to import something a bit larger than the whole world to solve the whole world's problems.
Dependencies introduce risks, but not using them at all puts you at a competitive disadvantage against those who are using them and thus achieve faster development time and time to market.
What you need is a process to manage dependencies:
1) Only consider open-source dependencies.
2) Introducing new dependencies requires a review. Not just a code review on the pull request introducing it, but checking the license, estimating how much work it would be to rip out, auditing it for history of security vulnerabilities or bugs, whether it is live and still gets updates, how vibrant the community is, and so on.
3) If possible, review your dependencies for possible security issues. Doing this at scale is expensive and the economics of this are still an unsolved problem (I have my ideas: https://blog.majid.info/supply-chain-vetting/).
4) Do not adopt a dependency you are not willing and able to take over the maintenance of, or fork if necessary. At a minimum, it means you have built it from source at least once, not just used binary packages maintained by someone else.
5) Preemptively fork all your dependencies. People can and do delete their repos out of pique, see left-pad.
Both 4) and 5) are very important, but often forgotten.
Even for my own personal (small) projects, i've gotten hit with problems when i take an extended leave of absence and then return to a project, only to find my dependencies have become either completely outdated and unusable, or the repo was entirely deleted (with only a remote copy to build with).
I've since adopted the "fork" method; the dependency's source is forked (privately), and get the dependency built; this is recursively done with their dependencies (stopping at the language level libraries), just to get familiar with their build chain. Only then, will i feel good enough to add the dependency to my project (and use their publicly released artefacts from their publicly hosted library repository). It's a bit of work, but i think this effort is spent upfront, and removes future effort if the project lives long enough to see the ecosystem move/change directions (and you dont want to follow).
Sometimes i do find that source libraries (rather than pre-built binaries) to be better in this sense. I merely need to fork and link the sources into the project, rather than have to learn anything about build chains of dependencies...
For #4, while I agree sometimes, I also recognize that SQLite (and other similar types of dependencies) are 100% going to outlive the product I'm building. So it's a bit arrogant to say my product is going to outlive them. I'm not going to build the Linux kernel just because it's a dependency.
I think it still applies if consider you can build SQLite from a fork of the source and it probably needs no maintenance. In some ways, you could consider the SQLite package that ships with a given distro/OS a mini-fork
Still, surprises happen. Key contributor(s) could become ill, license changes can happen, ownership/stewardship change. Although super popular projects you can _probably_ rely on someone else forking
Iirc one of the MariaDB engines was maintained by some singular dude that would go offline for months at a time and maintained it as a hobby.
4 and 5. Not at least once, all of your code should be at the very least buildable with a network cable yoinked out, preferably without any binary artifacts, but that is not always possible.
With 5, forking may be a bit excessive, as keeping your fork up to date adds a big maintenance burden. Vendoring dependencies (just pushing them into git, git LFS, or operating a caching proxy) is valid though, especially for long-lived software projects. Maybe less so for NodeJS based ones as their dependencies use many small files, but there's Yarn and maybe PNPM that will keep dependencies in more convenient archive files instead.
This is amazingly insightful, thank you. I'm going to copy this to my own procedures files, on every dependency I use in the future.
The only problem I really see is very tall dependency trees, seen in e.g. the JavaScript world.
Being in the energy sector dependencies is something we intentionally avoid because we'd actually have to go through and review changes. What has helped this immensely is AI code assistance. One of the primary uses is to use it to write CLI tools for code and config generation in the tool chain. All of it is around making your life easier, without pulling external dependencies.
An example is that we create openapi docs with LLM's. Then we serve them with Go's net/http + FileServer, meaning that we never leave the standard library. Of course the LLM itself is a third party dependency, but when you use it to write CLI tools that then do the code generation, it never actually sees the code. That also means that the quality of those CLI tools are less important, because it is their output that matters.
It only goes so long of course. I'm not sure any of our front-end engineers would want to live in a world where they weren't allowed to use React, but then, their products are treated as though they are external anyway.
Anyway, it's a lot easier to make engineers stop using "quality of life" dependencies when you give them something that still makes their lives easy.
Doesn't the LLM spit out the code of the dependency it has been trained on? How is that any different from just forking the dependency and treating it like your own?
One advantage might be that you would only implement the (small) part of the library that you actually need, potentially avoiding bugs such as Log4Shell.
The risk of code duplication is a serious one though, and it would be nice if AI could automatically detect and clean this.
I guess we are in an annoying "in-between" state of things, with a pretty unsure future.
The benefit of using a library directly is your 3rd party library checks will warn you when a CVE is found in the version you are using. If an LLM creates the same functionality from copying a version of a library, you might be getting a version that already has known vulnerabilities, and you probably won't be pulling in any fixes/improvements in future until you find them.
Why not both?
Fork the dependency and use that, to have a stable non-changing base which you use. And additionally, make the original project a dependency but don't actually use it. This way you'll get CVE information from your tooling.
If you fork a dependency and change features, the CVE information on original depenency is now no longer valid for your code. Your additions or removals can induce new CVEs, or render CVE for original lib a moot point.
You would (probably) be avoiding commonly known exploits while introducing subtle AI-induced bugs like printing logs out of order or not locking/ordering resources properly.
Oh yes, I fully agree. It will be quite horrible if we don't handle things properly.
Not necessarily, you can task the AI Agent with writing your tool without the use of external dependencies. Often the tool itself will be absolutely horrible, but this doesn't matter if the output of the tool is fine. As in the case of OpenAPI documentation, you basically get the LLM to write the equivalent of C#'s Swashbuckle (or whatever it's called these days) for your language. What it produces is horrible in comparison to the third party dependencies you would have pulled 5 years ago, but the output is the same.
You can also let it use outside dependencies, but then you're right, then it would make little difference in regards to security.
We figured this out because Go didn't have a "Swashbuckle", and nobody wanted to write the documentation or use codegens to go in the opposite direction. When it turned out to be so easy to use LLM's for these things, it grew into it's own thing in regards to replacing a lot of dependencies. Obviously the LLM is still the dependency, but you can replace all other external dependencies for a lot of things like this. Which means you won't acidentially pull something malicious.
I imagine we're going to see more and more of these in-house dependency replacement tools. Coming from Go, I obviously use SQLC (which is run inside it's own isolated container that is only available for requests from developers and is only updated by hand). If we couldn't run SQLC in total isolation then I imagine we would have had to build our own "SQLC". I haven't tried but I'm pretty confident that you could use LLM's to build a similar (but much worse in quality) tool. In an ideal world, we would have been allocated the resources we needed, but in reality, it would have just made us slower as "IT" is still viewed as a cost center on par with HR except that we aren't even "IT".
It’s easier to incrementally review the generated, specific code of an AI agent than it is to review a generic, overly featured long-lived library— and to keep on top of (and review) all changes to that library over time.
[flagged]
Completely agree. It's one of the most important skills to know which dependency is good and which is bad.
My two cents. If a dependency is paid, than it is usually bad. Because the company providing that dependency has an incentive to lock you in.
As another point, "dependency minimalism" is a nice name for it. https://x.com/VitalikButerin/status/1880324753170256005
As well, paid dependencies usually only have one source of support, and when the company goes under or stops supporting the product you are in rough seas.
Given very few companies last forever, you have to assess if the trajectory of your project would be impacted by being coupled to their ability to support you.
For sure, this goes into the terrain of acquisition though, for which there are long-running procedures and assessment criteria. Long-term support / company longevity is one of them.
But likewise, these companies have the incentive to look like they have long-running success and can be relied upon for years / decades to come.
> when the company goes under or stops supporting the product
Or, even worse, gets acquired by someone like Salesforce
Exactly. That's another point
I've experienced some bad paid dependencies forced on us by a non-engineering team. I've had a few good experiences with "open core" kinds of dependencies that are widely used by the community, e.g. sidekiq, and therefore less likely to suddenly vanish one day as they would almost certainly get forked and maintained by others in the same boat.
The upside of paying for something is that, assuming the owner or company is stable, I don't have to worry about some unpaid solo maintainer burning out and never logging back in.
> assuming the owner or company is stable
and continues to be stable for the lifetime of your product.
> My two cents. If a dependency is paid, than it is usually bad. Because the company providing that dependency has an incentive to lock you in.
Vendor lock-in is a risk for both purchased components and FOSS ones where the organization is unwilling to assume maintenance. The onus is on the team incorporating third-party component(s) to manage their risk, identify alternatives as appropriate, and modularize their solutions.
Developers would say this and then deploy to AWS Lambda or Vercel with a straight face
If a dependency is paid and it is bad, then maybe you just aren't paying enough for it.
If my code has a dependency then I want there to be people whose feel it is their job to support it.
Either there have to be enough people who are paid to support it, or there have to be enough people whose self-worth and identity is so wrapped up in the code that they take it as a point of honor to support it.
I don't need a company that's likely to abandon a code product and leave me hanging. I also don't need a free software author who says "bugs are not my problem - you have the source, go fix it yourself." If those are my choices, I'd rather write it myself.
> I want there to be people whose feel it is their job to support it.
their "feeling" will not change reality, which might conflict. For example, a specialized database vendor would prefer that you cannot move away, and even if they feel like they want to support you, there are other economic factors to consider which could override this.
I think the idea is that if you are paying, the dependency needs to implement some sort of open standard/interface to which there are at least one other implementation you could move to. The vendor cannot lock you in with this requirement, since you always would have the option to move (despite it being a bit expensive, it's just there as a threat).
https://opensourcemaintenancefee.org/ uses payments as an incentive to keep projects going, so dependencies can be updated. .NET Rocks! interviewed them https://www.dotnetrocks.com/details/1948
Well, as a battle-scarred old war horse, I can say that "It Depends™."
I have found that "It Depends™" is a mantra for almost everything that I do, and experience has given me the ability to choose a fork in the road, when an "It Depends™" moment happens.
When I was younger, my world was governed by "hard and fast" rules. There could be no exceptions, and some of the worst code I wrote, consisted of the gymnastics required to shoehorn an unsuitable library/paradigm into my projects.
I tend to use a lot of self-developed dependencies. They are really distillations of things that I do frequently, so I factor them into standalone packages. That way, I don't have to reinvent the wheel, and I'm confident of the Quality and provenance.
But if there's functionality that is required, and beyond me, for whatever reason, I am always open to including dependencies, as long as I can reconcile myself to their quality and other issues.
People love to claim this, especially on this site, but in my experience it's the opposite. Many people like writing new code and will do it even when it's detrimental to the business, but 9 times out of 10 even using a "bad" dependency is far more effective than writing in-house.
Dependencies are a double-edged sword.
Most vocal people work on "disposable" end of software. It's cheaper for software giants to just throw engineer-hours at rewriting piece of code that has fallen into organizational obscurity than to maintain (hehe) maintainability. There is usually no sense for small branding/webshop factories to churn out high quality, maintainable code.
However, I suggest you revisit the reason why the dreaded "enterprise patterns" exist in the first place. The main reason to use these architectures is so that five years down the line, when documentation is badly outdated, there is no organizational memory left behind that component, original developers have transitioned to either different teams/departments or left the company altogether, the component is still well isolated, analyzable and possible to work on.
Introduction of external dependency(-ies) carry two inherent business risks: either support for dependency will be dropped, meaning you will have to either absorb maintenance burden yourself or switch dependencies, or it will introduce breaking changes, meaning you have to stick to unmaintained version or update your product code. Both situations will eventually impact your feature flow, whatever it is.
Compromise between trunk and leaf (push of dependencies vs pull of deps) is intrinsic to modularization and is always there, however with internal components this compromise is internal, rather external.
> Many people like writing new code and will do it even when it's detrimental to the business, but 9 times out of 10 even using a "bad" dependency is far more effective than writing in-house.
If you are a SaaS company - most probably yes as it is the short-term outcome that is determinate of business success. However, if you work in any industry with safety and support requirements on software or put the burden of long term support on yourself, long-term horizon is more indicative of business success.
Remember, writing new code is almost never the bottleneck in any mature-ish organization.
> five years down the line, when documentation is badly outdated, there is no organizational memory left behind that component, original developers have transitioned to either different teams/departments or left the company altogether, the component is still well isolated, analyzable and possible to work on.
This will be far more true for an external dependency - even one that's no longer actively developed - than for an internally developed component, IME. Just at the most basic level an external dependency has to have some level of documentation and at least be usable by someone other than the original author to even get picked up.
> Introduction of external dependency(-ies) carry two inherent business risks: either support for dependency will be dropped, meaning you will have to either absorb maintenance burden yourself or switch dependencies, or it will introduce breaking changes, meaning you have to stick to unmaintained version or update your product code. Both situations will eventually impact your feature flow, whatever it is.
Sure, you need stay up to date, potentially even take over maintenance yourself, or accept the risk of not doing so, and none of that is free. But writing an internal implementation basically puts you in the worst-case scenario by default - you have to maintain the code yourself, and it's probably less maintainable than an external codebase.
> But writing an internal implementation basically puts you in the worst-case scenario by default - you have to maintain the code yourself, and it's probably less maintainable than an external codebase
This very much depends on the type of dependency we are talking about. if it's externally-facing, sure, you'll have to maintain a suitable security posture and keep up with spec/requirement changes coming in from the outside world.
If it's an internal-facing dependency, it's reasonably likely that you never have to touch it again once the problem is solved. When I worked at Amazon we had internal libraries that hadn't really changed in the past decade, because they were designed to operate inside controlled environments that insulated them from the changing world outside.
Would you say Log4j is an internal or an external dependency?
External, unfortunately. A library that only wrote log files would be internal, but log4j is one of those open-source solutions that has fallen prey to the kitchen-sink fallacy - bundling network transport and service discovery into your logging library creates a massive attack surface that isn't strictly related to the library's stated function.
> > five years down the line, when documentation is badly outdated, there is no organizational memory left behind that component, original developers have transitioned to either different teams/departments or left the company altogether, the component is still well isolated, analyzable and possible to work on.
> This will be far more true for an external dependency - even one that's no longer actively developed - than for an internally developed component, IME. Just at the most basic level an external dependency has to have some level of documentation and at least be usable by someone other than the original author to even get picked up.
Not sure if I can agree with the sentiment, especially with the "at least be usable by someone other than the original author to even get picked up." part. Components are easily usable either because they sort of adhere to the UNIX philosophy "do one thing and do it well", which makes them inherently well isolated, or they are easy to integrate, falling on the kitchen-sink end of the spectrum, making them inherently hard to isolate. I think it has more to do with the functional domain of the component, rather than source.
> Sure, you need stay up to date, potentially even take over maintenance yourself, or accept the risk of not doing so, and none of that is free.
The more of a kitchen-sink the component is, the more maintenance effort there is and the higher the risks. However, in my previous comment I wanted to highlight integration risks: components on the kitchen-sink end of the spectrum are more likely to break interfaces due to feature work, nudging organization towards trunk based development. It's not inherently worse or better than leaf based development, it's a compromise, however my key point remains that larger external components shift the compromise externally, which is impossible to plan. In my experience, internally-facing external components that diverge too much simply get code frozen instead of getting required support, whereas internal components get maintenance effort allocated.
Can I ask how seriously your company takes security vulnerabilities and licensing? I used to have a pretty lax attitude toward dependencies, but that changed significantly when I joined a company that takes those things very seriously.
I've worked with many companies over the years. License management should be automatic or mostly-automatic if you're taking dependency management seriously (particularly these days where so many projects use well-known open-source licenses and everything is machine-readable metadata), and I've never seen a single in-house codebase (even at e.g. F500 financial institutions) that took security more seriously than your average open-source library.
The author is from New Zealand. You have to understand the Number 8 wire[1] mindset the has taken root there to put this in context.
[1]https://en.wikipedia.org/wiki/Number_8_wire#:~:text=Accordin...
I think just about every experienced developer I know - most of whom are not from NZ - would agree with this article. We've all been burned by using the wrong dependency for the job at some point, or the pain of having to deal breaking changes when upgrading a library that really wasn't necessary to use in the first place.
To push back on this article's thrust, those are actual issues that can occur with dependencies. It's just that they usually don't, and most dependencies are harmless, small, and transparent, so articles like this and people complaining about dependencies are almost always overstating their case.
And therefore, adopting a "zero dependencies" policy is absolutely an overreaction.
I'd argue harmless, small and transparent dependencies are the easiest to avoid. The extreme of it "is_even", but any library that could fit in 200 lines should probably be owned code.
Where the article hits is critical libraries that you heavily rely on and that your code is slowly formatted around. I'm thinking specific data parsers, fancy query brokers, time management libraries etc.
I don't think the author is actually arguing for zero dependencies here. While they do quote a "zero dependency policy" in one open source project as being "instructive", they also propose a "framework for evaluating dependencies" - which of course anyone advocating for zero dependencies would have no need for.
As a developer from NZ I agree
Purely vibes but as a Kiwi I feel like Number 8 Wire mentality has been dead for at least 20 years now.
TIL. I'm reminded of the Australian "She'll buff out, mate" meme.
There's also the distinction between a library - a tool that does one particular thing and does it well - and a framework - which dictates how you structure your entire application. And this is a mindset as well.
I mainly know the original quote in the context of Go, where the community as a whole (or at least the part that is in the Slack server) eschews big frameworks in favor of the standard library, lightweight architectures, and specialized libraries. A lot of people coming from other languages come into the communities and ask about whether there's anything like Spring for Go. Another common request is help with a major framework or abstraction like Gin (web API framework) or GORM (database ORM). But Gin is no longer really necessary as the SDK has good support for the common web API use cases, and GORM quickly breaks down as soon as you need anything more advanced than querying a single table, not to mention it's big and complicated but has only one lead developer (I believe), no major organization behind it.
But Gin and GORM dictate how you structure at least part of your application; especially the latter adds a layer of indirection so you no longer have to ask "how to I join two tables in SQL" but "how do I join two tables in GORM", the latter of which becomes very specialized very fast.
Anyway. Using the standard library is preferable, using small and focused libraries for specific problems that aren't solved by the standard library is also fine. Copy / pasting only the functions you need is also fine.
You can tell it is written by C programmer because they think installing dependencies is hard.
C programmers don't install dependencies. That would be insane. Better to "vendor" them into the same tree as everything else and build them at the same time.
if it's not hard why do we have docker ?
Because while it's not hard, it's boring. Setting up an environment is boring and repetitive work, and Docker reduces that.
If dependency == library, it is hard in C or Python, but it isn’t in Rust or Java.
You could easily run a Rust program on bare Linux, but using Docker might simplify the version management story.
But even if you use a language with a sane dependency management story, you probably depend on PostgreSQL, and running it in Docker can be easier than installing it on the system (since distros really like to mess with the PostgreSQL configuration).
Can Rust program link with Shared C library? If yes then I would argue they have the same problem with dependency version.
Yes it can and yes indeed it's sometimes a pain in the ass.
However, some (maybe even most) C libraries' Rust bindings deal with this automatically when installed via Cargo.
The ubiquity criteria also informs scaling - for example, if a tooling or backend dependency is being production deployed by a company at a scale of 10^2 or 10^3 times your use case, you're much less likely to hit insurmountable performance issues until you get to similar scale.
They're also much more likely to find/fix bugs affecting that scale earlier than you do, and many companies are motivated to upstream those fixes.
Their libraries sometimes don’t even work for low scale though.
The protocol buffer compiler for Swift actually at one point crashed on unexpected fields. Defeating the entire point of protos. The issue happens when only it tries to deserialize from JSON, which I guess none of them actually use due to large scale.
To clarify, I'm not thinking of code/libraries written by a huge company, more about open source code that has been scaled far beyond your deployment size by someone/anyone else.
Also, if you're using some feature that isn't regularly exercised (like your Swift protobuf example), it's probably doesn't have the variety of use to be covered by Hyrum's Law (see https://www.hyrumslaw.com ), which is definitely a different aspect of the Ubiquity criteria.
I don't agree.
Some of the worst bugs I've hit have been in libraries written by very large companies, supposedly "the best and brightest" (Meta, Google, Microsoft, in that order) but it takes forever for them to respond to issues.
Some issues go on for years. I've spent months working in issue trackers discussing PRs and whether or not we can convince some rules-lawyer it doesn't warrant a spec change (HINT: you never convince him), chasing that "it's faster/cheaper/easier to use a 3rd party package" dragon, only to eventually give up, write my own solution, fix the core issue, and do it in less time than I've already wasted. And probably improve overall performance while I'm at it.
I think a lot of it depends on the exact field you're working in. If you're working in anything sniffing close to consulting, work is a constant deluge of cockamamie requests from clients who don't understand they aren't paying you enough to throw together a PhD research thesis in a month with a revolving crew of junior developers you can't grow and keep because the consulting firm won't hire enough people with any amount of experience to give the small handful of senior developers they keep dragging into every God damned meeting in the building so we can have a chance to come up for air every once in a while.
I'm at a point where I have enough confidence in my skills as a software developer that I know pretty much for certain whether I can develop a given solution. There are very few I can't. I'm not going to try to train an AI model on my own. I won't try to make my own browser. A relational database with all the ACID trimmings, no.
But I'll definitely bang out my own agentic system running off of local inference engines. I'll for sure implement an offline HTML rendering engine for the specific reports I'm trying export to an image. I'll build a fugging graph database from scratch because apparently nobody can make one that I can convince anyone to pay for (budget: $0) that doesn't shit the bed once a week.
Most of the time, the clients say they want innovation, but what they really want is risk reduction. They wouldn't hire a consultant if it wasn't about risk, they'd put together their own team and drive forward. Being broadly capable and well-studied, while I may not be quite as fast at building that graph database example as an expert in Neo4j or whatever, we're also not getting that person and have no idea when they are showing up. If they even exist in the company, they're busy on other projects in a completely different business unit (probably not even doing that, probably stuck in meetings).
But I know I can get it done in a way that fits the schedule. Spending time reading the worst documentation known to mankind (Google's) because some drive-by said they did this once and used a Google product to do it is probably going to end in wasting you a lot of time only to realize that said drive-by didn't spend long enough actually listening to the meeting to understand the nuance of the problem. Time that you could have spent building your own and hitting your schedule with certainty.
Sorry, it's late and I'm tired from a full quarter of 12 hour days trying to rescue a project that the previous team did nothing on for the previous quarter because... IDK why. No adults in the room.
Hope you're getting paid well bro
Don't go into consulting. If you do, it's impossible to get out. No product-oriented companies will ever hire you. Wish someone told me that 20 years ago.
This echoes the Joel On Software Classic, "In Defense of Not-Invented-Here-Syndrome", which gives some of the reasons Microsoft Excel won the spreadsheet wars. Of particular note:
=-=-=
“The Excel development team will never accept it,” he said. “You know their motto? ‘Find the dependencies — and eliminate them.’ They’ll never go for something with so many dependencies.”
https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...
NIH can also be great if you only need a subset of a "mature" dependency but you need to really nail it.
Since it is a solved problem you'll likely find ample documentation, algorithms and code examples so there is much less that can go wrong than in a greenfield project.
At my last job we rewrote InfluxDB as it as was too resource-hungry for our embedded device and the result was fantastic. 10x the time series storage at a fraction of the memory and CPU utilization, much more robust and flash-friendly.
"One technique for making software more robust is to minimize what your software depends on – the less that can go wrong, the less that will go wrong. Minimizing dependencies is more nuanced than just not depending on System X or System Y, but also includes minimizing dependencies on features of systems you are using."
From http://nathanmarz.com/blog/principles-of-software-engineerin...
So after 2 mins i figured out what NIH means.
Would be nice to at least explain it once at the beginning thought.
There are many articles that use acronyms and not everyone knows WTH is going on.
"New, in-house" development? Would be my guess. Still left guessing.
"Not Invented Here".
My American brain, of course, immediately went to "National Institute of Health" and thought this was a software developer opining on the current administrations distaste for federal funding of anything.
https://en.wikipedia.org/wiki/Not_invented_here
Database drivers, crypto etc im always "installing", but for 97% of other stuff i tend to roll my own. And when i dont want to reinvent the wheel, i take time to vet the depency.
How many LOC?
Does it has deps of its own?
Is it a maintained library?
Can i fork/fix bugs?
I take neither side in a dependencies vs NIH war. I like using SQLite as a database, and I use Tk (of Tcl/Tk) as my user interface toolkit, for example. Neither of those is going anywhere, they are both maintained, and I build them myself.
I would argue that the right way to think about dependencies, as some have alluded to, is to do a risk/benefit analysis. On the benefit side, how much work would it take me to duplicate this component, how well will it suit my needs, and so on. On the risk side, everything from continued maintenance to security and licence issues (for both free/libre/open-source and proprietary components). Then risk/benefit asks for ways of ameliorating risk, such as by forking code, or paying for support.
The right way to think of adding a dependency is to ask and answer questions like this prior to adding it. This is not the way that JavaScript developers used left_pad, they just said “it's in npm, so it's good.”
I do technical diligence on companies getting purchased or invested in, largely by private equity. Previous to that I was a developer for 15 years (still am, but open source now). One big one not touched on in the article is the cost of security updates. I see this all the time... companies that are in a bad position because they have too many dependencies and have wound up unable (for business reasons) to dedicate adequate resources to keeping them all current, and then end up with past EOL deps with published vulnerabilities. This – literally – costs them a lot at acquisition time. The purchaser (who is typically very risk averse) factors in all the cost to get things up to date and takes it off the price. This can be very significant. (looking at you, React et. al!)
On the other hand, I've seen companies where they used as little in deps as they could get away with, and they wind up in very strong positions. There really is a big labour cost in chasing the latest versions of underlying frameworks when those move too fast.
And here I thought I was going to read a novel defense of the British healthcare system.
NHS?
Haha, I was doubly confused! You’re right, I should have said National Institutes of Health.
Either way you wrap each thing that acts as a dependency, even if it's internal. I treat dependencies another team in my company delivers the same as any other third party dependency. Never use the classes but always just wrap around them in a service or component.
When my 'task' is to publish things on a Kafka bus I create a publishing service that takes in an object that I control, only inside that service is there actual talk about the dependency and preferably even that's a bit wrapped. It's easy to go too far with this but a little bit of wrapping keeps you safe.
> Sometimes it's easier to write it yourself than install the dependency
This is definitely true, but only really relevant if you're either a solo dev or in for the long haul, so to speak. And it'll work way better for your use case too.
The problem is working with others. Others coming in are more apt to try to learn a known or semi-known system than "Dave who used to work here's crazy thoughts." Especially in this market where longevity is rare, and company specific knowledge is useless as resume fodder.
So from a dev standpoint it absolutely makes sense. From a business standpoint, probably markedly less so.
Figuring out how to get others to install a dependency is even worse!
What's NIH
Short for Not Invented Here syndrome... when developers ignore existing libraries or solutions and implement their own version. Usually this is meant negatively, i.e. developers thinking they -need- to implement their own version, at great expense, when they could have just used an off-the-shelf library. However this article is positing that writing your own code is a good thing in some cases.
It is a good thing in some cases for a couple of reasons - first, off the shelf libraries often do a lot more things than are needed for the project at hand which can have the same effect that adding extra lanes to a highway does on traffic - more effort in the same space that could have been avoided. Second, OTS libraries can be opinionated, which winds up bending the existing codebase in ways that aren't optimal or aligned with the other code in a given ecosystem.
Not Invented Here
National Institute of Health
This is what I thought too.
I find both the article and the title a bit too simplistic for the topic. This is the classic "build vs buy" problem and I think it's one of the hardest problems in softare development and maybe engineering in general.
I've been bitten by both sides of the decision many times and apart from trivial cases (like `is-even`), it's hard to come up with universal rule of thumb to decide this, especially for cases where it matters.
I can't say I am that impressed with ECMA-48. Back when it was VT-100s connected to a VAX or logging into a BBS over a 9600 baud modem with a PC I felt satisfied. Even when there was a bit of line noise.
Ever since 1990 or so when it has been using "telnet" or the equivalent on an xterm or CMD.EXE or some other GUI application that emulates I've never trusted it to work 100% right. Maybe 99.5%, but I expect the cursor to be a column off where it is supposed to be, for there to just be a little misalignment here and there. I mean, other people don't complain, but people don't complain when the center channel is burned out at the cinema or when the Linux desktop has a 179px wide label in a 59px wide space and such.
It's one reason I gave up on emacs and just use the 20% of vi functionality I know (hit 'i' and pretend that it's Notepad, as least ESC + :wq doesn't confound my muscle memory for ^S being search the way ^S and ^X^S do) when I am logged into a busted Linux install to get the package manager running again but really use IntelliJ IDEA as much as I can.
> Sometimes it's easier to write it yourself than install the dependency...
A few examples wouldn't hurt...
> Their breaking changes can trigger expensive re-writes of your own code to handle a new interface.
Don't update them or NIH-update them just like you'd do with the original NIH code. Still a net win in saving the time for the initial coding
> You need to ensure they end up on your clients' machine.
Vendoring exists?
The tolerance for the amount of time it takes to build a new thing is a lot looser than the tolerance for the amount of time it takes to fix an old thing in production with PMs screaming about down time.
Since none of the two alternatives guarantee no downtime bugs in production, you're missing some other factor that would make NIH preferable, like, maybe, testing (which the new thing would have but adding the old one wouldn't because ... tolerance)?
NIH is cool and all for greenfield projects but please remember that most of industrial work today is compatibility with old standards and behaviors, dealing with their bugs and edge cases. In that case, a third-party dependency that has been existing for 20 years will be better.
If you have a third-party dependency that survived for 20 years. But what if you are trying to choose what to rely on today, and to decide if it will even exist in 20 years? Certainly none of the fashionable JavaScript frameworks will.
A good question I ask myself is: Can I vendor it in? If I cannot, that’s usually something standard, but also a complex domain (crypto, network protocol, platform framework,…). Anything else, I assume it’s only a short term gain, so I make sure to explore the library and get a good understanding of its internals.
I am failing to find it, but Simon Willison had a post where he vendored some small (<1000 line) blob of code instead of adding it as a dependency. The trick was that he had some Github action(?) automation which would automatically check if the vendored version ever fell out of sync with upstream. Get the benefits of vendoring while minimizing your actual support burden.
Only a realistic strategy for small bits of code, but I have considered do the same for a few utility libraries which are a focused niche, but subtle enough that you would rather outsource the complexity.
Vendoring is somewhat an answer, but not always. I have some python code, vitrualenv is an obvious solution. However, older modules depend on older python3.9 behavior. And python 3.9 EOL is October 2025.
Vendoring is taking the responsibility to maintain it as part of your codebase instead of relying on third party, not just capturing a snapshot.
What's the difference vs a state where you wrote the code many years ago that defends on 3.9?
I mistakenly read the article thinking it was about the National Institute of Health. :-(
For me the "should I use a library for this" comes down to if I want to be able to dip under the abstraction.
For instance: terminal color codes. I'm building for linux, but would like my code to be reasonably portable to other platforms. By inserting my own \033[m... Codes, I take responsibility for doing colors on other platforms too, whereas a dependency absolves me of that.
On the other hand, if I need a graph algorithm like Bellman-Ford, I don't want to reach for a library, because that immediately forces me to use their structs for every graph node. I might also want to tune BF specifically to my graph's shapes, etc.
Interestingly I've only ever ran into this problem (as being a problem) in Advent of Code - you need a certain (more exotic) graph algorithm and you only find an example that goes contrarian to your current model.
At work I don't seem to need these weird algorithms, or they are in the language's standard library/default math or graph package and it's not worth reimplementing anyway.
> whereas a dependency absolves me of that
Except it doesn't. Users don't care if the reason something is broken on their platform is your own code or a dependency.
IME platform abstractions often tend to be a trap because they can only abstract 90% without making compromised whereas if you do it yourself you can pierce the abstraction for the remaining 10% and get the best result on all platforms.
Note that the ECMA-48 escape sequences themselves are the good dependency, not abstractions that hide them, like your tput command or curses or what have you.
I think the OP's article is saying that they are both good and bad depending on the context. You need to evaluate these things on a case by case basis. Writing a little TUI for yourself or your team? Sure, go nuts with escape codes. Making a well supported tool that runs on any developer's machine? Maybe consider curses, there are probably more edge cases than you think.
Trying to find product market fit as a startup? Who gives a duck, find some customers before you care too much about the tech =P cattle not pets etc
What is NIH? I couldn't find it in the article
"Not invented here." Refers to the tendency to avoid using things from outside one's company/organization.
I feel this post. I maintain software in Android, iOS, Python, embedded C, and of late adding Elixir to the mix.
Some in the community will wine about lack of Elixir ecosystem. But often, I’m fine just putting together what I need. It varies. I don’t want to do my own bandit sever or Phoenix live view stuff. But MQTT, no probs.
Often I find that the need for libraries can be an indictment of complexity against the problem space. Take Blootooth as an example. What a huge ball. Though at the end of the day, I’ve done straight to HCI python implementations that for some things are better than the “libraries” available. Don’t get me started on the “hold your hand” gpio libraries for the raspberry pi.
One type of dependency, that I kind of miss is the “copy this code to your project” dependency. You can take complete ownership right away. Whereas with more crafted dependencies, the dependency surface is more than just an algorithm writ large in code, but a whole philosophy/metaphor that may not be an exact fit for your own project.
Dozens of comments and no mention of DIP yet.
You should absolutely use dependencies, and your should be able to tear them out as soon as you don't like them anymore.
Including the dependency can even be part of your NIH if you're so inclined. Instantiate two copies of your code (dep/NIH) and test that they behave the same.
They could've at least added left-pad as an example of a bad dependency, instead of the cop-out
Recently I had a good time avoiding a few dependencies. I am developing a project using Django and it go to a size, where I felt I should add testing to it.
But to test things, I would need instances of model classes. However, I didn't want to have to specify all the required fields every time I needed an instance for a test. The fields might be mandatory, but not always relevant for each test. Such a field needs to be filled, but not necessarily by me in the test code. I knew from earlier projects, that Faker generates random data and in the past I have thought it shouldn't be difficult to write a thing that generates random strings and numbers and ids for fields, and recursively other required entities. So I decided to write it myself [1], rather than introducing factory boy and faker as dependencies.
Sure those libraries will have a lot more features, but my code is simple and does what I need and I gained additional understanding of the complications with Django's model handling.
I also suspect, that factory boy has to hack around some issues, that I avoided by always creating entities in the database, which most of my code is doing anyway. I am not sure how they solved creating instances of models, that have many to many fields without persisting the entities in the database for example, because Django raises an error when one tries to do set a value for such a many to many field, without storing the entity in the database first, so that it gets an id. Which makes sense. So I suspect there is quite some magic in that dependency, that I don't need for now.
What's more is, that my factory functions are just as easy to use, if not easier. I simply do `factory_for_model(MyModel)(kwargs)`. It is also easy to add more functionality like `gen_multiple(factory, count)` for example. Uniqueness is more difficult, due to how Django's fields and models are compared. But as far as I understand neither does Faker do uniqueness out of the box. I may be wrong about that though.
I also developed a decorator for pytest functions, that helps me avoiding long test names and instead puts a test description in the decorator arguments. Those sentence long test names have previously annoyed me and always caused linting to attack me for it.
[1]: https://codeberg.org/ZelphirKaltstahl/web-app-vocabulary-tra...
Test-only dependencies should be maintained separately from functional dependencies.
For standardized things like testing, I'm not sure if DIY makes sense. Even if you use 10% of its capabilities, sticking with an established test framework makes your codebase more maintainable and easier for others to contribute to.
The test framework is established though. I am still using pytest.
Awesome! A literal actual debate between absurd humongous dependencies and the known dumbness of "Not Invented Here". Pick a wrong side folks. (No, "respect for reality" doesn't count as a side. Try again.)
One huge overlooked advantage of using dependencies is knowledge transfer. Software projects (for the moment) require people to write them and people move between jobs. There is a lot of common requirements between different software applications, it would be terrible if everyone just relied on what the standard lib offers.
PS: I assume this would be akin to soviet communism, where the state gives you "everything" and the rest you fashion at home. (being extremely loose here, do forgive me)
Everybody loves NIH until what gets invented is dogshit. Seriously, it's easy to romanticise the idea but wait until you have to work on something that evolved over a decade without any refactoring or documentation, and worse, the author is a lifer who couldn't ever imagine that their code could use improvement. I'll migrate to whatever silly API react router has cooked up this year over living that again, thanks.
Dan Luu quoting Joel Spolsky:
“Find the dependencies — and eliminate them.” When you're working on a really, really good team with great programmers, everybody else's code, frankly, is bug-infested garbage, and nobody else knows how to ship on time.
[..] We didn't think that everyone else was producing garbage but, we also didn't assume that we couldn't produce something comparable to what we could buy for a tenth of the cost.
from https://danluu.com/nothing-works/
After many years using libraries without any sort of vetting process, I completely agree in principle that most code out there is bug infested garbage. One dependency had a concurrency bug so severe it almost cost our company our biggest customer early in our journey. We forked and completely rewrote the library as after looking at the source it was clear they didn’t have nearly as much care with their code as we did. This was the worst case but we faced many bugs in widely used libraries. We can’t replace them all as time is short but if we could we would probably replace most of them.
I'm honestly increasingly skeptical of any third party dependencies as I age through my career.
Before I'll agree to depend on a build tool, package or library, I either need to be convinced that:
With AI NIH will be the new normal.
I wish. As far as I can tell the Venn diagram of people building piles of shit with NPM and people building piles of shit with LLMs seems pretty close to a circle.
Why not ask AI how to use <appropriate library / framework> to implement <thing>?
Doesn’t seem like you can blame NIH on AI more than other motivations for NIH.
Edit to add: If AI makes NIH easier, then it implies that AI is good at solving problems, and speaks to AI’s credit.
From me using Claude Code, without a proper system prompt, Claude generates code rather than using a library, this week e.g. command line parameter and flag parsing. The difficulty is where the tipping point is to use a library, it can't be https://www.npmjs.com/package/is-even
> One of the biggest fallacies in coding is that dependencies have zero downsides.
Am I the only one who hates when articles start with a strawman like this? Or does the author of TFA really know people who think dependencies have zero downsides?
Just the word "dependency" brings to mind infantile helplessness. I cannot believe anyone is out there opining a lack of downsides.
>Instructive here is Tigerbeetle, a financial database written entirely with Vanilla Zig:
How does TigerBeetle's existence support the argument that rolling your own dependencies is better than using off-the-shelf tools?
TigerBeetle has existed as a company for less than three years, they haven't reached a 1.0 release, and they have a single-digit number of engineers. As far as I can tell, they haven't even reached the point of self-sufficiency, as they raised another round of VC funding a year ago.
The pain of rolling your own tools happens as you grow and your needs change. Rolling your own static site generator[0] is fine when you only have a handful of engineers, but I suspect over time, they'll burn more time on their homegrown HTML templating solution[1] than if they had just used a battle-tested, off-the-shelf solution like Hugo.
The only dependency TigerBeetle accepted is Zig, which also hasn't reached a stable 1.0 release and frequently forces developers to rewrite their code to deal with breaking changes, including a major one coming in Zig 0.15.0.[2]
I like TigerBeetle, but they're not proof of anything yet. For every TigerBeetle, there are a hundred profitable companies whose entire business depends on WordPress with 50 plugins they picked at random.
Aside from TigerBeetle, I find most of the arguments in this article weak. They sound like rationalization for a developer who enjoys rolling their own tools and seem not to consider actual business value.
The article also leaves out what I think is the most important consideration in evaluating dependencies: lock-in. How difficult is it to swap out this dependency later? Because if the answer is, "very easy" most of the other considerations don't matter.
There's no use doing an exhaustive survey of image processing libraries when you can swap out your choice with a few hours of dev work. But choosing something like a storage backend is very difficult to change later, so you want to put more thought into it.
[0] https://tigerbeetle.com/blog/2025-02-27-why-we-designed-tige...
[1] https://github.com/tigerbeetle/tigerbeetle/blob/86b85ab60395...
[2] https://github.com/ziglang/zig/pull/24329
> How does TigerBeetle's existence support the argument that rolling your own dependencies is better than using off-the-shelf tools?
At least from my own experience, without TigerStyle [0], there's no way we could have shipped TigerBeetle from 0 to production in 3.5 years, with the same quality, and to the same tolerances: https://jepsen.io/analyses/tigerbeetle-0.16.11
Whereas, for example, if we'd used off-the-shelf dependencies such as Raft or RocksDB, it would have been a nightmare, and taken longer, just to fix them up to solve our storage fault model and implement Protocol-Aware Recovery correctly. Even then, we'd have been stuck with a systems language of the last 30 years (C/C++), when we could have invested in a language for the next 30 and with a more powerful toolchain. Of course, we'd also have been without Deterministic Simulation Testing, because most software dependencies are typically not designed for DST.
But finally (and to your point about sustainability), without TigerStyle, we wouldn't have had the company, or the investment, team, and customers we have today, or be as far along financially. Not saying that you need to share our engineering philosophy. But it works for us.
[0] https://www.youtube.com/watch?v=w3WYdYyjek4
>Often they're so large and complex that simply writing the functionality yourself is faster than learning. Sometimes it's easier to write it yourself than install the dependency...
Sometimes for good reason. Re-implementing something like HTTP with TLS is non-trivial and surely not easier to rewrite yourself...
On the other hand, left-pad.