IMO the best practice is to leave dependencies unpinned, but use a lock file, and only update the lock file a few times a year.
Upgrade enough that you don’t get stuck, but not often enough to expose yourself to supply chain attacks every time CI runs.
That just means that you have whatever % chance of catching an attack on the days you do upgrade, and then in the event you do, stay compromised for a large % of the year until your next bump though?
Constantly upgrading offers more days with that % chance of catching one, but at least means you'll see the fix or release-pull sooner too.
The industry standard for the best peace of mind is for ALL dependencies to be pinned, both the lockfile and the dependencies.
Upgrades are done manually and all characters such as "^", "*", next to the version are removed for a fixed predictable version to avoid unexpected version bumps or package hijacked in-case if they are compromised.
Then you haven’t looked at a lot of open source repos? A ton of them are using renovate/dependabot which are doing exactly that for you. And it is certainly good practice.
It's another startup sales pitch. Their argument is the entire ecosystem is screwed, but it's okay, you can run their uber complicated toolchain in a CI pipeline and that will fix everything, by reviewing all of the code of your dependency updates. (because all the other backdoored code was never reviewed? because you're better at reviewing upstream code than the upstream maintainers?)
My take is the "hot take" nobody likes to hear. I think you should actually follow standard security best practices. Don't update constantly to the latest bleeding edge versions, but do update to the latest security patched versions. Do pin your versions (and SHA hashes of releases). Do keep an artifact repository. Do cryptographic verification of artifacts. Do validate every dependency you add, understand who made it, what it does. Do try to minimize your dependencies. Do review every new dependency and see what it is you're pulling into your application and whitelist the sources and their signed keys. Do use code signing. Do use OAuth, ephemeral keys, MFA, certificates.
Linux distributions have been secure for a long time by following these practices. Even their unstable branches. If they can do it, you can do it.
Exactly. And for bonus points, the first part of the article seems to be arguing for the common sense stuff you describe--but then the article suddenly pivots to "use our new shiny tool instead". Huh?
Yes, every time I get message that some dependency is needed to update, I get anxious what could go wrong not to mention recent situation with malware or robware silent distribution.
It doesn’t matter whether it is Wordpress, Python, Nodejs, PHP, to name a few.
I understand that updates are necessary but we need to change the way we do them.
> The old operating model was indeed fine in a much smaller, simpler tech world,
The thing with the 'old days' that does not happen now is developers would go out of their way to make sure new releases were API backward compatible. Now it seems anything goes. If the API changed, a new function may be created or the old function would use variable arguments to allow the older functionality.
With that said, providing backward compatibility seems to be hard these days due to a BOM (Bill of material/dependencies) that make a BOM of submarine look simple.
That ssh/xz issue kind of highlighted how complex things have gotten.
Usually, there are CI/CD pipelines, but also change and testing regimes of some sort. Dev/QA too before prod.
Here is my suggestion though:
- Don't use package repos on the internet, host local caches
- Update your package caches twice a week, not more frequently
- Unpin and update packages in dev as it makes sense, as soon as the package repos update
- QA whenever your team can actually do testing
- Prod, with your next release, no more frequently than 90 days
There is a reason public CVE disclosure is ballpark 90 days per ethical security research and testing norms. That's how long proper testing and release is expected to take.
If you have no specific reason to update a package, and it passes integration tests with more up to date packages, then don't update. Supply chain compromises aside, bugs have a way of creeping. Things get missed with tests. Either you take this careful approach, or you take the other extreme approach where you design everything so well and with so much costly infrastructure and people to manage it, that impact is priced into the availability and threat modeling, which almost no one can pull off except on paper (well.. I said almost...).
Agree on how scary it is to do npm upgrade right now.
But I don't understand why Dependabot is bad but Mendral is good. Why Dependabot will be an attack vector and Mendrol will not?
IMO the best practice is to leave dependencies unpinned, but use a lock file, and only update the lock file a few times a year. Upgrade enough that you don’t get stuck, but not often enough to expose yourself to supply chain attacks every time CI runs.
That just means that you have whatever % chance of catching an attack on the days you do upgrade, and then in the event you do, stay compromised for a large % of the year until your next bump though?
Constantly upgrading offers more days with that % chance of catching one, but at least means you'll see the fix or release-pull sooner too.
The industry standard for the best peace of mind is for ALL dependencies to be pinned, both the lockfile and the dependencies.
Upgrades are done manually and all characters such as "^", "*", next to the version are removed for a fixed predictable version to avoid unexpected version bumps or package hijacked in-case if they are compromised.
> The industry standard for the best peace of mind
I read about "industry standards" in software and never see them in the wild.
Odd to assume your own direct experience is uniformly distributed belief.
Then you haven’t looked at a lot of open source repos? A ton of them are using renovate/dependabot which are doing exactly that for you. And it is certainly good practice.
I see. Some open source repos are "the industry".
Got it.
It's a really weird hill to die on.
[flagged]
It's another startup sales pitch. Their argument is the entire ecosystem is screwed, but it's okay, you can run their uber complicated toolchain in a CI pipeline and that will fix everything, by reviewing all of the code of your dependency updates. (because all the other backdoored code was never reviewed? because you're better at reviewing upstream code than the upstream maintainers?)
My take is the "hot take" nobody likes to hear. I think you should actually follow standard security best practices. Don't update constantly to the latest bleeding edge versions, but do update to the latest security patched versions. Do pin your versions (and SHA hashes of releases). Do keep an artifact repository. Do cryptographic verification of artifacts. Do validate every dependency you add, understand who made it, what it does. Do try to minimize your dependencies. Do review every new dependency and see what it is you're pulling into your application and whitelist the sources and their signed keys. Do use code signing. Do use OAuth, ephemeral keys, MFA, certificates.
Linux distributions have been secure for a long time by following these practices. Even their unstable branches. If they can do it, you can do it.
> It's another startup sales pitch.
Exactly. And for bonus points, the first part of the article seems to be arguing for the common sense stuff you describe--but then the article suddenly pivots to "use our new shiny tool instead". Huh?
I think a way to do is to reduce how many dependencies are needed.
Yes, every time I get message that some dependency is needed to update, I get anxious what could go wrong not to mention recent situation with malware or robware silent distribution.
It doesn’t matter whether it is Wordpress, Python, Nodejs, PHP, to name a few.
I understand that updates are necessary but we need to change the way we do them.
If I had a solution, I would post it here…
All of this is really about web dev, not s/w dev in general.
npm and pip are curses on the planet.
One point right from the beginning of the article: Why would you EVER leave a public port open straight to phpMyAdmin?
This is what SSH tunneling is for...
Did you not run phpMyAdmin in the 90s/2000s? Nobody was that secure back then.
> The old operating model was indeed fine in a much smaller, simpler tech world,
The thing with the 'old days' that does not happen now is developers would go out of their way to make sure new releases were API backward compatible. Now it seems anything goes. If the API changed, a new function may be created or the old function would use variable arguments to allow the older functionality.
With that said, providing backward compatibility seems to be hard these days due to a BOM (Bill of material/dependencies) that make a BOM of submarine look simple.
That ssh/xz issue kind of highlighted how complex things have gotten.
you just update them and that's it?
Usually, there are CI/CD pipelines, but also change and testing regimes of some sort. Dev/QA too before prod.
Here is my suggestion though:
- Don't use package repos on the internet, host local caches - Update your package caches twice a week, not more frequently - Unpin and update packages in dev as it makes sense, as soon as the package repos update - QA whenever your team can actually do testing - Prod, with your next release, no more frequently than 90 days
There is a reason public CVE disclosure is ballpark 90 days per ethical security research and testing norms. That's how long proper testing and release is expected to take.
If you have no specific reason to update a package, and it passes integration tests with more up to date packages, then don't update. Supply chain compromises aside, bugs have a way of creeping. Things get missed with tests. Either you take this careful approach, or you take the other extreme approach where you design everything so well and with so much costly infrastructure and people to manage it, that impact is priced into the availability and threat modeling, which almost no one can pull off except on paper (well.. I said almost...).