I wonder if we should just universally accept that live patching should become part of the linux kernel? An automatic job that updates (much like some system packages in some distros) that installs (signed) live patches from upstream? Of course we would run into a problem where a malicious patch can now be distributed reliably to hundreds of thousands of machines, but we already have that at a lower level with normal application updates.
Canonical has thus far proved that it can be safe, but they're also a massive organization that is locking this feature for $200/yr for any commercial use.
It would be neat if such patches could retroactively replace tagged functions that have identical sematics so that means it would automatically get backported without extra effort from the maintainers.
Why would the source of the patches be less trusted than the source of (updated) kernels? I expect it to be the same, your distro.
$200/year is peanuts for any commercial use worth the name. The problem, of course, is the whole non-free infrastructure it has to introduce.
I wonder when large and critical OSS projects will start to be seen as a public good they are, with large corporations willingly financing them because not doing so is bad PR.
After the npm supply chain attacks people suggested automating delays before installing updates, now we're talking about automating update delivery... I'm afraid there won't be any easy or quick fix after decades of treating security as an afterthought.
Linux distros are not npm. It doesn't mean they are infallible to malicious actors, but I believe it is possible to make them infallible for some small set of packages at least.
Attacks are still possible, but if we look at xz backdoor attack[1] it was insanely complicated attack and it still failed. Its fail doesn't look promising, attack could succeed just the attacker was unlucky. Still it shows that the success is not guaranteed.
Theoretically npm can be improved in this way, if there were a separate "distro" for packaged, with dedicated maintainers for packages, who don't write code, just pull it from a mainstream and review it. It is not being done because of tragedy of commons, not because it is impossible.
Whenever you read about an incredibly unlucky criminal, there's a chance that the unlucky event is a parallel construction to the classified real reason why they were caught. Not sure how exactly that would have worked in this case.
> I wonder if we should just universally accept that live patching should become part of the linux kernel?
I think we can learn many lessons from the recent SNAFUs before going all wild on auto-patching.
One lesson for example is that you shouldn't compile into the kernel modules that only about 0.00001% of all Linux installations out there are ever going to use.
Another lesson is that even if the modules are compiled, but not into the kernel, they should probably be blacklisted (preventing them from loading) by default and only removed from the blacklist by people who really know they'll need these rarely used modules.
We're way past the "but it needs to work on all cases": we're now into the "users installing our distro are getting hacked left and right" territory.
In any case I think many things can be done before Linux distros reproduce the "security" practices of the NPM ecosystem.
> we're now into the "users installing our distro are getting hacked left and right" territory
Are we? Are users actually getting hacked, or have they theoretically been exposed to problems that could allow local privileged escalation if exploited but that nobody's seen used in the wild?
(Edit: To be clear, I'm skeptical but this isn't a completely rhetorical question. If there are actual reports of these vulns causing problems, that would strongly incentivize a stronger response.)
It used to be relatively standard even on the "big" distros to compile your own kernel if you needed something outside of the bog-standard. Modularization and all the related auto-detect auto-mod tools have resulted in most distros shipping a "works for almost everyone" kernel that has everything available as a module.
It seems like a reasonable middle ground for most distros is to put things in kernel modules, but then package those modules into separate packages. If you don't need somedriver.ko, then you don't `apt install linux-driver-somedriver`; if you do need it, just install the package and it just works without needing to compile anything and you get automatic updates and everything.
For Gentoo, of course, "just recompile the kernel as desired" is more reasonable, though they have binary packages including for the kernel and I don't see why the same idea shouldn't work there.
>but then package those modules into separate packages. If you don't need somedriver.ko, then you don't `apt install linux-driver-somedriver
But I don't want to know what drivers I need and will need next. Tomorrow I could buy a different wifi module and then what? Spend 3 hours googling which rtl378326973268632aahaxhabt.ko to install? Thanks but no thanks.
If we are looking at things like gvisor or firecracker, SELinux might be an alternative. From what I can see, SELinux prevented both copy fail and dirty frag, and maybe also fragnesia but I couldn't find any definitive answer on that one.
Last time I tried it was a pain to setup and a pain to use, but as a sysadmin there is a lot of thing that share those attributes. The only question if its worth it. If the current avalanche of patches continues it might.
> Like, running emerge -u @world on a regular basis
You can run emerge -u sys-kernel/whatever-kernel-u-use, maybe followed by `cd /usr/src/linux; make bzImage modules install modules_install...` well, probably you'd use genkernel or something like that, instead of hand-crafted scripts.
The point is: `emerge -u @world` can run into issues esp. if you customized a lot, it can't be automated fully, but I've never run into any issues with updating the kernel, and it can be automated.
It is not so hard to upgrade kernel, the issue is with the reboot you need do automate. Or with live patching, which doesn't seem encouraging, as you say.
Clearly, the future is LLM-generated patches that get instantly vibecoded and installed on all machines without any human review. In fact, this is such a good idea that it should be illegal and impossible to run your computer without being connected to such a system. There are no other alternatives. /sarcasm
Many distros deal with the problem of learning about these issues the same time as the public. Some have fast track processes to ensure patches can get into their stable/rolling releases but it is still a lot of work (especially as kernel updates usually mean that automatic updates won't fully shipped you (without alsp automatically rebooting after an update)).
All of them need to do it. There maybe differences, like different number of versions of kernel supported, so less of backporting, but still distros have to provide fixed kernels.
With Gentoo I believe it is more fun, because of all the options gentoo provides out of a box. More kernels, more work to do.
When I used Gentoo the normal was to install gentoo-sources, which gives you the kernel source code but doesn't compile it. You then have to compile and install the kernel yourself without any support from the package manager.
If you're running on a different platform then perhaps you need the raspberrypi or asahi kernel
Expanding on gentoo's recommendations:
I wonder if we should just universally accept that live patching should become part of the linux kernel? An automatic job that updates (much like some system packages in some distros) that installs (signed) live patches from upstream? Of course we would run into a problem where a malicious patch can now be distributed reliably to hundreds of thousands of machines, but we already have that at a lower level with normal application updates.
Canonical has thus far proved that it can be safe, but they're also a massive organization that is locking this feature for $200/yr for any commercial use.
It would be neat if such patches could retroactively replace tagged functions that have identical sematics so that means it would automatically get backported without extra effort from the maintainers.
Why would the source of the patches be less trusted than the source of (updated) kernels? I expect it to be the same, your distro.
$200/year is peanuts for any commercial use worth the name. The problem, of course, is the whole non-free infrastructure it has to introduce.
I wonder when large and critical OSS projects will start to be seen as a public good they are, with large corporations willingly financing them because not doing so is bad PR.
Public goods are not generally funded by large corporations.
After the npm supply chain attacks people suggested automating delays before installing updates, now we're talking about automating update delivery... I'm afraid there won't be any easy or quick fix after decades of treating security as an afterthought.
Linux distros are not npm. It doesn't mean they are infallible to malicious actors, but I believe it is possible to make them infallible for some small set of packages at least.
Attacks are still possible, but if we look at xz backdoor attack[1] it was insanely complicated attack and it still failed. Its fail doesn't look promising, attack could succeed just the attacker was unlucky. Still it shows that the success is not guaranteed.
Theoretically npm can be improved in this way, if there were a separate "distro" for packaged, with dedicated maintainers for packages, who don't write code, just pull it from a mainstream and review it. It is not being done because of tragedy of commons, not because it is impossible.
[1] https://en.wikipedia.org/wiki/XZ_Utils_backdoor
Whenever you read about an incredibly unlucky criminal, there's a chance that the unlucky event is a parallel construction to the classified real reason why they were caught. Not sure how exactly that would have worked in this case.
I dont belive in live patching unless you are AWS.
But I absolutely belive we should have a method for changing kernel configuration (e.g. kernel module blacklists) and syscall firewalls and alike.
> I wonder if we should just universally accept that live patching should become part of the linux kernel?
I think we can learn many lessons from the recent SNAFUs before going all wild on auto-patching.
One lesson for example is that you shouldn't compile into the kernel modules that only about 0.00001% of all Linux installations out there are ever going to use.
Another lesson is that even if the modules are compiled, but not into the kernel, they should probably be blacklisted (preventing them from loading) by default and only removed from the blacklist by people who really know they'll need these rarely used modules.
We're way past the "but it needs to work on all cases": we're now into the "users installing our distro are getting hacked left and right" territory.
In any case I think many things can be done before Linux distros reproduce the "security" practices of the NPM ecosystem.
> we're now into the "users installing our distro are getting hacked left and right" territory
Are we? Are users actually getting hacked, or have they theoretically been exposed to problems that could allow local privileged escalation if exploited but that nobody's seen used in the wild?
(Edit: To be clear, I'm skeptical but this isn't a completely rhetorical question. If there are actual reports of these vulns causing problems, that would strongly incentivize a stronger response.)
It used to be relatively standard even on the "big" distros to compile your own kernel if you needed something outside of the bog-standard. Modularization and all the related auto-detect auto-mod tools have resulted in most distros shipping a "works for almost everyone" kernel that has everything available as a module.
Perhaps we should tend toward the first.
It seems like a reasonable middle ground for most distros is to put things in kernel modules, but then package those modules into separate packages. If you don't need somedriver.ko, then you don't `apt install linux-driver-somedriver`; if you do need it, just install the package and it just works without needing to compile anything and you get automatic updates and everything.
For Gentoo, of course, "just recompile the kernel as desired" is more reasonable, though they have binary packages including for the kernel and I don't see why the same idea shouldn't work there.
>but then package those modules into separate packages. If you don't need somedriver.ko, then you don't `apt install linux-driver-somedriver
But I don't want to know what drivers I need and will need next. Tomorrow I could buy a different wifi module and then what? Spend 3 hours googling which rtl378326973268632aahaxhabt.ko to install? Thanks but no thanks.
So why can't someone (probably the distro) build a utility that detects the hardware and installs the required kernal module?
We can have security and convenience.
On older versions of Windows you used to get popups saying new hardware is detected, would you like to install the driver now?
> We recommend exploring ways to automate upgrading your kernel
Like, running emerge -u @world on a regular basis, or ...
/me searches
Okay, so https://wiki.gentoo.org/wiki/Live_patching exists but says,
> A note of caution: Kernel live patching is risky. Count on hard freezing or panics to become normal...
That's not encouraging.
---
Another approach: Can we make the kernel vulns less important? Has anyone had luck moving more things to run under gvisor or firecracker or such?
If we are looking at things like gvisor or firecracker, SELinux might be an alternative. From what I can see, SELinux prevented both copy fail and dirty frag, and maybe also fragnesia but I couldn't find any definitive answer on that one.
Last time I tried it was a pain to setup and a pain to use, but as a sysadmin there is a lot of thing that share those attributes. The only question if its worth it. If the current avalanche of patches continues it might.
> Can we make the kernel vulns less important?
How about strong virtualization? https://qubes-os.org
> Like, running emerge -u @world on a regular basis
You can run emerge -u sys-kernel/whatever-kernel-u-use, maybe followed by `cd /usr/src/linux; make bzImage modules install modules_install...` well, probably you'd use genkernel or something like that, instead of hand-crafted scripts.
The point is: `emerge -u @world` can run into issues esp. if you customized a lot, it can't be automated fully, but I've never run into any issues with updating the kernel, and it can be automated.
It is not so hard to upgrade kernel, the issue is with the reboot you need do automate. Or with live patching, which doesn't seem encouraging, as you say.
Clearly, the future is LLM-generated patches that get instantly vibecoded and installed on all machines without any human review. In fact, this is such a good idea that it should be illegal and impossible to run your computer without being connected to such a system. There are no other alternatives. /sarcasm
Is Gentoo an outlier or do all Linux distributions deal with this problem?
Many distros deal with the problem of learning about these issues the same time as the public. Some have fast track processes to ensure patches can get into their stable/rolling releases but it is still a lot of work (especially as kernel updates usually mean that automatic updates won't fully shipped you (without alsp automatically rebooting after an update)).
All of them need to do it. There maybe differences, like different number of versions of kernel supported, so less of backporting, but still distros have to provide fixed kernels.
With Gentoo I believe it is more fun, because of all the options gentoo provides out of a box. More kernels, more work to do.
Not all these directories are different kernel packages, but anything with -kernel or -sources at the end is.When I used Gentoo the normal was to install gentoo-sources, which gives you the kernel source code but doesn't compile it. You then have to compile and install the kernel yourself without any support from the package manager.
If you're running on a different platform then perhaps you need the raspberrypi or asahi kernel