When people say "this is a VM on xen" I really wish they would provide on essential piece of troubleshooting data. As a domU, is it PV, PVH, full HVM, or HVM with PV drivers? There's a huge difference for how a modern xen dom0 handles the bootloader and kernel boot for those.
At this point trixie is also extremely mature and has been out for a long time, and while I'm not saying this isn't a major issue, if lots of people are seeing it, it's also a solid argument in favor of upgrading those bookworm systems to the latest bookworm and then to trixie to begin following the 13.x stable branch. The 6.12.x kernel in trixie fixes a lot of issues related to both being a hypervisor and running inside a hypervisor.
Technologies such as the venerable chroot, schroot, lxc, lxd, docker, podman or systemd-nspawn have made sure that the applications component can always be independent of an unresolved compatibility across Debian versions.
I would agree with you if trixie had just been released, but a lot of the rough edges have been worn off by now. Running the present branch of debian stable (as opposed to debian testing) is already an extremely conservative choice and gets you a lot of revisions of packages that are quite old compared to what ships in other distros...
To give an example, there is a rough edge called Dovecot 2.4 (new config syntax, Director removed, master/master replication removed) that will not wear off by itself. Upgrading to 2.4 could, depending on the complexity of your setup, require days to weeks of planning, testing and implementation.
Upgrading complex setups is never easy, but people have had over a year to plan, test, and implement their upgrades.
These people will need to upgrade at some point, though that point will probably be at the very last moment midway through 2028 if they're still on oldstable today.
In many case, many software runs on the same server.
Docker don't help much because the image can't build from dockerfile, because some dependency are not downloading.
The hardware requirements of a base CLI only server trixie are not greater than the same base CLI environment of bookworm. we're talking like, 300MB RAM when system booted with full systemd and openssh and a few other standard things running. The only major difference is that 32-bit/i386 cpu support was dropped. If you have an original intel core duo (not core 2 duo) from 2006 that isn't 64-bit, or older, yeah I could see that being a problem... but otherwise,
> I would agree with you if trixie had just been released, but a lot of the rough edges have been worn off by now.
That doesn't help if you need to run a piece of software only certified to run under a certain Java version or, worse, a certain OS version. It's either "run on something really old" or "lose the ability for commercial support / you're on your own if shit happens".
Containers !~= VMs (except Kata Containers and similar). VMs ensure specific, isolated, supported compatible kernel versions in controlled environments because what's supported / should work isn't the same as what's technically possible / might work. At scale, tiny differences matter.
Containers seem lightweight and magical until the tyranny of the quirks of the kernel matters with some gotchas of supported configurations as opposed to unsupported gremlins running custom stuff that other vendors refuse to touch.
eh I'd like a "just ubuntu" requirement much more than the usual "only RHEL is supported" slop...
Yeah I get it, vendors want to keep their support matrix small. But RHEL (and its bunch of derivatives) just plain suck ass. At least offer support for something based on Debian...
OS rollback is possible if you've configured it. A combination of btrfs + timeshift has served me very well after a failed Ubuntu upgrade. You can just boot into the previous snapshot from Grub.
Although you can set this up after the fact (converting ext4 to btrfs is supported and you can update your mounts from any recovery distro), that's probably just something to keep in mind for your next fresh install. It's also extremely for any other kind of failed upgrade, though, not just full OS upgrades.
FYI, for the next time: with Debian, while rolling back is not supported, it often works.
I had issues too with Trixie on a laptop (after upgrade everything became unbearably slow, probably due to an Intel graphic chip which was dropped by Mesa or some other library) and did a rollback to bookworm. IIRC, I only had to put back the old apt source entries then apt update and apt dist-upgrade...
> IIRC, I only had to put back the old apt source entries then apt update and apt dist-upgrade...
The problem with rollbacks is not the packages themselves, that usually works without issues. The problem is with configuration and data items: if a package changes configuration format on upgrade (e.g. move from pkg.conf to pkg.conf.d), a downgrade will not restore the old version (e.g. rebuild pkg.conf from the .d snippets).
Also, apt will not downgrade packages without prodding. At the very least, you would need --allow-downgrades.
Sometimes problems arise when trying to upgrade from "version 1" to "version 3" without passing by "version 2". Moving from oldstable to testing can broke things. You need to upgrade first your sources to stable and then to testing.
Don't know if this is your problem without data. just my 2cents. But you may need to go back two versions and install it properly.
Also if some command "X" don't work, launch it as "strace X" to have a deeper look at the problem.
Not GP, but I've been using Debian for 25+ years and, as a user, I still don't understand how to report a bug the right way when it is unclear what package is the cause for a regression...
The proper way is to figure out which package is causing a regression.
As an alternative you can report a general bug that will end up on the debian-devel mailing list where someone might reassign it to the proper place. But low effort reports sent to generals are basically a waste of everyone's time.
This is only economical if you can afford the time wastage yourself. If you need a working system, you don't have the time to fix the bug yourself (hoping you won't find two more by the time you're done)
I'm not going to take seriously a single person's anecdote of "I upgraded to trixie, stuff broke" when I weigh it against the large number of people I know who run debian stable with very few operating-system-base caused problems for very serious production things.
Your comment is devoid of any useful technical detail on what broke or why, and quite frankly reads like a very linux noob complaint. Leading me to think you did something wrong in the upgrade process, or failed to manually handle any basic configuration files that you were prompted with diffs/merge/install/keep existing version in the upgrade process.
> Your comment is devoid of any useful technical detail on what broke or why, and quite frankly reads like a very linux noob complaint. Leading me to think you did something wrong in the upgrade process, or failed to manually handle any basic configuration files that you were prompted with diffs/merge/install/keep existing version in the upgrade process.
Why is your first reaction is to assume the person broke something in the upgrade process or did something in the config files? The parent comment is devoid of useful technical detail, but assuming the user is at fault is terrible. Debian is very conservative between releases with not upgrading packages from upstream and keeps a lot out of tree patches. It is very possible something broke between the major versions.
I mean, linux kernel broke userspace in a couple different instances between oldstable to stable. If his software happened to use those APIs it could crash. Just an example. But a normal person would fix their software (if they have the sources to do that!)
Yes, but there is software you may rely on that has not yet adjusted to Trixie or if it has you have not had the time/resources to test that upgrade path.
As someone who works in IT around administration of 30+ Debian boxes for a decade now: Updates are easy in theory only or if the software doesn't rely on any OS package. Sure, most updates of well written software are easy, but we also had to manually migrate database schemas and similar things.
>Either that or Santa Altman came down the chimney during the night and swapped all their RAM sticks out for bad ones.
Back in the early 1990s, when they had PC Clone shops, some PC Clone shops would buy from others when RAM chips were in short supply. Sometimes they were bad RAM chips and were sold to sabotage the other PC Clone shop.
Yeah I moved off Debian to red flag, they’re really good linuxers, runs really fast and in half the ram and cpu
(Happy I upgraded to trixie two weeks ago.)
> From: Salvatore Bonaccorso <carnil@debian.org>
> I think this is this one: https://lore.kernel.org/linux-ext4/20261102-ext4-ntfs3-null-...
https://salsa.debian.org/kernel-team/linux/-/merge_requests/...
Seems people are happy with the proposed fix.
When people say "this is a VM on xen" I really wish they would provide on essential piece of troubleshooting data. As a domU, is it PV, PVH, full HVM, or HVM with PV drivers? There's a huge difference for how a modern xen dom0 handles the bootloader and kernel boot for those.
At this point trixie is also extremely mature and has been out for a long time, and while I'm not saying this isn't a major issue, if lots of people are seeing it, it's also a solid argument in favor of upgrading those bookworm systems to the latest bookworm and then to trixie to begin following the 13.x stable branch. The 6.12.x kernel in trixie fixes a lot of issues related to both being a hypervisor and running inside a hypervisor.
You can get mostly the same kernel 6.12 from debían Trixie in debian bookworm using backports, it was recently updated too
Nobody runs oldstable by choice.
Often there are unresolved compatibility issues across Debian versions. Some are not necessarily tied to the kernel but other components.
"Just upgrade bro" can be a recipe for punishment.
A customer of mine has been running a Debian 10 until a couple of months ago.
"It's inside our VPN, no customer accesses that server, what could happen?"
We upgraded only to be able to run a newer Java version.
Technologies such as the venerable chroot, schroot, lxc, lxd, docker, podman or systemd-nspawn have made sure that the applications component can always be independent of an unresolved compatibility across Debian versions.
And it has been the case for literally decades.
I would agree with you if trixie had just been released, but a lot of the rough edges have been worn off by now. Running the present branch of debian stable (as opposed to debian testing) is already an extremely conservative choice and gets you a lot of revisions of packages that are quite old compared to what ships in other distros...
Trixie is 14 months old now
To give an example, there is a rough edge called Dovecot 2.4 (new config syntax, Director removed, master/master replication removed) that will not wear off by itself. Upgrading to 2.4 could, depending on the complexity of your setup, require days to weeks of planning, testing and implementation.
Upgrading complex setups is never easy, but people have had over a year to plan, test, and implement their upgrades.
These people will need to upgrade at some point, though that point will probably be at the very last moment midway through 2028 if they're still on oldstable today.
Spending mythical man months running an upgrade just to deliver the exact same service is a hard sell for the budget season.
As a small self hoster, I found it easier tot upgrade the stack to Stalwart than to fix the Dovecot config.
Not only did a parallel subcomment point out the year they had to find some weeks or even months to plan an upgrade... but there are also dovecot 2.3 containers for the desperate: https://hub.docker.com/layers/dovecot/dovecot/2.3.21.1/image...
In many case, many software runs on the same server. Docker don't help much because the image can't build from dockerfile, because some dependency are not downloading.
Hardware doesn't grow younger magically. Upgrading is not always an option, even as a release becomes mature.
The hardware requirements of a base CLI only server trixie are not greater than the same base CLI environment of bookworm. we're talking like, 300MB RAM when system booted with full systemd and openssh and a few other standard things running. The only major difference is that 32-bit/i386 cpu support was dropped. If you have an original intel core duo (not core 2 duo) from 2006 that isn't 64-bit, or older, yeah I could see that being a problem... but otherwise,
> I would agree with you if trixie had just been released, but a lot of the rough edges have been worn off by now.
That doesn't help if you need to run a piece of software only certified to run under a certain Java version or, worse, a certain OS version. It's either "run on something really old" or "lose the ability for commercial support / you're on your own if shit happens".
Can't you just put that in a container with an old version of Java?
Older Java versions aren't container aware, thus it might not run as expected, depending on which version it is.
sometimes, sure but
> "lose the ability for commercial support / you're on your own if shit happens"
is not something you can clever your way into. It's the vendors choice what support characteristics look like.
Containers !~= VMs (except Kata Containers and similar). VMs ensure specific, isolated, supported compatible kernel versions in controlled environments because what's supported / should work isn't the same as what's technically possible / might work. At scale, tiny differences matter.
Containers seem lightweight and magical until the tyranny of the quirks of the kernel matters with some gotchas of supported configurations as opposed to unsupported gremlins running custom stuff that other vendors refuse to touch.
oh yes, i've almost given up trying to run FPGA software on linux, official tools for some vendors require JDK 11 (sigh) or just ubuntu (double sigh)
eh I'd like a "just ubuntu" requirement much more than the usual "only RHEL is supported" slop...
Yeah I get it, vendors want to keep their support matrix small. But RHEL (and its bunch of derivatives) just plain suck ass. At least offer support for something based on Debian...
How do I best explain this?
I upgraded to Trixie. Stuff broke.
I rolled back to Bookworm, which was extremely painful, because unlike other OSes rollback is not possible and a reinstall is necessary.
Stuff worked again.
Maybe I'll give it another go in 2028 when it goes EOL. Or maybe not. I'm not under any obligation to keep trying.
OS rollback is possible if you've configured it. A combination of btrfs + timeshift has served me very well after a failed Ubuntu upgrade. You can just boot into the previous snapshot from Grub.
Although you can set this up after the fact (converting ext4 to btrfs is supported and you can update your mounts from any recovery distro), that's probably just something to keep in mind for your next fresh install. It's also extremely for any other kind of failed upgrade, though, not just full OS upgrades.
FYI, for the next time: with Debian, while rolling back is not supported, it often works.
I had issues too with Trixie on a laptop (after upgrade everything became unbearably slow, probably due to an Intel graphic chip which was dropped by Mesa or some other library) and did a rollback to bookworm. IIRC, I only had to put back the old apt source entries then apt update and apt dist-upgrade...
> IIRC, I only had to put back the old apt source entries then apt update and apt dist-upgrade...
The problem with rollbacks is not the packages themselves, that usually works without issues. The problem is with configuration and data items: if a package changes configuration format on upgrade (e.g. move from pkg.conf to pkg.conf.d), a downgrade will not restore the old version (e.g. rebuild pkg.conf from the .d snippets).
Also, apt will not downgrade packages without prodding. At the very least, you would need --allow-downgrades.
Was a snapshot before upgrading not possible?
Why isn’t Ghost or Time Machine a thing in Linux
(I guess tar czf would work)
essentially the same idea as ghost:
https://clonezilla.org/
You didn't have a backup? Or at least make a backup before upgrade?
Sometimes problems arise when trying to upgrade from "version 1" to "version 3" without passing by "version 2". Moving from oldstable to testing can broke things. You need to upgrade first your sources to stable and then to testing.
Don't know if this is your problem without data. just my 2cents. But you may need to go back two versions and install it properly.
Also if some command "X" don't work, launch it as "strace X" to have a deeper look at the problem.
How do i explain this?
If you don't report the bugs of what breaks (possibly while it's in testing) nobody can help you.
Not GP, but I've been using Debian for 25+ years and, as a user, I still don't understand how to report a bug the right way when it is unclear what package is the cause for a regression...
The proper way is to figure out which package is causing a regression.
As an alternative you can report a general bug that will end up on the debian-devel mailing list where someone might reassign it to the proper place. But low effort reports sent to generals are basically a waste of everyone's time.
This is only economical if you can afford the time wastage yourself. If you need a working system, you don't have the time to fix the bug yourself (hoping you won't find two more by the time you're done)
You can pay freexian to fix it for you.
I'm not going to take seriously a single person's anecdote of "I upgraded to trixie, stuff broke" when I weigh it against the large number of people I know who run debian stable with very few operating-system-base caused problems for very serious production things.
Your comment is devoid of any useful technical detail on what broke or why, and quite frankly reads like a very linux noob complaint. Leading me to think you did something wrong in the upgrade process, or failed to manually handle any basic configuration files that you were prompted with diffs/merge/install/keep existing version in the upgrade process.
> Your comment is devoid of any useful technical detail on what broke or why, and quite frankly reads like a very linux noob complaint. Leading me to think you did something wrong in the upgrade process, or failed to manually handle any basic configuration files that you were prompted with diffs/merge/install/keep existing version in the upgrade process.
Why is your first reaction is to assume the person broke something in the upgrade process or did something in the config files? The parent comment is devoid of useful technical detail, but assuming the user is at fault is terrible. Debian is very conservative between releases with not upgrading packages from upstream and keeps a lot out of tree patches. It is very possible something broke between the major versions.
I mean, linux kernel broke userspace in a couple different instances between oldstable to stable. If his software happened to use those APIs it could crash. Just an example. But a normal person would fix their software (if they have the sources to do that!)
Classic Linux bro response. Doing a very normal thing (system upgrade) leads to an abnormal result (broken software), blame the user.
Sruff broke and broken software literally doesn't mean anything.
FWIW people use the very same words when they forgot to turn on the monitor or plug their desktop computer to an outlet.
Yes, but there is software you may rely on that has not yet adjusted to Trixie or if it has you have not had the time/resources to test that upgrade path.
As someone who works in IT around administration of 30+ Debian boxes for a decade now: Updates are easy in theory only or if the software doesn't rely on any OS package. Sure, most updates of well written software are easy, but we also had to manually migrate database schemas and similar things.
I always set two different kernels at boot because of this, an LTS one and the mainline version.
Debian based distribution generally keep older kernel by default for that reason
Well, with every upgrade, apt suggests that old linux-image packages be removed.
That works out because it does that while you're already booted into the other kernel.
So what? Count the remaining ones.
I am glad I stayed with Ubuntu after the recent MPTCP kernel incident. SCNR
Weren't people criticising Ubuntu recently for putting the TCP bug kernel update through thorough testing?
no, for delaying patch for a bug 3+ weeks
Oof. This is bad.
I couldn't understand which Debian ran this strange kernel version, until I realized all my oldstable boxes run a backport kernel.
Phew.
Because oldstable was... pretty freaking old even when it was new.
Either that or Santa Altman came down the chimney during the night and swapped all their RAM sticks out for bad ones.
>Either that or Santa Altman came down the chimney during the night and swapped all their RAM sticks out for bad ones.
Back in the early 1990s, when they had PC Clone shops, some PC Clone shops would buy from others when RAM chips were in short supply. Sometimes they were bad RAM chips and were sold to sabotage the other PC Clone shop.
And this affects stable versions how?