> In this setup, UML is essentially a userspace process that cleverly employs concepts like files and sockets to launch a new Linux kernel instance capable of running its own processes. The exact mapping of these processes to the host — specifically, how the CPU is virtualized — is something I’m not entirely clear on, and I’d welcome insights in the comments. One could envision an implementation where guest threads and processes map to host counterparts but with restricted system visibility, akin to containers, yet still operating within a nested Linux kernel.
At least in the first generation of UML, the guest processes are in fact host processes. The guest kernel (a userland process) essentially runs them under ptrace() and catches all of the system calls made by the guest process and rewires them so they do operations inside of the guest kernel. They otherwise run like host processes on host CPU, though.
Completing the illusion, however, the guest kernel also skillfully rewires the guest ptrace() calls so you can still use strace or gdb inside of the guest!
It's good enough that you can go deeper and run UML inside of UML.
> What’s the real-world utility here? Is UML suitable for running isolated workloads? My educated guess is: probably not for most production scenarios.
Back in the day there were hosts offering UML VMs for rent. This is actually how Linode got its start!
Stopped making sense for the mass hosting use case, but it is still wildly useful for setting up and tearing down during dev work, especially when hacking on the kernel itself, which you can trivially attach GDB to
They didn't entirely. It is still maintained, developed even.
> It would seem to be a potentially useful middle ground between docker containers and KVM VMs
Back in the day I actually used it that way for running “VM”s and some firms even sold VPS accounts based on UML. Back then other virtualisation options were not nearly as mature as they soon became, or cost proper money (IIRC VMWare was good by that point but there were no free or reliable OSS options yet), and UML offered better isolation (a full environment including its own root) than simply chrooting a process tree (fuller containers were not a thing back then either, so all users fully existed on the host and you couldn't give out root access net.).
These days things like KVM and more advanced containerisation solve the problems most people want UML for and do so much more efficiently (UML performs badly, compared to other options, where there is a lot of kernel interaction, including any filesystem or network access).
UML is still very useful for its original intent though: testing and debugging certain kernel level items like filesystems (FUSE is competition here in many, but not all, cases), network drivers & filters, and so forth. When things go wrong you can trace into it in ways you can not (as easily) with VMS and containers.
…unless your tests are syscall heavy, then it can be 10-100× slower :(. It also doesn't support SMP. I really wish for a better version of this, we could really use it, especially since the "time stops while CPU is in use" part also means your tests won't randomly fail if the host is heavily loaded and things take longer. Unfortunately, doing something about it is outside my area of expertise.
I was fascinated when I first learned of [FreeBSD Jails], I wonder if right before containerization became a thing the concept was developed further for its requirements (could it have been?) it would have offered a more efficient containerization platform.
Jails are entirely different conceptually from UML; they share the host kernel and are roughly analogous to containers/namespaces. UML is an entirely separate kernel, running as user mode process.
Why do they initialize a disk image with /dev/urandom instead of /dev/zero? Given it's not an encrypted disk container, I don't see any valid reason to do so, but perhaps I'm not seeing something?
It was great. I remember trying it about twenty years ago. The very first time I fired it up, I just typed "linux" at a prompt, and a kernel booted - right there in the terminal.
And then panicked, because it had no root. But hey, I've got a root filesystem right here!
So the second time I typed "linux root=/dev/hda1" (because we had parallel ATA drives back then).
It booted, mounted root, and of course that was the root filesystem the host was booted off.
Anyway it recovered after a power cycle and I didn't need to reinstall, and most importantly I learned not to do THAT again, which is often the important thing to learn.
I used that very method to rescue a machine after the contents of /boot were damaged. Booting from a live CD and building a UML kernel (on another drive, there wasn't enough RAM to do it there & I didn't want to mount any of the main partitions just-in-case) allowed me to boot from the main root and rebuild initrd & friends easily. I felt proper clever!
In supernested, which is a script to see how far you can nest KVM, we do in fact mount the root disk in a VM, but using a snapshot so it's safe(-ish). http://git.annexia.org/?p=supernested.git;a=tree
I've often thought that if only UML would build on Darwin, we'd have a MacOS container solution that didn't need virtualisation. That involves two big unsolved problems though: building UML on not-linux, and building UML on not-x86.
Wait until you realise QEmu (and dosbox) can do this too, while running windows or Dune II, as can old versions of virtualbox (not sure about new versions)
> Today, we’ll explore how you can start an unconventional VM by running a Linux kernel as a process within the Linux kernel itself. This approach doesn’t require installing virtualization software like QEMU, nor does it need root privileges, which opens up some intriguing possibilities.
My point is that VM software, especially older software and emulation software, doesn't require virtualization or root privileges. And yes, this is confusing because QEmu became virtualization software (just like VirtualBox did). Neither originally used hardware virtualization for anything. Dosbox still doesn't.
Hell I wish someone made something that could build dockerfiles and immediately start them as VMs in emulation using just the normal socket api to emulate network.
Yeah, but if you specifically target a Linux kernel to run as a regular user process, you don't have to take detours through CPU emulation code. It should (in theory) be more efficient to call the host's mmap() rather than mucking around an emulated MMU.
> In this setup, UML is essentially a userspace process that cleverly employs concepts like files and sockets to launch a new Linux kernel instance capable of running its own processes. The exact mapping of these processes to the host — specifically, how the CPU is virtualized — is something I’m not entirely clear on, and I’d welcome insights in the comments. One could envision an implementation where guest threads and processes map to host counterparts but with restricted system visibility, akin to containers, yet still operating within a nested Linux kernel.
At least in the first generation of UML, the guest processes are in fact host processes. The guest kernel (a userland process) essentially runs them under ptrace() and catches all of the system calls made by the guest process and rewires them so they do operations inside of the guest kernel. They otherwise run like host processes on host CPU, though.
Completing the illusion, however, the guest kernel also skillfully rewires the guest ptrace() calls so you can still use strace or gdb inside of the guest!
It's good enough that you can go deeper and run UML inside of UML.
> What’s the real-world utility here? Is UML suitable for running isolated workloads? My educated guess is: probably not for most production scenarios.
Back in the day there were hosts offering UML VMs for rent. This is actually how Linode got its start!
The second generation was "skas" for Separate Kernel Address Space, some more background here: https://user-mode-linux.sourceforge.net/old/skas.html
The host kernel patch for skas was never merged, probably for good reason, but that and Xen/VM hardware support meant UML stopped making sense.
Stopped making sense for the mass hosting use case, but it is still wildly useful for setting up and tearing down during dev work, especially when hacking on the kernel itself, which you can trivially attach GDB to
Do you know why people stopped? It would seem to be a potentially useful middle ground between docker containers and KVM VMs
> Do you know why people stopped?
They didn't entirely. It is still maintained, developed even.
> It would seem to be a potentially useful middle ground between docker containers and KVM VMs
Back in the day I actually used it that way for running “VM”s and some firms even sold VPS accounts based on UML. Back then other virtualisation options were not nearly as mature as they soon became, or cost proper money (IIRC VMWare was good by that point but there were no free or reliable OSS options yet), and UML offered better isolation (a full environment including its own root) than simply chrooting a process tree (fuller containers were not a thing back then either, so all users fully existed on the host and you couldn't give out root access net.).
These days things like KVM and more advanced containerisation solve the problems most people want UML for and do so much more efficiently (UML performs badly, compared to other options, where there is a lot of kernel interaction, including any filesystem or network access).
UML is still very useful for its original intent though: testing and debugging certain kernel level items like filesystems (FUSE is competition here in many, but not all, cases), network drivers & filters, and so forth. When things go wrong you can trace into it in ways you can not (as easily) with VMS and containers.
Performance, mostly.
I worked for a hosting company that sold UML-based virtual machines, while we trialed Xen as the successor, before moving to use KVM instead.
But also KVM supported things like live-migration and virtio drivers which made custom interfaces and portability easier to deal with.
It's slow for many of the things people want to use it for.
In case you wonder how UML is currently used: https://netdevconf.info/0x14/pub/slides/8/UML%20Time%20Trave...
It's testing. Using timetravel mode you can skip sleeps and speedup your unit tests massively.
…unless your tests are syscall heavy, then it can be 10-100× slower :(. It also doesn't support SMP. I really wish for a better version of this, we could really use it, especially since the "time stops while CPU is in use" part also means your tests won't randomly fail if the host is heavily loaded and things take longer. Unfortunately, doing something about it is outside my area of expertise.
I was fascinated when I first learned of [FreeBSD Jails], I wonder if right before containerization became a thing the concept was developed further for its requirements (could it have been?) it would have offered a more efficient containerization platform.
FreeBSD Jails: https://docs.freebsd.org/en/books/handbook/jails/
Jails are entirely different conceptually from UML; they share the host kernel and are roughly analogous to containers/namespaces. UML is an entirely separate kernel, running as user mode process.
Why do they initialize a disk image with /dev/urandom instead of /dev/zero? Given it's not an encrypted disk container, I don't see any valid reason to do so, but perhaps I'm not seeing something?
Probably avoid zero write optimizations. This force actual allocation of disk space for the data, instead of pretending to do so.
So to make future performance more predictable?
It was great. I remember trying it about twenty years ago. The very first time I fired it up, I just typed "linux" at a prompt, and a kernel booted - right there in the terminal.
And then panicked, because it had no root. But hey, I've got a root filesystem right here!
So the second time I typed "linux root=/dev/hda1" (because we had parallel ATA drives back then).
It booted, mounted root, and of course that was the root filesystem the host was booted off.
Anyway it recovered after a power cycle and I didn't need to reinstall, and most importantly I learned not to do THAT again, which is often the important thing to learn.
I used that very method to rescue a machine after the contents of /boot were damaged. Booting from a live CD and building a UML kernel (on another drive, there wasn't enough RAM to do it there & I didn't want to mount any of the main partitions just-in-case) allowed me to boot from the main root and rebuild initrd & friends easily. I felt proper clever!
In supernested, which is a script to see how far you can nest KVM, we do in fact mount the root disk in a VM, but using a snapshot so it's safe(-ish). http://git.annexia.org/?p=supernested.git;a=tree
I've often thought that if only UML would build on Darwin, we'd have a MacOS container solution that didn't need virtualisation. That involves two big unsolved problems though: building UML on not-linux, and building UML on not-x86.
Had been using this quite some time ago, it is sad that it has only 1-CPU support, preventing some SMP bugs from emerging.
Wonder if it's hard to make it SMP, if too many places use something like #ifdef CONFIG_ARCH_IS_UM to tell whether it is single CPU, it might be hard.
SMP has been implemented recently and is queued for the next release.
Interesting
That’s giving very firecracker vibes
Wait until you realise QEmu (and dosbox) can do this too, while running windows or Dune II, as can old versions of virtualbox (not sure about new versions)
> Today, we’ll explore how you can start an unconventional VM by running a Linux kernel as a process within the Linux kernel itself. This approach doesn’t require installing virtualization software like QEMU, nor does it need root privileges, which opens up some intriguing possibilities.
That was addressed in the first few sentences.
My point is that VM software, especially older software and emulation software, doesn't require virtualization or root privileges. And yes, this is confusing because QEmu became virtualization software (just like VirtualBox did). Neither originally used hardware virtualization for anything. Dosbox still doesn't.
Hell I wish someone made something that could build dockerfiles and immediately start them as VMs in emulation using just the normal socket api to emulate network.
Yeah, but if you specifically target a Linux kernel to run as a regular user process, you don't have to take detours through CPU emulation code. It should (in theory) be more efficient to call the host's mmap() rather than mucking around an emulated MMU.