Until a few years back most of my systems used Linux From Scratch. It's amazing for learning your system and how all its libraries integrate. I created my own boot scripts, my own pacman package manager repository, etc. One of the main reasons I stopped using LFS was the amount of work required for updates.
Firefox is bad to compile yourself, because it's slow and often requires a library version you don't have, such as a new ICU version. Updating ICU also requires you to build packages that depend on it: harfbuzz, ffmpeg, qt5, poppler and the list goes on.
Chromium takes it to a whole new level. While Linux packages typically have an automake flow (configure, make, make install) Chromium requires ninja, not just ninja but a specific version depending on the Chromium version. Then you need a ton of libraries with specific versions (or custom patches) or use the built-in versions provided by chromium, which may or may not compile at all. Then there is clang, while most software is compiler agnostic, Chromium does not compile with gcc. Of course, it needs a specific clang release, sometimes even a pre-release, or again a load of patches to compile. It seems the chromium build process is mostly built by engineers desperate to reinvent the wheel. While everything can work fine with sane Linux defaults (GCC, automake, system libraries) they convinced themselves their software is so complex and unique it requires a special treatment.
Chromium looks like a build system for a complex integrated project where the engineers have one priority: make the application work as best it can. You will quickly find that when you focus on that then making the build system convenient, especially convenient for distribution packagers with their myriad of own rules and modifications that they want to make on top of your system, is quite a big blocker to the primary goal. You can see similar things with applications like OBS, where they bundle modified libraries to fix problems and then distributions unbundle them and release broken packages, causing no end of grief for users and the developers.
The Chromium build process is just a consequence of the monorepo mindset. For Google, depending on specific versions of tools and dependencies is not a problem because all their engineers will automatically get those versions. For the Linux distribution model it's a pain.
> Firefox is bad to compile yourself, because it's slow and often requires a library version you don't have, such as a new ICU version. Updating ICU also requires you to build packages that depend on it
> Chromium does not compile with gcc. Of course, it needs a specific clang release, sometimes even a pre-release, or again a load of patches to compile
And that is what is wrong with today software: it is an uncompilable mess.
It is "This program requires Windows 98 and Internet Explorer 4" brought to a new level.
There is not some arbitrary notion of wheel reinvention just for browsers. Calling browsers 'just browsers' is doing their feature set a huge disservice: they are bona fide operating systems or virtual machines, which have to load and run untrusted, unverified data and code within critical system libraries (many of which with kernel-mode access, such as GPU drivers, sound drivers, USB subsystems, and more), and do a huge amount of sandboxing to guard against escalations.
Within Google, a Chromium build uses remote build execution by default, upstream Clang by default (for the latest features and fixes, and Chromium developers regularly contribute to LLVM), and needs to cross-compile easily for at least a dozen different OS-architecture triples (Linux x86-64, Linux ARM, macOS, Windows x86-64, Windows on ARM, iOS, Android x86-64, Android ARM, Android TV, some BSDs, etc).
Everything you've listed—from ninja, to Clang, to libraries with custom patches, to Clang releases, including requiring pre-releases sometimes—is optimised for Google's scale, rather than the lone warrior developer. What you call 'sane Linux defaults' (especially the examples you have listed: GCC, automake, so-called 'system libraries') are highly antagonistic to all of the above.
They may make life easy for the so-called 'default case' of desktop Linux (arguable; Chromium was first released on Windows, and secondly since desktop Linux's most common C library, glibc has an unstable ABI, making back-portability impossible), but make it incredibly hard to get out a project of such immense complexity. All of Chromium's tooling and 'complexity' was developed in response to the 'sane' defaults being not so sane after all.
To get a GCC cross-compile for a given set of architecture triples, one needs a complete graph of GCC toolchains targeting every pair of host and target. On the other hand Clang is natively a cross-compiler; pass in `--sysroot` and the host compiler will redirect for the target instantly. Automake (and the autotools in general) is slow as molasses compared to ninja. It is based on computing and compilation from the 1970s, and their defaults probe for long-since-dead string comparison functions that are expected to work by default nowadays on most hosted systems. Not to mention it has an extraordinarily complicated configure step[1], and an equally appalling macro-based configure language.
Ninja is purely declarative, and is not even meant to be human-readable, as it is a back-end for a variety of build generator tools (CMake, XMake, build2, Bazel (Blaze internally at Google), Meson, probably more). Google have in fact rewritten Ninja to be a remote-build-capable tool from the outset[2], and they call it Siso. Siso has more than halved my Android build times even compared to Ninja because of its incredible use of RBE.
Frankly I think it should be evident why they have opted for 'rewriting the world' including vendoring and modifying system libraries, as well as including efforts like Skia, ANGLE, and more: directly relying on 'system libraries' is painful. I mentioned GPU drivers. Imagine rewriting the same code over and over to have precisely the same behaviour on D3D, Vulkan, OpenGL, Metal, and every single vendor GPU driver on every single system (and there are a lot more on mobile than on desktop: ARM Mali, Qualcomm Adreno, Samsung Xclipse, Imagination PowerVR, Huawei, MediaTek; this is not even close to exhaustive, just look at the Khronos vendor list).
When comments like yours are made I wonder if there is ever a consideration beyond 'it makes the developer's life hard'. As developers, our ultimate target audience are the users of our software. We should be willing to do anything come hell or high water, even making the dev-ex extraordinarily convoluted, if it means the user experience will be the smoothest and fastest possible. Sure, there is a balance, but I believe it should be strongly skewed towards the user story first.
>glibc has an unstable ABI, making back-portability impossible
Of all the libraries to call out for this, glibc is not the one. It's basically the only library on linux that actually has a strong commitment to providing a stable ABI, with its own symbol versioning system to allow it. It is a bit of a PITA to compile for an older version of that ABI, but it's very possible to be compatible with glibc across a wide range of versions and distributions (I don't know why there isn't a flag to do it: the most widely supported option is compiling on a system with the oldest glibc you want to support, and there's some header hacks which claim to be able to avoid that).
It's everything else necessary to interact with the system (especially anything desktop/multimedia) that's wildly unstable.
I singled it out because I've been badly burnt by it in the past, and I dislike how it works compared to Win32. Back-targeting old glibc from newer (a very common case because developers tend to have newer distros than their users), requires either changes in source code (not build code, which is where it ought to be) to explicitly pin versions, or the ridiculousness of firing up an entire Docker container just for old glibc. It has regularly caused issues for things like Steam on Linux.
If you build on random developer's machines rather than a unified build environment you've already failed at so many things that the glibc problem is certainly not your biggest concern.
Regardless it's still a PITA to maintain an ancient distro as a build environment and the alternatives are hacky. It might be the best of the linux libraries but it's still not good.
No need if you are targeting a current linux distro. You only need to do that if you are targeting the ancient distro, in which case you need it around to actually test your software on it as well.
All I can say is that Gentoo specifically forbids downgrading glibc. I think it's the only package where they did this. [1] The .ebuild is full of patches and workarounds.
Yeah, that's forwards-compatibility, which is different. Something built targeting a newer version of glibc is unlikely to work with an older one. (and gentoo will by default build against the latest you have installed, so yeah, downgrading has a good chance of breaking near everything).
> All I can say is that Gentoo specifically forbids downgrading glibc.
Nope. You're protected from accidentally doing it, not forbidden. Look:
# I_ALLOW_TO_BREAK_MY_SYSTEM=yes emerge -1v =sys-libs/glibc-2.43-r4
These are the packages that would be merged, in order:
<snip>
[ebuild UD ] sys-libs/glibc-2.43-r4:2.2::gentoo [2.44-r3:2.2::gentoo] <snip>
Total: 1 package (1 downgrade), Size of downloads: 0 KiB
>>> Verifying ebuild manifests
>>> Running pre-merge checks for sys-libs/glibc-2.43-r4
>>> Emerging (1 of 1) sys-libs/glibc-2.43-r4::gentoo
* glibc-2.43.tar.xz BLAKE2B SHA512 size ;-) ... [ ok ]
* glibc-2.43-patches-5.tar.xz BLAKE2B SHA512 size ;-) ... [ ok ]
* Checking whether python3_14 is suitable ...
* dev-lang/python:3.14 ... [ ok ]
* Using python3.14 to build (via PYTHON_COMPAT iteration)
>>> Unpacking source...
* Checking general environment sanity.
* Sanity check to keep you from breaking your system:
* Downgrading glibc is not supported and a sure way to destruction.
* Performing simple compile test for ABI=amd64 ...
<snip>
If you don't set that environment variable, the downgrade blows up at the "Sanity check to keep you from breaking your system:" step with a message that includes a call stack that pretty much tells you how to override the safety check:
* Call stack:
* ebuild.sh, line 143: Called src_unpack
* environment, line 3570: Called sanity_prechecks
* environment, line 3199: Called die
* The specific snippet of code:
* [[ ${I_ALLOW_TO_BREAK_MY_SYSTEM} = yes ]] || die "Aborting to save your system.";
Fair enough for poor Google, if only they did not strong-armed Chrome on everyone's computer by leveraging their default search engine/home page position, their ads and pop-under "Try Chrome" that were plastered everywhere 15 years ago or so.
It is: builds being painful enough that it's near impossible for smaller distro's to maintain a Chromium package, is community-hostile in itself.
A true community project is flexible w/ respect to dependencies, builds in a way that's doable by anyone who puts in some effort, and sends bugfixes in dependencies upstream so that -normally speaking- project can build against common (versions of) system libraries.
Doing the exact opposite screams vendor lock-in disguised as an open source effort. Of course no surprise this coming from Google (also notice similarities with the Android project).
It's relevant because Chrome market dominance still has a very strong influence on websites working (tested?) only with Chrome. Add to that Google's NIH syndrome and its "move fast" mantra at the cost of interoperability with the wider Linux community and then both things are relevant.
> which have to load and run untrusted, unverified data and code within critical system libraries (many of which with kernel-mode access, such as GPU drivers, sound drivers, USB subsystems, and more),
No, they don't have to. The fact that someone decided that a browser shall run remote code (JS) is sad.
gentoo's emerge is awesome. I keep a gentoo vm just to compile firefox, because it is impossible any other way. even though i don't use gentoo overall.
the comments trying to defens chrome (and old OBS) are completely corporate shills or missing the point. yeah obs at first did that because they had to, since it was such a small and niche project the devs had no pull upstream. after distros did all the unbundling job and helped them move patches upstream now every software benefit from those libraries fixes.
Google took chromium et al from open source and actively moved in the wrong direction, while having all the pull and influence like no open source project will ever have! its malice and/or incompetence. if you don't see that you're either shillings or lying to yourself. or have no knowledge of what open source is about.
> Linux defaults (GCC, automake, system libraries)
Last time I tried Gentoo desktop I ended up with 8 separate autotools versions, and gcc & glibc was not that far behind in count. And just like in https://xkcd.com/927/ every C developer tries to improve the situation by creating another build system.
> Last time I tried Gentoo desktop I ended up with 8 separate autotools versions, and gcc & glibc was not that far behind in count.
When was this?
AFAIK, you simply can't end up with more than one glibc version as that package hasn't had multiple Slots. [0] GCC is Slotted but I can't remember a time when there were more than three or four Slots defined for it. IME, it's rare for a package to declare an upper bound on GCC version, but it does happen. It's definitely true that there are many Slots for the various autotools tools.
[0] A "Slot" defines a particular subset of package versions that can be installed alongside other "Slots" of the same package. For example, every sys-kernel/gentoo-sources version has its own Slot, every minor version of dev-lang/python [1] has its own Slot, etc, etc, etc.
This is also why I stopped using Gentoo. I used it regularly from 2004-2005 (which I admit was a very long time ago now) on my graphite G4, because support was very good for PowerPC. But then one day emerge suggested I update glibc. I don’t know if I botched the update or if it really was suggested in error, but I was just starting my studies in CS so I did not know what that particular “library” did and said yes. The update must have included an ABI change. Not surprisingly, the machine did not come back up after a reboot.
Gentoo was fun while it lasted (fluxbox user here!). I decided that MacOS X wasn’t so bad after all after that.
I tried Gentoo some 20 years ago or so for fun at home. I guess my naive thinking was that software specifically compiled for my hardware would run noticeably faster and "better".
But I didn't last for very long. I remember starting the install for a bigger app before going to bed, only to find the thing still building when I woke up. And as for the dream of more tailored builds, I never really noticed anything much different - I probably was too uneducated about build configurations, or - more likely - the differences just didn't matter too much.
Eventually, I switched to Ubuntu and never looked back.
So, I'm curious - is anyone using Gentoo as their daily driver, and what is the main incentive for you?
I'm using it as my main OS. I have it installed on 4 machines: a RK3288 SBC, a RK3588 desktop, a x86-64 desktop and a x86-64 laptop. Main incentive for me is customization.
- It makes it very very easy to patch any package before they're installed - just put the patches in /etc/portage/patches/<pkg name>/, and that includes the kernel too.
- I can exclude package features that I don't like. For example, my Firefox doesn't try/offer to turn on the wifi.
- I can also exclude entire packages that I don't like, because the entire dependency chains can be eliminated by disabling USE flags. For example, none of my systems have systemd or dbus. Since last year I don't even have openrc anymore (the other Gentoo service manager). One of them doesn't even have udev. I tweaked it so much, I'm not even sure I can still call it Gentoo.
Daily Gentoo desktop user since 2007. Compilations/ricing is one thing, the variety and flexibility of the portage and USE flag system is another. For me it just makes sense to stay with this proven distro. I use Ubuntu variants on my work PC and they just have all this crap that I don't have to tolerate on my gentoo installs.
The docs were great for learning. And it made it so easy to recompile the kernel because it was right in the initial setup and you just had to walk through it again.
My understanding is that compiler flags yielding speed boosts where an actual thing back in the day, but not anymore really.
Nowadays the speed come from other things. For example a few years ago i realised my firefox on linux was doing video encoding/decoding on cpu without using the intel gpu’s hardware encoder/decoder. I did not have to rebuild firefox from scratch but rather fiddle with configurations and environment variables to make sure Firefox wouldn’t assume the lowest common denominator.
That saved battery on my laptop and yielded better performance.
Everything system-wide other than -march=native or equivalent has always been either placebo or sped thing up by introducing bugs. A specific software package sometimes didn’t build with appropriate flags and could benefit from tweaks, but having that do anything remotely meaningful is incredibly rare.
I used it on an off as my daily machine for around 15-20 years I think. It is fast in terms of execution-on-hardware speed, and genuinely puts you in control. For a certain kind of user this is more important than anything.
These days I'm on OSX mostly because of the frequency of business trips meant I needed something small and portable a few years ago, and now the hardware is so much better than what you can get for the PC world. Plus, I am hyper-conscious of my time and can't really afford to be fiddling with new system libraries, kernel configurations and so forth.
Frankly Gentoo was never polished, at some point the system state would break and you'd have to reinstall, and after enough iterations I'd started to err to Ubuntu at some point. But it was by far the most fun of any Linux I used, save perhaps Slackware which was my first.
I think in a few years' time the next machine I build will be a non-Apple ARM custom portable, hopefully running Gentoo, with a lot of custom machining and hardware features.
> It is fast in terms of execution-on-hardware speed, and genuinely puts you in control.
After installing Gentoo I felt I should make some kind of change, and I wanted to replace the GNOME foot in the GNOME menu with an icon I found of the Gentoo "g" in a wizard hat.
I found documentation suggesting that the menu's icon was controlled by some setting somewhere, and I changed that setting to point to the wizard-hat icon.
The menu was unaffected. If I viewed the menu bar's properties, the menu was associated with the wizard-hat icon in the properties display, but it continued to show the foot icon in its name.
I had nothing else to go on and gave up. I'm still mad about that.
> Frankly Gentoo was never polished, at some point the system state would break and you'd have to reinstall
Well, that did happen to me once. But it happened after I uninstalled glibc. I can't really blame Gentoo for that. I always saw it as positive that Gentoo allowed me to do something that ridiculous.
once? You obviously weren't using it long! All sorts of portage tree screwups would affect you periodically that weren't your fault. Details escape me, it just happened, my recollection is circa annually to 24 months. Usually accompanied by some tacit admission of guilt in an enews item. Longer installs were a rarity without a combination of paranoia and luck or a working snapshot setup.
One thing Gentoo would also let you do, unlike any other distro I'm aware of, was install managed cross-compilation toolchains with deep control preserved. An awesome feature if you are building images for variant embedded devices at any frequency.
> at some point the system state would break and you'd have to reinstall
My advice is to start using btrfs snapshots. Make a backup snapshot, run the updates, and if anything breaks, just "btrfs set-default" to the backup snapshot and everything is working again.
Looked in to ZFS snapshots and pretty sure I had it working for awhile, the challenge was the boot sequence required for a custom kernel and rootfs was pretty non-transparent when you wanted to combine the snapshots with encryption and GPT disk label support. Or that's my recollection, anyway. I think a lot of gentoo users these days probably just use genkernel and run scripts for initial root fs generation, ceding control, but that's sort of against the mentality that brought me to gentoo IMHO. Also, stick with openrc...
> Frankly Gentoo was never polished, at some point the system state would break and you'd have to reinstall
Not at all. I have even recovered from bricking the package manager (my fault, these days there are protections in place against that). Worst case you manually extract working binaries and then clean up the mess. And I have never run into issues that needed such drastic measures without doing stupid things.
It is however not a distro for those unwilling to understand how it works.
Yes, this is classic gentoo: recover using some massive hassle technique then tell everyone about it, incrementally growing your knowledge of system failure modes. But once you'd done that a few times you realised keeping your system cleanly structured and doing a clean reinstall was probably a far better strategy for personal time and sanity preservation if you actually wanted to get anything done...
This is what we get when we go all in on a monolithic bloated platform, and then let its direction be primarily determined by one company. We'd be better off with a simpler, albeit less featureful, platform, or at least one that's more modular.
It builds Chromium in a Docker container using Debian and spits out an AppImage. The AppImage seems to run fine on Gentoo, so I've been using that.
Note that for some time, I've had to patch their build process (it fails to pull a GIT repo in my environment -- so I have to shell into the container and run the pull manually). I've considered making this the basis for a Gentoo package... but I'll likely stick with my current process (or just use ungoogled-chromium's binaries).
If you've used source based distros, you've noticed certain problems over the years. Upgrading compilers or runtimes can bring up problems that need to be fixed (maybe by recompilation or dev work). Upgrading libraries with circular dependencies is more painful.
After a while, you just want things to be more modular, have well behaved dependencies and release practices that just work to isolate any failures. Overall, Gentoo has done very well improving in this regard.
Chrome and Android are large enough projects with enough resources that they can brute force through these issues which is harder for a small distro to do.
I've always used source-based packages, but chromium was the one I simply dropped in favour of Firefox.
On a 16c32t / 64G machine, compiling Firefox takes ~1h with LTO and PGO, while Chromium already took 5+h in 2019 without PGO. Then they removed jumbo-build because there were some issues and kept adding code and toolchain deps (jumbo-build is a build step that merges a few hundred source files before compiling to do the header processing once, shrinking the build from 5h+ to ~1-2h). Last time I checked the build had 70k+ build steps, with the final link step easily taking 10m+ with LLD. Answering some other comments: ccache doesn't help, because every version will have some minor header changes somewhere, so basically the entire cache gets invalidated even within the same major (e.g. 151.44.22.11 to 151.44.23.28).
Compiling Chromium took absolutely forever 5+ years ago. I think a whole system took like 20% of the time that Chromium would take to compile. I can't even imagine with rust extensions added on top now. Gentoo was so awesome though but I had to switch to debian because I would just play with my configuration all the time and break things. One day I hope to return to using it though or setup another machine to use it.
NTA but why so hostile? What do you even mean by "real work" and "real hardware"? And what has this to do with Linux compile times? Linux compiles way faster than Chromium in general.
Chromium is a pyramid of components and the higher you go up, the longer the compilation of single file takes, because they depend on so many things. This makes caching difficult. A small change at the bottom can invalidate many files, especially the heavy ones at the top. The build system ninja already does caching, but it only offers limited benefits. Then there is so much activity that if you plan to do a weekly build, you can do it from scratch.
ccache for portage is broken since at least 2010. Because each package source gets extracted and compiled in it's own versioned directory, with each new version the cache is invalidated by the different path.
ccache is not very effective accross different versions of software as it only takes one small change in a header that's included everywhere to make the cache completely useless.
Gentoo user since 2002 sounding off. I can't remember this being a problem in like five or ten years. Occasionally a library update will trigger rebuilds for a few packages, and revdep-rebuild hasn't actually had anything to do on my systems in years.
I don't know if devs got much better at not making changes that break binary compatibility, [0] Portage (and friends) got better at figuring out the minimal set of packages to rebuild on library update, or what, but, yeah. That hasn't been a problem for me for quite a long time.
[0] Perhaps the chucklefucks who would habitually be so careless and inconsiderate all moved to working in Javascript?
Am I the only one who noticed just how entitled this Žilka person is? If they are so good at maintenance maybe they should take it over and learn the hard way that things like this are not at all remotely simple to pull off and that maintainers are (usually) unpaid volunteers who have their own lives (and in some cases, they maintain many other packages).
Edit: I should note that it definitely looks like entitlement based on how the article presents it, and doesn't at all seem like this person is taking into consideration all the maintenance policies/rules/whatnot.
The main problems seem to be that it takes a long time to build all combinations of USE flags and that users complain that volunteers are not working fast enough, causing annoyance of the maintainers. Can you explain how AI would help here?
You still need the hardware resources to do builds. And thanks to Slop Generator providers gobbling up alllllllll the RAM and storage they can get their icky hands on, it simply practically infeasible for everyone not affiliated with a tech BigCo.
A dumb-as-rocks tinderbox can do the task of "build the thing in all valid configurations". Good luck getting the hardware needed to do all that testing in a timely manner on an -IIRC- 15K USD budget, or finding people to volunteer it in this economy. And good fucking luck getting LLMs to do the diagnostics, writeups, and -most importantly- shepherding required to get the bugs your testing reveals actually fixed in upstream.
Upstream not giving much of a fuck about whether anything other than Google® Chrome™ can be produced from Chromium is a big part of the reason it's being dropped from Gentoo. And yes, I 110% agree that Google is totally entitled to give zero fucks about non-Google users of Chromium. Gentoo is equally entitled to stop packaging Chromium.
Maintaining Gentoo ebuilds for complex packages such as Chromium is as peak Linux as one can get. You're talking about developers who _often_ discover and then submit upstream patches for the most obscure and difficult bugs that upstream projects haven't heard of. Often because Gentoo users are the only ones trying different compilation options, or they're the first to try building with more modern dependencies than the upstream project has yet attempted.
For just one of many examples, see [1] where the Gentoo developer had to submit a patch upstream for a Chromium build toolchain bug that to 99% of experienced Linux users is bordering on unintelligible due to how much specific knowledge is needed to understand the bug.
I tried some months ago, and it'll make a blank one fine, but if you need a complex configure or install phase, I felt I was better off doing it by hand.
Until a few years back most of my systems used Linux From Scratch. It's amazing for learning your system and how all its libraries integrate. I created my own boot scripts, my own pacman package manager repository, etc. One of the main reasons I stopped using LFS was the amount of work required for updates.
Firefox is bad to compile yourself, because it's slow and often requires a library version you don't have, such as a new ICU version. Updating ICU also requires you to build packages that depend on it: harfbuzz, ffmpeg, qt5, poppler and the list goes on.
Chromium takes it to a whole new level. While Linux packages typically have an automake flow (configure, make, make install) Chromium requires ninja, not just ninja but a specific version depending on the Chromium version. Then you need a ton of libraries with specific versions (or custom patches) or use the built-in versions provided by chromium, which may or may not compile at all. Then there is clang, while most software is compiler agnostic, Chromium does not compile with gcc. Of course, it needs a specific clang release, sometimes even a pre-release, or again a load of patches to compile. It seems the chromium build process is mostly built by engineers desperate to reinvent the wheel. While everything can work fine with sane Linux defaults (GCC, automake, system libraries) they convinced themselves their software is so complex and unique it requires a special treatment.
Chromium looks like a build system for a complex integrated project where the engineers have one priority: make the application work as best it can. You will quickly find that when you focus on that then making the build system convenient, especially convenient for distribution packagers with their myriad of own rules and modifications that they want to make on top of your system, is quite a big blocker to the primary goal. You can see similar things with applications like OBS, where they bundle modified libraries to fix problems and then distributions unbundle them and release broken packages, causing no end of grief for users and the developers.
The Chromium build process is just a consequence of the monorepo mindset. For Google, depending on specific versions of tools and dependencies is not a problem because all their engineers will automatically get those versions. For the Linux distribution model it's a pain.
> Firefox is bad to compile yourself, because it's slow and often requires a library version you don't have, such as a new ICU version. Updating ICU also requires you to build packages that depend on it
> Chromium does not compile with gcc. Of course, it needs a specific clang release, sometimes even a pre-release, or again a load of patches to compile
And that is what is wrong with today software: it is an uncompilable mess. It is "This program requires Windows 98 and Internet Explorer 4" brought to a new level.
Software has become balkanized.
There is not some arbitrary notion of wheel reinvention just for browsers. Calling browsers 'just browsers' is doing their feature set a huge disservice: they are bona fide operating systems or virtual machines, which have to load and run untrusted, unverified data and code within critical system libraries (many of which with kernel-mode access, such as GPU drivers, sound drivers, USB subsystems, and more), and do a huge amount of sandboxing to guard against escalations.
Within Google, a Chromium build uses remote build execution by default, upstream Clang by default (for the latest features and fixes, and Chromium developers regularly contribute to LLVM), and needs to cross-compile easily for at least a dozen different OS-architecture triples (Linux x86-64, Linux ARM, macOS, Windows x86-64, Windows on ARM, iOS, Android x86-64, Android ARM, Android TV, some BSDs, etc).
Everything you've listed—from ninja, to Clang, to libraries with custom patches, to Clang releases, including requiring pre-releases sometimes—is optimised for Google's scale, rather than the lone warrior developer. What you call 'sane Linux defaults' (especially the examples you have listed: GCC, automake, so-called 'system libraries') are highly antagonistic to all of the above.
They may make life easy for the so-called 'default case' of desktop Linux (arguable; Chromium was first released on Windows, and secondly since desktop Linux's most common C library, glibc has an unstable ABI, making back-portability impossible), but make it incredibly hard to get out a project of such immense complexity. All of Chromium's tooling and 'complexity' was developed in response to the 'sane' defaults being not so sane after all.
To get a GCC cross-compile for a given set of architecture triples, one needs a complete graph of GCC toolchains targeting every pair of host and target. On the other hand Clang is natively a cross-compiler; pass in `--sysroot` and the host compiler will redirect for the target instantly. Automake (and the autotools in general) is slow as molasses compared to ninja. It is based on computing and compilation from the 1970s, and their defaults probe for long-since-dead string comparison functions that are expected to work by default nowadays on most hosted systems. Not to mention it has an extraordinarily complicated configure step[1], and an equally appalling macro-based configure language.
Ninja is purely declarative, and is not even meant to be human-readable, as it is a back-end for a variety of build generator tools (CMake, XMake, build2, Bazel (Blaze internally at Google), Meson, probably more). Google have in fact rewritten Ninja to be a remote-build-capable tool from the outset[2], and they call it Siso. Siso has more than halved my Android build times even compared to Ninja because of its incredible use of RBE.
Frankly I think it should be evident why they have opted for 'rewriting the world' including vendoring and modifying system libraries, as well as including efforts like Skia, ANGLE, and more: directly relying on 'system libraries' is painful. I mentioned GPU drivers. Imagine rewriting the same code over and over to have precisely the same behaviour on D3D, Vulkan, OpenGL, Metal, and every single vendor GPU driver on every single system (and there are a lot more on mobile than on desktop: ARM Mali, Qualcomm Adreno, Samsung Xclipse, Imagination PowerVR, Huawei, MediaTek; this is not even close to exhaustive, just look at the Khronos vendor list).
When comments like yours are made I wonder if there is ever a consideration beyond 'it makes the developer's life hard'. As developers, our ultimate target audience are the users of our software. We should be willing to do anything come hell or high water, even making the dev-ex extraordinarily convoluted, if it means the user experience will be the smoothest and fastest possible. Sure, there is a balance, but I believe it should be strongly skewed towards the user story first.
[1]: https://upload.wikimedia.org/wikipedia/commons/8/84/Autoconf...
[2]: https://chromium.googlesource.com/build/+/refs/heads/main/si...
>glibc has an unstable ABI, making back-portability impossible
Of all the libraries to call out for this, glibc is not the one. It's basically the only library on linux that actually has a strong commitment to providing a stable ABI, with its own symbol versioning system to allow it. It is a bit of a PITA to compile for an older version of that ABI, but it's very possible to be compatible with glibc across a wide range of versions and distributions (I don't know why there isn't a flag to do it: the most widely supported option is compiling on a system with the oldest glibc you want to support, and there's some header hacks which claim to be able to avoid that).
It's everything else necessary to interact with the system (especially anything desktop/multimedia) that's wildly unstable.
Fair response.
I singled it out because I've been badly burnt by it in the past, and I dislike how it works compared to Win32. Back-targeting old glibc from newer (a very common case because developers tend to have newer distros than their users), requires either changes in source code (not build code, which is where it ought to be) to explicitly pin versions, or the ridiculousness of firing up an entire Docker container just for old glibc. It has regularly caused issues for things like Steam on Linux.
If you build on random developer's machines rather than a unified build environment you've already failed at so many things that the glibc problem is certainly not your biggest concern.
Regardless it's still a PITA to maintain an ancient distro as a build environment and the alternatives are hacky. It might be the best of the linux libraries but it's still not good.
No need if you are targeting a current linux distro. You only need to do that if you are targeting the ancient distro, in which case you need it around to actually test your software on it as well.
All I can say is that Gentoo specifically forbids downgrading glibc. I think it's the only package where they did this. [1] The .ebuild is full of patches and workarounds.
[1] See line 873. https://github.com/gentoo/gentoo/blob/master/sys-libs/glibc/...
Yeah, that's forwards-compatibility, which is different. Something built targeting a newer version of glibc is unlikely to work with an older one. (and gentoo will by default build against the latest you have installed, so yeah, downgrading has a good chance of breaking near everything).
> All I can say is that Gentoo specifically forbids downgrading glibc.
Nope. You're protected from accidentally doing it, not forbidden. Look:
If you don't set that environment variable, the downgrade blows up at the "Sanity check to keep you from breaking your system:" step with a message that includes a call stack that pretty much tells you how to override the safety check:Fair enough for poor Google, if only they did not strong-armed Chrome on everyone's computer by leveraging their default search engine/home page position, their ads and pop-under "Try Chrome" that were plastered everywhere 15 years ago or so.
This is not really relevant within the discussion of Chromium's build system.
It is: builds being painful enough that it's near impossible for smaller distro's to maintain a Chromium package, is community-hostile in itself.
A true community project is flexible w/ respect to dependencies, builds in a way that's doable by anyone who puts in some effort, and sends bugfixes in dependencies upstream so that -normally speaking- project can build against common (versions of) system libraries.
Doing the exact opposite screams vendor lock-in disguised as an open source effort. Of course no surprise this coming from Google (also notice similarities with the Android project).
It's relevant because Chrome market dominance still has a very strong influence on websites working (tested?) only with Chrome. Add to that Google's NIH syndrome and its "move fast" mantra at the cost of interoperability with the wider Linux community and then both things are relevant.
That may be true, but I don't see how it is the same topic as the build system.
Big company does big company things like cross-category marketing for a new product.
> which have to load and run untrusted, unverified data and code within critical system libraries (many of which with kernel-mode access, such as GPU drivers, sound drivers, USB subsystems, and more),
No, they don't have to. The fact that someone decided that a browser shall run remote code (JS) is sad.
gentoo's emerge is awesome. I keep a gentoo vm just to compile firefox, because it is impossible any other way. even though i don't use gentoo overall.
the comments trying to defens chrome (and old OBS) are completely corporate shills or missing the point. yeah obs at first did that because they had to, since it was such a small and niche project the devs had no pull upstream. after distros did all the unbundling job and helped them move patches upstream now every software benefit from those libraries fixes.
Google took chromium et al from open source and actively moved in the wrong direction, while having all the pull and influence like no open source project will ever have! its malice and/or incompetence. if you don't see that you're either shillings or lying to yourself. or have no knowledge of what open source is about.
> Linux defaults (GCC, automake, system libraries)
Last time I tried Gentoo desktop I ended up with 8 separate autotools versions, and gcc & glibc was not that far behind in count. And just like in https://xkcd.com/927/ every C developer tries to improve the situation by creating another build system.
> Last time I tried Gentoo desktop I ended up with 8 separate autotools versions, and gcc & glibc was not that far behind in count.
When was this?
AFAIK, you simply can't end up with more than one glibc version as that package hasn't had multiple Slots. [0] GCC is Slotted but I can't remember a time when there were more than three or four Slots defined for it. IME, it's rare for a package to declare an upper bound on GCC version, but it does happen. It's definitely true that there are many Slots for the various autotools tools.
[0] A "Slot" defines a particular subset of package versions that can be installed alongside other "Slots" of the same package. For example, every sys-kernel/gentoo-sources version has its own Slot, every minor version of dev-lang/python [1] has its own Slot, etc, etc, etc.
[1] ...that is, 3.9, 3.10, 3.11...
This is also why I stopped using Gentoo. I used it regularly from 2004-2005 (which I admit was a very long time ago now) on my graphite G4, because support was very good for PowerPC. But then one day emerge suggested I update glibc. I don’t know if I botched the update or if it really was suggested in error, but I was just starting my studies in CS so I did not know what that particular “library” did and said yes. The update must have included an ABI change. Not surprisingly, the machine did not come back up after a reboot.
Gentoo was fun while it lasted (fluxbox user here!). I decided that MacOS X wasn’t so bad after all after that.
I tried Gentoo some 20 years ago or so for fun at home. I guess my naive thinking was that software specifically compiled for my hardware would run noticeably faster and "better".
But I didn't last for very long. I remember starting the install for a bigger app before going to bed, only to find the thing still building when I woke up. And as for the dream of more tailored builds, I never really noticed anything much different - I probably was too uneducated about build configurations, or - more likely - the differences just didn't matter too much.
Eventually, I switched to Ubuntu and never looked back.
So, I'm curious - is anyone using Gentoo as their daily driver, and what is the main incentive for you?
I'm using it as my main OS. I have it installed on 4 machines: a RK3288 SBC, a RK3588 desktop, a x86-64 desktop and a x86-64 laptop. Main incentive for me is customization.
- It makes it very very easy to patch any package before they're installed - just put the patches in /etc/portage/patches/<pkg name>/, and that includes the kernel too.
- I can exclude package features that I don't like. For example, my Firefox doesn't try/offer to turn on the wifi.
- I can also exclude entire packages that I don't like, because the entire dependency chains can be eliminated by disabling USE flags. For example, none of my systems have systemd or dbus. Since last year I don't even have openrc anymore (the other Gentoo service manager). One of them doesn't even have udev. I tweaked it so much, I'm not even sure I can still call it Gentoo.
Daily Gentoo desktop user since 2007. Compilations/ricing is one thing, the variety and flexibility of the portage and USE flag system is another. For me it just makes sense to stay with this proven distro. I use Ubuntu variants on my work PC and they just have all this crap that I don't have to tolerate on my gentoo installs.
P.S.: I've got to admit, though, that I was always impressed by the really great quality of Gentoo's documentation!
The docs were great for learning. And it made it so easy to recompile the kernel because it was right in the initial setup and you just had to walk through it again.
My understanding is that compiler flags yielding speed boosts where an actual thing back in the day, but not anymore really.
Nowadays the speed come from other things. For example a few years ago i realised my firefox on linux was doing video encoding/decoding on cpu without using the intel gpu’s hardware encoder/decoder. I did not have to rebuild firefox from scratch but rather fiddle with configurations and environment variables to make sure Firefox wouldn’t assume the lowest common denominator.
That saved battery on my laptop and yielded better performance.
Everything system-wide other than -march=native or equivalent has always been either placebo or sped thing up by introducing bugs. A specific software package sometimes didn’t build with appropriate flags and could benefit from tweaks, but having that do anything remotely meaningful is incredibly rare.
I used it on an off as my daily machine for around 15-20 years I think. It is fast in terms of execution-on-hardware speed, and genuinely puts you in control. For a certain kind of user this is more important than anything.
These days I'm on OSX mostly because of the frequency of business trips meant I needed something small and portable a few years ago, and now the hardware is so much better than what you can get for the PC world. Plus, I am hyper-conscious of my time and can't really afford to be fiddling with new system libraries, kernel configurations and so forth.
Frankly Gentoo was never polished, at some point the system state would break and you'd have to reinstall, and after enough iterations I'd started to err to Ubuntu at some point. But it was by far the most fun of any Linux I used, save perhaps Slackware which was my first.
I think in a few years' time the next machine I build will be a non-Apple ARM custom portable, hopefully running Gentoo, with a lot of custom machining and hardware features.
> It is fast in terms of execution-on-hardware speed, and genuinely puts you in control.
After installing Gentoo I felt I should make some kind of change, and I wanted to replace the GNOME foot in the GNOME menu with an icon I found of the Gentoo "g" in a wizard hat.
I found documentation suggesting that the menu's icon was controlled by some setting somewhere, and I changed that setting to point to the wizard-hat icon.
The menu was unaffected. If I viewed the menu bar's properties, the menu was associated with the wizard-hat icon in the properties display, but it continued to show the foot icon in its name.
I had nothing else to go on and gave up. I'm still mad about that.
> Frankly Gentoo was never polished, at some point the system state would break and you'd have to reinstall
Well, that did happen to me once. But it happened after I uninstalled glibc. I can't really blame Gentoo for that. I always saw it as positive that Gentoo allowed me to do something that ridiculous.
once? You obviously weren't using it long! All sorts of portage tree screwups would affect you periodically that weren't your fault. Details escape me, it just happened, my recollection is circa annually to 24 months. Usually accompanied by some tacit admission of guilt in an enews item. Longer installs were a rarity without a combination of paranoia and luck or a working snapshot setup.
One thing Gentoo would also let you do, unlike any other distro I'm aware of, was install managed cross-compilation toolchains with deep control preserved. An awesome feature if you are building images for variant embedded devices at any frequency.
> at some point the system state would break and you'd have to reinstall
My advice is to start using btrfs snapshots. Make a backup snapshot, run the updates, and if anything breaks, just "btrfs set-default" to the backup snapshot and everything is working again.
Looked in to ZFS snapshots and pretty sure I had it working for awhile, the challenge was the boot sequence required for a custom kernel and rootfs was pretty non-transparent when you wanted to combine the snapshots with encryption and GPT disk label support. Or that's my recollection, anyway. I think a lot of gentoo users these days probably just use genkernel and run scripts for initial root fs generation, ceding control, but that's sort of against the mentality that brought me to gentoo IMHO. Also, stick with openrc...
> Frankly Gentoo was never polished, at some point the system state would break and you'd have to reinstall
Not at all. I have even recovered from bricking the package manager (my fault, these days there are protections in place against that). Worst case you manually extract working binaries and then clean up the mess. And I have never run into issues that needed such drastic measures without doing stupid things.
It is however not a distro for those unwilling to understand how it works.
Yes, this is classic gentoo: recover using some massive hassle technique then tell everyone about it, incrementally growing your knowledge of system failure modes. But once you'd done that a few times you realised keeping your system cleanly structured and doing a clean reinstall was probably a far better strategy for personal time and sanity preservation if you actually wanted to get anything done...
I remember triggering my third recompile of kde-base because I wanted a gui calculator. I moved away not long after that.
xcalc wasn't good enough?
> Measures need to be taken by somebody qualified that this doesn't keep happening in packages as important as these.
To the single guy maintaining it, and no offer of help. Crazy entitlement.
Use of the passive voice is a giveaway. People seem to write as if these are large corporate projects with managers and all, not volunteer driven.
Very related and interesting discussion that got posted in the comments about the new Crubit thingie: https://groups.google.com/a/chromium.org/g/embedder-dev/c/W_...
The irony being that ChromeOS is actually Gentoo under the hood
Companies hardly ever contribute anything to the projects they depend on.
This is what we get when we go all in on a monolithic bloated platform, and then let its direction be primarily determined by one company. We'd be better off with a simpler, albeit less featureful, platform, or at least one that's more modular.
Currently exploring a return to Gentoo here, for "reasons". My solution for a while (even before Gentoo) has been <https://github.com/ungoogled-software/ungoogled-chromium-por...>
It builds Chromium in a Docker container using Debian and spits out an AppImage. The AppImage seems to run fine on Gentoo, so I've been using that.
Note that for some time, I've had to patch their build process (it fails to pull a GIT repo in my environment -- so I have to shell into the container and run the pull manually). I've considered making this the basis for a Gentoo package... but I'll likely stick with my current process (or just use ungoogled-chromium's binaries).
If you've used source based distros, you've noticed certain problems over the years. Upgrading compilers or runtimes can bring up problems that need to be fixed (maybe by recompilation or dev work). Upgrading libraries with circular dependencies is more painful.
After a while, you just want things to be more modular, have well behaved dependencies and release practices that just work to isolate any failures. Overall, Gentoo has done very well improving in this regard.
Chrome and Android are large enough projects with enough resources that they can brute force through these issues which is harder for a small distro to do.
I always wondered how long it would last tbh.
I've always used source-based packages, but chromium was the one I simply dropped in favour of Firefox.
On a 16c32t / 64G machine, compiling Firefox takes ~1h with LTO and PGO, while Chromium already took 5+h in 2019 without PGO. Then they removed jumbo-build because there were some issues and kept adding code and toolchain deps (jumbo-build is a build step that merges a few hundred source files before compiling to do the header processing once, shrinking the build from 5h+ to ~1-2h). Last time I checked the build had 70k+ build steps, with the final link step easily taking 10m+ with LLD. Answering some other comments: ccache doesn't help, because every version will have some minor header changes somewhere, so basically the entire cache gets invalidated even within the same major (e.g. 151.44.22.11 to 151.44.23.28).
49982225 Last rites for Gentoo''s Chromium package https://lwn.net/SubscriberLink/1097760/2be4d9e3eeb59039/
49998685 Last rites for Gentoo''s Chromium package https://lwn.net/SubscriberLink/1097760/2be4d9e3eeb59039/
Compiling Chromium took absolutely forever 5+ years ago. I think a whole system took like 20% of the time that Chromium would take to compile. I can't even imagine with rust extensions added on top now. Gentoo was so awesome though but I had to switch to debian because I would just play with my configuration all the time and break things. One day I hope to return to using it though or setup another machine to use it.
So basically they'd be new CVE's appearing by the time someone built the previous release at this pace? :P
The kernel itself used to take just as long.
I could build the kernel on a pentium 3 in about 20 minutes.
When you build your own kernel you don't compile all the random crap like token ring support and 30 filesystems nobody uses.
Well unfortunately those of us doing real work run things on real hardware.
I'm sorry me being poor when I was a student bothers you so much. I will try to be born rich next time -_-'
[flagged]
NTA but why so hostile? What do you even mean by "real work" and "real hardware"? And what has this to do with Linux compile times? Linux compiles way faster than Chromium in general.
[flagged]
I was ok with Chromium alone taking ~6h. I moved away when I started experimenting with Rust.
No ccache?
Chromium is a pyramid of components and the higher you go up, the longer the compilation of single file takes, because they depend on so many things. This makes caching difficult. A small change at the bottom can invalidate many files, especially the heavy ones at the top. The build system ninja already does caching, but it only offers limited benefits. Then there is so much activity that if you plan to do a weekly build, you can do it from scratch.
ccache for portage is broken since at least 2010. Because each package source gets extracted and compiled in it's own versioned directory, with each new version the cache is invalidated by the different path.
https://bugs.gentoo.org/346063
ccache is not very effective accross different versions of software as it only takes one small change in a header that's included everywhere to make the cache completely useless.
As an ex-Gentoo user, the pain in a was always bundled Vs shared libraries.
Shared libraries breaking ABI could mean recompiling half your operating system
On the other hand, bundled libraries meant insanely long build times and a sense of impurity/uncleanliness
Unfortunately, it looks like projects like ABI tracker are long dead:
https://github.com/lvc/abi-tracker
Current Gentoo user here. This hasn't been a real issue for me in years.
Gentoo user since 2002 sounding off. I can't remember this being a problem in like five or ten years. Occasionally a library update will trigger rebuilds for a few packages, and revdep-rebuild hasn't actually had anything to do on my systems in years.
I don't know if devs got much better at not making changes that break binary compatibility, [0] Portage (and friends) got better at figuring out the minimal set of packages to rebuild on library update, or what, but, yeah. That hasn't been a problem for me for quite a long time.
[0] Perhaps the chucklefucks who would habitually be so careless and inconsiderate all moved to working in Javascript?
Am I the only one who noticed just how entitled this Žilka person is? If they are so good at maintenance maybe they should take it over and learn the hard way that things like this are not at all remotely simple to pull off and that maintainers are (usually) unpaid volunteers who have their own lives (and in some cases, they maintain many other packages).
Edit: I should note that it definitely looks like entitlement based on how the article presents it, and doesn't at all seem like this person is taking into consideration all the maintenance policies/rules/whatnot.
why don't they leverage ai for maintaining packages?
https://wiki.gentoo.org/wiki/Project:Council/AI_policy
> Their operations are causing concerns about the huge use of energy and water.
Pretty rich from a compile-from-source distribution
The main problems seem to be that it takes a long time to build all combinations of USE flags and that users complain that volunteers are not working fast enough, causing annoyance of the maintainers. Can you explain how AI would help here?
ai can work test the options and fix the builds
You still need the hardware resources to do builds. And thanks to Slop Generator providers gobbling up alllllllll the RAM and storage they can get their icky hands on, it simply practically infeasible for everyone not affiliated with a tech BigCo.
Yep.
A dumb-as-rocks tinderbox can do the task of "build the thing in all valid configurations". Good luck getting the hardware needed to do all that testing in a timely manner on an -IIRC- 15K USD budget, or finding people to volunteer it in this economy. And good fucking luck getting LLMs to do the diagnostics, writeups, and -most importantly- shepherding required to get the bugs your testing reveals actually fixed in upstream.
Upstream not giving much of a fuck about whether anything other than Google® Chrome™ can be produced from Chromium is a big part of the reason it's being dropped from Gentoo. And yes, I 110% agree that Google is totally entitled to give zero fucks about non-Google users of Chromium. Gentoo is equally entitled to stop packaging Chromium.
Maintaining Gentoo ebuilds for complex packages such as Chromium is as peak Linux as one can get. You're talking about developers who _often_ discover and then submit upstream patches for the most obscure and difficult bugs that upstream projects haven't heard of. Often because Gentoo users are the only ones trying different compilation options, or they're the first to try building with more modern dependencies than the upstream project has yet attempted.
For just one of many examples, see [1] where the Gentoo developer had to submit a patch upstream for a Chromium build toolchain bug that to 99% of experienced Linux users is bordering on unintelligible due to how much specific knowledge is needed to understand the bug.
[1] https://bugs.gentoo.org/956054#c2
Most ai wouldn't know how to write a proper ebuild to save their lives.
That's funny cause I use ai to maintain dozens of ebuilds. They are usually one shots.
I tried some months ago, and it'll make a blank one fine, but if you need a complex configure or install phase, I felt I was better off doing it by hand.
Cool, you can adopt chromium then!
Problem is solved!
I'd be happy to if I had enough compute to rebuild it 50 times in under an hour.