>Debian Policy still cites the FHS, even though the FHS has gone unmaintained for more than a decade.
What ongoing maintenance would a file system standard require? A successful standard of that type would have to remain static unless there was a serious issue to address. Regular changes are what the standard was intended to combat in the first place.
>The specification was not so much finished as abandoned after FHS 3.0 was released...
OK.
>...though there is a slow-moving effort to revive and revise the standard as FHS 4.0, it has not yet produced any results.
So it is not abandoned then. A slow moving process is exactly what you would want for the maintenance of a file system standard.
>Meanwhile, in the absence of a current standard, systemd has spun off its file-hierarchy documentation to the Linux Userspace API (UAPI) Group as a specification. LWN covered that development in August, related to Fedora's search for an FHS successor.
Ah. Systemd/Fedora want a standard that they can directly control without interference from others.
Author here: It was abandoned. I linked to one of the former maintainers who said as much. The current effort is by a few people who asked the LF to take out over, and have (so far) done little after an initial flurry of activity. That, too, is covered in the other article I wrote about the FHS recently.
Prior to the group who started an update effort, it had not been touched in about a decade. That’s not slow-moving: that’s abandoned.
The FHS ultimately belongs to the users collectively, not those maintaining it. I am old enough to remember the horror that existed before the influence of the FHS. It exists in the fact that it is to some extent respected, not because there is a file somewhere that says it is the FHS standard. If you want changes, then sure, do the politics required to develop support for those changes. You can't just declare a new standard and then do whatever you want.
Developers have this thing where they will think of a standard as a specification. Instead it is a statement of political will. Saying that a standard is "abandoned" due to lack of "maintenance" seems like an example of thinking of a standard as the instantation of a specification; an actual program.
I agree--given your contraints of law and 10 years. But what about a law that hasn't been updated for 150 years? There's plenty of those that we regularly ignore.
> Even the silly confusing ones have a long life. E.g. "Rule against perpetuities"
That wouldn't be my go-to example of a silly law. It's what prevents control of property from remaining permanently with the will of a dead person who managed to own the property outright. It says that, at some point, the will can have no more influence and full ownership vests in someone who's alive.
Much of the US/UK legal system is based on common-law rules that are several hundred years old. In some cases those old laws have been codified, in some cases not, but either way there's no need to drop them just because they're old. On the contrary, laws that have stood that long without needing to be changed have demonstrated that they are extraordinarily good ideas.
I’m not sure point to the UK is a good example. There are plenty of weird and obscure laws that simply aren’t enforced or followed anymore. Everything from laws about handling salmon suspiciously, through to various right around who can drive sheep across Tower Bridge.
Those laws survive not because anyone considers them a good idea, but simply because the issues caused by ignoring them are substantially smaller than the effort involved in removing them.
We also have a bunch of laws that are still followed, but only in the most technical sense. Every “Parliament route” train schedule falls into that category. Train services that must be provided at least once a day, sometimes only once a week, which nobody actually uses, and in some cases only travel to stations with no practical public entrances. Those laws don’t survive because anyone things they’re a good idea, it’s just easier to run the train, than it is to get parliament time to abolish the law.
There is no automatic, fixed timeframe after which a law simply stops being followed because it hasn't been updated or looked at; and remember, we're still applying the FHS, it's in active use even if it's not updated.
Laws remain in force until they are formally:
* Repealed (abolished) by the relevant legislative body (Parliament, Congress, etc.).
* Struck down by a court as unconstitutional or otherwise invalid.
A 150 year "delete" timer would genuinely undermine the foundation of the legal system. Lawyers, judges, and businesses rely on the continuity of core laws (e.g., contract, property, and tax law). If a 150-year-old property law suddenly lapsed, it could instantly void millions of land titles and commercial contracts...
> Laws remain in force until they are formally: * Struck down by a court as unconstitutional or otherwise invalid.
False. They are still in force - they have just become unenforceable. There's a crucial difference, as the US is currently finding out: as long as they are in the books, a Supreme Court decision can instantly render them enforceable again - even against the wishes of the population.
The proper thing to do would be to "garbage collect" unenforceable laws, but politicians are (understandably) hesitant to spend political capital on it when it doesn't provide any tangible return.
There are other reasons as well. The body responsible for enforcing a particular law can choose not to enforce it, thereby rendering the law useless. Or a law can become obsolete by changes in technology or society - the original law legislates something that just doesn't happen any more, say. Laws can also be written to handle a specific event that only occurs once. Once that event has passed, the law might as well not exist. It doesn't need to be repealed because it just doesn't apply any more.
In addition, laws are typically regularly amended to handle new societal developments, to clarify wording, or to fit better with other laws or changes in attitudes. A law that has gone 150 years without being amended at all is probably a law that falls into the categories above and is obsolete.
Of course, all this is getting somewhat off-topic, but the point is that laws absolutely can become outdated and unmaintained, either deliberately or by happenstance. And the inverse is also true: most laws that people deal with regularly are kept up-to-date to ensure that they still reflect the needs and wills of the society they're being used in.
Obviously the intent was "Thou shall not murder anyone"... Interpreting it otherwise doesn't make sense, and is inconsistent with the rest of the Bible.
That's the wrong way to look at it. The Bible does not refer to other laws as a source of authority. Saying murder is "just" illegal killing is the real meaningless statement. Clearly people in the Bible are allowed to fight and defend themselves. Murder is typically intentional and unnecessary killing of someone else for malicious reasons (or no reason at all). Malicious reasons would typically be spite, greed, or convenience.
> The Bible does not refer to other laws as a source of authority
The Bible is the source of law here. My point was that interpreting it solely in the way you deem "obvious" does not work: you cannot have "thou shalt not murder" on its own without additional rules clarifying what counts as "murder" and what counts as "lawful killing" - and the Bible contains plenty of those.
> Murder is typically intentional and unnecessary killing of someone else for malicious reasons (or no reason at all). Malicious reasons would typically be spite, greed, or convenience.
That's how you interpret it. Modern law allows for killing out of greed - if the soldier firing the bullet is different from the politician wanting to capture some resources. We allow countries to kill out of spite with retaliatory strikes. We allow cops to kill in self-defense - even when other methods to subdue are theoretically available, but inconvenient. On the other hand, we no longer allow stoning to death people violating the Sabbath.
Clearly, there is a nontrivial list of criteria separating murder from lawful killing, and this list is mutable. In practice this list is codified in the law, which means murder becomes "killing which is not otherwise allowed in the law", which is the point I was trying to make.
Looping back to the original discussion: contrary to what ikiris was originally claiming it is not "thou shalt not kill" but "thou shalt not murder", and we've been updating the definition of "murder" (and by extent the meaning of "thou shalt not murder") quite a lot over the last few thousand years, so it is false to claim that " 'thou shalt not kill' never needs an update ".
>The Bible is the source of law here. My point was that interpreting it solely in the way you deem "obvious" does not work: you cannot have "thou shalt not murder" on its own without additional rules clarifying what counts as "murder" and what counts as "lawful killing" - and the Bible contains plenty of those.
It is obvious to people who know the Bible lol. It may have lots of contradictions but this isn't one of them. Murder was understood in a particular way to these ancient people, that still applies to us today.
>That's how you interpret it. Modern law allows for killing out of greed - if the soldier firing the bullet is different from the politician wanting to capture some resources.
As I said, I am not referring to laws of any country. The laws of modern countries are irrelevant to interpretation of the Bible. The dictionary does not count either.
>Looping back to the original discussion: contrary to what ikiris was originally claiming it is not "thou shalt not kill" but "thou shalt not murder", and we've been updating the definition of "murder" (and by extent the meaning of "thou shalt not murder") quite a lot over the last few thousand years, so it is false to claim that " 'thou shalt not kill' never needs an update ".
The only thing that needs an update is the translation. The meaning is very clear and universal. Don't kill people except in self-defense. It is just as antisocial today as it was thousands of years ago.
"Now go, attack the Amalekites and totally destroy[a] all that belongs to them. Do not spare them; put to death men and women, children and infants, cattle and sheep, camels and donkeys.’”
"and when the Lord your God has delivered them over to you and you have defeated them, then you must destroy them totally.[a] Make no treaty with them, and show them no mercy."
Lol yes... I've heard some alternative explanations of this. The Ten Commandments are not recognized outside of Christianity. They are just some important general sounding rules in a book that is full of rules for Jews, who get into battles with other people at times. Many of those rules in the same book have prescribed punishments of stoning to death, as well. So clearly, killing is at least allowed for purposes of punishment and warfare. Common sense also tells us that no religion can realistically prohibit self-defense.
"Samuel said to Saul, “I am the one the Lord sent to anoint you king over his people Israel; so listen now to the message from the Lord. 2 This is what the Lord Almighty says: ‘I will punish the Amalekites for what they did to Israel when they waylaid them as they came up from Egypt. 3 Now go, attack the Amalekites and totally destroy[a] all that belongs to them. Do not spare them; put to death men and women, children and infants, cattle and sheep, camels and donkeys.’”"
Later on it says that he was not happy with the outcome because they didn't kill all the livestock and people... Of course this is a work of mythology, but the message clearly isn't that the Israelites should go and loot from these people.
The US constitution is still in force after 236 years, and even older laws are still enforced. US courts will sometimes look at precedent from England before the colonies existed.
Meanwhile some laws that are months old are ignored by law enforcement because nothing forces them to read it. It’s that effect which is why so many old laws are ignored rather than formally repealed. When nobody is ridding a horse nobody cares how you need to tie one up when visiting a store etc.
> The US constitution is still in force after 236 years
True, but it's been updated a lot more recently than that.
The last update was still much longer ago than 10 years, of course. The most recently ratified amendment to the Constitution - the Twenty-Seventh Amendment, ratified 1992 - was, incredibly enough, proposed in 1789 along with the ten we know as the Bill of Rights and another one which was never ratified. And of the twenty-seven amendments ratified so far, the one most recently proposed by Congress, the Twenty-Sixth Amendment, was both proposed and ratified in 1971.
Are you suggesting that appending the constitution in 1992 with: No law, varying the compensation for the services of the Senators and Representatives, shall take effect, until an election of Representatives shall have intervened.
Somehow has an impact on anything else? Because by that standard every change to any law updates all existing laws that were not changed. Or I’m just completely misunderstanding your point here.
My point is that merely referencing the centuries-old original age of a document is misleading when it’s been updated dozens of times ending just a few decades ago. (And many of the updates, including the one in 1971, have been far more impactful than the Twenty-Seventh Amendment which you quoted.)
It’s certainly true that the constitution is old and crusty overall and desperately needs an overhaul, but the discussion was about when old laws which haven’t been updated in a while are ignored or enforced.
The constitution is indeed one law, not several different laws, and it’s been updated far more recently than its original year of promulgation or ratification. And it’s still mostly enforced (with increasing exceptions but that’s another discussion entirely).
18 times. 27 total amendments with 1-10 all passing on December 15, 1791.
> a few decades ago
There hasn’t been a meaningful change in over 54 years.
> The constitution is indeed one law, not several different laws
Those recent amendments are a minimum different laws. If you want to call it one law then there’s either 2 federal laws in the US. One needs to be ratified by the states and the others don’t.
> No 18 times. 27 total amendments with 1-10 all passing on December 15, 1791.
Fair correction.
> Those recent amendments are a minimum different laws.
Only in the sense that any new enactment is a new law, but in that sense no law can be updated in a way that preserves its identity as the same law. Which is not useful for the discussion of whether an old law has or hasn’t been updated, since by that definition updates to any law are impossible.
Lawyers and judges don’t exclusively use that sense any more than programmers view a software patch as changing the basic identity (beyond something like a version number) of the patched software program.
They do use that sense when they are referring to individual enactments by Congress, just like programmers refer to patches and patch releases as well as to the things which exist across many patch( release)s over time. Which sense is useful depends on context, and which sense is meant depends on a mixture of context and exact phrasing.
> If you want to call it one law then there’s either 2 federal laws in the US.
There are far more than 2 federal laws in the US, but far fewer than one per enactment except when using the sense of “a federal law” that specifically refers to each enactment.
For example, most federal tax legislation explicitly amends the Internal Revenue Code of 1986, which is still the official name of our federal tax code. Similarly, most immigration legislation amends the Immigration and Nationality Act of 1954 (I could have the year wrong), even more than 70 years later. The many “patch” laws enacted in the meantime all have their own identities via Public Law numbers and often a name, but they do update a specific identifiable underlying law without replacing it.
Other federal laws, like most annual appropriation and authorization bills, stand alone and are not routinely updated but have a finite duration of relevance. Common provisions are often carried forward from one iteration to the next, but they are re-enacted as part of each separate iteration.
And then there are the many parts of the federal legal landscape where the US Code is the official authoritative version instead of a mere convenience version as it is for things like the tax code and immigration law. Amendments to the directly authoritative parts of the US Code explicitly amend the US Code instead of a separately named law, so those directly authoritative parts of the US Code are themselves (whether each or collectively) a single federal law in the sense I’m discussing.
Yes, this stuff is complicated and messy in both the law and the software worlds.
> One needs to be ratified by the states and the others don’t.
Yes, constitutional amendments are laws (in one sense) that amend a law (in the other sense) and which states need to ratify, and regular Acts of Congress are laws (in the first sense) that may or may not amend one or more pre-existing laws (in the second sense) and which states do not need to ratify.
> Yes, constitutional amendments are laws (in one sense) that amend a law (in the other sense) and which states need to ratify, and regular Acts of Congress are laws (in the first sense) that may or may not amend one or more pre-existing laws (in the second sense) and which states do not need to ratify.
Yea, obviously we agree with what’s going on this is just a question of arbitrary definitions that don’t impact anything.
> Which is not useful for the discussion of whether an old law has or hasn’t been updated, since by that definition updates to any law are impossible.
It’s definitely easier work with the law based on the provided organizational structures with tax code being separate from family law etc. Yay, congress is doing something reasonably efficiently.
However, timing matters as making something illegal ex post facto is explicitly banned by the constitution etc. Further, in case of conflict newer laws win even without explicitly declaring the old laws invalid. So each bill is a meaningfully different law, and there’s in effect one law at any given moment after resolving those conflicts.
Net result three mutually contradictory but still useful definitions. But IMO organizational structure is by far the least meaningful one from a legal standpoint while being the most useful one from a practical standpoint.
You mean like those local laws that say you can’t walk a cow backwards through the main streets? Or laws that say a motor vehicle must be preceded by a lamp carrier.
> The FHS ultimately belongs to the users collectively, not those maintaining it.
I completely agree that regular updates are not a requirement for standards to remain relevant, but it does require the ecosystem to still adhere to them - and the problem is that Linux users are increasingly deviating from the FHS.
The FHS does not accurately describe the situation on-the-ground, there are no plans to update the FHS to accurately describe the situation on-the-ground, and there are no plans to update the ecosystem to accurately implement the FHS.
Like it or not: the FHS is dead, and nobody seems interested in reviving it.
I still don't think you adequately explained why that would matter. To reiterate the question OP asked: what updates does it need that it hasn't been getting? It isn't as if one would expect a "put X stuff in y location" document to need maintenance.
The FHS 3.0 doesn’t reflect current practices, such as the /usr merge, nor the /sys directory. There’s other ways it’s either no longer followed or missing developments from the last 10 years.
>> Debian Policy still cites the FHS, even though the FHS has gone unmaintained for more than a decade.
> What ongoing maintenance would a file system standard require? A successful standard of that type would have to remain static unless there was a serious issue to address. Regular changes are what the standard was intended to combat in the first place.
It's 2025, anything that wants to be considered modern (and everything should want that), needs to be undergoing constant change and delivering regular "improvements."
>>...though there is a slow-moving effort to revive and revise the standard as FHS 4.0, it has not yet produced any results.
> So it is not abandoned then. A slow moving process is exactly what you would want for the maintenance of a file system standard.
The FHS people to get off their butts. There's no excuse for that pace now that we have such well-developed AI assistants. They should be pushing quarterly updates at a minimum, and a breaking change at least every year or two. It's been obvious for decades that "etc" is in urgent need of renaming to "config", "home" to "user", and "usr" to "Program Files" to keep up with modern UX trends.
I thought it would have been obvious by the "Program Files" at the end :).
Anyway, Linux community as a whole has an antiquated development process, and needs to modernize and follow the best practices of an industry-leading trend-setter, like MS Teams.
Surely Linux should be developed Google style, with a Web Scale perspective. uucp, tar, yacc, roff removed (with a 30 day notice, of course!), all the uses of "creat" amended to add the final "e", etc.
Wait you were kidding? I've already spun up a kubes cluster so we can feed FHS in a globally load-balanced CI/CD pipeline, I have an agentic LLM doing constant improvements so we can sprint to you never knowing where your files are!
It's like ASLR for files but no maps because maps aren't for trailblazers, they make the maps! It's very cutting edge and a value-add!
Counterpoint: Modern distro’s needs have evolved past the FHS in some cases, and everybody deviates from it slightly but incompatibly.
A standard does no good if it does not reflect reality. I think it is a worthwhile effort to try to bring it back in line with actual real world usage.
it's remarkable to me that NixOS manages to run so well despite breaking the FHS so thoroughly. and not just in superficial ways like not calling it /bin, I mean forsaking dynamic linking (hence /var/lib and /usr/lib), keeping man pages, resources and config bundled into the same derivation as the binary sometimes, and occasionally hacking up binary blobs to rewrite rpaths.
on the other hand, there's a place for legacy distros too.
An interesting take in the context of systemd's deviation from it being the source of bugs because of real world usage that conflicts with systemd's changes.
OS X doesn't have a merged /usr. On my Mac I see /bin as a separate directory, not a symlink into `/usr`. Does Linux have a compelling reason that OS X lacks?
The reason for having a separate /usr has long slid into obsolescence. Our storage is no longer constrained to require us to mount a remote partition which holds most of the binaries and other sundry required to boot a distribution.
I'm assuming you're referring to partitioning of boot-critical binaries into `/bin`, but "the reason for having a separate /usr" is even older and worse than that. In original Unix `/usr` was for home dirs[1], and was colonized by the operating system in 1971 when it no longer fit on a single 1.5MB RK05 disk. Nobody ever untangled that change and we've been living with the hack ever since.
I think the current strongest argument against it is that systemd complains when /usr is on a separate partition[1], and what its devs have weighed in on the matter[2].
sysvinit had no problem being told to mount /usr as soon as network was available, and if you set up an init script to run before /usr was available, but the script needed /usr, that was your own fault.
systemd relies on things in /usr being available, including to decide which scripts to run, and mounting /usr would be one of those scripts, so it has a chicken-and-egg problem.
But ah, it doesn't! Instead the world needs to make sure /usr is mounted before systemd even gets started, so systemd doesn't have to fix its bug.
Personally, I don't mind /usr/bin merging with /bin, the benefit I can see is no more squabbling over whether something should be in /bin or not (i.e. is this tool needed to boot the system, or not?)
The difference being that the authors of sysvinit didn't advertise obnoxious messages at boot time (https://systemd.io/SEPARATE_USR_IS_BROKEN/) and try to get the filesystem standards changed.
One is like "I'll run some scripts in order, everything else is on you", the other is like "I'll take care of everything, I'll do that, WHAT YOU DIDN'T MOUNT /USR ? SHAME ON YOU I DON'T WANT TO DEAL WITH THAT CORNER-CASE"
I read that message as "Whoever is the poor soul attempting to fix this system that doesn't boot, (because who otherwise inspects early boot messages?) whoever installed this system is running a known-fragile configuration."
And that's what I expect of systemd? That it should complain loudly whenever me, the daemons I'm attempting to run, or the overall system is doing things in a weird, known-bad, known-fragile way and warn me about it before it breaks if possible.
systemd is my hammer having it's own opinions about what nails I hit & where & into what & why.
I want a nail only driven half in and at some crooked angle, that's my business.
It's not my hammers job to agree or disagree that it's a bad nail hammering job as far as it knows. I don't wantto have to convince it of the validity of a use-case it didn't think of before, or thought of and decided it doesn't agree to support.
I just want that crude coat hanger and I don't care who else likes it or doesn't like it or who else thinks I should buy an actual coat hanger and attach it in some way that someone else approves of.
> systemd is my hammer having it's own opinions about what nails I hit & where & into what & why.
I feel like the exact opposite. systemd has way too many layers of customization. Things can be in /etc/systemd/system/, /run/systemd/system/, /usr/lib/systemd/system/* and those are just the "basic" override locations. And those edits and masks usually don't clash with each other, so your customizations and quirks stick, even if the rest of your system changes.
Now, if you are in a system that doesn't use systemd, if you mess with the "sacred and sanctioned" system service summoning scrolls, now you're on your own to diff whoever stuff the maintainer did, every time the summoning scroll changes, and to make sure that probably Turing-complete mess still summons a valid, working and safe system component instead of rampaging thru your system.
> Things can be in /etc/systemd/system/, /run/systemd/system/, /usr/lib/systemd/system/* and those are just the "basic" override locations. And those edits and masks usually don't clash with each other, so your customizations and quirks stick, even if the rest of your system changes.
I really encourage you to learn the difference between these three places. They are semantically distinct, and it's truly an advantage to have the system distinguish them. They're all sources of configuration, of course. But one is managed by the OS vendor, one by the sysadmin, and one by the sysadmin at runtime. Runtime isn't as significant a distinction, but having a clear, bright line between vendor-provided and sysadmin-managed config is huge.
There is no analogy that worked both ways here. Some fitting counter-analogy might exist, but this isn't it, because there are no razor blade analogs in the handle analog of init.
I used hammer as the example because init is actually like a hammer. It is a dead simple impliment with limited scope and knowable, known, and absolutely predictable behaviors and properties. It doesn't actually do much of anything, which makes it infinitely flexible. By not having any will of it's own, it allows you to do more than a more complex, automatic, and integrated tool.
It can't suddenly razor blade you because it doesn't have razor blades or any other special features in the first place.
YOU are the complex integrator. And if you are not, I have no sympathy for you or whatever difficulty you have with responsibility.
You can still have complex automation and management, built on top of simpler lower level agnostic layers and tools. There was never any requirement to remove that flexibility and elegant future-proof design in order to have all the things systemd promises (and really, merely promises, it does not actually deliver any better than anything else).
Systemd is also a grand scope example of tight coupling. It's so bizarre how these supposed genius coders can commit such a huge example of something they all know to avoid like the plague in just a slightly different context.
In general, systemd-ish projects expect you to bend the system to match the project's expectation while sysvinit-ish projects are infinitely flexible to match any system (jack of all trades, master of none; worse is better; etc).
From the creators of systemd we also have GNOME, PulseAudio, and Wayland. They have some design philosophy in common.
BTW most sysvinit distros barely even use sysvinit. sysvinit is a service monitor, similar to systemd but more primitive, but typically most of what it's configured to do is to launch some shell scripts on startup. We really have "systemd distros" and "ad-hoc script distros", not sysvinit distros ("ad-hoc" is not a pejorative). I don't know why they don't make init a shell script directly - you can do that, and it's typically done that way in initramfs.
> In general, systemd-ish projects expect you to bend the system to match the project's expectation while sysvinit-ish projects are infinitely flexible to match any system (jack of all trades, master of none; worse is better; etc).
Name one thing you can do in sysvinit (or "ad-hoc script"-init) that you cannot do in systemd. With systemd, you're still entirely free to write your own artisinal shell spaghetti and jam it in alongside the otherwise clean unit structure.
Assuming that your comment is in good faith, that comparison doesn't work in the way I think you're trying to make it work.
systemd operates (when used normally) at a higher level of abstraction than a bucket-of-scripts/sysvinit. Especially because systemd units are declarative, this higher level of abstraction affords you (the sysadmin) the ability to more precisely specify the behavior you want. However, you can of course run scripts from systemd, so you've got whatever escape hatches you like. My question was asking if there was anything in sysvinit that those escape hatches didn't cover adequately.
Assembly, on the other hand, operates at a far lower level of abstraction. Inappropriately low, in fact. Because of this, you actually have less practical ability to specify the precise behaviors you want. For example, you'd be hard-pressed to use assembly to start a webserver after the network is up but before some other piece of software that depends on the webserver.
no, it is that it adds complexity which is no longer needed
especially for image based stuff it's a pain
which includes OCI images for things like docker
but also image based distros like e.g. ostree (as used through rpm-ostree by Atomic Fedora desktops like Fedora Silverblue, but also in similar but different forms something Ubuntu has been experimenting with)
Your comment seems fully unrelated to my point that overlaying images is much more a pain if the things you might want to shadow and/or extend are distributed or even duplicated across many different places when they could be just be in one place.
On the contrary. While the issue is not space, and hasn't been for a long time, a lot of devices are "network/cloud-first" and it is modern to load programs over the internet. It seams all the more useful, to have the distinction between core binaries, that are needed to boot, connect to the network and should be there in case the network setup became toast for debugging, and programs that can be loaded from the network on demand.
macOS didn't merge /usr, but it did do something sorta related.
One of the purposes of usrmerge is to cleanly separate the read-only and read-write parts of the system. This helps with image-based distros, where /usr can be on its own read-only filesystem, and related use cases such as [1]. Usrmerge is not required for image-based distros to work [2], but it makes things cleaner.
macOS, starting in 2019, is also an 'image-based distro', in that it has a read-only filesystem for system files and a separate read-write filesystem for user data. However, the read-only filesystem is mounted at / instead of /usr. Several different paths under the root need to be writable [3], which is implemented by having a single read-write filesystem (/System/Volumes/Data) plus a number of "firmlinks" from paths in the read-only filesystem to corresponding paths in the read-write filesystem. Firmlinks are a bespoke kernel feature invented for this purpose.
Both approaches have their advantages and disadvantages. The macOS approach is nice in that the system filesystem contains _all_ read-only files/directories, whereas under "distro in /usr" scheme, you need a separate tmpfs at / to contain the mount points and the symlinks into /usr. But "distro in /usr" has the advantage of making the separation between read-only and read-write files simpler and more visible to the user. Relatedly, macOS's scheme has the disadvantage that every writable file has two separate paths, one with /System/Volumes/Data and one without. But "distro in /usr" has the opposite disadvantage, in that a lot of read-only files have two separate paths, one with /usr and one without. Finally, macOS's scheme has the disadvantage that it required inventing and using firmlinks. Linux can already achieve similar effects using bind mounts or overlayfs, but those have minor disadvantages (bind mounts are more annoying to set up and tear down; overlayfs has a bit of performance overhead). Actual firmlinks are not necessarily any better, though, since they don't have a clear story for being shared between containers (which macOS does not support). It is nice that "distro in /usr" doesn't require any such complexity.
Ultimately, the constraints and motivations on both sides are quite different. macOS couldn't have gotten everything read-only under one directory as easily because it has /System in addition to /usr. macOS doesn't have containers. macOS doesn't have different distros with different filesystem layouts and deployment mechanisms. And philosophically, for all that people accuse systemd of departing from Unix design principles, systemd seems to see itself as evolving the Unix design, whereas macOS tends to treat Unix like some legacy thing. It's no surprise that systemd would try to improve on Unix with things like "/bin points to /usr/bin" while macOS would leave the Unix bits as-is.
> What ongoing maintenance would a file system standard require?
adaption to _a lot_ of subtle changes to requirements
- very different security related requirements today
- very different performance related requirements/characteristics
- very different need for various edge cases
and lastly adapt based on what turned out to work well and what didn't
so some examples not already mentioned in the article
- /boot -- dead or at least differently used if you use efistub booting
- /etc/X11 -- half dead on wayland
- /etc/xml, /etc/sgml -- dead, should IMHO never have existed
- also why was /etc/{X11,xml,sgml} every explicit part of the standard when the spec for `/etc` already implies them as long as e.g X11 is used ??
- `/media` -- dead/half dead depending on distro, replaced by `/run/media/{username}/{mount}`
- `/sbin` -- "controversial"; frequent reoccurring discussions that it isn't needed anymore, didn't work out as intended etc. It was useful for very old style thin clients as `/sbin` was in storage but `/bin` was mounted. And there are still some edge cases where it can makes sense today but most fall under "workaround for a different kind of problem which is better fixed properly".
- `/tmp` -- "controversial", long history of security issues, `/tmp` dir per program fixes the security issues (e.g. systemd service PrivateTmp option) but requires having a concept of "programs" instead of just "running processes" (e.g. by systemd services or flatpack programs). Also `tmpfiles.d` can help here.
- `/usr/libexec` -- dead, nice idea but introduces unneeded complexity and can be very misleading in combination swith suid and similar
- `/usr/sbin` see `/sbin`
- `/usr/share/{color,dict,man,misc,ppd,sgml,xml}` -- should never have been in the standard they are implied by the definition of `/usr/share`; at least sqml,xml are dead. dict was for spell check/auto completion, except that neither works anymore like dict expects
- `/var/account` -- to specific to some subset of partially dead programs, shouldn't be in the standard
- `/var/crash` -- distro specific mess
- `/var/games` -- basically dead/security mess, I mean 99% of games today are user per-user installed (e.g. Steam) and even for such which are packed any variable download data is per user, making it shared creates a permission/security mess
- `/var/lock` -- as mentioned there are better technical solutions by now, e.g. using `flock` instead of "presence of file" and some other techniques. Tend to also avoid issues of crashed programs not cleaning up "lock files" leading to dead locks and needing manual intervention.
- `/var/mail` assumes a quite outdated form of managing mail which is quite specific to the mailing program, as it's very program specific it IMHO shouldn't be in the standard
- various legacy program specific, non "generic" file system requirements e.g. that `/usr/lib/sendmail` must exist and be a link to a sendmail compatible program and similar.
also missing parts:
- `/run/user/{uid}`
- `/var/run/user/{uid}`
- `/proc`
- `/sys`
- user side versions (e.g. from the XDG spec which is also somewhat in a zombie state from my personal experience with it , e.g. .config, .local/{bin,share})
- references to light weight sandboxing, e.g. per-program /temp etc.
- factory reset stuff (`/usr/share/factory`) needed for having a uniform way for devices sold with Linux and device specific distro customization(e.g. steam deck)
Well, it probably depends on which software's concern will be implementing a policy to prevent users from having permission to fill critical directories and prevent the system from operating normally, which is discussed in the article. Which is also a coordination problem because the most common user of disk is software itself, I think.
FHS seems to specifically imbue the user with the responsibility and consequences of filling up the disk.
> The FHS 3.0 is clearly reaching the end of its useful life, if not actually expired.
Interesting take.
I think that the FHS is still extremely helpful for packagers, sysadmins and others so they won't stomp on each other's feet constantly. It helps set expectations and prevents unnecessary surprises.
Just the fact that one particular FHS rule might be outdated or even harmful doesn't mean that the FHS as a whole has outlived its usefulness.
Standards are a double edged sword though. They are great for getting everyone to agree to the "most correct" answer. But they also freeze evolution in place. What happens when your standard doesn't support contemporary use cases? What if it's at direct odds with, say, modern security practices?
FHS hasn't changed in years. Since then, sandboxing, containers, novel package schemes, and more are the zeitgeist. What does the FHS say about them?
Looking at this specific use case, someone is saying /var/lock being world-writable is an unacceptable security risk, but that's very dependent on what your world/users look like. If anything it sounds to me like the maintainer is trying to make the FHS smaller and remove support for a lot of use cases. (Use cases that sound pretty valid to me, without digging in.)
Nothing keeps you from following the FHS inside your container or sandbox.
Are you referring to the location where container images live? Then `/var/lib/containers/` and `/var/lib/containers/storage/` would be perfectly FHS compliant.
The idea though is when you don't want to follow the FHS anymore, like systemd is doing.
Systemd frustrates and angers people with Poettering's complete disregard for bug reports, tradition, and basic common courtesy. At the same time, change needed to happen and change is gonna hurt. And big changes can't wait until they're just as stable as the old system: does anyone develop software like that in their own careers? I try not to ship complete crap but "just as stable as v1" is never a goal.
> Systemd frustrates and angers people with Poettering's complete disregard for bug reports, tradition, and basic common courtesy
Poettering is a Microsoft employee. It is normal that he follows the direction of the mothership. What is not normal is, that he has so many blind followers.
every distro has defined their own new file system layout standard
sure they all started out with the common ancestor of FHS 3.0, but diverged since then in various degrees
and some modern competing standards try to fix it (mainly UAPI Group)
(And yes some people will go one and one about how UAPI is just a way for systemd to force their ideas on others, but if you don't update a standard for 10+[1] years and aren't okay with others taking over this work either, idk. how you can complain for them making their own standard).
[1]: It's more like 20 years, but 10 years ago the Linux Fundation took over it's ownership.
Debian systemd maintainer Luca Boccassi has recently pushed through and dismissed several problematic and undesired breakages as "niche cases" in a way I personally find antithetical to what I expect from Debian.
I hope they have a change of mind in their approach.
Yes, the guy is legendary in terms of maximum arrogance. He did impressive work early on designing a complex system, but gets defensive when that overengineered moloch runs into real-world problems. Systemd has accumulated lots of small hacks to make it more versatile, let's hope a better solution will be available one day.
Apparently yes, since the parent to your comment has been flagged.
Personally I find an interesting observation, and microsoft contributing to linux in any way should be met with skepticism based on the entire last 30 years.
People are so quick to wipe away any wrongdoing from Microsoft as soon as they get thrown a bone, there's some interesting psychology here.
I would say no because microsoft seems to have a magical ability to over-complicate systems, UIs, etc. Not to mention the fact that they were out to crush Linux not so many years ago (through proxies even!). Trusting them to make good, unix-like choices seems ill-informed.
It's complicated. Microsoft devotes resources to it and they can afford to do so but they only have that luxury from being a massive user trampling megacorp.
If we could trust them to be devoting resources to it without any risk of abusing their access and power in the future, that would be sort of okay, but we can't.
Like, should Lockheed intentionally hire North Korean programmers at cheap rates because North Korea can afford to devote resources to helping Lockheed? The issue here is not primarily that North Korea is a massive citizen-trampling megastate. It's that Lockheed's interests are misaligned with North Korea's.
The goals of Debian is based on those who contribute to it. If Microsoft contributes, then it cannot be misaligned. That's what it means to be open. You may not like how that affects Debian, but to deny Microsoft the ability to participate would be equivalent to no longer being truly open.
Also individuals tend to prioritize work that benefits employer interest but that doesn't mean they can do things arbitrarily. It just shifts energy and focus towards certain areas. It's not a problem unless the company employs a large fraction of Debian maintainers which Microsoft doesn't.
No, Debian has a mission of its own, and has always excluded candidates for Developership if they are opposed to Debian's mission. It has never been open in the way you are imagining.
I think employing the project lead of systemd gives Microsoft a kind of influence that employing the packager of libjpeg-turbo wouldn't. Lennart is notorious for doing things arbitrarily, and what we are discussing here is that the Debian package maintainer for systemd is also doing things arbitrarily, and is also employed by Microsoft.
how does that work when Microsoft also makes a competing operating system?
If we think about it in logical terms, they could sabotage Debian by having “interests” that are suboptimal for their core demographic.
this is a similar to how euro-skeptics are the people who make the very unpopular laws inside the European Union, leading to all the negative press about the European Union. But they have to be listened to as they are democratically elected and it is a democracy.
I don't know of anyone that's been doing this for a while that hasn't been touched by systemd stupidity in some way. I still loathe the default behavior around the stub-resolver with unqualified names that "just worked" before Lennart decided he knew best.
I've been a Linux sysadmin professionally for 10 years now. I have never had problems with systemd, nor do I consider it to be stupid. Obviously you don't have to like it, but I think you're painting with too broad of a brush.
Doesn’t that mean you started being a sysadmin after systemd was widely adopted? Most people who dislike it probably actively worked with Linux prior to systemd.
Depends on what you mean by this in "been doing this"?
While work now mandates "If you want to use Linux, it has to be Ubuntu" (and I complied). On personal front - about a decade ago I've moved from "vanilla" Gentoo to Calculate Linux - which was and still is 100% Gentoo.
These days difference is even smaller, but already 10+ years ago Calculate had sane profiles as well as all software packages as pre compiled binaries matching those profiles.
And although systemd is one of configurable USE keywords on Calculate/Gentoo - it's still not the default.
So there probably are some folks that haven't been touched by systemd at all... For now.
To be fair, he does know how globbing works: ".*" should include "." and ".." under normal globbing rules. The 'rm' command (presumably) has a special case in it to avoid traversing those in recursive mode because doing so would be a footgun.
There is Devuan, if you want to Debian but without systemd. I suspect though that "natively" non-systemd distros will be more consistent, personally I've found happiness with Guix.
In this case, I think the upstream maintainer's response -- "Upstream systemd will do X, distros who want to are free to do Y" -- is legitimate. Consider the reverse: If systemd requires a writable /run/lock, then distros who want to be more safe won't really be able to (or will have to implement a much more intrusive patch).
Looking from the outside, it looks more that this is a failure of the Debian systemd package maintainer to follow Debian's rules. (Though since I'm not a part of that community, I recognize that there may be cultural expectations I'm not aware of.)
> "Upstream systemd will do X, distros who want to are free to do Y"
Yes this is a good response from upstream. I can work with that, but in that case, even this response didn't get reflected to mailing list discussion, or drowned out instantly.
My question was more general though, questioning systemd developers' behavior collectively (hence the projects' behavior) through time.
The systemd developers have a long history of reinventing the wheel and trying to force it on everyone. We only put up with them because they do some difficult work that nobody wants to do.
Speak for yourself, then. I’ve been using Linux since 2004, and the systemd components finally made system management easy. No more arcane init scripts. Handling of service dependencies. Proper timers. Simple configuration files. Administration knowledge that immediately carries over between all systems equally.
As a user, systemd has improved my productivity tremendously.
The kind of bad mouthing developers that work on solutions for complex problems, code that runs on billions of machines, reflects more of your own fragile ego than them.
> The systemd developers have a long history of reinventing the wheel and trying to force it on everyone. We only put up with them because they do some difficult work that nobody wants to do.
> As a user, systemd has improved my productivity tremendously.
Both can be true at the same time. Particularly in the beginning, there was a long string of really important things that used to Just Work that were broken by systemd. Things like:
1. Having home directories in automounted NFS. Under sysv, autofs waited until the network was up to start running. Originally under systemd, "the network" was counted as being up when localhost was up.
2. Being able to type "exit" from an ssh session and have the connection close. Under systemd, closing the login shell would kill -9 all processes with that userid, including the sshd process handling the connection -- before that process could close the socket for the connection. Meaning you type "exit" in an interactive terminal and it hang.
It's been a while since I encountered any major issues with systemd, but for the first few years there were loads of issues with important things that used to Just Work and then broke and took forever to fix because they didn't happen to affect the systemd maintainers. If you didn't encounter any of these, it's probably because your use cases happened to overlap theirs.
Yes, systemd and journalctl have massively simplified my life. But I think it could have been done with far less disruption.
My favorite systemd bug is when your network black-holes (not disconnected, SYN packets work but nothing else will come back). The entire system will hang.
There's no need to be rude. While I'm not anti-systemd; it didn't change my life tremendously, either.
People tend to bash init scripts, but when they are written well, they both work and port well between systems. At least this is my experience with the fleet I manage.
Dependencies worked pretty well in Parallel-SysV, too, again from my experience. Also, systemd is not faster than Parallel-SysV.
It's not that "I had to learn everything from scratch!" woe either. I'm a kind of developer/sysadmin who never whines and just reads the documentation.
I wrote tons of service files and init scripts during Debian's migration. I was a tech-lead of a Debian derivative at that time (albeit working literally underground), too.
systemd and its developers went through a lot phases, remade a lot of mistakes despite being warned about them, and took at least a couple of wrong turns and booed for all the right reasons.
The anger they pull on themselves are not unfounded, yet I don't believe they should be on the receiving end of a flame-war.
From my perspective, systemd developers can benefit tremendously by stepping down from their thrones and look eye to eye with their users. Being kind towards each other never harms anyone, incl. you.
Everything is a slight modification of something someone else has already done. However it does take effort to make things that work well together. Systemd may not have any novel tricks but it sure does synergize well. Standardization also goes a long way towards simplifying things for a lot of folks. Otherwise everyone constantly reinvents everything and certain integrations are going to invariably be broken or not work well. Modularity is great but it's not free.
The central problem with Linux is that it forces you to conform to everything being a files, including device and IO. I’ll stick to MSDOS where I can go about business poking memory myself to control the hardware. /s
My point being is that contracts between systems can improve our lives. We take the process, file, and protected memory models for granted today. However, I’m sure people hated the migration to protected systems as well because it “didn’t let them go about their business”
Linux is 34 years old, and some of the Unix-ism borrowed are even older. There is genuine cruft that has downsides. Different relative priorities of backwards comparability, maintainability and the various issues the legacy issues cause are reasonable.
Systemd basically arose out of a frustration at the legacy issues so the whole project exists as a modernizing effort. No wonder they consider backwards compatibility low priority.
Because that's what they've always done, and it continues to work for them?
Systemd doesn't work for me, but it has taken over most Linux distributions, so clearly it's got something people want that I don't understand. That was the case for PulseAudio too.
* Their is an option for the old behavior.
* It is a security issue and better solutions to replace exist.
* FHS isn't maintained.
I think everyone involved would prefer updates to the applications, which fix the issue. Debian opted - for now - for reliability for its users, which fits in their mission statement. On Arch /run/lock is only writeable for the superusers, which improves security. As user I value reliability and security and that legacy tools remain usable (sometimes by default, sometimes by a switch).
The "security issue" expressed is that someone creates 4 billion lock files. The entire reason an application would have a path to create these lock files is because it's dealing with a shared resource. It's pretty likely that lock files wouldn't be the only route for an application to kill a system. Which is a reason why this "security issue" isn't something anyone has taken seriously.
The reason is much more transparent if you read between the lines. Systemd wants to own the "/run" folder and they don't like the idea of user space applications being able to play in their pool. Notice they don't have the same security concerns for /var/tmp, for example.
> On Arch /run/lock is only writeable for the superusers, which improves security.
Does it? That means anyone who needs a lock gets superuser, which seems like overkill. Having a group with write permissions would seem to improve security more?
a global /run/lock dir is an outdated mechanism not needed anymore
when the standard was written (20 years ago) it standardized a common way programs used to work around not having something like flock. This is also reflected in the specific details of FHS 3.0 which requires lock files to be named as `LCK..{device_name}` and must contain the process id in a specific encoding. Now the funny part. Flock was added to Linux in ~1996, so even when the standard was written it was already on the way of being outdated and it was just a matter of time until most programs start using flock.
This brings is to two ways how this being a issues makes IMHO little sense:
- a lot of use cases for /var/lock have been replaced with flock
- having a global writable dire used across users has a really bad history (including security vulnerabilities) so there have been ongoing affords to create alternatives for anything like that. E.g. /run/user/{uid}, ~/.local/{bin,share,state,etc.}, systemd PrivateTemp etc.
- so any program running as user not wanting to use flock should place their lock file in `/run/user/{uid}` like e.g. pipewire, wayland, docker and similar do (specifically $XDG_RUNTIME_DIR which happens to be `/un/user/{uid}`)
So the only programs affected by it are programs which:
- don't run as root
- don't use flock
- and don't really follow best practices introduced with the XDG standard either
- ignore that it was quite predictable that /var/lock will get limited or outright removed due to long standing efforts to remove global writable dirs everywhere
i.e. software stuck in the last century, or in this case more like 2 centuries ago in the 2000th
But that is a common theme with Debian Stable, you have to fight even to just remove something which we know since 20 years to be a bad design. If it weren't for Debians reputation I think the systemd devs might have been more surprised by this being an issue then the Debian maintainers about some niche tools using outdated mechanisms breaking.
OK, but suppose you have a piece of software you need to run, that's stuck in the last century, that you can't modify: maybe you lack the technical expertise, or maybe you don't even have access to the source code. Would you rather run it as root, or run it as a user that's a member of a group allowed to write to that directory?
The systemd maintainers (both upstream and Debian package maintainers) have a long history of wanting to ignore any use cases they find inconvenient.
most very old software will depend on many other parts so you anyway often have to run it in a vm with a old Linux kernel + distro release or similar
and if not, you can always put it in a container in which `/var/lock` permissions are changed to not being root-only. Which you probably anyway should do for any abandon ware.
That’s true and compatibility is a grace. Software shall not need every month an update. I sign of its quality.
Is this case, usage of /var/lock was clumsy for a long time. And not cleaning up APIs creates something horrible like Windows. API breaks should be limited, to the absolute minimum. The nice part here is, that we can adapt and patch code on Linux usually.
On the other side Linux (the kernel), GLIBC/STDLIBC++, Systemd and Wayland need to be API stable. Everybody dislikes API-Instability.
No, I didn't read the whole article. I follow debian-devel directly. Watched all of it unravel, step by step. I know the resolution since the day it posted to debian-devel.
This was a general question to begin with.
> Their is an option for the old behavior.
The discussion never centered on an option for keeping old behavior for any legitimate reason. The general tone was "systemd wants it this way, so Debian shall oblige". It was a borderline flame-war between more reasonable people and another party which yelled "we say so!"
> It is a security issue and modern solutions to replace exist.
I'm a Linux newbie. Using Linux for 23 years and managing them professionally for 20+ years. I have yet to see an attack involving /var/lock folder being world-writeable. /dev/shm is a much bigger attack surface from my experience.
Migration to flock(2) is not a bad idea, but acting like Nero and setting mailing lists ablaze is not the way to do this. People can cooperate, yet some people love to rain on others and make their life miserable because they think their demands require immediate obedience.
> FHS isn't maintained.
Isn't maintained or not improved fast enough to please systemd devs? IDK. There are standards and RFCs which underpin a ton of things which are not updated.
We tend to call them mature, not unmaintained/abandoned.
> On Arch /run/lock is only writeable for the superusers. As user I value reliability and the legacy tools are usable.
I also value the reliability and agree that legacy tools shall continue working. This is why I use Debian primarily, for the same last 20+ years.
I mean /var/lock was kinda on the way of being super seeded when FHS3 was written 20 years ago. We known it is bad design since a similar amount of time.
If FHS hadn't been unmaintained for nearly 2 decades I'm pretty sure non-root /var/lock would most likely have been deprecated over a decade ago (or at least recommended against being used). We know that cross user writable global dirs are a pretty bad idea since decades, if we can't even fix that I don't see a future for Linux tbh.(1)
Sure systemd should have given them a heads up, sure it makes sense to temporary revert this change to have a transition period. But this change has be on the horizon for over 20 year, and there isn't really any way around it long term.
(1): This might sound a bit ridiculous, but security requirements have been changing. In 2000 trusting most programs you run was fine. Today not so much, you can't really trust anything you run anymore. And it's just a matter of time until it is negligent (like in a legal liability way) if you trust anything but your core OS components, and even that not without constraints. As much as it sucks, if Linux doesn't adept it dies. And it does adopt, but mostly outside of the GPG/FSF space and also I think a bit to slow on desktop. I'm pretty worried about that.
> > FHS isn't maintained.
> Isn't maintained or not improved fast enough to please systemd devs? IDK.
more like not maintained at all for 20+ years in a context where everything around it had major changes to the requirements/needs
they didn't even fix the definition of /var/lock. They say it can be used for various lock files but also specify a naming convention must be used, which only works for devices and also only for such not in a sub-dir structure. It also fails to specify that it you should (or at least are allowed to cleared the dir with reboot, something they do clarify for temp). It also in a foot note says all locks should be world readable, but that isn't true anymore since a long time. There are certain lock grouping folders (also not in the spec) where you don't need or want them to be public as it only leaks details which maybe an attacker could use in some obscure niche case.
A mature standard is one which has fixes, improvements and clarification, including wrt. changes in the environment its used in. A standard which recognizes when there is some suboptimal design and adds a warning, recommending not to use that sub-optimal desing etc. Nothing of the sort happened with this standard.
What we see instead is a standard which not only hasn't gotten any relevant updates for ~20 years but didn't even fix inconsistencies in itself.
For a standard to become mature it needs mature, that is a process of growing up, like fixing inconsistencies, clarifications, and deprecation (which doesn't imply removal later one). And this hasn't happen for a long time. Just because something has been used for a long time doesn't mean it's mature.
And if you want to be nit picky even Debian doesn't "fully" comply with FH3, because there are points in it which just don't make sense anymore, and they haven't been fixed them for 20 years.
> In 2000 trusting most programs you run was fine.
Yes. This is why Microsoft didn't decide to base Windows XP on the NT kernel and Windows 95 was nothing more than a (arguably very) pretty coat of paint on top of Windows 3.11.
It's also why multi-user systems with complicated permissions systems that ran processes in isolated virtual address spaces never got built in the decades prior to NT. All those OS researchers and sysadmins saw no reason to distrust the programs other users intended to run.
Probably because everyone lets them keep getting away with it.
Any time there's systemd criticism there's always a quick rebuttal "But it was too hard writing anything in any other init system before so stop complaining".
So there's enough pass being given from the start to Systemd and the developers because it always has been forced upon us.
In this case I assume their "fear" is that unprivileged users can exhaust resources (inodes, filesystem space) in an important tmpfs breaking the system. The proper solution for backward compatibility would probably be something like make /run/lock its own mountpoint, but they fixed it in their system (Fedora) so now it's no longer their problem. Just be thankful their software is portable to such strange niche operating systems like Debian. /s
First before venting I'll say this: thanks to stuff like hypervisors, VMs, non-systemd distros, minimal immutable Linux distro like Talos (made to run Kubernetes with a minimum number of executable) and OCI containers where, by definition, PID 1 is not systemd, there's thankfully a real way out of systemd, even on a Linux stack.
And I think more people should look into being, once again, 100% systemd-free.
> Can anyone tell why systemd developers run fast and loose with what they believe and bully everyone with a stick made out of their ideas?
Because the goal is to take control of Linux. That's why systemd is PID1. That's why Poettering works for Microsoft.
The real question is: why was that ultra-convoluted xz backdoor attempt only working on Linux systems that did have systemd? People shall try to wag the dog saying "but it's because this and that made is so that xz was loaded by OpenSSH, it's got nothing to do with systemd". It's got everything to do with systemd.
And the other question is: how many backdoors are operational, today, on systems that have systemd?
Systemd is Microsoft-level bloat, running as PID 1, spreading its tentacles everywhere in Linux distros, definitely on purpose.
Poettering is moreover an insufferable bully, as can be seen once again.
From TFA:
> So what do you recommend how to go on from here? Change Debian policy (as asked in #1111839), revert the change in systemd, find a Debian wide solution or let every package maintainer implement their own solution?
I suggest Debian just drops systemd once and for all. Debian can still be made systemd-free but it's a hassle. Just make Debian systemd free once again.
Meanwhile you'll find me running systemd-less distros on VMs and running containers giving the PID 1 finger to systemd.
I can't wait to switch my Proxmox to FreeBSD's bhyve hypervisor (need to find the time to do it).
But most of all: I cannot wait for the day a systemd-less hypervisor Linux like Proxmox comes out.
It's coming and people who write stuff like: "Don't use Docker, use systemd this and systemd that" are misguided.
systemd is to me the antithesis of what Linux stands for.
I hope Debian gets pissed enough at some point to fully drop systemd.
P.S: one of my machine runs this: https://www.devuan.org/ and honestly it's totally fine. So yup: power to all those running systemd-less distros, BSDs, etc.
I still cannot understand how it came about that everyone adopted systemd, despite it being a showcase of the most selfish, reckless and irresponsible maintainership ever. And it's not like it's something new, it's been like that for what, 10, 15 years now?
The last paragraph in the article states how Fedora uses lockdev(3) to somehow address the issue. It's not clear to me how a library that exposes a standardized management interface to these locks enables non-root users to create them without a world writable lock directory. Is it that lockdev(3) allows the system to use a different directory?
Secondly, it's suggested that the /run/lock directory and/or the solution of acquiring a lock for a shared resources is dated -- but, no mention of the "modern" way to address this? Is it implied that systemd offers a solution to this? What is it?
Another conflict: in Debian Trixie (current stable release), when you run `systemctl status …` you’re given the rather scary warnings:
State: degraded
…
Tainted: unmerged-bin
This is because Debian has a separate /usr/bin and /usr/sbin. There are no plans to either back off this warning by the systemd maintainers, or to patch it out by the Debian maintainers, and also no plans to merge /usr/sbin and /usr/bin.
So we’re left with the absurd situation that Debian users are being told the default clean install is ‘degraded’ and ‘tainted’ by its main init system!
If this is the case, I am surprised no one has patched this anti-feature out of the source. Both sides made that suggestion directly in the bug reports you linked. I expect it should be positively trivial to implement a patch, but someone must step up and do the work.
systemd upstream says "distros can patch it out if they want", the debian package maintainer doesn't want to carry a patch. "Someone needs to write a patch" is not the problem here.
Oops. I misread Luca’s reply as willingness to accept a patch, but he said to submit one upstream - where they explicitly don’t want it. So, yeah, deadlock.
Has anyone pushed on this issue? If a patch exists and upstream explicitly says it belongs in the distro, then I expect an explanation from the maintainer(s) would need to be presented to detail exactly why it’s either unsuitable/unmaintainable or left better as-is.
They should've never accepted systemd into Debian, or other submarines to monopolize knowledge around RH/IBM and steal attention away from original software packages (gnome, wayland, podman, hell even rpm, etc.)
It's been a very long time since I heard about uucico (Unix-to-Unix Copy-In Copy-Out program, part of the UUCP suite). Glad to see it's still being shipped! I wonder if any network uses it.
I've been running it the last year or two to get e-mail to a vintage DOS BBS that had a UUCP package. I was pleasantly surprised it was out of the box usable on both CentOS and Debian, and Postfix still ships with example UUCP email config.
like overriding it now makes a lot of sense, there needs to be grace periods etc.
but we live in a world where OSes have to become increasingly more resilient to misbehaving programs (mainly user programs, or "server programs" you can mostly isolate with services, service accounts/users etc.). And with continuous increases in both supply chain attacks and crappy AI code this will only get worse.
And as such quotas/usage limits of a temp fs being shared between all user space programs like lvm2 and dmraid is kinda a bad idea.
and for such robustness there aren't that many ways around this change, basically the alternatives are:
- make /var/lock root only and break a very small number of programs which neither use flock nor follow the XDG spec (XDG_RUNTIME_DIR is where your user scoped locks go, like e.g. for wayland or pipewire)
- change lvm2, dmraid, alsa(the low level parts) and a bunch of other things your could say are core OS components to use a different root only lock dir. Which is a lot of work and a lot of breaking changes, much more then the first approach.)
- use a "magic" virtual file system which presents a single unified view of /var/lock, but under the hood magically separates them into different tempfs with different quotas (e.g. based on used id the file gets remapped to /run/user/{uid}, except roots gets a special folder and I guess another folder for "everything else"???) That looks like a lot of complexity to support a very small number of program doing something in a very (20+ years) outdated way. But similar tricks do exist in systemd (e.g. PrivateTemp).
kinda only the first option makes sense
but it's not that it needs to be done "NOW", like in a year would be fine too, but in 5 years probably not
Why is that non-clickbait? Honestly "Debian Technical Committee overrides systemd /run/lock permission change" might be a better title than either, I don't know whether the thing or the actors are more interesting here. But you can only say so much in a title.
The "somebody might mailicuously exhaust memory via /run” argument seems silly IMHO... you can trivially put a hard limit on its size via the tmpfs mount option. I guess if you bend over backwards that's still a DOS in that new files can't be created in /run once it is full, but come on...
There is support for quotas in tmpfs! /me runs and hides under desk to avoid fruit being thrown at me
The problem is /var/lock is still used for various system root components.
Which includes services like lvm2, dmraid and audio drivers.
So you need at least a different /var/lock for "root" and every one else.
So you now can either fix all the root tooling to use a different lock folder. Which would break a lot of things.
Or you break a very small number of very old tools which (mostly) decided to neither use use flock (from 1996) nor follow the XDG spec (XDF_RUNTIME_DIR) from 2003.
So kinda obvious choice what way to go with ;), it just should be coordinated better and communicated better.
And yes you want quotas on tempfs on XDF_RUNTIME_DIR, without question. At least for a system more robust against misbehaving programs.
To be clear a lot of this security concerns have been irrelevant/ignored with thread models from the 2000th. But times (sadly) have changes and you shouldn't just blindly trust user space programs anymore. Even if we ignore malicious programs just thing about all the bugs AI will sneak in. And honestly I'm worried that Desktop Linux will fail to adopt to this changes. Server Linux is clearly adopting, but also more in the non-gnu Linux ecosystem while the more FSF/GNU parts of Linux seem mostly stuck in a past which doesn't look like it will have a future :/. This sucks without question but I just don't see a future without a lot of supply chain attack and AI coding induced crazy misbehavior of user space programs.
PS: Even if you just want a steam deck with the same degree of robustness as console (i.e. a game going rogue will have a hard time to hang the console fully not matter what it does, i.e. you can always press the menu button and close/kill it) you need this kind of subtle changes. And many other.
> Which includes services like lvm2, dmraid and audio drivers.
I know at least the first two have "ignore the lock" command line flags so you can get out of situations like /run being full. So whether it's a DOS depends on how you define DOS :)
> it just should be coordinated better and communicated better
I mean, that's the thing. I don't disagree it should be fixed. But is it really an important enough problem it justifies breaking users now? Not for me...
Letting upstream systemd single-handedly define what directories exist with what modes in your distro has never been the intended Modus Operandi.
Debian has a huge selection of packages available for it and clearly is going to have more headaches when it comes to preserving compatibility with all that software.
This is a trivial matter for Debian to handle appropriately, while systemd stays focused on its current priorities. I'm surprised this is being talked about at all outside the appropriate mailing lists... slow week for linux news?
> Is anyone surprised that the systemd folks basically said "fuck you, we'll do what we want to" to everyone else?
I don't think that's an accurate paraphrase of "Consider this more a passing of the baton from upstream systemd to downstreams: if your distro wants this kind of legacy interface, then just add this via a distro-specific tmpfiles drop-in. But there's no point really in forcing anyone who has a more forward-looking view of the world to still carry that dir."
That kind of drop-in is pretty routine, so I don't know why this became a big thing we're all discussing now.
as I will explain below this change is pretty much needed sooner or later with all alternatives being noticeable worse
In addition the way Debian handles things is in a very deep conflict to how most software development is done. Like time wise not practical viable stuff like expecting small OSS teams to have LTS releases and/or back porting fixes which had been fixed with some rewrite/breaking changes. Or fun stuff like for example shipping outdated version of your software with version of dependencies it never was supposed to work with. And in the later case not only will you get bug tickets for already fixed bugs all the time (which you don't really have time for) but als bugs which should be impossible with any "supported" version(/dependency combination). As a consequence a lot of software went from "lets make sure it works with every distro" to "lets make sure it works with everything which is at least somewhat close to upstream and everything else has to figure it out themself".
Which brings us to this discussion which wasn't communicated that well:
- due to the points below it makes sense that systemd says that is is how it is going to be upstream, not open for discussion
- but it's equally reasonable for Debian to decide to prioritize more backward compatibility over bit less robustness, or at least do so for some potentially long grace period
- and systemd really should have told them about the change before they shipped it and it broke things
- but any Debian maintainer thinking they have the right to force systemd to revert the change would be very entitled
- and any systemd dev saying Debian must not revert the change in their distro is equally entitled (it's not like this is likely to cause bugs tickets against systemd which otherwise wouldn't have happened)
so a lot of bad communication. But both the decision to the change and not care about weather Debian likes it, and the decision to revert it because they don't like it, are at the core very reasonable IMHO
---
/var/lock is shared by some very core components like lvm2 and dmraid, it running out of inodes is a serious issue
In 2000 it was okay to not overly consider misbehaving user space programs, in 2025 this is relevant security issue (due to having all of 1: higher standards for resilance/reliability, 2: more malicious code, supply chain attacks etc. 3: more accidentally badly behaving code/low quality code). So this needs to be fixed.
There are 3 solutions I can come up with:
1st make no root program use /var/lock, which is a lot of work and a lot of braking changes.
2nd make /var/lock root only, considering that user programs shouldn't really use it anymore since flock (1996), XDG_RUNTIME_DIR(2003) especially since /run/user/{uid} (still 7years ago or so). So outside of breaking some minor software using an approach outdated by like 20years no issue.
3rd some crazy magical virtual file system proxy mapping entries to different temp fs instances depending on owner user id. But is that worth to have to maintain that blob of complexity which can have security vulnerability for a handful of very old software?
Only really the solution systemd did makes sense. Debian probably would love the 3rd one IFF systemd maintains it for them, so it's not really an option (Debian could still write and maintain it themself, it doesn't need any deep systemd integrations.)
So realistically speaking there is only one solution, if you don't propose a better one just saying "I don't like that, don't do that" is not going to change anything.
And systemd developers are payed by companies which need that stuff, actually most companies would want this kind of in depth increases to robustness. Because a single time of a server not hanging can easily outweighs any cost imposed by read only /var/lock (which most times will be no cost at all). Heck even Desktop systems would profit from it . And for the few systems where companies need the old /var/lock they can just change it like Debian did.
"everyone else" in this case is pretty much only the debian ecosystem because they insist on enforcing a serial lock policy from the 1980s. It's fine if Debian wants to move at the speed of a Soviet committee but I don't think it should be expected (or would be healthy) for systemd to move at the same pace.
A software developer's primary job is to develop software for their users, not to comply with a third party distributor that repackages their software.
The beef isn't with systemd upstream which already has a very simple/boring workaround for this, it's with the debian package maintainer (some people here are wearing multiple hats).
Really the whole raison d'etre of debian is move at this pace to prioritize stability/compatibility. If you don't like that philosophy there are other distros but a package maintainer's primary job is to repackage software for that distro (which presumably users have chosen for a reason), not comply with upstream.
Somehow I feel that if all the time that has been invested in debating and discussing this had been spent on patching the affected apps, the problem would be properly solved.
I mean, yeah, I get it, systemd bad, democracy good, but these world-writable lock folders are actually a huge pain, and adding some shim code to upgrade to a more secure solution seems achievable?
Genuinely curious - why would world-writeable directory be bad for security? Assuming of course, it's on a separate filesystem mounted with sensible options. Here's what I see from "grep /run/lock /proc/mounts" in sid:
The classic is say you know a root process will write a file called foo.lock in /run/lock, and you (a bad person) have write access to that directory. Then you make foo.lock a symlink to some file (/bin/init or /bin/sh or ld.so for example would be very inconvenient choices) and when the root process writes its lock it destroys that file.
Now obviously people these days generally know about that so hopefully don’t use predictable file names but that’s one way.
Yup to both of you. But all of this is to say, running shellscripts as root (in particular) needs to be done with extreme care, because if people forget those precautions when writing C, they sure as heck don’t trouble themselves to do it when they’re writing shell.
I remember the time (around 2001-2002) when just about every binary was discovered to have some variant on this exact exploit. I happened to be linux sysadmin for a very large, high-profile set of linux boxes at the time. Happy times.
Hmm - I see there's now "lockdev" for managing access to things like serial lines, but what's the preferred method of expressing "only one instance of this program should run at any one time"?
Considering I never had to reinstall a Debian system because it got bloated or broke one day, I can accept this slow-cooking approach. Even support them on this regard.
Gentoo Linux has quite a different approach than Debian [0] but after the first month or two (once my new-to-Linux ass figured out what it was doing) I've never had to reinstall any Gentoo system. [1]
If you want what Debian provides, it's a poor choice for you... but -IME- it doesn't break on upgrade, unlike some Debian-derived distros I've tried in the past.
[0] Something along the lines of "Always try to package exactly what's provided by upstream, try hard to get distro patches upstreamed, and try to have the latest available upstream release in the 'testing channel'.".
[1] Well, I do have a machine that (aside from "side-loading" kernel updates from time to time) hasn't been updated in four years. While I'll try to update that one in the normal way, I'm probably going to need to reinstall.
I use GNU screen for that. Looking at cu, it looks to be just as tiny and has ssh-like tilda-escapes, including "~." to disconnect. Nice, gotta try it out, thanks!
What would you suggest? Personally I find cu the closest to ssh in ergonomics to any of the tui serial terminal programs, and it’s easily available across unixes like FreeBSD or Mac OS, which is a bonus.
This part seems fairly editorial:
>Debian Policy still cites the FHS, even though the FHS has gone unmaintained for more than a decade.
What ongoing maintenance would a file system standard require? A successful standard of that type would have to remain static unless there was a serious issue to address. Regular changes are what the standard was intended to combat in the first place.
>The specification was not so much finished as abandoned after FHS 3.0 was released...
OK.
>...though there is a slow-moving effort to revive and revise the standard as FHS 4.0, it has not yet produced any results.
So it is not abandoned then. A slow moving process is exactly what you would want for the maintenance of a file system standard.
>Meanwhile, in the absence of a current standard, systemd has spun off its file-hierarchy documentation to the Linux Userspace API (UAPI) Group as a specification. LWN covered that development in August, related to Fedora's search for an FHS successor.
Ah. Systemd/Fedora want a standard that they can directly control without interference from others.
Author here: It was abandoned. I linked to one of the former maintainers who said as much. The current effort is by a few people who asked the LF to take out over, and have (so far) done little after an initial flurry of activity. That, too, is covered in the other article I wrote about the FHS recently.
Prior to the group who started an update effort, it had not been touched in about a decade. That’s not slow-moving: that’s abandoned.
The FHS ultimately belongs to the users collectively, not those maintaining it. I am old enough to remember the horror that existed before the influence of the FHS. It exists in the fact that it is to some extent respected, not because there is a file somewhere that says it is the FHS standard. If you want changes, then sure, do the politics required to develop support for those changes. You can't just declare a new standard and then do whatever you want.
Developers have this thing where they will think of a standard as a specification. Instead it is a statement of political will. Saying that a standard is "abandoned" due to lack of "maintenance" seems like an example of thinking of a standard as the instantation of a specification; an actual program.
I know it's not the same, but imagine thinking a law is not longer meant to be followed because it hasn't been updated in 10 years.
I agree--given your contraints of law and 10 years. But what about a law that hasn't been updated for 150 years? There's plenty of those that we regularly ignore.
What's the timeline for software?
> what about a law that hasn't been updated for 150 years? There's plenty of those that we regularly ignore.
There might be minor alterations to details, but the core laws are mostly older than that. Murder, theft, etc don't change that much.
Even the silly confusing ones have a long life. E.g. "Rule against perpetuities"
> Even the silly confusing ones have a long life. E.g. "Rule against perpetuities"
That wouldn't be my go-to example of a silly law. It's what prevents control of property from remaining permanently with the will of a dead person who managed to own the property outright. It says that, at some point, the will can have no more influence and full ownership vests in someone who's alive.
Much of the US/UK legal system is based on common-law rules that are several hundred years old. In some cases those old laws have been codified, in some cases not, but either way there's no need to drop them just because they're old. On the contrary, laws that have stood that long without needing to be changed have demonstrated that they are extraordinarily good ideas.
I’m not sure point to the UK is a good example. There are plenty of weird and obscure laws that simply aren’t enforced or followed anymore. Everything from laws about handling salmon suspiciously, through to various right around who can drive sheep across Tower Bridge.
Those laws survive not because anyone considers them a good idea, but simply because the issues caused by ignoring them are substantially smaller than the effort involved in removing them.
We also have a bunch of laws that are still followed, but only in the most technical sense. Every “Parliament route” train schedule falls into that category. Train services that must be provided at least once a day, sometimes only once a week, which nobody actually uses, and in some cases only travel to stations with no practical public entrances. Those laws don’t survive because anyone things they’re a good idea, it’s just easier to run the train, than it is to get parliament time to abolish the law.
There is no automatic, fixed timeframe after which a law simply stops being followed because it hasn't been updated or looked at; and remember, we're still applying the FHS, it's in active use even if it's not updated.
Laws remain in force until they are formally:
* Repealed (abolished) by the relevant legislative body (Parliament, Congress, etc.).
* Struck down by a court as unconstitutional or otherwise invalid.
A 150 year "delete" timer would genuinely undermine the foundation of the legal system. Lawyers, judges, and businesses rely on the continuity of core laws (e.g., contract, property, and tax law). If a 150-year-old property law suddenly lapsed, it could instantly void millions of land titles and commercial contracts...
> Laws remain in force until they are formally: * Struck down by a court as unconstitutional or otherwise invalid.
False. They are still in force - they have just become unenforceable. There's a crucial difference, as the US is currently finding out: as long as they are in the books, a Supreme Court decision can instantly render them enforceable again - even against the wishes of the population.
The proper thing to do would be to "garbage collect" unenforceable laws, but politicians are (understandably) hesitant to spend political capital on it when it doesn't provide any tangible return.
There are other reasons as well. The body responsible for enforcing a particular law can choose not to enforce it, thereby rendering the law useless. Or a law can become obsolete by changes in technology or society - the original law legislates something that just doesn't happen any more, say. Laws can also be written to handle a specific event that only occurs once. Once that event has passed, the law might as well not exist. It doesn't need to be repealed because it just doesn't apply any more.
In addition, laws are typically regularly amended to handle new societal developments, to clarify wording, or to fit better with other laws or changes in attitudes. A law that has gone 150 years without being amended at all is probably a law that falls into the categories above and is obsolete.
Of course, all this is getting somewhat off-topic, but the point is that laws absolutely can become outdated and unmaintained, either deliberately or by happenstance. And the inverse is also true: most laws that people deal with regularly are kept up-to-date to ensure that they still reflect the needs and wills of the society they're being used in.
There are plenty of 150-year-old laws that we don't ignore, too.
How often does "thou shall not kill" need an update?
Not even in defense of your own life, family, others? You have a lot of people on HN that celebrate killing of then innocent.
Obviously the intent was "Thou shall not murder anyone"... Interpreting it otherwise doesn't make sense, and is inconsistent with the rest of the Bible.
The definition of "murder" is "unlawful killing", so you've reduced it to "unlawful killing is against the law" - which is meaningless.
That's the wrong way to look at it. The Bible does not refer to other laws as a source of authority. Saying murder is "just" illegal killing is the real meaningless statement. Clearly people in the Bible are allowed to fight and defend themselves. Murder is typically intentional and unnecessary killing of someone else for malicious reasons (or no reason at all). Malicious reasons would typically be spite, greed, or convenience.
> The Bible does not refer to other laws as a source of authority
The Bible is the source of law here. My point was that interpreting it solely in the way you deem "obvious" does not work: you cannot have "thou shalt not murder" on its own without additional rules clarifying what counts as "murder" and what counts as "lawful killing" - and the Bible contains plenty of those.
> Murder is typically intentional and unnecessary killing of someone else for malicious reasons (or no reason at all). Malicious reasons would typically be spite, greed, or convenience.
That's how you interpret it. Modern law allows for killing out of greed - if the soldier firing the bullet is different from the politician wanting to capture some resources. We allow countries to kill out of spite with retaliatory strikes. We allow cops to kill in self-defense - even when other methods to subdue are theoretically available, but inconvenient. On the other hand, we no longer allow stoning to death people violating the Sabbath.
Clearly, there is a nontrivial list of criteria separating murder from lawful killing, and this list is mutable. In practice this list is codified in the law, which means murder becomes "killing which is not otherwise allowed in the law", which is the point I was trying to make.
Looping back to the original discussion: contrary to what ikiris was originally claiming it is not "thou shalt not kill" but "thou shalt not murder", and we've been updating the definition of "murder" (and by extent the meaning of "thou shalt not murder") quite a lot over the last few thousand years, so it is false to claim that " 'thou shalt not kill' never needs an update ".
>The Bible is the source of law here. My point was that interpreting it solely in the way you deem "obvious" does not work: you cannot have "thou shalt not murder" on its own without additional rules clarifying what counts as "murder" and what counts as "lawful killing" - and the Bible contains plenty of those.
It is obvious to people who know the Bible lol. It may have lots of contradictions but this isn't one of them. Murder was understood in a particular way to these ancient people, that still applies to us today.
>That's how you interpret it. Modern law allows for killing out of greed - if the soldier firing the bullet is different from the politician wanting to capture some resources.
As I said, I am not referring to laws of any country. The laws of modern countries are irrelevant to interpretation of the Bible. The dictionary does not count either.
>Looping back to the original discussion: contrary to what ikiris was originally claiming it is not "thou shalt not kill" but "thou shalt not murder", and we've been updating the definition of "murder" (and by extent the meaning of "thou shalt not murder") quite a lot over the last few thousand years, so it is false to claim that " 'thou shalt not kill' never needs an update ".
The only thing that needs an update is the translation. The meaning is very clear and universal. Don't kill people except in self-defense. It is just as antisocial today as it was thousands of years ago.
Old Hebrew didn't had the same word for killing and murder, so your whole discussion is based on a translation decision.
Or when God tells you to.
"Now go, attack the Amalekites and totally destroy[a] all that belongs to them. Do not spare them; put to death men and women, children and infants, cattle and sheep, camels and donkeys.’”
"and when the Lord your God has delivered them over to you and you have defeated them, then you must destroy them totally.[a] Make no treaty with them, and show them no mercy."
etc
Lol yes... I've heard some alternative explanations of this. The Ten Commandments are not recognized outside of Christianity. They are just some important general sounding rules in a book that is full of rules for Jews, who get into battles with other people at times. Many of those rules in the same book have prescribed punishments of stoning to death, as well. So clearly, killing is at least allowed for purposes of punishment and warfare. Common sense also tells us that no religion can realistically prohibit self-defense.
jesus. what did the Amalekites do?
Without looking it up classic reasons are almost always thin veils over they have things that could enrich us.
Not really: 1 Samuel 15 starts like this:
"Samuel said to Saul, “I am the one the Lord sent to anoint you king over his people Israel; so listen now to the message from the Lord. 2 This is what the Lord Almighty says: ‘I will punish the Amalekites for what they did to Israel when they waylaid them as they came up from Egypt. 3 Now go, attack the Amalekites and totally destroy[a] all that belongs to them. Do not spare them; put to death men and women, children and infants, cattle and sheep, camels and donkeys.’”"
Later on it says that he was not happy with the outcome because they didn't kill all the livestock and people... Of course this is a work of mythology, but the message clearly isn't that the Israelites should go and loot from these people.
"Obviously the intent was" probably not the same as it obviously was 150 years ago!
It's still obvious, but in context. It isn't a recipe or street sign lol...
The US constitution is still in force after 236 years, and even older laws are still enforced. US courts will sometimes look at precedent from England before the colonies existed.
Meanwhile some laws that are months old are ignored by law enforcement because nothing forces them to read it. It’s that effect which is why so many old laws are ignored rather than formally repealed. When nobody is ridding a horse nobody cares how you need to tie one up when visiting a store etc.
> The US constitution is still in force after 236 years
True, but it's been updated a lot more recently than that.
The last update was still much longer ago than 10 years, of course. The most recently ratified amendment to the Constitution - the Twenty-Seventh Amendment, ratified 1992 - was, incredibly enough, proposed in 1789 along with the ten we know as the Bill of Rights and another one which was never ratified. And of the twenty-seven amendments ratified so far, the one most recently proposed by Congress, the Twenty-Sixth Amendment, was both proposed and ratified in 1971.
Are you suggesting that appending the constitution in 1992 with: No law, varying the compensation for the services of the Senators and Representatives, shall take effect, until an election of Representatives shall have intervened.
Somehow has an impact on anything else? Because by that standard every change to any law updates all existing laws that were not changed. Or I’m just completely misunderstanding your point here.
My point is that merely referencing the centuries-old original age of a document is misleading when it’s been updated dozens of times ending just a few decades ago. (And many of the updates, including the one in 1971, have been far more impactful than the Twenty-Seventh Amendment which you quoted.)
It’s certainly true that the constitution is old and crusty overall and desperately needs an overhaul, but the discussion was about when old laws which haven’t been updated in a while are ignored or enforced.
The constitution is indeed one law, not several different laws, and it’s been updated far more recently than its original year of promulgation or ratification. And it’s still mostly enforced (with increasing exceptions but that’s another discussion entirely).
> been updated dozens of times
18 times. 27 total amendments with 1-10 all passing on December 15, 1791.
> a few decades ago
There hasn’t been a meaningful change in over 54 years.
> The constitution is indeed one law, not several different laws
Those recent amendments are a minimum different laws. If you want to call it one law then there’s either 2 federal laws in the US. One needs to be ratified by the states and the others don’t.
> No 18 times. 27 total amendments with 1-10 all passing on December 15, 1791.
Fair correction.
> Those recent amendments are a minimum different laws.
Only in the sense that any new enactment is a new law, but in that sense no law can be updated in a way that preserves its identity as the same law. Which is not useful for the discussion of whether an old law has or hasn’t been updated, since by that definition updates to any law are impossible.
Lawyers and judges don’t exclusively use that sense any more than programmers view a software patch as changing the basic identity (beyond something like a version number) of the patched software program.
They do use that sense when they are referring to individual enactments by Congress, just like programmers refer to patches and patch releases as well as to the things which exist across many patch( release)s over time. Which sense is useful depends on context, and which sense is meant depends on a mixture of context and exact phrasing.
> If you want to call it one law then there’s either 2 federal laws in the US.
There are far more than 2 federal laws in the US, but far fewer than one per enactment except when using the sense of “a federal law” that specifically refers to each enactment.
For example, most federal tax legislation explicitly amends the Internal Revenue Code of 1986, which is still the official name of our federal tax code. Similarly, most immigration legislation amends the Immigration and Nationality Act of 1954 (I could have the year wrong), even more than 70 years later. The many “patch” laws enacted in the meantime all have their own identities via Public Law numbers and often a name, but they do update a specific identifiable underlying law without replacing it.
Other federal laws, like most annual appropriation and authorization bills, stand alone and are not routinely updated but have a finite duration of relevance. Common provisions are often carried forward from one iteration to the next, but they are re-enacted as part of each separate iteration.
And then there are the many parts of the federal legal landscape where the US Code is the official authoritative version instead of a mere convenience version as it is for things like the tax code and immigration law. Amendments to the directly authoritative parts of the US Code explicitly amend the US Code instead of a separately named law, so those directly authoritative parts of the US Code are themselves (whether each or collectively) a single federal law in the sense I’m discussing.
Yes, this stuff is complicated and messy in both the law and the software worlds.
> One needs to be ratified by the states and the others don’t.
Yes, constitutional amendments are laws (in one sense) that amend a law (in the other sense) and which states need to ratify, and regular Acts of Congress are laws (in the first sense) that may or may not amend one or more pre-existing laws (in the second sense) and which states do not need to ratify.
> Yes, constitutional amendments are laws (in one sense) that amend a law (in the other sense) and which states need to ratify, and regular Acts of Congress are laws (in the first sense) that may or may not amend one or more pre-existing laws (in the second sense) and which states do not need to ratify.
Yea, obviously we agree with what’s going on this is just a question of arbitrary definitions that don’t impact anything.
> Which is not useful for the discussion of whether an old law has or hasn’t been updated, since by that definition updates to any law are impossible.
It’s definitely easier work with the law based on the provided organizational structures with tax code being separate from family law etc. Yay, congress is doing something reasonably efficiently.
However, timing matters as making something illegal ex post facto is explicitly banned by the constitution etc. Further, in case of conflict newer laws win even without explicitly declaring the old laws invalid. So each bill is a meaningfully different law, and there’s in effect one law at any given moment after resolving those conflicts.
Net result three mutually contradictory but still useful definitions. But IMO organizational structure is by far the least meaningful one from a legal standpoint while being the most useful one from a practical standpoint.
You mean like those local laws that say you can’t walk a cow backwards through the main streets? Or laws that say a motor vehicle must be preceded by a lamp carrier.
yup, but no standard is a law.
law on its own can mandate the use of a specific standard, but a standard on its own is no law.
so much so that often doing non-standard stuff is the most successful route. dumb example: Apple and all of it proprietary, non standard stuff.
> The FHS ultimately belongs to the users collectively, not those maintaining it.
I completely agree that regular updates are not a requirement for standards to remain relevant, but it does require the ecosystem to still adhere to them - and the problem is that Linux users are increasingly deviating from the FHS.
The FHS does not accurately describe the situation on-the-ground, there are no plans to update the FHS to accurately describe the situation on-the-ground, and there are no plans to update the ecosystem to accurately implement the FHS.
Like it or not: the FHS is dead, and nobody seems interested in reviving it.
Huh, most programs do use it, and if they don't that's a bug?
I still don't think you adequately explained why that would matter. To reiterate the question OP asked: what updates does it need that it hasn't been getting? It isn't as if one would expect a "put X stuff in y location" document to need maintenance.
The FHS 3.0 doesn’t reflect current practices, such as the /usr merge, nor the /sys directory. There’s other ways it’s either no longer followed or missing developments from the last 10 years.
>> Debian Policy still cites the FHS, even though the FHS has gone unmaintained for more than a decade.
> What ongoing maintenance would a file system standard require? A successful standard of that type would have to remain static unless there was a serious issue to address. Regular changes are what the standard was intended to combat in the first place.
It's 2025, anything that wants to be considered modern (and everything should want that), needs to be undergoing constant change and delivering regular "improvements."
>>...though there is a slow-moving effort to revive and revise the standard as FHS 4.0, it has not yet produced any results.
> So it is not abandoned then. A slow moving process is exactly what you would want for the maintenance of a file system standard.
The FHS people to get off their butts. There's no excuse for that pace now that we have such well-developed AI assistants. They should be pushing quarterly updates at a minimum, and a breaking change at least every year or two. It's been obvious for decades that "etc" is in urgent need of renaming to "config", "home" to "user", and "usr" to "Program Files" to keep up with modern UX trends.
I genuinely can't tell if this is satire.
I thought it would have been obvious by the "Program Files" at the end :).
Anyway, Linux community as a whole has an antiquated development process, and needs to modernize and follow the best practices of an industry-leading trend-setter, like MS Teams.
Surely Linux should be developed Google style, with a Web Scale perspective. uucp, tar, yacc, roff removed (with a 30 day notice, of course!), all the uses of "creat" amended to add the final "e", etc.
And systemd already follows Microsoft best practices, such as "Fire and Motion" https://www.joelonsoftware.com/2002/01/06/fire-and-motion/
The X11 people are trying this hard. I'm really curious how Wayland will evolve but, the history of GTK and QT does not give me much hope.
Wait you were kidding? I've already spun up a kubes cluster so we can feed FHS in a globally load-balanced CI/CD pipeline, I have an agentic LLM doing constant improvements so we can sprint to you never knowing where your files are!
It's like ASLR for files but no maps because maps aren't for trailblazers, they make the maps! It's very cutting edge and a value-add!
(Obligatory /s)
Ok, so it's only half satire, or is this reply also a satire? I mean, MS Teams, really?
(Yes, the reply was also satire)
Counterpoint: Modern distro’s needs have evolved past the FHS in some cases, and everybody deviates from it slightly but incompatibly.
A standard does no good if it does not reflect reality. I think it is a worthwhile effort to try to bring it back in line with actual real world usage.
laughs in NixOS
it's remarkable to me that NixOS manages to run so well despite breaking the FHS so thoroughly. and not just in superficial ways like not calling it /bin, I mean forsaking dynamic linking (hence /var/lib and /usr/lib), keeping man pages, resources and config bundled into the same derivation as the binary sometimes, and occasionally hacking up binary blobs to rewrite rpaths.
on the other hand, there's a place for legacy distros too.
An interesting take in the context of systemd's deviation from it being the source of bugs because of real world usage that conflicts with systemd's changes.
The /usr merge is an example of something that a modern FHS might reflect.
OS X doesn't have a merged /usr. On my Mac I see /bin as a separate directory, not a symlink into `/usr`. Does Linux have a compelling reason that OS X lacks?
The reason for having a separate /usr has long slid into obsolescence. Our storage is no longer constrained to require us to mount a remote partition which holds most of the binaries and other sundry required to boot a distribution.
I'm assuming you're referring to partitioning of boot-critical binaries into `/bin`, but "the reason for having a separate /usr" is even older and worse than that. In original Unix `/usr` was for home dirs[1], and was colonized by the operating system in 1971 when it no longer fit on a single 1.5MB RK05 disk. Nobody ever untangled that change and we've been living with the hack ever since.
[1] https://lists.busybox.net/pipermail/busybox/2010-December/07...
In other words, the strongest argument for usrmerge is that there is no compelling argument against it?
I think the current strongest argument against it is that systemd complains when /usr is on a separate partition[1], and what its devs have weighed in on the matter[2].
[1] https://freedesktop.org/wiki/Software/systemd/separate-usr-i...
[2] https://www.freedesktop.org/wiki/Software/systemd/TheCaseFor...
sysvinit had no problem being told to mount /usr as soon as network was available, and if you set up an init script to run before /usr was available, but the script needed /usr, that was your own fault.
systemd relies on things in /usr being available, including to decide which scripts to run, and mounting /usr would be one of those scripts, so it has a chicken-and-egg problem.
But ah, it doesn't! Instead the world needs to make sure /usr is mounted before systemd even gets started, so systemd doesn't have to fix its bug.
Personally, I don't mind /usr/bin merging with /bin, the benefit I can see is no more squabbling over whether something should be in /bin or not (i.e. is this tool needed to boot the system, or not?)
> sysvinit had no problem being told to mount /usr [..] if you set up an init script to run before /usr was available,
> the world needs to make sure /usr is mounted before systemd even gets started, so systemd doesn't have to fix its bug.
Unironically in the same post despite being, to my untrained eye, the same thing.
The difference being that the authors of sysvinit didn't advertise obnoxious messages at boot time (https://systemd.io/SEPARATE_USR_IS_BROKEN/) and try to get the filesystem standards changed.
One is like "I'll run some scripts in order, everything else is on you", the other is like "I'll take care of everything, I'll do that, WHAT YOU DIDN'T MOUNT /USR ? SHAME ON YOU I DON'T WANT TO DEAL WITH THAT CORNER-CASE"
I read that message as "Whoever is the poor soul attempting to fix this system that doesn't boot, (because who otherwise inspects early boot messages?) whoever installed this system is running a known-fragile configuration."
And that's what I expect of systemd? That it should complain loudly whenever me, the daemons I'm attempting to run, or the overall system is doing things in a weird, known-bad, known-fragile way and warn me about it before it breaks if possible.
systemd is my hammer having it's own opinions about what nails I hit & where & into what & why.
I want a nail only driven half in and at some crooked angle, that's my business.
It's not my hammers job to agree or disagree that it's a bad nail hammering job as far as it knows. I don't wantto have to convince it of the validity of a use-case it didn't think of before, or thought of and decided it doesn't agree to support.
I just want that crude coat hanger and I don't care who else likes it or doesn't like it or who else thinks I should buy an actual coat hanger and attach it in some way that someone else approves of.
> systemd is my hammer having it's own opinions about what nails I hit & where & into what & why.
I feel like the exact opposite. systemd has way too many layers of customization. Things can be in /etc/systemd/system/, /run/systemd/system/, /usr/lib/systemd/system/* and those are just the "basic" override locations. And those edits and masks usually don't clash with each other, so your customizations and quirks stick, even if the rest of your system changes.
Now, if you are in a system that doesn't use systemd, if you mess with the "sacred and sanctioned" system service summoning scrolls, now you're on your own to diff whoever stuff the maintainer did, every time the summoning scroll changes, and to make sure that probably Turing-complete mess still summons a valid, working and safe system component instead of rampaging thru your system.
> Things can be in /etc/systemd/system/, /run/systemd/system/, /usr/lib/systemd/system/* and those are just the "basic" override locations. And those edits and masks usually don't clash with each other, so your customizations and quirks stick, even if the rest of your system changes.
I really encourage you to learn the difference between these three places. They are semantically distinct, and it's truly an advantage to have the system distinguish them. They're all sources of configuration, of course. But one is managed by the OS vendor, one by the sysadmin, and one by the sysadmin at runtime. Runtime isn't as significant a distinction, but having a clear, bright line between vendor-provided and sysadmin-managed config is huge.
Systemd is what happens when a software engineer says "What you have here is not a hammer but a Wood Fastening Device Driver..."
We used to throw such people into moats.
https://www-users.york.ac.uk/~ss44/joke/toast.htm
I don't want my hammer to have a razorblade embedded in the handle so if I grab it "wrong" my hand gets sliced up.
Look analogies work both ways!
Poettering would assert that the razorblade is a part of their vision and systemd only supports modern, razorblade resistant hands anyways.
There is no analogy that worked both ways here. Some fitting counter-analogy might exist, but this isn't it, because there are no razor blade analogs in the handle analog of init.
I used hammer as the example because init is actually like a hammer. It is a dead simple impliment with limited scope and knowable, known, and absolutely predictable behaviors and properties. It doesn't actually do much of anything, which makes it infinitely flexible. By not having any will of it's own, it allows you to do more than a more complex, automatic, and integrated tool.
It can't suddenly razor blade you because it doesn't have razor blades or any other special features in the first place.
YOU are the complex integrator. And if you are not, I have no sympathy for you or whatever difficulty you have with responsibility.
You can still have complex automation and management, built on top of simpler lower level agnostic layers and tools. There was never any requirement to remove that flexibility and elegant future-proof design in order to have all the things systemd promises (and really, merely promises, it does not actually deliver any better than anything else).
Systemd is also a grand scope example of tight coupling. It's so bizarre how these supposed genius coders can commit such a huge example of something they all know to avoid like the plague in just a slightly different context.
In general, systemd-ish projects expect you to bend the system to match the project's expectation while sysvinit-ish projects are infinitely flexible to match any system (jack of all trades, master of none; worse is better; etc).
From the creators of systemd we also have GNOME, PulseAudio, and Wayland. They have some design philosophy in common.
BTW most sysvinit distros barely even use sysvinit. sysvinit is a service monitor, similar to systemd but more primitive, but typically most of what it's configured to do is to launch some shell scripts on startup. We really have "systemd distros" and "ad-hoc script distros", not sysvinit distros ("ad-hoc" is not a pejorative). I don't know why they don't make init a shell script directly - you can do that, and it's typically done that way in initramfs.
> In general, systemd-ish projects expect you to bend the system to match the project's expectation while sysvinit-ish projects are infinitely flexible to match any system (jack of all trades, master of none; worse is better; etc).
Name one thing you can do in sysvinit (or "ad-hoc script"-init) that you cannot do in systemd. With systemd, you're still entirely free to write your own artisinal shell spaghetti and jam it in alongside the otherwise clean unit structure.
name one thing you can do in systemd that you can't do in assembly code
Assuming that your comment is in good faith, that comparison doesn't work in the way I think you're trying to make it work.
systemd operates (when used normally) at a higher level of abstraction than a bucket-of-scripts/sysvinit. Especially because systemd units are declarative, this higher level of abstraction affords you (the sysadmin) the ability to more precisely specify the behavior you want. However, you can of course run scripts from systemd, so you've got whatever escape hatches you like. My question was asking if there was anything in sysvinit that those escape hatches didn't cover adequately.
Assembly, on the other hand, operates at a far lower level of abstraction. Inappropriately low, in fact. Because of this, you actually have less practical ability to specify the precise behaviors you want. For example, you'd be hard-pressed to use assembly to start a webserver after the network is up but before some other piece of software that depends on the webserver.
no, it is that it adds complexity which is no longer needed
especially for image based stuff it's a pain
which includes OCI images for things like docker
but also image based distros like e.g. ostree (as used through rpm-ostree by Atomic Fedora desktops like Fedora Silverblue, but also in similar but different forms something Ubuntu has been experimenting with)
if you're building containers with full systemd instead of the pid0 shim you're doing systemd wrong
did you accidentally post on the wrong comment?
Your comment seems fully unrelated to my point that overlaying images is much more a pain if the things you might want to shadow and/or extend are distributed or even duplicated across many different places when they could be just be in one place.
On the contrary. While the issue is not space, and hasn't been for a long time, a lot of devices are "network/cloud-first" and it is modern to load programs over the internet. It seams all the more useful, to have the distinction between core binaries, that are needed to boot, connect to the network and should be there in case the network setup became toast for debugging, and programs that can be loaded from the network on demand.
it makes things more complicated without any benefits (at least not anymore)
this doesn't matter for OS X which main changes mostly tend to be diverging away from it's roots into a fully proprietary direction
but it does matter if you build image based Linux distros which might be the future of Linux
Merged /usr is one (increasingly accepted?) means of implementing image-based distros. New OS version, new /usr parition.
macOS didn't merge /usr, but it did do something sorta related.
One of the purposes of usrmerge is to cleanly separate the read-only and read-write parts of the system. This helps with image-based distros, where /usr can be on its own read-only filesystem, and related use cases such as [1]. Usrmerge is not required for image-based distros to work [2], but it makes things cleaner.
macOS, starting in 2019, is also an 'image-based distro', in that it has a read-only filesystem for system files and a separate read-write filesystem for user data. However, the read-only filesystem is mounted at / instead of /usr. Several different paths under the root need to be writable [3], which is implemented by having a single read-write filesystem (/System/Volumes/Data) plus a number of "firmlinks" from paths in the read-only filesystem to corresponding paths in the read-write filesystem. Firmlinks are a bespoke kernel feature invented for this purpose.
Both approaches have their advantages and disadvantages. The macOS approach is nice in that the system filesystem contains _all_ read-only files/directories, whereas under "distro in /usr" scheme, you need a separate tmpfs at / to contain the mount points and the symlinks into /usr. But "distro in /usr" has the advantage of making the separation between read-only and read-write files simpler and more visible to the user. Relatedly, macOS's scheme has the disadvantage that every writable file has two separate paths, one with /System/Volumes/Data and one without. But "distro in /usr" has the opposite disadvantage, in that a lot of read-only files have two separate paths, one with /usr and one without. Finally, macOS's scheme has the disadvantage that it required inventing and using firmlinks. Linux can already achieve similar effects using bind mounts or overlayfs, but those have minor disadvantages (bind mounts are more annoying to set up and tear down; overlayfs has a bit of performance overhead). Actual firmlinks are not necessarily any better, though, since they don't have a clear story for being shared between containers (which macOS does not support). It is nice that "distro in /usr" doesn't require any such complexity.
Ultimately, the constraints and motivations on both sides are quite different. macOS couldn't have gotten everything read-only under one directory as easily because it has /System in addition to /usr. macOS doesn't have containers. macOS doesn't have different distros with different filesystem layouts and deployment mechanisms. And philosophically, for all that people accuse systemd of departing from Unix design principles, systemd seems to see itself as evolving the Unix design, whereas macOS tends to treat Unix like some legacy thing. It's no surprise that systemd would try to improve on Unix with things like "/bin points to /usr/bin" while macOS would leave the Unix bits as-is.
[1] https://lwn.net/Articles/890463/ [2] https://blog.verbum.org/2024/10/22/why-bootc-doesnt-require-... [3] https://eclecticlight.co/2023/07/22/how-macos-depends-on-fir...
I'm very happy that the "independent standard" facade of the UAPI group fell, and its actions are now directly attributed to systemd's interests.
> What ongoing maintenance would a file system standard require?
adaption to _a lot_ of subtle changes to requirements
- very different security related requirements today
- very different performance related requirements/characteristics
- very different need for various edge cases
and lastly adapt based on what turned out to work well and what didn't
so some examples not already mentioned in the article
- /boot -- dead or at least differently used if you use efistub booting
- /etc/X11 -- half dead on wayland
- /etc/xml, /etc/sgml -- dead, should IMHO never have existed
- also why was /etc/{X11,xml,sgml} every explicit part of the standard when the spec for `/etc` already implies them as long as e.g X11 is used ??
- `/media` -- dead/half dead depending on distro, replaced by `/run/media/{username}/{mount}`
- `/sbin` -- "controversial"; frequent reoccurring discussions that it isn't needed anymore, didn't work out as intended etc. It was useful for very old style thin clients as `/sbin` was in storage but `/bin` was mounted. And there are still some edge cases where it can makes sense today but most fall under "workaround for a different kind of problem which is better fixed properly".
- `/tmp` -- "controversial", long history of security issues, `/tmp` dir per program fixes the security issues (e.g. systemd service PrivateTmp option) but requires having a concept of "programs" instead of just "running processes" (e.g. by systemd services or flatpack programs). Also `tmpfiles.d` can help here.
- `/usr/libexec` -- dead, nice idea but introduces unneeded complexity and can be very misleading in combination swith suid and similar
- `/usr/sbin` see `/sbin`
- `/usr/share/{color,dict,man,misc,ppd,sgml,xml}` -- should never have been in the standard they are implied by the definition of `/usr/share`; at least sqml,xml are dead. dict was for spell check/auto completion, except that neither works anymore like dict expects
- `/var/account` -- to specific to some subset of partially dead programs, shouldn't be in the standard
- `/var/crash` -- distro specific mess
- `/var/games` -- basically dead/security mess, I mean 99% of games today are user per-user installed (e.g. Steam) and even for such which are packed any variable download data is per user, making it shared creates a permission/security mess
- `/var/lock` -- as mentioned there are better technical solutions by now, e.g. using `flock` instead of "presence of file" and some other techniques. Tend to also avoid issues of crashed programs not cleaning up "lock files" leading to dead locks and needing manual intervention.
- `/var/mail` assumes a quite outdated form of managing mail which is quite specific to the mailing program, as it's very program specific it IMHO shouldn't be in the standard
- various legacy program specific, non "generic" file system requirements e.g. that `/usr/lib/sendmail` must exist and be a link to a sendmail compatible program and similar.
also missing parts:
- `/run/user/{uid}`
- `/var/run/user/{uid}`
- `/proc`
- `/sys`
- user side versions (e.g. from the XDG spec which is also somewhat in a zombie state from my personal experience with it , e.g. .config, .local/{bin,share})
- references to light weight sandboxing, e.g. per-program /temp etc.
- factory reset stuff (`/usr/share/factory`) needed for having a uniform way for devices sold with Linux and device specific distro customization(e.g. steam deck)
so yes, it's quite outdated
> `/usr/libexec` -- dead,
Definitely not dead, the XDG portals and Polkit agents live here.
depends on the Distro,
but yes not dead but more like unnecessary added complexity without any benefit for modern Linux systems
"Programs in a standard location, but not in the path" isn't something you need anymore?
Well, it probably depends on which software's concern will be implementing a policy to prevent users from having permission to fill critical directories and prevent the system from operating normally, which is discussed in the article. Which is also a coordination problem because the most common user of disk is software itself, I think.
FHS seems to specifically imbue the user with the responsibility and consequences of filling up the disk.
A more neutral phrasing would be.
> Debian Policy still cites the FHS, and FHS has remained static for over a decade.
> The FHS 3.0 is clearly reaching the end of its useful life, if not actually expired.
Interesting take.
I think that the FHS is still extremely helpful for packagers, sysadmins and others so they won't stomp on each other's feet constantly. It helps set expectations and prevents unnecessary surprises.
Just the fact that one particular FHS rule might be outdated or even harmful doesn't mean that the FHS as a whole has outlived its usefulness.
Standards are a double edged sword though. They are great for getting everyone to agree to the "most correct" answer. But they also freeze evolution in place. What happens when your standard doesn't support contemporary use cases? What if it's at direct odds with, say, modern security practices?
FHS hasn't changed in years. Since then, sandboxing, containers, novel package schemes, and more are the zeitgeist. What does the FHS say about them?
Looking at this specific use case, someone is saying /var/lock being world-writable is an unacceptable security risk, but that's very dependent on what your world/users look like. If anything it sounds to me like the maintainer is trying to make the FHS smaller and remove support for a lot of use cases. (Use cases that sound pretty valid to me, without digging in.)
> What does the FHS say about them?
Nothing keeps you from following the FHS inside your container or sandbox.
Are you referring to the location where container images live? Then `/var/lib/containers/` and `/var/lib/containers/storage/` would be perfectly FHS compliant.
The idea though is when you don't want to follow the FHS anymore, like systemd is doing.
Systemd frustrates and angers people with Poettering's complete disregard for bug reports, tradition, and basic common courtesy. At the same time, change needed to happen and change is gonna hurt. And big changes can't wait until they're just as stable as the old system: does anyone develop software like that in their own careers? I try not to ship complete crap but "just as stable as v1" is never a goal.
> Systemd frustrates and angers people with Poettering's complete disregard for bug reports, tradition, and basic common courtesy
Poettering is a Microsoft employee. It is normal that he follows the direction of the mothership. What is not normal is, that he has so many blind followers.
yes but also no
every distro has defined their own new file system layout standard
sure they all started out with the common ancestor of FHS 3.0, but diverged since then in various degrees
and some modern competing standards try to fix it (mainly UAPI Group)
(And yes some people will go one and one about how UAPI is just a way for systemd to force their ideas on others, but if you don't update a standard for 10+[1] years and aren't okay with others taking over this work either, idk. how you can complain for them making their own standard).
[1]: It's more like 20 years, but 10 years ago the Linux Fundation took over it's ownership.
Debian systemd maintainer Luca Boccassi has recently pushed through and dismissed several problematic and undesired breakages as "niche cases" in a way I personally find antithetical to what I expect from Debian.
I hope they have a change of mind in their approach.
This is usual systemd maintainer behavior. Poetterings "I don't consider it much of a problem" is legend: https://github.com/systemd/systemd/issues/5644
Yes, the guy is legendary in terms of maximum arrogance. He did impressive work early on designing a complex system, but gets defensive when that overengineered moloch runs into real-world problems. Systemd has accumulated lots of small hacks to make it more versatile, let's hope a better solution will be available one day.
Whenever I'm using Guix I enjoy the simplicity of Shepherd [1]
[1] https://shepherding.services/manual/html_node/Introduction.h...
Interesting coincidence that both are @Microsoft.
Do we really want Microsoft employees setting standards for Debian?
Apparently yes, since the parent to your comment has been flagged.
Personally I find an interesting observation, and microsoft contributing to linux in any way should be met with skepticism based on the entire last 30 years.
People are so quick to wipe away any wrongdoing from Microsoft as soon as they get thrown a bone, there's some interesting psychology here.
Also Microsoft is doing all kinds of abusive things to their users in Windows 11.
I would say no because microsoft seems to have a magical ability to over-complicate systems, UIs, etc. Not to mention the fact that they were out to crush Linux not so many years ago (through proxies even!). Trusting them to make good, unix-like choices seems ill-informed.
It's complicated. Microsoft devotes resources to it and they can afford to do so but they only have that luxury from being a massive user trampling megacorp.
If we could trust them to be devoting resources to it without any risk of abusing their access and power in the future, that would be sort of okay, but we can't.
Like, should Lockheed intentionally hire North Korean programmers at cheap rates because North Korea can afford to devote resources to helping Lockheed? The issue here is not primarily that North Korea is a massive citizen-trampling megastate. It's that Lockheed's interests are misaligned with North Korea's.
The goals of Debian is based on those who contribute to it. If Microsoft contributes, then it cannot be misaligned. That's what it means to be open. You may not like how that affects Debian, but to deny Microsoft the ability to participate would be equivalent to no longer being truly open.
Also individuals tend to prioritize work that benefits employer interest but that doesn't mean they can do things arbitrarily. It just shifts energy and focus towards certain areas. It's not a problem unless the company employs a large fraction of Debian maintainers which Microsoft doesn't.
No, Debian has a mission of its own, and has always excluded candidates for Developership if they are opposed to Debian's mission. It has never been open in the way you are imagining.
https://www.debian.org/intro/philosophy
I think employing the project lead of systemd gives Microsoft a kind of influence that employing the packager of libjpeg-turbo wouldn't. Lennart is notorious for doing things arbitrarily, and what we are discussing here is that the Debian package maintainer for systemd is also doing things arbitrarily, and is also employed by Microsoft.
how does that work when Microsoft also makes a competing operating system?
If we think about it in logical terms, they could sabotage Debian by having “interests” that are suboptimal for their core demographic.
this is a similar to how euro-skeptics are the people who make the very unpopular laws inside the European Union, leading to all the negative press about the European Union. But they have to be listened to as they are democratically elected and it is a democracy.
No we don't.
I don't know of anyone that's been doing this for a while that hasn't been touched by systemd stupidity in some way. I still loathe the default behavior around the stub-resolver with unqualified names that "just worked" before Lennart decided he knew best.
I've been a Linux sysadmin professionally for 10 years now. I have never had problems with systemd, nor do I consider it to be stupid. Obviously you don't have to like it, but I think you're painting with too broad of a brush.
Doesn’t that mean you started being a sysadmin after systemd was widely adopted? Most people who dislike it probably actively worked with Linux prior to systemd.
Depends on what you mean by this in "been doing this"?
While work now mandates "If you want to use Linux, it has to be Ubuntu" (and I complied). On personal front - about a decade ago I've moved from "vanilla" Gentoo to Calculate Linux - which was and still is 100% Gentoo.
These days difference is even smaller, but already 10+ years ago Calculate had sane profiles as well as all software packages as pre compiled binaries matching those profiles.
And although systemd is one of configurable USE keywords on Calculate/Gentoo - it's still not the default.
So there probably are some folks that haven't been touched by systemd at all... For now.
The fool doesn't know how globbing works, but considers his uninformed guess good enough without testing it or reading (and understanding) the spec.
To be fair, he does know how globbing works: ".*" should include "." and ".." under normal globbing rules. The 'rm' command (presumably) has a special case in it to avoid traversing those in recursive mode because doing so would be a footgun.
To be fair, POSIX-compliant shells are allowed to exclude `.` and `..`, and some do, and POSIX says this may be required in the future.
https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V...
Thankfully they updated it in the last 10 years … oh wait. Hmm. Seems this part of the standard is abandoned. We should create a new standard!
Link to such an dismissal: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1112535#64
Here's another fun thread, which resulted in removal of systemd-resolved from debian-testing, because of a spat between him and the technical committee: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1101532
Needless to say, such actions are ultimately hurting the users.
Wow, I would instantly get fired if I acted that way at my job.
That's actually insanely concerning that he essentially got mad and withheld the patch after
There is Devuan, if you want to Debian but without systemd. I suspect though that "natively" non-systemd distros will be more consistent, personally I've found happiness with Guix.
https://www.devuan.org
I've used it as my home computer for four years, and it seems to work fine.
Debian works fine without systemd (at least for now) though Devuan does indeed make like easier.
[flagged]
Can anyone tell why systemd developers run fast and loose with what they believe and bully everyone with a stick made out of their ideas?
In this case, I think the upstream maintainer's response -- "Upstream systemd will do X, distros who want to are free to do Y" -- is legitimate. Consider the reverse: If systemd requires a writable /run/lock, then distros who want to be more safe won't really be able to (or will have to implement a much more intrusive patch).
Looking from the outside, it looks more that this is a failure of the Debian systemd package maintainer to follow Debian's rules. (Though since I'm not a part of that community, I recognize that there may be cultural expectations I'm not aware of.)
> "Upstream systemd will do X, distros who want to are free to do Y"
Yes this is a good response from upstream. I can work with that, but in that case, even this response didn't get reflected to mailing list discussion, or drowned out instantly.
My question was more general though, questioning systemd developers' behavior collectively (hence the projects' behavior) through time.
The systemd developers have a long history of reinventing the wheel and trying to force it on everyone. We only put up with them because they do some difficult work that nobody wants to do.
Speak for yourself, then. I’ve been using Linux since 2004, and the systemd components finally made system management easy. No more arcane init scripts. Handling of service dependencies. Proper timers. Simple configuration files. Administration knowledge that immediately carries over between all systems equally.
As a user, systemd has improved my productivity tremendously.
The kind of bad mouthing developers that work on solutions for complex problems, code that runs on billions of machines, reflects more of your own fragile ego than them.
> The systemd developers have a long history of reinventing the wheel and trying to force it on everyone. We only put up with them because they do some difficult work that nobody wants to do.
> As a user, systemd has improved my productivity tremendously.
Both can be true at the same time. Particularly in the beginning, there was a long string of really important things that used to Just Work that were broken by systemd. Things like:
1. Having home directories in automounted NFS. Under sysv, autofs waited until the network was up to start running. Originally under systemd, "the network" was counted as being up when localhost was up.
2. Being able to type "exit" from an ssh session and have the connection close. Under systemd, closing the login shell would kill -9 all processes with that userid, including the sshd process handling the connection -- before that process could close the socket for the connection. Meaning you type "exit" in an interactive terminal and it hang.
It's been a while since I encountered any major issues with systemd, but for the first few years there were loads of issues with important things that used to Just Work and then broke and took forever to fix because they didn't happen to affect the systemd maintainers. If you didn't encounter any of these, it's probably because your use cases happened to overlap theirs.
Yes, systemd and journalctl have massively simplified my life. But I think it could have been done with far less disruption.
My favorite systemd bug is when your network black-holes (not disconnected, SYN packets work but nothing else will come back). The entire system will hang.
Obligatory XKCD: https://xkcd.com/438
There's no need to be rude. While I'm not anti-systemd; it didn't change my life tremendously, either.
People tend to bash init scripts, but when they are written well, they both work and port well between systems. At least this is my experience with the fleet I manage.
Dependencies worked pretty well in Parallel-SysV, too, again from my experience. Also, systemd is not faster than Parallel-SysV.
It's not that "I had to learn everything from scratch!" woe either. I'm a kind of developer/sysadmin who never whines and just reads the documentation.
I wrote tons of service files and init scripts during Debian's migration. I was a tech-lead of a Debian derivative at that time (albeit working literally underground), too.
systemd and its developers went through a lot phases, remade a lot of mistakes despite being warned about them, and took at least a couple of wrong turns and booed for all the right reasons.
The anger they pull on themselves are not unfounded, yet I don't believe they should be on the receiving end of a flame-war.
From my perspective, systemd developers can benefit tremendously by stepping down from their thrones and look eye to eye with their users. Being kind towards each other never harms anyone, incl. you.
Everything is a slight modification of something someone else has already done. However it does take effort to make things that work well together. Systemd may not have any novel tricks but it sure does synergize well. Standardization also goes a long way towards simplifying things for a lot of folks. Otherwise everyone constantly reinvents everything and certain integrations are going to invariably be broken or not work well. Modularity is great but it's not free.
You think systemd could be a psyop? Gain influence by paying devs to do the ugly but necessary work, but then sow loads of dissent at the same time...
I’ve long thought a similar thought regarding the browser wars. The project(s) being psyops are left as an exercise to the reader.
That has been their method since the beginning; why would they change from a tactic which works for them?
The central problem with systemd is that they don't want to let you go about your business, they want you to conform to their rule.
The central problem with Linux is that it forces you to conform to everything being a files, including device and IO. I’ll stick to MSDOS where I can go about business poking memory myself to control the hardware. /s My point being is that contracts between systems can improve our lives. We take the process, file, and protected memory models for granted today. However, I’m sure people hated the migration to protected systems as well because it “didn’t let them go about their business”
Linux is 34 years old, and some of the Unix-ism borrowed are even older. There is genuine cruft that has downsides. Different relative priorities of backwards comparability, maintainability and the various issues the legacy issues cause are reasonable.
Systemd basically arose out of a frustration at the legacy issues so the whole project exists as a modernizing effort. No wonder they consider backwards compatibility low priority.
Because that's what they've always done, and it continues to work for them?
Systemd doesn't work for me, but it has taken over most Linux distributions, so clearly it's got something people want that I don't understand. That was the case for PulseAudio too.
well the primary author does work for Microsoft
a company that considers "consent" to be a dirty word
Did you read the same article?
I think everyone involved would prefer updates to the applications, which fix the issue. Debian opted - for now - for reliability for its users, which fits in their mission statement. On Arch /run/lock is only writeable for the superusers, which improves security. As user I value reliability and security and that legacy tools remain usable (sometimes by default, sometimes by a switch).> It is a security issue
The "security issue" expressed is that someone creates 4 billion lock files. The entire reason an application would have a path to create these lock files is because it's dealing with a shared resource. It's pretty likely that lock files wouldn't be the only route for an application to kill a system. Which is a reason why this "security issue" isn't something anyone has taken seriously.
The reason is much more transparent if you read between the lines. Systemd wants to own the "/run" folder and they don't like the idea of user space applications being able to play in their pool. Notice they don't have the same security concerns for /var/tmp, for example.
they don't like the idea of user space applications being able to play in their pool
i think that is somewhat reasonable. but then systemd should have its own space, independent of a shared space: /var/systemd/run or /run/systemd/ ?
> then systemd should have its own space, independent of a shared space
This would go contrary to an unstated goal: making everyone else to dance to systemd's tune, for their own good.
Unstated, but obvious. From a long while back: [0][1]
[0] <https://lore.kernel.org/all/20140402144219.4cafbe37@gandalf....>
[1] <https://lore.kernel.org/all/CA+55aFzCGQ-jk8ar4tiQEHCUoOPQzr-...>
> On Arch /run/lock is only writeable for the superusers, which improves security.
Does it? That means anyone who needs a lock gets superuser, which seems like overkill. Having a group with write permissions would seem to improve security more?
no that isn't what it means at all
a global /run/lock dir is an outdated mechanism not needed anymore
when the standard was written (20 years ago) it standardized a common way programs used to work around not having something like flock. This is also reflected in the specific details of FHS 3.0 which requires lock files to be named as `LCK..{device_name}` and must contain the process id in a specific encoding. Now the funny part. Flock was added to Linux in ~1996, so even when the standard was written it was already on the way of being outdated and it was just a matter of time until most programs start using flock.
This brings is to two ways how this being a issues makes IMHO little sense:
- a lot of use cases for /var/lock have been replaced with flock
- having a global writable dire used across users has a really bad history (including security vulnerabilities) so there have been ongoing affords to create alternatives for anything like that. E.g. /run/user/{uid}, ~/.local/{bin,share,state,etc.}, systemd PrivateTemp etc.
- so any program running as user not wanting to use flock should place their lock file in `/run/user/{uid}` like e.g. pipewire, wayland, docker and similar do (specifically $XDG_RUNTIME_DIR which happens to be `/un/user/{uid}`)
So the only programs affected by it are programs which:
- don't run as root
- don't use flock
- and don't really follow best practices introduced with the XDG standard either
- ignore that it was quite predictable that /var/lock will get limited or outright removed due to long standing efforts to remove global writable dirs everywhere
i.e. software stuck in the last century, or in this case more like 2 centuries ago in the 2000th
But that is a common theme with Debian Stable, you have to fight even to just remove something which we know since 20 years to be a bad design. If it weren't for Debians reputation I think the systemd devs might have been more surprised by this being an issue then the Debian maintainers about some niche tools using outdated mechanisms breaking.
> software stuck in the last century
OK, but suppose you have a piece of software you need to run, that's stuck in the last century, that you can't modify: maybe you lack the technical expertise, or maybe you don't even have access to the source code. Would you rather run it as root, or run it as a user that's a member of a group allowed to write to that directory?
The systemd maintainers (both upstream and Debian package maintainers) have a long history of wanting to ignore any use cases they find inconvenient.
most very old software will depend on many other parts so you anyway often have to run it in a vm with a old Linux kernel + distro release or similar
and if not, you can always put it in a container in which `/var/lock` permissions are changed to not being root-only. Which you probably anyway should do for any abandon ware.
It is my opinion that three things are true:
1) A piece of software can be complete.
2) It is virtuous when a piece of software is complete. We're freed to go do something else with our time.
3) It's not virtuous to obligate modifications to any software just because one has made changes to the shape of "the bikeshed".
That’s true and compatibility is a grace. Software shall not need every month an update. I sign of its quality.
Is this case, usage of /var/lock was clumsy for a long time. And not cleaning up APIs creates something horrible like Windows. API breaks should be limited, to the absolute minimum. The nice part here is, that we can adapt and patch code on Linux usually.
On the other side Linux (the kernel), GLIBC/STDLIBC++, Systemd and Wayland need to be API stable. Everybody dislikes API-Instability.
No, I didn't read the whole article. I follow debian-devel directly. Watched all of it unravel, step by step. I know the resolution since the day it posted to debian-devel.
This was a general question to begin with.
> Their is an option for the old behavior.
The discussion never centered on an option for keeping old behavior for any legitimate reason. The general tone was "systemd wants it this way, so Debian shall oblige". It was a borderline flame-war between more reasonable people and another party which yelled "we say so!"
> It is a security issue and modern solutions to replace exist.
I'm a Linux newbie. Using Linux for 23 years and managing them professionally for 20+ years. I have yet to see an attack involving /var/lock folder being world-writeable. /dev/shm is a much bigger attack surface from my experience.
Migration to flock(2) is not a bad idea, but acting like Nero and setting mailing lists ablaze is not the way to do this. People can cooperate, yet some people love to rain on others and make their life miserable because they think their demands require immediate obedience.
> FHS isn't maintained.
Isn't maintained or not improved fast enough to please systemd devs? IDK. There are standards and RFCs which underpin a ton of things which are not updated.
We tend to call them mature, not unmaintained/abandoned.
> On Arch /run/lock is only writeable for the superusers. As user I value reliability and the legacy tools are usable.
I also value the reliability and agree that legacy tools shall continue working. This is why I use Debian primarily, for the same last 20+ years.
I mean /var/lock was kinda on the way of being super seeded when FHS3 was written 20 years ago. We known it is bad design since a similar amount of time.
If FHS hadn't been unmaintained for nearly 2 decades I'm pretty sure non-root /var/lock would most likely have been deprecated over a decade ago (or at least recommended against being used). We know that cross user writable global dirs are a pretty bad idea since decades, if we can't even fix that I don't see a future for Linux tbh.(1)
Sure systemd should have given them a heads up, sure it makes sense to temporary revert this change to have a transition period. But this change has be on the horizon for over 20 year, and there isn't really any way around it long term.
(1): This might sound a bit ridiculous, but security requirements have been changing. In 2000 trusting most programs you run was fine. Today not so much, you can't really trust anything you run anymore. And it's just a matter of time until it is negligent (like in a legal liability way) if you trust anything but your core OS components, and even that not without constraints. As much as it sucks, if Linux doesn't adept it dies. And it does adopt, but mostly outside of the GPG/FSF space and also I think a bit to slow on desktop. I'm pretty worried about that.
> > FHS isn't maintained. > Isn't maintained or not improved fast enough to please systemd devs? IDK.
more like not maintained at all for 20+ years in a context where everything around it had major changes to the requirements/needs
they didn't even fix the definition of /var/lock. They say it can be used for various lock files but also specify a naming convention must be used, which only works for devices and also only for such not in a sub-dir structure. It also fails to specify that it you should (or at least are allowed to cleared the dir with reboot, something they do clarify for temp). It also in a foot note says all locks should be world readable, but that isn't true anymore since a long time. There are certain lock grouping folders (also not in the spec) where you don't need or want them to be public as it only leaks details which maybe an attacker could use in some obscure niche case.
A mature standard is one which has fixes, improvements and clarification, including wrt. changes in the environment its used in. A standard which recognizes when there is some suboptimal design and adds a warning, recommending not to use that sub-optimal desing etc. Nothing of the sort happened with this standard.
What we see instead is a standard which not only hasn't gotten any relevant updates for ~20 years but didn't even fix inconsistencies in itself.
For a standard to become mature it needs mature, that is a process of growing up, like fixing inconsistencies, clarifications, and deprecation (which doesn't imply removal later one). And this hasn't happen for a long time. Just because something has been used for a long time doesn't mean it's mature.
And if you want to be nit picky even Debian doesn't "fully" comply with FH3, because there are points in it which just don't make sense anymore, and they haven't been fixed them for 20 years.
> In 2000 trusting most programs you run was fine.
Yes. This is why Microsoft didn't decide to base Windows XP on the NT kernel and Windows 95 was nothing more than a (arguably very) pretty coat of paint on top of Windows 3.11.
It's also why multi-user systems with complicated permissions systems that ran processes in isolated virtual address spaces never got built in the decades prior to NT. All those OS researchers and sysadmins saw no reason to distrust the programs other users intended to run.
Probably because everyone lets them keep getting away with it.
Any time there's systemd criticism there's always a quick rebuttal "But it was too hard writing anything in any other init system before so stop complaining".
So there's enough pass being given from the start to Systemd and the developers because it always has been forced upon us.
In this case I assume their "fear" is that unprivileged users can exhaust resources (inodes, filesystem space) in an important tmpfs breaking the system. The proper solution for backward compatibility would probably be something like make /run/lock its own mountpoint, but they fixed it in their system (Fedora) so now it's no longer their problem. Just be thankful their software is portable to such strange niche operating systems like Debian. /s
Funny part is that it used to be a separate file system, before Luca decided to kill it. https://lwn.net/Articles/1041948/
First before venting I'll say this: thanks to stuff like hypervisors, VMs, non-systemd distros, minimal immutable Linux distro like Talos (made to run Kubernetes with a minimum number of executable) and OCI containers where, by definition, PID 1 is not systemd, there's thankfully a real way out of systemd, even on a Linux stack.
And I think more people should look into being, once again, 100% systemd-free.
> Can anyone tell why systemd developers run fast and loose with what they believe and bully everyone with a stick made out of their ideas?
Because the goal is to take control of Linux. That's why systemd is PID1. That's why Poettering works for Microsoft.
The real question is: why was that ultra-convoluted xz backdoor attempt only working on Linux systems that did have systemd? People shall try to wag the dog saying "but it's because this and that made is so that xz was loaded by OpenSSH, it's got nothing to do with systemd". It's got everything to do with systemd.
And the other question is: how many backdoors are operational, today, on systems that have systemd?
Systemd is Microsoft-level bloat, running as PID 1, spreading its tentacles everywhere in Linux distros, definitely on purpose.
Poettering is moreover an insufferable bully, as can be seen once again.
From TFA:
> So what do you recommend how to go on from here? Change Debian policy (as asked in #1111839), revert the change in systemd, find a Debian wide solution or let every package maintainer implement their own solution?
I suggest Debian just drops systemd once and for all. Debian can still be made systemd-free but it's a hassle. Just make Debian systemd free once again.
Meanwhile you'll find me running systemd-less distros on VMs and running containers giving the PID 1 finger to systemd.
I can't wait to switch my Proxmox to FreeBSD's bhyve hypervisor (need to find the time to do it).
But most of all: I cannot wait for the day a systemd-less hypervisor Linux like Proxmox comes out.
It's coming and people who write stuff like: "Don't use Docker, use systemd this and systemd that" are misguided.
systemd is to me the antithesis of what Linux stands for.
I hope Debian gets pissed enough at some point to fully drop systemd.
P.S: one of my machine runs this: https://www.devuan.org/ and honestly it's totally fine. So yup: power to all those running systemd-less distros, BSDs, etc.
[flagged]
Oh, so can we hit people on the nose just because what we say is right?
That's a new one.
What if they are also right, and we start fighting?
Maybe people can discuss instead of bullying each other?
Being able to code doesn't give license to being rude.
I still cannot understand how it came about that everyone adopted systemd, despite it being a showcase of the most selfish, reckless and irresponsible maintainership ever. And it's not like it's something new, it's been like that for what, 10, 15 years now?
The last paragraph in the article states how Fedora uses lockdev(3) to somehow address the issue. It's not clear to me how a library that exposes a standardized management interface to these locks enables non-root users to create them without a world writable lock directory. Is it that lockdev(3) allows the system to use a different directory?
Secondly, it's suggested that the /run/lock directory and/or the solution of acquiring a lock for a shared resources is dated -- but, no mention of the "modern" way to address this? Is it implied that systemd offers a solution to this? What is it?
PS lockdev has such a poorly written manpage.
Another conflict: in Debian Trixie (current stable release), when you run `systemctl status …` you’re given the rather scary warnings:
This is because Debian has a separate /usr/bin and /usr/sbin. There are no plans to either back off this warning by the systemd maintainers, or to patch it out by the Debian maintainers, and also no plans to merge /usr/sbin and /usr/bin.So we’re left with the absurd situation that Debian users are being told the default clean install is ‘degraded’ and ‘tainted’ by its main init system!
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1085370
https://github.com/systemd/systemd/issues/35438
If this is the case, I am surprised no one has patched this anti-feature out of the source. Both sides made that suggestion directly in the bug reports you linked. I expect it should be positively trivial to implement a patch, but someone must step up and do the work.
systemd upstream says "distros can patch it out if they want", the debian package maintainer doesn't want to carry a patch. "Someone needs to write a patch" is not the problem here.
Oops. I misread Luca’s reply as willingness to accept a patch, but he said to submit one upstream - where they explicitly don’t want it. So, yeah, deadlock.
Has anyone pushed on this issue? If a patch exists and upstream explicitly says it belongs in the distro, then I expect an explanation from the maintainer(s) would need to be presented to detail exactly why it’s either unsuitable/unmaintainable or left better as-is.
[flagged]
They should've never accepted systemd into Debian, or other submarines to monopolize knowledge around RH/IBM and steal attention away from original software packages (gnome, wayland, podman, hell even rpm, etc.)
It's been a very long time since I heard about uucico (Unix-to-Unix Copy-In Copy-Out program, part of the UUCP suite). Glad to see it's still being shipped! I wonder if any network uses it.
I've been running it the last year or two to get e-mail to a vintage DOS BBS that had a UUCP package. I was pleasantly surprised it was out of the box usable on both CentOS and Debian, and Postfix still ships with example UUCP email config.
Non-clickbait title -- Debian Technical Committee overrides /run/lock permission change
More-clickbaity title -- Debian Technical Committee tells its doofus maintainers to stop worshipping Poettering so much
"DTC serves crow to 2 systemd maintainers with a history of accepting money from Microsoft"
and also might be just temporary
like overriding it now makes a lot of sense, there needs to be grace periods etc.
but we live in a world where OSes have to become increasingly more resilient to misbehaving programs (mainly user programs, or "server programs" you can mostly isolate with services, service accounts/users etc.). And with continuous increases in both supply chain attacks and crappy AI code this will only get worse.
And as such quotas/usage limits of a temp fs being shared between all user space programs like lvm2 and dmraid is kinda a bad idea.
and for such robustness there aren't that many ways around this change, basically the alternatives are:
- make /var/lock root only and break a very small number of programs which neither use flock nor follow the XDG spec (XDG_RUNTIME_DIR is where your user scoped locks go, like e.g. for wayland or pipewire)
- change lvm2, dmraid, alsa(the low level parts) and a bunch of other things your could say are core OS components to use a different root only lock dir. Which is a lot of work and a lot of breaking changes, much more then the first approach.)
- use a "magic" virtual file system which presents a single unified view of /var/lock, but under the hood magically separates them into different tempfs with different quotas (e.g. based on used id the file gets remapped to /run/user/{uid}, except roots gets a special folder and I guess another folder for "everything else"???) That looks like a lot of complexity to support a very small number of program doing something in a very (20+ years) outdated way. But similar tricks do exist in systemd (e.g. PrivateTemp).
kinda only the first option makes sense
but it's not that it needs to be done "NOW", like in a year would be fine too, but in 5 years probably not
Why is that non-clickbait? Honestly "Debian Technical Committee overrides systemd /run/lock permission change" might be a better title than either, I don't know whether the thing or the actors are more interesting here. But you can only say so much in a title.
Yeah. I expected better from LWN.
IMO the interesting bit here isn't the specific technical change but the interpersonal one of overriding the maintainer(s) decision.
Thus the title reflects the most interesting bit of the story.
The "somebody might mailicuously exhaust memory via /run” argument seems silly IMHO... you can trivially put a hard limit on its size via the tmpfs mount option. I guess if you bend over backwards that's still a DOS in that new files can't be created in /run once it is full, but come on...
There is support for quotas in tmpfs! /me runs and hides under desk to avoid fruit being thrown at me
The problem is /var/lock is still used for various system root components.
Which includes services like lvm2, dmraid and audio drivers.
So you need at least a different /var/lock for "root" and every one else.
So you now can either fix all the root tooling to use a different lock folder. Which would break a lot of things.
Or you break a very small number of very old tools which (mostly) decided to neither use use flock (from 1996) nor follow the XDG spec (XDF_RUNTIME_DIR) from 2003.
So kinda obvious choice what way to go with ;), it just should be coordinated better and communicated better.
And yes you want quotas on tempfs on XDF_RUNTIME_DIR, without question. At least for a system more robust against misbehaving programs.
To be clear a lot of this security concerns have been irrelevant/ignored with thread models from the 2000th. But times (sadly) have changes and you shouldn't just blindly trust user space programs anymore. Even if we ignore malicious programs just thing about all the bugs AI will sneak in. And honestly I'm worried that Desktop Linux will fail to adopt to this changes. Server Linux is clearly adopting, but also more in the non-gnu Linux ecosystem while the more FSF/GNU parts of Linux seem mostly stuck in a past which doesn't look like it will have a future :/. This sucks without question but I just don't see a future without a lot of supply chain attack and AI coding induced crazy misbehavior of user space programs.
PS: Even if you just want a steam deck with the same degree of robustness as console (i.e. a game going rogue will have a hard time to hang the console fully not matter what it does, i.e. you can always press the menu button and close/kill it) you need this kind of subtle changes. And many other.
> Which includes services like lvm2, dmraid and audio drivers.
I know at least the first two have "ignore the lock" command line flags so you can get out of situations like /run being full. So whether it's a DOS depends on how you define DOS :)
> it just should be coordinated better and communicated better
I mean, that's the thing. I don't disagree it should be fixed. But is it really an important enough problem it justifies breaking users now? Not for me...
Someone might exhaust your disk if it’s not a memory fs, not sure how it’s any different.
Wow, talk about making a mountain of a mole hill.
Letting upstream systemd single-handedly define what directories exist with what modes in your distro has never been the intended Modus Operandi.
Debian has a huge selection of packages available for it and clearly is going to have more headaches when it comes to preserving compatibility with all that software.
This is a trivial matter for Debian to handle appropriately, while systemd stays focused on its current priorities. I'm surprised this is being talked about at all outside the appropriate mailing lists... slow week for linux news?
Is anyone surprised that the systemd folks basically said "fuck you, we'll do what we want to" to everyone else?
> Is anyone surprised that the systemd folks basically said "fuck you, we'll do what we want to" to everyone else?
I don't think that's an accurate paraphrase of "Consider this more a passing of the baton from upstream systemd to downstreams: if your distro wants this kind of legacy interface, then just add this via a distro-specific tmpfiles drop-in. But there's no point really in forcing anyone who has a more forward-looking view of the world to still carry that dir."
That kind of drop-in is pretty routine, so I don't know why this became a big thing we're all discussing now.
Well yes, but the complication is that Luca is both systemd developer and debian developer, so the passing of the baton did not really happen here.
no, not really surprised at all
as I will explain below this change is pretty much needed sooner or later with all alternatives being noticeable worse
In addition the way Debian handles things is in a very deep conflict to how most software development is done. Like time wise not practical viable stuff like expecting small OSS teams to have LTS releases and/or back porting fixes which had been fixed with some rewrite/breaking changes. Or fun stuff like for example shipping outdated version of your software with version of dependencies it never was supposed to work with. And in the later case not only will you get bug tickets for already fixed bugs all the time (which you don't really have time for) but als bugs which should be impossible with any "supported" version(/dependency combination). As a consequence a lot of software went from "lets make sure it works with every distro" to "lets make sure it works with everything which is at least somewhat close to upstream and everything else has to figure it out themself".
Which brings us to this discussion which wasn't communicated that well:
- due to the points below it makes sense that systemd says that is is how it is going to be upstream, not open for discussion
- but it's equally reasonable for Debian to decide to prioritize more backward compatibility over bit less robustness, or at least do so for some potentially long grace period
- and systemd really should have told them about the change before they shipped it and it broke things
- but any Debian maintainer thinking they have the right to force systemd to revert the change would be very entitled
- and any systemd dev saying Debian must not revert the change in their distro is equally entitled (it's not like this is likely to cause bugs tickets against systemd which otherwise wouldn't have happened)
so a lot of bad communication. But both the decision to the change and not care about weather Debian likes it, and the decision to revert it because they don't like it, are at the core very reasonable IMHO
---
/var/lock is shared by some very core components like lvm2 and dmraid, it running out of inodes is a serious issue
In 2000 it was okay to not overly consider misbehaving user space programs, in 2025 this is relevant security issue (due to having all of 1: higher standards for resilance/reliability, 2: more malicious code, supply chain attacks etc. 3: more accidentally badly behaving code/low quality code). So this needs to be fixed.
There are 3 solutions I can come up with:
1st make no root program use /var/lock, which is a lot of work and a lot of braking changes.
2nd make /var/lock root only, considering that user programs shouldn't really use it anymore since flock (1996), XDG_RUNTIME_DIR(2003) especially since /run/user/{uid} (still 7years ago or so). So outside of breaking some minor software using an approach outdated by like 20years no issue.
3rd some crazy magical virtual file system proxy mapping entries to different temp fs instances depending on owner user id. But is that worth to have to maintain that blob of complexity which can have security vulnerability for a handful of very old software?
Only really the solution systemd did makes sense. Debian probably would love the 3rd one IFF systemd maintains it for them, so it's not really an option (Debian could still write and maintain it themself, it doesn't need any deep systemd integrations.)
So realistically speaking there is only one solution, if you don't propose a better one just saying "I don't like that, don't do that" is not going to change anything.
And systemd developers are payed by companies which need that stuff, actually most companies would want this kind of in depth increases to robustness. Because a single time of a server not hanging can easily outweighs any cost imposed by read only /var/lock (which most times will be no cost at all). Heck even Desktop systems would profit from it . And for the few systems where companies need the old /var/lock they can just change it like Debian did.
"everyone else" in this case is pretty much only the debian ecosystem because they insist on enforcing a serial lock policy from the 1980s. It's fine if Debian wants to move at the speed of a Soviet committee but I don't think it should be expected (or would be healthy) for systemd to move at the same pace.
A software developer's primary job is to develop software for their users, not to comply with a third party distributor that repackages their software.
The beef isn't with systemd upstream which already has a very simple/boring workaround for this, it's with the debian package maintainer (some people here are wearing multiple hats).
Really the whole raison d'etre of debian is move at this pace to prioritize stability/compatibility. If you don't like that philosophy there are other distros but a package maintainer's primary job is to repackage software for that distro (which presumably users have chosen for a reason), not comply with upstream.
> Any process could write as much as it wanted to /run, which
> could effectively DoS the system by exhausting space or inodes
Does ulimit and quotas sufficiently address this concern?
Somehow I feel that if all the time that has been invested in debating and discussing this had been spent on patching the affected apps, the problem would be properly solved.
I mean, yeah, I get it, systemd bad, democracy good, but these world-writable lock folders are actually a huge pain, and adding some shim code to upgrade to a more secure solution seems achievable?
Genuinely curious - why would world-writeable directory be bad for security? Assuming of course, it's on a separate filesystem mounted with sensible options. Here's what I see from "grep /run/lock /proc/mounts" in sid:
The classic is say you know a root process will write a file called foo.lock in /run/lock, and you (a bad person) have write access to that directory. Then you make foo.lock a symlink to some file (/bin/init or /bin/sh or ld.so for example would be very inconvenient choices) and when the root process writes its lock it destroys that file.
Now obviously people these days generally know about that so hopefully don’t use predictable file names but that’s one way.
> and when the root process writes its lock it destroys that file.
Unless you do open("/run/lock/foo.lock", O_WRONLY|O_CREAT|O_EXCL|O_NOFOLLOW)
Yep. And for good measure, first open with O_CREAT as tempfile with random name, then rename() it to predictable "foo.lock".
Yup to both of you. But all of this is to say, running shellscripts as root (in particular) needs to be done with extreme care, because if people forget those precautions when writing C, they sure as heck don’t trouble themselves to do it when they’re writing shell.
I remember the time (around 2001-2002) when just about every binary was discovered to have some variant on this exact exploit. I happened to be linux sysadmin for a very large, high-profile set of linux boxes at the time. Happy times.
> Now obviously people these days generally know about that so hopefully don’t use predictable file names but that’s one way.
Annoying side effect: now you gotta guess which process created the darn lockfile.
A more sensible approach is to do sanity checking on the lockfile and its contents (i.e. does the contained PID match one's own binary).
> now you gotta guess which process created the darn lockfile
Or you can use “lsof” to just tell you.
The argument is also that you could effectively DoS the system by exhausting space or inodes.
Hmm - I see there's now "lockdev" for managing access to things like serial lines, but what's the preferred method of expressing "only one instance of this program should run at any one time"?
I don't know what the preferred method is. But so far, flocking on my own executable works for me.
As always, it will just take another decade until debian has figured it out.
Considering I never had to reinstall a Debian system because it got bloated or broke one day, I can accept this slow-cooking approach. Even support them on this regard.
Gentoo Linux has quite a different approach than Debian [0] but after the first month or two (once my new-to-Linux ass figured out what it was doing) I've never had to reinstall any Gentoo system. [1]
If you want what Debian provides, it's a poor choice for you... but -IME- it doesn't break on upgrade, unlike some Debian-derived distros I've tried in the past.
[0] Something along the lines of "Always try to package exactly what's provided by upstream, try hard to get distro patches upstreamed, and try to have the latest available upstream release in the 'testing channel'.".
[1] Well, I do have a machine that (aside from "side-loading" kernel updates from time to time) hasn't been updated in four years. While I'll try to update that one in the normal way, I'm probably going to need to reinstall.
Yes, the slow-cooking approach is one of the main reasons why I prefer Debian.
once it's ready they'll push it, that's how it should be
I am in no rush. I like something being stable.
Debian discussions make political discussions seem quick and fast acting by comparison
> He said that he uses cu ""almost constantly for interacting with embedded serial consoles on devices a USB connection away from my laptop""
Whyyyyyyyyyyyyyyy
There are a million better ways of doing this.
I use GNU screen for that. Looking at cu, it looks to be just as tiny and has ssh-like tilda-escapes, including "~." to disconnect. Nice, gotta try it out, thanks!
I got used to using minicom way way back. Has a nice TUI and can do stuff.
I only moved to using screen about 2 years ago after over 2 decades of minicom
cu might be good for anything that you have a fixed config, otherwise I'll just go with minicom
What would you suggest? Personally I find cu the closest to ssh in ergonomics to any of the tui serial terminal programs, and it’s easily available across unixes like FreeBSD or Mac OS, which is a bonus.
cu is the regular way of doing it
I don't see the problem. Minicom and even picocom are bloated compared to cu
One comment [0] highlights a point in favor of the current implementation:
> create a lock file for every dial-in line to prevent its use by programs looking for a dial-out line.
[0]: https://lwn.net/Articles/1042594/