I'm somewhat alarmed that the context that this bug was running in was capable of root login. Is there a reason that an SSH login process would, by default, have enough capabilities to facilitate direct root login?
If it runs as your user and can only log in as you, then I wouldn't expect it to be able to become root. But if it can log you in as different users, I would expect that 1. it needs to run from root, and 2. it can log in as root.
We did Tailscale-like SSH reverse tunnels at scale first in 2013 and the main issue has always been that there are no good libraries. Bash scripting around the OpenSSH binaries is pretty much the only way to go.
There's Paramiko, but Python is still a huge liability in memory-constrained systems.
libssh, libssh2. These are totally independent and unrelated code bases, libssh is maintained by Red Hat mainly for ansible and some other tools, libssh2 was created for curl. libssh2 is client-only, libssh can also be used to implement servers.
Making a useful multi-uid-capable daemon that can’t become root but is not so nerfed that no one uses it is nontrivial. If nothing else, what policy would you use? Why do you think that no other uid is equivalent to root?
At $work we use Tailscale mostly because we were running into too many random issues with NAT with our standard DIY Wireguard setup, especially when people were working from hotels and other places with half-ass network setups.
But we don't trust Tailscale. Just look at the thousands of unresolved Github issues, many of which are actually quite important/useful but have been ignored for months and years.
We very much fear at $work that there are vulnerabilities in the Tailscale product awaiting discovery. Especially as, AFAIK, Tailscale have never had a formal security audit on their software.
So we install it on hardened bastion hosts in an old-school "jump host" model. So people can still get access to where they need to be, but we don't need to install Tailscale's unaudited shit on every single server / vm / etc.
And we only use Tailscale as a glorified VPN, we don't use their hundreds of extra random features like this SSH one.
In terms of SSH, we use old-school OpenSSH and SSH certificates. Its really not that difficult and its really not expensive, you can do offline signing with Yubikeys, no need for expensive HSMs.
> Spoiler alert: This applies to all software. This is why security preaches defense in depth.
Thank you Mr/Mrs Pedant.
However this is a VPN product I am talking about which has an integral part in the defence in depth of which you speak.
Therefore it is only right and proper that I, and others, should be able to trust them to a greater degree. Never 100% of course, as you say. But if you are selling me a security product then you should be able to demonstrate you have put some damn effort into securing it.
"insecure argument handling" in a security product is not a good look. It stinks of sloppy coding practices followed up by a lack of security auditing.
> And we only use Tailscale as a glorified VPN, we don't use their hundreds of extra random features like this SSH one.
Same, you are not alone.
The Tailscale VPN stuff just works. None of their competitors can claim this that I'm aware of. We tried several other products and none of them were as reliable and 'just worked'. I imagine they spend a lot of time just keeping that stuff working.
Word of warning. Netbird does use maginally more data than Tailscale when idle. While you might be tempted by the Jetbird thirdparty app, I found it used ALOT more data and didn't work consistontly even with lazy connections.
> Tailscale on iOS has always been highly unreliable for me
Its better than it used to be.
But the fundamental problem is that the Wireguard app is a simple GUI around `wireguard-go` built as a static C library via cgo. But Tailscale uses a fork of `wireguard-go` and then adds control client, DERP, NAT traversal etc. on top of it.
So there's quite a lot of "bloat" on top of the Wireguard code in Tailscale iOS and therefore your problem might not be Wireguard vs Wireguard implementation question but something happening elsewhere in the Tailscale code.
Interesting! I use it only occasionally, but I don't leave the connection on. I only turn the iOS Tailscale connection on when I need to access resources behind Tailscale.
Are you having issues with it when it's always connected?
It randomly and frequently turns off, then the connected services of course no longer work, then you turn it on manually, they still don’t work, then you hard kill safari a few times and/or toggle it on and off and it eventually works. On Linux or Windows the client is rock-solid however.
Meanwhile with wireguard: It just works. Every time. Every where. Unless someone blocks UDP.
I have a hub and spoke wireguard nerwork that has been working somewhat perfectly for years, but there always are a couple of servers that just dont want to play nice. They connect to the hub, all good, and after a few minutes/hrs connection drops and handshakes fail. Using wireguard-go on docker (older linux machines) for all of them. Its a pain.
In what way has it been unreliable? All of my self hosted services sit on my local network with access provided by Tailscale. I navigate to the address and they just work.
I've been planning a similarly "paranoid" (but apparently not that paranoid) Tailscale setup, for the same reasons.
Another concern I have is whether a compromise of Tailscale's own infra could let an attacker just add itself to my network. Apparently the "Tailnet Lock" feature mitigates this, but it is off by default. If I was an APT, compromising Tailscale would be priority number 1!
> Another concern I have is whether a compromise of Tailscale's own infra could let an attacker just add itself to my network. Apparently the "Tailnet Lock" feature mitigates this, but it is off by default.
Yeah, we use "tailnet lock" to sort of cover that. AFAIK its the only option available.
I say "sort of" because "tailnet lock" is a bit half-assed in its design and implementation.
For example, you cannot sign new nodes from mobile devices (e.g. iOS). And as we know, locking down iOS is easier than a full desktop machine. So that's a bit of a missed opportunity. And as usual, there's a Tailscale Github issue open for it for years ...
And in practice, the "tailnet lock" addition is done through the `tailscale` software on the desktop/laptop. Its click a button or run a CLI without further authentication/authorisation required. So basically anyone could do it. And the keys are, of course, open to exfiltration, you can't use a Yubikey or anything like that. And you can't require multi-signer. Complete joke really.
Also if you use "tailnet lock" with the Tailscale Mullvad integration, its a one-way street. You can sign Mullvad nodes so they can be used as exit nodes, but it is impossible to revoke signatures from obsolete/replaced Mullvad nodes (with tailscale clients, you simply delete them from the org which means they have to re-auth and re-join under a new ID ... but you can't do that with the Mullvad integration, so the 'ghost' exit node could easily return). Your only option is to remove the signing node that signed the Mullvad nodes and start again from scratch. Another joke.
So, yeah, "better than nothing" is the way I would summarise "tailscale lock".
Comically you can sign Tailnet lock from iOS, but it’s an insane workflow.
You need to generate a QR code then scan it from the signing mobile device, which opens a secret menu option to sign (fine just brings up a confirmation dialog).
Incredibly annoying but perhaps more secure vs the threat of randomly tapping at prompts
> If I was an APT, compromising Tailscale would be priority number 1!
Crowdstrike would be at a much higher priority. They already have a cloud connected root console to all the machines in entire corporations. Compromising your network is small potatoes compared to that.
Genuine question, if you use bastion hosts, why do you need Tailscale? Why not to expose tcp/22 to the internet and allow public key authentication only (or, certificate based one, if you prefer fancy)? OpenSSH security track record seems to be better than that of Tailscale.
> Genuine question, if you use bastion hosts, why do you need Tailscale? Why not to expose tcp/22
Fair question. The short answer is my wording was a little off.
We do have pure SSH bastion hosts. Those are OpenBSD-based with a few tweaks, but basically they are locked-down and heavily monitored to within an inch of their life.
The reason Tailscale is there is mostly so the non-techies can access intranet portals and such-like.
But Tailscale does have a useful party-trick for the techies too, in that you can do DNS-suffix based routing (without needing to use Tailscale's DNS).
So they can automagically hop via a Tailscale bastion host to "*.$region.rds.amazonaws.com" or whatever. You just specify the DNS suffix and anything on that wildcard will work. It therefore enables us to further harden access to that stuff to a source IP of the Tailscale bastion hosts (in addition to the usual security stuff, of course).
Tailscale takes care of the automagic load-balancing/re-routing to the Tailscale bastion hosts, so we just have a bunch installed in random geographically-dispersed places and as long as >0 are up and running the Techies will always have a route to where they need to be.
Obviously the above is over-simplified summary before the pedants start picking it apart. ;)
> Tailscale contracts with cybersecurity firm Latacora to conduct
And these published audits of the `tailscale` software are where ?
Even half-serious VPN providers like Mullvad publish in public their regular security audits of their app and infrastructure.
There is zero reason Tailscale cannot do the same.
And frankly, given the nature of this vulnerability, "insecure argument handling" I'm not entirely sure it has been audited ? Or if it has, they should be looking for a new auditor ASAP !
> I'd class Mullvad as the most serious VPN provider
I agree. Its annoying that lots of places have recently been blanket-banning Mullvad IP ranges.
Their sister-company Tilitis[1] is also doing interesting things with the Tkey product.
The present version has limitations due to the original security model but the up-and-coming version has a revised security model to make things a bit more "real-world" useful whilst not making any security trade-offs.
Am aware of them but IIRC they are both unaudited which kind of brings us back to square one ? We would still end up running them at arms-length as we do with Tailscale at the moment.
Isn't Headscale server-side only ?
Also a bit strange that "Tailscale vs Netbird" doesn't feature more prominently on their "Compare Netbird" page. It is hidden behind "Load More". ;D
Yes. You're still running the native Tailscale client code on the hosts, which this evidence reveals can't be as trusted as Tailscale would like us to believe.
I also wouldn't trust Headscale fully. It had a critical defect at some point that, IIRC, would allow an attacker to rotate the key of a registered node without auth to a value chosen by the attacker. And its primary maintainer is a member of the Tailscale team, apparently maintained with full approval of their employer and with reasonable transparency between the projects, but nonetheless the overlap is a little close for comfort.
Frankly, as you've said, I don't trust any of these solutions to be anything more than a convenient way to jump onto a bastion or another host of minimal consequence to get into the network and jump onwards.
> we don't trust Tailscale. Just look at the thousands of unresolved Github issues, many of which are actually quite important/useful but have been ignored for months and years.
This is a poor measure of quality. I've spent considerable time knee-deep in these issues in particular and the vast, vast majority of them are feature requests, bug reports awaiting more information from the submitter, or bug reports that cannot easily be reproduced (likely conflicts with other software). On the whole, Tailscale does an excellent job of "just working" in practically every possible environment. They do an incredible job thriving in the diverse ecosystem of software networking and their work will naturally never be remotely done. There will always be gaps, it's the nature of the beast. I'm not aware of any other product that does a better job here.
If you've spent any time dealing with enterprise software you'd know that there is an infinite firehose of these sort of issues. We're lucky that Tailscale keeps these public. Many other vendors track these sort of issues privately.
If there are particular issues which jeopardize the security posture of Tailscale deployments that have been open for a significant amount of time, my clients and I would love to know. Please share!
> We very much fear at $work that there are vulnerabilities in the Tailscale product awaiting discovery. Especially as, AFAIK, Tailscale have never had a formal security audit on their software.
I can't take this seriously. If you were a customer you could, you know, ask them? Or inspect their SOC2 documents?
I absolutely guarantee they undergo regular formal security audits. There's no question.
> So we install it on hardened bastion hosts in an old-school "jump host" model. So people can still get access to where they need to be, but we don't need to install Tailscale's unaudited shit on every single server / vm / etc.
Bastion hosts are a terrible model in 2026. I can't take this approach seriously.
> we don't need to install Tailscale's unaudited shit on every single server / vm / etc.
You never need to do this. Simply create an exit node into your subnet, and everything you want becomes routable.
It sounds to me like your architecture struggles with a zero-trust network approach if you view this as a blocker. I've got some slots available if you need a consultation!
> In terms of SSH, we use old-school OpenSSH and SSH certificates. Its really not that difficult and its really not expensive, you can do offline signing with Yubikeys, no need for expensive HSMs.
It's expensive in terms of engineer-hours, especially compared to Tailscale. It's also easy to get wrong, and end up with the same vulnerability as TFA, or worse.
But my main issue is that if you see this as equivalent it's telling me that you're not really the target audience. Your set-up is far more vulnerable than the out-of-the-box experience you get with Tailscale. If you don't mind, then I'm happy for you.
Lastly, I don't work for Tailscale and I'm not affiliated with them in any way beyond being a happy user that has solved a lot of problems very easily with their product. I highly recommend it to practically everyone. It's great.
I'm having a hard time taking your comment seriously. It just seems like non-constructive FUD.
> I absolutely guarantee they undergo regular formal security audits. There's no question.
Well, they clearly don't if they have an "insecure argument handling" vulnerability.
As others here have said already here, its an "venerable and ancient class of bugs".
Its the sort of thing that should be picked up by modern defensive programming that includes fuzz testing.
And it is CERTAINLY the sort of thing that should be flagged by any competent security audit. "insecure argument handling" is bread-and-butter for security auditors.
> I can't take this seriously. If you were a customer you could, you know, ask them? Or inspect their SOC2 documents?
SOC2, ISO27001 and all that shit is not the same thing.
As I said, anyone serious who is proud of having had their software audited as clean would publish their reports in public. Nothing to hide. And it strengthens your case with customers.
It can always be a suitably redacted management summary written by the auditor. That's what everyone else does.
> SOC2, ISO27001 and all that shit is not the same thing.
Hard agree. SOC2/ISO27001 are a thing that might be useful, but are mostly a framework dreamt up by auditors and other people who like wearing suits (and making lots of money).
They have some normative controls. But a SOC2 audit is not going to survive contact with an engineer who can push code and whose job depends on the company they work for making money. Where they have to ship product features, and don't have weeks to justify every line of code they write.
A SOC2 audit is not going to survive contact with a script kiddie in their bedroom (or, a nation-state APT) who has plenty of time and an LLM to help them find vulnerabilities or weird edge cases they can compose to attack a system.
These frameworks might serve a purpose, but corporate 'compliance' departments have been lulled into a false sense of security that a satisfactory SOC2 audit means the product is secure and obviates them from asking more technical details or for a fuller audit. It's not. That doesn't matter in a lot of cases, but probably does for a network security solution - I would absolutely be asking deeper questions if I was deploying any of these products en masse.
But then again, the insurers probably asked for a SOC2 certificate, so I guess like most things in life, it's not about whether the systems are secure - it's about whose insurance is ultimately covering the loss.
As an open source maintainer of a fairly large project, we get reports from Ada Logics and similar firms every once in a while and those are absolutely not the same as a proper security audit (which we've also had commissioned in the past). Reports are just a description of a particular issue, not the deeper analysis of the general structure of the codebase that you get from a good audit.
I’m a heavy Tailscale user, so I do trust them quite a bit, but I never used the Tailscale SSH feature.
I feel like OpenSSH’s security record is pretty unbeatable, not sure why I’d swap over for such a security-sensitive tool.
The SSH vulnerability here only applies if the attacker is already on the network. It violates your Tailscale ACLs, but it's not arbitrary external root ssh access. Arguably that's a more secure starting point than vanilla ssh to publicly accessible machine.
OTOH, if you run vanilla ssh on a publicly accessible machine where only port 22 is open, sshd only allows publickey-based authentication and the only accepted key types are FIDO2/U2F hardware-backed keys, it's probably more secure again (less attack surface).
Unless you're using short lived SSH CA certificates, SSH key access has only modest security. You would need to buttress it with TOTP 2FA or enforce Yubikey generation/storage of keys, neither of which are at all default.
Even then, it's obvious you are running SSH, and they can fingerprint the OS, and external logging shows which machines are connecting, and hence have said SSH keys. If they have SSH open they become targets; if they come from CGNAT, carriers can leak location via CGNAT behaviour.
With a plain VPN like WireGuard when they get access to your network, they don't have plain ssh, not to mention root ssh access to hosts. This is a serious issue.
I just don't use stranger's machines to access my personal stuff. Possibly compromised stranger's machines.
I don't see the benefit about that, as I have more laptops than I need.
I used it for a bunch of remote monitor boxes to have a way of centrally managing ssh access to things that were often on- and off-line. It was simple and convenient and access was easily revocable.
I also run self-hosted Wireguard. Initially on a Debian box, nowadays it is integrated into my router (admittedly, this is closed source). For around 6 years at this point.
The whole thing could not be easier and simpler. It has never randomly broken on me. It is fast. It is free. There is no middle man, no vendor.
I never understood the popularity of Tailscale, though that is on me. I'm sure it is a great product, I just never tried it, do not seem the target audience.
What confuses me is the often accompanying, sometimes aggressive anti-selfhosting stance in these sorts of threads. I do not see this in other topics, e.g. someone mentioning they run Jellyfin isn't met with "why not Plex?". Where does that come from? We are on HackerNews, not ProductShillNews, aren't we? I guess self hosting Wireguard is too boring to warrant any further discussion? The VPN equivalent of a Toyota Corolla.
My WireGuard uses (either at home or at work) are very much mobile client to single network
Where Tailscale comes into its own is automatic managing of mesh networking (like an “sdwan” solution). The other thing it excels at is firewall busting - if you have a firewall (with or without address translation) which only allows outgoing traffic to be established (with UDP timeouts for session) then Tailscale also works in a similar way to turn/stun.
If I needed that capability then I’d be looking at Headscale. I don’t need it though.
Remember that this is hackernews, not slashdot. Where the community used to be far smaller and the technology far smaller it was quite normal for everyone to understand basic building blocks of ip addresses, use open source software, wear t-shirts threatening to replace people with a small shell script etc.
It’s not the same community, many people here have no real understanding of computer fundamentals, but instead have expertise in specific narrow areas. They also have little interest in things like free software, but do have an interest in building a new billion dollar company to sell to a behemoth.
Some would consider that an anti-feature. Firewalls are not to be busted. Nothing good lies at the extreme end of working around overly strict policies. Change the policy instead.
I think Tailscale is popular because of how plug and play it is for most people. Although the main reason I use it over self hosting wireguard is the NAT busting it does, which has so far worked flawlessly for me with no setup aside from installing on both devices. There is nothing wrong with self hosting wireguard, but it doesn't actually do the same job as tailscale.
Wireguard by itself also doesn't allow for 2FA or expiring keys. Not as relevant for private use, but some orgs need it for compliance. The idea was always that things like that need to be implemented by an application on top of it, so you end up with something like tailscale eventually.
i have my homelab only reachable via tailscale and can access everything i would ever want on the go that way. it was a matter of 15 min to get it all working.
> I never understood the popularity of Tailscale, though that is on me.
> I guess self hosting Wireguard is too boring to warrant any further discussion?
It's popular because you don't have to deal with NAT punching. It "just works", all the time. And Wireguard is not too boring, it's just not enough on its own.
I'm all for self-hosting and this is exactly why I prefer to use Tailscale and not have to manage jump-hosts and STUN points on some cloud, given that I won't be able to make it as reliable as Tailscale and as cheap as Tailscale (effectively $0). So this is literally the only tradeoff I made while self-hosting everything else.
it wasn't meant as a rebuttal, I'm genuinely asking. tailscale + headscale was just recommended to me, hence that's what I'm using for self hosting. is wireguard's client roughly equivalent to tailscale's? especially tailscale's always-on nature is very appealing.
I found Netbird to be easier to implement. Among its peers it seems to be the most complete open source solution. It IS a bit fast moving though so there might be issues. And they just added an Agent network thing that nobody asked for but it's par for the course these days.
Because wireguard to tailscale is like git to git GUIs. It's solid base but never should be used separately without a proper wrapper if one wants to keep one's sanity.
If you don't care to invest half an hour into learning some basics of how computer networking and in particular CIDR notation and subnet masks work, maybe it is not for you.
The proper fix would be to not use getent CLI tool in their logic, but instead use proper system APIs for looking up user account entries, like one of earlier comments here already mentions. This is shocking amateur hour!
My guess is they hastily threw together something hacky in early development, and forgot to replace it with a real, safe solution later.
The issue here is there is NO single system API for looking up user account entries on Linux.
It's implemented in libc. So you need to link to libc. Tailscale is a Go binary, and they probably prefer it to be statically-linked. glibc NSS implementation also REQUIRES you to load `.so` so you just can't emulate it in Go.
But of course there is, it's part of POSIX, implemented in libc. And if you're using a higher level language, they all have their own wrappers around libc/POSIX APIs. Here is golang's: https://pkg.go.dev/os/user
// userLookupGetent uses "getent" to look up users so that even with static
// tailscaled binaries without cgo (as we distribute), we can still look up
// PAM/NSS users which the standard library's os/user without cgo won't get
// (because of no libc hooks). If "getent" fails, userLookupGetent falls back
// to the standard library.
Others have pointed out that os/user.Lookup is a platform-independent way to resolve this, but additionally you don’t _need_ to link against glibc to use it.
If you are writing go, you usually want to set CGO_ENABLED=0 by default, to avoid inadvertently introducing nonportable code. In this way, only the pure Go implementations are used and there is no need to link (statically or dynamically) against a libc implementation to compile and run your programs.
There is a whole class of security issues where fixes are worse that the issues themselves. Case in point, the OpenSSH itself that sends 100 packets on each keystroke to avoid timing attacks.
Why is that worse than the issue itself? If someone could figure out, say, my root password via an ssh timing attack, that seems bad. Sending 100 packets for each keystroke to protect against this seems cheap in the face of that.
I don't see the point of publishing a security bulletin if you are not going to timely push the fix to artifacts on all affected platforms. Tailscale needs to do better on their release process, docker hub shows last update was 8 days ago.
I don't mind having a bulletin so much as the claim that it's fixed in 1.98.9 or newer, when that release doesn't appear to exist yet. Feels pretty weird practice to advise upgrading to a non-existent version.
1.98.9 has already been tagged since bulletin was published (don't know why they chose on github to tag but not release).
1.98.9 version exists! That's not the question. It should already have been made available for Linux distros assuming this resource from Tailscale is accurate https://pkgs.tailscale.com/stable/?v=1.98.9
Not necessarily, it's a clean room implementation. Even if leading dashes was known/documented/tested to implement they might have done it differently. And maybe it was an implementation detail that it was ever allowed, but that's a weird username, headscale implementation happened not to allow it, and nobody ever noticed the discrepancy.
This project is not associated with Tailscale Inc.
However, one of the active maintainers for Headscale is employed by Tailscale and he is allowed to spend work hours contributing to the project. Contributions from this maintainer are reviewed by other maintainers.
I presume Ada Logics has access to Anthropic's Mythos model via Project Glasswing, and Ada Logics discovered this exploit during their vulnerability research.
It takes over port 22 on the Tailscale interface only. Only had problems with this when I’ve wanted to hit a host’s non-Tailscale ssh service via Tailscale. Otherwise it’s been great for me
It lets organizations (Tailscale) control the timing and narrative around the disclosure more directly. Organizations sometimes avoid the bureaucracy of going through CVE Numbering Authorities by self-publishing. Often a CVE assignment follows self-disclosure, especially when there's pressure to interoperate with vuln-scanning/compliance tooling
In my experience it is really very quick and easy to get a CVE if you contact MITRE directly, and on paper you do not need to disclose too much. The GitHub CNA is (ironically) very slow by comparison but that is a GitHub issue.
Tailscale can also just apply to be their own CNA and issue CVEs for their products themselves, eliminating any such issues entirely.
If it used one of the standard system APIs for looking up user accounts (e.g. getpwnam(3)), there's no way it can happen.
This is incredibly bad engineering, on level of a SQL injection, in 21st century. Something a highschool student experimenting with scripting could come up with, but not a supposedly professional software company.
They actually do use "os/user".Lookup() as a fallback if fetching via getent(1) fails, so they might be using NSS modules to dynamically define users and then not linking their binaries with glibc (musl and the opt-in pure-Go implementation of "os/user".Lookup() do not support NSS).
Still, requesting the information from systemd-userdb would be a better way of doing this if it's really necessary.
A better fix is to call “getent passwd” with no user controlled arguments and then parse the resulting list. This gets rid of the input sanitization problem entirely.
Most variants don't, Alpine in fact does not, nor does the uclibc shell script, nor the commonly copied in implementation in many busybox environments.
A correct implementation would be to just call glibc directly, this seems like a hasty fix to get the patch out the door. The history of vulns from bad shell escaping is as old as bash, whenever possible you probably shouldn't be mixing code and data, especially in a security critical application like this.
The fact that there is no portable way to link the relevant functions that works reliably across all distributions of Linux is a failure of POSIX and GNU, and unfortunately is largely the Linux distribution story in a nutshell.
Your answer is mostly correct, except that when you tug on that thread the shelf comes off the wall, the plaster comes with it, and then it cracks the water pipes on the way to the floor.
Indeed, but they did have code to use "os/user".Lookup() in a fallback path so I wonder if the issue is that they have some NSS module for user definitions (which wouldn't work with osusergo/static or musl-linked binaries)?
If so, this is kind of an understandably ugly problem, though there is still a better option than shelling out -- systemd-userdb.
I argue the opposite: there’s no better fix for this. You can write the most elegant fix, whatever it is, and prevent that from happening only on the codebase that’s fixed. That doesn’t mean that the codebase will always be the only authority on authentication.
The username policy fixes this issue for good, regardless of whatever you write in the future, or whatever new mechanism is introduced.
It’s a restriction for sure, but it’s not a nonsense restriction? Who would have a username starting with a hyphen? I didn’t even know it was possible until today.
This is such a venerable and ancient class of bugs, going at least as far back as AIX 3. Glad to see they're still makin' 'em like they used to.
(If you had SSH access to a host in your Tailscale ACL, you could log in as `-i` and get a root login.)
I'm somewhat alarmed that the context that this bug was running in was capable of root login. Is there a reason that an SSH login process would, by default, have enough capabilities to facilitate direct root login?
If it runs as your user and can only log in as you, then I wouldn't expect it to be able to become root. But if it can log you in as different users, I would expect that 1. it needs to run from root, and 2. it can log in as root.
We did Tailscale-like SSH reverse tunnels at scale first in 2013 and the main issue has always been that there are no good libraries. Bash scripting around the OpenSSH binaries is pretty much the only way to go.
There's Paramiko, but Python is still a huge liability in memory-constrained systems.
libssh, libssh2. These are totally independent and unrelated code bases, libssh is maintained by Red Hat mainly for ansible and some other tools, libssh2 was created for curl. libssh2 is client-only, libssh can also be used to implement servers.
Making a useful multi-uid-capable daemon that can’t become root but is not so nerfed that no one uses it is nontrivial. If nothing else, what policy would you use? Why do you think that no other uid is equivalent to root?
It's so old skool, it's almost new again... almost... Still waiting for that third (or fourth?!) wave of XXE (or similar) bugs.
you can also add parameters to env vars in some popular cloud providers for the same effect.
At $work we use Tailscale mostly because we were running into too many random issues with NAT with our standard DIY Wireguard setup, especially when people were working from hotels and other places with half-ass network setups.
But we don't trust Tailscale. Just look at the thousands of unresolved Github issues, many of which are actually quite important/useful but have been ignored for months and years.
We very much fear at $work that there are vulnerabilities in the Tailscale product awaiting discovery. Especially as, AFAIK, Tailscale have never had a formal security audit on their software.
So we install it on hardened bastion hosts in an old-school "jump host" model. So people can still get access to where they need to be, but we don't need to install Tailscale's unaudited shit on every single server / vm / etc.
And we only use Tailscale as a glorified VPN, we don't use their hundreds of extra random features like this SSH one.
In terms of SSH, we use old-school OpenSSH and SSH certificates. Its really not that difficult and its really not expensive, you can do offline signing with Yubikeys, no need for expensive HSMs.
>We very much fear at $work that there are vulnerabilities in the Tailscale product awaiting discovery
Spoiler alert: This applies to all software. This is why security preaches defense in depth.
> Spoiler alert: This applies to all software. This is why security preaches defense in depth.
Thank you Mr/Mrs Pedant.
However this is a VPN product I am talking about which has an integral part in the defence in depth of which you speak.
Therefore it is only right and proper that I, and others, should be able to trust them to a greater degree. Never 100% of course, as you say. But if you are selling me a security product then you should be able to demonstrate you have put some damn effort into securing it.
"insecure argument handling" in a security product is not a good look. It stinks of sloppy coding practices followed up by a lack of security auditing.
> And we only use Tailscale as a glorified VPN, we don't use their hundreds of extra random features like this SSH one.
Same, you are not alone. The Tailscale VPN stuff just works. None of their competitors can claim this that I'm aware of. We tried several other products and none of them were as reliable and 'just worked'. I imagine they spend a lot of time just keeping that stuff working.
I haven't had problems with NetBird on Android and Linux. I have also used tailscale and found it comparable.
Neat, I haven't heard of/tried NetBird yet. Always more stuff out there to tinker with!
Word of warning. Netbird does use maginally more data than Tailscale when idle. While you might be tempted by the Jetbird thirdparty app, I found it used ALOT more data and didn't work consistontly even with lazy connections.
Tailscale on iOS has always been highly unreliable for me, while the regular Wireguard app just works.
> Tailscale on iOS has always been highly unreliable for me
Its better than it used to be.
But the fundamental problem is that the Wireguard app is a simple GUI around `wireguard-go` built as a static C library via cgo. But Tailscale uses a fork of `wireguard-go` and then adds control client, DERP, NAT traversal etc. on top of it.
So there's quite a lot of "bloat" on top of the Wireguard code in Tailscale iOS and therefore your problem might not be Wireguard vs Wireguard implementation question but something happening elsewhere in the Tailscale code.
Interesting! I use it only occasionally, but I don't leave the connection on. I only turn the iOS Tailscale connection on when I need to access resources behind Tailscale.
Are you having issues with it when it's always connected?
It randomly and frequently turns off, then the connected services of course no longer work, then you turn it on manually, they still don’t work, then you hard kill safari a few times and/or toggle it on and off and it eventually works. On Linux or Windows the client is rock-solid however.
Meanwhile with wireguard: It just works. Every time. Every where. Unless someone blocks UDP.
I have a hub and spoke wireguard nerwork that has been working somewhat perfectly for years, but there always are a couple of servers that just dont want to play nice. They connect to the hub, all good, and after a few minutes/hrs connection drops and handshakes fail. Using wireguard-go on docker (older linux machines) for all of them. Its a pain.
In what way has it been unreliable? All of my self hosted services sit on my local network with access provided by Tailscale. I navigate to the address and they just work.
I've been planning a similarly "paranoid" (but apparently not that paranoid) Tailscale setup, for the same reasons.
Another concern I have is whether a compromise of Tailscale's own infra could let an attacker just add itself to my network. Apparently the "Tailnet Lock" feature mitigates this, but it is off by default. If I was an APT, compromising Tailscale would be priority number 1!
> Another concern I have is whether a compromise of Tailscale's own infra could let an attacker just add itself to my network. Apparently the "Tailnet Lock" feature mitigates this, but it is off by default.
Yeah, we use "tailnet lock" to sort of cover that. AFAIK its the only option available.
I say "sort of" because "tailnet lock" is a bit half-assed in its design and implementation.
For example, you cannot sign new nodes from mobile devices (e.g. iOS). And as we know, locking down iOS is easier than a full desktop machine. So that's a bit of a missed opportunity. And as usual, there's a Tailscale Github issue open for it for years ...
And in practice, the "tailnet lock" addition is done through the `tailscale` software on the desktop/laptop. Its click a button or run a CLI without further authentication/authorisation required. So basically anyone could do it. And the keys are, of course, open to exfiltration, you can't use a Yubikey or anything like that. And you can't require multi-signer. Complete joke really.
Also if you use "tailnet lock" with the Tailscale Mullvad integration, its a one-way street. You can sign Mullvad nodes so they can be used as exit nodes, but it is impossible to revoke signatures from obsolete/replaced Mullvad nodes (with tailscale clients, you simply delete them from the org which means they have to re-auth and re-join under a new ID ... but you can't do that with the Mullvad integration, so the 'ghost' exit node could easily return). Your only option is to remove the signing node that signed the Mullvad nodes and start again from scratch. Another joke.
So, yeah, "better than nothing" is the way I would summarise "tailscale lock".
Comically you can sign Tailnet lock from iOS, but it’s an insane workflow.
You need to generate a QR code then scan it from the signing mobile device, which opens a secret menu option to sign (fine just brings up a confirmation dialog).
Incredibly annoying but perhaps more secure vs the threat of randomly tapping at prompts
> Comically you can sign Tailnet lock from iOS, but it’s an insane workflow.
Interesting, thanks for pointing out the insane workflow.
Although looking around, footguns still remain in terms of relying too much on iOS and Tailnet Lock, e.g. https://github.com/tailscale/tailscale/issues/20475
> If I was an APT, compromising Tailscale would be priority number 1!
Crowdstrike would be at a much higher priority. They already have a cloud connected root console to all the machines in entire corporations. Compromising your network is small potatoes compared to that.
Genuine question, if you use bastion hosts, why do you need Tailscale? Why not to expose tcp/22 to the internet and allow public key authentication only (or, certificate based one, if you prefer fancy)? OpenSSH security track record seems to be better than that of Tailscale.
> Genuine question, if you use bastion hosts, why do you need Tailscale? Why not to expose tcp/22
Fair question. The short answer is my wording was a little off.
We do have pure SSH bastion hosts. Those are OpenBSD-based with a few tweaks, but basically they are locked-down and heavily monitored to within an inch of their life.
The reason Tailscale is there is mostly so the non-techies can access intranet portals and such-like.
But Tailscale does have a useful party-trick for the techies too, in that you can do DNS-suffix based routing (without needing to use Tailscale's DNS).
So they can automagically hop via a Tailscale bastion host to "*.$region.rds.amazonaws.com" or whatever. You just specify the DNS suffix and anything on that wildcard will work. It therefore enables us to further harden access to that stuff to a source IP of the Tailscale bastion hosts (in addition to the usual security stuff, of course).
Tailscale takes care of the automagic load-balancing/re-routing to the Tailscale bastion hosts, so we just have a bunch installed in random geographically-dispersed places and as long as >0 are up and running the Techies will always have a route to where they need to be.
Obviously the above is over-simplified summary before the pedants start picking it apart. ;)
Tailscale contracts with cybersecurity firm Latacora to conduct traditional assessments, advisory services, design reviews, auditing and testing.
https://tailscale.com/security
> Tailscale contracts with cybersecurity firm Latacora to conduct
And these published audits of the `tailscale` software are where ?
Even half-serious VPN providers like Mullvad publish in public their regular security audits of their app and infrastructure.
There is zero reason Tailscale cannot do the same.
And frankly, given the nature of this vulnerability, "insecure argument handling" I'm not entirely sure it has been audited ? Or if it has, they should be looking for a new auditor ASAP !
Agree, but I'd class Mullvad as the most serious VPN provider, personally.
> I'd class Mullvad as the most serious VPN provider
I agree. Its annoying that lots of places have recently been blanket-banning Mullvad IP ranges.
Their sister-company Tilitis[1] is also doing interesting things with the Tkey product.
The present version has limitations due to the original security model but the up-and-coming version has a revised security model to make things a bit more "real-world" useful whilst not making any security trade-offs.
[1] https://www.tillitis.se/
Did you try Headscale? https://github.com/juanfont/headscale or netbird? The latter has been great for me.
> Did you try Headscale? https://github.com/juanfont/headscale or netbird?
Am aware of them but IIRC they are both unaudited which kind of brings us back to square one ? We would still end up running them at arms-length as we do with Tailscale at the moment.
Isn't Headscale server-side only ?
Also a bit strange that "Tailscale vs Netbird" doesn't feature more prominently on their "Compare Netbird" page. It is hidden behind "Load More". ;D
(not top poster)
> Isn't Headscale server-side only ?
Yes. You're still running the native Tailscale client code on the hosts, which this evidence reveals can't be as trusted as Tailscale would like us to believe.
I also wouldn't trust Headscale fully. It had a critical defect at some point that, IIRC, would allow an attacker to rotate the key of a registered node without auth to a value chosen by the attacker. And its primary maintainer is a member of the Tailscale team, apparently maintained with full approval of their employer and with reasonable transparency between the projects, but nonetheless the overlap is a little close for comfort.
Frankly, as you've said, I don't trust any of these solutions to be anything more than a convenient way to jump onto a bastion or another host of minimal consequence to get into the network and jump onwards.
> we don't trust Tailscale. Just look at the thousands of unresolved Github issues, many of which are actually quite important/useful but have been ignored for months and years.
This is a poor measure of quality. I've spent considerable time knee-deep in these issues in particular and the vast, vast majority of them are feature requests, bug reports awaiting more information from the submitter, or bug reports that cannot easily be reproduced (likely conflicts with other software). On the whole, Tailscale does an excellent job of "just working" in practically every possible environment. They do an incredible job thriving in the diverse ecosystem of software networking and their work will naturally never be remotely done. There will always be gaps, it's the nature of the beast. I'm not aware of any other product that does a better job here.
If you've spent any time dealing with enterprise software you'd know that there is an infinite firehose of these sort of issues. We're lucky that Tailscale keeps these public. Many other vendors track these sort of issues privately.
If there are particular issues which jeopardize the security posture of Tailscale deployments that have been open for a significant amount of time, my clients and I would love to know. Please share!
> We very much fear at $work that there are vulnerabilities in the Tailscale product awaiting discovery. Especially as, AFAIK, Tailscale have never had a formal security audit on their software.
I can't take this seriously. If you were a customer you could, you know, ask them? Or inspect their SOC2 documents?
I absolutely guarantee they undergo regular formal security audits. There's no question.
> So we install it on hardened bastion hosts in an old-school "jump host" model. So people can still get access to where they need to be, but we don't need to install Tailscale's unaudited shit on every single server / vm / etc.
Bastion hosts are a terrible model in 2026. I can't take this approach seriously.
> we don't need to install Tailscale's unaudited shit on every single server / vm / etc.
You never need to do this. Simply create an exit node into your subnet, and everything you want becomes routable.
It sounds to me like your architecture struggles with a zero-trust network approach if you view this as a blocker. I've got some slots available if you need a consultation!
> In terms of SSH, we use old-school OpenSSH and SSH certificates. Its really not that difficult and its really not expensive, you can do offline signing with Yubikeys, no need for expensive HSMs.
It's expensive in terms of engineer-hours, especially compared to Tailscale. It's also easy to get wrong, and end up with the same vulnerability as TFA, or worse.
But my main issue is that if you see this as equivalent it's telling me that you're not really the target audience. Your set-up is far more vulnerable than the out-of-the-box experience you get with Tailscale. If you don't mind, then I'm happy for you.
Lastly, I don't work for Tailscale and I'm not affiliated with them in any way beyond being a happy user that has solved a lot of problems very easily with their product. I highly recommend it to practically everyone. It's great.
I'm having a hard time taking your comment seriously. It just seems like non-constructive FUD.
> I absolutely guarantee they undergo regular formal security audits. There's no question.
Well, they clearly don't if they have an "insecure argument handling" vulnerability.
As others here have said already here, its an "venerable and ancient class of bugs".
Its the sort of thing that should be picked up by modern defensive programming that includes fuzz testing.
And it is CERTAINLY the sort of thing that should be flagged by any competent security audit. "insecure argument handling" is bread-and-butter for security auditors.
> I can't take this seriously. If you were a customer you could, you know, ask them? Or inspect their SOC2 documents?
SOC2, ISO27001 and all that shit is not the same thing.
As I said, anyone serious who is proud of having had their software audited as clean would publish their reports in public. Nothing to hide. And it strengthens your case with customers.
It can always be a suitably redacted management summary written by the auditor. That's what everyone else does.
>> Or inspect their SOC2 documents?
> SOC2, ISO27001 and all that shit is not the same thing.
Hard agree. SOC2/ISO27001 are a thing that might be useful, but are mostly a framework dreamt up by auditors and other people who like wearing suits (and making lots of money).
They have some normative controls. But a SOC2 audit is not going to survive contact with an engineer who can push code and whose job depends on the company they work for making money. Where they have to ship product features, and don't have weeks to justify every line of code they write.
A SOC2 audit is not going to survive contact with a script kiddie in their bedroom (or, a nation-state APT) who has plenty of time and an LLM to help them find vulnerabilities or weird edge cases they can compose to attack a system.
These frameworks might serve a purpose, but corporate 'compliance' departments have been lulled into a false sense of security that a satisfactory SOC2 audit means the product is secure and obviates them from asking more technical details or for a fuller audit. It's not. That doesn't matter in a lot of cases, but probably does for a network security solution - I would absolutely be asking deeper questions if I was deploying any of these products en masse.
But then again, the insurers probably asked for a SOC2 certificate, so I guess like most things in life, it's not about whether the systems are secure - it's about whose insurance is ultimately covering the loss.
Have you tried asking them for a copy of such a report? Or is your plan to just continue complaining until they make one public?
The finding in TFA was the result of a security audit.
As an open source maintainer of a fairly large project, we get reports from Ada Logics and similar firms every once in a while and those are absolutely not the same as a proper security audit (which we've also had commissioned in the past). Reports are just a description of a particular issue, not the deeper analysis of the general structure of the codebase that you get from a good audit.
> The finding in TFA was the result of a security audit.
Don't you fucking dare.
Might I point you to the words "We would like to thank Anthropic and Ada Logics for reporting this issue.".
It was not commissioned by Tailscale. It was DONE BY OTHERS AND REPORTED TO TAILSCALE. Just like the fucking disclosure tells you.
Tailscale should not be relying on the random goodwill of others to do random audits of unknown coverage at random intervals.
That is not a serious approach to security.
They should be commissioning their own, paid out of their own pocket, at regular intervals, and publishing the results.
I’m a heavy Tailscale user, so I do trust them quite a bit, but I never used the Tailscale SSH feature. I feel like OpenSSH’s security record is pretty unbeatable, not sure why I’d swap over for such a security-sensitive tool.
The SSH vulnerability here only applies if the attacker is already on the network. It violates your Tailscale ACLs, but it's not arbitrary external root ssh access. Arguably that's a more secure starting point than vanilla ssh to publicly accessible machine.
OTOH, if you run vanilla ssh on a publicly accessible machine where only port 22 is open, sshd only allows publickey-based authentication and the only accepted key types are FIDO2/U2F hardware-backed keys, it's probably more secure again (less attack surface).
Unless you're using short lived SSH CA certificates, SSH key access has only modest security. You would need to buttress it with TOTP 2FA or enforce Yubikey generation/storage of keys, neither of which are at all default.
Even then, it's obvious you are running SSH, and they can fingerprint the OS, and external logging shows which machines are connecting, and hence have said SSH keys. If they have SSH open they become targets; if they come from CGNAT, carriers can leak location via CGNAT behaviour.
In contrast tailscale makes this much harder.
With a plain VPN like WireGuard when they get access to your network, they don't have plain ssh, not to mention root ssh access to hosts. This is a serious issue.
Yeah pretty much just use tailscale as a vpn.. do one thing as they say.
I've used it before to access my tailnet machines through a browser on a machine I can't download software on.
I just don't use stranger's machines to access my personal stuff. Possibly compromised stranger's machines. I don't see the benefit about that, as I have more laptops than I need.
That's cool. Others are not like you and may have use/need of things that you have no use/need for is what I'm trying to communicate here.
I used it for a bunch of remote monitor boxes to have a way of centrally managing ssh access to things that were often on- and off-line. It was simple and convenient and access was easily revocable.
Convenience for the most part but in general, I agree. I like having it as an option.
> usernames were passed as arguments to getent(1) to retrieve the corresponding passwd entry
Always try to use actual API/system calls (in this case getpwnam) instead of calling sub-processes.
I'll stick to my 100% self-hosted Wireguard setup, thank you very much.
Why not tailscale plus head scale for self hosting?
I do not understand this rebuttal.
I also run self-hosted Wireguard. Initially on a Debian box, nowadays it is integrated into my router (admittedly, this is closed source). For around 6 years at this point.
The whole thing could not be easier and simpler. It has never randomly broken on me. It is fast. It is free. There is no middle man, no vendor.
I never understood the popularity of Tailscale, though that is on me. I'm sure it is a great product, I just never tried it, do not seem the target audience.
What confuses me is the often accompanying, sometimes aggressive anti-selfhosting stance in these sorts of threads. I do not see this in other topics, e.g. someone mentioning they run Jellyfin isn't met with "why not Plex?". Where does that come from? We are on HackerNews, not ProductShillNews, aren't we? I guess self hosting Wireguard is too boring to warrant any further discussion? The VPN equivalent of a Toyota Corolla.
My WireGuard uses (either at home or at work) are very much mobile client to single network
Where Tailscale comes into its own is automatic managing of mesh networking (like an “sdwan” solution). The other thing it excels at is firewall busting - if you have a firewall (with or without address translation) which only allows outgoing traffic to be established (with UDP timeouts for session) then Tailscale also works in a similar way to turn/stun.
If I needed that capability then I’d be looking at Headscale. I don’t need it though.
Remember that this is hackernews, not slashdot. Where the community used to be far smaller and the technology far smaller it was quite normal for everyone to understand basic building blocks of ip addresses, use open source software, wear t-shirts threatening to replace people with a small shell script etc.
It’s not the same community, many people here have no real understanding of computer fundamentals, but instead have expertise in specific narrow areas. They also have little interest in things like free software, but do have an interest in building a new billion dollar company to sell to a behemoth.
Thanks for the first 3 paragraphs, but big no to the last two. The vocal minority shapes communities.
> it excels at is firewall busting
Some would consider that an anti-feature. Firewalls are not to be busted. Nothing good lies at the extreme end of working around overly strict policies. Change the policy instead.
I think Tailscale is popular because of how plug and play it is for most people. Although the main reason I use it over self hosting wireguard is the NAT busting it does, which has so far worked flawlessly for me with no setup aside from installing on both devices. There is nothing wrong with self hosting wireguard, but it doesn't actually do the same job as tailscale.
Wireguard by itself also doesn't allow for 2FA or expiring keys. Not as relevant for private use, but some orgs need it for compliance. The idea was always that things like that need to be implemented by an application on top of it, so you end up with something like tailscale eventually.
as someone who uses tailscale: exactly this.
i have my homelab only reachable via tailscale and can access everything i would ever want on the go that way. it was a matter of 15 min to get it all working.
NAT busting is a great point.
> I never understood the popularity of Tailscale, though that is on me.
> I guess self hosting Wireguard is too boring to warrant any further discussion?
It's popular because you don't have to deal with NAT punching. It "just works", all the time. And Wireguard is not too boring, it's just not enough on its own.
I'm all for self-hosting and this is exactly why I prefer to use Tailscale and not have to manage jump-hosts and STUN points on some cloud, given that I won't be able to make it as reliable as Tailscale and as cheap as Tailscale (effectively $0). So this is literally the only tradeoff I made while self-hosting everything else.
How do I install wire guard on my mom's Apple TV?
it wasn't meant as a rebuttal, I'm genuinely asking. tailscale + headscale was just recommended to me, hence that's what I'm using for self hosting. is wireguard's client roughly equivalent to tailscale's? especially tailscale's always-on nature is very appealing.
I also tested tailscale, headscale and netbird and found all these drain your battery on mobile.
Others report no issues but I had massive drain on iOS even with only 4 connections open.
Native wireguard is unnoticeable.
I found Netbird to be easier to implement. Among its peers it seems to be the most complete open source solution. It IS a bit fast moving though so there might be issues. And they just added an Agent network thing that nobody asked for but it's par for the course these days.
i really dislike that there is no way to do dhcp for new clients and that i have to manually define peers in each "exit node"
Because wireguard to tailscale is like git to git GUIs. It's solid base but never should be used separately without a proper wrapper if one wants to keep one's sanity.
But are there any good self-hosted wrappers?
I ditched wireguard for tailscale for the ease of managing it. I'd much rather run my own independently but CBA with the config editing hassle.
tailscale is self-hostedable
haha self hosted wireguard, an opportunity to find out AllowedIPs: 0.0.0.0/0 does the opposite of what you think it will do
If you don't care to invest half an hour into learning some basics of how computer networking and in particular CIDR notation and subnet masks work, maybe it is not for you.
I see that you don't know what AllowedIPs does :)
Yeah right, I manually set up 4 servers, 2 pcs and 2 phones in a wireguard mesh network and I don't understand what it does.
Look I'm the one who made the error here. All I am saying is that AllowedIPs doesn't do what it sounds like it does.
Allow any (ip4) traffic to enter the tunnel, install a route in the default routing table to make that happen (well the second depends on the client)
Does it do the opposite of that?
Is this what your referring to? https://utcc.utoronto.ca/~cks/space/blog/linux/WireGuardAllo...
[dead]
> Tailscale SSH now rejects usernames with leading dashes.
Is the proper fix not restricting users not possible in these poorly designed ancient systems?
Similarly re another issue: why not just fix the permission issues instead of restricting users?
> Tailscale now disallows the use of UIDs or numeric-only usernames via SSH to avoid this ambiguity
The proper fix would be to not use getent CLI tool in their logic, but instead use proper system APIs for looking up user account entries, like one of earlier comments here already mentions. This is shocking amateur hour!
My guess is they hastily threw together something hacky in early development, and forgot to replace it with a real, safe solution later.
The issue here is there is NO single system API for looking up user account entries on Linux.
It's implemented in libc. So you need to link to libc. Tailscale is a Go binary, and they probably prefer it to be statically-linked. glibc NSS implementation also REQUIRES you to load `.so` so you just can't emulate it in Go.
Then, "link to libc". Which libc? glibc? musl?
But of course there is, it's part of POSIX, implemented in libc. And if you're using a higher level language, they all have their own wrappers around libc/POSIX APIs. Here is golang's: https://pkg.go.dev/os/user
Which uses libc via CGO or parses /etc/passwd with no CGO, which won't work for some cases.
https://github.com/tailscale/tailscale/blob/e4144230f410204a...
Thanks, that's what I didn't get from the other discussion, so it seems to be a rather trivial fix - just use the existing language functionality
> The issue here is there is NO single system API for looking up user account entries on Linux.
Yes there is, and you answered in the next line, it is implemented in libc.
If you want to check authentication use libc don't try to implement crypto and authentication yourself.
Others have pointed out that os/user.Lookup is a platform-independent way to resolve this, but additionally you don’t _need_ to link against glibc to use it.
If you are writing go, you usually want to set CGO_ENABLED=0 by default, to avoid inadvertently introducing nonportable code. In this way, only the pure Go implementations are used and there is no need to link (statically or dynamically) against a libc implementation to compile and run your programs.
With CGO disabled, it only reads /etc/passwd, while the glibc getpwnam(3) can query LDAP etc.
That’s not the use case of the tailscale daemon though, which was using getent passwd.
To clarify: I’m not against CGO, just introducing a glibc dependency by default. I would only introduce it when I needed it.
There is a whole class of security issues where fixes are worse that the issues themselves. Case in point, the OpenSSH itself that sends 100 packets on each keystroke to avoid timing attacks.
Why is that worse than the issue itself? If someone could figure out, say, my root password via an ssh timing attack, that seems bad. Sending 100 packets for each keystroke to protect against this seems cheap in the face of that.
Because it breaks our mental model of how things work.
Or it means it is time to update the model. Models need to change as the world they model changes, right?
The OpenBSD developers could have opted for a warning, like the one you see about post-quantum encryption when connecting to old servers.
Instead, they just sneaked the change upon the world.
The example doesn't seem worse, but more importantly - are these issues part of the class (and why)?
I don't see the point of publishing a security bulletin if you are not going to timely push the fix to artifacts on all affected platforms. Tailscale needs to do better on their release process, docker hub shows last update was 8 days ago.
I don't mind having a bulletin so much as the claim that it's fixed in 1.98.9 or newer, when that release doesn't appear to exist yet. Feels pretty weird practice to advise upgrading to a non-existent version.
1.98.9 has already been tagged since bulletin was published (don't know why they chose on github to tag but not release).
1.98.9 version exists! That's not the question. It should already have been made available for Linux distros assuming this resource from Tailscale is accurate https://pkgs.tailscale.com/stable/?v=1.98.9
Edit: Their changelog also mentions the version: https://tailscale.com/changelog#all
Ah, OK, its the weird GitHub non-release that threw me. Thank you!
so you can disable or mitigate it…
tailscale ssh: replacing a 25-year-old battle-tested codebase with a startup's Go rewrite and then acting surprised when it has bugs
So, giving access via tailscale but using OpenSSH is safe, right?
Yes, this only involves their wrapper that is managed by ACL rules.
as much as handing control to a remote third part is, yes.
Good point. I self host headscale but it also has the ssh feature, probably also insecure.
Not necessarily, it's a clean room implementation. Even if leading dashes was known/documented/tested to implement they might have done it differently. And maybe it was an implementation detail that it was ever allowed, but that's a weird username, headscale implementation happened not to allow it, and nobody ever noticed the discrepancy.
How is that true if:
This project is not associated with Tailscale Inc.
However, one of the active maintainers for Headscale is employed by Tailscale and he is allowed to spend work hours contributing to the project. Contributions from this maintainer are reviewed by other maintainers.
Nice, this makes me feel a little better.
>>> We would like to thank Anthropic and Ada Logics for reporting this issue.
it seems anthropic also use tailscale or it's just being discovered by the mythos model?
I presume Ada Logics has access to Anthropic's Mythos model via Project Glasswing, and Ada Logics discovered this exploit during their vulnerability research.
Tailscale SSH has caused me other problems in the past because it takes over port 22. I'm not a fan.
It takes over port 22 on the Tailscale interface only. Only had problems with this when I’ve wanted to hit a host’s non-Tailscale ssh service via Tailscale. Otherwise it’s been great for me
That is what it's supposed to do, though. It's not a secret.
[dead]
It appears that tailscale ssh is off by default? How can I validate that? When I try to log in, I just get:
Why own numbering instead of CVE?
It lets organizations (Tailscale) control the timing and narrative around the disclosure more directly. Organizations sometimes avoid the bureaucracy of going through CVE Numbering Authorities by self-publishing. Often a CVE assignment follows self-disclosure, especially when there's pressure to interoperate with vuln-scanning/compliance tooling
And sometimes it’s just impossible to get a CVE number in a reasonable amount of time, or indeed at all.
In my experience it is really very quick and easy to get a CVE if you contact MITRE directly, and on paper you do not need to disclose too much. The GitHub CNA is (ironically) very slow by comparison but that is a GitHub issue.
Tailscale can also just apply to be their own CNA and issue CVEs for their products themselves, eliminating any such issues entirely.
Some reasons why an org might want to become their own CNA: https://daniel.haxx.se/blog/2024/01/16/curl-is-a-cna/
To be clear creating your own custom IDs isn't the same thing as being a CNA, but some of the motivations are the same.
pure logic error, the undergoing tailscale rust rewrite can't help this too:)
If it used one of the standard arguments-handling crates (e.g. "clap") there's no way it can happen.
If it used one of the standard system APIs for looking up user accounts (e.g. getpwnam(3)), there's no way it can happen.
This is incredibly bad engineering, on level of a SQL injection, in 21st century. Something a highschool student experimenting with scripting could come up with, but not a supposedly professional software company.
Agreed.
that said, the limit impact perhaps is only affected multiple users in tailnet if the ACL is not configure correctly
As single tailnet+single user, perhaps it's just okay
Why does the page only show up to 008? Did TS remove the notice?
Edit: my bad, it's between 008 and 007 ?=
> "Tailscale SSH now rejects usernames with leading dashes."
Really? That's the fix?
A proper fix is to use "--" to separate arguments.
A proper fix is not to shell out to a command at all; use getpwnam(3) or similar.
They actually do use "os/user".Lookup() as a fallback if fetching via getent(1) fails, so they might be using NSS modules to dynamically define users and then not linking their binaries with glibc (musl and the opt-in pure-Go implementation of "os/user".Lookup() do not support NSS).
Still, requesting the information from systemd-userdb would be a better way of doing this if it's really necessary.
EDIT: Yeah, some older comments confirm that was their reasoning. https://github.com/tailscale/tailscale/commit/1fc1077052dfa7...
“--“ doesn’t work on all versions of getent.
A better fix is to call “getent passwd” with no user controlled arguments and then parse the resulting list. This gets rid of the input sanitization problem entirely.
that's stupid. getent passwd will fetch the user list from the network depending on setup. Have fun fetching 40000 records from the network every call
Are there any actual systems that can run Tailscale and that have faulty getent?
Yes
Which ones? Alpine supports it.
Most variants don't, Alpine in fact does not, nor does the uclibc shell script, nor the commonly copied in implementation in many busybox environments.
Reference: https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/m...
Their fix just future-proofs it in case the same bug gets reintroduced.
A correct implementation would be to just call glibc directly, this seems like a hasty fix to get the patch out the door. The history of vulns from bad shell escaping is as old as bash, whenever possible you probably shouldn't be mixing code and data, especially in a security critical application like this.
The fact that there is no portable way to link the relevant functions that works reliably across all distributions of Linux is a failure of POSIX and GNU, and unfortunately is largely the Linux distribution story in a nutshell.
Your answer is mostly correct, except that when you tug on that thread the shelf comes off the wall, the plaster comes with it, and then it cracks the water pipes on the way to the floor.
This is just a dirty fix. It adds weird restrictions and masks issues.
Refactoring external invocations to use safe argument handling is a better way to fix it. Along with tests that exercise weird names.
http://github.com/tailscale/tailscale/commit/e4144230f410204...
All the hallmarks of an LLM fix right there...
Indeed, but they did have code to use "os/user".Lookup() in a fallback path so I wonder if the issue is that they have some NSS module for user definitions (which wouldn't work with osusergo/static or musl-linked binaries)?
If so, this is kind of an understandably ugly problem, though there is still a better option than shelling out -- systemd-userdb.
I argue the opposite: there’s no better fix for this. You can write the most elegant fix, whatever it is, and prevent that from happening only on the codebase that’s fixed. That doesn’t mean that the codebase will always be the only authority on authentication.
The username policy fixes this issue for good, regardless of whatever you write in the future, or whatever new mechanism is introduced.
It’s a restriction for sure, but it’s not a nonsense restriction? Who would have a username starting with a hyphen? I didn’t even know it was possible until today.
> I argue the opposite: there’s no better fix for this
The better fix would be to not have the username pass through a parser looking for cli flags in the first place.
How do you propose to do that when username is a possible argument?
I'll fix this entire HN thread for you all and you'll thank me later:
Sadly, yet another path to root via Tailscale.
If their scope grows, and they run so much as root, it won't be their last.
[dead]