SSH has *ANOTHER* built-in solution, in the form of the SSHFP DNS record.
If the DNS record for the host has an SSHFP (SSH FingerPrint) record, SSH will compare it to the retrieved public key(s) and refuse the connection if there is a mismatch. It can be configured to require DNSSec for this, or to only reject if it gets a secure rejection (to prevent DoS).
It works perfectly, has no notable down sides (just add a DNS record when you generate the host's SSH key), and has been around for many years.
I recommend reading the description of the option `VerifyHostKeyDns` in the `ssh_config` man page.
If set to `yes`, you get automatic trust-on-first-use (no user prompt) if you use DNSSec, and you get the current asking-the-user behavior if your DNSSec is broken or you are under attack.
Obviously it's more secure if you use DNSSec, because that way you can reflexively deny any request to manually verify a host key, but it provides value regardless.
That site doesn't mention that when DNSSec is absent, the behaviour of SSH is identical to what happens if you hadn't used the SSHFP record at all, except that for unsophisticated attackers it also displays "no matching host key found in DNS".
So even without DNSSec using the SSHFP records is an improvement over not using them because some of the time it tells you for certain you're being interfered with.
There is no situation in which an insecure DNS response is auto-trusted by the SSH client.
The article describes a method of securely seeding private key data to newly provisioned servers. It's a bit of bootstrapping problem, and not something a PKI can solve.
You usually need some form of trusted communication with a new server until you can give it its real identity, in the form of host names and cryptographic keys. In an enterprise environment this can usually be done with some sort of isolated management or provisioning vlan. In a cloud environment, I've seen all sorts of more or less hacky solutions but since it depends a lot on specific details of your networks and APIs, bespoke solutions are fine.
A big class of attacker is nation state attackers who do not want to risk discovery.
A big way to deter them is to keep remote log files which, if analyzed, will reveal any attack.
For example, if both ssh-client and ssh-server kept a fingerprint of the session key in some append-only logfile, then a later administrator could compare the logfiles to know if an MITM happened.
Suddenly, nation state attackers won't be interested in MITM-ing at all.
Unfortunately it appears openssh doesn't even have an option to create such a logfile!! Why not??
Linodes response of 'we have observed no illegal activity and are closing this ticket' is as close as you can get to 'yes, this is a legal wiretap and we aren't allowed to tell you about it'.
> Couldn’t the MITM ssh server just forward the client’s fingerprint to the legitimate server?
The client sends not only the public key, but also a signature, and that signature depends on the output from the key exchange, so it's "bound" to the shared keys negotiated between the client and the server. If the MITM server does separate key exchanges with the client (pretending to be the real server) and the server (pretending to be the real client), the signature won't match; if it forwards the key exchange between the real client and the real server, it won't be able to decrypt the packets.
That's the best thing about SSH public key authentication (and HTTPS client certificates): even when MITM can impersonate the server to the client (because the client didn't verify the host key), it can't impersonate the client to the real server.
Let me see if I understand correctly: Client takes its own public key and the server's public key and creates this signature.
MITM can take its public key and the client's public key and send the resulting signature to the server instead of forwarding what it received from the client.
Do pretty much the same exact thing: MITM PK + Server's PK -> Client. Now client has a signature as well. The signatures that client and server have are different but that is OK as long as MITM can see and change all communication.
It has been a while since I went through the details of the protocol, so I must be missing something. What is it?
> MITM can take its public key and the client's public key and send the resulting signature to the server instead of forwarding what it received from the client.
That's not possible, since the MITM doesn't know the client's private key (and using a different public key will be rejected by the server).
> Do pretty much the same exact thing: MITM PK + Server's PK -> Client. Now client has a signature as well. The signatures that client and server have are different but that is OK as long as MITM can see and change all communication.
You're confusing the Diffie-Hellman Key Exchange with the Public Key Authentication Method. When you MITM the key exchange, the shared secrets the client and server have are different (one side has a secret derived from the client and MITM keys, the other side has a secret derived from the MITM and server keys), but that works as long as the MITM can see and change all communication (basically, decrypting it and encrypting it again).
But since the secrets are different, the session identifier is also different. The MITM can't forward the signature from the client since the server will fail to verify it due to the mismatch in the session identifier; the MiTM can't create a new signature with the client public key since it doesn't have the corresponding private key; and the MITM can't create a valid signature with its own public key (and the corresponding private key) since that key won't be in the authorized keys list for that user account in the server.
> Couldn’t the MITM ssh server just forward the client’s fingerprint to the legitimate server?
Fingerprints are derived from the certificates/private keys. Unless I don't understand some basic crypto, or SSH works in some obtuse way, I do not think it would be possible for the MITM attacker to present the server with the true client's fingerprint unless they also had had the client's private key.
To try to add some explanation here: the server doesn't send the fingerprint. The server sends the public key, your computer just shows a fingerprint because it's more human-friendly and sufficiently secure.
If they forward the real key, so it matches the fingerprint, and you use it, they can't MITM the request because they can't read the contents.
I think the idea is the attacker didn't compromise both the local machine and the remote log sink machine. If you want to get really fancy the techniques used in cert revocation logs/blockchains could be used.
Blockchain is completely unnecessary (as it always is; I thought people stopped trying to ram that garbage into everything years ago).
I was answering this question from GP:
> Unfortunately it appears openssh doesn't even have an option to create such a logfile!! Why not??
The answer is because in Linux systems the logging logistics are handled at the system level, just like starting and running openssh itself. The answer to "why not" is because that's the logging system's job, not openssh's.
rsyslogd is one simple and direct way to distribute logs to remote machines.
A few do. Though a more common way of securing the loop is to take the desired user public key during the build process.
Assuming you trust that the host control panel (or API server) hasn't been hacked, which you are assuming anyway if you trust a host fingerprint given to you that way, that should be as secure. For a small bit of extra assurance, to protect from an extra very unlikely attack, generate a new key pair just for this VM's creation so that you know you aren't connecting to some other VM that happens to have a known public key of yours and you've been redirected to by DNS poisoning.
AFAIK this is not secure. The SSH protocol doesn’t require the server to know the client’s public key for the connection to succeed. Instead, the client actively sends its public key to the server while authenticating, and the server just decides whether to allow a connection or not. OpenSSH will typically match the client’s key against a fixed list of keys, but nothing stops a rogue server from allowing connections from any client key. In fact, OpenSSH itself can be configured to do this using AuthorizedKeysCommand.
It does need to know the public key. The AuthorizedKeysCommand does not provide a yes/no answer, it has to respond with a list of full public keys (or certificates) that are authorized to connect for a given user id. The SSH server then uses the public key to challenge the client to demonstrate it has the private key.
However, an easy attack in the same ballpark, is to accept the connection without any password or public key auth, and then accept agent forwarding, and ask that agent connection to authorize a connection to a target server, with the user's keys. Never forward your agent connection to an untrusted host. Though -- I imagine this pattern is common when setting up a new host -- trust the first connection, and forward your agent so you can pull resources (like git repos) from the new host to set it up ...
The author essentially bootstraps their servers with a known trusted host key, so that first connection is recognized, instead of having to trust a new and recently generated host key when you first connect.
It's a neat little trick if you're often deploying VPS in shared cloud environments.
How to deploy secrets during bootstrap to a new virtual machine running in the Cloud that does not leave a trace in the infrastructure. And in a way that I can completely automate the deployment.
One answer is providing the secrets in cloudinit - but this leaves a trail on the host/provider's infrastructure, I do not know if those configs I paste into the portal then get saved off somewhere.
The other option (more secure) is having the keys/secrets generated on the host itself at first boot. But then this is difficult to automate as I would need to scrap them (even just the public parts) in a secure way. One option would be to have the public keys printed to the terminal/VNC - but this is much more trouble than it is worth to automate.
I'm not sure on a good solution. This is taking quite an adversarial security model though, assuming the host/provider is not completely trustworthy. Of course not owning the hardware means that the host/provider could be performing other attacks without my knowledge (copying memory, etc.)
You can bootstrap from your custom ISO with some embedded starter key, upload ISO, loads into RAM and opens SSH, connect and run a playbook to encrypt the root drive where you deploy your OS with your SSH key. If you went with encrypted root, you might need to enter pass via console, or dropbear server in a pre-boot env you connect to via SSH to enter the key if you dont trust the console, or setup some custom network unlock mechanism, etc. But once unlocked your provider can still dump your keys from memory. There are also things like AMD SEV-SNP for some more confidential use cases.
Public keys go over untrusted channels. That's why they're public.
I'm not confident you understand how crypto works.
You do realize the entire threat model here is a house of cards perched atop someone else's software hosted on someone else's hardware all of which you implicitly trust and discard in favor of some unlikely cloak and dagger interception scheme.
Public keys can go over channels that an attacker can read. They cannot go over channels that an attacker can modify. (Which would include the SSH connection itself, until such time as you’ve verified the key through a trustworthy channel.)
So you inject a host ssh key pair at VM creation using cloud-init, then connect one time trusting that key and immediately replace the host keys in the VM. So there is still a brief race condition where the attacker could impersonate the VM if he had intercepted your cloud-init file, correct?
Also I don't see anywhere that the script reloads the sshd daemon, which AFAIK is necessary to get it to start using the new host keys and stop using the deleted initial host key.
I've been fighting against this chicken and egg problem in my homelab. I have a step CA SSH CA set up along with automated deployment, bootstrapping, and cert renewals with an ansible playbook. It sets StrictHostKeyChecking yes in the ssh_config for my domain so I'm happily protected against a very unlikely attacker that has a foothold in my network. Theoretically, it means I should never have to worry about host keys again.
Unfortunately, ansible does everything over.... SSH. So if I spin up a new VM or host, I have to manually trust the certificate for the first connection, which is the whole point of this article. I always have console access so I can log in and check the pub key.
There are various ways around this, including the authors suggestion with cloud init. None are very helpful for new physical hosts, though. I'm leaning towards a feature step CA supports that lets hosts authenticate themselves with an X509 cert, which you can easily get with ACME. It's so easy that you could even do it over console on a new physical host.
What I really wish is that there were something like the ACME TLS-ALPN challenge but for SSH servers. They can already present a self signed certificate, so it would just be a matter of connecting all the plumbing.
True. But it means you have to get it into the installer/image which just doesn't seem worth it for something that, as you point out, doesn't happen all that frequently.
> inject a temporary SSH host (private) key via cloud-init...
Reading the comments here I'm tempted to believe that if cloud-init is available and if we consider Heztner (and OVH etc.) provides a secure access to cloud-init (i.e. the box running cloud-init is really the box you think it is), then there are many different ways to solve this problem.
I've been thinking of setting up a simple server that publishes the public keys at a known endpoints. You point an A record for one of your subdomains at the machine and it can provision a TLS cert. Then you can be reasonably confident your connection hasn't been MitM (assuming your trust your cert provider) when you query for those public keys.
The one presented in the article has fewer moving parts though. I'm also curious what ideas are bouncing around your head if you're willing to share.
I was about to downvote this for being obviously false, but after some research this does appear to be true, because ssh uses some channel binding mechanism to prevent your public key authentication from being replayed/reused by the "man" in the middle.
This is one of those situations where it's necessary to be very precise about the security properties.
Specifically, if you bind authentication to the connection, then an attacker who impersonates the server (in this case because it's the first connection, but in other settings because they have a fake certificate), then client authentication is not portable to another connection, so the attacker can't mount a classic MITM attack. However -- and this is a big however -- that doesn't mean that there aren't serious security problems. For example:
* If you use SSH to copy a secret such as an API key to the server, then the attacker still knows the API key.
* If you download some file (e.g., a script) from the server and then trust it, the attacker can use that to provide a malicious script.
>* If you use SSH to copy a secret such as an API key to the server, then the attacker still knows the API key.
That's much harder to pull off though, because you need to replicate the environment close enough so that the victim doesn't suspect anything. Do they put their config files in /var/lib or random docker volumes? Do they use docker compose or docker-compose, etc.
If you know its their first connection to a fresh VPS and assume they haven't used a web-based display to set up anything yet, you just need to guess their install image, which is probably off-the-shelf.
Basically, the client signs the shared key obtained through Diffie-Hellman key exchange, which then gets verified by the server. This ensures that the client and the server have the same shared key, hence no man-in-the-middle.
So this is a tangent on a thought i had after reading the title, but it might be a cool idea that I'll not have the time to do anything with so feel free to use it:
Human checkable fingerprints for pubkeys/hashes dont really work. None of the schemes i've seen hold up under somebody willing to spend compute to get a near-enough collision to fool most people most of the time.
But we can take those random bits and transform them and feed them into a seeded image generation LLM, and then have a person remember/compare the deterministic output image.
You might even make the case its the perfect machine to create memorable-2-human image artifact from random data.
There are a bunch of methods for transforming a hash into something that's easier to compare. You've probably already seen the RandomArt thing that openssh uses for comparing host keys on first use. Some apps produce a sequence of emoji for fingerprint comparison. It's a small but fertile little research niche.
I can't offhand think of anything that an LLM image generator would do to improve the process; it'd be an interesting research task. You'd need a way to transform the 256-bit hash into LLM input in a way that would maximize the perceptual difference in generated images. The problem is that it's absolutely critical that two different implementations work the same, which means the spec would need to specify the exact set of model weights to use.
There was ages ago a project called monkeysphere that let you sign the host ssh key with your gpg key and verify it automatically. The downside was that it was very slow.
> The technique appears to be new: I haven't found a proper write-up of this, nor of any other provider-independent solution (but I'd welcome a correction).
To be frank, anyone that serious about security would probably log in via console, generate and retrieve the host key that way. And then any client would have strict verification enabled.
It's kinda the 101 of communication using public keys cryptography. You have to get hold of the public key in a secure manner first (direct contact or attestation by a third party).
Section 3.1 in Bruce Scheiner's Applied Cryptography discuss how to automatically solves MITM. But that's only important for M:N communications (TSL). For 1:1 communications where you can have secure exchange before hand, no need to go that far.
I guess they don't teach public key-enabled SPA secure port knocking anymore to defend ports of jumpboxes. And using jumpboxes and an internal vlan rather than putting control ports on the wild internet, right?
You get a box and someone is sitting on its IP address (in the middle), proxying to the real one, so everything is getting logged. Other comments say that this mitm stops working when you use public key authentication.
> Other comments say that this mitm stops working when you use public key authentication.
It doesn't completely stop working; a MITM can still pretend to be the server, it just can't authenticate to the real server on your behalf. You could be doing all your work in a fake server controlled by the attacker, while the real server sits there untouched.
It does note that it only protects against an attacker "who learns the cloud-init user-data at any point after the script terminates".
If the attacker can get the cloud-init user-data while the script is still running (in the time between sending the cloud-config.yaml and connecting with SSH to the machine) that would still allow MitM, but would require more effort on the attacker's part to leak the cloud-init data.
The point of the script was that leaking the cloud-init data after the script has completed is harmless.
Yes, I'm just saying if you think you've set up a server with Hetzner, but Smersh is able to intercept your first interaction with it and present you a server that you think is the one you created, then it doesn't matter how much you try to harden the compromised server. But if you get MITM later in the the process, the above is the scenario you are worried about.
The box itself is probably fine. It's the path between you and it. In shared infrastructure one compromised hop somewhere upstream is enough and now you're SSHing into the wrong thing without realizing it.
> The technique appears to be new: I haven't found a proper write-up of this, nor of any other provider-independent solution.
Maybe I'm missing something but SSH already has a built-in solution for this, key-certs. Just sign the server key with a private CA key you trust.
SSH has *ANOTHER* built-in solution, in the form of the SSHFP DNS record.
If the DNS record for the host has an SSHFP (SSH FingerPrint) record, SSH will compare it to the retrieved public key(s) and refuse the connection if there is a mismatch. It can be configured to require DNSSec for this, or to only reject if it gets a secure rejection (to prevent DoS).
It works perfectly, has no notable down sides (just add a DNS record when you generate the host's SSH key), and has been around for many years.
It is very insecure unless you use dnssec, isn’t it?
Just means an attacker also needs to mitm DNS if you MITM the host. Not trivial, but depending on setup might not be harder.
I recommend reading the description of the option `VerifyHostKeyDns` in the `ssh_config` man page.
If set to `yes`, you get automatic trust-on-first-use (no user prompt) if you use DNSSec, and you get the current asking-the-user behavior if your DNSSec is broken or you are under attack.
Obviously it's more secure if you use DNSSec, because that way you can reflexively deny any request to manually verify a host key, but it provides value regardless.
Correct. Very insecure unless your client app goes out of its way to perform DnSSEC.
But wait, there's more: SSH config, resolv.conf, DNS RR setup.
A lomg checklist for successful SSHFP deployment:
https://egbert.net/blog/articles/dns-rr-sshfp.html
That site doesn't mention that when DNSSec is absent, the behaviour of SSH is identical to what happens if you hadn't used the SSHFP record at all, except that for unsophisticated attackers it also displays "no matching host key found in DNS".
So even without DNSSec using the SSHFP records is an improvement over not using them because some of the time it tells you for certain you're being interfered with.
There is no situation in which an insecure DNS response is auto-trusted by the SSH client.
Is there something like this for Wireguard?
Wireguard has no key distribution mechanism.
You can use software like Headscale/Tailscale/Netbird on top.
The article describes a method of securely seeding private key data to newly provisioned servers. It's a bit of bootstrapping problem, and not something a PKI can solve.
You usually need some form of trusted communication with a new server until you can give it its real identity, in the form of host names and cryptographic keys. In an enterprise environment this can usually be done with some sort of isolated management or provisioning vlan. In a cloud environment, I've seen all sorts of more or less hacky solutions but since it depends a lot on specific details of your networks and APIs, bespoke solutions are fine.
A big class of attacker is nation state attackers who do not want to risk discovery.
A big way to deter them is to keep remote log files which, if analyzed, will reveal any attack.
For example, if both ssh-client and ssh-server kept a fingerprint of the session key in some append-only logfile, then a later administrator could compare the logfiles to know if an MITM happened.
Suddenly, nation state attackers won't be interested in MITM-ing at all.
Unfortunately it appears openssh doesn't even have an option to create such a logfile!! Why not??
Example of a nation state attacker that risked discovery and got discovered: https://notes.valdikss.org.ru/jabber.ru-mitm/.
How do you know this was a nation state attacker?
Linodes response of 'we have observed no illegal activity and are closing this ticket' is as close as you can get to 'yes, this is a legal wiretap and we aren't allowed to tell you about it'.
Oof. The answers by Hetzner and Linode read as if they were straight out of a dystopian novel.
Couldn’t the MITM ssh server just forward the client’s fingerprint to the legitimate server?
If so, the legitimate server wouldn’t have anything in their logs that would help detect such an attack.
OpenSSH does log other telemetry though.
> Couldn’t the MITM ssh server just forward the client’s fingerprint to the legitimate server?
The client sends not only the public key, but also a signature, and that signature depends on the output from the key exchange, so it's "bound" to the shared keys negotiated between the client and the server. If the MITM server does separate key exchanges with the client (pretending to be the real server) and the server (pretending to be the real client), the signature won't match; if it forwards the key exchange between the real client and the real server, it won't be able to decrypt the packets.
That's the best thing about SSH public key authentication (and HTTPS client certificates): even when MITM can impersonate the server to the client (because the client didn't verify the host key), it can't impersonate the client to the real server.
Let me see if I understand correctly: Client takes its own public key and the server's public key and creates this signature.
MITM can take its public key and the client's public key and send the resulting signature to the server instead of forwarding what it received from the client.
Do pretty much the same exact thing: MITM PK + Server's PK -> Client. Now client has a signature as well. The signatures that client and server have are different but that is OK as long as MITM can see and change all communication.
It has been a while since I went through the details of the protocol, so I must be missing something. What is it?
What you're missing is that, to create a signature, you need to know the private key corresponding to the public key; it's an asymmetric algorithm.
> Client takes its own public key and the server's public key and creates this signature.
According to https://www.rfc-editor.org/rfc/rfc4252#section-7 client takes its own public key, the "session identifier", and a few other things, and creates this signature (using the private key corresponding to that public key). According to https://www.rfc-editor.org/rfc/rfc4253#section-7.2 that "session identifier" is a byproduct of the key exchange.
> MITM can take its public key and the client's public key and send the resulting signature to the server instead of forwarding what it received from the client.
That's not possible, since the MITM doesn't know the client's private key (and using a different public key will be rejected by the server).
> Do pretty much the same exact thing: MITM PK + Server's PK -> Client. Now client has a signature as well. The signatures that client and server have are different but that is OK as long as MITM can see and change all communication.
You're confusing the Diffie-Hellman Key Exchange with the Public Key Authentication Method. When you MITM the key exchange, the shared secrets the client and server have are different (one side has a secret derived from the client and MITM keys, the other side has a secret derived from the MITM and server keys), but that works as long as the MITM can see and change all communication (basically, decrypting it and encrypting it again).
But since the secrets are different, the session identifier is also different. The MITM can't forward the signature from the client since the server will fail to verify it due to the mismatch in the session identifier; the MiTM can't create a new signature with the client public key since it doesn't have the corresponding private key; and the MITM can't create a valid signature with its own public key (and the corresponding private key) since that key won't be in the authorized keys list for that user account in the server.
> Couldn’t the MITM ssh server just forward the client’s fingerprint to the legitimate server?
Fingerprints are derived from the certificates/private keys. Unless I don't understand some basic crypto, or SSH works in some obtuse way, I do not think it would be possible for the MITM attacker to present the server with the true client's fingerprint unless they also had had the client's private key.
To try to add some explanation here: the server doesn't send the fingerprint. The server sends the public key, your computer just shows a fingerprint because it's more human-friendly and sufficiently secure.
If they forward the real key, so it matches the fingerprint, and you use it, they can't MITM the request because they can't read the contents.
> Fingerprints are derived from the certificates/private keys
SSH fingerprints are derived from the public key. https://superuser.com/a/453022
Ah thanks for the explanation. It’s been a long long time since I’ve delved this deeply into the topic.
Because log processing is handled in the kernel/root/system? Is this a trick question?
See also: rsyslogd
I think the idea is the attacker didn't compromise both the local machine and the remote log sink machine. If you want to get really fancy the techniques used in cert revocation logs/blockchains could be used.
Blockchain is completely unnecessary (as it always is; I thought people stopped trying to ram that garbage into everything years ago).
I was answering this question from GP:
> Unfortunately it appears openssh doesn't even have an option to create such a logfile!! Why not??
The answer is because in Linux systems the logging logistics are handled at the system level, just like starting and running openssh itself. The answer to "why not" is because that's the logging system's job, not openssh's.
rsyslogd is one simple and direct way to distribute logs to remote machines.
how would you create REAL write only logs?
syslog > /dev/lpt0 printer?
Forwarding to WORM tape is one way. Of course that still involves network and likely a buffer somewhere.
[dead]
It baffles me why VPS providers don't display the fingerprints on their dashboards or expose them through the API.
A few do. Though a more common way of securing the loop is to take the desired user public key during the build process.
Assuming you trust that the host control panel (or API server) hasn't been hacked, which you are assuming anyway if you trust a host fingerprint given to you that way, that should be as secure. For a small bit of extra assurance, to protect from an extra very unlikely attack, generate a new key pair just for this VM's creation so that you know you aren't connecting to some other VM that happens to have a known public key of yours and you've been redirected to by DNS poisoning.
AFAIK this is not secure. The SSH protocol doesn’t require the server to know the client’s public key for the connection to succeed. Instead, the client actively sends its public key to the server while authenticating, and the server just decides whether to allow a connection or not. OpenSSH will typically match the client’s key against a fixed list of keys, but nothing stops a rogue server from allowing connections from any client key. In fact, OpenSSH itself can be configured to do this using AuthorizedKeysCommand.
It does need to know the public key. The AuthorizedKeysCommand does not provide a yes/no answer, it has to respond with a list of full public keys (or certificates) that are authorized to connect for a given user id. The SSH server then uses the public key to challenge the client to demonstrate it has the private key.
However, an easy attack in the same ballpark, is to accept the connection without any password or public key auth, and then accept agent forwarding, and ask that agent connection to authorize a connection to a target server, with the user's keys. Never forward your agent connection to an untrusted host. Though -- I imagine this pattern is common when setting up a new host -- trust the first connection, and forward your agent so you can pull resources (like git repos) from the new host to set it up ...
The author essentially bootstraps their servers with a known trusted host key, so that first connection is recognized, instead of having to trust a new and recently generated host key when you first connect.
It's a neat little trick if you're often deploying VPS in shared cloud environments.
This is something I have struggled with:
How to deploy secrets during bootstrap to a new virtual machine running in the Cloud that does not leave a trace in the infrastructure. And in a way that I can completely automate the deployment.
One answer is providing the secrets in cloudinit - but this leaves a trail on the host/provider's infrastructure, I do not know if those configs I paste into the portal then get saved off somewhere.
The other option (more secure) is having the keys/secrets generated on the host itself at first boot. But then this is difficult to automate as I would need to scrap them (even just the public parts) in a secure way. One option would be to have the public keys printed to the terminal/VNC - but this is much more trouble than it is worth to automate.
I'm not sure on a good solution. This is taking quite an adversarial security model though, assuming the host/provider is not completely trustworthy. Of course not owning the hardware means that the host/provider could be performing other attacks without my knowledge (copying memory, etc.)
I've been meaning to investigate Nitro Enclaves[1] for exactly this but haven't made the time yet. Have you looked into them and found them lacking?
[1] https://aws.amazon.com/ec2/nitro/nitro-enclaves/
1. Use cloudinit but give it a one time password to download the secrets on first boot.
2. Use certificates and your own CA.
3. Use the virtual serial console for first login.
4. Use cloudinit to add a custom software repo, then use that to install a custom package that does the initial work.
You can bootstrap from your custom ISO with some embedded starter key, upload ISO, loads into RAM and opens SSH, connect and run a playbook to encrypt the root drive where you deploy your OS with your SSH key. If you went with encrypted root, you might need to enter pass via console, or dropbear server in a pre-boot env you connect to via SSH to enter the key if you dont trust the console, or setup some custom network unlock mechanism, etc. But once unlocked your provider can still dump your keys from memory. There are also things like AMD SEV-SNP for some more confidential use cases.
I'm supposed to believe MitM with the same exact keypair is somehow possible? Private keys are never exchanged. Did everybody forget how crypto works?
Yes you implicitly trust the public key on first login.... then just... immediately compare it with what's on your box?
Might as well seal your doors with duct tape to prevent ghosts from entering your home because this is equally effective.
The author of this post understands that private keys are never exchanged. Read it more carefully.
How do you compare? What trusted channel do you use to retrieve the real public key?
Public keys go over untrusted channels. That's why they're public.
I'm not confident you understand how crypto works.
You do realize the entire threat model here is a house of cards perched atop someone else's software hosted on someone else's hardware all of which you implicitly trust and discard in favor of some unlikely cloak and dagger interception scheme.
Public keys can go over channels that an attacker can read. They cannot go over channels that an attacker can modify. (Which would include the SSH connection itself, until such time as you’ve verified the key through a trustworthy channel.)
A public key is useless without the private key. Which the attacker in this unlikely scenario doesn't have.
So you login the first time and they either match, or they don't. If they don't you start over. The end.
Ignore the fact that most people will probably use the box to host a poorly coded vulnerable service anyway.
If you’re being MITM’d, they’ll also match, because you’ll end up connected to an environment of the attacker’s choosing.
> in favor of some unlikely cloak and dagger interception scheme
someone who definitely understands how crypto works, describing the most basic possible MITM
So you inject a host ssh key pair at VM creation using cloud-init, then connect one time trusting that key and immediately replace the host keys in the VM. So there is still a brief race condition where the attacker could impersonate the VM if he had intercepted your cloud-init file, correct?
Also I don't see anywhere that the script reloads the sshd daemon, which AFAIK is necessary to get it to start using the new host keys and stop using the deleted initial host key.
I've been fighting against this chicken and egg problem in my homelab. I have a step CA SSH CA set up along with automated deployment, bootstrapping, and cert renewals with an ansible playbook. It sets StrictHostKeyChecking yes in the ssh_config for my domain so I'm happily protected against a very unlikely attacker that has a foothold in my network. Theoretically, it means I should never have to worry about host keys again.
Unfortunately, ansible does everything over.... SSH. So if I spin up a new VM or host, I have to manually trust the certificate for the first connection, which is the whole point of this article. I always have console access so I can log in and check the pub key.
There are various ways around this, including the authors suggestion with cloud init. None are very helpful for new physical hosts, though. I'm leaning towards a feature step CA supports that lets hosts authenticate themselves with an X509 cert, which you can easily get with ACME. It's so easy that you could even do it over console on a new physical host.
What I really wish is that there were something like the ACME TLS-ALPN challenge but for SSH servers. They can already present a self signed certificate, so it would just be a matter of connecting all the plumbing.
Though the name is cloud-init, you can cloud-init physical hosts. Seems excessive though, how often are you adding new physical hosts to the homelab?
True. But it means you have to get it into the installer/image which just doesn't seem worth it for something that, as you point out, doesn't happen all that frequently.
> inject a temporary SSH host (private) key via cloud-init...
Reading the comments here I'm tempted to believe that if cloud-init is available and if we consider Heztner (and OVH etc.) provides a secure access to cloud-init (i.e. the box running cloud-init is really the box you think it is), then there are many different ways to solve this problem.
I've been thinking of setting up a simple server that publishes the public keys at a known endpoints. You point an A record for one of your subdomains at the machine and it can provision a TLS cert. Then you can be reasonably confident your connection hasn't been MitM (assuming your trust your cert provider) when you query for those public keys.
The one presented in the article has fewer moving parts though. I'm also curious what ideas are bouncing around your head if you're willing to share.
provision the hosts with an SSH CA, use the CA as a trust root in openssh. they are various version out there from the big players.
MitM is not possible if one uses public key authentication.
I was about to downvote this for being obviously false, but after some research this does appear to be true, because ssh uses some channel binding mechanism to prevent your public key authentication from being replayed/reused by the "man" in the middle.
This is one of those situations where it's necessary to be very precise about the security properties.
Specifically, if you bind authentication to the connection, then an attacker who impersonates the server (in this case because it's the first connection, but in other settings because they have a fake certificate), then client authentication is not portable to another connection, so the attacker can't mount a classic MITM attack. However -- and this is a big however -- that doesn't mean that there aren't serious security problems. For example:
* If you use SSH to copy a secret such as an API key to the server, then the attacker still knows the API key.
* If you download some file (e.g., a script) from the server and then trust it, the attacker can use that to provide a malicious script.
>* If you use SSH to copy a secret such as an API key to the server, then the attacker still knows the API key.
That's much harder to pull off though, because you need to replicate the environment close enough so that the victim doesn't suspect anything. Do they put their config files in /var/lib or random docker volumes? Do they use docker compose or docker-compose, etc.
If you know its their first connection to a fresh VPS and assume they haven't used a web-based display to set up anything yet, you just need to guess their install image, which is probably off-the-shelf.
Sure. I'm not saying it's not better to use public key authentication (it is!). Just that it's still possible to have problems.
Basically, the client signs the shared key obtained through Diffie-Hellman key exchange, which then gets verified by the server. This ensures that the client and the server have the same shared key, hence no man-in-the-middle.
So this is a tangent on a thought i had after reading the title, but it might be a cool idea that I'll not have the time to do anything with so feel free to use it:
Human checkable fingerprints for pubkeys/hashes dont really work. None of the schemes i've seen hold up under somebody willing to spend compute to get a near-enough collision to fool most people most of the time.
But we can take those random bits and transform them and feed them into a seeded image generation LLM, and then have a person remember/compare the deterministic output image.
You might even make the case its the perfect machine to create memorable-2-human image artifact from random data.
There are a bunch of methods for transforming a hash into something that's easier to compare. You've probably already seen the RandomArt thing that openssh uses for comparing host keys on first use. Some apps produce a sequence of emoji for fingerprint comparison. It's a small but fertile little research niche.
I can't offhand think of anything that an LLM image generator would do to improve the process; it'd be an interesting research task. You'd need a way to transform the 256-bit hash into LLM input in a way that would maximize the perceptual difference in generated images. The problem is that it's absolutely critical that two different implementations work the same, which means the spec would need to specify the exact set of model weights to use.
There was ages ago a project called monkeysphere that let you sign the host ssh key with your gpg key and verify it automatically. The downside was that it was very slow.
> The technique appears to be new: I haven't found a proper write-up of this, nor of any other provider-independent solution (but I'd welcome a correction).
To be frank, anyone that serious about security would probably log in via console, generate and retrieve the host key that way. And then any client would have strict verification enabled.
It's kinda the 101 of communication using public keys cryptography. You have to get hold of the public key in a secure manner first (direct contact or attestation by a third party).
Section 3.1 in Bruce Scheiner's Applied Cryptography discuss how to automatically solves MITM. But that's only important for M:N communications (TSL). For 1:1 communications where you can have secure exchange before hand, no need to go that far.
Attacker might have MITMed the console.
If they MITMed the console then they have probably MITMed the entire deployment process and now have the one time use key already.
I guess they don't teach public key-enabled SPA secure port knocking anymore to defend ports of jumpboxes. And using jumpboxes and an internal vlan rather than putting control ports on the wild internet, right?
zero touch prod might be something folks should keep as a north star - SSH should be break glass and a last resort if possible.
https://www.usenix.org/conference/srecon19emea/presentation/...
but how real is this threat? you buy a box from hetzner or some VPS provider and its rooted when you try to ssh ?
If you don't assume the network between you and Hetzner is compromised, you might as well just use rsh instead of ssh.
Hope your home internet isnt compromised if it is thats a whole other problem
You get a box and someone is sitting on its IP address (in the middle), proxying to the real one, so everything is getting logged. Other comments say that this mitm stops working when you use public key authentication.
> Other comments say that this mitm stops working when you use public key authentication.
It doesn't completely stop working; a MITM can still pretend to be the server, it just can't authenticate to the real server on your behalf. You could be doing all your work in a fake server controlled by the attacker, while the real server sits there untouched.
If that were the case then I don't think the OP approach works either.
It does note that it only protects against an attacker "who learns the cloud-init user-data at any point after the script terminates".
If the attacker can get the cloud-init user-data while the script is still running (in the time between sending the cloud-config.yaml and connecting with SSH to the machine) that would still allow MitM, but would require more effort on the attacker's part to leak the cloud-init data.
The point of the script was that leaking the cloud-init data after the script has completed is harmless.
Yes, I'm just saying if you think you've set up a server with Hetzner, but Smersh is able to intercept your first interaction with it and present you a server that you think is the one you created, then it doesn't matter how much you try to harden the compromised server. But if you get MITM later in the the process, the above is the scenario you are worried about.
The box itself is probably fine. It's the path between you and it. In shared infrastructure one compromised hop somewhere upstream is enough and now you're SSHing into the wrong thing without realizing it.
[dead]
Seems like way easier way would be sshing for the first time and just typing `sudo reboot`. If VM reboots, it is yours
Or cat-ing some secrets that would be on target machine but not attacker
A mitm does not redirect your ssh login to their machine