Stop MITM on the first SSH connection, on any VPS or cloud provider

This little script stops attacks on the first SSH connection to a new VM, even on providers (like Hetzner Cloud) that don't offer a proprietary solution; we only need cloud-init, which is widely supported.

Summary (for experts; read on for a longer explanation): inject a temporary SSH host (private) key via cloud-init, and trust that temporary SSH host key just long enough to generate and retrieve the "real" (long-term) SSH host keys. The script is a simple but hardened implementation of this technique; the comments in the script discuss implementation choices. The technique appears to be new: I haven't found a proper write-up of this, nor of any other provider-independent solution (but I'd welcome a correction).

This technique actually protects the first connection, whereas just answering "yes" when ssh asks "The authenticity of host [...] can't be established" (i.e. Trust On First Use) leaves you open to an attacker rerouting your traffic to a proxy, or to an attacker generously deciding to provide your VM (... for now).

This technique also makes leaks of the cloud-init userdata harmless. Injecting a long-term SSH host (private) key via cloud-init does allow you to authenticate the first connection (by adding the public part of the injected key to ~/.ssh/known_hosts), but leaves valuable (private) key material in the cloud-init userdata, where an attacker can often obtain it from

• the metadata service, which any process on the VM can typically read; e.g. on a Hetzner VM:

  $ curl http://169.254.169.254/hetzner/v1/userdata
  #cloud-config
  ssh_keys:
    ecdsa_private: |
      -----BEGIN OPENSSH PRIVATE KEY-----
      [...]
      -----END OPENSSH PRIVATE KEY-----
    ecdsa_public: ecdsa-sha2-nistp256 AAAAE2Vj[...]tI= temporary host key for [...]
  

  An attacker can often trick some process into divulging this data via SSRF (which is often not blocked, even where the provider does offer a solution); or from
• the provider's (other) systems (e.g. Hetzner explicitly warns against storing "passwords or other sensitive information"); or from
• your administrator workstation.

Security analysis / threat model

Throughout, we trust the (Open)SSH protocol and implementation, and we do not rely on you, the administrator, detecting the attack.

We are secure against a network attacker

We protect

• the integrity of the administrator's workstation, and
• the VM

against an attacker

• who has full control over the network ("man-in-the-middle"), and
• who learns the cloud-init user-data at any point after the script terminates (successfully or not),

because the attacker never learns any key material at a time when it is still valuable.

To prevent accidental use of the temporary SSH host key, the script keeps it in a temporary directory; the temporary SSH host key is never in ~/.ssh/known_hosts.

Hacking the administrator workstation still doesn't get an attacker the (long-term) SSH host (private) key

We protect

• (only) the VM, including its (long-term) SSH host (private) key

against an attacker

• who has full control over the network ("man-in-the-middle"), and
• who has full control over the administrator workstation, but
• who does not actually connect to the VM (in a real-life scenario, this is presumably proven by logging),

because the (long-term) SSH host (private) key was never on the administrator workstation, and because the attacker does not actually connect to the VM.

(An attacker who does connect to the VM will likely be able to learn the SSH host key, e.g. via ssh root@<VM> cat /etc/ssh/ssh_host_*.)

Hacking the VM and/or provider still doesn't get an attacker the administrator's workstation

We protect

• (only) the integrity of the administrator's workstation

against an attacker

• who has full control over the network ("man-in-the-middle"), and
• who has full control over the VM and/or provider

because we assume (Open)SSH is secure.

As an additional safeguard for this scenario, the script does not just write output from the VM to ~/.ssh/known_hosts, but relies on OpenSSH key rotation to place the long-term SSH host keys there, which

• prevents the compromised host from feeding malicious data into ssh's known_hosts parser, and
• ensures that we only write keys to ~/.ssh/known_hosts that the VM actually controls, and
• ensures that we correctly handle OpenSSH options like HashKnownHosts (and any relevant options that might be added in the future).

Aside: ... but can an attacker really make the network attack work?

It depends. In particular, the attacker likely fails if you actually detect that all your connections are, and always have been, to the wrong machine, and cannot be convinced to enter a password (on the first or on any later connection), and don't configure ssh to forward an agent or X11 connection.

As a simplified non-exhaustive list, with thanks to ssh-mitm,

• the attacker (likely) succeeds if they can fool you by giving you access to an attacker-controlled machine (rather than to the real target host); otherwise,
• the attacker succeeds if they can fool you into giving up information that lets them log in to the real host, i.e.
  • if you log in (to the attacker's machine) with a password; or
  • if you log in with any authentication method, then enter a password when prompted; or
  • if you log in with any authentication method, and forward an ssh-agent connection; otherwise,
• the attacker likely fails: they need access to the real host to fool you, but can't log in to the real host using your input.

The attacker may instead/additionally succeed at attacking your workstation if you use any authentication method and forward an X11 connection.