Initial connection security in the cloud

So you’ve started your first virtual machine in the cloud somewhere, and you’re ready to make something people will love. You just need to put it out there, so you open up your terminal, type in the magic letters ssh and some IP address, and suddenly you’re in some cypherpunk dystopia and have to wrangle with the moral dilemma of whether to trust an arbitrary sequence of hexadecimal integers and wonder if your dog will ever look at you the same if you just close your eyes and type yes, just this one time

What did you just risk? Does using modern technology always come with such pain in the beginning, that eases as you learn that this is the way it’s done? Are there any tools that could help you use the cloud that doesn’t end up with you questioning whether you are indeed a good, trustworthy person, deep down? I’m no therapist, so while I might not be able to provide solace, I hope I can provide some understanding, and a bumpy path forward.

Risk

What risk do we run when we accept untrusted ssh connections to cloud servers? If you are using password authentication over untrusted connections with ssh, it is possible for a man-in-the-middle (MitM) to snoop on everything sent over the connection to your server. They’ll answer your connection as if they were the server, just presenting their own ssh public key, and you’ll accept it since you don’t know the actual public key of your server. The attacker will receive your password, use that to authenticate to your actual server, and transparently connect the two of you, listening to everything that is sent. They also now probably have root access to your server and can install whatever backdoor they desire.

However, if you are using public key authentication, you are no longer sending any secrets over the wire, and this attack does no longer work, but the attacker is left with one more trick. Instead of snooping on the connection to the actual server, the attacker can now just claim they are the actual server. Since for an initial connection you probably just created the server, the attacker only needs to present you with a seemingly fresh box and you’ll be none the wiser that this isn’t the actual server you just created. Chances are that you’ll have sent something of value, like TLS certificates, or database credentials, or maybe you configured a new user with your password, before you notice that something is wrong. Maybe you’ll just write it off as a somewhat expected failure of cloud services, networks are best-effort services anyway, and create a new box, and this time everything worked. Oh well, probably all right then. Depending on what the attacker was able to get from you with their fake server, you might have fully compromised your organization. Maybe the attackers server was given access to your puppet/salt/chef master, and now has access to all your configuration data and secrets. Maybe the attacker now has access to your intranet and less-than-perfectly-protected internal services.

Two things should be clear by now, we do need a way to tell that a server we’re connecting to is the server we expected, and we should really not use password authentication, especially to untrusted servers. The latter is usually easy, since most cloud services will not enable password authentication for ssh and require you to configure a public ssh key on the box before you start it. Thus our primary threat is that the server we’re connecting to is not the server we just created. If you use tools like terraform or salt-cloud or something similar to automatically provision your cloud services, you might be at even greater risk, since you’ll be creating resources more often, and every new connection is another chance for an attacker to impersonate an actual server. And if you haven’t guessed it yet, none of the common cross-provider cloud tools like terraform, libcloud, salt-cloud and probably more, verify the initial connection. They mostly trust that the first connection is not tampered with, and will save the public key presented for later connections. This is the common policy for ssh, known as TOFU (trust on first use).

Quoting Raymond Hettinger, “there must be a better way!”.

A way forward

In an ideal scenario we’ll get the public key from our cloud provider before connecting to the server, enabling us to validate that the connection is indeed solid before we hand over the keys to our kingdoms. As a backup plan, if we can put a something only we know on the server before we connect to it, and verify that it’s there before we send anything else, we can protect ourselves from sending anything of value to the attacker in the event that we connected to the wrong server. The latter can be accomplised by using cloud-init, a common tool for running a script or adding services when a server first boots. If we create a nonce locally before creating the new server, add a cloud-init configuration to place that nonce in a known location on the filesystem, we can verify after connecting that the file does indeed contain our nonce before we do anything else. If it’s there, we reached the correct server, and future connections can trust the public key that was presented. If it was not, odds are you have an attacker on your network and you have just connected to their server. Abort the connection, remove the public key you accepted from your ssh known hosts, and talk to your security team.

So where does the common cloud providers stand when it comes to these capabilities?

Provider capabilities

To break down what we’ve learned so far, we can establish a trusted connection to a cloud server only if we can either get the ssh public key from the provider, OR if they provide cloud-init or similar tools for the initial connection and only accept connections using public keys. So what are our options?

Checking the docs for a couple of common providers yields the following capability table:

¹ Using the get-console-output API (docs)

² Unless you’ve added one of your ssh keys to the instance when creating it, Digital Ocean will email you a password you can use to connect as root. Never use this feature.

³ Called startup scripts

⁴ Called StackScripts. Since there’s no way to upload your ssh public key to the instance before it starts you also have to use StackScripts to add your public key to the instance.

In other words, there is hope! While AWS is the only provider that enables getting the server public key programmatically (as far as I’m able to tell), all the others do provide a way to run something similar to cloud-init. Linode doesn’t make secure connections easy, but you can also ssh into the box through their ssh gateways, for which you can get their key fingerprints, but that requires authenticating with your account password instead of ssh public keys, and then connecting with the root password for your box from the gateway.

From this small bit of research it seems it is indeed possible to securely connect to cloud servers, but like mentioned earlier, most of the tools to automate cloud provisioning does not utilize these capabilities. But we now know what has to be done, go make some pull requests!