Instance Metadata Service Risks

October 8, 2023

Many cloud providers offer an ‘instance metadata’ service. Each refers to it with a different name but I’ll just call it IMDS hereafter. The IMDS is a HTTP interface exposed to VM instances which provides information about the instance itself, and often temporary credentials that can be used to interact with cloud APIs. In Google Cloud, these credentials belong to the service account assigned to the instance and can be used to perform any action the service account can. In AWS, the credentials belong to the instance profile and can perform any action that the assigned role can perform.

An example of how this could be consumed is by a log shipping agent. The agent could fetch temporary credentials from the IMDS and use them to upload logs to a cloud storage bucket.

IMDS credentials are a convenient way to grant applications running on instances access to the cloud environment. It eliminates the need to store static, long-lived credentials on the instance itself. But it also introduces risks.

In 2019, Capital One was hacked using Server-Side Request Forgery to direct requests to the IMDS and gain access to STS credentials. Although cloud providers have now mitigated this attack vector, fundamentally, there are still risks. While static credentials on disk can be protected by file permissions such that only specific users/processes can access them, IMDS credentials can be accessed by any process on the instance, save for ones heavily sandboxed by systemd.

It is possible to use firewalling to restrict access to the IMDS to specific UIDs but this still isn’t ideal because it’s not fine-grained. What if you have a service that needs ordinary metadata but not credentials?

imds-filterd is a a project which aims to grant fine-grained access to the IMDS by UID/GID. Unfortunately, it only works on FreeBSD because it uses a kernel-specific feature to see which UID connected to its TCP socket.

Furthermore, AWS and Google Cloud’s metadata services only support one role/service account per instance. If you have multiple services that require different permissions on an instance, they’ll have to share, which is poor security practice. A service may get access to something it doesn’t need and audit logs will be less useful. There’s no such limitation with static file-sourced credentials.

These days I’m leaning away from using IMDS credentials. From an attacker’s perspective, it’s easier to forge a request than to gain root access and read a credentials file. I’ve been trying to come up with some kind of drop-in “fix” like imds-filterd that will work under Linux but had no success yet.

EC2’s Instance Identity Document

Another little-known feature of the IMDS in AWS and Google Cloud is an endpoint that returns a payload containing information about the instance along with a signature by the cloud provider. AWS calls this Instance Identity Document. Google Cloud calls it VM identity.

I encountered this feature while reading about SPIRE which uses the signed document for host attestation. The idea is that the signed document can be used to prove that the instance is running in a cloud environment and that it is the instance it claims to be. This is useful for establishing trust between services running on different instances.

HashiCorp Vault provides the ec2 and gcp auth methods, enabling instances to authenticate with their signed instance identity documents.

I noticed a worrying flaw in the implementation of this feature in AWS. The Instance Identity Document does not contain an issue or expiry time, nor a nonce. A signed document is valid forever and can be reused as many times as you like! If you have a service using these documents for authentication, you’ve got a permanent door key unless extra steps are taken to validate it.

{
  "devpayProductCodes" : null,
  "marketplaceProductCodes" : [ "1abc2defghijklm3nopqrs4tu" ], 
  "availabilityZone" : "us-west-2b",
  "privateIp" : "10.158.112.84",
  "version" : "2017-09-30",
  "instanceId" : "i-1234567890abcdef0",
  "billingProducts" : null,
  "instanceType" : "t2.micro",
  "accountId" : "123456789012",
  "imageId" : "ami-5fb8c835",
  "pendingTime" : "2016-11-19T16:32:11Z",
  "architecture" : "x86_64",
  "kernelId" : null,
  "ramdiskId" : null,
  "region" : "us-west-2"
}

HashiCorp Vault does take a number of extra steps to mitigate this risk. For example, after verifying the signature, it checks that the instance described in a document it receives is actually running. The pendingTime field in the payload (which describes the last boot time) is checked to ensure it never goes backward but according to Vault’s comments in code, this field only updates if the instances is stopped and started again, and not if it’s rebooted. Doesn’t help much if this happens rarely.

// When the presented client nonce does not match the cached entry, it
// is either that a rogue client is trying to login or that a valid
// client suffered a migration. The migration is detected via
// pendingTime in the instance metadata, which sadly is only updated
// when an instance is stopped and started but *not* when the instance
// is rebooted. If reboot survivability is needed, either
// instrumentation to delete the instance ID from the access list is
// necessary, or the client must durably store the nonce.

Vault introduces a nonce of its own that must be sent after the initial authentication. Even this mitigation is flawed because the nonce is never rotated so it only needs to be captured once. Maybe a disgruntled employee grabs it on their last day.

It’s also concerning that Vault has AWS’s certificates hardcoded. If AWS’s private keys ever leak, the only fix is to update Vault.

SPIRE’s code implements fewer mitigations.

I raised my concern regarding the lack of issue/expiry time in the payload to AWS’s security team and got a nonchalant WONTFIX/NOTABUG response. ¯\_(ツ)_/¯

Hello,

Thank you for bringing your security concern to our attention. We greatly appreciate and encourage reports from the security community worldwide.

We do not believe the behavior you describe in this report presents a security concern, rather, it is expected behavior.

If you discover or become aware of other concerns specific to AWS products and services, please do not hesitate to contact us again at aws-security@amazon.com

Best regards,
[REDACTED]
AWS Security
https://aws.amazon.com/security

Google Cloud does things properly and uses JWTs, which includes these fields. The expiry is 1 hour after the issue time.

{
  "iss": "[TOKEN_ISSUER]",
  "iat": "[ISSUED_TIME]",
  "exp": "[EXPIRED_TIME]",
  "aud": "[AUDIENCE]",
  "sub": "[SUBJECT]",
  "azp": "[AUTHORIZED_PARTY]",
  "google": {
    "compute_engine": {
      "project_id": "[PROJECT_ID]",
      "project_number": "[PROJECT_NUMBER]",
      "zone": "[ZONE]",
      "instance_id": "[INSTANCE_ID]",
      "instance_name": "[INSTANCE_NAME]",
      "instance_creation_timestamp": "[CREATION_TIMESTAMP]",
      "instance_confidentiality": "[INSTANCE_CONFIDENTIALITY]",
      "license_id": [
        "[LICENSE_1]",
        ...
        "[LICENSE_N]"
      ]
    }
  }
}