Keep containers away from cloud metadata

A stagelet machine runs the code of a pull request in containers, and a build worker runs its build steps in containers. Every cloud VM can reach a metadata service at the link-local address 169.254.169.254. This post shows what a container could get from that service, and how Stagely blocks it on all six supported clouds.

What the metadata service gives away

The metadata service answers any request that comes from the machine. A container on a Docker bridge network sends its packets through the host, and the host routes them to 169.254.169.254 like any other address.

The service describes the machine, such as its addresses and SSH public keys. On five of the six clouds, it also serves the machine’s user data. Azure is the exception (see below).

Stagely’s user data holds a secret. When the customer agent (stagely-agent) creates a machine, it generates an Ed25519 host key and puts it in the user data. Cloud-init makes it the machine’s only SSH host key. The agent keeps the public half and connects only to a machine that presents that key.

So the host key tells the agent that it talks to the right machine. Over that SSH connection, the agent signs Docker in to the registry and writes the stagelet’s secret values to /opt/stagelet/secrets.env. A build worker also gets the commit’s source that way. The key alone does not open the machine. But an attacker who holds the key, and who can also get between the agent and the machine, could pose as the machine and receive all of that. Build workers stay up while stagelets come and go, so a stolen build worker key stays useful while that worker runs.

A header check does not help. Google Cloud’s metadata server requires the header Metadata-Flavor: Google, and Azure’s requires Metadata: true. Code in a container can send any header.

One rule in DOCKER-USER

On every machine that runs containers, the user data ends with these lines. They come from a test of the cloudinit package:

bootcmd:
  - iptables -N DOCKER-USER 2>/dev/null || true
  - iptables -C DOCKER-USER -d 169.254.169.254 -m conntrack --ctstate NEW -j REJECT 2>/dev/null || iptables -I DOCKER-USER -d 169.254.169.254 -m conntrack --ctstate NEW -j REJECT

• The first line creates the DOCKER-USER chain if Docker has not created it yet.
• The second line checks for the rule with -C, and inserts it with -I only when it is missing. A second run adds nothing.
• The rule matches packets that open a new connection to 169.254.169.254. REJECT sends an ICMP error back, so the client in the container fails at once instead of waiting for a timeout.

Cloud-init runs bootcmd early in every boot. So each boot adds the rule again, before Docker starts, and no container runs without it. Docker keeps the rules that it finds in DOCKER-USER when it starts.

Why DOCKER-USER and not INPUT

The iptables filter table has three built-in chains. INPUT sees packets for the host’s own sockets. FORWARD sees packets that the host routes. OUTPUT sees packets that the host sends itself.

A packet from a container to 169.254.169.254 is not for the host. The host routes it, so the packet passes FORWARD. A rule in INPUT never sees it.

Docker manages FORWARD. It adds rules there that jump to DOCKER-USER first, and Docker documents DOCKER-USER as the place for rules that run before its own. A rule appended to FORWARD runs after Docker’s rules, which can accept the packet first.

A rule in OUTPUT would block the wrong sender. It would cut off the host itself, and the host still needs these services. On Azure, for example, the VM agent must reach the platform on ports 80 and 32526.

DOCKER-USER covers only containers whose packets the host forwards. A container in the host’s network namespace would skip it. The Compose subset refuses that case: Stagely rejects network_mode, a network named host, network: host in a build, and build entitlements. A Dockerfile RUN --network=host needs the network.host entitlement, so a build step cannot skip the rule either.

AWS: no token for a container

AWS adds a second control. With IMDSv2, a client first sends a PUT to /latest/api/token, then sends the token with each GET. When tokens are required, a request without a valid token gets 401 - Unauthorized.

The PUT response carries an IP hop limit, which is its time to live. Stagely sets the limit to 1. The response reaches the host, but the host cannot forward it to a container. AWS says why: “going to the container is considered an additional network hop.” So a container gets no token, and without a token it gets no metadata.

The agent sets both options when it creates an instance (api.go):

// IMDSv2 with a hop limit of 1: containers, one hop further, get no
// metadata token, so they can't read the user data and its host key.
"MetadataOptions.HttpTokens":              {"required"},
"MetadataOptions.HttpPutResponseHopLimit": {"1"},

The bootstrap module of stagely setup aws sets the same metadata_options on the agent host and the gateway host. So on AWS, either control alone stops the read. See AWS setup.

Azure: the wireserver too

Azure has a second platform address, 168.63.129.16, which Microsoft calls the wireserver. The Azure VM agent uses it to report that the VM is ready and to get the VM’s deployment configuration. The same address is also the VM’s DNS server.

Containers have no reason to talk to it, except for DNS. So the Azure provider blocks one more address (azure.go):

var wireserver = cloudinit.HostBlock{Address: "168.63.129.16", ExceptPort: 53}

For an address with an exception, metadataRules writes one rule for TCP and one for UDP, such as:

DOCKER-USER -d 168.63.129.16 -p tcp ! --dport 53 -m conntrack --ctstate NEW -j REJECT

Port 53 stays open for build steps. Containers on Docker’s default bridge get a copy of the host’s /etc/resolv.conf, so they query the VM’s DNS server directly. Containers on a Compose network use Docker’s embedded DNS server, which forwards lookups to the host’s DNS servers.

Azure does not serve custom data, where Stagely puts the host key, through its Instance Metadata Service. The rule for 169.254.169.254 stays on Azure anyway. That service still describes the VM, and it hands out tokens for any managed identity that the VM has. See Azure setup.

Google Cloud: keep DNS open

On Google Cloud, the metadata server and the DNS server share one address. Google requires each VM to use 169.254.169.254 as its name server. A plain REJECT would break name lookups in build steps.

So the Google Cloud provider writes its own rules (gcp.go):

metadataReject = "DOCKER-USER -d 169.254.169.254 -m conntrack --ctstate NEW -j REJECT"
dnsReturnUDP   = "DOCKER-USER -d 169.254.169.254 -p udp --dport 53 -j RETURN"
dnsReturnTCP   = "DOCKER-USER -d 169.254.169.254 -p tcp --dport 53 -j RETURN"

Each command inserts its rule at the top of the chain. The RETURN rules go in after the REJECT, so they end up above it. A DNS packet leaves DOCKER-USER before it meets the REJECT, and Docker’s own rules handle it from there. Every other new connection to the metadata server fails. See Google Cloud setup.

DigitalOcean, Hetzner Cloud and Vultr

DigitalOcean gets the plain rule from the start of this post. Hetzner Cloud and Vultr get it too, with one more block. Their firewalls do not filter private networks, so the same chain also rejects new connections from containers to the whole range of the private network.

No cloud credentials behind it

A metadata service also gives a machine the credentials of its cloud identity. On AWS, an application reads the credentials of the instance’s IAM role from iam/security-credentials/. On Google Cloud, the service accounts of an instance generate access tokens that the metadata server serves. On Azure, IMDS hands out tokens for the VM’s managed identities.

Stagely attaches none of these. The agent’s RunInstances request names no instance profile. Its Compute Engine request names no service account. Its Azure VM request has no identity. On DigitalOcean, Hetzner Cloud and Vultr, the API token stays on the agent host. Machines get only registry credentials, and the agent hands those over SSH. So a container that got past the rules would still find no cloud API credentials there. The security model lists where each credential lives.