When cloud firewalls skip the private network

A stagelet runs the code of one pull request on its own machine. That machine shares a private network with other stagelets, the build workers, the agent host and the gateway host. The agent host holds your cloud credentials. The gateway host lets a visitor reach a stagelet only with a pass from Stagely. So code in one stagelet must not reach the other machines on that network.

On DigitalOcean, AWS, Google Cloud and Azure, the cloud’s firewall also filters traffic inside the private network. The bootstrap modules that stagely setup applies use it there: stagelet machines and build workers accept traffic only from the agent and the gateway. On Hetzner Cloud and Vultr, the cloud’s firewall does not filter that traffic. So on these two clouds, each machine adds its own iptables rules at every boot.

What the cloud firewalls cover

Hetzner’s firewall FAQ asks “Can Firewalls secure traffic to my private Hetzner Cloud Networks?” and answers “Not yet, because we consider the private networks to be ‘secure’” (Hetzner Docs).

Vultr’s docs say that a firewall group applies its rules to the instance’s main network interface. A VPC gives the instance a separate private networking interface. Traffic inside the VPC does not pass the group’s rules.

Stagely still uses both firewalls on the public side. Stagelet machines and build workers sit in a firewall, or a Vultr firewall group, with no rules, so they accept nothing on their public address. The agent host accepts only SSH, from the address ranges that you gave setup. The gateway host accepts ports 80 and 443 from anywhere, and SSH from your ranges.

What an open private network allows

Without host rules, a machine on the network can open a connection to any port that another machine listens on there. For stagelets, this means:

• Code in one stagelet can connect to the ports that another stagelet publishes on its private address. That path does not go through the gateway, so no one checks a pass.
• Code in one stagelet can open connections to the agent host and the gateway host on their private addresses, for example to SSH. The cloud firewall limits SSH to your ranges only on the public side.
• The agent host holds the cloud credential: a read and write token for the Hetzner Cloud project, or a Vultr API key that can act on the whole account.

Pull request code is untrusted. A pull request from a fork deploys only when someone with write access asks, but its code can come from anyone. So the private network needs the same boundary as the public side.

Rules at every boot, before Docker starts

The agent writes the rules into the cloud-init user data of each stagelet machine and build worker that it creates. Setup does the same for the agent host and the gateway host. The rules sit in a bootcmd section. The bootcmd module of cloud-init runs commands “very early in the boot process”, and its module frequency is always (cloud-init docs). So the rules come back after every reboot, and they are in place before Docker starts a container. A golden image does not change this, because the rules come from the user data, not from the image.

The network that setup creates uses the range 10.42.0.0/20. These are the last commands in the bootcmd list of a stagelet machine on that network, with two allowed addresses. They come from the unit test that checks the generated user data:

iptables -C DOCKER-USER -d 10.42.0.0/20 -m conntrack --ctstate NEW -j REJECT 2>/dev/null || iptables -I DOCKER-USER -d 10.42.0.0/20 -m conntrack --ctstate NEW -j REJECT
iptables -N STAGELY-PRIVATE 2>/dev/null || iptables -F STAGELY-PRIVATE
iptables -A STAGELY-PRIVATE -s 10.42.0.2 -j RETURN
iptables -A STAGELY-PRIVATE -s 10.42.0.3 -j RETURN
iptables -A STAGELY-PRIVATE -j REJECT
iptables -C INPUT -s 10.42.0.0/20 -m conntrack --ctstate NEW -j STAGELY-PRIVATE 2>/dev/null || iptables -I INPUT -s 10.42.0.0/20 -m conntrack --ctstate NEW -j STAGELY-PRIVATE
iptables -C DOCKER-USER -s 10.42.0.0/20 -m conntrack --ctstate NEW -j STAGELY-PRIVATE 2>/dev/null || iptables -I DOCKER-USER -s 10.42.0.0/20 -m conntrack --ctstate NEW -j STAGELY-PRIVATE

The STAGELY-PRIVATE chain holds the allow list. An allowed source address returns to the chain that jumped there, and every other address gets a reject. Setup fills the list with the private addresses of the agent host and the gateway host, through -hcloud-network-allow or -vultr-vpc-allow. The jumps match only --ctstate NEW, the first packet of a connection, so replies to connections that the machine opened still pass. Each iptables -C ... || iptables -I ... pair checks for a rule before it inserts it, so a chain never holds the same rule twice.

Why two chains

INPUT sees the packets for the machine’s own ports, such as SSH. It does not see the packets for the ports that Docker publishes. Docker changes their destination to the container’s address in the nat table, so the kernel forwards them. Docker’s docs say that these packets “are diverted before it reaches the INPUT and OUTPUT chains” (Docker docs). Forwarded packets go through Docker’s rules in FORWARD, which jump to DOCKER-USER first. Docker calls that chain a place for user rules “processed before rules in the DOCKER-FORWARD and DOCKER chains” (Docker docs).

So a rule in INPUT alone leaves every published port open to the network. A rule in DOCKER-USER alone leaves SSH open. Both chains jump to the same allow list. The boot commands create DOCKER-USER when Docker has not created it yet, and Docker keeps the rules in that chain when it starts.

Containers stay off the network

The first command in the excerpt works in the other direction. It rejects every new connection from a container to the other machines on the network: other stagelets, build workers, the agent host and the gateway host. It matches on the destination, not on the source. So a container that writes a packet with the agent’s address as its source still gets the reject. The same chain also rejects new connections from containers to the metadata service at 169.254.169.254, which serves the machine’s user data.

This rule works only when container traffic goes through DOCKER-USER. The Compose rules make sure of that. Stagely refuses privileged, network_mode, cap_add, network drivers other than bridge, external networks and the host network. So each container sits on a Docker bridge, and the kernel forwards its traffic through the chain.

The agent host and the gateway host run no containers, and they serve nothing on the network. Setup gives them the same chain with an empty allow list, in INPUT only. So they reject every new connection from the network. Their own connections still work: the agent drives each machine over SSH, and the gateway forwards visitors over HTTP. The replies to these connections are not new, so they pass.

What the rules trust

The rules decide by source address. They trust that a packet on the network carries its sender’s own address. A container cannot break that trust, because its new connections to the network never leave the machine. Root on a machine can change that machine’s rules and choose any source address. Then only the cloud network’s own handling of forged addresses is left, and these rules do not control it. The Compose rules cover this side: they refuse the settings that give a container access to its machine.

The rules also have these limits:

• They cover IPv4 only. The private network that setup creates is IPv4, and the code refuses to write the rules for an IPv6 range or address.
• A Hetzner network holds up to 100 attached resources, servers and load balancers together (Hetzner Docs). With the agent host and the gateway host on it, an environment runs at most 98 stagelet machines and build workers at a time.

What the tests check

Unit tests compare the generated user data with the expected text: the excerpt above for a stagelet machine, and a reject for every new connection from the network on the agent host and the gateway host. They also check that the code refuses IPv6.

The provider tests for Hetzner Cloud and Vultr go further. Each one creates a stagelet machine and a neighbor on one network, and allows only the test’s own address. It runs a Compose project on the stagelet. Then it checks that:

• the service answers on the stagelet’s private address, from the allowed address;
• the neighbor can open connections over the network, so a refused connection proves something;
• the neighbor cannot reach the stagelet’s published port or its SSH port;
• a container on the stagelet gets “connection refused” when it connects to an address on the network, and when it asks the metadata service.

The setup pages for Hetzner Cloud and Vultr list these rules in short. The security model shows how they fit with the other boundaries.