Registry tokens that cannot push

Stagely runs two kinds of machines in your cloud account. A build worker gets the full source of a commit, builds its images and pushes them to your registry. A stagelet machine pulls those images and runs the code of a pull request. It never needs to push.

The customer agent (stagely-agent) keeps your cloud credentials on the agent host, and they stay there. For each job, the agent gets fresh registry credentials for each machine that the job uses. This post shows how it does this on five clouds, and why it cannot on the sixth.

One function for each job

The agent gets registry credentials through a function of this type. On five clouds, the cloud’s provider in the agent supplies it (agent.go):

// RegistryCredentials returns the user name and password that a machine's
// Docker signs in to host with for one job: credentials that can push for a
// build worker, and pull only, where the cloud allows it, for a stagelet
// machine. Machines run untrusted code, so a cloud should hand out
// short-lived credentials that reach only the registry.
type RegistryCredentials func(ctx context.Context, host string, push bool) (username, password string, err error)

The agent calls it before each build, each deploy and each reset. push is true only for a build worker. The agent then signs the machine’s Docker in with docker login --password-stdin, so the password is not on a command line. It also hides the user name and the password in the job’s output.

A build worker runs the Dockerfiles of each pull request. So after each build, the agent signs the worker out, and the next build does not get a login that can push. If the cleanup fails, the agent deletes the worker.

The Compose rules refuse settings that give a container access to its machine. The agent does not rely on them alone. A credential on a machine should be worth little if it leaks. It should expire soon and reach only the registry. On a stagelet machine, it should only pull.

DigitalOcean and Vultr: ask the API

DigitalOcean and Vultr each have an API call that returns Docker credentials for a registry in the account. The agent sends the same two parameters to both (DigitalOcean, Vultr):

q := url.Values{"read_write": {strconv.FormatBool(readWrite)}, "expiry_seconds": {strconv.Itoa(int(expiry.Seconds()))}}

read_write is true only for a build worker. When it is false, DigitalOcean’s credentials are read-only and “cannot be used to push images”. expiry_seconds is 7200, so the credentials expire after two hours, which leaves room for a long build before its push.

The agent must set the expiry. On both clouds, credentials without one never expire (DigitalOcean, Vultr). The DigitalOcean API token and the Vultr API key stay on the agent host. On Vultr, the registry’s root user stays off the machines too.

AWS: a narrow session, then an ECR token

Amazon ECR signs Docker in with an authorization token for the user AWS. The GetAuthorizationToken call takes no permission parameter. The token’s “permission scope matches that of the IAM principal used to retrieve the authentication token” (ECR user guide). A token from the agent’s own key would carry all of the agent’s ECR permissions, push included.

So the agent first makes a narrower principal. It calls STS GetFederationToken with a session policy (aws.go):

name, actions := "stagely-registry-pull", registryPullActions
if push {
	name, actions = "stagely-registry-push", slices.Concat(registryPullActions, registryPushActions)
}
repositories := "arn:aws:ecr:" + region + ":" + account + ":repository/*"
if prefix != "" {
	repositories = "arn:aws:ecr:" + region + ":" + account + ":repository/" + prefix + "/*"
}

The policy allows ecr:GetAuthorizationToken, and the three pull actions on the repositories under the prefix. For a build worker, it also allows the push actions and ecr:CreateRepository there. The prefix is the environment’s name, unless you set -registry-prefix. AWS gives the session “the intersection of the IAM user policies and the session policies that you pass” (GetFederationToken).

The agent then calls GetAuthorizationToken with the session’s temporary credentials. The token can pull under the prefix, and it can push only for a build worker. It is valid for 12 hours, and the session lasts 12 hours to match. The agent checks the host before and after it asks. Only the account’s own registry, <account>.dkr.ecr.<region>.amazonaws.com in the agent’s region, gets a token.

This design needs an IAM user. AWS says: “You must call the GetFederationToken operation using the long-term security credentials of an IAM user.” Role sessions cannot: AWS’s comparison of STS credentials says that credentials from AssumeRole “Cannot call GetFederationToken or GetSessionToken.” So the agent needs a long-term access key. With temporary credentials in AWS_SESSION_TOKEN, the first build fails at the registry sign-in.

Google Cloud: a push account and a pull account

stagely setup gcp creates two service accounts for the environment. One may push to the Artifact Registry repository. The other may only pull (main.tf):

resource "google_artifact_registry_repository_iam_member" "pull" {
  location   = google_artifact_registry_repository.images.location
  repository = google_artifact_registry_repository.images.repository_id
  role       = "roles/artifactregistry.reader"
  member     = google_service_account.registry_pull.member
}

The agent’s service account gets roles/iam.serviceAccountTokenCreator on both accounts, so it can make their tokens. Before each job, the agent calls generateAccessToken for the push account or the pull account. The machine signs in to Artifact Registry as oauth2accesstoken with that token.

The token lasts one hour, the longest that Google allows by default. An organization policy can raise the limit to 12 hours, but the agent still asks for one hour. The service account key in GOOGLE_CREDENTIALS stays on the agent host.

Azure: a new token for each job

Azure Container Registry can sign Docker in with a token, and each token has one scope map. Two system scope maps fit: _repositories_pull “Can pull any repository of the registry”, and _repositories_push “Can push to any repository of the registry” (token-based repository permissions). Setup creates one registry for the environment, with its admin user turned off. So these scope maps reach only the environment’s own registry.

Before each job, the agent creates a token with one of the two scope maps, and a password for it that expires after two hours:

scope := "_repositories_pull"
if push {
	scope = "_repositories_push"
}

The agent makes a new token for each job because a token has only two passwords, password1 and password2. A new password replaces the old one with the same name. Jobs run at the same time, so with one shared token, a new job could replace the password of a job that still uses it. The agent names its tokens stagely-job-<random>, and it deletes the ones whose passwords have expired. The service principal stays on the agent host.

The six clouds side by side

Hetzner Cloud: the exception

Hetzner Cloud has no container registry, so setup creates none. You bring your own, such as Docker Hub, GitHub Container Registry, GitLab’s registry or CNCF Distribution on your own server. Setup writes your user name and password into the agent’s environment file. The agent wraps them in StaticCredentials (agent.go):

// StaticCredentials returns the same user name and password for every job.
func StaticCredentials(username, password string) RegistryCredentials {
	return func(context.Context, string, bool) (string, string, error) {
		return username, password, nil
	}
}

The function ignores push. Every machine signs in with the same password, and this includes stagelet machines. A build worker must push, so the password can push. A copy of it is on each machine that runs pull request code, and Stagely does not make it expire. The Hetzner API token still stays on the agent host.

The same is true on any cloud when you start the agent with -registry-username. The static credentials then replace the cloud’s own.

So scope the registry token yourself. Give it push and pull access only under the -registry prefix, and nothing else. Then a leaked copy reaches only the images under that prefix.

Where to read more

The security model lists what stays on the agent host. Set up Stagely in your cloud account lists the credentials that setup needs. Each cloud page has a registry section: DigitalOcean, AWS, Google Cloud, Azure, Hetzner Cloud and Vultr.