Nobody encrypts container images

A container image is a manifest pointing at a stack of gzipped tarballs. The registry serves those blobs by digest to anyone with pull access, and tar -x does the rest. The tooling to encrypt the layers has existed for years and almost nobody uses it. Maybe that’s for a reason?

I’m working with a company that doesn’t trust whoever runs the registry with the bytes in it. So I went and found out how far encrypting the layers gets you: how an encrypted layer works, what the tooling does today, and what it costs to run. The short version is that it’s cheap to add and expensive to operate.

But first the problem. Things like this are very easy:

$ cek cat localhost:5001/acme/example:1.0 /app/config.yaml
api_key: sk-live-4eC39HqLyjWDarjtT1zdp7dc

That’s cek, by the way, the container exploration kit: a CLI I wrote to look inside images without running them. docker save and tar -x get you the same file in four commands, but cek does it in one, so I’ll be leaning on it throughout this post. If you want to follow along: brew install cek, or go install github.com/bschaatsbergen/cek@latest.

pull access is read access

Whenever I bring this up, the answer is “our registry is private”. It is, in the sense that you need credentials to pull from it. But if that’s all that protects what’s in your images, that isn’t really defense in depth, is it?

• 2016: Vine’s entire source code, API keys included, sat in a Docker registry that was reachable from the internet, one docker pull away.
• 2021: Codecov’s Bash Uploader was backdoored with a credential lifted out of a Docker image layer, exposing up to 29,000 customers.
• 2023: RWTH Aachen analyzed 337,171 images from Docker Hub and 8,076 private registries and found secrets in 8.5% of them, including 52,107 private keys.

And it’s rarely just an API key. Licensed binaries, certificates, migration scripts with the internal hostnames still in them: most of us have written a COPY . . at some point, and not always with a clear picture of everything that dot picks up. And there are plenty of security researchers and engineers out there who can reverse engineer a lot from an image alone.

So who can read it? Anyone who can reach the blobs, and that’s a longer list than the people you gave pull access to:

That last one I ran into myself:

So why does nobody encrypt? There was no tooling until recently, “our registry is private” ends most conversations, and frankly, when I talk about this, most people aren’t aware it’s possible at all. Would your CISO know?

signing is not encryption

cosign proves who built an image and that nobody changed it since. It does nothing about who can read it. A signature answers whether this is the image you meant to run, not who gets to see what’s inside. Different questions, different keys, and the order you apply them in matters, which I’ll get to.

layers are just blobs

The image from the start, as the registry sees it:

$ cek inspect localhost:5001/acme/example:1.0
...
Layers:
#  Digest                                                                   Size    Media Type
1  sha256:3f26bc2dec0b515f1c2818f6e13a8f1da1f88179a008445d4e587233386bff78  3.9 MB  application/vnd.oci.image.layer.v1.tar+gzip
2  sha256:fe29c73d75d248ca2ee5d78cddc35e10e8d482993169c473788eecf59602b8bb  293 B   application/vnd.oci.image.layer.v1.tar+gzip
3  sha256:f63565c21574e32c8241f0e970a3266977d083b58f31e355252c9e8e2244ff09  5.0 MB  application/vnd.oci.image.layer.v1.tar+gzip
4  sha256:818c5761af2389ff952e9c3c9f7556ad7639ae91c267b0aa68c7bdd0df34b5e8  5.0 MB  application/vnd.oci.image.layer.v1.tar+gzip
5  sha256:b482cc4abecc63f5dc10be9947663f48ec68ac181c2fc32a8aa8dcd8aa9a2481  1.3 MB  application/vnd.oci.image.layer.v1.tar+gzip

Layer 1 is alpine, the rest is one COPY per file. The 293 byte one is the config with the API key in it.

A registry only does content addressing: it stores a blob of bytes under the digest of those bytes and hands it back when asked for that digest. Whether the bytes are plaintext or ciphertext is not its concern, they’re just bytes.

So nothing stops you from encrypting a layer before it goes up, and the registry will store it like any other.

That is what ocicrypt does, one layer at a time:

The layer is encrypted with a fresh random AES-256 key. The mode is counter mode, which turns AES into a stream cipher, so the ciphertext is exactly as long as the plaintext, and a keyed hash over the ciphertext, an HMAC, lets the reader catch tampering before decrypting anything.

That key, together with its nonce and the digest of the plaintext, is then itself encrypted with the reader’s RSA public key, so only the holder of the matching private key can recover it. The result is packaged as a JWE, a standard JSON format for an encrypted payload plus the wrapped key and the algorithms used, and stored as an annotation on the layer’s entry in the manifest, under org.opencontainers.image.enc.keys.jwe.

The media type gets an +encrypted suffix so a runtime knows not to untar it. The image config still lists the digests of the plaintext layers, the diff IDs, so a decrypted layer has the same identity it had before.

The spec change that defines those media types and annotations was opened in April 2019 and is still open. Seven years in review is apparently long enough that everyone just shipped it:

encrypt

Using ocicrypt, it’s roughly this, errors dropped:

cc, _ := config.EncryptWithJwe([][]byte{rsaPub})

for _, layer := range layers[1:] { // skip alpine
	plain, _ := layer.Compressed()
	enc, finalize, _ := ocicrypt.EncryptLayer(cc.EncryptConfig, plain, desc)

	ciphertext, _ := io.ReadAll(enc)
	annotations, _ := finalize()
}

EncryptLayer returns a reader that encrypts as you drain it. finalize has to run after that, because the plaintext digest and the HMAC are only known at the end: the digest gets sealed into the wrapped key, the HMAC goes in a public annotation. The ciphertext becomes the new layer and the annotations go on its descriptor.

You don’t have to write that. skopeo does it with a flag on copy, and buildah and podman have the same flag on push. I build to a local OCI layout first and let skopeo encrypt while it pushes, so the plaintext never touches the registry:

$ docker buildx build -o type=oci,dest=build,tar=false .
$ skopeo copy --encryption-key jwe:keys/rsa.pub \
    --encrypt-layer 1 --encrypt-layer 2 --encrypt-layer 3 --encrypt-layer 4 \
    oci:build \
    docker://localhost:5001/acme/example:1.0-enc

Note the --encrypt-layer flags: without them skopeo encrypts everything, and I’m skipping layer 0, which is alpine. Nothing in it is secret, and leaving it in plaintext keeps it deduplicated with every other alpine layer in the registry and on the node, which matters later.

Don’t feed skopeo docker-daemon: as the source though. The daemon hands out uncompressed layers, skopeo compresses them on the way and seals the digest of the uncompressed bytes into the JWE, so anything that checks that digest after decrypting rejects the layer.

$ cek inspect localhost:5001/acme/example:1.0-enc
Image: localhost:5001/acme/example:1.0-enc
Digest: sha256:9cfbf34a1f294a77af32fe6e9ba963519a9df8828d454a9654b4a154010e4e2f
...
Entrypoint: /app/example
Env:
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Layers:
#  Digest                                                                   Size    Media Type
1  sha256:3f26bc2dec0b515f1c2818f6e13a8f1da1f88179a008445d4e587233386bff78  3.9 MB  application/vnd.oci.image.layer.v1.tar+gzip
2  sha256:e8e8af8d9f5736237ec34d6102510298acb1b357803ae21b37af7154492fc249  293 B   application/vnd.oci.image.layer.v1.tar+gzip+encrypted
3  sha256:d9b7f61bb3b549be30dfb912babf5a04c0c6bb25f2f7c872ee00bdc245f258c2  5.0 MB  application/vnd.oci.image.layer.v1.tar+gzip+encrypted
4  sha256:26ed23043bf81cadcf14c7e6c221ad51231aa9c188c4f81364900db24c1749c3  5.0 MB  application/vnd.oci.image.layer.v1.tar+gzip+encrypted
5  sha256:ef731b892f8cce957b5b29bf8b7e0ae6d4bbf5ba33053dabc549b6285abd6a09  1.3 MB  application/vnd.oci.image.layer.v1.tar+gzip+encrypted

Layer annotations:
#  Key                                    Value
2  org.opencontainers.image.enc.keys.jwe  eyJwcm90ZWN0ZWQiOiJleUpoYkdjaU9pSlNVMEV0VDBGRlVDSXNJbVZ1WXlJ...
2  org.opencontainers.image.enc.pubopts   eyJjaXBoZXIiOiJBRVNfMjU2X0NUUl9ITUFDX1NIQTI1NiIsImhtYWMiOiJk...
3  org.opencontainers.image.enc.keys.jwe  eyJwcm90ZWN0ZWQiOiJleUpoYkdjaU9pSlNVMEV0VDBGRlVDSXNJbVZ1WXlJ...
3  org.opencontainers.image.enc.pubopts   eyJjaXBoZXIiOiJBRVNfMjU2X0NUUl9ITUFDX1NIQTI1NiIsImhtYWMiOiJC...
...

Layer 1 kept its digest. Layers 2 to 5 have new digests, a new media type and two annotations each. The JWE decodes to a header, a wrapped key and a payload:

$ cek manifest localhost:5001/acme/example:1.0-enc \
    | jq -r '.layers[] | select(.annotations) | "\(.digest)  " +
      (.annotations["org.opencontainers.image.enc.keys.jwe"] | @base64d | fromjson | .protected | @base64d)'
sha256:e8e8af8d9f5736237ec34d6102510298acb1b357803ae21b37af7154492fc249  {"alg":"RSA-OAEP","enc":"A256GCM"}
sha256:d9b7f61bb3b549be30dfb912babf5a04c0c6bb25f2f7c872ee00bdc245f258c2  {"alg":"RSA-OAEP","enc":"A256GCM"}
sha256:26ed23043bf81cadcf14c7e6c221ad51231aa9c188c4f81364900db24c1749c3  {"alg":"RSA-OAEP","enc":"A256GCM"}
sha256:ef731b892f8cce957b5b29bf8b7e0ae6d4bbf5ba33053dabc549b6285abd6a09  {"alg":"RSA-OAEP","enc":"A256GCM"}

Four layers, four JWEs, and sort -u over their encrypted_key fields gives four distinct values, so each layer really has its own key. The payload behind that header holds the layer’s AES key, its nonce and the plaintext digest, and only the matching RSA private key opens it. The other annotation is public and says which cipher did the layer:

$ cek manifest localhost:5001/acme/example:1.0-enc \
    | jq -r '.layers[2].annotations["org.opencontainers.image.enc.pubopts"] | @base64d'
{"cipher":"AES_256_CTR_HMAC_SHA256","hmac":"BwyRKHznnr+KDQjzjx8Ui3aMQ4wityprelGqb2cCS1k=","cipheroptions":{}}

the registry doesn’t care, Docker does

The push went through without a complaint. The registry stored the encrypted image exactly the way it stored the plaintext one, so look at the bytes:

$ cek blob --layer 3 localhost:5001/acme/example:1.0 | head -c 32 | xxd
00000000: 1f8b 0800 0000 0000 00ff e4b1 4390 3050  ............C.0P
00000010: 9765 9bb6 6ddb b66d dbb6 6ddb b66d dbce  .e..m..m..m..m..

$ cek blob --layer 3 localhost:5001/acme/example:1.0-enc | head -c 32 | xxd
00000000: 611d 6af3 ed6c ec60 8fe5 73da ea95 2c2d  a.j..l.`..s...,-
00000010: 5646 f432 3540 cd53 a864 6c71 b457 3a24  VF.25@.S.dlq.W:$

1f8b is the gzip magic number. The second one is noise, and cek gives up on it:

$ cek cat localhost:5001/acme/example:1.0-enc /app/config.yaml
Error: failed to read layers: layer 2: failed to read tar header: unexpected EOF

Docker gives up too, with a better error:

$ docker pull localhost:5001/acme/example:1.0-enc
...
failed to get stream processor for application/vnd.oci.image.layer.v1.tar+gzip+encrypted: exec: "ctd-decoder": executable file not found in $PATH

Docker Desktop pulls through containerd, and containerd’s default config maps +encrypted to a stream processor called ctd-decoder. That binary comes from imgcrypt, and it isn’t installed, which is the correct failure.

One trap when you try this yourself: the image config carries the plaintext diff IDs, and Docker matches cached layers by diff ID. If the node pulled the plaintext image earlier, the encrypted one “works” from cache and you’ll think it decrypted something. docker rmi the plaintext image first, and know that with the containerd image store even that isn’t always enough: the unpacked layers can outlive the image until garbage collection gets to them.

decrypt, then run

cc, _ := config.DecryptWithPrivKeys([][]byte{rsaKey}, [][]byte{{}})

for _, desc := range manifest.Layers {
	ciphertext, _ := layer.Compressed()
	plain, _, _ := ocicrypt.DecryptLayer(cc.DecryptConfig, ciphertext, desc, false)

	// sha256(plain) == the digest sealed in the JWE
}

The private key unwraps each layer key, each layer decrypts on its own, and the plaintext digest that was sealed into the JWE at encryption time is what you check each result against.

At runtime nobody runs a tool for this. The node decrypts while it unpacks. containerd’s default config already routes the encrypted media types to a stream processor, ctd-decoder from imgcrypt, and tells it where the node keeps its keys. This is straight from containerd config default on 2.2:

[stream_processors.'io.containerd.ocicrypt.decoder.v1.tar.gzip']
  accepts = ['application/vnd.oci.image.layer.v1.tar+gzip+encrypted']
  returns = 'application/vnd.oci.image.layer.v1.tar+gzip'
  path = 'ctd-decoder'
  args = ['--decryption-keys-path', '/etc/containerd/ocicrypt/keys']

[plugins.'io.containerd.cri.v1.images'.image_decryption]
  key_model = 'node'

So a stock containerd node needs the binary on its PATH and a key in that directory, nothing else. The node key model is what makes the kubelet’s pulls go through the decoder with the node’s keys, so the pod spec doesn’t change at all: the kubelet pulls, containerd decrypts each layer as it unpacks it, the container starts. Docker on its own can’t do this, which is the empty cell in the table above.

encrypt, then sign

cosign signs the manifest digest. Encryption changes every layer digest, so the manifest digest changes with it, and a signature over the plaintext image says nothing about the encrypted one, so you encrypt first and sign second.

$ cosign sign --key keys/cosign.key --use-signing-config=false --tlog-upload=false -y \
    localhost:5001/acme/example:1.0-enc
$ cosign verify --key keys/cosign.pub --insecure-ignore-tlog \
    localhost:5001/acme/example:1.0-enc
$ cek tags localhost:5001/acme/example
1.0-enc
1.0
sha256-9cfbf34a1f294a77af32fe6e9ba963519a9df8828d454a9654b4a154010e4e2f
...

The extra flags keep cosign away from the public Sigstore services: no transparency log, no signing config fetched. The sha256- tag is the signature: an OCI artifact in the same registry whose subject is the digest it covers, and that digest is the encrypted manifest’s. Signing proves who built it, encryption decides who can read it.

where the key comes from

ocicrypt has five ways to wrap a layer key. Four take a key you hold:

The demo uses JWE with an RSA key in a file because it’s the easiest to show, and the worst way to run it: now there’s a private key file on every node that pulls, and a private key file on every node is the problem this post opened with, one level up.

The fifth is the one I think is worth running: a key provider, reached over gRPC or as a local binary. Instead of holding a key, the node asks a service to unwrap the layer key, and the service decides. The demo has a toy provider that stands in for a KMS and only releases keys when it considers the caller attested, which here is a flag rather than real evidence. With the flag off every unwrap comes back as attestation failed: key release denied and the pull stops at the first layer. Flip it, and the same image decrypts on the same node. The only thing that changed is what the provider believed about the caller. Attest, release the key, decrypt, start the container: layers that only open on a machine that can prove what it booted, which is the version of this worth running.

That same decoder on the node is where a key provider gets called instead of a key file, which puts the key service in the path of every pod start.

what’s still plaintext

Only layers are encrypted. The manifest and the image config are not, which is why cek inspect on the encrypted image still printed the entrypoint, the environment, the layer count and every layer’s size. The API key in config.yaml is gone, the one you put in ENV is not, and a 5 MB layer is a 5 MB layer whatever is in it.

what it costs

Image encryption is cheap to add and expensive to operate. Encrypting was a couple of hundred lines of Go, or a flag in skopeo, and everything after that is the cost.

Layer dedupe and caching break. Every encryption run draws new keys, so the same layer encrypted twice is two different blobs. Registries dedupe by digest, so a layer shared by twenty images is stored twenty times, and a rebuild that changed nothing still pushes new bytes. Leaving base layers in plaintext, as the demo does, is the easiest relief.

Every pull is slower and costs CPU. Every node decrypts and HMACs every encrypted layer on every fresh pull. On an autoscaler that’s a lot of nodes doing the same work at the same time.

You built a hard dependency on a KMS. A node that needs a key service to start a pod has that service in the path of every scale-up and every node replacement. A KMS outage now means no pod that uses one of these images can start.

Runtime support is uneven. Docker can’t pull these, CRI-O decrypts but can’t encrypt, and every scanner, admission controller and local dev setup has to learn a media type it has never seen.

key management is the actual problem

ocicrypt doesn’t solve key management, it moves it.

Distribution. Which nodes get which keys, and how the first key reaches a fresh node without that key becoming the credential someone copies instead.

Rotation. A new reader key means unwrapping and re-wrapping every layer key it covered, which is a new manifest for every image you’ve ever encrypted. If the old key leaked, the layer keys it could open are burnt too, so that’s a re-encrypt and re-push of everything.

Revocation. You can’t take a key back from a node that had it. That node decrypted the layers onto its disk. Revoking stops the next pull, not the last one.

wrapping up

Pull access is read access, and a login in front of a blob store doesn’t change what’s in the blobs. Encrypting the layers instead works today, with ocicrypt, skopeo, podman and containerd, and what it leaves you with is key management, which is the real problem and yours to solve.

If the contents are the product, say model weights or a licensed binary, and you have nodes that can attest, encrypt the layers and release the keys on attestation. If the problem is an API key in config.yaml, don’t encrypt the image, take the key out.

Working notes. If something here is wrong, tell me.