Sign and Verify Container Images with Cosign

Overview#

Part 3 ended with an image that passes a vulnerability gate. Nothing yet connects that result to what a cluster will pull: a tag is a mutable pointer, and ci-demo:1.4.2 can mean a different image tomorrow. A signature fixes that by binding a statement (“I vouch for this”) to the image digest, which cannot change without changing the signature’s subject.

This part signs the image with Cosign using a local key pair against a local registry, verifies it, then deliberately points the tag at the old vulnerable image to show what verification catches. Everything in the executed sections was run with Cosign 3.1.3 and registry:2. Keyless signing with Sigstore’s public infrastructure requires an OIDC identity and network access to Fulcio and Rekor; that section is configuration and is marked as such.

Prerequisites

• Docker, and the ci-demo:1.4.2 and ci-demo:1.4.0 images from part 3
• A local registry: docker network create lab && docker run -d --name registry --network lab -p 5000:5000 registry:2
• Cosign from its image; a shell function is defined below

Push, and find the digest#

Push the fixed image to the local registry and read back what the registry stored:

docker tag ci-demo:1.4.2 localhost:5000/ci-demo:1.4.2
docker push localhost:5000/ci-demo:1.4.2
docker inspect --format '{{index .RepoDigests 0}}' localhost:5000/ci-demo:1.4.2

localhost:5000/ci-demo@sha256:5743684a366b5d6e91df18c41f9f3a9ceab8452d62b2c3fea07015552ce1b636

The registry agrees, which is the point: the digest is computed from the manifest, not assigned by a client.

curl -sI -H 'Accept: application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json' \
  localhost:5000/v2/ci-demo/manifests/1.4.2 | grep -i docker-content-digest

Docker-Content-Digest: sha256:5743684a366b5d6e91df18c41f9f3a9ceab8452d62b2c3fea07015552ce1b636

Save it; every command from here uses the digest, never the tag:

DIGEST=sha256:5743684a366b5d6e91df18c41f9f3a9ceab8452d62b2c3fea07015552ce1b636

Sign with a key pair#

Cosign runs from its image on the same Docker network as the registry, so it reaches the registry as registry:5000. The private key is protected by a passphrase; for the lab it is empty.

cosign() { docker run --rm --network lab -e COSIGN_PASSWORD="" -e COSIGN_YES=true \
  -v "$PWD:/work" -w /work ghcr.io/sigstore/cosign/cosign:v3.1.3 "$@"; }

cosign generate-key-pair

Private key written to cosign.key
Public key written to cosign.pub

Cosign 3 signs according to a signing config that names the transparency log, certificate authority and timestamp authority to use. The default config points at Sigstore’s public services, which a private-key, local-registry setup has no use for, so create one that names none of them:

cosign signing-config create \
  --no-default-rekor --no-default-fulcio --no-default-oidc --no-default-tsa \
  --out signing-config.json
cat signing-config.json

{"mediaType":"application/vnd.dev.sigstore.signingconfig.v0.2+json", "rekorTlogConfig":{}, "tsaConfig":{}}

Sign by digest. --allow-http-registry is needed only because the lab registry has no TLS:

cosign sign --key cosign.key --signing-config signing-config.json --allow-http-registry \
  "registry:5000/ci-demo@$DIGEST"

Signing artifact...
Pushing signature to: registry:5000/ci-demo

The signature is stored in the registry, next to the image, as an OCI artifact that references the image digest. The registry now has one more tag, derived from the digest:

curl -s localhost:5000/v2/ci-demo/tags/list

{"name":"ci-demo","tags":["1.4.2","sha256-5743684a366b5d6e91df18c41f9f3a9ceab8452d62b2c3fea07015552ce1b636"]}

Nothing about the image changed. The signature is a separate object that says “the holder of cosign.key signed digest 5743684a…”.

Verify#

Verification needs the public key and the same reference. --insecure-ignore-tlog tells Cosign not to expect a transparency-log entry, which matches the signing config; it prints a warning every time, which is appropriate.

cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog "registry:5000/ci-demo@$DIGEST"

Verification for registry:5000/ci-demo@sha256:5743684a366b5d6e91df18c41f9f3a9ceab8452d62b2c3fea07015552ce1b636 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The signatures were verified against the specified public key

[{"critical":{"identity":{"docker-reference":"registry:5000/ci-demo@sha256:5743684a…"},"image":{"docker-manifest-digest":"sha256:5743684a366b5d6e91df18c41f9f3a9ceab8452d62b2c3fea07015552ce1b636"},"type":"https://sigstore.dev/cosign/sign/v1"},"optional":{}}]

The JSON at the end is the signed payload: the digest, the reference it was signed under, and the claim type. Exit code 0. Verifying by tag (registry:5000/ci-demo:1.4.2) also passes right now, because the tag resolves to the signed digest. Right now.

Move the tag#

Push the vulnerable image from part 2 (ci-demo:1.4.0, 183 findings) over the same tag:

docker tag ci-demo:1.4.0 localhost:5000/ci-demo:1.4.2
docker push localhost:5000/ci-demo:1.4.2
curl -sI -H 'Accept: application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json' \
  localhost:5000/v2/ci-demo/manifests/1.4.2 | grep -i docker-content-digest

Docker-Content-Digest: sha256:1f5c38e53aba6b41fdc24c6186cde3c0d76d80d4ba75fe48e427cd3859b2e72f

Same tag, different digest. Anyone pulling ci-demo:1.4.2 now gets the image with 183 findings, and nothing in the pull tells them. Verification does:

cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog registry:5000/ci-demo:1.4.2
echo "exit=$?"

Error: no signatures found
exit=10

And the signed digest is still exactly as trustworthy as it was:

cosign verify --key cosign.pub --allow-http-registry --insecure-ignore-tlog "registry:5000/ci-demo@$DIGEST"
echo "exit=$?"

exit=0

That pair of results is the entire argument for signing. The tag lied; the digest could not. Restore the tag before moving on (docker tag ci-demo:1.4.2 localhost:5000/ci-demo:1.4.2 && docker push …).

Keyless signing and identity-based verification#

A private key has to live somewhere, and wherever it lives is where an attacker goes to sign their own image. Sigstore’s keyless mode removes the long-lived key: the signer authenticates with an OIDC identity (a CI job’s identity token, or a person’s account), Fulcio issues a short-lived certificate binding that identity to an ephemeral key, the signature is made, and the certificate plus signature are recorded in the Rekor transparency log. The key expires in minutes; the log entry is permanent and public.

Verification then checks who signed rather than which key:

# Configuration only: requires network access to Sigstore's public services and a real OIDC identity.
cosign verify \
  --certificate-identity-regexp '^https://gitlab.com/example-group/ci-demo//' \
  --certificate-oidc-issuer https://gitlab.com \
  registry.example.com/ci-demo@sha256:…

The identity is the CI project path and the issuer is the platform, so a signature from a different project, or from a developer laptop, fails verification even though it is a valid Sigstore signature. In a GitLab job the id_tokens keyword provides the OIDC token (SIGSTORE_ID_TOKEN) and cosign sign needs no key at all. This was not run here, because it requires the public Sigstore services and a GitLab project identity; the flags are from the Cosign 3.1 documentation and the public Sigstore signing config.

The choice between the two modes is a choice about where trust lives. Key pairs are simple and work offline; the key is the thing to protect (a KMS-held key, cosign sign --key awskms://…, keeps it out of the pipeline entirely). Keyless ties trust to the CI platform’s identity and to a public log, which is stronger evidence and a harder dependency.

Where the verify step goes#

A signature is only a control if something refuses to run unsigned images. Three places, in increasing strength:

1. The deploy job. cosign verify … || exit 1 before kubectl set image, with the deploy using the digest the verify step resolved. Cheap, and it covers the pipeline’s own deployments.
2. Admission control. A policy in the cluster that rejects any pod whose image is not signed by the expected key or identity. Kyverno’s verifyImages rule and Sigstore’s policy-controller both do this, and both also mutate the image reference to the digest so a later tag move cannot help. The Secure Kubernetes path covers Kyverno admission policies.
3. Registry policy. Some registries can refuse to serve unsigned images or enforce signatures at pull time; this depends on the product.

Do the first today; it takes one line. The second is what makes the control hold for images that did not come through your pipeline.

Security Considerations#

• cosign.key in this lab has an empty passphrase and lives in the working directory. In a pipeline the key belongs in a KMS or a secret store the signing job can use but not read, or you use keyless.
• --insecure-ignore-tlog and the empty signing config are correct for an air-gapped registry with a private key and a private log or none. With the public Sigstore instance, keep the transparency log on: it is what lets anyone detect a signature made with a stolen key.
• A signature proves who signed, not that the image is safe. Part 3’s scan and part 5’s attestations say what was checked; the signature says the check applies to this digest.

Troubleshooting#

Conclusion#

Signing turned “the image called 1.4.2” into “the image with digest 5743684a…, vouched for by this key”, and a moved tag went from silent to a failed check. The signature says who; it does not yet say what was checked. Part 5 attaches the SBOM and the idea of build provenance to the same digest so the verifier can ask both.

References#

• Cosign documentation
• Cosign: signing with a key pair
• Sigstore: keyless signing
• Sigstore signing configuration
• GitLab: Cosign keyless signing with CI id_tokens
• Kyverno: verify images