Rollback and Recovery: Versioned Artifacts, Rollout Undo and the Database Problem

Overview#

A rollback is a deployment of a version you already trust. It is only cheap if that version still exists as exactly the bytes that were running before (an immutable artifact, an image digest) and if the thing that changed is the application and not its data. This part walks through a real failed rollout on a kind cluster, reverses it with kubectl rollout undo, and then deals with the part undo cannot touch: the database.

The outputs below are from the web Deployment built in part 3, on kind 0.31 with Kubernetes 1.35.

Prerequisites

• The kind cluster and deploy-v1.yaml from part 3, applied and rolled out
• kubectl 1.32 or newer

What makes rollback possible#

Three properties, established before anything goes wrong:

Versioned, immutable artifacts. The archive from part 1 is named after its commit and kept for 30 days. Rolling back is re-running the deploy job with the previous name. If the artifact had been overwritten, or expired, or built again from the old commit, there would be nothing to roll back to, only something new that hopefully behaves like the old one.

Tags that do not move, or better, digests. A container tag is a pointer. nginx:1.29.1-alpine can be re-published tomorrow with different contents; nginx@sha256:… cannot. What was actually running is recorded on the pod:

kubectl get pods -l app=web -o jsonpath='{.items[0].status.containerStatuses[0].imageID}{"\n"}'

docker.io/library/nginx@sha256:42a516af16b852e33b7682d5ef8acbd5d13fe08fecadc7ed98605ba5e3b26ab8

That digest is the only honest description of the previous version. A deploy job that sets image to a digest (kubectl set image deployment/web web=nginx@sha256:42a516…) makes every revision in the rollout history unambiguous, and a rollback to it is a rollback to those bytes.

A definition of “failed”. A rollout that is still “in progress” after twenty minutes is a failure nobody declared. The next section gives the rollout a deadline.

Detecting a failed rollout#

Roll out a version that cannot start: an image tag that does not exist stands in for any new version that crashes or never becomes ready.

kubectl set image deployment/web web=nginx:1.29.1-alpine-doesnotexist
kubectl annotate deployment/web kubernetes.io/change-cause="web: bad tag" --overwrite
kubectl rollout status deployment/web --timeout=45s; echo "exit=$?"

deployment.apps/web image updated
Waiting for deployment "web" rollout to finish: 1 out of 3 new replicas have been updated...
error: timed out waiting for the condition
exit=1

rollout status waited 45 seconds, saw no progress and exited 1. In a pipeline that non-zero exit fails the deploy job, which is the detection mechanism: the deploy job is the rollout, and it is red. Look at what the cluster did in the meantime:

kubectl get deploy web
kubectl get pods -l app=web

NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    3/3     1            3           119s

NAME                   READY   STATUS         RESTARTS   AGE
web-5484757784-6ldr2   1/1     Running        0          79s
web-5484757784-8r7pn   1/1     Running        0          50s
web-5484757784-gplzx   1/1     Running        0          47s
web-85dcbfc44-zzwfg    0/1     ErrImagePull   0          46s

Because part 3 set maxUnavailable: 0, the three old pods are still serving. The controller created one new pod (maxSurge: 1), it failed to pull, and the rollout stopped there: users saw nothing. This is the single most valuable property of a well-configured rolling update, and it is why the readiness probe and maxUnavailable: 0 were not optional.

The Deployment’s conditions tell the same story from the controller’s side:

kubectl get deploy web -o jsonpath='{range .status.conditions[*]}{.type}={.status} ({.reason}){"\n"}{end}'

Available=True (MinimumReplicasAvailable)
Progressing=True (ReplicaSetUpdated)

Progressing is still True because the controller has not given up yet. It gives up after spec.progressDeadlineSeconds, which defaults to 600:

kubectl get deploy web -o jsonpath='{.spec.progressDeadlineSeconds}{"\n"}'

600

After that deadline the condition becomes Progressing=False with reason ProgressDeadlineExceeded, and rollout status exits non-zero immediately instead of waiting for its own --timeout. Ten minutes is long for a service that should be ready in thirty seconds; setting progressDeadlineSeconds: 120 in the manifest makes the controller declare failure sooner. (This run used the 45-second --timeout on rollout status instead of waiting for the deadline, so the ProgressDeadlineExceeded transition itself was not observed.)

Rolling back with what you have#

Every rollout creates a ReplicaSet, and the Deployment keeps the last revisionHistoryLimit (default 10) of them. rollout history lists them with the change-cause annotation, which is why the deploy job should always set it:

kubectl rollout history deployment/web

REVISION  CHANGE-CAUSE
1         <none>
2         web: nginx 1.29.1-alpine
3         web: bad tag

rollout undo returns to the previous revision:

kubectl rollout undo deployment/web
kubectl rollout status deployment/web --timeout=120s
kubectl rollout history deployment/web
kubectl get deploy web -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

deployment.apps/web rolled back
Waiting for deployment "web" rollout to finish: 1 old replicas are pending termination...
deployment "web" successfully rolled out
REVISION  CHANGE-CAUSE
1         <none>
3         web: bad tag
4         web: nginx 1.29.1-alpine

nginx:1.29.1-alpine

Two details of that output matter. The rollback finished in seconds, because the old ReplicaSet still existed and its image was already on the node: nothing was built, nothing was pulled. And revision 2 is gone, replaced by revision 4 with the same template: a rollback is a new rollout of an old template, and the history reflects that. Rolling back to a specific point works the same way:

kubectl rollout undo deployment/web --to-revision=1
kubectl rollout status deployment/web --timeout=120s

deployment.apps/web rolled back
deployment "web" successfully rolled out

Revision 1 (nginx:1.27.5-alpine) became revision 5. rollout history deployment/web --revision=4 prints the full pod template of a revision, which is how you confirm what a number refers to before you undo to it.

Verify after rolling back#

A rollback is a deployment and gets the same checks as one:

1. rollout status returned 0 (above).
2. The running image is the one you meant: the jsonpath for .spec.template.spec.containers[0].image above, or the pod imageID for the digest.
3. The service answers correctly. The probe pod from part 3 (curl -sI http://web) is the minimum; the smoke test the pipeline runs after a deploy is the real one.
4. The metric that triggered the rollback recovers. If it does not, the deployment was not the cause, and the next step is investigation, not another rollback.

Rollback criteria#

Decide before the incident. A workable rule: roll back when a deployment is followed by a user-visible failure (error rate, latency, a failed smoke test) and the deployment is the most recent change, without first attempting to diagnose. Diagnosis happens afterwards, on the old version, with the new one’s logs and the failed pods still available (kubectl get pods --show-labels lists the failed ReplicaSet’s pods until it is scaled to zero). The cost of an unnecessary rollback is a redeploy; the cost of a slow one is the outage.

The database problem#

rollout undo reverses the application. It does nothing to the database, and the database is where rollback actually goes wrong. If version 2 ran a migration that renamed a column, version 1 does not know the new name; the rollback “succeeds” and every request fails.

The way out is to never make a schema change the old version cannot live with, so that application rollback never needs data rollback. The pattern is expand/contract:

1. Expand. Add the new column, table or index. Do not remove or rename anything. Deploy this migration on its own; the current application ignores the addition.
2. Deploy the application that writes to both old and new (or reads new with a fallback to old). If it fails, roll it back: the expanded schema is still compatible with the previous version.
3. Backfill existing rows, in batches, as a job, not as part of a deploy.
4. Contract. Only after the new version has been stable for as long as you would ever roll back to (a release or two), remove the old column in a separate migration.

Two consequences follow. Migrations run as their own step (a Job, a pipeline stage before the deploy), never in the container’s start-up, where three replicas starting at once would race to run it. And a “rollback” of the database is not a thing you plan to do; a restore is, and it is a different procedure with a different owner, tested on a schedule, that loses every write since the backup. Keep application rollback frequent and boring, and database restore rare and rehearsed.

This section describes the pattern; no migration was run in this environment.

Recovery thinking#

Rollback is one recovery. The others are worth naming, because the pipeline should make each of them a deploy:

• Roll forward. The fix is small and obvious, and rolling back would lose something (a feature flag, a dependency bump that closed a vulnerability). Ship the fix through the same pipeline, with the same gates.
• Scale. The new version is fine and the load is not. kubectl scale, or the autoscaler, is the recovery.
• Kill switch. The new code path is behind a flag; turn it off. This is the fastest recovery there is, and it requires having built the flag before the incident.
• Restore. Data is wrong or gone. This is the slow path, and the only one where practising matters more than tooling.

Whichever it is, the deploy job that makes it happen should be the ordinary one: same artifact store, same credentials, same audit trail. An emergency procedure that bypasses the pipeline is a second, untested pipeline.

Security Considerations#

• The rollback path uses the same credentials as deploy. Do not give on-call a broader role “for emergencies”; give the pipeline a rollback job with the deploy job’s scope.
• Old ReplicaSets keep old pod templates, including environment variables and image references. Lower revisionHistoryLimit if those templates carry anything sensitive, and prefer secrets mounted by reference.
• Digests, not tags, in the deploy job. A rollback to a tag can pull an image that is not the one that was running.

Troubleshooting#

Production Recommendations#

• Deploy by digest and keep release artifacts for longer than your maximum rollback window.
• Set progressDeadlineSeconds, run kubectl rollout status in the deploy job, and set change-cause on every rollout.
• Make rollback a pipeline job that redeploys a previous artifact with the deploy job’s credentials; test it on staging on a schedule.
• Use expand/contract for every schema change, run migrations as a separate step, and rehearse restores.
• Decide the rollback criteria in writing before the first incident.

Conclusion#

Rollback is cheap when it is a redeploy of bytes you still have, detected by a job that knows what “failed” means, and when the data underneath was changed in a way the old version can survive. That closes the loop the path opened: commit, build, test, artifact, deploy, and back. The DevSecOps Pipeline path adds the security gates to this same shape, and the GitOps article shows what rollback looks like when Git, not kubectl, is the source of truth.

References#

• Kubernetes: rolling back a Deployment
• Kubernetes: Deployment status and progress deadline
• kubectl rollout
• OCI image specification: digests
• Expand/contract (parallel change) pattern