Deployment Strategies: Rolling, Blue/Green and Canary on Kubernetes

Overview#

A deployment strategy is a decision about risk: how many users see the new version at once, how fast the old version disappears, and how you get back if the new one is wrong. Kubernetes gives you one strategy for free (rolling update) and the primitives to build the other two (blue/green and canary) without a service mesh or a progressive-delivery controller. This part builds all three on a kind cluster and checks, with requests, that traffic went where the strategy said it would.

The artifact being deployed is the image from the previous parts’ point of view: built once, tagged by version. Here it is nginx:1.27.5-alpine (the current release) and nginx:1.29.1-alpine (the new one), because nginx reports its version in the Server header, which makes traffic easy to attribute. Every output below is from a kind 0.31 cluster running Kubernetes 1.35.

Prerequisites

• A disposable cluster: kind create cluster --name cicd-lab (kind 0.31, Kubernetes 1.35)
• kubectl 1.32 or newer
• Internet access to pull two small nginx images and a curl image

Rolling update#

The default Deployment strategy replaces pods a few at a time. Two fields control the blast radius: maxUnavailable (how many existing pods may be down during the rollout) and maxSurge (how many extra pods may run temporarily). maxUnavailable: 0 with maxSurge: 1 means capacity never drops: a new pod must become ready before an old one is removed.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels: { app: web }
spec:
  replicas: 3
  selector:
    matchLabels: { app: web }
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels: { app: web }
    spec:
      containers:
        - name: web
          image: nginx:1.27.5-alpine
          ports: [{ containerPort: 80 }]
          readinessProbe:
            httpGet: { path: /, port: 80 }
            periodSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector: { app: web }
  ports: [{ port: 80, targetPort: 80 }]

The readiness probe is what makes “ready” mean something. Without it a pod is ready the moment its container starts, and the rollout happily replaces working pods with ones that are still booting.

kubectl apply -f deploy-v1.yaml
kubectl rollout status deployment/web --timeout=180s

deployment.apps/web created
service/web created
Waiting for deployment "web" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "web" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "web" rollout to finish: 2 of 3 updated replicas are available...
deployment "web" successfully rolled out

Now roll to the new version. In a pipeline this is the deploy job; here it is one command, plus an annotation so the rollout history says what changed and why:

kubectl set image deployment/web web=nginx:1.29.1-alpine
kubectl annotate deployment/web kubernetes.io/change-cause="web: nginx 1.29.1-alpine" --overwrite
kubectl rollout status deployment/web --timeout=180s

Waiting for deployment "web" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "web" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "web" rollout to finish: 1 old replicas are pending termination...
deployment "web" successfully rolled out

rollout status exits 0 when the rollout completes and non-zero when it does not within the timeout, which is what makes it usable as a pipeline step: the deploy job fails if the rollout fails. Part 4 shows what a failed rollout looks like and how to reverse it.

When rolling is right. Stateless services that can run two versions side by side for a minute. That is most web services. It is wrong when old and new cannot coexist (an incompatible protocol between replicas, a schema the old version cannot read), and it gives no way to test the new version before users hit it.

Delete the rolling deployment before the next experiment so the label selectors do not overlap:

kubectl delete -f deploy-v1.yaml

Blue/green#

Blue/green runs two complete copies of the service and moves traffic between them by changing what the Service selects. The new version (“green”) is fully deployed, can be tested through its own address, and receives production traffic in one atomic switch. Rolling back is the same switch in reverse.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-blue
spec:
  replicas: 2
  selector:
    matchLabels: { app: web, track: blue }
  template:
    metadata:
      labels: { app: web, track: blue }
    spec:
      containers:
        - name: web
          image: nginx:1.27.5-alpine
          readinessProbe:
            httpGet: { path: /, port: 80 }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-green
spec:
  replicas: 2
  selector:
    matchLabels: { app: web, track: green }
  template:
    metadata:
      labels: { app: web, track: green }
    spec:
      containers:
        - name: web
          image: nginx:1.29.1-alpine
          readinessProbe:
            httpGet: { path: /, port: 80 }
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector: { app: web, track: blue } 
  ports: [{ port: 80, targetPort: 80 }]

The track label is the switch. Apply, wait for both deployments, and confirm the Service only has blue endpoints:

kubectl apply -f bluegreen.yaml
kubectl rollout status deploy/web-blue && kubectl rollout status deploy/web-green
kubectl get endpointslices -l kubernetes.io/service-name=web \
  -o jsonpath='{range .items[0].endpoints[*]}{.targetRef.name}{"\n"}{end}'

web-blue-747577b8f7-frnfr
web-blue-747577b8f7-9vxb5

Check from inside the cluster which version answers. A throwaway curl pod is enough:

kubectl run probe --rm -i --restart=Never --image=curlimages/curl:8.14.1 -- \
  sh -c 'for i in 1 2 3 4 5; do curl -sI http://web | grep -i "^server"; done'

Server: nginx/1.27.5
Server: nginx/1.27.5
Server: nginx/1.27.5
Server: nginx/1.27.5
Server: nginx/1.27.5

Green is running and healthy but receives nothing. This is the moment to test it: a second Service selecting track: green, or a port-forward to a green pod, gives the new version a private address. When it passes, switch:

kubectl patch service web -p '{"spec":{"selector":{"app":"web","track":"green"}}}'
kubectl run probe --rm -i --restart=Never --image=curlimages/curl:8.14.1 -- \
  sh -c 'for i in 1 2 3; do curl -sI http://web | grep -i "^server"; done'

service/web patched
Server: nginx/1.29.1
Server: nginx/1.29.1
Server: nginx/1.29.1

The endpoint slice now lists only green pods; existing connections to blue pods finish, new ones go to green. Rolling back is the same patch with track: blue, and in this run it took effect on the next request:

service/web patched
Server: nginx/1.27.5

When blue/green is right. When the new version must be verified in the production environment before any user sees it, when a rollback has to be instant, and when you can afford two copies of the service for the duration. The cost is that doubled capacity, plus the discipline to actually test green before switching. It does not help with database changes: both colours share the database, which is why part 4 treats schema changes separately.

kubectl delete -f bluegreen.yaml

Canary#

A canary sends a small share of real traffic to the new version and watches. With plain Kubernetes the share is set by pod count: a Service that selects app: web balances across every matching pod, so three stable pods and one canary pod give the canary roughly a quarter of requests.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-stable
spec:
  replicas: 3
  selector:
    matchLabels: { app: web, track: stable }
  template:
    metadata:
      labels: { app: web, track: stable }
    spec:
      containers:
        - name: web
          image: nginx:1.27.5-alpine
          readinessProbe:
            httpGet: { path: /, port: 80 }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-canary
spec:
  replicas: 1
  selector:
    matchLabels: { app: web, track: canary }
  template:
    metadata:
      labels: { app: web, track: canary }
    spec:
      containers:
        - name: web
          image: nginx:1.29.1-alpine
          readinessProbe:
            httpGet: { path: /, port: 80 }
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector: { app: web } 
  ports: [{ port: 80, targetPort: 80 }]

Apply it and count where forty requests land:

kubectl apply -f canary.yaml
kubectl rollout status deploy/web-stable && kubectl rollout status deploy/web-canary
kubectl run probe --rm -i --restart=Never --image=curlimages/curl:8.14.1 -- \
  sh -c 'for i in $(seq 1 40); do curl -sI http://web | grep -i "^server"; done | sort | uniq -c'

     30 Server: nginx/1.27.5
     10 Server: nginx/1.29.1

Exactly the 3:1 ratio the pod counts predict, though on a real cluster expect it to be approximately, not exactly, proportional: kube-proxy balances per connection, not per request, and clients that keep connections open stick to one pod.

The canary phase is an observation window. What you watch is the new version’s error rate, latency and whatever business metric matters, compared with the stable pods over the same window. If it holds, promote by shifting the ratio until stable is gone:

kubectl scale deploy/web-canary --replicas=3
kubectl scale deploy/web-stable --replicas=0
kubectl run probe --rm -i --restart=Never --image=curlimages/curl:8.14.1 -- \
  sh -c 'for i in $(seq 1 20); do curl -sI http://web | grep -i "^server"; done | sort | uniq -c'

     20 Server: nginx/1.29.1

If it does not hold, kubectl scale deploy/web-canary --replicas=0 removes the canary and the stable pods carry on as if nothing happened; that is the whole appeal.

When a canary is right. When the risk you care about only shows under real traffic (performance regressions, rare inputs, third-party integrations) and you have the metrics to see it. Pod-count canaries are coarse: the smallest share is one pod out of N. Finer control (1 % of requests, header-based routing, automated analysis and rollback) needs an ingress controller or service mesh that splits traffic by weight, or a controller such as Argo Rollouts or Flagger. Those tools automate the same steps shown here.

kubectl delete -f canary.yaml

Environment promotion and gates#

Whatever the strategy, the artifact should reach production by promotion, not by a rebuild: the same image digest that passed in staging is the one the production Deployment is set to. In the pipeline from part 1 that is the manual deploy-production job receiving the artifact deploy-staging used. On Kubernetes it is the same manifest applied to a second cluster or namespace with one value changed: nothing.

Gates are the conditions between promotion steps. In order of how much they catch:

1. Readiness and rollout status. The pod answers its probe; rollout status returns 0. Free, always on, catches crashes and misconfiguration.
2. A smoke test after deploy. A job that calls the deployed service (a health endpoint, one real request) and fails the pipeline otherwise. The probe pod above is a minimal version.
3. Metrics over a window. Error rate and latency compared with the previous version, either read by a human during a canary or by a controller that aborts automatically.
4. A manual approval. Someone accountable clicks. Useful for timing and compliance; not a substitute for the three above, because a human cannot see what a probe sees.

Manual approvals belong before production, and only there. An approval before staging trains people to click without looking.

Security Considerations#

• Deploy by digest, not tag, so the pods that pass the canary are byte-for-byte the pods you promote (image: nginx@sha256:…). Part 4 covers why this also makes rollback honest.
• The deploy job’s credentials should be able to update one Deployment in one namespace, nothing more; the least-privilege RBAC lab builds exactly that service account.
• A Service selector patch is a small change with a large effect. Restrict who can edit Services in production namespaces, and make the switch a pipeline job with an audit trail rather than a kubectl from a laptop.

Troubleshooting#

Production Recommendations#

• Default to rolling updates with maxUnavailable: 0 and real readiness probes; they cover most services at no cost.
• Use blue/green when you need to test in production before exposure or need an instant switch back, and budget the double capacity.
• Use a canary when the risk only shows under real traffic, and only if you have the metrics to judge it.
• Promote digests, gate on rollout status and a smoke test everywhere, add metrics where you can, and keep manual approval for production alone.

Conclusion#

All three strategies are label and selector arithmetic on a cluster you already have. Rolling replaces gradually, blue/green switches atomically, canary exposes proportionally, and each has a rollback that is the strategy run backwards. Part 4 makes that rollback reliable, including the case the strategies above do not cover: the database.

References#

• Kubernetes: Deployments and rolling updates
• Kubernetes: configure liveness, readiness and startup probes
• Kubernetes: Service and EndpointSlices
• Argo Rollouts
• Flagger
• kind