Kubernetes Workloads and Controllers: Choosing and Reading Them

Overview#

Every workload on Kubernetes is pods, and pods are disposable. What makes a service survive a node failure, a database keep its disk, or a nightly report run once is the controller that owns the pods: it watches the actual state, compares it with the declared one, and creates or deletes pods until they match. Choosing the controller is choosing what “the same” means when a pod is replaced: same count, same name, same disk, one per node, run to completion.

This part applies one of each to a kind cluster with a control plane and two workers, reads their state, deletes their pods, and watches what comes back. The Secure Kubernetes path covers what these workloads are allowed to do; this path covers running them.

Prerequisites

• kind 0.31 and kubectl 1.32 or newer
• A three-node cluster: kind create cluster --name k8s-ops-lab --config kind-ops.yaml with one control-plane and two worker roles
• The sample image ci-demo:1.4.2 from the supply chain path, loaded with kind load docker-image ci-demo:1.4.2 --name k8s-ops-lab; any small HTTP server image works in its place

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker

One of each#

The manifest below declares a stateless API (Deployment), a database (StatefulSet with a headless Service), a per-node agent (DaemonSet), a one-off migration (Job) and a scheduled report (CronJob).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  labels: { app: api }
spec:
  replicas: 3
  selector:
    matchLabels: { app: api }
  template:
    metadata:
      labels: { app: api }
    spec:
      containers:
        - name: api
          image: ci-demo:1.4.2
          imagePullPolicy: Never # the image was loaded into kind, not pulled
          ports: [{ containerPort: 3000, name: http }]
          readinessProbe:
            httpGet: { path: /healthz, port: http }
          resources:
            requests: { cpu: 50m, memory: 64Mi }
            limits: { memory: 128Mi }
---
apiVersion: v1
kind: Service
metadata:
  name: db
spec:
  clusterIP: None # headless: DNS returns the pods, not a virtual IP
  selector: { app: db }
  ports: [{ port: 5432, name: pg }]
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db
  replicas: 2
  selector:
    matchLabels: { app: db }
  template:
    metadata:
      labels: { app: db }
    spec:
      containers:
        - name: db
          image: postgres:17-alpine
          env:
            - { name: POSTGRES_PASSWORD, value: lab-only } # a Secret in anything but a lab
            - { name: PGDATA, value: /var/lib/postgresql/data/pgdata }
          ports: [{ containerPort: 5432, name: pg }]
          volumeMounts: [{ name: data, mountPath: /var/lib/postgresql/data }]
  volumeClaimTemplates:
    - metadata: { name: data }
      spec:
        accessModes: [ReadWriteOnce]
        resources: { requests: { storage: 1Gi } }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-agent
spec:
  selector:
    matchLabels: { app: node-agent }
  template:
    metadata:
      labels: { app: node-agent }
    spec:
      containers:
        - name: agent
          image: alpine:3.22
          command: ['sh', '-c', 'while true; do date; sleep 30; done']
---
apiVersion: batch/v1
kind: Job
metadata:
  name: migrate
spec:
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: alpine:3.22
          command: ['sh', '-c', 'echo "applying migration 0042"; sleep 5; echo done']
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: report
spec:
  schedule: '*/2 * * * *'
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 2
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: report
              image: alpine:3.22
              command: ['sh', '-c', 'echo "report at $(date)"']

kubectl apply -f workloads.yaml
kubectl rollout status deploy/api && kubectl rollout status statefulset/db && kubectl rollout status daemonset/node-agent
kubectl wait --for=condition=complete job/migrate

rollout status works for the three long-running controllers; a Job is waited for with wait --for=condition=complete. Fifteen seconds later:

kubectl get deploy,rs,statefulset,daemonset,job,cronjob

NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/api   3/3     3            3           15s

NAME                             DESIRED   CURRENT   READY   AGE
replicaset.apps/api-6c5dc7ffff   3         3         3       15s

NAME                  READY   AGE
statefulset.apps/db   2/2     15s

NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/node-agent   2         2         2       2            2           <none>          15s

NAME                STATUS     COMPLETIONS   DURATION   AGE
job.batch/migrate   Complete   1/1           11s        15s

NAME                   SCHEDULE      TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/report   */2 * * * *   <none>     False     0        <none>          15s

And the pods, with the node each landed on:

kubectl get pods -o wide --sort-by=.metadata.name

NAME                  READY  STATUS     RESTARTS  NODE
api-6c5dc7ffff-9pdcl  1/1    Running    0         k8s-ops-lab-worker2
api-6c5dc7ffff-d65d7  1/1    Running    0         k8s-ops-lab-worker
api-6c5dc7ffff-k8zxw  1/1    Running    0         k8s-ops-lab-worker2
db-0                  1/1    Running    0         k8s-ops-lab-worker2
db-1                  1/1    Running    0         k8s-ops-lab-worker
migrate-7jqh2         0/1    Completed  0         k8s-ops-lab-worker
node-agent-j8m2m      1/1    Running    0         k8s-ops-lab-worker2
node-agent-l8xtx      1/1    Running    0         k8s-ops-lab-worker

The naming already tells the story. Deployment pods carry a ReplicaSet hash and a random suffix; StatefulSet pods are db-0 and db-1; DaemonSet pods are one per worker; the Job pod ran and stayed, Completed, for its logs.

Deployment: the same count#

A Deployment does not manage pods directly. It manages ReplicaSets, and the ReplicaSet manages pods; that indirection is what makes rolling updates possible (a new ReplicaSet scales up while the old one scales down), and it is why kubectl get rs is worth reading during a rollout. api-6c5dc7ffff is the ReplicaSet for the current pod template; a changed template gets a new hash and a new ReplicaSet, and the CI/CD Engineering path shows that transition in full.

Delete a pod and the ReplicaSet replaces it with a new one, new name, new IP, any node:

kubectl delete pod api-6c5dc7ffff-9pdcl
kubectl get pods -l app=api -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,CREATED:.metadata.creationTimestamp

NAME                   STATUS    CREATED
api-6c5dc7ffff-d65d7   Running   2026-09-15T21:20:42Z
api-6c5dc7ffff-k8zxw   Running   2026-09-15T21:20:42Z
api-6c5dc7ffff-mm4js   Running   2026-09-15T21:21:24Z

mm4js is new. Nothing depended on 9pdcl by name, so nothing noticed. That is the contract: use a Deployment when every replica is interchangeable and state lives elsewhere. Most services qualify.

StatefulSet: the same name and the same disk#

A StatefulSet gives each pod a stable identity: an ordinal name, a DNS name through the headless Service, and a PersistentVolumeClaim created from the template and bound to that ordinal.

kubectl get pvc

NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-db-0   Bound    pvc-dd0674e7-4420-4b99-a4ee-63f95501e5db   1Gi        RWO            standard       32s
data-db-1   Bound    pvc-f0759243-efc5-49ad-ad26-630563ed7f96   1Gi        RWO            standard       24s

The claims are named <template>-<statefulset>-<ordinal>, and db-0 was created before db-1 (ordered startup is the default; podManagementPolicy: Parallel changes it). Each pod is addressable:

kubectl run dns --rm -i --restart=Never --image=busybox:1.37 -- nslookup db-0.db.default.svc.cluster.local

Name:    db-0.db.default.svc.cluster.local
Address: 10.244.2.6

Delete db-0 and the controller recreates a pod with the same name, attached to the same claim:

kubectl delete pod db-0 --wait=false
kubectl get pod db-0 -o jsonpath='{.metadata.name} {.status.phase}{"\n"}'
kubectl wait --for=condition=ready pod/db-0
kubectl get pod db-0 -o jsonpath='{.metadata.name} {.status.phase} volume={.spec.volumes[0].persistentVolumeClaim.claimName}{"\n"}'

db-0 Pending
db-0 Running volume=data-db-0

The data in data-db-0 survived; a Deployment with a volumeClaimTemplates-less pod would have lost it. That is the contract: use a StatefulSet when pods need durable identity, per-pod storage, or ordered operations (databases, queues, anything with a leader election that uses pod names). The cost is operational: the controller will not replace a pod on a failed node until it is sure the old one is gone, because two db-0s writing one volume is worse than none.

DaemonSet: one per node#

A DaemonSet runs one pod on every node that matches, and adds one when a node joins. The cluster has three nodes and the DaemonSet reports DESIRED 2, because the control plane carries a taint:

kubectl get node k8s-ops-lab-control-plane -o jsonpath='{.spec.taints}{"\n"}'

[{"effect":"NoSchedule","key":"node-role.kubernetes.io/control-plane"}]

A DaemonSet that must run there too (a log shipper, a node exporter) adds a matching toleration to its pod template; one that should not (anything that is not infrastructure) leaves it out. Use a DaemonSet for per-node infrastructure: log collection, metrics agents, CNI and storage plugins. It is the wrong tool for “I want several copies”; that is a Deployment with replicas.

Job and CronJob: run to completion#

A Job creates pods until the required number of completions succeed, retrying failures up to backoffLimit. The pod is kept after success so its logs are readable:

kubectl describe job migrate | grep -E 'Completions|Duration|Pods Statuses'
kubectl logs job/migrate

Completions:      1
Duration:         11s
Pods Statuses:    0 Active (0 Ready) / 1 Succeeded / 0 Failed
applying migration 0042
done

A CronJob is a Job factory on a schedule. Two minutes after creation it had produced its first Job:

kubectl get jobs; kubectl get cronjob report

NAME              STATUS     COMPLETIONS   DURATION   AGE
migrate           Complete   1/1           11s        2m26s
report-29825122   Complete   1/1           3s         68s
NAME     SCHEDULE      TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
report   */2 * * * *   <none>     False     0        69s             2m27s

Two fields decide whether a CronJob behaves in production. concurrencyPolicy: Forbid skips a run if the previous one is still going, which is what a report or a backup wants. successfulJobsHistoryLimit bounds how many completed Jobs (and their pods) are kept; without it, a minutely CronJob leaves thousands of objects behind. Use a Job for a migration, an import, a one-off task with a definite end; a CronJob for the same thing on a schedule. The rollback article explains why migrations belong in a Job rather than in the application container’s start-up.

Reading a controller#

Three commands cover most of what you need to know, for any of the five kinds:

• kubectl get <kind> for the counts. READY 2/3 on a Deployment, DESIRED 2 / READY 1 on a DaemonSet or COMPLETIONS 0/1 on a Job is the first sign something is wrong.
• kubectl describe <kind> <name> for the events and conditions: why a pod was not scheduled, when the last rollout happened, what the Job’s failure count is.
• kubectl get pods -l <selector> -o wide for where the pods are and what state each is in. The selector is in the controller’s spec; the pods are not “inside” it in any other sense.

The next parts build on this: part 2 makes readiness and resources mean something, and part 3 works through what to do when the counts are wrong.

Security Considerations#

• The StatefulSet above passes the database password as a plain environment variable. In a lab that is acceptable; anywhere else it is a Secret mounted or referenced with secretKeyRef, and the Kubernetes security checklist covers the rest of the workload surface.
• A DaemonSet that tolerates the control-plane taint runs on the node that holds the cluster’s keys. Give it only the host access it needs, and prefer a read-only root filesystem and a non-root user.
• imagePullPolicy: Never is a lab convenience for a kind-loaded image. In production, pull by digest; the supply chain path explains why.

Troubleshooting#

Cleanup#

Keep the cluster for the rest of the path. Remove only the CronJob so it stops producing Jobs:

kubectl delete cronjob report

Conclusion#

Five controllers, five definitions of “the same”: count, identity plus disk, one per node, done once, done on a schedule. Read the counts with get, the reasons with describe, and pick the controller by what must survive a pod’s death. Part 2 turns to what “ready” and “healthy” mean for the pods themselves.

References#

• Kubernetes: Deployments
• Kubernetes: StatefulSets
• Kubernetes: DaemonSet
• Kubernetes: Jobs and CronJobs
• Kubernetes: taints and tolerations
• kind: configuration