Production Cloud Hardening: A Checklist That Connects the Controls

Overview#

A hardening checklist is usually a list of settings. This one is a list of questions, each with the command that answers it, because a setting that was applied once and never checked again is a belief. Every verification below was run while writing this path: against the kind cluster from Parts 1 to 4, against the images from Part 3, or read-only against this site’s production edge. The cloud-provider items that could not be exercised without an account are marked as such.

The list is organised by the layer that fails, not by the tool that fixes it, and each section ends with the gap that layer leaves and the section that covers it. That is the real content: controls are chosen so that each one’s blind spot is another one’s job.

A request from the internet passes identity, which decides who may call an API; exposure, which decides which sockets are reachable; and transport and headers, which make the one public port safe to serve. Behind them, secrets and images keep credentials out of the artifact and the store, visibility records every decision the other layers made, and backups exist for when the rest fail. The failure path from each gate lands in the audit log rather than disappearing.

1. Identity decides who can call the API at all (Part 1). Its gap: a socket that does not check identity.
2. Exposure decides which sockets are reachable and from where (Part 2). Its gap: the one port that must be public.
3. Transport and headers make the public port safe to serve: TLS, HSTS, a CSP. Their gap: what the code behind the port does with a secret.
4. Secrets and images keep credentials out of the artifact and the storage layer (Part 3). Their gap: a package with a known vulnerability, and a control that stopped working.
5. Visibility records every decision the other layers made (Part 4). Its gap: the moment the data is gone.
6. Backups and recovery are the layer that exists for when the others fail. Their gap: they are a copy of everything, including the secrets.

1. Identity#

Run on the kind cluster, the legacy token Secret from Part 1 shows up in the first check, which is the point of the check; delete it. The can-i --list for the api account returned only the self-review resources (selfsubjectreviews, selfsubjectaccessreviews, selfsubjectrulesreviews) and the discovery URLs every authenticated identity has.

The gap: identity says nothing about a listener that never asks. Section 2.

2. Exposure#

The fourth line is the one Part 2 failed on its own laptop: the internal network was reachable by IP. The check is a connection attempt, not a review of the diagram.

The gap: one port has to be public. Section 3.

3. Transport and headers#

The public port on this site is Cloudflare’s edge, and its configuration is documented in Hardening a Static Site on Cloudflare. The checks below were run against production on 2026-09-16 and are the ones to repeat after any edge or origin change:

curl -sI http://sachinchaurasiya.com/ | grep -iE '^(HTTP|location)'
curl -sI https://www.sachinchaurasiya.com/ | grep -iE '^(HTTP|location)'
curl -sI https://sachinchaurasiya.com/ | grep -iE '^(strict-transport|content-security|x-content-type|referrer|permissions|x-frame|cross-origin)'
curl -s -o /dev/null -w '%{http_code}\n' https://sachinchaurasiya.com/does-not-exist

HTTP/1.1 301 Moved Permanently
Location: https://sachinchaurasiya.com/
HTTP/2 301
location: https://sachinchaurasiya.com/
strict-transport-security: max-age=31536000; includeSubDomains
content-security-policy: frame-ancestors 'none'
cross-origin-opener-policy: same-origin
permissions-policy: accelerometer=(), camera=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), payment=(), usb=(), interest-cohort=()
referrer-policy: strict-origin-when-cross-origin
x-content-type-options: nosniff
x-frame-options: DENY
404

Plain HTTP and the www host both redirect to the one canonical origin, the hardening headers are present on every response, and an unknown path is a real 404, not a 200 with an error page. The TLS side from Part 2: TLS 1.3 negotiated by default, TLS 1.2 available, TLS 1.1 refused by the server with a protocol-version alert, certificate valid until Dec 13 2026, issued by Google Trust Services through Cloudflare. DNSSEC is on: the zone publishes a DS record (2371 13 2 …).

The gap: none of this constrains what the application does with a credential. Section 4.

4. Secrets, images and patching#

Patching is a schedule, not a task: a rebuild of every image on a cadence (weekly is common) whether or not the application changed, because the base image and the transitive dependencies change without you. The Grype article on this site shows what one rebuild does to a finding count; the number it produces on your image is the metric to trend.

The gap: a control that was disabled, a permission that was widened, a scan that was skipped. Section 5.

5. Visibility#

The third line is the one that ties the path together. Each earlier part ended with an action that should have left a record; Part 4 found them. The test to repeat is the same: do the thing, then find it in the log.

The gap: logs describe the past. Section 6.

6. Backups and recovery#

Backups are the control for the failure of every other control, and they are also a copy of the data those controls protect. On the kind cluster:

kubectl -n kube-system exec etcd-cloud-lab-control-plane -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/lib/etcd/backup-2026-09-16.db
kubectl -n kube-system exec etcd-cloud-lab-control-plane -- etcdutl snapshot status /var/lib/etcd/backup-2026-09-16.db --write-out=table

Snapshot saved at /var/lib/etcd/backup-2026-09-16.db
+---------+----------+------------+------------+---------+
|  HASH   | REVISION | TOTAL KEYS | TOTAL SIZE | VERSION |
+---------+----------+------------+------------+---------+
| f05ebad |     2261 |        395 |     2.0 MB |   3.6.0 |
+---------+----------+------------+------------+---------+

Copied off the node and searched:

strings etcd-snapshot.db | grep -c 'rotated-password'

1

That snapshot was taken before Part 3 enabled encryption at rest; the Secret’s value is in it in plain text, and would be in every backup taken since the cluster was created. The surprise came after encryption was enabled and every Secret rewritten: a fresh snapshot still contained rotated-password once. etcd keeps previous revisions of every key until they are compacted, so the plaintext version written before the rewrite was still in the store’s history, and a snapshot is the whole store.

etcdctl compact 3408      # the current revision, from `etcdctl endpoint status --write-out=json`
etcdctl defrag
etcdctl snapshot save /var/lib/etcd/backup-enc2.db
strings etcd-snapshot-enc2.db | grep -c 'rotated-password'

compacted revision 3408
Finished defragmenting etcd member[127.0.0.1:2379]. took 269.489102ms
0

Only after a compaction and a defragmentation did the snapshot contain no plaintext (and three k8s:enc:secretbox prefixes). The API server compacts automatically on an interval, so this window closes on its own in a running cluster; a backup taken inside the window does not. The rule that follows: a backup has the access controls of the most sensitive thing that was ever in it, and encryption at rest is what makes that manageable, once the history has been compacted.

Recovery has two shapes. Rolling back a deployment (the previous image, the previous ReplicaSet) needs no backup and is covered in the CI/CD path. Recovering state (a deleted namespace, a corrupted volume, a lost account) needs the backup, and the only evidence that it works is a restore that was actually performed. Terraform state on a versioned, access-controlled bucket, as Secure Terraform Foundations recommends, is the same principle for infrastructure: the state file is a backup of the mapping between code and reality, and it contains secrets.

The gap: backups contain everything. Section 4’s encryption at rest and Section 1’s access control on the backup location close it.

Dependency hygiene, briefly#

Two items do not fit a layer and belong on the list anyway. Lockfiles with --frozen-lockfile installs, so the build uses the dependencies review saw; and allow-listed build scripts, so a dependency cannot run code at install time. Both are in place in this repository’s pnpm validate and are described in the supply chain path. They are here because a compromised dependency arrives with the application’s identity and inside every boundary above.

Not verified here#

• Cloud-provider IAM behaviour, security group enforcement and VPC flow logs: the Terraform in Parts 1 and 2 was validated and scanned, not applied. The provider’s behaviour is documented, and the design is standard, but no account was used.
• Cloudflare WAF and security events: the managed ruleset is enabled for this site and the events view exists; no events were exported for this article.
• Restore: etcd snapshots were taken, inspected and compared; none was restored into a cluster in this run.
• Docker bridge isolation on Linux: not tested; the macOS host in Part 2 showed the opposite.

Security implications#

• A checklist item without a command is an opinion. Every row above names one, and the ones that could not be run are listed under “not verified” rather than assumed.
• The controls overlap on purpose. When Part 2’s network boundary failed, Part 1’s password on the cache held; when Section 3’s CSP cannot see a compromised build dependency, Section 4’s lockfile and scan do. Removing a layer because another one “already covers it” removes the overlap, which is the design.
• Backups and audit logs are the two artifacts that outlive an incident and are read by the attacker’s successor as well as yours. They need the strictest access on the list.

Running this in production#

• Turn the tables into a script that runs on a schedule and after every change, and fails on any row that changed answer. Most rows are a curl, a kubectl or a scanner with an exit code.
• Record the date each row was last verified next to the row. Rows nobody has run in a quarter are the ones to run first.
• Keep the “not verified” list honest. It is the list of controls whose failure you would not see, and it should shrink over time.
• Repeat Parts 1 to 4 on the next system before this list, not after it. The list is the summary; the parts are where the understanding is.

References#

• OWASP: HTTP Headers Cheat Sheet
• Kubernetes: operating etcd clusters (backing up, restoring)
• Kubernetes: security checklist
• Cloudflare: managed rules and DNSSEC
• Trivy: vulnerability scanning options (--ignore-unfixed, --exit-code)
• CIS Benchmarks (cloud providers and Kubernetes)