Cloud Identity and Least Privilege: Humans, Workloads and CI

Overview#

Every cloud provider, every Kubernetes cluster and every CI platform answers the same two questions on each request: who is this, and what may they do. Everything else in this path (network boundaries, secrets, logs) assumes those answers are right. When they are wrong, no firewall rule helps: the request arrives with a valid credential and is served.

This part separates the identities a system has, looks at the credentials that carry them, and shows why the shape of a credential matters more than how carefully it is stored. The hands-on sections run on a kind cluster and on Terraform configuration validated locally; nothing here needs a cloud account, and the one section that would (a CI job assuming a role) is marked as configuration only.

The identities a production system has#

The rows that go wrong most often are the last three, because they are the ones people give a static key to “so it works”. A human forgets a password; a workload cannot forget the key in its environment, and neither can anyone who copies that environment.

A human signs in at the identity provider with MFA, a workload presents the identity of where it runs, and a CI job presents the ID token its platform issued. All three exchange their claim at the token service for short-lived credentials bound to a role, and every authorised request is written to the audit log with the identity and credential used. A static access key stored in an environment file goes straight to the API, bypassing the token service and its expiry.

1. A human signs in through the identity provider with MFA and receives a session with an expiry. What the session may do is a role, not a list of permissions on the person.
2. A workload gets its identity from where it runs: a Kubernetes service account token projected into the pod, an instance profile on a virtual machine, a managed identity on a container service. The application never sees a long-lived key.
3. A CI job presents the ID token its platform issued for this exact job and exchanges it, through the cloud provider’s token service, for credentials that expire in minutes.
4. The authorisation policy is the same regardless of who arrives: a role with the smallest set of actions the task needs, on the narrowest set of resources that can be named.
5. Every decision is logged with the identity, the credential used and the source address. Part 4 reads those logs.

The trust boundary is the token service: everything to its left is a claim, everything to its right is a credential the provider will honour. A static access key skips the boundary entirely, which is the whole problem.

Two tokens for the same identity#

Kubernetes is a convenient place to see the difference between a bound, short-lived credential and a legacy static one, because it issues both kinds for the same service account. The commands below ran on a kind 0.31 cluster (Kubernetes 1.35).

kubectl create namespace payments
kubectl create serviceaccount api --namespace payments
kubectl create token api --namespace payments --duration=10m > token.txt

Decode the payload of the token (it is a JWT; the middle segment is base64url JSON):

{
  "aud": ["https://kubernetes.default.svc.cluster.local"],
  "exp": 1789532252,
  "iat": 1789531652,
  "iss": "https://kubernetes.default.svc.cluster.local",
  "jti": "d2dd3e2b-0e2a-4815-801d-ba7952f70e19",
  "kubernetes.io": {
    "namespace": "payments",
    "serviceaccount": { "name": "api", "uid": "593159a0-a473-48d6-8757-ff933bb4db58" }
  },
  "sub": "system:serviceaccount:payments:api"
}

Four claims carry the security properties. exp - iat is 600 seconds: the token is useless after ten minutes, whoever holds it. aud names the only audience that should accept it, so a token captured by one service cannot be replayed against another. jti is a unique ID for this token, which the audit log records. And the serviceaccount.uid binds it to this specific account object: delete and recreate the account and the token is invalid even before it expires.

Now the legacy form, the kind that lands in a CI variable and stays there for years:

apiVersion: v1
kind: Secret
metadata:
  name: api-static-token
  namespace: payments
  annotations:
    kubernetes.io/service-account.name: api
type: kubernetes.io/service-account-token

kubectl apply -f static-token.yaml
kubectl get secret api-static-token --namespace payments -o jsonpath='{.data.token}' | base64 -d

{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "payments",
  "kubernetes.io/serviceaccount/secret.name": "api-static-token",
  "kubernetes.io/serviceaccount/service-account.name": "api",
  "sub": "system:serviceaccount:payments:api"
}

No exp, no aud, no jti. It is the same identity with a credential that never expires and that any API server signed with the same key will accept for any purpose. Kubernetes stopped creating these automatically in 1.24 for exactly this reason; the only way to get one now is to ask for it, as above.

With a kubeconfig holding only the bound token:

kubectl --kubeconfig api-kubeconfig.yaml auth whoami

ATTRIBUTE                                           VALUE
Username                                            system:serviceaccount:payments:api
UID                                                 593159a0-a473-48d6-8757-ff933bb4db58
Groups                                              [system:serviceaccounts system:serviceaccounts:payments system:authenticated]
Extra: authentication.kubernetes.io/credential-id   [JTI=d2dd3e2b-0e2a-4815-801d-ba7952f70e19]

The credential-id is the token’s jti. When the same request is made with the legacy token (here with curl and a bearer header), the audit log records the identity but no credential ID at all: there is nothing to identify one static token from another copy of it. Part 4 shows both entries side by side.

kubectl --kubeconfig api-kubeconfig.yaml get secrets

Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:payments:api" cannot list resource "secrets" in API group "" in the namespace "payments"

Authentication succeeded and authorisation refused: the account exists but has no role binding. That is the correct starting state for every new identity. The least-privilege RBAC lab builds the role for a deployer from here, and the Secure Kubernetes path covers the cluster-wide controls.

CI identity without a key#

The delivery identity is where static keys concentrate, because the pipeline needs to reach the cloud and a key is the first thing that works. The alternative is federation: the CI platform is an OpenID Connect issuer, the cloud provider trusts that issuer, and each job exchanges its short-lived ID token for short-lived cloud credentials. The Terraform below declares that arrangement for GitLab.com and AWS. It was validated with terraform validate and scanned with Checkov; it was not applied to an AWS account, so the token exchange itself was not exercised here.

resource "aws_iam_openid_connect_provider" "gitlab" {
  url            = "https://gitlab.com"
  client_id_list = ["https://gitlab.com"]
}

# Read-only role: what a merge-request `terraform plan` needs and nothing more.
data "aws_iam_policy_document" "plan_trust" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.gitlab.arn]
    }
    condition {
      test     = "StringEquals"
      variable = "gitlab.com:aud"
      values   = ["https://gitlab.com"]
    }
    # Any branch or merge request of this one project may plan.
    condition {
      test     = "StringLike"
      variable = "gitlab.com:sub"
      values   = ["project_path:${var.gitlab_project_path}:ref_type:branch:ref:*"]
    }
  }
}

resource "aws_iam_role" "ci_plan" {
  name                 = "ci-terraform-plan"
  assume_role_policy   = data.aws_iam_policy_document.plan_trust.json
  max_session_duration = 3600
}

The apply role is identical except that its sub condition is StringEquals on ref:main: only a job on the protected default branch can assume it. The split mirrors the plan/apply table in Secure Terraform Foundations; this is the identity side of that table.

On the GitLab side the job asks for a token and exchanges it (configuration only, not run here):

plan:
  id_tokens:
    AWS_TOKEN:
      aud: https://gitlab.com
  script:
    - >
      export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s"
      $(aws sts assume-role-with-web-identity
      --role-arn "$PLAN_ROLE_ARN" --role-session-name "gitlab-$CI_JOB_ID"
      --web-identity-token "$AWS_TOKEN" --duration-seconds 900
      --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' --output text))
    - terraform plan -input=false

Three properties fall out of this that no amount of careful key storage gives you. The credentials live for fifteen minutes (--duration-seconds 900). They are tied to a job ID that appears in both the CI log and the cloud audit trail. And there is nothing to rotate, because there is nothing stored.

Scoping what the role can do#

The permissions policy for the plan role started as a single statement, and the scanner did not like it:

Check: CKV_AWS_356: "Ensure no IAM policies documents allow "*" as a statement's resource for restrictable actions"
	FAILED for resource: aws_iam_policy_document.plan

The finding was half right. rds:Describe* can be scoped to database ARNs and was not; ec2:Describe* cannot be scoped at all (EC2 describe calls do not support resource-level permissions), so "*" is the only valid resource for that statement. The fix is to split the statement and record the remaining exception with its reason, the pattern from IaC Security in CI/CD:

data "aws_iam_policy_document" "plan" {
  #checkov:skip=CKV_AWS_356: ec2:Describe* does not support resource-level permissions; the RDS statement is scoped
  statement {
    sid       = "ReadState"
    actions   = ["s3:GetObject", "s3:ListBucket"]
    resources = ["arn:aws:s3:::${var.state_bucket}", "arn:aws:s3:::${var.state_bucket}/*"]
  }
  statement {
    sid       = "DescribeNetwork"
    actions   = ["ec2:Describe*"]
    resources = ["*"]
  }
  statement {
    sid       = "DescribeOwnDatabases"
    actions   = ["rds:Describe*"]
    resources = ["arn:aws:rds:eu-west-1:*:db:app-*"]
  }
}

docker run --rm -v "$PWD:/tf" -w /tf bridgecrew/checkov:3.3.17 -f /tf/ci-identity.tf --framework terraform --quiet --compact

Passed checks: 44, Failed checks: 0, Skipped checks: 1

For contrast, the same directory holds the configuration this design replaces: an IAM user with an access key and "Action": "*", "Resource": "*". Checkov raised eleven findings on those three resources, among them CKV_AWS_286 (privilege escalation), CKV_AWS_287 (credentials exposure), CKV_AWS_288 (data exfiltration) and CKV_AWS_273 (an IAM user where SSO should be). None of them is a false positive. A wildcard policy on a user with a static key is every one of those risks at once.

Privilege escalation paths#

Least privilege is not a property of a single policy; it is a property of the graph. The classic escalations are permissions that let an identity change what identities can do:

• Attach or create policies (iam:AttachUserPolicy, iam:PutRolePolicy, iam:CreatePolicyVersion): a read-only user with any of these is an administrator with one extra step.
• Pass or assume roles (iam:PassRole with a broad resource, sts:AssumeRole on an admin role): the identity does not need the permission itself, only the ability to hand a more powerful role to something it controls.
• Bind roles in Kubernetes: create on rolebindings lets an account grant itself any role that exists in the namespace. RBAC refuses a binding to permissions the binder does not hold, but a badly scoped ClusterRole with bind or escalate verbs removes that check.
• Write to where code runs from: push access to the deployment repository or to the container registry is cluster-admin with a delay, because whatever is pushed will be executed with the workload’s identity.

One detail observed on the kind cluster while writing Part 4: kubectl create rolebinding with --role pointing at a Role that does not exist succeeds (201 Created). The binding grants nothing until someone creates a Role with that name, at which point it grants everything in it. Audit for dangling bindings, not only for over-broad roles.

Rotation and break-glass#

Rotation is a workaround for credentials that should not have existed in that form. Where a credential has to be static (a database password for a legacy application, an API key for a vendor without OIDC), rotation must be automated and observed: a scheduled job replaces the secret, the consumers pick it up (Part 3 shows which consumers do and do not), and an alert fires if the old value is still being used a day later.

Break-glass is the opposite trade-off made deliberately. One account, outside SSO, with administrator rights, whose credentials are stored offline and whose every use is alerted on. It exists because the identity provider is itself a dependency that can fail, and its value is that it is boring: used twice a year in a drill, never used silently.

Security implications#

• A credential without exp has no upper bound on the damage from a leak. Treat every static key found in a CI variable, an environment file or an image (Part 3) as a finding, not as a convention.
• Federation moves the trust to the issuer. The sub condition is the whole control: ref:* on the plan role means any branch can plan, which is intended; ref:* on an apply role would let anyone who can push a branch change production.
• max_session_duration and --duration-seconds bound a compromised job, not a compromised issuer. If the CI platform is compromised, every role it can assume is compromised for as long as the trust exists.
• Kubernetes bound tokens are only as scoped as their aud. A token minted for the API server should not be accepted by your own service; issue one with a different audience for that (kubectl create token --audience).

Troubleshooting#

Running this in production#

• Inventory identities before permissions: list every user, role, service account and CI credential and mark which ones hold a static key. That list is the backlog.
• Scope by task, not by team. The plan role and the apply role belong to the same pipeline and have different permissions; that is the pattern for every automation.
• Put the audit log in front of the identity work: without Part 4, you cannot tell which permissions are used and which can be removed.
• Alert on break-glass use and on any new IAM user or access key creation. Both should be rare enough that an alert is never noise.

References#

• Kubernetes: Service account tokens (bound tokens, TokenRequest API)
• Kubernetes: RBAC privilege escalation prevention (bind, escalate)
• GitLab: OpenID Connect in GitLab CI/CD (id_tokens)
• AWS: Creating OIDC identity providers and AssumeRoleWithWebIdentity
• AWS: Actions, resources and condition keys for Amazon EC2 (resource-level support)
• Checkov: AWS IAM policy checks