IaC Security in CI/CD: Gates, Exceptions and Baselines

Overview#

A scanner that runs on a laptop protects the laptop’s owner. A scanner that runs in the merge request, before terraform plan, with the result visible next to the diff and the merge blocked when it fails, protects the account. This part builds that pipeline: format and validate, Checkov against a baseline, a plan with read-only credentials attached to the merge request, and an apply that only runs from the protected branch.

The fmt, validate and checkov jobs were executed with gitlab-ci-local 4.75.1 against the fixed configuration from part 2; the baseline behaviour was exercised with Checkov directly. The plan and apply jobs need an AWS account and OIDC trust and were not run; their configuration is shown and marked as such.

A developer pushes a Terraform change and opens a merge request. The pipeline runs Checkov against a baseline first: a new finding fails the job and blocks the merge request; a finding the team accepts is recorded as a skip with a reason and the job passes. Only then does terraform plan run with read-only credentials, and the plan is attached to the merge request for review. Apply runs from the protected main branch after the merge.

1. Checkov, with a baseline. Runs on the source before any credentials are involved. Findings that are already in the baseline are known debt; a new finding fails the job and blocks the merge.
2. terraform plan, read-only. Runs only if the scan passed, with credentials that can read state and describe resources but change nothing. The saved plan and its text rendering are job artifacts.
3. Review. The reviewer sees the HCL diff, the Checkov report and the plan in one place.
4. terraform apply, from main. A manual job on the protected branch, with the apply role, applying the saved plan that was reviewed.
5. Exception recorded. When a finding is accepted rather than fixed, a #checkov:skip with a reason lands in the same merge request. The gate passes because the decision is written down, not because it was disabled.

Prerequisites

• The configuration and .checkov.yaml from part 2
• A GitLab project with a protected default branch, or gitlab-ci-local to run the jobs locally
• For plan and apply: an AWS account with an OIDC trust for GitLab and two IAM roles (read-only, apply)

The pipeline#

stages: [validate, scan, plan, apply]

variables:
  TF_IN_AUTOMATION: 'true'
  TF_INPUT: '0'

fmt:
  stage: validate
  image:
    name: hashicorp/terraform:1.14
    entrypoint: ['']
  script:
    - terraform fmt -check -diff -recursive

validate:
  stage: validate
  image:
    name: hashicorp/terraform:1.14
    entrypoint: ['']
  script:
    - terraform init -backend=false
    - terraform validate

checkov:
  stage: scan
  needs: [] # no inputs: runs alongside fmt and validate
  image:
    name: bridgecrew/checkov:3.3.17
    entrypoint: ['']
  script:
    - checkov --directory . --baseline .checkov.baseline
      --output cli --output junitxml --output sarif --output-file-path reports
  artifacts:
    when: always # the report matters most when the job fails
    paths: [reports/]
    reports:
      junit: reports/results_junitxml.xml
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

plan:
  stage: plan
  needs: [fmt, validate, checkov] # no plan for a change that failed the scan
  image:
    name: hashicorp/terraform:1.14
    entrypoint: ['']
  id_tokens:
    AWS_OIDC_TOKEN:
      aud: https://gitlab.com
  script:
    - terraform init
    - terraform plan -out=tfplan -var-file=env/prod.tfvars
    - terraform show -no-color tfplan > plan.txt
  artifacts:
    paths: [tfplan, plan.txt]
    expire_in: 1 day
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

apply:
  stage: apply
  needs: [plan]
  image:
    name: hashicorp/terraform:1.14
    entrypoint: ['']
  id_tokens:
    AWS_OIDC_TOKEN:
      aud: https://gitlab.com
  environment:
    name: production
  script:
    - terraform init
    - terraform apply -input=false tfplan
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

Running the three credential-free jobs locally:

gitlab-ci-local --variable CI_COMMIT_BRANCH=main --variable CI_DEFAULT_BRANCH=main fmt validate checkov

checkov  finished in 36 s
checkov  exported artifacts in 595 ms
validate > - Installed hashicorp/aws v6.64.0 (signed by HashiCorp)
validate > Terraform has been successfully initialized!
validate $ terraform validate
validate > Success! The configuration is valid.
validate finished in 2.17 min
 PASS  fmt
 PASS  validate
 PASS  checkov

validate took two minutes because init downloaded the AWS provider; on a real runner, cache .terraform/providers keyed on .terraform.lock.hcl (the same pattern as the npm cache in the CI/CD Engineering path) and it drops to seconds.

Three structural decisions carry the security value. checkov needs no credentials, so it can safely run on merge requests from anyone. plan needs checkov to pass, so an insecure change never even gets planned. apply has no merge-request rule at all; it exists only on the default branch, which is protected.

Fail only on what is new: the baseline#

A repository that has existed for a year has findings. Turning the gate on with all of them failing means either the gate is disabled the same afternoon or every merge request is red for a month. The baseline is the middle path: record the current findings, fail only on findings that are not in the record.

Create it once, from the default branch, and commit it:

docker run --rm -v "$PWD:/tf" -w /tf bridgecrew/checkov:3.3.17 --directory /tf --create-baseline

On the insecure configuration from part 2 the file records three resources and their check IDs:

{
  "failed_checks": [
    {
      "file": "/main.tf",
      "findings": [
        {
          "resource": "aws_db_instance.app",
          "check_ids": [
            "CKV2_AWS_30",
            "CKV2_AWS_60",
            "CKV_AWS_118",
            "CKV_AWS_129",
            "CKV_AWS_157",
            "CKV_AWS_16",
            "..."
          ]
        }
      ]
    }
  ]
}

Scanning against it reports nothing and exits 0: the known debt is tolerated. Add one new resource with the same mistakes (a bucket with nothing but a name) and scan again:

Passed checks: 0, Failed checks: 7, Skipped checks: 0

	FAILED for resource: aws_s3_bucket.uploads
	FAILED for resource: aws_s3_bucket.uploads
	...
exit=1

Seven findings, all on the new resource, and the job is red. The twenty-two existing findings did not move.

Two rules make a baseline honest rather than a permanent excuse. It is regenerated only when findings are fixed (the file shrinks) and never to absorb new ones (a merge request that adds to .checkov.baseline is a merge request that adds an insecure resource, and the reviewer should read it as such). And it carries a date: schedule a monthly pipeline that scans without --baseline, so the full debt is visible and someone owns burning it down. On the fixed configuration the baseline is {"failed_checks": []}, which is where every repository should be heading.

Pass and fail policy#

Open-source Checkov has no severity levels; those come with the Prisma Cloud platform and an API key. What it has is check IDs, and a policy is expressed as lists of them.

--soft-fail-on turns specific checks advisory: they are reported and do not affect the exit code. --hard-fail-on is the inverse, useful with --soft-fail to make a small set of checks blocking while everything else is advisory. Tested on the insecure configuration with three checks selected:

checkov --directory . --check CKV_AWS_24,CKV_AWS_17,CKV_AWS_16 --soft-fail-on CKV_AWS_16

Passed checks: 0, Failed checks: 3, Skipped checks: 0
Check: CKV_AWS_24: "Ensure no security groups allow ingress from 0.0.0.0:0 to port 22"
Check: CKV_AWS_16: "Ensure all data stored in the RDS is securely encrypted at rest"
Check: CKV_AWS_17: "Ensure all data stored in RDS is not publicly accessible"
exit=1

With all three soft-failed the same scan exits 0. So a workable policy for a team starting out is:

framework: [terraform, secrets]
# Blocking: exposure and data protection. Everything else is advisory until the team decides otherwise.
soft-fail: true
hard-fail-on:
  - CKV_AWS_24 # SSH from 0.0.0.0/0
  - CKV_AWS_260 # RDP from 0.0.0.0/0
  - CKV_AWS_17 # RDS publicly accessible
  - CKV2_AWS_6 # S3 bucket without a public access block
  - CKV_AWS_16 # RDS unencrypted
  - CKV_AWS_145 # S3 not encrypted with KMS
  - CKV_AWS_41 # hardcoded AWS credentials in a provider block

and tighten it over time by moving IDs from advisory to blocking as the baseline shrinks. Keep the file next to the code so the policy is reviewed like code; a policy that lives in a CI variable is invisible.

Ordering by check ID has a limit: an ID list says nothing about which resource, so hard-fail-on cannot express “blocking for production, advisory for sandboxes”. For that, run Checkov per directory with a different config (--config-file env/prod/.checkov.yaml), which is also how a monorepo with several root modules gets per-module policies.

Exceptions that survive review#

Part 2 introduced the two mechanisms: #checkov:skip=ID:reason in the resource, and skip-check in .checkov.yaml. In a pipeline the question is who can add one and how it is seen.

• A skip is a code change, so it goes through the same merge request and the same reviewer as the resource. Use a CODEOWNERS rule (/.checkov.yaml @platform-security, **/*.tf @platform) so a repository-level skip requires the security owner’s approval while a per-resource skip requires the code owner’s.
• The reason is the review. “Single-AZ accepted for this non-critical database; RPO covered by snapshots” can be checked; “false positive” cannot. Reject the second kind.
• Put a date or ticket in reasons for exceptions that should expire (“until the compute module attaches this SG, PLAT-142”), and let the monthly full scan list them.
• False positives exist: a check that fires on a resource the provider handles differently, or on a module output it cannot see through. Skip it on that resource with the reason false positive: <why>, and report it upstream; do not skip the check globally because it was wrong once.

Preventing the bypass#

Every gate has a way around it, and the ways around this one are all governance, not tooling:

Pushing to main directly. Protect the branch: no direct push, merge only through a merge request, at least one approval from someone other than the author, and “pipeline must succeed” turned on. Without the last setting a red checkov job is a warning, not a gate.

Editing the pipeline in the same merge request. A change that deletes the checkov job or adds allow_failure: true to it is a change to .gitlab-ci.yml. CODEOWNERS that requires the platform team’s approval for that file is the control. So is keeping the scan definition in a shared include (include: project: platform/ci-templates) that the application repository cannot edit.

Applying from a laptop. The apply role should be assumable only by the CI job on the protected branch (the OIDC trust policy’s condition on project_path and ref). A person with a personal admin key can still apply, which is why personal admin keys are the thing to remove.

Skipping without a reason. Fail the merge request on it. A small script that greps for checkov:skip= without a colon-separated reason, or a reviewer who knows to look, is enough; Checkov itself requires the reason syntax to recognise the skip at all.

Turning the whole thing advisory “temporarily”. This is what the baseline is for. Advisory mode was never needed; only the existing debt was.

Making it usable#

A gate that engineers hate gets routed around. The parts that keep it tolerable:

• Speed. Checkov on this configuration took 36 seconds, most of it container start; it runs in parallel with fmt and validate because none of them need each other. Cache the provider directory so validate is fast too.
• Local parity. The same .checkov.yaml and the same image tag mean docker run … bridgecrew/checkov:3.3.17 on a laptop gives the same answer the pipeline will. Put the command in the README.
• Reports where people look. The JUnit artifact renders in the merge request’s test tab; the SARIF file feeds code-scanning views and IDE plugins. Nobody should have to open a job log to see why it failed.
• Findings that mean something. The blocking list is short and every entry in it is a real exposure. Advisory findings are visible and not shouted about.

Security Considerations#

• Pin both images (hashicorp/terraform:1.14, bridgecrew/checkov:3.3.17). A floating latest on the scanner changes the policy set without a merge request.
• plan.txt and tfplan contain resource attributes, including values marked sensitive in tfplan. One-day expiry and project-private artifacts, as in the job above.
• TF_IN_AUTOMATION and TF_INPUT=0 stop Terraform from ever prompting in a job, which otherwise hangs the runner until the timeout.

Troubleshooting#

Production Recommendations#

• Scan on every merge request with no credentials; plan with read-only credentials; apply only from the protected branch, manually, from the saved plan.
• Commit .checkov.yaml and .checkov.baseline; regenerate the baseline only to shrink it; scan without it monthly.
• Block on exposure and data-protection checks, advise on the rest, and tighten over time.
• Require reasons on every skip and code-owner approval on policy and pipeline files.
• Enable “pipeline must succeed” on the protected branch, and remove personal admin credentials.

Conclusion#

The gate is three jobs and two protected files. Checkov decides whether a change may be planned, the baseline keeps old debt from blocking new work while making it visible, exceptions are written down where the reviewer sees them, and branch protection turns the job’s exit code into a decision. Together with part 1 and part 2, that is a repository where an insecure change has to argue its case in writing before it reaches production.

References#

• Checkov: CLI reference (--baseline, --soft-fail-on, --hard-fail-on, outputs)
• Checkov: suppressing and skipping policies
• GitLab: protected branches
• GitLab: code owners
• GitLab: OpenID Connect for AWS
• Terraform: running Terraform in automation