Building Reliable Pipelines: Pinning, Caching, Artifacts and Failure Handling

Overview#

A pipeline is reliable when the same commit produces the same result on Monday and on Friday, on your laptop and on the runner, and when a failure tells you what broke instead of “retry it”. Most unreliability has one of three causes: an input that was not pinned, a cache that was trusted as if it were a build output, or a job whose failure mode was never designed. This part fixes all three on the pipeline from part 1.

Everything below was run with gitlab-ci-local 4.75.1 in Docker; the log lines are from those runs. Two features (retry and timeout) are configured but were not triggered, because the sample jobs neither hung nor hit a runner failure; they are marked as such.

Prerequisites

• The sample project and pipeline from part 1
• Docker and gitlab-ci-local

Deterministic builds#

A build is deterministic when its inputs are fixed. The inputs are the source (fixed by the commit), the toolchain (fixed by the image) and the dependencies (fixed by the lockfile). Pin all three.

Pin the image#

default:
  image: node:22.20.0-alpine3.22

node:22 is a moving tag: it points at a different Node and a different Alpine as they are released, so a pipeline that passed last month can fail today without a change in the repository. An exact tag fixes the version. A digest fixes the bytes:

docker image inspect node:22.20.0-alpine3.22 --format '{{index .RepoDigests 0}}'

node@sha256:dbcedd8aeab47fbc0f4dd4bffa55b7c3c729a707875968d467aaaea42d6225af

default:
  image: node:22.20.0-alpine3.22@sha256:dbcedd8aeab47fbc0f4dd4bffa55b7c3c729a707875968d467aaaea42d6225af

The tag@digest form keeps the tag readable and makes the digest authoritative. Renovate or Dependabot can bump both together; a human reviewing the merge request sees which Node they are approving.

Pin the dependencies#

npm ci is the install command for CI. It installs exactly what the lockfile records and refuses to run when the lockfile and package.json disagree. Editing package.json without regenerating the lock produces this, before a single package is installed:

npm error code EUSAGE
npm error `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync.
npm error Invalid: lock file's semver@7.7.2 does not satisfy semver@7.7.1

That error is the feature. npm install would have resolved a new version, rewritten the lockfile inside the job, and produced a build that no commit describes. The same rule exists everywhere: pnpm install --frozen-lockfile, pip install -r requirements.txt with pinned versions (or uv sync --locked), go mod download with go.sum committed, bundle install --frozen.

Record what was built#

The build script writes the version and commit into the output:

const commit = process.env.CI_COMMIT_SHORT_SHA || 'local';
fs.writeFileSync(
  'dist/version.json',
  JSON.stringify({ version: pkg.version, commit }, null, 2) + '\n',
);

A running instance can then answer “what are you?” with the commit it was built from, which is the first question in every incident.

Caching without lying#

A cache is a speed-up for files that can be recreated: a package manager’s download directory, a compiler’s intermediate output. It is not a build output, and a pipeline must produce the same result with an empty cache. Three rules keep that true.

Key the cache on the thing that invalidates it. Keying on package-lock.json means a dependency change gets a fresh cache instead of a stale one:

variables:
  npm_config_cache: .npm # npm's download cache lives inside the workspace so the runner can save it

cache:
  key:
    files: [package-lock.json]
  paths: [.npm/]
  policy: pull # most jobs only read the cache

build:
  cache:
    key:
      files: [package-lock.json]
    paths: [.npm/]
    policy: pull-push # one job writes it

Cache the download directory, not node_modules. With npm ci --prefer-offline the install still runs, still validates the lockfile, and simply reads tarballs from .npm/ instead of the network. Caching node_modules and skipping the install is faster and wrong: it bypasses the lockfile check and can carry a stale native module across Node versions.

Let one job write. policy: pull-push on build and policy: pull everywhere else avoids three jobs racing to upload the same cache.

On the first run the cache is created after build; on the second, every job imports it before its script starts:

.npm/: found 17 artifact files and directories
build             cache created in '.gitlab-ci-local/cache/0_package-lock-b190b099c7e9af26d2df0122d8eb5d4bb444dbbf' in 354 ms

build             imported cache '0_package-lock-b190b099c7e9af26d2df0122d8eb5d4bb444dbbf' in 63 ms
lint              imported cache '0_package-lock-b190b099c7e9af26d2df0122d8eb5d4bb444dbbf' in 82 ms

The key contains a hash of the lockfile. Change a dependency and the hash changes, the old cache is simply not found, and npm ci downloads what it needs. The pipeline is slower for that one run and never wrong.

Artifacts, and what they are for#

Artifacts carry a job’s output to later jobs and to people. Three uses, three configurations:

build:
  artifacts:
    paths: [dist/]
    expire_in: 1 week # build output: consumed by later jobs, then discarded

unit-tests:
  artifacts:
    when: always # publish the report even when tests fail
    reports:
      junit: junit.xml # GitLab renders this in the merge request

package:
  artifacts:
    paths: ['app-*.tar.gz', 'app-*.sha256']
    expire_in: 30 days # the release artifact: kept long enough to roll back to

when: always matters more than it looks. The default (on_success) discards artifacts of a failed job, which is exactly when you want the test report. expire_in is a retention decision: build output can go quickly, release artifacts must outlive the deployment they might be rolled back to.

Artifacts are also the boundary between build and deploy. The package job produces one archive and its checksum; the deploy jobs receive them and verify before doing anything:

deploy-staging    $ sha256sum -c "app-${CI_COMMIT_SHORT_SHA}.sha256"
deploy-staging    > app-4e205ab8.tar.gz: OK

Ordering: needs, dependencies, and what runs in parallel#

Stages are a coarse ordering. needs is the precise one: a job with needs starts as soon as those jobs have finished, and receives only their artifacts.

lint:
  stage: test
  needs: [] # starts immediately; does not wait for build

unit-tests:
  stage: test
  needs: [build] # starts the moment build has exported dist/

package:
  stage: package
  needs: [build, unit-tests, lint]

In the run from part 1, lint started alongside build and finished before it, which is the point: the pipeline’s duration is its longest chain, not the sum of its stages.

Two refinements are worth knowing. needs: [{ job: build, artifacts: false }] orders a job after another without downloading its artifacts, useful for a large build output a job does not read. And when a stage has many independent jobs, parallel: 4 splits one job into four copies (each sees CI_NODE_INDEX), which is how test suites are sharded. Neither was needed for a project this size, and neither was exercised in these runs.

Pipelines can depend on pipelines as well. A trigger: job starts a child pipeline (from another YAML file in the same project) or a downstream pipeline (in another project) and, with strategy: depend, waits for its result. Use it when a monorepo’s components have genuinely separate pipelines, not to split one pipeline into pieces for tidiness.

Failure handling#

A job fails when a command exits non-zero, and by default everything that needs it is skipped. Design the exceptions explicitly.

package:
  timeout: 10m
  retry:
    max: 2
    when: [runner_system_failure, stuck_or_timeout_failure]

timeout bounds how long a hung job can hold a runner; the project-level default on GitLab.com is one hour, which is too long for a job that should take seconds. retry with a when list retries only infrastructure failures (the runner died, the job was stuck). Retrying on script_failure too is how flaky tests are hidden instead of fixed. Both settings are present in the sample pipeline and neither fired during these runs; a timeout is easy to test by adding sleep 700 to the script, which was not done here.

allow_failure: true lets a job go red without stopping the pipeline. It has one good use, an advisory check while its rules are being tuned, and one bad one, a gate nobody wanted to fix.

interruptible: true on the default lets GitLab cancel a running pipeline when a newer commit arrives on the same branch. Mark deploy jobs interruptible: false, or exclude them, so a deployment that has started is not killed halfway.

Keep build and deploy apart#

The build side of the pipeline is deterministic and can run anywhere, including on a merge request from a fork. The deploy side has credentials and side effects. Keeping them in separate jobs, joined by an artifact, means:

• deploy jobs can be restricted to protected branches (rules on $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH) while build and test run for every merge request;
• a deploy can be re-run without rebuilding, which is the mechanism behind rollback in part 4;
• the artifact can be scanned, signed and stored between the two, which is where the DevSecOps Pipeline path adds its gates.

Security Considerations#

• A pinned digest protects against a re-published tag; it does not protect against a compromised image. Scan the base image (Trivy) and keep the pin updated.
• npm ci verifies the integrity hashes in the lockfile. Commit the lockfile; a .gitignore entry for it disables that check.
• Report artifacts (junit.xml, scanner output) can contain source paths and, if not redacted, secrets. Keep artifact visibility limited to project members.

Troubleshooting#

Production Recommendations#

• Pin images by tag and digest; let a bot open the bump merge requests.
• Make the lockfile check non-negotiable: npm ci, --frozen-lockfile, --locked, whatever the ecosystem calls it.
• Key caches on lockfiles, cache download directories only, and let one job write.
• Give every job a timeout that reflects what it should take, and retry only infrastructure failures.
• Keep release artifacts for at least as long as you might roll back to them.

Conclusion#

Reliability is pinning what goes in, caching only what can be recreated, and deciding in advance what a failure means. With those in place the artifact at the end of the pipeline is worth deploying, which is what part 3 does next.

References#

• GitLab: caching in CI/CD
• GitLab: cache:key:files
• GitLab: retry, timeout and interruptible
• GitLab: downstream pipelines
• npm ci
• Docker: pull an image by digest