Argo CD ApplicationSets for Multi-Environment Delivery

Overview#

An Argo CD Application points one path in one repository at one namespace. With three environments that is three Applications that differ in four fields and are identical everywhere else. With ten services that is thirty, and the first time someone fixes a sync option in twenty-nine of them, the drift has begun. An ApplicationSet replaces the copies with a template and a generator: the template says what every Application looks like, the generator says how many there are and what differs.

GitOps Deployment with Argo CD, part 3 of the Secure Kubernetes path, covers the install, the AppProject boundary, automated sync with prune and self-heal, and hardening the control plane. This article assumes that and goes one level up: one ApplicationSet that produces the development, staging and production Applications for a service, with different sync behaviour per environment, and the promotion, boundary and deletion behaviour that follows. Everything below was run on a kind 0.31 cluster with Argo CD 3.5.3 and a Gitea 1.24.7 container as the Git server; the outputs are from that run.

A Git repository holds one application base and one directory per environment, each with a small config file. The ApplicationSet controller reads those files with the Git file generator and produces one Argo CD Application per environment, in the AppProject the file names. Development and staging Applications sync automatically with prune and self-heal; the production Application waits for a person to trigger the sync. Each Application deploys into its own namespace, and the prod AppProject refuses any other destination.

1. The repository holds one Kustomize base per application and one directory per environment. Each environment directory contains an overlay and a small config.json with the values the ApplicationSet needs: the environment name, the target namespace, the AppProject and whether sync is automated.
2. The ApplicationSet uses the Git file generator to read every config.json. Each file becomes one set of parameters and one generated Application. A new environment is a new directory; nothing else changes.
3. Development and staging are generated with syncPolicy.automated (prune and self-heal), so a merged commit reaches the cluster within the polling interval and manual drift is reverted.
4. Production is generated without automated sync. It shows OutOfSync when Git moves ahead and stays that way until a person triggers the sync. The gate is the review that precedes the commit plus the deliberate sync after it.
5. Each environment lands in its own namespace, and the prod AppProject lists only the production namespace as an allowed destination, so a mistake in a config file cannot point production at another environment.

Repository layout#

gitops/
├── apps/
│   └── ci-demo/
│       ├── base/
│       │   ├── deployment.yaml
│       │   ├── service.yaml
│       │   └── kustomization.yaml
│       └── envs/
│           ├── dev/    kustomization.yaml  config.json
│           ├── stage/  kustomization.yaml  config.json
│           └── prod/   kustomization.yaml  config.json
├── appsets/
│   └── ci-demo.yaml
└── projects/
    ├── non-prod.yaml
    └── prod.yaml

The base is the application as it is everywhere: a Deployment with probes and resource requests, a Service, and an envFrom pointing at a ConfigMap the overlay generates. The overlay is what differs per environment, and it is deliberately short:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: ci-demo-prod
resources:
  - ../../base
images:
  - name: ci-demo
    newTag: '1.4.1' # the version production runs; promotion changes this line
replicas:
  - name: ci-demo
    count: 2
configMapGenerator:
  - name: ci-demo-config
    literals:
      - APP_ENV=prod
      - LOG_LEVEL=warn

{
  "env": "prod",
  "namespace": "ci-demo-prod",
  "project": "prod",
  "automated": false
}

Two files per environment, each with one job. The kustomization decides what runs (image tag, replicas, configuration); the config file decides how Argo CD treats it (project, namespace, sync policy). The split matters because the people who change them are often different: an application team bumps newTag, a platform team changes what automated means.

The ApplicationSet#

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: ci-demo
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ['missingkey=error']
  # An environment that disappears from Git is left running, not deleted: removal is a deliberate, separate step.
  syncPolicy:
    applicationsSync: create-update
  generators:
    - git:
        repoURL: http://gitea:3000/platform/gitops.git
        revision: main
        files:
          - path: 'apps/ci-demo/envs/*/config.json'
  template:
    metadata:
      name: 'ci-demo-{{ .env }}'
      labels: { app: ci-demo, env: '{{ .env }}' }
    spec:
      project: '{{ .project }}'
      source:
        repoURL: http://gitea:3000/platform/gitops.git
        targetRevision: main
        path: '{{ .path.path }}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{ .namespace }}'
      syncPolicy:
        syncOptions: [CreateNamespace=true]
  templatePatch: |
    {{- if .automated }}
    spec:
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions: [CreateNamespace=true]
    {{- end }}

Four decisions are encoded here.

The generator is the Git file generator, not the directory generator. Both would find the three environment directories; the file generator additionally reads each file’s keys into the template parameters, so the namespace and project come from the environment itself rather than from a naming convention (ci-demo-{{ .path.basename }}). The file’s directory is still available as .path.path and is used as the source path. A list generator would have put the same values inline in the ApplicationSet, which works for three environments and stops working when the application team needs to change one without editing a platform-owned resource. A matrix generator (Git directories × a list of clusters) is the next step when the same environments exist on several clusters; it was not needed here.

missingkey=error makes a config file that forgets a key fail generation with a message that names the key, instead of rendering a placeholder into project: or namespace: and letting Argo CD reject, or worse accept, the result later.

templatePatch is where the sync policy differs. The template cannot express “automated for some environments”, because a template is one shape; the patch is a Go template rendered per set of parameters and merged over the generated Application. Only the environments whose file says "automated": true get the block. The alternative is two ApplicationSets with two generators filtering on a field, which is the older pattern and works on versions without templatePatch.

applicationsSync: create-update changes what happens when a generator stops producing an environment. The last section shows both behaviours.

The projects are the ones from the earlier article, with one difference: prod allows exactly one destination namespace.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: prod
  namespace: argocd
spec:
  description: Production. Manual sync only; changes arrive through reviewed merges.
  sourceRepos: ['http://gitea:3000/platform/gitops.git']
  destinations:
    - { server: https://kubernetes.default.svc, namespace: 'ci-demo-prod' }
  clusterResourceWhitelist:
    - { group: '', kind: Namespace }

What gets generated#

kubectl apply -f projects/ && kubectl apply -f appsets/ci-demo.yaml
kubectl -n argocd get applications -o custom-columns='NAME:.metadata.name,PROJECT:.spec.project,PATH:.spec.source.path,NS:.spec.destination.namespace,AUTOMATED:.spec.syncPolicy.automated,SYNC:.status.sync.status,HEALTH:.status.health.status'

NAME            PROJECT    PATH                      NS              AUTOMATED                       SYNC        HEALTH
ci-demo-dev     non-prod   apps/ci-demo/envs/dev     ci-demo-dev     map[prune:true selfHeal:true]   Synced      Healthy
ci-demo-prod    prod       apps/ci-demo/envs/prod    ci-demo-prod    <none>                          OutOfSync   Missing
ci-demo-stage   non-prod   apps/ci-demo/envs/stage   ci-demo-stage   map[prune:true selfHeal:true]   Synced      Healthy

Twenty seconds after the apply: two environments running, production generated and waiting. What each namespace actually runs, read back from the cluster rather than from Git:

ci-demo-dev    replicas=1/1 image=ci-demo:1.4.2 APP_ENV=dev   LOG_LEVEL=debug
ci-demo-stage  replicas=2/2 image=ci-demo:1.4.2 APP_ENV=stage LOG_LEVEL=debug
ci-demo-prod   (nothing yet)

Syncing production by hand#

Argo CD’s CLI and UI trigger a sync by writing an operation field on the Application; kubectl can do the same, which is useful in a pipeline that is not allowed to hold an Argo CD token:

kubectl -n argocd patch app ci-demo-prod --type merge -p '{
  "operation": {
    "initiatedBy": { "username": "release-manager" },
    "sync": { "revision": "main", "prune": false, "syncOptions": ["CreateNamespace=true"] }
  }
}'

sync=Synced health=Healthy phase=Succeeded revision=c8297ca… by=release-manager
ci-demo-prod   replicas=2/2 image=ci-demo:1.4.1

The first attempt at this failed with namespaces "ci-demo-prod" not found: a manual operation carries its own syncOptions, and the CreateNamespace=true in the Application’s syncPolicy applies to automated syncs, not to an operation written by hand. The retry above includes it. The initiatedBy.username lands in the Application’s status.history, which is the audit trail for who released what:

ID  REVISION   DEPLOYED AT            BY
0   c8297ca…   2026-09-16T05:20:13Z   release-manager

Promotion is a commit#

The image tag lives in the overlay, so promoting a version is changing one line in the next environment’s kustomization and merging it. The same commit that would be reviewed for any other change:

sed -i 's/newTag: "1.4.1"/newTag: "1.4.2"/' apps/ci-demo/envs/prod/kustomization.yaml
git commit -am "prod: promote ci-demo to 1.4.2" && git push

ci-demo-prod   sync=OutOfSync   deployment still runs ci-demo:1.4.1

Production noticed the commit (after a refresh; the default poll is three minutes) and did nothing. OutOfSync is the whole gate: Git says 1.4.2, the cluster says 1.4.1, and a person with permission to sync decides when they meet. Development, on the same repository a minute later:

sed -i 's/count: 1/count: 2/' apps/ci-demo/envs/dev/kustomization.yaml
git commit -am "dev: two replicas" && git push

ci-demo-dev    sync=Synced revision=7b90a4b… auto=true   replicas 2/2

And self-heal, which is what makes “automated” safe rather than merely convenient:

kubectl -n ci-demo-dev scale deploy ci-demo --replicas=5
# twenty seconds later
kubectl -n ci-demo-dev get deploy ci-demo -o jsonpath='{.spec.replicas}'   # 2

The promotion model this gives you is linear and boring, which is the point: a version reaches staging by a commit to envs/stage, reaches production by a commit to envs/prod, and both commits are diffs against the environment they change. Rollback is the reverse commit. git revert on the promotion commit puts 1.4.1 back in Git; for an automated environment that is the rollback, and for production it is a second manual sync. Argo CD also keeps the last ten synced revisions in status.history and can sync to any of them, but a rollback that exists only in Argo CD’s history and not in Git is a drift waiting to be self-healed away. The mechanics of rolling back the workload itself, including the database problem, are in Rollback and Recovery.

The boundary holds#

A config file is easy to get wrong. This commit points production at the staging namespace:

{ "env": "prod", "namespace": "ci-demo-stage", "project": "prod", "automated": false }

The ApplicationSet regenerated ci-demo-prod with the new destination, and Argo CD refused it:

dest=ci-demo-stage sync=Unknown
InvalidSpecError: application destination server 'https://kubernetes.default.svc' and namespace 'ci-demo-stage'
  do not match any of the allowed destinations in project 'prod'
stage untouched: image=ci-demo:1.4.2 replicas=2

Nothing was applied anywhere, the staging Deployment was not touched, and the error names the project rule that stopped it. Reverting the commit restored dest=ci-demo-prod on the next generation. The AppProject is the control; the ApplicationSet template is only a convenience, and a template that let the config file choose the project and the project allowed every namespace would have no boundary at all. Keep the set of projects small and platform-owned, and let the config file pick from them.

When an environment disappears#

Delete apps/ci-demo/envs/stage/ and push. With the default applicationsSync (sync), the ApplicationSet controller deleted the ci-demo-stage Application within one generation cycle, and because generated Applications carry the resources finalizer, the Deployment, Service and ConfigMap in ci-demo-stage went with it (the namespace itself stayed: CreateNamespace creates it but does not track it):

application gone after ~195s
ci-demo-dev ci-demo-prod
No resources found in ci-demo-stage namespace.

That is correct behaviour and also the way a mistaken git rm, a bad rebase or a broken generator path takes an environment offline. The ApplicationSet was then changed to applicationsSync: create-update (the version shown above) and the directory deleted a second time:

ci-demo-dev ci-demo-prod ci-demo-stage
still present: sync=Synced health=Healthy
ci-demo   2/2   2     2     3m51s

The Application stayed, the workload stayed, and the ApplicationSet’s conditions reported success. Removing the environment is now a second, explicit step (kubectl delete application ci-demo-stage), which is the right cost for something that cannot be undone by a revert. create-only goes further and never updates a generated Application either; preserveResourcesOnDeletion: true keeps the workload but deletes the Application. For production, create-update is the floor.

Secrets and configuration#

The overlays above hold configuration, not secrets, and that is a rule rather than a limitation of the layout. A Secret in a Kustomize overlay is a secret in Git, which Secrets and Configuration Security explains the cost of. The patterns that fit this repository: a SealedSecret or an External Secrets ExternalSecret object in the overlay (a reference, not a value), or the application reading from a secret manager with its own workload identity (Part 1 of Cloud Security Foundations). What must never be in the environment directory is the value itself, however many people can read the repository today.

Argo CD itself needs a repository credential. In this lab the Gitea repository was public and the ApplicationSet pulled it anonymously; in production the credential is a repository Secret in the argocd namespace with a read-only deploy token, created outside Git.

Security implications#

• The generator reads files from the revision it polls. A branch protection on main and a review on envs/prod/ are what stand between a commit and a production Application; the ApplicationSet does not distinguish a reviewed commit from a forced push.
• goTemplate with missingkey=error fails loudly on a bad file. Whether a missing project key would otherwise end up as default, the project with no restrictions, depends on how the rendered value is parsed; failing generation removes the question.
• Manual sync on production is a gate only if the people who can write operation on the Application are the people who should release. Argo CD RBAC (applications, sync, prod/*) and Kubernetes RBAC on the applications.argoproj.io resource both need to say so.
• applicationsSync: create-update protects against generator mistakes, not against a bad commit to an environment that still exists. Self-heal on non-production means a bad merge is live in minutes; that is the trade the automated environments make on purpose.

Troubleshooting#

Running this in production#

• One ApplicationSet per application (or per team), generated from files the application team owns, with projects the platform team owns. The generator’s path glob is the contract between them.
• Start every ApplicationSet with applicationsSync: create-update; loosen it for ephemeral preview environments, where deleting on branch removal is the feature.
• Alert on argocd_app_info with a sync_status of OutOfSync for production Applications older than the release window, and on ApplicationSet ErrorOccurred conditions. The observability article covers where those metrics go.
• Add a Git webhook so generation and sync react to a push instead of waiting for the poll; three minutes is a long time to watch a pipeline.

References#

• Argo CD: ApplicationSet controller
• Argo CD: Git generator (files and directories)
• Argo CD: template patch
• Argo CD: ApplicationSet policies (applicationsSync) and resource deletion
• Argo CD: projects and sync options
• Kustomize: images, replicas and configMapGenerator