GitOps Deployment with Argo CD

Overview#

GitOps means the desired state of the cluster lives in Git and a controller continuously reconciles the cluster toward it. Argo CD is the most widely used controller for this model. The payoff is not just automation: every change has an author, a review and a revert path, and the cluster credentials that used to live in CI runners disappear.

This article sets up Argo CD with automated sync, pruning and self-heal, scopes what each team can deploy with AppProject, and hardens the Argo CD control plane itself.

Architecture#

A developer merges application code. CI builds, scans and pushes an image to the registry, then opens a merge request that updates the image digest in a separate deploy repository. Argo CD watches the deploy repository, renders the manifests, compares them with the live cluster state and applies the difference. Nothing in CI holds cluster credentials; the only path into the cluster is a merged commit in the deploy repository.

1. CI pipeline. Builds the image, scans it, pushes it to the registry. It has registry credentials and Git credentials for the deploy repository. It has no cluster credentials.
2. Container registry. Holds images by digest. The cluster pulls from here; CI never pushes anything into the cluster directly.
3. Deploy repository. Kubernetes manifests (Kustomize overlays or Helm values) with image digests pinned. CI opens a merge request here to bump a digest; a human or an automated policy merges it. This merge is the production change.
4. Argo CD. Runs inside the cluster, polls the deploy repository (or receives a webhook), renders manifests, diffs them against the live state and applies the difference. With self-heal enabled it also reverts manual changes.
5. Kubernetes. The AppProject scopes what Argo CD may touch: which repositories, which namespaces, which resource kinds. That boundary is what stops a deploy repository from becoming a path to ClusterRoleBinding.

The trust boundary is the cluster edge. Only Argo CD’s service account applies manifests, and the only way to change what it applies is a commit in the deploy repository.

Everything below was run against Argo CD 3.5.2 on a kind cluster, using the upstream example application repository as the deploy repository; outputs are real.

Prerequisites

• A Kubernetes cluster (kind/minikube is fine) and kubectl access
• A Git repository for environment manifests (Kustomize or Helm)
• The argocd CLI (brew install argocd)

Implementation#

Install#

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl -n argocd rollout status deploy/argocd-server --timeout=600s

For production use the HA manifests (manifests/ha/install.yaml) or the Helm chart with redis-ha enabled and pin a specific release instead of stable.

The stable manifest at the time of writing installs Argo CD 3.5.2: seven workloads, of which the application controller (a StatefulSet), the repo server and the API server are the ones that matter for what follows.

Get the initial admin password and log in through a port-forward (no public exposure yet):

argocd admin initial-password -n argocd
kubectl -n argocd port-forward svc/argocd-server 8080:443 &
argocd login localhost:8080 --username admin --insecure   # local port-forward only

Everything in this article can also be done with kubectl alone, because Applications and AppProjects are custom resources. The CLI is convenient for diff and sync; it is not required.

Define a project boundary first#

An AppProject limits which repositories and destinations an application may use. Create one per team before creating applications.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: payments
  namespace: argocd
spec:
  description: Payments team workloads
  sourceRepos:
    - https://gitlab.com/example/payments-deploy.git
  destinations:
    - namespace: payments
      server: https://kubernetes.default.svc
  clusterResourceWhitelist: [] # no cluster-scoped resources
  namespaceResourceBlacklist:
    - group: ''
      kind: ResourceQuota
    - group: ''
      kind: LimitRange
  roles:
    - name: deployer
      policies:
        - p, proj:payments:deployer, applications, sync, payments/*, allow
      groups:
        - payments-engineers # SSO group

clusterResourceWhitelist: [] is the line that matters most. It means the project may not create any cluster-scoped resource, and that includes Namespace. The consequence, which the verification section shows, is that the namespace must exist before the first sync, created by the platform team rather than by the application’s own sync. That is the correct division: creating namespaces is a cluster-level act.

The Application#

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-api
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io # delete cluster resources when the app is deleted
spec:
  project: payments
  source:
    repoURL: https://gitlab.com/example/payments-deploy.git
    targetRevision: main
    path: apps/payments-api/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: payments
  syncPolicy:
    automated:
      prune: true # delete resources removed from Git
      selfHeal: true # revert manual kubectl changes
    syncOptions:
      - CreateNamespace=false # the project forbids cluster-scoped resources; see below
      - ServerSideApply=true
      - PrunePropagationPolicy=foreground
    retry:
      limit: 3
      backoff: { duration: 10s, factor: 2, maxDuration: 2m }

Apply both and watch the first sync:

kubectl apply -f project-payments.yaml -f app-payments-api.yaml
argocd app get payments-api --refresh
argocd app sync payments-api        # only needed the first time if automated sync is off

Helm sources#

For Helm-based apps point source at the chart and keep environment values in Git:

source:
  repoURL: https://gitlab.com/example/payments-deploy.git
  targetRevision: main
  path: charts/payments-api
  helm:
    valueFiles: [values.yaml, values-prod.yaml]

Argo CD renders the chart with helm template; it does not use Helm releases or hooks the same way helm install does — Helm hooks are mapped to Argo CD sync hooks. Keep that in mind if a chart relies on post-install jobs.

Configuration#

Ordering — use argocd.argoproj.io/sync-wave annotations (CRDs and namespaces at -1, databases at 0, apps at 1).

Ignore expected drift — mutating webhooks and HPAs change fields after apply. Declare it rather than fighting it:

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers: [/spec/replicas] # managed by HPA

App of apps — a bootstrap Application that points at a directory of Application manifests lets you recreate an entire cluster from one command. Combine it with ApplicationSet generators for many-cluster fleets.

Verification#

The run below used the upstream argocd-example-apps repository’s guestbook path as the deploy repository, with the project and application above adjusted to point at it.

The project boundary is real#

With CreateNamespace=true in the sync options and clusterResourceWhitelist: [] in the project, the first sync does not create the namespace; it fails, and says why:

kubectl -n argocd get application guestbook -o jsonpath='{.status.operationState.message}'

one or more synchronization tasks are not valid: resource :Namespace is not permitted in project payments. Retrying attempt #3 at 9:25PM.

Creating the namespace out of band and setting CreateNamespace=false lets the sync through:

kubectl create namespace payments
kubectl -n argocd get application guestbook -o json | jq '{sync: .status.sync.status, health: .status.health.status, phase: .status.operationState.phase}'

{"sync": "Synced", "health": "Progressing", "phase": "Succeeded"}

Progressing becomes Healthy once the pods pass their readiness checks. That progression is what the argocd app get view and the UI show; the fields are the same.

Self-heal reverts manual changes#

kubectl -n payments scale deploy/guestbook-ui --replicas=3
kubectl -n payments get deploy guestbook-ui -o jsonpath='{.spec.replicas}'; sleep 20
kubectl -n payments get deploy guestbook-ui -o jsonpath='{.spec.replicas}'

deployment.apps/guestbook-ui scaled
3
1

The scale command is a manual change to live state. With selfHeal: true, Argo CD’s watch on the Deployment sees the change, marks the application OutOfSync, starts an automated sync and reapplies the manifest, all within the twenty seconds. The application’s events tell the story:

OperationStarted     Initiated automated sync to '8088f4c0d970abb09e250248cc97e35623447cb5'
ResourceUpdated      Updated sync status: Synced -> OutOfSync
OperationCompleted   Partial sync operation to 8088f4c0d970abb09e250248cc97e35623447cb5 succeeded
ResourceUpdated      Updated sync status: OutOfSync -> Synced

Without self-heal the application would show OutOfSync and stay at 3 until someone synced.

A destination outside the project is refused#

kubectl -n argocd apply -f - <<'YAML'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: escape, namespace: argocd }
spec:
  project: payments
  source: { repoURL: https://github.com/argoproj/argocd-example-apps.git, targetRevision: HEAD, path: guestbook }
  destination: { server: https://kubernetes.default.svc, namespace: kube-system }
YAML

The object is admitted (it is just a custom resource), but Argo CD refuses to work with it and records why:

kubectl -n argocd get application escape -o jsonpath='{.status.conditions[0].message}'
kubectl -n argocd delete application escape

application destination server 'https://kubernetes.default.svc' and namespace 'kube-system' do not match any of the allowed destinations in project 'payments'

The condition type is InvalidSpecError; nothing is synced. This is the boundary that makes it safe to let application teams create their own Application objects: the project, owned by the platform team, decides where they can land.

Security Considerations#

• Disable the built-in admin after configuring SSO (Dex or OIDC): argocd-cm → admin.enabled: "false".
• RBAC in argocd-rbac-cm: default role role:readonly; grant sync per project via SSO groups, never role:admin to groups.
• Secrets are not GitOps-friendly in plaintext. Use External Secrets Operator or SOPS-encrypted manifests decrypted in-cluster; never commit Secret objects with real data.
• Repository credentials should be read-only deploy tokens scoped to the deploy repo.
• Webhooks from GitLab/GitHub to Argo CD must carry a secret; otherwise anyone can trigger refreshes.
• Restrict clusterResourceWhitelist per project so a team cannot create ClusterRoleBindings through GitOps.

Troubleshooting#

Production Recommendations#

• Separate application repos (code) from deploy repos (manifests); CI opens merge requests against the deploy repo to bump image digests.
• Turn on notifications (Slack/Teams) for OnSyncFailed and OnHealthDegraded.
• Back up Application, AppProject and the Argo CD config maps — they are the cluster’s source of truth for delivery.
• Pin Argo CD to a specific minor version and upgrade on a schedule; read the upgrade notes for RBAC changes.
• Once there is more than one environment, generate the Applications instead of copying them: Argo CD ApplicationSets for Multi-Environment Delivery.

Conclusion#

Argo CD changes the question from “who ran kubectl apply?” to “who merged that?”. The install is a few commands; the value comes from project boundaries, automated sync with self-heal, and treating Argo CD itself as a privileged system.

References#

• Argo CD documentation
• Argo CD security
• Sync options · Diffing customization