GitOps Interview Questions
10 questions for junior, mid-level and senior GitOps interviews, with model answers, code examples and common mistakes. Answer each one out loud first, then reveal the answer.
Bootstrapping Argo CD and Letting It Manage Itself
// interview question
Argo CD manages your apps. Who manages Argo CD? Walk me through how you would bootstrap it from a fresh cluster and where its own config lives in your repo.
Sample answer
Argo CD has a chicken-and-egg problem on day one. The cluster has nothing in it, so kubectl is the only tool that can install the first thing. After that, you want Argo CD to manage itself the same way it manages everything else. The flow I use: 1. One bootstrap kubectl apply puts Argo CD into the cluster. I use the upstream install manifest pinned to a version, not latest. That gets you the namespace, the CRDs, the controller, the repo server, the API server. 2. Once Argo CD is up, I apply one Application CR by hand. This Application points at a directory in the manifests repo, something like argocd/bootstrap. That directory contains another Application that points at Argo CD's own Helm chart or Kustomize overlay. So Argo CD starts syncing itself. 3. The same bootstrap directory contains AppProjects, repository credentials (as sealed secrets or external secrets), and a parent Application that points at argocd/applications, which holds every other Application CR for the cluster. After that initial kubectl apply, you never touch Argo CD with kubectl again. Bumping the Argo CD version is a Git commit changing the chart version or image tag in argocd/bootstrap. Adding a new app to the cluster is a Git commit dropping a new Application YAML into argocd/applications. Repo layout I keep for this: argocd/ bootstrap/ # Argo CD itself, plus the root app-of-apps kustomization.yaml values.yaml # Helm values for the argo-cd chart root-app.yaml # the parent Application that points at applications/ projects/ # AppProjects: per-team or per-environment applications/ # one Application YAML per workload or per env One gotcha. Argo CD syncing itself can lock you out if a bad change rolls in. If the controller fails to start because of a config change you just merged, no controller means no reconcile, which means no rollback through Git. Keep the bootstrap Application on manual sync, or at minimum keep a tested rollback commit ready. Some teams keep a separate management cluster running its own Argo CD that watches the workload clusters' Argo CDs, so a broken self-managed instance can still be repaired through GitOps.
Why this matters
Most junior candidates can install Argo CD. Fewer can explain how it ends up managing itself, and even fewer have thought about what happens when self-management goes wrong. This question separates someone who has watched a tutorial from someone who has set up Argo CD in a real environment. Listen for the bootstrap kubectl apply being acknowledged as a one-time thing, a clear repo location for Argo CD's own config, and any awareness of the self-lockout risk.
Code examples
One-time bootstrap commands
# 1. Install Argo CD itself with kubectl (one time only)
kubectl create namespace argocd
kubectl apply -n argocd \
-f https://raw.githubusercontent.com/argoproj/argo-cd/v2.11.0/manifests/install.yaml
# Wait for the controller to be ready
kubectl wait --for=condition=available --timeout=300s \
deployment/argocd-server -n argocd
# 2. Apply the root Application that makes Argo CD self-manage
kubectl apply -f argocd/bootstrap/root-app.yaml
# After this point, no more kubectl. Everything is Git.Root Application that bootstraps the rest
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: root
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: https://github.com/org/manifests
targetRevision: main
path: argocd/applications
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: true
selfHeal: trueArgo CD managing its own install via Helm
# argocd/applications/argocd-self.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: argocd
namespace: argocd
spec:
project: platform
source:
repoURL: https://argoproj.github.io/argo-helm
chart: argo-cd
targetRevision: 7.3.0
helm:
valueFiles:
- $values/argocd/bootstrap/values.yaml
sources:
- repoURL: https://github.com/org/manifests
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
# No automated sync: a bad change to Argo CD itself should not auto-deploy
syncOptions:
- ServerSideApply=trueCommon mistakes to avoid
- Installing Argo CD with kubectl, then continuing to use kubectl for every config change instead of having it manage itself
- Putting Argo CD's own Application on the same auto-sync settings as workloads, then getting locked out when a bad chart bump crashes the controller
- Forgetting about the repository credentials problem and hard-coding tokens into Git on day one
Likely follow-ups
- What happens if you merge a bad change to the Argo CD chart values and the controller crashloops? How do you recover?
- Would you put Argo CD on auto-sync for itself? Why or why not?
- How do you handle Argo CD version upgrades across many clusters?
- Where do the credentials that Argo CD uses to read your private manifests repo come from? They cannot be in the same repo in plaintext.
Answer out loud first, then check yourself against the model answer.
Structuring a Git Repo for Argo CD Multi-Environment Deployments
// interview question
How would you structure a Git repo for Argo CD when you have dev, staging, and prod environments?
Sample answer
Two common patterns exist: environment-per-directory and environment-per-branch. Pick directories. Branches per environment create merge pain, cherry-picking, and drift. Use one branch (usually main) with separate directories for each environment. A layout that works for most teams: apps/ myapp/ base/ # shared manifests (Deployment, Service, ConfigMap) kustomization.yaml overlays/ dev/ # dev patches: replicas, image tag, hostname staging/ prod/ Each environment gets its own Argo CD Application pointing at its overlay path. Base holds what is the same everywhere. Overlays patch only what differs (replicas, resource requests, image tags, ingress hosts, ConfigMap values). If you grow past 20 to 30 environments or clusters, switch from hand-written Application CRs to an ApplicationSet with a Git directory generator so you do not write the same boilerplate over and over.
Why this matters
This is one of the first questions in any Argo CD interview. It separates candidates who have read a tutorial from those who have run this in production. Listen for: a clear preference for directories over branches with a real reason (merge complexity, drift, audit clarity), separation of base from environment-specific patches, and awareness that the layout has to scale past the first three environments. If they immediately reach for branches-per-env, dig in and see if they have actually tried it.
Code examples
Repo layout for a single app across three environments
apps/
web/
base/
deployment.yaml
service.yaml
configmap.yaml
kustomization.yaml
overlays/
dev/
kustomization.yaml
replicas-patch.yaml
staging/
kustomization.yaml
replicas-patch.yaml
prod/
kustomization.yaml
replicas-patch.yaml
hpa.yaml
argocd/
applications/
web-dev.yaml
web-staging.yaml
web-prod.yamlArgo CD Application pointing at the prod overlay
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: web-prod
namespace: argocd
spec:
project: production
source:
repoURL: https://github.com/org/manifests
targetRevision: main
path: apps/web/overlays/prod
destination:
server: https://kubernetes.default.svc
namespace: web
syncPolicy:
syncOptions:
- CreateNamespace=trueCommon mistakes to avoid
- Using a branch per environment and then drowning in cherry-picks and merge conflicts
- Putting environment-specific values in a single Helm values file with conditional blocks for prod, then forgetting which branch sets what
- Pointing all three environments at the same Argo CD Application path so a dev change ships to prod
Likely follow-ups
- Why not use a branch per environment? People still recommend that on blog posts.
- How do you handle a config value that has to differ in every single environment?
- What changes when dev and prod live in different clusters?
- When would you switch from one Application per environment to an ApplicationSet?
Answer out loud first, then check yourself against the model answer.
Application Code vs Kubernetes Manifests in Separate Repos
// interview question
Should your application source code and Kubernetes manifests live in the same repo or in separate repos? What is your take?
Sample answer
Default to separate repos. The application code repo holds Go, Python, Node, whatever. A second manifests repo holds the Argo CD Applications, Kustomize overlays, and any cluster config. This is the pattern most production GitOps setups end up using, and there are four good reasons for it. First, blast radius and permissions. Devs commit code freely. Manifests changes for prod usually want code owners, branch protection, and a smaller set of reviewers. Splitting the repos lets you set those rules cleanly. Second, noise. Argo CD watches the manifests repo. If manifests share a repo with app code, every README edit shows up as a change Argo CD has to decide what to do with. Third, image promotion loops. CI builds an image from the code repo and writes the new tag into the manifests repo. If both live in one repo, the bump commit triggers another CI run, which can build, push, and bump again. You end up adding skip-ci tags everywhere to stop the loop. Fourth, manifests are usually shared. A monitoring stack, ingress controller, and three apps all live in one manifests repo. They do not belong with any single app's code. When the monorepo is fine: one team, one app, no compliance separation. Do not over-engineer that case.
Why this matters
This question reveals whether a candidate has shipped GitOps in production or only read the docs. Strong answers cite specific operational problems (CI loops, noisy reconciliation, permission boundaries) rather than vague principles like separation of concerns. Watch out for candidates who insist on one model as universal truth. The right answer is conditional, and the conditions matter.
Code examples
CI step that bumps the image tag in the manifests repo
# .github/workflows/build.yml
name: build-and-promote
on:
push:
branches: [main]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build and push image
run: |
IMAGE_TAG=$(git rev-parse --short HEAD)
docker build -t ghcr.io/org/web:${IMAGE_TAG} .
docker push ghcr.io/org/web:${IMAGE_TAG}
echo "IMAGE_TAG=${IMAGE_TAG}" >> $GITHUB_ENV
- name: Bump image in manifests repo
env:
GH_TOKEN: ${{ secrets.MANIFESTS_REPO_TOKEN }}
run: |
git clone https://x-access-token:${GH_TOKEN}@github.com/org/manifests.git
cd manifests/apps/web/overlays/dev
kustomize edit set image web=ghcr.io/org/web:${IMAGE_TAG}
git config user.email bot@example.com
git config user.name ci-bot
git add kustomization.yaml
git commit -m "web: bump dev to ${IMAGE_TAG}"
git pushTwo-repo layout
# Repo 1: org/web (application code)
src/
Dockerfile
Makefile
.github/workflows/build.yml
# Repo 2: org/manifests (Kubernetes manifests, watched by Argo CD)
argocd/
applications/
web-dev.yaml
web-prod.yaml
apps/
web/
base/
overlays/
dev/
prod/
platform/
ingress-nginx/
cert-manager/
prometheus/Common mistakes to avoid
- Giving a single deploy key write access to the whole manifests repo, so any compromised CI job can push to prod
- Mixing platform infrastructure manifests and per-app manifests in one folder with no clear ownership
- Insisting on monorepo or polyrepo as a religion instead of matching the choice to team size and compliance needs
Likely follow-ups
- How does the CI pipeline write to the manifests repo without ending up in an infinite loop?
- What does Argo CD Image Updater do, and would you use it for prod?
- How do you keep manifest changes in sync with app changes that need both to deploy together?
- What is the right scope for one manifests repo? One per team? One per cluster? One global?
Answer out loud first, then check yourself against the model answer.
Helm vs Kustomize for Per-Environment Config
// interview question
Helm or Kustomize for handling environment differences in an Argo CD repo? Walk me through how you would pick and what the repo layout looks like for each.
Sample answer
Default to Kustomize for your own apps. Use Helm for third-party software you do not own. Use both together when you have a Helm chart from a vendor but need environment-specific tweaks the chart does not expose as values. Reasoning. Kustomize patches a flat YAML base. What you see is what gets applied. No templating engine, no Go template debugging, no quoting traps. The downside is no logic, no loops, no conditionals. That is fine for your own apps because you control the base. Helm shines when you do not own the base. cert-manager, ingress-nginx, kube-prometheus-stack: each ships a chart with hundreds of toggles. Reimplementing that in Kustomize is a waste of time. Pin the chart version, write a values file per environment, done. Repo layout with Kustomize (your own apps): apps/ web/ base/ deployment.yaml service.yaml kustomization.yaml overlays/ dev/ kustomization.yaml prod/ kustomization.yaml Repo layout with Helm (third-party): platform/ ingress-nginx/ values-base.yaml values-dev.yaml values-prod.yaml cert-manager/ values-base.yaml values-prod.yaml Argo CD Application points at the chart in its repo (or an OCI registry) and references the right values files from your manifests repo using the multi-source feature. The combined pattern, Helm chart rendered by Kustomize, looks like this: platform/ingress-nginx/ base/ kustomization.yaml # uses helmCharts to pull and render the chart values.yaml # base values overlays/ prod/ kustomization.yaml # patches the rendered output values.yaml # prod-specific values Kustomize renders the chart with its values, then your overlay patches the rendered output. This is what you reach for when the chart does not let you change something through values and forking the chart is worse than patching the output. One thing not to do. Do not build your own apps as a Helm chart just so you can use values files per environment. That is the most common 'we picked Helm and regretted it' story. You will spend weeks debugging Go template indentation. Use Kustomize.
Why this matters
This is a question where opinions matter, and a strong candidate will have one. The instinct to say 'it depends' is fine if it is followed by specific criteria. Listen for the ownership criterion: who controls the base manifests determines whether templating logic earns its keep. Bonus points for awareness of the multi-source Application feature in Argo CD, and for mentioning the combined helmCharts inside Kustomize pattern, which is widely used in production but rarely taught.
Code examples
Argo CD Application with multi-source: chart from one repo, values from manifests repo
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: ingress-nginx-prod
namespace: argocd
spec:
project: platform
sources:
- repoURL: https://kubernetes.github.io/ingress-nginx
chart: ingress-nginx
targetRevision: 4.10.0
helm:
valueFiles:
- $values/platform/ingress-nginx/values-base.yaml
- $values/platform/ingress-nginx/values-prod.yaml
- repoURL: https://github.com/org/manifests
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: ingress-nginx
syncPolicy:
syncOptions:
- CreateNamespace=trueKustomize rendering a Helm chart so you can patch the output
# platform/ingress-nginx/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
helmCharts:
- name: ingress-nginx
repo: https://kubernetes.github.io/ingress-nginx
version: 4.10.0
releaseName: ingress-nginx
namespace: ingress-nginx
valuesFile: values.yamlProd overlay patching the rendered chart output
# platform/ingress-nginx/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- target:
kind: Deployment
name: ingress-nginx-controller
patch: |
- op: add
path: /spec/template/spec/nodeSelector
value:
workload: ingress
- op: add
path: /spec/template/spec/tolerations
value:
- key: workload
operator: Equal
value: ingress
effect: NoScheduleCommon mistakes to avoid
- Building your own application as a Helm chart so you can have values files, then drowning in Go template debugging for what was supposed to be a simple replica count change
- Pinning a chart to a major version like 4.x and letting Argo CD pull whatever 4.y.z is latest at sync time, so what gets deployed depends on when the sync runs
- Copying entire third-party Kubernetes manifests into your repo because Helm 'feels too complex', then never updating them and falling years behind
Likely follow-ups
- Argo CD needs --enable-helm to render Helm inside Kustomize. Where do you set that, and what is the security trade-off?
- A vendor chart does not expose the setting you need as a value. Helm post-renderer or Kustomize patch on top? Why?
- How do you pin a chart version so a new release does not change behavior under you?
- Same setup but the chart lives in an OCI registry instead of a Helm repo. What changes in the Application spec?
Answer out loud first, then check yourself against the model answer.
Using Kustomize Overlays for Per-Environment Config
// interview question
Walk me through how you would use Kustomize overlays to handle config that differs between dev and prod in an Argo CD setup.
Sample answer
Put what is shared in a base. Put what differs in overlays, one per environment. Argo CD detects the kustomization.yaml in the overlay path and runs kustomize build to render the manifests. Base holds: Deployment, Service, baseline ConfigMap, common labels. Overlays patch: replicas, resource requests and limits, image tags, ingress hosts, environment-specific ConfigMap values, env vars, and HPA settings. Key rules I follow: 1. Anything that differs between environments lives in overlays, never the base. 2. Image tags belong in overlays so CI can bump dev without touching prod. 3. Patch with strategic merge or JSON patches. Do not copy the whole Deployment YAML into each overlay; that is exactly the duplication Kustomize is supposed to remove. 4. Use configMapGenerator with behavior: merge to add or override values, not replace the whole map. 5. Secrets do not go in Kustomize. Use external-secrets, sealed-secrets, or a secrets manager. On the Argo CD side, the Application points at the overlay path (apps/web/overlays/prod) and Argo CD figures out it is Kustomize from the kustomization.yaml file. No extra config needed. Edge case: if the same overlay applies to two clusters with one tiny difference (a region label, a DNS suffix), do not create a fourth overlay. Use ApplicationSet parameters to pass that one value in. Otherwise overlays multiply fast.
Why this matters
This question tests whether the candidate has actually shipped Kustomize, not just read about it. Watch for people who treat overlays like full copies of the base. The point of Kustomize is the patch, not the duplication. Strong candidates know the difference between configMapGenerator behaviors (create vs merge vs replace), know where image tags should live, and have a clear take on secrets being out of scope.
Code examples
Base kustomization.yaml
# apps/web/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
- configmap.yaml
commonLabels:
app.kubernetes.io/name: web
images:
- name: web
newName: ghcr.io/org/web
newTag: placeholderProd overlay patching replicas, resources, image, and config
# apps/web/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: web
resources:
- ../../base
- hpa.yaml
patches:
- path: replicas-patch.yaml
- path: resources-patch.yaml
images:
- name: web
newTag: v1.42.0
configMapGenerator:
- name: app-config
behavior: merge
literals:
- LOG_LEVEL=warn
- DB_HOST=prod-db.internal
- FEATURE_X=trueReplicas patch (strategic merge)
# apps/web/overlays/prod/replicas-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 12Common mistakes to avoid
- Copying the whole Deployment YAML into each overlay instead of writing a small patch
- Putting secrets into a ConfigMap because Kustomize does not support encrypted resources out of the box
- Hardcoding prod image tags inside the base, so a dev image bump cannot happen without touching prod
Likely follow-ups
- How does Argo CD know to render this with Kustomize? Do you have to tell it?
- Same overlay needs to apply to two clusters but with a different region label in each. How do you handle that without forking the overlay?
- When would you reach for Helm instead of Kustomize? Or both together?
- How do you preview the rendered output of the prod overlay before merging a change?
Answer out loud first, then check yourself against the model answer.
Organizing a Shared Manifests Repo for Multiple Teams
// interview question
Three teams share one Argo CD instance and one manifests repo. How do you lay it out so each team can ship without waiting on each other and can't accidentally deploy each other's stuff?
Sample answer
Two things have to work: ownership at the directory level so reviews go to the right people, and AppProjects in Argo CD so the cluster enforces the boundary even if a PR slips through. Repo layout, top level split by team: teams/ payments/ apps/ checkout/ ledger/ projects/ payments.yaml # AppProject for payments team search/ apps/ indexer/ query-api/ projects/ search.yaml platform/ apps/ ingress-nginx/ cert-manager/ projects/ platform.yaml argocd/ applicationsets/ payments.yaml # one ApplicationSet per team search.yaml platform.yaml Then four things wired together: 1. CODEOWNERS at the repo root maps directory paths to team groups. teams/payments/** routes reviews to @org/payments-platform. Branch protection on main requires a review from the right code owners, so a search team PR cannot land changes under teams/payments without a payments reviewer. 2. One AppProject per team. The AppProject restricts which source repo paths the team's Applications can read from, which destinations (cluster plus namespace) they can deploy to, and which Kubernetes resource kinds they are allowed to create. The payments team's AppProject cannot deploy into the search-prod namespace even if someone tries. 3. One ApplicationSet per team with a Git directory generator pointed at that team's apps folder. New app appears under teams/payments/apps, a new Application appears automatically, owned by the payments project. 4. RBAC on the Argo CD UI mirrors the projects. Payments engineers can sync payments Applications, see them, restart pods. They cannot touch search Applications. The platform team can see everything. The AppProject is the load-bearing piece. CODEOWNERS protects the Git side. The AppProject protects the cluster side. You need both. CODEOWNERS alone is fine until a platform engineer with write-everywhere access merges a typo. The AppProject stops the deploy at the controller before it ever reaches the API server. One thing I avoid. Do not put every team's apps in a single flat apps/ directory with team prefixes in the names like apps/payments-checkout. It works for five apps and falls apart at fifty. Directory boundaries cost nothing and CODEOWNERS matches them naturally.
Why this matters
This is a strong mid-level question because multi-tenant Argo CD is where most teams hit their first real organizational pain. Strong candidates know that GitOps repo structure has to combine with both Git review controls (CODEOWNERS, branch protection) and Argo CD's own RBAC (AppProjects). Weak candidates lean entirely on one side, usually CODEOWNERS, and miss that the cluster has no idea what GitHub thinks. The AppProject is what stops a bad deploy at the controller. Watch for that.
Code examples
CODEOWNERS file routing reviews per team directory
# .github/CODEOWNERS
# Each team owns its own directory under teams/
teams/payments/** @org/payments-platform
teams/search/** @org/search-platform
teams/platform/** @org/platform-engineering
# Argo CD config requires platform review
argocd/** @org/platform-engineering
# Anything else falls back to platform
* @org/platform-engineeringAppProject restricting where the payments team can deploy
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: payments
namespace: argocd
spec:
description: Payments team workloads
sourceRepos:
- https://github.com/org/manifests
# Only allow Applications whose path starts with teams/payments
sourceNamespaces:
- argocd
destinations:
- server: https://kubernetes.default.svc
namespace: payments-*
clusterResourceWhitelist: []
namespaceResourceWhitelist:
- group: ''
kind: ConfigMap
- group: ''
kind: Service
- group: apps
kind: Deployment
- group: autoscaling
kind: HorizontalPodAutoscaler
roles:
- name: payments-engineer
policies:
- p, proj:payments:payments-engineer, applications, sync, payments/*, allow
- p, proj:payments:payments-engineer, applications, action/*, payments/*, allow
groups:
- org:payments-platformApplicationSet that auto-generates Applications for the team
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: payments
namespace: argocd
spec:
generators:
- git:
repoURL: https://github.com/org/manifests
revision: main
directories:
- path: teams/payments/apps/*/overlays/*
template:
metadata:
name: 'payments-{{path[3]}}-{{path.basename}}'
spec:
project: payments
source:
repoURL: https://github.com/org/manifests
targetRevision: main
path: '{{path}}'
destination:
server: https://kubernetes.default.svc
namespace: 'payments-{{path.basename}}'
syncPolicy:
automated:
prune: true
selfHeal: trueCommon mistakes to avoid
- Relying only on CODEOWNERS without an AppProject, so once a PR merges any team can technically deploy anywhere
- Giving every team a single shared AppProject called default with no destination restrictions, which is the same as having no project at all
- Flat-namespacing apps like apps/payments-checkout instead of using directories, then losing CODEOWNERS routing as the repo grows
Likely follow-ups
- A team wants to deploy a CRD their app needs. The AppProject namespaceResourceWhitelist does not include CustomResourceDefinition. What is the right way to handle that request?
- How do you stop the platform team from becoming the bottleneck for every new AppProject change?
- What goes wrong if two teams both want to manage their own ApplicationSets without platform review?
- How would you give a team read-only access to prod Applications but full access to their dev ones?
Answer out loud first, then check yourself against the model answer.
Handling Secrets in an Argo CD Manifests Repo
// interview question
Secrets can't go into Git in plaintext. How do you handle them in an Argo CD setup, and what does that look like in your repo structure?
Sample answer
Three approaches, in order of how I would pick today. Pick External Secrets Operator for most teams. Pick Sealed Secrets if you do not have an external secrets manager. Avoid SOPS-only setups for teams larger than a handful of people because key distribution gets ugly. External Secrets Operator (ESO). The actual secret lives in AWS Secrets Manager, Vault, GCP Secret Manager, or similar. In Git you commit an ExternalSecret CR that says 'fetch this key from that store and create a Kubernetes Secret named X in namespace Y'. Argo CD syncs the ExternalSecret. ESO does the fetch. No ciphertext in Git, the secrets manager handles rotation, IAM controls access. This is what I default to. Sealed Secrets (Bitnami). Run the sealed-secrets controller in the cluster. It generates a public key. You encrypt secrets locally with kubeseal using that public key. The encrypted SealedSecret CR is safe to commit. The controller decrypts it in-cluster and creates a normal Secret. The public key is per cluster, so a SealedSecret encrypted for dev does not work in prod. That is a feature. Repo layout with ESO (per env, since secret references differ): apps/ web/ base/ external-secret.yaml # template with placeholders kustomization.yaml overlays/ dev/ external-secret-patch.yaml # patches secretStoreRef + remote key path prod/ external-secret-patch.yaml platform/ external-secrets/ cluster-secret-store-dev.yaml cluster-secret-store-prod.yaml Repo layout with Sealed Secrets: apps/ web/ base/ deployment.yaml kustomization.yaml overlays/ dev/ sealed-secret.yaml # encrypted with dev cluster public key prod/ sealed-secret.yaml # encrypted with prod cluster public key Things I never do regardless of approach: 1. No plaintext Secret manifests in Git, even base64-encoded. Base64 is not encryption. 2. No single key shared across all environments. The whole point is that a leaked dev key cannot decrypt prod secrets. 3. No secret values in Argo CD Application parameters or Helm values files committed to Git, because they end up in argocd-server memory and CLI history. 4. No mixing: pick one approach per cluster. Two systems means twice the rotation pain and unclear ownership. One Argo CD-specific gotcha. When ESO or sealed-secrets creates the underlying Secret, Argo CD might show the parent Application as OutOfSync because the Secret it created is not tracked in Git. Add the Secret to ignoreDifferences, or annotate the ExternalSecret/SealedSecret so Argo CD knows the child Secret is managed by another controller. Otherwise self-heal will fight with the secret controller and you get reconcile loops.
Why this matters
Secrets in GitOps is the question that exposes who has actually shipped GitOps versus who has read about it. Strong candidates can name at least two approaches, articulate the trade-offs, and explain the Argo CD-specific sync issues that come up when a child Secret is created by another controller. Weak candidates either say 'put them in a ConfigMap' (terrifying) or 'use sealed-secrets' without being able to explain key rotation. Watch for awareness that base64 is not encryption.
Code examples
ExternalSecret CR committed to Git, no plaintext anywhere
# apps/web/overlays/prod/external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: web-db-creds
namespace: web
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-prod
kind: ClusterSecretStore
target:
name: web-db-creds # the Kubernetes Secret that will be created
creationPolicy: Owner
data:
- secretKey: DATABASE_URL
remoteRef:
key: prod/web/db
property: connection_string
- secretKey: DATABASE_PASSWORD
remoteRef:
key: prod/web/db
property: passwordCreating a SealedSecret with kubeseal
# Fetch the public cert from the prod cluster's sealed-secrets controller
kubeseal --fetch-cert \
--controller-namespace=kube-system \
--controller-name=sealed-secrets \
> prod-pub-cert.pem
# Create a normal Secret locally (never commit this file)
kubectl create secret generic web-db-creds \
--namespace=web \
--from-literal=DATABASE_PASSWORD='real-password-here' \
--dry-run=client -o yaml > secret.yaml
# Encrypt it with the prod public key
kubeseal --format=yaml \
--cert=prod-pub-cert.pem \
< secret.yaml > apps/web/overlays/prod/sealed-secret.yaml
# Delete the plaintext local copy and commit the sealed version
rm secret.yaml
git add apps/web/overlays/prod/sealed-secret.yamlArgo CD Application ignoring Secret diffs created by ESO
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: web-prod
namespace: argocd
spec:
project: production
source:
repoURL: https://github.com/org/manifests
targetRevision: main
path: apps/web/overlays/prod
destination:
server: https://kubernetes.default.svc
namespace: web
syncPolicy:
automated:
prune: true
selfHeal: true
# The Secret 'web-db-creds' is created by ESO, not Git. Do not fight it.
ignoreDifferences:
- group: ''
kind: Secret
name: web-db-creds
jsonPointers:
- /dataCommon mistakes to avoid
- Committing base64-encoded Secret YAML and thinking it is safe because it is not obvious plaintext
- Using one sealed-secrets key across all clusters, defeating the whole point of per-cluster encryption
- Not configuring ignoreDifferences on the Application, then watching Argo CD selfHeal fight with ESO in an endless reconcile loop
Likely follow-ups
- How do you rotate a secret with ESO versus with Sealed Secrets? What is the engineer experience for each?
- A sealed-secrets controller's private key gets compromised. What is your recovery plan?
- You are running multi-cluster Argo CD. How does that affect sealed-secrets specifically?
- Why is putting secrets in a Helm values file in Git, even for dev, still a bad idea?
Answer out loud first, then check yourself against the model answer.
App-of-Apps Pattern vs ApplicationSets
// interview question
Explain the app-of-apps pattern in Argo CD. When would you pick it over ApplicationSets?
Sample answer
App-of-apps is a pattern, not a feature. One parent Argo CD Application points at a directory of YAML files. Those YAML files are themselves Argo CD Application CRs. Argo CD syncs the parent, which creates the children, which deploy actual workloads. It gives you one bootstrap entry point per cluster: install the parent, everything else shows up. ApplicationSet is a controller that does something similar but template-driven. You write one Application template plus a generator (Git directory, list, cluster, matrix, pull request, SCM provider) and the controller produces N Applications from it. Add a new directory under overlays, an Application appears. Open a pull request, a preview Application appears. Close it, the preview gets cleaned up. Use ApplicationSets when: 1. You ship the same app across many environments or clusters and the only difference is a parameter (path, cluster URL, namespace). 2. You want pull-request preview environments. 3. Environments come and go often enough that hand-editing Application YAML is annoying. Use app-of-apps when: 1. Each Application is genuinely different. A monitoring stack, an ingress controller, and three apps do not fit one template. 2. You want a clear, hand-reviewed list of cluster components in Git. Real setups use both. App-of-apps bootstraps the platform layer (cert-manager, ingress, monitoring, the ApplicationSet controller itself). ApplicationSets generate per-environment Applications for actual workloads. The platform is curated; the apps are templated. One gotcha to mention. Deleting an ApplicationSet by default deletes every Application it generated, and Argo CD will then delete the workloads those Applications managed. Set preserveResourcesOnDeletion: true on the ApplicationSet spec if you do not want that cascade, or use a Finalizer carefully.
Why this matters
This is a senior-leaning question because the answer hinges on operational experience: knowing when templating pays for itself, and what happens when you remove the template. Listen for candidates who can name a generator type other than Git directory (PullRequest, Cluster, Matrix are the interesting ones) and who flag the deletion behavior. Bonus points for mentioning that the ApplicationSet controller is itself usually deployed via app-of-apps so it can manage everything else.
Code examples
App-of-apps parent Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: platform-bootstrap
namespace: argocd
spec:
project: platform
source:
repoURL: https://github.com/org/manifests
targetRevision: main
path: argocd/applications
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: true
selfHeal: trueApplicationSet with Git directory generator
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: web-envs
namespace: argocd
spec:
generators:
- git:
repoURL: https://github.com/org/manifests
revision: main
directories:
- path: apps/web/overlays/*
template:
metadata:
name: 'web-{{path.basename}}'
spec:
project: default
source:
repoURL: https://github.com/org/manifests
targetRevision: main
path: '{{path}}'
destination:
server: https://kubernetes.default.svc
namespace: 'web-{{path.basename}}'
syncPolicy:
automated:
prune: true
selfHeal: trueApplicationSet that fans out across registered clusters
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: monitoring
namespace: argocd
spec:
generators:
- clusters:
selector:
matchLabels:
env: prod
template:
metadata:
name: 'monitoring-{{name}}'
spec:
project: platform
source:
repoURL: https://github.com/org/manifests
targetRevision: main
path: platform/monitoring
destination:
server: '{{server}}'
namespace: monitoringCommon mistakes to avoid
- Hand-writing one Application YAML per environment across 30 environments instead of using an ApplicationSet
- Not knowing that deleting the ApplicationSet by default cascades and deletes the generated Applications plus their workloads
- Treating ApplicationSet generators as magic and not being able to predict what Applications the template will produce for a given input
Likely follow-ups
- What happens if you delete an ApplicationSet? How do you stop it from deleting prod by accident?
- Walk me through how the PullRequest generator builds preview environments and what cleans them up.
- How would you bootstrap a brand new cluster from zero with just one kubectl apply?
- When would the Matrix generator make sense, and what is the trap with it?
Answer out loud first, then check yourself against the model answer.
Promoting Changes from Dev to Staging to Prod with Argo CD
// interview question
Walk me through how a change moves from dev to staging to prod in an Argo CD setup. How do you stop something from slipping into prod that should not be there?
Sample answer
Promotion in GitOps is a Git operation, not a kubectl operation. A pull request into the manifests repo updates the prod overlay. The Argo CD controller picks up the new commit and reconciles. The three things to get right: what triggers each environment, how image tags move between environments, and how prod is gated. A flow that works: 1. Developer merges code into the application repo. CI builds an image and pushes it with an immutable tag, the Git SHA or a semver. 2. CI commits the new tag to apps/web/overlays/dev/kustomization.yaml on main of the manifests repo. Argo CD dev Application has auto-sync on and self-heal on. It picks up the change and ships to dev within a couple of minutes. 3. Promotion to staging is a pull request bumping apps/web/overlays/staging/kustomization.yaml to the same tag. Merge after a smoke test or an automated check. Argo CD syncs staging. 4. Promotion to prod is another pull request, this time bumping the prod overlay. Branch protection requires code owners, a passing CI check, and a manual reviewer. After merge, Argo CD syncs prod, or a human clicks sync if prod is set to manual. Non-negotiables I enforce: - Same image artifact through every environment. Do not rebuild between staging and prod. The binary that passed staging is the one that ships. - Immutable tags. No latest, no main. Use the Git SHA or a release version. - Prod sync policy is different from dev. Either manual sync, or auto-sync with sync windows so it can only run during business hours. - selfHeal is fine for dev but think hard about it on prod. It reverts any manual change, which can cause grief during an incident if the on-call engineer needs a quick patch. If you keep selfHeal off on prod, you also need a clear policy that the only way to change prod is through Git. Gotcha that bites teams: auto-sync off does not mean self-heal off. They are separate. If self-heal is on, Argo CD will still revert manual edits, even though it will not pull new commits automatically. Configure both knobs deliberately. Hotfixes: have a documented bypass that still goes through Git. Commit straight to the prod overlay path with senior approval. Never apply manifests with kubectl from a laptop just because the PR flow feels slow.
Why this matters
This question covers the operational side of GitOps end to end. Strong candidates have a clear, opinionated answer with specific knobs they would set. They will mention immutable tags, sync windows, branch protection on the manifests repo, and the difference between auto-sync and self-heal. Weak candidates describe a flow but cannot answer 'what stops a developer from running kubectl apply against prod' or 'what happens when selfHeal is on during an incident.' This is where you find out if they have actually run Argo CD past the demo stage.
Code examples
Prod Application with manual sync and stricter policy
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: web-prod
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: production
source:
repoURL: https://github.com/org/manifests
targetRevision: main
path: apps/web/overlays/prod
destination:
server: https://prod-cluster.example.com
namespace: web
syncPolicy:
# No automated block: prod sync requires a human click
syncOptions:
- PrunePropagationPolicy=foreground
- ApplyOutOfSyncOnly=true
revisionHistoryLimit: 20AppProject with sync windows that limit prod to business hours
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: production
namespace: argocd
spec:
description: Production workloads
sourceRepos:
- https://github.com/org/manifests
destinations:
- server: https://prod-cluster.example.com
namespace: '*'
syncWindows:
- kind: allow
schedule: '0 9 * * 1-5'
duration: 8h
applications:
- 'web-prod'
- 'api-prod'
manualSync: true
- kind: deny
schedule: '0 0 * * 6,0'
duration: 24h
applications:
- '*-prod'
manualSync: falsePromotion script: bump staging to the tag currently in dev
#!/usr/bin/env bash
set -euo pipefail
APP=web
MANIFESTS_DIR=$(pwd)
# Read the image tag currently deployed to dev
DEV_TAG=$(yq '.images[] | select(.name == "web") | .newTag' \
apps/${APP}/overlays/dev/kustomization.yaml)
echo "Promoting ${APP} to staging at tag ${DEV_TAG}"
# Create a branch and open a PR
BRANCH=promote-${APP}-staging-${DEV_TAG}
git checkout -b ${BRANCH}
(cd apps/${APP}/overlays/staging && \
kustomize edit set image web=ghcr.io/org/web:${DEV_TAG})
git add apps/${APP}/overlays/staging/kustomization.yaml
git commit -m "promote ${APP} to staging: ${DEV_TAG}"
git push -u origin ${BRANCH}
gh pr create \
--title "promote ${APP} to staging: ${DEV_TAG}" \
--body "Promotes the dev image to staging. No code change."Common mistakes to avoid
- Rebuilding the image between staging and prod so a different binary ships than was tested
- Using a floating tag like latest or main on prod, so what gets deployed depends on when sync runs rather than what is in Git
- Enabling auto-sync and self-heal on prod with the same settings as dev, then being surprised when self-heal reverts an on-call hotfix
Likely follow-ups
- A change works in staging and breaks in prod because of a config that differed. How do you stop that next time?
- How do database migrations fit into this flow? They are not idempotent the way manifests are.
- What is your rollback story? A bad change went out, the Argo CD UI is open, what do you do?
- How does this change if you are running progressive delivery with Argo Rollouts in front of the Deployment?
Answer out loud first, then check yourself against the model answer.
Scaling a Manifests Repo Across Many Services, Environments, and Clusters
// interview question
You have 15 microservices, 4 environments (dev, staging, preprod, prod), and prod runs in 3 regional clusters. That's potentially 90 Application CRs. How do you structure the manifests repo so it doesn't fall apart?
Sample answer
At this size, hand-written Application CRs are dead weight. Switch to ApplicationSets generated from the directory layout, push environment differences into structured parameter files, and split the repo into a workloads side and a platform side. Three changes get you there. First, layout. Two axes: app and environment-or-cluster. I keep the app axis as the primary directory split because most changes happen inside one app. workloads/ checkout/ base/ overlays/ dev/ staging/ preprod/ prod-us-east/ prod-eu-west/ prod-ap-south/ ledger/ base/ overlays/ ... same shape ... clusters/ dev/ config.yaml # cluster metadata: name, server, region, labels staging/ preprod/ prod-us-east/ prod-eu-west/ prod-ap-south/ argocd/ applicationsets/ workloads.yaml # one ApplicationSet that fans out across apps x envs projects/ Second, one ApplicationSet does the fan-out. A matrix generator combines a list of apps with the cluster list. The template produces one Application per (app, cluster) pair. Add a new app: drop a directory under workloads/. Add a new region: register the cluster and add an overlay. Both flows are pure Git, no YAML duplication. Third, push the actual environment differences into well-named files, not into a wall of overlay patches. Each overlay holds a values.yaml or a config.yaml with the parameters that differ: replica count, resource requests, DNS suffix, feature flags. The overlay's kustomization.yaml is a thin shim that pulls in the base, applies a couple of patches, and merges the values file into a ConfigMap. When someone wants to know 'what is different about eu-west prod', they read one file, not five. Things that go wrong at this scale if you do not plan for them: 1. Argo CD performance. 90 Applications all polling the same Git repo means your repo server becomes a bottleneck. Enable Git webhook integration so commits trigger immediate refresh instead of every Application polling on its own interval. Tune timeout.reconciliation in argocd-cm. 2. Repo server memory. Rendering Kustomize or Helm for 90 Applications eats RAM. Scale up the argocd-repo-server replicas, give them more memory, and consider running a separate Argo CD instance for production clusters so a dev repo storm cannot starve prod reconcile. 3. Blast radius of a base change. A change to workloads/checkout/base/deployment.yaml hits 6 environments at once. Use sync waves and sync windows so prod-eu-west does not sync at the same moment as prod-us-east. Or, for the riskiest changes, use a release branch that overlays target instead of main, so you can promote regions one at a time. 4. Discoverability. With 90 Applications, the Argo CD UI gets hard to navigate. Use labels on the ApplicationSet template (app, env, region, team) and rely on the UI's label filters instead of scrolling. One pattern I avoid at this size: per-cluster manifests repos. It feels clean and it kills you the first time you need to roll out a security patch to 6 clusters at once. One repo, many Applications, generated from one template. Drift between regions is the enemy.
Why this matters
This is the senior question where you find out whether the candidate has actually operated Argo CD at scale or just configured it for one team. Strong candidates will name specific scaling pain points (repo server memory, polling vs webhook, sync windows) and reach for ApplicationSets and matrix generators without prompting. They will also call out drift between regions as a first-class concern, which most people learn the hard way. Weak candidates will describe the small-scale layout they used in a previous job and pretend it works at 10x scale.
Code examples
Matrix ApplicationSet generating Applications across apps and clusters
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: workloads
namespace: argocd
spec:
generators:
- matrix:
generators:
# Axis 1: each app under workloads/
- git:
repoURL: https://github.com/org/manifests
revision: main
directories:
- path: workloads/*
# Axis 2: each registered cluster, filtered to envs we care about
- clusters:
selector:
matchExpressions:
- key: env
operator: In
values: [dev, staging, preprod, prod]
template:
metadata:
name: '{{path.basename}}-{{name}}'
labels:
app: '{{path.basename}}'
env: '{{metadata.labels.env}}'
region: '{{metadata.labels.region}}'
spec:
project: '{{metadata.labels.env}}'
source:
repoURL: https://github.com/org/manifests
targetRevision: main
path: 'workloads/{{path.basename}}/overlays/{{name}}'
destination:
server: '{{server}}'
namespace: '{{path.basename}}'
syncPolicy:
automated:
prune: true
selfHeal: trueCluster Secret with labels the ApplicationSet selector uses
# Each cluster registered with Argo CD as a Secret with labels.
# These labels drive which Applications get generated for which cluster.
apiVersion: v1
kind: Secret
metadata:
name: prod-eu-west
namespace: argocd
labels:
argocd.argoproj.io/secret-type: cluster
env: prod
region: eu-west
tier: production
type: Opaque
stringData:
name: prod-eu-west
server: https://prod-eu-west.example.com
config: |
{
"bearerToken": "...",
"tlsClientConfig": { "caData": "..." }
}Sync windows that stagger prod region rollouts
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: prod
namespace: argocd
spec:
description: Production workloads, region-staggered rollout
sourceRepos:
- https://github.com/org/manifests
destinations:
- server: '*'
namespace: '*'
syncWindows:
# us-east first, business hours
- kind: allow
schedule: '0 9 * * 1-4'
duration: 1h
clusters: ['prod-us-east']
manualSync: true
# eu-west 2 hours later, gives us time to catch issues
- kind: allow
schedule: '0 11 * * 1-4'
duration: 1h
clusters: ['prod-eu-west']
manualSync: true
# ap-south last
- kind: allow
schedule: '0 13 * * 1-4'
duration: 1h
clusters: ['prod-ap-south']
manualSync: true
# No prod syncs Friday afternoon or weekends
- kind: deny
schedule: '0 14 * * 5'
duration: 66h
clusters: ['prod-*']
manualSync: falseCommon mistakes to avoid
- Hand-writing 90 Application YAMLs instead of generating them, then never daring to refactor because every change touches every file
- One manifests repo per cluster, which feels organized until you need to roll a CVE fix to all of them at once
- Letting Argo CD poll Git on its default interval across 90 Applications instead of wiring up webhooks, then wondering why changes take 3 minutes to show up
Likely follow-ups
- At what point would you split into multiple Argo CD instances instead of one? What is the line?
- Your repo server is OOMing. Walk me through how you would diagnose and fix that before throwing more memory at it.
- A base manifest change needs to roll to 6 prod clusters but you want to halt after 2 if anything looks wrong. How do you wire that?
- How would you keep regions from drifting when one region needs a temporary patch the others do not?
Answer out loud first, then check yourself against the model answer.
More interview topics
Also worth your time on this topic
GitOps with Argo CD: Structuring Your Repository for Multi-Environment Deployments
A practical guide to laying out your Git repository for Argo CD across dev, staging, and production. See real folder structures, Kustomize and Helm patterns, and the pitfalls that bite teams in production.
Argo CD Multi-Environment Repository Structure Checklist
How to organize your Git repositories when running Argo CD across dev, staging, and production. Covers folder layout, app-of-apps, ApplicationSets, secrets, RBAC, and promotion flow.
60-90 minutes
GitOps with ArgoCD - Automated Kubernetes Deployments
Implement GitOps workflows using ArgoCD for automated, declarative, and auditable Kubernetes application deployments.
120 minutes