Chaos Engineering Interview Questions
5 questions for junior, mid-level and senior Chaos Engineering interviews, with model answers, code examples and common mistakes. Answer each one out loud first, then reveal the answer.
Litmus Building Blocks: ChaosEngine vs ChaosExperiment
// interview question
You install Litmus on a cluster and want to kill a pod to see what happens. Walk me through the pieces Litmus gives you, and what is the actual difference between a ChaosExperiment and a ChaosEngine?
Sample answer
Litmus is built around a few custom resources plus an operator that watches them. A ChaosExperiment is the reusable template: it describes one fault (pod-delete, pod-network-latency, node-drain), the litmus-go container image that runs it, the default tunables, and the RBAC permissions that fault needs. You install these from the ChaosHub, and on their own they do nothing. Think of ChaosExperiment as the recipe for what could go wrong. A ChaosEngine is the order ticket: it binds one or more ChaosExperiments to a specific target through appinfo (appns, applabel, appkind), names the chaosServiceAccount to run as, lets you override env, and attaches probes. Applying a ChaosEngine is what actually triggers a run. The chaos-operator reconciles that ChaosEngine, spins up a chaos-runner pod, and the runner launches the experiment job that injects the fault. The third resource is ChaosResult, which holds the verdict: Pass, Fail, or Awaited, plus probeSuccessPercentage and the failStep. That is your source of truth for whether the hypothesis held, not just whether the pod came back. So the short version: ChaosExperiment is the recipe, ChaosEngine is the order, ChaosResult is the review.
Why this matters
This question tells you fast whether someone has actually run Litmus or only read about chaos engineering in the abstract. The ChaosExperiment versus ChaosEngine split is the single thing beginners get muddled. Listen for the operator and runner flow, and for the candidate naming ChaosResult as the place you read the outcome. If they say 'the pod rescheduled so it passed' without mentioning the verdict resource, they have done the tutorial once and stopped thinking.
Code examples
A minimal ChaosEngine that runs the pod-delete experiment
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: nginx-chaos
namespace: default
spec:
appinfo:
appns: default
applabel: 'app=nginx'
appkind: deployment
chaosServiceAccount: pod-delete-sa
engineState: active
experiments:
- name: pod-delete
spec:
components:
env:
- name: TOTAL_CHAOS_DURATION
value: '30'
- name: PODS_AFFECTED_PERC
value: '50'Install the experiment definition, then read the verdict
# ChaosExperiment definitions come from the ChaosHub
kubectl apply -n default -f \
https://hub.litmuschaos.io/api/chaos/3.0.0?file=charts/generic/pod-delete/experiment.yaml
# After applying the ChaosEngine, the verdict lives in the ChaosResult
kubectl get chaosresult nginx-chaos-pod-delete -n default \
-o jsonpath='{.status.experimentStatus.verdict}'
# -> Pass / Fail / AwaitedCommon mistakes to avoid
- Thinking that installing Litmus alone runs chaos; you still need the experiment definition, an RBAC service account, and a ChaosEngine to trigger anything
- Using ChaosEngine and ChaosExperiment as if they are the same object
- Treating a rescheduled pod as a pass instead of reading the ChaosResult verdict
Likely follow-ups
- Where does the ChaosExperiment definition actually come from, and what is the ChaosHub?
- What is the chaos-runner pod doing, and how is it different from the chaos-operator?
- If the deleted pod comes back but ChaosResult never reaches Pass, what would you go look at first?
Answer out loud first, then check yourself against the model answer.
Running Your First Pod-Delete Experiment Safely
// interview question
I hand you a fresh cluster with a demo nginx deployment. Take me from nothing to a controlled pod-delete experiment. What are the steps, and how do you keep it from turning into an outage?
Sample answer
Six steps. First, install Litmus, which gives you the chaos-operator and the CRDs (helm chart or the operator manifest is fine). Second, install the pod-delete ChaosExperiment from the ChaosHub into the target namespace. Third, set up RBAC: a dedicated ServiceAccount with a Role and RoleBinding scoped to that namespace, granting only what pod-delete needs. Do not reach for the bundled litmus-admin here. Fourth, create a ChaosEngine pointing at the target with appns=default, applabel=app=nginx, appkind=deployment, and the service account from step three. Fifth, keep the blast radius small in that ChaosEngine: PODS_AFFECTED_PERC set to 50 or even target a single pod, a short TOTAL_CHAOS_DURATION like 30 seconds, and FORCE=false so you mimic a graceful eviction instead of a hard kill. Sixth, watch it: tail the runner and experiment pod logs, watch the ChaosResult verdict, and keep an eye on the app's availability the whole time. The safety rules that matter: start in staging, target a Deployment so the ReplicaSet actually reschedules the pod (pod-delete against a bare pod just leaves you down), kill one replica at a time, have a probe or at least a live dashboard so you observe steady state rather than assume it, and know the abort before you start: kubectl delete chaosengine, or set engineState to stop.
Why this matters
This is the practical 'have you done this with your own hands' question. Listen for ordered steps and especially the RBAC step, which beginners skip and then spend an hour confused about why the runner gets 'forbidden' listing pods. Good candidates mention blast-radius tunables, the FORCE semantics, targeting a controller-backed pod, and the abort path. Someone who says 'apply the ChaosEngine and watch it' without the service account or blast radius has not run this on anything they cared about.
Code examples
Install Litmus and the pod-delete experiment, then watch the run
# 1) Operator + CRDs
helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
helm install litmus litmuschaos/litmus --namespace litmus --create-namespace
# 2) Experiment definition into the target namespace
kubectl apply -n default -f \
https://hub.litmuschaos.io/api/chaos/3.0.0?file=charts/generic/pod-delete/experiment.yaml
# 4-6) After applying RBAC + ChaosEngine, watch it run
kubectl get pods -n default -w # runner + experiment job appear
kubectl describe chaosresult nginx-chaos-pod-delete -n defaultChaosEngine with a small, graceful blast radius
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: nginx-chaos
namespace: default
spec:
appinfo:
appns: default
applabel: 'app=nginx'
appkind: deployment
chaosServiceAccount: pod-delete-sa
engineState: active
experiments:
- name: pod-delete
spec:
components:
env:
- name: TOTAL_CHAOS_DURATION
value: '30'
- name: CHAOS_INTERVAL
value: '10'
- name: PODS_AFFECTED_PERC
value: '50'
- name: FORCE
value: 'false' # graceful eviction, like a real node drainThe abort switch, before you start
# Stop chaos and let Litmus revert it (best effort)
kubectl patch chaosengine nginx-chaos -n default \
--type merge -p '{"spec":{"engineState":"stop"}}'
# Or just delete the engine
kubectl delete chaosengine nginx-chaos -n defaultCommon mistakes to avoid
- Skipping the RBAC service account, hitting 'forbidden' errors from the runner, then papering over it with cluster-admin
- Leaving FORCE at a value that hard-kills pods, so the test never exercises graceful shutdown the way a real eviction would
- Running against a bare pod with no controller, so nothing reschedules and the 'experiment' is just an outage
Likely follow-ups
- What does FORCE=true actually change, and when does that make your experiment unrealistic or dangerous?
- What happens if you run pod-delete against a standalone pod with no Deployment or ReplicaSet behind it?
- How would you target one specific pod by name instead of a percentage of the deployment?
- Where exactly do you read whether the run passed, and what does Awaited mean?
Answer out loud first, then check yourself against the model answer.
How Litmus Decides Pass or Fail: Probes
// interview question
Your pod-delete experiment shows Pass, but during the run users got 502s for about 20 seconds. How can the experiment pass while the service was actually down, and how do you fix that?
Sample answer
It passes because, with no probe attached, the verdict mostly checks that the fault was injected and the target recovered to a healthy pod count. It never checked whether your service kept serving traffic. A bare ChaosEngine gives you green that means almost nothing. The fix is probes, which is how you encode the steady-state hypothesis and feed probeSuccessPercentage into the verdict. Litmus has four probe types: httpProbe hits an endpoint and asserts on the status or response, cmdProbe runs a command and compares output, k8sProbe asserts on the state of a Kubernetes resource, and promProbe runs a PromQL query and asserts on the result (for example, error rate stays under a threshold). But the type matters less than the mode. SOT and EOT run the probe once, at the start or end of the test. Edge runs it at both ends. Continuous polls throughout the chaos window at probePollingInterval, and that is the one that catches the 20 seconds of 502s, because it samples while the failure is happening. So add an httpProbe in Continuous mode against the service, with a tight polling interval and a sane retry budget. If availability dips during chaos, the probe fails, probeSuccessPercentage drops, and ChaosResult becomes Fail. Now Pass actually means the failure was injected and the service stayed up. The probe is the experiment's assertion. Without one you are just deleting pods and hoping.
Why this matters
This separates people who ran the happy-path tutorial from people who validate a hypothesis. The scenario is real and common: a green verdict that hides a real availability gap. Listen for probes, and specifically Continuous mode versus the once-at-the-edges modes, plus the framing that the probe is the assertion that makes the verdict meaningful. A strong candidate reaches for promProbe to assert on an SLO like error rate or latency, not just a 200 from an endpoint.
Code examples
ChaosEngine with a Continuous httpProbe plus a Prometheus SLO probe
experiments:
- name: pod-delete
spec:
probe:
- name: frontend-stays-up
type: httpProbe
mode: Continuous
httpProbe/inputs:
url: http://nginx.default.svc.cluster.local
method:
get:
criteria: ==
responseCode: '200'
runProperties:
probeTimeout: 2s
interval: 1s
attempt: 3
probePollingInterval: 2s
- name: error-rate-under-1pct
type: promProbe
mode: Continuous
promProbe/inputs:
endpoint: http://prometheus.monitoring:9090
query: rate(http_requests_total{code=~"5.."}[1m])
comparator:
criteria: '<'
value: '1'
runProperties:
probeTimeout: 5s
interval: 5s
attempt: 2
probePollingInterval: 5sRead the probe result, not just the verdict
kubectl get chaosresult nginx-chaos-pod-delete -n default \
-o jsonpath='{.status.experimentStatus.probeSuccessPercentage}'
# 100 means every probe poll held the hypothesis.
# Anything less means the service dipped while the fault was live.Common mistakes to avoid
- Running experiments with no probe at all and trusting the green Pass it produces
- Using SOT or EOT mode when the failure window is mid-experiment, so the availability dip is never sampled
- Setting probes so strict (zero retries, tiny timeout) that you get flaky failures and the team stops trusting the results
Likely follow-ups
- What is the difference between Continuous and Edge mode, and when would Edge actually be the right choice?
- How do probeTimeout, attempt, and probePollingInterval interact, and how do you keep a probe from producing flaky verdicts?
- How would you assert on a real SLO like p99 latency staying under 300ms during the chaos window?
- How would you turn probeSuccessPercentage into a pass or fail gate in a CI pipeline?
Answer out loud first, then check yourself against the model answer.
From One Experiment to Continuous Chaos at Scale
// interview question
You have proven a single pod-delete works. Leadership now wants chaos as an ongoing practice across dozens of services and several clusters, not a one-off demo. What changes, and what would you do differently at scale?
Sample answer
A single hand-applied ChaosEngine does not become a program by running it more often. Four things change. First, you chain faults into scenarios. Litmus drives Argo-workflow-based chaos experiments that sequence multiple faults plus probes into one run, for example kill a pod, then add network latency, while a probe asserts the SLO holds the whole time. ChaosCenter becomes the control plane, and in Litmus 3.x you install chaos infrastructure (the delegate agents) into each target cluster and drive runs centrally. Second, you put the definitions under GitOps. Store the ChaosEngine, workflow, and probe YAML in a repo and deploy it with Argo CD, so chaos is reviewed, versioned, and reproducible instead of kubectl-applied from someone's laptop. Third, you schedule and gate. Run experiments on a cron with a ChaosSchedule for steady drift detection, and wire them into your delivery pipeline as a resilience gate where a dropped probeSuccessPercentage blocks promotion. Reserve structured GameDays for the big multi-team scenarios. Fourth, you observe. Export chaos metrics to Prometheus, overlay chaos events on your golden-signal Grafana dashboards, and make a promProbe the auto-halt so a burning SLO ends the run and reverts. What I would do differently at scale is mostly organizational: enforce blast-radius and RBAC policy through admission control so nobody can run with litmus-admin, give each team a namespaced scope, track a resilience score per service over time so you can show leadership the curve, and treat every chaos finding like an incident with an owner and a fix rather than a dashboard nobody reads. The goal is regressions caught automatically, not heroics.
Why this matters
This is a senior systems-thinking question. The jump you are listening for is from a custom resource to a practice: workflows that chain faults, GitOps for versioning and review, scheduling, CD gating, observability with auto-halt, and the org mechanics that keep it alive. The 'what would you do differently at scale' part should surface policy enforcement and treating findings as owned work. A weak answer just says 'run it more and put it on a schedule' with no governance, no observability, and no story for who acts on the results.
Code examples
Recurring chaos with a ChaosSchedule, fenced to work hours and weekdays
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosSchedule
metadata:
name: nginx-weekday-chaos
namespace: default
spec:
schedule:
repeat:
properties:
minChaosInterval:
hour:
everyNthHour: 6
workHours:
includedHours: 9-17
workDays:
includedDays: 'Mon,Tue,Wed,Thu' # never Friday, never overnight
engineTemplateSpec:
appinfo:
appns: default
applabel: 'app=nginx'
appkind: deployment
chaosServiceAccount: pod-delete-sa
experiments:
- name: pod-delete
spec:
components:
env:
- name: PODS_AFFECTED_PERC
value: '25'A Prometheus probe used as an auto-halt SLO gate
probe:
- name: availability-slo-gate
type: promProbe
mode: Continuous
promProbe/inputs:
endpoint: http://prometheus.monitoring:9090
query: |
sum(rate(http_requests_total{job="nginx",code!~"5.."}[1m]))
/ sum(rate(http_requests_total{job="nginx"}[1m]))
comparator:
criteria: '>='
value: '0.99' # availability dips below 99% -> probe fails -> run halts
runProperties:
probeTimeout: 5s
interval: 10s
attempt: 1
probePollingInterval: 10s
stopOnFailure: trueCommon mistakes to avoid
- Treating chaos as a one-off demo instead of a versioned, scheduled, and owned practice
- Scheduling chaos with no freeze or incident awareness, so it can fire in the middle of a real outage
- Collecting chaos results that nobody acts on, with no resilience score and no owner for findings
Likely follow-ups
- How do you keep scheduled chaos from firing during a real incident or a change freeze?
- In a shared cluster, how do you prevent one team's chaos from spilling into another team's services?
- How do you actually measure whether the chaos program is improving reliability, rather than just generating activity?
- Build versus adopt: how would you choose between Litmus, Chaos Mesh, and a managed offering for a program this size?
Answer out loud first, then check yourself against the model answer.
Scoping Litmus Safely: RBAC and Blast Radius
// interview question
Your security team sees that Litmus can delete pods and inject network faults cluster-wide, and they want it gone. How do you scope Litmus so you can still run chaos in production without handing it the keys to the cluster?
Sample answer
There are two levers, RBAC and blast radius, and you pull both. On RBAC: Litmus ships litmus-admin, a cluster-wide service account that is fine for a sandbox and wrong for production. Each ChaosExperiment already declares exactly the verbs and resources its fault needs, so build a least-privilege ServiceAccount per experiment or per team namespace using a Role and RoleBinding instead of a ClusterRole wherever the fault allows it. Run Litmus in namespaced scope so a blast cannot cross tenant boundaries. On blast radius: the ChaosEngine appns and applabel narrow the target set, PODS_AFFECTED_PERC and KILL_COUNT cap how many pods go at once, and for node or infra faults NODE_LABEL fences which nodes can be touched. Sequence multiple faults serially rather than in parallel so you do not compound failures you cannot reason about. Then layer operational safety on top: run chaos in its own namespace, set resource quotas, attach probes in Continuous mode that fail fast and abort the run when an SLO breaks, and rehearse the whole thing in staging first. And know the kill switch cold: set engineState to stop or delete the ChaosEngine, which reverts the chaos best-effort and tears down the runner. The pitch back to security is simple. A least-privilege service account, namespaced scope, a bounded blast radius, and an auto-halt probe is a smaller and known standing risk than the unknown failure modes you are already shipping to production blind.
Why this matters
This is a senior security and operations conversation, not a feature recall. Listen for least-privilege per experiment, namespaced versus cluster scope, the specific blast-radius env vars, serial sequencing, an auto-halt probe, and the kill switch. The framing for security matters too: a good candidate sells bounded risk, not 'trust me'. The red flag is anyone who shrugs and says give it cluster-admin or just use litmus-admin.
Code examples
A least-privilege service account scoped to one namespace for pod-delete
apiVersion: v1
kind: ServiceAccount
metadata:
name: pod-delete-sa
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role # Role, not ClusterRole: stays inside this namespace
metadata:
name: pod-delete-role
namespace: default
rules:
- apiGroups: ['']
resources: ['pods', 'pods/log', 'events']
verbs: ['get', 'list', 'create', 'delete', 'deletecollection']
- apiGroups: ['apps']
resources: ['deployments', 'replicasets']
verbs: ['get', 'list']
- apiGroups: ['batch']
resources: ['jobs']
verbs: ['get', 'list', 'create', 'delete']
- apiGroups: ['litmuschaos.io']
resources: ['chaosengines', 'chaosexperiments', 'chaosresults']
verbs: ['get', 'list', 'patch', 'update']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: pod-delete-rb
namespace: default
subjects:
- kind: ServiceAccount
name: pod-delete-sa
namespace: default
roleRef:
kind: Role
name: pod-delete-role
apiGroup: rbac.authorization.k8s.ioBounding the blast radius in the ChaosEngine
experiments:
- name: pod-delete
spec:
components:
env:
- name: PODS_AFFECTED_PERC
value: '25' # at most a quarter of matched pods
- name: KILL_COUNT
value: '1' # one at a time
- name: TOTAL_CHAOS_DURATION
value: '60'
- name: SEQUENCE
value: 'serial' # do not compound faults in parallelCommon mistakes to avoid
- Reaching for litmus-admin or cluster-admin in production because it is the path of least resistance
- Leaving blast-radius env vars at defaults, so PODS_AFFECTED_PERC takes out far more than intended
- Assuming chaos always reverts cleanly and never testing the abort path or the runner-dies-mid-run case
Likely follow-ups
- How would you actually enforce that nobody runs chaos with litmus-admin, for example with an admission policy or OPA Gatekeeper?
- How do you isolate two teams running chaos in the same shared cluster so one cannot affect the other?
- If the chaos-runner pod dies mid-experiment, how do you make sure the injected fault still reverts?
- How do you stop a blast from cascading into dependencies you did not explicitly target?
Answer out loud first, then check yourself against the model answer.
More interview topics
Also worth your time on this topic
Litmus Chaos Experiment Checklist: Your First Run
How to install Litmus on Kubernetes and run a controlled failure experiment from a written hypothesis to a verdict you can act on, without breaking production by accident.
90-150 minutes
Running Your First Chaos Engineering Experiment with Litmus
A hands-on walkthrough of installing LitmusChaos on Kubernetes, killing pods on purpose, and watching whether your app actually recovers. Real YAML, real output, no theory.
GitOps with ArgoCD - Automated Kubernetes Deployments
Implement GitOps workflows using ArgoCD for automated, declarative, and auditable Kubernetes application deployments.
120 minutes