Observability Interview Questions
6 questions for junior, mid-level and senior Observability interviews, with model answers, code examples and common mistakes. Answer each one out loud first, then reveal the answer.
Traces and Spans Explained
// interview question
A request hits your API gateway, which calls two backend services, and one of those queries a database. Walk me through what that looks like as a distributed trace. What is a span, and how do spans connect to each other?
Sample answer
The whole request is one trace, identified by a single trace ID that follows the request everywhere it goes. Each unit of work along the way is a span: one span for the gateway handling the request, one for each backend call, and one for the database query. Every span records a name, start and end timestamps, a status, and key-value attributes like http.method or db.statement. Spans link together through parent-child relationships: the gateway span is the root, the two backend spans are its children, and the database span is a child of the backend that made the query. The child knows its parent because the parent's span ID travels with the request. When a backend visualizes this, you get a waterfall view showing exactly where the 800ms went: maybe 50ms in the gateway, 100ms in service A, and 650ms stuck in that database query. That is the core value: instead of grepping logs across four services and guessing, you see the full request path and its timing in one place.
Why this matters
This is the entry-level question for any observability conversation. You are checking whether the candidate has the mental model right: trace as the whole journey, span as one hop, parent-child links forming a tree. Strong candidates explain it through a concrete scenario and mention what spans actually contain (attributes, status, timestamps). Weak candidates recite 'tracing tracks requests' without being able to describe how two spans end up connected. If they bring up the waterfall view and what it tells you about latency, that is a good sign they have used tracing, not just read about it.
Code examples
Creating a parent and child span manually
from opentelemetry import trace
tracer = trace.get_tracer("checkout-service")
with tracer.start_as_current_span("process_order") as parent:
parent.set_attribute("order.id", order_id)
parent.set_attribute("order.items", len(items))
# Child span is linked automatically via the active context
with tracer.start_as_current_span("charge_card") as child:
child.set_attribute("payment.provider", "stripe")
result = charge(order_id)
if not result.ok:
child.set_status(trace.StatusCode.ERROR, result.error)Simplified span as exported via OTLP
{
"traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
"spanId": "00f067aa0ba902b7",
"parentSpanId": "b7ad6b7169203331",
"name": "SELECT orders",
"kind": "SPAN_KIND_CLIENT",
"startTimeUnixNano": "1717500000000000000",
"endTimeUnixNano": "1717500000650000000",
"attributes": {
"db.system": "postgresql",
"db.statement": "SELECT * FROM orders WHERE id = $1"
},
"status": { "code": "STATUS_CODE_OK" }
}Common mistakes to avoid
- Confusing traces with logs, describing tracing as 'centralized logging with request IDs' and missing the timing and hierarchy aspects
- Saying spans connect through the trace ID alone, when the parent-child link comes from the parent span ID
- Not being able to name a single thing a span contains beyond 'the request'
Likely follow-ups
- How does the trace ID get from service A to service B in the first place?
- What is the difference between a span attribute and a span event, and when would you use each?
- If a service publishes to a queue and a consumer picks it up an hour later, should that be one trace or two?
Answer out loud first, then check yourself against the model answer.
Monitoring and Alerting Strategy
// interview question
How do you design a monitoring and alerting strategy? What metrics would you track and how do you avoid alert fatigue?
Sample answer
Start with the four golden signals: latency, traffic, errors, and saturation. Use RED method for services (Rate, Errors, Duration) and USE method for resources (Utilization, Saturation, Errors). To avoid alert fatigue: alert on symptoms not causes, set appropriate thresholds with historical data, use severity levels (page vs ticket), implement alert grouping and deduplication, require runbooks for every alert, and regularly review and tune alerts. Only page for actionable issues that require immediate human intervention.
Why this matters
Effective monitoring enables proactive issue detection and faster incident resolution. Poor alerting leads to alert fatigue where important signals get ignored. This is a critical skill for maintaining reliable systems.
Code examples
Prometheus alerting rules example
groups:
- name: service-alerts
rules:
# High error rate - page immediately
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: High error rate detected
runbook: https://wiki/runbooks/high-error-rate
# High latency - create ticket
- alert: HighLatency
expr: |
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 0.5
for: 10m
labels:
severity: warning
annotations:
summary: P95 latency above 500msCommon mistakes to avoid
- Alerting on every metric threshold crossing
- Not including runbooks with alerts
- Setting thresholds without historical baseline data
Likely follow-ups
- What is the difference between metrics, logs, and traces?
- How do you decide between warning and critical alert severity?
- What is an error budget and how does it relate to alerting?
Answer out loud first, then check yourself against the model answer.
Auto vs Manual Instrumentation
// interview question
You need to roll out tracing across 40 services owned by six different teams. Do you go with auto-instrumentation or manual instrumentation, and how do you decide?
Sample answer
Start with auto-instrumentation for breadth, then add manual spans where the business logic lives. Auto-instrumentation hooks into known libraries: HTTP servers and clients, database drivers, gRPC, queue clients. In Java you attach the agent jar with a JVM flag and get traces with zero code changes, in Python you wrap the process with opentelemetry-instrument. That gets all 40 services emitting spans in days, not quarters, and you immediately see the service map and cross-service latency. What auto-instrumentation cannot give you is domain context. It shows 'POST /api/checkout took 2s' but not which step of checkout was slow or which customer tier was affected. So the second phase is manual: teams add spans around meaningful operations and attach attributes like order.value or tenant.id to the spans that auto-instrumentation already created. Language matters too. Java and Python have strong runtime agents. Go has no agent in the traditional sense, so you instrument in code or look at eBPF-based approaches, which means your Go services need more team buy-in up front. The practical rollout: platform team ships a base image or init container with the agent preconfigured, exporting to a standard endpoint, and each team only touches code when they want richer spans. Forcing six teams to hand-instrument everything before seeing any value is how tracing rollouts die.
Why this matters
This question tests judgment, not API knowledge. The trap is treating it as either-or: strong candidates immediately say both, in sequence, and explain why breadth-first wins for adoption. You also learn whether they have dealt with organizational reality: getting six teams to do anything is harder than any technical step, so answers that include a platform-level rollout path (base images, agents injected at deploy time, sane defaults) signal real experience. Bonus points for knowing the per-language differences, especially that Go cannot just attach an agent.
Code examples
Zero-code instrumentation for Java and Python services
# Java: attach the agent at startup, no code changes
java -javaagent:/otel/opentelemetry-javaagent.jar \
-Dotel.service.name=payment-service \
-Dotel.exporter.otlp.endpoint=http://otel-collector:4317 \
-jar payment-service.jar
# Python: wrap the entrypoint
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
OTEL_SERVICE_NAME=order-api \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
opentelemetry-instrument gunicorn app:appKubernetes Operator injecting auto-instrumentation via annotation
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: platform-defaults
spec:
exporter:
endpoint: http://otel-collector.observability:4317
propagators:
- tracecontext
- baggage
---
# Teams opt in with one pod annotation, no image changes:
# instrumentation.opentelemetry.io/inject-java: "true"Enriching auto-created spans with business context
from opentelemetry import trace
def checkout(request):
# The HTTP server span already exists, created by
# auto-instrumentation. Add domain attributes to it.
span = trace.get_current_span()
span.set_attribute("order.value_usd", request.total)
span.set_attribute("customer.tier", request.user.tier)
tracer = trace.get_tracer("checkout")
# Manual span for the step auto-instrumentation cannot see
with tracer.start_as_current_span("calculate_discounts"):
apply_discounts(request)Common mistakes to avoid
- Picking one approach for everything: all-manual stalls the rollout for months, all-auto leaves you with traces that lack any business meaning
- Assuming every language works like Java, then discovering the Go services emit nothing because nobody planned for in-code instrumentation
- Ignoring semantic conventions, so every team invents its own attribute names and cross-service queries become impossible
Likely follow-ups
- How would you handle the Go services in that fleet, where attaching a runtime agent is not an option?
- Auto-instrumentation just doubled your span volume and your tracing bill. What do you do?
- How do you enforce consistent attribute naming, like tenant.id versus tenantId, across six teams?
Answer out loud first, then check yourself against the model answer.
Context Propagation Across Services
// interview question
Service A calls service B over HTTP. How does B know the request belongs to an existing trace? What actually travels on the wire?
Sample answer
A single HTTP header called traceparent, defined by the W3C Trace Context spec. It looks like 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01: version, the 128-bit trace ID, the parent span ID, and flags such as the sampled bit. On the sending side, the OpenTelemetry SDK's propagator injects this header into the outgoing request. On the receiving side, B's instrumentation extracts it, so the server span B creates uses the same trace ID and points at A's span as its parent. There is also an optional baggage header for passing your own key-value pairs across services, like a tenant ID. The places this breaks are the interesting part: async work that spawns a thread or goroutine without carrying the context along, message queues where you have to inject the context into message headers yourself, and proxies or legacy services that strip or drop unknown headers. Mixed environments are another trap: older systems use Zipkin's B3 headers, so during a migration you run a composite propagator that reads and writes both formats. The symptom of broken propagation is always the same: traces that cut off at a service boundary, with the downstream work showing up as orphaned root spans.
Why this matters
Propagation is where tracing theory meets reality. Anyone can say 'the trace ID flows between services', but candidates who have actually run tracing in production know the traceparent header by name and have debugged broken traces caused by a proxy stripping headers or a queue consumer starting fresh traces. Listen for the failure modes: async boundaries, message queues, B3 versus W3C. A candidate who only describes the happy path has probably never had to fix a trace that went dark halfway through.
Code examples
What the traceparent header looks like on the wire
GET /api/orders/42 HTTP/1.1
Host: service-b.internal
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
baggage: tenant.id=acme,deployment.canary=true
# Format: version-traceId-parentSpanId-flags
# The last byte (01) means this trace was sampled upstreamManually injecting context into a queue message
from opentelemetry.propagate import inject, extract
from opentelemetry import trace
# Producer: HTTP auto-instrumentation will not help here,
# you have to carry the context inside the message yourself
headers = {}
inject(headers)
queue.publish(body=payload, headers=headers)
# Consumer: restore the context so the span joins the trace
ctx = extract(message.headers)
tracer = trace.get_tracer("order-consumer")
with tracer.start_as_current_span("process_message", context=ctx):
handle(message)Composite propagator for a B3 to W3C migration
from opentelemetry import propagate
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.propagators.b3 import B3MultiFormat
from opentelemetry.trace.propagation.tracecontext import (
TraceContextTextMapPropagator,
)
# Accept and emit both header formats while old Zipkin-era
# services are still in the call path
propagate.set_global_textmap(
CompositePropagator([TraceContextTextMapPropagator(), B3MultiFormat()])
)Common mistakes to avoid
- Believing the backend stitches traces together by timestamps or IP addresses instead of an explicit header carried with the request
- Forgetting that queues, cron handoffs, and background jobs need manual context injection because there is no HTTP request to carry the header
- Putting sensitive or bulky data in baggage without realizing it is forwarded to every downstream service on every call
Likely follow-ups
- A trace consistently ends at one specific service and the downstream spans show up as separate root traces. How do you debug that?
- How would you propagate context through Kafka, where there is no HTTP request between producer and consumer?
- What is baggage useful for, and what is the risk of putting too much in it?
Answer out loud first, then check yourself against the model answer.
Why Run an OpenTelemetry Collector
// interview question
Why run an OpenTelemetry Collector at all instead of having every application export directly to your tracing backend? And how would you deploy it in Kubernetes?
Sample answer
Direct export couples every application to your backend. Change vendors, rotate an API key, or add span redaction, and you are redeploying 40 services. With a Collector in the middle, apps export plain OTLP to a local endpoint and everything else becomes pipeline config: receivers take data in, processors transform it, exporters fan it out. The processors are where the real value is. The batch processor cuts export overhead, memory_limiter stops the Collector from OOMing under a span flood, attributes and redaction processors strip PII before it leaves your cluster, and tail sampling can only happen here because no single app sees the whole trace. The Collector also buffers and retries when the backend is down, so a backend outage does not mean dropped telemetry or back-pressure into your apps. In Kubernetes the standard layout is two tiers. A DaemonSet agent on every node gives apps a node-local endpoint, adds k8s metadata like pod and namespace, and does cheap work like batching. A central gateway Deployment behind a Service handles the expensive, stateful work: tail sampling, filtering, authentication to external backends, and being the single egress point your firewall rules allow. Small clusters can skip the gateway and run just the DaemonSet pointed straight at the backend. You add the gateway when you need tail sampling or want one place to control egress and credentials.
Why this matters
This separates people who have operated telemetry pipelines from people who have only instrumented an app. The core insight you want to hear is decoupling: telemetry policy lives in pipeline config, not application code. Then probe the deployment topology. Strong candidates know the agent-versus-gateway split and, more importantly, why the split exists: tail sampling and credential management belong in a central tier, host metadata enrichment belongs on the node. If they mention memory_limiter or what happens when the backend goes down, they have run this in production.
Code examples
Collector pipeline: receive OTLP, protect memory, batch, fan out
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 80
spike_limit_percentage: 25
k8sattributes: {}
attributes:
actions:
- key: http.request.header.authorization
action: delete
batch:
send_batch_size: 8192
timeout: 5s
exporters:
otlp/tempo:
endpoint: tempo.observability:4317
tls:
insecure: true
otlp/vendor:
endpoint: ingest.vendor.com:443
headers:
api-key: ${env:VENDOR_API_KEY}
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, k8sattributes, attributes, batch]
exporters: [otlp/tempo, otlp/vendor]Two-tier deployment with the OpenTelemetry Operator
# Node agent: one per node, apps export to their local node
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: node-agent
spec:
mode: daemonset
config:
exporters:
otlp:
endpoint: otel-gateway.observability:4317
---
# Gateway: central tier for tail sampling and backend egress
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: otel-gateway
spec:
mode: deployment
replicas: 3Common mistakes to avoid
- Describing the Collector as just a proxy and missing the processing layer, which is the actual reason to run one
- Running a single replica gateway with no memory_limiter, so one chatty service OOMs the whole telemetry pipeline
- Putting tail sampling on the DaemonSet agents, where no node ever sees all spans of a trace, so sampling decisions are made on partial data
Likely follow-ups
- The Collector itself becomes a single point of failure for your telemetry. How do you make the pipeline survive a Collector crash or a bad config rollout?
- How would you monitor the Collector itself, and which signals tell you it is dropping spans?
- When would you choose a sidecar Collector per pod over a node DaemonSet?
Answer out loud first, then check yourself against the model answer.
Sampling Strategies at Scale
// interview question
Your platform handles 50,000 requests per second and tracing every one of them is blowing up the observability bill. How do you approach sampling, and what is the tradeoff between head and tail sampling?
Sample answer
First decide what questions the traces need to answer, because that drives the strategy. Head sampling decides at the root of the trace, before anything happens: a parent-based ratio sampler keeps, say, 1% of traces, and the decision rides along in the traceparent sampled flag so every downstream service agrees. It is cheap and simple, but it is blind: the slow request and the 500 error are dropped with the same probability as the boring 200. Tail sampling flips that. Every span is exported, buffered in the Collector gateway, and the keep-or-drop decision happens after the trace completes, with full knowledge: keep 100% of traces with errors, 100% of traces slower than two seconds, and 1% of everything else as a healthy baseline. That is exactly what you want for debugging, but you pay for it: the gateway buffers every in-flight trace in memory, and all spans of a given trace must land on the same Collector instance, which means a load-balancing tier that routes by trace ID in front of your sampling tier. At 50k rps that buffer is real memory and the loadbalancing exporter is not optional. In practice I combine them: light head sampling or rate limiting at the SDK to take the edge off volume, tail sampling at the gateway for the keep-the-interesting-ones logic. Two things people forget: any metrics derived from spans, like RED dashboards from the spanmetrics connector, must be generated before sampling or your latency percentiles are fiction, and you should write down that trace-based alerting now sees sampled data, so absence of a trace no longer proves absence of a problem.
Why this matters
This is a senior-level cost and architecture question disguised as a feature comparison. Anyone can define head versus tail. What you are listening for: the operational consequences of tail sampling (memory buffering, trace-ID-aware load balancing across Collector instances), the policy thinking (keep all errors and slow traces, sample the healthy baseline), and the second-order effects, especially that span-derived metrics computed after sampling are statistically wrong. Candidates who jump straight to 'just use tail sampling' without mentioning the buffering and routing cost have read the docs but never run it at volume.
Code examples
Tail sampling policy: keep errors and slow traces, sample the rest
processors:
tail_sampling:
decision_wait: 10s
num_traces: 100000
policies:
- name: keep-errors
type: status_code
status_code:
status_codes: [ERROR]
- name: keep-slow
type: latency
latency:
threshold_ms: 2000
- name: baseline
type: probabilistic
probabilistic:
sampling_percentage: 1Routing tier so all spans of a trace hit the same sampler
# First Collector tier: no sampling, just route by trace ID
exporters:
loadbalancing:
routing_key: traceID
protocol:
otlp:
tls:
insecure: true
resolver:
k8s:
service: otel-sampling-tier.observability
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [loadbalancing]Head sampling at the SDK, propagated to all children
# Keep 5% of traces, decided once at the root span.
# parentbased_* makes children honor the parent's decision
# instead of re-rolling the dice at every hop.
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.05Common mistakes to avoid
- Recommending tail sampling without mentioning trace-ID-aware load balancing, which silently produces broken sampling decisions on partial traces across multiple Collector replicas
- Computing latency percentiles from spans after sampling, so dashboards are skewed toward whatever the policies kept
- Using plain traceidratio instead of a parent-based sampler, so each service makes its own decision and you end up storing fragments of traces
Likely follow-ups
- Your error rate spikes to 10x during an incident, which means tail sampling suddenly keeps 10x the volume. How do you protect the pipeline?
- A product team complains a specific customer's traces are never there when they need them. How do you support per-tenant sampling?
- How does sampling interact with billing-by-span vendors versus self-hosted backends like Tempo, and does that change your strategy?
Answer out loud first, then check yourself against the model answer.
More interview topics
Also worth your time on this topic
Distributed Tracing with OpenTelemetry: From Instrumentation to Visualization
A walkthrough of instrumenting a real service with OpenTelemetry, running the Collector, and finding the slow span in Jaeger when a request hops across five microservices.
Monitoring & Observability Checklist
Comprehensive checklist for implementing monitoring, logging, tracing, and alerting across your infrastructure and applications.
60-90 minutes
Microservices Observability with OpenTelemetry
Implement comprehensive observability for microservices using OpenTelemetry for distributed tracing, metrics, and logging across multiple services.
140 minutes