How to Use Datadog APM Traces to Find a Slow Endpoint
Your dashboard says /api/checkout has a p95 of 2.4 seconds. Three team members open the code, each finds a different suspect, and all three are wrong. A trace shows you the calls in order, with timings, and takes about four minutes to answer the question properly.
Datadog APM is not included with infrastructure monitoring. It’s billed separately — $31 per host per month for APM, on top of the $15-$23 host fee. That’s why enabling it is a decision, not a checkbox.
How do you enable Datadog APM?
On the agent side, edit /etc/datadog-agent/datadog.yaml:
apm_config:
enabled: true
apm_non_local_traffic: false # set true only if traces come from other containers
Then instrument the application. Datadog ships tracing libraries for Python, Node, Java, Go, Ruby, .NET and PHP. For a Node service:
npm install dd-trace --save
node -r dd-trace/init server.js
For Python, pip install ddtrace and run with ddtrace-run python app.py. Both auto-instrument common frameworks — Flask, Django, Express, FastAPI — and emit spans for HTTP handlers and database calls without touching application code.
Restart and hit an endpoint. Within a minute, APM → Traces shows entries. If it’s empty, check DD_TRACE_AGENT_URL (default http://localhost:8126) and confirm the agent is listening with sudo datadog-agent status | grep -A5 apm.
The full-host APM pricing tier covers all services on a host. If you’re running four containers on one machine, that’s still one APM host — a detail that changes the maths in containerised setups.
How do you read a trace flame graph?
Open a slow trace from the Service → Traces list. The waterfall reads top to bottom, each bar a span, width proportional to duration.
Look for these patterns, in order of frequency:
| Pattern | What it means | Typical fix |
|---|---|---|
| One wide child bar under a narrow parent | A single downstream call dominates | Cache it, batch it, or make it async |
| Many medium bars in sequence | N+1 query in a loop | Batch the query, or eager-load |
| Gap between parent start and first child | Blocking work in the handler itself | Serialisation, sync crypto, CPU loops |
| Wide bar with no children | Untraced code — a library without instrumentation | Add manual spans to find the real cost |
That last row matters. If a span is 1.8 seconds wide and has no children, the time is in code you haven’t instrumented. Auto-instrumentation covers common frameworks and database drivers, not your internal utility library. Add manual spans where it hurts:
from ddtrace import tracer
with tracer.trace("pricing.recalculate", service="checkout") as s:
s.set_tag("sku_count", len(skus))
result = recalculate(skus)
Now the next slow trace tells you whether pricing.recalculate is the problem or an innocent bystander.
How do you go from p95 spike to root cause?
The workflow that works, in four prompts:
- Filter first. In APM → Traces, add
service:checkout resource_name:/api/checkout @duration:>2s.@durationis in nanoseconds, so 2s is2000000000. - Group by a dimension. Group by
http.status_codeorversion. If onlyversion:2.4.1is slow, you have a deploy problem, not a capacity problem. That single grouping saves hours. - Open the trace with the widest gap. Read the waterfall, and check whether the slow span is consistent across traces or spiky.
- Correlate with infrastructure. In the trace detail, the Host tab links to host metrics at that exact minute. A 2-second query that lines up with 100% disk I/O has a different fix than the same query at idle.
Datadog’s Watchdog automatically surfaces anomalies like “this endpoint got 40% slower after 14:20”. Useful as a starting point, unreliable as a conclusion — it flags the symptom, and you still do the trace walk.
What is an SLO and should you bother?
An SLO (service level objective) is a target like “95% of checkout requests succeed in under 1 second over 28 days”. Datadog computes it from your traces and shows a burn-down.
It’s worth the setup when you have several teams arguing about priorities. “Checkout is at 94.2% against a 95% target with 11 days left” settles arguments that p95 charts don’t.
It’s not worth it for a single-service product. The configuration overhead (event-based or metric-based, which monitor backs it, what counts as good) is a couple of hours, and with one team you’ll read the trace list anyway.
If you do set one up: pick a 28-day window (aligns with monthly reviews), measure from real user traces rather than synthetic checks, and use the error budget display rather than a single number. 95% vs 94% sounds like noise until you see the budget is 80% consumed.
What does APM actually cost at scale?
Datadog APM is $31/host/month with host-based indexing at 1 million spans per month included, then per-million sampling charges. Trace volumes on a busy service blow through that quickly — a service handling 50 requests/second with 20 spans each generates 2.6 billion spans a month.
The standard answer is ingestion controls: set retention filters so you keep 100% of errors and slow traces, and 1% of the rest. That keeps the bill near the base rate while preserving the traces you’d actually debug with. Configure it under APM → Ingestion Settings.
Teams that skip this step get a $4,000 invoice and turn APM off. Teams that spend 30 minutes on retention filters keep it.
Who shouldn’t pay for APM?
A monolith with 20 endpoints and one developer. Structured logs with request timing, plus infrastructure metrics, answer almost every latency question at that scale, for free.
APM earns its price when services call each other and “which one is slow” stops being obvious from logs alone. Microservices, a queue in the middle, a third-party API dependency — that’s where the flame graph pays for itself in one incident.
Quick Summary
- APM is separate from infrastructure monitoring: $31/host/month on top of the host fee.
- Enable via
apm_config.enabled: trueplus the language library, not the agent alone. - In a flame graph, a wide childless span means uninstrumented code — add manual spans there.
- Group slow traces by
versionbefore anything else; a single bad deploy is the most common cause. - Set ingestion retention filters early or bill shock will end the experiment.
- SLOs are worth it across multiple teams, not for a single-service product.</}
