How to Trace RAG Failures with Braintrust Logging and Playground

November 9, 2026

An eval score of 71% is a diagnosis without a location. Braintrust logging answers the next question — which step in the pipeline produced the bad answer — by recording every model call, retrieval and tool invocation as a trace with its inputs and outputs.

The practical value shows up fast. When the answer is wrong, you can usually narrow it to one of three places in under a minute: retrieval returned nothing useful, reranking chose the wrong chunk, or generation ignored the right chunk.

What is a span, and why does the structure matter?

A trace is a tree of spans. One request creates a root span; retrieval, reranking and generation each become child spans with their own duration, input and output.

If you log only the final answer, you have a collection of strings and no way to attribute failure. If you log spans, the trace itself shows the answer was generated in 1.4 seconds from a chunk that mentions a different product. That is a fixable finding.

Structure rule: one span per external call and per pipeline stage. Not one span per function. People over-instrument and end up with forty spans of which thirty-eight are string formatting.

How do you add logging to an existing app?

The Python integration wraps any function:

import braintrust
from braintrust import traced

@traced
def retrieve(query):
    return index.search(query, k=8)

@traced
def answer(query):
    chunks = retrieve(query)
    return generate(query, chunks)

@traced on the top-level function creates the root span; nested calls become children automatically because Braintrust propagates context. That is the whole integration for a Python service.

For anything that already uses OpenTelemetry, there is an OTLP endpoint — point your existing exporter at it and skip the SDK decorators. That path is the right one for polyglot backends; the decorator is right for a single Python service.

What should you log on a retrieval span?

Not the whole chunk text at full length. Log what you would need to reproduce the problem:

  • The query as sent to the index, including any rewriting the model did first. – The scores per result, not just the IDs. – Chunk IDs and the source document IDs, so you can join back to your own store. – The number of results before and after a rerank or filter.

A drop from 8 results to 2 after filtering explains a lot of bad answers, and it is invisible if you only log the final list.

How do you find the failures worth investigating?

Do not read traces randomly. Filter.

  1. Low scores from your eval scorers — every eval item links to the trace that produced it. 2. High latency — sort by duration; one slow retrieval usually accounts for the whole p99. 3. User feedback flags — wire the thumbs-down button to attach a score to the trace. A hundred thumbs-down traces are the best dataset you will ever get. 4. Errors — 429s and timeouts from the model provider show up as failed spans, which lets you distinguish “the model was rate limited” from “the model was wrong.”

How does the playground help after you find a bad trace?

Open a trace and you can replay its inputs in the playground: change the system prompt, swap the model, adjust temperature, and see the new output next to the old one. Then save it as a dataset item so the score change is permanent rather than a one-off impression.

The workflow that pays off: every thumbs-down trace gets replayed once with a prompt change. If the change fixes it, the item goes into the eval set. If it does not, the problem is not the prompt — check the retrieval span.

What does it cost to log everything?

Braintrust’s free tier includes a monthly span allowance that comfortably covers a small product; paid plans are per-seat with higher limits. The operational cost to watch is storage of long prompts: a trace with 8 chunks of 1,500 tokens each plus system prompt is roughly 13,000 tokens of stored text. Truncating chunk text to the first 400 characters in the span cuts that by 80% and rarely loses the diagnostic value.

Sampling is the other lever. Log 100% of errors and feedback-flagged traces, and sample 5–20% of successes.

Where does this approach fall short?

Traces show what happened in one request. They do not show you that your retrieval index is stale for an entire customer, or that two tenants’ documents leaked into the same index. Those are data-quality problems and they need a different kind of check — a periodic audit query, not a trace viewer.

Also, correlation becomes hard past a certain scale. If your service handles 50,000 requests a day, no human reads traces; you need the scorer-derived aggregates to surface problems, and the trace is only used after a metric moves. Set up dashboards on score distribution per prompt version, not per request.

Quick Summary

  • Braintrust traces are span trees; the structure is what lets you attribute a bad answer to retrieval, rerank or generation.

  • Add @traced to the top-level function, or point an existing OpenTelemetry exporter at the OTLP endpoint.

  • Log query text, per-result scores and chunk IDs on retrieval spans, and nothing else, to keep traces cheap.

  • Filter traces by low eval score, latency, errors and thumbs-down before reading anything.

  • Replay a bad trace in the playground, then save the improved item into the dataset so the fix is permanent.