How to Evaluate an LLM Prompt with Braintrust in Under an Hour

November 9, 2026

Braintrust is an evaluation platform for LLM applications. You define a dataset, a task that turns inputs into outputs, and scorers that grade the outputs. Then you compare versions instead of guessing.

The reason to bother: most prompt changes are made on vibes. You tweak the system message, try four examples, it looks better, you ship. Two weeks later a different set of inputs breaks and nobody knows which change caused it.

An eval takes about an hour to stand up and pays for itself the second time you change a prompt.

What are the three pieces of a Braintrust eval?

Dataset — inputs plus, ideally, expected outputs. Task — your function that calls the model. Scorers — functions that grade the output, either deterministic (exact match, regex, valid JSON) or model-graded (“is this answer supported by the source”).

Everything else in the product is built around these three. Get them right and the UI is just a viewer.

Who this is for: anyone iterating on prompts or RAG with more than a handful of users. Who it is not for: a single-prompt prototype where you have five test cases in your head, or teams with no way to define “correct” — in which case you need a labelling effort before you need a platform.

How do you set up the project?

Install the SDK and set your key:

pip install braintrust autoevals
export BRAINTRUST_API_KEY="sk-..."

Create a project in the dashboard, or let braintrust.init(project="my-app") create it on first use. The autoevals package is separate and worth installing: it ships ready-made scorers for factuality, closed QA, JSON validity and summarisation that you would otherwise write from scratch.

How do you build a useful dataset?

Start with 20 to 50 examples. Real ones.

Pull them from production logs if you have them, from support tickets if you do not, or write them by hand from the product spec. What you must not do is generate all of them from the same model you are evaluating — you will get a dataset that matches the model’s style and scores of 95% that mean nothing.

Structure each record as input and expected (or metadata when there is no single right answer). For a support classifier, input is the ticket text and expected is the category. For a RAG answer, expected may be empty and the scorer checks groundedness instead of exact match.

Twenty examples catch gross regressions. Fifty catch most. Beyond 200 you are doing statistics, not prompt work, and you should probably be doing a labelled eval run instead of eyeballing.

How do you write the task function?

The task is a plain Python function. Wrap the model call, return a string or a dict:

def task(input):
    resp = client.messages.create(model=M, max_tokens=400,
        messages=[{"role": "user", "content": input}])
    return resp.content[0].text

Keep it deterministic enough to compare: pin the model version, set temperature to 0 for scoring runs, and log the model name into metadata so a run is traceable later.

How do you pick scorers that are not useless?

Exact match is a trap for generative output. Two answers can be equally correct with different wording, and exact match scores one zero.

Use a mix:

  • Deterministic first. Valid JSON, contains required fields, under N characters, no banned phrases. These are free and catch most breakage.

  • Model-graded second. Factuality or a custom LLM judge with a rubric. Costs money per item, so 50 examples times 2 scorers is 100 judge calls per run, which is fine.

  • Human review for the ambiguous cases. The dashboard lets you click through low-scoring items. Do this once per run; it is where you find out that your scorer, not your prompt, is wrong.

A custom judge takes a rubric string. Write the rubric as a checklist, not an adjective: “Does the answer cite a specific number from the source? Does it avoid claiming anything the source does not state?”

How do you compare two prompts?

Run the experiment twice with different prompt versions under the same experiment name. Braintrust groups runs and shows a diff per scorer: which items improved, which regressed, and by how much.

The number to look at is not the average. It is the count of regressions — items that went from passing to failing. An average that rises from 78% to 82% while three items break is usually a bad trade, because those three broke for a reason you can find.

Attach prompt versions as metadata or use the experiment parameter so lineage is visible. When a score drops six weeks later, you need to know which prompt and which model produced the run.

What does it cost and where does it get slow?

Braintrust’s free tier covers a project with a reasonable dataset size and monthly eval runs. Paid plans are per-seat with higher limits and SSO.

The slow part is always the model calls, not the platform. 50 items at 2 seconds each is 100 seconds; multiply by scorers that also call a model and a full run takes three or four minutes. Run evals on a subset (20 items) during iteration and the full set before merging.

What should you not try to measure this way?

Long-conversation quality. A single input-output pair cannot capture whether a ten-turn conversation stayed coherent, because the tenth turn depends on the previous nine. Multi-turn scenarios need a simulated user and a separate harness.

Also skip eval-driven development for things that are architecture-bound. If your RAG answers are wrong because retrieval returns the wrong chunk, a better prompt will not fix the score, and the eval will send you tuning the wrong variable for a week.

Quick Summary

  • A Braintrust eval is three things: a dataset, a task function, and scorers; everything else is a viewer.

  • Start with 20–50 real examples from logs or tickets, never generated by the model you are testing.

  • Combine deterministic scorers with one model-graded judge, and write judge rubrics as checklists rather than adjectives.

  • Compare runs by regression count, not by average score; a higher average with three broken items is usually a loss.

  • Run evals on 20 items during iteration and the full set before merge, since model calls dominate the runtime.