How to Measure RAG Accuracy in Contextual AI Before You Ship

You measure RAG accuracy in Contextual AI by building a labelled question set, scoring each answer for faithfulness to the cited passages, and tracking retrieval hit rate separately from generation quality. If you skip this, you are shipping on vibes.

This follows the setup walkthrough for Contextual AI. It assumes you already have a datastore answering questions and now need numbers.

Why does a grounded chatbot still give wrong answers?

Two failure modes look identical to a user. The retriever may miss the right passage entirely, so the model answers from general knowledge. Or it retrieves the right passage and the generator misreads it. Faithfulness scoring fixes the second; hit rate fixes the first.

Measure them apart. A single “accuracy” number hides which half of your pipeline needs work.

How do you build an evaluation set?

Fifty questions is enough to start. For each, record the expected answer and the document that should be cited. Build them from real user questions, not from your own reading of the docs — the gaps in your self-made questions are exactly the ones that hurt.

Cover three shapes:

• Direct lookup — one passage answers it.
• Multi-hop — needs two documents combined.
• Unanswerable — nothing in the corpus fits, so the correct response is “I don’t know”.

The unanswerable category is the one teams skip and then regret. A model that never says “I don’t know” is not accurate, it is agreeable.

Which metric do you track for launch?

Use a small dashboard rather than one score:

Metric	What it catches	Starting target
Retrieval hit rate	Missing passages	above 85%
Faithfulness	Misread passages	above 90%
Refusal accuracy	Hallucinated answers	above 80%

Those numbers are a floor, not a goal. Below them, users lose trust fast; above them, you can ship and keep improving.

How do you improve a low score?

If retrieval hit rate is the problem, look at chunking first. Long, structure-heavy documents split badly and the right passage never surfaces. Adding better metadata filters — date, product line — often moves the number more than changing models.

If faithfulness is low but retrieval is fine, the generator is over-reading. Tighten the prompt to require citation of a specific passage and re-run the same evaluation set. Same questions, new prompt, compare directly.

How often should you re-run the evaluation?

Every time documents change or you touch the prompt. Rerun it on a schedule too — a monthly check catches corpus drift, where new documents quietly overwrite old answers. Keep the question set frozen so the numbers stay comparable.

Quick Summary

• Build 50 labelled questions before measuring anything.
• Score retrieval hit rate and faithfulness separately.
• Include unanswerable questions to test refusal behavior.
• Start targets: 85% hit rate, 90% faithfulness, 80% refusal accuracy.
• Re-run after every document or prompt change to catch drift.