This follows the setup walkthrough for Contextual AI. It assumes you already have a datastore answering questions and now need numbers.

Why does a grounded chatbot still give wrong answers?

Two failure modes look identical to a user. The retriever may miss the right passage entirely, so the model answers from general knowledge. Or it retrieves the right passage and the generator misreads it. Faithfulness scoring fixes the second; hit rate fixes the first.

Measure them apart. A single “accuracy” number hides which half of your pipeline needs work.

How do you build an evaluation set?

Fifty questions is enough to start. For each, record the expected answer and the document that should be cited. Build them from real user questions, not from your own reading of the docs — the gaps in your self-made questions are exactly the ones that hurt.

Cover three shapes:

• Direct lookup — one passage answers it.
• Multi-hop — needs two documents combined.
• Unanswerable — nothing in the corpus fits, so the correct response is “I don’t know”.

The unanswerable category is the one teams skip and then regret. A model that never says “I don’t know” is not accurate, it is agreeable.

Which metric do you track for launch?

Use a small dashboard rather than one score:

Those numbers are a floor, not a goal. Below them, users lose trust fast; above them, you can ship and keep improving.

How do you improve a low score?

If retrieval hit rate is the problem, look at chunking first. Long, structure-heavy documents split badly and the right passage never surfaces. Adding better metadata filters — date, product line — often moves the number more than changing models.

If faithfulness is low but retrieval is fine, the generator is over-reading. Tighten the prompt to require citation of a specific passage and re-run the same evaluation set. Same questions, new prompt, compare directly.

How often should you re-run the evaluation?

Every time documents change or you touch the prompt. Rerun it on a schedule too — a monthly check catches corpus drift, where new documents quietly overwrite old answers. Keep the question set frozen so the numbers stay comparable.

Quick Summary

• Build 50 labelled questions before measuring anything.
• Score retrieval hit rate and faithfulness separately.
• Include unanswerable questions to test refusal behavior.
• Start targets: 85% hit rate, 90% faithfulness, 80% refusal accuracy.
• Re-run after every document or prompt change to catch drift.

It competes with rolling your own RAG stack. The trade-off is control versus time — Contextual AI gets you to a working grounded chatbot faster and gives up some fine-grained tuning.

How do you get a first dataset running?

Four steps to a queryable knowledge base:

1. Create an account and a new datastore in the dashboard.
2. Upload documents — PDF, HTML, DOCX and plain text are the common inputs.
3. Wait for indexing; a few hundred pages finishes in minutes.
4. Run a test query in the playground and check the cited sources.

Each answer comes back with the passages it used. That citation trail is the feature worth paying attention to — it is how you tell a grounded answer from a confident hallucination.

What makes Contextual AI different from a hobby RAG setup?

Three things stand out:

• Chunking that respects structure — tables and headings are kept intact rather than split on a fixed token count.
• Reranking built in — retrieved passages get reordered by a model before they reach the generator.
• Evaluation tooling — you can score answers for faithfulness against the source documents.

A homegrown pipeline can do all of this, but not in an afternoon.

Is Contextual AI free?

There is a trial or limited free usage for evaluation, and production use is priced by usage — storage for documents plus query volume. Exact numbers move, so budget around your document volume rather than memorizing a figure. For a personal project the free tier is enough; a company with thousands of documents should expect a real monthly bill.

Where does the platform get in your way?

You are on their chunking and embedding choices. If your domain needs a specialized embedding model — legal citations, chemical names — you have less room to swap it than with a code-your-own stack.

It is also overkill for a fifty-page knowledge base. Below a few hundred documents, a simple embeddings script and a hosted vector store will cost less and give you identical answers.

Quick Summary

• Contextual AI is an end-to-end RAG platform: upload, index, query.
• Answers return with citations, which helps separate grounded from hallucinated output.
• It handles structure-aware chunking, reranking and evaluation out of the box.
• Free for evaluation; production priced by document volume and queries.
• Poor fit for small corpora or domains needing custom embedding models.

For teams that only need embeddings and not a full RAG stack, look at Voyage AI.