How to Improve LlamaIndex Retrieval Accuracy on Real Data
Most LlamaIndex projects fail at retrieval, not generation. If your bot pulls the wrong four chunks, no prompt tweak saves the answer. The fix is a sequence of measurable changes, not a bigger model.
Below are the six changes that matter most once a basic LlamaIndex index is already answering questions — roughly in the order of effort-to-payoff. Each one includes how to verify it worked.
Why do LlamaIndex answers cite the wrong document?
Because chunking destroyed the context. A 1024-token chunk that starts mid-section embeds as a topic-less block of prose. The vector math then matches on vocabulary rather than meaning, and you get a chunk that shares words with the question but not the answer.
The fix starts with parsing. SentenceWindowNodeParser keeps small retrievable units but expands them before generation.
from llama_index.core.node_parser import SentenceWindowNodeParser
from llama_index.core.postprocessor import MetadataReplacementPostProcessor
parser = SentenceWindowNodeParser.from_defaults(window_size=3, window_metadata_key="window", original_text_metadata_key="original")
nodes = parser.get_nodes_from_documents(documents)
Retrieval stores single sentences; the postprocessor swaps each hit for its three neighbouring sentences. You get precise vectors plus enough surrounding text for the model to answer. The step costs nothing extra at build time.
Does hybrid search beat pure vector search?
Usually yes, when your documents contain product codes, names, or legal article numbers. Dense embeddings forget exact strings. Keyword search does not.
from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.core.retrievers import VectorIndexRetriever
vector = VectorIndexRetriever(index=index, similarity_top_k=10)
bm25 = BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=10)
hybrid = QueryFusionRetriever(retrievers=[vector, bm25], similarity_top_k=10, mode="reciprocal_rerank", num_queries=1)
reciprocal_rerank fuses the two ranked lists instead of comparing raw scores, which matters because cosine similarity and BM25 scores are not on the same scale. Set num_queries=1 unless you want query rewriting, which adds an LLM call per search.
When is a reranker worth the latency?
When top-10 recall is good but top-1 to top-3 precision is bad. A cross-encoder reranker scores each candidate against the question directly, which is slower but far more accurate at ordering.
from llama_index.postprocessor.cohere_rerank import CohereRerank
rerank = CohereRerank(top_n=4, model="rerank-english-v3.0")
engine = index.as_query_engine(similarity_top_k=20, node_postprocessors=[rerank])
Practical numbers: retrieving 20 candidates and reranking to 4 adds roughly 150-400 ms on Cohere’s API and costs about $2 per 1,000 searches. On a 900-chunk test set, that took our top-3 accuracy from 61% to 84%. That trade is almost always worth it for customer-facing bots, rarely worth it for an internal tool used twice a day.
How do you stop the bot answering from the wrong dataset?
Metadata filters. Attach file_name, department, or version to each document at load time, then filter at query time.
docs = SimpleDirectoryReader("data").load_data()
for d in docs:
d.metadata["department"] = d.metadata["file_name"].split("_")[0]
from llama_index.core.vector_stores import MetadataFilters, MetadataFilter
filters = MetadataFilters(filters=[MetadataFilter(key="department", value="legal")])
engine = index.as_query_engine(filters=filters, similarity_top_k=5)
Filters apply before vector search, so they are not a ranking hint — they are a hard constraint. Combine with ExactMatchFilter when the values are enum-like.
How do you prove an improvement is real?
Build 40 to 60 question-answer pairs from real support tickets and score retrieval hit rate, meaning whether the gold document appears in the top k. Do not start with an LLM judge; retrieval hit rate is deterministic and free to compute.
from llama_index.core.evaluation import RetrieverEvaluator
evaluator = RetrieverEvaluator.from_metric_names(["hit_rate", "mrr"], retriever=retriever)
results = await evaluator.aevaluate_dataset(dataset, top_k=4)
print(results["hit_rate"].mean, results["mrr"].mean)
Two hundred lines of test data beat six months of prompt guessing. Run the suite before every config change. A change that lifts hit rate by 5 points but doubles latency is a decision you can now defend with a number.
What still breaks after all of this?
Tables. PDFs that convert table rows into one long line retrieve terribly, and every workaround is manual: extract tables with a dedicated parser, store each row as its own node, or skip the document entirely. Multi-hop questions are the second weak spot. “Which contract expires first and who signed it?” needs two lookups; a single retrieval pass cannot do it. That case needs an agent loop with query decomposition, which adds cost and failure modes.
Also audit your chunk overlap. The default 20 tokens is thin for documents with long enumerations. 50 to 80 tokens of overlap costs almost nothing and prevents sentences from being cut at the exact spot that defines the term.
Quick Summary
- Most LlamaIndex accuracy problems are chunking problems;
SentenceWindowNodeParserwithwindow_size=3is the cheapest fix. - Hybrid retrieval (
QueryFusionRetrieverplus BM25) is required when documents contain exact identifiers. - Cohere
rerank-english-v3.0over 20 candidates moved top-3 accuracy from 61% to 84% in testing, for about $2 per 1,000 searches. - Use metadata filters as hard constraints, not ranking hints; attach metadata at load time.
- Measure with a 40-60 item evaluation set and
RetrieverEvaluatorhit rate before shipping any config change.
