How to Build a RAG Pipeline with LlamaIndex in Python
LlamaIndex is a Python framework for connecting large language models to your own data. The most common thing people build with it is retrieval-augmented generation: ask a question, the app finds relevant passages in your documents, and the model answers using those passages instead of guessing.
You can get a basic version running in about thirty lines. The quality depends on two settings, not on model choice.
What you need installed
Python 3.9 or newer and an API key from a model provider. Install the core package and a reader for your file type.
pip install llama-index llama-index-readers-file
Set your API key as an environment variable so it never lands in your source file.
How do you load and index documents?
Point a loader at a folder, then build an index. LlamaIndex splits documents into chunks automatically, and the chunk size is the first setting that matters.
- Default chunk size is around 1024 tokens
- Drop to 256-512 for dense technical docs where answers live in single paragraphs
- Keep larger chunks for narrative content that needs context
After indexing, persist to disk so you do not re-embed on every run.
How do you query the index?
Build a query engine and call it with a string. The engine embeds your question, retrieves the closest chunks, and passes them to the model with instructions to answer from context. By default it pulls two chunks. The second setting that matters is top_k.
| Setting | Default | When to change |
|---|---|---|
| chunk_size | ~1024 tokens | Lower for dense docs |
| similarity_top_k | 2 | Raise to 4-6 for broad questions |
| response_mode | compact | Use tree_summarize for multi-doc summaries |
Is LlamaIndex free?
Yes. The library is open source under the MIT licence, so there is no licence fee. You pay only for model and embedding API calls, and you can avoid most of that by running a local model through Ollama. Embedding a 500-page corpus typically costs well under a dollar with a hosted provider.
Why are your answers wrong even though it runs?
Three usual causes. Chunks are too large, so the relevant sentence is buried in noise. Top_k is too low, so the answer is not in the retrieved context at all. Or the documents overlap and contradict, which the model happily reflects back as a confident wrong answer. Print the retrieved chunks before blaming the model.
Key Takeaways
- LlamaIndex is an open-source Python framework for RAG and document querying.
- A basic pipeline needs LlamaIndex plus a document reader and a model API key.
- Chunk size and top_k decide answer quality more than model choice.
- The library is MIT-licensed; you only pay for API calls.
- Debug by printing retrieved chunks before changing the model.
