How to Build a RAG Chatbot with LlamaIndex in 30 Minutes
LlamaIndex is an open-source Python framework for connecting LLMs to your own data. You install it with pip, load documents into an index, and query that index with natural language in under 50 lines of code.
If you are new to LlamaIndex itself, this walkthrough builds a small RAG chatbot over a folder of PDFs and Markdown notes. It uses LlamaIndex 0.11, OpenAI’s text-embedding-3-small for vectors, and gpt-4o-mini for generation. Expect about 30 minutes if you already have a Python environment and an API key.
What does LlamaIndex actually do?
LlamaIndex sits between your files and an LLM. It chunks documents, embeds them, stores vectors, and on each question it fetches the closest chunks and hands them to the model as context.
That last part matters. Without retrieval, an LLM answers from training data and makes up citations. With retrieval, it answers from the paragraphs you actually own.
The framework ships three pieces you will touch on day one: SimpleDirectoryReader for loading, VectorStoreIndex for storage, and a query engine for asking questions. Everything else — agents, workflows, response synthesizers — builds on top of those.
Who is this for, and who should skip it?
Good fit: Python developers who have 20 to 5,000 documents and want control over chunking, prompts, and model choice. Also teams that need to swap between OpenAI, Anthropic, and local Ollama models without rewriting glue code.
Poor fit: non-coders, and anyone whose entire corpus is five short pages. A long system prompt is cheaper and simpler at that size. If you just want a hosted chat box on a help center, look at a product like Chatbase instead of writing Python.
One honest caveat: LlamaIndex’s API has moved fast. Tutorials from 2023 use GPTSimpleVectorIndex, which no longer exists. Anything older than version 0.10 will fight you. Pin your version in requirements.txt.
Step 1: Install and set up the environment
Use Python 3.10 or newer. A virtualenv keeps the dependency blob contained, because the full llama-index package pulls in a long tail of integrations.
python -m venv .venv
source .venv/bin/activate
pip install llama-index==0.11.20 python-dotenv
Then export your key. LlamaIndex reads OPENAI_API_KEY by default:
export OPENAI_API_KEY=sk-...
The bare llama-index package installs the core plus common integrations. If you want a leaner install, use llama-index-core and add only the readers you need.
Step 2: Load your documents
Drop your files in a data/ folder. PDFs, .md, .txt, and .docx all work without extra code.
from llama_index.core import SimpleDirectoryReader
documents = SimpleDirectoryReader("data").load_data()
print(len(documents), "documents loaded")
One Document object is created per file, not per page. That detail bites people later: a 60-page PDF becomes one huge node, and retrieval gets fuzzy. Split it before indexing if your files are long.
Step 3: Build the index
from llama_index.core import VectorStoreIndex
index = VectorStoreIndex.from_documents(documents)
index.storage_context.persist(persist_dir="./storage")
By default LlamaIndex uses 1024-token chunks with 20 tokens of overlap. For most prose, 512 tokens retrieves better because the matching paragraph dominates the embedding instead of being diluted.
Cost reference: embedding 1,000 pages of text with text-embedding-3-small runs about $0.02. Re-indexing is cheap; the generation calls are what add up.
Step 4: Ask questions
query_engine = index.as_query_engine(similarity_top_k=4)
response = query_engine.query("What does the contract say about cancellation?")
print(response)
for node in response.source_nodes:
print(node.metadata.get("file_name"), round(node.score or 0, 3))
similarity_top_k=4 means the four nearest chunks go into the prompt. Three to five is the usable range. Ten chunks of 1024 tokens each will flood a small context window and the answer quality drops, not rises.
Always print source_nodes. It is the fastest way to see whether the retriever found the right passage or whether the model just guessed from context.
Step 5: Wrap it in a web endpoint
from fastapi import FastAPI
from pydantic import BaseModel
from llama_index.core import StorageContext, load_index_from_storage
app = FastAPI()
ctx = StorageContext.from_defaults(persist_dir="./storage")
idx = load_index_from_storage(ctx)
engine = idx.as_query_engine(similarity_top_k=4)
class Ask(BaseModel):
question: str
@app.post("/ask")
def ask(a: Ask):
r = engine.query(a.question)
return {"answer": str(r), "sources": [n.metadata.get("file_name") for n in r.source_nodes]}
Run it with uvicorn main:app --port 8000 and POST {"question": "..."} to /ask. That is a functional internal chatbot.
Which settings actually change answer quality?
Three knobs do most of the work.
- Chunk size. 512 tokens for prose, 256 for Q&A sheets and tables. Below 128 you lose context; above 1500 you dilute it.
similarity_top_k. Start at 4. Raise to 6 only if answers cite partial passages.- System prompt. The default tells the model to use only the context. Keep that sentence — removing it turns your RAG bot back into a hallucinating chatbot.
Embedding model choice matters less than people expect at small scale. text-embedding-3-small is 1,536 dimensions and fine for English corpora under ~50k chunks.
Common failure modes to expect
Repeated imports fail because storage/ was written by a different LlamaIndex version. Delete it and rebuild. Answers ignore a document because the PDF was scanned images — LlamaIndex reads text, not pictures, so run OCR first. Queries return nothing because the index was built in one process and every request rebuilds it; load once at startup instead.
Also watch the token bill. A chatbot answering 2,000 questions a day with 4 chunks of 512 tokens sends roughly 4 million input tokens daily, near $0.60 at gpt-4o-mini pricing. That is fine. Swap in a bigger model and the same traffic costs 30 times more.
Quick Summary
- LlamaIndex is a free open-source Python framework (Apache 2.0) for retrieval over your own documents; the pip package is
llama-index, current minor line 0.11. - Minimum viable setup is three calls:
SimpleDirectoryReader,VectorStoreIndex.from_documents,index.as_query_engine. - Use 512-token chunks and
similarity_top_k=4as defaults; adjust only with evidence fromsource_nodes. - Embedding 1,000 pages with
text-embedding-3-smallcosts roughly $0.02; generation dominates spend. - Skip it if you have no Python skills or fewer than a few dozen documents.
