How to Build a PDF Question-Answering App with LlamaIndex

November 16, 2026

LlamaIndex is a Python framework for connecting language models to your own data. For a PDF question-answering app it provides the three pieces you would otherwise write yourself: document loading, chunking with embeddings, and a retrieval layer that feeds the right passages into the prompt.

The hello-world is five lines. The version that works on real documents takes a bit more attention to chunking.

How do you install and index a folder?

pip install llama-index pulls the core package. Then, with an OpenAI key in the environment:

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex
docs = SimpleDirectoryReader("pdfs/", required_exts=[".pdf"]).load_data()
index = VectorStoreIndex.from_documents(docs)
print(index.as_query_engine().query("What is the refund window?"))

That works, and it is also where most people stop and then wonder why answers cite the wrong page. The defaults chunk at 1024 tokens with 200 tokens of overlap, store vectors in memory, and retrieve the top 2 chunks. None of those defaults fit most real corpora.

Note the memory part. VectorStoreIndex keeps everything in RAM. A few hundred PDFs is fine; a few thousand is not, and you will need a vector store like Qdrant, Chroma, or pgvector.

How do you pick a chunk size that works?

Chunk size decides what the retriever can find. Too small and a passage loses its context; too large and retrieval finds a page that mentions the topic but the answer is buried somewhere inside.

Reasonable starting points by document type:

  • Contracts and policy documents: 512 tokens with 64 overlap. Dense, self-contained clauses.
  • Technical manuals: 768 with 128. Longer procedures need continuity.
  • Support tickets or short notes: 256. Each item is already a unit.

Set these when constructing the index, not after: VectorStoreIndex.from_documents(docs, chunk_size=512, chunk_overlap=64).

Overlap is a hedge against a split landing mid-argument. 10 to 15 percent of the chunk size is enough; more just duplicates text and inflates your embedding bill.

How do you make answers cite sources?

Attach metadata at load time and ask the engine to include it. SimpleDirectoryReader adds file_name and page_label where the PDF parser supports it.

Then set similarity_top_k higher than the default — 4 to 6 — and use a response mode that summarises across nodes rather than stuffing one. More retrieved chunks means more context for the model, and more tokens per question.

A simple improvement that costs nothing: prefix each chunk with its file name and page number before embedding. That way the retrieved text itself carries provenance and the model tends to repeat it.

If two documents contradict each other, raise top_k and let the answer reflect both rather than silently picking one.

What breaks on real PDFs?

Scanned documents. If a PDF is images, the text extractor returns nothing and you get an empty index with no error. Check by printing the first 300 characters of each loaded document before indexing — if they look like garbage or are empty, you need OCR first.

Multi-column layouts get scrambled. A two-column academic paper often extracts as interleaved sentences.

Tables extract as a wall of numbers with no structure. For anything where the answer lives in a table, chunking will not save you; you need a table-aware parser or a different approach entirely.

And there is the cost question. Embedding 5,000 pages at roughly 500 tokens each means millions of tokens through the embedding model, once. Ask about re-indexing cost before you build a nightly rebuild job that quietly runs every night.

Quick Summary

  • LlamaIndex handles loading, chunking, embedding, and retrieval for your own documents.
  • Defaults are 1024-token chunks, top-2 retrieval, and in-memory storage — change all three.
  • Use 512-token chunks for contracts, 768 for manuals, 256 for notes.
  • Print the first lines of every loaded doc; scanned PDFs return empty text silently.
  • Move to a real vector store beyond a few thousand documents.