How to Index One Million Documents in LlamaIndex on a Budget
A million documents at LlamaIndex defaults is a five-figure embedding bill and a job that cannot finish before it crashes. The way through is to stop treating indexing as a one-shot script and treat it as a pipeline with caching, batching, and a persistent store.
The difference is not clever code. It is three configuration choices that do the heavy lifting.
Why does the naive approach fail at scale?
VectorStoreIndex.from_documents(docs) holds every node in memory, computes embeddings in one pass, and rebuilds from scratch whenever you run it. At a million documents you hit three walls:
- Memory: node objects plus vectors can exceed a modest server’s RAM well before the end.
- Cost: a full re-embed on every run charges for the same text repeatedly.
- Time: an interrupted run has no checkpoint, so a crash at 90 percent means starting over.
All three are solved by the same mechanism: IngestionPipeline with a docstore and a vector store, both persistent.
How does the ingestion pipeline change things?
It separates transformations from storage and adds a document-level cache. Nodes that have already been processed with the same transformation hash are skipped on subsequent runs.
Configure it with an explicit docstore and vector store, then run in batches:
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.storage.docstore import SimpleDocumentStore
pipeline = IngestionPipeline(transformations=[splitter, embed_model],
docstore=docstore, vector_store=vector_store)
for batch in chunked(files, 500):
nodes = pipeline.run(documents=load(batch), num_workers=4)
docstore.persist("store/docstore.json")
Persisting after each batch is what makes the job resumable. Without that persist call, a crash still loses everything, which defeats the purpose.
For a million documents, swap SimpleDocumentStore for a Redis or Postgres docstore. A JSON file with a million nodes will not load in reasonable time.
Which embeddings are cheap enough?
Embedding cost scales linearly with tokens, so the model choice is the budget. A small embedding model at roughly 384 to 768 dimensions costs a fraction of a large one and retrieves well enough for most internal search.
Two rules:
- Re-embedding everything to switch models costs the full price again. Pick once, and store the model name alongside your vectors so a future migration is deliberate.
- Embedding a million documents at 500 tokens each is 500 million tokens. Multiply that by your provider’s per-token rate before you start. If the number is uncomfortable, downsample first — deduplicate near-identical documents, drop boilerplate pages, and exclude anything under 200 words.
Deduplication is usually the biggest single saving. Corporate corpora often contain 20 to 30 percent near-duplicates from document versioning.
How do you keep retrieval fast at that scale?
A single flat index over a million vectors returns results, but latency and memory both suffer. Three practical measures:
- Index the metadata. Filter by source, date, or department before vector search rather than after.
- Use a vector store with native filtering and quantisation. Scalar quantisation to int8 typically reduces memory by about 4x with minimal recall loss.
- Retrieve 20, rerank to 5. A reranker on the candidate set costs a little latency and improves answer quality more than raising top_k on the initial search.
What still costs money after indexing?
Queries. Every question embeds once and sends 4 to 6 chunks to the generation model. At a few hundred queries a day, the generation side dominates the bill, not retrieval. Cache frequent questions at the application layer — internal search traffic is heavily skewed to a handful of repeated questions.
Re-indexing is the other recurring cost. Set a policy: incremental updates through the docstore for changed documents only, with a full rebuild scheduled at most twice a year.
Quick Summary
- Use
IngestionPipelinewith a persistent docstore and vector store, and persist after each batch. - The docstore cache skips already-embedded nodes, so re-runs are cheap and resumable.
- Choose a small embedding model once and store the model name with the vectors.
- Deduplicate before embedding; corporate corpora run 20-30 percent near-duplicates.
- Retrieve 20, rerank to 5, and filter by metadata before the vector search.
