How to Use Chroma as a Local Vector Store for RAG

October 12, 2026

Chroma is an open-source vector database that runs inside your Python process and persists to a local folder. No server, no Docker, no account. That makes it the default first choice for retrieval-augmented generation prototypes.

Quick answer: pip install chromadb, create a persistent client pointed at a directory, add documents with collection.add, and query with collection.query. Everything stays on disk between runs.

Chroma will not scale to millions of vectors with high concurrency, and it does not pretend to. For a single-machine app or a prototype, that ceiling is far away.

How do you install and create a persistent store?

pip install chromadb

import chromadb
client = chromadb.PersistentClient(path="./chroma_db")
col = client.get_or_create_collection(name="docs")

The path argument is the difference between a throwaway in-memory store and one that survives a restart. Use Client() for the ephemeral version when you are prototyping and want a clean slate each run.

How do you add documents?

Chroma can embed for you if you give it text and let it use its default embedding function, or you can pass vectors you computed yourself.

col.add(
    documents=["Chroma persists to disk", "Chroma runs in-process"],
    metadatas=[{"src": "guide"}, {"src": "faq"}],
    ids=["d1", "d2"],
)

IDs are required and must be unique. Re-adding an existing ID errors unless you use upsert.

How do you query with filters?

res = col.query(
    query_texts=["does chroma save data?"],
    n_results=2,
    where={"src": "guide"},
)

where filters on metadata before ranking, which is how you scope a search to one customer or document type. Metadata values should be simple types — strings, numbers, booleans. Nested objects are not supported.

Is Chroma free?

The library is Apache 2.0 and free. The hosted Chroma Cloud service has a free tier and paid plans above it. Self-hosted, your only cost is disk and whatever embedding provider you use.

How do you update and delete data?

col.update(ids=[...], metadatas=[...]) changes metadata without re-embedding.
col.upsert(...) inserts or replaces in one call.
col.delete(ids=[...]) removes vectors permanently.
client.delete_collection("docs") wipes everything — no confirmation.

When should you move to something bigger?

Chroma handles hundreds of thousands of vectors on a laptop fine. Move on when you need multi-user concurrency, horizontal scale, or filtered search across tens of millions of rows. Weaviate and similar servers exist for that jump. The API shape is similar enough that migration is mostly a rewrite of the store layer, not the whole app.

Who this suits: solo developers and small teams who want retrieval working today without infrastructure. Who should skip it: production systems serving many concurrent users, where a server-based database is the honest choice from day one.

Quick Summary

Chroma is a free, open-source, in-process vector store that persists to a folder.
Use PersistentClient(path=...) so data survives restarts.
Add with add, filter with where on flat metadata.
Handles hundreds of thousands of vectors locally without a server.
Migrate to a server-based store when concurrency becomes the bottleneck.

Tags:

Chroma