How to Connect Chroma to LangChain for Retrieval Augmented Generation

Chroma stores vectors; LangChain orchestrates the prompt and model call. Together they form the standard RAG stack. This is the concrete wiring between them.

Quick answer: Wrap Chroma with langchain_chroma.Chroma, call .as_retriever(), and pipe the retriever into a prompt and model. Embeddings must match between indexing and querying or results are noise.

The single most common bug in this stack is a mismatched embedding model. You index with one model and query with another, and every result looks almost random. Keep them in one place.

How do you install the integration?

pip install langchain langchain-openai langchain-chroma chromadb

Note it is langchain-chroma, a separate package since the 0.2 split. Importing langchain_community.vectorstores.Chroma still works but is on the deprecation path.

How do you index documents into Chroma?

from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma
from langchain_text_splitters import RecursiveCharacterTextSplitter

emb = OpenAIEmbeddings(model="text-embedding-3-small")
texts = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
              .split_text(open("manual.txt").read())

store = Chroma.from_texts(texts, embedding=emb,
                          persist_directory="./chroma_db",
                          collection_name="manual")

retriever = store.as_retriever(search_kwargs={"k": 4})

Use the same OpenAIEmbeddings instance for indexing and querying. If you ever change models, you must re-index — the old vectors are meaningless to the new model.

How do you build the retrieval chain?

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

prompt = ChatPromptTemplate.from_template(
    "Answer using this context only. If unsure, say so.\n"
    "{context}\n\nQuestion: {question}")

chain = ({"context": retriever, "question": RunnablePassthrough()}
         | prompt | ChatOpenAI(model="gpt-4o-mini", temperature=0)
         | StrOutputParser())

print(chain.invoke("How do I reset the device?"))

The instruction to answer only from context is what stops the model inventing an answer. It is not a guarantee, but it measurably cuts hallucination on retrieval gaps.

What does this stack cost?

Chroma and LangChain are both free and open source. You pay for embeddings once at index time (about $0.02 per million tokens with text-embedding-3-small) and for the chat model on every query. A small documentation site costs cents per month to index and a fraction of a cent per question with a mini model.

How do you keep answers accurate?

• Log the retrieved chunks alongside the answer so you can see what the model actually saw.
• Raise or lower k and watch quality — too few chunks miss context, too many dilute it.
• Add metadata filters so a query only searches the relevant collection.
• Re-index when your source documents change; stale vectors give confident wrong answers.

Who this suits: developers building doc Q&A, internal search, or support assistants. Who should skip it: if your source fits in one prompt and never changes, you do not need retrieval — just paste the text and save the complexity.

Quick Summary

• Use langchain-chroma and the same embedding model for index and query.
• Chunk at 800 characters with 100 overlap as a starting point.
• as_retriever() plus a prompt template is the core RAG chain.
• Embeddings cost pennies; the chat model dominates query cost.
• Re-index on source changes or answers go stale.