How to Connect Langflow to a Local Ollama Model for Private RAG

November 14, 2026

Every hosted LLM pipeline sends your documents to someone else’s server. For legal, medical, or internal-only content, that is a blocker. Langflow has native Ollama support, so you can build the same RAG flow with a local model and a local embedding model, and nothing leaves the machine.

The cost is hardware and speed. A decent local setup answers a question in 3 to 8 seconds; OpenAI answers in under 2.

What hardware do you actually need?

Run a 7B model at 4-bit quantization and you need about 5 to 6 GB of VRAM. A 13B model needs 9 to 10 GB. Anything larger is not worth it for RAG, because the retrieval step carries most of the answer quality.

Practical options:

  • 8 GB VRAM (RTX 3060, M1/M2 MacBook with 16 GB unified memory): Llama 3.1 8B or Qwen 2.5 7B in Q4.
  • 12 to 16 GB (RTX 4070, M3 Pro): 13B models comfortably, or 8B with a long context.
  • CPU only: works with 3B models, expect 20 to 40 seconds per answer. Fine for testing, not for daily use.

How do you install and start Ollama?

One command on macOS and Linux:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:8b
ollama pull nomic-embed-text

Ollama runs a server on port 11434 by default. Confirm it works with curl http://localhost:11434/api/tags.

If Langflow runs in Docker, localhost inside the container is not your host. Use host.docker.internal:11434 on macOS or add --network host on Linux.

How do you swap hosted nodes for local ones in Langflow?

Replace two nodes, not five.

  1. Chat model: replace the OpenAI node with an Ollama node, base URL http://host.docker.internal:11434, model llama3.1:8b.
  2. Embeddings: replace the OpenAI embeddings node with an Ollama Embeddings node using nomic-embed-text (768 dimensions).
  3. Leave the rest alone — File, Split Text, Chroma, Prompt template all work unchanged.

Critical detail: if you reindex existing documents with a new embedding model, clear the vector store first. Mixing 1536-dimension OpenAI vectors with 768-dimension Nomic vectors in one collection returns garbage.

What quality loss should you expect?

Noticeable but smaller than people assume, when retrieval is good.

An 8B model with well-chosen context chunks answers document questions about as well as a frontier model in most internal cases. It is weaker at multi-step reasoning across several chunks, and weaker at following long instructions with many constraints.

Where local collapses: long documents where the answer requires synthesizing 10 or more chunks. Frontier models hold much longer contexts without degrading.

How do you make it faster?

Three levers, in order of impact:

  • Lower the context window. Ollama’s default can be generous; set num_ctx to 4096 for RAG where each answer uses 4 chunks of roughly 1000 characters.
  • Cut chunk count. Four chunks at 512 tokens beats eight at 256 for speed and usually quality.
  • Quantize harder. Q4_K_M is the usual sweet spot; Q3 is faster and noticeably dumber.

Who should not go local?

Anyone without a GPU and a real workload. A CPU-only setup is a demo, not a tool.

Teams that need audit trails and consistent output. Local models drift more between versions, and a model update can quietly change your answers.

Quick Summary

  • Langflow runs a full RAG flow against Ollama with only two node swaps: chat model and embeddings.
  • A 7B model at Q4 needs about 5 to 6 GB VRAM; 13B needs 9 to 10 GB.
  • Use host.docker.internal:11434 when Langflow runs in Docker, not localhost.
  • Clear the vector store when switching embedding models; mixing 1536-dim and 768-dim vectors breaks retrieval.
  • Expect 3 to 8 seconds per answer with a GPU, and a real drop on questions needing 10+ chunks of context.