How to Build a Graph RAG Pipeline with Neo4j and LLMs

Vector RAG fails in a specific way: ask a question that requires joining two facts and the retriever hands back five chunks that each look relevant but none that answer. A graph fixes that, because the join is stored rather than inferred.

Direct answer: to build graph RAG on Neo4j, load entities and relationships into nodes, store chunk embeddings on those nodes, then query with a hybrid pattern that combines the vector index and a Cypher traversal before sending the subgraph to the LLM as context.

Why does a graph beat plain vector search here?

A vector index retrieves by similarity. It has no notion of “which supplier ships the part that failed in this ticket.” That is a two-hop question, and embeddings do not store hops.

Neo4j 5.x ships a native vector index (db.index.vector), so you do not need a separate vector database. Chunks live as nodes, vectors live as properties, and edges encode the structure your documents actually have.

The trade-off is real: schema design takes longer than dropping PDFs into a vector store. Budget a day for the schema, half a day for ingestion, and expect the first retrieval results to be mediocre until you tune hop limits.

How do you set up Neo4j locally?

The fastest route is Docker:

docker run \
  -p 7474:7474 -p 7687:7687 \
  -e NEO4J_AUTH=neo4j/testpass123 \
  -e NEO4J_PLUGINS='["apoc"]' \
  neo4j:5.26

Aura has a free tier instead if you prefer managed — 200 MB storage and one instance, which is tight but workable for a prototype. Either way, connect with the Python driver:

pip install neo4j openai

from neo4j import GraphDatabase
driver = GraphDatabase.driver("bolt://localhost:7687", auth=("neo4j", "testpass123"))

How should the schema look?

Start with three labels, not twelve. Entity, Chunk, and Document. Over-modelling is the most common mistake in graph RAG — teams create 15 node types and then cannot write a query.

CREATE CONSTRAINT ent_id IF NOT EXISTS
FOR (e:Entity) REQUIRE e.id IS UNIQUE;

CREATE VECTOR INDEX chunk_vec IF NOT EXISTS
FOR (c:Chunk) ON (c.embedding)
OPTIONS { indexConfig: {
  `vector.dimensions`: 1536,
  `vector.similarity_function`: 'cosine'
}};

The dimension must match your embedding model. text-embedding-3-small is 1536; text-embedding-3-large is 3072 and triples index memory for maybe 3% better recall on domain text. Use small unless you have measured a gap.

How do you extract entities during ingestion?

Two options. Rule-based extraction with spaCy is cheap and predictable; LLM extraction is better on messy prose but costs about $0.0005 per chunk at GPT-4o-mini prices.

prompt = """Extract entities and relations as JSON:
[{"name":..., "type":..., "related":[{"target":..., "rel":...}]}]
Text: {chunk}"""

Then merge with Cypher, using MERGE not CREATE so duplicate entities collapse:

UNWIND $rows AS row
MERGE (e:Entity {id: row.id})
SET e.name = row.name, e.type = row.type
MERGE (c:Chunk {id: row.chunk_id})
MERGE (c)-[:MENTIONS]->(e)

Run ingestion in batches of 500 rows. Single-row transactions on 50,000 chunks take hours; batched UNWIND takes minutes.

How do the hybrid queries work?

This is the part people get wrong. Do not just embed the question and take the top 5. Retrieve seeds by vector, then expand one hop and return the subgraph as text.

CALL db.index.vector.queryNodes('chunk_vec', 5, $qvec)
YIELD node AS seed, score
MATCH (seed)-[:MENTIONS]->(e:Entity)<-[:MENTIONS]-(c:Chunk)
WHERE c <> seed
RETURN seed.text AS seed_text,
       collect(DISTINCT e.name) AS entities,
       collect(DISTINCT c.text)[0..5] AS neighbours

Flatten that into a prompt with the entities listed first, then the chunks. LLMs weight a short structured entity list more heavily than five paragraphs of raw text, and that list is often the whole answer.

What limits should you set?

Cap the expansion. One hop keeps context under 3,000 tokens. Two hops on a dense graph can pull 40 chunks and blow past the window while adding noise.

Also cap fan-out per entity with [0..5]. A single hub entity like “Apple” connects to hundreds of nodes, and without a limit your query returns its whole neighbourhood.

Keep a updated_at on every node. Graph RAG goes stale in a way vector RAG does not — a deleted relationship stays queryable until you remove it.

Who should not bother?

If your documents are independent — a pile of support articles with no shared entities — a plain vector store is cheaper and simpler. Graph RAG pays off when the domain is genuinely relational: supply chains, org charts, code dependency trees, medical ontologies.

Also skip it if you cannot afford the ingestion latency. Adding a document means extraction, embedding, and merging — 2-8 seconds per document, versus milliseconds for a vector upsert.

Key Takeaways

• Neo4j 5.x has a native vector index, so graph and vector retrieval live in one database.
• Use three labels (Entity, Chunk, Document) and expand by one hop, five neighbours max.
• Batch ingestion with UNWIND in rows of 500; single-row MERGE is 10-50x slower on large corpora.
• Match vector dimensions to the embedding model — 1536 for text-embedding-3-small.
• Worth it for relational domains; overkill for independent documents where vector search already works.