How to Cut a Long-Chat Memory Bill with Zep Summaries and Search
Most teams adopting Zep for the first time send it everything and pull everything back. That is the expensive way to use it. The cheaper pattern is to decide per request whether you need a summary, a search result, or nothing at all.
This is the advanced half of Zep work: what the summary pipeline actually produces, which endpoint answers which question, and how to keep a memory block from quietly doubling your input tokens.
What does Zep’s summary pipeline actually produce?
On top of the fact graph, Zep keeps a running summary of the session. A background job compares the new turns against the existing summary and writes an updated one that references established entities, so the summary stays coherent across hundreds of turns.
What comes back from a summary query is a short paragraph, usually 150–250 words: what the user is trying to do, what was decided, what is still open. It is not a transcript and it does not include every fact.
That is a feature. A transcript in a prompt invites the model to reason about old details that no longer apply. A summary states the current position.
When should you use search instead of the summary?
Search whenever the user’s question has a specific noun in it: a ticket number, an account name, a product code. Search does keyword plus embedding retrieval across the graph and returns the matching facts with their timestamps.
The decision rule that has held up in practice:
- “What is this customer’s setup?” — summary.
- “What did they say about invoice 88421?” — search.
- “Remind me who they are” — summary.
- “Anything about the migration?” — search, then summarise the hits yourself.
A hybrid works too: fetch the summary as background, add three search hits for the named entity. Total comes to roughly 500 tokens and covers both cases.
How do you keep the memory block from growing over time?
This is where most bills go sideways. The memory endpoint returns facts for the whole session by default. As a support conversation approaches 200 turns, the fact count rises into the hundreds and a single retrieval can return several thousand tokens.
Three controls:
- Scope the read. Fetch memory per session, not per user, when the conversation is self-contained. User-level memory is for facts that outlive a single ticket.
- Cap search results.
limit=5is usually enough.limit=20costs four times the tokens and adds maybe one useful fact. - End sessions. Once a ticket closes, ending the session freezes its summary. New conversations start clean and the old one becomes searchable history rather than active memory.
A team running 5,000 daily requests that shaved 1,200 tokens off the memory block saves almost 6 million input tokens a day. That is the whole game.
How does custom extraction change the result?
Zep’s default extractor looks for people, organisations, projects, preferences and events. If your domain has nouns it will not classify (SKUs, lab codes, prescription names), those get stored as prose and retrieval for them degrades.
Custom instructions fix it. You describe the entity types and fact patterns you care about, and extraction starts emitting them as typed nodes. Two cautions: custom extraction increases cost per ingest, and a badly written instruction set will produce noisy entities that pollute search.
Write instructions that mirror how your users actually talk. “Track server names, regions and deployment dates” works. “Track all technical details” does not.
What about latency in a real request path?
Ingestion is asynchronous, so it does not block the response. Retrieval is synchronous, and a memory read is typically a few tens of milliseconds against the cloud API; a graph search with an LLM reranking step is slower, sometimes several hundred milliseconds.
If you are on a strict sub-second budget:
- Read the summary in parallel with your main model call, not before it.
- Cache the memory block per session for 60 seconds. Most users send another message within that window.
- Skip the rerank step for trivial questions by using the summary alone.
How does this compare with trimming the history yourself?
Trimming works and costs nothing. But it has a failure mode nobody talks about: the model loses the reason behind a constraint. History gets cut at turn 40, and the “don’t mention competitors” rule from turn 3 is gone. The next response names a competitor.
Zep’s summary keeps decisions and constraints as first-class facts, which is why it survives the cut. If your assistant has no constraints like that, plain trimming is fine and cheaper.
When is Zep the wrong call?
If your assistant is single-session by design (a document Q&A tool where every conversation is independent), memory is dead weight and an extra network hop. If you need an exact, replayable log of every message for compliance, a graph extraction layer is the wrong storage primitive; keep your own transcript store alongside it.
Also skip it if you cannot tolerate a vendor in the request path. Self-hosting Graphiti is a real option, but it means owning Neo4j or FalkorDB, an extraction model, and the consolidation jobs.
Quick Summary
- Zep keeps a running session summary of roughly 150–250 words; it is the cheapest thing you can put in a prompt.
- Use summary for “who is this”, search for named nouns like ticket or invoice numbers, and combine both for about 500 tokens.
- Scope reads to a session and cap search at around 5 results, or fact accumulation quietly doubles your input tokens.
- Custom extraction instructions help domain nouns such as SKUs, but raise ingest cost and can pollute search if vague.
- Skip Zep if conversations never repeat or if you need exact transcripts for compliance rather than extracted facts.
