How to Build a Real-Time Voice Agent with Sesame CSM
A voice agent is three parts: speech-to-text, an LLM, and text-to-speech. Sesame CSM handles the last part with more natural delivery than classic TTS. The engineering challenge is making the three stages feel like one conversation.
Direct answer: Chain a streaming STT model into your LLM, feed the token stream into CSM in sentence-sized chunks, and play audio as it is generated to hide generation time.
How Do the Three Stages Fit Together?
A loop, not a pipeline you run once.
- Capture audio from the microphone.
- Transcribe with a streaming STT model, emitting partial results.
- Send the final transcript to your LLM.
- Collect LLM tokens until you have a full clause or sentence.
- Pass that chunk to CSM with speaker and recent context.
- Play the generated audio immediately; repeat as more tokens arrive.
Step four is the trick. Waiting for the complete LLM answer before speaking adds seconds of silence. Chunking at clause boundaries lets the first audio start while the rest is still being written.
What Is a Realistic Latency Budget?
Under 800 ms from end of user speech to first audio feels responsive. Above 1.5 s it feels broken. That total splits across STT, LLM, and CSM, so each stage must sit in a few hundred milliseconds.
| Stage | Target | Optimization |
|---|---|---|
| STT finalize | 200-300 ms | Use endpointing, not silence timers |
| LLM first token | 200-400 ms | Smaller model, short system prompt |
| CSM first audio | 200-400 ms | Stream chunks, cache context |
Every millisecond you do not spend on optimization is silence the user hears.
How Do You Keep the Voice Consistent?
Keep the speaker reference and a short rolling window of prior turns in every CSM call. Drop the context and the delivery drifts — the voice loses the thread of the conversation and starts sounding flat and disconnected.
But do not feed unbounded history. A growing context raises per-call cost and can slow generation. Six to ten recent turns is a good working window; summarize anything older.
What Audio Problems Show Up in Practice?
Four recurring ones. Clicks at chunk boundaries — solve by overlapping a few milliseconds of audio. Robotic joins when you split mid-word — split on sentence or clause boundaries instead. Echo when using speakers rather than headphones — run echo cancellation on the input. And drift in pacing as context grows, which the rolling window fixes.
Test with headphones first. Half of reported “robotic voice” complaints turn out to be feedback from open speakers.
Where Does CSM Stop Being Enough?
It is not a turn-detection system. Deciding when the user finished speaking is a separate problem, usually solved with voice activity detection. CSM also does not generate the words — your LLM does — so a weak LLM produces a natural-sounding agent answering poorly.
And without a GPU, this whole architecture is not viable. Plan your infrastructure before writing pipeline code.
Who Should Build This
Teams that need privacy-preserving voice assistants, like healthcare or internal tools where audio cannot leave your network. Not for anyone who needs a hosted, zero-ops solution — that is a managed voice API, and it will be simpler though less controllable.
Quick Summary
- A CSM voice agent chains STT, an LLM, and CSM in a streaming loop.
- Chunk LLM output at clause boundaries to start audio early.
- Target under 800 ms to first audio for a natural feel.
- Keep a rolling 6-10 turn context to hold the voice steady.
- Handle chunk clicks and echo before blaming the model.
