How to Cut Voice Agent Latency with Cartesia Streaming
Cartesia markets low-latency speech synthesis, and developers assume swapping in the API fixes a laggy voice agent. Sometimes it does. Often the delay sits somewhere else in the chain, and a faster TTS engine removes 200ms of a two-second pause.
This is the measurement-first approach: find where the time goes, then optimise the part that matters.
Where does the time actually go?
A voice agent round trip has five segments:
| Segment | Typical latency | Reducible by |
|---|---|---|
| End-of-speech detection | 300–800ms | Silence threshold tuning |
| Speech-to-text finalisation | 150–500ms | Streaming STT |
| LLM first token | 300–1,500ms | Model choice, shorter prompts |
| TTS time-to-first-byte | 100–300ms | Streaming synthesis |
| Network transport | 50–300ms | Region, WebRTC |
Add them up and the end-of-speech detection plus LLM first token usually dominate. TTS is rarely the biggest block — which is why buying a faster TTS engine alone disappoints.
How do you measure each segment?
Instrument every hop with timestamps. You need five numbers per turn, logged server-side: speech_end, transcript_final, first_llm_token, first_audio_byte_sent, audio_played.
Build a histogram of 200 turns, not an average. Voice delay is perceived at the 90th percentile — a user remembers the two-second pause, not the 400ms median.
Which mouth-to-ear delay is acceptable?
The conversational target is under 800ms from speech end to first audio locally, adding network round trip for remote users. Under 300ms starts feeling immediate. Above 1.2 seconds users talk over the agent.
Those numbers tell you where to spend effort. If end-of-speech detection is 700ms, halving it beats every TTS optimisation available.
What should you change in the TTS layer?
Three settings, in order of impact:
- Stream, do not batch. Request audio per sentence as the LLM produces it. Waiting for a full paragraph adds the whole generation time.
- Cap the first segment. Ask the model for a short opening sentence, then stream the rest. The first 15 words buy a second of generation time.
- Check the region. Synthesis in the wrong region adds 100 to 200ms of transport for nothing.
Once the first audio byte is under 200ms, further TTS work yields little. Move upstream.
What does this cost in complexity?
Streaming means cancellation logic — the user interrupts, and you must stop synthesis and drop queued audio. That is the part teams underestimate: barge-in handling is usually where a streaming pipeline produces bugs, not latency.
Also, smaller chunks mean more requests. Per-request overhead and rate limits become a real constraint at scale; check the concurrency and request rate your plan allows.
Who should do this work
Teams shipping a live voice product where conversation quality is the product. Anyone who has already swapped TTS providers and still hears complaints about lag.
It is unnecessary for recorded narration, where total synthesis speed matters more than time-to-first-byte — and for prototypes, where a 1.5-second pause is cheap to fix later.
Quick Summary
- Model five segments; end-of-speech detection and LLM first token usually dominate, not TTS.
- Log five timestamps per turn and judge on the 90th percentile, not the mean.
- Target under 800ms speech-end to first audio; under 300ms feels immediate.
- Stream per sentence, request a short opening sentence first, and check the synthesis region.
- Streaming adds barge-in cancellation complexity — that is where bugs appear.
