How to Build a Voice AI Agent With Pipecat in Python
Pipecat is an open-source Python framework for building real-time voice AI agents. You assemble a pipeline of services — a transcription model, an LLM, and a text-to-speech engine — and it handles the streaming, interruption, and turn-taking between them. It installs with pip and runs on a normal Python 3.10+ environment.
The point of a framework here is latency. Chaining STT, LLM, and TTS naively adds seconds of silence. Pipecat streams each stage so the bot starts talking before the full response is generated.
What is Pipecat?
Pipecat is a Python library for real-time voice and multimodal agents. A pipeline is a chain of processors, each doing one job: capture audio, transcribe, generate, synthesize, play back.
It ships integrations for common providers — Deepgram, Whisper, OpenAI, Cartesia — so you swap components without rewriting the glue.
Is Pipecat free?
The framework is free and open source. The AI services you plug in are not — STT, LLM, and TTS each bill separately.
Rough numbers: real-time speech-to-text from a hosted provider costs around a cent per minute or less, and TTS is similar. A heavy voice bot can pass a few dollars an hour of call time, so budget per-minute before launching.
How do you install and run a basic bot?
- Create a virtual environment and install the framework: pip install pipecat-ai.
- Install the service packages you need, such as the Deepgram, OpenAI, and Cartesia extras.
- Set API keys as environment variables for each provider.
- Write a pipeline that chains transport input, STT, the context with your LLM, TTS, and audio output.
- Run the script and connect the provided client, or a WebRTC transport, to speak to it.
The project’s example scripts are the fastest reference; start by copying one and swapping the model names.
What do the pipeline components do?
| Stage | Job | Example |
|---|---|---|
| Transport | Move audio in and out | WebRTC, WebSocket |
| STT | Speech to text | Deepgram, Whisper |
| LLM | Reason and respond | OpenAI, Anthropic |
| TTS | Text to speech | Cartesia, ElevenLabs |
Interruption handling lives in the pipeline, not your code. When the user speaks mid-response, the framework can stop playback and re-listen, which is the behavior that makes a bot feel alive rather than scripted.
What makes latency good versus bad?
Use a streaming STT model that returns partial transcripts, not one that waits for an utterance to end. Pick an LLM that starts tokens quickly, and a TTS engine that streams the first audio chunk fast. The sum of first-chunk times is what the user feels.
Pipecat’s own metrics let you log each stage’s time-to-first-byte. Watch those numbers; a single slow stage ruins the whole exchange no matter how fast the others are.
Where does it get hard?
Barge-in — interrupting the bot — is genuinely tricky to tune. Too sensitive and background noise cuts the bot off. Too lax and the user talks over it. Expect to spend real time on voice activity detection thresholds.
Deployment also is not trivial: real-time audio needs a stable WebRTC or WebSocket server, not a simple REST host. If you want a text chatbot, this framework is overkill.
Who should use Pipecat?
Developers building phone agents, voice assistants, or multimodal real-time apps who know Python and can manage cloud services. If you want a hosted voice bot with a dashboard, a managed platform is faster. Pipecat is for people who need control over the pipeline.
Quick Summary
- Pipecat is a free, open-source Python framework for real-time voice and multimodal AI agents.
- It chains transport, STT, an LLM, and TTS into a streaming pipeline to cut latency.
- Provider services bill separately — budget roughly a few dollars per hour of call time.
- Barge-in tuning and real-time deployment are the hard parts; it suits developers, not no-code users.
