How a voice agent works

Before you build one, it helps to know what actually happens between someone speaking and your agent answering. Almost every configuration choice you'll make later maps onto one of the stages below.

New to the vocabulary? The glossary defines every term on this page.

The problem being solved

A text chatbot has it easy. A message arrives, complete and unambiguous, and the reply can take as long as it likes.

A phone call has neither luxury. Audio arrives as a continuous stream with no natural boundaries, and humans expect a reply within a few hundred milliseconds of finishing a sentence. Pause too long and the caller says "hello?". Reply too early and you've cut them off.

So a voice agent is really solving two problems at once: what to say, and when to say it.

The pipeline

Every call runs through the same five stages.

1. Voice activity detection (VAD)

The agent is always listening, but it must not react to every sound. VAD decides, many times a second, whether the incoming audio is speech or not speech — breathing, keyboard clatter, a television in another room.

VAD also decides the moment that matters most: when the caller has stopped talking. Cut too early and you interrupt someone mid-thought. Wait too long and the agent feels slow. This single judgement is responsible for most of a call's perceived responsiveness.

2. Speech to text (ASR)

Once a stretch of speech is captured, it goes to an ASR model, which returns text.

ASR is imperfect in specific, predictable ways, and it's worth knowing them because your prompt has to cope with them:

• Homophones. "their" and "there" are the same sound.
• Spoken punctuation. An email address arrives as fayaz at gmail dot com.
• Spelled-out numbers. "two thirty" rather than 2:30.
• Hallucinated filler. Given silence or noise, transcription models sometimes emit a plausible short phrase that nobody said.

The agent never hears audio. It only ever reads the transcript, with all of the above baked in.

3. The language model (LLM)

The model receives the conversation so far, your system prompt, and the list of tools it may call. It then decides one of two things: say something, or call a tool first.

Two details matter for how a call feels:

• It streams. The reply is produced word by word, and the agent starts speaking the first sentence while the model is still writing the second. Waiting for the full reply would add a second or more of silence to every turn.
• Tool calls take real time. Looking up an order, checking a calendar, searching your knowledge base — each is a round trip that happens while the caller is waiting. This is why the agent says something like "let me check that for you" before it goes quiet.

4. Text to speech (TTS)

The reply is spoken. Because the LLM streams, TTS works a sentence at a time rather than waiting for the whole paragraph.

TTS providers differ more than you'd expect — not just in how the voice sounds, but in how quickly audio starts arriving and how consistently. A provider with a good average and an occasional four-second stall produces a noticeably worse call than one that is slightly slower but steady.

See Choosing a voice for the trade-offs.

5. Back to the caller

Audio is streamed back over the phone line or the browser widget. Two things run continuously underneath:

• Echo cancellation stops the agent from hearing its own voice through the caller's speaker and mistaking it for the caller talking.
• Barge-in watches for the caller starting to speak while the agent is mid-sentence. When it fires, the agent stops immediately — the remaining audio is discarded, the LLM is cancelled, and the caller has the floor.

Turn-taking is the hard part

The pipeline above is straightforward. What makes voice agents difficult is deciding whose turn it is, and every hard case is a judgement call:

The caller does this	The agent should
Finishes a sentence	Reply
Pauses mid-sentence to think	Wait, say nothing
Says "mm-hmm" while the agent talks	Keep talking — that's a listening noise, not an interruption
Says "no, wait" while the agent talks	Stop immediately
Says nothing for 30 seconds	Check they're still there
Coughs while the agent talks	Keep talking

There is no setting that gets all six right in every environment. A caller on speakerphone in a noisy office and a caller on a headset need different sensitivities, which is why these thresholds are tuned for a typical phone call rather than exposed as dashboard controls. What you can do is hear the effect — see Testing your agent.

Where the time goes

Every stage adds latency, and it accumulates into the pause the caller actually experiences:

The platform targets roughly 700 ms end to end. Treat that as a target rather than a promise — the model and TTS provider you pick, the number of tools a turn calls, and upstream provider load all move it, sometimes by several seconds. Choosing a voice covers which choices cost you the most.

What you actually control

You don't configure the pipeline directly. You configure six things, and they land on the stages above:

What you set	Which stage	Covered in
System prompt	LLM	Writing the system prompt
Voice + TTS provider	TTS	Choosing a voice
LLM model	LLM	Anatomy of an agent
Knowledge base	LLM (via a tool call)	Knowledge base
Tools	LLM (via tool calls)	Tools
Speech thresholds	VAD + barge-in	Build your first agent

Next

• Core concepts — the objects you'll work with: workspaces, agents, campaigns
• Anatomy of an agent — what an agent is actually made of
• What happens during a call — the same pipeline, minute by minute