Ask someone what makes a voice agent good and they will talk about the model. Does it understand, does it say the right thing, does it hallucinate.
Run one on real phone calls for a while and you find out that even a capable model cannot hide bad timing. What they notice is being talked over. What they notice is finishing a sentence and hearing nothing for two seconds. The thing that decides whether an agent feels like a person or like an IVR is not what it says. It is when.
I have been building voice agents at Spyne for about two and a half years now, first on a managed platform, now on LiveKit Agents, and turn-taking is the problem I keep coming back to. This is what I have learned about it, most of it the hard way.
The problem in one sentence
Humans manage turn-taking with prosody, grammar, eye contact and a lifetime of practice, and we still talk over each other constantly. A voice agent gets one mono audio stream over a phone line and a few hundred milliseconds to make the same call.
Silence is a bad signal
A common starting point is a silence threshold. The caller is done when they have not made noise for N milliseconds.
It is wrong in both directions at the same time. Make N short and you cut people off. Callers pause to think. They pause to find a document. Someone reading out a VIN or a phone number pauses between every group of digits, and an eager agent will jump in at every single gap, which is exactly as infuriating as it sounds. Make N long and every exchange in the whole conversation inherits that delay. The caller finishes, the line goes quiet, and the agent sounds slow even when the model answered in 300ms.
There is no correct N. Whether a pause is the end of a turn depends on what was just said, and silence cannot tell you that.
Make endpointing care about the words
The fix is to make the endpointing decision content-aware. “What is a good callback number?” followed by six digits and a pause is probably not a finished turn. “Yeah that works, book it” almost certainly is.
LiveKit supports context-aware turn detection and adaptive endpointing. The decision can use what was said as well as how long the caller has been silent. The available detectors and tuning options depend on the SDK version and pipeline. In my experience this one change does more for how an agent feels than a faster model does.
I care about this layer enough that I have spent time prototyping my own detectors against the LiveKit Agents framework, including one that asks an LLM to score whether the turn is complete, and a hook for plugging custom audio-based detectors into the pipeline. Neither is something I would have been able to try on a closed platform. That is a large part of why I like working on an open one.
The latency budget
Turn-taking and latency are the same problem from two angles. From the moment the caller stops talking, everything downstream is on the clock: the endpointing delay itself, STT finalising, the model’s time to first token, TTS time to first byte, and the network in between. The acceptable gap depends on the task: a quick acknowledgment and a database lookup have different expectations. Measure the full gap the caller hears, not just the model’s response time.
Stream model output into speech and play audio as it becomes available. Treat generation from an unfinished user turn as speculation: cancel it when the transcript changes, and keep irreversible tools behind a committed turn. LiveKit calls this preemptive generation. Model choice and service tier can affect latency; output caps mainly bound generation and do not guarantee a faster first token. Part of what I have upstreamed to the LiveKit Agents SDKs is exposing exactly those OpenAI Responses API controls, because in a voice pipeline they are latency features first.
Do not add artificial delays just to sound human. When a tool really takes time, a brief acknowledgment such as “let me check that” can explain the wait. Repeated filler on every turn becomes its own annoyance.
Barge-in is the other half
Endpointing decides when to start talking. Barge-in decides when to stop, and it is harder.
When the caller talks over the agent you have to notice fast, which means running voice activity detection on the inbound stream while the outbound stream is playing, which on a phone call also means dealing with the echo of the agent’s own voice. Then you stop the audio, flush whatever TTS is queued, and cancel the in-flight generation, otherwise the agent comes back after the interruption and finishes answering the old question.
Then there is the part nobody warns you about. The agent said half a sentence. Did the caller hear the confirmation number or not? Your conversation state has to record what was actually played, not what was generated. I have watched agents confidently “repeat” something the caller never heard.
And not every noise is an interruption. “Mm-hmm,” “okay,” “right” are the caller telling you to keep going. An agent that stops dead for every acknowledgment feels broken in a way that is hard to describe but instant to notice.
The phone makes all of it worse
Everything above gets harder on a real call. 8kHz narrowband audio. Speakerphones in cars. Showroom noise in the background. Carrier jitter. Answering machines whose greetings are designed to sound like a person picking up. If your test setup is a laptop microphone in a quiet room, you have not tested turn-taking.
This is most of why I ended up building a telephony simulator that plays scripted, imperfect callers at the agent over real carrier paths and grades what it does. Turn-taking regressions do not show up in unit tests. They show up as a slow drift in how many callers hang up.
Measure the failures separately
A single average latency hides the trade you are tuning. Keep a labeled set of complete turns, mid-sentence pauses, digit sequences, backchannels and genuine interruptions. Split results by language and audio conditions.
| Measure | Definition | Why it matters |
|---|---|---|
| Response gap | Last caller speech sample to first audible agent sample | Includes endpointing, generation and transport; track p50 and p95 |
| Premature reply rate | Replies started before a labeled user turn ended, divided by labeled turns | Exposes an agent that is fast because it cuts people off |
| Interruption stop time | Genuine interruption onset to the last audible agent sample | Measures playback stopping, not just a cancellation event |
| False interruption rate | Backchannels or noise that stopped playback, divided by labeled non-interruptions | Catches an agent that cannot finish a sentence |
Replay these cases after changing a detector, STT provider or audio buffer. Listening finds the failure; a labeled corpus makes the regression repeatable.
What I would tell someone starting today
Make endpointing semantic. A turn-detection model with adaptive thresholds is the highest-leverage component in the pipeline.
Budget latency end to end, stream where useful, and measure provider settings against real calls.
Treat barge-in as state management, not just audio. Track what the caller actually heard.
Keep talking through “mm-hmm.”
Acknowledge a real wait without adding one.
Test against bad audio over real carriers, and grade it automatically, so listening sessions have repeatable checks behind them.
None of this shows up in a model comparison table. That is roughly why it is where the difference is. The teams whose agents feel human do not have a secret model. They took turn-taking seriously.