A voice agent doing outbound calls has to answer a question before any of the interesting ones. Not “what should I say” or “is the caller done talking,” but “is there a person here at all?”
Inbound calls usually make this less urgent, although forwarded calls, automated callers and IVRs can still put a machine on the line. The moment you start dialling out at any volume, answering-machine detection becomes one of the highest-leverage pieces of the whole system, and I almost never see it discussed.
Our agents at Spyne dial real people all day. Lead follow-ups, appointment confirmations, callbacks. A big share of those dials never reach a human. What the agent does in the first three seconds after the call connects decides whether the whole thing feels like a professional operation or a robodialer, and whether we are paying full pipeline cost to have a conversation with a greeting.
Why it matters more than it sounds
Miss a machine and the agent delivers its opening line to a voicemail box, then stands there waiting for a reply from nobody. Multiply that by a few thousand dials and you are burning STT, LLM and TTS minutes on silence, and leaving half-finished nonsense in people’s voicemail. I have listened to some of these. They are not flattering.
Miss in the other direction and it is worse, because it happens to a person. They pick up, say hello, and get dead air while the classifier makes up its mind. Or the call just drops, because the system decided they were a machine. Every one of those is someone who now believes your business robocalled them and hung up.
The cost pushes you one way. The human experience pushes the other. You cannot configure your way out of that tension; it has to be engineered.
The signals, such as they are
Nothing on a phone line announces “this is a recording.” You infer it.
The strongest cheap signal is how long the first stretch of speech is before a pause. People answer short and rising: “Hello?” Machines play a long, fluent, uninterrupted sentence: “Hi, you’ve reached...” That works most of the time. Then a receptionist answers with a scripted 15-word greeting, or someone’s voicemail just says “leave a message,” and it does not.
The second signal is what happens when you say nothing. A person who said hello and hears nothing says “hello?” again. A recording keeps going on its schedule. The interactivity itself is a probe, if you can afford the second it costs.
The third is the end of the greeting. A beep is useful evidence, but not every mailbox emits one and not every tone is a mailbox beep. Twilio’s AMD API distinguishes greeting endings detected through a beep, silence or another signal, and can also return unknown. Message playback needs a bounded greeting-end policy, not an indefinite wait for a tone.
Sometimes the carrier tells you the call diverted. Often it does not, and prerecorded intercepts (“the number you have dialled...”) add a third category that is neither a person nor a mailbox.
Implementations combine audio classification, timing rules and sometimes transcript cues, and the platforms expose AMD with exactly the knobs you would expect: how long to listen, how aggressive to be, whether to wait for the beep. Whatever the implementation, the shape of the problem is the same. You are trading confidence against dead air for real people. That is the endpointing trade in a different costume, which is why I think of AMD as turn-taking’s evil twin.
Detection is half of it
The other half is what to do, and that belongs in configuration rather than buried in agent code. In our platform, voicemail behavior is part of the runtime profile that describes every call, next to transports, transfer rules and duration limits, because different use cases genuinely want different answers.
Hang up and try later. One option for follow-ups when a message would not be useful and another attempt is permitted. The interesting part is the retry schedule: backing off, varying the time of day, capping attempts per lead. Redial behavior is also a compliance and reputation question, which I will get to in another post.
Leave a message. Useful for some confirmations and callbacks. This path needs greeting-end detection, a message written for voicemail rather than for a conversation, and a record that a message was left, so the next touchpoint does not repeat it word for word.
Classify the failure before retrying. A confirmed invalid destination can be suppressed; a busy line, temporary network failure or ambiguous intercept needs a different policy. An AMD guess alone should not permanently ban a number.
And when the classifier says human, the opening line has to work for someone who has already said hello twice. Openings that survive an awkward start do better than ones that assume a clean pickup.
Unknown is a result, not an exception
Keep classification separate from the action it triggers. An unknown result should not silently become “machine.” Give it a bounded fallback appropriate to the use case, and log the decision so you can audit human calls that ended early.
Synchronous detection delays the opening while it listens. Asynchronous detection lets the call proceed, but its result can arrive after the agent has started speaking. That result must be checked against current call state before it stops playback, starts a voicemail message or ends the call. A late callback after a human transfer should not hang up the transferred conversation.
The part everyone skips
You cannot test AMD by calling your own phone. The failures live in the variety: hundreds of voicemail greetings of different lengths, accents and hold music, beeps at different delays and pitches, carrier intercepts, silent pickups, people answering mid-sentence to their kids.
A lot of why I built my telephony simulator was this. It plays scripted callers and answering-machine scenarios at the agent over real carrier paths, and grades what the agent did. AMD is a statistical component, so the test has to be statistical too. Not “did it pass,” but “what is the false-human rate and the false-machine rate across the corpus this week, compared with last week.” Define the denominators: false-human rate is machines labeled human divided by labeled machines; false-machine rate is humans labeled machine divided by labeled humans. Track unknown rate and decision latency separately, so abstaining on difficult calls cannot masquerade as better accuracy. Once those numbers are on a dashboard, tuning turns into engineering. Before that it is folklore.
What it taught me
AMD looks like a small telephony detail. It is actually a compressed version of everything hard about voice: ambiguous audio, a real-time deadline, errors that cost different amounts in each direction, and a policy layer that product genuinely cares about.
The useful question is not whether AMD is enabled. It is how often each decision is wrong, how long it takes, and what happens to the person on the line when it is wrong.