Conversational Voice AgentsConversationalVoiceAgents

The first version ran on a managed platform. At zero agents in production that was the right trade. Media transport, telephony, model orchestration and interruption handling all arrived on day one, for a per-minute fee. What ran out was reach. The incidents that matter are never “the call failed”. They are “the agent went quiet for two seconds and the caller hung up”. Answering that needs per-stage latency and the ability to change the execution path. So I wrote my own contract a year early, while still managed. The move to LiveKit then added a second worker instead of starting a rewrite.

Every call begins as a typed runtime profile. It names providers and models, prompts, tools, transport, recording, voicemail behavior, idle handling, duration limits, transfers and reporting. Unknown providers fail explicitly instead of falling back in silence. The control plane hands that profile to a worker and keeps the call state. The LiveKit worker and the Pipecat worker share generated wire contracts, not engine internals, so each keeps its native pipeline and lifecycle. Lifecycle events travel back idempotent and HMAC signed. OpenTelemetry spans cover the real-time pipeline, so one call looks the same in my tooling whatever executed it. The engine stays fixed for the life of a call.

• Write the contract before needing it. Building runtime profiles while still on the managed platform bought nothing visible for months. It is the only reason the engine swap moved traffic by profile rather than by flag day.
• Own the phone system. The model pipeline was not the hard part. SIP trunking, caller identity and attestation, detection and transfers were. Infrastructure cost fell from $0.26 to $0.10 a minute, and the engineering to operate it is not inside that figure.
• Keep two workers alive. Every contract change now has to cross both. In return, a new runtime can be graded against the same profiles before any traffic reaches it.
• Detect answering machines asynchronously. Waiting for a verdict adds dead air for real people, so the call proceeds while detection runs. The cost is that the verdict can land after the agent has started speaking, and every consumer has to re-check call state first. That is the one I would revisit.

Take an outbound follow-up. The dial connects, and detection runs while the opening line is still queued. A verdict of machine routes the call to the voicemail behavior in its profile. That is either a hang-up and a retry on a backoff, or a message written for a mailbox, logged so the next touchpoint does not repeat it. A verdict of unknown stays its own result with a bounded fallback, never quietly rewritten to machine. The failure that matters is the other direction. A person picks up, says hello twice, and a late callback ends the call. It surfaces as a short call whose transcript and end reason disagree. So false-human and false-machine rates are tracked separately, with unknown rate and decision time beside them.

Today the platform runs inbound, outbound and web calls with transport-specific routing. It models recording, cold transfers and the boundaries of a warm transfer. Reported metrics cover model usage, latency, transcripts, end reasons and failure context, keyed by call, tenant, use case, engine and model. Turn-taking is the standing open problem rather than a solved one. Endpointing, barge-in and backchannel handling get tuned against a real call distribution. The implementation is maintained in private repositories, so this page describes the architecture and the decisions rather than the source.