Building AI phone agents that callers do not hang up on

The difference between an AI agent people use and one they immediately ask to bypass is rarely the language model. It is the conversational mechanics: how fast the agent responds, whether it can be interrupted, and what it does when it reaches the edge of its competence.

Latency is the product

Humans notice silence after about 500 milliseconds and start talking over the agent at around a second. Budget the whole loop: speech recognition, reasoning, speech synthesis and the network hops between them. Streaming every stage, and colocating them, is the difference between a conversation and a walkie-talkie.

Barge-in is not optional

Callers interrupt, especially when the agent reads a list. The agent must stop speaking within a couple of hundred milliseconds, keep what it already said as context, and treat the interruption as the new topic. Agents that finish their paragraph regardless feel like an IVR with better vocals.

Design the escalation before the conversation

• Decide the hard boundaries first: payments above a threshold, medical or legal advice, angry callers. These route to people, always.
• Hand over with a spoken summary and the transcript attached, so the caller never repeats themselves.
• Measure containment honestly: a call the agent resolved badly is worse than a fast escalation.

Start narrow, instrument everything

The agents that succeed launch with one job, such as appointment booking, reach high resolution rates there, and expand. Review transcripts weekly, tag failure modes and fix the top one. It compounds quickly.