Why a voice AI agent interrupts callers, and how to stop it

What the caller experiences

Your customer rings to ask about an order. Halfway through the sentence they pause to read the order number off a screen. The agent answers the half-question. They start again, louder, and the agent, which is now speaking, either talks over them or stops dead. By the third turn they are speaking in short bursts to stay ahead of it, and they hang up with the impression that your company has installed something that does not listen.

People who run voice agents describe this as one problem. On real phone lines it is three, and each one needs a different fix.

What is happening underneath

A voice agent cannot see the caller. Its only signal that a person has finished speaking is that the line has gone quiet, so it waits for a pause of a certain length and then replies. That decision is called end-of-turn detection, and every interruption you have ever heard from a voice agent is that decision being made too early.

The obvious fix is to make the pause longer. That works, and it is wrong, because the same delay is then added to every reply on every call, and past about two seconds of silence callers assume the line has dropped. A vendor who solves interruptions by turning up one number has moved your complaint from “it cuts people off” to “it is slow”, which is the complaint described in response time in a voice agent.

The three causes below are the ones we found on the way to our first completed phone call, which took four attempts. Two of them had a passing test asserting the opposite of what the phone line did.

Cause one: the pause threshold was tuned on one language

Measured against a labelled set of real calls, the pause that means “I have finished” is not the same length in English as in Hindi. English speakers leave shorter gaps inside a sentence. Hindi speakers leave longer ones. A threshold tuned on English recordings is therefore set to cut Hindi speakers off in the middle of a thought.

The obvious escape is to stretch the threshold to suit Hindi, at which point every English caller waits at the end of every sentence. And a caller who moves between the two languages inside one sentence, which is how most Indian customers talk on the phone, behaves like neither. The mechanics are set out in callers who mix Hindi and English.

The fix is a threshold set per language, measured against recordings of the calls the agent will take rather than against a default. This is measurement work somebody has to have done. It is not a setting you can turn on.

Cause two: a gap in the mobile network looks like a pause

Our tests assumed that audio arrives in the order it was sent and at a steady rate. On a laptop in an office that is true. On a mobile network it is not: a run of packets gets held up in transit and reaches the agent as a short silence, followed by a burst.

A rule that counts silence counts that gap. The caller never stopped talking. The agent decided they had, started its reply, and then the delayed audio arrived on top of it. From the caller’s side this is the agent interrupting them for no reason at all, mid-word, on a call where they had not paused.

This one is invisible in a demo, because the demo runs over a connection that does not do it. It is the reason a system that behaved well in a browser tab behaves badly on a customer’s phone, and it is one of the defects that four phone calls found and the tests did not.

The fix is to stop treating “no audio arrived” as the same thing as “the caller went quiet”. Those are different facts, and a system on a real phone line has to tell them apart.

Cause three: somebody else in the room is talking

Background conversation is speech. On an Indian mobile call there is nearly always some: a colleague, a television, a shop. A system that cannot tell speech aimed at it from speech that happens to be nearby will take the person behind the caller as the caller’s turn, answer it, and appear to have skipped the question that was asked.

The same weakness shows up in the other direction with the small noises a listening person makes. “Hmm”, “haan”, “okay” are not turns; they are the caller telling the agent to carry on. Taken as an interruption, they stop the agent mid-sentence. Taken as the end of a turn, they trigger a reply to nothing. Telling a listening noise from a real interruption is its own problem, and the work on it is described in public by people who have hit it.

The fix is a system that decides whether the speech it hears is the caller addressing it, which is harder than deciding whether it is speech at all. The opposite failure, an agent that stops dead when a door slams, is covered in barge-in.

How to tell which one you have

Three tests, each in a live demo on a real phone number rather than a browser tab.

1. Pause mid-sentence for two seconds, as though reading an order number off a screen, in Hindi and then in English. If the agent cuts in on one language and not the other, you have cause one.
2. Take the call on a mobile while walking, somewhere the signal is ordinary. If the agent interrupts you mid-word when you had not paused, you have cause two.
3. Have someone talk near you while you are on the call. If the agent answers them, or repeats itself, you have cause three.

Then ask the vendor one question: is the pause threshold the same for every language, and what was it measured against? The honest answer is specific. The other answer is a number. There is a longer list in nine questions to ask a voice AI vendor, and a way to run the same test on your own line, on one call type, for two weeks, starting with after-hours reception and the other calls to hand over first.

We found all three on our own line before a customer did. The fourth call completed. The first three are why this page exists, and interruption is one of eight entries in why voice AI agents fail on real phone calls.