End-of-turn detection: how a voice agent knows you stopped

What it is

A voice agent cannot see the caller. The only signal it has that a person has finished talking is that the audio has gone quiet, so it waits for a pause and, when the pause is long enough, starts its reply. That decision is end-of-turn detection. Underneath it is voice activity detection, the component that decides whether what is coming down the line is speech or not.

Everything a caller experiences as rudeness or slowness in a voice agent traces to this one decision going wrong in one of two directions.

The two directions it goes wrong

Too short. The agent takes an ordinary mid-sentence pause as the end of the turn and starts answering. The caller was about to say the second half of the sentence. Both now talk at once, the agent has answered the wrong question, and the caller has learned to speak in short bursts to stay ahead of it. This is the failure people describe as the agent interrupting them, and it is set out call by call in why a voice AI agent interrupts callers.

Too long. The agent waits long enough to be sure, on every turn, and every reply arrives after a silence. Callers do not experience this as caution. They experience it as a slow line, and past about two seconds they start saying “hello?”, which is covered in response time in a voice agent.

There is no value of one number that avoids both, because the callers on the other end are not all the same.

Why one number cannot work

Measured against a labelled set of real calls, the pause that means “I have finished” is not the same length in English as it is in Hindi. English speakers leave shorter gaps inside a sentence; Hindi speakers leave longer ones. A threshold tuned on English cuts Hindi speakers off. Stretched to suit Hindi, it leaves every English caller waiting. And a caller who moves between the two languages inside one sentence, which is how most Indian customers talk, behaves like neither. That is why Voicely sets the threshold per language against recordings, and why the mechanics of callers who mix Hindi and English matter more than a language count.

What else looks like a pause, and what else looks like speech

Two things confuse the decision that have nothing to do with the caller.

A gap in the network. On a mobile connection, audio does not arrive in order or at a steady rate. A run of packets held up in transit reaches the agent as a short silence, and a rule that counts silence counts it. The caller never stopped talking. The agent thinks they did. This one is invisible in a demo run over an office connection.

Somebody else in the room. Background conversation is speech. A system that cannot tell speech aimed at the agent from speech that happens to be nearby will treat the television as the caller’s turn, answer it, and appear to have skipped the question. The mirror image, an agent that stops mid-word when a door slams, is described in barge-in.

What to ask a vendor

• Is the pause threshold the same for every language? If the answer is one number, the system was tuned on one language.
• What was it measured against? “Our defaults” is a different answer from “recordings of calls like yours”.
• In the demo, pause for two seconds in the middle of a sentence, as though looking up an order number. Then take the same call on a mobile while walking.