Why voice AI agents fail on real phone calls — and the fix for each

Where pilots die

Nearly every voice AI demo works. It runs in a browser tab on a laptop with a good microphone, in a quiet room, on a script the vendor chose. Then the agent is pointed at a real number, and the first caller is on a moving train, switching between Hindi and English in one sentence, talking over the answer because they already know what they want.

That gap is where pilots die. Our own first complete phone call took four attempts, and each of the first three found a defect that every automated test had passed. The eight failures below are the ones we have watched on real lines, in the order a buyer tends to meet them, each with the fix and with the question that shows whether a vendor has done the work.

It was tested in a browser, and the browser lied

A laptop microphone on an office connection makes three things true that a mobile network makes false: audio arrives in order, it arrives at a steady rate, and the moment the system finishes sending a reply is the moment the caller finishes hearing it. A test suite written from those assumptions confirms them every time. The suite goes green because the code matches the assumption, and the phone line does not.

The fix is to treat production as the test: assertions that can only be checked on a live call run on live calls. The story of how we learned that is in what four phone calls taught us that the tests did not.

The question: what has gone wrong on a real call, and what did you change? A vendor with no list has either not been on a phone line or is not looking.

The servers are on another continent

A voice call is a loop. Audio goes to a server and an answer comes back, on every turn. If the servers are far away, that round trip is paid before any work has begun. From our Mumbai deployment, reaching speech recognition takes about 20 ms. Reaching the same models served from the United States takes about 300 ms, and that gap is spent again on every turn of every call. Telephony has known for decades how little delay a conversation tolerates, which is what ITU-T G.114 is about.

The fix is to serve the whole call from the country the callers are in: speech, reasoning and voice, not one of the three. Why this matters more on Indian lines is in why voice AI built for English struggles on Indian calls.

The question: where is each part of the call served from? “Our cloud has a region in India” answers none of the three.

It interrupts callers, or leaves them waiting

An agent decides a caller has finished by waiting for a pause. Set that pause too short and it cuts people off mid-sentence. Set it too long and every reply carries dead air, and past about two seconds callers assume the line has dropped. The pause that means “I have finished” is not the same length in English as in Hindi, and a gap in a mobile network looks exactly like a pause to a rule that counts silence.

The fix is a threshold set per language against recordings of real calls, and a system that treats “no audio arrived” as a different fact from “the caller went quiet”. The three causes and the demo test for each are in why a voice AI agent interrupts callers; the mechanism is end-of-turn detection.

The question: is the pause threshold the same for every language? One number means the system was tuned on one language.

Callers cannot interrupt it, or a door slams and it stops

The opposite pair of failures. Many systems stop listening the moment they have handed the audio over, which is seconds before the caller has finished hearing it, so the agent can be interrupted at the start of a reply and not at the end. Others are tuned for a quiet room and stop mid-word whenever a bus passes, which on an Indian mobile call is most of the time.

The fix is to keep listening until the caller has finished hearing the reply, and to tell speech aimed at the agent from noise that happens to be nearby. Both tests take one minute in a demo and are set out in barge-in.

The question: can I interrupt it in the last two seconds of a long answer? Then, while it is talking, make a noise that is not speech.

It has one language setting and your customers use two

Most platforms model language as a setting: one per agent or per call. That is correct in a market where each caller speaks one language and wrong in one where a caller begins a sentence in Hindi, names a product in English and finishes in Hindi. Whichever value you pick, it is wrong for part of every call, and the expensive damage is in the timing rather than the transcript.

The fix is a system tuned for callers who mix Hindi and English inside a sentence, which is measurement work rather than a feature.

The question: show me one sentence with two languages in it, and give me the transcript. A language count is the tell.

A person is fetched only after the agent has failed

There are two ways to build a handover. As a fallback, the agent tries everything, fails, apologises and transfers, and your customer arrives at your colleague after ninety seconds of explaining something to a machine that was never going to help. As a rule, you decide in advance which calls a person takes, and the agent transfers those early, sometimes in the first sentence.

The fix is the second shape: a subject the agent never touches, a value above a threshold, an upset caller, a caller who asks. The rules and what your colleague should see on pickup are in human handover.

The question: who writes the rules for fetching a person, and what does my colleague see when the call arrives?

Nobody can show you the call that went wrong

Every vendor publishes a speed number. Almost none say what was measured, where the clock started, or whether it was a browser or a phone. An average across a thousand calls tells you nothing about the one where a customer hung up, and if the wait on each turn was not recorded at the time, it cannot be reconstructed afterwards.

The fix is to record how long the caller waited on every turn of every call, and keep it, so that “it felt slow” becomes a row you can look at. What the numbers mean and why published ones rarely compare is in response time in a voice agent.

The question: for real calls last week, what did the slowest one call in twenty look like? A vendor who records per turn answers in a minute. A vendor who cannot will offer an average.

It was pointed at the wrong first job

A pilot on a call type your team handles twice a week cannot show anything inside a fortnight. A pilot on a call type that takes a month of shadowing to learn will spend the fortnight discovering exceptions. The pilots that go well are almost always the boring ones: high volume, low variation, currently done badly because nobody has the hours.

The fix is to pick the call your team makes most often and dislikes most, agree what it costs you today, and measure the pilot against that number. The six jobs that fit, and where a person still belongs in each, are on the calls to hand over first.

The question: which call type do you recommend we start with, and what will we measure it against?

What to do with this list

Take the eight questions into every demo, and then stop trusting demos. Run the same two-week pilot with each shortlisted vendor on your own inbound line, on the same call type, with the same handover rules, so the comparison is between products rather than between sales decks. The fuller version of the questions, with the wrong answer to listen for on each, is in nine questions to ask a voice AI vendor.

We found every failure on this list on our own line before a customer did. That is the only reason we can write it down.