What four phone calls taught us that the tests did not

The number

Our first complete production phone call ran 112 seconds over 16 turns and ended cleanly. Getting there took four attempts.

The first three each found a defect. Not one of those defects was caught by the test suite, and the suite was not thin. Every one of them had a passing test that asserted the opposite of what the phone line did.

That is the whole post, and it is the reason this company argues what it argues on the rest of this site. The wider list of why voice AI agents fail on real phone calls starts here.

Why the tests agreed

A test is written by the same person, from the same understanding, as the code it covers. When the misunderstanding is about the outside world, the test inherits it and then confirms it. The suite goes green, and the green means the code matches the assumption rather than that the assumption matches reality.

Voice telephony is unusually rich in assumptions that feel obviously true and are not:

• That when the speech vendor says it has finished delivering audio, the caller has finished hearing it. Those are different moments, seconds apart, and the difference decides whether callers can interrupt the agent during the second half of a sentence.
• That a pause long enough to mean “I have finished speaking” is one number. That decision, end-of-turn detection, needs at minimum one number per language, which is why callers who mix Hindi and English break systems tuned on one of them.
• That audio arrives in the order and at the rate it was sent. On a mobile network it does neither.

Every one of those is invisible in a browser tab. A laptop microphone and a local connection make all three assumptions true. A carrier makes none of them true.

What we changed as a result

Two things, and both are now visible to customers.

We record the timing per turn, on every call, and keep it. How long the caller sat in silence before hearing an answer is the number a voice product is judged on, and it cannot be worked out after the fact. Storing it only when someone complains is storing it too late. This is also the number we suggest asking any vendor for, in the piece on questions to ask a voice AI vendor.

We treat production as the test. Assertions that can only be checked on a live call run on live calls, in production, on real traffic. The first of them fired on the first real phone call and named the exact problem. A scheduled probe samples; production sees everything.

What this means if you are buying

Three things follow, and they are worth more than any feature list.

A demo tells you the vendor can hold a conversation, not that they can hold a phone call. Those are different products. Ask what has gone wrong on a real call.

Ask for the evidence per call, not the average. A vendor who records what each caller experienced can show you the bad call. A vendor who reports one figure across a thousand calls has no way to explain the one that lost you a customer.

Run the pilot on your own line. Two weeks on one call type, on real callers, measured against what that job costs you today. The vendor whose system survives that is the one to sign with, and it will not always be the one whose demo was smoothest. There is a short guide to picking which call type to start with.