Where the time goes in a voice agent’s reply
Our agents answer in about 1.2 seconds, the median across browser sessions. That number is the sum of several smaller ones, and a few of them were not doing anything useful. This is what we measured, what we changed, and what we chose not to do.
Every reply passes through three stages: speech recognition hears the caller, a language model writes the answer, and a voice speaks it. We record the latency of every turn, so the numbers below come from real sessions and from benchmarks we wrote for the purpose. They are medians from small samples, measured on our own setup, so read them as directions rather than laws.
A 519 ms wait that was doing nothing
We first compared two speech setups: a newer one, where the speech model itself decides when the caller has finished, and an older one. The newer setup looked slower, 1157 ms against 654 ms per turn. It was also far better at not interrupting people: 0 false interruptions in 7 tries, against 4 in 10.
Most of that gap turned out to be a wait that should not have been there. The turn closes when two things have arrived: the caller’s final transcript, and the signal that they have stopped speaking. The speech model sends the transcript first and the stop signal second. But inside our pipeline the stop signal travels on a faster lane, so it overtook the transcript queued behind it, found no words yet, and did nothing. The turn then sat out its full timeout with the answer ready to be asked for.
Measured on its own, that wait was 519 ms on every turn. We now close the turn when the transcript lands, and the same measurement reads 13 ms. A test measures both the old and the new behaviour, so if anything underneath us changes, we will know.
The lesson we took: every component passed its own unit tests. The bug lived in the seam between two of them, and only showed up when we built the whole pipeline the way production builds it and timed it.
Not interrupting beats raw speed
We kept the newer setup. A reply that arrives a little later is a small cost. An agent that talks over someone who paused to think is a call that goes wrong. People pause in the middle of sentences, read out numbers slowly and trail off, and a fixed silence timer either cuts them off or makes every exchange drag.
Reasoning tokens are dead air
Some models think before they answer, and on a phone call that thinking is silence: the caller hears nothing until the first word of the reply. On one model we tested with a real agent prompt, four prompts per setting, the median time to the first token was:
| Reasoning | Median time to first token |
|---|---|
| On, the model’s default | 4558 ms |
| Minimal effort | 2448 ms |
| Off | 1972 ms |
Off more than halves the pause, so it is our default, and on the model we run today it is worth about 475 ms a turn. Some newer models refuse to switch reasoning off at all, and we test for that before offering one.
What a router costs
We call models through a router, which lets us switch providers and fail over. It is also a hop on every turn. Measured from a laptop against the router’s own reported upstream timing, the router and the network to it added roughly 250 to 480 ms, depending on prompt size. Some of that is the distance from the machine running the test. We keep the router for now, because the flexibility is worth more than a few hundred milliseconds until latency is the binding constraint, and going direct to a provider is a configuration change when it is.
The greeting skips the model
The first call to a model in a session is slower than the rest. In one measurement it took 2.16 s cold against 0.52 s mid-call. So an agent with a written greeting speaks it straight away, without asking the model for it, and the caller hears a voice the moment the call connects.
What we do not do: fake it with “um”
A common suggestion is to add filler, an “um” or a “gotcha”, to mask the delay. We decided against it. Filler is a disguise for the delay, not a fix for it, and we had already spent time removing the habits that make agents sound like software, like “I can certainly help you with that”. Putting scripted hesitation back would keep the same delay and bring the tic back with it. The work goes into the number instead.
Where it stands
About 1.2 s median per reply in the browser, with turn-taking we trust more than raw speed. The parts left are mostly the model’s own time to first token and the network between us and it. If you build voice agents and have seen turns that stall for no visible reason, check how your turn closes: ours was waiting on itself.