Voice Agent Latency and Cost: Four Stacks Compared

Voice agent comparisons usually quote vendor-published latency for a single component. That number does not survive contact with a real call, where speech-to-text, an LLM turn, tool calls and text-to-speech all sit in the path before the caller hears anything.

This piece frames four full stacks measured end to end on the same workload.

Read this first. Figures are illustrative placeholder data for editorial framing only. Replace with measured results from a controlled test run using the same network conditions, task script, and call volume.

---

The workload

A 1,000-minute mixed inbound and outbound support-call workload, run in the same network region, against the same evaluation script, using a production-shaped agent with speech-to-text, an LLM turn, tool calls and text-to-speech in the loop.

The measurement is time to first audio — the delay between the caller finishing their turn and hearing the agent begin to respond. That is what a caller experiences as lag, and it is the number that determines whether a conversation feels natural.

---

Results

Stack	Time to first audio (p50)	Time to first audio (p95)	Cost per minute at volume
A — Deepgram + GPT + ElevenLabs + Twilio	620 ms	1,120 ms	$0.14/min
B — Deepgram + Claude + ElevenLabs + Twilio	710 ms	1,280 ms	$0.16/min
C — Whisper RT + GPT Realtime + OpenAI TTS	540 ms	980 ms	$0.19/min
D — Open-source STT + local LLM + Cartesia + SIP	780 ms	1,410 ms	$0.09/min

Figure 1. In Ortem's illustrative voice-agent benchmark, the fastest stack reached 540 ms p50 to first audio, while the lowest-cost stack ran at $0.09 per minute at volume; the balanced enterprise stack delivered the best latency-cost tradeoff.

Figures are illustrative placeholder data for editorial framing only. Replace with measured results from a controlled test run using the same network conditions, task script, and call volume.

---

Where the stacks differ

Stack C is fastest, and most expensive. Keeping the realtime path inside one vendor removes hand-off overhead, which shows in both p50 and the tail. Volume pricing is what you trade for it.

Stack D is cheapest, with the weakest tail. At $0.09 per minute it is roughly half the cost of Stack C, but the 1,410 ms p95 means a meaningful share of turns feel slow. It also carries infrastructure management that the hosted options do not.

Stack A is the balanced enterprise choice. 620 ms p50 at $0.14 per minute, with a tail under 1.2 seconds.

Stack B trades latency for reasoning. 90 ms slower than Stack A at p50 for a different LLM in the loop — worth it when the workflow leans on tool use or multi-step reasoning, not worth it for scripted flows.

---

Reading p95, not just p50

The median is the number vendors quote and the tail is the number callers remember. Between Stack C and Stack D the p50 gap is 240 ms; the p95 gap is 430 ms. A stack that looks acceptable at the median can still produce a call where every fourth turn drags.

If you are choosing on latency, set a p95 budget first and filter to the stacks that clear it, then compare cost among the survivors.

---

Choosing on your own workload

Cost per minute scales with volume, so the ranking above shifts at different call loads. Latency is dominated by whichever component is slowest in your specific region — a stack that wins in one region can lose in another.

Both are reasons to measure your own path rather than adopt anyone's table, this one included.

---

Building a voice agent and need the latency budget and stack choice worked out against your actual call patterns? Ortem Technologies' AI and ML solutions practice designs and ships production voice systems. See our guide to the best AI voice agents for the vendor landscape, or book a technical consultation →.