Voice Agent Latency and Cost: What Four AI Voice Stacks Delivered

For voice AI, latency is not a dashboard vanity metric. It determines whether a conversation feels natural or forces callers to wait, interrupt, repeat themselves, or request a human agent.

The useful measure is not simply a vendor's claimed speech-to-text or text-to-speech speed. Teams need to measure the complete chain: telephony transport, speech recognition, turn detection, model inference, API or CRM lookup, speech synthesis, and audio delivery back to the caller.

Read this first. Figures are illustrative placeholder data for editorial framing only. Replace with measured results from a controlled test run using the same network conditions, task script, and call volume.

Short answer: In this illustrative test, the fastest stack reached 540 ms p50 time to first audio, while the lowest-cost stack operated at $0.09 per minute. The strongest enterprise balance came from a modular Deepgram, GPT, ElevenLabs, and Twilio architecture at $0.14 per minute.

Ortem Technologies builds AI solutions, cloud systems, backend platforms, and customer-facing digital products. Our ClearVoice case study describes a financial-services voice AI agent using Twilio, Deepgram, ElevenLabs, Salesforce, and a core-banking API integration.

---

Test conditions

The benchmark used four representative production-style stacks against one fixed script: caller authentication, balance or policy lookup, FAQ response, appointment scheduling, and a handoff scenario.

Control	Test condition
Call volume	1,000 combined inbound and outbound minutes
Network	Same telephony region and stable business-grade network path
Script	Same caller utterances, pauses, interruptions, API lookup sequence, and escalation rule
Measurement	Time from caller end-of-turn to first audible agent response
Cost	Blended telephony, STT, LLM, TTS, and infrastructure cost per call minute
Exclusions	One-time build, integration, and optimisation costs are reported separately

Both p50 and p95 should be published. P50 shows the typical caller experience; p95 shows the slow tail that can damage perceived quality during peak load, retries, or slow external API calls.

---

Voice agent latency and cost

Stack	Core components	Time to first audio p50	Time to first audio p95	Blended cost at volume
Stack A	Deepgram + GPT + ElevenLabs + Twilio	620 ms	1,120 ms	$0.14/min
Stack B	Deepgram + Claude + ElevenLabs + Twilio	710 ms	1,280 ms	$0.16/min
Stack C	Whisper RT + GPT Realtime + OpenAI TTS	540 ms	980 ms	$0.19/min
Stack D	Open-source STT + local LLM + Cartesia + SIP	780 ms	1,410 ms	$0.09/min

Figure 1. Illustrative controlled benchmark: Stack C produced the fastest median first-audio response, while Stack D produced the lowest per-minute operating cost. Stack A offered the most balanced latency-cost profile.

Figures are illustrative placeholder data for editorial framing only. Replace with measured results from a controlled test run using the same network conditions, task script, and call volume.

---

How to interpret the results

Stack A: balanced enterprise deployment. At 620 ms p50 and $0.14 per minute, Stack A is appropriate for customer support, booking, account enquiries, and workflows requiring external data lookups. It avoids over-optimising for a headline latency number at the expense of reliability, observability, and integration flexibility.

Stack B: higher reasoning allowance. This design incurred a slightly slower response profile because model and tool-routing decisions were more involved. It is more suitable where the agent must interpret policy language, apply complex decision rules, or make multi-step backend requests.

Stack C: speed-first conversational experience. This stack gave the fastest p50 and p95 result. The trade-off is higher per-minute spend, making it better suited to premium voice experiences, short calls, high-value conversion flows, or applications where responsiveness directly affects completion.

Stack D: cost-optimised architecture. Running more components under direct infrastructure control reduced call-minute cost. However, it produced the slowest p95 tail and carried a greater platform-management burden: GPU capacity, model updates, observability, failover, and incident response.

---

The cost formula to publish

Use a transparent blended calculation: cost per minute equals telephony plus speech-to-text plus LLM plus text-to-speech plus infrastructure.

For a 10,000-minute monthly programme, the illustrative operating cost would range from $900 per month for Stack D to $1,900 per month for Stack C. This comparison excludes implementation, integration, security, monitoring, and human-supervision costs.

---

How to improve voice-agent latency

• Stream audio input and speech output rather than waiting for a full response.
• Place voice, model, and application infrastructure in compatible regions.
• Keep system prompts, retrieval context, and tool descriptions concise.
• Return a short acknowledgement while a longer backend request completes.
• Cache predictable answers and session information where appropriate.
• Instrument each stage separately: STT, LLM first token, tool calls, TTS first audio, and telephony transport.
• Design a clean human handoff for complex or low-confidence scenarios.

---

Building a voice agent and need the latency budget and stack choice worked out against your actual call patterns? Ortem Technologies' AI and ML solutions practice designs and ships production voice systems. See our guide to the best AI voice agents, or book a technical consultation →.