The 800ms Barrier: Profiling the Latency Chain of a Real-Time Gemini 3.1 Voice Agent

Sub-second Audio-to-Audio (A2A) latency for conversational AI has moved from a research constraint to a production requirement. The gap between "technically impressive" and "feels like talking to a human" is measured in milliseconds, not model quality.

In my latency audit of a real-time voice agent built on Gemini 3.1 Flash Lite, the total A2A loop ranged from 420ms to 1,800ms depending on transport protocol, audio chunking strategy, and network conditions. The 800ms barrier—the threshold where users perceive a conversational turn as "instant"—is achievable but requires optimizing every stage of the pipeline: Speech-to-Text (STT), LLM inference, Text-to-Speech (TTS), and the network transport layer between them. The bottleneck is never where you expect it to be.

Short answer: The LLM first-token latency (TTFT) is not the dominant factor in voice agent responsiveness. Audio chunking strategy and the STT-to-LLM handoff account for up to 40% of total A2A latency. Gemini 3.1 Flash Lite achieves ~320ms TTFT on a clean prompt, but the full A2A loop—including 120ms of audio buffering, 180ms of STT processing, and 150ms of TTS synthesis—pushes the perceived latency to 770ms minimum. WebRTC transport saves ~80ms over HTTP/2 streaming by eliminating connection setup overhead per chunk. On a 16GB M2 Air, thermal throttling after 8 minutes of continuous conversation adds 60ms to each component, pushing a previously stable 750ms loop past the 800ms perceptual barrier.

The architecture of the A2A latency chain

A real-time voice agent is not a single API call. It is a pipeline of four serial stages, each with its own latency profile, each adding buffer tax that compounds the total round-trip time.

Audio Input → VAD → STT → LLM → TTS → Audio Output

Voice Activity Detection (VAD) determines when the user has stopped speaking. STT converts the audio stream to text. The LLM generates a response. TTS converts that response back to audio. The user hears the result.

Every stage introduces at least one buffering decision: how much audio to accumulate before sending it, how many tokens to generate before starting TTS, how large each network packet should be. These decisions are not about model quality. They are about the engineering of time.

The latency chain is serial, not parallel. Each stage must complete—or at least reach a stable checkpoint—before the next can begin. If STT takes 180ms, LLM TTFT takes 320ms, and TTS takes 150ms, the theoretical minimum is 650ms. In practice, it is always higher because of the network and buffering overhead I profiled below.

Benchmark setup: MacBook Air M2 (16GB)

I built a Python voice agent using Gemini 3.1 Flash Lite via the Google AI SDK. The agent ran on a baseline 16GB M2 Air connected to a 500Mbps fiber connection (measured ping to Google Cloud: 12ms). I measured latency across 200 conversational turns with an automated script that played pre-recorded audio questions and timestamped each stage transition.

Metrics I tracked: 1. VAD detection latency: Time from audio end to turn-handoff signal. 2. STT processing time: Time from audio chunk receipt to final transcript. 3. LLM Time to First Token (TTFT): Time from prompt submission to first generated token. 4. LLM full generation time: Time from first token to final token. 5. TTS synthesis time: Time from text receipt to first audio byte. 6. Total A2A latency: Time from user speech end to first synthesized audio byte playback.

I tested two transport protocols: HTTP/2 streaming (the default SDK behavior) and WebRTC DataChannel (a custom implementation using aiortc). I also tested three audio chunking strategies: 200ms frames, 500ms frames, and VAD-triggered full utterance sends.

The VAD tax: when does the user actually stop speaking?

Voice Activity Detection is the unsung hero—and villain—of voice agent latency. A VAD model must decide whether a pause in speech is a mid-sentence hesitation or a genuine turn boundary. Get it wrong and you either interrupt the user (rude) or wait too long (slow).

I used Silero VAD v5, a lightweight ONNX model that runs locally. On the M2 Air, Silero processes audio in 32ms frames and makes a speech/non-speech decision in under 2ms per frame. The model itself is not the bottleneck. The patience threshold is.

I tested three patience settings: - Aggressive (200ms silence trigger): Average VAD latency of 210ms. False turn-end rate of 18% (the agent interrupts mid-sentence). - Balanced (500ms silence trigger): Average VAD latency of 510ms. False turn-end rate of 6%. - Conservative (800ms silence trigger): Average VAD latency of 810ms. False turn-end rate of 1%.

The balanced setting is the only one that works for production. But 510ms of VAD latency before the STT pipeline even starts means you have already consumed 510ms of your 800ms budget. This is the first reason sub-800ms A2A latency is so difficult to achieve.

STT processing: the chunking bottleneck

Once VAD triggers, the accumulated audio is sent to the STT engine. I tested two STT backends: Google's Cloud Speech-to-Text (via the Gemini SDK's built-in audio handling) and OpenAI's Whisper Large V3 (self-hosted via faster-whisper).

The chunking strategy matters enormously. Sending 200ms audio frames means the STT engine receives a new packet every 200ms and must maintain rolling state. Sending the full utterance on VAD trigger means one larger payload but no state management overhead.

Cloud STT adds a network round-trip per chunk. With 200ms frames, that is five round-trips per second of audio. Even at 12ms ping, the cumulative HTTP overhead adds up. Whisper local eliminates the network hop entirely but consumes ~400MB of RAM and triggers memory pressure on the M2 when running alongside the LLM.

The VAD-triggered full utterance strategy is the fastest because it makes exactly one STT call. But it requires VAD to be accurate—if VAD fires too early, the STT receives an incomplete sentence and produces a partial transcript. The LLM then responds to a truncated query.

The LLM TTFT: Gemini 3.1 Flash Lite under the microscope

Gemini 3.1 Flash Lite is Google's answer to low-latency conversational AI. The marketing claims ~200ms TTFT. My benchmarks tell a more nuanced story.

I measured TTFT across three scenarios: 1. Clean prompt: A fresh conversation with no prior history. TTFT: 320ms (p50), 480ms (p95). 2. Short context (5 turns): TTFT: 340ms (p50), 510ms (p95). 3. Long context (50 turns, ~15k tokens): TTFT: 520ms (p50), 780ms (p95).

The "200ms" claim is achievable only for trivially short prompts with warm model instances. In a real conversational agent where the prompt grows with each turn, TTFT degrades linearly with context length.

This is the same context window problem I covered in my KV cache eviction benchmark. Gemini's attention mechanism must process every token in the prompt before generating the first response token. A 50-turn conversation with 15k tokens requires the model to attend over 15,000 key-value pairs before producing a single output token.

The mitigation is context pruning. If I evict the oldest 30% of the conversation history (keeping only the first 4 tokens as attention sinks and the most recent 10 turns), TTFT drops from 520ms back to 380ms. The accuracy cost is measurable but acceptable for casual conversation. For technical queries where earlier context matters, pruning causes the model to lose track of constraints.

The TTS synthesis delay

Text-to-Speech is the final stage. I tested two TTS engines: Google's Cloud TTS (via the Gemini SDK) and OpenAI's tts-1-hd API. Both operate on a streaming basis—they begin emitting audio bytes as soon as the first sentence is generated.

The critical metric here is time to first audio byte (TTFAB). This is not the time to synthesize the full response. It is the time from text receipt to the first playable audio sample.

Google's TTS is faster because it uses a lighter phoneme-to-waveform pipeline. OpenAI's tts-1-hd produces more natural prosody but adds ~60ms of preprocessing overhead.

The TTS engine also needs a minimum text buffer before it can start synthesizing. Google Cloud TTS requires at least one complete sentence (roughly 15-20 tokens). This means the LLM must generate 15-20 tokens before TTS can begin. At Gemini 3.1 Flash Lite's generation speed of ~80 tokens/second, that is an additional 190ms of delay before TTFAB even starts.

This is the hidden coupling between LLM generation and TTS synthesis. They are not truly parallel. TTS is blocked on LLM producing a complete linguistic unit.

The 800ms barrier: a full budget breakdown

Here is the complete latency budget for the best-case configuration (VAD balanced, VAD-triggered full utterance, Gemini Flash Lite with short context, Google Cloud TTS, WebRTC transport):

Even in the best-case configuration, the total is 1,415ms. That is nearly double the 800ms target. The VAD patience alone consumes 510ms.

To reach 800ms, you must make tradeoffs that compromise the user experience:

• Reduce VAD patience to 200ms: Saves 300ms but adds 18% false turn-end rate. The agent interrupts the user.
• Use speculative TTS: Start synthesizing audio based on partial LLM output. Saves ~120ms but risks generating audio for incomplete sentences that get abandoned if the LLM changes direction mid-generation.
• Context pruning: Keep context under 5k tokens. Saves ~200ms on TTFT but loses conversational memory.

The "sub-800ms voice agent" is achievable, but only by accepting one or more of these compromises.

Speculative TTS: the risky shortcut

Speculative TTS is the voice agent equivalent of speculative decoding—you start working before you have the full answer. Instead of waiting for the LLM to produce a complete sentence, the TTS engine begins synthesizing audio as soon as the first few tokens arrive.

The risk is non-monotonic generation. LLMs do not always produce linear text. They might generate "The best approach is to—" and then backtrack with "Actually, let me reconsider." If TTS has already synthesized audio for the first clause, you are now playing audio that contradicts the model's final answer.

I measured the speculative TTS failure rate across 500 conversational turns: - Sentence-level speculation (wait for period/terminal punctuation): 3% backtrack rate. - Clause-level speculation (wait for comma): 12% backtrack rate. - Token-level speculation (start after 5 tokens): 28% backtrack rate.

Sentence-level speculation is the only strategy that works in production. It saves ~80ms on average but requires the TTS engine to discard partially synthesized audio when the LLM backtracks. The audio glitch is perceptible—a 50ms silence followed by the corrected audio—but most users tolerate it once per conversation.

WebRTC vs. HTTP/2: the transport layer tax

Most voice agent tutorials use HTTP/2 streaming because it is the default in every SDK. HTTP/2 multiplexes streams over a single TCP connection, which is efficient for high-throughput data but suboptimal for low-latency audio.

The problem is TCP head-of-line blocking. If one audio chunk is lost and requires retransmission, all subsequent chunks are delayed until the retransmission completes. In a real-time voice pipeline, a lost packet adds 200-500ms of stall time—far worse than the 12ms of network latency.

WebRTC uses UDP with a custom reliability layer (SCTP over DTLS). Lost audio packets are simply skipped rather than retransmitted. The audio stream continues with a minor glitch rather than a full stall.

In my benchmark across 200 turns with artificially induced 2% packet loss: - HTTP/2 streaming: p50 latency of 890ms, p95 latency of 1,650ms (spikes from retransmission). - WebRTC DataChannel: p50 latency of 810ms, p95 latency of 920ms (minor glitches but no stalls).

WebRTC saves ~80ms on p50 latency and ~730ms on p95 latency. The p95 improvement is the real win—it eliminates the catastrophic latency spikes that make a voice agent feel "unresponsive" even when the average is fine.

The downside is implementation complexity. WebRTC requires a signaling server, ICE candidate negotiation, and DTLS certificate management. It adds roughly 200 lines of infrastructure code compared to a simple HTTP POST.

The silence filler problem: what plays while you wait?

A 1,400ms response time means 1.4 seconds of silence after the user stops speaking. Human conversation has a median turn-taking gap of 200-300ms. A 1,400ms silence feels like the agent is ignoring you.

Production voice agents use "filler utterances" to bridge the gap: "Hmm," "Let me think," "One second." These fillers serve two purposes: they signal that the agent is still processing, and they buy time for the pipeline to complete.

I tested three filler strategies: - Fixed filler (always "Hmm"): Predictable but robotic. Users reported it felt "fake" after 2-3 turns. - Adaptive filler (chooses from a set based on context): Better UX. "Let me check" for factual queries, "Interesting" for opinions. - No filler: Users interrupted the agent 40% more often, perceiving the silence as a system failure.

The adaptive filler strategy is the only one that maintains a natural conversational rhythm. But it adds ~50ms of TTS overhead to synthesize the filler itself. This is a net positive because it reduces the perceived latency even as it increases the measured latency.

Thermal throttling on the M2 Air: the slow death

Running a voice agent on a fanless MacBook Air reveals a constraint that cloud benchmarks ignore: thermal throttling. The M2 Air has no active cooling. After 8 minutes of continuous conversation (approximately 40 turns), the SoC temperature hits the thermal limit and the clock speed drops from 3.5GHz to 2.8GHz.

The impact on latency is gradual but measurable: - Turn 1-10: VAD at 32ms/frame, total A2A at 1,415ms. - Turn 20-30: VAD at 38ms/frame, total A2A at 1,480ms. - Turn 40-50: VAD at 45ms/frame, total A2A at 1,540ms. - Turn 60+: VAD at 52ms/frame, total A2A at 1,610ms.

The local components (VAD, local STT, context management) are the most affected by throttling. The cloud components (LLM, Cloud TTS) are unaffected because they run on Google's hardware. But the total loop still degrades because the local stages are on the critical path.

For production deployments, this means a local-first voice agent on consumer hardware will gradually slow down during long sessions. The mitigation is a cooling pad or a device with active cooling (MacBook Pro, Mac Mini). On an M2 Pro with a fan, I observed zero throttling even after 60 minutes of continuous conversation.

The context window memory tax

As I documented in my open source AI memory review, raw context windows are a poor substitute for structured memory. The same principle applies to voice agents. A 50-turn conversation with full history forces the LLM to process 15,000+ tokens of prior conversation before generating each response.

The latency impact is direct. Gemini 3.1 Flash Lite's TTFT scales from 320ms (clean) to 520ms (15k tokens). That is 200ms of additional silence after every user turn, purely because the model is re-reading everything you have already said.

Implementing a context quarantine—keeping only the first 4 tokens (attention sinks), the most recent 10 turns, and a semantic summary of older turns—reduces the effective context to ~5k tokens. TTFT drops to 380ms. The 140ms savings is meaningful but comes at the cost of long-term consistency. If the user says "remember I mentioned my budget is $50" at turn 3 and asks about it at turn 40, the pruned context will have lost that detail.

The economics of sub-800ms voice AI

Every optimization has a cost. Here is the tradeoff matrix I built for reaching the 800ms target:

The only combination that reaches sub-800ms is: aggressive VAD (200ms) + context pruning (5k tokens) + speculative TTS. This saves 520ms from the baseline 1,415ms, bringing it to 895ms. Adding the adaptive filler reduces perceived latency to approximately 700ms.

The cost: the agent interrupts the user 18% of the time, forgets details from earlier in the conversation, and occasionally generates backtracked audio. Whether this is acceptable depends on your use case. A customer support bot can tolerate these tradeoffs. A medical consultation assistant cannot.

Case study: the failure modes

Not everything worked. Here is what broke during testing:

WebSocket connection drops: The Gemini SDK's streaming connection timed out after 5 minutes of inactivity. The agent silently stopped responding. Fix: implement a heartbeat ping every 30 seconds.

STT hallucination on silence: Whisper Local V3 occasionally transcribed background noise (fan hum, keyboard clicks) as speech tokens. The LLM received phantom input and generated nonsensical responses. Fix: add a pre-STT noise gate that discards audio below -40dB RMS.

TTS buffer underrun: When the LLM generated extremely short responses ("Yes.", "No."), the TTS engine received fewer tokens than its minimum buffer requirement and stalled for 200ms waiting for more. Fix: pad short responses with a filler token ("Yes, I agree.").

Concurrent turn overlap: If the user started speaking again before the agent finished its TTS playback, the VAD triggered a new turn while the previous TTS was still playing. The agent talked over itself. Fix: implement a turn-lock that disables VAD during TTS playback.

Practitioner's checklist: auditing your voice agent latency

1. Profile every stage independently: Do not measure total latency alone. Instrument STT, LLM TTFT, TTS TTFAB, and network handoffs separately.
2. Optimize VAD patience first: It is the largest single contributor to total latency. Find the minimum patience that keeps false turn rate under 10%.
3. Use VAD-triggered full utterance sends: Avoid per-frame STT calls unless your use case requires real-time transcription (live captioning).
4. Prune context aggressively: Keep the effective prompt under 10k tokens. Summarize older turns rather than sending them raw.
5. Implement speculative TTS at sentence granularity: Never speculate at the token or clause level.
6. Use WebRTC for production: The implementation complexity is justified by the p95 latency improvement.
7. Add a heartbeat: Streaming connections die. Detect and reconnect before the user notices.

The future of real-time voice AI

The 800ms barrier is not a model problem. It is a systems engineering problem. Gemini 3.1 Flash Lite, GPT-4o Mini, and Claude Haiku all have TTFT in the 200-400ms range. The difference between a "fast" and "slow" voice agent is not the LLM—it is the VAD configuration, the chunking strategy, the transport protocol, and the TTS pipeline.

The next frontier is full-duplex voice: the ability for the agent to listen while it speaks, detecting user interruptions and backing off mid-sentence. This requires a fundamentally different architecture where STT, LLM, and TTS run in parallel rather than serially. Google's Gemini Live API already supports this at the model level, but the latency budget for full-duplex is even tighter because the agent must react to interruptions in under 300ms.

I covered the broader KV cache management problem that affects all LLM-based systems. Voice agents are simply the most latency-sensitive application of these infrastructure constraints. Every millisecond saved in the context window is a millisecond closer to a conversation that feels human.

Teams building voice-first products need engineers who can translate these latency constraints into clear architectural documentation and implementation guides. My work page has examples of how I've turned complex inference infrastructure into high-signal content for DevTools companies.

FAQ

What is the single biggest latency contributor in a voice agent? VAD patience. A balanced 500ms silence trigger accounts for 36% of total A2A latency. Reducing it saves the most time but increases the false turn-end rate. There is no free optimization here.

Is Gemini 3.1 Flash Lite faster than GPT-4o Mini for voice? In my benchmarks, yes—marginally. Gemini 3.1 Flash Lite achieved 320ms TTFT vs. GPT-4o Mini's 380ms on identical prompts. The difference is more pronounced on long context, where Gemini's TTFT scaling is flatter.

Can I run the entire voice agent pipeline locally? Partially. VAD (Silero) and STT (Whisper) run locally. TTS can run locally with Piper or Coqui. The LLM is the bottleneck—running a model small enough for local inference (7B parameters) produces significantly worse conversational quality. A hybrid approach (local VAD+STT, cloud LLM+TTS) is the current sweet spot.

How does packet loss affect voice agent latency? On HTTP/2 streaming, a 2% packet loss rate causes p95 latency to spike from 890ms to 1,650ms due to TCP retransmission. WebRTC handles the same loss with p95 at 920ms because it skips lost packets rather than blocking.

What is the "context quarantine" pattern? It is a context pruning strategy that isolates the first 4 tokens (attention sinks), the most recent N turns (the active conversation), and a compressed semantic summary of older turns. It prevents the prompt from growing indefinitely while preserving the model's numerical stability.

Should I use filler utterances? Yes. A 1,400ms silence is perceived as unresponsiveness. A 200ms "Hmm, let me check" bridges the gap and signals active processing. The ~50ms TTS cost is negligible compared to the UX improvement.