TTFT Chain: Queue to Prefill to First Token, vs Throughput

NET

QUEUE

PREFILL (KV cache)

FIRST TOKEN

stream / throughput →

TTFT = network + queue + prefill

throughput = tokens / sec after

TTFT is dominated by prefill: longer prompts cost more. Caching the prefix can drop it ~80%.