Drafting turns one-token steps into candidate runs

Several research groups developed speculative decoding methods around 2022 and 2023 to reduce the sequential cost of autoregressive generation. A small model or another draft mechanism proposes a run of tokens.

The target model evaluates that run in one pass, accepts a valid prefix under the decoding rule, and resumes from the first rejection.

The method can be lossless with respect to the target distribution when the acceptance and correction procedure is implemented correctly. A plain small-model fallback does not provide that guarantee.

DRAFT AND VERIFYThe target model verifies a proposed token run in parallelAcceptance advances the sequence; rejection returns control to the target.
A faster process drafts several tokensThe draft is a proposal, not final output.
target accepts prefix
Commit accepted tokensAdvance by more than one sequential target step.
target rejects token
Stop at the first rejectionDiscard the unsupported remainder of the draft.
sequence continues
Draft the next runRepeat from the updated prefix.
correction required
Use the target-model tokenResume drafting from the corrected prefix.

The verification rule preserves the target distribution. A faster draft alone does not.

Acceptance rate determines whether the draft helps

A draft model that predicts common boilerplate well may advance several tokens per target-model verification. If the domain, language, or style makes its proposals diverge, frequent rejection can erase the benefit.

Quantization reduces numeric precision, batching combines requests, and test-time compute spends more work to improve an answer. Speculative decoding instead tries to produce target-model tokens with fewer sequential target passes.

The benchmark needs the whole serving context

The speculative decoding guide explains the mechanism and evaluation variables. The technique is a poor fit when the target model is already small or verification overhead dominates.

The common misuse is publishing one speedup without the draft and target models, hardware, batch size, prompt and output lengths, acceptance behavior, and decoding settings.

Serving load can change the result

A technique that improves single-request latency may compete for compute that a busy server could use for batching other requests. Draft-model memory also reduces the capacity available to the target model and its KV cache.

The relevant evaluation matches the production objective: time to first token, time per output token, total request latency, or throughput under load. “Faster” is incomplete until the metric and workload are named.

Continue with these glossary entries: