Speculative Decoding: Hit Four Times, Miss and Lose

Status: Draft (2026-10-27), n-gram measured, EAGLE, MTP, and suffix unmeasured. Measured on 2026-10-27 using a single 8GB-class laptop GPU (Ada generation), vLLM 0.31.0, and Qwen3-Reranker-0.6B. Model-based methods (EAGLE, MTP, draft models) require downloading draft weights, and the suffix method requires downloading a new additional package, so they were not measured this time. Added on 2026-10-27. The results re-measured with the conversational Qwen3-0.6B and vLLM 0.32.0 have been added to Section 3-1. The figures for 0.31.0 and the reranker model remain unchanged. Writing Rules: Measured data is the default; unmeasured items are marked as "Unmeasured", and unverified items are marked as "Unverified".

Table of Contents

• 1. Principle: Guess and Verify at Once
• 2. Draft Methods in vLLM
• 3. Direct Measurement
  • Method
  • Result 1. Single User: Four to Six Times Faster When Copying
  • Result 2. Random Output: Varies by Prompt
  • Result 3. Increasing Concurrency Reduces Gains, and Random Output Causes Losses
  • Result 4. Output and Cost
• 3-1. Re-measurement on 2026-10-27: Conversational Model and 0.32.0
• 4. When to Enable
• 5. Unverified Items
• References

1. Principle: Guess and Verify at Once

A decode step spends most of its time reading weights from memory (Part 2). However, whether you check one token or five tokens in a single step, the amount of weight read is the same. Therefore, you can cheaply guessthe next few tokens, and have the large model validate the entire draft in a single step.

초안(싼 방법): " keeps a queue of"        ← 4토큰 제안
검증(큰 모델 한 걸음): " keeps" ✓ " a" ✓ " queue" ✓ " of" ✓ + 다음 토큰 하나
→ 한 걸음에 5토큰 확정

It stops at the first incorrect token, replacing that position with the token chosen by the large model. Thus, at least one token is always gained per step. In sampling (temperature > 0), rejection sampling preserves the same distribution as the large model alone, while in greedy decoding, the output must be identical. Leviathan et al. (ICML 2023) reported 2x to 3x speedups on T5-XXL without altering outputs, and Chen et al. (DeepMind, 2023) reported 2x to 2.5x speedups in a distributed environment on Chinchilla 70B (based on the abstract).

The gains are determined by two figures:

• Acceptance Length: How many tokens are confirmed on average per step. The better the draft matches, the longer this becomes.
• Cost of Verification Step: How much more expensive a step checking k drafts together is compared to a normal step. When there is remaining GPU compute (small batch size), it is almost the same, but if the batch size is large and the GPU is already busy, checking k additional tokens lengthens the step accordingly.

Therefore, the official documentation introduces speculative decoding as a feature to "reduce inter-token latency at medium-to-low request rates and memory-bound workloads."

2. Draft Methods in vLLM

Configurations are provided as a single JSON.

vllm serve <모델> --speculative-config '{"method":"ngram","num_speculative_tokens":4,"prompt_lookup_min":2,"prompt_lookup_max":5}'

The n-gram prompt_lookup_max·min is the length range of the token sequence to find; if both are omitted, it defaults to 5. In 0.32.0, Model Runner V2 introduced a length-varying GPU n-gram proposer. In 0.31.0, ngram_gpu was already an option, but in actual measurements, both methods issued a warning stating, "Model Runner V2 does not yet support this method, using the V1 executor." The CPU version ngram also issued a warning stating, "Disabling asynchronous scheduling."

Some features cannot be used together. The compatibility table in the 0.32.0 documentation notes that speculative decoding cannot be used with LoRA, pooling models, and prompt embedding inputs, and warnings were logged indicating that min_p and logit_bias are ignored.

3. Direct Measurement

Method

• Environment: A single 8GB-class laptop GPU (Ada generation), vLLM 0.31.0, Qwen3-Reranker-0.6B (bf16).
• Workload sets. All generate 256 tokens; ignore_eos.
  • Copying: Given a self-written English paragraph and a Python function (approx. 300 tokens), instructions are "Copy verbatim twice." Greedy decoding. This is a case where the output is almost identical to the input, making n-gram most effective.
  • Word chaining: Given 200 words formed by randomly chaining 30 words, prompt to continue writing. Greedy decoding. This model has a strong tendency to repeat phrases, often reusing input words in the output.
  • Random output: Given 200 tokens randomly sampled from the entire vocabulary, generate with temperature 1.0. This is the case expected to have the least overlap. Using the same seed ensures the same prompt is received even across different servers.
• Servers: Default (no speculation), n-gram proposals 2, 4, 8, and GPU n-gram proposal 4. Lookup length 2 to 5.
• Iterations: Four batches per condition (first batch for warm-up), three batches for high concurrency. Acceptance rates were calculated using the difference in vllm:spec_decode_num_accepted_tokens_total and ..._draft_tokens_total before and after the batch.

Result 1. Single User: Four to Six Times Faster When Copying

For k=4, the server was launched separately twice (692.6, 704.1 tok/s) and yielded nearly identical values. When increasing proposals to 8 in copying, an average of 7.7 out of 8 drafts matched, confirming about 8.7 tokens per step including the extra token appended by the large model. However, throughput was 6.3x instead of 8.7x. This means the step validating 9 tokens simultaneously was about 1.4x longer (7.8 ms) than a normal step (5.7 ms). Even with a single weight read, increasing the tokens to validate increases computation and attention.

"Word chaining" also being three times faster is because this model repeats identical words (e.g., signal river meadow signal river meadow …). General conversational models likely wouldn't match this.

Result 2. Random Output: Varies by Prompt

Random output at temperature 1.0 varies significantly by prompt, so results are listed per batch (concurrency 1, tok/s).

• Prompt 2 was a random prompt, yet the model quickly repeated identical phrases, yielding a 97% acceptance rate.
• Prompt 3 was slower than default across all k values (-13%, -10%, -18%). At k=4, only 22 out of 175 proposals matched. Unmatched drafts make the validation step heavier, and finding n-grams also takes time. The more proposals made (k=8), the greater the penalty.

Result 3. Increasing Concurrency Reduces Gains, and Random Output Causes Losses

As the batch size increases, the GPU is already busy, so steps lengthen proportionally to the increase in tokens to validate. Even for copying, which has a high acceptance rate, the gain dropped to 1.41x at concurrency 32, and random output, with an acceptance rate around 50%, suffered an 11% to 14% loss at concurrencies 8 and 32. The document's explanation that "gains diminish at high request rates" is borne out by the numbers.

Result 4. Output and Cost

• Greedy Output Equality: Generating 128 tokens each across 12 fixed prompts (6 copying, 6 word chaining) using greedy decoding showed that the n-gram k=4 server produced outputs identical down to the character for 10 prompts compared to the default server, with 2 word-chaining prompts differing. The documentation states that while speculative decoding is algorithmically lossless, floating-point discrepancies arising from the validation step computing multiple tokens at once can alter outputs. Empirical measurements matched this warning.
• KV Cache Size: On the speculative decoding server, the KV cache capacity decreased by 23% from 40,208 tokens to 30,832 tokens. This appears to be due to memory measurements changing alongside the same warning (reverting to V1 executor), and the cause is unverified. Servers with tight KV cache should check this line before enabling.
• Startup Time: The speculative decoding server took 78 and 96 seconds to prepare, taking longer than the default server with a matched compilation cache (approx. 32 seconds).
• Acceptance by Position: vllm:spec_decode_num_accepted_tokens_per_pos_total counts separately which token position in the draft matched. Across all records of the k=4 server, matches from the 1st to 4th positions were 84.2%, 79.4%, 77.0%, and 74.7%, confirming an average of 4.15 tokens per step. This curve, dropping towards the later positions, serves as the basis for choosing k.

3-1. Re-measurement on 2026-10-27: Conversational Model and 0.32.0

The previous measurements used a reranker model that tended to repeat phrases, favoring n-grams. Measurements were retaken using conversational Qwen3-0.6B (bf16) and vLLM 0.32.0 across four workloads.

• Copying: Same as before (raw prompt, greedy, fixed 256 tokens).
• Returning code: In a chat format, \
• Explain: \
• Random Output: Same as before (200 random tokens, temperature 1.0).

• For self-generated answers, n-grams barely matched. The acceptance rate for explanation was 3% and throughput decreased by 3%. The small loss is because many steps skip generating drafts when a matching n-gram cannot be found (1,188 proposals, much fewer than the random output of the reranking model).
• Return Code became three times faster. However, this 0.6B model failed to follow the instruction to change the name and returned the function as-is. Therefore, this value represents an \
• For greedy output, all 12 were identical in characters to the base server. In the 0.31.0 reranking model, it was 10 out of 12.
• In 0.32.0, KV memory usage hardly decreased. In 0.31.0, the KV memory of the speculative decoding server decreased by 23%, and there was a warning saying \

4. When to Enable

• Tasks where output echoes input(code modification, document correction, sentence quoting in summarization, repeated agent calls) yield significant gains with just n-grams. They require no preparation and can be turned on immediately.
• To reduce response latency in general conversation , model-based methods like EAGLE or MTP are appropriate. They require a draft head trained for that model (unmeasured).
• High-concurrency servers targeting throughputsee reduced gains, and a low acceptance rate leads to a loss. Throughput before and after enabling must be measured under actual load. Features that adjust proposal length based on load, like dynamic speculative decoding in 0.32.0, target this issue (unmeasured).
• If enabled, put vllm:spec_decode_num_accepted_tokens_total ÷ vllm:spec_decode_num_draft_tokens_total on the dashboard and tune k using position-based acceptance.

5. What Was Not Verified

• EAGLE, EAGLE-3, MTP, draft models, and suffix require downloading new draft weights or the arctic-inference package, so they remain unmeasured.
• The n-gram acceptance rate for general chat models was measured using only a single 0.6B model in Section 3-1. Code editing outputs that actually modify several lines are unmeasured.
• The cause of the 23% reduction in KV memory for the speculative decoding server in 0.31.0 is only presumed to be related to reverting to the V1 runner and was not confirmed (2% reduction in 0.32.0).

References

• vLLM Documentation, Speculative Decoding, N-Gram, Suffix Decoding, EAGLE (same for 0.31.0 and 0.32.0): https://docs.vllm.ai/en/latest/features/speculative_decoding/
• Leviathan et al., \
• Chen et al., \
• Li et al., \
• Li et al., \
• Oliaro et al., \
• Saxena, \