Carryover Drafting: Recycling Rejected States for Speculative Decoding
Jahyun Koo, Sunghyeon Woo, Jaeeun Kil, Jeongtae Lee, Sungjae Lee, Kyomin Jung, Minsub Kim
cs.LG · cs.AI
Quick verdict
Recycles rejected verifier hidden states as one-round drafter KV; +6.5–14.7% acceptance, +7.9–14.4% speedup; credible, but epoch-fairness and no-code caveats.
Summary
The paper observes that speculative-decoding verification already computes target hidden states for rejected draft tokens, which block-diffusion drafters such as DFlash discard. Carryover Drafting recycles these rejected hidden states as temporary KV context for the drafter's next proposal: states pass through the existing projection interface, a single learned embedding marks them as rejected, and the carried context is replaced each round, bounding overhead at one proposal block (15 tokens). Since standard parallel drafter training never produces inference-time rejected states, the authors add parallel draft–verify–draft training: detached proposals from sampled anchors are verified by a frozen target in parallel, yielding inference-aligned rejected states, with gradients only through the second drafting pass. Across Qwen3-4B and Gemma 4 12B targets, DFlash and a DSpark-derived Markov drafter, and eight datasets with confidence scheduling disabled, average acceptance length improves 6.5–14.7% and end-to-end vLLM speedup 7.9–14.4%, peaking at +28.8% on translation (Qwen3-4B Markov, T=1); relative gains persist at concurrency 1–32. Matched-anchor recall and re-convergence analyses support the mechanism. Limitations: Carryover trains 3 extra epochs beyond the 10-epoch baseline with no continued-baseline control; main tables report no confidence intervals (844 prompts, 80 per dataset); drafting overhead is only inferred from net speedup; no code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Rejected target hidden states computed during verification carry predictive information about future tokens that improves subsequent drafts.
- Adding rejected states through the existing KV interface with a single learned embedding improves average acceptance length by 6.5–14.7% across two targets, two drafters, and eight datasets.
- The acceptance gains translate into end-to-end vLLM speedup gains of 7.9–14.4% over the corresponding baselines, peaking at +28.8% on translation, and persist at concurrency 1–32.
- Parallel draft–verify–draft training exposes the drafter to inference-aligned rejected states while preserving parallelism across training positions, using only recorded target responses for supervision.
- Carryover gains correlate with re-convergence of rejected states with the eventual continuation (r=0.96 across 8 datasets), suggesting stronger drafters leave more useful rejected states.
Strengths
Minimal, well-bounded mechanism
Algorithm 1 differs from the baseline only in retaining proposal rows m+1:B, adding one learned embedding Ecarry, and exposing them as read-only KV replaced each round; added context is bounded by B=15 rows per drafter layer, and no gate or extra projector sits on the drafting path.
Elegant training solution to a real mismatch
Parallel draft–verify–draft training (Section 2.2, Algorithm 2) obtains inference-aligned rejected states via a detached first draft and frozen-target parallel verification under a block-diagonal causal mask, keeping anchors independent and gradients only in the second pass — no sequential rollouts, no synthetic continuations.
Unusually thorough end-to-end evaluation
Diagnostic depth beyond headline numbers
Honest scope and correctness argument
Weaknesses
Training-budget asymmetry without control
No uncertainty quantification on main results
Overhead never measured in isolation
Reproducibility gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Carryover Drafting (arXiv 2609.14717, ICLR 2027 submission; Seoul National University + NAVER Cloud) targets LLM inference acceleration. Its premise: speculative-decoding verification already computes target hidden states at rejected draft positions, and block-diffusion drafters (DFlash, DSpark-style) discard them. The method feeds these rejected states back as temporary KV context for the next draft through the drafter's existing projection interface, tagged by a single learned embedding (Ecarry), with the carried block replaced every round so overhead is bounded by one proposal block (B = 15). Central claims: (1) average acceptance length improves 6.5–14.7% over matched baselines across two targets (Qwen3-4B, Gemma 4 12B) and two drafters (parallel DFlash; Markov, a DSpark-derived rank-256 semi-autoregressive variant) on eight datasets; (2) end-to-end vLLM speedup improves 7.9–14.4%, peaking at +28.8% (Qwen3-4B Markov, translation, T=1: 1.94×→2.50×); (3) a parallel draft–verify–draft training scheme supplies inference-aligned rejected states without sequential rollouts.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Carryover Drafting in speculative decoding?
It is a method that recycles the target model's hidden states computed on rejected draft tokens as temporary KV context for the drafter's next proposal. A single learned embedding marks these states as rejected, and the carried context is replaced every round, so the extra memory is bounded by one 15-token proposal block. The paper reports average acceptance-length gains of 6.5–14.7% and vLLM speedup gains of 7.9–14.4% across two target LLMs and two drafters.
How much faster is LLM inference with Carryover Drafting?
In the paper's vLLM measurements on a single H200 GPU, end-to-end speedup over autoregressive decoding improves by 7.9–14.4% relative to the corresponding baseline drafter, with the largest gain (+28.8%) on translation using Qwen3-4B with a Markov drafter at temperature 1 (1.94× to 2.50×). Relative gains persist at serving concurrency from 1 to 32.
Why do rejected draft tokens contain useful information?
When the target model verifies a draft, it computes hidden states for every proposed position, including rejected ones. These states encode the target's internal view of the rejected continuation, which often partially agrees with the text eventually generated (the paper calls this re-convergence). The drafter can attend to these states as extra context to improve its next proposals.
Is code available for Carryover Drafting?
The paper text does not state a code or model release. Reproduction relies on the described algorithms and public components (UltraChat and Magpie prompts, the Red Hat speculator benchmark, vLLM), but retraining the drafters requires GPU-scale compute and target-regenerated training responses.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as extracted text; Figure 4 recall curves, Figure 5 scatter fits, and Figure 6 attention heatmaps were assessed from captions and in-text numbers only.
- The abstract in the source was truncated mid-sentence, but the body and appendices contain the complete claims and algorithms.
- 2026-dated citations (DFlash, DSpark, DFlash 2, Lyanna, Draft-OPD, etc.) could not be verified against their sources.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.