Predict, Don't Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Models
Haobo Xu, Sirui Chen, Yuanchen Bei, Lingjie Chen, Yuchen Yan, Dongqi Fu, Jingrui He, Hanghang Tong
cs.CL · cs.AI
Quick verdict
PILL predicts DLM infill length via a probe + parallel decoding, beating CAL (+4.8 code pass rate, 1.82x faster); convincing but single-run, no variance stats.
Summary
Diffusion language models (DLMs) must fix the generation length before decoding, making infilling highly length-sensitive. The paper proposes PILL, a backbone-frozen pipeline with three stages: (1) a 3-layer MLP probe predicts the target length from bidirectional hidden states (mean-pooled last-4 prefix tokens, mask token, mean-pooled first-4 suffix tokens) in one forward pass; (2) five candidate lengths (radius r=2) are decoded in parallel via a slot-wise attention mask and a position-ID interpolation scheme that prevents under-generation; (3) a single post-hoc scoring pass approximates pseudo-log-likelihood with causally-masked probes, combining inner-coherence and suffix-continuation scores (alpha=0.5). Across five DLMs (LLaDA-8B Base/Instruct, LLaDA-MoE, Dream-7B, Dream-Coder-7B) and eight code and text benchmarks, PILL beats the strongest baseline CAL by +4.8 average pass rate on code and +6.0 BLEU-2 on text while running 1.82x faster (Table 2: 71.35 vs 64.74 pass@1 on HumanEval-S at 1.18s vs 2.13s). Ablations support each stage (Tables 3-5); notably, oracle-best-initialized CAL still beats PILL on the two HumanEval tasks (Table 16). Limitations: single-run deterministic decoding with no variance or significance testing; the probe needs roughly 196k supervised samples per backbone; evaluation covers models up to 8B and focuses on single-span infilling. Code is released on GitHub.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A lightweight MLP probe predicts the infill span length from bidirectional hidden states in one forward pass, eliminating preset initial lengths.
- Multi-slot parallel decoding with slot-wise attention masks and position-ID interpolation generates 2r+1 length candidates at near single-candidate cost.
- A single-pass, causally-masked post-hoc score combining inner coherence and suffix continuation (alpha=0.5) reliably selects the best candidate.
- Across five DLMs and eight benchmarks, PILL beats the strongest baseline CAL by +4.8 average pass rate on code and +6.0 BLEU-2 on text while running 1.82x faster.
- PILL adds only two extra forward passes versus roughly 12-14 for iterative baselines (Table 2).
Strengths
Genuinely cheaper mechanism with verified internal consistency
PILL replaces iterative length search with one probe pass + one scoring pass (2.00 extra forward passes vs 13.79 for CAL, Table 2); the 1.82x speed claim matches the reported 1.18s vs 2.13s on HumanEval-S/LLaDA-8B-Base, and Figure 1(c)/Table 2 agree on <10% overhead over vanilla decoding (1.10s to 1.18s).
Position-ID interpolation is a transferable insight, sharply quantified
The under-generation analysis in Section 4.3 (position-gap example producing '3' or '6' instead of '36') is concretized in Table 4: interpolation reaches 51.6 avg pass@1 vs 28.2/31.6 for left/right padding, persists under oracle selection (81.9 vs 73.2/74.2), and nearly matches independent single-slot decoding (83.6/52.9).
Honest baseline protocol and disclosed losses
Comprehensive ablation and error anatomy
Breadth beyond the main setting
Weaknesses
No variance or significance reporting anywhere
Supervised probe creates an asymmetry and a possible domain advantage
Baseline adaptation concerns
Scope limits acknowledged but real
Minor internal inconsistency in probe-training documentation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PILL (arXiv 2609.02108) targets a real deployment pain point for diffusion language models: the mask-span length must be fixed before decoding, yet correct infilling is highly length-sensitive (Fig. 1a's code example is solved only at the exact length). The authors claim a three-stage, backbone-frozen pipeline - (1) one-pass length probing, (2) parallel multi-slot decoding of five candidate lengths, (3) one-pass post-hoc selection - that beats the strongest adaptive baseline (CAL) by +4.8 average pass rate on code and +6.0 BLEU-2 on text across five DLMs and eight benchmarks, at 1.82x CAL's speed with only two extra forward passes. The claims are specific and numerically anchored, and the presented tables mostly support them; my overall assessment is positive, with reservations about statistical reporting and probe-training asymmetries.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PILL and what problem does it solve?
PILL is an inference-time method for diffusion language models (DLMs). DLMs must fix the number of tokens to generate before decoding, which hurts infilling (filling a gap between a prefix and a suffix) because the right length is unknown in advance. PILL predicts the length with a small trained probe, decodes several nearby lengths in parallel, and picks the best candidate with a one-pass coherence score - adding only two extra forward passes and requiring no preset length.
How much faster is PILL than prior adaptive-length methods?
On HumanEval-S with LLaDA-8B-Base (Table 2), PILL takes 1.18s per sample versus 2.13s for CAL and 2.29s for DAEDAL, using 2 extra forward passes versus about 13-14. The paper reports PILL as 1.82x faster than CAL while also being 6.6 points more accurate (71.35 vs 64.74 pass@1).
Does PILL require fine-tuning the diffusion model?
No. The backbone stays frozen. The only trained component is a small 3-layer MLP length probe (about 100 minutes on one A40 per backbone), trained on public code and text corpora (Py150, LeetCode, CodeContests, C4) stated to be disjoint from the evaluation datasets. This contrasts with methods like DreamOn, FlexMDM, and DDOT that fine-tune the DLM itself.
What are PILL's main limitations?
Results come from single runs with deterministic decoding and no variance estimates; the probe needs roughly 196k supervised samples per backbone; evaluation covers DLMs up to 8B on a single GPU and focuses on single-span infilling (multi-span appears only in the appendix); and on the two HumanEval tasks, CAL given its oracle-best initial length still outperforms PILL (73.48 vs 71.35 on HumanEval-S, Table 16).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with five marked '[...sampling gap...]' regions; parts of the Section 5.2 results narrative, some references, and portions of appendices (A.2, parts of A.3/A.12) were skipped. All headline tables (1, 2, 3, 4, 5, 7, 8, 10-17) are present, so the key quantitative claims were verifiable.
- Table 12's extracted row labels are merged (PILL and CAL values run together); the reading that PILL scores 59.24-71.35 across 1k-100k samples with CAL at 64.74 is confirmed by the accompanying narrative sentence ('67.96 vs 64.74 at 10k').
- The 196k-vs-100k probe-sample inconsistency noted in this review is an issue in the paper's own text, not an extraction artifact.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.