How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method
Konstantin Grotov, Valentin Malykh
cs.LG
Quick verdict
Claims draft-model gate cuts coding-agent errors 6–8pp and tokens 14–19%; promising and honest, but exit-code-only oracle and no confidence intervals.
Summary
The paper presents Speculative Uncertainty (SU), a method for estimating failure risk of black-box LLM coding agents from output tokens alone. It inverts speculative decoding: a small open-weight draft (Qwen3-4B) teacher-forces the agent's generated trajectory in a single forward pass, yielding per-token surprisal, gap, and entropy. Features are computed separately over reasoning and action spans, and an ℓ1-logistic calibrator maps them to a probability that the next tool call executes without error. Trained on 10,000 SWE-rebench OpenHands trajectories from Qwen3-Coder-480B, SU reaches .77–.80 AUROC, approaching the white-box HTC baseline (.82–.85) and beating verbalized confidence (.57–.59). Deployed as a pre-execution veto gate, it cuts per-call execution error from 21% to 15% on SWE-Bench Verified and from 14% to 6% on DA-Code while saving 14–19% of tokens, but task success drops 5 points on SWE-Bench Verified (49%→44%). Distilled drafts transfer zero-shot to Claude 3.5 Sonnet (.69 AUROC). Main limitations: the oracle is only the exit code, all results are single-run estimates without confidence intervals, calibration remains imperfect, and no code release is stated.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A small draft model scoring a black-box agent's output tokens via inverted speculative decoding recovers a failure signal approaching white-box calibrators (.77–.80 AUROC vs HTC .82–.85).
- Separating reasoning and action spans is decisive: pooling phases costs 5–6 AUROC because the failure signal has opposite sign in the two phases.
- A pre-execution veto gate cuts per-call execution error by 6–8 pp and token cost by 14–19% on SWE-Bench Verified and DA-Code, transferring zero-shot across benchmarks and agents.
- The distillation objective barely matters (.76 vs .77); drafts aligned on one open agent transfer to closed agents well above the untrained baseline (.69 vs .60 AUROC on Claude 3.5 Sonnet).
Strengths
Genuine black-box constraint met at inference
SU reads only realized output tokens via one teacher-forced draft pass (§3.2, §B.2); no logits, weights, activations, or repeated sampling — exactly the regime where HTC (white-box) and sampling-based UQ are inapplicable to closed APIs.
Phase separation is ablation-verified, not just asserted
Appendix A/Figure 2 show the failure signal has opposite sign in reasoning vs action spans; dropping phase separation (SU-Uniform, Table 1) costs 5–6 AUROC (.79→.74 SFT held-out; .76→.71 and .77→.72 on OOD splits).
Honest negative results and miscalibration disclosure
Deployment-grade evaluation detail
Practically interesting cross-agent transfer
Weaknesses
Oracle is much shallower than the framing
No statistical uncertainty on headline numbers
Abstract omits the task-success regression
Advantage over HTC is access, not accuracy
Reproducibility gaps around Claude and code
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a method paper on uncertainty quantification for LLM agents, evaluated on agentic software engineering. The central claim: a black-box agent's failure risk can be recovered from output tokens alone by inverting speculative decoding (§3.2) — a small open-weight draft (Qwen3-4B) teacher-forces the agent's generated trajectory in one forward pass, producing three per-token signals (speculative surprisal, argmax gap, entropy). Phase-aware features (48 statistics split across reasoning/action spans plus 2 span lengths, §3.3, Table 8) feed an ℓ1-logistic calibrator predicting whether the next tool call executes without error. The deployment claim (§4.3, Table 2): a pre-execution veto gate cuts per-call execution error by 6–8 pp and tokens by 14–19% on SWE-Bench Verified and DA-Code, transferring zero-shot across benchmarks and agents. Scope is narrower than the framing: the verifiable objective Ω is strictly exit-code-0 (§B.1), so 'confidently wrong' here means 'crashes', not semantically wrong patches that run.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Speculative Uncertainty (SU)?
SU is a method that estimates how likely a black-box LLM agent's next action is to fail by having a small open-weight 'draft' model re-read the agent's already-generated text in a single forward pass — the reverse of speculative decoding, where a draft accelerates generation. Token-level surprisal, gap, and entropy features, split by reasoning vs action phase, feed a small linear calibrator that outputs a failure-likelihood score.
Does the SU veto gate improve task success?
No. On SWE-Bench Verified the resolved rate fell from 49% to 44%, and on DA-Code it was roughly flat (64% to 63%). The demonstrated gains are fewer execution errors (−6 to −8 percentage points per call) and 14–19% lower token cost from avoiding execute-fail-retry loops.
Can SU be used with closed-source models like Claude?
Yes — that is the main selling point. It needs only the agent's output text, no logits or internals. A draft aligned on the open Qwen3-Coder agent transferred zero-shot to Claude 3.5 Sonnet at .69 AUROC, versus .60 for an untrained draft.
What are the main limitations of the paper?
Evidence is limited to a code-execution (exit-code) oracle, so only crashing actions are predicted, not semantically wrong code that runs. All results are single-run estimates without confidence intervals, the score is not well calibrated as a probability (ECE worsens after SFT), and the paper states no code release.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 and Figure 2 are vector graphics: their plotted curves cannot be verified from the text extraction; assessment of the phase-structure evidence relies on captions and in-text descriptions (§Appendix A).
- The extracted text runs complete from abstract through Appendix C (Table 8) with no cut-off at the end; no content is assumed missing beyond the figures noted above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.