Testing-Driven Reliability Audit of Trajectory-Based Early Outcome Prediction for LLM Agents: Target-Specific Calibration Transfer Persists Within a Single Benchmark
cs.AI
Quick verdict
Rigorous frozen audit: calibration errors persist only in two agent/head cells on SWE-bench, not broadly; no cross-benchmark replication; credible but narrow.
Summary
The paper audits whether a frozen, reference-free dual-head early-outcome predictor (inherited from EarlyEval) stays calibrated when transferred to unseen agents, and whether transfer failures are broad or concentrated. Using 4,989 public SWE-bench Verified trajectories across 10 model labels, the author runs a leave-one-agent-out audit, an oracle label-shift correction, a leave-two-agents-out shared-predictor control over 45 pairs, a persistence battery (task-cluster bootstrap, task halves, partner jackknife), a stopping-threshold sweep (0.900–0.975), and a fixed-scaffold TerminalBench boundary test. Broad same-predictor heterogeneity is not supported: median pairwise corrected-gap differences are 0.0180 (SUCCESS, 45 pairs) and 0.0385 (FAILURE, 35 pairs), below pre-registered criteria. However, two target/head cells persist: gpt-5-mini/SUCCESS (median corrected gap 0.1377, 9/9 same sign) and claude-opus-4.6/FAILURE (0.1107, 8/8), stable across cohorts, resampling, jackknife, thresholds, and prior correction. TerminalBench fails to replicate: the success target yields zero decisions (indeterminate) and the failure target's 45-decision interval includes zero. Limitations: positive evidence sits on one benchmark; discovery and persistence validation reuse the same folds; some denominators are thin; no mechanism analysis. The frozen pre-registration and exact-reproduction checks are unusually strong, but the contribution is an audit of an inherited pipeline rather than a new method.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Broad same-predictor pairwise calibration heterogeneity across agents is not supported on SWE-bench Verified: median pairwise corrected-gap differences are 0.0180 (SUCCESS, 45 pairs) and 0.0385 (FAILURE, 35 pairs), failing pre-registered criteria on both heads.
- Two specific target/head cells — gpt-5-mini/SUCCESS (median corrected gap 0.1377) and claude-opus-4.6/FAILURE (0.1107) — retain persistent calibration-transfer errors across source-cohort changes, task resampling, task halves, partner jackknife, and stopping thresholds 0.900–0.975.
- Oracle label-shift correction reduces pooled gaps (FAILURE head 0.1063→0.0421) but leaves the largest per-cell errors essentially intact (0.1423, 0.1106, 0.1053), so the target prior explains only part of the spread.
- A fixed-scaffold TerminalBench boundary test does not establish cross-benchmark replication: the success target produced zero decisions (INDETERMINATE) and the failure target's 45-decision interval includes zero.
Strengths
Frozen pre-registration and integrity verification are exceptional
Gates were defined before analysis (§3.6–3.8); phases hash-verify inputs; the 0.950 anchor reproduces all 44,901 decision rows to 1.11×10⁻¹⁶ and TerminalBench reproduces 15,889 rows with zero mismatches (§3.10); a negative intermediate result (late-reversal SIGNAL_NOT_YET_ROBUST, §4.1) is retained rather than dropped.
Shared-predictor leave-two-agents-out control is a genuine design advance
§3.6 trains one predictor+calibrator per 8-agent cohort and scores both pair members with it, removing retraining as a confound for pairwise comparisons; all 45 folds completed with zero failures, with pair statistics computed on shared task support.
Negative and boundary results are reported as first-class outcomes
Persistence battery is fully specified and multi-layered
Honest scoping of conclusions
Weaknesses
No methodological novelty; entire pipeline inherited
Discovery and validation use the same fold occurrences
Unjustified frozen gate constants and multiplicity over 20 cells
Thin denominators under several key estimates
Second-benchmark test is largely uninformative and provenance-unverified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a reliability audit, not a method paper. The audited object is the frozen reference-free dual-head early-outcome predictor (P0B_ReferenceFree_LightGBM_Dense_AF_FoldLocalFE) and stopping policy from EarlyEval [1]; the question is whether its calibrated confidence survives transfer to unseen agents, and whether failures are broad across agent pairs or concentrated in target/head cells. Four claims are advanced: (1) broad same-predictor heterogeneity is unsupported — median pairwise corrected-gap differences 0.0180 (SUCCESS, 45 pairs) and 0.0385 (FAILURE, 35 pairs), both failing pre-registered criteria (§4.3); (2) two cells persist — gpt-5-mini/SUCCESS (median corrected gap 0.1377, 9/9 same sign) and claude-opus-4.6/FAILURE (0.1107, 8/8) — across cohorts, resampling, task halves, jackknife, and thresholds 0.900–0.975 (§4.4–4.5, Table 2); (3) oracle prior correction shrinks pooled gaps (FAILURE 0.1063→0.0421) but not the largest cells (§4.2); (4) the TerminalBench boundary test establishes no cross-benchmark replication (§4.6, Table 3). The title's 'within a single benchmark' scoping is accurate.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
On SWE-bench Verified trajectories, a frozen early-outcome predictor stays broadly calibrated when transferred to unseen agents, but two specific agent/head combinations (gpt-5-mini on the success head, claude-opus-4.6 on the failure head) remain miscalibrated by about 0.11–0.14 in probability units, persisting across retraining cohorts, resampling, thresholds, and prior correction. The pattern did not replicate on TerminalBench.
Does this mean early stopping of LLM agents is unsafe?
Not in general. Aggregate calibration was benign (pooled corrected gaps of 0.0170 and 0.0421), and the broad heterogeneity hypothesis was rejected. The paper's practical point is narrower: before applying a fixed high-confidence stopping threshold (e.g., 0.95) to a new agent system, audit that predictor's calibration on that specific target, because aggregate metrics can hide a few badly miscalibrated cells.
Is the code and data available?
Yes. The analysis code (MIT license) and derived frozen artifacts (CC BY 4.0, with SHA-256 manifests) are in the public GitHub repository early-outcome-calibration-audit. Both trajectory datasets are third-party public sources (SWE-bench Verified trajectories and a community-compiled TerminalBench set) consumed at pinned revisions without redistribution.
Why wasn't the finding confirmed on the second benchmark?
The TerminalBench test was largely uninformative rather than refuting: the success target produced zero stop decisions at every threshold (an absence of measurement, labeled INDETERMINATE), and the failure target produced only 45 decisions over 14 tasks with a confidence interval including zero. Failure to replicate under these conditions does not show the benchmark caused the discrepancy, and the paper explicitly disclaims that.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 are assessed from captions only; underlying plot data not visible in the extract.
- Supplement S9.3 (dense-feature list, fold shapes) referenced but not included in the extracted text.
- External claims (GitHub repo contents, dataset revisions, pinned upstream commit) could not be verified from the text alone.
- Section A/B table formatting in the extract is dense; values were cross-checked against §4.6 prose and were consistent.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.