What Makes a Terminal-Bench Task Hard? Separating Genuine Hardness from Fake-Hardness on an Adjudicated Agentic Corpus
Edward Lue Chee Lip, Boden Moraski, Tim Knappe, Lang Xiong, Sarvesh Gharat, Antonio Mari, Ivan Bercovich
cs.LG · cs.AI
Quick verdict
Claims all-fail TB3 tasks split into 78 certified-unsolved vs 47 defective; credible audit protocol but labels rest on LLM judges and single oracle runs.
Summary
The paper audits what a zero pass rate actually certifies in an agentic benchmark. Using a frozen Terminal-Bench 3 / Frontier-Bench 0.1 production record (1,081 pull requests, 639 scored tasks, 28,801 trials, $105,933 logged spend, snapshot 3c5be84efd707da8), the authors adjudicate the 125 tasks with no honest pass via an ordered validity screen combining reference-solution (oracle) runs, empty-solution (nop) controls, cheat-variant trials, trajectories, telemetry, and review records. Only 78 survive as certified-unsolved candidates; 14 have broken oracles, 8 are infrastructure-limited, 4 are exploit-only-passable, and 21 have uncertified solvability. A strictness ladder shows only 25 of 78 have two or more recorded oracle runs; 53 rest on a single reference run. Reliability checks include judge-panel unanimity (48/53 contested trials), adversarial re-checks (12/12), and infrastructure-threshold sensitivity (1-3 task changes). Analysis of 555 rejected pull requests shows agents fail many invalid tasks, and a cost signal (minimum output tokens to first honest pass) separates 217 harder from 129 tractable tasks with AUC 0.750 (CI 0.700-0.801). Main limitations: rejection taxonomy and trial labels rely on a single-pass LLM judge with limited hand-label agreement (41/53), no code or data release is stated, and all results bind to one frozen snapshot.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Of the 125 all-fail tasks in the frozen Terminal-Bench 3 record, only 78 meet the certified-unsolved criteria; 47 have broken oracles, infrastructure domination, bypass-only passability, or uncertified solvability.
- The certified-unsolved label is deliberately narrow — one accepted reference route, no infrastructure domination, no strict bypass — and does not prove intrinsic hardness or verifier completeness.
- Analysis of 555 rejected pull requests shows agents fail many invalid tasks, so pass rate is a coarse signal that cannot diagnose why a task is hard.
- Among 346 tasks with at least one honest pass, minimum output tokens to first pass gives a graded difficulty signal (AUC 0.750, 95% CI 0.700-0.801) separating 217 harder from 129 tractable tasks.
Strengths
Genuine item-validity audit with real controls, not just score analysis
Section 2 joins 21,219 ordinary trials with 1,605 oracle runs, 1,309 nop controls and 4,668 cheat-variant trials; the five-way decomposition of the 125 all-fail tasks (78/14/8/4/21) converts one leaderboard number into maintainer-actionable labels — a protocol most benchmark papers lack.
Unusual label honesty and a self-imposed strictness ladder
Table 1 (125 → 85 → 79 → 78 → 25) and the repeated statement that certification 'does not prove intrinsic hardness, verifier completeness, or failure at the intended capability' scope the claims correctly; the 53-of-78 single-oracle-run weakness is disclosed rather than hidden.
Rejection-record analysis quantifies why pass rate cannot diagnose difficulty
Cost signal reported with effect size and CI, and with stated caveats
Reliability and sensitivity checks exist at all
Weaknesses
Headline rests on the weakest rung of its own ladder
LLM-judge dependence with small hand-label denominators
No code or data release; external verification impossible
Adjudication evidence is single-pipeline and absence-based
Cost-signal construct overlap
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper audits the 125 tasks in a frozen Terminal-Bench 3 / Frontier-Bench 0.1 record (snapshot 3c5be84efd707da8: 1,081 PRs, 639 scored tasks, 28,801 trials, $105,933 spend) that no agent solved honestly, asking what the shared zero certifies. Central claims: (1) an ordered validity screen splits the all-fail set into 78 certified-unsolved candidates, 14 broken oracles, 8 infrastructure-limited, 4 exploit-only-passable, and 21 with uncertified solvability; (2) the certified label is deliberately narrow — one accepted reference route, no infrastructure domination, no strict bypass — and does not prove intrinsic hardness; (3) the 555 closed-unmerged PRs show agents fail many invalid tasks, so pass rate cannot diagnose difficulty; (4) minimum output tokens to first honest pass grades difficulty among passing tasks (AUC 0.750). The claims are scoped carefully, and the paper's candor about its own label limits is a model for benchmark auditing — though the headline framing leans on the weaker-evidence end of its own ladder.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a 'certified-unsolved' task in this paper?
An all-fail Terminal-Bench 3 task where the authors' own reference solution passed (oracle run), infrastructure failures did not dominate the attempts, no strict evidence of verifier bypass was found, and yet every evaluated agent failed honestly. The label certifies one working route plus the absence of visible disqualifiers — not that the task is intrinsically hard.
How many of Terminal-Bench 3's all-fail tasks are actually broken?
Of 125 tasks with no honest pass, 47 failed validity screens: 14 broken oracles, 8 infrastructure-limited, 4 passable only by bypassing the verifier, and 21 with insufficient evidence. 78 survived as certified-unsolved candidates, though only 25 have two or more reference-solution runs recorded.
Why can't pass rate alone measure task difficulty?
The paper shows rejected tasks with very different defects share similar low pass rates: ambiguous tasks average 0.17 and broken/nondeterministic tasks 0.18, close to genuinely too-hard tasks at 0.10, while hackable-verifier tasks average 0.41. A leaderboard zero can come from a real capability gap or from defects at any layer of the evaluation stack.
What is the cost-versus-difficulty finding?
Among 346 tasks with at least one honest pass, the minimum output tokens used by any first passing trial separates 217 harder tasks (median 24,829 tokens) from 129 tractable ones (median 12,388) with AUC 0.750 (95% CI 0.700-0.801) — a graded signal where the binary pass rate saturates, though it is observational and confounded by model choice and harness policy.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Abstract text is truncated mid-sentence ('to show that p'); full claims were reconstructed from Sections 1-5.
- Figures 1-3 are available only as embedded text descriptions; all counts and labels were parsed from caption text, not from rendered charts.
- No code/data availability statement appears in the provided text; the codeAvailable=false judgment reflects absence of any release claim.
- Several references and at least one author overlap with the Terminal-Bench ecosystem; the paper does not discuss this self-audit conflict structure.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.