Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts
Guojun Zhu, Xunheng Huang, Peng Yin, Jiahui Xie, Sanguo Zhang, Doudou Zhou
cs.AI · cs.LG · stat.ML
Quick verdict
CHASE counters benchmark-wide shortcuts in LLM harness evolution via protocol counterfactuals; formalization strong, evidence thin (3 rounds, no repeats).
Summary
The paper targets a generalization failure in automated agent-harness optimization: a Proposer LLM that repeatedly improves a harness against a released benchmark can encode benchmark-wide protocol shortcuts (e.g., in OfficeQA, 95.2% of 66,951 unit statements sit immediately before a table), which task holdout cannot detect because held-out tasks share the same protocol. The authors formalize a gain-destruction estimand contrasting harness improvement under the released benchmark versus validity-preserving protocol counterfactuals, and an idealized shortcut-neutralized benchmark B0. Their method, CHASE, alternates Proposer harness edits with a Challenger that searches for executable protocol transformations destroying current gains; a validity firewall checks semantic preservation, and freshly confirmed counterfactuals join a finite archive constraining later evolution and final selection. Theorems 1-3 link finite archives to B0 via composition (K(A), rho), bound the number of confirmable rounds, and give stopping certificates. On OfficeQA (3 Proposer rounds, 246 questions split by source component), CHASE reaches 68.86% on certification versus 67.98% for the unchanged harness and 30.37% on the separate Pro V2 corpus; on synthetic Syn-Ledger its shortcut-neutralized gain approaches an oracle optimizing directly on B0 (-17.79% vs -18.91% delta-BS). Main limitations: only three rounds, a single confirmed counterfactual on the sole real benchmark, no error bars or seeds, and theoretical thresholds never instantiated at the experimental scale; no code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Task holdout cannot detect benchmark-wide harness shortcuts because held-out tasks inherit the same benchmark protocol (formalized as Delta_TS near zero while Delta_BS is large).
- Challenger-searched, firewall-checked, freshly-confirmed protocol counterfactuals can be added to a finite archive that constrains harness evolution without knowing the shortcut mechanism or B0.
- Theorems 1-3: a finite counterfactual archive certifies shortcut-neutralized gain up to composition constants (K(A), rho), the number of confirmable Challenger rounds is bounded, and a non-confirmed round yields an approximate stopping certificate.
- On OfficeQA and Syn-Ledger, CHASE retains released-benchmark gains (highest Dcert and Pro V2 scores among methods) while achieving near-oracle shortcut-neutralized gains, unlike a fixed generalization gate baseline.
Strengths
Genuinely new problem axis with quantified motivation
Section 1 cleanly separates benchmark-wide protocol shortcuts from task-specific ones, and supports the motivating anecdote with corpus statistics (Table C.1: 58.1% of 246 questions mention numerical scales; 95.2% of 66,951 unit statements are immediately followed by a table; 67.6% of 94,303 table blocks are preceded by a unit statement) rather than leaving it anecdotal.
Formal estimand and non-trivial statistical theory
Section 3-4 define delta-b and B0, then Theorem 1 certifies shortcut-neutralized performance from a finite archive via composition (K(A)) plus a composition-excess constant rho; Theorem 2 bounds the number of admissible rounds by a packing argument; Theorem 3 gives a stopping certificate for non-confirmed rounds. The included proofs (Supp A) are standard but correct-looking (symmetrization + Hoeffding + union bounds).
Careful evaluation hygiene
Honest mechanism reporting, constraints demonstrably bind
Weaknesses
Headline comparisons are underpowered
Theory is never instantiated on the experiments
Implemented counterfactual diverges from the narrated one
Thin baseline coverage for the generalization claim
Reproducibility gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a benchmark-wide shortcut in agent harness evolution?
It is a protocol-level regularity shared by all tasks in a benchmark rather than tied to specific questions. In the paper's example, in the OfficeQA Treasury-document corpus a unit statement such as 'in millions of dollars' sits immediately before a table in 95.2% of cases, so a harness can learn to read the line above any table for units. Switching to held-out questions does not expose this because the layout regularity persists across tasks.
How does CHASE detect cheating harnesses?
After each round in which a Proposer LLM improves the agent harness against the released benchmark, a Challenger LLM proposes executable changes to the benchmark protocol (for example, moving where evidence appears in search results) that keep questions, answers, documents, tools, and scoring intact. Transformations that pass executable validity checks and destroy the evolved harness's improvement on fresh, never-seen confirmation tasks are archived and become constraints that later harness versions must satisfy.
What results does the CHASE paper report?
On OfficeQA, CHASE's final harness scores 68.86% on 76 held-out certification questions versus 67.98% for the unmodified harness and 64.04% for a HarnessCompass baseline, and 30.37% on the separate 90-question Pro V2 corpus versus 27.04% for the unmodified harness. On the synthetic Syn-Ledger benchmark, CHASE's gain after shortcut neutralization (-17.79% delta-BS) is close to an oracle that optimizes directly on the shortcut-free benchmark (-18.91%).
Is code or data available for this paper?
The paper links public OfficeQA datasets on Hugging Face, but the reviewed text does not state a code release for CHASE itself. Experiments use a proprietary GPT-5.6-class model in all three roles (target agent, Proposer, Challenger), and key implementation details such as the validity firewall checks and Syn-Ledger construction are in a supplementary whose full contents could not be verified.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 could not be analyzed: the extraction renders Figure 3 and surrounding pages as massively duplicated Treasury Bulletin table text, so all figure-based judgments rely on captions and body prose.
- Supplementary C.5 (token/wall-time costs), parts of C.4 (Generalization Gate prompt), and Section D (Syn-Ledger construction, reference neutralizers, placebos) are missing or truncated in the extracted text; Syn-Ledger assessment rests on Table 2 alone.
- The number of independent runs/seeds is never stated in the available text; the statistical-adequacy critique relies on reported sample sizes and rollout counts.
- Proofs of Theorems 1-3 were checkable from the included supplementary text, but supplementary pages beyond Section C.3 were unavailable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.