Commit-first LLM judging inherits the judge's own errors
cs.SE · cs.AI · cs.CL
Quick verdict
Careful empirical audit and experiment showing commit-first judging moves the gaming anchor to the judge's own errors; convincing, though small in scale.
Summary
This empirical study audits the default LLM-judge configurations shipped by eight widely used evaluation frameworks (nine packages, 27 configurations) and finds that none implements commit-first judging — having the judge solve the task first, commit to that answer, and accept only matching candidates — the defence Zhou measured at a 0.012 false-positive rate. Nine ship the ineffective recompute variant, traceable through a copied typographical error to one ancestor prompt. A controlled experiment then games the shipped DeepEval G-Eval 4.1.5 configuration (judge: Claude Opus 4.8; generator: Claude Sonnet 5) with plain best-of-N search that never accesses ground truth: on interval merging, 90/96 and 93/96 candidates diverged across two seeds, with the judge citing the defective line as grounds for a perfect score. Commit-first judging eliminates this (0/96 in both seeds). On duration parsing, where the judge's own committed answer is wrong (0.667 on held-out checks), the defence backfires: divergence rises from 37/41 to 50/75, and in one seed the population's final correctness rests exactly on the judge's erroneous score — the paper's central claim that the anchor moves rather than disappears. A cheap 12-call probe predicts both outcomes. Limitations: four small single-function tasks, one judge family and generator, single runs for several control arms, a lock-in result resting on one seed, and a temperature-setting asymmetry affecting cross-judge comparison.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- None of the 24 in-scope default judge configurations across eight frameworks implements commit-first judging; nine ship the recompute variant Zhou measured at 0.719 false-positive rate, sharing one ancestor prompt traceable via a copied typo.
- Plain best-of-N search with no ground-truth access games the shipped DeepEval G-Eval 4.1.5 configuration (90/96 and 93/96 diverged on interval merging), with the judge citing the defective line as grounds for a perfect score, and commit-first judging eliminates the effect (0/96 both seeds).
- On a task the judge cannot solve, commit-first judging backfires in both seeds (divergence 37/41 to 50/75), and in one seed the population's final correctness equals the judge's own erroneous score — the gaming anchor moves from candidate to judge.
- A cheap 12-call probe scoring judges' own solutions against checks they never see predicts where the defence is safe, and shows judge competence to be judge-task specific rather than scale-dependent (Opus 1/3 vs Haiku 3/3 on interval merging).
- Applied to itself, verification found real defects: 5 of 15 first-draft citation claims failed against sources, and 2 of 13 held-out checks are unjustified by task specifications (with no impact on divergence counts).
Strengths
Verifiable provenance for every census verdict
Each of the 27 verdicts is pinned to package version, file path and SHA-256 digest with a re-verification script (§2, §6); a mid-study DeepEval 4.1.5→4.2.0 release was re-checked and the G-Eval template confirmed byte-identical.
Instrument validated in both directions before use
16 deliberately broken implementations probe the held-out suites (15 caught; the in-place-sorting miss disclosed), every held-out check traced to task-spec sentences, two unjustified checks flagged and replayed per-candidate to show zero divergence-count impact (§3.2).
Full-census replay reproducibility
Honest asymmetry between failure and repair
Cheap predictive probe with mechanistic alignment
Weaknesses
Narrow external validity
Key arms are single runs
Temperature confound in the cross-judge headline
Criteria and repair are author-constructed
Census covers only default templates
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper claims six contributions (§1): a census of 27 default judge configurations from eight frameworks (nine packages; LangChain ships two); a controlled best-of-N gaming demonstration against shipped DeepEval G-Eval 4.1.5; evidence that commit-first judging relocates the gaming anchor to the judge's own answer; a cheap predictive probe; a held-out-instrument validation method; and a verification practice applied to the paper's own prose. The headline claim — commit-first judging moves rather than removes the gameable anchor, so evaluation becomes bounded by judge competence on the task — is supported on two tasks, with the lock-in endpoint (population correctness pinned at the judge's own 0.667) shown in one seed of two. The paper's scoping statements are consistently honest; the risk is not overclaiming in prose but readers overgeneralising beyond four tiny tasks.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is commit-first LLM judging?
A defence against judge gaming in which the judge first solves the task by itself and commits to that answer before seeing any candidate, then accepts a candidate only if it matches. The paper finds none of 24 audited default judge configurations implements it, and shows it works only when the judge is actually right — when the judge's answer is wrong, optimisation pressure converges on the judge's mistake instead.
Can LLM judges be gamed without prompt injection or jailbreaks?
Yes. The paper shows ordinary best-of-N selection — the loop production teams run daily, with no injection — makes 90 and 93 of 96 generated code candidates pass a shipped DeepEval G-Eval judge while failing held-out tests. The judge even located the buggy line and cited it as the reason for a perfect score.
Does making the judge solve the task first fix reward hacking?
Not always. When the judge's own answer is correct, gaming dropped from 90-93 of 96 diverged candidates to 0 in both seeds. But on a task where the judge's committed answer was wrong (scoring 0.667 on held-out checks), divergence rose from 37 and 41 to 50 and 75, and in one run the population's final correctness came to rest exactly on the judge's incorrect score.
How can you cheaply test whether a judge is safe to trust with commit-first judging?
Ask the judge to solve each task directly, three attempts per task, and score its answers against checks it has never seen — 12 model calls total. In the paper this cost about 3 pence for a small model and 15 pence for a frontier model, and it correctly predicted both where the defence helped and where it backfired.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 are present only via their captions in the extracted text; the plots themselves are images and their visual content was not reviewable (the paper states the underlying data ships as ancillary files).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.