Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus
Elias Hossain, Niloofar Yousefi, Ser-Nam Lim
cs.AI
Quick verdict
Claim: correlated errors make LLM-judge consensus overstate evidence (10 judges ≈ 3.5 independent). Well-supported, candid about limits, though headline ρ partly reflects shared item difficulty.
Summary
This empirical study quantifies how correlated errors among LLM judges inflate consensus evidence. On 1,133 listwise-complete RewardBench v2 pairs, ten open-weight judges (five models × two prompt styles) show mean pairwise error correlation 0.21; via the design-effect approximation, ten judges behave like roughly 3.5 independent ones (2.89–3.75 across datasets and Table 1 variants). Frontier banks are more correlated still: GPT-5.6-sol, Claude Opus 5, and Grok 4.5 average ρ=0.56 on 400 RewardBench pairs and co-fail at 7.7× the independent rate. Treating votes as independent flips significance conclusions in up to 28% of 100-pair Focus panels. The paper distinguishes global co-failure from vulnerable-subgroup failure, which moves correlation in opposite directions; its CorrFilter and Bias-Cluster filters target these regimes, yet neither beats plain majority in most tested settings, and a router fails non-inferiority versus a regime oracle in 22 of 24 comparisons. The practical deliverable is an item-level-inference checklist and trusted-set calibration. Limitations: small frontier banks, synthetic regimes whose natural prevalence is unestablished, and only position bias examined as a shared vulnerability.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 6 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The ten-judge open-weight bank's average pairwise error correlation of 0.21 means it provides only the statistical information of about 3.5 independent judges.
- Higher-accuracy frontier judges (GPT-5.6-sol, Claude Opus 5, Grok 4.5) have more correlated errors (ρ̄=0.56), co-failing at 7.7× the independent rate, including across providers.
- Treating judge votes as independent observations flips the significance verdict in up to 28% of 100-pair Focus panels versus an item-level test.
- Global co-failure and vulnerable-subgroup failure move measured correlation in opposite directions, so no single fixed dependence score can rank agreement correctly in both regimes (Prop. 4).
- The proposed filters and routing rarely beat plain majority voting, so the recommended practice is item-level inference plus calibration on roughly 100 trusted examples.
Strengths
Quantifies a real, consequential problem rather than anecdoting it
Ten-judge open-weight bank on 1,133 listwise-complete RewardBench v2 pairs yields ρ̄=0.206 and n_eff=3.51 (§4.1); the same loss recurs on UltraFeedback (3.36) and PKU-SafeRLHF (2.89) and across 68 sub-banks (ρ̄ 0.13–0.28, App. F.3).
Goes beyond a single correlation number
Distinguishes global co-failure from vulnerable-subgroup failure (§4.4, Prop. 4): in position-aligned poisoning, ρ̄ falls 0.22→0.08 while n_eff rises 3.4→6.0 as majorities get worse — a clean demonstration that ρ̄ alone can mislead.
Concrete statistical-consequence evidence
Unusually honest negative results and preregistration
Careful measurement hygiene
Weaknesses
Shared item difficulty inflates the headline correlation
Frontier-judge conclusions rest on few pairs and three judges
Key regimes are synthetic; natural prevalence unestablished
Corrective machinery largely fails; practical contribution is a checklist
Position bias is the only deeply studied shared vulnerability
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether consensus among LLM judges carries the statistical evidence its vote count suggests. Its thesis: judge errors are correlated, so agreement overstates evidence. Four claims are staked out: (1) the ten-judge open-weight bank (five 7–9B models × pairwise/Likert prompts) has mean pairwise error correlation ρ̄=0.206 on 1,133 listwise-complete RewardBench v2 pairs, giving effective ensemble size n_eff=3.51 via the design effect 1+(n−1)ρ̄≈2.85; (2) frontier banks correlate more (GPT-5.6-sol/Claude Opus 5/Grok 4.5: ρ̄=0.56 on 400 pairs, 7.7× the independent co-failure rate); (3) treating votes as independent flips significance verdicts in up to 28% of 100-pair Focus panels; (4) two failure regimes — global co-failure and vulnerable-subgroup failure — move ρ̄ in opposite directions, so correlation magnitude alone is insufficient.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 6 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'Agreement Overstates Evidence' claim about LLM judges?
The paper claims that when several LLM judges agree on an answer, that agreement is weaker evidence than the raw vote count suggests, because judges make correlated errors — they share training data, prompts, and blind spots. In its main setup, ten judges behave statistically like about 3.5 independent ones (mean pairwise error correlation 0.21 on RewardBench v2), and frontier judges like GPT, Claude, and Grok correlate even more strongly (0.56) despite higher accuracy.
How many independent judges are ten LLM judges really worth?
On the paper's main calibration set (1,133 RewardBench v2 pairs with five open-weight models each prompted two ways), the effective ensemble size is 3.51 — i.e., about 3.5 independent judges' worth of statistical information. It varies from roughly 2.9 to 3.75 depending on dataset and bank configuration. Frontier three-judge banks have an effective size around 1.4–1.8.
Can majority voting over multiple LLM judges be wrong even when most judges agree?
Yes, in the specific failure patterns the paper studies. When a subgroup of judges shares a vulnerability (the paper injects position bias), majorities can be systematically wrong even while overall correlation drops and the bank looks more 'independent'. In controlled experiments at 20% contamination, supermajority precision fell to 0.791, and a cluster-aware filter recovered part of the loss (+2.3 points).
Does the paper offer a better aggregation method than majority voting?
Largely no, by its own account. Its specialized filters (CorrFilter, Bias-Cluster) only help under specific injected failure regimes, an automatic router is significantly worse than a regime oracle in most comparisons, and on external factuality datasets the selector always picks plain majority. The recommended practice is: treat each preference pair (not each vote) as the statistical unit, and measure judge accuracy and error dependence on about 100 trusted examples before deployment.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Text extraction is partial: sections are omitted mid-stream (e.g., part of §4.3 and Table 2, Table 7-9, Table 20-21 fragments), so some figures/tables could not be inspected.
- Minor internal inconsistencies in the extracted text (Table 1 caption's 13% exact-tie claim versus reported Acc. 0.63; App. K narrative '2.39 of 10' versus Table 24's 2.43/10) may reflect extraction artifacts rather than paper errors; flagged, not asserted.
- Figures 1-8 are described only via captions and embedded numbers; visual claims could not be independently checked.
- Code availability is stated ('Project page · Code', 'released implementation') but no repository URL is visible in the extracted text, so it could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.