How Many Humans Are 32 LLM Judges Worth?
Chao Li, Yingying Yu, Yunfeng Li
cs.CL · cs.LG
Quick verdict
Claims 32 LLM judges match 2.3-3.8 humans by error on ChaosNLI (4.2-6.5 spectrally); rigorous, well-audited, but one pool and in-sample selection limit scope.
Summary
The paper asks how many human annotators a 32-model LLM judge panel replaces, arguing the answer depends on the measurement target. Matching a fixed 32-judge panel to empirical human label distributions on three ChaosNLI tasks (MNLI-m, SNLI, alphaNLI; 1,000 items each) yields distributional-error effective sizes nu_MSE of 2.304/3.750/3.445 versus spectral sizes nu_H of 4.242/6.459/6.499, while binary-error design-effect votes stay near two (1.971-2.227). An exact quadratic-form identity shows distributional error depends on member energy and residual orientation relative to the averaging direction, which the participation ratio discards; a realizable hard-label construction raises PR by 14.3% while worsening error by 25%. Holding observed moments fixed, extrapolated nu_MSE asymptotes are 2.392-3.990, with 32 judges already at 94.0-96.3%. An external Civil Comments check (CC-1000) gives nu_H = 2.84 versus nu_MSE = 1.49. Exhaustively enumerating all 5- and 7-judge panels, the accuracy-top-k baseline is never on the Pareto front; an at-most-two-swap greedy search gains 24.8-56.0% nu_H at +0.10-1.10pp accuracy, and the selected panels also improve nu_MSE by 11.3-33.2%. Limitations: one fixed model pool, selection evaluated on the same items used to choose panels, asymptotes resting on fixed-moment assumptions, and descriptive presentation diagnostics with cache reuse.
Review complete
Assessment at a glance
0
risk flags identified
● high 0
● medium 0
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A 32-judge panel's human-equivalent size is target-specific: distributional-error matching gives nu_MSE = 2.304/3.750/3.445 on MNLI-m/SNLI/alphaNLI versus spectral matching nu_H = 4.242/6.459/6.499, while binary-error effective votes remain 1.971-2.227.
- An exact quadratic-form identity shows distributional error depends on member energy and residual orientation that the participation ratio discards, so higher spectral diversity can coexist with worse recovery (a realizable hard-label construction raises PR 14.3% while error rises 25%).
- Holding observed moments fixed, the distributional-error asymptotes are 2.392-3.990 and the 32-judge panels already reach 94.0-96.3% of them, implying sharply diminishing returns beyond 32 judges.
- In the fixed pool, panels beating the accuracy-top-k baseline on both accuracy and nu_H exist in all eight dataset-size cases; at most two swaps achieve +24.8-56.0% nu_H at +0.10-1.10pp accuracy while also improving nu_MSE by 11.3-33.2%.
Strengths
Exact identity turns a correlation into a mechanism
Eq. (10) decomposes error E into energy scale Omega and eigenvector alignments b_j; the hard-label construction of Appendix C.4 makes the conflict realizable: PR rises 2 to 16/7 (+14.3%) while E rises 1/4 to 5/16 (+25%) under equal member energies and nonnegative correlations - not just an asymptotic curiosity.
Selection claims rest on exhaustive enumeration, not sampling
Sec. 4.10 enumerates all 201,376 (k=5) and 3,365,856 (k=7) panels per dataset across four datasets; accuracy uses exact integer arithmetic with lcm(1,2,3) tie scaling (App. A.6), eliminating tie-handling ambiguity; the top-k baseline is dominated in all eight dataset-size cases.
Exceptional audit depth
Candor about protocol relativity
Mechanistic account of why top-k panels fail
Weaknesses
Selection results are in-sample
Asymptote is an assumption-bound extrapolation
CC-1000 check is confounded
Single deterministic response per judge
No uncertainty intervals on headline sizes
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks how many human annotators a 32-model LLM judge panel replaces and argues the question has no single answer: human-equivalent size is target-specific. Using a fixed pool of 32 commercial LLMs on three ChaosNLI tasks (MNLI-m, SNLI, alphaNLI; 1,000 entropy-stratified items each with 100 human labels) plus an external Civil Comments subset (CC-1000), it reports three distinct effective sizes for the same panels: distributional-error matching (nu_MSE = 2.304/3.750/3.445), spectral matching via participation ratio calibrated against human-draw Monte Carlo (nu_H = 4.242/6.459/6.499), and binary-error design-effect votes (n_eff = 1.971-2.227; Table 3). Central claims: (1) an exact quadratic-form identity (Eq. 10) explains the nu_H/nu_MSE gap through member energy and residual orientation that PR discards; (2) fixed-moment extrapolation puts nu_MSE asymptotes at 2.392-3.990 with 32 judges at 94.0-96.3%; (3) in-pool selection beats the accuracy-top-k baseline jointly on accuracy and nu_H in all eight dataset-size cases.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 0 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How many humans are 32 LLM judges worth?
By this paper's measurements on ChaosNLI, it depends on the target: about 2.3-3.8 humans when matching distribution-recovery error (nu_MSE), about 4.2-6.5 when matching spectral diversity of residuals (nu_H), and only about 2 effective votes under the binary-error design effect. The paper's core point is that there is no single number.
Does adding more LLM judges keep improving the panel?
Returns diminish fast. Extrapolating the error curve while holding observed moments fixed, the distributional error approaches a ceiling of about 2.4-4.0 human-equivalent raters, and the 32-judge panels already sit at 94.0-96.3% of that ceiling, so adding judges adds little.
Should an LLM judge panel be built from the most accurate models?
Not necessarily. Exhaustive search over all 5- and 7-member panels in the paper's pool shows the top-accuracy baseline is never on the Pareto front; replacing one or two top-ranked models with lower-ranked, less correlated ones improved both majority-vote accuracy (+0.10-1.10 points) and effective size (+24.8-56% nu_H) in all eight tested cases.
What is the participation ratio in this paper and why does it matter?
The participation ratio is an eigenvalue-based measure of the effective spectral dimension of a panel's residual correlation matrix, between 1 and k. The paper proves PR alone cannot predict distribution-recovery error because it discards member energy and residual orientation, which an exact quadratic-form identity shows the error depends on.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2, 3, 6, 7 and D2-D4 are only partially recoverable from the extracted text; visual claims (e.g., anisotropy/angle displacements in Fig. 2, Pareto plots in Fig. 7) are taken on trust from captions and prose.
- Minor numeric inconsistency: max tie rate at k=4 is 12.3% in Sec. 4.10 but 12.4% in Appendix A.6; likely rounding, unconfirmed.
- The CC-1000 judge list (including 'GPT5.5', absent from Table A1) is not fully specified in the extracted text.
- The metadata abstract is truncated mid-sentence, but the embedded full text covers all sections and appendices, so the review rests on the full text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.