When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings
cs.CL
Quick verdict
Local 7B/8B judges are self-consistent yet weakly track humans (r 0.275–0.340); a sound but small study whose GPT-2 generator limits generalization.
Summary
This paper asks whether an LLM judge that produces stable repeated scores is therefore reliable by human standards. The author scores 300 responses — three per question across 100 questions in five categories (factual knowledge, instruction following, mathematics, reasoning, writing) — all generated by an instruction-tuned GPT-2 (124M). Nine human annotators rated each response 0–100 (mean pairwise Pearson 0.967, ICC(2,k) 0.996), and two locally deployed judges, LLaMA-3-8B and Qwen2.5-7B (via Ollama), scored each response three times under a shared rubric. Human–judge agreement is weak: Pearson 0.275 (LLaMA) and 0.340 (Qwen), MAE 27.71 and 18.64, with positive bias (+23.04, +9.54). Yet exact self-consistency is high: 97.3% and 92.3%. Agreement varies sharply by category (e.g., LLaMA r=0.026 on mathematics vs 0.592 on factual knowledge) and decoding setting, while length control changes correlations only modestly. Bootstrap CIs (5,000 resamples, fixed seed) support the main estimates. The central conclusion — self-consistency and human alignment are distinct properties — follows, but scope is narrow: the generator is tiny and dated, producing ~130-character responses; judge decoding settings are unspecified (greedy decoding would make high self-consistency near-tautological); and the rubric text, code/data, and annotator backgrounds are undisclosed, limiting reproducibility and generalization.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Two local open-weight judges (LLaMA-3-8B, Qwen2.5-7B) correlate only weakly with averaged human scores (Pearson 0.275 and 0.340) despite 97.3% and 92.3% exact self-consistency over three repeated evaluations.
- Self-consistency and human alignment are distinct properties, so consistency alone must not be treated as evidence of judge reliability.
- Human-judge agreement varies substantially across question categories (e.g., LLaMA-3-8B r=0.592 on factual knowledge vs 0.026 on mathematics) and across GPT-2 decoding configurations.
- Response length does not explain the disagreement: length-controlled partial correlations change the raw values by at most 0.024.
- Both judges systematically over-score relative to humans, with signed bias of +23.04 (LLaMA-3-8B) and +9.54 (Qwen2.5-7B) points.
Strengths
The core distinction is operationalized, not just argued
Section 3.2 and Eq. (1) implement 1,800 judge evaluations (300 responses x 2 judges x 3 repeats) against an independent 9-annotator human reference, so the self-consistency vs. alignment contrast is measured directly. Section 4.3 makes the point vividly: LLaMA-3-8B is 97.3% exactly consistent yet reaches only Pearson r=0.275 with humans.
Human reference validity is established before blaming the judges
Section 4.1 / Table 2 reports mean pairwise Pearson 0.967 (range 0.939-0.990) and ICC(2,k)=0.996 among the nine annotators, ruling out annotator noise as the primary explanation for the low human-judge correlations in Table 3.
Above-average statistical transparency
Multi-angle robustness checks go beyond the headline
Weaknesses
Evidence comes from a near-degenerate generator regime
Self-consistency may be a decoding artifact
Reproducibility artifacts are absent despite Section 7.2's claim
Implausibly perfect human agreement is unexamined
No uncertainty on the subgroup results that carry the narrative
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper investigates a narrow but well-posed question: does an LLM judge that reproduces its own scores across repeated runs thereby deserve trust as a substitute for human raters? Using 300 responses from an instruction-tuned GPT-2 (124M) — three per question over 100 questions in five categories — the author compares nine-annotator mean human scores against two locally deployed judges, LLaMA-3-8B and Qwen2.5-7B (both via Ollama), each run three times under a shared rubric (Judge Prompt v2). The headline claims (Abstract, Sections 4.2-4.3) are: (1) human-judge correlation is weak (Pearson 0.275 / 0.340) while exact self-consistency is high (97.3% / 92.3%); (2) therefore self-consistency and human alignment are distinct properties that must be reported separately. Both claims are supported by the data presented; the open question is how far they generalize beyond this deliberately weak generator.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
Two small local LLM judges (LLaMA-3-8B and Qwen2.5-7B) gave nearly identical scores when re-evaluating the same responses (exact match in 97.3% and 92.3% of three-run repetitions), yet their scores correlated only weakly with the average of nine human raters (Pearson 0.275 and 0.340) and showed large absolute errors (MAE 18.6-27.7 points on a 0-100 scale). The message: a judge agreeing with itself is not evidence that it agrees with humans.
Does high self-consistency mean an LLM judge is reliable?
Not according to this study. Self-consistency only measures the stability of a judge's own repeated outputs. Both tested judges were highly stable but over-scored humans by +23.0 and +9.5 points on average, and their agreement with humans varied wildly by task category. The paper argues both properties should always be measured and reported separately.
Which models and data were used in the study?
LLaMA-3-8B and Qwen2.5-7B, run locally via Ollama, judged 300 short responses (about 130 characters on average) generated by an instruction-tuned GPT-2 (124M) for 100 questions in five categories: factual knowledge, instruction following, mathematics, reasoning, and writing. Each response was rated 0-100 by nine human annotators and scored three times by each judge under a shared rubric.
What are the main limitations of this study?
The responses come from a very small, outdated generator, so results may not transfer to modern LLMs; only two 7-8B judges and one rubric were tested; the rubric text, code, and data are not released; judge decoding settings are unspecified, so near-perfect self-consistency may partly reflect deterministic decoding; and no per-category uncertainty is reported for the subgroup analyses.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 appear in the text only as captions; no numeric values were taken from figure interiors, all cited numbers come from tables and body text.
- Equations 2-4 contain PDF-extraction artifacts (e.g., repeated radical signs in Eq. 4), but the formulas were unambiguously interpretable as mean and standard deviation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.