Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels
David Ababio Awuni, David Ababio Awuni, Luke E. K. Achenie, Benjamin Tei Partey, Elvis Gyasi Owusu, Elvis Gyasi Owusu, Nii-Nai Derrick Sowah
cs.CL · cs.AI · cs.CY · cs.LG
Quick verdict
Corrected estimator shows a robust 6.7pp same-family lift across four LLM judge families; sound methods, though magnitude is design-sensitive and scope is limited.
Summary
The paper audits family-conditioned preference — extra support a judge gives a candidate from its own model family — in a fully crossed panel of four open-weight families (Llama 3.1, Qwen 2.5, Gemma 2, Yi 1.5), each at two parameter scales, over 194 English prompts and 9,312 pairwise judgments. The authors show the natural row-wise per-family statistic is confounded with candidate quality (r = 0.95 with Bradley–Terry ability) and derive a column-wise estimator that holds the candidate family fixed. All four families then show positive same-family lifts (3.4–8.4 pp); global FPS is 0.067 (95% CI [0.053, 0.084], permutation p = 0.0002). The effect survives nested fractional-logit quality controls, an independent blind human-consensus anchor, and a float16 judging replication (FPS 0.066). Adding judge-side likelihood advantage attenuates the controlled coefficient by 61% (0.299 to 0.116), reported explicitly as descriptive attenuation, not causal mediation. Position bias is treated separately: 55.4% of AB/BA pairs reverse (Gemma 78.8%), exceeding the 50% ceiling of an independent content-noise model, and panel composition changes 18.5% of pairwise outcomes. Limitations: open-weight, English-only scope; each family pools two parameter scales; effect magnitude is design-sensitive (multiverse range 0.039–0.231); the reproducibility archive is announced but not yet publicly released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The standard row-wise per-family preference statistic is confounded with candidate quality (r = 0.95 with Bradley–Terry ability), and a column-wise estimator that holds the candidate family fixed isolates the judge–candidate family interaction.
- All four tested families show a statistically significant positive same-family lift (3.4–8.4 pp), with global FPS 0.067 (95% CI [0.053, 0.084], permutation p = 0.0002).
- The effect persists under nested panel-derived quality controls, a blind human-consensus anchor, and an unquantized float16 judging replication (FPS 0.066).
- Adding judge-side likelihood advantage reduces the fully controlled same-family coefficient by 61% (0.299 to 0.116), which the authors treat as descriptive attenuation rather than causal mediation.
- AB/BA reversal of 55.4% (Gemma 78.8%) exceeds the 50% bound implied by an independent content-noise model, establishing systematic position dependence as a failure mode distinct from family preference, and panel composition alone changes 18.5% of pairwise outcomes.
Strengths
Genuine identification insight, not just a new metric
Section 4 derives that the row-wise contrast equals (4/3)(b_f + γ_ff), mixing candidate main effect with interaction, while the column-wise contrast equals (4/3)γ_ff. The demonstration is concrete: naive scores track BT ability at r = 0.95 (Qwen +0.160, Yi −0.004) vs r = −0.24 after correction, and the permutation null per-family means match (4/3)b_f to four decimals (Sec 4.2).
Triangulated robustness across independent stacks
Nested controls barely move the coefficient (0.270 to 0.301, Table 2); off-diagonal-only BT leaves abilities unchanged to four decimals; a blind human anchor shifts the coefficient from 0.370 to 0.382 (Sec 6.2); an unquantized float16 replication gives FPS 0.066 vs 0.067 (Sec 9); all 72 multiverse specifications and all leave-one-out analyses stay positive.
Careful causal hygiene on the likelihood channel
Sharp formal test for position bias
Practical stakes quantified against external references
Weaknesses
Family is conflated with parameter scale
Reconciliation entangles per-family FPS with judge-specific reversal rates
Human anchor is small and only moderately reliable
Surface-format confounds are acknowledged but unmeasured
Reproducibility claims are unverifiable at review time
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is family-conditioned preference in LLM-as-a-judge evaluation?
It is the tendency of an LLM judge to give extra support to a candidate answer produced by a model from the same model family (for example, a Qwen judge favoring a Qwen candidate) beyond ordinary quality differences. This paper measures it in a fully crossed panel of four open-weight families and finds a 3.4–8.4 percentage-point same-family lift, with a global family preference score of 6.7 percentage points.
How large is the same-family judge bias compared with leaderboard gaps?
The paper's 6.7 percentage-point global lift exceeds 8 of 9 adjacent length-controlled win-rate gaps in the AlpacaEval 2.0 top-10 snapshot it compares against (median gap 1.09 pp). The authors also show that changing panel composition alone alters 18.5% of pairwise outcomes relative to a family-balanced reference, so the effect is practically large even though it is only a scale comparison.
Why was the naive way of measuring judge family bias wrong?
Comparing how much a judge supports its own family against how much it supports other families mixes in candidate quality: a strong candidate family makes its same-family judge look biased even without favoritism. In this paper's data the naive statistic correlated r = 0.95 with candidate ability. Holding the candidate family fixed and comparing different judges on the same candidates removes the candidate-strength confound.
What explains same-family preference in LLM judges, and does it survive controls?
Judge-side likelihood — how predictable the judge finds the text — is closely related: same-family responses had about 0.69 log-units higher per-token likelihood, and adding a likelihood-advantage term to the controlled model cut the family coefficient by 61%. The authors call this descriptive attenuation, not a proven cause, and a smaller statistically significant family effect remains after panel quality controls, blind human-consensus labels, and a float16 replication.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 are available only as text-rendered descriptions with embedded numbers in the extraction; visual details could not be inspected directly.
- The reproducibility archive is described as prepared but not yet published, so no reported number could be verified against raw data.
- This is a preprint (arXiv:2609.17857) and has not undergone peer review; 2026-dated citations could not be verified offline.
- Appendices A–D are included in the extraction, but any supplementary material beyond them was not available.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.