A Closed-Form Estimator and Diagnostic Battery for Anchor-Judge Error Correlation, Under a Single-Common-Factor Model
stat.ME · cs.CL · cs.LG
Quick verdict
Closed-form estimator for anchor–judge error correlation with calibrated diagnostics; rigorous and honest, but no real panel has passed its adequacy pre-test.
Summary
The paper asks when the clean-anchor assumption in LLM-judge evaluation can be replaced by an estimate. Under a single-common-factor model (judges load unitly on a quality and a shared common-mode factor), two or more judges and two or more anchors point-identify the quality variance, the common-mode variance, and each anchor's contamination correlation ρk in closed form (Theorem 1), with an exact non-identification boundary at βk = σ²c and a positive-measure weak-identification neighbourhood where the variance blows up (Table 1: fifteen-fold sd growth as ρ1 goes 0.98→0.99). Proposition 1 shows a uniformly loaded second factor leaves ρk unbiased; only asymmetric loadings bias it. The estimator ships with a calibrated three-test battery (judge-covariance dispersion; over-identification; a family-block test from judge metadata), a family-blocked estimator removing family-level residual bias exactly, item-bootstrap intervals with measured coverage (0.953), and a weak-identification screen. Ordinally, ρk is unidentified with all-ordinal data at any anchor count but identified with ordinal judges plus three or more continuous anchors; a mixed-scale estimator is validated semi-synthetically on HANNA. Simulations recover injected ρ within 0.02; the closed form is statistically indistinguishable from maximum likelihood, while a CT-C(M-1)-style clean-anchor estimator degrades linearly with trusted-anchor contamination. Key limitations: validation is asymmetric — both real panels tested correctly fail the adequacy pre-test, so the estimator has never been validly applied to real data; the six-provider study is a robustness observation under oracle calibration; panel-wide shared residuals remain undetectable.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under a single-common-factor model, at least 2 judges and 2 anchors point-identify the quality variance, common-mode variance, and each anchor's contamination ρk in closed form, with an exact failure boundary at βk = σ²c (Theorem 1, Corollary 1).
- Only asymmetric second-factor loadings bias ρk; a uniformly loaded second factor is absorbed into the quality variance and leaves ρk unbiased (Proposition 1).
- A calibrated three-test diagnostic battery plus a family-blocked estimator detects the harmful violations, and both real panels tested correctly fail the model-adequacy pre-test (Sections 5-7).
- With all-ordinal data ρk is unidentified at any number of anchors, but with ordinal judges and at least 3 continuous anchors it is identified and estimable (Section 7).
- The closed form is statistically indistinguishable from full-information ML and strictly more robust than a CT-C(M-1)-style clean-anchor estimator, which degrades linearly as the trusted anchor's contamination grows (Section 6).
Strengths
Closed-form transparency with an exact failure boundary
Theorem 1 inverts three observable moment types (K, Mk, Pkl) with no clean-anchor assumption; Corollary 1 derives the exact failure condition βk = σ²c (measure zero) and Table 1 quantifies the weak-identification neighbourhood (sd grows from ±0.004 at ρ1 = 0.98 to ±0.061 at 0.99). Proofs are three lines of moment algebra, SymPy-verified (Appendix A).
Violation taxonomy with plainly stated blind spots
Proposition 1 shows a uniformly loaded second factor (g = h) is absorbed into σ²t and leaves ρk unbiased; Section 5 then states that Test A cannot see homogeneous judge residuals and quantifies Test B's weak power against them (0.06 at N=500, 0.16 at N=2000, 0.99 only at N=10⁴), advising against use on shared-template panels altogether.
Calibrated battery with measured operating characteristics
Ordinal identification hierarchy is a genuine contribution
Unusually disciplined honesty and reproducibility
Weaknesses
The estimator has never been validly applied to a real panel
Oracle calibration is circular for the headline real-judge recovery
Unit-loading assumption contradicted by the author's own data
Pre-registered success criterion amended after failing
Thin seed replication for headline recovery claims
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a specific failure mode in LLM-as-a-judge methodology: anchored decompositions of judge error assume the anchor's error is uncorrelated with the judges' shared error. Sunkavalli asks when this 'clean anchor' assumption can be replaced by an estimate of each anchor's contamination correlation ρk. Under a single-common-factor model (Assumption 1: judges Jj = t + c + ej with unit quality loadings, anchors Ak = t + βk c/σ²c + uk), Theorem 1 gives a closed form for (σ²t, σ²c, {ρk}) from ≥2 judges and ≥2 anchors using three observable moment types; Corollary 1 derives the exact failure boundary βk = σ²c (measure zero) and characterizes the positive-measure weak-identification neighbourhood around it. The author is explicit that identifiability is inherited from MTMM (Campbell & Fiske 1959; Eid 2000); the claimed contributions are the closed form, the exact boundary, Proposition 1's violation taxonomy, and the diagnostic battery.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is anchor contamination in LLM-as-a-judge evaluation?
When an external reference set (an anchor) is used to measure how much error an LLM-judge panel shares, contamination is the correlation ρk between the anchor's own error and the judges' shared common-mode error. Standard anchored decompositions assume ρk = 0 (a 'clean anchor'). This paper instead estimates ρk for each anchor in closed form, so a biased anchor no longer silently corrupts the judge-error estimate.
Has this estimator been validated on real LLM judge panels?
Not end-to-end. Both real panels tested — three human raters on the HANNA benchmark and a six-provider LLM panel — correctly fail the paper's model-adequacy pre-test (Test A), so the estimator was deliberately not applied to them as valid measurements. The six-provider study is only a robustness observation under oracle calibration; the paper states plainly that no real panel has yet passed the pre-test.
Does the method work with Likert (1-5) judge scores?
Partly. If every variable (judges and anchors) is ordinal, the contamination ρk is not identified at any number of anchors. If the judges are ordinal but there are at least three continuous-scored anchors, ρk becomes identifiable via a mixed-scale estimator — though it carries a finite-sample upward bias near ρ = 0 at small N and roughly an order of magnitude more variance than the continuous case.
How is this different from standard factor analysis or MTMM models?
Identifiability itself is classical — the model is a two-factor MTMM-style model — and the closed form matches full-information ML (semopy) statistically. The claimed additions are the judge/anchor-specific closed form, the exact non-identification boundary, a proposition on which model violations bias the estimates, and a calibrated diagnostic battery tailored to LLM-judge panels.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The abstract quoted in the supplied metadata is cut mid-sentence ('and we say'), but the embedded full text contains the complete abstract; the extracted paper text runs from title through Appendix A proofs and references with no visible truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.