How Sensitive Are LLM Leaderboard Claims to Hidden Model Selection?
Chen Yang, Xianyang Zhang, Jun Chen
stat.ML · cs.LG
Quick verdict
Rigorous sensitivity curves quantify how many hidden LLM variants a leaderboard margin survives; strong theory, though few audited claims are contested.
Summary
The paper asks how many privately evaluated model variants a published LLM leaderboard margin can support while remaining statistically meaningful. Because neither the number of hidden variants nor their correlation is public, the authors invert the usual multiplicity correction: under a Gaussian margin model and a least-favorable (equal-means) null they derive a per-claim sensitivity curve reporting the maximum hidden count as a function of an assumed lower bound on within-family correlation, using a one-factor Dunnett tail plus Slepian-based conservative bounds. A key point is that the relevant correlation must match the ranked score and the sampling model: in a controlled Qwen2.5 fine-tune family the pooled item correlation is 0.90, but the score-matched correlation is 0.46 under item resampling and 0.92 when MMLU subjects are resampled. An item-based audit of 394 adjacent-rank claims on the Open LLM Leaderboard finds 391 lack statistical support even before selection correction (median z=0.17); at k=27 the single contested rank-1 claim (z=2.61) requires a correlation floor rising from 0.689 to 0.849 under nuisance adjustments, while a v2 claim (z=3.01) survives all displayed adjustments. Masking replays are conservative in 12/12 families. Limitations: hidden-family correlation remains an assumption, all public verdicts are conditional with unverified applicability, and the package is promised on acceptance.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A published leaderboard margin certifies a provider-level advantage only up to a computable maximum number of hidden variants k̄(z; ρw⁻, ρb), derived from a least-favorable (equal-means) Gaussian null.
- The correlation governing the hidden-selection correction must match the ranked score functional and the sampling model; pooled-item and score-matched correlations differ substantially (0.90 vs 0.46 in a controlled family) and can even reverse family orderings.
- 391 of 394 adjacent-rank claims on the Open LLM Leaderboard v1 lack statistical support even before any selection correction (median z = 0.17).
- For the few claims passing the uncorrected test, certification at k=27 hinges on an assumed within-family correlation floor (v1 rank-1 requires ≥0.849 under the joint adjustment), which public data cannot estimate.
Strengths
Well-posed inversion of an unidentifiable quantity
Rather than estimating the unobservable search size k, the paper outputs a per-claim sensitivity curve k̄(z; ρw⁻, ρb) (Eq. 4, §3), in the spirit of Rosenthal's file-drawer count, with correct limiting behavior at the Šidák, Dunnett, and clone boundaries (§3, B.6: c.05(27; 0.999999, 0.5)=1.6467 vs 1.6449).
The score-matched correlation insight is consequential and well-proven
Proposition 2 distinguishes pooled-item from score-matched estimands; the exact variance decomposition shows GSM8K is 8.6% of items but ~91% of composite variance (Figure 2a, 9), explaining the 0.90-pooled vs 0.46-score-matched gap, and 4/12 observational families even reverse ordering.
Conservative inference validated against ground truth
Controlled families with known generating process
Unusual scope discipline
Weaknesses
Practical yield of the selection machinery is tiny
Decisive input is stipulated, not estimated
Nuisance adjustments rest on extreme-tail asymptotics
External validity and provenance gaps
Dense layered presentation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses a narrow but consequential inference problem: given a published margin between a provider's submitted model and a fixed public comparator, how many privately evaluated sibling variants could that margin still certify against, if the siblings correlate at least ρw⁻? The deliverable is not a corrected ranking but a per-claim sensitivity curve k̄(z; ρw⁻, ρb) (Eq. 4), explicitly in the spirit of Rosenthal's file-drawer count. Three headline claims: (i) the certificate machinery — exact Gaussian tail, conservative heterogeneous bounds, nuisance adjustments — is valid under stated assumptions; (ii) the correlation entering the curve must match the ranked score and the sampling model (Proposition 2): pooled-item and score-matched estimates differ dramatically (0.90 vs 0.46 in the worked family, 0.92 under MMLU subject resampling); (iii) on the Open LLM Leaderboard, 391/394 adjacent claims fail even the uncorrected test, and for the few that pass, verdicts hinge on unobservable correlation floors. Scope statements are honest: pair-conditional, fixed-k, applicability unverified for every audited row.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper conclude about LLM leaderboard rankings?
That almost all adjacent gaps on the audited Open LLM Leaderboard (391 of 394 claims) are too small to distinguish from noise even before considering hidden model selection; for the few significant gaps, certification depends on assumptions about privately tested sibling models, which the paper makes explicit through sensitivity curves rather than pretending to estimate them.
What is a hidden-selection sensitivity curve?
For a given published margin, it is the largest number k of privately evaluated sibling models for which the margin still passes a significance test, computed as a function of an assumed lower bound on how correlated those siblings are. The more correlated the siblings, the less a maximum over k tries inflates the margin, so more hidden tries can be tolerated at the same correlation floor.
Why does the paper use 27 hidden variants?
27 is the variant count Meta reportedly tested for a Llama-4 release on a battle platform (Singh et al., 2025). The paper uses it as a stated sensitivity reference, not an estimate of any provider's search; it sits at about the 97th percentile of public per-organization submission counts on the audited board.
Is code or data available for this paper?
The analyses use public per-item leaderboard data (395 models × 28,659 items, via the tinyBenchmarks release). The authors describe a package, selective-evals, with tests, scripts, and machine-readable results included as supplementary material, with a public release promised upon acceptance.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extracted text contains two '[...section omitted...]' gaps (one near the §3–§4 boundary, one within the references/Appendix I); a few transitional passages were reconstructed from surrounding text.
- Figures are available only as captions and in-text descriptions; plotted values could not be visually verified.
- The selective-evals package and machine-readable results are described but not yet publicly verifiable (release promised on acceptance).
- Some reference entries are truncated mid-list in the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.