Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints
cs.AI · cs.LG
Quick verdict
Preregistered audit: LLM-judge stability gates fail on shared endpoints despite clean engineering; exemplary audit hygiene, but scope is narrow.
Summary
This paper reports a fully preregistered negative result: two research campaigns that required an LLM observer to read solution progress from partial reasoning traces both terminated at instrument-validation gates and never reached their scientific objective. Across 52,988 audited request attempts, same-window repeat rankings achieved a Spearman median of 0.400 against a frozen 0.90 threshold, and byte-identical next-day replays agreed on 78 of 100 full rankings against a required 0.99, while execution-integrity metrics (delivery, schema validity, request hashes, metadata constancy) sat at ceiling. The authors decompose the failure into three mechanisms: label-mapping bias comparable in magnitude to the signal, score gaps up to seven orders of magnitude below the instrument's noise floor, and platform nondeterminism amplified by permutation readouts. Neither metric substitution nor sampling repaired it (a 748,000-call simulated design passed 0 of 500 times). Preregistered supplements show the instability is a same-day floor (0.805 versus 0.800 cross-day, replicated over five further days), shared by four providers (medians 0.74-0.88), worsened 8.4-fold under load even when self-hosted, and the readout separates constructed errors by type, not magnitude. The paper contributes an L0-L2 snapshot-identity ladder, eight design rules, and a 24-item reporting checklist. Key limitations: mechanisms were measured on one observer model, one synthetic arithmetic task family, and self-manufactured near-degeneracy; the second campaign's preregistration timing deviates; simulations rest on proxy readouts.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Execution-layer perfection (delivery, schema validity, byte-identical requests, constant metadata) implies nothing about measurement-layer reliability of black-box LLM observers on shared endpoints (C1, §4).
- Judge instability decomposes into label-mapping bias, near-degenerate score separations, and platform nondeterminism, unified by the amplifier property of exact-permutation readouts (C2, §5).
- Neither metric substitution nor sampling (0/500 simulated passes up to 748,000 calls) repairs the failure on the tested grid (C3/C4, §6-§7).
- No exposed metadata field - including a stable system_fingerprint - certifies replay stability; the floor recurs on four providers (medians 0.74-0.88) and is a same-day, not overnight, effect (C5, §8, §10).
- A pilot at roughly 2% of the study's call volume (noise floor, gap distribution, operating characteristic) would have exposed both unreachable gates before spend (§9.2).
Strengths
Uncompromising audit chain
Append-only JSONL logs with canonical and wire-order SHA-256 request hashes, raw-response hashes, reserve-before-network caps and strict-prefix resume (§3.5); every number resolves to a SHA-256-pinned archive member (§12), and all 11 exclusion events are itemized with disposition and cost (Appendix A.10, §12.5).
Failure decomposed into separable mechanisms, not one blob
M1 (mapping bias, bmap median 0.31 logits in the fresh window, interacting with correctness: +0.531 vs -0.250), M2 (median min gap 3.04e-10 vs noise floor ~1e-3 to 1e-2, §5.2), M3 (byte-identical drift, §5.3), unified by the permutation-amplifier argument that resolves the apparent paradox of scalar stability (|Δr|=0.0067) coexisting with ranking collapse (0.400).
Prospective supplements pre-empt obvious objections
Self-correction on the record
Practical, evidence-traced design guidance
Weaknesses
Narrow external validity of the mechanism claims
The headline degeneracy was self-manufactured
Central sampling refutation (C4) is simulation over proxies
Preregistration is not uniformly clean
L2 control arm is thin
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a preregistered negative-result audit of LLM observers used as measurement instruments on shared inference endpoints. The measured system is deliberately narrow: a two-model pipeline over deterministic exact-rational arithmetic, where a generator (gpt-4o-mini) writes four step-by-step candidate solutions per task (80 tasks x 4 candidates) and a black-box observer (qwen3.7-plus, temperature 0) reads partial reasoning prefixes either as a symmetrised two-label logprob (D1-S) or as a structured full ranking of ten aliases (I0-R). The claims (C1-C6, §4-§9) are that (i) execution-layer perfection implies nothing about measurement reliability, (ii) instability decomposes into mapping bias (M1), near-degenerate separations (M2), and platform nondeterminism (M3) unified by the amplifier property of permutation readouts, (iii) neither metric substitution (§6) nor sampling to 748,000 calls (§7) repairs it, (iv) exposed metadata certifies nothing (§8), and (v) gates frozen on never-measured quantities test nothing (§9). Scoping discipline is exemplary: §11 and the conclusion repeatedly bound every claim to the tested providers, metadata fields, protocol, and windows.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What did this paper find about LLM-as-judge reliability?
That the same request sent to the same model name on shared API endpoints often returns different results: same-window repeat rankings agreed at Spearman 0.400 against a required 0.90, and byte-identical next-day replays reproduced the exact ranking only 78/100 times against a required 0.99 - even though delivery, schema validity, request hashes and all recorded metadata were perfect.
Why does an LLM judge give different answers to identical inputs?
The authors attribute it to platform nondeterminism on shared serving infrastructure (batch-dependent kernels under concurrent load), amplified by readouts that demand an exact ranking where tiny score perturbations reorder near-tied candidates. The effect exists within a single afternoon (same-day agreement 0.805 vs cross-day 0.800), appears on all four tested providers (medians 0.74-0.88), and re-emerges even on a self-hosted batch-invariant server under concurrent load (8.4x more disagreement).
Can more sampling or a different metric fix LLM judge instability?
Not on the tested grid: a simulated design of up to 500 samples per point (748,000 calls) passed the frozen ranking gate 0 of 500 times, partly because a third of candidate groups were exactly tied; and the same replay batch reads 0.780 (full ranking), 0.950 (top-1), or 0.986 (pairwise) depending on the metric. The paper instead recommends continuous aggregated readouts with preregistered equivalence bands and prospective k-fold aggregation.
Is the paper saying LLM judges are useless?
No. It explicitly does not conclude that judge evaluation is unusable. Its claim is that judge instruments have measurable noise floors, gap distributions and audit boundaries, and that gates frozen without measuring them test nothing. The constructive output is a pilot-then-freeze pipeline, eight design rules, and a 24-item reporting checklist, each traced to a measured failure.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with three marked gaps (parts of §3-§5, the start of §10.1's result table, and rows of Appendix A.10 plus §12.3); every judgment and number cited in this review comes from sections fully present in the excerpt, and the paper's appendices A.11, A.12, B, C, D are present, so confidence remains high.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.