Replication Without Persistence in Hosted LLMs: Measurement Sensitivity in Action-Time Belief Evaluation
cs.AI · cs.CL · cs.LG
Quick verdict
复现成功而持续性失败：三问分离论证严谨，但单环境、捆绑混淆与依赖支持度的指标限制了结论范围
Summary
The paper asks whether a published behavioural finding about hosted LLMs replicates, by separating three questions: replication under the original configuration, measurement sensitivity when the instrument is rebuilt, and persistence across model identifiers. In Regent Chess, a chess variant whose hidden state is recorded exactly, a previously reported Gemini 3.1 Flash-Lite belief deficit replicates on fresh games under the historical configuration (+0.0530, 95% CI [+0.0329, +0.0714]). A same-day back-to-back comparison on the same identifier shows the rebuilt configuration lowers the endpoint by 0.0429, with the rebuilt arm unresolved around zero. Under the rebuilt instrument the sign reverses between Gemini 3.1 and Gemini 3.7, with Gemini 3.8 and a GPT-5.6 transport showing the same pattern; the serving-period contribution is statistically inconclusive. The central lesson—that replication, sensitivity, and persistence can dissociate—motivates indexing claims by identifier, serving period, instrument, and inference configuration. Limitations include reliance on one environment and one opponent, a six-component configuration bundle whose effects cannot be isolated, support-dependent endpoints that complicate cross-condition magnitude comparisons, and no code or data link in the extracted text.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The previously reported Gemini 3.1 Flash-Lite action-time belief deficit replicates on fresh games under its historical configuration (+0.0530, 95% CI [+0.0329, +0.0714]).
- Rebuilding the measurement-and-inference configuration under the same identifier on the same day shifts the endpoint down by 0.0429 (95% CI [+0.0182, +0.0667]), with no component of the six-part bundle isolated.
- Under the common rebuilt instrument, the deficit fails to persist: the endpoint reverses sign from Gemini 3.1 (+0.0242/+0.0262 at 4K/16K) to Gemini 3.7 (−0.0131/−0.0104), and also reverses for Gemini 3.8 and the GPT-5.6 transport.
- Serving-period contribution is unresolvable at available precision (−0.0166, 95% CI [−0.0483, +0.0157]), and the design cannot separate release from product tier across identifiers.
- Hosted-model behavioural claims should be indexed by tested identifier, serving period, measurement instrument, and inference configuration — the three validation questions (replication, sensitivity, persistence) can dissociate within a single evaluation.
Strengths
Genuinely novel three-way validation decomposition
Section 1 cleanly separates replication (fresh data, historical configuration H), measurement sensitivity (same identifier, same day, H vs R bundle), and persistence (common instrument R, new identifiers) — and the paper demonstrates all three dissociate: +0.0530 [+0.0329,+0.0714] replicates, H−R = +0.0429 [+0.0182,+0.0667], and cells A/C reverse sign.
Unusually disciplined confirmatory/exploratory hygiene
Cells A/B/C were sealed before outcomes known; cell D is honestly labelled descriptive because its outcome preceded the classification-rule freeze; the Gemini 3.8 amendment is documented as operational-only; the endpoint-status label 'replicates' is explicitly distinguished (Section 2) from a study-design replication claim.
Same-day H/R control bounds the confound the design cannot remove
Independent reimplementation audit with quantified agreement
Honest instrument limitation analysis
Weaknesses
Total dependence on one synthetic environment, one opponent, one provider lineage
H/R bundle is fully confounded — measurement sensitivity is a black-box association
Support-dependent endpoint invites artifact explanations for the headline reversal
Heavy reliance on post-hoc, exploratory analyses without multiplicity control
No code or data availability statement in the extracted text
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper is a replication-and-measurement study of one previously published behavioural finding: Gemini 3.1 Flash-Lite's severe action-time belief miscalibration in Regent Chess (only 1 of 62 high-confidence capture beliefs correct historically). Its contribution is not a new model capability but a decomposition of what "rerunning a result" means for hosted models, separating (1) **replication** (fresh games, historical configuration H), (2) **measurement sensitivity** (same identifier, same day, rebuilt configuration R), and (3) **persistence** (common instrument R, subsequently tested identifiers).
The claims are deliberately and narrowly scoped. The abstract promises the finding "recurs on fresh games under its historical configuration (+0.0530, 95% CI [+0.0329, +0.0714])", that the H/R same-day contrast is +0.0429 [+0.0182, +0.0667], and that under R the sign reverses between Gemini 3.1 and 3.7. Section 9 explicitly renounces causal claims about model versions, backend-drift demonstrations, and any generalisation beyond one environment and one opponent. This scope discipline is the paper's most consistent virtue.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did the original finding about Gemini 3.1 Flash-Lite replicate?
Yes, in the narrow sense of a fresh-data replication: rerunning the original study's historical configuration on newly generated games recovered the deficit (+0.0530, 95% CI [+0.0329, +0.0714]), the same side of the comparator as before. But under a rebuilt measurement configuration the same identifier's endpoint dropped by 0.0429 in a same-day comparison, and under that rebuilt instrument the finding reversed direction for Gemini 3.7 — so it replicates under the original harness yet does not persist across newer model versions.
What is Regent Chess and why was it used?
Regent Chess is a chess variant in which a player can secretly transfer 'royal status' (the Regent) among their own pieces; capturing the opponent's Regent wins. The environment's engine records the true Regent identity exactly and independently of what players report, so a model's stated belief about the hidden Regent can be scored against ground truth at the moment it acts — a property few interactive benchmarks offer.
What does 'replication without persistence' mean here?
It means the same behavioural finding can recur when you rerun the original measurement setup on new data (replication), while simultaneously failing to carry over to the next model versions tested under one common new instrument (persistence). The paper shows both happened in the same study, which is why it argues the three validation questions must be asked and reported separately.
How should LLM evaluation results be reported so they can be compared later, according to this paper?
The paper recommends indexing every behavioural claim about a hosted model by four things: the tested public identifier, the serving period in which it was measured, the measurement instrument (harness, elicitation, scorer), and the inference configuration (thinking settings, sampling parameters). A rerun that shares only the model name may be measuring a different object.
Does the paper release code and data?
The extracted text claims all reported quantities derive from a frozen, independently audited result set verified by independent reimplementations, but the provided text contains no repository link or data availability statement; provenance appendix A.6 explicitly omits repository and commit identifiers for anonymity. External reproducibility of the exact numbers is therefore not currently possible from the text alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The abstract and two body sections were omitted by truncation; analysis rests on the remaining main text and appendices as provided.
- Figures 1–4 could not be visually analyzed; figure content is taken from captions and in-text descriptions only.
- No code or data availability statement appears in the extracted text; codeAvailable=false reflects its absence, not a confirmed negative.
- All numbers cited are exactly as stated in the paper; internal consistency checks (e.g., contrast arithmetic) were not independently recomputable from raw artifacts.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.