How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure
cs.CL · cs.AI · cs.LG
Quick verdict
A self-audit showing LLM prompt-structure rankings are statistically fragile; unusually honest, well-instrumented, but rests on tiny samples from one endpoint.
Summary
This paper asks how much confidence a ranked LLM-evaluation table deserves, using LLM-based prompt-structure inference — asking a model to emit a typed intermediate representation (IR) from a natural-language prompt — as the case study. Eight open model variants across five families (8B–675B) were queried repeatedly with caching disabled; 293 raw IRs were persisted. Reproducibility is poor even though the synthesized corpus is mostly annotated structure the models were told to copy: mean node-set Jaccard spans 0.39–0.96, and 72% of 127 prompt-model cells are never node-set-perfect. The paper's central contribution is a claim-level self-audit of its own evaluation: under a joint cluster bootstrap, only the bottom of the ranking is firm (the two least reproducible models hold rank in 99%/86% of replicates; the middle four in 27–48%; the top two in 68% each); two defensible merge rules change four of eight rows and move the headline 7.1 percentage points; ground-truth checks show reproducibility cannot be read as accuracy (recall 0.86, explicit precision 0.82, orderings diverge); and four of eight endpoints were retired within ten weeks, making the study non-rerunnable. The audit also retracts the authors' earlier accessibility claim after identifying a 60-second client timeout as the likely instrument artifact. Main limitations: tiny, unbalanced per-model samples (n=3 to 22), a single endpoint and instrument, and no downstream-task validation. The paper is a methodological exemplar of adversarial self-scrutiny rather than a source of generalizable effect estimates.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Repeated identical LLM calls for prompt-structure inference do not reliably return the same node set, even when the structure is explicitly annotated for copying (mean node-set Jaccard 0.39–0.96; 72% of 127 cells never node-set-perfect).
- Under a joint cluster bootstrap over prompts, the model ranking reliably identifies only the worst model (gpt-oss:120b last in 99% of replicates), not the best (top two hold 68% each).
- Two equally defensible campaign-merge rules change four of eight table rows and move the study-wide headline by 7.1 percentage points, showing conclusions hinge on undocumented analysis choices.
- Reproducibility cannot be read as accuracy: against ground-truth annotations the orderings diverge (e.g., mistral-large-3:675b beats ministral-3:8b on stability but is the less accurate of the two).
- Four of eight model endpoints were withdrawn within ten weeks of measurement (all on 2026-07-15), so the study as specified can no longer be run; the 293 persisted raw outputs are what kept it auditable.
Strengths
Genuine claim-level self-audit
Table 2 enumerates seven claims; only 'gpt-oss:120b is least reproducible' survives strongly (99% last-place retention). §5.1 and Appendices C.1/F document two specification errors (endpoint-by-construction criterion; relative-vs-absolute tolerance flipping 50 vs 80 of 98 cells) instead of hiding them; §7 formally withdraws the DeepSeek/Zhipu accessibility finding after identifying the 60,000 ms client timeout as the likely cause.
Sound rank-uncertainty methodology
The joint cluster bootstrap (§5.2) resamples prompts once per replicate and rescores all models, preserving pairing; 9,583/10,000 replicates usable. The authors correctly label retention frequency a strict diagnostic, not a posterior, and cite rank confidence intervals (Neuhof & Benjamini) as the more principled instrument.
Complete artifact release enabling offline regeneration
Metric hygiene and confound disclosure
Honest small-sample reporting
Weaknesses
Very small, unbalanced samples undermine the headline table
Single endpoint, window, and instrument
Fragile stability-correctness association
Confounded cross-regime comparison and non-random missingness
Unresolvable exclusion of DeepSeek
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper studies whether LLM-based prompt-structure inference — asking a model to emit a typed intermediate representation (Rule/Condition/Breakpoint/Directive nodes) from a natural-language prompt — returns the same structure on repeated identical calls, then subjects its own evaluation of that phenomenon to a claim-level audit. Eight open model variants across five families (8B–675B), one commercial endpoint, caching disabled, R = 2–5 runs per cell, 293 raw IRs persisted. Claim families: (1) the measurement is unstable (§4, Table 1: Jaccard 0.39–0.96, 35/127 cells node-set-perfect); (2) the resulting ranking is statistically fragile (§5); (3) conclusions hinge on undocumented analysis choices — campaign merge precedence, tolerance units, an endpoint-by-construction criterion, a 60 s timeout (Table 2 collects seven claims; one survives strongly); (4) reproducibility is not validity (§6); (5) endpoint retirement makes the study non-rerunnable (§7). The self-audit is the stated main contribution.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
That repeated, identical calls to LLMs for prompt-structure inference often return different node sets (mean Jaccard 0.39–0.96 across eight models, with 72% of 127 prompt-model cells never agreeing perfectly), and that the resulting model ranking reliably identifies only the worst model: under bootstrap resampling the two lowest models keep their ranks in 99% and 86% of replicates while the top two hold theirs only 68% of the time. The central point is that a small-sample evaluation leaderboard looks more definitive than its evidence supports.
Can the study be reproduced or re-run?
Not as originally specified: four of the eight model endpoints were retired within ten weeks of measurement (all on 2026-07-15). However, the 293 raw per-run outputs, the corpus and the analysis scripts are released, so the drift, stabilisation, merge-sensitivity and correctness analyses can be regenerated offline without querying any model.
Does a reproducible LLM also give correct answers here?
No. Checking the inferred structures against ground-truth annotations gives mean recall 0.86 and explicit-node precision 0.82, and the stability ranking disagrees with the accuracy ordering — for example, the stability table would pick mistral-large-3:675b over ministral-3:8b even though the latter is the more accurate of the two.
Why did the authors withdraw their earlier DeepSeek/Zhipu accessibility claim?
Because their own client had a hardcoded 60,000 ms analysis timeout with no override flag; slower reasoning-heavy models would be logged as connection failures, matching the original error signature exactly. A probe of 11 currently served models showed most finish validly under a generous output budget, the historical failure could not be reproduced, and five successful GLM runs already contradicted the family-level claim — so the cause is unresolved, with the client instrument as the leading candidate.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The abstract is truncated mid-sentence in the source text ('...reporting rank stability, per-cel…'); the recommendations are restated in full in §10, so impact is minimal.
- Tables were extracted as run-on text; no figures appear in the extraction and could not be assessed.
- The artifact repository URL is not visible in the extracted text, so public availability of the released corpus/scripts could not be verified from the text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.