Editorial routing shapes how computational results are qualified in AI-assisted scientific writing
cs.CL
Quick verdict
Claims routing a comparison to a repo makes LLM writers omit qualifications; matched design and full archive are strong, but single-provider, exploratory.
Summary
The paper asks whether editorial instructions about where supporting evidence belongs change what an LLM writes in a manuscript. Using fixed computational evidence (12 water-adsorption simulation settings whose Henry constants span about eightyfold in NbOFFIVE-1-Ni, and a constructed electronic-structure dataset of six treatments that reverse a Ba/Sr ordering), the author had Claude Opus 5, Sonnet 5 and Haiku 4.5 complete 25-word manuscript sentences via Claude Code. In the output-matched comparison, requesting identical outputs but assigning the detailed comparison to a group repository dropped numerical qualifications from Sonnet's sentences (30/30 to 0/30) and less so for Opus (30/30 to 23/30). Destination mattered: working notes outperformed Supporting Information, which outperformed the repository, an ordering not explained by reader accessibility. A targeted placement rule largely restored qualification (Sonnet 25/30) while a generic accuracy reminder did not. In 120-word paragraphs, numerical qualifications persisted but Sonnet's cross-setting summaries fell to 24/30. Cells are n=30 with exact binomial intervals and no hypothesis tests; the author flags the exploratory design, post-hoc secondary classifications, single-provider models and missing human annotation validation. A Zenodo archive supports regeneration of all reported numbers.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Assigning a detailed comparison to a group repository causes LLM writers to omit numerical qualifications alongside the reported result while retaining them in the repository (Sonnet 30/30 to 0/30; Opus 30/30 to 23/30, water sentence task, Table 1).
- The effect tracks the destination's communicative role (working notes > Supporting Information > group repository) and is not explained by reader accessibility (S3.2).
- A targeted placement rule largely restores sentence-level qualification (Sonnet 25/30, Opus 30/30), whereas a generic accuracy reminder does not (Sonnet 0/30).
- In 120-word paragraph contributions, complete omission disappears but cross-setting summaries are still redistributed away from the manuscript (Sonnet 30/30 to 24/30).
- Therefore, documentation somewhere in an AI workflow does not ensure communication of qualifications where readers encounter the reported result.
Strengths
Output-matched design cleanly isolates the routing clause
The output-matched control requests byte-identical outputs and differs from routing by one line (Suppl. S1.7), so the Sonnet drop from 30/30 to 0/30 (Table 1) is attributable to the destination assignment rather than the extra repository request; the sentence-only control (Sonnet 25/30) closes the remaining gap.
Destination x accessibility design tests the mechanism
Table S4 shows a strong destination ordering for Sonnet under routing (repository 0/30, Supporting Information 13/30, working notes 28/30), and the accessibility arm (S3.2: Sonnet 0/30 at all three stated access levels; Opus 23/30 vs 21/30) rules out reader accessibility as the driver.
Mitigation contrast shows reversibility and its limits
Exceptional provenance and transparency
Honest, well-bounded claims
Weaknesses
Single provider, small cells, no inferential tests
Partial, non-human validation of classifications
Confounded instruction contrasts, acknowledged but unresolved
Limited ecological validity
Endpoint threshold depends on a withheld reference estimate
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a communication question, not an accuracy question: given identical computational evidence, does an instruction about where a detailed comparison 'belongs' change what an LLM writes next to the reported result? Two datasets anchor the design: (i) GCMC water adsorption in NbOFFIVE-1-Ni across twelve parameter sets (UFF/Dreiding x three TIP4P water models x DDEC/DDEC6 charges), Henry constants spanning ~80-fold with the designated S5 at calc/expt = 0.4299; and (ii) a constructed electronic-structure set in which six treatments split on the Ba/Sr ordering and the lowest-MAE treatment is among the reversers. Three Anthropic models (Opus 5, Sonnet 5, Haiku 4.5) complete a <=25-word sentence via Claude Code 2.1.251. The central claim: routing the comparison to a group repository suppresses manuscript qualification (Sonnet 30/30 to 0/30; Opus 30/30 to 23/30, Table 1), and the authors explicitly disclaim any deception interpretation.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'editorial routing' in this paper?
It means instructing an AI writing assistant where a detailed comparison belongs - for example a group repository, Supporting Information, or its own working notes - while the assistant drafts the manuscript text that reports the headline computational result.
Did the AI models hide or falsify results?
No. The models almost always retained the comparison in their repository notes (29-30 of 30 sessions); the finding is that they sometimes omitted a brief numerical qualification from the reader-facing manuscript sentence when told the comparison 'goes in the repository'. The author explicitly states the results are not evidence of deception or misconduct.
Which AI models were tested and how many runs per condition?
Claude Opus 5, Claude Sonnet 5 and Claude Haiku 4.5, run through Claude Code version 2.1.251, with 30 sessions attempted per model per condition. Results are reported as counts such as 30/30 or 0/30 with exact binomial confidence intervals, and no hypothesis tests.
Can this study be reproduced?
The archived session outputs, prompts, session-level scoring table, batch mapping and analysis code are deposited on Zenodo (DOI 10.5281/zenodo.22731289), so every reported count can be regenerated from the archive; re-running the sessions themselves would require API access to the same named model versions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are present only as captions in the extracted text; the visual content of Figs. 1-3 could not be independently verified.
- Model identities are given by alias (claude-opus-5, claude-sonnet-5, claude-haiku-4-5-20251001); behavior of these specific versions cannot be independently confirmed.
- The exact instructions for Supporting Information and working-notes destinations are referenced to the Zenodo archive rather than fully reproduced in the provided text.
- This is a single-author preprint; no external peer-review status is available.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.