SCIRIGOR:Evaluating Open-Ended Scientific Analysis Beyond Final Scores
Bowen Liu, Shuo Nie, Bodong Du, Xiaomeng Li
cs.AI
Quick verdict
SCIRIGOR: evidence-graph benchmark shows agent self-consistency does not establish scientific correctness; rigorous build, no run-level variance reported.
Summary
SCIRIGOR is a benchmark and evaluation framework for scientific coding agents that scores not final artifacts but whether each generated claim retains a complete, verified data-to-claim support path within the same run. Each case requires an executable analysis, structured results, a visualization, and atomic claims. The framework reconstructs a typed evidence graph from runtime artifacts, separates artifact fidelity from cross-artifact edge validity, applies weakest-link scoring over pre-approved alternative support paths, and localizes the earliest unsupported relation. The benchmark contains 100 figure-level cases from 27 open-access articles (six domains, 17 subfields, 302 input assets), each passing two clean reproductions and a human audit; curation was GPT-5.6-assisted. Across 11 agent/model configurations, the headline finding is that agents' claims agree with their own results at nearly identical rates whether those results are faithful (91.8%) or unfaithful (91.0%) to the scientific target, while no system exceeds 62.6% soft evidence-chain score or 18.0% strict whole-chain success (GPT-5.5 xhigh); 78.3% of failed trajectories first break at the contract/execution (30.1%) or data/transformation (48.2%) stage. Limitations: no run-level variance or significance testing on main results, thin life/health (9) and physical (7) domain coverage, Claude Sonnet 5 absent from the error analysis, and evaluator internals described only conceptually.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Agents' claims agree with their own results at nearly identical rates whether those results are faithful (91.8%) or unfaithful (91.0%), so internal coherence does not establish scientific correctness.
- No evaluated system exceeds 62.6% on the soft evidence-chain score or 18.0% on strict whole-chain success (both GPT-5.5 xhigh).
- 78.3% of failed trajectories first break at the contract/execution (30.1%) or data/transformation (48.2%) stage, making upstream computation the dominant bottleneck.
- Task complexity primarily reduces chain completeness (Strict RG-EC falls from 45.7% to 25.0% as input assets go from 1 to ≥5) rather than surface claim plausibility.
Strengths
Novel scored object: the complete claim-support path
Sections 3.2/3.4 score claims over D→T→R→V→C evidence-graph paths with max-over-paths / min-within-path weakest-link aggregation (Eq. 3), directly exposing the core finding that Result→Claim consistency is nearly invariant to result fidelity (91.8% vs 91.0%, Fig. 3c) — a blind spot Table 1 shows no prior benchmark covers jointly.
Unusually rigorous benchmark construction
Section 4 + Appendices C–E: evidence closure requires two clean executions of frozen author code against all code-consumed inputs; 302 declared assets verified by SHA-256 and file-open traces; unresized source-pixel crops with logged coordinates and hashes; a final human audit for answer leakage and cross-artifact scope alignment. All 100 cases passed.
Actionable failure localization
Source-grounded alternatives prevent false negatives
Weaknesses
No run-level variance or significance testing
Coverage and reporting gaps
LLM-assisted curation introduces potential circularity
Evaluator internals under-specified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SCIRIGOR (arXiv 2609.06192) reframes evaluation of scientific coding agents around 'evidence-grounded scientific analysis': given a question, analysis-ready data, and study context, an agent must produce an executable analysis (T̂), structured results (R̂), a visualization (V̂), and atomic claims (Ĉ) such that every primary claim retains a verified same-run support path (Sec. 3.1, Eq. 1). The paper claims four contributions — task formulation, an evidence-graph evaluation framework, a 100-case benchmark, and an empirical diagnosis across 11 configurations. The headline empirical claims: (1) Result→Claim consistency is nearly invariant to result fidelity (91.8% faithful vs 91.0% unfaithful, n=364/354, Fig. 3c); (2) no system exceeds 62.6% SoftECS or 18.0% strict whole-chain success (GPT-5.5 xhigh, Table 2); (3) 78.3% of the 936 failed trajectories first break at contract/execution (30.1%) or data/transformation (48.2%, Fig. 3a). The motivating log-scale CI-transform example (Sec. 1, App. F.2) crisply shows why final-output scoring fails.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SciRIGOR?
SciRIGOR is a benchmark and evaluation framework for scientific coding agents. Instead of scoring final artifacts alone, it checks whether every scientific claim an agent reports is supported by the code, numerical results, and figure produced in the same run, using a typed evidence graph and weakest-link scoring over complete data-to-claim paths. It contains 100 figure-level cases drawn from 27 open-access journal articles across six domains.
What are SciRIGOR's main results?
Across 11 agent/model configurations, agents' claims agreed with their own results at nearly the same rate whether those results were faithful (91.8%) or unfaithful (91.0%) to the scientific target. The best system (GPT-5.5 xhigh) reached only 62.6% on the soft evidence-chain score and 18.0% strict whole-chain success, and 78.3% of failures first broke at the execution or data/transformation stage.
Why does self-consistency not imply correctness in SciRIGOR?
Because agents propagate their own numbers coherently: a claim can match the agent's own results and figure even when those results were computed wrongly — for example, applying a log-scale transform to a regression estimate but not to its confidence bounds. SciRIGOR's conditional analysis found Result→Claim consistency passes about 91% of the time regardless of whether the underlying result is actually correct, so independent checks against recomputed oracles are necessary.
How was the SciRIGOR benchmark built?
Cases come from open-access empirical articles that published both source data and plotting code. Each case was admitted only after 'evidence closure' (two clean executions of the frozen author code against all code-consumed inputs), task standardization (the agent sees only the question, study context, and declared data), and a human audit for clarity and answer leakage. 100 cases across six domains and 17 subfields passed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as captions plus extracted numeric annotations (e.g., Fig. 3a stage shares, Fig. 4 bin means/n); the rendered plots themselves were not assessable.
- No repository URL, license, or download pointer appears anywhere in the extracted text; release status is inferred from wording such as 'The release covers…' and 'The released metadata records…'.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.