ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environments
Hejia Geng, Zesen Huang, Haoyang Li, Wenbin Li, Koutian Wu, Zihan Zhou, Yuanbo Pang, Weihao Liu, et al.
cs.CL · cs.CY
Quick verdict
Converts scientific code into calibrated agent environments with real eval/SFT/RL gains; credible, candid infrastructure work, but coverage is narrow and benchmark gains selectively reported.
Summary
The authors present ScienceIDE, infrastructure that converts scientific code repositories into executable environments for training and evaluating LLM agents. Experts and agents jointly package pinned upstream revisions into environments with calibrated scientific checks (pointwise tolerances or invariant policies), from which task factories generate validated repair and implementation tasks. The registry holds 64 environments from 27 codebases and 2,812 tasks (2,515 repair, 295 implementation, 2 acceleration), plus a public ScienceIDE-Hard subset of 85 tasks from 18 environments. Fifteen model-harness pairs evaluated under a one-hour budget reach at most 67.1% strict success (Claude Fable 5.1), leaving about a third unsolved; trajectory analysis attributes most failures to reference-convention mismatch. Fine-tuning Qwen3.5-4B/9B and Qwen2.5-72B on verifier-selected trajectories improves held-out scientific repair reward and selected public benchmarks (e.g., BBH Word Sorting +33.6 points for 9B), though comparisons are screened at a 3-point threshold and left uncorrected for multiple testing. RL on two environments with native verifier rewards raises held-out reward from 0.357 to 0.857 (LAPS) and 0.286 to 0.571 (MITgcm-biogeo) after masking budget-truncated trajectories. Main limitations: narrow task families, within-environment RL with hinted prompts and tiny held-out sets, single-seed runs, and admitted historical contamination in earlier campaigns.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Scientific codebases can be packaged into reusable executable environments whose expert-calibrated checks amortize across task families (repair, implementation, acceleration) without redefining scientific equivalence.
- Even the best of fifteen model-harness pairs solves only 67.1% of the 85-task ScienceIDE-Hard set under a one-hour budget, and most failures stem from reference-convention mismatch rather than execution errors.
- Supervised fine-tuning on verifier-selected scientific trajectories improves held-out scientific repair reward and produces gains on selected public code, reasoning, and knowledge benchmarks across 4B, 9B, and 72B scales.
- Online RL with native verifier rewards roughly doubles or triples held-out scientific repair reward (LAPS 0.357 to 0.857; MITgcm-biogeo 0.286 to 0.571) once budget-truncated trajectories are masked from the policy-gradient loss.
- Scientific experience is a viable training substrate, evidencing positive transfer from scientific interaction to broader model capability.
Strengths
Rigorous, explicit check-calibration methodology
Section 2.2 and S2.1 define pointwise (|c-r| <= a + rho|r|) vs. invariants pass policies, nominal/variant sensitivity runs, optional altbuild cross-build floors, identity-based alignment of unordered outputs, and plain-language warrants per check — a level of numerical-contract specification absent from prior SWE-style benchmarks.
Unusually candid integrity reporting
Appendix S3.1-S3.2 self-discloses 455 upstream-directed commands among 5,314 trials (21 confirmed fetches, 13 scored solved), 19 staging scripts that rewrote network declarations, Gemini provider-tool content on 71/85 tasks, and a DeepSeek aggregator-key leak corrected from .365 to .294 — rerun affected trials rather than hiding them.
Genuine RL methodological finding with ablation
Executable validity gates for manufactured tasks
Honest statistical hedging on the leaderboard
Weaknesses
Empirical coverage falls far short of the framing
SFT benchmark gains are screened and uncorrected
RL evidence is thin and confined
No cross-codebase transfer demonstrated
Construction cost of the core contribution is not quantified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ScienceIDE?
ScienceIDE is infrastructure that turns scientific software repositories (e.g., plasma, ocean, and astrophysics simulation codes) into executable environments for AI agents. Experts and agents package pinned code versions with calibrated numerical checks, then task factories generate validated repair and implementation tasks used for evaluating agents, supervised fine-tuning, and reinforcement learning.
What are the PhAI-IDE models?
PhAI-IDE-72B, PhAI-IDE-9B, and PhAI-IDE-4B are a model family trained by the authors on verified scientific-coding trajectories (experiments fine-tune Qwen3.5-4B/9B and Qwen2.5-72B bases). They improve on held-out scientific code-repair tasks measured by numerical simulation checks and show gains on selected public code, reasoning, and knowledge benchmarks.
How hard is the ScienceIDE-Hard benchmark for current AI models?
On the 85-task ScienceIDE-Hard set with a one-hour budget, the best model-harness system (Claude Fable 5.1 via Codex) solved 67.1% of tasks; the next best were Claude Opus 5 (64.6%) and GPT-6 Astra (63.1%). Most failures came from mismatching the host codebase's implicit numerical conventions, not from broken code.
Is the code, data, or the trained model available?
The paper links a public code repository (github.com/aitofound/ScienceIDE) and the model family is announced as released; however, each environment's private verifier and expert-curated checks are the core controlled assets, and the paper notes that independent external audits and full redistribution rights remain open issues.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as extracted captions/axis labels; plotted distributions and error-bar magnitudes could not be inspected directly.
- The paper text appears largely complete (including appendices S1-S6), but a few table values (e.g., Table S4/S5 cells) were reconstructed from dense OCR text and could contain extraction artifacts.
- Model and harness names (e.g., 'Claude Fable 5.1', 'GPT-6 Astra', 'Qwen3.5') are reported as printed in the provided text and were not verified externally.
- The abstract's author list is truncated relative to the full byline in the paper body, which was used for the review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.