OpenAI4S: Code as Action, Science as Sessions
Gongbo Zhang, Hao Li, Yu Wang, Mujie Lin, Liuzhenghao Lv, Yicheng Mao, Yimi Wang, Jun Zhu, et al.
cs.AI · cs.CL · cs.SE
Quick verdict
Claims its agent harness (persistent kernels, provenance) beats Claude Code 7.83 vs 5.79–6.36 on 36 science scenarios; gains partly built into its evaluator.
Summary
OpenAI4S is an open-source scientific research agent built on the principle of 'code as action, science as sessions': complete code cells execute in persistent Python and R kernels while JSON tool calls handle orchestration. An append-only action ledger, per-cell execution records, versioned artifacts, environment records and checkpoints support session recovery, branching and export, complemented by configurable sandboxing, permissions and biosecurity screening. Evaluated on 36 scenarios across six tasks (retrosynthesis, molecular dynamics, protein binder design, protein mutation, catalyst screening, mineral spectroscopy) with a deterministic-first, binary-criterion evaluator, OpenAI4S scores 7.83 overall versus 5.79–6.36 for the Claude Code harness driven by Kimi-K3, GLM-5.2 and Opus-4.8, with the largest gains on long-horizon molecular dynamics and binder-design workflows, where the harness reliably turns computation into final artifacts and reports. The results demand caution: the comparison pits complete systems with different models and does not isolate the harness's contribution; the evaluator was built by the same authors and rewards trajectory retention and artifact properties the system enforces by design; the safety layers fail open by default and received no empirical safety evaluation; and the evaluated configuration does not match the documented code baseline, with reproducibility incomplete for all systems tested.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 6 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A dual-plane harness combining persistent Python/R kernels with structured session management (append-only action ledger, versioned artifacts, checkpoints) reliably converts long-horizon scientific workflows into inspectable, delivered repositories.
- On 36 scenarios across six scientific tasks, OpenAI4S reaches an overall score of 7.83 versus 5.79–6.36 for Claude Code driven by Kimi-K3, GLM-5.2 and Opus-4.8 (Table 3).
- The aggregate advantage concentrates on long-horizon, computation-intensive workloads (MD +2.95, binder design +0.88 versus the best competitor) and on converting workflow logic into final artifacts, traceable numbers and bounded claims (Fig 6).
- Sandboxing, permission mediation and code/trajectory screening provide complementary safeguards, though only BLOCK verdicts stop execution and several screens fail open.
- Environment specification and full rerunnability remain weak for every evaluated system, including OpenAI4S itself.
Strengths
Concrete harness architecture with unusually honest boundary reporting
Section 3 and the supplementary notes (S1–S4) specify dual-plane execution, synchronous host RPC, and state-lifetime limits (e.g., background jobs don't inherit the foreground namespace; R lacks mid-cell RPC) with failure semantics per mechanism.
Carefully engineered evaluation protocol
Section 4.2 decomposes three dimensions into binary criteria, runs deterministic checks before a constrained LLM judge with AND-combination and a hard score cap (2.0 default), and documents the withheld-data detector's validation: 17 false positives in a raw-string version triggered an AST rewrite and uniform rescoring of 155 evaluations.
Strong and specific empirical margins
Disclosed self-counterexample
Explicit mechanism-of-gain analysis
Weaknesses
Internally inconsistent headline statistics
Complete-system comparison; model confound unresolved
Evaluator built by the same team rewards the system's own design goals
No scientific-agent baseline and no significance testing
Safety layers fail open and are never empirically evaluated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
OpenAI4S (arXiv 2609.15096) is a systems paper: an open-source scientific research agent whose organizing principle is *code as action, science as sessions* — scientific actions are complete code cells in persistent Python/R kernels, while orchestration runs through JSON tool calls (Section 3, Fig 2). Around this dual plane the authors build session-level provenance (append-only Action Ledger, per-cell records, SHA-256-versioned artifacts, environment generations, checkpoints with branch/revert), bounded recovery (recipe-based cell replay with explicit coverage reporting), and safeguards (Seatbelt/bubblewrap sandboxing, permission broker, heuristic/LLM code screening, keyword-gated biosecurity trajectory screening). The empirical claim is that this harness beats a general-purpose coding harness on 36 scenarios across six scientific tasks, scoring 7.83 vs 5.79–6.36 (Table 3) with a human ground-truth ceiling of 8.70.
The paper is unusually candid about its own boundaries — capture/recovery limits, fail-open screens, evaluator circularity — and even includes a self-counterexample (budget-constrained catalyst screening, Fig 7: OpenAI4S 4.4 vs baselines 4.9 on packaging alone). That candor is a genuine strength; it does not, however, neutralize the structural problems below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 6 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is OpenAI4S?
OpenAI4S is an open-source (MIT) scientific research agent from Peking University and collaborators. Following the principle 'code as action, science as sessions', it lets an LLM do computational science by writing complete code cells that run in persistent Python/R kernels, while orchestration, permissions and monitoring go through structured JSON tool calls. It records an append-only action ledger, per-cell execution records, versioned artifacts and workspace checkpoints so research sessions can be inspected, resumed, branched and exported.
How does OpenAI4S differ from Claude Code or other coding agents?
Claude Code-style harnesses interleave model reasoning with individual tool calls and commands. OpenAI4S instead keeps Python and R kernels alive so datasets, fitted models and simulation state persist across steps, lets one code cell compose many operations with synchronous access to audited host services, and adds research-session provenance: versioned artifacts, environment records, checkpoints, branching and recovery. In the paper's benchmark it also ships ~604 curated 'Skills' (code recipes) for chemistry, biology and materials tasks.
What results does the paper report?
On 36 research scenarios spanning six tasks (retrosynthesis, molecular dynamics, protein binder design, protein mutation, catalyst screening, mineral spectroscopy), OpenAI4S scores 7.83 overall on the authors' repository-level evaluator, versus 5.79 (Kimi-K3), 6.36 (GLM-5.2) and 6.12 (Opus-4.8) for the Claude Code harness; a human-built reference workflow scores 8.70. Gains are largest on long, compute-heavy molecular dynamics and binder-design workflows.
Is OpenAI4S's benchmark and code publicly available?
Yes, per the paper: the system is MIT-licensed at github.com/PKU-YuanGroup/OpenAI4S with PyPI and desktop distributions, and scenario definitions plus the LLM-judge evaluation pipeline are available at 'SciCodeBench-OpenAI4S'. Note, however, that the supplementary materials state the documented code commit is not the exact revision used in the reported experiments, so exact-results reproduction may require contacting the authors.
What are the main limitations of OpenAI4S?
Per the authors: in-memory kernel state does not survive restarts and recovery only replays eligible cells; environment reconstruction on another machine is future work; safety screens can fail open (only BLOCK verdicts stop execution, missing screeners allow it); the comparison used unmatched models (doubao-pro versus three frontier models); and the evaluator rewards properties the system itself enforces, so the measured advantage over coding harnesses should not be read as an isolated causal effect of any single component.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–7 could not be analyzed as images; all figure readings (including the Fig 5C vs Section 4.4 discrepancy and panel subtitle/caption mismatches) come from extracted caption and body text, which may garble sub-labels.
- The abstract's author list (8 names) does not match the full-text byline (17+ authors plus community); metadata and extraction may differ from the published version.
- Domain-level aggregation figures appear only as in-text numbers with no domain table in the extracted text; the inconsistency flagged may partially stem from missing table content.
- The abstract appears truncated mid-sentence at the end of the embedded text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.