OpenDiscoveryTrace: Process Traces for Evaluating AI Scientist Workflows
cs.AI
Quick verdict
Public 558-trajectory process-trace dataset for AI scientists; promising process-vs-output finding, but rests on unanchored LLM judging and keyword heuristics.
Summary
The paper introduces OpenDiscoveryTrace, a public dataset of 558 trajectories recording how seven AI models work through 124 scientific tasks across drug discovery, materials science, genomics, and literature analysis. Unlike output-only benchmarks, each step stores a nine-field trace (thought, tool call, observation, error, revision trigger, confidence, among others) under a standardized six-phase workflow. Three frontier models (GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro) ran all 124 tasks; four open-weight models contributed 120 single-response runs, plus six tool-scaffolded Qwen2.5-7B runs and 60 live-retrieval variants. Pilot analysis of 363 LLM-judged trajectories shows comparable success rates (84–89%) yet large process differences: Claude averages 2.5 errors per trajectory versus 0.08 for GPT-5.4 (Cliff's δ=0.613), with 66.7% tool misuse for Claude versus 83.6% reasoning errors for GPT-5.4. Five benchmark tasks come with logistic-regression, random-forest, LSTM, and Transformer baselines; none beats the majority baseline on outcome prediction. Key limitations: success on open-ended tasks rests entirely on an LLM judge with no human anchor yet, frontier web search was simulated rather than live, and error typing uses keyword heuristics that may favor verbose models. Dataset (CC-BY-4.0) and code are public.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing AI-scientist benchmarks evaluate only final outputs; OpenDiscoveryTrace is presented as the first public dataset of structured process traces (558 trajectories, 9 fields per step).
- Frontier models show statistically indistinguishable success rates (84–89%) but roughly 30× different explicit error rates (Claude 2.5 vs GPT-5.4 0.08 per trajectory; H=122, p<10⁻⁴, Cliff's δ=0.613).
- Error profiles are qualitatively different: 66.7% of Claude's 462 error events are tool misuse versus 83.6% reasoning errors among GPT-5.4's 61 events (Table 4).
- 68.1% of first errors occur at step 0, suggesting early monitoring could prevent most failures (Appendix H.2).
- No baseline (logistic regression, random forest, gradient boosting, LSTM, Transformer) exceeds the majority baseline (0.693) on trajectory outcome prediction, which the authors read as task non-triviality.
Strengths
Targets a real evaluation gap with a reusable schema
Output-only AI-science benchmarks discard reasoning; the 9-field per-step schema with error, revision_trigger, confidence, and recovery/outcome blocks (Appendix A) is concrete and immediately reusable for auditing agent behavior.
Matched-task, within-domain design with effect sizes
Table 5 compares models only on the 31 shared tasks per domain: the error divergence is significant in all four domains (H=29.5–34.0, p<0.0001) while steps are n.s., and Table 3 reports Cliff's δ=0.613 plus Wilson CIs (H.3) — the core claim is not a pooled artifact.
Unusually transparent about artifacts and weak spots
Internally consistent numbers
Weaknesses
Headline error metric is heuristic-contaminated
Success labels unanchored for open-ended tasks
Simulated retrieval undercuts the literature domain
Benchmark delivery incomplete
Reporting slips
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
OpenDiscoveryTrace is a workshop dataset paper (AI for Science @ ICML 2026) claiming the first public corpus of structured process traces from AI-scientist agents: 558 trajectories spanning 124 tasks across drug discovery, materials science, genomics, and literature analysis, produced by three frontier models (GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro; 124 tasks each), four open-weight models (30 single-response trajectories each), six tool-scaffolded Qwen2.5-7B runs, and 60 live-retrieval variants. The central empirical claim: output-only evaluation is blind to process — frontier success rates are statistically indistinguishable (Table 3: 88.6% / 83.9% / 89.3%, n.s., p=0.38) while per-trajectory error rates differ roughly 30× (Claude 2.5 vs GPT-5.4 0.08; H=122, p<10⁻⁴, Cliff's δ=0.613). Five benchmark tasks are proposed; four receive baseline results. Phases 2–4 (scale-up to 1,800+ trajectories, human expert annotation, full live retrieval) are roadmap, not delivered results. The header footnote also claims a workshop 'Best Dataset Award', which a reviewer cannot verify from the text.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the OpenDiscoveryTrace dataset?
A public dataset of 558 trajectories recording how seven AI models — GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro, Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini, and Qwen2.5-1.5B — worked through 124 scientific tasks in drug discovery, materials science, genomics, and literature analysis. Every step stores a nine-field trace including the model's thought, tool calls, observations, errors, revision triggers, and self-reported confidence. It is released under CC-BY-4.0 on Hugging Face, with the harness and analysis code on GitHub.
What is the main finding of the OpenDiscoveryTrace paper?
On 363 LLM-judged trajectories, the three frontier models achieved similar success rates (84–89%), yet their processes differed sharply: Claude Opus 4.6 averaged 2.5 errors per trajectory versus 0.08 for GPT-5.4 (about 30×, Cliff's δ=0.613), and the error types differed too — 66.7% of Claude's error events were tool misuse while 83.6% of GPT-5.4's were reasoning errors. An output-only benchmark would have rated the three models as interchangeable.
Are there important caveats about the OpenDiscoveryTrace results?
Yes. Success on open-ended tasks was decided entirely by an LLM judge with no human validation yet (planned as Phase 3); web search was simulated rather than live for the frontier-model trajectories; and error classification partly relies on keyword heuristics that could favor more verbose models. The authors themselves flag these as limitations and future work. Also, only four of the five promised benchmark tasks have baseline results.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 is a workflow schematic; only its caption is text-extractable, so no numbers were lost from it.
- The extraction contains the full body, references, and Appendices A–N including Tables 1–10, ending cleanly at page 12; no truncation is assumed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.