Dr. Claw: An AI Scientist Workspace for Vibe Research
Dingjie Song, Hanrong Zhang, Dawei Liu, Yixin Liu, Zongxia Li, Zhengqing Yuan, Siqi Zhang, Henry Peng Zou, et al.
cs.AI · cs.CL · cs.CV · cs.LG
Quick verdict
Claims orchestration wrapper lifts completeness (0.952 vs 0.873, CI spans 0) over bare agent; honest useful demo, but pilot-scale and no component ablation.
Summary
Dr. Claw is an open-source workspace that wraps existing command-line coding agents (codex and Claude SDK adapters) in a human-in-the-loop orchestration layer for research, rather than introducing another autonomous agent. The layer adds four persistent state objects — Task Graph, Artifact Store, Decision Log, Execution Trace — plus a skill library (58 stage-mapped skills across five stages, 171 in the deployed catalogue) and multi-executor coordination, unifying planning, execution, and writing into one auditable loop. Evaluation holds the executor fixed (codex, gpt-5.4) and compares against the bare agent on three open-ended medical tasks scored against 21 research best-practice elements: pooled completion 0.952 vs 0.873, Dr. Claw winning two tasks and tying one, with gains concentrated in research hygiene (limitations 0.33→1.00, subgroup analysis 0.33→1.00, real citations 0.00→0.67); the bootstrap CI [-0.00, +0.14] includes zero. A single-condition walkthrough shows non-destructive recovery from an induced failure. A retrospective study of seven PhD students associates Dr. Claw with shorter time bands, higher blind-rated quality, and better experience. Main limitations, largely acknowledged by the authors: one replicate per task, no component ablation, a coverage-not-correctness metric, a bare-executor-only baseline, and a single medical domain.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Wrapping a fixed command-line coding agent with four persistent state objects and a skill library raises research completeness under open-ended goals (pooled 0.952 vs 0.873), consistently across three tasks, though the bootstrap CI includes zero.
- The completeness gain comes from research-hygiene skills — limitations section (0.33→1.00), subgroup analysis (0.33→1.00), real citations (0.00→0.67) — not from modeling capability, which saturates in both conditions.
- Only orchestrated runs persist a queryable, recoverable process trail (task graph, execution trace, decision log), enabling in-place recovery from an induced failure with all prior artifacts retained.
- A retrospective three-condition study of seven PhD researchers associates Dr. Claw with shorter completion-time bands, higher blind-rated output quality, and better experience than no-AI or general web/desktop assistants.
Strengths
Calibrated honesty about evidence strength
Section 5.1 reports the pooled delta (+0.079) with a bootstrap CI [-0.00, +0.14] that includes zero; the Limitations section explicitly disclaims ablation power, correctness of outputs, and state-of-the-art-baseline status — rarer candor than is typical.
Well-chosen evaluation regime and deterministic scoring
Open-ended prompts are used precisely where orchestration can add value, the executor and permission profile are held fixed (codex/gpt-5.4, danger-full-access/approval-never), and the 21-element completeness rubric is scored deterministically from produced files with no model-in-the-loop judgment (Section 5.1).
Mechanism-level attribution of the gain
Auditability framed as architecture, not score
Release and replication scaffolding
Weaknesses
Pilot scale with no statistically established effect
No component ablation
Internal numeric inconsistency in the skill library
Human study is retrospective and measurement-asymmetric
Editorial slips and narrow comparison set
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Dr. Claw (arXiv 2609.00365) is a system-demonstration paper for an open-source workspace that wraps existing CLI coding agents (codex and Claude SDK adapters) rather than introducing a new executor. The claimed contributions are threefold: (i) formalizing 'Vibe Research' as a controllable, human-in-the-loop orchestration paradigm; (ii) implementing four persistent state objects (Task Graph, Artifact Store, Decision Log, Execution Trace) plus a skill library (58 stage-mapped skills, 171 in the deployed catalogue) and multi-executor coordination; (iii) a pilot comparison against the bare wrapped executor, complemented by a retrospective human study. The headline claim: holding the executor fixed (codex/gpt-5.4), Dr. Claw scores higher on research completeness (0.952 vs 0.873 pooled) while persisting an auditable, recoverable process trail.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'Vibe Research' as defined in the Dr. Claw paper?
The authors' term for a controllable, human-in-the-loop research paradigm: the researcher states high-level goals, constraints, and acceptance criteria in natural language; AI compiles them into an executable research loop covering literature review, ideation, coding, analysis, and drafting; humans retain research direction, key trade-offs, and final acceptance rather than delegating to a fully autonomous agent.
Is Dr. Claw a new AI agent or a new model?
No. It is an open-source workspace (AGPL-3.0, with GPL-3.0 upstream components) that wraps existing command-line coding agents such as codex and Claude SDK adapters, adding a task graph, artifact store, decision log, execution trace, and a skill library (58 stage-mapped skills; 171 in the deployed catalogue) on top of the wrapped executor.
Does Dr. Claw actually beat a plain coding agent?
In the paper's pilot on three open-ended medical tasks with the same gpt-5.4 executor, pooled completeness was 0.952 vs 0.873, and Dr. Claw was never worse on any task. The 95% bootstrap CI [-0.00, +0.14] includes zero, so the difference is directional, not statistically established; the gains came from hygiene skills such as limitations sections, subgroup analyses, and real citations.
Where is the Dr. Claw code, and can the results be reproduced?
Code is at github.com/OpenLAIR/dr-claw. The appendix provides a replication checklist (fixed repository revision, lockfile, permission profile, stage definitions, blinded rubrics, exported pipeline state). However, exact numbers depend on a proprietary backend model (gpt-5.4 via codex) and single agent runs per task, so exact reproduction is unlikely; the system itself can be installed and run from the repository.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- No truncation detected: the provided text runs through the complete reference list and Appendix B.2 (ending with the replication checklist) in complete sentences, so all cited sections were available for review.
- Figure-embedded numbers (Figures 4, 5, 8) are interleaved by PDF extraction; per-task/per-condition attribution beyond values stated in prose (pooled 0.952/0.873, hygiene pass rates, human-study statistics) was not attempted from the garbled figure text.
- The supplied paper metadata lists 8 authors, while the extracted title block lists 13 (adding Zhiling Yan, Yuxuan Zhang, Yanfang Ye, Philip S. Yu, Lichao Sun); this review reflects the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.