Qiushi Engine on AstaBench E2E-Bench-Hard
Wenhao Li, Shuxing Yang, Fujia Chen, Jincheng Mi, Yuang Pan, Rui Zhao, Zichen Li, Junyao Wu, et al.
cs.AI · cs.SE
Quick verdict
Reports 0.816 on AstaBench E2E-Bench-Hard, 3.3x best perfect-task rate; disciplined self-evaluation, but attribution and judge validity rest on self-reported evidence.
Summary
This report analyzes Qiushi Engine v0.8's performance on all 40 E2E-Bench-Hard tasks in AstaBench, a rubric-based benchmark of end-to-end autonomous research. The closed-source system ran through the authors' web UI with DeepSeek deepseek-v4pro-preview as backend; exported outputs were scored with the pinned official scorer (claude-sonnet-4-6 judge). The official leaderboard records 0.816 at $15.209 per task; local recomputation yields 0.8159 ± 0.0187 (median 0.833). Across 507 scored rubric items, 416 (82.1%) were met. Four tasks satisfied every item — a 10% perfect-completion rate, about 3.3 times the ~3% best rate reported for AstaBench's published agents — although four successes imply a wide confidence interval (~4%–23% Wilson). Misses concentrate in statistical support (16 samples), core components (11), metrics (10) and ablations (8). Main limitations: no component ablations or backend comparison (closed engine limits attribution), no human validation of the LLM judge, token statistics transcribed from UI screenshots, and Meta-Trace process analysis missing parsed states for at least six samples.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Qiushi Engine v0.8 (DeepSeek deepseek-v4pro-preview backend) achieves an official AstaBench E2E-Bench-Hard score of 0.816 at $15.209 per task, locally recomputed as 0.8159 ± 0.0187 over all 40 test tasks.
- Qiushi perfectly completes 4/40 tasks (10%), which the paper reports as 7 percentage points and ~3.3× above the ~3% best full-task completion rate reported for AstaBench's official agents.
- 416 of 507 rubric items (82.1%) were judged met; failures concentrate in statistical support (16 samples), external-resource-dependent core components (11), specified metrics (10), and ablations (8), rather than being uniform.
- Score-loss mechanisms are identifiable and distinct: resource-allocation pressure, external dependency cascades (idea_16), technical misinterpretation (idea_14), and premature conclusion (idea_12).
Strengths
Complete test-set coverage with a verifiable score chain
All 40/40 E2E-Bench-Hard test tasks were run; the locally recomputed 0.8159 ± 0.0187 (Appendix E, six consistency checks) matches the official 0.816, and the archive documents per-item scorer verdicts, SHA-256 inventories (§4.4, Appendix C) and a five-script reproducible extraction pipeline.
Careful descriptive discipline
Macro (81.59%) vs micro (82.1%) averages are distinguished (§8.1); every correlation (tokens r=0.001, steps r=−0.010, repair fraction r=0.24) is explicitly labeled non-causal; §4.6 maps each claim to what its evidence source can and cannot support.
Actionable failure taxonomy
Explicit model-role separation and telemetry honesty
Weaknesses
Attribution is structurally impossible from this evidence
Headline perfect-task-rate claim lacks stated uncertainty
No validation of the LLM-as-judge
Process analyses rest on incomplete self-generated traces
Cost-efficiency is weak and weakly measured
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a self-evaluation report: the Qiushi Engine team documents its own system's v0.8 run over **all 40 test tasks** of AstaBench E2E-Bench-Hard (arXiv:2609.08196), with DeepSeek deepseek-v4pro-preview as backend. Headline claims: official leaderboard score **0.816 at $15.209/task**, local recomputation **0.8159 ± 0.0187** (median 0.833, SD 0.119, range 0.50–1.00); **416/507 rubric items met (82.1%)**; and **4/40 (10%) tasks with every rubric item satisfied**, framed as ~3.3× the ~3% best rate reported in the AstaBench paper for its own agents. Sections 2 and 4 carefully explain the scoring mechanism and disentangle three model roles (agent, replay label, judge), and §4.6/§8.7 are unusually candid about what the evidence cannot establish.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What score did Qiushi Engine achieve on AstaBench E2E-Bench-Hard?
The official AstaBench leaderboard records 0.816 (81.6%) at a standardized cost of $15.209 per task, with DeepSeek deepseek-v4pro-preview as the agent's model backend. The paper's local full-precision recomputation over all 40 test tasks is 0.8159 ± 0.0187 (median 0.833), and the next-highest displayed entry (Ai2's customized ReAct with Claude Opus 4.7) scores 65.5%.
What does an E2E-Bench-Hard score of 0.816 actually mean?
It is not a quality percentage. Each task carries roughly 8–17 binary rubric items asking whether concrete research actions (dataset preparation, baseline implementation, statistical tests, ablations) are supported by the agent's report, code, and artifacts together. 0.816 means about 82% of those concrete items were judged satisfied; misses here concentrate in repeated runs, statistical testing, ablations, and unavailable external models — not in core implementation.
How does Qiushi Engine compare with other agents on this benchmark?
In the September 2026 leaderboard snapshot used by the paper, Qiushi (81.6%, $15.209/task) leads Ai2's customized ReAct with Claude Opus 4.7 (65.5%, $11.452) and Asta Panda (56.5%). It is also the costliest listed configuration, and a 'Faker' baseline that fabricates plausible outputs still satisfies 25.4% of rubric items. Perfect completion of all rubric items per task was 10% for Qiushi versus roughly 3% reported as the best rate for the benchmark's own agents.
Who makes the models used in this evaluation and what roles do they play?
Three model roles are distinct: DeepSeek's deepseek-v4pro-preview was the backend powering the Qiushi agent during the research runs; Anthropic's claude-sonnet-4-6 was the rubric judge used by the official scorer to grade outputs; and a mockllm label appears only in offline replay packaging of cached outputs. The score applies to the whole Qiushi–model–tools configuration, not to DeepSeek alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The text is extracted from a single pre-print v1 PDF by the submitting authors; no external verification of the underlying archive was possible in this review.
- The authors list in the supplied metadata (9 names) does not match the PDF byline (17 names including corresponding author Yihao Yang); this review treats the byline as authoritative.
- No evidence of truncation: references, appendices A–E, and Table 12 are complete in the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.