Evaluating Verified Autonomy in Quantum Engineering
Naixu Guo, Changhao Li, Siyu Cheng, Qicheng Tang, Binzhao Luo, Bikun Li, Yuxuan Du, Shihao Ru, et al.
quant-ph · cs.AI
Quick verdict
Evidence-bound quantum-agent benchmark exposes a 78%-to-3% reliability spread across 17 systems; strong design but simulation-only, n=3 trials, harness-confounded.
Summary
The paper presents Quantum-Harbor, a sandboxed virtual quantum laboratory built on the Harbor agent-evaluation framework, and QIQCBench, a 49-task benchmark authored and reviewed by domain experts. LLM agents interact through MCP tool interfaces with simulated quantum hardware (36 'qtypes': transmon, trapped ion, neutral atom, spin/NV), while a hidden sidecar records every experiment in an evidence log; the grader recomputes numerical claims from raw records or replays calibrations on fresh hidden data, targeting 'verified autonomy' - conclusions supported by execution evidence. Tasks span six categories and five quantum-stack layers; 39 are pass-fail, seven are Elo-ranked races, three are excluded test cases. Seventeen frontier systems (OpenAI, Anthropic, DeepSeek, Z.ai, Alibaba, Tencent, Moonshot, ByteDance), each with three attempts per task under a 3-hour budget, score 78% (GPT-5.6-sol) down to 3% (GLM-5.2); racing Elo spans 1,694-582. No-submission dominates failures (937 of 1,510), and only 34% of system-task pairs that ever pass pass all three attempts. Limitations: all runs are simulation-only, different vendors used different agent harnesses, per-task samples are thin (n=3), and the extracted text states no public release of benchmark artifacts.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing benchmarks grade final outputs without checking whether results are supported by the agent's execution record; Quantum-Harbor's evidence log and fresh-data replay make 'verified autonomy' measurable.
- QIQCBench comprises 49 expert-authored tasks covering six categories and five quantum-stack layers, with hidden parameters and thresholds kept out of all published material.
- Across 17 frontier agentic systems, verified pass rates span 78% to 3%, with no-submission the largest failure mode (937 of 1,510 failures), exposing a gap between capability demonstration and reliable operation.
- The benchmark separates systems that perform alike on other frontier agentic benchmarks (AutomationBench-AA, GDPval-AA).
- Failures arise mainly from experimental judgment - stale priors, budget mismanagement, uninformative sampling - suggesting harness-level improvements could substantially raise reliability.
Strengths
Evidence-bound grading design
The hidden sidecar's evidence log plus a grader that recomputes reported numbers from raw records and replays calibrations on fresh hidden draws (Fig. 2b,c) - e.g., the lattice-surgery decoder verified on fresh syndrome records - directly targets the fabrication failure mode the authors cite (refs 56, 57), rather than grading final answers alone.
Demonstrated discriminative power
Systems within 54.0-62.2% of each other on AutomationBench-AA and within 31 Elo on GDPval-AA spread across nearly the full range here (78% to 3% pass rate; Elo 1,694-582 over 7 racing tasks), supporting the claim that this benchmark separates frontier systems other agentic benchmarks do not.
Task-construction and honesty discipline
Informative failure taxonomy
Breadth with transparent negative findings
Weaknesses
Simulation-only evidence
Harness-model confound
Thin statistics for rankings
Threshold and difficulty provenance
Release and contamination unaddressed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is QIQCBench and Quantum-Harbor?
Quantum-Harbor is a sandboxed virtual quantum laboratory where AI agents plan and run quantum experiments on simulated hardware through tool interfaces, with every action logged for verification. QIQCBench is the 49-task, expert-authored benchmark built on it, spanning calibration, control, error correction, compilation, simulation, sensing, and networking.
Which AI model scored best on QIQCBench?
OpenAI's GPT-5.6-sol led with a 78% pass rate on the 39 pass/fail tasks (each system gets 117 trials: 39 tasks x 3 attempts), followed by GPT-5.6-terra at 57% and Claude Opus 4.8 at 51%. The lowest was GLM-5.2 at 3%.
Why do AI agents fail at quantum engineering tasks?
The largest failure mode is running out of the 3-hour time budget without submitting an answer (937 of 1,510 failures) - agents keep running experiments rather than wrapping up. Most remaining failures are scientific: the grader rejects the physics, often due to stale assumptions, uninformative sampling, or budget mismanagement.
Was QIQCBench evaluated on real quantum hardware?
No. All reported evaluations ran on simulated quantum hardware with hidden noise parameters. The framework claims support for live connections to IBM, Rigetti, IonQ, and Quantinuum devices plus a replay mode, but this study is simulation-only.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are rendered as text in the extraction; per-task pass matrices (Fig. 3, Fig. 4d) could not be inspected directly.
- Supplementary Material (per-task tables, audit protocols, simulation implementation) was not included in the extracted text.
- No data/code availability statement appears in the extracted portion; one may exist in a section not captured by extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.