Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents
Jingjie Ning, Shanshan Zhong, Xiaochuan Li, Ji Zeng
cs.MA · cs.AI · cs.SE
Quick verdict
Proposes DCP, a preregistered three-gate audit for AI discovery claims; statistically consistent but self-applied on narrow synthetic tasks.
Summary
The paper introduces the Discovery Certification Protocol (DCP), a preregistration-based audit framework for claims produced by AI research agents. DCP splits one numerical outcome into three separable evidential questions. Gate 1 validates useful improvement on a sealed evaluation. Gate 2 gives fresh matched agents the registered background, initial observations, and captured Web bytes while withholding the target run's research history; any valid method reaching the registered score threshold supplies a recovery witness that vetoes certification. Optional Gate 3 estimates the effect of truthful versus neutral feedback from a shared checkpoint, with independent null calibration. DCP Core requires adequate controls, zero recoveries, and a finite-sample recovery bound; DCP Evidence adds a calibrated feedback effect. Two full audits—SQLite index optimization (DeepSeek-v4-flash) and virtual catalyst optimization (DeepSeek-v4-pro)—each recorded 0/96 recoveries (upper bound 0.0468), 30/30 versus 0/30 truthful/neutral recoveries, and passing 60-pair null studies; device, knapsack, and affine cases exercise the Core, recovery-refuted, and audit-incomplete decisions. A deterministic LLM-free verifier replays decisions from frozen evidence bundles. Main limitations: every audit was produced by the authors themselves (certificates remain issued=false, countersigning unexercised); tasks are small and synthetic; certificates are scoped to one challenger model and budget; and key thresholds are registered but thinly justified.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A useful sealed score, the absence of an alternative recovery route, and a beneficial feedback effect are three distinct evidential questions, and any valid method reaching the registered score threshold (score ≥ x−ε) constitutes a recovery witness that vetoes certification.
- With preregistration, adequate positive controls, and n independent fresh episodes, zero recoveries yields a finite-sample upper bound on episode recovery probability (1−0.01^(1/96)=0.0468 in both complete audits, below ρ=0.05), supporting DCP Core.
- Truthful experimental feedback shows a large checkpoint-conditional effect over a null-calibrated neutral channel (30/30 vs 0/30 recoveries, 99% interval [0.6379, 1] against the required 0.51 margin), supporting DCP Evidence in both complete audits.
- The full decision — Core, Evidence, recovered, or audit-incomplete — is deterministically recomputable offline by an LLM-free verifier from frozen, hashed evidence bundles.
Strengths
Exceptional internal statistical consistency
Reported bounds reconcile exactly when recomputed: 1−0.01^(1/96)=0.0468381 (C.1, C.2), 1−0.02^(1/80)=0.0477239 (C.3/C.4), control-recall bounds 0.005^(1/45)=0.8889268 and 0.005^(1/60)=0.9155 (C.4), and even the intentionally underpowered affine case yields 0.005^(1/2)=0.0707 (C.6). Episode sizing follows the stated formula ⌈log α/log(1−ρ)⌉=90 for α=0.01, ρ=0.05, with 96 registered (A.3).
Conceptually clean separation of three evidential questions
Utility (Gate 1), alternative-route recovery (Gate 2), and checkpoint-conditional feedback effect (Gate 3) are given distinct estimands, decisions (Core vs Evidence), and error budgets (A.2), avoiding the common conflation of 'high score' with 'discovery'.
The veto demonstrably fires on a designed negative
Replayability architecture with disclosed costs
Weaknesses
All audits are self-produced; the trust boundary is unexercised
External validity untested on realistic tasks
Gate 1 is nearly vacuous in the flagship audit
Recovery bound attaches to a low-diversity challenger distribution
Key thresholds registered but unargued
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper proposes the Discovery Certification Protocol (DCP), a preregistration-style audit framework for claims produced by AI research agents (§1, §3). Its central argument: a high sealed score answers only one of three distinct evidential questions about a claimed discovery — whether the artifact is useful (Gate 1); whether the research history was necessary, tested by giving fresh agents the registered background K, initial observations E0, and the target run's captured Web bytes Wobs while withholding run history L⋆ (Gate 2); and whether truthful experimental feedback causally improves outcomes over a neutral policy (optional Gate 3). Two decisions are defined: DCP Core (adequate controls, zero qualified recoveries, finite-sample bound p_upper ≤ ρ = 0.05 on a fresh episode's recovery probability) and DCP Evidence (Core plus a null-calibrated feedback effect exceeding registered margins δevidence + δsham = 0.51). The authors claim the decisions are portable across research domains, admit any valid alternative method as a recovery witness, and are deterministically replayable offline by an LLM-free verifier from frozen evidence bundles. Historical priority is explicitly out of scope (§3.1, Appendix D).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Discovery Certification Protocol (DCP)?
DCP is a proposed audit framework for claims made by AI research agents. Instead of trusting a reported score, an auditor (1) validates the artifact on a sealed evaluation, (2) gives fresh AI agents everything the original run started with — including captured web pages — but not its research history, and checks whether any of them can reach the same score by a different route, and (3) optionally measures whether truthful experimental feedback actually helps compared to uninformative feedback. Certificates are issued only under preregistered rules and can be recomputed offline from frozen evidence bundles.
What is the difference between DCP Core and DCP Evidence?
DCP Core certifies that a useful result survived a sealed test and that, in a registered number of fresh agent episodes with the research history withheld, no alternative route reached the score threshold (with a statistical upper bound on recovery probability). DCP Evidence adds a causal claim: from a shared checkpoint, truthful experimental feedback produced better outcomes than a calibrated neutral (uninformative) feedback policy by at least a registered margin.
Did the DCP audits find that AI agents copied or leaked prior work?
No. In the paper's two full audits (SQLite index optimization and virtual catalyst optimization, using DeepSeek models), zero of 96 fresh agent episodes reached the recovery threshold in either task, with an upper bound of 4.68% on the per-episode recovery probability. One deliberately designed case (multidimensional knapsack) did produce a recovery, demonstrating that the protocol's veto can fire when an alternative route exists.
Is DCP code or data available?
The paper states that installable dcp-audit and dcp-harness packages, task-specific evidence producers, example evidence bundles, and figure-generation scripts are released (PyPI/GitHub links on the first page), and that decision replay is deterministic and offline. Note the authors report all certificates as issued=false: independent provenance verification and countersigning are described but not yet exercised.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Inline math is flattened by PDF extraction (e.g., Eq. 3 renders as '1 − α1/n' and the clip formula loses its fraction layout); superscript forms were reconstructed from Appendix A.3 and are noted as reconstructions where cited.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.