ClaimReceipt: Verifying Evidence Sufficiency and Coverage in Agent Evaluations
cs.AI · cs.CR · cs.MA
Quick verdict
Receipt spec makes agent-eval claims recomputable and coverage checkable; careful preregistration, but validation is self-referential and no code is released.
Summary
The paper introduces ClaimReceipt, a typed receipt specification plus selective verifier that separates two evidentiary questions in agent evaluation: whether a reported claim is recomputable from retained evidence (sufficiency) and whether the retained records cover the committed experiment set (coverage). A manifest signed before ingress commits assignments, protocol fingerprints, and analysis contracts; per-transaction receipts carry canonical commitments (HMAC-SHA256 for low-entropy private values) and replay-tier boundaries. The specification was frozen before implementation. On 1,392 historical buyer-seller records from six retrospective epochs, the CR-2 verifier reproduces all five manually labeled audit verdicts, exactly replays 600 deterministic and 792 post-generation records, renders all 13 field groups non-redundant, and detects 11/11 frozen semantic faults with 0/8 false positives on benign variations. A prospective CR-3 epoch (30 assignments, Qwen2.5-3B, k=1) shows coverage-qualified verdicts: a withheld terminal receipt yields INCONCLUSIVE_COVERAGE, while withheld private openings preserve coverage but make economic claims inconclusive, matching a preregistered prediction. Instrumentation costs 0.021% of inference time and 9.9 KB per transaction. Main limitations: self-referential validation (the authors froze the expected results and supplied the five labels), a single hotel testbed with one model family, designed rather than organic faults, an independent-reader legibility probe that matched only 3/11 verdicts post hoc, and no released code or separately operated trust roles.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Sufficiency and coverage are formally distinct evidentiary properties, and a typed receipt schema bound to a signed pre-ingress manifest makes both machine-checkable per claim (Eq. 1, §3.3).
- On 1,392 historical records the CR-2 verifier reproduces all five manually labeled audit verdicts and exactly replays 600 deterministic plus 792 post-generation claims (Tables 3-4).
- All 13 declared field groups are non-redundant, and 11/11 frozen semantic faults are detected with 0/8 false positives on benign variations (Tables 5-6, §5.4).
- Prospective CR-3 on 30 transactions returns preregistered verdicts under complete, missing-terminal, and missing-private-opening conditions, at 0.021% inference-time overhead and 9.9 KB per transaction (Table 7).
- The authors' own frozen specification fails an independent legibility probe (no valid protocol-conformant run; 3/11 post-hoc agreement), showing it is not yet unambiguous to an independent reader (§5.4).
Strengths
Clean conceptual separation
Eq. 1 defines claim-relative sufficiency over executions, and §2-§3 split sufficiency, coverage, and transport integrity apart — clarifying exactly what hashing/signing cannot decide: which semantic distinctions a claim needs and whether omissions occurred.
Genuine preregistration discipline
v0.3 frozen with a published SHA-256 before implementation (§4.1); E7's signed manifest (08:13:17.826Z) precedes all 30 ingress tickets and carries an OpenTimestamps Bitcoin attestation at height 963833; the missing-opening prediction was hash-frozen (72c554...4c0f) and matched exactly (§5.5).
Honest negative results
Precisely scoped replay claims
Non-redundancy ablation tied to a frozen matrix
Weaknesses
Circular, thin ground truth
Self-authored fault suite
Narrow external validity
Specification legibility not established
No artifact release; single-operator trust
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ClaimReceipt (arXiv 2609.01992) targets two evidentiary questions the authors argue generic logging conflates: sufficiency — whether a reported claim is recomputable from retained evidence, formalized in Eq. 1 as equal evidence projections implying equal claim truth — and coverage — whether retained records constitute the complete committed experiment set, checkable only against a pre-committed manifest. The deliverables are a frozen v0.3 specification (SHA-256 published in §4.1), a two-tier reference verifier (CR-2 retrospective, CR-3 prospective), and an evaluation on 1,392 historical buyer-seller records from the authors' prior A2A commerce audit plus a 30-transaction prospective epoch E7. Headline claims: 5/5 manual audit verdicts reproduced (Table 3); 600/600 deterministic and 792/792 post-generation replays (Table 4); all 13 field groups non-redundant (Table 6); 11/11 frozen semantic faults detected with 0/8 benign false positives; preregistered verdicts under evidence withholding at 0.021% inference-time overhead and 9.9 KB per transaction (Table 7).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ClaimReceipt?
ClaimReceipt is a specification and software verifier for AI agent evaluations. For each claim about an experiment it checks whether the retained evidence is sufficient to recompute the reported number and whether the retained records cover the full set of experiments committed in advance, returning PASS, INVALID, or INCONCLUSIVE per claim.
How well does ClaimReceipt perform?
On 1,392 historical buyer-seller records it reproduced all five manually labeled audit verdicts and exactly replayed 600 deterministic plus 792 post-generation records. It detected all 11 designed semantic faults with zero false positives on 8 benign variations, and in a 30-transaction prospective run its verdicts matched a preregistered prediction. Instrumentation cost about 0.02% of model inference time and about 9.9 KB per transaction.
What are the main limitations of the paper?
The validation is self-referential: the authors wrote the specification, verifier, and fault suite, and the labeled ground truth is only five cells from their own prior study. It was tested on one hotel-style marketplace testbed with one model family; the independent legibility probe failed (3/11 post-hoc agreement); and no code or specification artifact is stated to be publicly available.
Why isn't hash-chaining or signed logging enough for auditable agent experiments?
The paper argues that signatures and hash chains only prove retained bytes were not mutated. They cannot tell you which semantic details a specific claim needs (e.g., hidden buyer profiles, parser outputs, discarded attempts) or whether runs are missing from the committed set — which is what ClaimReceipt's typed claims and pre-committed manifest add.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Table 3 and Table 7 cell alignment is partially garbled by PDF extraction; cell-by-cell readings were cross-checked against the prose in §5.1 and §5.5.
- Figure 1's diagram text is extracted as loose fragments; interpretation relies on its caption.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.