Clean Scores, Buried Evidence, and Confident Wrong: A Receipt-Based Audit of Frontier Agentic QA
cs.IR · cs.AI · cs.CL · q-fin.GN
Quick verdict
Buried-evidence audit of six frontier agents: accuracy drops, cost rises; rare statistical candor, but n=41 and grader sensitivity cap the claims.
Summary
The paper introduces room02, a synthetic 41-item finance-and-insurance 'data room' benchmark with author-verified provenance, and runs one flagship model from six labs (OpenAI, Anthropic, Zhipu, Alibaba, Google, Meta) under clean (gold documents only) versus buried (full room with noise) conditions, three seeds per arm, eliciting forced-declaration JSON answers with confidence and evidence files. Buried evidence reduced accuracy (deltas -4.1 to -17.8 pp), raised tool calls 1.6-3.2×, and raised cost per correct answer 4.7-7.0× (1.6× for Meta), while models stated high confidence on most wrong answers (72% at ≥80 on a 0-100 scale). The paper also reports a shallow image-based chart quiz covering five of six labs, and an n=1 production fabrication incident in which Fable 5 mixed accurate tables with fabricated structural claims. Distinctively, the author audited the frozen containment-based grader: two items were ruled defective, one manufactured spurious lab differences, and after exclusions only two of six labs' accuracy drops (Alibaba, Google) clearly exclude zero by item-level bootstrap; no model ranking is claimed. Main limitations: a small synthetic corpus (41 items) designed by the single author, frozen scores questioned by the author's own audit, withheld raw result files, and independent human verification of the claim ledger still pending.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Burying evidence in a full data room instead of mounting only gold documents reduces frontier-agent accuracy (-4.1 to -17.8 pp point estimates) and multiplies tool calls (1.6-3.2×) and cost per correct answer (4.7-7.0× for five of six labs).
- Self-stated confidence does not flag errors: 72% of wrong buried-arm answers were stated at ≥80 confidence on a 0-100 scale, and calibration did not prevent a documented production fabrication (Fable 5, n=1).
- Agentic evaluation needs claim-level receipts — statement-level provenance a third party can verify against frozen files — plus condition-aware scoring and human-adversarial verification.
- The grading instrument shapes conclusions: the frozen containment grader contained two defective items and one that manufactured inter-lab differences, and only 2 of 6 labs' burial effects exclude zero on the cleaned population.
Strengths
Exceptional statistical self-scrutiny
§8 separates frozen scores, human-reviewed corrections, and agent-provisional re-scorings; it reports that the 4-of-6 significant burial effects shrink to 2-of-6 (Alibaba -13.7 pp, Google -17.9 pp) after excluding defective items, that the two indices' correlation spans -0.47 to +0.77, and explicitly refuses to rank models.
Provenance infrastructure as a contribution
A 420-row claim ledger maps every table cell to a file and locator; hash manifests self-verify (64/64 OK, D.3); tables regenerate byte-identically on independent re-run (D.2); a synthetic-layer Zenodo deposit (CC-BY-4.0) is published (D.4).
Grader audit is instructive beyond this paper
Multi-metric design reveals hidden behavior
Honest depth-label discipline
Weaknesses
Small single-author synthetic corpus
Headline effect is fragile
Frozen scores rest on a grader the paper itself demotes
Reproducibility is partial by construction
Cost comparability is shaky
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper measures what happens to frontier agentic QA when evidence must be found rather than handed over. Its instrument is room02, a synthetic finance-and-insurance data room of 41 items with declared hop counts 2-5 (d2=8, d3=9, d4=17, d5=7), each authored with known provenance. One flagship model from each of six labs — GPT-5.6-sol (OpenAI), Fable 5 (Anthropic), GLM-5.3 (Zhipu), Qwen3.8-Max (Alibaba), Gemini-3.1-Pro (Google), Muse-Spark 1.2 (Meta) — runs under clean (gold documents only) and buried (full room with noise) conditions, three seeds per arm (123 item-seeds), answering via a forced-declaration JSON schema (answer, confidence 0-100, evidence_files). Central claims: (1) burial cuts accuracy and multiplies tool use and cost; (2) stated confidence fails to flag errors; (3) agentic evaluation needs claim-level receipts rather than answer-level scores; (4) the grading instrument itself can change which model looks better. Secondary elements: a shallow image chart-quiz (Table 1b; five of six labs, plus Grok-4.6 outside the panel) and an n=1 documented fabrication by Fable 5 (Box 1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What happens when AI agents must search for evidence instead of being handed it?
In this study, moving evidence from a clean document set into a full synthetic 'data room' lowered accuracy by 4-18 percentage points, raised tool calls 1.6-3.2×, and raised cost per correct answer roughly 5-7× for five of the six frontier models tested (Meta was the outlier at 1.6×).
Does the paper say which AI model is best at document work?
No. The author explicitly reports no model ranking: with 41 items × 3 seeds, essentially every pairwise gap between labs falls within sampling noise, and orderings change depending on which scoring variant is used.
What is a 'receipt' in AI evaluation?
In this paper, a receipt is verifiable statement-level provenance: each claim is traced to a specific file and locator that a third party can check against frozen documents — unlike a model's own citation strings or confidence scores, which cannot be independently checked on their own.
Is the room02 benchmark publicly available?
Partly. The synthetic layer (configs, scripts, outputs, manifests) is on Zenodo under CC-BY-4.0, but the raw model-result files and the fabrication-case transcript are withheld, so the exact numbers regenerate only from the author's private frozen folder, not from the public deposit.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-9 are not analyzable from the extracted text; all quantitative assessment rests on tables and prose.
- The paper's internal integrity claims (hash manifests, byte-identical regeneration, ledger verification) are taken as reported and could not be independently checked.
- Extraction appears complete through Appendix D, but the Zenodo deposit contents were not inspected.
- Review covers v1 (14 Sep 2026) only.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.