One note in three: a verified census of three deployed AI scribes, and the instrument that counted it
Sebastian Fox, Luke Markham, Ryan Lail, Michael Karotsieris
cs.CL · cs.AI · cs.CY
Quick verdict
One note in three (31.3%) carries a verified failure from AI scribes; exemplary instrument self-audit, but anonymized products limit independent checks.
Summary
The authors audit three commercial ambient AI scribes on a shared corpus of 142 consultations (57 recorded UK primary-care encounters, 45 US ambulatory simulations, 40 authored scenarios), yielding 565 notes. Twelve LLM discovery passes proposed 13,678 candidate errors; 5,898 cleared an importance filter and faced an adversarial verification panel of two model families with a tiebreak model; 618 findings survived. The headline is that 31.3% [27.0, 35.6] of notes carry at least one verified failure (24.8% [20.8, 29.0] excluding invented identity and dates, the two classes a patient record might prefill), concentrated in allergy and medication information, invented patient identity, and history documented as physical examination on telephone consultations. A controlled re-review of a 115-note sample shows the review instruction alone moves candidate verification from 9.3% to 79.0%, and the reviewing model family roughly doubles the note-level flag rate, so published audits' divergent omission shares (54-86%) are plausibly instrument-driven. Blinded clinician sittings upheld 20/21 and 12/12 sampled findings. Key limitations: no clinician-written comparator, anonymized products with withheld notes, a single stochastic run, unmeasured discovery-stage sensitivity, and most human validation performed by a physician author.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- 31.3% [27.0, 35.6] of 565 notes from three deployed AI scribes carry at least one adversarially verified failure (24.8% [20.8, 29.0] when the two classes a patient record would prefill — invented identity and dates — are removed).
- With model, evidence and settings fixed, the review instruction alone moves the share of candidates verified from 9.3% to 79.0%, and the reviewing model family roughly doubles the note-level flag rate (27.8% vs 54.8% on the same sampled notes).
- The divergent omission shares of published scribe audits (54-86% of errors vs this census's 23.1%) are consistent with instrument differences of the measured size; on the one shared note-level denominator the census matches Taylor et al. (15.4% [12.0, 19.0] vs 18%).
- Verified failures concentrate in allergy and medication information, invented patient identity, and history written up as physical examination on telephone consultations that could contain none.
- The complete counting instrument — prompts, model pins, seeds, verdicts and pipeline — is released and re-runnable, and every rate is a floor under the published strict standard.
Strengths
Controlled decomposition of the counting instrument itself
Section 4.1 / Table 3 re-reviews 1,295 candidates from 115 drawn notes under four standards with everything else fixed: strict 9.34% [7.88, 11.05] vs lenient 79.00% [76.69, 81.13] on the same model, with exact McNemar 902 discordant vs 0 (p = 5.9e-272) and verified strict nesting (the lenient standard keeps 100% of panel survivors). This is a genuinely novel, falsifiable measurement of LLM-judge sensitivity that no compared audit performs.
Adversarial verification with full self-accounting
Appendix A.3.1 publishes the panel's internals: the harsher skeptic (also the discovery model) refutes 90.6% of its own proposals, cross-family unaided agreement is 86.2% with Cohen's kappa 0.44, and every one of the 812 tiebreak calls is recorded — the gpt-5.4 tiebreak sided against its own family's skeptic 531 times vs 281, directly answering the self-leniency and shared-provenance objections.
Blinded human checks on disjoint samples, honestly reported
Quantified, falsifiable treatment of its own weak points
Full instrument release enabling re-runs
Weaknesses
The audit's object is not independently verifiable
No clinician-written comparator arm
Discovery funnel only half-audited
Human validation is thin and conflict-adjacent
Design confounds the cross-product comparison the headline rests on
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This arXiv preprint (2608.31017, cs.CL) audits three anonymized commercial ambient AI scribes on one shared corpus — 142 consultations: 57 PriMock57 UK primary-care recordings, 45 ACI-Bench US ambulatory encounters, 30 trap-seeded and 10 trap-blind authored scenarios — producing 565 notes (Table 1). Its claims are twofold: a census (31.3% [27.0, 35.6] of notes carry at least one verified failure; 24.8% [20.8, 29.0] excluding the two classes a patient record would prefill), and a methodological thesis that failure rates are joint products of scribes and counting instruments, demonstrated by decomposition (9.3% strict vs 79.0% lenient candidate verification; 27.8% vs 54.8% note-level flags by reviewer family). The abstract's headline is honest about the first number but leads with the strict end of a range the paper itself shows spans 28-97% of notes.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What did this study find about AI scribe error rates?
On 142 shared consultations, 31.3% of the 565 notes written by three commercial ambient AI scribes contained at least one error that survived an adversarial AI verification panel explicitly built to refute findings (24.8% when excluding invented patient identity and dates). Sampled findings were upheld by human clinicians (20/21 and 12/12). Common failures were allergy and medication errors, invented patient names and sexes, and physical-exam findings documented for telephone consultations where no exam occurred.
Which AI scribe products were tested in this paper?
The products are not named. They are anonymized as Scribes A (accessed via API, producing two note templates per consultation), B and C (which received replayed audio). The authors state they will not identify them and withheld the notes themselves over terms-of-service concerns; only the 618 verified findings, prompts and pipeline are released.
Can the 'one note in three' error rate be trusted?
It is a floor under a deliberately strict verification standard, not a true rate. The paper's own controlled experiment shows the review instruction alone moves the share of verified candidates from 9.3% to 79.0%, and an equally strict reviewer from the other model family reads roughly twice the note-level rate. The paper's core argument is that any error rate must be published alongside the instrument that produced it.
Is the code and data from this study available?
Yes, partly. All 618 verified findings with transcript-side evidence quotes, panel verdicts, prompts, model versions, run manifests and the re-runnable pipeline are released (HuggingFace dataset ComposoAI/OmissionBench, GitHub composo-ai/omission-bench, Zenodo DOI). The vendor notes are withheld, so per-product results cannot be independently re-derived without vendor accounts.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with two '[...sampling gap...]' breaks; portions of Sections 3.4-4.1 (including the Section 3.5 retracted-device case and the opening of 4.1), the tail of the references, and parts of Appendices A.4.3/A.5 were skipped. Judgments above rest on the extensive visible text on both sides of the gaps.
- The metadata abstract is truncated mid-sentence ('alone at tha'); full abstract content was recovered from the paper's Introduction, which restates it.
- Figures 1-3 and Tables 2-5 are partially garbled by extraction (caption/body interleaving); cited numbers were cross-checked between body text and appendices where duplicated.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.