Verifiable by Construction: Claim-Level Evaluation of Verbatim Citation in Clinical Question Answering
Jiashuo Zhang, Yuling Chen, Yvonne Commodore-Mensah, Michael Oberst
cs.CL
Quick verdict
Solid harness decomposing 12 LLMs' verbatim-citation clinical QA into coverage/verbatim/support stages; credible findings, but no ablations or CIs.
Summary
The paper presents an end-to-end evaluation framework asking whether LLM answers to clinical questions are 'verifiable by construction': every factual claim should carry an inline verbatim quote that by itself substantiates the claim. The authors build a retrieval-based clinical QA system over four AHA/ADA practice guidelines, generate 222 synthetic clinical questions with gpt-4o-mini, freeze the retrieval stage (LLM-guided selection via deepseek-v4-flash), and evaluate twelve LLMs from five vendors under identical prompts. An automated pipeline decomposes each answer into stages — claim filtering, citation coverage, verbatim compliance (exact/normalized/elided matching) and claim support judged by an LLM — aggregated into a claim funnel whose endpoint is the certified claim rate (CCR). Nine of twelve models attach verbatim quotes to over 90% of claims, yet none fully substantiates all claims: best CCRs are 78.3% (gpt-5-mini), 75.6% (gpt-5.4) and 72.0% (deepseek-v4-pro), while claude-opus-5 reaches 98.0% verbatim coverage but only 37.1% CCR, mostly through 'unsupported additions'. The judge was validated against two human annotators (κ 0.81–0.86) and a second LLM judge (κ 0.803). A recovery probe shows most missing evidence already sits in the sections models read (claude-opus-5 rises to 85.1% supported). Limitations: synthetic question set, a single system prompt without ablations, author annotators, and no confidence intervals on model comparisons.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Nine of twelve mainstream LLMs attach verbatim-compliant quotes to over 90% of their claims from prompting alone (Table 1).
- No model fully substantiates all claims; the best certified claim rates are 78.3% (gpt-5-mini), 75.6% (gpt-5.4) and 72.0% (deepseek-v4-pro) (Table 1).
- claude-opus-5 achieves 98.0% verbatim-compliant quote coverage but only 37.1% CCR, dominated by 'unsupported addition' shortfalls (Tables 1–3, Figure 4).
- Most missing evidence already exists in the sections models read; recovery raises claude-opus-5 from 37.1% to 85.1% supported (Finding 3, Table 11).
- The deepseek-v4-flash judge agrees with two human annotators at κ 0.858/0.810 and with a second judge (gpt-5-mini) at κ 0.803.
Strengths
Stage-decomposed funnel cleanly localizes failures
Separating citation coverage, verbatim compliance and support lets the paper attribute end-to-end gaps: claude-haiku-4.5 fails at verbatim quoting (69.0% VCR, Table 2), while claude-opus-5 quotes perfectly (100.0% VCR) yet fully supports only 37.8% of its 645 cited claims (Table 3) — two distinct failure modes a single metric would blur.
Deterministic verbatim checks plus validated LLM judges
Exact/normalized/elided matching tiers with explicit normalization rules (Appendix A.6) remove judge discretion from quoting; the support judge is validated against two blinded annotators (inter-annotator κ 0.853; 0.858/0.810 vs judge on 70 stratified pairs, Appendix A.13) and a second judge, gpt-5-mini (κ 0.803).
Retrieval confound controlled and benchmarked
Recovery probe converts a negative result into a diagnosis
Full transparency of the harness
Weaknesses
Judge–generator identity overlap
Unvalidated synthetic question set with circular ground truth
Single frozen prompt, no ablations
No uncertainty quantification
Config comparability and domain narrowness
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper evaluates whether LLM clinical QA answers can be "verifiable by construction": every factual claim carries an inline verbatim quote that alone substantiates it, so a clinician can verify without opening the source. The scope is deliberately verifiability, not correctness. Twelve LLMs from five vendors answer 222 synthetic questions built from four AHA/ADA guidelines (2019 primary prevention, 2025 hypertension, 2026 cholesterol, 2026 ADA Standards of Care). Central claims: (1) nine of twelve models attach verbatim quotes to over 90% of claims from prompting alone; (2) no model fully substantiates everything — best certified claim rates (CCR) are 78.3% (gpt-5-mini), 75.6% (gpt-5.4) and 72.0% (deepseek-v4-pro); (3) claude-opus-5's 98.0% verbatim coverage collapses to 37.1% CCR, dominated by "unsupported addition" shortfalls; (4) most missing evidence already sits in the sections models read (claude-opus-5 recovers to 85.1% supported).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'verifiable by construction' mean in this paper?
It describes answers designed so that every factual claim carries an inline quote copied verbatim from the source clinical guideline, where the quote alone is enough to confirm the claim — the reader never needs to open the cited document to check it.
Which LLM was best at citing clinical guidelines with verbatim quotes?
On the certified claim rate (claims carrying a verbatim quote that fully supports them), gpt-5-mini led at 78.3%, followed by gpt-5.4 at 75.6% and deepseek-v4-pro at 72.0%. No model certified all its claims, and three lightweight models (gpt-4.1-mini, claude-haiku-4.5, glm-4.7-flash) fell below 25%.
Why does claude-opus-5 score only 37.1% despite quoting accurately?
Its quotes are verbatim (100% citation-level compliance) but usually cover only part of each claim — mostly 'unsupported additions', where the claim adds details the quote never states. When a separate LLM re-searches the sections the model already read, 85.1% of its claims find full supporting evidence.
Is the dataset or code from this paper public?
The authors state that code and the 222-question synthetic dataset will be released at github.com/oberst-lab/verifiable-by-construction. The underlying AHA/ADA guideline texts are licensed publications (Circulation, Diabetes Care) and are not republished.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1, 3, 5, 8 and 10 are garbled in extraction; their content was assessed from captions and surrounding text only.
- Code/data availability ('will be available' at a GitHub URL) is stated in the paper but not verifiable from the text.
- Appendix page-break artifacts exist and minor prose may be lost, but all headline tables (1–11) extracted intact.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.