VERGE: Verification-Enhanced Refinement for Grounded Extraction of Early-Onset Colorectal Cancer Symptoms in Clinical Notes
Nikkie Hooman, Monarch Nigam, Amy E. Hughes, Rasmi G. Nair, Mehak Gupta
cs.CL
Quick verdict
Bounded verify-refine loop cuts false positives in EOCRC note extraction (P .764→.849, MCC .730); credible, but single-annotator labels limit clinical reach.
Summary
The paper presents VERGE, a four-agent LLM workflow that extracts six colorectal red-flag symptoms plus family-history risk status from free-text clinical notes for early-onset colorectal cancer (EOCRC). An Extractor combines UMLS terminology, MedCPT retrieval with reciprocal rank fusion and GRADE-based prioritization to propose labels with quoted evidence; a Claim Composer standardizes them; a Verifier checks textual grounding and clinical validity; a Refiner corrects flagged claims within a bounded five-round loop, escalating unresolved cases to human review. On 4,033 clinician-labeled note–finding pairs (578 notes, 326 patients, Parkland Health), VERGE raised precision from 0.764 to 0.849 and MCC from 0.681 to 0.730 over the single-agent Extractor with recall essentially unchanged (0.705 to 0.702); a patient-cluster bootstrap shows the precision, F1 and MCC gains are reliable. A rule-based medspaCy+ConText baseline reached similar precision (0.846) with lower recall. Error analysis shows misattributed rather than fabricated evidence dominates. Limitations: single-clinician gold labels without inter-rater agreement, a single safety-net institution, and unstable family-history performance (F1 0.33 to 0.29) on only 18 positive cases.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A bounded Verifier–Refiner loop raises precision from 0.764 to 0.849 and MCC from 0.681 to 0.730 over a single-agent Extractor while recall is essentially unchanged (0.705 to 0.702).
- The dominant hallucination failure in EOCRC symptom extraction is misattributed genuine evidence (60.5% of flagged claims fail clinical validity), not fabricated citations.
- Most flagged claims resolve autonomously; only 1.5% of claims (61 of 4,033) require human review.
- The verification benefit is not backbone-specific: a Ministral-3-14B variant reaches MCC 0.723 versus 0.730 for Llama-3.1-8B.
Strengths
Appropriate clustered significance analysis
Table VI's patient-cluster bootstrap with paired within-resample differences gives 95% CIs: precision +0.085 (0.057–0.112), F1 +0.035 (0.020–0.051), MCC +0.049 (0.030–0.069) all exclude zero while recall −0.003 (−0.018–0.013) spans it — matching the claimed precision-at-flat-recall design rather than reverse-engineering it.
Informative error taxonomy with a worked example
Table V shows clinical-validity failures (222 of 367 flagged claims, 60.5%) dominate grounding failures (67 total), and Fig. 2 walks through a fully grounded but wrong-target quote (rectal-bleeding evidence cited for an abdominal-pain claim) that Layer 1 alone would pass.
Honest publication of a negative exploratory result
Fair baseline treatment
Symmetric verification design with full audit trail
Weaknesses
Single-clinician gold standard without inter-rater agreement
Family-history regression masked by aggregate metrics
No held-out split between development and evaluation
Rule-based baseline at a single un-tuned operating point, without significance testing
Backbone generality and self-judging under-specified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
VERGE — Structured Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is VERGE and what problem does it solve?
VERGE is a four-agent LLM workflow that reads free-text clinical notes from adults aged 18–49 and extracts six colorectal cancer red-flag symptoms — abdominal pain, rectal bleeding, rectal pain, diarrhea, constipation, weight loss — plus family history of colorectal cancer. Each positive label must be backed by a quote from the note; the quote is checked for textual grounding and clinical validity, corrected if flagged, and escalated to a human if unresolved after five correction rounds.
How much better is VERGE than a single LLM call?
On 4,033 clinician-labeled note–finding pairs, precision rose from 0.764 to 0.849 and MCC from 0.681 to 0.730 versus the same model used as a single agent, while recall stayed essentially flat (0.705 to 0.702). A patient-cluster bootstrap shows the precision, F1 and MCC gains are statistically reliable; 9.1% of claims entered refinement and only 1.5% needed human review.
Is the data or code public?
The labeled notes come from Parkland Health EHR data (2010–2020, adults 18–49) under UT Southwestern IRB protocol STU-2024-0355 and are not public. The paper states that prompt design and model settings are available in the authors' GitHub repository; the models used (Llama-3.1-8B-Instruct, Ministral-3-14B) and MedCPT are public, but the headline numbers cannot be re-derived without the restricted notes.
What was the most common error VERGE caught?
Misattribution: the extractor cited real, correctly quoted note text that supported the wrong finding — for example, rectal-bleeding evidence quoted for an abdominal-pain claim. Among 367 flagged claims, 60.5% failed clinical-validity checks versus about 18% with grounding failures, so grounding-only verification would have missed most consequential errors.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 were assessed from their embedded text labels only; visual layout and any content rendered solely as graphics could not be verified.
- The metadata abstract supplied with the paper is cut mid-sentence, but the complete abstract appears intact in the body text, so nothing is missing from the basis of this review; the extraction ends cleanly at reference [27].
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.