SentryLine: Evidence-Grounded Question Answering over Evolving Documents in Oncology Care
Tampu Ravi Kumar, Gaurav Najpande, Muhammad Ali Khan, Kaneez Zahra Rubab Khakwani, Karan Kathuria, Shorya Azriel Moses, Yuvraj Kalia, M Bassam Sonbol, et al.
cs.CL
Quick verdict
Strong system+benchmark for guideline-drift QA, but no significance tests, judge-family bias, and unconditioned baselines temper headline gains.
Summary
The paper presents SentryLine, a clinical question-answering system over ASCO living guidelines for breast and prostate cancer. It combines a vectorless hierarchical retrieval pipeline (PageIndex), role routing between clinician- and patient-facing registers, and a verification layer — factual, temporal, and citation checks — that adds drift-detection notes when a guideline update supersedes a recommendation. The authors also introduce ASCOBench: 405 three-turn conversations across four question categories built from 57 ASCO guideline PDFs, with the 105-conversation test split reviewed by two clinicians who produced gold answers. Using Claude Sonnet 4.6 as an LLM judge calibrated against expert ratings (quadratic-weighted κ 0.74 pooled), SentryLine is evaluated on three backbones (Gemini 3.1 Pro, Llama 3.3 70B, DeepSeek-R1) against Direct CoT, BM25/dense/naive RAG, and ASCO's production assistant — e.g., Gemini Reasoning FC reaches 77.1% versus 27.1–31.2% for RAG baselines and 10.0% for the assistant. Ablations indicate the verifiers trade fully-correct rate for lower incorrect rates. Main weaknesses: no statistical significance testing despite small per-category test cells (Role-Specific n=48), a judge sharing provenance with the model family that generated the benchmark, unreported human–human annotator agreement, role instructions given only to SentryLine in the role-specific comparison, and ablations that omit the strongest backbone. The paper states code and data will be released under an MIT license.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A vectorless hierarchical retrieval pipeline (PageIndex) with role routing and independent verification outperforms chunk-based RAG baselines on ASCOBench across three generation backbones.
- The verification layer trades fully-correct rate for lower incorrect rates, e.g., removing the temporal verifier raises DeepSeek Role-Specific incorrect answers from 5.6% to 26.4% (Table 4).
- SENTRYLINE substantially outperforms ASCO's production Guidelines Assistant on all four question categories, e.g., Reasoning FC 77.1% vs 10.0% under the Gemini backbone (RQ4).
- ASCOBench is the first benchmark explicitly targeting temporal guideline drift in oncology QA, with clinician-validated gold answers on the 105-conversation test split.
Strengths
First benchmark explicitly targeting temporal guideline drift in clinical QA
ASCOBench (Section 4) is grounded in 57 ASCO breast/prostate guideline PDFs with a Contrasting category that embeds an outdated recommendation as a decoy; the 105-conversation test split was gold-annotated by two clinicians under a documented accept/revise/rewrite protocol (Appendix A.3).
Verification-first architecture with ablation evidence for its safety value
The temporal verifier demands verbatim excerpts from superseding passages and discards unsupported drift flags (Section 3.2); removing it raises DeepSeek Role-Specific incorrect rate from 5.6% to 26.4% (Table 4), and Appendix C explicitly accounts for the FC-cost versus Inc-reduction tradeoff.
Multi-backbone evaluation separating retrieval/verification design from the generator
Judge calibration against human experts is reported, not just asserted
Candid limitations and ethics treatment
Weaknesses
No statistical inference anywhere despite small test cells
Judge-family circularity with the benchmark generator
Role-Specific comparison is asymmetric by construction
Incomplete ablations and unreported annotator agreement
Register, efficiency, and assistant-comparison claims are unmeasured
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SentryLine is a clinical QA system over ASCO living guidelines (breast and prostate cancer; 43 + 14 guideline PDFs, Section 4) built in three stages: vectorless hierarchical retrieval (PageIndex tree navigation, Section 3.1), role-routed generation (clinician vs. patient templates, temperature 0.1), and independent factual/temporal/citation verification with drift notes (Section 3.2). The companion contribution, ASCOBench, contains 405 three-turn conversations across four categories (Factual, Reasoning, Contrasting, Role-Specific); the 105-conversation test split was gold-annotated by two clinicians. Central claims: (1) hierarchical structured retrieval beats chunk-based RAG; (2) verification converts confident errors into hedged answers; (3) SentryLine beats five baselines including ASCO's production assistant (RQ4: Reasoning FC 77.1% vs. 10.0% under Gemini). The claims are consistently supported as descriptive results, but their evidentiary strength is bounded by the evaluation design.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SentryLine?
SentryLine is a clinical question-answering system built on ASCO living guidelines for breast and prostate cancer. It retrieves guideline passages with a hierarchical, non-embedding index (PageIndex), adapts answers to a clinician or patient reading level, attaches page-level citations with factual, temporal, and citation verification reports, and flags when a guideline update supersedes an older recommendation.
What is ASCOBench?
ASCOBench is a benchmark of 405 three-turn clinical conversations in four question categories (factual, reasoning, contrasting, role-specific), built from 57 ASCO breast and prostate cancer guideline PDFs. The 105-conversation test split was reviewed by two clinicians who produced gold reference answers, and systems are scored against them using an LLM-as-judge protocol calibrated to expert ratings.
How does SentryLine compare to ASCO's own AI assistant?
On the ASCOBench test set with the Gemini backbone, the paper reports fully-correct rates of 56.4% vs 20.0% on factual questions, 77.1% vs 10.0% on reasoning, 59.3% vs 13.3% on contrasting, and 65.3% vs 8.3% on role-specific questions, all in SentryLine's favor. The comparison is descriptive — no significance testing is provided and abstentions by the assistant are not discussed.
Is the code or dataset publicly available?
The paper states that SentryLine is released under an MIT license with the full pipeline, both role prompt templates, the 405-conversation ASCOBench dataset with stratified splits, clinician annotations, and evaluation scripts, plus a demo screencast. Actual availability should be verified against the linked repository, which this review could not access.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–6 are image-only in the extraction; their internal content is not available. All numbers cited in this review come from body text and Tables 1–4, which extracted cleanly.
- The provided text ends naturally after Appendix C with no sign of truncation; references and all appendices A–C are present.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.