CRISS: A Retrieval-Augmented AI Chatbot for Assisting Cancer Registrars
Vani Seth, Mohammad Beheshti, Anirudh Kambhampati, Vishwa Bhayani, Lucinda Ham, Prasad Calyam, Iris Zachary
cs.AI · cs.CL
Quick verdict
RAG assistant for cancer registrars shows large grounding gains over baselines, but LLM-judge-only scoring and no human validation temper the claims.
Summary
The paper presents CRISS, a retrieval-augmented generation (RAG) chatbot that helps cancer registrars and Oncology Data Specialists (ODSs) find and interpret coding and staging guidance from national standards (SEER manuals, Solid Tumor Rules, MPH rules). Registry documents are chunked (1,500 characters, 200 overlap), embedded, and retrieved (top-6, multi-hop, recency re-ranking) to ground LLM answers with citations. The authors benchmark 22 configurations - open-weight, proprietary, and non-RAG baselines using web search - on 208 ODS-annotated questions across easy/medium/hard tiers (~4,600 responses), scoring with ROUGE/BERTScore/cosine similarity, a grounding metric, and an LLM-as-a-judge ensemble (GPT-4o plus Gemini 3.1 Pro) across six dimensions and four failure flags. RAG grounding roughly doubled versus non-RAG (0.62/0.56/0.59 vs 0.29/0.26/0.29). Proprietary RAG models led the easy/medium tiers while open-weight RAG models led the hard tier, which the authors attribute partly to lower caution (15.3% vs 23.1%) rather than better reasoning. Limitations: no human validation of the judges or registrar user study, no significance testing, partly self-reported retrieval metrics, a single-institution question bank, and mismatches between prose and figure numbers for hard-tier failure flags.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Domain-specific RAG substantially improves evidence grounding for cancer registry questions, with mean grounding scores of 0.62/0.56/0.59 (easy/medium/hard) versus 0.29/0.26/0.29 for web-search non-RAG baselines.
- Proprietary RAG models perform strongest on easy and medium questions, while open-weight (local) RAG models rank highest on hard questions.
- The hard-tier advantage of local models reflects response behavior (lower caution: 15.3% vs 23.1% refusal/caution rates) rather than demonstrated better clinical reasoning.
- Citation-grounded RAG can support training and helpdesk use for registrars while preserving human oversight of final coding decisions.
Strengths
Concrete, transferable system design
Chunking (1,500/200 chars), metadata schema, provider embeddings, top-k=6, recency-aware reranking, and multi-hop retrieval are specified in enough detail (§2.1, Appendix B) for others to rebuild the pipeline
Broad, tiered model comparison
22 configurations including open-weight (Qwen3.5-122B, gpt-oss-120B, Gemma), proprietary, and web-search non-RAG baselines over ~4,600 responses across three difficulty tiers (§3.3, Figure 5)
Unusually candid negative analysis
Operationally meaningful rubric
Deployment realism
Weaknesses
Unvalidated LLM judges
No inferential statistics
Internal numerical inconsistencies
Structurally unfair baseline comparison
No user validation or release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CRISS is a retrieval-augmented generation (RAG) chatbot that grounds cancer-registry coding guidance (SEER manuals, Solid Tumor Rules, MPH/Hematopoietic rules, SSDI; Table 1) in a citation-supported chat interface. The paper makes three claims: (1) domain-specific RAG yields accurate, citation-grounded answers across difficulty tiers; (2) it improves access to and interpretation of guidance; (3) it can support training and helpdesk use while humans retain final coding authority. Claim 1 is directly tested. Claims 2 and 3 are motivated but never evaluated with users - no registrar study, task-timing measurement, or comprehension assessment appears anywhere in the text.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CRISS?
CRISS is a retrieval-augmented generation (RAG) chatbot built by University of Missouri researchers to help cancer registrars and Oncology Data Specialists quickly find citation-supported answers to coding and staging questions using national registry standards such as SEER manuals and Solid Tumor Rules.
How much better is CRISS than just asking a regular chatbot?
On 208 registry questions in three difficulty tiers, RAG configurations roughly doubled the grounding score compared with non-RAG chatbots using web search (e.g., 0.62 vs 0.29 on easy questions). Proprietary RAG models scored best on easy/medium tiers, while open-weight RAG models scored highest on hard ones, though the best hard-tier score was only 2.94 out of 5.
Was CRISS tested with real cancer registrars?
No. Evaluation relied on automatic metrics and LLM-as-a-judge scoring. A prospective study with practicing cancer registrars is explicitly listed as future work, so usability claims are not yet backed by user evidence.
Is the code or the question bank publicly available?
The paper does not report a public code, question bank, or data release. The 208-question evaluation set was developed at a single institution and its ground-truth answers were annotated by Oncology Data Specialists.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as flattened text runs; prose-vs-figure flag-rate mismatches (§5.3 vs Figure 7) may partly be extraction artifacts, though the internal prose inconsistencies stand on their own.
- Appendix table data (e.g., Table 6 per-model containment) is partially flattened; some values are unrecoverable from the text.
- The knowledge-base source documents and full judge outputs were not available for inspection, so rubric application could not be independently audited.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.