SIFTING: A Novel LLM-Based Framework for Structured and Transparent Information Extraction from Clinical Free-Text Reports, with Application to Tumor Staging in Lung Cancer
Mirco Hess, Gerben van Veenendaal, Joris Wakkie, Yiwen Soo, Malcolm H. Lawson, John D. Maclay, Arjun Nair, Neal Navani
cs.CL
Quick verdict
Proposes SIFTING, a constrained-decoding LLM framework for traceable tumor T-staging (90% on 130 reports); solid, but beats only weak baselines, not GPT-5.
Summary
The paper introduces SIFTING, an LLM-based information-extraction framework that decomposes document-level tasks into a tree of segment-level classification and generation steps, using constrained decoding (perplexity-based option selection, DFA-restricted quoting, regex-constrained values) so every finding is anchored to source text. Applied to T-staging of lung cancer from 130 de-identified radiology reports (external set from Segmed), a 4-bit quantized Llama-3.3-70B (35 GB, single L40S GPU) achieved 90% accuracy (95% CI 84–95) against a four-expert consensus reference standard, and 91% with stages aggregated to T1–T4. A bootstrap non-inferiority analysis (5,000 resamples, 0.5% margin) found that hybrid panels substituting one human annotator with SIFTING were statistically non-inferior to all-human panels (82.2% vs 81.6% mean agreement, p<0.001). The same model in a conventional single-prompt setup scored only 68%; parity was reached only with o3 (90%) and GPT-5 (92%). Limitations: a single task and modest dataset, no code release, proprietary data, a manually built prompt tree requiring domain expertise, per-class support as low as 4 (T1a), differing denominators (115 accepted cases vs 118 for competitors), and no multiple-comparison correction across many model tests.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SIFTING constrains LLM decoding and processes documents segment-by-segment so every extracted finding is linked to exact source text, reducing hallucination by design without any model training.
- SIFTING-T-stage reached 90% accuracy (95% CI 84–95) against a four-expert consensus standard on 130 lung cancer radiology reports, and 91% with stages aggregated to T1–T4.
- A bootstrapped hybrid-panel analysis indicates SIFTING-T-stage is statistically non-inferior to individual expert annotators in inter-rater agreement (p<0.001, 0.5% margin).
- The same 4-bit Llama-3.3-70B rises from 68% (single prompt) to 90% within SIFTING, matching o3 and approaching GPT-5 (92%) while adding traceability.
Strengths
Traceability is enforced by construction, not requested in a prompt
DFA-restricted quoting and regex-constrained generation (Eqs. 2–3, §2.2.1) guarantee outputs are source substrings or formatted values; Figure 6 demonstrates color-coded positive/possible/negative anchors a clinician can audit.
Unusually rigorous reference standard and honest error analysis
Two-phase blinded-then-consensus annotation by three pulmonologists and one thoracic radiologist (§2.1) with explicit ambiguity rules (Appendix A); 53/130 cases required discussion, candidly reported, and the Table 3 error taxonomy openly attributes 6/12 errors to lesion-origin ambiguity.
Clean isolation of the framework effect
Interchangeability analysis is methodologically interesting and conservatively framed in the method itself
Weaknesses
Denominator asymmetry complicates the headline comparison
No multiple-comparison correction across model tests
Evidence base is narrow relative to the framework's broad framing
Reproducibility is low: proprietary data, no code, partially disclosed prompts
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SIFTING is a framework paper with one clinical demonstration. Its claims: (1) segment-level processing plus constrained decoding (§2.2.1) yields structured extraction with every finding anchored to source text, reducing hallucination by design and requiring no model training; (2) SIFTING-T-stage reaches 90% accuracy (95% CI 84–95, n=115 accepted) against a four-expert consensus standard on 130 lung cancer radiology reports; (3) it is statistically interchangeable with individual expert annotators (hybrid-panel non-inferiority, p<0.001); (4) the framework lifts the same 4-bit Llama-3.3-70B from 68% to 90%, matching reasoning LLMs. The Conclusion's framing ('can be extended to other tasks') runs ahead of evidence gathered on a single task and dataset.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SIFTING framework?
SIFTING is an LLM-based information-extraction method that splits a document into sentence-level segments and runs a tree of small classification and generation tasks, using constrained decoding so every output is either one of a fixed option set, an exact quote from the source text, or a regex-formatted value. Each finding is anchored to the phrase it came from, making results auditable by clinicians.
How accurate is SIFTING at lung cancer tumor staging?
On 130 lung cancer radiology reports, SIFTING-T-stage matched a four-expert consensus T-stage in 90% of cases (95% CI 84–95); with stages merged into main categories T1–T4, accuracy was 91%. The same model used in a conventional single-prompt way scored only 68%.
Which LLM does SIFTING use and can it run locally?
The study used a 4-bit quantized Llama-3.3-70B (35 GB), fully self-hosted on a single NVIDIA L40S GPU with 48 GB memory, averaging 2.3 minutes per report. The framework requires token-level control of model generation, so closed API-only models cannot be used with it.
Is the SIFTING code or dataset publicly available?
No. The data statement says the radiology reports are proprietary under a commercial license (Segmed) and cannot be shared, and the paper mentions no code release, so independent replication currently requires reimplementing the framework and curating new data.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All figures (1–7, B.8) are images; only their captions appear in the extracted text, so the Appendix B single-prompt content and diagram internals could not be reviewed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.