EviStreams: Human-in-the-Loop AI Data Extraction for Systematic Reviews in Medicine
Sai Karthik Kosuri, Ankita Shashikant Bhosale, Michael Glick, Alonso Carrasco-Labra, Chris Callison-Burch, Chris Callison-Burch
cs.CL
Quick verdict
System paper claims field specs, not model choice, drive extraction quality; evaluation is honest but calibration and judge circularity limit certainty.
Summary
EviStreams is an open-source, no-code web platform that embeds human control into LLM-assisted data extraction for systematic reviews. A methodologist defines typed fields (description, hints, rules, examples) in a form builder; the system proposes an extraction plan — a DAG compiled mechanically into DSPy signatures — that a reviewer must approve before any code runs; and extracted values undergo reviewer-blinded dual review with adjudication, each value linked to its verbatim source passage. The authors evaluate three frontier model families (Claude Sonnet 4.6, GPT-5.5, Gemini 3.1 Pro) on four clinical corpora totaling 125 studies from completed Cochrane/JADA reviews. Micro-F1 spans 0.741–0.898 across corpora, while model choice shifts results by at most ~0.03–0.05; collapsing the multi-signature pipeline into one prompt changes F1 by at most 0.036. A richer field specification improves macro-F1 in 40 of 51 model–form comparisons (78%; mean ΔF1 0.049, median 0.014), with one form falling from 0.724 to 0.247 when hints, rules, and examples are removed. Of 1,003 reviewed discrepancies, 68% are methodological (ground-truth/convention side) and 30% genuine model errors. Key limitations: no inter-annotator agreement for reference data, pilot calibration on the same corpora scored against, and a free-text judge from the Claude family deciding 34% of comparisons.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Human review at three stages (program approval, specification calibration, blinded dual review with adjudication) lets domain experts control AI extraction while preserving systematic-review evidentiary standards.
- Across four clinical corpora and three frontier model families, extraction quality is governed far more by the field specification than by model choice or pipeline decomposition.
- Richer field specifications improved macro-F1 in 40 of 51 model-form comparisons (78%), with the oral-cancer index_test form collapsing from 0.724 to 0.247 macro-F1 without hints, rules, and examples.
- About 68% of 1,003 reviewed extraction discrepancies are methodological (ground-truth, convention, or scoring side) rather than genuine model errors, which account for roughly 30%.
Strengths
Unusually candid error analysis
§3.3 reviews 1,003 discrepancies across 52 studies and attributes 68% to the ground-truth/convention/scoring side versus 30% genuine model errors, with per-subtype causes in Table 7 (e.g., 75 missing rows, 63 false-NR, 60 wrong-arm/time-point), summing consistently (683+298+22=1,003).
Independent evaluation harness with sound record scoring
§3 states the type-matched scoring harness is implemented independently from the extraction pipeline to reduce evaluation coupling, uses Hungarian assignment for record-level alignment, and is released with the four corpora.
Clean one-at-a-time ablation design
Real deployment with protocol-faithful human review
Self-declared limitations cover the main threats
Weaknesses
Specification-calibration circularity
No statistical uncertainty quantification
Unquantified ground-truth reliability
LLM-judge circularity
No quantitative comparison with prior systems
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
EviStreams is a system demonstration paper presenting an open-source, no-code platform for AI-assisted data extraction in systematic reviews, with human control at three stages: program review (an LLM-proposed extraction DAG approved before any code runs, §2.1), specification calibration (typed fields with description/hints/rules/examples, refined on 3–5 pilot papers, §2.4), and prediction review (reviewer-blinded dual extraction plus adjudication, §2.4). The headline empirical claim: across four clinical corpora and three frontier model families, extraction quality is governed more by the field specification than by model choice or pipeline decomposition. A secondary framing claim is full auditability — every value links to its verbatim source passage, and the per-field decision chain (AI pre-fill, R1, R2, adjudicator) is preserved for export.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is EviStreams?
An open-source (Apache-2.0), no-code web platform from University of Pennsylvania researchers that helps systematic-review teams use LLMs for data extraction while keeping humans in control: a reviewer approves the AI-proposed extraction plan before it runs, refines typed field specifications from pilot runs, and completes a reviewer-blinded dual review with adjudication. Every extracted value is linked to its verbatim source passage. Live demo: evistreams.com/demo.
Does the choice of LLM matter for systematic-review data extraction?
In this study, only modestly. Across four clinical corpora totaling 125 studies, the Claude, GPT, and Gemini families differed by at most roughly 0.03-0.05 micro-F1, while the between-corpus range was 0.741-0.898. The authors conclude that the field specification — not model choice — is the main quality lever, so providers can be chosen on cost or governance.
How accurate is AI data extraction according to this paper?
Overall micro-F1 ranged from 0.741 (oral cancer, the hardest corpus) to 0.898 (antibiotic prophylaxis, the easiest). In the paper's error analysis of 1,003 reviewed discrepancies across 52 studies, 68% were methodological (ground-truth, convention, or scoring-side issues) and about 30% were genuine model errors such as missing rows or wrong-arm values.
Is the code and evaluation data available?
Yes: the backend and frontend are on GitHub under Apache-2.0, and the authors state the evaluation harness and the four clinical corpora are released with the paper. Actually running the pipeline requires access to commercial LLM APIs (Anthropic, OpenAI, Google), and results depend on those models' current versions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 are unavailable as images; assessment relies on captions and body text only
- Table 3 capability symbols (●/◗/—) rendered imperfectly in extraction; capability judgments verified against the caption legend
- Appendices appear truncated to summaries (A-E) without full raw tables; per-field results could not be independently recomputed
- PDF-to-text garbling in some footnotes/references; model names (GPT-5.5, Gemini 3.1 Pro) are as printed in the source
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.