Checkpoints Are Not Enough: Trust Calibration in CoSLR, a Human-AI System for Systematic Literature Reviews
MD Aidul Islam, Malik Abdul Sami, Muhammad Waseem, Zeeshan Rasheed, Kai-kristian Kemell, Zheying Zhang, Pekka Abrahamsson
cs.CL · cs.AI · cs.IR
Quick verdict
CoSLR adds mandatory human checkpoints to LLM-based SLR, but a survey (N=63) found 34.9% would accept unverified AI synthesis; honest, exploratory, yet self-report-only.
Summary
CoSLR is a multi-agent, RAG-grounded system supporting the SLR lifecycle—objective/research-question drafting, Boolean search-string generation, retrieval, screening, analysis, and cited report generation—with mandatory human checkpoints between every AI draft and its acceptance. A survey-based design-science evaluation with 63 participants (students, researchers, professionals, ~15-minute remote sessions, 33-item questionnaire) found positive usability perceptions (grouped means 3.78 for usability, 3.68 for learning support and productivity on a 5-point scale) alongside the lowest-rated dimension, trust in unverified output (3.02): 22 of 63 participants (34.9%) would accept AI summaries and reports without additional checking, 22 were neutral, and 19 would verify. Trust correlated with expected time savings (Spearman ρ=.30, p=.017), while no researcher/non-researcher difference was detectable (Mann–Whitney p=.38). The paper's core contribution is framing checkpoint acceptance as a trust-calibration problem. Key limitations: self-reported intent rather than observed verification, 15-minute sessions with no completed review, no control condition, and convenience sampling.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A mandatory human checkpoint safeguards an SLR only if researchers use it to verify rather than merely confirm; interface design must address trust calibration rather than assume oversight (Sec. 1, 5.3, 8).
- In a survey of 63 participants after ~15-minute interactions, 22 (34.9%) reported they would accept AI-generated summaries and reports without additional human checking (Sec. 5.3).
- Willingness to accept unverified output is positively associated with expected time savings (Spearman ρ=0.30, p=.017), suggesting the efficiency perception partly reflects a disposition toward AI (Sec. 5.3, 7.1).
- Participants perceived the system as usable and time-saving (grouped usability mean 3.78; 76.2% estimated ≥25% time savings), with no detectable researcher/non-researcher difference in trust (Mann–Whitney p=.38) (Sec. 5.1–5.3).
- Checkpoint strength (how much active engagement is demanded) should be a design variable calibrated to stage risk, with effortful verification reserved for inclusion/exclusion and final synthesis (Sec. 6.4, 8).
Strengths
Reframes evaluation around trust calibration, not just usability
RQ3 (Sec. 4.1) and Sec. 5.3 directly test whether mandatory checkpoints produce verification, motivated by the 'verify rather than confirm' distinction; the headline finding (22/63, 34.9% would accept unverified output) is a genuinely useful negative result for the human-in-the-loop literature.
Unusually careful statistical reporting for an exploratory survey
Sec. 5.3 and 7.4 pre-specify three tests, report exact p-values, give effect sizes (Spearman ρ=.30, Kendall τb=.27, rank-biserial r=-.13), use non-parametric methods appropriate to ordinal items, and explicitly refuse to over-read the null (p=.38) — no p-hacking smell.
Honest, self-critical threats-to-validity section
Concrete, workflow-grounded design discussion
Weaknesses
The central 'calibration' claim rests on a single self-reported Likert item
15-minute sessions, no completed review, no comparison condition
Inconsistent usability statistic between abstract and body
Insufficient implementation detail for scientific reproducibility
Editorial control issues
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CoSLR (Sec. 3) is a multi-agent, RAG-grounded assistant for the systematic literature review (SLR) lifecycle: an Objective Builder, RQ Builder, and Search Query Generator draft planning artifacts; retrieved papers pass through PyMuPDF+OCR extraction, chunking, and embedding into a vector index; a RAG question-answering interface and Report Generation Agent produce cited synthesis. Three mandatory human checkpoints gate the workflow: search-strategy approval, paper selection (inclusion/exclusion), and report validation (Fig. 1, Sec. 3.4).
The paper's claims are deliberately two-layered. The system-level claim (RQ1/RQ2) is that users perceive the integrated pipeline as useful and time-saving. The more interesting claim (RQ3, and the title) is that *mandatory checkpoints do not by themselves ensure verification*: 22 of 63 participants (34.9%) rated trust in unverified output 4–5 on a 5-point Likert item after ~15 minutes of use, 22 were neutral, and only 19 (30.2%) chose 1–2. Supporting analysis: trust correlates with expected time savings (Spearman ρ=0.30, p=.017; Kendall τb=0.27, p=.015), and no researcher/non-researcher difference is detectable (Mann–Whitney U=400.5, p=.38, rank-biserial r=−0.13, medians 3 vs 3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the title 'Checkpoints Are Not Enough' mean?
The paper argues that forcing users to pass through mandatory human checkpoints in an AI literature-review tool does not by itself guarantee verification: a checkpoint can be clicked through as a rubber stamp. In their survey, 22 of 63 participants (34.9%) said they would trust AI-generated summaries and reports without additional checking, so the interface provided the opportunity to verify without ensuring it happened.
What is CoSLR and how does it work?
CoSLR is a human-AI collaborative system for systematic literature reviews built from multiple LLM agents plus retrieval-augmented generation (RAG). It covers three phases: planning (drafting objectives, research questions, and Boolean search strings), screening/extraction (retrieving papers, extracting text, building a vector index), and reporting (grounded question answering and cited report generation). After each AI-produced draft, the workflow pauses at a mandatory checkpoint where a researcher must approve or edit before proceeding.
What were the main results of the CoSLR user study?
In a survey of 63 participants (about 15 minutes of system interaction each), usability was rated positively: grouped mean 3.78/5 for usability, 3.68 for learning support and productivity. The lowest-rated dimension was trust in unverified output (3.02): 22 of 63 (34.9%) would accept AI summaries/reports without additional checking, 22 were neutral, and 19 would verify. Trust correlated with expected time savings (Spearman ρ=0.30, p=0.017); no researcher/non-researcher difference was detected (p=0.38).
What are the study's main limitations?
Findings rest on self-reported perceptions, not observed verification behavior or objective task performance. Sessions lasted about 15 minutes and no participant completed an actual systematic review. There was no control condition (e.g., manual workflow or an existing tool), the time-savings question only offered positive brackets, and recruitment used convenience sampling skewed toward software-engineering students and researchers.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The metadata-supplied abstract ends mid-sentence ('the effect'), but the paper body embedded below contains a complete abstract; nothing else in the body text appears truncated.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.