ClinAgent: A ReAct-Based Agent for Conversational Access to Clinical Trial Information
Antonino Vaccarella, Riccardo Cantini, Domenico Talia, Paolo Trunfio, Marianna Talia, Rosamaria Lappano, Marcello Maggiolini
cs.AI · cs.CL · cs.IR
Quick verdict
A ReAct agentic-RAG system for querying clinical trials; open-source and promising, but undisclosed eval-set size and no statistical rigor cap confidence.
Summary
ClinAgent is a conversational system that lets clinicians and researchers query clinical trial information in natural language. A ReAct-style LLM agent (LangChain/LangGraph, with persisted multi-turn memory) iteratively selects among three tools: a ClinicalTrials.gov search wrapper with schema-validated parameter extraction, a PubMed interface over the NCBI Entrez API, and a Python REPL that runs Pandas analyses on a locally cached CSV of retrieved trials; the system is open source. The authors evaluate three backends — Gemini 3.0 Flash, DeepSeek V3.2 in thinking mode, and non-thinking DeepSeek V3.2 — with a three-phase framework: goal achievement, plan quality and tool-use efficiency (crediting successful replans), and expert Likert ratings on eight criteria. Gemini attains a perfect goal-achievement rate (1.00), the highest overall score (0.80), and the best expert ratings; DeepSeek thinking leads plan quality (0.97) but collapses on expert-rated factuality (0.28). Key limitations: the size and composition of the query set and the number of experts are never reported; no statistical testing or inter-rater agreement is provided; and no non-agentic baseline (single-pass RAG or plain LLM) is compared, so the claimed advantage of the agentic design is not directly demonstrated.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 3
● low 0
4 strengths · 5 weaknesses · 1 author questions
AI text signal = Binoculars zero-shot detector score over 19 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ClinAgent reliably answers natural-language clinical trial queries by looping ReAct-style over ClinicalTrials.gov, PubMed and a cached-data Python analyzer, reaching a goal-achievement rate up to 1.00.
- Backend choice entails a trade-off: DeepSeek V3.2 (thinking) leads plan quality (0.97) while Gemini 3.0 Flash leads overall performance (0.80) and expert ratings.
- Reasoning-enabled (thinking) mode reduces factual reliability in this setting, with expert-rated factuality of 0.28 versus 0.65 for the other backends.
- No existing system combines agentic RAG with multi-source retrieval across both a trial registry and biomedical literature for conversational clinical information access.
Strengths
Concrete, fully open-source system design
Section 3 specifies a real architecture: Pydantic-validated parameter extraction for ClinicalTrials.gov API calls, local CSV caching to avoid redundant requests, a Pandas REPL over the cache, PubMed via NCBI Entrez, and MemorySaver-persisted multi-turn state; code is released on GitHub.
Evaluation separates outcome from process
Phase 1 (GAR) measures whether answers are produced, while Phase 2 disentangles Plan Quality from Tool Use Efficiency; the recursive success function (Eq. 3) elegantly credits failures followed by successful replans, and Teff (Eq. 4) penalizes first-attempt failures.
Actionable backend trade-off finding
Clear related-work positioning
Weaknesses
Evaluation denominator never reported
Expert evaluation lacks rigor details
No non-agentic baseline
Overall score aggregation under-specified
No error analysis, latency or cost
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper introduces ClinAgent (Sections 1, 3): a ReAct-style agentic RAG system for conversational access to clinical trial information, built on LangChain/LangGraph with persisted multi-turn memory (MemorySaver). An LLM agent iterates a reasoning-acting-observation loop over three tools: a ClinicalTrials.gov search wrapper (LLM parameter extraction validated by a Pydantic schema, results cached to CSV), a PubMed interface over NCBI Entrez, and a Python REPL running Pandas analyses over the cached dataset. The second contribution is a three-phase evaluation framework (Section 4) applied to three backends: Gemini 3.0 Flash and DeepSeek V3.2 in thinking and non-thinking modes. Central claims: (i) the system reliably answers factual, comparative, literature and mixed queries; (ii) backends exhibit a planning-versus-factuality trade-off; (iii) no prior system combines agentic RAG over both registry and literature (Section 2). The code is released on GitHub.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 1 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ClinAgent?
ClinAgent is an open-source conversational AI system that answers natural-language questions about clinical trials. It uses a ReAct-style LLM agent (built with LangChain/LangGraph) that iteratively calls three tools: a ClinicalTrials.gov search module, a PubMed literature search, and a Python REPL that analyzes a locally cached dataset of retrieved trials.
Which LLM backend worked best in the ClinAgent evaluation?
Gemini 3.0 Flash scored highest overall (0.80), with a perfect goal-achievement rate (1.00) and the best expert ratings. DeepSeek V3.2 in thinking mode achieved the best plan quality (0.97) but the lowest expert-rated factuality (0.28); non-thinking DeepSeek V3.2 scored 0.72 overall.
Is ClinAgent's code available?
Yes. The paper states the system is publicly available as open source at github.com/antoninovaccarella/clinical-chat. However, the evaluation query set and expert ratings are not reported in enough detail to reproduce the published scores.
What are the main limitations of the ClinAgent paper?
The number of evaluation queries and the number of expert raters are never reported; no statistical tests or inter-rater agreement are provided; there is no non-agentic baseline such as single-pass RAG; and the surprisingly low factuality of the thinking-mode backend (0.28) is not analyzed with error examples.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 is flattened into fragmentary inline text by PDF extraction; the diagram's wiring beyond its caption could not be verified.
- The query-set size and expert-panel details are absent from the entire provided text — a paper-level omission, not an extraction cut.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.