Can LLMs Follow Medical Expert Logic? A Benchmark for Hierarchical Logical Consistency in Risk-of-Bias Assessment
Jiayu Huang, Zichen Tang, Qianhui Ling, Zemin Kuang, Haihong E
cs.AI
Quick verdict
A rigorous white-box audit of LLM medical-reasoning logic; strong design and stats, though headline collapse figures rest on single T=0 runs.
Summary
The paper introduces LogiMed-RoB, a benchmark of 860 randomized controlled trials and 14,820 queries that audits whether large language models follow the Cochrane Risk-of-Bias 2.0 expert decision logic rather than merely matching final labels. A Hierarchical Logical Consistency framework formalizes the RoB 2.0 workflow as deterministic rules and scores Atomic Consistency, Domain Consistency, Aggregation Consistency, and Evidential Faithfulness (Blind Guess and Reasoning Failure rates at Jaccard threshold τ=0.8). Track A (626 instances; 13,772 signaling-question queries) probes hierarchical logic; Track B (1,048 items from ROBIN and RoBBR) probes evidential grounding. Across 10 LLMs, Gemini 3.1 Pro reaches 98.88% atomic consistency yet only 45.13% end-to-end complete consistency, several open-weight models fall near 0%, blind guess rates reach 48.28%, and reasoning failure rates span 18.63–40.05%; domain D2 is the universal bottleneck. Supporting analyses include McNemar asymmetry tests, quadratic-weighted kappa, Wilson confidence intervals, threshold sensitivity, and rule-enforcement ablations yielding 2–24 point gains. Main limitations: all results come from single T=0 runs without variance estimates; blind-guess figures depend heavily on the τ=0.8 threshold; Track A gold answers were extracted by the author team without reported inter-annotator agreement; and parse-failure exclusion (e.g., Gemini 3.1 Pro parsed 554/626 trials) may bias comparisons.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLMs exhibit severe error compounding in RoB assessment: Gemini 3.1 Pro's 98.88% Atomic Consistency collapses to 45.13% end-to-end Complete Consistency, with several open-weight models near 0%.
- High final-label accuracy conceals flawed reasoning: Blind Guess Rates reach 48.28% and Reasoning Failure Rates span 18.63–40.05% even when high-quality evidence is retrieved.
- Domain D2 (deviations from intended interventions) is the universal logical consistency bottleneck across all 10 evaluated models.
- Enforcing the Cochrane decision rules as post-hoc symbolic deduction improves domain-level accuracy over direct LLM prediction in 7 of 8 tested models, by 2–24 percentage points.
- Outcome-accuracy-centric evaluation creates an illusion of clinical readiness, motivating white-box logical verification before clinical deployment.
Strengths
Genuine white-box evaluation paradigm
Section 3.2 formalizes RoB 2.0 as a deterministic rule system (answer space {Yes, PY, PN, No, NI, NA}, prerequisite DAG, symbolic domain deduction Mi, Worst-of aggregation), enabling audits of the reasoning chain — CAR, LF, VR — rather than final-label accuracy that Table 1 shows prior RoB datasets settle for.
Statistical rigor beyond accuracy tables
Appendix A applies exact binomial tests (all p<.0001), McNemar tests with directional b/c decomposition per model and per domain (Tables 7, 12), quadratic-weighted kappa 91.88–100.00 for aggregation, and χ² for evidence–decision dependence; Appendix C adds Wilson 95% CIs per model–domain (Table 9).
Self-skeptical validation of the headline effect
Actionable negative-to-positive result
Process transparency
Weaknesses
Single-run protocol with no variance estimates on headline metrics
BGR construct validity and threshold dependence
Possible selection bias from parse-failure exclusion
Track A gold extraction lacks reported inter-annotator agreement
Dramatic framing of an arithmetically expected result
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
LogiMed-RoB (arXiv 2609.11185) is a benchmark paper with a methodological core: the Hierarchical Logical Consistency (HLC) framework, which audits whether LLM outputs conform to the Cochrane RoB 2.0 expert decision process instead of merely matching final labels. The dataset spans 860 RCTs and 14,820 queries in two tracks: Track A (RoB 2.0; 626 outcome-level instances — 467 from Cochrane, 159 from Figshare — with 22 signaling questions each, yielding 13,772 queries) tests Atomic, Domain, and Aggregation Consistency; Track B (RoB 1.0; 1,048 items reused from ROBIN and RoBBR) tests Evidential Faithfulness. Central claims: (1) error compounding — Gemini 3.1 Pro's 98.88% Atomic Consistency (CAR) collapses to 45.13% end-to-end Complete Consistency, with open-weight models near 0% (Table 4); (2) an evidence-reasoning gap — Blind Guess Rates up to 48.28% (Llama 4 Maverick) and Reasoning Failure Rates of 18.63–40.05% at τ=0.8; (3) domain D2 (deviations from intended interventions) is the universal bottleneck; (4) white-box logical verification is necessary for clinical deployment.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the LogiMed-RoB benchmark?
LogiMed-RoB is a benchmark of 860 randomized controlled trials and 14,820 queries that tests whether large language models follow the Cochrane Risk-of-Bias 2.0 expert decision logic — answering signaling questions, deriving domain risks, and aggregating an overall judgment — rather than merely producing a correct final label. It scores four dimensions: Atomic, Domain, and Aggregation Consistency plus Evidential Faithfulness.
What is the Error Compounding Effect reported in the paper?
It is the finding that small per-question mistakes multiply through the hierarchical RoB 2.0 decision chain: Gemini 3.1 Pro answers 98.88% of atomic signaling questions consistently, yet only 45.13% of trials remain consistent across the entire chain, and several open-weight models drop to nearly 0% end-to-end consistency.
Which models were evaluated and which performed best?
Ten LLMs: GPT 5.1, Claude Sonnet 4.5, Gemini 3 Flash, Gemini 3.1 Pro, DeepSeek V3.2, Llama 3.3 70B, Qwen3 32B, Llama 4 Maverick, Baichuan M2, and Baichuan M3. Gemini 3.1 Pro led overall (98.88% atomic consistency, 45.13% end-to-end), DeepSeek V3.2 was the strongest open-weight model, and proprietary models generally outperformed open-weight ones on hierarchical consistency.
Is the LogiMed-RoB dataset publicly available?
The authors state it is the first publicly released benchmark of its kind for RoB assessment, with a project page (bupt-reasoning-lab.github.io/LogiMed-RoB) and GitHub repository (BUPT-Reasoning-Lab/LogiMed-RoB); they contrast this with nine prior RoB datasets marked as not publicly released.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The excerpt contains two '[...sampling gap...]' regions; Table 3 is cut mid-row (only GPT 5.1's partial row is visible), so per-model Table 3 values cited in this review come from the surrounding prose rather than the full table.
- The head of Appendix F (Track B window-cropping construction) falls inside a sampling gap; statements about Track B construction rely on Section 3.3.1 and the visible tail of Appendix F.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.