Large Language Models Versus Physicians in Traditional Chinese Medicine: A Real-World Clinical Case Evaluation
Jiacheng Xie, Xiaoting Tang, Yang Yu, Jinpu Li, Shouli Li, Congcong Jing, Yantao Yang, Zhiyong Zhao, et al.
cs.CL · cs.CY
Quick verdict
Frontier LLMs beat TCM physicians on blinded expert scores in a real-case benchmark, but blinding-leakage risk, single-run inference, and a self-evaluated model temper the headline.
Summary
The study benchmarks 16 LLMs (12 general-purpose, 4 TCM-specialized) against 60 practicing TCM physicians on 60 real outpatient cases drawn from a 349-case library collected across 62 Chinese hospitals (2018-2023). Anonymized reports were scored by five senior, source-blinded experts on nine diagnostic/therapeutic dimensions (5-point Likert); inter-rater reliability was high (ICC(2,k) 0.748-0.940). DeepSeek-R1, Claude Opus 4, and GPT-5 exceeded the per-case physician mean by 0.667/0.629/0.627 points on average, winning 58/60, 57/60, and 55/60 head-to-head cases; TCM-specialized models (BianCang, Huatuo, HuatuoGPT-2) lagged far behind with templated, repetitive outputs. Prescription analyses showed LLMs use more ingredients and higher doses, with systematic herb-preference and dose deviations (e.g., Bupleuri Radix +7.56 g), plus case-expansion hallucinations and occasional protected-species ingredients. An exploratory arm (n=7) found DeepSeek assistance raised junior physicians' scores (P=0.016). Key limitations: single-run stochastic inference with unreported repetition settings, fragile blinding given distinctive model styles, physician self-selection of cases, author involvement in one evaluated model (BenCao), and static vignettes that exclude iterative clinical workflow.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Cutting-edge general-purpose LLMs (DeepSeek-R1, Claude Opus 4, GPT-5) achieved higher blinded expert scores than a 60-physician comparator across nine TCM diagnostic and therapeutic dimensions on 60 real outpatient cases, with average advantages of 0.63-0.67 points and 55-58/60 case-level wins.
- Despite high expert scores, LLM prescriptions diverge systematically from physician practice in herb selection, number of ingredients, dosage calibration, and treatment strategy.
- TCM-specialized models underperform strong general-purpose models, frequently producing templated diagnoses, repetitive prescriptions, and disease substitution (e.g., BianCang diagnosing 'Feiwei disease' in 42/60 cases).
- DeepSeek-generated support significantly improved junior physicians' blinded diagnostic scores in an exploratory seven-physician before/after analysis (+0.836 points, P=0.016).
- Hallucinations persist across model classes, taking the form of additive case expansion in general LLMs and cross-case templating in TCM-specialized models, supporting the need for physician oversight.
Strengths
Genuine real-world evaluation data with ethical provenance
349 de-identified outpatient cases from 62 hospitals (2018-2023) with IRB approval (2025-001), patient and physician consent, and first-time release; the 60-case benchmark spans 227 disease types with four-exam findings, labs/imaging, and longitudinal follow-up (Results; Fig. 1; Table 1), addressing the examination-question gap the authors document in the introduction.
Comparison design with blinding and measured reliability
Identical structured output template for physicians and LLMs, pooled anonymized reports, five blinded senior raters on a dedicated platform, and ICC(2,k) of 0.748-0.940 overall (Table 2), with source-stratified ICCs reported rather than asserted.
Statistical framework goes beyond mean comparisons
Safety-relevant analyses beyond aggregate scores
Transparency artifacts and candid limitations
Weaknesses
Blinding integrity is questionable and style may drive scores
Single-run inference under default settings with unreported repetition
Asymmetric case assignment between groups
Conflict of interest in an evaluated model and unspecified rater independence
Small-sample and reporting-precision issues in secondary analyses
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did LLMs outperform TCM doctors in this study?
Under blinded expert scoring on 60 real outpatient cases, top general-purpose LLMs averaged 0.63-0.67 points higher than the physician mean on a 5-point scale across nine dimensions, with DeepSeek-R1 winning 58 of 60 case-level comparisons. However, expert opinion scores are not clinical outcomes, and the study simultaneously documents unsafe prescribing patterns and hallucinations, so the authors frame LLMs as decision-support candidates requiring physician oversight, not replacements.
Which AI model performed best for traditional Chinese medicine tasks?
DeepSeek-R1 ranked first overall (best average rank 4.34, wins in 58/60 cases), followed by Claude Opus 4 and GPT-5. Notably, the four TCM-specialized models evaluated (BianCang, HuaTuoGPT-2, BenCao, Huatuo) all performed worse than leading general-purpose models, with several showing templated or repetitive outputs.
Is it safe to use ChatGPT or DeepSeek to get Chinese herbal prescriptions?
No, not without a physician. The study found LLM prescriptions deviated from physician practice with more ingredients and higher doses, included unsupported herbs (occasionally expensive or protected-species materials like pangolin scales or antelope horn mentioned in the discussion), and exhibited hallucinations such as inventing comorbidities. The authors explicitly call for physician supervision and safety constraints before any prescription use.
What data was the study based on and is it public?
The evaluation used 60 de-identified real outpatient cases selected from 349 cases collected at 62 Chinese hospitals between 2018 and 2023, with IRB approval and informed consent. The 60 cases are included in the supplementary materials, and analysis code, prompts, and model configurations are available on the authors' GitHub repository (TCM-Evaluation).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-6 are available only as captions in the extracted text; exact point estimates, confidence intervals, and critical-difference visualizations could not be independently verified.
- Supplementary tables (scoring rubric, prompting template, per-case physician assignments) were not included in the extract, so rubric design and assignment balance could not be assessed.
- Small arithmetic inconsistencies noted in Table 1 may partly reflect text extraction rather than the source document.
- This is an arXiv preprint (2609.17544); peer-review status and any resulting revisions are unknown.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.