Performance of Clinical AI System and Physicians and Frontier Language Models in primary care diagnostics
Andy Nkansah, Hanna Plotnitskaya, Stanislau Salavei, Anna Kozlova, Piotr Gibas, Julian Milek, Viktar Harbachou, Aleksey Ropan, et al.
cs.CL · cs.AI · cs.HC
Quick verdict
Company-run study: Doctorina beats 8 physicians, 4 frontier LLMs in 150 synthetic Polish consultations; transparent but COI and post-hoc endpoint limit weight.
Summary
This company-run controlled study compared Doctorina — a patient-facing clinical AI system built by the authors' company around OpenAI models — with eight Polish physicians and four standalone frontier LLMs (Gemini 3.1 Pro Preview, Claude Opus 5, GPT-5.6-sol, Kimi K3) in 150 synthetic Polish primary-care consultations using a case-bounded simulated patient. Doctorina reached 82.0% Top-1 diagnostic concordance versus 57.0% for physicians (25.0 percentage points; 95% CI 17.7–32.7) and scored higher on physician-rated workup (89.4 vs 66.9) and initial treatment (83.7 vs 61.2). Among standalone models, Kimi K3 came closest diagnostically (78.7%), while Claude Opus 5 led management scores (90.3/86.6). A second eligible Doctorina run reproduced the advantage over physicians but reversed the management comparisons against Opus and Kimi. The design is unusually transparent — adaptive consultations, three-clinician case review, disclosed post-hoc endpoint changes, sensitivity analyses and published analysis code — yet key limitations temper the headline: the manufacturer evaluated its own product; the primary endpoint hierarchy was post-specified; run selection and all-attempt zero-coding favor the primary arm; the two review blocks were separate, source-visible and over two months apart; and cases plus the simulated patient are synthetic, so clinical-practice generalization requires prospective validation.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Doctorina achieved 82.0% Top-1 diagnostic concordance versus 57.0% for the eight physicians (+25.0 points; 95% CI 17.7–32.7) in 150 synthetic Polish primary-care consultations.
- Doctorina also outperformed physicians on primary-or-reference-differential concordance (97.3% vs 85.0%) and on physician-rated workup (89.4 vs 66.9) and initial treatment (83.7 vs 61.2).
- Doctorina's diagnostic point estimates led all four standalone frontier LLMs, while Opus, Doctorina and Kimi formed a closely spaced leading group on management, with Opus numerically highest.
- A second eligible Doctorina execution (run 479) reproduced the advantages over physicians across all four outcomes.
- Diagnostic correctness and management quality were only weakly correlated (Spearman 0.073–0.289), supporting separate assessment of diagnosis, workup and treatment.
Strengths
Adaptive consultation design with a shared, tightly controlled information environment
All six groups faced the same 150 cases, constructed references and endpoints through a case-bounded simulated patient (o3-2025-04-16, temperature 1), so scored outputs depended on respondent-directed questioning rather than fixed vignettes — the design explicitly extends AMIE-style evaluations (Fig. 2).
Rigorous case construction with epidemiological grounding
Cases were clinician-authored with field-by-field three-of-three clinician acceptance and revision loops; the diagnosis distribution was scaled from 2024 National Health Fund ICD-10 service data, yielding 127 distinct labels across 150 cases (Methods, Table 2).
Unusually candid reporting of design weaknesses
Genuine sensitivity and robustness program
Reproducibility package above field norm
Weaknesses
Structural conflict of interest in a self-evaluation
Post-hoc primary endpoint undermines confirmatory status
Analytic choices favor the primary arm on management outcomes
Asymmetric, source-visible review blocks
Limited external validity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper reports a controlled reader-by-case comparison of Doctorina v2.1.3 — a patient-facing clinical AI product built by A.I. Doctor Medical Assist (five of nine authors hold company roles) around OpenAI models (o3 doctor agent; gpt-5, gpt-5-mini and o3-mini supporting agents) — against eight Polish family/internal-medicine physicians (five specialists, three second-year residents) and four standalone frontier LLMs (Gemini 3.1 Pro Preview, Claude Opus 5, GPT-5.6-sol, Kimi K3). All respondents handled the same 150 synthetic Polish primary-care cases through a case-bounded simulated patient (o3-2025-04-16, temperature 1). Central claims: Doctorina reached 82.0% Top-1 diagnostic concordance versus 57.0% for physicians (+25.0 points, 95% CI 17.7–32.7); 97.3% versus 85.0% on primary-or-reference-differential concordance; and higher physician-rated workup (89.4 vs 66.9) and initial-treatment (83.7 vs 61.2) scores. It led all standalone models on both diagnostic endpoints; Opus, Doctorina and Kimi were closely spaced on management, with Opus numerically highest (90.3/86.6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Doctorina and who made it?
Doctorina is a patient-facing clinical AI product — an orchestration scaffold (doctor, assistant, final-recommendation, translation, reference and follow-up agents) around OpenAI models (o3, GPT-5, GPT-5-mini, o3-mini) — developed by A.I. Doctor Medical Assist LTD, the company employing five of the paper's nine authors.
Did the AI really beat real doctors?
In this study, yes — but only in a synthetic, text-based simulation: 150 invented Polish primary-care cases with an LLM-simulated patient, versus eight Polish physicians working unaided in one six-hour session. Doctorina matched the constructed reference diagnosis more often (82.0% vs 57.0% Top-1) and got higher workup/treatment ratings. No real patients or clinical workflows were involved.
Which model performed best in this study?
Among standalone LLMs, Kimi K3 scored highest on diagnosis (78.7% Top-1 concordance) and Claude Opus 5 on management (90.3 workup, 86.6 treatment on the 0–100 scale). The full Doctorina system led all standalone models on both diagnostic endpoints (82.0% Top-1; 97.3% broader endpoint), but its management edge over Opus and Kimi reversed in the second sensitivity run.
Is the study data and code publicly available?
Partially. Deidentified analysis ledgers (Supplementary Data 1–10), analysis code with a fixed bootstrap seed, and validation code accompany the article. Full case narratives, dialogues and execution records are available only on request, under a signed confidentiality and data-use agreement.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Supplementary Data 1–10, the supplementary code and all figures/tables were not part of the provided text; every number cited here is as reported in the manuscript body and could not be independently recomputed. The provided body text itself appears complete (ends with the full reference list).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.