Understanding the Limits of Agentic ICD Coding
Chong Yock Eng, Yushi Cao, Yiming Chen, Kezhi Mao, Hongchao Jiang
cs.CL · cs.AI · cs.LG
Quick verdict
Claims neural and workflow ICD coders fail orthogonally on rare vs multi-step codes, and a tool-augmented agent partially recovers; credible but small-sample.
Summary
This empirical study evaluates three paradigms for automated ICD-10-CM coding — a neural classifier (PLM-ICD), a fixed workflow agentic system (CodeSeeker), and a free-form tool-augmented LLM agent — on rarity-stratified MIMIC-IV discharge summaries. The authors sample 150 notes (50 each across rare, uncommon, common strata by training frequency) plus a 153-note multi-step subset requiring Chapter 19/20 injury and external-cause codes. They report two orthogonal failure modes: PLM-ICD shows a 0.43 micro-F1 gap between rare (0.227) and common (0.656) codes, while CodeSeeker scores near zero (0.010–0.027) on multi-step codes despite strong rare-code performance (0.560 with Opus 4.6). The free-form agent with nine discovery-gated tools reaches 0.276–0.363 micro-F1 on the multi-step subset with Claude Opus 4.6, with wide confidence intervals. Failure analysis attributes 66–86% of missed multi-step codes to the discovery stage, not verification. Ablations show tool access roughly doubles rare-code F1, and removing prompt-level sequencing instructions drops GPT-5.2's common-code F1 from 0.322 to 0.091. Limitations include a single institution, MIMIC-IV annotation noise, a small multi-step subset confined to two ICD chapters, indistinguishable rankings among agentic systems, fixed commercial model versions, and no stated code release; the abstract's 'recovers up to 0.34' figure matches no table cell.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Neural ICD classifiers show a 0.43 micro-F1 gap between rare (0.227) and common (0.656) codes, driven by training-distribution bias.
- Workflow-driven agentic systems (CodeSeeker) score near zero (0.010–0.027) on multi-step injury/external-cause codes despite strong rare-code performance (0.560 with Opus 4.6).
- A free-form agentic framework with nine discovery-gated tools and prompt-level sequencing guidance recovers substantial multi-step performance (0.276–0.363 with Opus 4.6).
- Discovery, not verification, is the multi-step bottleneck: 66–86% of missed codes were never surfaced by any tool, and only 0–3% of misses are wrong-7th-character errors.
- No single system dominates: PLM-ICD leads on common codes, LLM-based systems on rare codes, and the agentic framework on multi-step codes.
Strengths
Controlled rarity stratification targets a documented benchmark blind spot
150 notes (50 per bin by minimum training frequency) plus a Chapter 19/20 multi-step subset directly probe the rare-code degradation (Edin 2023; Gan 2025) that aggregate MIMIC metrics mask; per-bin micro-F1 with bootstrap CIs (§2.1–2.2, Table 1).
Discovery-gated tool layer is a clean anti-hallucination mechanism
Submission rejects codes not previously surfaced by search/browse (§2.3); FP analysis shows 74–95% of false positives were verified-then-submitted, so the gate holds and residual error is clinical judgment (§4, Table 3).
Failure localisation is concrete and actionable
Honest uncertainty reporting
Component-level ablations across models
Weaknesses
Headline generalisation rests on a single workflow system
Small samples produce wide intervals that outrun abstract-level claims
Cross-paradigm comparability is asymmetric
Substantial prompt-encoded expertise confounds the agentic gain
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.13806) evaluates three paradigms for automated ICD-10-CM diagnosis coding on MIMIC-IV discharge summaries: a trained neural classifier (PLM-ICD, threshold 0.45), a fixed workflow agentic system (CodeSeeker/CLH, with Opus 4.6 and GPT-5.2 backbones), and the authors' free-form agentic configuration — one LLM with nine tools across search/verification/reference/submission, gated so a code can only be submitted if a tool previously surfaced it. Six further LLMs run zero-shot. The central claim is that two orthogonal failure modes exist: neural models collapse on rare codes (0.227 vs 0.656 micro-F1, a 0.43 gap; Table 1 left), while workflow systems collapse on multi-step injury/external-cause codes (0.010–0.027; Table 1 right) that the agentic configuration partially recovers (0.276/0.363/0.280 across rarity bins with Opus 4.6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What are the two failure modes in automated ICD-10-CM coding identified by this paper?
The paper finds that neural classifiers (PLM-ICD) degrade sharply on rare codes — micro-F1 of 0.227 on rare versus 0.656 on common codes, a 0.43 gap — while the workflow agent CodeSeeker, despite handling rare codes well (0.560), scores near zero (0.010–0.027) on injury and external-cause codes that require multi-step guideline navigation. The two failure modes are independent of each other.
How does the tool-augmented LLM agent perform on ICD coding?
With access to nine structured tools (alphabetical index search, tabular verification, external-cause index, etc.) and a rule that a code can only be submitted if first discovered via a tool, Claude Opus 4.6 reached 0.276–0.363 micro-F1 on the multi-step injury/external-cause subset, versus near-zero for the workflow system, though confidence intervals were wide. On common codes the trained neural model PLM-ICD still led all systems (0.656).
What data was used and can the study be reproduced?
The evaluation used MIMIC-IV discharge summaries (a credentialed-access dataset): 150 notes stratified by code rarity plus a 153-note subset with ICD Chapter 19/20 codes. The paper describes the full tool environment, system prompt, and compute budgets, but states no code release; reproduction also requires commercial LLM APIs (e.g., Claude Opus 4.6, GPT-5.2) and training PLM-ICD.
Why did the workflow system fail on injury and external cause codes?
CodeSeeker's fixed Analyze-Locate-Assign-Verify pipeline could not surface paired external-cause and place-of-occurrence codes: 66–86% of missed multi-step codes were never returned by any retrieval step, concentrated in Chapter 20 where the External Cause Index was underused. Verification was not the problem — fewer than 3% of misses involved a wrong 7th-character extension.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2–3 are charts; their data-point values are not in the extracted text, so PLM-as-tool effects (§4, Figure 3) are judged from the prose, not exact numbers.
- The PDF byline reads 'Eng Chong Yock' while the metadata lists 'Chong Yock Eng' — treated as the same first author.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.