LearnActCoder: Role-Aware Error Memory for Adaptive Clinical Coding Agents
Meysam Ghaffari, Bhaskar Sen, Nasim Sabetpour, Nina Fatehi, Animesh Agarwal, Carlos Morato
cs.AI · cs.MA
Quick verdict
LearnActCoder claims structured error memory yields mixed precision/recall gains; framing is honest, but key routing ablations are missing.
Summary
The paper introduces Learn-Then-Act, an inference-time framework that converts prediction errors from a small 25-note labeled batch into a structured Mistake Knowledge Database (MistakeKDB) for clinical coding agents. False-negative lessons are injected into a recall-oriented Coder prompt and false-positive lessons into a precision-oriented Judge prompt, within a fixed GPT-5 Coder–Judge pipeline with lookup-table grounding. On 150 matched MIMIC-III notes, structured memory significantly improves mean per-note CPT F1 from 11.5% to 17.4% (+5.9 points, Wilcoxon p=2.4e-05), while raw-example and reflection-style memories of the same error pool stay near baseline; ICD-9 does not improve significantly. On 250 matched MIMIC-IV notes, memory raises ICD-10 precision (+4.2 points) at a recall cost (−3.1 points), leaving F1 unchanged; a 1,000-note held-out run shows a stable ICD operating point. Effects are heterogeneous, and an appendix component ladder shows the full system's F1 below a zero-shot single call, which the authors interpret as operating-point control rather than performance loss. The review flags the missing routing ablation, the non-adjudicated 25-note LEARN set, single-model evaluation, low absolute CPT performance, and no code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
4 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Converting errors from a small labeled LEARN batch into a structured MistakeKDB and routing FN lessons to a recall-oriented Coder and FP lessons to a precision-oriented Judge improves CPT F1 by 5.9 percentage points on 150 matched MIMIC-III notes, without weight updates.
- Raw-example and reflection-style memories of the same underlying error pool stay near the no-memory CPT baseline, suggesting structured representation is what matters for the procedural-code gain.
- On matched MIMIC-IV notes, memory raises ICD-10 precision (+4.2 points) at a recall cost (−3.1 points), leaving F1 statistically unchanged; the effect is operating-point control, not uniform accuracy gain.
- The same 25-note MIMIC-III memory maintains a stable ICD-9 operating point from 250 to 1,000 held-out notes (43.8% vs 43.7% mean per-note F1).
Strengths
Matched, paired evaluation design with honest null reporting
Matched-note ablations with paired-bootstrap CIs (B=10,000; Wilcoxon p-values; W/L/T counts) and explicit null reporting: ICD-9 CI crossing zero (Table 1) and MIMIC-IV F1 unchanged (Table 2) are stated as nulls rather than spun as wins
Well-controlled memory-format ablation isolating representation
Table 3 holds the same 1,028-error pool, notes, and pipeline fixed across no-memory/raw/reflection/structured; structured memory alone moves CPT F1 (17.4 vs 11.5–12.1), isolating representation from mere memory exposure
Careful epistemic labeling of labels and grounding
System-level latency transparency
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
LearnActCoder (Optum AI; arXiv 2609.19721) proposes **Learn-Then-Act**, an inference-time adaptation framework for clinical coding agents. Errors from a 25-note labeled LEARN batch are distilled by a MistakeAnalyzer into a structured **Mistake Knowledge Database (MistakeKDB)**; false-negative lessons are injected into a recall-oriented Coder prompt, false-positive lessons into a precision-oriented Judge prompt, with model weights and the Coder–Judge workflow fixed. The paper's central empirical claim is deliberately narrow: a significant **CPT F1 gain of +5.9 points on 150 matched MIMIC-III notes** (11.5→17.4, Table 1), while ICD-9 (+1.5, p=0.11) and MIMIC-IV ICD-10 F1 (+1.0, p=0.30) show no significant change — on MIMIC-IV the memory trades precision (+4.2) for recall (−3.1). The authors consistently frame the contribution as **operating-point control rather than universal accuracy improvement**, which is a model of honest framing (Abstract, §5.1, §6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 0 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main result of the LearnActCoder paper?
In a 150-note matched MIMIC-III evaluation, structured mistake memory raised mean per-note CPT F1 from 11.5% to 17.4% (+5.9 points, paired-bootstrap 95% CI [+3.3, +8.6], Wilcoxon p=2.4e-05). This is a large relative gain on low absolute performance. ICD-9 F1 showed no significant change, and on MIMIC-IV memory raised ICD-10 precision (+4.2 points) at a recall cost (−3.1 points), leaving F1 statistically unchanged.
How does LearnActCoder work?
A recall-oriented Coder proposes codes, a lookup database checks that identifiers exist, and a precision-oriented Judge reviews clinical support, for up to three Coder–Judge rounds using GPT-5. A separate MistakeAnalyzer converts errors on a 25-note labeled batch into a structured Mistake Knowledge Database; false-negative lessons are injected into the Coder's prompt and false-positive lessons into the Judge's prompt at inference time, with no weight updates.
Does the mistake memory improve overall coding accuracy?
No. Results are heterogeneous: the significant gain is CPT F1 on MIMIC-III; ICD-9 improvement is not significant, and on MIMIC-IV memory shifts ICD-10 coding toward higher precision at lower recall with F1 unchanged. Absolute CPT/HCPCS performance also remains low (~15–17% F1), and one appendix component analysis shows the full system's F1 below a zero-shot single call.
What are the paper's main limitations?
Evaluation is retrospective on MIMIC data. The role-aware routing strategy was not ablated against symmetric or swapped routing, the 25-note LEARN batch was machine-generated and not coder-adjudicated, only one model family (GPT-5) was tested, absolute CPT performance remains low, and no code release is stated in the text.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 and prompt templates are not analyzable from the extracted text; no MistakeKDB entry examples are shown in the provided text
- The text may omit acknowledgment/data-use and code-release statements if they were beyond the extracted sections
- Appendix A.4 (component ladder) is summarized in one paragraph; per-component CIs are not provided in the text
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.