Towards a Deterministic Math Solver for Clinical Language Models
Felipe Ocampo Osorio, Sebastián Andrés Cajas Ordoñez, Maximin Lange, Rafi Al Attrach, Sahil Kapadia, Zakaria Laouabdia Sellami, Angelo Antonio Talio, Leo Anthony Celi
cs.AI · cs.SE
Quick verdict
Tests LLM-written Python vs direct arithmetic on MedCalc-Bench: solver wins only at 32B (+7.05pp, CI barely clears zero); rigorous, honest evaluation.
Summary
The paper asks whether clinical language models should compute directly or generate case-specific Python that a restricted local executor runs deterministically. On MedCalc-Bench Verified (1,100 cases, 55 calculators), the authors compare Program-Solve against direct arithmetic with formula, gold variables and whole-note access matched, plus a hand-written 22-calculator library, using Qwen2.5-7B and Qwen2.5-32B-AWQ (plus Mistral-7B and Phi-3.5-mini) over five seeds. Handing off to the solver is not a reliable advantage at 7B (75.31% vs 72.02%, +3.29pp, calculator-cluster CI [-3.49, 10.38]) but is at 32B (90.53% vs 83.47%, +7.05pp [0.47, 14.60]). The library is exact on its 440 covered cases yet scores 40.0% overall under abstention; a Gold-first hybrid reaches 81.64%/91.07%. Removing formula and variable access collapses the program route by roughly 46-47pp. The authors also audited the benchmark: 16 of 55 calculators have version/use/coefficient concerns, and 28 supplied formula texts were wrong or incomplete, fixed before rerunning. Limitations include the quantization confound (4-bit 32B vs bf16 7B), only two primary checkpoints with divergent secondary models, an exploratory prompt selected on the test split, and gold variables supplied in the main comparison, limiting deployment inference.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- With formula text, gold variables and whole-note access matched, program generation is not reliably more accurate than direct arithmetic at Qwen2.5-7B (+3.29pp, cluster CI [-3.49, 10.38]) but is at Qwen2.5-32B-AWQ (+7.05pp [0.47, 14.60]).
- The program route's advantage over the 22-calculator hand-written library comes almost entirely from coverage: the library is exact on its 440 supported cases (100% with gold variables) but abstains on the other 660, scoring 40.0% overall.
- Formula and variable access dominate performance: removing both collapses Program-Solve by 47.27pp (7B) and 45.82pp (32B), and the design does not separate formula recall from variable extraction.
- The benchmark itself is unreliable: 16 of 55 calculators have version, use or coefficient concerns, and 28 of 55 supplied formula texts were wrong or incomplete (10 could not reproduce the benchmark's own number) until the authors corrected them.
- The clinically defensible configuration is a hybrid: the verified library where it covers, program generation elsewhere (81.64% at 7B, 91.07% at 32B), with explicit abstention when neither can be trusted.
Strengths
Genuinely matched comparison isolating execution
Program-Solve and Open-book arithmetic share formula text, gold variables and whole-note access (Section 2, Table 1), so the headline pair (+3.29pp at 7B, +7.05pp at 32B) manipulates only the execution route — a design most program-aided papers skip.
Statistically careful and self-skeptical reporting
Primary inference is a calculator-cluster bootstrap (10,000 draws, seeds pooled) with ICC 0.68-0.81 and neff 67-79 reported (Table 7); secondary tests Holm-corrected; the text explicitly states intervals crossing zero establish neither difference nor equivalence.
Benchmark auditing as a first-class contribution
Failure analysis with audited taxonomy
Negative and boundary results reported, not buried
Weaknesses
Quantization confounds the scale claim
Headline 32B interval is thin
Gold variables limit deployment relevance
Opportunistic library weakens the library comparison
Single-annotator audits
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks one controlled question: given a clinical calculator's formula, the gold variables and the full note, does an open-weight LLM gain accuracy by writing case-specific Python for a restricted executor (Program-Solve) rather than doing the arithmetic itself, and how does either fare against a hand-written function library? The headline claim is deliberately narrow and scale-dependent: the solver handoff is not a reliable advantage at 7B (75.31% vs 72.02%, +3.29pp, calculator-cluster CI [-3.49, 10.38]) but is one at 32B (90.53% vs 83.47%, +7.05pp [0.47, 14.60]). Secondary claims: the 22-calculator library's 40.0% full-set score is a coverage artifact (exact on its 440 cases, abstaining on 660); a Gold-first hybrid is the strongest configuration (81.64% / 91.07%, Table 15); and benchmark formula provenance is itself a problem — 16 of 55 calculators flagged, four superseded by current guidelines (Table 11).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Program-Solve?
An interface where the language model does not do arithmetic itself. Given the clinical note, the calculator's formula text and the variables, the model writes a short case-specific Python program, and a restricted local executor (allow-listed builtins, only the standard math/datetime modules, 5-second and 256 MB caps, no calculator-specific code) runs it and returns the number or date deterministically.
Does letting an LLM write code improve clinical calculation accuracy?
In this matched study on MedCalc-Bench Verified, it depends on the model: Qwen2.5-32B gained a reliable +7.05 points over direct arithmetic (90.53% vs 83.47%), Qwen2.5-7B showed no reliable gain (+3.29 points, confidence interval crossing zero), and Mistral-7B actually did 7.62 points worse with code. Execution helps some open-weight models, not others.
What did the authors find wrong with MedCalc-Bench?
Auditing all 55 calculators, they flagged 16 with version, usage or coefficient concerns (four formulas superseded by current clinical guidelines, e.g. MELD-Na replaced by MELD 3.0), and found 28 of the 55 supplied formula texts wrong or incomplete — 10 could not even reproduce the benchmark's own gold answers. They corrected all formula texts and reran every arm before reporting results.
Is a hand-written calculator library better than letting the model generate code?
Where the library covers a calculator it is perfect (100% on its 440 supported cases), but it must abstain on the other 660, capping full-set accuracy at 40.0%. The best configuration in the paper is a hybrid: use the verified library when it covers the case, program generation otherwise, reaching 81.64% (7B) and 91.07% (32B).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 and Figure 2 internal labels (axis titles, legend entries) are garbled by PDF text extraction; all table numbers cited in this review extracted cleanly from the appendix tables.
- The provided text ends after Table 24 with no sign of truncation; apparent shortfalls such as Table 9 listing only the top six calculators per family are the authors' own design, not extraction loss.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.