A decision-basis contract for auditable LLM-assisted medical billing verification: deterministic rules, verbatim evidence, and fail-closed abstention
Jan Hölter, Kevin Geis, Benjamin Raab, Boris Bauke
cs.SE · cs.AI
Quick verdict
PoC audit contract for LLM billing checks: transparent fail-closed design, honestly reported, yet no consistent accuracy gain in single-run synthetic tests.
Summary
This proof-of-concept paper proposes a decision-basis contract for LLM-assisted medical billing verification that separates deterministic checks of a versioned fee catalog (release resolution, code availability, summed-count maxima, symmetric exclusions) from LLM assessment of free-text documentation (supported, contradicted, or required-attribute-missing). Support and contradiction require a verbatim evidence span; unavailable rule context or failed assessments convert to fail-closed abstention rather than acceptance. Four open-weight models (gemma4:31b, medgemma:27b, qwen3:14b, qwen3:4b) were run locally at temperature 0 on a synthetic 14-code GOÄ-style catalog with two releases and 36 author-curated cases (83 items, 79 reference labels), under three conditions: full contract, ablation without documentation-requirement fields, and an end-to-end baseline. Outcome agreement showed no consistent advantage: the system beat the baseline for medgemma:27b (22 vs 19 of 36) and qwen3:14b (33 vs 32) but lost for gemma4:31b (28 vs 35) and qwen3:4b (28 vs 32). Explicit documentation requirements improved identification of the 14 missing-information items for all four models. The evidence gate converted 21 raw verdicts to DECISION BASIS INCOMPLETE, 19 of which had matched the reference. Limitations: single run per model with no variance or inferential statistics, synthetic LLM-drafted data without independent annotation, and the claimed benefit of inspectability remains unvalidated with human reviewers.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A two-layer decision-basis contract separating deterministic catalog checks from LLM documentation assessment makes every billing decision traceable to explicit basis entries (Section 3.1, Algorithm 1).
- Fail-closed mechanisms — the evidence gate and unavailable-assessment handling — make a false SUPPORTED outcome unreachable by construction for valid inputs.
- Explicit documentation-requirement fields improve raw identification of the 14 required-attribute-missing items for all four models (e.g., gemma4:31b 13 vs 7, medgemma:27b 5 vs 0).
- Outcome agreement shows no consistent advantage over the end-to-end baseline across the four models (Section 4, Table 3).
Strengths
Fully specified, implementable contract
Section 3.1, Table 1 and Algorithm 1 fix every basis code, the assessor interface, and the outcome precedence (NOT SUPPORTED > NOT EVALUABLE > INSUFFICIENT DOCUMENTATION > SUPPORTED); the completeness invariant that prevents a false SUPPORTED is stated and enforced in the implementation.
Exceptional reproducibility materials
Appendix A publishes the full assessor and baseline system prompts plus a schema'd example; Appendix B prints the 14-code synthetic catalog and all 36 cases with verbatim documentation; models are pinned to Ollama 0.31.1 with temperature 0 and 8192-token context.
Honest, non-overclaiming results narrative
Diagnostics that localize failure modes
Weaknesses
Evaluation lacks statistical power
Core benefit (auditability) never measured
Reference dataset validity is thin
Literal evidence gate is brittle
No gate-isolated ablation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This cs.SE/cs.AI proof of concept (Section 1) proposes a decision-basis contract for LLM-assisted medical billing verification. The deterministic layer resolves the applicable catalog release and checks code availability, summed-count maxima and symmetric exclusions; the semantic layer has an LLM classify each claimed item as DOCUMENTATION SUPPORT, DOCUMENTATION CONTRADICTION or REQUIRED ATTRIBUTE MISSING (Section 3.1, Table 1, Algorithm 1). Support and contradiction survive only with a verbatim evidence span; unavailable rule context, assessor failure or missing evidence produces fail-closed abstention (DECISION BASIS INCOMPLETE / NOT EVALUABLE), with outcome precedence NOT SUPPORTED > NOT EVALUABLE > INSUFFICIENT DOCUMENTATION > SUPPORTED. The paper claims: (1) traceable decisions via explicit basis entries; (2) construction-level prevention of false SUPPORTED; (3) documentation-requirement fields help all four models detect missing information; (4) no consistent outcome-agreement advantage over an end-to-end baseline. All four claims are supported within the paper's self-declared scope; the practical value of claim (1) is explicitly deferred to future work.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does the decision-basis contract make LLM billing verification more accurate than a plain LLM baseline?
Not consistently. In the paper's single-run evaluation, the contract system beat the end-to-end baseline on 2 of 4 models (medgemma:27b 22 vs 19 of 36 cases; qwen3:14b 33 vs 32) and lost on 2 (gemma4:31b 28 vs 35; qwen3:4b 28 vs 32). The claimed benefit is inspectability and traceability of decisions, not accuracy.
What is fail-closed abstention in this paper?
If the applicable catalog release cannot be resolved, the assessor fails, or a support/contradiction verdict lacks a verbatim evidence span from the documentation, the system withholds SUPPORTED and records an explicit abstention reason (DECISION BASIS INCOMPLETE or NOT EVALUABLE) instead of guessing. It guarantees support is never granted on unavailable information, not that granted judgments are correct.
What data was the paper evaluated on?
A fully synthetic 14-code catalog modeled on the German GOÄ fee schedule with two releases, plus 36 German cases drafted by an LLM and curated by two authors — 83 claimed items and 79 reference verdicts in total. The full catalog and all 36 cases are printed verbatim in the paper's Appendix B.
Which models were evaluated and how?
Four open-weight models run locally through Ollama 0.31.1 at temperature 0 with an 8192-token context: gemma4:31b, medgemma:27b, qwen3:14b and qwen3:4b. Each was run once per condition (system, system-ablation, end-to-end baseline) over all 36 cases; no repeated runs or variance estimates were made.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- German umlauts are extracted with spacing artifacts (e.g., H¨olter, GO¨A); the text is otherwise complete through reference [12] — no truncation observed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.