Four Ledgers, Not One Score: Responsible Communication of LLM-Judge Calibration in Biomedical ML
cs.AI
Quick verdict
Planted-card review ratios 0.710/0.218 are yield and composition, not judge precision/recall; rigorous single-workflow forensic audit, prevalence unknown.
Summary
This paper argues that synthetic planted errors are neither detector outputs nor human ground truth, and that collapsing them into one ledger produces metric-shaped numbers that do not measure an LLM judge. The authors formalize four ledgers—planted perturbations (P), independent detector outputs (D), source-linked human dispositions (H), and human-added discoveries (A)—and prove an audit identity: when plant keys substitute for detector outputs, numerical 'precision' reduces to proposal-confirmation yield (Eq. 3) and 'recall' to the confirmed-plant share, with no term depending on the detector, so detectors with different outputs are observationally equivalent. They then forensically audit a private synthetic Japanese care-handoff workflow: 69 planted cards across 47 targets; final review of 22 targets yields 22 confirmed and 9 rejected imports plus 79 human-added cards; passing imports to a generic scorer yields 0.710 and 0.218—identified as yield (Wilson 95% CI 53.4–83.9%) and composition (additions 78.2%), not precision/recall. The audit also finds source-name collisions, row-shadowing read paths, forced severity defaults, and vacuous 1.0 fallbacks. Contributions include a claim contract (Table 2), a four-ledger storage schema, and a six-condition calibration gate. Main limitations: a single private workflow, self-audit without independent database validation, only three double-annotated targets, and descriptive intervals ignoring clustering; the public artifact reproduces arithmetic but cannot replay the private extraction.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A planted mutation key is neither a detector output nor human ground truth, so mixing provenances in one ledger destroys the meaning of precision/recall-shaped counts.
- When plant keys substitute for detector outputs, numerical 'precision' (0.710=22/31) and 'recall' (0.218=22/101) are exactly proposal-confirmation yield and submitted-ledger composition, identifying no operating characteristic of any detector.
- Implementation choices—naming collisions, newest-row shadowing, forced severity defaults, and 1.0 fallbacks for zero denominators—actively manufacture overstated capability claims.
- Judge precision/recall may be claimed only under a six-condition calibration gate (pinned blind detection ledger, dispositioned proposals, documented miss search, frozen items, versioned matching); otherwise the metric is unidentifiable, not zero.
Strengths
Clean identifiability argument, not just an estimation complaint
Section 3's audit identity formally shows that substituting P for D and H_P for H_D makes Eq. 1 reduce to Eq. 3 and Eq. 2 to the confirmed-plant share; since no term depends on a realization of detector g, detectors with arbitrarily different outputs are observationally equivalent under the stored record. This upgrades the critique from 'the estimate is noisy' to 'the estimand is unidentifiable.'
Explicit claim contract for every reported ratio
Table 2 pairs each number (0.710, 0.218, 79/101, 22/47, 3/22) with its identified quantity and the specific unsupported inference; Section 5 correctly labels the 31.9–87.0% range a deterministic missing-review bound rather than a confidence interval. Rare metric hygiene for evaluation papers.
Concrete, repairable implementation forensics
Verifiable arithmetic via a public artifact
Weaknesses
Existence proof on a single private workflow
Self-audit without independent verification
No judge is ever measured, despite the framing
Quotable interval statistics on fragile samples
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a measurement-validity and responsible-communication paper grounded in a single forensic case study of a private, synthetic Japanese care-handoff evaluation workflow (Section 4). The authors claim that (1) synthetic planted errors are neither detector outputs nor human ground truth, so mixing provenances collapses distinct estimands; (2) a four-ledger model (Plant P, Detection D, Disposition H, Discovery A) plus an audit identity shows that 0.710 and 0.218, computed by passing imported plant keys to a generic scorer, are proposal-confirmation yield (Eq. 3) and submitted-ledger composition (Eq. 4), not judge precision/recall; (3) implementation choices manufactured those metric-shaped numbers (Section 6); and (4) a storage contract plus six-condition calibration gate (Section 7) must precede any judge-calibration claim. The paper explicitly claims existence-proof status only—no prevalence estimate, no model ranking, no clinical validity (Abstract, Section 8).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What do the numbers 0.710 and 0.218 in this paper actually measure?
They come from a private review workflow: 0.710 = 22 confirmed out of 22+9=31 imported proposals (a proposal-confirmation yield), and 0.218 = 22 out of 22+79=101 cards in the submitted final-review ledger (the confirmed-plant share). The paper argues neither is judge precision or recall, because no independent detector outputs were ever stored for the audited proposals.
What are the 'four ledgers' the title refers to?
Four separately stored record types: (1) planted perturbations created by the synthetic data factory, (2) independent detector (LLM-judge) outputs, (3) human dispositions (confirm/reject) of source-linked proposals, and (4) human-added discoveries beyond the displayed proposals. Mixing them makes metric-shaped numbers that lose their scientific meaning.
Is the data or code from the paper public?
A public GitHub artifact contains a frozen count sheet, a standard-library script reproducing all ratios and Wilson intervals, an executed audit notebook, a claim crosswalk, and a synthetic fixture. The private workflow data is not released; the artifact reproduces the arithmetic but cannot replay the private extraction.
Does this paper show that LLM judges are unreliable?
No. It shows that one workflow's stored records cannot support any precision/recall claim about its judge at all—an existence proof of a provenance/storage failure mode. It makes no claim about judge quality or about how often such failures occur.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Equations are extracted as linearized text; notation (Eq. 1–5) reconstructed from surrounding prose.
- No figures appear in the extracted text; Tables 1–3 were extracted as text and appear complete.
- The public artifact repository was not inspected (no network access); its contents are taken from the paper's own description.
- Minor PDF-extraction noise (running headers interleaved); abstract is truncated mid-sentence at 'not an estimate of its prevalence' but Section 8 restates the same claim.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.