LWCal: Loss-Weighted Calibration for Tabular Classifiers with Noisy Calibration Labels
Zeming Liu, Hang Lyu, Hang Lyu, Jingtao Zhang, Yuan Xie, Yuan Xie
cs.LG · cs.AI
Quick verdict
LWCal re-weights noisy calibration labels for isotonic calibration; simple, honestly evaluated, but only synthetic noise on small tabular benchmarks.
Summary
The paper tackles post-hoc probability calibration when the calibration set's labels are themselves noisy — a setting the authors argue is overlooked relative to noisy-label training. LWCal fits weighted isotonic regression on calibration scores, down-weighting examples whose noisy labels incur high cross-entropy against the base model's held-out probability (median/MAD normalization, weights clipped to [0.05, 1]). Gated-LWCal blends the map back toward the raw score when the apparent label-model disagreement rate exceeds a fixed 0.29 threshold. Across nine binary tabular tasks (six scikit-learn-derived, three synthetic), six seeds, symmetric and asymmetric corruption at 10-40%, and three tree ensembles, LWCal attains the lowest average ECE (0.113 vs 0.188 raw) and Gated-LWCal the best proper-score tradeoff (ECE 0.122, NLL 0.396 vs 0.438) over 432 random-forest cells; paired bootstrap CIs exclude zero against raw, Platt, isotonic, and beta calibration. Gains hold at 10-30% noise and across base learners, but vanish at 40%, honestly reported as a stress boundary, with further failures on a sparse and a tiny dataset. Main limitations: noise is entirely synthetic injection; datasets are small and generic; the gate constants were fixed from undocumented pilot behavior; and no naturally noisy corpus is evaluated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Standard post-hoc calibrators become unreliable when calibration labels are corrupted: in the main noisy-cell aggregate, Platt (ECE 0.187) and beta (0.189) are no better than the raw random forest (0.188).
- LWCal achieves the lowest average ECE (0.113, a 40.0% reduction from raw 0.188) over 432 noisy random-forest cells, using no clean labels, no noise-rate estimate, and no retraining.
- Gated-LWCal delivers the best proper-score tradeoff (Brier 0.125, NLL 0.396 vs raw 0.138/0.438), with paired bootstrap CIs excluding zero versus raw, Platt, isotonic, and beta on ECE, Brier, and NLL.
- Benefits are confined to the moderate-noise regime (10-30%); at 40% corruption the raw score is competitive on NLL (0.594 vs Gated-LWCal 0.640), which the authors present as a deployment warning.
- The apparent contradiction rate d̂, computable without clean labels, supports a three-tier operational protocol (Table XI) for when to trust, gate, or abstain from post-hoc calibration.
Strengths
Distinct, well-motivated problem formulation
Separates noisy calibration labels from noisy training labels and clean test labels (Section III.A), positioning against the Platt/isotonic/beta literature that assumes trustworthy calibration splits (Section II); the clean η=0 rows are excluded from headline aggregates but kept for audit.
Honest reporting of negative and boundary results
At 40% noise the raw classifier has the best NLL (0.594 vs Gated-LWCal 0.640, Table IV) and is presented as a stress boundary; Table VIII shows failures on Synthetic Sparse (LWCal ECE 0.166 vs raw 0.146) and tiny Iris (Gated NLL 0.441 vs raw 0.399); the clean-label oracle (ECE 0.057, NLL 0.340) is framed as headroom, not a claim.
Broad factorial evaluation with matched-cell inference
Deployment-aware and auditable design
Hyperparameter and ablation transparency
Weaknesses
External validity limited to small generic tasks with synthetic noise
Undocumented gate-constant selection
Circularity of the reliability signal
Metric and multiple-comparison gaps
Thin competitive baseline coverage
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper targets a narrow but real gap in post-hoc calibration: the calibration split itself carries noisy labels, while test labels stay clean. The authors formalize the setting (noisy train + noisy calibration + clean test, Section III.A) and propose two deterministic calibrators: LWCal, weighted isotonic regression with median/MAD-normalized loss weights (Eqs. 1-2, 5), and Gated-LWCal, which backs off toward the raw score using the apparent contradiction rate d̂ (Eqs. 3, 6). Three claims are supported empirically: (1) standard calibrators fail under calibration-label noise — in Table III, Platt ECE 0.187 and beta 0.189 are indistinguishable from the raw random forest's 0.188; (2) LWCal attains the lowest average ECE (0.113, a 40.0% reduction) while Gated-LWCal offers the best proper-score tradeoff (NLL 0.438→0.396, Brier 0.138→0.125) over 432 noisy random-forest cells; (3) gains hold across symmetric/asymmetric noise and three tree ensembles in the 10-30% regime, with a declared failure boundary at 40%.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is LWCal and when should I use it?
LWCal is a CPU-only post-hoc calibrator for binary tabular classifiers whose calibration-set labels may be noisy. It down-weights calibration examples whose noisy labels strongly contradict the base model's held-out probabilities, then fits weighted isotonic regression. It fits settings with suspected 10-30% label corruption and no clean calibration labels; when apparent contradiction exceeds roughly 40%, the paper recommends auditing labels before deploying new probability thresholds.
How much does LWCal improve calibration?
In the main random-forest study over 432 noisy cells, expected calibration error falls from 0.188 (raw) to 0.113 with LWCal and 0.122 with Gated-LWCal, and NLL falls from 0.438 to 0.396 for the gated variant. Gains are largest at 10-30% injected noise and disappear at 40%, where the uncalibrated score is competitive on NLL.
Does LWCal need clean labels or a noise-rate estimate?
No. It uses only calibration probabilities and noisy labels. Weights come from median/MAD-normalized losses, and the gate uses an apparent contradiction rate computed on the same split. Clean labels appear only in the final evaluation, never in fitting.
Is code and data available?
Yes. The paper points to a public artifact (github.com/ZemingL/lwcal) containing scripts, CSV result tables, figures, LaTeX tables, and a one-command reproduction. The experiment uses only local scikit-learn datasets and synthetic generators, runs in about six minutes on CPU with fixed seeds, and claims rounded headline tables are stable across reruns.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-6 are referenced but not analyzable from the extracted text; figure-level claims (reliability curves in Fig. 4, tradeoff geometry in Fig. 3, trimming sweep shape in Fig. 6) rest on captions and tables.
- Contributions cite five noise rates while Table IV reports four noisy rates (10-40%), presumably counting the clean η=0 row as the fifth; the 324-cell robustness total implies three rate levels although the text says 20-30%.
- The public artifact (github.com/ZemingL/lwcal) could not be inspected from the paper text; determinism, file layout, and runtime claims are taken on trust.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.