Error-Type-Aware Loss Reweighting for Robust Named Entity Recognition with Noisy LLM Labels
Elena Merdjanovska, Jonas Golde, Alan Akbik
cs.CL
Quick verdict
Error-type-aware token masking for NER with noisy LLM labels; simple, well-evaluated, modest 0.8-2.0 F1 gains, but no significance testing.
Summary
Large language models increasingly annotate training data for NER, but their heterogeneous errors — missing, hallucinated, mistyped, or partially bounded mentions — propagate to fine-tuned models. The authors argue existing noise-robust losses treat all suspicious tokens alike, which suits classification but not sequence labeling. They propose error-type-aware loss reweighting: tokens where the model disagrees with the label are partitioned by error type (missing vs. entity, optionally type/FP), a two-component Beta mixture is fitted to per-partition confidence scores to derive an adaptive masking threshold, and suspect tokens are hard-masked within a single fine-tuning run — no clean data, no extra training stages. Evaluation covers four datasets (OntoNotes, NoiseBench, BC5CDR, Wikigold), three LLM label variants each from gpt-oss-120b and Qwen3-235B under four prompting schemes, DistilBERT and XLM-RoBERTa-large with three seeds, against CE, GCE, focal, BMM-bootstrap and corrected-NLL losses plus oracle masking bounds and extended distant/crowd/weak noise sources. Dataset-level averages improve 0.8–2.0 F1 with DistilBERT and 1.2–1.6 with XLM-R; the maximum is +4.6 on Wikigold at 24.1% noise. Lmissing (missing-entity masking) is the safest default; global masking often underperforms CE. Limitations: choosing the right loss variant requires knowledge of the error distribution that the no-clean-data setting itself forbids; only English, encoder-only models; appendix tables show severe training instability (stds up to ±44.5); no significance testing is reported.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Fine-tuning on LLM-annotated NER data inherits the annotator's errors, capping downstream F1 within about 3 points of direct prompting across four datasets (Fig. 1a).
- Partitioning disagreeing tokens by error type and masking each partition with its own Beta-mixture confidence threshold outperforms both cross-entropy and global token masking.
- Error-type-aware losses improve dataset-level average F1 by 0.8-2.0 points with DistilBERT and 1.2-1.6 with XLM-RoBERTa-large, with a maximum +4.6 on Wikigold at 24.1% noise.
- Lmissing is the safest default loss, while global masking (Lall) and combined masking (Lmissing,entity) underperform because they remove too much training signal.
- Ideal masking nearly recovers clean-data training performance, establishing substantial headroom for practical masking approaches (Sec. 5.3, Fig. 2).
Strengths
Realistic no-clean-data evaluation protocol
Models train on noisy labels and are evaluated on clean test splits, with model selection on a noisy dev set (Sec. 4) — explicitly adopting the harder setting Zhu et al. (2023) argue for, unlike Self-Cleaning or multi-voter methods that need clean subsets.
Error-type decomposition is empirically grounded, not assumed
Table 1 quantifies FN/FP/Type/Partial shares per label variant; Sec. 5.2 shows the best loss matched the dominant error category in 9 of 12 variants; Appendix B (Table 6) shows missing/partial errors propagate at ~79-81% vs 55% for type errors, explaining why Lmissing works.
Oracle decomposition quantifies headroom
Honest negative results and mechanism analysis
Reproducibility support
Weaknesses
No statistical significance testing
Loss-selection circularity
Unexplained training instability in appendix
Wrong-choice cost is real but unaddressed
Hyperparameter sensitivity unreported
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a deployment pattern that is becoming standard: annotate with an LLM, fine-tune a small NER model. Its empirical premise is solid — the fine-tuned model stays within ~3 F1 points of the prompting annotator itself (Fig. 1a: OntoNotes prompting 66.2 vs fine-tuned 65.5, far below the 82.0 clean-label ceiling), and Appendix B shows error inheritance is type-dependent: ~78.9% of the student's missing errors and ~81.1% of partial-boundary errors land on LLM-error positions, but only 55.1% of type errors. The central claims are that (1) CE ignores this noise, (2) classification-oriented noise-robust losses transfer poorly because NER noise is heterogeneous and O-dominated, and (3) per-error-type masking yields 0.8-2.0 F1 (DistilBERT) and 1.2-1.6 (XLM-R-large) dataset-average gains, max +4.6 on Wikigold at 24.1% noise. I verified all headline ranges against Tables 2/3 and 7/8 — the arithmetic is honest (e.g., NoiseBench avg 73.7→75.7 via Lentity; OntoNotes 58.7→59.5 via Lmissing; XLM-R NoiseBench best 77.1→78.3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What problem does this paper solve?
When LLMs annotate NER training data, their mistakes (missed entities, hallucinated entities, wrong types, partial boundaries) are inherited by the smaller model fine-tuned on that data. The paper proposes loss functions that detect suspicious tokens from model confidence and mask them from training, with separate rules per error type instead of one rule for all noise.
How much does the proposed method improve results?
On dataset-level averages it improves entity-level test F1 by 0.8 to 2.0 points over standard cross-entropy with DistilBERT, and by 1.2 to 1.6 points with XLM-RoBERTa-large. The largest single improvement is +4.6 F1 on Wikigold with 24.1% label noise. The gain depends on matching the loss variant to the dominant error type; the missing-entity loss Lmissing is the safest default.
Does the method need any clean labeled data?
No — masking thresholds are estimated by fitting a Beta mixture model to the model's own confidence scores, and model selection uses a noisy development set. However, choosing WHICH error-type loss to use ideally requires knowing the annotation's error distribution, which the authors note cannot be measured without some clean labels; hence Lmissing is recommended as the default.
Is code and data available for this paper?
Yes. The authors release code at github.com/elenamer/error-type-aware-losses. Experiments use four public datasets (NoiseBench/CoNLL03, OntoNotes, BC5CDR, Wikigold) with LLM annotations generated from gpt-oss-120b and Qwen3-235B using prompting schemes described in the appendix.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are present only as captions plus in-text descriptions; the plotted values (e.g., exact oracle curves in Fig. 2 and threshold trajectories in Fig. 3a) could not be independently verified from the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.