English Word Sense Disambiguation in 2026: When the Labels Become the Bottleneck
Vassili Philippov, Amro Salman, Dmitrii Andreev, Penny Hands, Emil Kaiumov, Pavel Katunin, Anton Nikolaev
cs.CL
Quick verdict
Claims WSD gold labels—not models—are now the bottleneck; a rigorous study with strong controls, though the human-equivalence claim leans on its own inventory.
Summary
The paper argues that in English all-words word sense disambiguation (WSD), gold labels rather than models now decide benchmark outcomes. It contributes lexEN, a conservative, auditable correction layer over Maru2022's ALL_NEW: a model panel flags 363 of 4,917 items, three professional lexicographers review them independently, and a frozen two-of-three rule changes 211 labels and removes 56 items, yielding 4,861 instances with full lineage. A companion harness, SenseBench, adds immutable prompts, raw-call artifacts re-verified in CI, and a 57-model leaderboard. Frontier LLMs converge near 95% accuracy (best 95.6%, GPT-5.5) with the top three families statistically indistinguishable; re-scoring fixed predictions against Raganato, Maru, and lexEN labels lifts LLMs by up to 10.2 points but supervised systems by under one point on the lexEN-specific step. Relabeling SemCor with GPT-5.5 or Gemma 4 31B and retraining BEM, ESCHER, and ConSeC unchanged yields +2.4 to +5.5 points on test sets the relabeling never touched; the released Glite LENS bi-encoder (298M) reports 83.6 Raganato ALL and 87.4 Maru ALL_NEW at roughly $0.13 per million items. Expert agreement at fine WordNet granularity is moderate (Fleiss kappa 0.537), coarsening raises it to 0.740, and at coarse granularity a top model falls inside the expert agreement band. Main limitations: the reviewed subset is model-selected, the training-corpus relabeling is not human-verified, and multiple-choice scores are ceilings.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- In the near-saturated regime, residual errors in WSD gold labels — not model capability — decide benchmark rankings, and a three-lexicographer adjudicated layer (lexEN) corrects 211 labels and removes 56 items from Maru2022's ALL_NEW.
- Frontier LLMs converge near 95% accuracy on lexEN-v1 under inventory-constrained multiple choice, with the top three families statistically indistinguishable and accuracy trading off against cost across a >1,200x price span.
- Relabeling SemCor with frontier models and retraining unchanged BEM, ESCHER, and ConSeC lifts them by 2.4 to 5.5 points on test sets the relabeling never touched, showing training-label noise helped cause the decade-long 79-83% plateau.
- Fine-grained WordNet senses are partly ill-posed even for experts (three-reviewer Fleiss kappa 0.537); coarsening raises expert agreement and model accuracy together, placing a top model inside the expert agreement band at coarse granularity.
- Glite LENS, a 298M bi-encoder trained on relabeled SemCor, is the strongest reported bi-encoder (83.6 Raganato ALL, 87.4 Maru ALL_NEW) and serves at roughly $0.13 per million items, making cost the next binding constraint.
Strengths
Unusually rigorous benchmark governance
Section 3 and Table 3: three independent lexicographers review 363 model-flagged items with provenance hidden, a frozen two-of-three rule is applied mechanically (124 unanimous, 183 two-of-three, 56 removed), and the release is content-hashed, versioned, and carries per-item reviewer choices, rationales, cannot-answer typing, and a contamination canary — every single label change is reconstructible from released artifacts.
Controlled causal complement on the training side
Section 6 and Table 12: relabeling SemCor (226,036 instances) and retraining BEM/ESCHER/ConSeC at frozen published configurations isolates training labels as the only variable, with gains of +2.4 to +5.5 points on Raganato/Maru test sets the relabeling never saw; two independent relabelers (GPT-5.5, open Gemma 4 31B) replicate the near-identical disagreement profile (Figure 3), and the first-sense split analysis (+11.7 to +17.6 points on non-first-sense items, McNemar p <= 3.8e-33) rules out convention alignment.
Statistical methodology above field norm
Candid handling of circularity risks
Cost made a first-class, measured axis
Weaknesses
Gold corrections carry a structural pro-LLM tailwind
The headline equivalence claim rests partly on an author-authored inventory
Training-corpus relabeling is unverified model output at scale
Multiple-choice design caps external comparability of headline numbers
One headline leaderboard row is a two-model composite
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Is English word sense disambiguation solved in 2026?
According to this paper, it depends on granularity. At coarse sense granularity, the best LLMs agree with expert lexicographers about as well as the experts agree with each other, so the task is effectively at the human ceiling there. At fine WordNet granularity it is not solved: even professional lexicographers agree only moderately (kappa about 0.54 on hard items) because the inventory draws finer distinctions than anyone can reliably apply. The remaining differentiator between systems is cost, not accuracy.
What is lexEN and how was it built?
lexEN is a corrected evaluation benchmark for word sense disambiguation built on top of Maru2022's ALL_NEW test set. A panel of language models flagged 363 of 4,917 items whose labels they disputed; three professional lexicographers then reviewed those items independently, without seeing the original labels or each other's choices, and a fixed two-of-three agreement rule changed 211 labels and removed 56 unanswerable items, leaving 4,861 items with a recorded rationale and provenance for every change.
How accurate are LLMs at word sense disambiguation according to this paper?
On the lexEN benchmark, where the candidate WordNet senses are supplied as a multiple-choice list, the best models score about 95% (GPT-5.5: 95.6%), and the top three model families are statistically indistinguishable. These figures are upper bounds because models pick from a provided list. On the 307 hardest expert-reviewed items, fine-grained accuracy drops to about 66%, recovering to roughly 87% when senses are scored at coarse granularity.
What is Glite LENS and how much does it cost to run?
Glite LENS is a released 298-million-parameter bi-encoder trained on SemCor relabeled by GPT-5.5. It reports 83.6 on the standard Raganato test set and 87.4 on Maru's corrected version — described as the strongest bi-encoder result to date — and serves disambiguation at roughly $0.13 per million items, orders of magnitude cheaper than querying a frontier LLM per item, because candidate sense embeddings are precomputed once and inference needs only one context encoding.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extracted text is truncated mid-way through the Limitations section, and Appendices B-D (worked examples, component ablations, reviewer brief) are not included, so some claims (e.g., the component-level attribution in Section C) could only be assessed from their in-text summaries.
- Figures 1-7 were available only as captions and in-text numbers; no visual inspection of plots was possible.
- The paper cites several 2025-2026 systems and models (GPT-5.5, Gemini-3.1-Pro, Claude-Fable-5, SANDWiCH) whose external results could not be cross-checked offline; assessments of the related-work positioning rely on the paper's own characterization.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.