WiC is Not WSD: A Study on LLMs and Lexical Ambiguity Resolution
Yi Zhou, Kiamehr Rezaee, Danushka Bollegala, Mohammad Taher Pilehvar, Jose Camacho-Collados
cs.CL
Quick verdict
Claims WiC difficulty stems from unspecified sense granularity; ablations and human error analysis support it, though significance tests are absent.
Summary
The paper argues that the Word-in-Context (WiC) task remains difficult for LLMs largely because it provides no explicit sense inventory specifying the granularity at which 'same meaning' should be judged, unlike Word Sense Disambiguation (WSD). The authors evaluate six open models (Llama3, DeepSeek-R1-distill, Mistral families; 7B–70B) on coarse-grained WiC built from CWSD-20 (2,798 pairs), a new fine-grained WiC dataset constructed from WordNet 3.1 examples (1,185 pairs), and coarse-grained WSD, crossing candidate-sense options (Options±) with chain-of-thought (CoT±). Providing candidate senses improves mean accuracy in all configurations (coarse WiC 74.9→81.0 without CoT, 78.5→84.9 with; fine WiC 67.7→71.9), and WiC-through-WSD decomposition mainly helps small models (Mistral-S 61.5→82.3). WSD remains easier than WiC even with the same inventory (92.7 vs 84.9 mean). Human re-annotation of 140 fine and 120 coarse sampled errors — by two annotators, only 40 double-annotated each, no agreement statistics — finds 76/140 fine-grained 'errors' judged model-correct, suggesting label ambiguity and granularity mismatch rather than comprehension failure; overly fine-grained distinctions dominate. Limitations include absence of significance testing, a dated model set, English-only scope, and an under-specified new dataset without a stated code/data release.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Providing an explicit candidate sense inventory improves WiC accuracy in all evaluated settings, with the largest gains for smaller models.
- WiC remains harder than WSD even under the identical sense inventory (84.9 vs 92.7 mean accuracy), so comparison under unspecified granularity adds difficulty beyond single-context sense selection.
- Many apparent WiC errors reflect dataset label ambiguity or model–annotator granularity mismatch rather than lexical comprehension failure (76/140 fine-grained sampled errors judged model-correct).
- LLMs tend toward overly fine-grained sense distinctions, and chain-of-thought prompting can amplify this in fine-grained settings.
Strengths
Hypothesis-driven, well-controlled design
The 2×2 factorial (Options± × CoT±) across coarse WiC, fine WiC and WSD (Section 4.2–5, Table 3) directly tests the granularity hypothesis, and the same-inventory comparison (WiC 84.9 vs WSD 92.7 mean) isolates the comparison step rather than just benchmarking.
Contamination-aware new benchmark
The fine-grained WiC dataset (Section 4.2, Tables 2/8: 1,185 pairs, 924 target words, near-balanced 594/591) is built from WordNet 3.1 usage examples explicitly to avoid memorisation of the widely circulated original WiC.
Informative human error analysis
Practical decomposition finding and per-word transparency
Weaknesses
No statistical inference anywhere
Human annotation rigor is thin
Dated and narrow model pool
Inconsistent contamination policy and thin dataset documentation
Abstract overclaim nuance
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks why Word-in-Context (WiC) remains hard for LLMs and hypothesises that the root cause is not the comparison of two contexts per se but the absence of an explicit sense inventory pinning down the granularity at which 'same meaning' is judged — the property distinguishing WiC from Word Sense Disambiguation (Section 3). Central claims: (1) adding candidate senses (Options+) improves WiC accuracy in all settings; (2) WiC stays harder than WSD under the identical inventory (84.9 vs 92.7 mean with Options+CoT+, Table 3); (3) human re-annotation shows many apparent WiC errors are label ambiguity or model–annotator granularity mismatch, with models erring toward overly fine-grained distinctions. Scope: six open-weight models, English only, no proprietary systems.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why do LLMs struggle with the Word-in-Context (WiC) task?
According to this paper, the main cause is not poor comprehension but the absence of an explicit sense inventory: WiC never tells the model how fine-grained the 'same meaning' judgment should be. When candidate senses are added to the prompt, accuracy rises in every tested setting, and human analysis shows many apparent errors are granularity mismatches rather than misunderstanding.
What is the difference between WiC and Word Sense Disambiguation (WSD)?
WSD gives the model a predefined list of candidate senses for the target word and asks it to pick the right one; WiC gives two sentences containing the same word and asks whether the meaning is the same, with no inventory. This paper shows the two are not interchangeable as evaluations: even when WiC is given the same sense inventory, it remains harder (84.9% vs 92.7% mean accuracy).
Is there label noise in the WiC benchmark?
The paper's human re-annotation found that 76 of 140 sampled fine-grained 'model errors' were actually judged correct by human annotators, and 39 of 120 in the coarse setting, suggesting substantial label ambiguity — WiC scores may understate models' true lexical understanding.
Does chain-of-thought prompting help with WiC?
It helps when sense distinctions are coarse (mean accuracy up from 74.9% to 78.5% without candidate options, and 81.0% to 84.9% with options) but slightly hurts on fine-grained distinctions (67.7%→66.8%), because free-form reasoning can amplify small contextual differences into overly fine-grained sense splits.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 are available only as captions; the violin-plot distributions in Figure 2 could not be inspected (mean values taken from prose).
- Review is based on extracted text; table formatting details (e.g., bolding of best scores in Table 3) verified only where repeated in prose.
- No code or data release statement appears in the extracted text; availability may exist outside it (e.g., a footnote or external page).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.