KoNeoBench: A Curated Evaluation Dataset for LLM Understanding of Korean Neologisms
Soha Lee, Soojin Lee, Heesung Yang, Hyunju Song, Hyunji Lee, Jinsan An, Jeongwan Shin, Jin Hyun Park, et al.
cs.CL · cs.AI
Quick verdict
首次声称拥有专家精选的韩国新词基准；GPT-5.4 在语言学训练的人类测试中仍处于落后；资源扎实，但判定验证/统计数据较薄弱。
Summary
论文介绍了 KoNeoBench，这是一个包含 2020–2024 年主要新闻媒体中出现的 1,785 个韩国新词的基准，并按照韩国国立国语院的年度方法进行筛选：在 1,627,371 个自动提取的候选词中，专家词典学评审保留了 1,269 个，加上通过语用标记手动发现的 516 个。条目包含定义、实证使用示例、构词类型、语义类别（16 向）和专业领域（43 向）。四项任务考察了 LLM 的理解能力：完形填空上下文识别（带/不带助词）、396 个混合词/缩写词的词源识别、分类（寻找异类和直接分类）以及词典风格定义生成。评估了九个模型（2023–2026 年，包括 GPT-4.1/5.4、韩国专用的 EXAONE 和 Solar、Qwen、LLaMA）。GPT-5.4 领先，但在上下文识别方面仅达到 66.16%，语义分类达到 66.80%，词源恢复的 F1 值仅为 49.96；没有模型在任务 2 中超过 50 的 F1 值。定义生成（Gemini 判别，专家验证过的准则，在 250 个项目上与人类对比 r=0.708）最能区分各个模型。一个 20 人的基线显示，语言学专业学生在任务 1-2 中的表现优于 GPT-5.4（例如，75.0% 对 60.0%）。局限性：单判别评分仅在 GPT-5.4 的输出上进行了验证，模型之间缺乏显著性检验，仅限新闻的覆盖范围排除了语义新词和社交媒体词汇，且未报告标注者间的一致性。
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing Korean benchmarks center on established vocabulary, making KoNeoBench the first benchmark for multifaceted evaluation of LLM understanding of Korean neologisms.
- Current LLMs show clear limitations on Korean neologisms, with average scores around 30–50/100 and no model exceeding 50 F1 on source-word recovery (best: GPT-5.4 at 49.96).
- Korean's agglutinative structure adds measurable difficulty: attaching case particles (Task 1 Type 2) costs an average 2.24% accuracy across models.
- Definition generation (Task 4) is the most discriminative task and correlates with the others, offering a practical comprehensive measure since annual neologism dictionaries supply gold definitions.
- Providing usage examples substantially improves performance (mean +13.70 on Task 3 Type 2 semantic categorization), suggesting retrieval-augmented approaches as a mitigation.
Strengths
Expert lexicographic curation at real scale
From 1,627,371 automatic OOV candidates (2020–2024, Table 2), expert review retained 1,269 (acceptance rate 0.06–0.09% per year), plus 516 manual additions via pragmatic markers; first-occurrence dates verified against Naver News covering ~130 outlets.
Korean-specific task design
Case-particle variants in Task 1 (Type 1 vs Type 2) isolate agglutinative-morphology effects (avg −2.24% with particles); Task 3 contrasts odd-one-out grouping with direct labeling to separate lexical grouping from explicit category knowledge; Task 1 distractors use relation-based hard negatives, not just random words.
Transparent evaluation protocol
Human baseline with statistical tests and honest judge validation
Actionable finding on usage examples
Weaknesses
Single LLM judge, validated on one model only
No variance or significance reporting for the main comparison
Contamination not audited
Gold-annotation reliability unquantified
Coverage limits are consequential
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
KoNeoBench (arXiv 2609.19916) targets a real gap: Korean evaluation resources (KLUE, KoBEST, HAE-RAE Bench, Ko-H5) are built on established vocabulary, and prior neologism benchmarks (NEO-BENCH, SLANG) are English-centric. The authors assemble 1,785 neologisms attested 2020–2024 via the National Institute of Korean Language's annual survey pipeline: LRNounExtractorv2 candidate extraction against the Standard Korean Dictionary headword list, rule-based filtering against ~330M attested forms, and expert validation that accepts only 1,269 of 1,627,371 candidates (0.06–0.09% per year, Table 2), plus 516 manual additions found via pragmatic markers. Each entry carries a dictionary-style definition, an attested usage example, word-formation type, 16-way semantic category, and 43-way specialized domain. Central claims: (1) this is the first multifaceted Korean neologism benchmark; (2) current LLMs are weak on it (averages ~30–50/100); (3) definition generation is the most discriminative task; (4) usage examples substantially help.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is KoNeoBench?
KoNeoBench is a benchmark of 1,785 Korean neologisms (newly coined words) attested in major Korean news from 2020–2024 and curated by lexicography experts. It tests how well large language models understand recent Korean words through four tasks: contextual identification, source-word recovery, semantic/domain categorization, and dictionary-style definition generation.
How well do LLMs perform on KoNeoBench?
Performance is limited. The best model, GPT-5.4, reaches 66.16% accuracy in contextual identification and 66.80% in semantic categorization, but only 49.96 F1 on source-word recovery — no model exceeds 50 F1 on that task. In a 20-person human baseline, linguistics majors significantly outperform GPT-5.4 on the identification and source-recovery tasks.
Where can I download KoNeoBench?
The dataset is publicly released at https://github.com/bcmilab/ko-neobench/ in CSV format, including for each of the 1,785 neologisms a dictionary-style definition, an attested usage example, word-formation analysis, semantic category, and specialized-domain label.
Why are Korean neologisms hard for LLMs?
The paper identifies three reasons: (1) time drift and low frequency — models score lower on more recently coined items; (2) Korean agglutinative morphology — attaching case particles costs about 2.2% accuracy on average; and (3) irregular word formation — truncated blends and abbreviations make it hard to recover source words from the surface form.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Several sections were marked omitted in extraction (parts of Appendix B.1 evaluation setup, the tail of the Acknowledgments); Figure-embedded numbers (Figures 5, 21–28) are partially garbled, so figure-level claims rely on the surrounding prose.
- Figures could not be analyzed visually; year-wise trends (Figure 6b) and prediction distributions (Figure 5) are taken from the text description only.
- The extracted header lists 11 authors while the metadata lists 8; corresponding authors are Kilim Nam and Hyeyoung Park.
- Licensing terms of the released CSV (derived from National Institute of Korean Language surveys) are not stated in the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.