An LLM-Associated Register Shift in Korean Journal Abstracts: A Morphology-Aware Excess-Vocabulary Study, 2018-2026
cs.CL · cs.DL
Quick verdict
Adapts excess-vocabulary bounds to 398k Korean abstracts, finding a post-2024 LLM register shift; rigorous placebos and controls make the core result credible.
Summary
Aron Lee adapts the excess-vocabulary method of Kobak et al. to Korean: 398,296 KCI journal abstracts (2018–Aug 2026) tokenized into morphological lemma–part-of-speech units with suffix fusion, plus 47,165 Vietnamese VJOL abstracts as an external comparison. Placebo runs on pre-ChatGPT target years at every extrapolation horizon put the false-positive floor at 0.1–2.2 pp (single word) and at most 2.9 pp (set statistic). Korean abstracts are quiet in 2023, turn in late 2024 and flatten in mid-2026: 시사하다 appears in 21.4% of 2026 abstracts against 5.3% expected, while plain verbs like 알아보다 fall to a quarter of trend. The single-word lower bound on LLM-processed abstracts is 3.5%, 10.5% and 16.1% for 2024–2026; a split-half set bound reaches 33.0% (95% CI 30.8–35.4); a Holzwarth-style estimator gives 41.9% (2025) and 72.1% (2026) cross-fitted. Positive controls with five models across sixteen length-matched conditions reproduce most rising markers (구조적 excepted); topic matching and style/topic annotation leave 14.7 of the 33.0 points; translationese markers fall. Main limitations: the attribution assumption is untestable, the main five-base-year/four-year-horizon configuration has no pre-LLM placebo analogue, annotator agreement is weak (κ=0.24), and this is a non-peer-reviewed single-author working paper — though code and frequency tables are released.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Korean scholarly abstracts show an LLM-associated register shift absent in 2023, beginning late 2024 (one year after the English abstracts of the same articles), rising through 2025 and flattening in mid-2026.
- Under assumptions A1–A2, at least 16.1% (single-word) and 33.0% (split-half set, 95% CI 30.8–35.4) of January–August 2026 Korean KCI abstracts must have been LLM-processed, with a Holzwarth-style estimator at 72.1% cross-fitted.
- The rising markers are formal-register Sino-Korean items, not translations of the English excess vocabulary, and marker turnover tracks model generations across three providers including a Korean open-weight model.
- Topic drift, translation routes and selection artefacts do not account for the shift: topic-matched within-journal pairs leave the difference at 34.1 of 34.0 pp, translationese surface marks fall, and the split-half design removes selection optimism.
Strengths
Placebo discipline at matched horizons
Section 5.8 and Table 10 run the full statistic on pre-ChatGPT target years at every extrapolation horizon the data allow (h=1–4, 2–5 base years), yielding floors of 0.1–2.2 pp (ratio≥1.5 single word) and −0.8 to +2.9 pp (set statistic); the 2023 bound sits inside its floor, and all later bounds are read only above the matched floor.
Split-half selection removes selection optimism
Markers for the set bound are chosen on 929 journals (half A) and measured on the disjoint 924 (half B), giving 16.3 vs 16.1 pp for the single-word 2026 bound and a 33.0 pp set bound (CI 30.8–35.4, Table 11/App. F) — directly addressing the max-over-units upward null bias that plagues this literature.
Multi-model positive control with length matching
Within-article Korean–English pairing
Unusually honest uncertainty reporting
Weaknesses
Attribution assumption A2 is untestable
Main configuration lacks a placebo analogue
구조적 anomaly exposes an unmodeled source
Weak annotator agreement on the style/topic boundary
Practical measurement caveats
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This version-8 working paper (Zenodo, single author, not peer-reviewed) adapts Kobak et al.'s excess-vocabulary method — observed document frequency minus a least-squares extrapolation of 2018–2022 — from English to Korean, on 398,296 KCI abstracts (2018–Aug 2026) plus 47,165 Vietnamese VJOL abstracts as an external comparison. Its central claims: (1) Korean scholarly abstracts show no movement in 2023, turn in late 2024 and flatten in mid-2026 (Section 5.2, Figs. 1/6); (2) under stated assumptions A1–A3 (Section 4.7), a conditional lower bound on LLM-processed abstracts of 3.5/10.5/16.1% for 2024–2026 (single word, ratio ≥1.5) and 7.8/20.6/33.0% for the split-half set statistic (33.0, CI 30.8–35.4, Table 11); (3) a Holzwarth-style mixture estimator gives 41.9% (2025) and 72.1% (2026) under cross-fitting; (4) the shift is a formal-register signature built from Sino-Korean academic verbs — not translations of English markers — and survives topic and translation controls (Sections 5.13–5.14).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What percentage of Korean academic abstracts are written with AI?
This study gives a conditional lower bound: at least 16.1% of January–August 2026 Korean KCI abstracts must have been LLM-processed by a single-word statistic, and at least 33.0% by a set statistic (95% CI 30.8–35.4). A Holzwarth-style estimator suggests up to 72.1%, but all these figures assume the word-frequency excess above pre-2023 trends is caused by LLMs — an assumption the paper flags as untestable.
When did Korean academic writing start changing because of ChatGPT and LLMs?
According to this paper, Korean journal abstracts show nothing in 2023, begin turning in the second half of 2024 — one year later than the English abstracts of the same articles — and the rise flattens around mid-2026. The clearest marker is the verb 시사하다 ("suggest"), present in 21.4% of 2026 abstracts versus 5.3% expected from the pre-2023 trend.
How do you measure LLM vocabulary in Korean when words are not separated by spaces?
The authors replaced whitespace tokens with morphological lemma–part-of-speech units, fused Sino-Korean noun+하다 derivations into single verb lemmas, and validated the statistic with placebo runs on pre-ChatGPT years (false-positive floors of 0.1–2.2 percentage points) plus 1,437 model-written abstracts from five models as a positive control.
Is this paper peer-reviewed?
No. It is version 8 of a working paper self-published via Zenodo by a single independent researcher (INTFRAME Research, Seoul). However, it releases the analysis code, per-year frequency tables and a public Korean AI style dictionary, so the computations can be checked independently.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: several marked [...sampling gap...] stretches (parts of Sections 5.4–5.5, 5.10–5.11, 6.2–6.3, references and appendices) were not visible; judgments touching those parts rest on cross-references within the visible text.
- Figures appear in the extraction only as axis labels and captions; figure-level judgments rest on captions and in-text numbers rather than the rendered plots.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.