TurEngMix: A Text Corpus and Benchmark for Turkish-English Code-Mixed Language Identification and Named Entity Recognition
Ilayda Dogan, Phuong-Anh Nguyen-Le, Julia Mendelsohn
cs.CL
Quick verdict
First Turkish-English code-mixed LID/NER benchmark; valuable resource, but GPT-4o-as-filter circularity and tiny minority classes limit conclusions.
Summary
The paper introduces TurEngMix, a corpus of 5,549 public posts (486,974 tokens) from the Turkish forum Ekşi Sözlük, collected via topic tags likely to contain Turkish-English code-mixing and enriched through langdetect plus two GPT-4o filtering stages. From 250 sampled posts the authors build a benchmark of 15,012 tokens with token-level language ID labels (following Solorio et al. 2014) and NER labels (CALCS 2018, BIO), plus an INTEGRATED category for English stems bearing Turkish suffixes. Inter-annotator agreement on a 203-token sample was high (κ=0.96 LID, 0.94 NER). Evaluating GPT-4o, Qwen3-8B, and three fine-tuned encoders, monolingual tokens are labeled reliably (F1>0.84) but MIXED tokens are hard (best LID F1 0.45 for GPT-4o three-shot; encoders at most 0.22). For NER, GPT-4o three-shot reaches 0.6140 macro F1 while encoders stay below 0.18; error rates on morphologically integrated tokens are 5.2× (GPT-4o) and 6.3× (Qwen) those on other tokens. Limitations: GPT-4o served as both corpus filter and baseline, favoring it; the benchmark is small and class-imbalanced (189 MIXED tokens; Time n=3, absent from the encoder test set); LLMs were scored on all 250 posts while encoders were scored only on the 25-post test split.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- TurEngMix is the first large Turkish-English code-mixed corpus (5,549 posts, 486,974 tokens) and the first jointly LID+NER annotated benchmark (15,012 tokens) for the language pair.
- All evaluated models label monolingual Turkish and English tokens reliably (per-class F1 > 0.84) but fail disproportionately on MIXED tokens, where the best LID F1 is 0.45 (GPT-4o three-shot) and fine-tuned encoders reach at most 0.22.
- Morphological integration is the dominant difficulty: NER error rates on tokens with English stems bearing Turkish suffixes are 5.2× (GPT-4o) and 6.3× (Qwen3-8B) those on other tokens, both significant at p<0.001.
- Decoder LLMs outperform fine-tuned encoders on code-mixed NER (macro-F1 0.6140 vs ≤0.176), while LID macro-F1 is comparable across model families (0.42-0.52).
Strengths
Fills a genuine, well-documented resource gap
§2.2 establishes that the only prior public Turkish-English written corpus has 391 samples (Yirmibeşoğlu & Eryiğit 2018) and that LinCE/GLUECoS omit the pair; TurEngMix contributes 5,549 posts / 486,974 tokens plus the first jointly LID+NER annotated benchmark (15,012 tokens).
Typologically distinctive focus on morphological integration
Beyond standard code-mixing categories, the INTEGRATED metadata (386 tokens; 2.57% of tokens but present in 66.0% of posts, §4.2) isolates English stems with Turkish suffix chains, and §6.3.3 gives a targeted statistical error analysis of exactly these tokens.
Strong agreement statistics for the core labels
Headline claim is statistically grounded
Transparent, reproducible release
Weaknesses
GPT-4o circularity between corpus construction and evaluation
Decoder and encoder models are evaluated on different data
Rare classes are too small to support per-class conclusions
Baseline design does not test the paper's own remedy
Annotation rests on a single primary annotator with a thin validation sample
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
TurEngMix (arXiv:2609.06963, cs.CL) is a resource-and-benchmark paper for Turkish-English code-mixed NLP. It claims three contributions (§1): (1) a 5,549-post, 486,974-token corpus of naturally occurring Ekşi Sözlük posts; (2) the first Turkish-English benchmark with joint token-level LID and NER annotation (15,012 tokens from 250 posts); (3) a systematic evaluation of decoder LLMs (GPT-4o, Qwen3-8B) and fine-tuned encoders (XLM-RoBERTa, BERTurk, TurkishBERTweet). The motivating gap is real: the only prior public Turkish-English written corpus contains 391 samples (Yirmibeşoğlu & Eryiğit 2018), and LinCE/GLUECoS cover other pairs (§2.2). The scientific headline is that morphologically integrated tokens — English stems with Turkish suffix chains such as *influencerlarımız* — drive disproportionate model errors.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is TurEngMix?
TurEngMix is a publicly released corpus of 5,549 Turkish social media posts from the forum Ekşi Sözlük (486,974 tokens), rich in Turkish-English code-mixing, plus a 15,012-token benchmark annotated for token-level language identification and named entity recognition. It is licensed CC BY-NC-SA 4.0 for non-commercial use.
Why is Turkish-English code-mixing hard for NLP models?
Turkish is agglutinative, so English word stems can take chains of Turkish suffixes inside a single token (e.g., influencerlarımız, 'our influencers'). The paper shows models make errors on such morphologically integrated tokens at 5.2-6.3 times the rate of other tokens, because the token is partly English and partly Turkish.
Which models performed best on the TurEngMix benchmark?
GPT-4o with three-shot prompting led both tasks: 0.5238 macro-F1 for language identification and 0.6140 macro-F1 for NER. Fine-tuned encoders (BERTurk best at 0.4734 for LID) were competitive on LID but fell below 0.18 macro-F1 on NER. All models scored above 0.84 F1 on plain Turkish/English tokens but struggled on mixed-language tokens.
Where can I download the TurEngMix dataset and code?
The corpus, annotations, guidelines, and scripts are public: Hugging Face dataset ilydoa/TurEngMix and GitHub repository kemnguyenle/TurEngMix, released under the CC BY-NC-SA 4.0 license (non-commercial use with share-alike attribution).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are images in the PDF; only their captions were extracted, so visual distributions (post-length histogram, topic-tag frequencies, the annotation screenshot) could not be verified beyond the numbers stated in the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.