Recognized but Not Produced: A Generation Benchmark for Culturally Specific Kinship Terms
Sahil Pardasani, Madhusudan Singh
cs.CL · cs.AI
Quick verdict
Kinship-term generation benchmark (Hindi/Tamil/Korean, 5 LLMs) vs matched MC baseline: format gap is real; honest analysis, but scoring lacks human validation.
Summary
The paper introduces a generation benchmark for culturally specific kinship terms in Hindi, Tamil, and Korean, testing whether five open-weight LLMs (GPT-OSS-120B, GLM-5.1, Kimi-K2.6, Sarvam-M, Llama-3.3-70B) can produce correct kinship terms inside birthday greetings and wedding invitations when the relationship is explicitly specified (L3 prompts). A matched four-option selection baseline covers the same 77 relation-language-perspective cells for two of the five models. Scoring uses a frozen rule-based native-script substring matcher over 4,500 generations (three runs per prompt). L3 generation accuracy spans 72.29% (GLM-5.1) down to 24.24% (Llama-3.3-70B); GPT-OSS-120B selects the correct option in 90.67% of 75 valid cells but generates an accepted term in only 36.00% of matched attempts, which the authors interpret as an evaluation-format gap rather than proof of intact lexical knowledge. The paternal-lineage advantage is large in Hindi (+43.3 points pooled) but weak or reversed in Korean, while Tamil shared-term pairs expose roughly 10-point measurement noise. Llama-3.3-70B produced no Hangul in any of its 162 Korean L3 responses. Main limitations: substring scoring lacks a human-audited false-positive/negative rate; the selection baseline covers only two of five models under different decoding (temperature 0, one run); distractors are broad inventory terms; and cross-model comparisons are uncontrolled for general language proficiency.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Open-weight LLMs select the correct culturally specific kinship term far more reliably than they generate it in a complete message: on identical cells, GPT-OSS-120B scores 90.67% selection versus 36.00% generation (75 valid cells) and Llama-3.3-70B 77.92% versus 24.24% (77 cells).
- On fully specified L3 prompts, generation accuracy varies sharply across the five models, from 72.29% for GLM-5.1 down to 24.24% for Llama-3.3-70B over 77 cells (Table 1).
- The paternal-minus-maternal accuracy advantage is language- and model-specific: large in Hindi (+43.3 points pooled), weak or reversed in Korean (+4.0), with Tamil shared-term pairs exposing about 10 points of measurement noise.
- Llama-3.3-70B exhibits a total Korean script-production failure, emitting no Hangul in any of its 162 Korean L3 responses.
Strengths
Matched generation-vs-selection design on identical cells
Table 3 compares option-supported selection and mean L3 generation on the same relation-language-perspective cells (GPT-OSS-120B: 90.67% vs 36.00% over 75 cells, gap 54.67 with CI [45.11, 64.00]; Llama-3.3-70B: 77.92% vs 24.24% over 77 cells), directly isolating the format difference the title claims.
Clustered, sensitivity-checked statistics
Table 1 CIs use a cell-level bootstrap keeping all six attempts per cell together, avoiding pseudo-replication; position-macro scores (91.07%/78.29%) guard against the unbalanced A-D distribution (13/24/21/19), and a blank-as-wrong sensitivity (88.31% for GPT-OSS-120B) is reported.
Self-calibrating control for measurement noise
Unusually honest claim calibration and reconciliation
Reproducibility scaffolding
Weaknesses
Scorer validity is unquantified
Selection baseline is confounded and incomplete
Cross-model ranking is uncontrolled
Gold standard validated only at table level
Lineage analysis is descriptive with strong heterogeneity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This arXiv preprint (2609.26942, cs.CL) asks a focused question: when an open-weight LLM is told the exact relationship ('father's elder brother'), can it generate the correct culturally specific kinship term in Hindi, Tamil, or Korean inside a natural message — and how does that compare with simply selecting the term from four displayed options? The benchmark spans 25 relations lexically distinct in at least one of the three languages, two communicative templates (birthday greeting, wedding invitation), and two specificity levels (L1 underspecified hypernyms like 'uncle'; L3 fully specified relations). Five models x 300 prompts x 3 runs yield 4,500 generations; a four-option selection baseline covers the same 77 relation-language-perspective cells for two of the five models. Headline claims: (1) GPT-OSS-120B selects correctly in 90.67% of 75 valid cells but generates an accepted term in only 36.00% of matched attempts (Llama-3.3-70B: 77.92% vs 24.24%); (2) L3 generation spans 72.29% (GLM-5.1) to 24.24% (Llama-3.3-70B) over 462 attempts each; (3) the paternal-lineage advantage is language- and model-specific; (4) Llama produced no Hangul in any of its 162 Korean L3 responses. The authors pre-empt overclaiming: the selection condition is labeled 'option-supported,' and the gap is explicitly framed as an evaluation-format difference, not proof that lexical knowledge is intact.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this kinship generation benchmark paper?
Large language models are much better at picking the right culturally specific kinship term from displayed options than at producing it inside a natural message. On the same relation-language cells, GPT-OSS-120B selected the correct option in 90.67% of 75 valid cells but generated an accepted term in only 36.00% of matched generation attempts; Llama-3.3-70B showed 77.92% versus 24.24%. The authors call this an evaluation-format gap, not proof that the models' lexical knowledge is intact.
Which languages and models does the benchmark cover?
Three non-Western languages — Hindi, Tamil, and Korean — and five open-weight LLMs: GPT-OSS-120B, GLM-5.1, Kimi-K2.6, Sarvam-M, and Llama-3.3-70B. Models wrote birthday greetings and wedding invitations across 25 kinship relations, producing 4,500 generations scored by a rule-based native-script matcher. On fully specified (L3) prompts, accuracy ranged from 72.29% (GLM-5.1) to 24.24% (Llama-3.3-70B).
Why does the paper argue multiple-choice tests are misleading for kinship terms?
Because the multiple-choice condition displays the candidate terms in the target script and only requires outputting a Latin letter (A-D), it removes spelling, native-script production, and message composition. The paper pairs it with open-ended generation on identical test cells and shows a 40-55 point difference, so multiple-choice accuracy can overstate culturally specific lexical competence.
Is the benchmark code and data available?
Yes. The paper states that data, code, and a documented offline reproduction workflow are available at github.com/sahilpardasani/kinship-generation-benchmark, including per-call metadata and row-level audit files. Human output-audit records are not included because a stratified human annotation of model outputs has not yet been conducted.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- No figures exist in the extracted text; all quantitative claims come from embedded tables, which appear complete (main text through Appendix B present).
- The matcher's native-script variant inventory and the row-level audit files live in the external GitHub release and could not be inspected, so scorer behavior is taken on the paper's description.
- Appendix A.2 examples are transliterated in the PDF itself (LaTeX rendering limitation), so the exact native-script prompt surface used at evaluation time could not be verified from the text.
- Publication venue/status could not be verified; the Acknowledgments mention camera-ready formatting, suggesting a workshop or conference submission.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.