VakyArth: Evaluating Pragmatic Competence in LLMs across Indic Languages
Usneek Singh, Poorvaja Veera Balaji Kumar, Parth Nanda, Anand Madhusoodanan, Geyang Guo, Wei Xu, Junyi Jessy Li
cs.CL · cs.AI
Quick verdict
First Indic pragmatics benchmark (578 items, 4 languages): native-authored, systematic literal-reading failures found; small per-cell n, no human ceiling.
Summary
VakyArth is presented as the first pragmatic-competence benchmark for Indic languages: 578 native-authored items across Hindi, Punjabi, Tamil, and Malayalam, spanning five phenomena (deixis, speech acts, implicature, social pragmatics, coherence) evaluated through MCQ, NLI, and translation. Four native speakers with graduate NLP training authored and reviewed all items; inputs are Latin transliterations; five instruction-tuned models (8B-111B) are tested with 3-shot prompting. Reported findings: MCQ accuracy exceeds NLI accuracy in all 20 model-language aggregates; on MCQ errors models pick the literal-meaning distractor 59.1% of the time (33.3% chance, p<0.001); when the gold NLI label is contradiction, models predict entailment 53.8% of the time; Gemma-4-31B is strongest overall (MCQ 0.86, NLI 0.43, COMET 0.81) despite lacking Indic specialization; Indo-Aryan languages show a translation advantage over Dravidian ones. Human ratings of 104 sampled translations show COMET correlates best with humans (rho=0.86) yet rewards fluent but pragmatically inverted outputs. A script ablation finds only a modest, uneven native-script advantage (76.5% vs 70.6%). Stated and observed limitations: no human baseline on MCQ/NLI items, small per-cell samples without confidence intervals, an unspecified 'four models' error-analysis subset, single-turn items only, and coverage of four of hundreds of Indic languages.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- VakyArth is the first pragmatic-competence benchmark for Indic languages, with 578 native-authored items across Hindi, Punjabi, Tamil, and Malayalam, five phenomena, and three task formats.
- Multilingual LLMs fail systematically rather than randomly on Indic pragmatics: 59.1% of MCQ errors select the literal-meaning distractor (vs 33.3% chance, p<0.001), and models predict entailment 53.8% of the time when the gold NLI label is contradiction.
- MCQ accuracy exceeds NLI accuracy in all 20 model-language aggregates, indicating pragmatic interpretation benefits from answer-choice scaffolding.
- Translation performance does not reliably track pragmatic understanding, and automatic metrics (even the best-correlating COMET, rho=0.86) reward fluent but pragmatically unfaithful translations, especially for implicature and deixis.
- Indo-Aryan languages show a translation advantage over Dravidian languages, attributed to pretraining coverage, and Gemma-4-31B is the strongest model despite lacking Indic specialization.
Strengths
Genuine coverage gap addressed with native authorship
No prior pragmatic evaluation exists for any Indic language (§2); VakyArth contributes 578 items (Table 1: Hindi 185, Punjabi 125, Tamil 134, Malayalam 134) written from scratch by four native speakers with graduate NLP training, full-team review, and deliberate inclusion of code-mixing reflecting real digital usage (§3.3).
Multi-format diagnostic design yields non-obvious findings
The same phenomena under MCQ/NLI/translation (§3.2) expose a receptive-productive dissociation: MCQ beats NLI in 20/20 model-language aggregates (§5.2), and translation scores do not track MCQ/NLI, showing recognition and cross-lingual expression are distinct capabilities.
Directional error analysis rather than aggregate scores
Human evaluation designed for pragmatic fidelity, plus metric audit
Honest, nuanced script ablation
Weaknesses
No human baseline or ceiling on MCQ/NLI items
Underpowered per-cell statistics with no uncertainty
Internal accounting inconsistencies
NLI design choices unaccompanied by baselines
Dated model set limits currency of conclusions
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
VakyArth (arXiv 2609.01788, Georgia Tech / UT Austin) targets a demonstrable gap: pragmatic-competence evaluation for Indic languages. The authors claim (1) no prior pragmatic evaluation exists for any Indic language (§2), (2) multilingual LLMs fail systematically on meanings rooted in Indic conventions, and (3) recognizing pragmatic meaning (MCQ/NLI) is dissociable from expressing it (translation). The benchmark spans 578 items across Hindi and Punjabi (Indo-Aryan) and Tamil and Malayalam (Dravidian), five phenomena following the Ma et al. (2025) taxonomy, and three task formats. Claims are mostly scoped to the five evaluated models, which is appropriate discipline for a benchmark paper.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is VakyArth?
VakyArth is a benchmark for testing whether large language models understand pragmatics - implied, context-dependent meaning - in four Indian languages (Hindi, Punjabi, Tamil, Malayalam). It contains 578 questions written by native speakers covering five phenomena (deixis, speech acts, implicature, social pragmatics, coherence) in three formats: multiple choice, natural language inference, and translation. The authors state code and data are released at github.com/Usneek1/VakyArth.
Which LLMs were tested and which performed best on VakyArth?
Five instruction-tuned models from 8B to 111B parameters: Llama-3-8B, Gemma-4-31B, Qwen3-32B, Sarvam-105B (which has dedicated Indic pretraining), and Command-A-111B. Surprisingly, Gemma-4-31B was strongest overall despite lacking Indic specialization, with 0.86 MCQ accuracy, 0.43 NLI accuracy, and a 0.81 COMET translation score.
Why do LLMs fail on Indian-language pragmatics according to this paper?
The paper finds models fail in a systematic direction: they default to the literal meaning of utterances. On multiple-choice errors, models picked the literal-reading distractor 59.1% of the time versus a 33.3% random rate; on inference items whose correct label was 'contradiction', models predicted 'entailment' 53.8% of the time. Typical failures include missing that Hindi 'kal' can mean yesterday or tomorrow, or that a Punjabi host saying 'nothing special was made' is offering food politely, not criticizing it.
Can automatic metrics like COMET detect pragmatically wrong translations?
Often no. Although COMET (wmt22-comet-da) correlated best with human raters overall (Spearman rho = 0.86 on 416 outputs), the paper documents cases where COMET scored 0.65-0.86 on translations that native speakers rated 1-2 out of 5, because the outputs were fluent but semantically or pragmatically inverted (e.g., translating 'don't come here' as 'come here'). Human judgment remains necessary for implicature and deixis.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Table 2 was extracted as flattened 15-number rows; the per-phenomenon MCQ/NLI/COMET column mapping was inferred from row structure and could not be visually verified against the rendered table.
- The GitHub repository's current contents and completeness were not verified (no network access); release status is as stated in the paper.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.