SWORD: Wikidata-based Distortions Reveal Hidden Cross-Lingual Inconsistencies in LLM Factual Error Rejection
Sanghyeok Park, Minji Kang, Hosung Kwak, Jinhyuk Yun
cs.CL
Quick verdict
SWORD reveals hidden cross-lingual gaps in LLM false-statement rejection via Wikidata distortions; credible, though translations unverified and stats thin.
Summary
The paper introduces SWORD, a benchmark testing whether LLMs consistently reject factually false statements across eight languages (EN, DE, FR, ES, IT, PT, KO, JA). From ~2.1M Wikidata triples filtered for eight-language label coverage, the authors sample 500 and generate four distortion types per triple — random subject/object substitution and property-based (semantically closer) substitution — verified false by Wikidata non-existence, then verbalize all statements via Gemini-2.5-Flash and evaluate five models on true/false classification. Two headline findings: (1) counterintuitively, 59/80 (73.75%) of model-language-distortion-pair combinations show equal or higher accuracy rejecting plausible property-based distortions than nonsensical random ones, suggesting reliance on distributional familiarity; (2) models with comparable baseline accuracy (LLaMA-3.1: Korean ORG 0.884 vs German 0.854) diverge sharply under distortion — Korean drops to 0.294/0.298 on PSS/POS versus German 0.556/0.580 (26–28 pp gap, 47–49% relative), a pattern absent in Gemini and Qwen3; Mistral shows a similar 21–23 pp gap for Japanese. Jaccard analyses reveal language-family clustering under distortion for LLaMA/Mistral. The authors candidly attribute Qwen3's apparent stability to a False-response bias and flag Gemini's self-preference. Main limitations: no significance testing or confidence intervals; statement quality unverified in four of eight languages; Wikidata absence as the falsity oracle; English-only prompting; single-run Gemini. Code and benchmark are released via an anonymous repository.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Models show equal or higher accuracy rejecting semantically plausible property-based distortions than nonsensical random substitutions in 59/80 (73.75%) of model-language-distortion-pair combinations, suggesting reliance on distributional familiarity rather than genuine factual verification.
- Models with comparable baseline accuracy across languages (e.g., LLaMA-3.1 Korean ORG 0.884 vs German 0.854) show gaps of up to 28 percentage points (47–49% relative) under property-based distortion, a pattern invisible to true-statement benchmarks like MMMLU.
- The disparity is model-specific: LLaMA-3.1 and Mistral-Nemo degrade sharply on East Asian languages under distortion while Gemini-2.5-Flash and Qwen3 do not.
- Qwen3's apparent cross-lingual stability under distortion is largely a response bias toward answering False, evidenced by very low ORG accuracy in KO/JA (Qwen3-14B: 0.216/0.245).
Strengths
A priori difficulty control via graph structure
node2vec embeddings on the 2,119,631-triple graph (walk length 10, 10 walkers, skip-gram; Section 3.1, Figure 1b,c) verify that property-based substitutions are semantically closer to originals than random ones before any model is run — directly answering the post-hoc-difficulty critique the authors level at FACTS and FactBench.
Unusually candid confound analysis
The paper reinterprets Qwen3's 'stable' cross-lingual results as a False-response bias (ORG KO/JA accuracy as low as 0.216/0.245 for Qwen3-14B, Table 2), flags Gemini's generator-evaluator self-preference (Section 4, citing Panickssery et al. 2024), and reports Unsure-as-incorrect (Table 4) and Unsure-excluded (Table 5) scoring side by side, showing conclusions are unchanged.
A sharp, well-anchored headline effect
Robustness checks beyond the main table
Weaknesses
No inferential statistics anywhere
Generation quality unverified in half the languages
Wikidata absence as the falsity oracle
Ambiguous distortion definitions in Table 1
Single-run Gemini and English-only prompting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SWORD is a benchmark and diagnostic study asking a narrow but important question: can multilingual LLMs consistently *reject* false statements, and does that ability degrade unevenly across languages? From 2,119,631 Wikidata triples whose subject/property/object labels exist in all eight languages (EN, DE, FR, ES, IT, PT, KO, JA), the authors sample 500, generate four distortion types per triple (random subject/object substitution; property-based, semantically closer substitution), and verbalize everything with Gemini-2.5-Flash into 20,000 true/false items. Five models (Mistral-Nemo, LLaMA-3.1-8B, Gemini-2.5-Flash, Qwen3-8B/14B) are evaluated. The central claims: (1) models reject plausible property-based distortions at least as accurately as nonsensical random ones in 59/80 combinations (73.75%); (2) models with near-identical true-statement accuracy diverge dramatically under distortion — LLaMA-3.1 Korean falls to 0.294/0.298 (PSS/POS) versus German 0.556/0.580, a 26–28-point gap (47–49% relative) — while MMMLU shows only ~5 points between those languages; (3) the effect is model-specific (LLaMA, Mistral yes; Gemini, Qwen3 no); (4) Qwen3's apparent stability is a False-answer bias, not stable factuality. Claims (1)–(3) are well supported descriptively; the 'distributional familiarity' mechanism is interpretive and untested.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SWORD benchmark?
SWORD (Systematic Wikidata-based Object-Relation Distortion) is a benchmark measuring whether LLMs consistently reject factually false statements across eight languages (English, German, French, Spanish, Italian, Portuguese, Korean, Japanese). It creates false statements by perturbing Wikidata subject-property-object triples via random entity substitution or semantically closer property-based substitution, then asks models to judge each resulting sentence true or false.
Why do models perform better at rejecting plausible false statements than absurd ones in this study?
In 59 of 80 model-language combinations (73.75%), the five tested models classified semantically plausible distortions as false more accurately than nonsensical random substitutions. The authors interpret this as reliance on distributional familiarity — what entities usually co-occur — rather than actual fact verification, since absurd statements fall outside the models' stable behavioral range.
Which models showed cross-lingual problems rejecting false statements?
LLaMA-3.1-8B and Mistral-Nemo showed sharp degradation on Korean and Japanese under distorted statements — up to 28 percentage points below German despite comparable true-statement accuracy — while Gemini-2.5-Flash and Qwen3 did not. The paper notes Qwen3's stability partly reflects a bias toward answering 'False' rather than stable factual verification.
Is the SWORD dataset and code publicly available?
The paper states the benchmark and code are publicly available via an anonymous repository (anonymous.4open.science/r/SWORD-EMNLP), including prompt templates reproduced in the appendix; a de-anonymized release would be expected upon publication.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-embedded statistics (difficulty-bin percentages in Figures 3/8/10 and the cosine-similarity distributions in Figure 1b,c) were extracted as unstructured numeric blocks rather than labeled values; figure-level magnitudes were therefore assessed via captions, tables, and recomputation from Tables 2–3 rather than the plots themselves.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.