IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks
Saikat Mondal, Mamta, Deeksha Varshney, Oana Cocarascu, Asif Ekbal
cs.CL · cs.AI
Quick verdict
Claims persuasion jailbreaks transfer to Indic languages (ASR up to 78%); solid benchmark, but translated prompts and judge bias temper conclusions.
Summary
The paper introduces IndicSafeEval, a black-box benchmark for persuasion-based jailbreak attacks in four Indian languages (Hindi, Bengali, Marathi, Punjabi). Starting from 300 harmful seed queries across ten risk categories (Shen et al., 2024b), the authors use GPT-5.1 with few-shot exemplars to rewrite each query under six persuasion strategies (logical appeal, authority endorsement, misrepresentation, anchoring, priming, confirmation bias), yielding 1,800 English prompts that are machine-translated into the four Indic languages (7,200 Indic prompts; 9,000 total). Five open-source models are judged by Gemini-2.5-Flash on a 5-point scale; ASR counts scores 4-5. Average ASR reaches 77.68% (Qwen3-8B), 73.89% (Gemma3-4B), 70.30% (Sarvam-M), 69.12% (Llama-3.1-8B), versus 38.85% for Hindi-centric Llama-3-Nanda-10B. Authority endorsement, misrepresentation, and logical appeal are the strongest strategies; confirmation bias the weakest. Category-level results (e.g., Qwen3-8B: government decision-making 86.22%, hate speech 60.00%) show heterogeneous vulnerability. A 100-sample-per-language human study agrees with the LLM judge within 4-9 points (binary Cohen's kappa 0.8056-0.8890; Punjabi 1.00). Larger models remain vulnerable: Llama-3.3-70B (46.8%) and GPT-4o-mini (46.1%) on Physical Harm. Main limitations: prompts are machine-translated despite the paper's own critique of translated benchmarks; no significance tests despite n=30 cells; the judge over-rates humans by ~6 points on average; and the full adversarial set is access-restricted.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Safety alignment of open-source LLMs degrades sharply under persuasion-based jailbreaks in Indic languages, with average ASR of 69.12-77.68% for four of five models versus 38.85% for Hindi-centric Llama-3-Nanda-10B (Table 8).
- Attack success depends systematically on persuasion strategy — authority endorsement, misrepresentation, and logical appeal are strongest while confirmation bias is weakest — and on risk category, with Qwen3-8B averaging 86.22% ASR on government decision-making versus 60.00% on hate speech (Table 19).
- Lower attack success in low-resource Indic languages (Punjabi, Marathi) reflects weaker language proficiency rather than stronger safety alignment (§5.1, Appendix C.6).
- Persuasion-based jailbreaks generalize beyond small open models: Llama-3.3-70B and GPT-4o-mini reach 46.8% and 46.1% average ASR on the Physical Harm category (Table 7).
Strengths
Genuine coverage gap filled at scale
9,000 prompts (1,800 English + 7,200 across Hindi/Bengali/Marathi/Punjabi) spanning 10 risk categories x 6 persuasion strategies x 5 open-source models — the first large-scale persuasion-jailbreak benchmark for Indic languages, per §3 and the related-work gap analysis.
Beyond-binary safety metrics
SRR/HLR/PCR (§5.6, Table 8) expose distinct failure modes that a single ASR would hide: Llama-3.1-8B partial compliance hits 24.61% in Punjabi, and Llama-3-Nanda-10B hint leakage reaches 21.24% in Marathi despite low ASR.
Judge validated against native-speaker humans
Honest interpretation of low-resource results
Generalization beyond 8B-scale models
Weaknesses
No statistical inference anywhere
Construction contradicts the paper's own positioning
Single proprietary judge with bias and anomaly
Reproducibility friction
Internal inconsistencies and copy-editing lapses
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
IndicSafeEval (arXiv:2609.03781v2) is a benchmark and empirical study of persuasion-based jailbreak attacks against open-source LLMs in four Indian languages: Hindi, Bengali, Marathi, and Punjabi. From 300 harmful seed queries across ten risk categories (Shen et al., 2024b), GPT-5.1 rewrites each query under six persuasion strategies from the Zeng et al. (2024) taxonomy — logical appeal, authority endorsement, misrepresentation, anchoring, priming, confirmation bias — yielding 1,800 English prompts that Google Translate converts into 7,200 Indic prompts (9,000 total, §3.2). The central claims: (1) safety alignment degrades sharply under persuasion in Indic languages (average ASR 69-78% for four of five models, Table 8); (2) vulnerability varies by language, strategy, and risk category; (3) apparent robustness in low-resource languages reflects weaker proficiency, not stronger safety; (4) the attack vector generalizes to Llama-3.3-70B (46.8% ASR) and GPT-4o-mini (46.1%) on Physical Harm (Table 7).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is IndicSafeEval and what does it measure?
IndicSafeEval is a benchmark of persuasion-based jailbreak attacks against large language models in four Indian languages (Hindi, Bengali, Marathi, Punjabi). It combines 300 harmful seed questions across 10 risk categories with 6 human-like persuasion strategies, producing 7,200 Indic-language adversarial prompts (9,000 including English), and measures how often models produce policy-violating answers (Attack Success Rate, judged on a 5-point scale).
Which LLMs were most vulnerable to persuasion jailbreaks in Indian languages?
Per the paper, Qwen3-8B was most vulnerable with 77.68% average attack success rate, followed by Gemma3-4B (73.89%), Sarvam-M (70.30%), and Llama-3.1-8B (69.12%); the Hindi-centric Llama-3-Nanda-10B-Chat was most resistant at 38.85%. The larger Llama-3.3-70B (46.8%) and proprietary GPT-4o-mini (46.1%) were also successfully attacked on the Physical Harm category.
Is the IndicSafeEval dataset publicly available?
Only partially. The implementation is on GitHub (MonSaikat/IndicSafeEval), but the ethical statement says the full adversarial prompt set is not released publicly — access is restricted to vetted researchers under an acceptable-use agreement, which limits independent auditing and exact reproduction.
Does a lower attack success rate in Punjabi or Marathi mean models are safer in those languages?
Not necessarily. The authors explicitly warn that lower ASR in lower-resource languages may reflect weaker language proficiency — shorter, less coherent responses that fail harm criteria — rather than stronger safety alignment, and they support this with a harmfulness ranking of responses to identical prompts across languages.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Provided text is a sampled excerpt with '[...sampling gap...]' markers: Appendix C.5 (delta-ASR), Tables 13-18 and 22-26, and details of A.1-A.4 were not visible; assessments of those analyses rely on in-text references only.
- The annotator compensation figure extracted as '| 37,200 per month' (B.3) lost its currency glyph in PDF extraction (likely rupee symbol) — an extraction artifact, not a paper error.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.