ParsHate: A Benchmark Dataset for Hate and Target Detection in Persian
Zahra Bokaei, Walid Magdy, Bonnie Webber
cs.CL · cs.DB
Quick verdict
第一个十年的波斯语仇恨言论基准，具备丰富的目标/隐式/片段标注；这是一个宝贵的资源，尽管核心统计数据存在不一致。
Summary
ParsHate 是一个手动标注的基准数据集，涵盖 2013–2022 年的 10,000 条波斯语推文，号称是首个长达十年的波斯语仇恨言论数据集，其中 31% 为仇恨内容。每年贡献 1,000 条推文，一半随机抽取，另一半则基于基于 Llama-3 的波斯语仇恨分类器的十个得分区间进行分层，旨在避免关键词驱动的采样偏差。三位母语标注员对二元仇恨、七个类别（含子类型）的多标签目标、显式与隐式仇恨及其策略标签、隐式目标以及字符级的理由片段进行了标注，其中 Fleiss' κ 系数从 0.69（仇恨）降至 0.44（隐式片段），并进行了裁决。实验表明，在多年份训练下，微调后的 LLaMA-3 达到了 79.0 的宏观 F1，而零样本 GPT-5 为 68.0；在 PHATE 上训练的迁移模型在 2020–2022 年的宏观 F1 得分为 81.3–90.9，但在 2013–2015 年仅为 57.3–65.2，作者将此归因于时间分布漂移，并辅以 TTR 和命名实体掩码分析支持。两个微调后的 LLM 在隐式仇恨上表现崩溃（LLaMA-3 的 F1 为 4.4，而显性仇恨为 84.5），且在政治主导（占仇恨推文的 76.45%）下的多标签目标预测中（宏观 F1 约为 24–27）表现不佳。主要局限性：§3.4 中的逐年仇恨百分比无法与 31% 的数据相协调，摘要中的 25.5% 目标宏观 F1 与 Table 6 不符，隐式仇恨仅占仇恨实例的 7%，且未提供显著性检验或常规的多语言基线。
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ParsHate is the first decade-spanning (2013–2022) Persian hate-speech benchmark, with 10,000 manually annotated tweets, 31% hateful, and the richest Persian annotation schema (targets, implicit strategies, spans).
- Hybrid random plus score-stratified per-year sampling reduces keyword-driven bias while, per the authors, preserving natural label distributions.
- Models trained on recent data degrade sharply on earlier years: the PHATE-trained transfer model scores 81.3/90.9/88.4 macro-F1 on 2020–2022 but 57.9/57.3/65.2 on 2013–2015, evidencing temporal distribution shift.
- Current LLMs collapse on implicit hate (fine-tuned LLaMA-3: F1 4.4 implicit vs 84.5 explicit) and on multi-label target prediction (macro-F1 about 24–27) under politics dominance (76.45% of hate tweets).
- Span-level rationales function both as an interpretability resource and as a diagnostic (LLaMA-3-FT token-level span F1: 58 explicit vs 14 implicit).
Strengths
Richest annotation stack for Persian to date
Combines binary hate, seven-category multi-label targets with fine-grained subtypes, explicit/implicit hate with three strategy labels, implicit-target marking, and character-level majority-vote spans (§3.2, Table 1) — no prior Persian resource (Pars-Off, PHATE, Pars-HAO, PHICAD) covers all four dimensions.
First longitudinal protocol for Persian hate speech
Decade coverage (2013–2022) with per-year sampling enables the cross-temporal experiments in §4.1: the PHATE-trained transfer model scores 81.3/90.9/88.4 macro-F1 on 2020–2022 but 57.9/57.3/65.2 on 2013–2015, with TTR and NE-masking analyses (§4.2) offering a plausible lexical mechanism.
Bias-aware sampling design
Transparent quality control
Diagnostic error analysis
Weaknesses
Internal inconsistencies in headline statistics
Thin implicit-hate evidence base
No uncertainty quantification and thin baselines
Self-referential sampling loop and diluted natural-distribution claim
Presentation defects
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ParsHate (arXiv 2609.16393) introduces a manually annotated corpus of 10,000 Persian tweets from 2013–2022 (31% hateful), positioned as the first decade-spanning Persian hate-speech benchmark. Beyond binary hate labels it adds: multi-label targets across seven categories with fine-grained subtypes; explicit/implicit hate with three strategy labels; explicit/implicit target marking; and character-level rationale spans. Evaluation covers zero-shot (GPT-5, LLaMA-3, Gemma-3), transfer from a PHATE-trained SOTA model, year-specific and all-years fine-tuning, plus TTR/NE-masking, multi-label target, and span-localisation analyses. Most claims are dataset-scoped and defensible, but several headline statistics conflict with the paper's own tables (see Statistical Evidence).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the ParsHate dataset?
ParsHate is a publicly released benchmark of 10,000 Persian tweets from 2013–2022, manually annotated for hate speech (31% hateful), the targets of hate across seven multi-label categories (politics dominates at 76.45% of hate tweets), explicit vs implicit hate with strategy labels, implicit targets, and character-level text spans justifying each label. It is described as the first decade-long Persian hate-speech dataset.
How well do current models perform on ParsHate?
Fine-tuned LLaMA-3 reaches 79.0 macro-F1 for binary hate detection when trained on all years, versus 68.0 for zero-shot GPT-5. But performance collapses on implicit hate (LLaMA-3 F1 4.4 vs 84.5 on explicit hate) and on multi-label target prediction, where the best models reach only about 24–27 macro-F1, failing mostly on rare categories such as gender, occupation, and ethnicity.
Why does the paper claim temporal distribution shift matters?
A model trained on recent data (PHATE, 2020–2023) scores 81.3–90.9 macro-F1 on tweets from 2020–2022 but only 57.3–65.2 on tweets from 2013–2015. The authors link this to greater lexical diversity in earlier years, measured via type–token ratio and named-entity masking analyses, and show that training on all years (79.0 macro-F1) yields more stable performance across the decade.
Is the ParsHate dataset publicly available?
Yes. The paper states the dataset is made publicly available via GitHub (github.com/zbokaei/ParsHate), with annotation guidelines (Persian original plus English translation) and the annotation interface to be released; tweets are de-identified and released for research purposes only because the content is offensive and sensitive.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 were extracted as garbled digit streams, so year-level distribution claims could not be verified visually and some derived tallies cannot be cross-checked.
- Several citation placeholders ('?') survive in the extracted text; a few referenced works (e.g., the TOXIGEN and toxic-spans citations) could not be identified with certainty.
- This review was conducted offline against the provided extraction only; the actual contents of the GitHub repository (data files, splits) could not be inspected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.