ArGuard Shared Task: Harmful Content Detection in Arabic Memes and LLM Prompts
Firoj Alam, Md. Rafiul Biswas, Mohamed Bayan Kmainasi, Ali Ezzat Shahroor, Hamdy Mubarak, George Mikros, Abul Hasnat, Wajdi Zaghouani
cs.CL · cs.AI
Quick verdict
ArGuard joins Arabic meme + LLM-prompt safety in one shared task; valuable resource, but test-label noise and absent significance tests weaken team rankings.
Summary
This overview paper presents ArGuard, a shared task that for the first time evaluates harmful-content detection in Arabic across two settings in one framework: multimodal hateful memes (Track A) and harmful LLM prompts (Track B), each with a binary and a fine-grained subtask. Fifty-eight teams registered, 35 entered the final evaluation, and 27 submitted system-description papers. Track A builds on the 5,000-meme AHA-Memes dataset with a ten-label multi-label taxonomy; Track B merges ArabicDialectSafety with AraSafe test prompts (27,047 prompts for B1). Participating systems span Arabic encoders (MARBERTv2, AraBERT), vision-language models (Qwen3-VL with LoRA), decoder LLMs (Jais), and lexical baselines. Best macro-F1 scores were 0.823 (A1), 0.419 (A2), 0.984 (B1), and 0.790 (B2), identifying fine-grained meme classification as the hardest setting owing to sparse labels and train-test shift. Strengths include the unified benchmark, a fine-grained taxonomy with reported inter-annotator agreement, and candid disclosure that inexact AraSafe label remapping introduced noise into test labels. Main limitations: no significance tests or confidence intervals despite a 0.002-point top-two gap in B2, macro-F1 computed over classes with as few as three test instances, and a winning Track B system trained on AraSafe prompts — the same source as the test set — an advantage the organizers do not analyze. The resource is valuable; fine-grained rankings deserve caution.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ArGuard is the first shared task to jointly evaluate harmful-content detection in Arabic memes and Arabic LLM prompts under one framework with binary and fine-grained subtasks.
- Fine-grained classification is substantially harder than binary detection, with the largest gap on memes (best macro-F1 0.823 on A1 vs 0.419 on A2), driven by sparse multi-label classes and train-test shift.
- OCR text carries most of the predictive signal for meme hate detection, although the strongest systems gain from multimodal VLM ensembles.
- Train-test shifts in class priors and sources dominate Track B, rewarding threshold calibration, prior correction, and source-aware data use.
- Both a fine-tuned Jais decoder and Arabic encoder ensembles reach near-ceiling binary prompt-safety performance (0.984 macro-F1).
Strengths
First unified evaluation of Arabic meme harm and LLM prompt safety
Section 1 motivates the gap: prior efforts (OSACT4/5, ArAIEval, MAHED; HarmBench, WildGuard, ArabicDialectSafety) evaluated memes and prompts separately and were largely English-centric for LLM safety. ArGuard's two tracks and four subtasks (A1/A2/B1/B2) close this structural gap.
Genuinely fine-grained meme taxonomy with reported annotation quality
A2 moves beyond binary labels to ten labels (seven hateful attack types, Humor/Sarcasm, Other), with inter-annotator agreement reported per layer: κ = 0.91 binary, 0.75 hateful types, 0.67 non-hateful subtypes (Section 3.1), on 5,000 manually annotated memes.
Candid disclosure of a known defect in the test labels
Thorough, multi-angle system analysis
Full release of data and evaluation infrastructure
Weaknesses
Test-set gold labels are partly noise by the authors' own admission
No statistical uncertainty anywhere
A2 metric rests on vanishingly small per-class supports
Winning Track B system trained on test-source data; advantage unexamined
Unjustified design choices and odd reporting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ArGuard (arXiv:2609.29349) is the organizers' overview of a 2026 shared task that, for the first time, places Arabic multimodal meme harm and Arabic LLM-prompt safety under one evaluation framework. Four subtasks: A1 binary hateful-meme detection, A2 ten-label multi-label meme taxonomy (seven attack types, Humor/Sarcasm, shared Other), B1 binary prompt safety, B2 seven-way harm-category classification. The headline claims are that the released datasets and protocols constitute a durable Arabic safety resource; that fine-grained classification is much harder than binary (0.823→0.419 macro-F1 on memes); that OCR text carries most of the meme signal with VLM ensembles on top; and that Track B train-test prior/source shifts reward calibration. As an overview paper, its claims are descriptive syntheses of participant results rather than novel modeling — the evaluation is only as valid as the underlying test labels.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the ArGuard shared task?
ArGuard (2026) is a shared task on detecting harmful content in Arabic. It has two tracks and four subtasks: Track A covers hateful Arabic memes (A1: binary hateful/not-hateful; A2: fine-grained multi-label classification into attack types like Mocking or Dehumanization plus non-hateful subtypes), and Track B covers harmful prompts submitted to LLMs (B1: binary safe/unsafe; B2: classification into seven harm categories such as Self-Harm, Fraud and Deception, or Hate Speech).
What were the best results in the ArGuard shared task?
The best systems reached macro-F1 of 0.823 on binary meme hate detection (A1, iMak AI Lab), 0.419 on fine-grained meme classification (A2, Ahmed Younis), 0.984 on binary prompt safety (B1, tie between AK_QMUL and Lattice AI), and 0.790 on harm-category classification (B2, AK_QMUL). Fine-grained meme classification was by far the hardest setting.
Are the ArGuard datasets publicly available?
Partially. The Track A dataset (AHA-Memes, 5,000 annotated Arabic memes) is publicly downloadable on Hugging Face under a non-commercial research license. The Track B prompt dataset (27,047 prompts) requires controlled access via a registration form, due to the harmful nature of the content.
Why was fine-grained Arabic meme classification (subtask A2) so hard?
Three reasons given in the paper: the fine-grained labels are sparse and heavily imbalanced (e.g., only 17 memes labeled Exclusion in the whole dataset, 3 in the test set), the blind test set deliberately shifted in distribution (fewer hateful memes, more Humor/Sarcasm/Contempt than training data), and models confuse hateful Mocking with benign Sarcasm and Humor. The best macro-F1 was only 0.419 versus 0.823 for the binary version of the task.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are available only through captions; the actual meme and prompt examples could not be verified from the extracted text.
- Tables 3, 4, 9, and 10 were linearized during extraction; per-team technique attributions were cross-checked against the prose in Sections 4-5 and Appendix A.
- The 27 participant system-description papers were not available; assessments of participant methods rely entirely on the organizers' summaries.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.