ARAFA: An LLM-Generated Arabic Fact-Checking Dataset
Christophe Khalil, Shady Elbassuoni, Rida Assaf
cs.CL · cs.IR
Quick verdict
Largest Arabic fact-check dataset (182k LLM-built pairs); valuable resource but test-set contamination and validator circularity weaken benchmark claims.
Summary
The paper introduces ARAFA, a 181,976-pair Arabic fact-checking dataset built by an automated LLM pipeline: GPT-4o generates claims with verbatim evidence from ~10,000 Arabic Wikipedia pages, mutates 50,000 supported claims into counterfactuals using five mutation strategies, and Claude Sonnet 3.5 validates each claim-evidence pair as supported, refuted, or not-enough-information (NEI) via a weighted scoring rule (30/30/40 weights, threshold 88). A 400-pair human-annotated sample (200 supported, 200 refuted) shows the validator achieving 86%/88% accuracy, with non-expert inter-annotator Cohen's kappa of 0.89/0.94. Fine-tuning four open-source models, the best (AraModernBert-Base) reaches 77% macro F1, while generative models (Llama-3.1-8B, Qwen2.5-7B) dominate NEI detection (85-89% F1) but underperform on supported claims. Key limitations: the 400-pair test set was drawn from ARAFA yet the 'complete' dataset was used for training, raising contamination concerns; validator selection and the NEI threshold were tuned on the same 400 pairs later used as gold standard, inflating label-quality estimates; the test set's class composition (no stated NEI items) conflicts with reported NEI F1 scores; and evaluation is purely in-domain with no cross-dataset transfer or zero-shot LLM baselines. The framework is presented as transferable to other low-resource languages.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ARAFA is the first and largest large-scale Arabic fact-checking dataset, containing 181,976 claim-evidence pairs labeled supported, refuted, or NEI.
- LLM-based validation with Claude Sonnet 3.5 achieves 86%/88% accuracy on supported/refuted claims, comparable to human annotator reliability (kappa 0.89/0.94).
- ARAFA is sufficient to train competitive Arabic fact-checking models, with the best fine-tuned model (AraModernBert-Base) reaching 77% macro F1.
- The three-task generation framework is source- and language-agnostic and can be adapted to build similar datasets for other low-resource languages.
Strengths
Scale and openness versus prior Arabic resources
At 181,976 pairs (111,303 supported / 42,109 refuted / 28,564 NEI, Table 1), ARAFA is ~30x larger than all prior Arabic fact-checking sets combined (AraFacts ~6,222; AraStance ~4,063; Table 2), and is released with prompts via GitHub and Zenodo.
Linguistically grounded counterfactual mutation
Section 3.2 defines five Arabic-aware mutation strategies (temporal, quantitative, qualitative, relationship, contextual); the Lebanon/Mediterranean vs Red Sea example concretely demonstrates why evidence-grounded mutation beats blind negation.
Empirically tuned generation pipeline
Quantified sample representativeness and candid agreement reporting
Weaknesses
Probable train/test contamination
Circular validation of label quality
Undeclared test-set class composition
No external validity for the 'benchmark' claim
No uncertainty quantification anywhere
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
ARAFA Peer Review Report
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the ARAFA dataset?
ARAFA is a large-scale Arabic fact-checking dataset with 181,976 claim-evidence pairs in Modern Standard Arabic, generated automatically from Arabic Wikipedia: GPT-4o created the claims and extracted the evidence, and Claude Sonnet 3.5 validated each pair as supported, refuted, or not-enough-information (NEI). It is publicly available on GitHub and Zenodo.
How accurate are ARAFA's labels?
On a 400-pair human-annotated sample, the validation model (Claude Sonnet 3.5) matched expert labels on 86% of supported and 88% of refuted claims; non-expert annotators agreed with each other at Cohen's kappa 0.89 (supported) and 0.94 (refuted). Caveat: the validation model and its scoring threshold were selected on this same sample, so independent estimates could be lower.
Which models perform best on ARAFA?
The authors fine-tuned AraBERTv2, AraModernBert-Base, Llama-3.1-8B, and Qwen2.5-7B. AraModernBert-Base achieved the best macro F1 (77%). Encoder models were best at supported claims (F1 87-88%), while generative models excelled at detecting not-enough-information cases (F1 85-89%), suggesting an ensemble of both types.
Can ARAFA be used for other languages?
Not directly - ARAFA covers Modern Standard Arabic only and is built from Arabic Wikipedia. However, the authors argue the generation framework (prompts, mutation strategies, validation scoring) is language-agnostic and could be adapted to build similar fact-checking datasets for other low-resource languages.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Arabic text in all tables and examples is garbled by text extraction (mojibake), so claim quality and linguistic examples could not be directly verified.
- Figures 1-5 are referenced but not analyzable from extracted text; numeric claims read off them (e.g., 20%/4% duplication rates) are taken on trust.
- The supplementary prompts (CoT, Arabic rules, validation) are cited but not included in the extracted text, limiting prompt-level assessment.
- Review is based on arXiv v1 single-column extraction; minor truncation around tables is possible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.