SWARM: A Multilingual Human-Annotated Dataset for Russian Propaganda Detection in Search Engine Results
Manuel Tonneau, Abhinav Dubey, Farhan Shaikh, Ilaria Vitulano, Martha Stolze, Hale Dedeoglu, Clara Riechert, Ella Kuka, et al.
cs.CL
Quick verdict
SWARM: 2,183 search results in 9 languages labeled for Russian propaganda-narrative support; blocklists fail (F1 0.39); best LLM 0.73, still unreliable.
Summary
The paper introduces SWARM, a dataset of 2,183 search-engine results in nine languages, collected through an agent-based audit that issued 360 queries derived from twenty recurring Russian propaganda narratives to Google, Bing, Yandex, and DuckDuckGo across six regions on 29-30 January 2024. Trained annotators, with third-coder adjudication, labeled each document for whether it supports its narrative question. Against these labels, a source-based domain blocklist reaches only 0.39 positive-class F1, recovering 35% of supporting documents, 69% of whose misses sit on mainstream domains; supervised classifiers (XLM-R, e5+logistic regression, SetFit) reach 0.47-0.51; zero-shot LLMs reach 0.62-0.71 on English input and 0.73 for GPT-5.4 on native input, with smaller LLMs over-predicting support by conflating topic with endorsement. English translation shifts F1 by at most 0.02. Key limitations: moderate inter-annotator agreement (Krippendorff's alpha = 0.54; 0.41 English), gold labels assigned on gpt-5-nano translations spot-checked on only 78 items, a two-day snapshot with residual source-selection skew, and an unexplained 2,129 vs 2,183 evaluation-set discrepancy in the results-table captions. Data are gated research-use-only; evaluation code is public.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Source-based blocklists miss most narrative-supporting search results: 35% recall at 43% precision (F1 0.39), with 69% of the 251 missed supporting documents on mainstream news, government, IGO, or fact-checker domains.
- Zero-shot LLMs beat supervised classifiers on the minority support class (0.62-0.71 vs 0.47-0.51 positive-class F1 on English input), with GPT-5.4 reaching 0.73 on native-language input.
- Smaller LLMs systematically over-predict support, mistaking topical relevance for narrative endorsement - the same boundary on which human coders disagree most.
- SWARM is the first multilingual, human-annotated dataset for propaganda-narrative support in search engine results.
- Translating documents to English changes positive-class F1 by at most 0.02, so per-language evaluation is not confounded by the translation pipeline.
Strengths
New label target with clear positioning against prior schemes
SWARM is the first resource with human, document-level narrative-support labels over multilingual search results (Table 1), filling the gap between outlet-reputation labels (Proppy, Rashkin, Solopova) and narrative-typing schemes that record which narrative appears but not whether it is endorsed (SemEval-2025 T10, HALT-PROP).
Unusually transparent selection-bias accounting
Appendix H decomposes all 503 non-retained URLs (293 extraction failures, 204 annotated-but-unassessable, 6 ties), shows the Wayback recovery pass removed most pro-Kremlin skew (blocklist membership 18% non-retained vs 15% retained; .ru 13% vs 9%, Table 12), and frames reported support rates as a lower bound rather than hiding the effect.
Ablations that preempt the obvious objections
Error analysis links model failures to human difficulty
Per-language reporting with uncertainty
Weaknesses
Gold labels mediated by machine translation from the smallest model under test
Moderate, subset-based, non-independent label reliability
Unexplained evaluation-set discrepancy in both main results tables
Supervised comparison is configuration-handicapped
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SWARM targets a task prior propaganda NLP has not labeled: does a retrieved web document *support* a specific recurring pro-Kremlin narrative, judged on content rather than source. The authors position it against three schemes they critique (§2, Table 1): persuasion-technique span labeling (Da San Martino et al. 2019), outlet-reputation labels (Proppy, Rashkin, Solopova), and narrative typing without support (SemEval-2025 T10, HALT-PROP). Three claims anchor the paper: (1) blocklists miss most narrative support - 35% recall, F1 0.39, with 69% of the 251 missed supporting documents on mainstream news/government/IGO/fact-checker domains (§5, Table 11); (2) zero-shot LLMs beat supervised baselines on the minority class (0.62-0.71 vs 0.47-0.51 F1 English input; GPT-5.4 0.73 native, Table 8); (3) smaller LLMs conflate topical relevance with endorsement. The presented data support all three directions, with the caveats below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SWARM dataset?
SWARM is a dataset of 2,183 search-engine results in nine languages (Arabic, English, German, Hindi, Polish, Portuguese, Russian, Spanish, Ukrainian), each labeled by trained annotators for whether the linked document supports one of twenty recurring Russian propaganda narratives. It was built by auditing Google, Bing, Yandex, and DuckDuckGo in January 2024 and is released under gated research-only access on Hugging Face, with public evaluation code on GitHub.
How well can AI models detect Russian propaganda narratives in search results?
In the paper's benchmark, the best model (GPT-5.4 reading native-language text) reaches a positive-class F1 of 0.73 - useful but not reliable enough to treat predictions as ground truth. Smaller models over-predict support, flagging any document that merely discusses a propaganda topic. Supervised classifiers reach only about 0.5 F1 because the support class has just 388 positive examples.
Why isn't a propaganda website blocklist enough?
The source-based blocklist flags 15% of documents and finds only 35% of the propaganda-supporting ones, at 43% precision; 69% of the supporting documents it misses appear on mainstream news, government, or fact-checker domains that no blocklist contains, so content-level judgment of each document is required.
Is the SWARM dataset publicly available?
Yes, but gated: researchers must request access on Hugging Face and agree to non-commercial research use, no redistribution, and honoring takedown requests from rights holders. The release includes both the English-translated and source-language text, and the evaluation code reproducing every table is public on GitHub.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 is extracted as inline text interleaved with body paragraphs; only its textual labels were available, so no judgment is made on the figure's visual content.
- The string 'GPT-5.1 native' appears in the extracted Table 8 while all other rows and the abstract say GPT-5.4; this is flagged as an inconsistency without knowing whether it originates in the PDF itself or in the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.