Agentic Detection of Online Conspiracies
cs.CL · cs.LG
Quick verdict
Claims context-aware agentic LLM lifts Hebrew conspiracy-tweet F1 0.535→0.730 over text-only; persuasive, though no significance tests, small skewed eval set.
Summary
The paper reframes conspiracy-tweet detection as a socially embedded interpretation task: the same surface text may endorse, mock, report on, or criticize a conspiracy, so systems must infer the speaker's intent (illocutionary force). Using a unique archive of 270 million Hebrew tweets (Dec 2018-2023, est. 80-90% coverage of public Hebrew Twitter), the authors built a keyword-retrieved pool of 33,121 conspiracy-related candidates and manually annotated 504 tweets sampled to oversample cases where GPT-5 and Gemini 3 Flash disagreed under text-only conditions; two domain experts fixed the labels (331 non-conspiracy, 173 conspiracy). An agentic framework built on LangChain/LangGraph with Gemini 3 Flash can call four tools — tweet metadata, author profile, 30-day user history, and retweet-based ego-network context — all temporally masked to data preceding the target tweet. Against a text-only baseline (accuracy 0.603, F1 0.535), the full agentic system reaches accuracy 0.790 and F1 0.730; a degenerate agent given the same contexts preloaded scores 0.670 F1, and a two-agent debate scores 0.558. Tool-use traces show selective retrieval (no tools in 10.9% of cases) at roughly 1.5x input characters. Limitations include the absence of significance testing, a small skewed evaluation set, a single language and model family, unoptimized hyperparameters, and no stated code or data release.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Context-aware workflows consistently outperform text-only classification for Hebrew conspiracy tweets, with F1 rising from 0.535 to 0.730.
- Agency matters beyond context availability: an agent that decides when to retrieve performs better than a non-agentic model given the exact same contexts preloaded (F1 0.730 vs 0.670).
- The agent is selectively economical: it uses no tools in 10.9% of cases and matches text-only accuracy (94.5%) on those easy cases, spending extra tokens mainly on hard ones.
- Conspiracy detection should be treated as a socially embedded interpretation task targeting the utterance's illocutionary force rather than surface content.
Strengths
Mechanism-isolating experimental design
Table 4's preloaded vs on-demand conditions give the agent and the degenerate baseline the identical four context sources, so the F1 gap (0.730 vs 0.670) and the fact that agentic History-only (0.698) beats degenerate-full (0.670) isolate the value of selective retrieval rather than mere context availability
Unique, leak-controlled data asset
270M tweets from 3.3M users (§3.1) with tools temporally masked to pre-date the target tweet (§4.2) — genuine, leak-controlled social-context retrieval that COCO/YouNICon-style datasets cannot support; coverage estimated via Decahose extrapolation (fn. 8)
Adversarial evaluation construction
Behavioral analysis beyond headline accuracy
Unusually transparent prompt documentation
Weaknesses
No inferential statistics or run variance
Gold-label circularity risk and unreported IAA
Evaluation set is 88.5% model-disagreement cases
Missing supervised baseline; model confound in debate
Fixed, unoptimized tool hyperparameters
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper reframes Hebrew conspiracy-tweet classification as a *socially embedded interpretation* task: identical surface text may endorse, report, criticize, or mock a conspiracy, so the target is the speaker's illocutionary force, not topic presence (§1, adopting the Douglas & Sutton 2023 definition). Four claims are advanced: (i) social context materially improves detection; (ii) *agency* — an agent deciding when and which context to retrieve — beats preloading the identical context into a 'degenerate agent'; (iii) the agent is selective and therefore token-economical; (iv) a two-agent debate underperforms. Evidence: a 270M-tweet Hebrew archive (claimed 80-90% coverage, Dec 2018-2023), 33,121 keyword-retrieved candidates, and 504 manually annotated tweets oversampled from GPT-5/Gemini 3 Flash text-only disagreements.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main result of this paper?
On 504 manually labeled Hebrew tweets, an agentic LLM (Gemini 3 Flash with four context-retrieval tools) lifts conspiracy-detection F1 from 0.535 (text-only) to 0.730, and beats a non-agentic model given the exact same contexts preloaded into its prompt (F1 0.670).
Why is an agent better than just giving the model all the context?
The authors argue that preloading everything adds noise and distracts the model. The agent decides per case whether to retrieve context — using no tools in 10.9% of cases — and on the harder tool-using cases accuracy rises from 56.1% (text-only) to 77.1%.
Is the dataset or code public?
No release is stated in the paper. The evidence rests on a private archive of 270 million Hebrew tweets (2018–2023) and closed models (Gemini 3 Flash, GPT-5); user histories and network data raise privacy barriers the authors discuss in an ethical note.
What makes this evaluation adversarial?
The annotated set oversamples cases where GPT-5 and Gemini 3 Flash disagreed on text-only classification (88.5% of the 504 tweets), deliberately concentrating on adversarial, context-dependent cases; absolute scores therefore look lower than they would on a random sample.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 (workflow diagrams) are not analyzable from the extracted text
- Original Hebrew texts in Appendix B are empty in the extraction; judgments rely on the provided English translations
- Appendix D retrieval-lexicon tables (Tables 10-22) appear as captions without contents in the extracted text
- Corpus end date is inconsistent across the paper (abstract 'early 2023' vs §1 'June 2023' vs §3.1 'March 2023'); unresolved from text
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.