From Detection to Characterization: A Large-Scale Study of Ragebait on Japanese X
Zhiyang Qi, Kazuhiro Ito, Jinghui Chen, Hibiki Nakamura, Zhangxuan Chen, Erina Murata, Masaki Chujyo, Fujio Toriumi
cs.SI · cs.CL
Quick verdict
LLM-labeled ragebait detector for Japanese X hits 84% acc vs GPT labels; large-scale findings plausible but topic-confounded, untested against human gold.
Summary
The paper builds a ragebait detection framework for Japanese-language posts on X and uses it for a large-scale characterization. GPT-5.4 mini pseudo-labels 20,000 randomly sampled posts (4.11% ragebait); an initial RoBERTa detector (74% test accuracy) retrieves 30,000 high-engagement candidates for relabeling, yielding a balanced dataset of 18,558 posts. A 200-post human check finds moderate agreement (κ≈0.50) between GPT and two annotators. Six Japanese pretrained models are fine-tuned; majority voting over the three best (Tohoku BERT base v3, Rinna RoBERTa, LINE DistilBERT) reaches 84.05% accuracy and 84.04% Macro-F1 on a 2,000-post held-out set — measured against GPT labels, not a human gold standard. Applied to about 17.4M posts (October 2022–June 2023), 6.42% are flagged. Structural topic modeling (K=25) shows concentration in politics, discrimination, vaccines, and moral conflict (top four topics: 52.22% vs 14.78% mean prevalence). Flagged posts show faster reply/quote growth, and their reactions contain far more negative sentiment and anger/disgust (anger in 35.53% vs 6.40% of quotes). Main limitations: pseudo-label circularity, engagement-based selection bias, and no topic-controlled or statistical testing for the reaction analyses.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A majority-vote ensemble of Tohoku BERT base v3, Rinna RoBERTa base, and LINE DistilBERT base detects ragebait with 84.05% accuracy and 84.04% Macro-F1 on a 2,000-post held-out set labeled by GPT-5.4 mini.
- 6.42% (1,118,931 of 17,435,881) of keyword-filtered Japanese posts recovered from Oct 2022–Jun 2023 repost-event records are classified as ragebait.
- Ragebait concentrates in party politics, gender/discrimination, vaccines/infectious disease, and moral conflict — 52.22% vs 14.78% mean topic prevalence for the top four topics.
- Ragebait posts show faster reply/quote growth and draw far more negative reactions, with anger in 35.53% vs 6.40% of quotes and disgust in 69.46% vs 21.18%.
Strengths
Efficient two-stage pseudo-labeling pipeline
Random sampling yields only 4.11% positives (822/20,000); detector-assisted retrieval lifts stage-two yield to 29.2% (8,747/30,000), and all pipeline counts reconcile in Fig. 1 (822+8,747=9,569 → 9,279 after dedup).
Honest annotation-reliability reporting
Two independent annotators vs GPT on 200 posts: κ=0.500/0.498, human–human κ=0.570 (Sec. III-F), with explicit framing that GPT labels are not gold standard and classifier scores must be read under annotation ambiguity.
Multi-model comparison with single-use test set
Converging multi-dimensional characterization
Privacy-aware release
Weaknesses
No human gold-standard evaluation of the final detector
Topic confound in the headline analyses
No statistical inference anywhere
Corpus inconsistency and selection bias
Single mini-judge labels
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets ragebait — content crafted to provoke anger for engagement — in Japanese-language posts on X. It makes four linked claims: (1) an LLM-assisted pseudo-labeling pipeline plus an ensemble of Japanese PLMs yields a usable detector (84.05% accuracy, 84.04% Macro-F1, Table II ensemble row); (2) applied to ~17.4M posts (Oct 2022–Jun 2023), 6.42% are flagged; (3) flagged content concentrates in party politics, gender/discrimination, vaccines, and moral conflict (Sec. V-B: top-four topics 52.22% vs 14.78% mean prevalence); (4) flagged posts show faster reply/quote growth (Sec. V-C) and reactions richer in negative sentiment, anger, and disgust (Table III: anger in quotes 35.53% vs 6.40%). The framing is candid about annotation ambiguity (Sec. III-F), but causal language in V-C/V-D and the abstract ('elicit', 'trigger') outruns the correlational design.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'ragebait' mean in this paper?
Content intentionally created to provoke anger or outrage so that it attracts attention and engagement — distinct from clickbait, which exploits curiosity. The paper notes 'rage bait' was Oxford's Word of the Year 2025. Operationally, a post counts as ragebait if it appears intended to anger or provoke readers and likely to elicit anger, indignation, disgust, or strong discomfort.
How accurate is the ragebait detector?
The final three-model majority-vote ensemble reaches 84.05% accuracy and 84.04% Macro-F1 on 2,000 held-out Japanese posts. Important caveat: this is measured against labels produced by GPT-5.4 mini, and human–GPT agreement on a 200-post check was moderate (κ≈0.50), so real-world accuracy against human consensus is unknown.
What fraction of Japanese X posts are ragebait?
In this study, 6.42% (about 1.12M of 17.4M) of posts collected October 2022–June 2023 were flagged. This applies only to a filtered corpus (posts recovered from repost-event records, commercial content removed), not all of X; in a separate dataset of posts receiving replies/quotes, the flagged rate was 2.45%.
Is the dataset or code publicly available?
Partially. The authors release the source-post IDs, the labeling prompt, and the trained model checkpoints via GitHub; the original post texts and labels are withheld to protect user privacy, and the underlying 1% X sample is not public.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Values inside figures (word-cloud magnitudes Fig. 2, log-odds scores Fig. 3, per-topic prevalences Fig. 4, growth curves Fig. 5) are not extractable from the text; diffusion claims are assessed from the prose and captions only.
- Example posts appear only as GPT-5.5 English translations, and the exact labeling prompt wording beyond its Sec. III-C paraphrase is not in the provided text, so the Japanese-language labeling criteria cannot be audited from this extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.