Do Social Patterns Hold in Synthetic Data? Analyzing Cyberbullying Dynamics in LLM-Generated and Authentic Dialogues
Arefeh Kazemi, Hamza Qadeer, Sinan Asci, Joachim Wagner, Brian Davis
cs.CL · cs.CY
Quick verdict
Synthetic CB dialogues keep coarse structure but distort magnitude, roles, and escalation; credible framework, yet only 10 authentic conversations anchor it.
Summary
The paper asks whether LLM-generated cyberbullying dialogues reproduce the social dynamics of authentic interactions, beyond downstream task utility. The authors compare an authentic WhatsApp teen role-play corpus (10 conversations, 2,192 messages) against SynBullying dialogues generated by GPT-4o, Grok-2, and Llama-3.3-70B (40 conversations each), measuring turn-taking, power and repair dialogue acts, pronoun use, humor, CB-type distributions, profanity, toxicity, and binned escalation curves, plus a blinded three-annotator evaluation of 99 snippets. Findings: synthetic data preserves coarse structure — role participation (e.g., Grok Cramér's V 0.042), bully-over-victim power asymmetry, and broad marker orderings — but systematically distorts magnitude, role-specific allocation, category distributions, and temporal dynamics. Distortions are model-dependent: GPT suppresses harm (66.7% CB presence in human evaluation; attenuated profanity and toxicity), Grok amplifies it (37.6% bully profanity; inverted escalation correlation -0.33 despite the lowest toxicity JSD of 0.0123), and LLaMA best matches magnitude and trend (L2 0.236; social realism 4.36 vs authentic 4.38) while smoothing role distinctions. The framework is well motivated and the structure-versus-dynamics dissociation is genuinely informative. Main limitations: conclusions rest on only 10 authentic role-play conversations with no uncertainty estimates; GPT-4o both annotates the synthetic data and powers several classifiers, risking circularity; and statistical reporting contains internal inconsistencies (Section 5.3.2).
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 3
● low 0
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLM-generated CB dialogues consistently preserve high-level interactional structure: role participation patterns, directional bully-over-victim power asymmetry, and broad behavioral-marker orderings.
- All three models systematically distort behavioral magnitude, role-specific allocation of behaviors, CB-type distributions, and temporal escalation dynamics.
- Distortions are strongly model-dependent: GPT suppresses harmful content, Grok amplifies aggression while preserving directional orderings, and LLaMA is the most balanced approximation while smoothing role distinctions.
- Synthetic CB data is useful for global conversational structure but should not be treated as a faithful substitute for authentic interactions when behavioral realism and social dynamics matter.
Strengths
Theoretically grounded, multi-level metric framework
Section 4 operationalizes each metric (turn-taking, power acts, repair, pronouns, humor, CB types, profanity, toxicity, escalation) with a definition, social-science motivation (Sacks et al. 1974; Fairclough 2013; Pennebaker-style pronoun analysis), and an explicit formula — far more rigorous than typical 'we compared distributions' papers.
Clean demonstration that distributional similarity does not imply dynamic fidelity
Grok attains the lowest toxicity JSD (0.0123) yet inverts the escalation trajectory (Pearson -0.3294, Table 4); GPT's apparent profanity alignment is explicitly attributed to safety suppression 'rather than faithful reproduction' (5.3.2). This structure-vs-magnitude dissociation is the paper's most valuable analytic insight.
Human evaluation exposes a blind spot of the automatic metrics
Honest, actionable model-wise synthesis
Weaknesses
The entire 'authentic' gold standard is 10 role-play conversations
LLM-in-the-loop circularity
Statistical reporting gaps and an internal inconsistency
Structural confounds between the compared corpora
Thin human evaluation with near-zero raw agreement on Likert items
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper on synthetic cyberbullying dialogues?
LLM-generated cyberbullying conversations reproduce the coarse social structure of authentic ones — role participation patterns and the bully-over-victim power asymmetry — but systematically distort how much aggression is expressed, which roles express it, the distribution of cyberbullying types, and how toxicity escalates over time. The effects are model-specific: GPT-4o suppresses harmful content, Grok-2 amplifies it, and Llama-3.3-70B is the most balanced overall but flattens role differences.
Can synthetic LLM data replace real cyberbullying datasets for training detectors?
According to this paper, only partially. Synthetic data is suitable for tasks that rely on global interactional structure, such as data augmentation or training role-aware detectors. It is not a faithful proxy for studying behavioral intensity, rare categories (e.g., discrimination-related insults), repair behavior, humor, or escalation dynamics — and models trained on it may learn surface signals while missing real social mechanisms.
Which LLM produced the most realistic synthetic cyberbullying conversations?
Llama-3.3-70B was the most balanced: it best matched authentic toxicity magnitude (lowest L2 distance, 0.236), tracked the escalation trend (Pearson 0.49), and scored 4.36/5 on social realism versus 4.38 for authentic conversations. However, it smoothed role distinctions. GPT-4o suppressed harm so much that human judges found cyberbullying in only 66.7% of its conversations; Grok-2 amplified profanity and toxicity to unrealistic levels.
What data was used to benchmark the synthetic conversations?
The authentic reference is an English-language WhatsApp corpus of 10 role-play conversations (2,192 messages) from Sprugnoli et al. (2018), built with teen participants in four bullying-trigger scenarios. The synthetic counterpart is the SynBullying dataset: 40 multi-turn conversations per model (GPT-4o, Grok-2, Llama-3.3-70B) covering the same scenarios, with messages annotated for harmfulness and cyberbullying type by GPT-4o.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures appear in the extracted text only as axis labels; plotted values (e.g., Figure 1 role shares, Figure 8 escalation curves) could not be visually verified.
- Section 5.3.2 references a placeholder 'Table X', and the metadata abstract is truncated mid-sentence; the full-text abstract was used instead, suggesting minor extraction/manuscript incompleteness.
- No code or data availability statement appears in the extracted text, so 'codeAvailable: false' reflects absence of a statement rather than confirmed unavailability; the underlying WA and SynBullying datasets come from prior publications.
- Exact prompts, decoding parameters, and classifier versions for generation/annotation are not included in the extracted text, limiting reproducibility assessment.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.