The Garden of Forking Prompts: How Users Explore Narrative Space in Story Generation
Advait Deshmukh, Nora Benedict, Melanie Walsh, Maria Antoniak
cs.CL
Quick verdict
Builds WildStories (275k prompts) and WildEdits (24k edit trees) from real chatbot logs; grounded edit taxonomy is valuable, headline effects modest.
Summary
The authors mine the public WildChat corpus (4.8M conversations) to study how real users iteratively revise story-generation prompts. After English and >=20-word filtering, a Qwen3.5-9B classifier identifies 275,635 story prompts (WildStories; precision 0.78, recall 1.00 against human labels), annotated for story format, prompt components, and explicitness (43.5% explicit). Trigram-shingle clustering (Jaccard >=0.5) then yields WildEdits: 24,291 edit trees, 100,200 prompt pairs, and 165,846 labeled edit actions along four directions (add/remove/change/extend) crossed with fourteen targets (plot, character, fandom, etc.), derived via open coding of 106 pairs plus LLM-assisted refinement and validated by humans (alpha=0.75; 91% agreement). Key findings: edits concentrate on plot and CHANGE; 94.2% of linked prompts have parents in a different conversation, so session-delimited methods miss most editing; within paired users, explicit trees are 1.78x larger (1.54x excluding refusal loops); deeper editing brings only a small specificity gain (median +.039, r=.179). A taxonomy-grounded permutation pipeline matches real-edit drift vectors better than Luminate and a naive baseline (L2 0.15 vs 0.28/0.74). Limitations: classifier noise propagates through all analyses, user identity rests on merging hashed IPs, and the benchmarking case study lacks downstream task evaluation.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Users iteratively edit a shared base story prompt, and 94.2% of prompts with an identified parent have that parent in a different WildChat conversation, so session-delimited analyses miss most iterative editing.
- A taxonomy of four edit directions crossed with fourteen targets, derived from open coding of real revision pairs, captures the dimensions along which users explore narrative space, with edits concentrating on plot and CHANGE.
- Explicit-content trees reflect sustained user interest rather than mere refusal loops: they are 1.78x larger within paired users and remain 1.54x larger when restricted to refusal-free trees.
- Deeper editing is associated with only a small increase in lexical specificity of generated stories (Wilcoxon median change +.039, rank-biserial r=.179).
- A permutation pipeline grounded in the edit taxonomy produces prompt variations whose drift vectors match real user edits more closely than Luminate (L2 0.74) or a naive baseline (0.28), at 0.15.
Strengths
Scale and novelty of the data contribution
WildStories (275,635 prompts, 10.8% of filtered WildChat) and WildEdits (24,291 trees, 100,200 pairs, 165,846 actions) are, per §4, the first edit-tree dataset for story prompts; 94.2% of linked prompts have parents in a different conversation, a finding that quantifies what session-delimited methods miss.
Validation discipline for LLM-in-the-loop labeling
Every model labeling stage is checked against blinded human annotation: story detection precision 0.78/recall 1.00; format 0.75/0.85; components macro 0.72/0.87; explicitness 0.90/0.97 (Krippendorff's alpha 0.72-0.81); edit labels alpha=0.75 with 91% agreement. Annotators were not shown model predictions.
Taxonomy grounded in observed behavior, not generated dimensions
Careful disentangling of confounds in headline analyses
Thorough ethics handling
Weaknesses
Label noise propagates from an imprecise detector
User identity rests on an unvalidated IP-merging assumption
Cluster validation is thin and suspiciously clean
No multiple-comparison control across the association analyses
Case study validates distribution match, not benchmark utility
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This resource-and-analysis paper mines WildChat (4.8M conversations) to study iterative story-prompt editing. It delivers WildStories (275,635 English story prompts, 10.8% of the filtered 2,560,447 pairs; median 114 words), labeled for story format, prompt components, and explicitness (43.5% explicit vs 8.5% in WildChat overall), and WildEdits (24,291 edit trees over 139,147 prompts; 100,200 parent-child pairs; 165,846 labeled actions). Central claims: (1) users iteratively edit a shared base prompt and 94.2% of linked prompts have parents in a different conversation, so session-delimited analyses miss most editing (§4.1, §7); (2) a 4-direction x 14-target taxonomy (Table 1) captures the dimensions users actually manipulate, with edits concentrated on plot and CHANGE; (3) explicit trees reflect sustained interest, not mere refusal loops (1.78x larger; 1.54x excluding refusal trees, §5); (4) deeper editing yields only a small lexical-specificity gain (median +.039, r=.179); (5) taxonomy-grounded permutation matches real-edit drift vectors better than Luminate or a naive baseline (L2 0.15 vs 0.28/0.74, §6). No causal or model-superiority claims are made — appropriately for the evidence presented.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the WildStories dataset?
WildStories is a dataset of 275,635 English story-generation prompts extracted from the public WildChat chatbot log corpus, each labeled with the requested story format (prose, roleplay, script, or narration), prompt components (instructions, jailbreak, story stub, premise, story summary, example), and whether the content is explicit — 43.5% of the prompts are explicit, versus 8.5% of WildChat overall.
What is WildEdits and how was it built?
WildEdits contains 24,291 edit trees tracing how users iteratively revised story prompts. Prompts were clustered with trigram-shingling similarity (Jaccard >= 0.5) on 300-token prefixes, then ordered into trees by timestamp and maximum similarity; 100,200 parent-child pairs were labeled with a taxonomy of 4 edit directions (add, remove, change, extend) crossed with 14 targets such as plot, character, and fandom.
Why does the paper argue that static story benchmarks miss real user behavior?
Because 94.2% of prompts with an identified parent had that parent in a different chatbot conversation, methods that treat each conversation as one session miss most iterative editing; real prompts are also far longer and more specific (median 114 words) than typical benchmark prompts like movie synopses or r/WritingPrompts seeds.
Is explicit prompting in the data just users fighting model refusals?
No. Models refuse about 23% of explicit prompts (vs 1.9% of non-explicit), but within-user comparisons restricted to refusal-free trees still find explicit trees 1.54x larger. Also, 83.1% of jailbreak prompts appear at the root of edit trees, suggesting jailbreaking is usually an up-front strategy rather than a reaction to refusal.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures were extracted as text; visual distributions (Figures 3-14, e.g., the Sankey transitions and PMI heatmaps) were assessed only via caption statistics and axis labels.
- Table 2 (PMI words) extraction is compressed and some cell boundaries may be garbled; cited example words were cross-checked against surrounding text.
- Appendix prompts for annotation were fully included, but per-figure underlying data files were not verifiable from text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.