How LLMs Build Fictional Worlds: Setting and Narrative Space in AI-Generated Creative Storytelling
Katrin Rohrbacher, Björn Nieth, Emmanuelle Salin, Bjoern Eskofier, Michaela Mahlberg
cs.CL
Quick verdict
Claims LLMs overproduce atmospheric 'perceived space' vs action-driven human fiction; well-validated cross-lingual study, though baseline is era-limited fiction.
Summary
The paper asks whether large language models construct fictional settings differently from human authors. Building on a narratological framework of five narrative-space categories (action, perceived, visual, descriptive, no space), the authors fine-tune BERT-based classifiers for German and English and apply them to 8,000 AI-generated stories (1,000 each for GPT 4.1, Llama 3.3 70B, Mistral Small 3.2 24B, Gemma 3 27B, in both languages) and length-matched excerpts from Project Gutenberg fiction (English 1800–1920, German 1780–1940). Each story continues the first sentence of a human source text with a genre label, generated chapter-by-chapter at temperature 1. The central finding: human fiction is dominated by action space, while all four LLMs significantly overproduce perceived space (GPT 4.1 ~0.47 vs ~0.19 human in English openings); ordinal-beta GLMMs with story random effects confirm author effects and author×section interactions, robust to prompt variants, chapter-boundary effects, memorization checks (13-gram), and genre covariates. Deviations are model-specific and language-sensitive: open-source models approach the German human baseline while GPT 4.1 stays elevated. Spatial profiles alone identify source models above chance (human recall 0.97 English). Main limitations: the human baseline is historical public-domain fiction, generation relies on a single role-based prompt family, and classifier accuracy on the pivotal perceived category is imperfect (κ 0.646 inter-annotator; Mistral German 62.7%). Code, data, and annotation resources are released.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Human-authored fiction is dominated by action space, whereas all four tested LLMs significantly overproduce perceived space (e.g., GPT 4.1 ~0.47 vs ~0.19 in English openings; +0.14–0.25 above baseline across sections).
- The human–LLM divergence is stable across narrative time and robust to prompt formulation, chapter-boundary artifacts, memorization, and genre covariation.
- Deviation magnitude is model-specific (GPT 4.1 largest; Gemma 3 closest to baseline) and language-sensitive (open-source models converge toward the German human baseline).
- Spatial category distributions alone identify the generating source well above chance (human recall 0.97 English, 0.79 German; Appendix E), making setting a stylistic marker of AI-generated fiction.
Strengths
Matched paired design at unusual scale
8,000 stories, each continuing the first sentence of a real source text under the same genre label (Sec. 3.2), with length-matched human excerpts following Lucy & Bamman (2021); authorship is effectively the only manipulated factor.
Theory-grounded measurement validated on the target domain
Five categories from lived-space narratology (Table 3); the English classifier (macro F1 0.82, Table 5) is validated on 600 hand-annotated AI-generated sentences (κ 0.725, Table 7), and the pivotal action-vs-perceived confusion is shown rare (8/600 annotators; 4/300 EN and 2/300 DE classifier-vs-gold, Fig. 5).
Unusually complete robustness battery
Cross-linguistic replication with a substantive finding
Candid validation and release practice
Weaknesses
Era-restricted human baseline
Single role-framed prompt family
Label noise concentrates on non-focal margins
Effect-size reporting is prose-only for headline contrasts
One generation run per story; arc truncated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper operationalizes *setting* — one dimension of storyworld construction — as five narrative-space categories (action, perceived, visual, descriptive, no space; Table 3) from Rohrbacher (2025b)'s lived-space framework. Using fine-tuned BERT-family classifiers (German original; English replication on roberta-base), the authors compare 8,000 LLM-generated stories (1,000 per model per language; GPT 4.1, Llama 3.3 70B, Mistral Small 3.2 24B, Gemma 3 27B) with length-matched excerpts from Project Gutenberg fiction (English 1800–1920; German 1780–1940). Headline claims: (1) human fiction is action-space-dominated while all four LLMs significantly overproduce perceived space; (2) the divergence is stable across narrative time and robust to prompt, chapter-boundary, memorization, and genre checks; (3) deviations are model-specific and language-sensitive; (4) spatial profiles alone identify the generating source above chance (Appendix E).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What did this paper find about how LLMs write settings in fiction?
Compared with human fiction from Project Gutenberg, the four LLMs studied (GPT 4.1, Llama 3.3, Mistral 3.2, Gemma 3) produce significantly more 'perceived space' — atmosphere- and mood-laden sentences — and relatively less 'action space', where characters physically interact with their environment. For example, in the first 15 sentences of English stories, GPT 4.1 reaches a perceived-space proportion of about 0.47 versus about 0.19 in the human corpus.
Is the AI-vs-human difference consistent across a whole story, or only at the opening?
It persists across ten equal narrative sections and is robust to prompt rewording, chapter-boundary artifacts of the generation procedure, memorization checks (13-gram overlap), and genre differences. Its size varies by model and language: open-source models sit closer to the German human baseline, while GPT 4.1 stays elevated in both English and German.
Can you detect AI-written fiction from these spatial patterns alone?
Partly. A random forest trained only on per-section spatial-category proportions identified the source above chance: human texts were easiest to spot (recall 0.97 in English, 0.79 in German) and Mistral 3.2 hardest (0.56 in English). So setting is a useful but not decisive stylistic marker of AI-generated prose.
What data, code, and models are publicly available from this study?
The 8,000 generated stories, metadata and opening sentences of the matched human sample, the English setting classifier (RoBERTa-based, five categories), and annotation resources are released at github.com/BjoernNieth/worldbuilding-AI. Regenerating the full corpus additionally requires OpenAI API access for GPT 4.1 and GPU inference for the open-weight models.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Numeric values plotted in Figures 2–4, 6 and 9 exist in the extraction only as prose approximations (e.g., '~0.47', '0.14–0.25'); exact per-section point estimates could not be verified from figure content.
- No truncation observed — the extraction includes appendices A–I and references — but figure internals are rendered as raw label strings, so figure-level effect sizes rest on the prose descriptions and printed tables.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.