Authorship attribution and aesthetic evaluation of AI poetry: a case study with Haiku
Livia Oddi, Simone Scardapane, Toru Sugimoto, Donatella Genovese
cs.CL · cs.AI
Quick verdict
Finds near-chance human detection of top-model haiku; ratings track perceived, not true, authorship. Sound design, tiny stimulus set limits generalization.
Summary
The paper asks whether human readers can distinguish AI-generated Japanese haiku from human-written ones, and whether aesthetic impressions track true authorship. The authors generated 72 regular 5-7-5 haiku (three per season) with six LLMs - GPT-5, Gemini 2.5, StableLM-7B, Gemma-2B, ELYZA-adapted LLaMA-2, and LLM-JP - via few-shot prompting with line-by-line stochastic sampling, and mixed them with 12 prize-winning human haiku in 12 questionnaires completed by 144 Japanese university students (12 per set), who rated seven impression dimensions and judged authorship. Recognition accuracy was near chance for GPT-5 (0.493), Gemini 2.5 (0.503), and StableLM-7B (0.524), and moderate for LLM-JP (0.590), Gemma-2B (0.611), and LLaMA-2 (0.670), with strong item-level variability. Regularized logistic regression and Bayesian mixed GLMMs show impression ratings predict perceived authorship (cross-validated AUC 0.793; favourability OR 1.72) but not correct recognition (AUC 0.505), indicating an attribution bias; exploratory Grok-as-judge results mirror the human pattern. Main limitations: only 12 unique human haiku, each AI model represented by 12 haiku judged by just 12 participants; model-level accuracies lack confidence intervals or significance tests; filtering generated candidates for constraint satisfaction introduces selection effects; participant-level AUCs below 0.5 are reported without explanation. Code and prompts are stated to be publicly available.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Human readers distinguish haiku generated by GPT-5, Gemini 2.5, and StableLM-7B from prize-winning human haiku only at chance level (accuracy 0.493-0.524), while LLM-JP (0.590), Gemma-2B (0.611), and LLaMA-2 (0.670) are moderately detectable.
- Recognition is strongly item-dependent rather than a stable model-level signal (84 unique haiku, Figure 1).
- The seven impression ratings predict perceived authorship (CV-AUC 0.793; favourability OR 1.72) but not correct recognition (CV-AUC 0.505; favourability OR 0.83), showing aesthetic cues act as attribution heuristics rather than authorship diagnostics.
- Exploratory Grok-as-judge classification mirrors the human pattern, judging all GPT-5, Gemini 2.5, and StableLM-7B haiku as human-written (50% accuracy) while reaching 71-83% on the other three models.
Strengths
Culturally grounded evaluation design
Ratings and attribution tasks were run with 144 Japanese university students (mostly native speakers) on seasonal 5-7-5 haiku with a curated, prize-winning human reference set from named contests - Sophiakai, Haiku Nippon, EJC (§4.1) - avoiding the typical English-only, uncurated-comparison setup of AI-poetry studies.
A clear, well-framed core finding
§5.4 shows the same seven ratings predict perceived authorship (CV-AUC 0.793, favourability OR 1.72) but not correct recognition (CV-AUC 0.505, favourability OR 0.83) - a clean dissociation, qualitatively corroborated by the Grok judge (Appendix C) and framed carefully as attribution bias rather than a Turing test (§1).
Transparent generation pipeline
Honest, well-scoped claims
Weaknesses
No uncertainty on the headline numbers
Small, partly reused stimulus set
Selection bias via constraint filtering
Unexplained statistical anomalies
Generation protocol may under-represent model outputs
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper studies human detection and aesthetic evaluation of Japanese haiku generated by six LLMs (GPT-5, Gemini 2.5, StableLM-7B, Gemma-2B, ELYZA-adapted LLaMA-2, LLM-JP), asking (i) whether readers can attribute authorship and (ii) whether the cues driving perceived humanness also support correct attribution. Central claims: near-chance recognition for GPT-5 (0.493), Gemini 2.5 (0.503), and StableLM-7B (0.524); moderate detectability for LLM-JP (0.590), Gemma-2B (0.611), and LLaMA-2 (0.670); strong item dependence; and a dissociation whereby impression ratings predict perceived authorship (CV-AUC 0.793) but not true recognition (CV-AUC 0.505). The authors scope the claims to constrained few-shot seasonal haiku (§1, §8), which is appropriate.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can readers tell AI-generated haiku from human-written haiku?
In this study with 144 Japanese university students, recognition accuracy was near chance (0.49-0.52) for haiku generated by GPT-5, Gemini 2.5, and StableLM-7B, and moderately above chance (0.59-0.67) for LLM-JP, Gemma-2B, and LLaMA-2. Accuracy depended heavily on the individual poem rather than on the model alone.
Did people rate AI poetry worse than human poetry?
No. Overall impression ratings (fluency, poeticness, coherence, etc., on a 1-5 scale) were similar for AI-generated and human haiku. However, ratings tracked what participants believed the source was: more favourably rated poems were more likely to be judged human-written (cross-validated AUC 0.793), even when that judgment was wrong - the same ratings predicted correct recognition at only AUC 0.505.
Was GPT-5's haiku detectable as AI-written?
Not by the human participants: accuracy was 0.493, essentially chance (of 144 judgments on human-written haiku, 75 were correct; of 144 judgments on GPT-5 haiku, 67 were correct). An exploratory Grok-based judge likewise classified all GPT-5 haiku as human-written, yielding 50% accuracy on the balanced set.
What are the main limitations of this haiku AI study?
Only 12 human and 12 AI haiku per model, each judged by 12 Tokyo-area university students; AI poems were pre-filtered for formal rule compliance (acceptance rates 62.5%-100% by model); per-model accuracies are reported without confidence intervals or significance tests; and the human reference set consisted solely of prize-winning contest haiku, which may flatter the human class.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-6 appear in the extraction only as captions and axis labels; plotted distributions (e.g., Figure 1 histogram bins) could not be visually verified.
- Japanese prompt strings are blanked or garbled in the extracted text (§3.3), so exact prompt wording is partly unverifiable.
- Only three example haiku (Table 3) are shown; the remaining 81 stimuli cannot be inspected from the text.
- The cited GitHub repository and raw response data could not be checked from the text alone; availability is taken from the authors' statement.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.