A Repeated-Measurement Study for Cultural Analytics of English Song Lyrics Using Five Large Language Models
E. Cho Smith, Samuel Ho, Dawn Laux
cs.LG
Quick verdict
Large-scale LLM lyric-annotation reliability study; construct-level honesty strong, but unfixed decoding params and thin transfer baselines weaken conclusions.
Summary
This arXiv preprint from Purdue researchers evaluates five 2025-era large language models (GPT-4o-mini, o3-mini, Claude 3.7 Sonnet, DeepSeek-R1, Gemini 2.0 Flash) as zero-shot annotators of four psychological constructs (self-esteem, self-control, seeking belonging, seeking recognition) in 69,130 English song lyrics from the Music4All corpus, using the authors' PNACSS four-binary-dimension coding scheme. Across three repeated annotation rounds per model, self-esteem showed the strongest within-model reliability (Fleiss' κ .717–.826), seeking recognition the weakest (.467–.683), with joint 16-category κ between .482 and .614. Cross-model agreement was consistently lower than within-model stability (e.g., seeking belonging κ ≈ .44 across five models but only .337 between Gemini and GPT-4o-mini consensus labels). A BigBird transfer experiment showed silver labels contain learnable signal (binary macro-F1 .684–.732) but severe positive-class over-prediction on a human-validated external set of 1,572 lyrics (seeking recognition precision .207, recall .886). Limitations include unfixed decoding parameters, highly uneven within-model sample sizes (n = 1,454 to 55,982), a partially re-annotated third round, an external set drawn from the authors' own prior work, and no public code or data due to lyric copyright.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 4
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLM annotation reliability for latent constructs in lyrics is construct-dependent: self-esteem is most stable (within-model Fleiss' κ .717–.826) and seeking recognition least stable (.467–.683).
- Cross-model convergence is consistently weaker than within-model repeatability, so a single LLM's reproducibility does not imply that different LLMs recover the same measurement.
- Consensus silver labels contain learnable signal for supervised classification (binary macro-F1 .684–.732), but transferability does not establish construct validity.
- Under natural class imbalance, downstream classifiers over-predict positive classes (seeking recognition: precision .207, recall .886), reproducing silver-label biases rather than unbiased measurement.
- Repeated-measurement stability and cross-model convergence should be reported, alongside model version, prompt, and provenance, before LLM annotations are treated as scalable measurements in cultural analytics.
Strengths
Unusually rigorous repeated-measurement design at scale
Three annotation rounds × five models over 69,130 English lyrics (Sec. IV.A), with construct-level Fleiss' κ and 95% CIs (Table IV) — a large advance over the prior 5–12-lyric studies the paper itself cites as motivation [5].
Exceptional provenance transparency
Sec. IV.F documents that only 15,592 of 69,130 GPT-4o-mini Round 3 files survived, and frames the 2026 re-annotation of 53,538 items as 'provenance-controlled replication' rather than recovery; Gemini's lost Round 3 is likewise disclosed with a two-round sensitivity check reproducing the construct ordering (SE .721 > SC .652 > SR .600 > SB .593).
Reliability/validity separation held throughout
Methodologically aware agreement analysis
Weaknesses
Severely unequal evidence base across models
Unfixed decoding parameters undermine the core construct
Thin, baseline-free transfer experiment
Partially circular external validation
Reproducibility closed by design
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv:2609.04428, Purdue) asks whether LLM outputs can be trusted as *measurements* of latent constructs in cultural analytics. It evaluates five 2025-era models (GPT-4o-mini, o3-mini, Claude 3.7 Sonnet, DeepSeek-R1, Gemini 2.0 Flash) as zero-shot annotators of four binary PNACSS constructs — self-esteem (SE), self-control (SC), seeking belonging (SB), seeking recognition (SR) — over 69,130 English lyrics from Music4All (63.3% of 109,269 after token-length and language filters, Sec. IV.A). Three research questions structure the work: (RQ1) within-model repeated-measurement reliability, (RQ2) cross-model convergence, (RQ3) transferability of consensus silver labels to BigBird classifiers. Construct validity is explicitly scoped out, and the central normative claim — report repeated-measurement stability and cross-model convergence before treating LLM annotations as corpus-level evidence — is well-taken and consistently applied.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which psychological construct did LLMs annotate most reliably in song lyrics?
Self-esteem. Across five 2025-era LLMs and three repeated annotation rounds over 69,130 English lyrics, within-model Fleiss' κ for self-esteem ranged from .717 to .826 — the highest of the four constructs studied. Seeking recognition was least stable, with κ from .467 to .683.
Do different LLMs agree with each other when labeling song lyrics?
Only partially. Five-model Fleiss' κ ranged from about .42 (seeking recognition) to .61 (self-esteem) across two annotation rounds — consistently below each model's own repeat stability. In the sharpest pairwise contrast, Gemini 2.0 Flash and GPT-4o-mini agreed on 80.33% of seeking-belonging consensus labels, yet chance-corrected Cohen's κ was only .337.
Can LLM-generated labels replace human coders in cultural analytics?
This paper argues not without safeguards. Repeat stability was moderate to strong, but cross-model convergence was weaker, and BigBird classifiers trained on consensus LLM labels over-predicted positive classes on a human-validated external set of 1,572 lyrics (seeking recognition: precision .207 at recall .886). The authors recommend treating LLM labels as weak supervision and reporting repeated-measurement and cross-model reliability first.
Is the dataset or code from this study publicly available?
No. Because Music4All song lyrics are copyrighted, the paper states that neither the corpus, the derived annotations, nor the code that processes lyric text can be released; researchers must obtain Music4All from its original creators under its research licensing terms.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2 and 3 (per-model category distributions and confusion matrices) are images; their underlying numbers are not present in the extracted text, so distribution-shape judgments rely on the paper's verbal descriptions (e.g., the Category 8 / LLPN concentration claim).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.