A Temporal Knowledge Graph for Music Festival Lineup Forecasting
Julia Gastinger, Thilo Dieing, Christian Meilicke, Heiner Stuckenschmidt
cs.LG
Quick verdict
Real-world TKG benchmark for festival lineup forecasting with thoughtful metrics; embedding models OOM on the full graph and gains over recurrence stay modest.
Summary
The authors construct a temporal knowledge graph (TKG) from setlist.fm and MusicBrainz covering 380 German music festivals from 1971 to 2025, with more than 90K festival-performance quadruples (the full concert variant: 278,936 nodes, 23 relations, ~5.07M training quadruples), released as a public benchmark for temporal link prediction. Lineup forecasting is formalized bidirectionally — predicting festivals for a given artist and artists for a given festival — and evaluated on 2025 data. Because one lineup can contain hundreds of correct answers, the authors argue filtered MRR/Hits@k mislead and add R-Precision and normalized 10-precision. Six TKG forecasting models and four zero-shot LLMs are compared: rule-based methods dominate, with CognTKE best at artist prediction (11.9% normalized top-10 precision on concertp) and CountTRuCoLa best at festival prediction (16.5% R-Precision, 40.4% top-10 on concert), while the strongest LLM (quantized Llama 3.3 70B) reaches only 1.7% for artist prediction and shows clear popularity bias and duplicated outputs. Main limitations: embedding models (RE-GCN, CEN, CognTKE) run out of memory on the full graph and are evaluated only on the reduced variant; enriched metadata brings no consistent gains; evidence rests on a single test year without significance testing.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A TKG of 380 German festivals (1971–2025, >90K performance quadruples) built from setlist.fm and MusicBrainz is released as a public benchmark for TKG forecasting with a reproducible construction pipeline.
- MRR and Hits@k are misleading for many-correct-answer queries such as lineup prediction; R-Precision and normalized 10-precision better reflect practical utility.
- Specialized TKG methods clearly outperform zero-shot LLMs on this task (best artist-prediction P@10: 11.9% for CognTKE on concertp vs 1.7% for quantized Llama 3.3 70B), and LLMs show popularity bias and duplicated rankings.
- Current TKG models do not yet exploit enriched context: embedding models fail on memory on the full graph, and adding releases, venues, and label relations yields no consistent gain over performance history alone.
Strengths
Application-grounded benchmark design
Directly answers five documented defects of existing TKG benchmarks (Sec. 3, e.g., >70% of WIKI/YAGO test facts already true at the previous timestep); entities and relations are interpretable, the task maps to a real practitioner question, and a documented construction pipeline plus lightweight ontology (Table 1) ship with the data.
Genuinely forecasting-oriented data
Direct Recurrency of concertp is 2.9% vs 92.7% for YAGO (Sec. 4.4.2), so persistence heuristics cannot trivially win; the COVID-19 disruption is quarantined by the chronological 1971–2023/2024/2025 split (Sec. 4.4.1).
Well-motivated metric addition
Careful, transparent LLM comparison
Full reproducibility package
Weaknesses
Flagship variant untested by neural models
No statistical uncertainty reporting
LLM baseline deliberately constrained
Coverage bias unquantified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This resource/benchmark paper introduces a temporal knowledge graph (TKG) of German music festival lineups, mined from setlist.fm and MusicBrainz, spanning 1971–2025 with 380 festivals and >90K 'performs at festival' quadruples. The full `concert` variant contains 278,936 nodes, 23 relations, and ~5.07M training quadruples (Table 2); `concertp` keeps only the performance relation (25,399 nodes). The authors make three claims: (1) the dataset is a realistic, interpretable alternative to event-news benchmarks whose queries are hard to interpret (Sec. 3 enumerates five defects, e.g., >70% of WIKI/YAGO test facts already true at the previous timestep per Gastinger et al. 2024b); (2) lineup forecasting needs new metrics because one lineup can hold hundreds of correct answers — hence R-Precision and normalized 10-precision (Sec. 5.1); (3) specialized TKG methods clearly beat zero-shot LLMs. All three are supported by the presented evidence, with caveats.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the festival lineup TKG and where can I download it?
It is a temporal knowledge graph of 380 German music festivals from 1971 to 2025, built from setlist.fm and MusicBrainz, containing more than 90K 'performs at festival' quadruples plus concerts, venues, releases, genres, and artist metadata. The final datasets are hosted at madata.bib.uni-mannheim.de/822/ and the construction/evaluation code at github.com/JuliaGast/festival-lineup-tkg.
Can LLMs predict music festival lineups better than specialized forecasting models?
Not in this study. Four zero-shot LLMs (Llama 3.1 8B, quantized Llama 3.3 70B, Mixtral 8x7B, GPT-4o-mini) were far behind TKG methods: the best LLM reached 1.7% normalized top-10 precision for artist prediction versus 11.9% for CognTKE, and the LLMs showed strong popularity bias (e.g., Billie Eilish predicted in the top-3 for 47.1% of queries despite not playing German festivals in 2025) and duplicated entries.
Why did the authors propose R-Precision instead of MRR for this task?
A single festival lineup can include hundreds of correct artists, so the standard filtered MRR produces misleadingly tiny scores even for rankings where half of the top predictions are correct (their worked example: a 50%-correct ranking scores 5.2% MRR). R-Precision and normalized top-10 precision score the ranked list against the number of true answers, matching how a user would consume a shortlist.
Why are embedding-based TKG models missing from the full concert dataset results?
RE-GCN, CEN, and CognTKE ran out of GPU memory (on a 48GB RTX 6000 Ada) on the full 5M-quadruple graph even after reducing embedding dimension and batch size, so they were evaluated only on the reduced concertp variant that keeps just the 'performs at festival' relation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–8 appear in the extraction only as axis-label fragments; visual claims (Fig. 6 activity trends, Fig. 7 gender clustering example, Fig. 8 cutoff analysis) could not be independently verified.
- Table 3/6/7 values were parsed from dense extracted text; spot-checked against the running narrative, but a digit-level misparse is possible.
- The apparent DRec inconsistency between Table 2 (2.9% for concert) and Table 6 (26.9% for concert) may partly stem from extraction, though both captions read identically.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.