TRILOGUE: A Trilingual Spoken Dialogue Fact-Checking Benchmark with Evidence and Paired Audio
Chaewan Chun, Meruyert Aristombayeva, Jiyoung Choi, Mahjabin Nahar, Delvin Ce Zhang, Dongwon Lee
cs.CL
Quick verdict
Claims the first large trilingual spoken-dialogue fact-checking benchmark (12K dialogues, 390h); thorough validation, but LLM-generated and LLM-labeled.
Summary
TRILOGUE is a trilingual (English, Russian, Kazakh) benchmark for spoken dialogue fact-checking built by generating podcast-style dialogues from news articles and controlled LLM hallucinated rewrites. It contains 11,957 dialogues, 187,544 turns, and 390.3 hours of paired audio, combining synthetic TTS with 4,998 human-recorded Russian and Kazakh dialogues, plus Whisper large-v3 ASR transcripts and word-level alignments. It supports three tasks: turn-level check-worthiness detection, source-conditioned sentence evidence retrieval, and three-way claim verification (True/False/NEI) under claim-only, gold-evidence, and retrieved-evidence inputs, in monolingual, pooled, and leave-one-language-out regimes. Baselines (TF-IDF+LR, XLM-R, mDeBERTa; BM25, E5) show check-worthiness is most stable (mDeBERTa ~86-87 macro-F1), claim-only verification hardest, and retrieved evidence nearly closes the gap to gold evidence. Kazakh is hardest throughout (Whisper WER 68.5%; Kazakh TTS fails human audio-quality thresholds). Human validation covers 600 dialogues (~5%) with 92.8-94.4% factual-label agreement; inter-annotator kappa is 0.317 for English and 0.67-0.71 elsewhere; NEI labels are unstable. Main limitation: dialogues, perturbations, and labels are all LLM-produced, so the benchmark measures verification of controlled, model-authored misinformation rather than naturally occurring claims.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- TRILOGUE is, to the authors' knowledge, the first large-scale multilingual spoken dialogue fact-checking benchmark combining paired audio, ASR transcripts, turn-level labels, and sentence-level evidence (11,957 dialogues, 187,544 turns, 390.3 hours across EN, RU, KZ).
- Retrieved source evidence substantially narrows the gap to gold-evidence verification (mDeBERTa EN macro-F1: 89.2 with BM25-retrieved vs 89.3 with gold vs 49.2 claim-only).
- ASR degradation costs are strongly language-dependent: negligible for English, roughly 12 F1False/Recall@5 points for Kazakh evidence-based tasks, with Kazakh Whisper WER at 68.5%.
- Cross-lingual transfer and Kazakh remain the main bottlenecks, driven by lower-resource speech tools rather than human-spoken audio quality alone.
- Dialogue context improves false-claim detection by about 5-9 F1False points over isolated claim turns (Table 9).
Strengths
Novel, large-scale resource filling a documented gap
First benchmark (per Sections 1-2) combining paired audio, ASR transcripts, turn-level labels, and sentence evidence across EN/RU/KZ: 11,957 dialogues, 187,544 turns, 390.3 hours, versus prior MAD (English-only, small) and podcast resources with only partial fact-checking annotations.
Unusually transparent multi-layer validation
Human audit of 600 dialogues (~240 person-hours; Tables 4-5), automatic DNSMOS/WER/CER metrics (Table 3), and a full-corpus o4-mini judge audit (93.7% check-worthiness and 92.1% factual agreement); failures are reported rather than hidden (KZ TTS 0.0% overall-good; English kappa 0.317).
Leakage-aware splits and rich diagnostics
Mechanistic Kazakh ASR error analysis
Weaknesses
Fully model-authored data and labels
NEI class is marginal and demonstrably noisy
Official Kazakh split contains audio that failed human quality gates
Narrow baseline and task scope
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
TRILOGUE (arXiv:2609.04452) is a benchmark paper for spoken dialogue fact-checking in English, Russian, and Kazakh. The central claim (Section 1) is that it is the first large-scale multilingual spoken dialogue fact-checking benchmark combining paired audio, ASR transcripts, turn-level check-worthiness and factual labels, and sentence-level source evidence. The corpus (Table 1) holds 11,957 dialogues, 187,544 turns, and 390.3 audio hours; 112,160 check-worthy turns carry True/False/NEI labels (57,554/45,615/8,991). Three tasks are defined: check-worthiness detection, source-conditioned sentence evidence retrieval, and three-way claim verification under claim-only, gold-evidence, and retrieved-evidence inputs, evaluated monolingually, pooled, and leave-one-language-out.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the TRILOGUE dataset?
TRILOGUE is a trilingual (English, Russian, Kazakh) benchmark for fact-checking claims in podcast-style spoken dialogues. It contains about 11,957 dialogues, 187,544 turns, and 390.3 hours of paired audio with ASR transcripts, word-level timestamps, turn-level check-worthiness and True/False/NEI factuality labels, and sentence-level evidence from the source news articles.
Where can I download TRILOGUE and under what terms?
The dataset is released on Hugging Face (chaewanC/TRILOGUE) for research use. Terms require preserving the factual labels and source provenance when presenting content, and prohibit voice cloning, impersonation, speaker identification, and biometric profiling of the human-recorded audio.
Is the data real conversations with real misinformation?
No. Dialogues are LLM-generated (Gemini 2.5 Flash / DeepSeek-V3.1) from news articles and controlled hallucinated rewrites. Audio is synthetic TTS plus about 5,000 human-read recordings in Russian and Kazakh. Labels come from Gemini 2.5 Pro, validated by humans on a 600-dialogue subset (about 5% of the corpus).
What do the baseline results show?
Check-worthiness detection is the most stable task (about 86-87 macro-F1 for mDeBERTa in English). Claim verification without evidence is hard (about 49 macro-F1), but adding gold or BM25-retrieved source evidence raises it to roughly 89. Kazakh is the hardest language (Whisper WER about 68.5%, and a roughly 12-point penalty from ASR noise on retrieval and verification).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Tables 6, 7, and 16 are extracted as flattened numeric grids; cell-to-row/column pairing was reconstructed from the linear text and individual cells could be misread.
- Figure 1 and Table 10 are extracted as flattened label text, so pipeline-stage and placeholder mappings were reconstructed from surrounding captions; content appears complete, with appendices A-F present and no truncation at the end of the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.