Chinese Competitive Debating Dataset and Benchmark
Zongrui Yang, Haoyuan Li, Zhongsheng Wang, Zhirui Zeng, Pengqian Han, Yi Zhou, Yi Zhou, Yuting Wang, et al.
cs.CL · cs.AI
Quick verdict
Novel expert-annotated Chinese debate benchmark (148 matches, 3 tasks); design and label-reliability analysis are solid, but results lack significance tests.
Summary
This paper introduces a dataset and zero-shot benchmark for competitive Chinese-language debate. The authors self-organized 182 matches (150 4v4, 32 2v2), each adjudicated in situ by three professional judges from a 120-judge pool under a shared rubric; after excluding 34 matches with incomplete scoring, the release contains 148 matches, 2,698 stages, and 20,542 exchange units (about three million Chinese characters), with manually verified transcripts and segmentation, preserved stage scores, votes, best-debater ballots, adjudication rationales, and 20 appeal re-adjudications. Three tasks are defined: winner-tendency prediction, stage-score prediction, and best-debater prediction. Zero-shot, nine LLMs reach at best 66.2% winner accuracy (vs. a 50.7% always-affirmative baseline), a Pearson correlation of 0.250 with mean human stage scores, and 56.8% best-debater accuracy. Strengths include transparent label-reliability analysis (70.3% pairwise winner agreement, ICC(1,3) = 0.523) and clearly flagged non-comparable human references. The main limitations are the absence of confidence intervals or significance tests behind the headline numbers, a modest 148-match scale, LLM-assisted transcript correction, and no download URL in the text.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- This is the first debate dataset combining fine-grained transcripts with stage-level scores assigned by professional judges during real competitions under a unified rubric (Table 1).
- Zero-shot LLMs reach at most 66.2% winner accuracy, r = 0.250 with mean human stage scores, and 56.8% best-debater accuracy, far below judge-level agreement on the outcome and stage tasks.
- Match outcome, stage quality, and speaker contribution are distinct adjudication capabilities that require separate metrics (§4).
- Judge labels have only moderate reliability (~70.3% pairwise winner agreement, ICC(1,3) = 0.523), so model agreement should be benchmarked against human-judge references rather than an assumed ground truth.
Strengths
Unique in-situ professional supervision
Table 1 shows this is the only corpus whose stage-level scores were assigned during real adjudication by credentialed judges under one shared rubric (Table 5), with judge texts and votes preserved; ORCHID/CEDAR/DEFINED lack in-situ stage scoring.
Multi-granularity, verified text
148 matches / 2,698 stages / 20,542 exchanges with sentence-level line traceability; full manual review adjusted 71 segmentations and fixed 304 ASR errors (§2.2.3); appeal re-adjudications kept separately linked (§2.2.4).
Transparent label reliability
Disciplined evaluation protocol
Metric design that self-corrects
Weaknesses
No inferential statistics
LLM-in-the-loop transcripts
Scale/diversity unreported
Asset availability unstated in text
Thin exclusion and subset reporting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a dataset-and-benchmark paper. Its claims: (1) the first resource combining fine-grained competitive Chinese-debate transcripts with stage-level scores assigned in situ by professional judges under a shared rubric (Table 1 positions it against ORCHID, Chen et al. 2026b, DEFINED, CEDAR, Conch); (2) a three-task benchmark — winner-tendency prediction, stage-score prediction, best-debater prediction (§2.3); (3) zero-shot evidence that current LLMs adjudicate debates poorly relative to professional judges (66.2% winner accuracy, Pearson r = 0.250 on stage scores, 56.8% best-debater accuracy); (4) an explicit framing that judge labels operationalize persuasion, not ground truth (§2.5). Claims are mostly well scoped; the novelty claim depends on the accuracy of the Table 1 characterization of prior corpora, which this review could not independently verify.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this dataset contain?
148 verified transcripts of competitive Chinese-language debate matches (out of 182 recorded), split into 2,698 scoring stages and 20,542 exchange units (~3 million characters). Each match carries independent stage scores, votes, best-debater ballots, and written rationales from three professional judges under a shared rubric, plus a second adjudication for the 20 appealed matches.
How well do LLMs judge debates according to this benchmark?
Zero-shot, the best of nine tested models reached 66.2% winner-prediction accuracy (trivial baseline: 50.7%), a 0.250 Pearson correlation with mean human stage scores, and 56.8% best-debater accuracy. Best-debater prediction is close to a conservative human reference (57.6%), while outcome and stage scoring lag well behind.
Are the judge scores ground truth?
No. The authors show only moderate agreement among the three judges (about 70.3% pairwise winner agreement; ICC(1,3)=0.523 for stage scores) and explicitly treat votes as measuring which side persuaded more judges, not objective correctness.
Is the dataset public?
The paper states it was publicly released in September 2026 and marks it as public/anonymized, arguing that pre-release models cannot be contaminated; however, no download URL is visible in the extracted text, so availability should be verified directly.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 and Table 5's rubric details are only partially recoverable from the extracted text; visual content was not assessed.
- Table extraction order was garbled in places (e.g., Table 2 embedded in §3.2 prose), but all metric values were legible.
- No dataset URL or license text appears in the extraction; availability claims rest on Table 1 and §2.4.
- Table 3/7 lacks a gpt-5.6-sol row while Table 2 has one — possibly an extraction artifact rather than a paper omission.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.