Neo-Classic: A Benchmark for Evaluating Linguistic-Aesthetic Reasoning in Classical Chinese Poetry
Han Zhang, Han Zhang, Zihan Gu, Zhiyuan Wang, Tianyi Ma, Jiacheng Lu, Xinyan Zhang, Yuhao Wei, et al.
cs.CL
Quick verdict
A benchmark of 1,406 contemporary metrical poems showing a 20–50% memorization gap and near-zero reordering in LLMs; credible design but weak statistics.
Summary
The paper introduces NEO-CLASSIC, a benchmark for separating genuine linguistic-aesthetic reasoning in Classical Chinese Poetry (CCP) from memorization of canonical pre-training texts. The authors curate 1,406 strictly metrical poems by 30 contemporary, pseudonymized poets (628 Shi, 778 Ci), verified by a rule-based engine supporting tonal-rescue variants and AST-based rhyme rules, alongside historical control corpora (Tang/Song). Five task groups with 41 variations probe authorship attribution, Ci-template identification, cloze, couplet matching, and poem reordering. Across nine models, accuracy drops 20–50% from historical to contemporary texts (e.g., DeepSeek-V3.2 authorship: 84.0% Tang vs 34.2% contemporary standard). In Lyu reordering, standard exact-match accuracy is near zero (0–3%; at most 13% with CoT); expert-prompt scaffolding lifts Gemini-3-Pro to 36% versus 48% for ten human experts. Error decomposition shows models obeying formal constraints poorly but outperforming experts at semantic ordering once constraints hold, while experts show the reverse profile. Limitations include absence of statistical significance testing on 100-item task samples, prose-table inconsistencies (e.g., §4.2.2 Gemini CiTone figures vs Table 6), a temperature mismatch between §4 and Appendix F, and a small expert sample (10 participants × 10 items). Overall, a thoughtful contamination-aware benchmark whose headline gaps are plausible but statistically under-supported.
Review complete
Assessment at a glance
0
risk flags identified
● high 0
● medium 0
● low 0
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SOTA LLMs lose roughly 20–50% accuracy when tasks shift from historical to formally compliant contemporary poetry, evidence of reliance on memorized pre-training texts (the 'Memorization Gap').
- Sentence reordering of regulated verse (Lyu) exposes a global-planning failure: standard exact-match accuracy is 0–3%, rising to at most 36% (Gemini-3-Pro) under expert-level guidance versus 48% for human experts.
- Models prefer semantically plausible synonyms over tonally correct candidates in cloze (Tone Mixed: human experts 87.0% vs Qwen3-Max 66%, DeepSeek-V3.2 61.8%), indicating weak phonological-constraint sensitivity.
- Error decomposition shows complementary profiles: the best model reaches only 59% rhyme validation and 55% parallelism identification versus 86%/80% for experts, but once formal constraints hold, models outperform experts at semantic discourse ordering (expert semantic-only failure rate 26%).
Strengths
Contamination-resistant constructionist corpus
1,406 authorized works by 30 pseudonymized contemporary poets, all passing a custom strict metrical check; leakage tests against chinese-poetry show 7-gram hits of only 0.09% (9 occurrences) and zero 14-gram overlap (§3.4). Thematic tagging (Table 1, 79.4% classical themes) anticipates and partially defuses the vocabulary domain-shift objection.
Serious prosody verification infrastructure
Appendix E describes variant-block tonal matching covering Ao-Jiu rescues and an AST-based rhyme verifier handling thousands of Cipai templates; the same engine metric-checks the Tang control corpus, keeping both sides of the historical/contemporary comparison structurally comparable.
Dual human baselines with error decomposition
Well-motivated global-planning probe
Weaknesses
No uncertainty quantification
Internal numeric inconsistencies
Decoding-configuration contradiction
Memorization-gap interpretation partly confounded
Mid-analysis model substitution
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
NEO-CLASSIC (arXiv 2609.19154) — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 0 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Neo-Classic benchmark and how does it avoid pretraining-data contamination?
It pairs a 'constructionist' corpus of 1,406 strictly metrical poems (628 Shi, 778 Ci) written after 2010 by 30 pseudonymized, authorized contemporary poets — verified by a custom rule engine and 7/14-gram leakage checks — with historical Tang/Song control corpora. Because the contemporary poems cannot exist in pre-training data, performance on them isolates rule-based reasoning from memorized retrieval.
What is the 'Memorization Gap' reported in the paper?
It is the accuracy drop LLMs suffer when the same task moves from historical poems (widely present in training data) to formally equivalent contemporary ones. The paper reports drops of roughly 20–50% across authorship attribution, Ci-template identification, cloze, and couplet matching, while human experts stay stable.
How well do LLMs reorder lines of regulated verse (Lyu)?
Very poorly without help: standard exact-match accuracy on 8-line Lyu reordering is 0–3% for frontier models (at most 13% with plain chain-of-thought). With expert-level guidance prompts, Gemini-3-Pro reaches 36%, still below the 48% achieved by ten human poetry experts.
Is the Neo-Classic dataset and code publicly available?
The authors state that code and data are uploaded as supplementary materials, with academic-use authorization from the 30 poets and pseudonymized author identities; local-model evaluation used vLLM on an A100 with a fixed seed, though API models may not honor seeds.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extracted text is truncated: §4.4 (pre-training composition study) and parts of Table 5/Appendix D are cut, and Figures 1–3 are garbled text; headline numbers were cross-checked against the surviving Table 6 grid.
- The flagged prose-vs-table discrepancies (§4.2.2 Gemini CiTone; Table 4 vs Table 6 for DeepSeek Lyu CoT-expert) could partly stem from extraction noise; they are raised as questions rather than asserted errors.
- Exact conditions under which human experts took each task (prompting, time limits) are not fully specified in the visible text.
- arXiv-listed model names (e.g., 'Gemini-3-Pro', 'DeepSeek-V3.2') could not be verified against external sources; review accepts the paper's nomenclature as given.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.