MultiGhostBench: A Multilingual Benchmark for Long-Form LLM-Generated Text Attribution under Distribution Shifts
Matteo Greco, Anudeex Shetty, Andrea Tagarelli, Jey Han Lau
cs.CL · cs.AI · cs.CY · cs.DL · cs.IR
Quick verdict
首次推出的多语言长文本 LLM 归属基准（928 本书，6 种语言）；结果可信，但微小的测试单元和缺乏显著性检验限制了研究结论的说服力。
Summary
MultiGhostBench 是一个用于 LLM 作者归属（AA）的多语言长文本基准：由五个近期的 LLM（Gemini-pro、Gemini-flash、DeepSeek-v3.2、Qwen3-235B、gpt-oss-120b）生成 928 本书（平均约 59,600 字），涵盖六种语言（EN, DE, IT, ES, RU, ZH）和三种文字系统，通过扩展英文 GhostWriteBench 流水线构建，翻译后的提示词已由母语人士验证。它支持语言内评估（ID、OOD-Domain、OOD-Author）以及新的 OOD-Language 设置。在低/高资源机制下，通过在开发集上校准的置信度阈值拒绝评估了来自三个家族的九个检测器。结果：没有任何方法能通吃；更多数据有帮助；基于 Transformer 的 XLM-RoBERTa 和 DETECTIVE 可以跨语言迁移（例如 IT→ES 0.981 macro-F1），而 N-GRAM、TRACE 指纹和基于指标的方法则会出现跨语言崩溃（通常接近于零）；OOD-Author 通常比 OOD-Domain 导致更大幅度的下降（例如 XLM-RoBERTa 意大利语高资源 ID 0.956 → OOD-Author 0.203）。优势包括内部一致的表格、透明的成本（约 972 美元）以及公开的数据/代码。主要局限性：任何地方都没有显著性检验或方差估计；单格测试集可能小至 1-4 本书（表 5）；人类质量评估仅涵盖 Gemini-pro 小说，每本书有两名标注员，且标注员间一致性极低（0.214）；闭源生成器使得精确再生成为不可能。
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- MultiGhostBench is the first benchmark designed to jointly evaluate multilingual, long-form (~59K-word) LLM authorship attribution under domain, author, and language shifts.
- No single attribution method performs consistently best across languages, resource regimes, and shift settings.
- Performance generally degrades under distribution shift, with unseen-author (OOD-Author) drops typically exceeding unseen-domain drops.
- Transformer-based detectors (XLM-RoBERTa, DETECTIVE) retain generator-related signal across languages, while N-GRAM, TRACE fingerprints, and token-statistic methods are largely language-bound.
Strengths
First joint multilingual, long-form AA benchmark with three shift axes
Table 1 shows prior resources cover at most short texts (<500 words), one language, or one shift axis; MultiGhostBench uniquely combines 59.6K-word books, six languages, three scripts, and domain+author+language shifts (Sections 1 and 3).
Internal consistency of reported results
Narrative statistics reproduce from the appendix tables on spot-checking: Spanish best low-resource source 0.866, Italian best high-resource source 0.894, Chinese hardest target 0.508/0.582 (Section 6.2, Table 17), and N-GRAM best OOD-Author detector in five of six languages (Table 12) — careful bookkeeping across ~200 configurations.
Broad and fairly adapted method comparison
Unusual construction transparency
Weaknesses
No statistical uncertainty quantification anywhere
Tiny per-cell test sets
Weak human quality evaluation
Single evaluator for the fingerprint method
Fragile open-set threshold calibration
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MultiGhostBench (arXiv:2609.02379) is a benchmark paper for LLM authorship attribution (AA): given a long text, identify which of five generators — gemini-pro, gemini-flash, deepseek-v3.2, qwen3-235b, gpt-oss-120b — wrote it. The corpus contains 928 books (avg 59.6K words; Table 2) in six languages and three scripts, built by translating the GhostWriteBench pipeline of Shetty et al. (2026). Its claimed novelty (Table 1) is being the first resource to jointly cover long-form texts, multilinguality, and three shift axes: OOD-Domain (unseen Gutenberg genres), OOD-Author (leave-one-generator-out, open-set with rejection), and the new OOD-Language. Four headline claims: (1) first joint benchmark of this kind; (2) no universal winner among nine detectors; (3) performance generally degrades under shift, with OOD-Author drops exceeding OOD-Domain; (4) Transformer detectors transfer cross-lingually while statistical and fingerprint methods do not. The text supports claims 2-4; claim 1 is plausible but rests entirely on the Table 1 comparison.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MultiGhostBench?
A research benchmark of 928 full-length 'books' (averaging ~59,600 words) written by five recent LLMs — Gemini-pro, Gemini-flash, DeepSeek-v3.2, Qwen3-235B, and gpt-oss-120b — in six languages (English, German, Italian, Spanish, Russian, Chinese) and three scripts. It tests whether authorship-attribution methods can identify which LLM wrote a text, including when the test text differs in genre, generator, or language from the training data. Data and code are on GitHub (GrecoMT/MultiGhostBench).
Which detection methods work best for attributing LLM-generated books?
In the paper's evaluation, no single method wins everywhere. Fine-tuned XLM-RoBERTa and DETECTIVE are strongest overall and the only families that transfer well across languages (e.g., an Italian-trained XLM-RoBERTa reaches 0.981 macro-F1 on Spanish books). A character/word N-GRAM classifier is best under unseen-author conditions in high-resource settings (up to 0.98-1.00 macro-F1). Statistical metrics (rank, entropy, GLTR) and the TRACE fingerprint method work within a language in some configurations but collapse to near zero when the test language differs from the training language.
How was the MultiGhostBench dataset created and how much did it cost?
The authors extended an English book-generation pipeline: each book starts from genre and time-period constraints drawn from Project Gutenberg, then is written section by section, conditioned on an outline, the previous section, and a running plot summary. Prompt templates were translated into each language and checked by native speakers. Texts were cleaned (trimming 1,500 tokens from each end) and language-verified with GlotLID. Total generation cost via OpenRouter was about $972, dominated by Gemini-pro ($760).
What are the main limitations of MultiGhostBench?
Only five generators and six languages (no low-resource languages), human quality checks limited to Gemini-pro fiction with very low annotator agreement (0.214), no statistical significance testing behind the method comparisons, some evaluation cells containing as few as 1-4 books, and closed-source generators (Gemini), meaning the corpus cannot be regenerated exactly. Adversarial settings and mixed human-AI authorship are also out of scope.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are graphics whose numeric content is available only via captions and the accompanying appendix tables; heat-map values cited here were cross-checked against Tables 11-17 rather than read off the figures.
- Appendix tables were flattened by PDF extraction, so individual cell alignments (e.g., Tables 15-18) were verified against narrative averages rather than assumed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.