Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models
Kaifeng Tan, Yudong Li, Linlin Shen
cs.CL · cs.AI
Quick verdict
Dynamic compression benchmark on fresh text for 80 base LLMs; scaling, long-context and MMLU findings are solid, though correlations lack size controls.
Summary
The paper introduces Uncheatable Eval, a dynamic benchmark that evaluates base language models by how well they losslessly compress newly published text, reducing benchmark-contamination risk. Each model's next-token negative log-likelihoods define an ideal arithmetic-coding length, normalized by UTF-8 byte size into a compression rate (CR; lower is better), requiring no instructions or reference answers. A curation pipeline (quality filtering, MinHash deduplication, NFC normalization, truncation to 3,584 tokens under eight tokenizers) builds 14 categories — fiction, news, encyclopedias, scientific papers, code — with 500 samples each from a July 2026 collection. Across 80 models, Gemma-4-31B leads Overall CR at 6.00; Qwen excels on code and scientific text, RWKV on general text. Three analyses: (1) CR follows CR(P)=aP^b+c (all-category R²=0.915, exponent −0.290; code −0.317, fiction −0.146; Pareto frontier R²=0.992); (2) on 32 KiB scientific documents, CR improves with position for Qwen3-8B (slope −0.426), Falcon-H1-7B (−0.454) and RWKV7-G1J-7.2B (−0.314), with the recurrent model's short-context advantage decaying at long positions; (3) technical-text CR correlates strongly with zero-shot MMLU (Spearman ρ=−0.884, 95% CI [−0.942, −0.783]). Limitations: contamination cannot be fully excluded, the CR–MMLU correlation is not controlled for model size, and uncertainty estimates are reported only for that correlation.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Compression rate on regularly refreshed, newly published text provides a contamination-resistant evaluation of base LLMs without instructions or reference answers.
- Across 80 models, compression rate follows a power law in parameter count, CR(P)=aP^b+c, with all-category R²=0.915 (exponent −0.290) and steeper scaling for code (−0.317) than fiction (−0.146).
- Attention-based, hybrid, and recurrent architectures differ in how CR improves with document position on 32 KiB scientific documents, with RWKV7's short-context advantage decaying to a 0.367-point deficit by 16–32 KiB.
- Lower compression rate on technical text is strongly associated with higher zero-shot MMLU accuracy (Spearman ρ=−0.884, 95% CI [−0.942, −0.783]).
Strengths
Contamination-resistant, instruction-free protocol
Evaluating freshly collected text via ideal arithmetic-coding length (Eqs. 1–4) scores base models without task instructions or reference answers, directly addressing the memorization problem the paper motivates in §1.
Careful cross-tokenizer comparability
Every sample is truncated to 3,584 tokens under eight tokenizers so all models score identical text; the byte-wise long-context construction (Eq. 5) distributes per-token loss over UTF-8 bytes, and Appendix E residual correlations show bytes/token is largely orthogonal to CR (|ρ| ≤ 0.245 in 13 of 14 categories).
Broad, internally consistent evaluation
Informative scaling and checkpoint ablations
Bootstrap-backed headline correlation
Weaknesses
CR–MMLU correlation not controlled for model size
Uncertainty reported for exactly one analysis
Architecture comparison rests on three families with confounds
Dataset provenance underspecified
Tokenizer sensitivity survives in one category
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Uncheatable Eval (arXiv 2609.27510, Sep 2026) is a benchmark paper with an empirical payload. It proposes evaluating base LLMs by losslessly compressing freshly published text, exploiting the prediction–compression equivalence: a document's ideal arithmetic-coding length under a model equals its base-2 negative log-likelihood (Eqs. 1–2). The authors claim three results: (1) compression rate (CR) follows a power law in parameter count; (2) attention, hybrid, and recurrent architectures differ in how CR improves with context length; (3) technical-text CR strongly associates with zero-shot MMLU accuracy. The target is explicitly base models, where limited instruction-following confounds task benchmarks.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Uncheatable Eval?
A dynamic benchmark that evaluates base language models by how well they losslessly compress newly published text. A model's next-token probabilities define an ideal arithmetic-coding length, reported as a compression rate (bits needed per UTF-8 byte, lower is better), so there are no fixed test answers that could have been memorized during training.
Why use compression instead of task accuracy?
Lossless compression and next-token prediction are mathematically equivalent: a model that predicts text better needs fewer bits to encode it. This works directly on base models without prompts, instructions, or reference answers, avoiding confounds from weak instruction-following.
What are the main findings of the paper?
Across 80 models and 14 text categories: compression rate follows a power law in model size (R²=0.915), recurrent models like RWKV lose their short-context compression advantage as documents grow to 32 KiB, and better compression of technical text strongly correlates with zero-shot MMLU accuracy (Spearman ρ=−0.884).
Can models still have seen the test data?
Yes, in principle. The authors acknowledge that using recently published text reduces but does not eliminate contamination risk — publication dates and training cutoffs may be inaccurate, and the dataset must be refreshed regularly as new models may train on its public texts.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 and 5–7 appear in the extraction only as axis-label text; visual claims were assessed from the tables and in-text numbers, which cover all headline results.
- Appendix B per-model long-context curves could not be inspected (axis dumps only); the architecture analysis relies on Table 3 and §4.3 text.
- This is an arXiv preprint (2609.27510, Sep 2026), not yet peer reviewed.
- Per-category data sources are not enumerated in the text, so dataset-level reproducibility could not be fully assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.