What Do We Expect from LLMs? Mapping the Design of LLM Benchmarks
Chao Wang (Independent Researcher)
cs.AI · cs.CL
Quick verdict
Systematic map of 14,767 arXiv LLM-benchmark papers (2022–26): transparent, well-caveated descriptive study, but seven-field coding accuracy is never measured.
Summary
This systematic mapping study analyzes 14,767 arXiv papers submitted between January 2022 and August 2026 that introduce or substantially update reusable LLM evaluation resources, asking what researchers expect LLMs to do rather than how well models perform. From a metadata frame of 1,153,355 submissions, a local classifier (M0) routes the top ~15% of each month to GLM-5.3-Flash abstract screening, after which a single full-text request re-assesses eligibility and assigns seven multi-label design fields covering target systems, domains, setups, modalities, material origins, scoring sources, and task languages. Key results: agent/tool-system targets rise from 9.0% (Jan–Aug 2024) to 28.3% (2026), LLM-based scoring from 25.8% to 40.3%, and interactive setups from 9.5% to 28.2%, while fixed task collections remain near 95%. A Kitagawa decomposition attributes 13.95 of a 14.51-point LLM-scoring increase to within-group change. Model-generated materials plateau around 50% after earlier growth. A stratified 100-record eligibility audit confirmed 99 inclusions, and sensitivity bounds keep the four largest increases positive under an assumed 5% label error. Principal limitations: the seven coding fields come from one LLM with no representative accuracy estimate; M0's stricter training definition may cause differential recall; paper records do not equal independent benchmark families; all analyses are declared exploratory and post hoc.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Agent/tool-system benchmark proposals rose from 9.0% of the January–August 2024 cohort to 28.3% in 2026, with interactive setups rising 9.5%→28.2% while fixed collections stayed near 95%.
- LLM-based scoring grew within both agent (26.3%→44.0%) and non-agent (25.7%→38.8%) groups; 13.95 of 14.51 percentage points of the aggregate increase is within-group change, not composition.
- Model-generated evaluation materials expanded early (15.6% in 2022 to ~50% by 2024) then plateaued, showing model participation grows along distinct dimensions rather than toward uniform automation.
- Generated material and LLM scoring co-occur in 22.2% of records, but co-occurrence declines relative to the independence baseline (lift 1.32→1.22) and paper-level labels cannot establish self-evaluation loops.
- As AI constructs, takes, and judges tests, the evaluation ecosystem risks reproducing participating models' shared preferences — a conditional risk this mapping motivates but does not itself establish.
Strengths
Complete pipeline accounting
Fig. 1 and §B.2 track all 1,153,355 metadata records through every stage, including 25,079 undetermined abstract decisions and 52 unavailable sources, rather than reporting only the final 14,767 inclusions.
Composition-vs-within-group analysis
The Kitagawa symmetric decomposition (Eq. B1, §3.4) shows 13.95 of 14.51 points of LLM-scoring growth is within-group change, preempting the 'just more agent papers' confound; replicated under three alternative windows in §B.5 (e.g., 14.64/13.84 for same-year versions).
Audit extends upstream of inclusion
Honest epistemic labeling
Negative and non-monotonic results retained
Weaknesses
Seven-field coding accuracy unmeasured
Single-model screening and coding
Recall bias from stale M0 training target
Version-read labels vs first-submission cohorts
No benchmark-family resolution
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Structured Preliminary Review: 'What Do We Expect from LLMs? Mapping the Design of LLM Benchmarks'
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How many LLM benchmark papers exist?
This study identifies 14,767 arXiv papers from January 2022 to August 2026 that introduce or substantially update a reusable LLM evaluation resource, out of 1,153,355 submissions scanned. These are paper records, not independent benchmark families, since one paper can contribute several resources.
Are LLM benchmarks becoming more agentic?
Yes, sharply. The share of benchmark papers targeting agent or tool-using systems rose from 9.0% of the January–August 2024 cohort to 13.0% in 2025 and 28.3% in 2026, and interactive evaluation setups grew from 9.5% to 28.2%, while fixed task collections stayed around 95%.
How common is LLM-as-a-judge scoring in benchmarks?
LLM-based scoring appeared in 25.8% of benchmark papers in January–August 2024, 35.5% in 2025, and 40.3% in 2026. It rises within both agent papers (26.3% to 44.0%) and non-agent papers (25.7% to 38.8%), so growth is not explained by more agent papers alone.
Is this benchmark-mapping study reproducible?
Largely yes: the dataset, frozen prompts and schemas, and analysis code are public (github.com/xxcg322/LLM-Bench-Map), and tables and figures can be rebuilt from archived model outputs without new API calls. The coding stage itself relied on the GLM-5.3-Flash API, so re-running it would not deterministically reproduce labels.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–5 are present only as captions in the extracted text; visual chart data could not be analyzed.
- The public repository (LLM-Bench-Map) was not inspected (offline review); reproducibility claims rest on the paper's own description.
- Coding-label quality cannot be independently verified from the text; this review relies on the paper's reported audits and appendices.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.