Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation
Koutian Wu, Junjie Zhou, Ergan Shang, Jiayu Wang, Pengqian Han, Junkai Wang, Wanghan Xu, Songyuanyi Lu, et al.
cs.AI · cs.IR
Quick verdict
A living search engine over 1,283 AI benchmark records with a rare full-catalog census and strong provenance, yet its own retrieval quality goes unevaluated.
Summary
The paper presents Benchmark Radar, a living database and search engine for AI benchmarks. Daily discovery draws on 37 sources (13 direct connectors and 24 first-party feeds) and feeds a searchable catalog of 1,283 source records from four providers — LLM Stats (687), OpenCompass Hub (461), Artificial Analysis (25), and model reports (110) — carrying 12,916 numeric score observations on 790 records. Retrieval uses BM25F lexical search, exposed through a web dashboard, an offline CLI with SHA-256-verified archives, and an agent skill. A full-catalog census audits every record without filters: 493 records lack scores; only 82 of 790 scored records satisfy a verifiable percentage-scale rule; 615 records carry release dates; and 1,208 distinct cited documents support 1,278 records. The taxonomy assigns 1,279 records to 11 domains and 63 sub-domains using publishers' own labels. A worked example shows a coding agent using the CLI for prior-art search on credit assignment in agentic training. Key limitations: retrieval accuracy, suitability judgment, and time savings are never quantitatively evaluated; the worked example lacks a controlled baseline; two relevant earlier studies were missed by the search and surfaced by a coauthor; classification is unvalidated; and four of 46 discovery snapshots are simulated backfills. Reproducibility support is strong: pinned commit, input hashes, auditable census scripts, and a DOI.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Benchmark Radar unifies daily benchmark discovery across 37 sources with a searchable 1,283-record catalog (12,916 score observations on 790 records), sharing stable benchmark IDs across web dashboard, CLI, exports, and an agent skill.
- The full-catalog census shows evaluation evidence is rarely directly comparable: only 82 of 790 scored records have verifiable percentage scales, and neither model-announcement nor document-publication dates establish evaluation dates.
- Preserving unscored records with artifact links (464 of 493) and per-source identities keeps candidate benchmarks inspectable, unlike leaderboard-only resources that drop them.
- A coding agent using the offline CLI plus the public skill can assemble a prior-art comparison table for designing a new agentic-training evaluation, as demonstrated in Section 5's worked example.
Strengths
Full-population census discipline
Section 3.5 and Appendix A audit all 1,283 records with no date, score, or interface filters, and strictly separate units: 12,916 numeric observations, scored models counted per record by source model ID, and 1,208 distinct cited documents. Table 5's GPQA Diamond contrast (586 models with 1 cited page on Artificial Analysis vs 19 models and 27 documents on the model-reports record) shows why the unit separation matters.
Honest measurement-eligibility accounting
Table 6 shows only 82 of 790 scored records satisfy the declared-percentage-scale rule; percentage headroom is computed nowhere else, and B.2 notes that neither model-announcement dates (12,594 scores) nor document-publication dates (322) establish evaluation dates. The paper resists the cross-benchmark comparability claims most leaderboard resources make casually.
Strong reproducibility infrastructure
Unscored records and provenance are preserved
Careful faceted taxonomy
Weaknesses
The retrieval system is never evaluated
Abstract promises saturation analysis the body does not deliver
Severe source concentration and a metadata-only bulk
Discovery provenance and source-health caveats
Unvalidated classification and unexplained duplicate counts
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Benchmark Radar (arXiv 2609.11115v3) is a system/resource paper for a living database and search engine covering AI benchmarks: LLM evaluation, agentic and tool-use, coding, reasoning, safety, and domain-specific suites. The v0.11.0 release claims a searchable catalog of 1,283 source records from four providers (Table 1: LLM Stats 687, OpenCompass Hub 461, Artificial Analysis 25, model reports 110), 12,916 numeric score observations on 790 records, and daily discovery across 37 sources (13 direct connectors, 24 first-party feeds). The abstract also promises examination of benchmark saturation, adoption trends, and the limits of score comparisons — a promise only partially kept. Declared contributions: the living system, shared web/offline access to evidence, and a reproducible full-catalog census.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Benchmark Radar?
A free, public database and search engine for AI benchmarks. It combines daily discovery of benchmark papers, repositories, datasets, and releases with a searchable catalog of 1,283 benchmark records from four sources and 12,916 recorded model-score observations, accessible via a web dashboard and an offline command-line client.
How many benchmarks and scores does Benchmark Radar contain?
Version 0.11.0 holds 1,283 source records (LLM Stats 687, OpenCompass Hub 461, Artificial Analysis 25, model reports 110); 790 records carry 12,916 numeric score observations and 493 have no scores. Records are per-source, so one real benchmark can appear under more than one source.
Is the code and data available?
Yes. Source code is on GitHub (ktwu01/benchmark-radar, MIT license), checksummed release data is on GitHub Releases, the dashboard is at benchmark-radar.org with downloadable JSON and RSS, and a Zenodo DOI covers citation. The paper's census can be re-run with the provided audit scripts and their check modes.
Did the authors measure how good the search is?
No. The paper states that retrieval precision, task suitability, and time saved remain to be evaluated. The worked example is illustrative with no controlled baseline, and the authors note that their own related-work search missed two relevant earlier studies, which a coauthor found.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3, 4, 8, and 10-13 are screenshots of the live dashboard and CLI; their interactive content cannot be verified from the extracted text (captions were extracted and are consistent with the prose).
- No truncation detected: the provided text runs through Appendix F's licensing terms, a natural ending point for the paper.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.