API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces
Jennifer Wang, Joachim Baumann, Daniel E. Ho, Sanmi Koyejo
cs.AI · cs.SE
Quick verdict
Claims API benchmark scores diverge from chatbot interfaces (3.4pp accuracy gap); rigorous audit, but unverified checkpoint identity confounds attribution.
Summary
The paper audits whether benchmark scores measured through model APIs transfer to the corresponding consumer chatbot interfaces. The authors send identical prompts from nine benchmarks — six OpenLLM Leaderboard tasks distilled into Metabench (858 items), plus BBQ, AA-Omniscience, and AITA — to seven systems across ChatGPT, Claude, and Gemini, in paired API and web-interface conditions with five runs each, using fresh browser profiles, synchronized request timing, and disabled personalization and tools. API evaluations score on average 3.4 percentage points higher in accuracy (mixed-effects p=0.002) and 2.1 points higher in test–retest agreement (95% CI 1.8–2.4), with large per-cell variation (up to +28.4 pp for Claude Haiku 4.5 on AITA) and only partial rank transfer (mean Spearman rho=0.52). A GPT-only comparison shows the interface gap rivals the GPT 5.3-to-5.4 model-version difference (4.5 pp). Sweeps of exposed API controls — approximated system prompts, temperature/top-p, and reasoning budgets — shift consistency but do not close the accuracy gap. The authors attribute results cautiously: they cannot verify that interface and API identifiers serve the same checkpoint, and scoring relies on a gpt-4o-mini judge validated by human audit (98.7% agreement over 1,280 items). Main limitations: the gap may partly reflect different deployed models rather than interface-layer effects; coverage is one subscription tier per platform and standardized, near-ceiling benchmarks.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Across seven systems and nine benchmarks, API evaluations average 3.4 pp higher accuracy (p=0.002) and 2.1 pp higher test-retest agreement than the corresponding chatbot interfaces.
- The access-surface gap can rival model-generation differences: GPT 5.4 Thinking's mean interface gap (+5.0 pp) exceeds the mean absolute GPT 5.3-to-5.4 Instant API-only difference (4.5 pp, Appendix C).
- System rankings transfer only partially across surfaces (mean Spearman rho=0.52), with within-provider inversions on BBQ and WinoGrande.
- Exposed API controls — approximated system prompts, sampling parameters, and reasoning budgets — shift behavior in some cases but do not reliably close the accuracy gap.
Strengths
Unusually rigorous confound control for black-box interface auditing
Per-query barriers synchronize API and interface submissions within seconds (§3.4); trials rotate across same-tier accounts; every query runs in a fresh ephemeral Chrome profile with personalization and tools disabled. Appendix H adds targeted robustness checks: account-level routing (χ² all n.s., max SD 1.83 pp), single-turn request-level checks (Table 26), and time-of-collection incremental R² ≤ 0.11% (Table 27).
Sound and fully reported statistics
Overall gap via LPM-ME with benchmark random slopes (β1=+3.40 pp, SE=1.11, p=0.002, CI [1.22, 5.58]) cross-checked by stratified paired bootstrap (+3.19 pp); 63 per-cell tests carry Benjamini-Hochberg FDR (Tables 5, 8); 10,000-resample paired bootstrap for reliability (CI [1.8, 2.4]); variance components reported in §B.1.
Validated, condition-shared scoring pipeline
Exemplary transparency about what is and is not controlled
Novel scope and reusable infrastructure
Weaknesses
Checkpoint identity unverifiable, capping causal interpretation
Controls-insufficiency conclusion rests on a narrow grid
Metric heterogeneity inflates the headline average
Anomalous GPT HellaSwag baselines feed the flagship magnitude comparison
Residual tool confound for one provider
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a controlled audit study asking whether LLM benchmark measurements taken through model APIs generalize to the consumer chatbot interfaces built on those models — what the authors call context validity (§2.1, after Schmuckler 2001). Across 7 systems (GPT 5.3 Instant, GPT 5.4 Thinking, Claude Haiku/Sonnet/Opus 4.6, Gemini 3 Flash Fast/Thinking) and 9 benchmarks (six OpenLLM Leaderboard tasks via the 858-item Metabench distillation, plus BBQ-200, AA-Omniscience-200, and 100 AITA original-flip pairs), identical prompts were issued to both surfaces with five runs per condition. Four central claims: (1) API accuracy averages +3.4 pp over interface; (2) API test-retest agreement averages +2.1 pp; (3) rankings transfer only partially (mean Spearman rho=0.52); (4) exposed API controls do not close the gap. The claims are well supported as descriptive transfer statements; the stronger mechanism narrative (§6.2's 'middle layers') is appropriately hedged in the text but not always in the abstract.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do chatbot interfaces really score lower than APIs on LLM benchmarks?
In this Stanford audit of ChatGPT, Claude, and Gemini (7 systems, 9 benchmarks, 5 runs per condition, March-May 2026), the web interfaces averaged 3.4 percentage points lower accuracy and 2.1 points lower test-retest consistency than the corresponding model APIs, with per-benchmark gaps ranging from +0.3 to +11.9 points and several ranking inversions within the same provider.
Why do API and chatbot results differ for the same model?
The exact cause cannot be isolated from outside: chat interfaces add hidden system prompts, provider-chosen sampling and reasoning defaults, routing, tools such as web search, and possibly a different model checkpoint than the matched API identifier. The paper shows that exposed API controls do not close the gap, and the authors explicitly cannot verify that the interface serves the same checkpoint.
Can the audit be reproduced or extended?
The automation tool is open-sourced at github.com/jennjwang/interface-audit, and a dataset of 102,048 item-run pairs is referenced for release, so re-analysis is feasible. However, replicating the interface collection requires paid accounts (ChatGPT Enterprise, Claude Pro, Google AI Pro) on live platforms and may conflict with platform terms of service, which the authors flag in §8.
Does this mean benchmark leaderboards are useless?
No — the paper argues API scores remain valid for the tested API configuration but are context-bound. Rankings transferred only partially to interfaces (mean Spearman rho=0.52 across nine benchmarks), and the effect was strongest exactly where leaderboards draw fine distinctions, such as MMLU (rho=0.11) where scores cluster within a few points.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are image content and their visual layout is not present in the extracted text; all underlying quantitative values, however, appear in the appendix tables, so no numeric claim in this review depends on the figures.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.