IBIB: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier
Blake Stenstrom, Charangan Vasantharajan, Brian Sathianathan
cs.CL · cs.AI · cs.LG
Quick verdict
Claims a route-binding benchmark protocol; unusually self-critical results, but sealed corpus and vendor conflict of interest keep scores unverifiable.
Summary
The paper argues that enterprises consume served routes rather than model checkpoints, and that benchmarks scoring advertised identifiers conflate model capability with serving-stack properties. It proposes IB2, a protocol with three parts: a gold-blind ten-predicate capability-binding preflight verifying that a route can execute the evaluation contract before any scored task reaches it; reliability-inclusive first-pass scoring that keeps failures in the primary metric; and structurally score-blind adjudication of non-OK responses. A reference instantiation—seven suites, 128 locked tasks, 987 deterministic assertions over documents, spreadsheets, charts, tools, and governed SQL—is evaluated across eleven systems but kept sealed. Key results: identical Qwen3.6-27B weights scored 26.12, 62.25, and 68.27 on three FP8 routes, two of which later failed binding predicates retrospectively; four of seven suites are saturated in a six-system band, with discrimination concentrated in governed database and multi-tab work; nominal tier cuts fail multiplicity and BCa checks; and excluding failures from denominators reorders the leaderboard. Limitations include a single run per configuration, a sealed unfalsifiable corpus, a fully consumed challenge reserve, and authors all employed by the instrument's vendor.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Deployed capability is a property of the full serving tuple (model, route, precision, effort, envelope, tools, output contract, harness), so a benchmark must bind the serving route before scoring rather than score an advertised model identifier.
- Route-level capability availability is measurable and identifier-blind: identical Qwen3.6-27B weights scored 26.12, 62.25 and 68.27 on three FP8 routes, two of which later failed distinct finalized gate predicates.
- Reliability-inclusive first-pass scoring changes conclusions, not wording: excluding failed responses from denominators would make GPT-5.6 Luna the point leader (+11.45 vs first-pass) and lift Kimi K3 to fourth.
- Four of seven suites are saturated under a six-system band, with composite discrimination concentrated in governed database work (rho=31.4) and multi-tab joins (rho=12.5), so interval-backed resolution groups, not ranks, are the reportable output.
- The protocol — route binding, gold-blind adjudication, fail-closed resume, saturation criterion — is the transferable contribution and can be applied to any bank with a published request contract, while the specific 128-task instrument is depreciated by the authors' own analysis.
Strengths
Structural anti-p-hacking design that is verifiable, not just promised
Algorithm 2 (Table 5) routes every non-OK response through a typed allowlist projection in which gold references, grader output and scores are unreachable by construction, with a fail-closed HOLD for unmodeled classes; Algorithm 3 admits run extensions only on exact run-plan SHA-256 equality, forbidding per-task splicing. Section 4.3 argues the invariant precisely: correctness is what a biased selector would need and exactly what is unreachable.
Exceptional self-critical reporting, repeatedly against the authors' own interest
The paper reports that its PARTIALCOVERAGE branch is unreachable in v0.13.2 (Section 4.2), that the binding gate missed GLM's strict-JSON-under-reasoning failure and names the missing predicate without claiming to have run it (Section 6.5, Table 4), that difficulty labels are non-predictive with 93.8% Expert/Frontier while 4 of 7 suites saturate (Section 5.2, Appendix B.7), and that its own 'four of seven' saturation headline is a six-band statement that flips at band sizes 5 and 7 (Appendix B.7).
Unusually disciplined statistical reporting
A genuinely novel reporting outcome for capability benchmarks
Governance and ethics disclosures above field norm
Weaknesses
The primary artifact is unverifiable by any reader
Structural vendor conflict of interest over the instrument itself
Headline route-binding evidence is 2/3 retrospective
The key quantitative route/endpoint effect is confounded by design
Instrument quality issues the authors themselves document
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper (arXiv 2609.10494, Iterate.ai) defines the measured object as an eight-component tuple — model, serving route, precision, reasoning effort, observed envelope, tool implementation, output contract, harness (Definition 1, Section 3.1) — and argues that all 18 audited capability benchmarks instead score advertised model identifiers (Table 1). Its four claims are precisely scoped: route binding must precede scoring; route availability is a reportable outcome identifier-blind benchmarks cannot express; reliability-inclusive scoring changes orderings; and saturation analysis, not ranks, is the honest output. The authors explicitly disclaim what they cannot support: no documented case of a published benchmark misattributing a route limit to a model, no run-to-run variance estimates (n=1), no human baseline, no procurement-grade interpretation (Section 1.4). This scoping discipline is itself a methodological strength.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is IB2 (IBIB)?
IB2 (Iterate Business Intelligence Benchmark, written as IB twice) is a protocol from Iterate.ai for evaluating enterprise AI systems at the level of the full serving route — provider, precision, context/completion envelope, tool parser, output contract and harness — rather than the advertised model identifier. It combines a gold-blind capability-binding preflight, reliability-inclusive first-pass scoring, and structurally score-blind adjudication of failed responses.
Is the IBIB benchmark dataset public?
No. The reference instantiation — 128 locked tasks and 987 assertions across documents, spreadsheets, charts, tools and governed SQL — stays sealed. The authors release the protocol algorithms, classification tables, request contract, manifest schemas, synthetic conformance fixtures and dated audit ledgers via a planned GitHub release (IterateAI/IBIB), stating that the procedure is the artifact, not the corpus.
Why can the same LLM score very differently on different providers?
Because serving routes impose their own limits. In the paper, identical Qwen3.6-27B FP8 weights scored 26.12 on DeepInfra (which rejected a 13-image request at a 4-image route limit), 62.25 on CoreWeave (which stopped at exactly 32,768 completion tokens despite advertising 262,144), and 68.27 on a third route that passed the full ten-predicate binding gate before its run.
What did the IB2 evaluation find?
Across eleven systems: four of seven suites are saturated among frontier systems; real separation comes from governed SQL (trimmed range 31.4) and multi-tab spreadsheet joins (12.5). Excluding failed responses from scores would make GPT-5.6 Luna the leader. The lowest scorer, the composed GLM system (41.10), failed mostly on the evaluator's strict bare-JSON output contract, which the authors acknowledge partly reflects their own parser choice rather than reasoning ability.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text contains three explicit [...sampling gap...] markers; skipped content includes parts of Section 4.1, the full Table 6/7 composite presentation, parts of Sections 6.1 and 8.3-10, and tails of Appendices A-D. Judgments touching those sections rely on cross-references visible in the retained text.
- Table 7 (the primary composite with tier diagnostics) is not directly visible in the excerpt; its values were corroborated via Tables 18-19 and the stated 41.10-88.34 range in Section 6.2.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.