ERPBench: Evaluating LLM Agents for Enterprise Decision-Making Across Competitive Market Ecologies
Xinran Zhang, Pengrui Lu, Lyumanshan Ye, Pengfei Liu
cs.AI
Quick verdict
ERPBench: LLM-agent rankings flip between Solo and Arena markets (DeepSeek→Gemini). Paired design is solid but single-run, one-Arena scope limits generality.
Summary
ERPBench is an execution-instrumented benchmark asking whether conclusions about LLM enterprise-decision agents survive a change of competitive market ecology. It simulates a cereal manufacturer over six rounds of 30 simulated days, with coupled pricing, production, procurement, inventory, finance, carbon costs, and an MCI demand model. The same 100 problems run under two matched ecologies: Solo, where each evaluated agent faces five fixed rule-based opponents, and Arena, where six evaluated LLM agents share one market. Across six model families this yields 1,200 trajectories and 7,200 decision rounds; the primary metric is round-6 DDM terminal valuation. The leaderboard reorders: DeepSeek leads Solo (252.29M mean valuation, mean rank 1.67) while Gemini leads Arena (263.95M, 1.76); the ecologies agree on the task-level winner for only 21 of 100 problems (mean Spearman 0.182), and Gemini's bottom-rank rate falls from 22% to 0%. Paired Wilcoxon tests with Benjamini-Hochberg correction and problem-level bootstrap CIs show Gemini's +94.54M gain is significant (q=1.3e-15). An execution-audit layer logs interventions, dominated by production-capacity skips (82.5%). Key limitations: one trajectory per model-problem-ecology cell (no repeated stochastic runs), a single six-model Arena composition, unpinned service-side model endpoints, audit logging available only 'where available', and a fixed 100-problem, six-round horizon.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The leading enterprise-decision model differs across matched competitive market ecologies: DeepSeek leads Solo (252.29M mean valuation, mean rank 1.67) while Gemini leads Arena (263.95M, 1.76).
- The Solo and Arena ecologies identify the same task-level winner on only 21 of 100 problems, with mean within-problem Spearman rank correlation 0.182 (CI [0.090, 0.273]).
- Tail reliability reorders across ecologies: Gemini's bottom-rank rate falls from 22% to 0% and P90 normalized regret from 0.73 to 0.27 in Arena, while DeepSeek's P90 regret rises from 0.16 to 0.39.
- Terminal valuations require execution-audit qualification: 82.5% of logged interventions are production-capacity skips, and fallback/traceback events are stronger signals of output-pipeline fragility than capacity skips.
- The ecology effect is problem-conditioned rather than a uniform offset, varying across difficulty strata (Fig. 6).
Strengths
Paired matched-ecology protocol
The same 100 problem identifiers, seeds, difficulty strata, and scaffold are preserved across Solo and Arena (§3.2), enabling per-problem contrasts Δ(V) = V(Arena) − V(Solo) via Eq. (1). The headline reversal is therefore a within-problem paired contrast, not a cross-sample leaderboard comparison — a genuinely better measurement design than conventional single-ecology leaderboards.
Statistical discipline with honest nulls
Table 2 reports two-sided Wilcoxon signed-rank tests with Benjamini-Hochberg correction across six model-wise tests plus problem-level percentile bootstrap CIs; non-significant results are printed alongside significant ones (Doubao +6.90M, q=.557; Claude −2.83M, q=.866). Gemini's +94.54M [+78.74, +110.40], q=1.28e-15, and the Spearman CI [0.090, 0.273] excluding zero are properly supported.
Execution-audit layer with careful interpretation
Coverage audit and volume accounting
Candid limitations section
Weaknesses
One trajectory per cell — no repeated-run variance
Arena result is conditional on one six-model composition
Audit-log coverage unquantified
Unpinned commercial endpoints undermine reproducibility
No external anchor for the valuation scale
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ERPBench (arXiv 2609.04667, cs.AI) is a benchmark paper posing a narrow, well-posed measurement question: do conclusions about LLM enterprise-decision agents survive a change of *competitive market ecology*? The substrate is an ERPsim-style simulation of a cereal manufacturer — 12 SKUs, three distribution channels, six rounds of 30 simulated days — coupling pricing, production, procurement, inventory, finance, carbon costs, and a multinomial competitive interaction (MCI) demand model (§3.1). The same 100 problems run under two matched ecologies: **Solo** (each evaluated agent vs five deterministic rule-based opponents: conservative, aggressive, balanced, growth-oriented, risk-averse) and **Arena** (six evaluated LLM agents sharing one market). Six families — Claude Opus 4.6, DeepSeek-V4-Flash, Gemini 3.1 Pro Preview, GPT-5.5, Doubao Seed 2.0 Pro, Qwen 3.7 Max — yield 1,200 model-level trajectories and 7,200 decision rounds; the primary outcome is round-6 DDM terminal valuation.
The four central claims are all supported by the reported numbers: (1) the leader flips — DeepSeek tops Solo at 252.29M mean valuation (mean rank 1.67) while Gemini tops Arena at 263.95M (1.76), Table 1; (2) task-level winner agreement is only 21/100 problems with mean within-problem Spearman 0.182 [0.090, 0.273] (§4.5), including a 35-problem DeepSeek→Gemini transition; (3) tail risk reorders — Gemini bottom-rank 22%→0% and P90 regret 0.73→0.27 in Arena, DeepSeek P90 regret 0.16→0.39 (§4.6); (4) terminal scores need execution-audit qualification (§4.3–4.4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ERPBench?
ERPBench is a benchmark for evaluating LLM agents on enterprise resource planning (ERP) decisions — pricing, production, procurement, inventory, and finance — in a simulated cereal manufacturing company over six decision rounds. Its distinguishing feature is a paired protocol: the same 100 problems are run twice, once in a Solo ecology (the LLM agent against five fixed rule-based competitors) and once in an Arena ecology (six LLM agents competing in one shared market), to test whether model rankings survive the change.
Why do DeepSeek and Gemini swap first place between Solo and Arena?
The paper argues the two ecologies measure different competences. Against stable rule-based opponents (Solo), DeepSeek performs best (252.29M mean terminal valuation, mean rank 1.67). When six evaluated LLM agents shape the same market (Arena), Gemini performs best (263.95M, rank 1.76): it gains +94.54M on the same problems while DeepSeek loses 17.58M. The authors read this as DeepSeek being the stronger independent optimizer and Gemini the stronger competitive responder; the two settings agree on the per-task winner for only 21 of 100 problems.
Is ERPBench's code and data publicly available?
The paper states that code and benchmark resources are available in the GitHub repository GAIR-NLP/erp-bench. Note, however, that the reported leaderboard depends on commercial model endpoints (e.g., Gemini 3.1 Pro Preview, GPT-5.5) whose routing and versions may change over time, so exact numbers are snapshots of the observed service configuration rather than permanently reproducible results.
Which models were evaluated in ERPBench?
Six model families: Claude Opus 4.6, DeepSeek-V4-Flash, Gemini 3.1 Pro Preview, GPT-5.5, Doubao Seed 2.0 Pro, and Qwen 3.7 Max. Each was run on the same 100 ERP problems in both ecologies with the same agent scaffold, producing 1,200 model-level trajectories and 7,200 decision rounds across 700 simulator executions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2–6 are image-only in the extraction; their quantitative content here comes from captions and body text, so per-stratum values in Figure 6 and per-problem detail in Figure 4 could not be independently verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.