E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation
Wei Fan, Xinjie Shen, Xudong Guo, Jianhong Tu, Yang Su, Yinger Zhang, Lianghao Deng, Fengyu Wang, et al.
cs.LG · cs.CL
Quick verdict
First deterministic year-long e-commerce agent benchmark; reproducible open design is strong, but 5-run variance and vendor self-evaluation temper the rankings.
Summary
The paper introduces E-Commerce Bench, an open-source benchmark in which an LLM agent runs up to four simulated online stores over a 365-day year, starting from a ¥100,000 stake and scored primarily on year-end total assets. The environment grounds 6,886 products and 576 suppliers in real Taobao & Tmall data and adds a calendar of promotions, disasters, and supply shocks. For reproducibility, both market sides are deterministic: a fixed, seeded demand model governs purchases and returns, and a negotiation kernel makes every supplier pricing and accept/reject decision, with an LLM only verbalizing; 152 of 576 suppliers run scripted fraud. Eighteen models each play five episodes (90 total). GPT-5.6 Sol earns most (¥1,431,425, 14.31×) but ranks 16/18 on fraud avoidance; Claude Opus 4.7 leads bargaining (CSE+ 0.811) and fraud avoidance from mid-pack profit; Qwen3.8-Max-Preview tops the open-weight tier (¥416,252, 4.16×, 38% above GLM 5.2 high). Ten episodes end bankrupt; 16 of 18 models fail to beat a permutation null on repeat-order price anchoring. Limitations: only five episodes per model, with fraud-axis standard deviations exceeding means for 15/18 models; authors are Alibaba's Qwen team evaluating their own model; a single calibrated world may limit generalization.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- E-Commerce Bench is the first open-source benchmark combining deterministic counterpart negotiation, dynamic events, and concurrent multi-store management over a year-long horizon (Table 1).
- Because both market sides are deterministic (seeded demand model plus negotiation kernel with an LLM only verbalizing), outcome differences among models belong to agent policy rather than sampling chance.
- No single model dominates: GPT-5.6 Sol earns most (14.31×) yet ranks 16/18 on fraud avoidance, and six of seven profiled models fall below the field median on at least one axis.
- Qwen3.8-Max-Preview leads open-weight models at ¥416,252 (4.16×, 38% above GLM 5.2 high) and is the only model with clear long-horizon learning, beating its own shuffled prices by more than two standard deviations.
- Sixteen of eighteen models show no sign of pushing repeat-purchase prices down across 8,647 repeat purchases from the same supplier, exposing deficient experiential learning.
Strengths
Determinism engineered into both market sides
A seeded demand model (Eq. 1) and a negotiation kernel (Eqs. 10–18) make every price and accept/reject decision rule-based; an LLM renderer must relay the kernel quote and an accept is refused unless it matches within ±0.005 (Appendix D.4), so wording cannot alter transactions and outcome differences are attributable to agent policy.
Real-data grounding with calibrated information asymmetry
6,886 SKUs, 60 categories, 576 suppliers and 12 store types derived from Taobao & Tmall (§3.6, Tables 4–5); cost floors, elasticities and return bands stay hidden from the agent, and opening harshness is solved rather than sampled (Eq. 21), making the opening-quote CSE+ exactly 0.5 by construction — a useful sanity anchor for the metric.
Seven-axis capability profiles instead of one leaderboard
Learning metric with a proper null
Unusual implementation transparency
Weaknesses
Five episodes per model with extreme variance
Vendor self-evaluation conflict of interest
Negotiation metrics score self-selected session sets
Single world instance
Fraud axis rides on renderer sampling noise
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
E-Commerce Bench (arXiv 2608.30730; Qwen Team, Alibaba, with HKUST and Taobao & Tmall Group) is an open-source continuing (non-episodic) agent benchmark: one LLM agent runs up to four of twelve store types across a simulated 2026 calendar year from a ¥100,000 bank stake, scored primarily on end-of-year total assets (§3.1, Eq. 22). The paper claims three contributions (§1): the first open-source benchmark uniting deterministic counterpart negotiation, dynamic events, and concurrent multi-store management over a year; full determinism on both market sides via a seeded demand model (§3.4.3) and a negotiation kernel (§3.5) with an LLM only verbalizing; and a seven-axis evaluation showing no model dominates. Table 1 argues novelty against Vending-Bench 1/2, YC-Bench, MerchantBench, and RetailBench. Claims are well-scoped and, on the text provided, supported by the described design.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is E-Commerce Bench?
An open-source benchmark from Alibaba's Qwen team in which an LLM agent runs up to four simulated online stores for a 365-day year starting from ¥100,000, scored on year-end total assets plus six capability axes. The market is deterministic on both sides — a fixed demand model drives customers and a rule-based negotiation kernel drives suppliers — over real Taobao/Tmall product and supplier data, so runs reproduce exactly.
Which model performed best on E-Commerce Bench?
On money, GPT-5.6 Sol: it grew ¥100,000 into ¥1,431,425 (14.31×) averaged over five episodes. But it ranked 16th of 18 on fraud avoidance. Claude Opus 4.7 bargained best (CSE+ 0.811) and avoided fraud best while finishing mid-pack on profit, and Qwen3.8-Max-Preview was the best open-weight model at ¥416,252 (4.16×).
How can the benchmark be reproducible if suppliers negotiate in free text?
Supplier chat is rendered by an LLM, but every price, concession, and accept/reject decision comes from a deterministic kernel. An acceptance only executes if it matches the kernel's standing quote within ±0.005, and all randomness is seeded, so two identical agent trajectories meet identical economics — only the wording differs.
How many models were evaluated and how reliable are the rankings?
Eighteen frontier and open-weight models, each playing five independent 365-day episodes (90 total); ten episodes ended in bankruptcy. The authors report standard deviations throughout, and the fraud-avoidance axis is explicitly noisy — its standard deviation exceeds its mean for 15 of 18 models — so fine-grained ranks there should be read cautiously.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: multiple [...sampling gap...] regions skip parts of §3.3, §3.6, and several appendices (parts of B and D, E.2–E.6, F, G, H). Per-model result tables (e.g., the full Table 2 values and Tables 11–16) were not visible, so complete per-axis orderings beyond those quoted in §4 could not be verified.
- The excerpt cuts off mid-sentence in Appendix E.1 (surplus efficiency definition); this is a sampling artifact, not evidence about the paper's actual content or ending.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.