Evaluating Open-Weight E-Commerce Agents with Environment-Grounded Verification
Nimit Shah, Haitz Sáez de Ocáriz Borde
cs.LG
Quick verdict
Precommitted-scenario e-commerce eval turns success rates into verifiable failure diagnoses; strong design, but simulator/judge dependence limits conclusions.
Summary
The paper introduces a deterministic e-commerce evaluation environment in which each trial's customer persona, difficulty tier, hidden target cart, and per-item reveal schedule are precommitted before interaction. A Claude Sonnet 4.5 user simulator plays a customer who progressively reveals requests while the evaluated agent operates a 10,000-product control catalog through eight tools; the environment logs every tool call with contemporaneous state, enabling post-trial verification by state replay rather than terminal success alone. Deterministic checks score cart precision/recall, phantom actions, and numeric claims against catalog rows; a judge handles only language-dependent questions (attribute support, conversational rubrics), and tool-call correctness is scored against the simulator's predicted call chains. Across eight open-weight agents (20B-35B, MoE and dense) at 160 trials each and 44 metrics, exact-cart success spans 0.30-0.57, and the profiles separate failure modes terminal scores merge: Qwen3.5-35B over-purchases (recall 0.85, precision 0.89, 44.1 calls/trial), Qwen3-30B under-acts (15.4 calls, 63% budget exhaustion), and GPT-OSS-20B plus Qwen3-30B show attribute-hallucination rates of 0.236/0.252 versus a ~0.055 cluster, with ordinary numeric rates. A verification ablation (Table 2) maps ten questions to the evidence each requires. Limitations: simulator and primary judge are the same proprietary model; the GPT-5.4 cross-check covers only two agents and five rubrics without human calibration (clarification r=0.49); tier subsamples are small (n=17-33); checkpoint revisions are not pinned; no code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Precommitting the target cart, persona, and per-item reveal schedule lets an evaluator grade intermediate actions (searches, adds, attribute claims) against what the customer had requested at that time, which terminal-success scores cannot do.
- Deterministic state replay plus catalog and tool-log checks settle most verification questions without a judge; only attribute support and conversational quality require model judgment, as formalized in the Table 2 ablation.
- Across eight open-weight agents at 160 trials each, capability profiles separate under-action (Qwen3-30B: 15.4 calls, 63% budget exhaustion), over-purchase (Qwen3.5-35B: recall 0.85, precision 0.89, 44.1 calls), attribute hallucination (GPT-OSS-20B/Qwen3-30B: 0.236/0.252), and poor search — all obscured by exact-cart success rates of 0.30-0.57.
- Tool-call correctness can be scored against the user simulator's predicted expected-call chain, with category-weighted penalties that treat unjustified cart writes more severely than extra reads.
Strengths
Precommitted reveal schedules make request timing gradeable
Section 3.3 stamps each target item with the customer-message index at which it is first requested, so a search issued before the request is distinguishable from one issued after. Table 2 explicitly notes τ-bench's published reward cannot make this distinction — a genuine gap in conversational agent benchmarks that this design closes.
Evidence-tiered verification ablation is reusable methodology
Table 2 (Section 5.1) systematically maps ten evaluation questions to Rule/Model/Hybrid/Infer verdicts across four retained-evidence conditions (final cart / transcript / state / full simulator record), making explicit what each evidence tier can and cannot establish. This template generalizes beyond e-commerce.
Diagnostic channels demonstrably separate failure modes
Unusually complete specification and cost transparency
Candid limitations section
Weaknesses
Simulator/judge confound on a single proprietary model
Bad-search metric entangled with a weak retriever
Thin inferential discipline across a 44-metric battery
Reproducibility gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper introduces an e-commerce agent evaluation environment whose defining move is precommitment: before any interaction, a seeded generator fixes the customer persona, difficulty tier, hidden target cart (drawn from a 10,000-product control catalog whose 19-category mix averages six retailers, Appendix A), and a per-item reveal schedule stamping when each requested item first becomes relevant (Section 3.3). A Claude Sonnet 4.5 user simulator plays the customer; the evaluated agent assists through eight tools over a MiniLM-L6-v2 retrieval backend. The environment logs every tool call with contemporaneous state, so a post-trial verifier settles most questions by deterministic state replay, reserving the LLM judge for language-bound questions (attribute support, conversational rubrics), while tool-call correctness is scored against the simulator's predicted expected-call chain with category-weighted penalties (Section 4.3, Appendix E). Central claims: (1) request timing becomes gradeable, unlike τ-bench's final-state reward; (2) a 44-metric battery separates failure modes terminal success merges; (3) the eight-model study (20B–35B, MoE and dense) demonstrates this diagnostically. Table 2's verification ablation formalizes which questions each evidence tier can answer — the paper's most transferable contribution.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is environment-grounded verification in agent evaluation?
It means scoring an agent's intermediate behavior against evidence the environment retained during the trial — the hidden target cart, a precommitted schedule of when each customer request becomes relevant, the full tool log with contemporaneous state, and a state-replayed cart — instead of judging only the final outcome. Deterministic replay answers factual questions (was the item in the cart, was the price claim correct); an LLM judge handles only language-dependent questions such as whether a product description is supported by the catalog row.
Which models were evaluated and how did they perform?
Eight open-weight models between 20B and 35B parameters: GPT-OSS-20B, Gemma4-26B/31B, Qwen3-30B, Qwen3.5-27B/35B, and Qwen3.6-27B/35B, each running 160 trials. Exact-cart success ranged from 0.30 (Qwen3-30B) to 0.57 (Qwen3.6-35B). The profiles also flagged under-action (Qwen3-30B), over-purchase (Qwen3.5-35B, 44.1 tool calls per conversation), and attribute-hallucination rates up to 0.252 (GPT-OSS-20B and Qwen3-30B) while numeric claims stayed accurate.
How does this benchmark differ from τ-bench or WebShop?
Both score terminal outcomes: WebShop rates the purchased product against a request given up front, and τ-bench compares the final database with an annotated goal. Here, each item's reveal turn is fixed before the dialogue begins, so a search issued before the customer mentioned a product can be distinguished from the same search issued after; per-turn expected tool calls, cart state, and catalog claims are all graded individually.
Is the benchmark or code publicly available?
The paper does not state a code, data, or transcript release, and model checkpoints are identified only by repository IDs without pinned immutable revisions. The environment's construction is documented in enough detail (catalog shares, basket formulas, penalty schedules) that reimplementation is conceivable, but the published scores cannot currently be reproduced independently because they depend on Claude Sonnet 4.5 as simulator/judge.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 and Figure 2 are rendered as jumbled text in the extraction; Figure 2 values were cross-read against Tables 15-16 instead of the figure labels.
- A code-availability or repository statement could exist outside the extracted text; 'no code release stated' reflects the provided text only.
- Refusal-probe wording (Appendix J) is summarized rather than fully quoted in the extraction, limiting judgment on probe construction.
- The paper is dated September 2026 and cites 2026 releases (Gemma 4, Qwen3.5/3.6, GPT-5.4) that could not be independently verified offline.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.