ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software
Kratika Bhagtani, Kusha Sridhar, Maziyar Baran Pouyan, Yuying Zhao, Eugene Siow
cs.AI · cs.CV · cs.MA
Quick verdict
ERPBench grades screenshot-only agents by ERPNext database state, exposing silent wrong-value saves; strong idea, but 30 tasks, 5 runs, no significance tests.
Summary
ERPBench is a benchmark that evaluates screenshot-only computer-use agents on a live, self-hosted ERPNext instance, grading every task against ground-truth values in the underlying database rather than on-screen confirmation. The suite has 30 tasks in three complexity tiers: 20 single-field edits/creates (T1), 4 multi-field record creations (T2), and 6 multi-screen chained workflows (T3, three starting from a blank page). A production harness with a Flask FSM, risk classifier, and human-approval gate runs autonomously during benchmarking via an auto-approver. Three graders score each run: a database grader (via Frappe API), a four-stage grader (Navigation, Interaction, Commit, Database), and a log grader; a five-mode failure taxonomy localizes breakdowns. Evaluating six agents over five runs per task, Claude Sonnet 4.6 reaches 94-100% success while the best open models reach only 34% on T1 and near zero on T2/T3; UI-TARS-7B commits saves in 85% of T2 runs but leaves the correct value in just 3%, illustrating silent failures invisible on screen. A three-person human baseline (including a first-time user) reaches 87-100%. Main limitations: the task set is small, five runs per task with no confidence intervals or significance tests, model scale and scaffolds differ across the six systems, the human baseline comprises the authors, and release of the benchmark is promised but not yet available.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Screenshot-only agents can be meaningfully graded against persistent database state on a live, self-hosted open-source ERP, exposing silent failures that screenshot-level or interaction-level evaluation misclassifies as success.
- Strong general GUI performance does not transfer to enterprise reliability: models scoring 34.8-42.5 on OSWorld-Verified reach only 9-23% on ERPBench's easiest tier.
- Reaching the right form and saving is insufficient: UI-TARS-7B invokes saves in 85% of T2 runs yet writes the correct value in only 3% of them.
- Enterprise failures localize post-navigation - in grounding, save-step, and perception stages - rather than in planning.
- The same harness supports human-gated production deployment and fully autonomous benchmarking, aligning evaluation with deployment conditions.
Strengths
Database-level ground truth catches silent failures
Grading every target field through the Frappe API (Sec. 3.3) exposes failures screenshots hide: UI-TARS-7B saves in 85% of T2 runs but is database-correct in only 3%, and reaches 95% navigation yet only 9% database success on T1 (Table 3) — precisely the interaction-level blind spot the paper targets.
Live, self-hostable, non-simulated environment
ERPNext in Docker with UUID-suffixed fixture seeding per run (Sec. 3.1) isolates runs without full database resets and, unlike UI-CUBE's web mocks or the proprietary Salesforce/ServiceNow benchmarks (Table 1), requires no license or vendor account.
Stage-wise grading plus a named failure taxonomy
Human baseline separating ERP expertise from task difficulty
Weaknesses
Very small task suite on a single ERP
No uncertainty quantification
Scale and scaffold confound in the six-agent comparison
Stage-grader verification underspecified
Reproducibility currently aspirational
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ERPBench?
ERPBench is a benchmark that tests AI computer-use agents on a real, open-source ERP system (ERPNext) using only screenshots and simulated mouse/keyboard input, then checks whether the correct values actually landed in the application's database - not just whether the screen looked right. It includes 30 tasks at three difficulty levels and grading tools that pinpoint where an agent's workflow broke down.
Why do AI agents fail at ERP tasks even when they navigate to the correct screen?
The paper shows agents often reach the right screen but then fail to type exact values, trigger saves correctly, or persist the right data. For example, one model (UI-TARS-7B) initiated a save in 85% of runs on multi-field record creation yet wrote the correct value in only 3% - a 'silent failure' invisible in screenshots because the interface still shows a success confirmation.
Which models were evaluated on ERPBench and how did they perform?
Six agents were tested: Claude Sonnet 4.6 succeeded on 94-100% of tasks across the three difficulty tiers, while the best open-weight models (Holo3-35B-A3B, Qwen3-VL-32B) managed only about 32-34% on the easiest tier and near 0% on harder tiers. A first-time human user of ERPNext still scored 87-100%, so the gap is not just about ERP expertise.
Is ERPBench code and data publicly available?
As of the paper (arXiv 2609.17885, September 2026), the authors state they 'intend to release' the benchmark and harness, but no repository or download link is provided yet. The underlying ERP (ERPNext) is open source and self-hostable via Docker, which should make reproduction feasible once the task suite is published.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 3's per-model taxonomy percentages are partially garbled in the extracted text (number sequences without labels for some models); individual shares may be misattributed, though the overall post-navigation clustering conclusion is clear from the prose.
- Figures 1-2 (system architecture, harness UI) could not be visually inspected; descriptions rely on captions and embedded labels.
- Prompt templates, model inference configurations, and the pinned ERPNext version are absent from the provided text, possibly in supplementary material not included here.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.