APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
Jintian Feng, Long Chen, Xiao Yu, Jiayi Dai, Chenglong Liu, Haoru Wang, Zizhen Xue, Yuxuan Shi, et al.
cs.AI
Quick verdict
Proposes a reproducible 557-task benchmark over 17 simulated apps; rigorous, self-audited evaluation, but live-app transfer and variance reporting unproven.
Summary
AppSim-Bench is a benchmark of 557 tasks across 17 simulated replicas of high-frequency Chinese and English mobile apps (WeChat, JD, Amazon, etc.) designed to make GUI-agent evaluation both realistic and reproducible. Instead of live commercial apps, agents operate sandboxed simulators with controllable backend data, resettable initial states, and deterministic outcome-based verification (state, answer, hybrid task types), built via a coding-agent-assisted, human-verified workflow. The authors evaluate 19 agents (15 general-purpose from five model families; 4 GUI-specialized) under temperature 0 and a 50-step cap. Claude-Opus-4.7 leads at 50.27% accuracy; 28.55% of tasks are solved by no agent and only 0.36% by all. Accuracy declines monotonically with task length, is weakest on threshold-filtering tasks (best 42.86%), and correlates negatively with action overhead and budget exhaustion. Supporting audits include a blind human realism study (simulation mistaken for the real app in 19.0% of decided static judgments), a placeholder-residual bound of 4.28%, and a 0/3,933 answer-extraction error audit. Main limitations: realism is validated visually and behaviorally only within the simulator, with no evidence that model rankings transfer to live apps; accuracy is reported without confidence intervals or significance tests, so close rankings (e.g., 50.27% vs 49.91%) are uninterpretable; construction quality relies on internal review without inter-annotator agreement.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Controllable simulated apps with deterministic outcome-based verification make mobile GUI-agent evaluation reproducible without sacrificing task-relevant realism (557 tasks, 17 apps).
- Autonomous mobile execution is far from solved: the best of 19 evaluated agents (Claude-Opus-4.7) completes 50.27% of tasks, and 28.55% are solved by none.
- Performance degrades monotonically with task length and is weakest on numerical-reasoning tasks, especially threshold filtering (best 42.86%).
- Inefficient trajectories - high action overhead and 50-step budget exhaustion (0.44% success) - are strong empirical markers of failure.
Strengths
Reproducibility engineered end-to-end
Resettable per-task initial state, controllable backend data, deterministic outcome checks across state/answer/hybrid types, and scoring with no LLM calls (Appendix B) directly target the stochasticity that undermines live-app benchmarks.
Unusually deep self-auditing
Blind A/B realism study (510 static + 255 dynamic judgments, Wilson CIs), placeholder-residual upper bound of 4.28% over 654 executions (Appendix E), 0/3,933 answer-extraction errors, and a 1/1,287 perception-error audit on JD (Appendix F); Uber Eats construction logged at ~2.3 review rounds per page (Appendix D).
Broad, balanced agent and ecosystem coverage
Findings are concrete and actionable
Candid limitations section
Weaknesses
No uncertainty quantification on headline numbers
External validity of the core claim is assumed, not tested
Budget cap entangled with the long-horizon finding
App-level evidence is unbalanced
ZH-EN gap narrative lacks confound controls
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
AppSim-Bench targets the realism-vs-reproducibility trade-off in mobile GUI-agent evaluation. It contributes 557 tasks over 17 simulated replicas of high-frequency apps - 9 Chinese (WeChat, JD, RedNote, Tencent Meeting, Amap, Ctrip, Bilibili, Ele.me, NetEase Cloud Music) and 8 English (Amazon, Booking.com, Instagram, Spotify, Uber Eats, WhatsApp, YouTube, Zoom). Three claims anchor the paper: (1) controllable simulated apps with deterministic outcome verification reconcile realism with reproducibility, where live commercial apps (recommendations, ads, account state) and system/open-source app suites (AndroidWorld) each fail one side (Table 1); (2) the benchmark discriminates 19 agents, led by Claude-Opus-4.7 at 50.27% with 28.55% of tasks (159) unsolved by any agent and only 0.36% solved by all; (3) failures concentrate in long workflows, numerical reasoning, and inefficient budget-exhausted trajectories.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AppSim-Bench?
A benchmark for evaluating mobile GUI (graphical user interface) agents. Instead of live commercial apps, it uses 17 simulated replicas of popular Chinese and English apps with controllable backend data and 557 natural-language tasks. Success is checked deterministically by the final outcome - a state change, a structured answer, or both - which makes runs reproducible across models and labs.
Which model performs best on AppSim-Bench?
In the paper's evaluation of 19 agents, Claude-Opus-4.7 leads with 50.27% task success, followed closely by Gemini-3-Pro (49.91%) and GPT-5.5 (49.73%). GUI-specialized models lag well behind: the best of them, MobileAgent-v3.5, reaches 31.42%. No agent solved 28.55% of the tasks, and only 0.36% were solved by all.
Why simulate apps instead of testing agents on real ones?
Live apps change constantly - recommendations, ads, account states and content differ between runs, so two agents may face different environments for the same task. AppSim-Bench rebuilds each app as a sandbox with resettable, predefined data and deterministic outcome checks, trading some realism (ads, personalization, live content) for fair, reproducible comparisons.
Is AppSim-Bench's code and data publicly available?
The paper states the project is available at github.com/Acrab-Agentic-Labs/AppSim, covering the simulated apps, task data and evaluation pipeline. The provided paper text does not include external verification of the release's completeness.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: regions marked [...sampling gap...] (parts of Section 3, Appendix B/E/F content, and parts of the prompt templates) were skipped by sampling; every number cited in this review appears in the provided text.
- Figures were present only as captions in the extracted text; figure-dependent judgments (e.g., Figure 4 task coverage, Figure 7 page topology) rely on caption descriptions.
- The tail of Table 5 (placeholder-residual per-app breakdown) is cut off by a sampling gap; conclusions avoid relying on its missing rows.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.