JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
Zixiang Chen, Yuheng Lu, Zihao Cheng, Zeming Liu, Jizeng Bai, Ziye Huang, Zhiyin Lin, Zihan Li, et al.
cs.AI
Quick verdict
First cross-device GUI benchmark across 3 OSes with typed task auto-composition; near-zero multi-device scores may partly reflect the shared planner harness.
Summary
The paper introduces JarvisGUI, a benchmark for evaluating GUI agents on workflows spanning multiple devices — Android, Windows, and Ubuntu — rather than a single platform. Tasks are modeled as typed input-output transformations, letting human-annotated atomic tasks be automatically composed into directed task graphs whose instructions (rewritten by Qwen3-32B) and parameterized Python evaluators are instantiated at runtime and checked against final environment state. The benchmark contains 118 atomic tasks across 31 apps and 150 compositional tasks (50 single-platform with dependencies, 50 cross-platform without, 50 cross-platform with), totaling 442 subtasks plus 65 file-transfer auxiliary tasks, executed in QEMU/KVM virtual machines with an AList web drive for transfers. Quality control combines pre/post evaluator checks, LLM-as-a-judge filtering that keeps only 5/5 scores, and manual audit of 50 composed tasks (92% correct; Cohen's Kappa ~0.92). Six open-source agents, all wrapped in a shared Planner-Grounder harness with Qwen3-VL-Plus as planner, reach 28.8-42.4% task success on atomic tasks but 0-2% on multi-device dependent workflows; the best system (HOLO2) attains 8.0% multi-task TSR and 19.0% sub-task success, and swapping in a Kimi K2.6 planner lifts multi-task TSR only to 11.3%. Bootstrap 95% CIs are reported. Main limitations: results may partly reflect the shared harness rather than native agent capability, each compositional category holds only 50 tasks, and code/data are promised but not yet released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- JarvisGUI is the first systematic benchmark for cross-device GUI agent tasks, spanning Android, Windows, and Ubuntu with realistic file and state transfer.
- A slot-typing-based task model (typed input-output transformations with parameterized evaluators) enables reliable, automatic, and verifiable composition of multi-step cross-device workflows.
- State-of-the-art open-source GUI agents achieve 28.8-42.4% TSR on atomic tasks but collapse to 0-2% TSR on multi-device dependent tasks, a gap invisible to single-device benchmarks.
- The cross-device bottleneck generalizes across planner backbones: swapping Qwen3-VL-Plus for Kimi K2.6 raises multi-task TSR only from 8.0% to 11.3%.
Strengths
Genuinely new problem formalization
Section 3.1/4.1 model tasks as typed tuples T=(I,O,D,Φ,P) with runtime materialization of outputs, instructions, and evaluators; type-checked edges enable automatic composition of cross-device workflows, directly addressing CRAB's 18-task limitation with 150 compositional tasks plus 65 transfer-assistant tasks (Table 2).
Real, reproducible multi-OS environment
Section 3.2 and Appendix A.4 describe QEMU/KVM-accelerated VMs inside ephemeral Docker containers on a unified virtual subnet, adapted OSWorld server-controllers for desktops, ADB for Android, a unified action space (Table 7), and deterministic final-state Python evaluators with pre/post execution checks.
Layered quality control with quantified agreement
Statistical transparency unusual for agent benchmarks
Honest scoping of known gaps
Weaknesses
Harness confound in headline results
Small subgroups with overlapping CIs and no significance tests
Transfer scaffold is telegraphed in the prompt
Reproducibility currently unverifiable
LLM-judge selection and limited human audit
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
JarvisGUI (arXiv 2609.10451, cs.AI) positions itself as the first systematic benchmark for cross-device GUI agents, evaluating workflows that span Android, Windows, and Ubuntu. Three claims anchor the paper: (1) existing benchmarks are single-device and statically defined, producing overly optimistic readiness assessments (Section 1, Table 1); (2) a slot-typing-based task formalism enables reliable, automatic composition and verifiable evaluation of cross-device workflows (Sections 3.1, 4.1); (3) state-of-the-art open-source agents collapse on cross-device dependencies — atomic-task TSR of 28.8-42.4% versus 0-2% on multi-device dependent (MD) tasks (Table 3). CRAB is acknowledged as the closest predecessor but characterized as limited to 18 cross-device tasks without realistic transfer modeling (Section 2.2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is JarvisGUI?
JarvisGUI is a benchmark for evaluating GUI (graphical user interface) agents on workflows that span multiple operating systems — Android, Windows, and Ubuntu — instead of a single device. It automatically composes multi-step tasks from typed, reusable task templates and scores agents by programmatically inspecting the final state of virtual machines, requiring agents to transfer files and maintain context across devices.
How well do current GUI agents perform on JarvisGUI?
In the paper, six open-source agents (UI-Venus, UI-TARS-1.5, MAI-UI, GUI-Owl, Qwen3-VL, HOLO2), all driven through a shared planner-grounder harness, succeed on 28.8-42.4% of single-device atomic tasks but on only 0-2% of tasks requiring state transfer across devices with dependencies. The best multi-task success rate is 8.0% (HOLO2), rising to 11.3% when the planner is swapped for Kimi K2.6.
How is JarvisGUI different from OSWorld or CRAB?
OSWorld evaluates agents on a single desktop OS with largely fixed tasks; CRAB covers cross-device execution but contains only 18 such tasks and does not model realistic file/data transfer, per the paper. JarvisGUI provides 150 automatically composed cross-device workflows (442 subtasks plus 65 transfer-assistant tasks) across three operating systems, with typed input-output dependencies and transfers via a network drive.
Is JarvisGUI's code and data available?
The paper states that reproduction scripts, benchmark data, evaluation code, Docker orchestration, and model configurations 'will be publicly available'. At review time the release could not be verified, and running the benchmark requires substantial infrastructure: QEMU/KVM virtualization of three operating systems and vLLM-served models (the planner in the main experiments is a paid Aliyun API model).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with four marked [...sampling gap...] regions; the end of Section 5.2, the conclusion, part of Appendix A.1, and Appendices A.5-A.6 were skipped, so no judgment in this review relies on those parts.
- The repository link in footnote 1 renders as the placeholder text 'JarvisGUI', so the actual code-release status could not be verified from the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.