OSWorld-Pro: Process-based Evaluation for Computer Use Agents
Zhilin Wang, Shaokun Zhang, Yifan Zhang, Hao Zhang, Jin Xu, Binfeng Xu, Jian Hu, Yunheng Zou, et al.
cs.CL · cs.AI · cs.LG
Quick verdict
Adds process-based CUA evaluation: 305 tasks, 2814 subgoals, 67k human labels; judge validation solid, yet single-judge, single-run design tempers trust.
Summary
OSWorld-Pro reframes computer-use agent (CUA) evaluation around process rather than final outcomes. The authors decompose 305 long-horizon Linux/GUI tasks into 2,814 sequentially dependent subgoals and publish 67,264 step-level human annotations (targeted subgoal, feasibility, progression, completion) collected over 5,000+ person-hours by 25 vetted annotators across three categories: rare applications (Diversity), coordination of ≥4 apps per task (mean 3.45), and robustness across 19 Linux distributions and 13 desktop environments. Because human judging at scale is costly, they validate LLM judges against these labels: GPT-5.6-Sol (max reasoning) reaches 94.1% subgoal-level agreement versus 95.6% between independent humans, but feasibility Macro-F1 lags badly (61.9 vs 91.6 human). Benchmarked with the OSWorld harness (250 turns, single run), Claude Opus 4.8 leads at 77.7% of tasks with all feasible subgoals completed, versus 83.4% for the top Opus model on OSWorld; the best open-weight model (Qwen 3.8 Flash Next, 125B) manages 55.1%. Process metrics expose failure modes outcome checks miss: click-progression as low as 39.0% (MiniMax M3), ~39.8 steps wasted per infeasible subgoal (Sonnet 5), and a 59-step subgoal-irrelevant plateau by Opus 5. Key limitations: all results flow through one judge model that is itself a benchmarked competitor, single runs without variance estimates, and no data/code release link appears in the text.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- OSWorld-Pro is the first process-based evaluation benchmark for CUAs: 305 tasks, 2,814 sequentially dependent subgoals, and 67,264 human step-level annotations.
- An LLM judge (GPT-5.6-Sol Max) approximates human judgment at the subgoal level (94.1% vs 95.6% inter-human agreement), though feasibility and progression judgment remains materially weaker than humans.
- OSWorld-Pro is substantially harder than OSWorld: the best model completes all feasible subgoals on 77.7% of tasks (Claude Opus 4.8) versus 83.4% on OSWorld, with open-weight models dropping to at most 55.1%.
- Process evaluation exposes failure modes invisible to outcome checks: click-coordinate errors (click progression as low as 39.0%), subgoal-irrelevant action plateaus (59 steps for Opus 5), and inefficient persistence on infeasible subgoals (39.8 steps for Sonnet 5).
Strengths
Unprecedented human ground truth for process evaluation
67,264 step-level labels (targeted/feasible/progressed/completed) across 305 tasks and 2,814 subgoals, >5,000 person-hours, 25 vetted annotators; inter-annotator Cohen's κ reported per field (0.987 targeted, 0.847 feasibility, 0.869 progression, 0.940 completion), honestly signaling which fields are subjective (§3, §4.1).
Multi-level, multi-config judge validation with honest failure reporting
Table 1 evaluates 3 GPT-5.6 sizes × 6 reasoning efforts at step/subgoal/task levels; the paper openly shows the best judge collapses on feasibility (Macro-F1 61.9 vs human 91.6) and quantifies the bias mechanism (models predict 5.1% infeasible vs 15.3% human ground truth at max effort).
Diagnostic yield that outcome benchmarks cannot provide
Substantial coverage expansion over OSWorld
Unusual transparency for a benchmark paper
Weaknesses
Judge monoculture with conflict of interest
No statistical rigor on headline deltas
Headline metric contradicts the paper's own thesis
No artifact release evidenced in the text
Trajectory provenance under-specified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
OSWorld-Pro (NVIDIA; arXiv 2609.24890) argues that outcome-only evaluation of computer-use agents (CUAs) — the OSWorld paradigm of functional verifiers on final deliverables — hides where and why agents fail. The deliverable is a process-based benchmark: 305 tasks (117 Diversity, 109 Coordination, 79 Robustness) decomposed into 2,814 sequentially dependent subgoals, with 67,264 step-level human annotations of (targeted subgoal, feasibility, progression, completion), reportedly over 5,000 person-hours from 25 annotators/reviewers in 5 countries (§2-3). Four claims: (1) first process-based CUA benchmark; (2) LLM judges approximate human judgment well enough (GPT-5.6-Sol Max: 94.1% subgoal-level vs 95.6% inter-human, Table 1); (3) the benchmark is harder than OSWorld (best 77.7% vs 83.4%); (4) process metrics expose new failure modes. Claims 1-3 are supported within stated limits; claim 4 is the strongest demonstrated contribution.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is OSWorld-Pro and how does it differ from OSWorld?
OSWorld-Pro is a benchmark that evaluates computer-use agents on process, not just final outcomes. OSWorld checks whether a task's final deliverable passes functional verifiers; OSWorld-Pro decomposes 305 long-horizon tasks into 2,814 sequentially dependent subgoals and scores step-by-step progress using 67,264 human annotations, with an LLM judge (GPT-5.6-Sol) validated against them.
How well do LLM judges match human annotations in OSWorld-Pro?
The best judge, GPT-5.6-Sol at maximum reasoning, agrees with humans on 94.1% of subgoal-level outcomes (independent humans agree with each other at 95.6%) and identifies targeted subgoals with 97.0 Macro-F1 versus the human 98.4. It is notably weaker on feasibility (F1 61.9 vs 91.6) and progression (74.7 vs 94.1), the more subjective fields.
Which models perform best on OSWorld-Pro?
Claude Opus 4.8 (max reasoning) leads at 77.7% of tasks with all feasible subgoals completed, followed by Opus 4.7 (76.7%), Sonnet 5 (76.4%), Opus 5 (75.7%) and GPT-5.6-Sol (74.8%). The best open-weight model, Qwen 3.8 Flash Next (125B), reaches 55.1%; MiniMax M3 drops from 75.2% on OSWorld to 28.9% here.
Why is OSWorld-Pro harder than OSWorld for agents?
Tasks average 9.2 sequentially dependent subgoals and 3.45 distinct applications (vs 1.34 in OSWorld), include rare apps, and span 19 Linux distributions and 13 desktop environments. Because subgoals must be completed in order and the task metric requires all feasible subgoals to be done, partial success still scores zero at the task level.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 are available only as flattened text/OCR; visual details of the failure case studies (Figs. 4-5) could not be independently verified.
- Tables were extracted as flat text; some cell-to-row mappings (notably Table 1 token/cost columns) were reconstructed from surrounding context.
- The cost figure $249.60 (Table 1) is read as the total full-judging-run cost per §G wording; the per-task vs per-run interpretation rests on that wording.
- No data/code release URL appears in the extracted text; codeAvailable=false reflects that absence, not an explicit refusal to release.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.