Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications
Satwik Ram Kodandaram, Monalika Padma Reddy, Xiaojun Bi, Jiawei Zhou, I. V. Ramakrishnan, Vikas Ashok
cs.HC · cs.AI
Quick verdict
Real-world CUA study with 8 blind users: GPT-5 tops at 52.5% (n.s.); rich dataset and failure taxonomy, but replay confounds and stats caveats remain.
Summary
The paper reports a three-week IRB-approved diary study in which eight blind screen-reader users used OLLA, a screen-reader-accessible interaction layer over computer-use agents (CUAs), on their own Windows machines, issuing 1,258 natural-language commands across 12 applications (consolidated into 304 task intents in eight categories). The deployed agent was GPT-5; afterwards the same commands were re-executed with Claude Sonnet, Gemini 2.5 Computer Use, UI-TARS-1.5-7B, and Qwen3-VL-8B in fresh application instances under a shared pipeline. GPT-5 achieved the highest success rate (52.5%, 95% CI 49.8–55.3), ahead of Claude Sonnet (48.5%), Gemini (43.9%), UI-TARS (39.8%), and Qwen3-VL (37.9%), though the GPT-5–Claude gap is not statistically significant (McNemar p=.071). Four annotators (Krippendorff's α=0.84) built reference steps and labeled outcomes; trace analysis attributes failures to UI grounding (24.6% of GPT-5 unsuccessful traces), hidden-path discovery (21.4%), prior-knowledge reliance (17.1%), constraint loss (20.6% of partial completions), and termination errors (17.6%). Interviews surface beyond-automation themes: explanation, situated troubleshooting, user-controlled execution, and learning support. Main limitations: only eight participants, Windows- and English-only scope, replay conditions differing from in-situ deployment for the comparison models, non-independent repeated commands, and task selection conditional on participants' choices.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Current CUAs complete a meaningful but unreliable share of real-world nonvisual desktop tasks: GPT-5 succeeded on 52.5% of 1,258 blind-user-issued commands, not significantly above Claude Sonnet's 48.5% (p=.071).
- Failures concentrate in recurring, quantified modes: UI grounding and hidden-path discovery, prior-knowledge overriding the observed UI, constraint loss across steps, and termination/commit-recognition errors.
- Blind users envision CUAs as collaborative support systems — explanation, situated troubleshooting, user-controlled execution, and learning — rather than end-to-end autonomous replacements.
- The released corpus of commands with full execution traces enables future work on clarification, recovery, and reward-guided assistive agents (Sections 7.1–7.2).
Strengths
Real-world deployment, not simulation
Three-week IRB-approved diary study on participants' own Windows machines (Sections 3.1, A.2) yielding 1,258 organically-issued commands across 12 applications — a deliberate contrast with OSWorld/WebArena-style benchmarks (Section 2.1) and prior simulated accessibility CUA work (Section 2.2).
Disciplined outcome coding
Four annotators independently performed each task to derive minimum reference-step sequences before labeling success/partial/failure; Krippendorff's alpha = 0.84 with consensus resolution (Section 3.3). Table 5 rows each sum exactly to 1,258 and Section 5 denominators reconcile (597 = 166 failures + 431 partials for GPT-5).
Honest statistics
Quantified, actionable failure taxonomy
Controlled multi-model comparison
Weaknesses
Deployment-vs-replay asymmetry
Independence violated in inference
Model configuration parity unclear
Denominator drift in Section 5
Dataset access under-specified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.00524, Stony Brook + Old Dominion) asks whether current computer-use agents (CUAs) can support blind screen-reader users in everyday desktop work, via three RQs: effectiveness (RQ1), breakdown taxonomy (RQ2), and beyond-automation roles (RQ3). Central claims: (1) GPT-5, deployed inside OLLA for three weeks with 8 blind users issuing 1,258 commands across 12 Windows applications, succeeded on 52.5% of attempts (95% CI 49.8–55.3), ahead of four models re-executed on the same commands (Claude Sonnet 48.5%, Gemini 2.5 CU 43.9%, UI-TARS 39.8%, Qwen3-VL 37.9%) — though the GPT-5 vs Claude gap is explicitly non-significant (McNemar chi-squared=3.25, p=.071, Section B.3); (2) failures cluster into quantified modes (grounding, hidden paths, prior-knowledge override, constraint loss, termination); (3) users want collaborative support, not end-to-end automation. The released command-plus-trace dataset is a stated contribution.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How often do computer-use agents succeed on tasks from blind users?
In this study, the best model (GPT-5) completed 52.5% of 1,258 real desktop commands issued by eight blind screen-reader users (95% CI 49.8–55.3%). Claude Sonnet followed at 48.5%, Gemini 2.5 Computer Use at 43.9%, UI-TARS at 39.8%, and Qwen3-VL at 37.9%; the GPT-5–Claude gap was not statistically significant (McNemar p=.071).
What is OLLA?
OLLA is the paper's open-source prototype: a screen-reader-accessible layer over computer-use agents. Blind users invoke it with a global hotkey, issue natural-language commands, hear audio feedback after each action, and can review what the agent did — while it logs screenshots, UI trees, model responses, and action traces for analysis.
Why do computer-use agents fail for blind users?
The study's trace analysis finds recurring modes: acting on controls that don't exist or are mis-located (about 25% of unsuccessful traces), failing to discover controls hidden behind menus (~21%), following memorized procedures instead of the visible UI (~17%), dropping user constraints mid-task (~21% of partial completions), and stopping early or looping instead of recognizing completion (~18% of partials).
Was this the first study of computer-use agents with blind users in real conditions?
It is among the first real-world deployments. Prior accessibility-focused CUA work cited in the paper (e.g., the A11yCUA dataset and BLIND-ACT) relied mainly on benchmark or simulated conditions, whereas this study ran for three weeks on participants' own Windows machines.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All seven figures are present only as captions in the extraction; in-figure values (e.g., Figure 2a/2b distributions, Figure 4a/4b taxonomy proportions) could not be verified against the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.