ElderBench: Benchmarking Autonomous Mobile Agents for Older Adults
Weide Zhan, Qumu Shaqu, Yuanqing Liu, Peng Zhang, Jiahao Liu, Kam Him Lam, Ning Gu, Zhan Hu, et al.
cs.AI
Quick verdict
First elderly-elicited mobile GUI benchmark with novel data and solid normalization study (+23-24 pts); judge validation, release, and stats gaps remain.
Summary
ElderBench is presented as the first benchmark evaluating mobile GUI agents on instructions naturally elicited from older adults rather than written to benchmark conventions. The authors collected 249 valid tasks via semi-structured interviews with 28 Chinese older adults (aged 59-84) across 20 Android apps, filtering 2 of 251 candidates. Linguistic analysis against MobileWorld instructions reports stark divergence: 78.31% of elderly instructions are under 20 characters, 47.80% elliptical or narrative, with 19.28% indirect speech, 12.85% referential ambiguity, and 38.15% under-specification. Evaluation covers 130 online tasks (closed-loop Android with a VLM-as-judge) and 119 offline tasks (human-demonstrated task graphs). No model exceeds 50% overall success: Gemini-3-Flash tops VLMs at 49.80%, GUI-Owl-1.5 tops agents at 45.38%. A stratified 100-task normalization study shows rewriting instructions into benchmark style raises successes from 37 to 61 (AutoGLM) and 23 to 46 (Qwen3-VL-Flash), significant by exact McNemar (p<0.001). Failure analysis of 63 AutoGLM online failures attributes 57.1% to intent/capability boundaries and 33.3% to planning/state tracking. Main limitations: the judge was validated on one agent only, online/offline subsets are not task-matched, feature-correlation and factorization-machine analyses lack significance testing or validation, and no public release of data or code is stated; the cohort is a small Chinese-speaking sample.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ElderBench is the first benchmark of mobile GUI agents built from naturally elicited older-adult instructions (249 tasks, 20 apps, 28 interviewees).
- Elderly instructions diverge sharply from benchmark language: 78.31% are under 20 characters, with 19.28% indirect speech, 12.85% referential ambiguity, and 38.15% under-specification.
- Current agents and VLMs degrade substantially on these instructions — no model exceeds 50% overall success rate (best: Gemini-3-Flash 49.80%, GUI-Owl-1.5 45.38%).
- Instruction normalization alone raises successes by +24/+23 tasks (exact McNemar p<0.001), showing linguistic mismatch is a major failure source alongside planning deficits.
- 57.1% of AutoGLM's online failures stem from intent-understanding/capability-boundary errors rather than low-level visual grounding (3.2%).
Strengths
Authentic elicited data, not synthetic rewrites
249 instructions collected via semi-structured interviews with 28 older adults (59-84) with no task templates or linguistic examples (App. B.1), 251 filtered to 249 by three authors. Unlike Mobile-Bench-v2's slot-removal approach (§2.2), ElderBench is the first with target-user elicitation (Table 1), preserving genuine indirect speech, disfluency, and referential ambiguity.
Controlled normalization with formal statistics
§5.3/Table 3 fixes tasks, device states, and protocols and rewrites only the wording on a stratified 100-task subset: AutoGLM 37→61 successes, Qwen3-VL-Flash 23→46, exact McNemar p<0.001, paired-bootstrap 95% CIs [14,34] and [12,34] points — cleanly attributing a large share of failures to linguistic formulation rather than task difficulty.
Multi-dimensional, well-documented evaluation design
Actionable failure taxonomy
Weaknesses
Judge validation is thin and partially circular
Online/offline results are not comparable and the gap is unquantified
Attribution analyses lack inferential support
No release statement for data, code, or task graphs
Narrow population and single linguistic baseline
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ElderBench (arXiv 2609.04850, cs.AI) claims to be the first benchmark evaluating mobile GUI agents on instructions *naturally elicited* from older adults, rather than the explicit, goal-oriented commands used by AndroidWorld, MobileWorld, or AndroidLab. Three research questions structure the paper (§1): how elderly instructions differ linguistically (RQ1), how well current agents/VLMs execute them (RQ2), and which linguistic and execution-level factors cause failures (RQ3). The artifact is 249 tasks from 28 interviewees across 20 Android apps. The headline empirical claims are: a severe capability ceiling (no model above 50% overall success, Table 2), a +24/+23-task gain from merely rewriting instructions into benchmark style (§5.3, Table 3), and a failure taxonomy dominated by intent understanding (57.1%, Table 4). On the presented evidence, the core claims are plausible; the evaluation methodology carries the caveats below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ElderBench?
ElderBench is a benchmark for evaluating autonomous mobile GUI agents on smartphone tasks phrased the way older adults actually speak. It contains 249 naturally elicited task instructions collected through interviews with 28 Chinese older adults (aged 59-84), spanning 20 Android applications, with both online (live Android interaction) and offline (task-graph-based) evaluation protocols.
How well do current AI agents perform on ElderBench?
No evaluated model exceeds 50% overall task success. The best VLM, Gemini-3-Flash, reaches 49.80% and the best GUI agent, GUI-Owl-1.5, reaches 45.38%. When instructions are rewritten into standard benchmark style, success counts rise sharply (AutoGLM from 37 to 61 out of 100 tasks), showing the language gap itself causes many failures.
Why do GUI agents fail on elderly users' instructions?
The paper's analysis of 63 failed AutoGLM runs on the online subset finds 57.1% are intent-understanding or capability-boundary errors (e.g., treating 'the sound is too low' as a question instead of a volume-adjustment request) and 33.3% are planning/state-tracking errors; low-level visual or tap errors account for only 3.2%.
Is ElderBench's data or code publicly available?
The paper text provided does not state any public release of the dataset, task graphs, or evaluation code; it references supplementary materials but gives no repository URL or license. Readers should check the arXiv listing or the authors' pages for an updated availability statement.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Numbers printed inside Figures 2-6 are garbled by PDF extraction (e.g., Figure 4's interleaved '73.7 112.28'); figure-level statistics here rely on the in-text reporting instead.
- The paper text lists Tun Lu as a ninth author; the supplied metadata lists only eight authors.
- Supplementary items referenced as 'complete distributions' are present only in condensed appendix figures/tables; per-task data cannot be verified from this extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.