MobileCybench: Evaluating Agent Vulnerability Discovery via Executable Probes
Andy K. Zhang, Ava Huang, Joey Ji, Wai Han, Thomas Qin, Nardos Demilew, Michael Tian-Yue Liu, Brian Song, et al.
cs.CR · cs.AI · cs.LG
Quick verdict
MobileCybench scores agent-found Android vulnerabilities via executable probes; well-designed and honest, but 13 apps, no significance tests, refusal confounds
Summary
MobileCybench introduces probes—executable checks of application-specific security properties (confidentiality, integrity, availability, access control)—for scoring exploits that AI coding agents find in Android apps. Because probes encode properties rather than known vulnerabilities, they can score exploits of bugs unknown when the probes were written; a post-hoc attribution step replays exploits against vulnerable and patched builds to map triggers to reference vulnerabilities. The benchmark covers 13 Android applications with 495 author-written probes, and evaluates 5 coding agents (OpenCode with GPT-5.5, GPT-5.6-Sol, and GLM-5.2; Claude Code with Opus 4.8 and Opus 5) under four settings: malicious-app versus remote-attacker, each with obfuscated APK only or with source code, with two runs per configuration (500 runs total; ~$6,973 API cost). The top agent, OpenCode/GPT-5.6-Sol, triggered probes in 53.8% of applications in the APK-only malicious-app setting and 16.7% in remote-attacker; source access raised the overall trigger rate from 28.8% to 32.8%. Benchmark construction surfaced 23 previously unreported vulnerabilities, 12 confirmed by maintainers (7 patched, 5 acknowledged) with 6 CVEs. Key limitations: only 13 apps and no statistical significance testing; sharply asymmetric safety-refusal rates (42.4% of Claude Code/Opus 5 runs, 25.0% Opus 4.8, 0% OpenCode) and differing cyber-authorization tiers confound cross-agent comparison; and APK-only does not rule out memorization, as the authors' own contamination audit shows.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Probes — executable checks of application-specific security properties — can score exploits of vulnerabilities that were unknown when the probes were written (§1, §2.2).
- The benchmark and its evaluation surfaced 23 previously unreported vulnerabilities, 12 of which maintainers confirmed (7 patched, 5 acknowledged) and 6 of which have public CVEs (§1, Appendix D).
- Agent capability is highly attack-setting-dependent: the top agent (OpenCode/GPT-5.6-Sol) triggers probes in 53.8% of applications APK-only as a malicious app but only 16.7% as a remote attacker (§4.2).
- Source-code access yields a modest trigger-rate gain (28.8% → 32.8% across all agents and settings) but several new distinct vulnerabilities (§4.2, Fig. 4).
- Attribution resolves triggered configurations to 19 of 24 attributable reference vulnerabilities, with only partial overlap across agents — 3 of 5 agents each detect a vulnerability the others miss in APK-only runs (§4.2).
Strengths
Property-based scoring decouples evaluation from known vulnerabilities
Probes encode security properties (CIAA) rather than specific bugs (§2.2), so a probe written before a bug is known can still score it — demonstrated by the Home Assistant location probe crediting CVE-2026-54318 (§D.6). Attribution (§2.3) adds vulnerable-vs-patched differential replay to map triggers to 19 of 24 reference vulnerabilities.
Real-world impact: 23 previously unreported vulnerabilities
Benchmark construction and agent runs surfaced 23 findings (§1, contribution 5); 12 maintainer-confirmed (7 patched, 5 acknowledged), 6 with public CVEs — external validation that the benchmark tasks reflect genuine exploitability, not synthetic bugs.
New attack surface for agent benchmarks: Android device-level channels
Unusually candid contamination and refusal auditing
Full cost and infrastructure accounting
Weaknesses
Tiny application sample with no statistical treatment
Safety-refusal and authorization asymmetry confounds cross-agent comparison
Memorization contamination only bounded, not eliminated
Probe sensitivity/specificity never quantified
Scaffold-version and endpoint instability acknowledged but unaddressed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MobileCybench proposes evaluating agent-discovered vulnerabilities with *probes*: executable checks of application-specific security properties (confidentiality, integrity, availability, access control — CIAA) that trigger when a replayed exploit violates them (§2.2). Because probes encode properties rather than known bugs, they can score exploits of vulnerabilities unknown at authoring time; an attribution step (§2.3) maps triggers to 24 reference vulnerabilities via vulnerable-vs-patched differential replay. The paper claims a 13-app / 495-probe benchmark, a 5-agent × 4-setting evaluation, and 23 previously unreported vulnerabilities (12 maintainer-confirmed, 6 CVEs). Claims are mostly well-supported by internal evidence; the related-work positioning (§5) is not in the extracted text, so external novelty is judged from the abstract's contrast with vulnerability-specific benchmarks (BountyBench, Cybench) and universal-property harnesses.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MobileCybench and what does it measure?
MobileCybench is a benchmark that measures how well AI coding agents can discover real vulnerabilities in Android applications. Instead of checking agents against a fixed list of known bugs, it uses 'probes' — executable checks of security properties (such as 'no other app may set the recorded phone location') — and triggers a probe when an agent's exploit actually violates that property in a clean replay environment.
How well did AI agents perform on MobileCybench?
Performance was highly setting-dependent. The best agent, OpenCode with GPT-5.6-Sol, triggered probes in 53.8% of the 13 applications when acting as a malicious app with only an obfuscated APK, but only 16.7% when acting as a remote attacker. Giving agents the app's source code raised the overall trigger rate across all agents from 28.8% to 32.8%.
Did the benchmark find any real, new vulnerabilities?
Yes. Building and running MobileCybench surfaced 23 previously unreported vulnerabilities in the 13 tested Android apps. 12 were confirmed by the apps' maintainers (7 patched, 5 acknowledged), and 6 received public CVE identifiers; the paper reports each was responsibly disclosed.
Is MobileCybench's code and data available?
The paper states that all code, target environments, and probe suites are available at github.com/bountybench/mobilecybench. Note that re-running the evaluation requires Android emulators, application backends, and paid API access to the five frontier coding agents.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extracted text has omitted sections (including Section 5 related work, parts of C.3/C.4, and most of Appendix D), so some judgments — e.g., benchmark novelty positioning — rest on abstract-level claims only.
- Figures (probe schematic, cost-frontier plots, attribution grid glyphs) are only partially recoverable from text; numeric citations rely on the prose and tables that survived extraction.
- Fictional/future model names and CVE IDs (GPT-5.6-Sol, Opus 5, CVE-2026-54318) are reproduced as stated in the text and were not externally verified per task rules.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.