AgentHijack: Visual Patch Attacks on Multimodal Computer-Use Agents
Zhihao Liu, Hongyu Sun, Zhiyuan Fu, Xiaonan Duan, Jice Wang, Shangru Zhao, Weizhi Meng, Wuxin Yang, et al.
cs.CR · cs.AI · cs.CY
Quick verdict
Trained visual patches make open CUAs run real commands (E2E-ASR 20.3%, n=600); layered metrics solid, but white-box, single-run design limits scope.
Summary
The paper presents AgentHijack, an end-to-end evaluation framework testing whether gradient-optimized visual patches embedded in web pages can drive computer-use agents (CUAs) to execute terminal commands with verifiable environmental effects. Patches (700x500, L-infinity budget of about 32/255) are trained per-backend (train-on-target) on author-controlled GitHub Pages screenshots and a locally deployed CSDN clone, then evaluated online across five open-source stacks: prompt_agent over Qwen3.5-4B/9B, Aguvis-7B, EvoCUA-8B, and UI-TARS-1.5-7B. The factorial suite (15 CIA-oriented objectives x 2 pages x 4 benign tasks = 120 cases per backend, 600 total) is scored at three layers: T-ASR 84.5% (507/600), parser acceptance TAPR 47.0% (282/600), and end-to-end environment success E2E-ASR 20.3% (122/600). Qwen3.5-4B is most vulnerable (E2E 55.8%) while UI-TARS-1.5-7B hits 100% text-level but 0% end-to-end, illustrating the value of layered metrics. Controls (random patch, clean patch, 3x3 position grid: 11/12 at the trained coordinate vs 0/96 shifted) support content and location specificity. Limitations: each case runs once with no variance estimates or statistical tests; the train-on-target, fixed-position setting sidesteps transfer and robustness questions; all benign tasks already require terminal use; no proprietary agents are tested; payloads are withheld and no code is released, so independent replication is difficult.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Gradient-trained local visual patches can survive the full CUA pipeline - screenshot, VLM generation, action parsing, execution - producing verifiable environment changes at 20.3% E2E-ASR across 600 online cases (Table 3).
- Text-level success does not equal harm: UI-TARS-1.5-7B attains 100% T-ASR but 0% E2E-ASR, which the authors argue justifies the layered T-ASR/TAPR/E2E-ASR metric scheme.
- The attack depends on learned content and precise spatial placement: random and clean patches fail to trigger, and shifting the patch to eight nearby positions drops success from 11/12 to 0/96 (Section 7).
- General-purpose open-source VLMs behind a prompt adapter (Qwen3.5-4B/9B) inherit visual environmental-injection risk, reaching 55.8% and 14.2% E2E-ASR respectively.
Strengths
Layered metric decomposition is genuinely diagnostic
T-ASR/TAPR/E2E-ASR (84.5%/47.0%/20.3%, Table 3) localize loss to generation, parsing, or execution - e.g., UI-TARS-1.5-7B collapses from 100% text hits to 0% environment success, a pattern a single ASR number would obscure.
Real online execution with an auditable evidence chain
Section 4.4 and Appendix B store VLM outputs, parsed actions, execution traces, screenshots, and VM-state checks per case, and environment success requires G_adv on the final VM state rather than text agreement.
Strong control battery supporting causal attribution
Transparent, internally consistent experimental design
Clearly bounded threat model and ethics posture
Weaknesses
No statistical inference over a 600-case design
Most-favorable attacker setting inflates severity
Utility claim unverifiable from reported numbers
Low reproducibility
Thin empirical positioning against closest prior work
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
AgentHijack (arXiv 2609.09212) is an attack-measurement study asking whether a gradient-trained visual patch placed on a web page can push a computer-use agent (CUA) beyond text manipulation into verifiable environment modification. The authors claim three things (Sections 1, 10): (1) trained patches survive the full screenshot-to-VLM-to-parser-to-execution chain, reaching 20.3% end-to-end success over 600 online cases; (2) single ASR-style numbers misrepresent risk, motivating the layered metrics T-ASR / TAPR / E2E-ASR (84.5% / 47.0% / 20.3%); (3) even general-purpose VLMs behind a prompt adapter inherit this risk. The threat model (Section 3.3) is explicitly white-box: train-on-target per backend, a fixed 700x500 patch at a known position, one author-controlled page region; no system-prompt, parser, or evaluator tampering.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AgentHijack's reported attack success rate?
Across 600 online cases on five open-source computer-use-agent stacks, 84.5% of runs produced the attacker's target string in the VLM output (T-ASR), 47.0% were accepted by the action parser (TAPR), and 20.3% (122/600) caused the intended environment change (E2E-ASR). The most vulnerable stack, Qwen3.5-4B behind a prompt adapter, reached 55.8% end-to-end.
How does the visual patch attack work?
A gradient-optimized visual patch (700x500 pixels, perturbation budget of about 32/255) is placed at a fixed position on an attacker-controlled web page. When a computer-use agent screenshots that page during an otherwise benign task, the patch induces the VLM to emit a terminal command (for example copying an SSH key or appending to .bashrc), which the agent then parses and executes in its environment.
Is this a real-world threat or a lab-only result?
The paper's own experiments are confined to author-controlled pages and isolated virtual machines, and use a train-on-target setting: the patch is gradient-trained against the exact target model and placed at a known position. The authors argue this models realistic risk (agents visiting web pages), but transfer to unknown pages, other positions, or closed commercial agents is untested, and their own control shows the attack fails entirely when the patch is shifted.
Does the paper release code, data, or payloads?
No code or data release is stated anywhere in the text. The full payload commands are deliberately withheld for safety (Appendix A), and task manifests are described as stored locally, which makes independent replication of the evaluation difficult.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are diagrams/screenshots whose visual content cannot be verified from the extracted text; only their captions were available.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.