Decision Hijacking: Prompt Injection Attacks on Jev's Typed Probabilistic Decisions
Tiantong Wu, Wei Yang Bryan Lim
cs.CR · cs.AI
Quick verdict
Registered study: injection shifts Jev's action probabilities but rarely hijacks its choices; rigorous work, but evidence rests on one proprietary model.
Summary
The paper asks whether indirect prompt injection transfers to Jev, a proprietary non-generative model (jev-1.13.0) that maps a state and a fixed action set to a typed choice plus reported probabilities instead of free-form text. The authors reconstruct InjecAgent's 510 direct-harm combinations as typed decisions in which the attacker controls only a designated slot of the latest tool result, then run four experiments totalling 54,060 calls. DH-1 shows original attack text raises mean attacker-target probability by 0.043 yet selects the target in only 1.8% of cases, and InjecAgent's override enhancement weakens the attack. DH-2's factorial ablation finds override markers reduce attacker probability (ignore-previous: -0.032). DH-3, using a registered SESOI of 0.02, finds contextual-relatedness assertions produce only small effects (+0.0030 for P2-N0). DH-4 grants an attacker black-box score feedback and a 72-configuration grammar optimizer: mean best-so-far probability doubles (0.043 to 0.084) while validated success, requiring three of five fresh calls, rises only from 1.8% to 3.5%. Exploratory analysis links successes to small initial margins or high attacker control over the observation. Key limitations: a single proprietary model version, no generative baseline on the same reconstruction, successes concentrated in one goal and context, small effect sizes, and no stated code release.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Malicious injected content reliably shifts Jev's reported action probabilities (mean +0.043 for the base InjecAgent attack) but selects the attacker target in only 1.8% of 510 cases.
- Injection-style override markers reduce rather than strengthen attacker influence in Jev (ignore-previous marker: -0.032), unlike their effect on generative agents.
- Assertions of contextual relatedness produce small positive probability effects (+0.0030 for P2-N0) that fall below the registered SESOI of 0.02.
- Black-box score feedback with a 72-configuration grammar optimizer doubles mean best-so-far attacker probability (0.043 to 0.084) but raises validated attack success only from 1.8% to 3.5%.
- Validated hijacking successes are associated, exploratorily, with small initial decision margins or high attacker control over the observation.
Strengths
Registered, pre-specified analyses
DH-3 hypotheses and design hash were recorded before analysis with a registered SESOI of 0.02 (Section 4.3, Appendix D); DH-2 pre-registers H1a/H1b/H2. This is well above the norm in prompt-injection evaluation papers.
Clean separation of probability shift vs validated hijacking
DH-4 screens on optimization calls but validates only on five fresh calls with a >=3/5 majority rule; case u16:a01 (pi-hat 1.0 at B=16, 0.4 at B=24, Appendix E.2) concretely demonstrates why screening successes are not validated successes.
Unusually thorough integrity reporting
Appropriate bootstrap for the crossed design
Honest claim scoping
Weaknesses
No generative baseline on the same reconstruction
Weak adaptive attacker bounds the DH-4 conclusion
Successes concentrated in two clusters
Tiny effect sizes with limited operational relevance
No code or data release; proprietary dependency
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether indirect prompt injection — established against generative LLM agents — transfers to Jev (jev-1.13.0, TypeSafe AI), a proprietary non-generative model that maps a state plus a fixed action set to a typed choice and reported probabilities. The authors define *decision hijacking*: attacker-controlled content shifting p(a|s,A) toward attacker-target actions within the allowed set, with no undeclared output possible. Using a deterministic reconstruction of InjecAgent's 510 direct-harm cases (30 attacker goals x 17 user contexts, Section 3.1), where only a designated slot of `latest_tool_result` is attacker-editable, they run four experiments (DH-1 to DH-4, 54,060 calls; Table 1). Central claims: (1) attacks shift probabilities (+0.043 mean) but rarely select the target (1.8%); (2) override markers reduce influence (A2-A0 = -0.032); (3) relatedness assertions fall below the registered SESOI of 0.02 (P2-N0 = +0.0030); (4) score-feedback optimization doubles best-so-far probability (0.043 to 0.084) while validated ASR rises only to 3.5%; (5) successes associate with small margins or high attacker control (exploratory).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'decision hijacking' in this paper?
It is the authors' term for prompt injection against a non-generative decision model: attacker-controlled text shifts the probability distribution over a fixed, allowed set of actions toward an attacker-chosen action, even though the model can never output anything outside the declared action set.
Is Jev vulnerable to prompt injection?
Largely not at the action-selection level, per this study: across 510 reconstructed InjecAgent cases, original attack text shifted reported probabilities (mean +0.043) but selected the attacker's target in only 1.8% of cases, rising to 3.5% after adaptive optimization with score feedback. Consistent probability shifts mean the risk is changed, not eliminated.
What is InjecAgent and how was it reused here?
InjecAgent (Zhan et al., 2024) is a public benchmark of indirect prompt injections against tool-using LLM agents. The authors rebuilt its 510 direct-harm cases as typed decisions where the attacker may edit only the malicious slot in the latest tool result, preserving user goals, attack text, and attacker-target tools.
Do 'ignore previous instructions' style attacks work better on Jev?
No, the opposite: adding the classic override marker lowered attacker-target probability by 0.032 on average, and InjecAgent's enhanced (override-prefixed) attacks were weaker than the plain attacks (0% vs 1.8% selection rate), unlike findings reported for generative agents.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-8 are available only as extracted captions and axis labels; visual details (error-bar overlap, ECDF shapes) were judged from in-text numbers.
- This is arXiv v1 (23 Sep 2026), apparently not yet peer-reviewed; the review reflects the provided text only.
- Jev is a proprietary commercial product; vendor independence of the evaluation could not be verified from the text.
- The reference list is short (13 entries); related-work coverage outside the cited lines was not assessable from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.