Structurally Close, Temporally Distant: Measuring Security Exposure in Long-Horizon LLM Agents
Md Jafrin Hossain, Nur Al Hasan Haldar
cs.CR · cs.AI · cs.CL · cs.LG
Quick verdict
Influence distance shows step count overstates attack proximity in agent logs (96.9% decoupled); careful and honest, but single-benchmark scope limits impact.
Summary
The paper argues that measuring long-horizon LLM agent attacks by execution-step count overstates how distant malicious input is from sensitive actions. The authors build a provenance-aware execution graph over trajectory logs with four deterministic edge classes (state write/read joins, typed identifier references, batch provenance, and same-tool call/result pairs) and define influence distance D_I as the shortest structural path from an untrusted source to a sensitive action, compared against sequence distance D_T in the ordered trajectory; the gap G = D_T − D_I is computed with no model in the loop. On 454 injection–sink pairs from 360 AgentDojo long-horizon trajectories across gpt-4o-mini, gpt-4o, Haiku 4.5, and Sonnet 4.6, 96.9% of pairs decouple (median gap 9 hops), and 91.0% remain decoupled after ablating the largest edge class. A banking-suite cross-check shows decoupling drops to 33.8% where trajectories are short (median D_T 5 vs 23) and memory-based provenance is absent. A logistic regression on 274 OpenAI pairs finds the gap does not independently predict attack success (β=0.066, p=.088), and a deterministic pre-execution gate at k=2–3 adds five blocked attack sinks with zero extra benign blocks — a gain that is not statistically significant (McNemar p=.0625). Limitations: a single benchmark family, code not yet released, and headline rates that depend on the contestable same-tool edge class.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Step count systematically overstates separation between malicious input and sensitive actions in stateful agent trajectories: 96.9% of 454 AgentDojo long-horizon injection-sink pairs have a shorter structural than transcript path (median gap 9 hops), and 91.0% remain decoupled after ablating the largest edge class.
- Influence distance D_I is computable deterministically from execution logs alone — via state write/read joins, typed identifier matching, batch provenance, and call/result joins — with no language model, embedding, or causal inference in the loop.
- The structural gap does not independently predict attack success after controlling for sequence distance, attack family, and backend (β_G=0.066, p=.088, n=274 OpenAI pairs) — a null the paper reports explicitly.
- A deterministic D_I-based pre-execution gate at k=2–3 blocks five additional attack sinks with zero additional benign blocks, though the paired gain is not statistically significant (McNemar p=.0625).
- Decoupling is mechanism- and length-dependent: in the banking suite only 33.8% of 231 pairs decouple, where trajectories are short (median D_T 5 vs 23) and no save-and-retrieve memory pattern exists.
Strengths
Honest treatment of negative and borderline results
Section 6 reports the gap coefficient as non-significant (β_G=0.066, p=.088, Table 3) and Section 7 flags the gate gain as not significant at 5% (exact McNemar p=.0625 on five discordant pairs); §3.5 and §6 repeatedly separate measurement from causal attribution.
Genuine robustness ablation of the weakest edge rule
The conservative graph dropping same-tool call/result edges (§5.2) moves decoupling from 96.9% to 91.0% and median gap from 9 to 3 (Table 9), and the gate result at k=2–3 is identical under both variants (Table 4) — the claim survives the harshest ablation.
Circularity safeguards are designed in, not bolted on
Cross-domain check with explicit scope honesty
Reproducibility documentation above the norm
Weaknesses
Core theory is trivially true by construction
Single benchmark family limits external validity
Both inferential results miss the significance bar and no power analysis is offered
Headline numbers lean on the most contestable edge class
Editing quality is below par
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses a measurement question in LLM-agent security: when analyses quantify indirect prompt injection risk by the number of execution steps between malicious input and a sensitive action, do they overstate separation in stateful agents? The authors formalize three conflated notions — temporal remoteness, structural exposure, exploitability (§3.2, C1) — and build a provenance-aware execution graph with four deterministic edge classes (state write/read joins, typed identifier references via source_ids, batch provenance, same-tool call/result; §3.3). Influence distance D_I is the shortest structural path from untrusted source to sensitive sink; sequence distance D_T is the transcript index difference; the gap G = D_T − D_I (§3.4). Central claims: (1) decoupling is pervasive — 96.9% of 454 pairs (§5.1); (2) the gap is computable with no model in the loop (C2); (3) the gap does not independently predict attack success (§6, Table 3); (4) a D_I-based gate adds coverage at matched thresholds (§7). The paper is disciplined about what G is not: not causal attribution, not a vulnerability score (§3.5, §6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is influence distance in this paper?
It is the length of the shortest path from an untrusted (potentially injected) input to a sensitive action in an execution graph built from the agent's trajectory log. Edges record deterministic provenance relationships: a state write joined to a later read of the same key, a tool-result identifier appearing in a later tool call, tool calls issued in one assistant batch, and a tool call joined to its result. It is compared with sequence distance (the position gap in the transcript), and their difference — the structural gap — shows how much closeness the step count hides. No language model is used in the computation.
What are the paper's main findings?
On 454 injection-to-sink pairs from 360 AgentDojo long-horizon trajectories across four models (GPT-4o-mini, GPT-4o, Claude Haiku 4.5, Sonnet 4.6), 96.9% of pairs have shorter structural paths than transcript paths (median 9 hops), and 91.0% remain after removing the largest, most contestable edge class. However, the gap does not independently predict attack success (p=.088), and a distance-based pre-execution gate at thresholds 2–3 adds five blocked attacks with no extra benign blocks — a gain that is not statistically significant (p=.0625).
Does a bigger structural gap mean attacks are more likely to succeed?
No — the paper tested this and did not find it. In a logistic regression over 274 OpenAI-model pairs controlling for sequence distance, attack family (immediate vs delayed injections), and backend, the gap coefficient was small and non-significant (β=0.066, p=.088). The authors explicitly frame the gap as a measure of candidate influence pathways, not a vulnerability score, and note the result may reflect limited statistical power.
Is code or data available for this paper?
Not yet. The paper states the code will be released publicly upon acceptance, and the trajectories or pair-level analysis table are not stated as released. The analysis pipeline is specified in enough detail (Appendix E) to reimplement on top of the public AgentDojo framework, which provides the execution harness and logging the study builds on.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The abstract as supplied in the metadata is cut mid-word ('gate bloc'), but the complete abstract is present in the embedded full text, so no content is missing from the review basis.
- Page-break artifacts (stray page numbers and values split across pages, e.g. the '93.3%]' continuing Table 2's confidence interval) are PDF-extraction noise, not missing content; references and appendices A–H appear complete.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.