Context Inference Attacks Without Jailbreaks
Prince Jha, Samuele Poppi, Nils Lukas
cs.CR · cs.LG
Quick verdict
Formalizes context inference on LLM agents via benign queries; attack results are strong and well-staged, but tested defenses are narrow and targets synthetic.
Summary
This paper introduces context inference attacks: determining whether a specific record entered an LLM agent's hidden inference-time context using only benign queries, with no jailbreak and no disclosure. The authors formalize a security game across three threat models of decreasing attacker knowledge — known context (TM1), unknown template and candidates (TM2), and agent-retrieved context (TM3, where records arrive via the agent's own tool calls) — and run one replay-based attack unchanged across all three: responses to K benign queries are scored by per-token NLL under candidate contexts built from attacker-held records, using the target model (grey-box) or a same-family surrogate (black-box), with offline query selection. Against Qwen2.5 models, LLaVA and Qwen2.5-VL VLMs, and hosted agents (Kimi-K2.6, Qwen3.5-35B-A3B), the attack reaches 100% ASR on small candidate sets and 63% at 1,024 candidates (TM1), 78.9 AUROC grey-box and 92.5 AUROC when a 14B surrogate scores a 32B target (TM2), and 81.8 AUROC against agent-retrieved contexts after filtering uninformative queries (TM3), while a GPT-4o-mini judge stays near chance. Instruction and logit-suppression defenses eliminate direct leakage (excess BLEU ≤ 0.0021) without stopping inference; only context dilution and shorter responses reduce it. Main limitations: synthetic targets, single-seed TM2 without confidence intervals, within-family surrogates only, mandatory full retrieval in TM3, and a narrow defense set.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Hidden inference-time context measurably changes LLM/VLM response distributions under benign queries alone, enabling context inference without any jailbreak or disclosure.
- A single replay-based NLL-scoring attack, unchanged across settings, achieves 100% ASR on small candidate sets (63% at |Z|=1024) in TM1, 78.9 AUROC grey-box and 92.5 AUROC with a same-family 14B surrogate scoring a 32B target in TM2, and 81.8 AUROC against agent-retrieved contexts in TM3.
- Disclosure-blocking defenses (instructions, logit suppression) hold excess BLEU overlap at or below 0.0021 yet leave likelihood inference intact; only context dilution and shorter responses reduce leakage, and no tested control eliminates it.
- Under the tested judge (GPT-4o-mini), the leakage is non-semantic: responses that support 92.5 AUROC leave the judge near chance throughout.
Strengths
Staged threat models make the agentic result interpretable
TM1→TM3 (Section 4) progressively remove attacker knowledge and change delivery while holding the attack fixed; the paper explicitly argues a null result in TM3 would otherwise be uninterpretable, and Definition 4.1 grounds all settings in one security game.
Direct-leakage audit confirms the defenses actually block disclosure
Table 1: mean excess BLEU ≤ 0.0021 (negative for one model), peak present–absent gap ≤ 0.006 across TM1–TM3 at N=10, K=5, so the attack results in Sections 6.2–6.4 were obtained against models that disclose nothing verbatim.
Unusually candid failure accounting
Consistent non-semantic baseline across every setting
Practical black-box and deployment-facing instantiation
Weaknesses
TM2 statistical evidence is thin
The non-semantic-leakage claim rests on one judge
Defense set is narrow relative to the stated threat
Synthetic and constrained targets limit external validity
Surrogate and query-pool caveats in the strongest numbers
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.01663, cs.CR/cs.LG; MBZUAI) claims that hidden inference-time context — records, credentials, retrieved pages assembled by an LLM agent — leaks through response *distributions* rather than through disclosure, and can be detected with benign queries alone. Contributions: a security game (Definition 4.1) spanning three staged threat models (TM1 known context, TM2 unknown template and candidates, TM3 agent-retrieved context), a single replay-based likelihood attack (Algorithm 1) with offline query optimization (Algorithm 2), and an evaluation across Qwen2.5 LLMs (1.5B–72B), LLaVA/Qwen2.5-VL VLMs (3B–32B), and hosted agents (Kimi-K2.6 via Fireworks, Qwen3.5-35B-A3B via DeepInfra, accessed August 2026). Headline numbers: 100% ASR on small candidate sets and 63% at |Z|=1024 (TM1); 78.9 AUROC grey-box and 92.5 AUROC with a 14B surrogate scoring a 32B target (TM2); 81.8 AUROC on TM3 after query filtering; up to 99% ASR on VLMs; a GPT-4o-mini judge near chance everywhere.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a context inference attack?
An attack that determines whether a specific record (a document, credential, or page) was placed in an LLM agent's hidden inference-time context by asking only ordinary, benign questions and statistically scoring the responses under candidate contexts — the model never discloses the content, and likelihood signals reveal its presence.
How is this different from prompt extraction or membership inference attacks?
Prompt/RAG extraction makes the model output the hidden content verbatim; this attack only detects presence. Unlike prior membership inference attacks, whose queries are built from the candidate being tested, these queries are fixed in advance, carry no adversarial instruction, and the leakage cannot be read from the response text by a GPT-4o-mini judge.
Does the attack work against black-box APIs without logit access?
Yes, when a surrogate model from the same family scores the responses: a 14B surrogate reached 92.5 AUROC against a 32B target. Cross-family transfer is much weaker (52% ASR at a 4-candidate set for the tested LLM pair), so an aligned surrogate is a practical requirement, not any model suffices.
What defenses work against context inference attacks?
Of the controls tested, only context dilution (more records) and shorter response caps reduce leakage, and neither eliminates it. Instructions and logit suppression stop direct quoting (excess BLEU ≤ 0.0021) but leave the statistical signal intact, so hidden context should not be treated as private merely because it is not quoted.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Values shown only inside figures (exact curve shapes and interval widths in Figures 2–7) could not be verified from the extraction; all numbers cited in this review come from the captions, tables, and prose, which are present in the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.