ASLEval: Measuring Privacy Exposure Displacement in LLM Agent Sessions
Guosen Wu, Huizhen Huang, Guoxiong Long, Tao Huang, Chen Hou
cs.CR · cs.AI
Quick verdict
ASLEval shows local privacy proxies miss ~47% of LLM-agent session exposure; rigorous protocol, limited by synthetic environments and small samples.
Summary
The paper introduces privacy exposure displacement: the gap between a local privacy-evaluation proxy (a guarded action, final answer, or attacker report) and target-grounded exposure across a requester-visible agent session. ASLEval operationalizes it as an authorization-aware protocol: an evaluator pre-registers a hidden target entity set, an authorization relation, an expected outlet, and the declared visible exits; a target-blind probe agent interacts with a black-box target agent under fixed budgets; a deterministic adjudicator maps all observations to the target set, counting only requester-visible/external exits as exposure while retaining internal tool traces as diagnostic path evidence. Across three synthetic enterprise-style environments (PrivacyInAction, AgentDojo Workspace, WorkBench), two independently implemented runtimes (Agent-R, Agent-F), and DeepSeek/GLM models, the expected-outlet view misses 46.9% of exposure recovered by the visible-exit union; an email guard cuts email coverage (-0.548) while session coverage stays near 1.0; attacker self-reports omit 23.9% of exposure with precision as low as 0.148; and schema-aligned internal evidence precedes visible exposure in 76.3% of 6,124 entity-sessions. Minimizing model-visible returns cuts overlap 45.7% but collapses task success from 0.800 to 0 on a 15-task slice. Human validation (kappa 0.747, F1 0.891) supports adjudication, with C2 candidate slices hardest (F1 0.762). Main limitations: synthetic data, 10-session C1 conditions, no stated artifact release.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Local evaluation proxies recover materially different portions of the same target-grounded exposure, with the expected-outlet view missing 46.9% of what the declared visible-exit union recovers (Table 3).
- An outlet-local guard improves its audited channel (email coverage -0.548, p=0.0312) while session-level exposure stays near saturation across two independent runtimes (Finding C1, Table 4).
- Attacker self-reports behave as noisy predictions, combining omissions (up to 23.9% of confirmed exposure unreported) with high false discovery (precision 0.148-0.429 across conditions, Table 5).
- Schema-aligned internal evidence is associated with, and temporally precedes, visible exposure at the request/probe level in 76.3% of matched entity-sessions, which the authors correctly present as association rather than causal mediation (Finding C3, Table 6).
- Minimizing model-visible tool returns reduces overlapping visible entities by 45.7% but eliminates deterministic task success (0.800 to 0) on a 15-task slice, so it is a mechanism probe, not a cost-free defense.
Strengths
Well-motivated evaluation-validity framing
The C1/C2/C3 displacement taxonomy (Sec. 1, Fig. 1) targets a real gap: Table 3 re-analyzes 245 identical exported sessions under five observation rules and shows coverage ranging from 0.026 (successful-action-only) to 0.887 (visible union), so the choice of proxy, not the agent, drives headline numbers.
Cross-runtime replication with matched conditions
C1 is replicated in two independently implemented runtimes (Agent-R on OpenAI Agents SDK, Agent-F nanobot JSON-schema runtime) holding model, workspace, target set, and guards fixed (Table 4): with the email guard on, email coverage drops to 0.317/0.344 while session coverage stays 0.955/1.000 and OLF is 0.683/0.656, with displacement in 7/10 and 8/10 sessions.
Explicit causal humility with claim-graded evidence
Quantified privacy-utility frontier instead of a defense claim
Human validation of the adjudication pipeline
Weaknesses
Small per-condition samples limit the replication claims
C2 numbers are entangled with adjudicator error
Fully synthetic environments, weakened external validity
No artifact availability statement
Unclear provenance of the 245-session re-analysis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is privacy exposure displacement in LLM agent evaluation?
It is the mismatch between what a local privacy-evaluation proxy measures (for example, a guarded email action or the final response) and the actual unauthorized exposure of pre-registered sensitive targets anywhere in the requester-visible session. The paper shows a guard can clean one channel while the same sensitive entities remain visible through console feedback, so the local proxy understates real session exposure.
How does ASLEval measure agent privacy?
Before a session, ASLEval fixes a hidden set of sensitive target entities, an authorization relation stating who may receive them, an expected outlet, and the set of declared visible exits (email, file share, final response, console). A blind probe agent then interacts with the target agent, and afterwards an adjudicator checks every visible observation against the target set. Internal tool arguments and returns are kept only as diagnostic path evidence and are not counted as exposure.
How is ASLEval different from AgentDAM, Privacy in Action, or AgentLeak?
AgentDAM asks whether private-data use was task-necessary; Privacy in Action judges whether disclosures are contextually appropriate; AgentLeak tracks leakage across final and internal channels of multi-agent stacks. ASLEval instead tests whether a local proxy fully represents unauthorized exposure of a fixed, pre-registered target set across the entire declared visible boundary, comparing local views against a visible-exit union on identical session logs.
Does the paper propose a defense against agent privacy leakage?
No. It deliberately tests one candidate mitigation, minimizing the content of model-visible tool returns, and finds it cuts measurable overlap by 45.7% but destroys task success (from 0.800 to 0) while multiplying tool calls, latency, and cost. The authors conclude that tool-interface defenses should be reported as privacy-utility frontiers rather than as leakage reductions alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only through captions; visual details such as the CI spreads and per-session points in Figure 3b could not be verified.
- The appendix (full run matrices, seeds, bootstrap intervals, FGE/GLM/AgentDojo C1 results, and offline top-k analyses) is not included in the extracted text, so several supporting claims rest on the main text alone.
- The deterministic adjudication and alias-canonicalization rules are described only at a high level in Section 3.3, limiting assessment of matching validity.
- Related work includes 2026-dated arXiv entries (e.g., 2602.11510, 2603.22751) that could not be verified offline.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.