CIPL: A Channel-Aware Framework for Recoverable Privacy Leakage in LLM Agents
Tao Huang, Tao Huang, Guosen Wu, Guosen Wu, Guolong Zheng, Guolong Zheng, Jiayang Meng, Chen Hou, et al.
cs.CR · cs.AI
Quick verdict
CIPL formalizes exposure-vs-recovery for black-box agent privacy leakage; results are coherent across 5 providers but lack statistical testing.
Summary
The paper introduces CIPL, a channel-aware framework for measuring black-box privacy leakage in LLM agents. Each target is decomposed into six stages — sensitive source, selection, assembly, execution, observation, extraction — and the framework measures the transition from internally selected sensitive units to attacker-recoverable output, reporting CER, AER and related metrics. Four controlled targets (two memory settings derived from public MEXTRA/EHRAgent artifacts, a synthetic 50-record RAG corpus, and a synthetic tool store with args-exfiltration and return-echo modes) plus a BrowserUse live-agent case study are evaluated on five providers (MiniMax-M2.5/M2.7, qwen3.5-plus, DeepSeek, GPT-4o) with n=30 queries, one retry, and five seeds. Memory targets stay saturated (CER=AER=1.0 everywhere), RAG leakage is frequent-but-partial (CER 0.33–0.55, AER 0.78–0.95), and tool/live-agent leakage depends strongly on observation surface, prompt-to-channel alignment, retrieval depth, and provider; cleaned weak-control prompts collapse leakage to near zero. A blinded 200-output semantic audit (kappa 0.951) finds five outputs with attacker-useful partial leakage that exact matching misses. Main limitations: no inferential statistics (one seed SD reaches 0.35), an unexplained CER discrepancy between main and ablation tables for the same rag_ctrl setting, near-degenerate saturated memory targets, and code/data only promised at publication.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Storage-component labels do not determine externally recoverable leakage: the transition from internally selected sensitive units to attacker-recoverable output must be measured explicitly (Propositions 1–2, §3).
- Under the shared protocol, memory targets are near-saturated (CER=AER=1.0 on all five providers), rag_ctrl leakage is frequent-but-partial (CER 0.33–0.55, AER 0.78–0.95), and tool/live-agent leakage is channel- and provider-conditioned (Tables 2 and 4).
- Leakage depends on prompt-to-channel alignment rather than mere internal exposure: cleaned weak controls collapse leakage to near zero on tool_ctrl and BrowserUse (Fig. 2a, Tables 5/H.2), and no prompt family is universally dominant (Table 3).
- Exact matching undercounts attacker-useful disclosure: a blinded 200-output semantic audit (kappa 0.951) found five outputs with partial semantic leakage and no exact canonical match (semantic_AER 0.855, semantic_CER 0.410).
- In the defense-aware extension, monitoring one channel does not close others: audit_deception holds AER ≤ 0.28 while audit_bypass reaches AER 0.92 (Appendix I, Table I.2).
Strengths
Well-posed formalization with explicit comparability conditions
The six-stage signature τ=(S, Sel, Asm, Exec, Obs, Ext) and the split between exposure set U and recovery set V (§3.1–3.2) turn 'storage labels ≠ leakage' into checkable Propositions 1–2; §3.4 states four protocol conditions (unit normalization, budget parity, fixed observation/extraction, reporting parity) that define exactly when cross-target comparison is legitimate.
Consistent shared protocol across heterogeneous targets
Five providers × four controlled targets plus BrowserUse under uniform n=30, one retry, five seeds; Appendix B reports full RN/EN/EE/CER/AER/ExecErr with per-seed SD, and execution errors are separated so runtime failure is not conflated with suppression (0 ± 0 across all main cells).
Honest negative and boundary results
Blinded semantic audit with strong reliability
Above-average reproducibility documentation
Weaknesses
No inferential statistics
Unexplained CER inconsistency between main and ablation tables
Degenerate saturated memory targets
Attack-pool and metric-definition gaps
Weak judge in defense extension and RQ3 scope
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CIPL addresses a real measurement gap: prior memory-extraction (MEXTRA), RAG-extraction, and tool-agent leakage studies each fix their own prompts, observation surfaces, and success criteria, so their numbers cannot be compared. The framework formalizes every target as a six-stage signature τ=(S, Sel, Asm, Exec, Obs, Ext) (§3.1) and measures the transition from internally selected sensitive units (U_j) to attacker-recoverable units (V_j), with two propositions (§3.5) establishing that storage labels are not leakage identifiers and that exact matching lower-bounds semantic recoverability. Central claims: (1) memory targets are near-saturated reference cases (CER=AER=1.0 on all five providers, Table 2); (2) retrieval-mediated leakage is frequent-but-partial (CER 0.33–0.55, AER 0.78–0.95); (3) tool/live-agent leakage is channel- and provider-conditioned (Tables 2, 4); (4) a blinded semantic audit surfaces exact-negative but attacker-useful disclosures (5/200 outputs, Table 6); (5) a monitored defense channel does not close unmonitored ones (audit_deception AER ≤ 0.28 vs audit_bypass AER up to 0.92, Table I.2). The claims are carefully scoped and mostly earned by the data.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CIPL and what problem does it solve?
CIPL (Channel Inversion for Privacy Leakage) is an evaluation framework for measuring black-box privacy leakage in LLM agents. It separates what sensitive content a system internally selects into computation from what an external attacker can actually recover, by modeling six stages (source, selection, assembly, execution, observation, extraction) and applying the same metrics (CER, AER) across memory-based, retrieval (RAG), tool-use, and live-browser agent settings — making results from different pipelines comparable.
How much do LLM agents leak private data according to this paper?
Under strong adversarial prompts, memory-based agents leaked everything (100% complete extraction on all five tested providers); RAG leakage was frequent but partial (complete-extraction rates 0.33–0.55); tool and live-browser leakage varied widely with observation surface, prompt alignment, and provider. Under benign task prompts, leakage collapsed to near zero, indicating leakage requires prompt-to-channel alignment, not just internal exposure of data.
Is the CIPL code or dataset available?
Not at review time. The data-availability statement promises evaluation code, configurations, prompts, and aggregated results 'upon publication, where permitted'; raw execution artifacts will not be redistributed. The RAG/tool corpora are manually constructed synthetic material, and the memory artifacts come from the public MEXTRA and EHRAgent repositories.
Which is riskier in live browser agents: final answers or execution traces?
It depends on the provider. In the BrowserUse case study, DeepSeek and GPT-4o leaked far more through intermediate process artifacts (AER 0.86 and 0.82) than through final answers (0.25 and 0.22), while qwen3.5-plus leaked more through final answers (0.76 vs 0.59); both MiniMax models leaked heavily through both surfaces. Deployments that expose execution logs to observers therefore face higher risk than final-answer filtering alone suggests.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Section 4.2's metric definitions (RN/EN/EE formulas) fell in a truncated region; EE = EN/(n·k) is reverse-engineered from table values, not verified against the text.
- Parts of Appendices D, G, and I are truncated in the extraction; some robustness tables and judge-prompt details are unverifiable.
- Figures are available only as embedded numeric values; trend claims from Figures 1–2 rest on the reported numbers.
- The reference list and part of Related Work were truncated; citation-completeness judgments are limited.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.