The Price of Safety: Benign-Case Utility and Token Overhead of Memory-Poisoning Defenses in LLM Agents
cs.CR · cs.AI
Quick verdict
Measures benign-traffic cost of 4 memory-poisoning defenses: write-time nulls, reranker −4.4pp & 33.6% false quarantines; honest but small-sample (5 convs).
Summary
This paper measures what memory-poisoning defenses cost LLM agents on traffic containing no attacks — the case that dominates deployment. Four defenses (write-time input sanitization, provenance tagging, LLM-based anomaly detection; read-time reranking) plus a four-way stack are evaluated against an identical undefended baseline on five LoCoMo conversations truncated to 150 turns, three replicates per condition (90 cells), with a frozen calibrated judge (claude-haiku-4-5, κ=0.909), isolated memory namespaces, and per-purpose token accounting via a proxy. Because the pipeline is non-deterministic even at temperature zero, replicates are collapsed by majority vote; effects are estimated as paired per-conversation deltas with stratified bootstrap CIs, exact McNemar tests, and sign-flip permutation tests. Write-time defenses show no detectable utility cost (CIs ≈ ±4.5 points, including zero) and 0% false quarantines. The reranker lowers core accuracy by 4.4 points (95% CI [−9.0, −0.05]; McNemar p=0.064) and quarantines legitimate memories on 33.6% of adjudicated items (up to 106 in one conversation) at 2.7% token overhead. The stack does not compound costs (−3.1 points, CI includes zero; 4.1% overhead). Limitations: only five conversations, a single backbone model (gemini-3.1-flash-lite), a borderline reranker result, and — by design — no attack-condition evaluation.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 1
● low 4
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Three write-time defenses (sanitization, provenance, LLM anomaly detection) show no benign-case utility cost detectable at ~±4.5pp resolution, with 0% false quarantine on attack-free traffic (Table 3).
- The read-time reranker lowers core accuracy by 4.4pp (95% CI [−9.0,−0.05], McNemar p=0.064) and quarantines legitimate memories on 33.6% of adjudicated items at 2.7% token overhead.
- Stacking all four defenses does not compound costs (−3.1pp, CI [−7.4,+1.0], 4.1% overhead), suggesting write-time defenses may partly offset reranker losses.
- A defense's pipeline position (write vs read), not whether it uses an LLM, predicts its benign-case price.
- Single-run evaluation at this scale is unreliable: the pipeline is non-deterministic at temperature zero, and one apparent defense effect was a scoring artifact (Sections 5.2, 4.4, Appendix D).
Strengths
Isolation-by-design harness
Section 4.1-4.2: all six conditions share backend, embedding, answering model, question set and frozen judge; each of 90 cells writes to an isolated namespace so quarantine effects cannot leak across conditions; per-purpose token attribution via a logging proxy even caught a 5x pricing error (Section 4.2).
Replication with honest noise accounting
Section 5.2/6.3 documents that temperature zero does not make the pipeline deterministic; replicates are collapsed by majority vote, and Appendix D shows a single-run artifact that replication overturned - a concrete demonstration of why replication matters at this scale.
FQR as a mechanical, noise-immune metric
Auditable self-correction of a scoring bug
Calibrated judge, no self-grading
Weaknesses
Sample size caps resolution
Headline reranker effect is statistically borderline
Single backbone and embedding model
No protection-side measurement
Stacked comparison confounded; data inconsistencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a measurement paper, not an attack or defense paper: it deliberately excludes attacks and asks only what four memory-poisoning defenses cost when traffic is entirely benign (Sections 1, 3.1). The stated claims: (1) three write-time defenses — lexical input sanitization, provenance tagging, and LLM-based anomaly detection — show no benign-case utility cost the authors can resolve, with 95% CIs of roughly ±4.5 points including zero and 0% false quarantines; (2) the single read-time defense, a consistency reranker, costs 4.4 points of core accuracy (95% CI [−9.0, −0.05], McNemar p = 0.064), quarantines legitimate memories on 33.6% of adjudicated items (up to 106 in one conversation, Table 5 conv-43 seed 3), at 2.7% token overhead; (3) stacking all four does not compound costs (−3.1 points, CI [−7.4, +1.0], 4.1% overhead); (4) pipeline position, not LLM usage, predicts benign-case price. A fifth meta-claim — that single-run evaluation at this scale is unreliable — is supported by run-to-run variance at temperature zero (Section 5.2) and by an abstention-scoring artifact the authors caught and corrected (Section 4.4, Table 1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the benign-case cost of memory-poisoning defenses in LLM agents?
In this paper's measurements on five LoCoMo conversations, three write-time defenses (input sanitization, provenance checking, LLM-based anomaly detection) showed no detectable accuracy loss on attack-free traffic, while a read-time reranker lowered core accuracy by 4.4 percentage points (95% CI [−9.0,−0.05], McNemar p=0.064) and quarantined 33.6% of legitimate memories it evaluated.
What is the false quarantine rate (FQR)?
It is the fraction of benign items a defense quarantines when the evaluation corpus contains no attacks at all, meaning every quarantine is a false positive by construction. In this paper the reranker reached 33.6% on benign traffic, with up to 106 false quarantines in a single conversation, while all write-time defenses had 0%.
Does stacking multiple memory defenses make the utility cost worse?
Not in this study: the stack of all four defenses showed a smaller accuracy loss (−3.1 points, confidence interval including zero) than the reranker alone, and its token overhead roughly summed the components (4.1% vs 1.2% + 2.7%). The authors caution the intervals overlap, so this is 'no evidence of compounding', not proof of protective interaction.
Is the code and data for this paper available?
Yes. The paper links a public repository (github.com/pritom02bh/memdefense) containing the harness, evidence-alignment filter, judge prompts, abstention detector, and complete per-cell run outputs; the underlying LoCoMo dataset is public. Re-running requires paid access to Gemini, OpenAI and Anthropic APIs.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 (forest plot, FQR chart, cost-quadrant scatter) are described in text but not analyzable from the extraction.
- Abstract text is truncated mid-sentence, but the full body and appendices cover all stacked-condition numbers.
- Tables were cross-checked from the flattened text layout; Table 3 and Table 5 values appear internally consistent (e.g., max 106 quarantines matches conv-43 seed 3).
- Review is based on the v1 preprint; the AI-detection signal run on the abstract was internally inconsistent across providers and was not used to judge authorship.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.