Revoked but Still Authoritative: An Empirical Study of Revocation Enforcement in Agent-Memory Systems
Yi Ting Shen, Kentaroh Toyoda, Alex Leung
cs.AI · cs.CR
Quick verdict
Claims no tested agent-memory system enforces soft revocation by default, leaving stale policies to cause unsafe actions (~43% of trials); measurement is careful, honest, though narrow.
Summary
This empirical security study measures whether five agent-memory systems (Graphiti, Zep, mem0, langmem, cognee) actually enforce soft revocation at retrieval time. The setup loads each store with a revoked policy and its replacement, across nine scenarios, nine decision models, and ten trials per cell, reporting an exposure rate (does the store return the revoked record) and an unsafe-action rate (does the agent act on it) under six defense conditions. Findings: under indirect insertion no system reliably records and enforces revocation; under direct insertion only Graphiti and mem0 in experimental configuration return revoked records, which rank first in every scenario and cause unsafe actions in 44.2% and 42.1% of trials respectively; flipping one retrieval flag in mem0 flips behavior from safe to unsafe (p<0.001). The authors propose a guard that intercepts reads and withholds revoked or conflicting records; it matches the store-level filter where backends expose marks but relies on lexical heuristics and underperforms no defense on some indirect-insertion configurations. Limitations include synthetic scenarios, a narrow action space, and end-to-end failure measurable on only two of five systems.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- No tested agent-memory system records and enforces soft revocation reliably under indirect insertion: cognee, langmem and mem0 never recorded a revocation in 0/9 scenarios, and Zep records it but exposes no status to callers.
Strengths
Controlled causal attribution via a within-system ablation
mem0 run twice on identical data with a single retrieval flag toggled flips unsafe actions from 0/810 to 341/810 (42.1%), matched-pair exact test p<0.001 (Table 12/13); this isolates retrieval, not the store, as the failure locus.
Transparent, honest statistical and scope reporting
Section B.2 states the exact-test power bound (17 discordant pairs need ≥13/4 to reach significance), Section A.2 explains why no per-condition CIs are reported, and Section 4.4 explicitly disqualifies its own with-rule vs no-rule comparison as non-controlled.
Failure taxonomy beyond the single read
Reproducibility provisioning
Weaknesses
Abstract claim is broader than the evidence
End-to-end quantification covers only two of five systems
Guard's negative and fragile cases are under-surfaced
Low statistical power and thin in-paper statistics
Confound between conditional-exception and precedence in the outlier scenario
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Revoked but Still Authoritative (arXiv:2609.08258)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'soft revocation' in agent memory systems?
It is the practice of keeping a fact in memory after it is contradicted, marking it invalid (e.g. via validity timestamps or an expiry field) instead of deleting it, so history stays auditable and point-in-time queries remain possible. This paper asks whether that 'invalid' mark actually prevents the store from returning the stale fact to an agent at retrieval time.
What did this study find about Zep, Graphiti, mem0, langmem and cognee?
Under organic, text-based policy updates (indirect insertion), none of the five systems reliably recorded and enforced revocation: cognee, langmem and mem0 recorded it in 0 of 9 scenarios, and Zep records it but never exposes the status to callers. Under direct insertion only Graphiti (by default) and mem0 with a flag override returned revoked records, and those led agents to the unsafe action in 44.2% and 42.1% of 810 trials respectively.
What is the proposed guard and does it fix the problem?
The guard is a middleware that intercepts every read between the agent and the memory backend, withholds records the backend marked revoked, and uses a lexical conflict test to withhold records contradicted by a newer fact. It matches the store-level filter when the backend exposes revocation marks, but on some indirect-insertion configurations it is slightly worse than no defense, so it is a mitigation, not a fix.
Can an agent's own writes undo the memory filter?
Yes. When an unsafe decision was journaled back into the store as an ordinary current record, later agents under the supposedly safe retrieval filter still took the unsafe action in 71.6% to 83.1% of runs across three follow-up hops (direct write-back mode), because the journal entry itself is not marked revoked.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Only the far end of the PDF can be truncated by extraction; the provided text ends mid-flow at the tool-actions discussion, but all numbered tables referenced in the review (Tables 1–16) appear complete within the extraction.
- Appendices referenced but partially substantive (e.g. ICC/design-effect values, Ω contents) are described by the paper itself as living in the released results bundle, so their absence from the review reflects the paper, not the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.