AIM: A Privacy-Aware Interoperable Memory Framework for Multi-Agent Multi-User LLM Systems
Zachary Johnson, Nigel Boachie Kumankumah, Somya Chatterjee, Tejas Sathyamurthi, Min Chen, Xinyi Alice Li, Xiao Wang, Emily Morgan Gelchie, et al.
cs.AI · cs.LG
Quick verdict
AIM brings index-level privacy control to multi-user LLM memory + MUMBench; promising design but synthetic circularity and no baselines weaken the claims.
Summary
The paper introduces AIM (Agentic Interoperable Memory), a framework letting multi-agent, multi-user LLM systems share one persistent memory store while enforcing per-user privacy. Each incoming message passes an operation-detection agent (multi-label create/read/update/delete/no-op), structured extraction, LLM-based visibility classification (public vs. private), and in-context deduplication; retrieval is a three-stage cascade — visibility-filtered HNSW dense search, tag rescoring, LLM reranking with recency fusion — with the visibility predicate enforced at the index level. The authors also present MUMBench, a synthetic benchmark of 672 GPT-4o-generated, manually validated interactions across 93 users and four domains, covering all five memory operations plus adversarial queries. Using five LLMs as pipeline backbone across three runs, the best configuration (GPT-5.4-mini) reaches 58.8% strict and 70.5% state-aware operation accuracy, 95.0% visibility accuracy, and 91.6% content quality; visibility peaks at 96.0% with GPT-5.4. Limitations are substantial: gold labels come from the same rule set the classifier is prompted with, raising circularity concerns; no external memory baselines (e.g., Mem0) are run on MUMBench; retrieval relevance peaks at only 45.0% and tag overlap at 35.5%; and AIM underperforms Mem0 substantially on single-user LOCOMO (judge score 0.280 vs. 0.671 on single-hop), evaluated in a single pass. The framework and benchmark are promising, but superiority claims rest on architectural argument rather than measured evidence.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- AIM is the first end-to-end pipeline unifying LLM-based memory extraction, automatic visibility classification, and deterministic index-level access enforcement in a system-agnostic service (§2).
- Enforcing the visibility predicate V(u,m) (Eq. 1) at the retrieval index makes private memories structurally inaccessible to non-owning users, with scope-restricted deduplication preventing cross-user leakage.
- MUMBench is the first public dataset evaluating create/read/update/delete/no-op decisions plus adversarial privacy queries in a multi-user setting (672 interactions, 93 users, four domains).
- Across three runs on MUMBench, AIM achieves up to 96.0% visibility classification accuracy, 58.8% strict and 70.5% state-aware operation accuracy, depending on backbone model.
- Smaller backbone models (GPT-5.4-mini) match or beat larger ones on memory-management metrics, so memory systems need not sacrifice efficiency for quality (§5.3).
Strengths
Fills a real benchmark gap
MUMBench (§4) is, per the paper's verified related-work survey, the first public dataset testing all five memory operations (create/read/update/delete/no-op) with adversarial cross-user queries; it explicitly targets gaps in LOCOMO and MemoryAgentBench, and answers the need flagged by Collaborative Memory (Rezazadeh et al., 2025).
Structural, deterministic privacy enforcement
The visibility predicate V(u,m) (Eq. 1) is pushed into the vector index as a server-side filter (§3.7 Stage 1), complemented by visibility-scoped deduplication (§3.5) and ownership-gated update/delete (§3.1) — private memories are excluded by construction, not by post-hoc filtering that a classifier error could leak.
Realistic multi-label operation model
Thoughtful tiered evaluation design
Unusually full disclosure of prompts and candor
Weaknesses
Benchmark-label circularity
No external baselines on the new benchmark
Retrieval metrics are weak and underpowered
Large unexplained regression on the only external comparison
Abstract composes headline numbers from different configurations
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
AIM (Microsoft; arXiv 2609.12320) is a memory service for multi-agent, multi-user LLM deployments. Its central claims: (1) it is the first end-to-end pipeline unifying LLM-based extraction, automatic public/private visibility classification, and deterministic index-level access enforcement (§2); (2) the visibility predicate V(u,m) (Eq. 1) makes private memories structurally unretrievable by non-owners; (3) MUMBench is the first public benchmark testing create/read/update/delete/no-op plus adversarial cross-user queries (672 interactions, 93 users, four domains, §4); (4) AIM reaches 96.0% visibility accuracy and 58.8%/70.5% strict/state-aware operation accuracy across three runs (abstract). Note immediately that the abstract composes these headline numbers from different backbones: 96.0% visibility is GPT-5.4's row in Table 1, while 58.8%/70.5% belong to GPT-5.4-mini; no single configuration achieves all three.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AIM and what problem does it solve?
AIM (Agentic Interoperable Memory) is a memory service for LLM agent systems in which many users and agents share one persistent memory store. It automatically labels each stored fact as private (retrievable only by its owner) or public (retrievable by everyone), and enforces that rule directly inside the vector-search index, so other users' private memories are structurally excluded from search results rather than filtered after the fact.
What is MUMBench?
MUMBench is a benchmark of 672 simulated chatbot interactions from 93 synthetic users across four domains (coding, customer support, education, travel planning), generated with GPT-4o and manually validated. Unlike single-user memory benchmarks such as LOCOMO, it tests create, read, update, delete and no-op decisions, public/private labeling, and adversarial attempts by one user to access or modify another user's memories.
How accurate is AIM?
On MUMBench across three runs, the best backbone (GPT-5.4-mini) achieved 58.8% strict operation accuracy, 70.5% state-aware operation accuracy, 95.0% visibility accuracy and 91.6% content quality; visibility peaked at 96.0% with GPT-5.4. Weaker spots: retrieval relevance peaked at only 45.0%, tag overlap at 35.5%, and on the single-user LOCOMO benchmark AIM scored well below Mem0 (e.g., 0.280 vs 0.671 judge score on single-hop questions, single pass).
Is the code or dataset available?
The paper describes MUMBench as the first public multi-user memory benchmark and publishes its full data-generation and judge prompts in the appendix, but the provided text contains no repository URL and no explicit code-release statement, so actual availability should be confirmed with the authors before relying on it.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Appendix A.3 carries a heading but no body text in the extraction (content may be a figure or pointer); a full JSON schema example is nonetheless present in A.5, so no schema information is lost.
- Table 3's per-domain public-memory counts (8+20+17+10) sum to 55 while the total row reads 56 in the provided text; I cannot determine whether this is a paper typo or an extraction artifact, and it does not affect any conclusion drawn here.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.