EvalMem: An Operation-Level Diagnostic Framework for Long-Term Memory Systems
Zeyu Liu, Jian Zhong, Rongduo Han, Ziyang Wu, Shunye Tang, Chenghao He, Yaxuan Yang, Yihang Qiu, et al.
cs.CL
Quick verdict
EvalMem 通过三个 Examiners 将记忆失败归因于编码/检索/生成；验证充分，检索缺陷的发现令人信服，但增益有限。
Summary
EvalMem 是一个诊断框架，它将长期记忆系统的故障分解为编码、检索和生成层。对于每个查询，三个并行的 LLM Examiners 会检查目标事实是否存储在记忆导出中，原生检索器是否呈现了可用的证据，以及骨干模型是否能够根据 oracle 证据进行回答；Attribution Agent 将它们的状态协调为带有上游掩蔽的 11 代码多标签缺陷集（例如，当事实从未存储时抑制检索失败）。一个优先召回的 agentic-RAG 探测器将 LoCoMo 上的评估器端观察遗漏从 29.8% 降低到 4.4%。作者在 LoCoMo、LongMemEval-S 和一个新的在policy DynaMem-Bench（962 个问题，10 个角色）上，使用带有标准化嵌入的三个骨干模型评估了七个记忆系统（TiMem、O-Mem、EverMemOS、MemOS、MemBox、MemoryOS、GAM）。检索在各处都是归因最多的层（例如，在 GPT-4o-mini 的 LoCoMo 上为 22.1%，而编码/生成为 7.7%/6.5%），并且动态交互提高了陈旧证据（NIR）缺陷。在此诊断的指导下，MemWiki（一种根据每个系统的记忆导出构建的辅助可搜索索引）在 LoCoMo 上将平均准确率提高了 +2.5pp，在 LongMemEval-S 上提高了 +2.3pp，其增益跟踪了基线检索缺陷（r=0.92）。判断可靠性已根据人工标注进行了验证（κ≥0.86；答案判断 κ=0.91）。局限性：对 LLM-judge 的依赖、单一语言、合成的 DynaMem-Bench、标准化的编码器配置以及适度的平均干预增益。
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- End-to-end QA accuracy cannot localize memory-system failures; EvalMem's three-Examiner design produces multi-label defect attribution validated against human annotators at κ≥0.86.
- A recall-first agentic-RAG probe using both the query and source evidence reduces evaluator-side observation misses on LoCoMo from 29.8% to 4.4%, making store-level encoding diagnosis trustworthy.
- Across seven memory systems and three datasets, retrieval is the most frequently attributed failure layer in every evaluated (dataset, backbone) cell — e.g. 22.1% vs 7.7%/6.5% on LoCoMo with GPT-4o-mini.
- MemWiki, an auxiliary search structure built from each system's own memory export, improves mean accuracy by +2.5pp on LoCoMo and +2.3pp on LongMemEval-S, with gains scaling with each system's baseline retrieval-defect rate.
Strengths
Explicit, auditable attribution logic
POS/NEG attribution matrices (Tables 4-5) plus seven verbatim examiner prompts (Appendix E.2) fully specify state-to-code mappings, including upstream masking rules (RF suppressed when encoding=MISS; NEG noise routed to NIR vs DMP by store cleanliness) — the diagnostic protocol itself is replicable.
Unusually thorough judge validation
Answer judge 96.7% agreement / κ=0.91 on 120 human-audited samples (Table 27); Examiners κ 0.86-0.90 with defect-set F1 0.92-0.95 over 450 decisions (Table 28); five diagnostic reruns show layer SD ≤0.25pp and Fleiss' κ ≥0.89 (Tables 9-12).
Ablations support the headline claim mechanistically
Honest reporting discipline
DynaMem-Bench built with fairness constraints
Weaknesses
Headline generality rests on a standardized configuration
Multi-label rates with full-query denominator are easy to over-read
Modest, heterogeneous intervention effect
All measurement flows through one LLM judge
Fully synthetic, single-language dynamic benchmark
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
EvalMem (arXiv 2609.22231, cs.CL; Nankai/vivo/Tsinghua/ZJU) targets a real evaluation gap: long-term memory systems for LLM assistants are scored almost exclusively by end-to-end QA accuracy, which cannot tell whether a wrong answer came from not storing, not retrieving, or not using a fact. The paper claims four things: (1) a three-Examiner diagnostic producing an 11-code multi-label defect set; (2) trustworthy store-level observation via a recall-first agentic-RAG probe, cutting observation misses on LoCoMo from 29.8% to 4.4% (recall of present evidence 70.2%→95.6%); (3) retrieval as the most frequently attributed failure layer across seven memory systems and three datasets (22.1% vs 7.7%/6.5% on default LoCoMo); (4) MemWiki, a diagnosis-guided auxiliary index, adding +2.5pp on LoCoMo and +2.3pp on LongMemEval-S. A new on-policy benchmark, DynaMem-Bench (962 questions, 10 personas), is a secondary contribution.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is EvalMem?
EvalMem is a diagnostic framework for LLM long-term memory systems. Instead of reporting only end-to-end QA accuracy, it runs three parallel examiners per query to determine whether a wrong answer is due to the fact not being stored (encoding), not surfaced by retrieval, or not used by generation, producing an 11-code multi-label defect diagnosis.
What is the main finding of the EvalMem paper?
Across seven memory systems evaluated on LoCoMo, LongMemEval-S, and the new DynaMem-Bench, retrieval failures are the most frequently attributed defect layer in every setting — for example 22.1% of all LoCoMo queries (GPT-4o-mini default) versus 7.7% for encoding and 6.5% for generation.
What is MemWiki and how much does it help?
MemWiki is a search-friendly auxiliary index built from a memory system's own memory export (factual records, tags, hypothetical questions, summaries). Enabling it improves mean accuracy by 2.5 percentage points on LoCoMo and 2.3 on LongMemEval-S, with the largest gains on systems that had the weakest native retrieval.
Is the EvalMem code and data available?
The paper links a public repository (github.com/ZeyuuLiu/EvalMem), publishes all examiner prompts verbatim in the appendix, and releases DynaMem-Bench personas and construction constraints; LoCoMo and LongMemEval-S are public datasets. Replication still requires API access to the judge models (GPT-5-mini) and the tested memory systems.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with [...sampling gap...] markers: mid-paper content (part of §2.4 MemWiki description, part of the §3 lead-in, portions of Appendix C.9-C.10, middle references, and segments of the examiner prompts) was skipped by sampling and not visible; all judgments are based on the visible head/tail.
- Table 17 (per-system McNemar b/c cells) is referenced in Appendix C.7 but its table body is not in the visible excerpt; quoted statistics (χ²=0.82, p=0.37 for TiMem) come from the prose.
- Figures are available only as extracted numeric annotations in text (e.g., Figures 4-5 heatmap values), so visual quality and layout could not be assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.