Interactive Memory Learning for Long-Term Conversations
Cai Ke, Jiangyue Yan, Han Zhang, Xin Liu, Zike Yuan, Yue Yu, Hui Wang, Ruifeng Xu
cs.AI · cs.CL
Quick verdict
Learnable two-agent memory RL for long-term chats via delayed rewards; strong gains, but LLM-judge circularity and thin reproducibility temper claims.
Summary
The paper proposes ICML (InteraCtive Memory Learning), reframing long-term conversation memory from a passive archive into a learnable policy optimized by online RL. Two agents co-evolve: a Planner decides which turns to store and a Trigger selects which memory to retrieve, coupled by a delayed cross-session truth reward that propagates future response-quality feedback back to earlier storage decisions. Cold-start is addressed by a retrospective session synthesis pipeline that backward-generates causally consistent storylines from a seed session and forward-annotates expert labels for supervised LoRA warm-up, followed by PPO-based online learning. Evaluated on MSC, Conversation Chronicles, and GapChat with GPT-4o and Gemini2.5 backbones and 1B-12B open-source policy models, ICML reports state-of-the-art results over Long Context, Mem0, A-Mem, MemoryOS, MemoryBank, LD-Agent, and THEANINE on BLEU-4/ROUGE-L/BertScore/Mauve and five LLM-judged dimensions. Human evaluation (5 evaluators, 150 samples) gives 66% generation and 70% memory win rates vs THEANINE (Fleiss' κ 0.65-0.68, p<0.01). Session-wise analysis shows memory miss rate falling from 48.2% to 3.2% with proxy-truth reward correlation rising to r=0.89. Main limitations: the training reward and much of the evaluation come from the same LLM judges (circularity risk), automatic-metric margins are small, expert data is LLM-synthesized without human audit, and no code release is stated.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Treating memory management as a learnable, online-RL policy (Planner + Trigger) outperforms static heuristic memory architectures across three long-term dialogue benchmarks.
- A delayed cross-session truth reward propagates future response-quality feedback back to earlier storage decisions, with proxy-truth reward correlation rising from r=0.62 to 0.89 over five training sessions (Table 8).
- Backward (retrospective) storyline synthesis produces better cold-start expert data than forward generation (e.g., CC/GPT-4o BLEU-4 2.40 vs 1.85, LLM-judge 4.82 vs 4.15, Table 12).
- ICML continuously improves with interaction: memory miss rate falls from 48.2% in session 1 to 3.2% in session 5 (Table 8).
- Human evaluators prefer ICML over the strongest baseline THEANINE, with 66.0% generation and 70.0% memory-retrieval win rates (95% CIs, Fleiss' κ 0.65-0.68, p<0.01, Table 9).
Strengths
Principled credit assignment to storage decisions
The delayed cross-session truth reward (Algorithm 1, lines 8-12) credits a Planner storage decision only when the Trigger later uses that memory and the response scores well — a concrete mechanism for sparse reward in memory learning, supported by proxy-vs-truth reward Pearson r rising from 0.62 (session 1) to 0.89 (session 5, Table 8).
Unusually broad empirical coverage
Three real-interaction benchmarks (MSC, CC, GapChat), 9 policy variants across Llama-3/Gemma-3/Qwen3 families (1B-12B), two frontier backbones (GPT-4o 128K, Gemini2.5 1M), and 7 baselines spanning both management-centric (Mem0, A-Mem, MemoryOS) and generation-centric (MemoryBank, LD-Agent, THEANINE) families (Table 1, Sec. 4.1).
Human evaluation with actual inferential statistics
Extensive ablations targeting each design choice
Weaknesses
Reward-evaluation circularity
Weak discriminative power and small margins in automatic metrics
Synthesized expert data is asserted, not audited
Missing reproducibility commitments
Test-time learning entangled with evaluation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets long-term open-domain conversation and argues that existing memory systems (summaries, fact stores, graph memories) are static heuristics: they archive passively and never learn from interaction feedback. ICML (InteraCtive Memory Learning — an unfortunate acronym collision with the ICML conference) reframes memory as a learnable policy: a **Planner** decides what to store, a **Trigger** decides when to retrieve, and both are optimized by online PPO with a **delayed cross-session truth reward**. Three claims are staked: (1) a learnable memory paradigm driven by environmental feedback; (2) being *first* to propose a plug-and-play online RL framework for this task; (3) state-of-the-art results on three datasets with quality improving as interactions accumulate. The contribution is a framework, not a model: 1B-12B policy models (Llama-3/Gemma-3/Qwen3) drive memory decisions while GPT-4o (128K) or Gemini2.5 (1M) generate responses.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ICML in this paper?
In this arXiv paper (2609.17088), ICML stands for InteraCtive Memory Learning — a multi-agent framework for long-term conversations in which a Planner agent decides which dialogue turns to store in memory and a Trigger agent decides when to retrieve them. The two agents are trained with online reinforcement learning (PPO) and synchronized by a delayed cross-session reward that credits earlier storage decisions for later response-quality gains. (Not to be confused with the International Conference on Machine Learning.)
What datasets and baselines does the paper use?
Evaluation covers three long-term open-domain conversation datasets built from real human interactions — Multi-Session Chat (MSC), Conversation Chronicles (CC), and GapChat (GC) — using GPT-4o (128K context) and Gemini 2.5 (1M context) as response-generation backbones. Compared baselines include Long Context, Mem0, A-Mem, MemoryOS, MemoryBank, LD-Agent, and THEANINE.
How much better is ICML than existing memory methods?
The authors report state-of-the-art automatic scores — for example, on Conversation Chronicles with a GPT-4o backbone, ICML-8B reaches BLEU-4 ≈2.40 and BertScore ≈47.74 versus ≈1.57/47.50 for a long-context baseline — plus higher LLM-judged Memorability. In human evaluation against the strongest baseline THEANINE, ICML won 66% of generation comparisons and 70% of memory-retrieval comparisons, with 95% confidence intervals and p<0.01 reported.
Is code available for ICML?
No code or data release is stated in the reviewed text. The appendix does provide key hyperparameters (LoRA warm-up and PPO settings), the training algorithm, and the prompts used for synthesis, judging, and reward computation, but the pipeline depends on closed-source models (GPT-4o, Gemini 2.5), so independent replication is currently difficult.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Middle sections of the extracted text are truncated (parts of Section 4.3+ analysis, Appendix B/D details, and Table 13's contents), and Table 1's row/column alignment is partially garbled in extraction; numbers were read carefully but some baseline configurations may be misattributed.
- Figures 1-4 are available only as captions and OCR fragments; visual magnitudes in the radar and bar charts could not be verified.
- The absence of a code-availability statement may partly be an extraction artifact if such a statement sat in an omitted section.
- Table 12 (Forward vs Backward) is truncated mid-row; only CC and part of MSC rows were visible, and the RMM comparison (Table 13) numbers were not readable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.