CHIME: Credit-Aware Hierarchical Memory Evolution for Long-Horizon Agentic Planning
Yongshi Ye, Tian Lan, Feihu Jiang, Muyang Ye, Bin Zhu, Qianghuai Jia, Longyue Wang, Zhao Xu, et al.
cs.AI
Quick verdict
CHIME claims credit-attributed plan/exec memory beats self-evolving baselines; eval gains 2.96–3.68 pts but modest; no variance tests; ReasoningBank missing.
Summary
The paper proposes CHIME, a training-free self-evolving memory framework for LLM agents targeting a credit assignment flaw: final task outcomes conflate plan quality with execution errors and environmental factors, so outcome-based memory feedback is biased. CHIME maintains two banks — planning and execution — and an LLM Credit Attribution Gate labels each episode's outcome as due to the plan, the execution, both, or neither, then updates only the corresponding bank; retrieval reranks candidates by confidence-weighted value scores. On four long-horizon benchmarks (τ²-Bench, VitaBench, BrowseComp-ZH, BFCL-v4) with Qwen3.5-Flash and DeepSeek-V4-Flash backbones, CHIME raises the frozen-memory eval average over the strongest baseline by 2.96 and 3.68 points and retains only 129 memory items versus 3,585 for A-MapReduce on τ²-Bench. Ablations show each component contributes (gate removal costs 2.44 eval points). Analyses report gate stability of 87.8–97.9%, monotonic value-accuracy correlation, cross-backbone transfer, and 2.25–3.12 further points from a stronger gate model. Main limitations: per-benchmark gains are uneven (CHIME trails A-MapReduce by 2.83 points on BFCL-v4 with DeepSeek), results are Avg@3 without variance or significance tests, ReasoningBank is discussed but not compared, and code is only promised.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Separating planning and execution memory banks and updating them via a credit attribution gate yields higher accuracy than outcome-based self-evolving memory (+2.96/+3.68 eval-average points on two backbones).
- CHIME accumulates effective memory with roughly 28x fewer items than A-MapReduce (129 vs 3,585 on τ²-Bench), while outcome-based baselines' accuracy declines at full accumulation.
- Learned memory values monotonically track downstream accuracy, and planning memories yield larger gains than execution ones (21.7%→50.8% vs 23.7%→35.4%).
- The Credit Attribution Gate is repeatable (87.8–97.9% agreement) and its generated experience rescues 56.3–67.6% of attributed failures.
- Accumulated memory transfers across backbones, beating A-MapReduce's transferred memory by 2.25–4.68 points.
Strengths
Well-posed problem framing with a crisp principle
The outcome-conflates-plan-and-execution argument is crisply formalized (Section 1, 3.1), and the three case studies (Figures 5–7) make the mechanism concrete: status-conditioned decomposition, exact tool-argument binding, and a strict-deadline failure routed to the planner bank.
Striking efficiency evidence for the noise thesis
Figure 3 (RQ2) shows outcome-based baselines peak at the 75% accumulation checkpoint and decline at 100% while CHIME keeps improving with 129 items vs 3,585 — direct evidence for the noise-accumulation thesis, not just a leaderboard win.
Multi-pronged validation beyond headline accuracy
Honest boundary disclosure
Weaknesses
No variance or significance reporting
Per-benchmark gains uneven; one clear loss
Missing comparison with ReasoningBank
Judge-dependent and partly circular analyses
Undefined epoch semantics in the protocol
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CHIME (arXiv:2609.02074) is a training-free, self-evolving memory framework for long-horizon LLM agents. Its diagnosis: prior self-evolving memory methods use the final binary task outcome as feedback, conflating plan quality with execution errors and environmental factors, so accumulated memory is biased (Section 1). The remedy is *attribute-before-memorize*: a hierarchical memory bank (planning bank + execution bank, Eqs. 2–3), a Credit Attribution Gate implemented as structured self-reflection labeling each outcome planning/execution/both/none (Eq. 6), and credit-aware evolution that updates values with confidence-weighted feedback (Eqs. 8–9) and writes new experience only into attributed banks (Eq. 10). Claims: SOTA over training-based and self-evolving baselines on four benchmarks, drastically smaller memory (129 vs 3,585 items), values that track downstream utility, reliable attribution, and cross-backbone transfer.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CHIME in one sentence?
CHIME is a training-free framework in which an LLM agent improves over time by storing experience in two separate memory banks (planning and execution) and writing feedback only into the bank that a credit-attribution step deems responsible for the task outcome — plan, execution, both, or neither.
How much does CHIME improve over prior methods?
On the frozen-memory eval split it beats the strongest of four baselines by 2.96 points on Qwen3.5-Flash and 3.68 points on DeepSeek-V4-Flash, averaged over four agent benchmarks (τ²-Bench, VitaBench, BrowseComp-ZH, BFCL-v4). On τ²-Bench it reaches the highest accuracy with only 129 stored memories versus 3,585 for the closest baseline.
Does CHIME require fine-tuning the model?
No — the backbone model stays frozen; only the external planning and execution memory banks evolve from interaction outcomes during the train stream, and they are frozen entirely during evaluation.
Is CHIME's code available?
The paper only promises a release at github.com/ATH-MaaS/Marco-DeepResearch; no code or data was verifiably available at review time, though hyperparameters and protocol details are documented in Appendix A and Table 7.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Inline math (Eqs. 1–10) is flattened by PDF extraction, but all tables (1–7), figure captions, and numeric results are legible; the provided text includes the full abstract, references, and Appendices A–C with no signs of truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.