Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability
cs.AI · cs.CL · cs.IR
Quick verdict
Controlled study: KG-fixed memory survives model swaps, NOTES degrades sharply, mixed embeddings fail silently; rigorous but one 8B pair, synthetic histories.
Summary
The paper asks whether an LLM agent's external memory survives a model upgrade, framing memory as a writer–reader pipeline and comparing four formats: verbatim long-context history (LC-RAW), chunked RAG, model-written compressed notes (NOTES), and a fixed-schema knowledge graph (KG-fixed). Using 48 synthetic histories with randomized answer codes, exact-match scoring, and two open-weight ~7–8B models (Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct-1M), the authors isolate one changed component at a time: writer swap, embedding swap, or repair source. Four contrasts were locked in a signed Git tag with a 5-point threshold and Holm correction. KG-fixed transfer is essentially lossless (swap Δ +0.0004±0.0020), whereas NOTES migration is strongly direction-dependent: Qwen reading Llama's notes loses 13.28 points while Llama reading Qwen's gains 9.91. A 50/50 mixed embedding index (bge-large v1.0→v1.5) recovers only 4.96 of an 11.90-point full-re-embedding gain. Diagnostics attribute ~80% of the NOTES deficit to store construction and ~81% of the RAG deficit to retrieval (a deliberately weak single-stage dense retriever). Store-only NOTES repair never reaches the 90% recovery target; raw-history repair succeeds in 34/48 cases only with Qwen as repairer. Limitations: a single cross-family model pair, scripted synthetic histories, a workload-derived KG schema favoring KG-fixed, non-public code/data ('available upon request'), and a lowered 60% evidence-coverage gate.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Fixed-schema knowledge-graph memory transfers essentially losslessly across a writer swap (Δ +0.0004 ± 0.0020), while model-written NOTES is strongly writer-coupled and direction-asymmetric (+9.91 / −13.28 pp).
- After a bge-large v1.0→v1.5 upgrade, a 50/50 mixed embedding index captures only 4.96 pp of the 11.90 pp full-re-embedding gain, failing silently because both models share 1024 dimensions.
- About 80% (0.467 ± 0.014 of 0.584) of the NOTES accuracy deficit comes from information lost at store construction, while ~81% (0.364 ± 0.012 of 0.450) of the RAG deficit comes from retrieval in the tested single-stage dense pipeline.
- Store-only NOTES repair never reaches the 90% recovery target in any of 48 histories; rebuilding from retained raw history succeeds in 34/48 cases but only with Qwen as the repair model.
Strengths
Preregistered hypothesis set with multiplicity control
Four contrasts (H1/H2/H4a/H7a) were frozen in a signed Git tag (hypothesis-lock-v1) before data collection, with a 5-pp action threshold, one-sided t-tests, and Holm correction; a seeded 10,000-resample bootstrap later replicated every decision (§3.5, Table 2). Null results (H1, H4a) are reported as unsupported rather than reworked post hoc.
Clean isolation design removing confounds
Writer/reader/embedder are separated (§2.1); randomized answer codes make chance ≈ 0 and exact-match scoring removes judge noise (§3.1); repeated live reads varied by only 0.0016; the two embedding spaces were verified distinct via cross-space cosine 0.904 (Appendix F).
Failure-stage decomposition is actionable
Direction-level reporting instead of averages
Exemplary audit trail
Weaknesses
External validity rests on one migration pair
NOTES effects measured on a weak, uneven baseline
KG-fixed advantage is structurally favored
Repair-direction result confounded by output budget
Artifacts not publicly deposited
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Review: Does Your Agent's Memory Survive a Model Upgrade?
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which memory format survives a model upgrade best according to this paper?
A fixed-schema knowledge graph (KG-fixed): after swapping the writing model, accuracy changed by only +0.0004 ± 0.0020 in this study. Model-written compressed notes (NOTES) were most fragile — with two ~7–8B open models on synthetic histories, one migration direction lost 13.28 points while the reverse gained 9.91.
Is it safe to mix old and new embedding vectors in one index after upgrading the embedding model?
In this study, no: a 50/50 mixed index of bge-large v1.0 and v1.5 vectors (both 1024-dimensional, so no dimension error warns you) recovered only 4.96 points of the 11.90-point accuracy gain from full re-embedding. The authors recommend building a separate index with the new model and cutting over only when complete.
Why did keeping the raw conversation history help repair agent memory?
Compression discards evidence permanently: rewriting existing NOTES never reached the 90% recovery target in any of 48 histories, while rebuilding from retained raw history succeeded in 34/48 cases — but only with Qwen as the repair model, since Llama hit its output-token limit on every attempt.
What are the main limitations of this study?
It tests a single cross-family pair of small open-weight models (Llama-3.1-8B vs Qwen2.5-7B) on 48 scripted synthetic histories with a KG schema built from the task's own fields; the retrieval pipeline is deliberately simple; code and data are only 'available upon request'; and the NOTES evidence-coverage gate was lowered to 60% during calibration.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure interiors (Figs. 1–3) are images absent from the extracted text; only captions and in-text numbers were available, so figure-level visual detail is unverified. All cited numbers come from the body text and tables, which appear complete through Appendix F.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.