What Eviction Destroys: A Restore-Counterfactual Audit of Forgetting in Agent Memory
cs.CL · cs.AI · cs.DB
Quick verdict
Restore-counterfactual audit splits agent-memory errors into irreversible/recoverable/residual; rigorous, honest, but one benchmark and same-provider judge.
Summary
The paper introduces the restore counterfactual: a per-question paired intervention for auditing eviction in external agent-memory stores. After an eviction policy runs at a token budget, the question's gold evidence is reinstated into the read-time context and the same frozen reader re-answers. Combining the correctness change with whether the evidence survived classifies each oracle-answerable error as irreversible (evidence destroyed; only retention helps), recoverable (evidence retained but retrieval missed it), or residual (still wrong with evidence restored; reader utilization failure). On LongMemEval-S (~102k-token histories), four eviction policies (FIFO, random, redundancy-aware, LLM-importance) are evaluated at 8k/30k/80k budgets under forced-gold and top-k retrieval, with GPT-4o-mini as reader/judge and GPT-5.4-mini as robustness reader. At 80k under top-k, the irreversible two-bin share is 0.67-0.73 for the three baselines and 0.60 for LLM-importance, rising to 1.00 at 8k; recoverable errors exist only under top-k, so budget-accuracy frontiers are incomparable across retrieval regimes. At matched accuracy, no policy dissociates in irreversible rate at 1.2-6 pp resolution, while the destructive control registers in all 9 contrasts. Ablations (placebo restore 8.3%, full-store injection 2%) bound the metric's slack; an independent GPT-5.5 judge agrees at 95.7% (kappa=0.90). Main limitations: single benchmark, single lexical ranker, reader-judge sharing one provider, and policy classes rather than published memory systems.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The restore counterfactual decomposes each oracle-answerable eviction error into an irreversible, recoverable, or residual bin, each implying a different remedy (retention, retrieval, reader).
- Under top-k retrieval at an 80k-token budget on LongMemEval-S, destruction is the majority of correctable eviction loss: two-bin share 0.67-0.73 for FIFO/random/redundancy-aware and 0.60 for LLM-importance, rising to 1.00 for all four at 8k.
- Recoverable errors occur only under top-k retrieval, so budget-accuracy frontiers are not directly comparable unless the retrieval regime is reported.
- At accuracy-matched comparisons, no tested policy shows a different irreversible rate at 1.2-6 percentage-point resolution, while the deliberately destructive control is detected in all 9 contrasts.
- Residual errors are predominantly reader utilization failures in cross-session aggregation, largely resolved by a stronger reader.
Strengths
Well-identified measurement instrument
The per-question paired restore (§3.1-3.2) separates failure modes that aggregate accuracy conflates: destruction, retrieval misses, and reader failures get distinct remedies, with an explicit estimator and a worked example (Fig. 2).
Validity work that quantifies its own slack
A1-A3 are stated with failure modes and then probed: over 2,276 irreversible cases (App. C), surviving-gold restore reclassifies 0.3%, full retained-store injection recovers 2.0%, and an equal-length placebo flips 8.3% vs 100% for gold — honestly reported as an upper bound on presentation effects.
Statistical honesty
Reader robustness replication
Reproducibility package
Weaknesses
Single benchmark and single ranker
Reader-judge coupling
Policy realism gap
Narrow error population
Headline negative rests on a post-hoc test
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This workshop paper (COLM 2026, "Context Beyond the Window"; arXiv 2609.08279) proposes the **restore counterfactual**: a per-question paired intervention for auditing eviction in external agent-memory stores. After a policy evicts history to meet a token budget, the question's gold evidence set Gq is reinstated into read-time context and the same frozen reader re-answers. Crossing the correctness change with whether Gq survived classifies each **oracle-answerable** error as irreversible (evidence destroyed; only retention helps), recoverable (retained but missed by retrieval), or residual (still wrong with evidence restored; §3.1-3.2). The paper claims: (R1) destruction is the majority component of correctable loss; (R2) budget-accuracy frontiers are not comparable unless the retrieval regime is reported; (R3) at matched accuracy no tested policy dissociates in irreversible rate; (R4) residual mass is reader-utilization failure, mostly cross-session arithmetic. It explicitly does not rank policies or propose a memory system.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the restore counterfactual in agent memory evaluation?
It is a per-question test: after an eviction policy has deleted history to fit a token budget, the gold evidence for that question is re-injected into the model's read-time context and the same frozen model re-answers. If only restoring destroyed evidence fixes the answer, the loss was irreversible (eviction destroyed it). If the evidence had survived and restoring it flips the answer, retrieval missed it (recoverable). If the answer stays wrong, the reader failed to use the evidence (residual).
What are the paper's main findings?
On LongMemEval-S with an 80k-token budget under top-k retrieval, 67-73% of correctable errors for FIFO, random, and redundancy-aware eviction were irreversible destruction (60% for LLM-importance scoring); at an 8k budget this reaches 100% for all policies. Recoverable (retrieval-miss) errors appear only under top-k retrieval, so budget-accuracy comparisons across papers are meaningless unless the retrieval regime is reported. At matched accuracy, no tested policy differed in irreversible rate beyond a 1.2-6 percentage-point resolution.
Is code or data available for this paper?
Yes. The evaluation code, per-question records for both readers, per-cell tables with bootstrap confidence intervals, the calibration report, and all prompts are released at github.com/megagonlabs/restore-counterfactual. The LongMemEval-S dataset is public and pinned by sha256 hash; models run via the OpenAI API with dated snapshot ids.
Which eviction policy is best according to this paper?
None can be ranked from this study. At accuracy-matched comparisons, no baseline pair and no LLM-importance comparison showed a different irreversible rate within the 1.2-6 percentage-point measurement resolution; only a deliberately destructive control stood out. The authors explicitly state the instrument diagnoses failure composition but does not select a best policy.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 were assessed from their captions and surrounding text only; plotted values were cross-checked against Table 1 and §5.1 where stated, but the rendered figures themselves were not verifiable from the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.