Scope Before You Persist: Preventing Cross-Family Interference in Agent Memory
Yezhou Cheng, Runjia Du, Zeming Liu, Qibai Chen, Hang Lyu, Yankai Zeng, Yilan Wei, Bojun Lin
cs.AI
Quick verdict
Claims scoped retrieval stops cross-family interference in agent memory; solid paired evidence, but from one templated benchmark and a single model.
Summary
The paper studies persistent natural-language memory in frozen LLM agents and argues that retrieval scope must match certification scope. On ProcStream-RSI, a 12-round code-repair stream with nine recurring task families and sealed hidden checkpoints, skill proposals from a frozen editor are judged by Orthogonal Regression Control (ORC), an execution-grounded gate combining public, private-test, and metamorphic channels via paired bootstrap bounds. Across eight main streams, global-memory ORC (0.713 mean trajectory utility) falls below the static agent (0.775): a locally accepted rule lifts its source family by 0.398 while degrading others by up to 0.461 — cross-family interference. A fixed-output intervention storing accepted skills in per-family slots raises utility to 0.816 (+0.103, p=0.031), and a prespecified 27-paired-stream extension shows +0.063 [0.037, 0.094], with 63 versus 12 accepted updates and zero versus six harmful acceptances. Transfer to HumanEval+ and a GPT-OSS-120B actor is modest. Strengths: paired randomized design, sealed manifests, an unusually transparent deviation log. Limitations: a single model, a templated in-distribution benchmark (the learned router's 100% accuracy is partly by construction), and per-family slots that structurally cannot harm other families.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Persistent skill edits certified on one task family harm unrelated families when deployed through a single global memory (cross-family interference: +0.398 source family vs up to −0.461 elsewhere).
- Matching retrieval scope to certification scope — storing accepted skills in per-family slots — raises mean trajectory utility from 0.713 (Global-ORC) to 0.816 under fixed proposals and gate decisions, eliminating harmful deployments (6/8 → 0).
- In 27 paired randomized-order streams, Scoped-ORC beats Global-ORC by 0.063 MTU [0.037, 0.094] (Holm p<10⁻⁴), accepting 63 vs 12 updates with 0/63 vs 6/12 harmful acceptances and ≥2 acceptances in 19/27 vs 2/27 streams.
- Even an execution-grounded gate (ORC) cannot prevent global drift because round-0 acceptance has no historical-family evidence, so strong within-scope selection (private/metamorphic AUROC 0.998) does not guarantee globally beneficial deployment.
Strengths
Clean causal isolation of the mechanism
The fixed-output intervention (§3.2, Table 4) holds proposals, gate decisions, programs, and scores fixed and changes only task-to-skill assignment, yielding +0.103 [0.052, 0.151] MTU with p=0.03125 — a genuine mechanism test, not a stronger-editor confound.
Honest reporting against the authors' own gate
Table 1 shows Global-ORC (0.713) below Static (0.775); Table 2's three prespecified contrasts are reported null; Appendix A concedes PPV drops from 1.00 (local) to 0.25 (global) with only 2/8 acceptances improving the next checkpoint.
Quantified interference mechanism
Unusual prespecification and transparency
Paired randomized extension design
Weaknesses
All main-study primary contrasts are null
Scoped arm's harm-freeness is partly structural
Templated, in-distribution benchmark
Extension estimand conflates scope with acceptance opportunity
Single editing backbone and endpoint dependence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The central claim (§1, §7) is that persistent agent memory requires two distinct decisions — whether an update is supported (certification) and where it applies (retrieval scope) — and that mismatching them causes "cross-family interference": a skill edit validated on one task family degrades unrelated families when stored in one global 360-character policy. Evidence spans three tiers on ProcStream-RSI (12-round code repair, nine recurring families, sealed hidden checkpoints): eight main paired streams, an 18-stream balanced randomized-entry replication, and a prespecified 27-stream paired extension, plus HumanEval+ and GPT-OSS-120B transfer diagnostics. Claims are mostly calibrated to evidence, but the main study's prespecified gate contrasts are null; the headline rests on the scope intervention and the extension.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is cross-family interference in agent memory?
It is the phenomenon where a prompt or skill edit that was validated on one task family is stored in a single global memory and degrades unrelated task families. In this paper, one accepted rule improved its source family by 0.398 while hurting another family by as much as 0.461 on hidden checkpoints.
What is ProcStream-RSI?
A benchmark of 12-round code-repair streams with nine recurring task families, sealed hidden checkpoints, and procedurally generated instances. It evaluates the full sequence of deployed agent policies rather than a best checkpoint, testing whether frozen-model agents keep improving via persistent skill edits.
What is ORC (Orthogonal Regression Control)?
An execution-grounded gate for skill edits: a candidate is deployed only if paired-bootstrap lower bounds show no regression across public, private-test, and metamorphic evaluation channels, with an additional veto on unsafe programs. It accepted only 1 update per stream in the main study.
Does scoped memory beat global memory for LLM agents?
In this benchmark, yes: across 27 paired streams, family-scoped retrieval improved mean trajectory utility by 0.063 (95% CI 0.037–0.094), accepted 63 updates versus 12 globally, and had zero harmful acceptances versus six. The evidence is limited to one templated code benchmark and a single model.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 appear only as captions in the extracted text; trajectory shapes and shading could not be verified.
- Reproducibility artifacts (code, manifests, traces) are claimed to ship with the supplement, but no public repository URL is given and availability was not verified.
- Several cited works are 2026 arXiv preprints; novelty versus the concurrent self-improvement literature was not independently checked.
- Minor extraction noise exists (e.g., ORC final-checkpoint CI [0.761] in Table 1 vs [0.762] in Table 4).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.