VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks
Liyang Fan, Yingcheng Shi, Yongbin Li, Chenghao Sun, Xin Chen, Xander Xu, Hu Wei, Shiwen Ni, et al.
cs.SE · cs.CL
Quick verdict
Rigorous executable benchmark: direct experience injection helps modestly (CIs cross zero), 11/12 memory-system pairings fail; selection bias limits scope.
Summary
VibeMemBench evaluates whether persistent memory improves executable repository coding. The authors construct 111 SWE-style targets from 90 SWE-rebench V2 repositories plus 3,634 same-repository history trajectories, using a four-stage SIEVE pipeline that validates source tasks, screens history by patch patterns, distills target-aware experience, and retains a target only when injected experience improves executable outcomes in a reference setting (deepseek-v4-flash with MiniSWEAgent, four seeds). Layer one transfers the frozen verified experience to five held-out solvers: injection raises Resolved by 1.1-4.5 points on four of five solvers (deepseek-v4-pro unchanged) and cuts agent steps on all five, but every bootstrap confidence interval crosses zero. An irrelevant-text control stays 0.7-3.4 points below baseline, ruling out text-injection effects. Layer two evaluates Mem0, SimpleMem, MemoryOS, and A-MEM retrieving from the same history: eleven of twelve solver-system pairings fail to beat matched memory-off baselines. Failure attribution over 231 failing pairings locates 69.3% at record form (transcript volume, instruction pollution), not retrieval ranking; a strip ablation shows random line deletion recovers as much as semantic stripping, implicating volume. Effects are conditioned on solver headroom: injection can only help below the four-of-four ceiling. Main limitations: every target is selected for verified reference uplift, so the benchmark measures delivery of known-useful experience rather than typical memory value; all headline gains are statistically non-significant; solvers omit proprietary frontier models.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Directly injecting execution-verified prior experience raises observed task resolution by 1.1-4.5 points on four of five transfer solvers (deepseek-v4-pro unchanged) and lowers agent steps on all five, though every bootstrap CI crosses zero.
- When four existing memory systems (Mem0, SimpleMem, MemoryOS, A-MEM) must construct and retrieve experience from the same history, eleven of twelve solver-system pairings fail to exceed the matched memory-off baseline.
- The dominant failure is the form of the supplied record (69.3% of 231 failing pairings), and a strip ablation shows the harm operates through transcript volume rather than instruction semantics.
- The downstream value of a memory record is conditioned on solver headroom rather than intrinsic to the record: the same record helps a weaker solver and harms a stronger one, and at the 4-of-4 ceiling injection can only lose seeds.
Strengths
Matched intervention protocol genuinely isolates memory
Section 3.4 and Table 11 fix model, agent, tools, sandbox, context budget, task order, and seeds; the memory-off control replaces records under the same fixed context budget, so paired differences in Resolved/tokens/steps are attributable to the memory condition alone.
Executable, auditable outcomes with explicit leakage control
Every released target keeps its test command, environment, and execution result; Table 8's visibility contract keeps target gold/test patches out of experience construction, retrieval, and solver context, and the selection audit (Table 10, 43.2% to 78.6%) is quarantined from transfer claims.
Disciplined negative-result analysis
The headroom mechanism is a substantive finding
Unusually honest statistical reporting
Weaknesses
All headline transfer gains are statistically non-significant
Selection-on-test design limits external validity
Memory systems evaluated outside their designed operating mode
Attribution boundary has low inter-rater reliability
Narrow solver and language coverage
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
VibeMemBench (arXiv 2609.23570) attacks a real measurement gap: repository benchmarks such as SWE-bench measure code changes but never isolate a memory intervention, while memory benchmarks (LoCoMo, LongMemEval, MemBench) score recall with no executable coding outcome. The paper builds 111 SWE-style targets from 90 SWE-rebench V2 repositories plus 3,634 same-repository history trajectories, and asks a strictly matched question: with agent, tools, sandbox, context budget, task order, and seeds fixed, does changing only the declared memory condition change executable task resolution? Three claims organize the paper: (1) execution-verified injected experience transfers positively to four of five held-out solvers (+1.1 to +4.5 pp on Resolved) while cutting agent steps on all five; (2) four existing memory systems fail to beat matched memory-off baselines in eleven of twelve solver-system pairings; (3) failures concentrate in record form (69.3% of 231 failing pairings), harm is traced to transcript volume rather than instruction semantics, and effects are conditioned on solver headroom. Scope boundaries are unusually candid: selection is by construction (Sections 3.3, 5), all CIs cross zero (Section 4.2), and no proprietary frontier solvers are tested (Section 6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is VibeMemBench?
It is a benchmark that tests whether memory systems help coding agents do executable repository work. It contains 111 SWE-style coding targets from 90 repositories (drawn from SWE-rebench V2) plus 3,634 prior-work trajectories from the same repositories. An agent edits the target codebase and executable tests decide success, so memory is judged by its effect on task resolution rather than by recall accuracy on stored facts.
What are the main results of the paper?
Directly injecting a prior experience whose usefulness was verified by execution raises task resolution by 1.1-4.5 percentage points on four of five solver models (one unchanged) and reduces agent steps on all five, but every reported confidence interval crosses zero. When four existing memory systems (Mem0, SimpleMem, MemoryOS, A-MEM) must build and retrieve experience from the same history themselves, eleven of twelve solver-system pairings fail to beat the no-memory baseline.
Why do existing memory systems fail on this benchmark?
The paper's failure attribution over 231 failing pairings finds 69.3% of failures come from the form of the retrieved record: transcript-style systems inject raw conversation chunks (instruction pollution, tool noise), while atomic systems compress the fix into generic facts. A strip ablation shows deleting random lines recovers about as much as deleting flagged lines, so raw record volume, not instruction content, causes the harm.
Is the benchmark biased because targets were selected for memory uplift?
Yes, by design and disclosed by the authors: each target was kept only because injecting its historical experience improved the executable outcome in a reference setting. The benchmark therefore measures whether known-useful experience is delivered and used, not how often memory helps on typical tasks; the discussion section states this boundary explicitly.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 and all tables arrive as flattened text in the extraction; numbers were cross-checked against the prose where possible, but column alignment could hide transcription errors.
- The metadata abstract is truncated mid-sentence; the full embedded text restores the complete abstract.
- The GitHub repository's actual contents (data release, manifests, judge labels) could not be inspected from the text; reproducibility claims rest on the paper's release description.
- Appendices beyond H-T are referenced (e.g., per-target listings) but not fully included, so some release-level claims cannot be independently verified here.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.