AdaRepair-Mem: Adaptive Experience Orchestration for Repository-Level Program Repair
Z. C. Luo, J. C. Guo, W. J. He, S. Y. Wang, J. C. Yu, F. M. Zhao, Y. Chen, T. Cao, et al.
cs.SE · cs.AI
Quick verdict
Adaptive memory retrieval for repo-level repair, +6.0pp SWE-Bench-Verified over ExpeRepair; plausible, but no significance tests and tiny ablation subsets.
Summary
This paper targets repository-level automated program repair with LLMs, arguing that existing memory-augmented systems (e.g., ExpeRepair) retrieve experiences suboptimally. A motivational study identifies three issues: memory stock is imbalanced across repositories (e.g., Django 205 vs Matplotlib 19 memories); more memories do not monotonically improve resolution; and memories are unevenly distributed across repair stages. AdaRepair-Mem adds three modules on top of ExpeRepair's unchanged pipeline: coverage-aware retrieval (falls back from same-repository to cross-repository and repair-type memories when fewer than 10 candidates exist, cross-repo cap 50), quality-aware selection (scores candidates by BM25 relevance, historical utility, specificity, and Jaccard redundancy, keeping top-3 per stage), and stage-aware routing (separate memory partitions for reproduction, localization, generation, refinement, validation). On SWE-Bench-Lite pass@1 rises 48.3%→51.0% (cost $1.89→$1.67) and on SWE-Bench-Verified 57.2%→63.2% (cost $1.74→$1.52). On a 150-instance stratified subset, gains are largest for medium-coverage repositories (41.2%→52.3%); token efficiency improves on all six analyzed repositories (e.g., astropy 0.95→2.12). File localization reaches 92.67% on SWE-Bench-Lite versus 79.00% (RepoRepair) and 74.67% (Agentless Lite). Limitations: no statistical significance testing, ablations on only 24 instances, single runs without variance, unclear memory provenance raising contamination concerns, and largely decorative formalization (Eq. 3 is an identity).
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing memory-augmented repository-level repair is limited by three retrieval deficiencies: imbalanced memory coverage across repositories, non-monotonic returns to memory volume, and stage-misaligned memory accumulation (Sec. I–II, Fig. 2).
- Replacing only ExpeRepair's retrieval with CAR+QAS+SAR raises pass@1 from 48.3% to 51.0% on SWE-Bench-Lite and from 57.2% to 63.2% on SWE-Bench-Verified while lowering per-issue cost (Table I).
- Gains concentrate where the design predicts: medium memory-coverage repositories improve most (41.2%→52.3%, +11.1pp), and high-coverage repositories are not harmed (Table III).
- Adaptive retrieval improves token efficiency on all six analyzed repositories (e.g., astropy 0.95→2.12 resolved tasks per 1,000 input tokens, Fig. 5), indicating better context use rather than larger prompts.
- Stage-routed memories deliver superior localization: 92.67% file localization on SWE-Bench-Lite vs 79.00% (RepoRepair) and 74.67% (Agentless Lite) (Table II).
Strengths
Clear empirical diagnosis before design
Fig. 2 grounds all three modules in data: memory stock imbalance (Django 205 vs SymPy 66 vs Matplotlib 19), a cap sweep on 24 issues showing 41.7%→45.8%→66.7% resolve rate as memory budget grows, and per-repository sensitivity (Δ=+4 on matplotlib), directly motivating CAR/QAS/SAR rather than ad hoc design.
Clean attribution via plug-in isolation
The framework replaces only ExpeRepair's retrieval strategy while keeping its five-stage pipeline and memory construction fixed (Sec. IV.A.4), so Table I gains (Verified 57.2%→63.2%, Lite 48.3%→51.0%) are attributable to the retrieval policy; the simultaneous cost drop ($1.74→$1.52) supports the selectivity narrative.
Multi-angle, internally consistent evidence
Reproducibility packaging
Weaknesses
No statistical rigor anywhere
Tiny, unrepresentative, outcome-informed ablation sets
Undefined memory provenance — contamination risk
Decorative and inconsistent formalization
Baseline choice gaps in the localization comparison
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
AdaRepair-Mem (arXiv 2609.20130; Zhejiang University + Tencent) is a retrieval-policy layer for memory-augmented repository-level automated program repair. It does not modify the repair pipeline; it wraps ExpeRepair's five-stage flow (reproduction → localization → patch generation → refinement → validation) with three modules: Coverage-Aware Retrieval (CAR), Quality-Aware Selection (QAS), and Stage-Aware Routing (SAR). Central claims: (1) existing memory retrieval is handicapped by repository-level memory imbalance (Fig. 2a: Django 205 vs Matplotlib 19 memories), non-monotonic returns to volume, and stage misalignment; (2) fixing retrieval alone raises pass@1 by +6.0pp on SWE-Bench-Verified (57.2%→63.2%) and +2.7pp on Lite (48.3%→51.0%) while cutting per-issue cost (Table I); (3) gains concentrate in low/medium-coverage repositories (Table III).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AdaRepair-Mem and how does it differ from ExpeRepair?
AdaRepair-Mem is a retrieval layer for memory-augmented repository-level program repair. It keeps ExpeRepair's repair pipeline and memory construction unchanged and replaces only the retrieval strategy with three modules: coverage-aware retrieval (falls back to cross-repository and same-repair-type memories when a repository has fewer than 10 local candidates), quality-aware selection (ranks memories by BM25 relevance, past success, specificity, and low redundancy, keeping the top 3 per stage), and stage-aware routing (separate memory pools for reproduction, localization, patch generation, refinement, and validation).
What results does AdaRepair-Mem report on SWE-Bench?
The paper reports pass@1 of 51.0% on SWE-Bench-Lite (up from 48.3% for ExpeRepair, same Claude 3.5 Sonnet + o4-mini setting) and 63.2% on SWE-Bench-Verified (up from 57.2%), with lower average per-issue cost ($1.67 vs $1.89 on Lite; $1.52 vs $1.74 on Verified). The largest gains appear on repositories with medium memory coverage (41.2%→52.3% on a 150-instance subset).
Is code or data available for AdaRepair-Mem?
Yes. Section VII links an anonymous replication package (anonymous.4open.science/r/AdaRepair-Mem-EB46) containing the implementation, benchmark configuration, ablation data, runtime data, plotting scripts, and the CSV files behind the paper's figures, so reported numbers can be checked without re-running paid LLM experiments.
What are the main criticisms of the AdaRepair-Mem paper?
The main issues are: no statistical significance testing or seed variance despite small deltas (the Lite gain is about 8 instances); ablation and generalization rest on only 24 hand-stratified instances that are unrepresentative of the full benchmark; the origin and decontamination of the memory bank on SWE-Bench is never specified, creating a possible leakage concern; and the mathematical formalization is inconsistent (Eq. 2 omits the 0.8/0.2 weights used in practice, and Eq. 3 is a trivial identity).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2, 5, 6, 7, 8, and 9 are embedded as low-fidelity text dumps; exact bar values (e.g., the four-LLM generalization chart) are only partially recoverable, so those claims were weighted accordingly.
- Table III's flattened extraction may misalign repository-to-coverage-group and Memo counts (read here as django 82 / sympy 31 high, flask 18 / sphinx 15 medium, astropy 3 / matplotlib 1 low); the 150-instance subset's per-group n is not recoverable.
- The abstract in the metadata is truncated mid-sentence, though the full text's abstract and conclusion are present.
- The anonymous replication package could not be inspected; reproducibility judgments rest on the paper's description of Section VII.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.