MEMOBench: A Process Level Memory Benchmark for Robotic Manipulation
Haiyang Sun, Haoxiao Wang, Junming Chen, Weicheng Fang, Zihao Su, Jingkun Yi, Wenyou Yi, Hao Chen, et al.
cs.RO · cs.AI
Quick verdict
Claims first process-level memory benchmark for manipulation; strong metrics and controls, but single-seed eval, modest gains and sim-only temper confidence.
Summary
MEMOBench is a simulation benchmark for evaluating memory in Vision-Language-Action (VLA) manipulation policies at the process level, not only via final task success. Built on LIBERO/robosuite with BDDL task specifications, it provides 30 history-dependent tasks across four memory types (temporal, spatial, object, procedural), 1,500 expert demonstrations, and 4,200 demonstration-level checkpoint instances from 84 templates. Each checkpoint pairs coarse-to-fine language with an executable simulator predicate and one mutually exclusive operation label — Storage, Update, or Compression — defining Memory Storage/Update/Compression Rates (MSR/MUR/MCR) alongside success rate. Evaluating six fine-tuned policies across replan intervals {5,20,35,50}, the best memory-augmented baseline (π0.5 + RoboMME memory module) reaches 31.9% average SR; high storage often coexists with weak update/compression (e.g., π0 hits 94.0% temporal MSR but 4.4% temporal SR at step 5). Checkpoint text also supervises three alignment losses on the same backbone, with contrastive alignment raising step-5 SR to 34.7%. Oracle memory-text injection recovers much of the update/compression gap (temporal SR 2.2%→54.9%), while shuffled-memory and granularity controls probe causality. Limitations: simulation only, the blind relabeling study shows Update/Compression label confusion (Fleiss' κ=0.799), and evaluation uses a single seed without significance testing.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- MEMOBench is the first robotic manipulation memory benchmark with process-level memory-operation annotations (Storage/Update/Compression) tied to executable simulator predicates.
- Current VLA policies, including memory-augmented ones, store initial information far better than they update or compress it: the best baseline (π0.5 + RoboMME memory module) reaches only 31.9% average SR, with e.g. 94.0% temporal MSR but 4.4% temporal SR for π0 at replan step 5.
- Checkpoint language is usable as training supervision: auxiliary semantic, contrastive, or framewise alignment losses yield modest gains (contrastive: 31.9→34.7% average SR at step 5) that specialize by memory operation.
- Oracle decomposition attributes much of the Update/Compression failure to memory content itself: appending ground-truth memory text lifts temporal SR from 2.2% to 54.9% and closes 45.5% of the MUR gap.
Strengths
First process-level memory-operation annotation layer
84 executable checkpoint templates (35 S / 32 U / 17 C) pair coarse-to-fine language with simulator predicates and mutually exclusive Storage/Update/Compression labels, enabling MSR/MUR/MCR (§3, §4) — a diagnosis final SR cannot provide (e.g., π0: 94.0% temporal MSR vs 4.4% temporal SR at step 5, Table 2).
Exemplary causal controls beyond the leaderboard
Oracle memory-text injection (Table 4: temporal SR 2.2%→54.9%, spatial MUR 0.3%→84.1%, procedural untouched), shuffled-subgoal training (Fig 5: 2% vs 8% final success), and granularity ablation (Fig 6) probe attribution rather than only ranking methods.
Unusually transparent annotation-quality analysis
Failure localization with concrete taxonomy
Detailed reproducibility reporting
Weaknesses
No variance or significance reporting
Update/Compression boundary is systematically ambiguous
Object memory family adds little diagnostic signal
Alignment gains are modest and replan-setting dependent
Simulation only, with best-of-four headline selection
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MEMOBench (arXiv:2609.07047, Zhejiang University) is a benchmark paper for **process-level memory evaluation** in robotic manipulation. The authors make three claims: (1) it is the first manipulation memory benchmark with process-level memory-operation annotations — Storage (∅→M), Update (M→M′), Compression (M→M̂) — each tied to an executable simulator predicate; (2) current standard and memory-augmented VLA policies are far from solving it (best: 31.9% average SR, π0.5 + RoboMME memory module at replan step 5), with storage strong but update/compression weak; (3) checkpoint language doubles as training supervision for memory alignment, yielding modest gains (§5.6). The suite: 30 history-dependent tasks on LIBERO/robosuite (BDDL), 1,500 demonstrations (927,444 steps, 340.56 GiB), and 84 checkpoint templates expanding to 4,200 demonstration-level instances (1,750 S / 1,600 U / 850 C).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MEMOBench?
A simulation benchmark for evaluating memory in robotic manipulation policies at the process level. Built on LIBERO/robosuite, it contains 30 history-dependent tasks, 1,500 expert demonstrations, and 4,200 checkpoint instances (from 84 templates), each labeled with one memory operation — Storage, Update, or Compression — and verified by an executable simulator predicate.
What metrics does MEMOBench report?
Besides task success rate (SR), it reports Memory Storage Rate (MSR), Memory Update Rate (MUR), and Memory Compression Rate (MCR), which measure whether a policy completes checkpoint predicates tied to encoding new information, revising state after changes, or consolidating accumulated history — separating memory fidelity from manipulation execution.
How well do current robot policies perform on MEMOBench?
Poorly on memory-hard task types: the best memory-augmented baseline (π0.5 with a RoboMME memory module) averages 31.9% success, and even MemER with privileged checkpoint supervision reaches 56.4%. Models often store initial information well (high MSR) but fail to update stale state or compress history — e.g., π0 posts 94.0% temporal MSR against only 4.4% temporal success.
Is MEMOBench's code and data available?
The paper states the code will be released under Apache-2.0 and the dataset, task specifications, and annotations under CC BY 4.0, via the project page github.com/Collab-Gen/MEMOBench. At publication this release is described as planned, not yet confirmed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-embedded text (Figures 1, 2, 4–7) extracts as fragmented labels; figure-level readings rely on captions and body text, which repeat the key numbers (e.g., 4/50, 3/50 success counts).
- The extraction includes all sections through Appendix H (Table 11) and appears complete; no truncation was detected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.