Driving on Memory
Christian Löwens, Thorben Funke, Alexandru Paul Condurache
cs.CV · cs.LG · cs.RO
Quick verdict
Claims NAVSIM scores are attainable without cameras using drive memories; a convincing audit, though single-seed results weaken statistical weight.
Summary
The paper introduces MemoryDrivoR, a diagnostic probe for end-to-end driving benchmarks: camera input from the evaluated scene is removed and replaced with latent tokens retrieved from a pose-indexed memory bank of previous training traversals, injected into an otherwise unchanged DrivoR planner. Because memories carry persistent scene structure (road layout, location-conditioned regularities) but not the current traffic state, high performance under this intervention implies the benchmark score does not demand reaction to current dynamics. On NAVSIMv1 navtest, MemoryDrivoR reaches 91.1 PDMS versus 93.7 for camera-based DrivoR and 72.9 for a no-camera control; on NAVSIMv2 navhard it scores 45.0 EPDMS versus 46.7. In contrast, on closed-loop Bench2Drive it drops to 34.7 DS (camera DrivoR: 61.0), and on RealEngine the closed-loop gap widens substantially, especially with inserted counterfactual actors. HD-map inputs close less of the gap (86.6), and geographical splits (Pittsburgh, Town11+13 held out of training) show no degradation, arguing against location memorization. Ablations attribute most memory gains to relative-pose conditioning; a probing experiment shows the resampler largely filters stale agent information. Main limitations: single-seed results without uncertainty estimates, a blurry static/dynamic boundary since memories retain partial agent information (agent AP 7.9 of 10.9), and Bench2Drive comparisons rest on the authors' own DrivoR reimplementation. Code is released.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- NAVSIM scores near camera-based state of the art are achievable without observing the evaluated scene: MemoryDrivoR reaches 91.1 PDMS on NAVSIMv1 and 45.0 EPDMS on NAVSIMv2 using only ego status and pose-retrieved memories from training traversals.
- The shortcut is benchmark-dependent: the identical intervention causes large performance drops on closed-loop Bench2Drive (34.7 vs 61.0 DS) and RealEngine closed-loop (72.3 vs 88.2 PDMS).
- High NAVSIM performance is not explained by memorizing test locations, since removing Pittsburgh (NAVSIM) or Town11+13 (Bench2Drive) from training does not degrade performance on those locations.
- Memory is a richer static-information source than HD maps: the HD-map baseline reaches only 86.6 PDMS / 40.8 EPDMS versus memory's 91.1 / 45.0.
- NAVSIM's short horizon lets the initial ego status partly reveal the logged maneuver, so high collision- and traffic-light-compliance subscores do not demonstrate reaction to current actors or signals.
Strengths
Clean diagnostic intervention design
Replacing camera tokens with pose-retrieved memories isolates the value of observing current dynamics; the comparison ladder on NAVSIMv1 (Tab. 1: Ego Status MLP 66.4 → no-camera DrivoR 72.9 → MemoryDrivoR 91.1 → camera DrivoR 93.7 PDMS) is interpretable at each step.
The cross-benchmark contrast is itself a finding
The same intervention barely hurts NAVSIMv1/v2 (91.1 PDMS, 45.0 EPDMS) but causes large drops on Bench2Drive (34.7 vs 61.0 DS, Tab. 3) and RealEngine closed-loop (72.3 vs 88.2, Tab. 4), localizing the weakness to short-horizon, non-reactive evaluation rather than to memory tricks in general.
Rules out the obvious confound
Unusually thorough ablation program
Honest confound discussion
Weaknesses
No uncertainty quantification
The static/dynamic boundary is leaky
Bench2Drive audit rests on a weak reimplementation
Asymmetric trajectory-selection protocol on Bench2Drive
Ego-status vs memory contribution not decomposed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks how much end-to-end driving benchmark performance actually depends on observing the current scene. It introduces MemoryDrivoR (Sec. 3): camera input at evaluation time is removed and replaced by latent scene tokens retrieved from a pose-indexed memory bank built from training-set traversals, injected into an otherwise unchanged DrivoR planner. Because retrieved memories encode persistent scene structure but not live traffic, residual performance measures how far static information plus ego status can carry a score. Central claims: (1) NAVSIMv1/v2 scores near camera-based SOTA are achievable this way (91.1 PDMS vs 93.7; 45.0 EPDMS vs 46.7); (2) closed-loop Bench2Drive and RealEngine resist the trick (34.7 vs 61.0 DS; RealEngine closed-loop 72.3 vs 88.2); (3) the effect is not location memorization, per geographical splits (Tab. 7).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'driving on memory' mean in this paper?
The model plans driving trajectories with no camera view of the scene being evaluated. Instead, it retrieves latent scene tokens recorded by previous drives through the same location (a pose-indexed memory bank), so it knows persistent structure like road layout and typical conditions at that spot, but not the live traffic around it.
What score does MemoryDrivoR achieve without cameras?
91.1 PDMS on NAVSIMv1 navtest and 45.0 EPDMS on NAVSIMv2 navhard, versus 93.7 and 46.7 for the camera-based DrivoR, and 72.9 and 28.3 for the same planner given only ego status with no scene context at all.
Does this prove cameras are unnecessary for autonomous driving?
No. The authors explicitly warn against that reading. The finding is a benchmark diagnosis: NAVSIM's 4-second, non-reactive horizon can be nearly solved from initial ego status plus static context. On closed-loop Bench2Drive and RealEngine, where actions change subsequent sensor input, the same memory-only model performs far worse than the camera-based planner.
Isn't the high NAVSIM score just memorization of test locations?
The authors test this with geographical splits: they remove Pittsburgh from NAVSIM training and Town11+13 from Bench2Drive training, then evaluate only on those held-out areas. Performance did not drop (it even rose slightly), arguing that models perceive static scene structure at test time rather than recalling specific training samples.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure plot contents (Figs. 1-9) are not visually verifiable from the text extraction; judgments about plots rely on their captions and numbers stated in the text, all tables were extracted legibly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.