Beyond the Leaderboard: Counterfactual Diagnosis of End-to-End and VLA Driving Policies Under Domain Shift
Ruolin Yang, Zilin Huang, Buoyue Wang, Zhengyang Wan, Yuhao Luo, Zihao Sheng, Sikai Chen
cs.CV · cs.AI
Quick verdict
Counterfactual edits diagnose six driving policies' pedestrian response; pre-registered transfer test is rigorous, though open-loop replay limits conclusions.
Summary
The paper argues that leaderboard ranks do not predict how end-to-end or vision-language-action driving policies behave near pedestrians at a new site. It proposes the 'Check-Up': a few hundred logged frames (nuScenes Boston/Singapore, plus 246 NAVSIM Singapore near-pedestrian scenes) are edited two ways at pixel level — the target pedestrian removed via SAM segmentation and LaMa inpainting, or re-lit by a fixed photometric night transform — with every edit verified by an independent Faster R-CNN detector. Plan changes define two causal axes: faithfulness F (pedestrian removal) and interference I (lighting). Five exams measure planned distance (exposure), hazard sensitivity, its scaling with required deceleration, specificity (plan steadiness when no reaction is needed), and the CFR lighting ratio. Six released policies are diagnosed. Where the pedestrian lies on the planned path, only 15 of 787 cells (1.9%) show genuine avoidance, with median clearance change at most 0.03 m; CFR < 1 for every policy, i.e., the irrelevant lighting edit moves plans more than the pedestrian does. A pre-registered Boston-to-Singapore transfer test confirms 4 of 7 predictions (exposure ρ=0.90 and specificity ρ=0.60 orderings, the lighting verdict, the collision outcome) while point values and the hazard-sensitivity verdict fail. Verdicts are priced: lighting settles within 14 frames, exposure within 51, hazard sensitivity needs hundreds. Limitations: open-loop replay, photometric-only night edit, small frame counts, six-policy correlations, and code not yet released.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Leaderboard rank on nuScenes open-loop error or NAVSIM EPDMS does not predict near-pedestrian behavior at a new site (|ρ| ≤ 0.43 across six released policies, Sec. IV-E).
- On 246 NAVSIM near-pedestrian scenes, only 1.9% of cells where the pedestrian lies on the planned path (15/787) are genuine avoidance, with median clearance change ≤ 0.03 m under the open-loop protocol.
- The night-style photometric perturbation moves every policy's plan further than removing the pedestrian (CFR = 0.23–0.47, all 95% CIs below 1, Table I).
- In the pre-registered Boston→Singapore transfer, the exposure (ρ = 0.90) and specificity (ρ = 0.60) orderings, the lighting verdict and the collision outcome transfer, while point values and the hazard-sensitivity verdict do not.
- Each verdict is priced in frames: lighting settles within 14, exposure within 51, but hazard-sensitivity conclusions need hundreds of frames (57–1,326, partly extrapolated).
Strengths
Pre-registered transfer test with fixed decision rules
Table II fixes seven predictions (P1–P7) and thresholds on 88 Boston frames before computing any Singapore statistic; three failures are reported prominently, including the authors' own auxiliary P6 failing with opposite sign (ρ = −0.90), and Alpamayo-1.5 is excluded from the registration-scored table to preserve discipline after a mid-study model replacement.
Interventions are verified, not assumed
An independent Faster R-CNN detector (COCO, score ≥ 0.5) confirms 100% target recovery on originals, 8.5% residual under removal (ER), 89.9% under re-lighting (EN), and the inpainting artefact is bounded at 5–17% of the removal effect by pasting patches on empty road (Sec. III-A.c).
Physical, untuned constants and human reference
Verdicts are priced with sample-size laws
Displacement decomposed into direction
Weaknesses
Headline avoidance rate mixes two definitions
The 'irrelevant edit' premise is contestable
Small-n statistics on the transfer orderings
Domain-shift scope narrower than the framing
Reproducibility currently unverifiable
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a decision problem, not a leaderboard: when choosing among released end-to-end / VLA driving policies for a new site, a benchmark rank (nuScenes open-loop L2, NAVSIM EPDMS) is an outcome measure that 'can be right for the wrong reason.' The proposed Check-Up builds pixel-level counterfactual pairs from a few hundred logged frames — the target pedestrian deleted (SAM prompts → LaMa inpainting; LiDAR points removed for the fusion policy) or the frame re-lit by a fixed photometric night transform (γ=2.2, gain 0.45, saturation 0.45, noise 7/255) — with every edit verified by an independent Faster R-CNN detector (100% recovery on originals, 8.5% residual detections after removal, 89.9% after re-lighting; inpainting artefact bounded at 5–17% of the removal effect by pasting patches on empty road). From paired plans it reads two causal axes — faithfulness F (pedestrian removal) and interference I (night edit) — and five exams: exposure, hazard sensitivity HS (four normalized surrogate-safety differences, Eq. 7), its Spearman scaling against required deceleration a_req (Eq. 8), specificity SP, and the causal faithfulness ratio CFR. Six released policies are examined (LTF, DiffusionDrive, DiffusionDriveV2, SimLingo, AutoVLA, Alpamayo-1.5) on nuScenes Boston/Singapore near-pedestrian frames and 246 NAVSIM Singapore scenes.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper's 'Check-Up' actually do?
It diagnoses driving policies by editing real logged frames two ways — deleting the pedestrian (SAM segmentation plus LaMa inpainting) or applying a night-style photometric change — verifying every edit with an independent detector, and reading how the planned trajectory changes. From those paired plans, five exams measure planned distance, hazard sensitivity, whether that response scales with danger, plan steadiness when no reaction is needed, and how much the irrelevant lighting edit moves the plan.
What does the 1.9% figure mean?
Across 246 NAVSIM near-pedestrian scenes and six released policies, the authors replayed scenes at 4 and 8 m/s and looked at cells where the pedestrian lies on the planned path. Only 15 of 787 such cells (1.9%) show genuine avoidance — the plan moves at least 0.5 m and buys at least 0.5 m of clearance. The median clearance change was at most 0.03 m, meaning most policies neither divert nor slow when facing a pedestrian they would reach.
Did the pre-registered transfer test succeed?
Partly. Seven predictions were fixed on Boston (left-hand-drive) frames and tested on Singapore (right-hand-drive) frames. Four held: the lighting verdict (CFR < 1 for every policy), the exposure ordering (ρ = 0.90), the specificity ordering (ρ = 0.60, exactly at the registered threshold), and the collision outcome. Three failed: point values drifted (only 60% of cells stayed inside the Boston confidence intervals), the hazard-sensitivity verdict crossed its 0.15 line, and an auxiliary orthogonality prediction reversed sign.
Is code or data available for this paper?
The paper states that code and the edited frames 'will be released,' but at the time of this review they were not yet available. All six policy checkpoints and the underlying datasets (nuScenes, NAVSIM) are public, so the pipeline is rebuildable, but exact reproduction currently requires reimplementing the SAM/LaMa editing and verification steps.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 are referenced but not analyzable from the extracted text; quantitative statements rely on in-text numbers and text-form tables.
- Table I is dense (7 columns × ~14 rows); column-to-policy attribution was cross-checked against in-text claims (e.g., exposure 0.50 = DiffusionDrive, 2.19 = SimLingo), but minor misalignment risk remains for a few cells.
- The abstract in the metadata is truncated mid-sentence; conclusions were drawn from the full-text sections instead.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.