IMPACT-VLA: Interaction-aware Multimodal Propagation Attribution via Counterfactual Trajectories for Vision-Language-Action Policies
cs.RO · cs.AI
Quick verdict
Closed-loop Shapley attribution over phase-modality blocks for VLA policies; careful stats on LIBERO, but gains over cheaper phase-restricted baseline unclear.
Summary
The paper introduces IMPACT-VLA, an attribution framework for vision-language-action (VLA) policies that asks when each input modality contributes to task success during closed-loop execution. A successful reference rollout is segmented into behavioral phases via CUSUM change-point detection on executed actions, boundaries are snapped to action-chunk (policy query) boundaries, and each phase-modality pair becomes a Shapley player. Blocks are scored by counterfactual closed-loop re-execution from intervention-altered states (mean/gray/blur/noise replacements) with 128 sampled permutations; cross-phase interactions, trajectory propagation, and paired behavioral/functional recovery metrics derive from the same rollouts. On 30 LIBERO tasks with OpenVLA-OFT, 25/30 tasks showed dominant-modality transitions across phases; IMPACT-VLA achieved the best deletion pAUC (0.810) and insertion pAUC (0.104) versus LOO, Static Action Perturbation, and structural ablations, but paired gains were statistically clear only over Static Action Perturbation — differences from the cheaper phase-restricted ablation were not. Held-out interactions showed complementary sign agreement of 0.900; later-block marginal gains for negatively interacting pairs rose ~3.3x under early replacement; functional recovery without behavioral recovery occurred in 62.5% of redundant-group edges. Main limitations: one policy, one benchmark, simulation only; phases fixed by a single reference rollout per task; and no code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Behavioral phases detected from action transitions and aligned to chunk boundaries form a valid attribution unit for chunk-based VLA policies.
- Closed-loop Shapley attribution over phase-modality blocks identifies task-critical blocks more faithfully than Static Action Perturbation (deletion pAUC 0.810 vs 0.670; insertion 0.104 vs 0.030).
- Dominant modality changes between adjacent behavioral phases in 25 of 30 LIBERO tasks (83.3%).
- Later-block marginal gains for negatively interacting pairs increase approximately 3.3x (0.025 to 0.082) under early-phase input replacement.
- Functional recovery of task success can occur without behavioral (trajectory) recovery, seen in 62.5% of gated redundant-group edges.
Strengths
Execution-aligned attribution unit
CUSUM changepoints on executed actions snapped to K=8 chunk boundaries (Sec. III-B) solve the intervention-validity problem for chunked VLAs; Table IV confirms action-transition-aligned boundaries beat uniform query-aligned phases (removal effect 0.948 vs 0.931) and modality-only Shapley (0.915), both paired CIs excluding zero.
Estimator calibration against exact enumeration
The synthetic closed-loop game (Sec. IV-B, Fig. 2) spans Additive, Delayed-Credit, and Genuine-Interaction scenarios and shows MAE < 0.03 and Spearman > 0.89 at the exact L=128 budget used in LIBERO — rare rigor for a sampled Shapley estimator.
Honest statistical machinery
Behavioral vs functional recovery distinction
Consequential closed-loop scoring
Weaknesses
Global formulation's payoff unproven
Single-reference-rollout dependence
Narrow external validity
No code or data release
Kernel-coupled evaluation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
IMPACT-VLA (arXiv:2609.15005, cs.RO/cs.AI) is a post-hoc interpretability method for chunk-based vision-language-action (VLA) policies. It claims three things: (1) phase-modality blocks — behavioral phases detected from executed-action transitions and aligned to policy-query (action-chunk) boundaries — are the right temporal unit for asking when each modality supports task success; (2) global cross-phase Shapley attribution computed by closed-loop counterfactual re-execution identifies task-critical information more faithfully than static or local alternatives; (3) cross-phase interactions, propagation analysis, and a behavioral-vs-functional recovery distinction reveal conditional dependencies invisible to local sensitivity. Evaluation covers OpenVLA-OFT on 30 LIBERO tasks (Spatial/Object/Goal, 10 each), with phase counts P=3-6 across tasks.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is IMPACT-VLA?
A post-hoc interpretability framework for vision-language-action (VLA) robot policies. It segments a successful robot rollout into behavioral phases, treats each phase-modality pair (e.g., wrist camera during the reaching phase) as a Shapley player, and estimates each player's contribution to final task success by replacing inputs and re-executing the policy-environment loop in closed loop.
What benchmark and policy were used to evaluate IMPACT-VLA?
30 LIBERO robot manipulation tasks (10 each from the Spatial, Object, and Goal suites) with OpenVLA-OFT as the evaluated policy, entirely in simulation. The phase structure for each task comes from a single successful reference rollout.
What did IMPACT-VLA find?
Dominant modalities shifted between phases in 25 of 30 tasks (83.3%). Its closed-loop attribution ranked task-critical blocks better than Static Action Perturbation (deletion pAUC 0.810 vs 0.670; insertion 0.104 vs 0.030). Later-phase blocks of negatively interacting pairs gained about 3.3x more when earlier inputs were replaced, and task success sometimes recovered without the robot trajectory returning to normal (62.5% of redundant-group edges).
Is code available for IMPACT-VLA?
The paper contains no code or data release statement. It builds on public components (the LIBERO benchmark and OpenVLA-OFT), but the authors do not state that their implementation, seeds, or reference trajectories are available.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 are referenced but not visually analyzable from the extracted text; figure-based content (e.g., Fig. 3 per-suite patterns, Fig. 4 interaction matrix, Fig. 5 deviation curves) relies on the authors' textual descriptions.
- The metadata abstract is truncated mid-sentence, but the full body text including all six tables and references appears intact.
- No appendix or supplementary material is visible; implementation details and compute costs may reside there.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.