A Closed-Loop Evaluation of Capability Loss and Recovery in Compressed Driving Policies
Ahmad Alfan Alfian Irfan, Nur Ahmad Khatim, Mansur Arief
cs.AI · cs.CV
Quick verdict
Closed-loop Duckietown study shows pruning, not quantization, first breaks a compressed driving policy; careful evidence but one tiny policy, one simulator.
Summary
The paper presents a stage-wise, closed-loop evaluation of a driving policy as it passes through a standard compression pipeline: structured pruning, knowledge distillation, and quantization. The authors formulate lane-following with pedestrian avoidance and stop-line compliance in Gym-Duckietown as a POMDP, fuse YOLO11n and MobileNetV3-small measurements into a 29-dimensional EKF-based belief, and train a 73,986-parameter PPO actor evaluated on five curricula over eight seeds each. Structured pruning to 64 hidden units (6,210 parameters, a 91.6% reduction) fails all five curricula. Distillation recovers behavior exactly as far as its rehearsal coverage extends: C4-focused rehearsal restores only the two stop curricula, while balanced rehearsal across all curricula restores full FP32 behavior. From that shared checkpoint, static INT8 PTQ again fails the two stop-and-resume curricula — freezing at the stop line (3/8 completions on C3 vs 8/8 for FP32) — while teacher-guided QAT fails oppositely by driving through stops (violations 4/8 and 7/8). An FP16 cast and quantization of the unpruned actor preserve all five curricula. Latencies are 12.9 μs (INT8), 19.4 μs (FP32), 24.5 μs (FP16). The study is candid about limits: one small policy, one simulator, ad-hoc acceptance thresholds, no factorial proof of operation ordering, and only eight seeds per decision without statistical tests.
Review complete
Assessment at a glance
3
risk flags identified
● high 0
● medium 2
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Structured pruning to width 64 (6,210 parameters, a 91.6% reduction) is the first pipeline stage at which the belief-based driving policy loses capability, failing all five curricula.
- Post-pruning distillation recovers capability exactly as far as its rehearsal coverage extends: C4-only rehearsal restores only the stop curricula, while balanced rehearsal restores the full tested FP32 behavior.
- From the same byte-identical checkpoint, INT8 (PTQ freezing at stop lines, QAT driving through them) re-breaks the two stop-and-resume curricula, while an FP16 cast preserves all five.
- Action-level fidelity metrics mispredict closed-loop task acceptance in both directions on this model family and cannot substitute for closed-loop testing.
Strengths
Stage-wise closed-loop attribution
Evaluating all ten configurations A0-A9 (Table III) inside the loop locates capability loss at pruning (A1 fails 5/5 curricula) and recovery at distillation — evidence a final-model score structurally cannot produce (Sec. V, Fig. 3).
Factor-isolating controls
A2 vs A3 differ only in rehearsal coverage; A6, A8, A9 branch from a byte-identical A3 checkpoint; A4 and A5 isolate quantization without distillation — clean causal contrasts (Sec. IV-A).
Failure-mode forensics
Pre-registered acceptance and engineering honesty
Weaknesses
n = 8 with no inferential statistics
Single point in architecture and pipeline space
Acceptance criterion under-specified
QAT conclusion under-supported
No code or data release stated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks four linked questions about compressing a learned driving policy: (i) at which pipeline stage does capability first break, (ii) can pruning damage be recovered, (iii) does rehearsal data decide what returns, and (iv) does recovery survive a later precision reduction (Sec. I). It answers them with a single belief-state PPO policy in Gym-Duckietown — lane following with pedestrian avoidance and stop-line compliance across five curricula C0-C4 (Table I) — pushed through structured pruning, distillation, and quantization into ten configurations A0-A9 (Table III), each judged by a pre-fixed closed-loop acceptance function (Eq. 4) rather than the training reward. The claims are deliberately scoped: the authors disclaim a factorial proof of operation ordering (Sec. V-C) and any claim about internal representations (Sec. VI), and frame the contribution as an empirical demonstration that a compression pipeline for an acting policy is a sequence of behavioral transitions.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 3 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What simulator and driving tasks does the paper use?
Gym-Duckietown, with five curricula of increasing difficulty: plain lane following (C0), a larger loop (C1), crossing pedestrians (C2), stop-sign compliance (C3, stop-then-resume), and combined pedestrians plus stop sign (C4). Each candidate policy is evaluated over eight fixed seeds per curriculum.
What is the paper's main finding?
That structured pruning (from 256 to 64 hidden units) is the stage where driving capability is first lost; distillation afterwards recovers only as much behavior as its rehearsal data covers; INT8 quantization of the recovered narrow actor then re-breaks the two stop-and-resume curricula (freezing or driving through stops), while an FP16 cast preserves them.
Does INT8 quantization always break driving policies?
No. In this study the unpruned 256-unit actor passed all five curricula after INT8 post-training quantization. The failures occurred on the pruned-and-distilled 64-unit actor, and specifically on curricula requiring the vehicle to stop and then resume driving.
Is code or data available for this paper?
The reviewed text does not state a code or data release. It describes a public simulator (Gym-Duckietown) and a 62,176-state balanced rehearsal set, but no repository link for the authors' own pipeline, thresholds, or telemetry is given.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Fig. 3's decision grid and Fig. 2's pipeline diagram are extracted as unaligned text streams, so per-candidate verdicts were taken from prose and Table IV rather than decoded from the figures; candidate A7's C3/C4 outcomes are not stated in the prose at all.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.