GzDRL: Reproducible and Scalable Deep Reinforcement Learning with Gazebo
Amal Dev Haridevan, Junjie Kang, Jinjun Shan
cs.RO · cs.LG
Quick verdict
Claims middleware-free Gazebo RL with deterministic stepping, top CPU throughput and zero-shot quadrotor transfer; results mostly support it, with baseline-fairness caveats.
Summary
The paper introduces GzDRL, a single-process reinforcement learning framework for Gazebo that replaces ROS-style middleware communication with a direct environment-stepping mechanism built on Gazebo's ECS API and controlled physics stepping. Adapted from EnvPool's batched execution, GazeboPool dispatches environment-indexed actions to C++ worker threads and returns zero-copy NumPy observations. Benchmarks on a Threadripper workstation and a Ryzen laptop show deterministic action-to-physics synchronization versus variable multi-step delays under ROS 2, bit-identical PPO training runs under identical seeds, and peak throughput of 78.6×10³ environment steps/s at 128 parallel environments on the workstation — about 3.6× the fastest evaluated baseline — and 43.4×10³ on the laptop, close to GPU-based Aerial Gym. In multi-agent settings GzDRL leads at 2–10 UAVs on the workstation but falls behind Aerial Gym at 20 UAVs and on the laptop. Domain randomization improves evaluated return on an inverted-pendulum-quadrotor task (643 vs. 440), and a frozen policy tracks trajectories on a physical Quanser QDrone2 with 20.2 cm mean position error without fine-tuning. Limitations: fairness of baseline tuning is undocumented, reproducibility is validated only for state observations and a single short PPO task, and sim-to-real is shown only for trajectory tracking on one platform with motion capture.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- GzDRL's middleware-free, single-process stepping applies each policy action at the immediately succeeding physics update, giving deterministic transitions where ROS 2 incurs variable multi-step delays and divergent trajectories over 100 trials (Sec. V-A, Figs. 4–5).
- Identical-seed PPO runs on GazeboPool are bit-identical — 100% matching checkpoint hashes and zero return variance — demonstrating experiment-level training reproducibility (Sec. V-B, Table II).
- GazeboPool reaches 78.6×10³ environment steps/s at 128 parallel environments on a Threadripper workstation, ~3.6× the fastest evaluated baseline and ~30× over ROS 2 vectorization, and stays near GPU-based Aerial Gym on a laptop (Sec. V-C, Table III).
- In multi-agent tasks GzDRL leads at 2–10 UAVs on the workstation (e.g., 35.5×10³ steps/s at NA=2) but is overtaken by Aerial Gym at 20 UAVs and on laptop hardware (Sec. V-D, Table IV).
- A frozen policy trained entirely in simulation tracks trajectories on a physical Quanser QDrone2 without fine-tuning, with 20.2±3.4 cm mean position error over nine hardware trials (Sec. V-G, Table V).
Strengths
Unusually rigorous reproducibility validation
Sec. V-B / Table II: five process-isolated same-seed PPO runs yield zero return variance (18.43±0.00), 0.00% CV, and 100% identical checkpoint parameter hashes, with every checkpoint evaluated deterministically in a fresh process on a fixed 20-episode set — a level of instrumentation rare in tools papers.
Honest scoping where the method loses
Despite the abstract's headline, the paper reports that Aerial Gym beats GzDRL at NA=20 on the workstation (5.53 vs. 3.18×10³ steps/s, Table IV) and on the laptop at NA≥5, and that laptop single-agent throughput is 43.38±0.56 vs. Aerial Gym's 44.56±0.42×10³ (Table III) — correctly worded as 'competitive' rather than superior.
Complete internal ablation of vectorization paths
End-to-end validation chain from determinism to hardware
Public code and precise tuning recipe
Weaknesses
Baseline fairness is undocumented
Reproducibility claims are narrow
Sim-to-real lacks a control condition
Sensor-rich workloads absent from all benchmarks
Domain randomization validated on one toy task, never on hardware
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
GzDRL (arXiv:2609.13243, submitted to IEEE) is a software-framework paper for the RL-in-robotics toolchain. It advances four claims: (1) a middleware-free, single-process stepping mechanism for Gazebo (Sec. IV-A, Eq. 3) that synchronizes policy actions with physics updates deterministically, eliminating the non-Markovian action/observation pairing introduced by ROS-style transport; (2) EnvPool-style batched vectorization (GazeboPool) with zero-copy NumPy observations; (3) experiment-level reproducibility of both state transitions and RL training; and (4) zero-shot sim-to-real deployment on a quadrotor. A headline secondary result is workstation CPU throughput of 78.6×10³ environment steps/s — the highest among the evaluated frameworks.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is GzDRL?
GzDRL is an open-source reinforcement learning framework for the Gazebo robotics simulator. Unlike typical setups that connect the RL agent to Gazebo through ROS middleware (which introduces variable delays and nondeterminism), GzDRL runs the agent and simulator in a single process, stepping physics directly so actions land at the exact next physics update. It also adapts EnvPool-style batched execution for high-throughput, vectorized data collection on CPUs.
How fast is GzDRL compared to other simulators?
On a workstation (Threadripper 3960X), GzDRL's GazeboPool reached 78.6×10³ environment steps per second with 128 parallel environments — about 3.6× faster than the best evaluated baselines (PyBullet-Drones, OmniDrones, Aerial Gym, each ~21×10³) and ~30× faster than ROS 2–Gazebo vectorization. On a laptop it reached 43.4×10³ steps/s, roughly on par with GPU-based Aerial Gym (44.6×10³). With 20 UAVs per environment, or on laptops with 5+ agents, GPU simulators overtake it.
Did GzDRL policies actually run on a real drone?
Yes. A policy trained purely in simulation was deployed without fine-tuning on a Quanser QDrone2 quadrotor, with inference onboard at 100 Hz on a Jetson Xavier NX and position feedback from a 120 Hz motion-capture system. Across nine flight trials it achieved a mean position error of 20.2±3.4 cm (3D RMS 26.2±7.4 cm) tracking seeded reference trajectories at up to ~0.72 m/s, with no safety events or actuator saturation.
Is the GzDRL code available?
Yes — the paper lists a public GitHub repository: https://github.com/amaldevh/gz-drl. The paper also documents the key tuning parameters (batch size NB ≈ NE/3, thread count NT = min(NE, NCPU)), which helps others reproduce the reported scaling and throughput behavior.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure images (Figs. 1–11) are not present in the extracted text; only captions and in-text descriptions are available, so figure-level quantities (e.g., latency distributions in Fig. 4, RMSE spread in Fig. 5) could not be independently verified. All tables I–V appear fully extracted and the text ends with a complete reference list.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.