Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching
Jaewoo Park, Minyoung Lee, Sukmin Seo, Moonbin Yim, Hyunwook Yoon, Dohoon Ryu, Daehee Kim, Myungseo Song, et al.
cs.RO · cs.AI
Quick verdict
9 MLLMs flown zero-shot as drone pilots in sim: small models navigate fine but fail at declaring arrival. Well-designed, rigorous; 20-episode cells stay noisy.
Summary
The paper introduces DroneCATS-Agent, a drone control architecture in which a multimodal LLM is a swappable component whose entire action space — go (pixel plus self-estimated depth), rotate, think, finished — is declared only in the prompt, with no fine-tuning or function calling, and DroneCATS, an AirSim benchmark that holds the agent fixed and varies the model. Eighty single-drone episodes span four tasks split by target motion and initial visibility (approaching, searching, tracking, search-and-track) on two maps, plus 20 four-drone commanding episodes with near-duplicate targets; success requires the model itself to declare arrival within 5 m of the visible target. Nine models are compared: GPT-5, Claude Opus 5, Gemini 3.7 Flash, Gemini Robotics-ER 2, a Qwen3.5 ladder (2B–27B), and Cosmos3-Edge-2B. Best approaching success is 65% (Gemini 3.7 Flash); searching never exceeds 40%. Small open models enter the success radius as often as or more often than frontier models but fail at declaring arrival: Qwen3.5-9B is within 5 m in 90% of approaching episodes yet succeeds in 35%, Qwen3.5-2B declares at 1.28 times its start distance, and Cosmos3-Edge-2B never declares. In commanding, Gemini 3.7 Flash reaches 80% while GPT-5 falls to 20%, and small models copy one coordinate across all four views (58–70% of all-go steps). Limitations: only 20 episodes per cell (9–11 points of binomial noise), simulation only, and a commanding-versus-single-drone comparison the authors themselves call unmatched.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Holding the agent fixed and varying the MLLM shows even the structurally simplest embodied task is unsolved: the best model (Gemini 3.7 Flash) succeeds in only 13 of 20 approaching episodes, and no model exceeds 40% on searching.
- Small open models fail in protocol adherence rather than navigation: Qwen3.5-9B enters the 5 m success radius in 90% of approaching episodes (more than any frontier model) but converts only 35%, while Qwen3.5-2B declares arrival at 1.28 times its start distance and Cosmos3-Edge-2B never declares.
- Multi-drone commanding amplifies the divide: Gemini 3.7 Flash wins 80% of N=4 episodes while GPT-5 drops to 20%, and small models copy one identical coordinate into all four views (70% of Qwen3.5-9B all-go steps, 58% for 27B, vs 0–1% for frontier).
- Prior systems engineer termination outside the model via distance thresholding, which is why this declaration-discipline gap has not appeared in the systems literature; making arrival a model claim makes it measurable and is a precondition for hierarchical delegation.
Strengths
Model as the only experimental variable
Unlike prior drone systems that vary pipelines and hold the model fixed (Fly0 sweeps backbones only for grounding; OnFly uses one 4B model for all baselines), DroneCATS-Agent pins prompt, parser, controller and flight stack, then swaps nine MLLMs through the identical verbatim prompt (Listing 1) with pinned decoding (A.3). Swapping changes nothing else, so model-attributed differences are clean.
Declaration-anchored criterion with a genuine sensitivity audit
Appendix B re-derives all 720 one-drone episode-runs offline and quantifies what alternative criteria would do: first-declaration judging flips 95 of 182 successes, a 10 s dwell window fails 65 of 99 tracking successes, and trajectory-proximity credit would inflate successes by a third (61 of 337 never-declaring episodes entered the 5 m radius). The metric is shown to drive the conclusions, not just report them.
Self-critical error analysis
The declaration-gap finding is localized with converging evidence
Weaknesses
Underpowered cells undercut fine-grained orderings
Depth is never validated
Commanding comparison is confounded
Simulation-only with an idealized flight stack
Inverse-scaling claim generalizes from thin evidence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
DroneCATS (NAVER Cloud Drone AI Team; arXiv 2609.01404) asks which parts of closed-loop drone control actually depend on the MLLM. It contributes two artifacts. DroneCATS-Agent places a swappable MLLM in the control loop with its entire action space — go (a 0–1000 pixel plus a self-estimated depth), rotate (in-place yaw), think (a paid deliberation step), finished (a self-declared arrival) — declared only in the prompt, with no fine-tuning and no function-calling schema (Listing 1). DroneCATS is an AirSim benchmark crossing two binary axes (target motion × initial visibility) into four task cells over 80 single-drone episodes on two maps, plus a 20-episode commanding suite at N=4 with near-duplicate targets identifiable only by close-range inscription. One criterion scores everything: an episode succeeds when the model declares arrival within δ=5 m of the visible target (Algorithm 1). Central claims: the simplest cell is unsolved (best 65%); small open models fail at protocol adherence, not navigation; commanding amplifies the divide. All three are supported descriptively, with real caveats about statistical power.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DroneCATS?
A simulation benchmark and agent architecture from NAVER Cloud that evaluates multimodal LLMs as zero-shot drone pilots. The model controls a drone in AirSim using four prompt-declared actions (go, rotate, think, finished) across approaching, searching, tracking, search-and-track, and four-drone commanding tasks. An episode succeeds only if the model itself declares arrival while within 5 m of the visible target.
Which model performed best in the DroneCATS evaluation?
Gemini 3.7 Flash overall: 65% success on approaching, 80% on tracking, and 80% on the four-drone commanding suite. GPT-5 nearly matched it on approaching (60%) but tracked a moving target in only 15% of episodes and fell to 20% in commanding. No model exceeded 40% on searching.
What is the paper's main finding about small models?
Small open models often fly into the 5 m success radius more reliably than frontier models (Qwen3.5-9B: 90% of approaching episodes versus 65–70% for frontier models) but then lose the episode by declaring arrival too early (Qwen3.5-2B at 1.28 times its start distance) or never declaring at all (Cosmos3-Edge-2B). The authors call this a protocol-adherence failure, not a navigation failure.
Is DroneCATS evaluated on real drones or only in simulation?
Only in simulation: AirSim 1.8.1 on Unreal Engine with the SimpleFlight controller. The authors argue the body-frame velocity setpoint their agent emits matches what real autopilots accept in offboard mode, so hardware flights are possible, but none are reported in this paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (1–13) are present only as caption text in the extraction; visual content such as the Figure 7/8 bar magnitudes and Figure 11/12 mosaic details was assessed from captions and surrounding prose, not from the images themselves.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.