Towards Embodied Air-Ground Cooperative Object Search: Benchmark, Dataset and Agentic Method
Boao Yu, Zimo Chen, Junreng Rao, Yue Hu, Zhengqiu Zhu, Yong Zhao, Rusheng Ju
cs.CV · cs.AI
Quick verdict
第一个 UAV-UGV 物体搜索基准测试 + 通过提升 8/9 个 VLM SRs（平均 +22.5pp）实现的智能体协议；可信且详尽，但缺乏显著性测试；仅有一个测试城镇。
Summary
本文定义了空地目标搜索（AGOS）：无人机（UAV）和无人车（UGV）必须在模拟城市环境中联合定位由裁剪的多视角参考图像指定的目标车辆，只有当 UGV 在 50 米内停下并从六视角观测中目视确认目标时，该轮测试才算成功。作者在开源 CARLA-Air 平台上构建了 AGOS-Bench，并将其定位为首个专注于 VLM 闭环 UAV-UGV 协同搜索的基准测试，通过成功率、oracle SR、SPL、导航误差以及四个新的分阶段指标（空中搜索效率、候选点引导、地面验证准确率、决策步骤）进行评估。配套的 AGOS-Dataset 包含分布在五个 CARLA 城镇的 7,700 个训练/验证轮次，以及位于不同城镇的 210 个留出测试轮次，分为三个难度级别，由基于自动 mTSP 的轨迹管线生成。AGOS-Agent 遵循搜索-移交-验证协议，将 VLM 骨干网络封装在特定角色的 UAV/UGV 策略中，将几何投影和路径规划委托给确定性工具。在固定的 210 个测试轮次上对九个 VLM 进行测试，八个骨干网络的总体成功率有所提高（平均 +22.5 个百分点），所有九个网络的决策步骤均有所减少（平均 -33.3）；Gemini-3.6-Flash 在困难集上的 SR 从 8.6% 提升至 55.7%（SPL 从 7.6% 提升至 44.0%），但仍低于人类参考的 89.3%。在 Qwen3-VL-4B 上进行的 2×2 消融实验表明，E2A 投影工具甚至可能损害成功率，而基于道路的规划则至关重要。主要局限性：缺乏统计显著性检验、仅有单一测试城镇、目标为静态、通信理想化，且组件级归因仅限于最弱的骨干网络。
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- AGOS-Bench is the first dedicated benchmark evaluating whether general-purpose VLMs can perform closed-loop UAV-UGV cooperative object search in urban environments.
- AGOS-Dataset provides 7,700 expert-trajectory episodes across five CARLA towns plus a 210-episode held-out test split on a disjoint town, spanning three difficulty levels.
- The training-free AGOS-Agent protocol improves overall success rate for eight of nine VLM backbones (mean +22.5 percentage points) and reduces decision steps for all nine (mean −33.3).
- On the hard split, Gemini-3.6-Flash improves from 8.6% to 55.7% SR with AGOS-Agent, yet the best VLM still trails the 89.3% human reference.
- Stage-wise metrics identify wide-area exploration, cross-view candidate grounding, and cautious ground verification as separable bottlenecks.
Strengths
Genuinely novel task formulation with a defensible 'first' claim
Table 1 systematically contrasts AGOS-Bench against CityAVOS (single-UAV), CityEQA/CityCube (QA), AirGroundBench (QA + instruction following), MECoBench (household) and AeroDuo (dual-UAV VLN); the coupled search–handoff–verify loop ending in a confirm/reject decision is not covered by any listed prior benchmark.
Unusual methodological transparency
Appendices A.1–A.5 specify the 40 m UAV grid at 80 m altitude with road-proximity filtering, the frozen OpenDRIVE UGV decision graph (~10 m anchors), closed-form definitions of all eight metrics, the mTSP expert pipeline with >80% visibility-overlap waypoint pairing, representative prompt fragments for both methods, and the full 2×2 ablation design.
Arithmetically consistent headline numbers
Honest negative and boundary results
Stage-wise diagnostic metrics that dissociate failure modes
Weaknesses
No statistical inference whatsoever
Headline comparison conflates protocol with tools
Unexplained discrepancy in the human-gap figure
Thin external validity and asymmetric metrics
Human reference under-specified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper formulates Air-Ground Object Search (AGOS): a UAV and a UGV must jointly locate a target vehicle, specified only by cropped multi-view reference images, in a simulated urban region; the episode succeeds only when the UGV halts within 50 m and visually confirms the target from its six-view observation (Task Description section). Three contributions are claimed: AGOS-Bench, positioned as the first benchmark dedicated to closed-loop UAV-UGV cooperative object search for VLMs (Table 1 contrasts it with CityAVOS, CityEQA, CityCube, AirGroundBench, MECoBench, AeroDuo); AGOS-Dataset, 7,700 train/validation episodes over five CARLA towns plus 210 test episodes on a disjoint town in three difficulty tiers; and AGOS-Agent, a training-free tool-augmented protocol. Headline results: across nine VLM backbones on the fixed 210-episode Town03 test set, overall SR improves for eight backbones (mean paired +22.5pp) and DS drops for all nine (mean −33.3); Gemini-3.6-Flash hard-split SR 8.6%→55.7%, SPL 7.6%→44.0%, versus a 89.3% human reference. Every headline aggregate I re-derived from Table 3 reconciles exactly — a nontrivial consistency check the paper passes.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AGOS-Bench and what task does it evaluate?
AGOS-Bench is a simulation benchmark built on CARLA-Air (CARLA plus AirSim in one Unreal Engine process) that tests whether vision-language models can coordinate a drone and a ground vehicle to find and verify a target vehicle in a virtual city. An episode succeeds only when the ground vehicle stops within 50 m of the target and correctly confirms it from a six-view observation. It offers easy/mid/hard difficulty levels and stage-wise metrics covering aerial search, candidate guidance, and ground verification.
How much does AGOS-Agent improve VLM performance?
Across nine VLM backbones on a fixed 210-episode test set, AGOS-Agent raised overall success rate for eight of nine models (mean +22.5 percentage points) and reduced decision steps for all nine (mean −33.3). The largest reported gain is Gemini-3.6-Flash on the hard split: success rate 8.6% → 55.7% and SPL 7.6% → 44.0%, still below the 89.3% human reference.
Is the AGOS dataset publicly available?
The paper states AGOS-Dataset is released as a companion resource: 5,500 training and 2,200 validation episodes over five CARLA towns plus 210 held-out test episodes on a separate town, in three difficulty levels, with expert trajectories intended for fine-tuning. It is built on the open-source CARLA-Air platform; the provided text shows no explicit code repository URL.
What are the main limitations of this work?
Per the authors: evaluation uses only 210 episodes from one test town; targets and distractors are static vehicles; communication, localization, and the map are idealized; control is high-level waypoint actions rather than low-level flight/driving; confirmation is scene-level rather than instance-level; and there is no simulator-to-real transfer. The paper also reports no statistical significance tests, and its component ablation covers only the weakest backbone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 4's caption annotation string is garbled by a font-encoding artifact in the PDF extraction (e.g., '&DVH %DVHOLQH'); the same decision counts (36/44 baseline vs 20/9 agent) are legible in the body text, so no information is lost.
- Figures 1 and 3 are images whose contents are not in the text extraction; the stage-wise values cited in this review (e.g., Gemini-3.6-Flash 68.5% ASE / 49.4% ACG / 71.5% GVA) come from the body prose that quotes them.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.