EgoPathBench: Evaluating Zero-Shot Egocentric Waypoint Decision-Making in Vision-Language Models
Yang Zhao, Zhuo Chen, Xubo Yang
cs.CV
Quick verdict
Geometry-grounded benchmark finds VLMs fail embodied route decisions (top 28.3/100); evaluator validation is strong but SFT gains lack control baselines.
Summary
The authors introduce EgoPathBench, a five-task benchmark testing whether vision-language models can make integrated, goal-directed waypoint decisions from a single egocentric RGB image. Each question overlays numbered waypoints anchored to registered 3D geometry (scenes from ScanNet, 3RScan, ARKitScenes, and Matterport3D via InternScenes); two traversability tasks score point- vs. embodied-agent candidate feasibility, and three route tasks score start validity, edge legality, goal arrival, and efficiency. The release contains 31,852 training, 1,345 validation, and 1,111 benchmark questions with geometry-verified reference routes and GPT-written spatial chain-of-thought. Across nine frontier VLMs, the best EgoPath Score is 28.3 (Gemini 3.1 Pro): Point Path success peaks at 35.9%, but embodied and intent route success collapses to 2.9% and 4.0%; diagnostics attribute failure to maintaining agent-specific edge legality across steps and to endpoint grounding, not output parsing (96.3-96.8% of outputs evaluable). A 50-question human reference (one volunteer) scores 54.2 vs. 28.6 for the best VLM on the same questions. Fine-tuning Qwen3.5-4B on the training split lifts its score from 3.9 to 38.9, above all zero-shot frontier models, and improves four external evaluations by 1.4-9.6 points. Key limitations: no control-SFT baseline isolating the source of external gains, a single-pass leaderboard whose intervals resample fixed outputs, a one-participant human reference, and public scene provenance without contamination probes. Code and data are promised upon publication.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing spatial-intelligence benchmarks measure isolated judgments, whereas EgoPathBench measures integrated waypoint decisions whose outcomes are verified against registered scene geometry.
- Across nine frontier VLMs the best EgoPath Score is 28.3, with Point Path success at most 35.9% but Embodied and Intent Path success only 2.9% and 4.0%, showing embodiment-plus-goal constraints as the binding failure mode.
- Route failures stem from maintaining agent-specific edge legality across intermediate steps and grounding the goal endpoint, not from output parsing (96.3-96.8% of predictions are evaluable routes).
- Supervised fine-tuning on the released 31,852-question training split raises Qwen3.5-4B from 3.9 to 38.9 and transfers, improving four evaluations across VSI-Bench, SpatialEval-VTQA, and 3DSRBench by 1.4-9.6 points.
- A same-interface human reference (one volunteer, 50 questions) reaches 54.2, roughly double the best VLM's 28.6 on identical questions.
Strengths
Scene-grounded evaluator rather than expert imitation
Every route question retains at least one geometrically verified reference route, and arbitrary predictions are scored by candidate feasibility, consecutive-edge legality, and goal arrival under a defined ordered contract (Table S4) - directly addressing the agreement-with-demonstrations limitation the authors attribute to NaviTrace's DTW-style scoring.
Paired task design enables factor attribution
Point vs. Embodied Traversability share views and candidate spaces (146 each), and Embodied vs. Intent Path share route geometry (309 vs. 201), isolating embodiment and target grounding. The VPR-SR gap decomposition (Fig. 3: first-edge legality 88.0-96.0% vs. full-route legality 4.8-46.8%) cleanly separates local action feasibility from sequence consistency and endpoint selection.
Unusually thorough failure diagnostics and controls
Evaluator robustness is empirically audited
Training resource is effective in-domain and transfers
Weaknesses
Training-resource study lacks a control SFT baseline
Human reference rests on one participant and 50 questions
Single-pass leaderboard with intervals over fixed outputs
No contamination probe despite fully public scene provenance
Reported intervals not verifiable in the extracted text
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is EgoPathBench and what does it measure?
EgoPathBench is a benchmark of 1,111 test questions (plus 31,852 training and 1,345 validation questions) that measures whether vision-language models can make integrated navigation decisions from a single first-person image with numbered waypoints. It has five tasks: two traversability tasks (which numbered points a point-like or 0.6 m-wide embodied agent could occupy) and three route tasks (ordered waypoint routes for a point agent, an embodied agent, and an intent-described target). Predictions are scored against registered 3D scene geometry for candidate feasibility, edge legality, and goal arrival, rather than by similarity to human demonstrations.
How well do current AI models perform on EgoPathBench?
Poorly. Across nine frontier vision-language models, the best overall score is 28.3 out of 100 (Gemini 3.1 Pro). Models do best on point-agent routes (up to 35.9% success) but succeed on only 2.9% of embodied-agent routes and 4.0% of intent-based routes. The main failures are keeping every step of a route legal for the agent's body and selecting a goal-consistent endpoint, not producing readable output, since over 96% of responses are parseable valid routes.
Is there a human baseline for EgoPathBench?
A preliminary one. A single volunteer answered 50 questions (10 per task) under the identical interface and scored 54.2 versus 28.6 for the strongest model on the same questions, with higher valid-path rates (70.0% vs. 26.7%). The authors explicitly describe this as an exploratory same-interface reference, not an estimate of a population-level human ceiling.
Is the EgoPathBench dataset and code available, and does its training data help other tasks?
The authors commit to releasing questions, annotations, chain-of-thought supervision, prompts, construction and evaluation code, and fixed model prediction records upon publication. As evidence the supervision is useful, fine-tuning Qwen3.5-4B on the training split raised its EgoPathBench score from 3.9 to 38.9 (above all nine zero-shot frontier models) and improved external spatial evaluations: VSI-Bench Route Planning (+4.13 Full / +4.38 Debiased), SpatialEval-VTQA (+9.6), and 3DSRBench (+1.4).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure images (Fig. 1-4 and qualitative panels S1-S10) could not be visually inspected; the review relies on extracted text, tables, and captions, so image/overlay quality itself was not assessable.
- Bootstrap interval values are referenced (10,000 resamples, seed 20260717) but not present in the extracted text, so statistical separation of close rankings could not be verified.
- Data and code are promised 'upon publication'; availability could not be verified at review time.
- Some supplementary analyses (model-level breakdowns, complete control protocols) are summarized in the main text rather than shown, and parts of the supplement may be truncated in extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.