PhysAI-Bench: A Benchmark for LLM-Based Agentic Decision-Making in Autonomous UAV-Centric Physical AI
Mohamed Amine Ferrag, Merouane Debbah, Abderrahmane Lakas, Manu Perumkunnil, Norbert Tihanyi
cs.AI
Quick verdict
Solid two-stage UAV decision benchmark (10,178 instances); best model hits 52% held-out, but labels derive from synthetic LLM traces, limiting validity.
Summary
PhysAI-Bench converts autonomous UAV mission traces into 10,178 four-way multiple-choice decision problems. Each instance is extracted from conversational episodes of the authors' earlier α3-Bench framework by truncating the dialogue immediately before an agent action, so models see only pre-decision context: mission state, sensor observations, MCP tool calls, A2A messages, and 6G network conditions. The reference label is the action actually executed in the trace; distractors are synthesized by LLM generators (DeepSeek V4.1, Qwen3.8-Max) kept blind to that action. Twenty-nine models from 14 organizations are evaluated under a two-stage protocol: twelve prompting-temperature configurations are screened on a 35-instance development set, and the winner is frozen for three runs on an episode-disjoint 500-instance held-out set. GPT-5.3 Chat leads with 52.00%, followed by GPT-5.2 Chat (49.40%) and Grok 4.5 (49.07%); development accuracies near 70% drop by 2.28-26.29 percentage points, showing selection optimism. Few-shot prompting generally helps; temperature has little effect. Main limitations: the ground truth is whatever the simulated agent did in synthetic LLM-generated traces, no human-expert accuracy is reported on the held-out set, and internal model counts are inconsistent (29 vs 30).
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- PhysAI-Bench isolates agentic action selection in Physical AI by converting UAV mission traces into 10,178 multiple-choice instances that expose only pre-decision information.
- Development-set configuration selection is systematically optimistic, with generalization gaps of 2.28-26.29 percentage points between selected-config dev accuracy and held-out accuracy.
- The best held-out decision accuracy is 52.00% (GPT-5.3 Chat), which the authors interpret as evidence that reliable agentic decision-making under physical and 6G constraints remains an open problem.
- Few-shot prompting generally improves performance across models, whereas decoding temperature (0.0-0.8) has comparatively limited influence.
Strengths
Leakage-aware, staged instance construction
Episodes are truncated before each decision (Eq. 13-14), context and candidate generators never see the executed action (Eq. 15, 'answer-blind'), stages are frozen, and answer positions are balanced 24.8-25.3% across A-D (Sec. III.C)
Disciplined two-stage evaluation protocol
Locked manifest, seeded episode-aware sampling excluding dev/demonstration episodes, prespecified tie-breaking rules (Sec. IV.D, Algorithm 2), and an explicit optimism-gap metric quantifying selection bias at 2.28-26.29 pp (Table IV)
Statistics that respect data structure
Breadth with deployment realism
Honest scope statements
Weaknesses
Label validity ceiling from synthetic trace-derived labels
Internal inconsistencies in model and generator counts
Headline ranking exceeds statistical support
Format compliance conflated with decision accuracy
Narrow decision mix and thin network coverage
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
PhysAI-Bench — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PhysAI-Bench?
A collection of 10,178 four-way multiple-choice decision problems derived from autonomous UAV mission traces. Each item shows only the information available before the decision (mission state, sensor observations, MCP tool calls, A2A messages, 6G network conditions), and the model must pick the action the trace agent actually executed.
Which model performs best on PhysAI-Bench?
On the 500-instance held-out set, GPT-5.3 Chat scores highest at 52.00% (3-shot, T=0.2), followed by GPT-5.2 Chat at 49.40% and Grok 4.5 at 49.07%. Random guessing would yield about 25%, and the top three models' confidence intervals overlap.
Is the PhysAI-Bench dataset publicly available?
Yes. The dataset is released at github.com/maferrag/physai-bench, and the held-out item identifiers, sampling seed, and frozen configurations are stored in a locked manifest. The paper does not explicitly commit to releasing the generation/evaluation code or prompt templates.
Why is the best accuracy only around 52%?
The tasks are hard and partially underdetermined: the reference label is whatever the simulated agent in a synthetic LLM-generated trace actually did, which the authors note is not necessarily the only valid decision. Parsing failures also count as errors under the strict metric, further depressing scores.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Section III.B.4 contains a garbled sentence ('internal reasoningring evaluation'), suggesting extraction corrupted part of the generator-pipeline description
- Figures 3-7 are only partially recoverable from the extracted text; bar/whisker values rely on embedded numeric labels
- Table II is densely typeset; individual price/latency/throughput cells may be misaligned in extraction
- Supplementary material (prompt templates, full distributions) is referenced but not present in the provided text
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.