ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs
Yizhan Li, Jianxin You, Mengyang Xiong, Yinhuan Chen, Zicheng Zhao, Dekun Wu, Dongqing Zhang, Bang Liu
cs.RO · cs.AI
Quick verdict
Reproducible simulated benchmark scores MLLM reactive hazard decisions; strong diagnostic design, but findings rest on descriptive stats, unvalidated metrics.
Summary
ReactHuman is a benchmark that asks whether multimodal large language models can react to sudden household hazards the way a competent person does. The evaluated model serves as the brain of a simulated Unitree G1 humanoid: a 240 Hz rigid-body simulation in Genesis runs a sudden event for roughly 0.6 seconds, freezes, and the model must commit to a structured plan — a walking command plus hand keyframes — that a pre-trained RL controller then executes, so every decision has an observable physical outcome. Seventeen event families span more than 1,000 bit-for-bit reproducible scenes with exact simulator-derived ground truth, including 14 adversarial objects whose appearance contradicts their physics; five metrics score reasonability, safety, and physical grounding separately. Evaluating seven MLLMs zero-shot on 306 scenes (2,138 decisions), the authors report 54.0% mean action accuracy and 80.8% mean safety; GPT-5.5 leads accuracy (63.7%), the 27B Gemma is safest (88.2%), and Claude Opus 4.8 is least safe (64.7%), with no scale trend. On Dodge scenes 35.9% of decisions violate a safety rule; models freeze under danger (162 of 410 violations), act from fixed dispositions, treat speed as binary (42% versus roughly 21% freeze rate at the slowest band), and never question an object's material across 280 adversarial decisions. Limitations include rigid-body-only simulation, open-loop single-shot decisions, keyword-matched intent scoring, and reliance on proprietary API snapshots.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ReactHuman is the first physics-grounded benchmark measuring whether MLLMs can convert physical understanding into immediate, safety-critical action, with exact annotation-free ground truth from 240 Hz rigid-body simulation.
- Across seven MLLMs on 306 scenes (2,138 decisions), models mishandle roughly one hazard in three (mean SAA 54.0%, safety 80.8%), and neither accuracy nor safety improves with model scale.
- Models act from fixed per-model dispositions rather than the observed scene, and safety failures concentrate on scenes requiring evasion (35.9% of Dodge-scene decisions violate a safety rule).
- Models perceive speed only as moving-or-not, freezing on 42% of slowest-band hazards versus roughly 21% at faster bands, and across 280 adversarial decisions they never revise appearance-based material judgments from observed motion.
- Even when the chosen action is correct, hands land a median 0.48 m from the true impact point — a failure only physical execution, not answer-only evaluation, can reveal.
Strengths
Decision quality isolated from latency and controller skill
Freeze-and-predict (§3.1) pauses simulated time so every model faces an identical decision problem, and the brain–spine–body split delegates balance and locomotion to a pre-trained RL controller, attributing scores to reasoning rather than policy quality.
Exact, scalable ground truth
Labels (action, 3D impact point, time-to-floor) are read directly from simulator state, and the LLM-plans/seeded-randomizer split (§3.4) yields bit-for-bit reproducible scenes extensible past 10^4 at zero annotation cost.
Execution exposes failures answer-only protocols hide
Diagnostic rather than leaderboard design
Adversarial appearance–physics probes
Weaknesses
No inferential statistics
Unvalidated metric machinery upstream of all results
Baseline and comparison gaps
Results tied to unverifiable proprietary API snapshots
Aggregate execution outcomes unreported
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ReactHuman (arXiv 2609.10895, AAAI-format) is a benchmark paper. Its novelty claim: existing physics evaluations are passive VQA (Physion, CLEVRER, PhysBench) and embodied benchmarks target deliberate long-horizon tasks (Habitat, BEHAVIOR-1K); none measure whether an MLLM converts physical understanding into an immediate, safety-critical action. The benchmark places the evaluated model as the brain of a simulated Unitree G1 humanoid facing sudden household hazards across 17 event families and 1,000+ bit-for-bit reproducible scenes (306-scene evaluated subset), with ground truth (action label, 3D impact point, time-to-floor) read from 240 Hz Genesis rigid-body simulation rather than human annotation. Central claims: (1) models mishandle roughly one hazard in three (mean SAA 54.0%, safety 80.8%, Table 1); (2) safety failures concentrate on Dodge scenes (35.9% violations); (3) action choice reflects fixed per-model dispositions, not the scene (Figure 4); (4) neither accuracy nor safety improves with scale (Gemma-3-27B safest at 88.2%, Claude Opus 4.8 least safe at 64.7%); (5) models treat speed as binary and never revise appearance-based physics judgments (280 adversarial decisions).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the ReactHuman benchmark?
ReactHuman is a benchmark that evaluates whether multimodal LLMs can react to sudden household hazards like a competent human. The model acts as the brain of a simulated Unitree G1 humanoid: a 240 Hz physics simulation runs a sudden event (e.g., a falling object) for about 0.6 seconds, freezes, and the model must output a walking command plus hand trajectory that a controller physically executes. It covers 17 event families and over 1,000 reproducible scenes with exact ground truth taken from simulator state.
How well do current MLLMs perform on ReactHuman?
In the paper's evaluation of seven models on 306 scenes (2,138 decisions), mean action accuracy was 54.0% and mean safety 80.8%. GPT-5.5 scored highest on accuracy (63.7%), the 27B Gemma model was the safest (88.2%), and Claude Opus 4.8 was the least safe (64.7%). No scale-related advantage appeared, and models mishandled roughly one hazard in three.
Where can I download the ReactHuman dataset?
The paper states the benchmark is released on Hugging Face at huggingface.co/datasets/Alan123/reacthuman-benchmark-scaled. Scenes ship with complete specifications so simulations can be re-run bit-for-bit, and the authors state the evaluation code (including the action-classification rule) is released as well.
How is ReactHuman different from physics VQA benchmarks like Physion or PhysBench?
VQA benchmarks ask passive questions about videos; ReactHuman requires an executable action with safety consequences. Every committed plan is carried out by a simulated humanoid, so a model that says catch but reaches half a meter from the impact point fails — a failure mode multiple-choice evaluation cannot detect. Ground truth also comes exactly from simulator state rather than human annotation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure imagery (Figures 1–5) is not present in the extraction; assessment of figures relies on their captions and embedded label text only.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.