FireWorldBench: Benchmarking Complex Physical World Intelligence through Coupled-Field Fire Dynamics

Summary

FireWorldBench evaluates whether multimodal large language models can reason about complex physical systems, using coupled-field fire dynamics as the testbed. It comprises 520 fire worlds — 494 controlled FDS simulations and 26 real-world-aligned event groups — spanning 47 scene archetypes in 7 environment families, and yields 9,074 text-image interleaved QA pairs in choice and open-ended report formats. Evaluation runs along a physical capability axis (temporal forecasting, field perception, cross-field coupling, causal attribution, counterfactual intervention) and a fire-scenario axis (onset, propagation, critical transition), each under two observation tracks: structured sensor records and rendered physical-field images. Six deterministic metrics, with no LLM judge, score accuracy, macro-F1, calibration, evidence grounding, mechanism recall, and gold-linked support. Across 25 model-track evaluations, the strongest closed model averages only 55.34% accuracy on structured inputs and the best visual-track result is 44.83%; performance degrades from forecasting toward intervention reasoning; structured inputs beat images by roughly 7–10 accuracy points; and parameter scaling shows diminishing returns. A LoRA-tuned FireWorldGPT improves over its base model. A 300-item audit reports 93% clarity and 97% expert-gold agreement but flags 8 answer-leakage and 4 future-leakage items. Key limitations: the real-world gold labels are simulation-completed rather than measured, real-subset samples are small, there is no human baseline, and model rankings lack significance testing.