SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMs
Pengfei Li, Naufal Suryanto, Sicheng Zhang, Mohammad Alsharid, Muzammal Naseer
cs.CV · cs.AI · cs.CL
Quick verdict
Large fire-smoke MLLM benchmark (83K imgs, 193K VQA): ten models avg 61.9% vs human 82.5%, though ~13% label noise and weakest-baseline framing temper claims.
Summary
SAFIRE is a large-scale benchmark for evaluating multimodal LLMs on fire and smoke understanding, motivated by the fact that visually similar flames carry context-dependent risk (fireworks vs. residential fire). It aggregates 83K captioned images across 20 scenarios in five groups (natural, industrial, accidental, recreational, civil-controlled) and derives 193K multiple-choice VQA pairs from a 9.7K-image subset, spanning ten dimensions from target counting to polysemy and danger reasoning. Annotations are generated by Qwen2.5-VL-72B, verified by GPT-5.4, and checked by majority voting among three non-Qwen MLLMs plus manual review; a 2,800-sample human audit reports 87.1% label accuracy. Evaluating ten open-source MLLMs (8B-38B) yields 61.9% average accuracy versus 82.5% for six human volunteers on a 700-question subset, with weaknesses concentrated in spatial correlation (42.9%) and general reasoning (53.3%); thinking-enabled variants shift accuracy by only -2.7 to +0.3 points. Zero-shot CLIP-style encoders reach up to 72.2% on 20-way scene classification, linear-probe adaptation with 7% of labels lifts the weakest baseline from 20.1% to 64.5%, and SFT on SAFIRE adds +7.81 points for Qwen3-VL-8B. Main limitations: a 12.9% residual label-error rate, single-run evaluations without confidence intervals, Qwen-family annotation bias (Qwen models average 64.6% versus 60.6% for others), and a headline encoder claim benchmarked against the weakest zero-shot model.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SAFIRE is the largest and most diverse fire-smoke resource to date: 83K captioned images across 20 scenarios and 193K multiple-choice VQA pairs over 10 dimensions, where none of 23 prior datasets (Table 1) include VQA.
- Ten open-source MLLMs (8B-38B) average 61.9% MCVQA accuracy versus an 82.5% human reference, exposing a large gap in safety-critical fire-smoke reasoning.
- Reasoning-enhanced (thinking) variants yield only marginal changes: -2.7 pp at 9B and +0.3 pp at 32B (Section 4.2).
- Lightweight adaptation is highly effective: 7% supervision lifts fire-scene classification from 20.1% to 64.5%, and SFT improves Qwen3-VL-8B by +7.81 pp on a held-out image-level split.
- The GPT-5.4-plus-majority-voting pipeline yields 87.1% annotation accuracy, and residual label noise does not overturn relative model rankings (Table 9).
Strengths
Unmatched task coverage for the fire-smoke domain
Table 1 shows 0 of 23 prior datasets include VQA pairs and only 2/23 include captions; SAFIRE contributes 193K MCVQA pairs over 10 reasoning dimensions plus 83K captions across 20 scenarios (Table 2), a qualitative leap over BowFire/DFAN/DetectiumFire-style classification or detection resources.
Bias and shortcut probing is unusually thorough for a model-annotated benchmark
Appendix F ablates scenario priors (image adds +34.87 pp vs. +9.91 pp for scenario name alone); Appendix I quantifies generator-family favoritism (Qwen avg 64.6% vs. non-Qwen 60.6%, human 82.5%); Appendix D/Table 9 shows model rankings survive label correction on the audited 2,800-sample subset.
Transparent residual-error accounting
Actionable downstream evidence, leakage-aware
Weaknesses
Ground truth carries a ~13% error rate
Headline encoder gain is framed against the weakest baseline
No uncertainty quantification anywhere
Qwen-centric annotation loop
Provenance/licensing risk for released images
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SAFIRE (Khalifa University; arXiv 2609.07823) is a benchmark paper targeting fire-smoke understanding in multimodal LLMs. The authors claim three things: (1) the largest and most diverse fire-smoke resource to date - 83K captioned images over 20 scenarios in five groups, plus 193K multiple-choice VQA pairs derived from a 9.7K-image subset across 10 evaluation dimensions; (2) that current open MLLMs are far from reliable in this safety-critical domain, averaging 61.9% MCVQA accuracy (Table 3) versus an 82.5% human reference (Appendix I); (3) that modest supervision goes a long way - 7% labels lift fire-scene classification from 20.1% to 64.5% (abstract) and SFT adds +7.81 pp to Qwen3-VL-8B (Table 13). The evidence mostly supports these claims, but several framings deserve softening.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SAFIRE benchmark?
SAFIRE is a public benchmark for testing how well multimodal AI models understand fire and smoke scenes. It contains about 83,000 captioned images across 20 scenarios (volcano, fireworks, gas stove, vehicle fire, etc.) and roughly 193,000 multiple-choice questions in ten skill dimensions, from counting fire sources to interpreting figurative language like 'on fire'. It is released for non-commercial research use.
How good are current multimodal LLMs at fire and smoke understanding?
On SAFIRE, ten open-source models (8B-38B parameters) average 61.9% accuracy on multiple-choice questions, well below the 82.5% achieved by six human volunteers on a 700-question sample. The weakest skills are spatial correlation (42.9%) and general reasoning (53.3%), while simple human-presence detection reaches 90.3%.
How reliable are SAFIRE's ground-truth labels?
Labels were generated by Qwen2.5-VL-72B, checked by GPT-5.4 and a three-model majority vote, with flagged samples human-reviewed. A 2,800-sample manual audit found 87.1% label accuracy. Re-evaluating models on corrected labels raised scores by 1.3-5.4 percentage points but preserved the relative model ranking.
Where can I download the SAFIRE dataset and code?
The paper points to the project page (risys-lab.github.io/SAFIRE) for datasets, models, and code, released for non-commercial research use. Note that parts of the image collection were scraped from commercial stock and social-video platforms, so check licensing terms before redistributing any imagery.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Encoder evaluation values (Figures 6-7) appear in text only as prose quotes (e.g., MetaCLIP2 72.2%, EvaCLIP F1=0.70); per-scenario encoder numbers exist only inside figures and are not extractable.
- The 20.1% zero-shot baseline behind the abstract's encoder claim appears only in the abstract and conclusion; it is not attributed to a named model anywhere in the extracted body text, so its exact configuration cannot be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.