Distilling Vision-Language Models for On-Device Fire Understanding
Mohammad Kazzazi, Zixuan Liu, Siavash Khajavi
cs.AI
Quick verdict
Claims distilled compact VLMs keep teachers' fire understanding on-device; convincing deployment study, but no direct fine-tuning baseline for the students.
Summary
The paper presents a teacher-student knowledge distillation framework for compressing fire-fine-tuned vision-language models (VLMs) onto resource-constrained embedded hardware. Teachers (Qwen2.5-7B, Qwen3-8B, Llama-3.1-8B, Gemma-2-9B) are adaptor-initialized on LLaVA pretraining data and LoRA-fine-tuned on the DetectiumFire dataset; students from 0.5B to 4B parameters are distilled with a KL-divergence objective plus a cross-entropy term. On the 1,452-image DetectiumFire evaluation set, compact students retain most teacher performance — Qwen2.5-0.5B keeps 99.96% of its teacher's fire/non-fire accuracy and 95.2-99.3% on reasoning tasks — while larger students add little. An ablation shows KL-only training collapses specificity to 3.82-67.62% (near-constant 'fire' predictions) without visibly hurting positive-class metrics, motivating the CE term. Models are exported to Q8_0 GGUF via llama.cpp and benchmarked on a Raspberry Pi 5-based commercial sensor (7 GiB cap): Qwen2.5-0.5B achieves 4,034 ms end-to-end latency and 1,880 MB RSS with the best post-export accuracy. Quantization shifts CLIP-based students toward false alarms (more so for larger models), while MobileCLIP-S2 configurations, run unoptimized in PyTorch, miss fires. Main limitations: no directly fine-tuned (non-distilled) student baseline to attribute gains to distillation; evaluation restricted to the authors' own dataset and sensor; the encoder comparison conflates runtime and architecture; and no statistical significance testing or code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Compact students distilled from fine-tuned 7-9B teachers retain most fire-understanding capability — Qwen2.5-0.5B retains 99.96% of its teacher's fire/non-fire detection accuracy and 95.2-99.3% on reasoning tasks (Table 1).
- Increasing student size yields little accuracy gain; Qwen2.5 fire/non-fire accuracy even decreases monotonically from 99.75% (0.5B) to 98.51% (3B).
- KL-only distillation catastrophically inflates false alarms (specificity 3.82-67.62%) while leaving positive-class reasoning metrics apparently intact, so domain-specific distillation must be validated on negative examples.
- Deployment-time quantization shifts CLIP-based students toward false alarms (larger students shift more: +8.8 avg vs +2.7 avg extra false alarms), while MobileCLIP-S2 configurations err toward missed fires.
- Qwen2.5-0.5B provides the strongest overall on-device trade-off: 4,034 ms E2E latency, 1,880 MB RSS, and the best post-export accuracy among students.
Strengths
Negative-example ablation uncovers a hidden catastrophic failure mode
KL-only distillation collapses fire/non-fire specificity to 3.82-67.62% (Qwen2.5-0.5B: 671.3 false alarms, 100% recall) while positive-class reasoning stays flat or improves (Table 8), and teachers provably lack the bias (≤2 false alarms, Table 9). A genuinely transferable lesson for domain KD evaluation.
Deployment measured on real target hardware, not a proxy
Raspberry Pi 5-based commercial sensor, 7 GiB cap, 4 CPU threads, no GPU; TTFT/E2E/RSS jointly reported (Table 4) with accuracy re-measured post-quantization (Table 3) and error-direction decomposition (Table 11). Two non-deployable configs (8,778 MB and 9,850 MB) are honestly flagged as reference-only.
Systematic coverage across families and scales with seed variance
Unusually complete implementation disclosure
Honest scoping of confounds
Weaknesses
Missing the decisive control: direct fine-tuning without a teacher
Closed evaluation loop on the authors' own assets
Encoder conclusion rides on a confounded comparison
Statistical reporting is descriptive only
Latency framing compares unlike quantities
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper studies whether domain-specialized VLMs — fine-tuned to understand fire scenes — can be compressed via teacher-student distillation into models small enough for fully on-device inference on a commercial fire sensor, without sacrificing safety-relevant behavior (low false alarms, low missed fires). Three claims are advanced: (1) compact students (0.5B–4B) retain most of their 7–9B teachers' fire-understanding capability (Qwen2.5-0.5B retains 99.96% of teacher fire/non-fire accuracy and 95.2–99.3% across reasoning tasks, Table 1); (2) increasing student size yields little gain — Qwen2.5 fire/non-fire accuracy falls monotonically from 99.75% (0.5B) to 98.51% (3B); (3) the distillation objective and deployment pipeline shift error *types*, not just error rates, with Qwen2.5-0.5B the best overall trade-off. Scope is explicit: four tasks (detection, burning object, environment, severity) on the 1,452-image DetectiumFire eval split (754 fire / 698 non-fire), plus on-device latency and memory. Privacy and latency motivations for on-device inference (Sec. 1) are well argued.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the best model size for on-device fire detection with a vision-language model?
In this paper, Qwen2.5-0.5B distilled from a fine-tuned Qwen2.5-7B teacher gave the best trade-off on a Raspberry Pi 5-based commercial sensor: 99.52% fire/non-fire accuracy after export (~99.75% before), 4,034 ms end-to-end latency, and 1,880 MB memory. Larger students (1.5B-4B) added little accuracy while costing more memory, more latency, and more post-quantization false alarms; two configurations exceeded the sensor's 7 GiB memory cap entirely.
Why can knowledge distillation produce models that raise false alarms on everything?
The paper shows that students trained only with the KL (teacher-distribution-matching) objective learn to predict 'fire' for nearly every image: fire/non-fire specificity fell to 3.82-67.62% across seven students (e.g., 671 false alarms out of 698 non-fire images for Qwen2.5-0.5B), even though the teachers show no such bias. Because the other evaluation tasks use only fire images, this collapse is invisible on them. Adding a cross-entropy term on ground-truth labels restored specificity to 97.5-99.9%.
Does quantization hurt a VLM's fire detection accuracy?
Modestly in aggregate, but it shifts error type. Converting student models to Q8_0 GGUF via llama.cpp kept fire/non-fire accuracy within 0.89 percentage points, yet systematically increased false alarms for CLIP-based students — about +2.7 on average for models ≤1.5B and +8.8 for models ≥2B — while fine-grained burning-object identification lost up to about 4 points. MobileCLIP-S2-based students, which ran unquantized in PyTorch, instead missed fires (39.3 and 14.7 misses), a more dangerous failure mode.
What dataset was used to train and evaluate these fire VLMs?
DetectiumFire, a multimodal fire dataset with VQA annotations for burning object, environment, and fire severity, created by the same research group (published at NeurIPS 2026 Datasets & Benchmarks). Evaluation uses 1,452 images: 754 fire and 698 non-fire. Teachers were additionally fine-tuned with LLaVA general-instruction data to preserve broad vision-language capability.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extracted text contains no figure content or figure references (tables only); any graphical material in the PDF, if present, could not be reviewed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.