HoliBench: A Cross-Platform Benchmarking and Deployment Toolkit for Foundation Models in CPS-IoT Applications
Inesh Chakrabarti, Zejun Xiong, Pragya Sharma, Mani Srivastava
cs.DC · cs.LG · cs.PF
Quick verdict
Unified FM accuracy-latency-energy profiling across 7 platforms; useful honest findings, but PAL validation is indirect and composability rests on one device.
Summary
HoliBench is an open-source toolkit that jointly profiles accuracy, latency, and energy for foundation models (LLMs, VLMs, time-series FMs) across heterogeneous hardware from a Raspberry Pi 5 and Jetson Orin AGX to consumer and datacenter GPUs and Apple Silicon Macs. A platform abstraction layer normalizes power telemetry from PMIC, NVML, jtop, and powermetrics with idle subtraction and temporal alignment; modality-aware drivers decompose inference phases; an ILP-based selector returns configurations meeting joint accuracy, latency, energy, and memory constraints. The authors profile 20 models, 3 quantization levels, and 8 claimed inference backends on 7 devices over a claimed 30+ tasks (Table 2 enumerates 23 dataset-task pairs; 100 samples per configuration). Key findings: bitsandbytes INT4 speeds 4B models 54-60% on an RTX 3070 but penalizes sub-1B models by 43-131% there and slows every tested model on the A5000; net energy scales linearly with latency (R2>0.98 on serving backends, 0.82-0.86 on HuggingFace), implying device-dominated, approximately constant inference power; accuracy beyond roughly 80% costs disproportionately more energy. A CARLA case study shows single-model profiles predict a two-model sequential pipeline's latency and power within 1.2% and 2.5% on one Jetson Orin. Limitations: composability validated on one device, one backend, sequential execution only; LLM-as-judge uncalibrated; MLX claimed as supported in Section 3.3 but admitted missing in Section 8; cross-platform energy comparisons mix GPU-, SoC-, and board-level telemetry scopes.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Standalone single-model profiles compose under sequential co-resident execution, predicting a two-model pipeline's latency and power within 1.2% and 2.5% on a Jetson Orin (Section 7).
- INT4 weight-only quantization latency effects depend on model size, runtime kernel, and GPU microarchitecture, reversing into penalties on most tested configurations (Section 6.3).
- Average inference power is approximately constant per device across models and output lengths, so net energy is predictable from latency alone (Section 6.2, R2>0.98 on serving backends).
- Accuracy gains beyond roughly 80% incur order-of-magnitude higher energy cost per point, with 12-fold energy spread among accuracy-equivalent configurations (Section 6.1).
- HoliBench's PAL yields commensurable cross-platform measurements, enabling constraint-aware configuration selection that accuracy-only evaluation cannot provide (Sections 3.1, 3.4, 6.4).
Strengths
Genuine gap addressed with working, extensible infrastructure
Sections 2-4 identify that capability benchmarks (HELM, lm-eval-harness) and hardware profilers (MELT, MLPerf) are disjoint, and deliver a four-layer stack (PAL, timing model, modality drivers, ILP selector) in ~15k lines of Python with Hydra configs, profiler registration predicates, and a pluggable solve() interface (Section 3.4).
Counterintuitive findings with mechanistic explanations
Section 6.3 shows INT4 (bitsandbytes) giving 54-60% speedups for 4B models on RTX 3070 but 43-131% penalties for sub-1B models, and Qwen3-4B reversing from 54% speedup (3070) to 48% slowdown (A5000), attributed to kernel code-path divergence between consumer and professional Ampere GPUs; Section 6.5 shows vLLM 1.8-16x faster on A5000 but up to 2.8x slower (Llama-3.2-3B) on Jetson under single-query workloads.
Composability result with real runtime validation and a documented failure mode
Methodologically sound energy accounting design
Unusually candid limitations section
Weaknesses
PAL never validated against ground-truth instrumentation
Headline quantization finding is confounded with a single runtime
Statistical reporting too thin for a measurement paper
Design-space claims exceed demonstrated evaluation
Composability scope narrowly validated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
HoliBench (arXiv 2609.12412, UCLA) is a systems/benchmark paper presenting an open-source toolkit that jointly measures accuracy, latency, and energy for three foundation-model modalities (autoregressive LLMs, VLMs, TSFMs) across seven platforms in four tiers (Table 1: Raspberry Pi 5, Jetson Orin AGX 64GB, RTX 3070, RTX 5070, RTX A5000, Mac Mini M2/M4). Its five claimed contributions (Section 1) are: unified measurement via a Platform Abstraction Layer; characterization of 20 models x 3 quantization levels x 8 claimed backends x a claimed 30+ tasks; three deployment-behavior findings (hardware-dependent quantization effects, device-determined inference power, diminishing accuracy returns under energy); profile composability for multi-model pipelines (1.2% latency / 2.5% power error); and constraint-aware configuration selection via an ILP solver. The gap claim — no existing tool spans FM modalities and the device-cloud spectrum with energy — is credible against the related-work survey (Section 2: MELT, Abstreiter et al., MLPerf Inference/Tiny, capability benchmarks).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does quantization always make LLM inference faster on edge devices?
No. According to this paper, bitsandbytes INT4 sped up 4B-parameter models by 54-60% on an RTX 3070 but slowed sub-1B models by 43-131% on that same card, and increased latency for every tested model on the RTX A5000 — including a 48% slowdown for Qwen3-4B and a 478% penalty for Qwen3-0.6B on the Jetson AGX Orin. The effect depends on model size, the quantization runtime's kernels, and the GPU microarchitecture.
What hardware does HoliBench support and measure?
The paper's testbed spans seven platforms in four tiers: Raspberry Pi 5, Jetson Orin AGX 64GB, RTX 3070, RTX 5070, RTX A5000, and Mac Mini M2/M4. Power is profiled through PMIC (RPi), jtop (Jetson), NVML (NVIDIA GPUs), and powermetrics (Apple Silicon), normalized by the toolkit's platform abstraction layer.
Is HoliBench's code available?
Yes. The paper links a public GitHub repository (github.com/beesfleas/HoliBench) containing the toolkit, additional results, demos, and an interactive configuration-selection UI.
Can measurements of single models predict how several models behave together on one device?
For sequential co-resident execution of two models on one Jetson Orin, the paper reports predictions within 1.2% latency and 2.5% power over 10-minute driving-simulator runs. The authors explicitly caution this does not yet cover concurrent execution, dynamic batching, or more than two co-resident models.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-9 are represented only by captions and in-text descriptions; plotted data points beyond values quoted in the prose could not be inspected from the text extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.