Is INT8 Portable? A Cross-Platform Measurement Study of Quantized Inference on Embedded and Automotive Accelerators
cs.AR · cs.LG · cs.PF
Quick verdict
Controlled 7-hardware-class study shows INT8 speedup, numerics, and deployability are not portable; convincing, though n=1 per class and one headline model.
Summary
This measurement study tests the 'quantize once, deploy anywhere' folklore for INT8 post-training quantization across seven hardware classes: ARM and x86 CPUs, a discrete GPU, a Jetson AGX Orin iGPU with NVDLA cores, and Qualcomm Hexagon HTP and DEEPX DX-M1 NPUs. Holding the ONNX artifact and quantization scales fixed, the authors report three failures. First, the INT8 speedup's sign is set by the CPU's dot-product ISA: ResNet-50 on the same ONNX Runtime MLAS path speeds up 1.83–2.11× on dot-product cores (Pi 5 A76, Jetson A78AE) but slows 1.65–1.76× on cores without them (Core i9 without VNNI, Cortex-A53). Second, INT8 outputs are not portable: FP32 predictions are bit-identical across all measured pairs, while INT8 agreement is bimodal — 1000/1000 when targets share an integer kernel, 958–965/1000 otherwise, regardless of boundary type, and invisible to top-1 accuracy. A power-of-two-scales mitigation transfers poorly, worsening agreement to 869–919/1000. Third, vendor NPUs reject bring-your-own QDQ graphs — Qualcomm silently (accuracy 0.75→0.005) and DEEPX loudly. A further study shows edge-NPU latency regimes are set by output/D2H transfer size, not compute, with a transition band of width equal to core count. Limitations: n=1 device per class, one headline model, differing runtime versions, single-run latencies without confidence intervals.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The sign of the INT8 speedup is determined by the CPU's dot-product ISA: identical ResNet-50 ONNX + ONNX Runtime MLAS gives 1.83–2.11× speedups on ARM dotprod cores and 1.65–1.76× slowdowns on cores without dot-product/VNNI (§4, Table 2).
- INT8 prediction agreement is bimodal under fixed scales — 1000/1000 when two targets share an integer kernel, 958–965/1000 otherwise — independent of whether the boundary is CPU↔CPU or CPU↔accelerator, while FP32 is bit-identical and top-1 accuracy is preserved (§5, Table 3).
- Vendor NPUs reject bring-your-own QDQ graphs in two opposite modes: Qualcomm HTP silently ignores external scales (top-1 0.75→0.005 while running without error) and DEEPX DX-M1 refuses compilation; only native quantization paths yield correct engines (§6).
- Edge-NPU latency regimes on a PCIe-attached NPU are set by output/D2H transfer size, not compute, with a compute-to-D2H transition band whose width equals the core count (§7, Tables 4–5).
- The power-of-two-scales determinism mitigation shown on a single GPU does not transfer across the x86↔A76 device boundary, worsening agreement from 958 to 869–919 per 1000 (§5, §11).
Strengths
Genuinely controlled cross-platform design
The same resnet50_int8_qdq.onnx with embedded QDQ scales runs on every target (§3.3), and the FP32 bit-identical control (1000/1000 on all measured pairs, §5) cleanly rules out harness or floating-point artifacts as the INT8 divergence source.
Honest quarantining of uncontrolled comparisons
The DEEPX 939/1000 pair is excluded from the invariance claim because its scales cannot be held fixed (§5, §6) — a distinction most papers would blur; §10 likewise pre-declares n=1, version, and subset limitations.
C4 is a real causal experiment, not correlation
Mechanism probing of a negative result
Artifact transparency
Weaknesses
Headline rests on a single model
Runtime-version confound is bounded, not removed
n=1 per hardware class
No uncertainty on smaller quantities
Safety framing outruns the evidence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper tests the "quantize once, deploy anywhere" folklore for INT8 post-training quantization with a controlled measurement campaign over seven hardware classes and eight targets (Table 1): Raspberry Pi 5 (Cortex-A76) and Jetson AGX Orin (A78AE) with ARM dotprod; i.MX8M-Nano (A53) and Core i9-10900K lacking dot-product/VNNI; an RTX 3080; the Orin iGPU plus 2× NVDLA v2; and two vendor NPUs (Qualcomm Hexagon HTP, DEEPX DX-M1). Five numbered claims: C1 — the INT8 speedup's sign is an ISA property; C2 (headline) — INT8 numerics are not portable across integer kernels even under fixed scales; C3 — vendor NPUs own quantization, with silent vs. loud BYO-QDQ failure modes; C4 — edge-NPU latency regimes are set by output/D2H size, not compute, with a transition band of width equal to core count; plus a reproducibility artifact of 32 reports with a claim-to-artifact map (Appendix A). Claims are consistently framed as relative, within-comparison deltas, and §2 positions the work honestly against concurrent kernel-swapping work (Chen) it does not claim to supersede mechanistically.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Is INT8 quantization always faster than FP32 on edge devices?
No. Running the identical ResNet-50 INT8 model on the same ONNX Runtime path, the paper measured 1.83–2.11× speedups only on CPU cores with dot-product instructions (ARM SDOT, x86 VNNI) and 1.65–1.76× slowdowns on cores without them — same model, same runtime, opposite outcomes.
Do two different devices running the same quantized model give the same predictions?
Not reliably. FP32 predictions matched on 1000/1000 inputs across devices, but INT8 predictions disagreed on roughly 3.5–4% of 1000 inputs whenever the integer kernels differed — even between two CPUs — while top-1 accuracy stayed unchanged, so standard accuracy reports hide the difference.
Can I deploy my own quantized QDQ ONNX model to Qualcomm or DEEPX NPUs?
The paper found Qualcomm's HTP silently ignored external QDQ scales — accuracy collapsed from 0.75 to 0.005 while compiling and running without error — and DEEPX's compiler rejected the graph outright. Only each vendor's native quantization path produced a correct engine (Qualcomm native: 0.735; DEEPX native: 0.7660).
What are the main limitations of this study?
One device per hardware class, headline agreement and speedup measured on a single model (ResNet-50), different ONNX Runtime versions across targets, and single-run latency measurements without confidence intervals — all acknowledged by the authors in their threats-to-validity section.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as extracted text; plots (Fig 1–3) were assessed from their tabulated numbers, which appear complete.
- The abstract is truncated mid-sentence in the source text, but the full body contains all key numbers and the missing abstract tail is recoverable from §1.
- arXiv v1 preprint, not yet peer-reviewed; concurrent-work citations (Chen arXiv:2608.13756, 2609.00363) and the released artifact repository were not independently verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.