A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware
Maysam Khatib, Moysis Symeonides, Demetris Trihinas, George Pallis, Marios D. Dikaiakos
cs.DC · cs.AI · cs.LG
Quick verdict
A careful, honestly-hedged measurement study of LLM serving on edge hardware; solid methodology but thin statistics and non-comparable energy boundaries.
Summary
This paper presents a controlled measurement study of self-hosted LLM inference across the edge continuum. The authors benchmark open-weight models — Llama 3.2 (1B/3B, F16/Q5_K_M/Q4_K_M), Mistral 7B, TinyLlama 1.1B, and Phi-2 (Q5/Q4 GGUF variants) — on a Jetson AGX Orin and a near-edge server operated in CPU-only and GPU modes, using Ollama as the serving stack and GPT-4o as a cloud reference. The workload is the MMLU validation split (1,531 zero-shot multiple-choice prompts, temperature 0, five repetitions, ten warm-up prompts), measuring accuracy, model footprint, prefill and per-token decode latency, and energy (mean power × runtime). Results: server-GPU execution is fastest; CPU-only serving is dominated in accuracy-latency space; parameter count and weight-file size do not reliably predict accuracy or latency; and Jetson Orin shows the lowest measured energy, though the authors caution that its wall-socket full-system measurement is not directly comparable to the server's partial GPU+CPU telemetry. A Pareto-frontier sensitivity analysis adding modeled 30/60 ms per-token streaming overheads shows on-device inference becoming competitive for latency-sensitive settings while server GPU remains efficient for high accuracy. Main limitations: sequential single-request workload only, a single serving stack, heterogeneous energy measurement boundaries, and synthetic rather than measured network delays.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Execution platform dominates latency: server-GPU execution yields the lowest prefill and per-token decode latency, and all CPU-only configurations are dominated in the accuracy-latency space for this workload (RQ2, RQ4).
- Parameter count and downloaded weight-file size do not reliably predict accuracy or latency; e.g., Llama 3.2 F16 shows slightly lower accuracy than smaller quantized Mistral Q5_K_M while outperforming 7B quantized models on Orin decode latency (RQ1).
- Jetson Orin exhibits the lowest measured energy per trial, attributed to lower platform power — explicitly framed as indicative because the server figure (nvidia-smi + RAPL only) is a partial lower-bound estimate (RQ3, footnote 3).
- Adding modeled 30/60 ms per-token delivery overheads shifts the accuracy-latency Pareto frontier so on-device Orin inference becomes competitive for delay-sensitive settings, while server GPU remains efficient for high accuracy (RQ5).
Strengths
Honest claim scoping and hedging
The explicit compute-side vs client-observed streamed-token latency distinction (Sec. 2), footnote 3 conceding the server energy figure is a partial lower bound, and the labeling of 30/60 ms as 'controlled what-if parameters' show unusually disciplined overclaiming avoidance; the Limitations section is candid.
Controlled isolation design
Same serving stack across devices, temperature 0, fixed zero-shot prompt with first-valid-label parsing and malformed-counts-as-incorrect, 10 warm-up prompts, five repetitions, sequential single-request workload, and loopback-only self-hosted latency — variables are consciously isolated rather than bundled (Sec. 2, 2.1).
Decision-relevant framing
Reproducibility support
Concrete anti-heuristic evidence
Weaknesses
No dispersion or inferential statistics
Cross-platform energy comparison crosses incomparable measurement boundaries
Single serving-stack confound
Cloud baseline is not decision-complete
Narrow, aging model coverage
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a controlled measurement study — not a new system — of self-hosted LLM serving across three execution modes: on-device (Jetson AGX Orin), near-edge server GPU (NVIDIA T4), and near-edge server CPU-only, plus GPT-4o as a cloud accuracy/latency reference. The declared contributions (Sec. 1) are a containerized benchmarking pipeline, a cross-platform comparison under a fixed MMLU QA workload, evidence that model size is an unreliable selection heuristic, and a Pareto-frontier sensitivity analysis of streamed-token delivery overheads. The claims are deliberately narrow and mostly supported at the trend level: platform dominates latency; CPU-only is dominated; size metrics do not predict outcomes; Orin shows the lowest *measured* energy; and modeled 30/60 ms delivery penalties erode the server-GPU latency advantage for delay-sensitive use.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which hardware and models were tested in this edge LLM benchmark study?
An NVIDIA Jetson AGX Orin (64GB) and a near-edge server with an Intel Xeon Gold 6230 CPU and NVIDIA T4 GPU, run in both CPU-only and GPU modes, with GPT-4o as a cloud reference. Models were Llama 3.2 1B/3B (F16, Q5_K_M, Q4_K_M), Mistral 7B, TinyLlama 1.1B Chat, and Phi-2 in Q5/Q4 GGUF quantizations, all served via Ollama.
What are the paper's main findings?
Server-GPU execution is the fastest; CPU-only serving is dominated in the accuracy-latency trade-off; parameter count and weight-file size alone do not reliably predict accuracy or latency; the Jetson Orin shows the lowest measured (wall-socket) energy; and once modeled 30/60 ms per-token streaming delivery overhead is added, on-device inference becomes competitive for latency-sensitive interactive use while server GPU stays best for high accuracy.
Is the energy comparison between the Jetson and the server fair?
Only partially. The Jetson Orin was metered at the power socket (full system draw), while the server used GPU (nvidia-smi) plus CPU-package (RAPL) telemetry only, excluding memory, storage, fans, and PSU losses. The authors state the server's reported energy is a lower-bound estimate and that absolute cross-platform values should be treated as indicative, not equivalent.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2–4 are plotted images; their underlying values could only be assessed from the prose descriptions (e.g., 17–22 ms/token CPU prefill, ~80% GPT-4o accuracy), not from the plots themselves.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.