GGUF-Metadata Prediction of Single-Sequence llama.cpp Throughput Across Three Systems
Xinyu Qiu, Chuhong Xu, Bo Su, Ziyao Chen, Ruiyang Xu, Shimeng Dai
cs.AI · cs.LG · cs.PF
Quick verdict
Metadata-based predictors cut held-out decode MAPE vs total-parameter baselines on 3 systems, but tiny held-out sets and RTX validity gaps limit claims.
Summary
The paper builds throughput predictors for single-sequence llama.cpp inference from GGUF metadata alone: a roofline-shaped decode model whose per-token byte estimate combines file size scaled by the active-parameter fraction with a per-layer KV-cache term, and a zero-prefix prefill model. Fitting one quantization-specific efficiency scale per host on reference configurations, it evaluates frozen family-based splits over 53 host–file configurations (318 phase–depth rows) on two Apple M4 Max machines and an RTX 5080. Active-parameter accounting plus metadata-aware KV lowers held-out decode MAPE from 49.4/55.3/51.9% (total-parameter B1) to 13.1/14.4/36.1% on MacBook/Studio/RTX, and per-layer KV avoids 10.7–13.7× cache overestimates for Gemma-4. Leave-one-host-out coefficients transferred to 11.6/16.8/36.0% test MAPE. Prefill is reported as a negative result (P2 test MAPE 18.7–108.2%), and low-bit ladder ordering differs across runtime stacks. Main limitations: held-out sets contain only 4, 5, and 2 configurations; RTX residency was not instrumented (the largest working set exceeds the GPU's ~17 GB VRAM) and its runtime version is unverified; error summaries are descriptive with no confidence intervals; depth-tripled rows are correlated.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 0 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A metadata-only decode predictor with active-parameter and per-layer KV accounting cuts held-out MAPE to 13.1/14.4/36.1% on the three hosts versus ~49–55% for total-parameter charging (Table 1).
- Per-layer KV metadata from the GGUF header avoids 10.7–13.7× cache-byte overestimates for sliding-window Gemma-4 architectures (Fig. 2b).
- Leave-one-host-out coefficients, paired with the target's calibrated bandwidth, transfer to 11.6/16.8/36.0% test MAPE — close to target-fitted accuracy but not a simultaneous model-and-host generalization result.
- The zero-prefix prefill model fails to replicate on RTX (108.2% test MAPE) and is explicitly declared unsupported under this protocol.
- Low-bit quantization ladder ordering differs across runtime stacks, so fitted efficiency coefficients are not universal.
Strengths
Clean ablation design isolates each structural correction
B1 vs B2 shares the median-ratio fitting rule, so the held-out drop 49.4→13.1% (MacBook), 55.3→14.4% (Studio), 51.9→36.1% (RTX) isolates active-weight accounting; the separate uniform→per-layer KV refit lowers 21.2→13.1, 21.0→14.4, 37.9→36.1%, confirming each mechanism independently (§4.1)
Unusually honest negative results and transfer limits
§4.3 declares prefill unsupported (RTX P2 108.2%, still 85.5% after median refit), RTX transfer limits are shown rather than hidden, and the MacBook gpt-oss-120B failure is carefully labeled a prompt-batch failure, not OOM (§3, §5)
Exceptional measurement provenance
Metadata-only KV correction is actionable and verifiable
Weaknesses
Statistically fragile headline numbers
RTX cohort has unquantified measurement-validity threats
Low-bit ladder comparison is doubly confounded
Prefill P2 coefficients are weakly identified
No learned-feature baseline bounds achievable accuracy
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a narrow, well-motivated problem: predicting single-sequence decode and prefill throughput of llama.cpp from GGUF header metadata (file size S, tensor shapes, expert routing, per-layer attention geometry) plus a small per-host reference set, without executing the candidate model. Three claims are tested: (1) active-parameter accounting materially improves held-out decode accuracy over total-parameter rooflines; (2) metadata-aware per-layer KV accounting matters for hybrid-attention architectures; (3) coefficients transfer only partially across systems, and prefill under this protocol fails. Each claim maps to a specific table or figure, and — unusually — the paper itself quantifies where each claim breaks (RTX, prefill, low-bit ladders).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 0 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How accurate is throughput prediction from GGUF metadata alone?
For decode on held-out model families the paper reports 13.1% (MacBook M4 Max), 14.4% (Mac Studio M4 Max), and 36.1% (RTX 5080) mean absolute percentage error, versus roughly 49–55% when charging total rather than activated parameters. The RTX figure comes from only two held-out files and is the least reliable.
Does the same predictor transfer across different computers or GPUs?
Partly: coefficients fitted on two machines and combined with the target's measured copy bandwidth give 11.6/16.8/36.0% test error, close to target-fitted accuracy. But the paper stresses that fitted efficiency factors are not universal, and the prefill model fails to replicate on the RTX 5080 (108.2% test error).
Why is prefill prediction reported as a failure?
The simple zero-prefix prefill model reaches 18.7% and 22.2% test MAPE on the two Apple hosts but 108.2% on the RTX 5080, and its per-quantization coefficients are weakly identified (most host-quantization training groups contain one configuration, so the low 4–6% training error is partly resubstitution). The authors explicitly treat prefill as unsupported under this protocol.
What are the biggest caveats of this study?
Tiny held-out sets (4, 5, and 2 configurations) with no confidence intervals and correlated depth rows; no RTX memory-residency telemetry although the largest workload (17.508 GB) exceeds the GPU's ~17.1 GB VRAM; an unverifiable RTX binary version; and a low-bit comparison where the IQ2 file also differs architecturally (64 layers/26.90B vs 65 layers/27.32B parameters).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 were extracted as text; visual patterns (scatter alignment in Fig. 1, bar shapes) could not be inspected directly
- Fig. 4b/appendix CV counts use different denominators by design (8/66 vs 7/57 MacBook prefill rows above 3% CV — all-selected vs scored-only); not an inconsistency, but readers should note it
- External identities (arXiv id, model names like Gemma-4/Qwen3.8) could not be verified against outside sources under the no-network rule
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.