Broken on Arrival: Silently Defective LLM Artifacts in Public Model Registries and How to Catch Them
cs.SE · cs.LG
Quick verdict
First functional census of 327 GGUF artifacts finds 5 defective official conversions; rigorous adjudication, honest stats, but thin community sample.
Summary
The paper reports the first ecosystem-scale functional census of distributed quantized LLM artifacts. The author executed 327 GGUF conversions — 305 official Ollama library artifacts across 15 model lines at every quantization level at or under 8 GB, plus 22 popular community HuggingFace conversions — using a 15-task code-generation smoke suite calibrated on Qwen2.5-Coder so that healthy configurations pass and a known-broken one fails. Suspects underwent full 164-task HumanEval+ evaluation, replication on a second backend (CUDA and Metal), and comparison against an independent distributor's conversion of the same model and quantization. Five official artifacts (1.6%) are confirmed defective — four Qwen2.5-Coder-3B low-bit builds and one phi3.5-mini q2_K — all scoring 0/164 on both backends while referee conversions score 6–14/15; 2 of 29 model-and-size conversion groups are affected. The adjudication chain reclassifies llama3.2-1B low-bit failures as capability collapse and exposes two TheBloke deepseek-coder conversions that fail on CUDA yet pass on Metal (backend-dependent failure). Surface output statistics catch only 3 of 5 defects at a 2.3% false-positive rate, so execution remains necessary. Main limitations: single-model calibration, a loose 0–27.8% bound on screen misses derived from ten sampled artifacts, a popularity-weighted community sample of 22, and one pinned runtime version.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Five official Ollama artifacts (1.6% of 305; 2 of 29 model-and-size conversion groups) are silently defective — four Qwen2.5-Coder-3B low-bit builds and one phi3.5-mini q2_K — scoring 0/164 on both CUDA and Metal while independent conversions of the same models score 6–14/15.
- The adjudication chain (family baseline, second backend, cross-distributor referee) separates genuine distribution defects from capability collapse (llama3.2-1B at 2–3 bit) and from backend-dependent failures (two TheBloke deepseek conversions failing on CUDA, passing on Metal).
- Surface output statistics cannot substitute for execution: the best rule catches 3 of 5 defects at a 2.3% false-positive rate, and two defects are statistically indistinguishable from healthy output.
- A 15-task calibrated smoke suite followed by baseline-relative and referee adjudication flags every confirmed defect at seconds-to-minutes per artifact, making registry-side acceptance testing practical.
Strengths
Novel measurement layer: execution, not metadata
First study to actually run distributed third-party conversions at ecosystem scale (305 official Ollama + 22 community artifacts), unlike the metadata-only mining studies it cites [4][15][16] and the single-model in-house evaluations [27][28]; the paper states no prior work executes distributed quantized artifacts at ecosystem scale.
Adjudication chain earns its keep in both directions
The family-baseline + dual-backend + cross-distributor-referee chain clears llama3.2-1B at 2/3-bit as capability collapse (same-level independent conversions fail equally: 0 and 3/15) and reclassifies the TheBloke deepseek-coder-6.7B pair as backend-dependent (CUDA 5–6/15 vs Metal 13–15/15), avoiding two wrong conclusions a single-stack threshold would have produced (Section IV-A, IV-C).
Honest statistical discipline
Prespecified controls against harness objections
Weaknesses
Single-family calibration with arbitrary task selection
RQ4 prospective-validity claim outruns its evidence
Backend-dependent phenomenon rests on n=2 from one repository
Community-channel comparison underpowered
Referee approximations and scope limits
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper conducts what it argues is the first ecosystem-scale *functional* census of distributed quantized LLM artifacts. The frame: all 305 eligible official Ollama library artifacts (instruct, code-capable, explicit quantization suffix, ≤8 GB, 15 registry model lines) — a census, not a sample — plus 22 files from the most-downloaded community GGUF repositories on HuggingFace, with the inventory frozen before testing. Four research questions structure the study: prevalence (RQ1), detectability without execution (RQ2), channel comparison and mechanisms (RQ3), and prospective validity of the screen (RQ4). Central claims: five confirmed silent defects in the official library (1.6% of artifacts; 2 of 29 model-and-size groups), a three-way adjudication separating defect / capability collapse / backend-dependent failure, proof by counterexample that surface statistics miss one defect class, and a released acceptance-testing tool (quantcheck). One caveat the paper itself discloses: the Qwen2.5-Coder-3B q3_K_M defect was known beforehand (it calibrated the suite), so only four official defects are new discoveries; the calibration-excluded rate is 4/281 (1.4%).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What did the study find wrong with Ollama models?
Five of 305 official library artifacts (1.6%) were confirmed silently defective: four Qwen2.5-Coder-3B low-bit quantizations (q2_K through q3_K_L) and one phi3.5-mini q2_K. All five score 0 of 164 HumanEval+ tasks on both CUDA and Metal backends, while independent conversions of the same models and quantizations solve 6–14 of 15 smoke tasks. Two of 29 model-and-size conversion groups are affected.
What is a silently defective model artifact?
A downloadable model file that runs normally and streams fluent output at normal speed but has lost essentially all task capability. It looks fine to a user and even to output-statistics heuristics — the paper's most deceptive defect class produces statistically normal output — and only actually executing tests reveals the damage.
Are HuggingFace community GGUF conversions broken?
In this census, 20 of 22 popular community conversions were healthy. The two exceptions were not defective files but backend-dependent failures: TheBloke deepseek-coder-6.7B conversions that lose most capability on CUDA while working fine on Apple Metal under the same runtime version. The sample is small (22) and popularity-weighted, so this is not a general verdict on the community channel.
What is quantcheck?
The acceptance-testing tool released with the paper (github.com/aditi-p31/quantcheck). It pulls a GGUF artifact, runs a 15-task code-generation smoke suite in seconds to minutes on a commodity GPU, and applies family-relative baselines and cross-distributor comparisons to flag likely-defective conversions before you rely on them.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's heatmap data extracts as an unformatted stream of numbers in the provided text; its interpretation here relies on the figure caption and prose (both intact), not the grid itself.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.