Global Ranks Survive, Selected Heads Shift: BOS-Sink Topology under 4-bit Weight-Only Quantization
Kuanlin Chen, Chen-Wei Kuo, Cheng-En Ou
cs.LG · cs.CL
Quick verdict
Shows 4-bit NF4 preserves global BOS-sink head ranks (ρs≥0.980) but not top-k sets (0.619–0.793) across 3 small LLMs; solid but narrow empirical study.
Summary
This empirical study asks whether head-level first-token (BOS) attention-sink maps calibrated in bf16 can be safely reused after 4-bit NF4 weight-only post-training quantization — a common shortcut in edge deployment. The authors introduce a Sink Topology Consistency (STC) metric family separating global rank preservation, top-k set overlap, and layerwise sink-mass shift, each at per-input and calibration levels, with a closed-form null baseline. Across Qwen2.5-0.5B, Qwen2.5-1.5B, and Llama-3.2-1B on 500 C4 texts (512–4,096 tokens), global Spearman rank survives at ≥0.980 while top-k Jaccard overlap drops to 0.619–0.793 (76.5–88.5% head retention; null ≈ 0.053). Global metrics mask local failures: Qwen terminal layers shift 6.2–7.9× the model mean with per-layer rank collapsing to 0.033, whereas Llama-3.2-1B drifts low and uniformly. A C4→LongBench probe shows corpus mismatch degrading overlap more than the precision change for both Qwen models but not for Llama. Full 4-bit recalibration reaches 90% of a split-half stability plateau at n=8 (Qwen) and n=32 (Llama); layer-selective updates never reach the criterion. Jetson Orin NX measurements show seconds-scale map recovery for two models, while Qwen2.5-1.5B hooks return NaN on the tested arm64 stack. Limitations: three models from two families, an fp16-vs-bf16 compute confound, no downstream task-accuracy evidence, a single domain pair, and no stated code release.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Across three small LLMs, bf16→4-bit NF4 global sink-head ranking correlates at ρs ≥ 0.980 (4,096 tokens) while top-k set Jaccard drops to 0.619–0.793 (76.5–88.5% retention; null ≈ 0.053).
- Global metrics hide layer-local failures: Qwen terminal layers shift 6.2–7.9× the model-mean sink mass with per-layer rank collapsing to 0.033 (Qwen2.5-0.5B layer 22), whereas Llama-3.2-1B drifts low and uniformly.
- Under a C4→LongBench shift, corpus mismatch reduces Cal-Set overlap more than the bf16→4-bit precision change for both Qwen models, but not for Llama-3.2-1B.
- Full 4-bit map recalibration reaches 90% of split-half stability at the smallest tested n=8 (Qwen) and n=32 (Llama); layer-selective updates never reach the criterion for the Qwen models.
- On Jetson Orin NX, a 16-sample sink-map recovery completes in 2.9–6.8 s for the two models with valid hooks; Qwen2.5-1.5B returns NaN on the tested arm64 stack.
Strengths
Metric decomposition with a proper null
STC separates global rank, top-k Jaccard, and layerwise JS mass shift at per-input and calibration levels (Sec. III-C), and benchmarks Set@k against a closed-form null (k/(2LHq−k)=0.053) plus 1,000 Monte Carlo draws — the 0.619–0.793 overlaps are unambiguously above chance, so the Rank–Set dissociation is a real effect, not an artifact.
Measurement hygiene and internal consistency
Paired passes fix checkpoint, tokens, teacher-forced path and eager backend (III-D); hook accumulation is validated against outputs.attentions (deviation <10⁻⁶ vs a 10⁻³ threshold); the k=34/34/52 arithmetic and the Jaccard→76.5–88.5% retention conversion are exactly reproducible from stated formulas; the layer-selective plateau 0.619 for Qwen2.5-0.5B exactly matches its Table II Cal-Set, as it must if updates never change top-k membership.
Candor about failures and non-results
Deployment-grounded evidence
Weaknesses
Compute confound in the central comparison
No task-level consequence
Narrow and confounded model evidence
Statistical thinness at decision points
Mechanism left open
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a narrow but well-posed deployment question: if a pipeline calibrates a head-level first-token (BOS) attention-sink map in bf16, which parts of that map remain valid after 4-bit NF4 weight-only PTQ? The central claims are (1) global rank preservation is high (Cal-STC-Rank ≥ 0.980 at 4,096 tokens, Table II) while top-k set overlap is materially lower (Cal-Set@k 0.619–0.793, i.e., 76.5–88.5% membership retention against a 0.053 null); (2) global statistics mask layer-local failures — Qwen terminal layers shift 6.2–7.9× the model-mean LayerShift, with per-layer rank collapsing to 0.033 (Qwen2.5-0.5B, layer 22), while Llama-3.2-1B drifts low and uniformly; (3) under a C4→LongBench shift, corpus mismatch costs more overlap than the precision change for both Qwen models but not Llama (Table V); and (4) full 4-bit recalibration reaches 90% of a split-half stability plateau at n=8 (Qwen) / n=32 (Llama), while layer-selective updates never reach the criterion. Claims are explicitly scoped to calibration validity, not task accuracy.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a BOS-sink head and why does 4-bit quantization threaten it?
Language models dump surplus attention onto the first token; heads specializing in this are called sink heads. Some edge pipelines precompute which heads these are and reuse the list after compressing the model. This paper measures, for three small models (Qwen2.5-0.5B/1.5B, Llama-3.2-1B), how much of that head list survives 4-bit NF4 weight-only quantization: the global head ranking stays almost identical (Spearman ≥ 0.980 at 4,096 tokens), but only 76.5–88.5% of the selected top-10% heads remain in the set.
Can I calibrate an attention-sink map once on a GPU server and reuse it on a Jetson device?
The paper's guidance: continuous global rankings mostly transfer, but discrete head sets, per-layer policies, and new domains should be re-validated in the deployed precision. On a Jetson Orin NX 8GB the 16-sample re-validation took 2.9–6.8 s for two of the three models; for Qwen2.5-1.5B the tested arm64 software stack returned NaN attention weights, so on-device recovery was not demonstrated for that model.
Does quantization change which attention layers matter most for the first-token sink?
Layer effects are model-specific in this study: the two Qwen models show sharp shifts concentrated in terminal layers (6.2–7.9× their model-mean shift; per-layer rank collapsing to 0.033 in Qwen2.5-0.5B layer 22), while Llama-3.2-1B drifts little and uniformly. The authors could not explain the difference via per-layer weight dynamic range or GQA sharing ratios.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 appear only as captions in the extracted text; distribution shapes and CI bands could not be independently inspected, though key values are duplicated in tables and prose.
- The extraction appears otherwise complete (all five tables, formulas, and references present); judgments assume the math rendering (Eq. 1–8, the JS-distance definition) is faithful.
- The review covers arXiv v1 (2609.23585), an IEEE-camera-ready-style preprint announcing IECON 2026 acceptance.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.