Attention Quantization for Tabular Foundation Models
Jonas M. Kübler, Benjamin Jäger, Klemens Flöge, Noah Hollmann, Frank Hutter
cs.LG · cs.AI
Quick verdict
Claims FP8 attention gives tabular foundation models up to 1.7x speedup with negligible accuracy loss; rigorous but Blackwell-only, 3 seeds, no code stated.
Summary
The paper introduces FP8 quantization of the attention computation in tabular foundation models, arguing that — unlike LLMs, where weight and KV-cache quantization dominate — efficiency work should target the attention calculation, whose cost scales quadratically with training-row count N. Using TabPFN-v3 as the workhorse and TabICLv2 for generality, the authors quantize queries, keys, and values with per-head dynamic absmax scaling to e4m3fn FP8 in a Triton kernel modeled on FlashAttention-2, targeting NVIDIA's Blackwell RTX Pro 6000. The central empirical finding is architectural: quantizing the train-train attention alone or the test-train attention alone degrades TabArena accuracy substantially (+2.8% and +2.4% relative error, -32 and -20 Elo), whereas quantizing both calls consistently is nearly lossless (+0.01-0.02%, about -1 Elo); Gaussian-noise ablations (Section 3.2, A.1) show this coordination requirement generalizes beyond quantization. Reusing the train-query scale for test queries preserves batch invariance. Under an N > 8192 gate, the kernel reaches up to 1.72x speedup over the FA2/SDPA baseline and end-to-end predict speedups up to 1.67x at 512k rows (Table 3); a 16-bit variant of their own kernel confirms the gains stem from FP8 tensor cores. Across TabArena (51 datasets) and BeyondArena (142 datasets, 56 above the gate) over three preprocessing seeds, accuracy changes stay within seed noise. Limitations: one primary GPU (H100/B200 unsupported without TMA redesign), only three seeds, no comparison to SageAttention-class kernels, and no code-availability statement.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- For tabular foundation models, attention — not weights or KV cache — is the right quantization target, because ICL FLOPs scale as O(N^2 + MN) in training rows N while weight multiplications scale linearly (Section 2, Figure 3).
- Per-head absmax FP8 quantization of Q/K/V is nearly lossless only when the train-train and test-train attention calls are quantized consistently: +0.01-0.02% relative error and about -1 Elo jointly, versus +2.4-2.8% and -20 to -32 Elo for single-call quantization (Table 1).
- The train/test coordination requirement is a general architectural property, reproduced by coordinated versus independent Gaussian noise on K/V across the two attention calls (Section 3.2, A.1).
- A Triton FP8 kernel gated at N > 8192 achieves up to 1.72x kernel and 1.67x end-to-end speedups on RTX Pro 6000, with accuracy changes within seed noise on TabArena (51 datasets) and BeyondArena (142 datasets) over three preprocessing seeds (Tables 2-3).
- Reusing the train-query quantization scale for test queries makes predictions batch invariant at no measurable accuracy cost (Table 1, Section 2).
Strengths
Correct problem targeting for a new model class
Section 1 and Figure 3 show ICL attention FLOPs scale as O(N^2 + MN) vs linear O(N + M) for weight multiplications, justifying attention-focused (not weight/KV-cache) quantization; the authors correctly note this differs from the LLM optimization playbook.
A genuinely novel, transferable finding
The train/test coordination requirement (Table 1: single-call quantization costs +2.38-2.81% relative error and -20 to -32 Elo; joint quantization only +0.01-0.02% and about -1 Elo) is validated beyond quantization via coordinated vs independent Gaussian noise on K/V (Section 3.2, A.1), indicating an architectural property of two-call ICL attention.
Clean speedup attribution
Unusually rigorous and honest statistical reporting
Practical engineering details with ablations
Weaknesses
Hardware transferability of the headline speedup is unproven
Small effective sample for the accuracy claims
No comparison to existing low-precision attention kernels
No code-availability statement
Generality evidence is thinner outside TabPFN-v3
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Attention Quantization for Tabular Foundation Models — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is attention quantization for tabular foundation models?
It is a post-training optimization that stores and multiplies the queries, keys, and values of the attention layers in 8-bit floating point (FP8) instead of 16-bit, using tensor-core FP8 matrix-multiply instructions to speed up inference. For tabular models like TabPFN, attention over training rows dominates the cost (it grows quadratically with training-row count), so quantizing attention matters more than quantizing model weights.
How much faster is FP8 attention in TabPFN-v3?
On an NVIDIA RTX Pro 6000 (Blackwell), the authors' Triton kernel reaches up to 1.72x kernel-level and up to 1.67x end-to-end prediction speedup at 512k training rows (Table 3); the full BeyondArena benchmark suite finishes about 1.4x faster (253.8 to 183.7 minutes). Below roughly 8,192 training rows the kernel is disabled because quantization overhead is not amortized (it is 0.30x, i.e., slower, at 1k rows).
Does FP8 quantization reduce TabPFN accuracy?
Not meaningfully, provided the train-train and test-train attention calls are quantized consistently. Quantizing only one of the two calls drops Elo by about 20-32 points on TabArena (relative error +2.4% to +2.8%), while quantizing both changes relative error by only +0.01% to +0.02% and Elo by about -1, within variation across three preprocessing seeds on TabArena and BeyondArena.
Why must train and test attention be quantized together?
In in-context tabular models, test rows attend to the same keys and values derived from the training rows. If only one of the two attention calls receives quantized inputs, the two calls effectively see different versions of the training data, and accuracy degrades sharply. Gaussian-noise experiments in the paper show this coordination requirement is a general property of the architecture, not something specific to FP8 arithmetic.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure contents (e.g., the Elo curves in Figure 3 and per-dataset points in Figure 4) are available only via captions and in-text descriptions; exact plotted values cannot be checked from the extracted text.
- 'codeAvailable: false' reflects the absence of any code/data availability statement in the provided text, not a confirmed statement that no code exists.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.