Scalability Analysis of Distributed Kolmogorov-Arnold Network Training on High-Performance Computing Systems
Guangneng Chen, David Garcia Selfa, Pablo Quesada Barriuso
cs.DC · cs.LG · cs.PF
Quick verdict
First DDP scalability study of KAN on 8 A100s; measurements credible and honestly scoped, but tiny synthetic-task models and no MLP control limit generality.
Summary
This paper presents what the authors describe as the first systematic scalability characterization of data-parallel Kolmogorov-Arnold Network (KAN) training, using PyTorch DDP on the FinisTerrae III supercomputer with up to 8 NVIDIA A100 GPUs across 4 nodes. Four dimensions are evaluated on a synthetic 2D regression task (f(x,y)=x^2+y^2) with reference PyKAN models of 2,726 to 21,542 parameters. Strong scaling of a fixed 100,000-sample workload yields 5.97x speedup and 74.7% parallel efficiency at 8 GPUs; weak scaling at 25,000 samples per GPU shows per-GPU throughput stable at 3,440-3,463 samples/s once the roughly 21% single-to-multi-GPU DDP entry cost is paid. Communication overhead (1.3%-6.1%) is non-monotonic and attributed to NCCL All-Reduce algorithm selection and node topology rather than KAN structure. Model-scaling shows training time grows 7.08x for 8x width while per-parameter throughput falls 56x, yet the parameter-to-memory ratio improves (about 74K to 118K parameters/GB). Key limitations, largely acknowledged by the authors: a single run per configuration (variability is within-epoch only), no matched-MLP control to support the no-KAN-specific-penalty claim, NCCL algorithm choices inferred rather than traced, and small model and workload scale on a single cluster. Code is publicly available. Results are credible as a system-level characterization, but generalization to real workloads remains untested.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 4 weaknesses · 0 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- KAN training under PyTorch DDP reaches 74.7% parallel efficiency and 5.97x speedup at 8 A100 GPUs, consistent with conventional DDP workloads rather than a KAN-specific effect (Table 1).
- The non-monotonic communication-overhead pattern (1.3%-6.1%, Table 3) is driven by NCCL All-Reduce algorithm selection and node topology, not by KAN's edge-wise gradient structure.
- Weak scaling is stable once the fixed ~21% single-to-multi-GPU DDP entry overhead is paid: per-GPU throughput varies by under 0.3% across the 2-8 GPU distributed configurations (Table 2).
- Training time scales near-linearly with model width (7.08x for 8x width) while the parameter-to-memory ratio improves from 74,268 to 118,066 parameters/GB (Table 4), attributed to amortized fixed per-process overhead.
Strengths
Explicit epistemic honesty about inferred mechanisms
Section 4.3 explicitly states NCCL_DEBUG traces were not captured, and Section 5.5 names the missing matched-architecture ablation as the most significant limitation, specifying its full design (parameter-matched MLP, 2G1N vs 2G2N and 4G4N vs 4G2N pairs) for exact follow-up.
Careful, self-aware statistical bookkeeping
Section 3.4 distinguishes population vs sample SD conventions and within-run vs cross-configuration CV; the Table 1 note quantifies how the first warm-up epoch inflates throughput sigma while leaving communication-overhead sigma largely unaffected.
Controlled topology isolation via allocation design
Precise positioning against operator-level work
Reproducibility provisions
Weaknesses
Single run per configuration
Central architecture-generality claim lacks its control
Table 1 vs Table 3 accounting gap
Scale limits generality
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper positions itself as the first application of standard HPC scalability methodology (strong/weak scaling, communication overhead, model-size scaling) to data-parallel KAN training, using PyTorch DDP on FinisTerrae III (up to 8x A100 across 4 nodes). Central claims: (1) 74.7% parallel efficiency and 5.97x speedup at 8 GPUs, consistent with conventional DDP workloads (Table 1); (2) non-monotonic communication overhead of 1.3%-6.1% (Table 3) driven by NCCL All-Reduce selection and topology, not KAN structure; (3) near-linear training-time growth with width (7.08x for 8x) with an improving parameter-to-memory ratio (74,268 to 118,066 params/GB, Table 4); (4) deployment guidance on GPU topology and model size. Claims are consistently hedged to the studied regime (Section 5.3) — commendable discipline.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 0 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What speedup does KAN training get on multiple GPUs?
On the FinisTerrae III supercomputer with 8 NVIDIA A100 GPUs across 4 nodes and PyTorch DDP, this paper measured a 5.97x speedup over a single GPU, corresponding to 74.7% parallel efficiency, for a small KAN on synthetic 2D regression data (Table 1).
Is distributed KAN training slower than MLP training?
The paper argues KAN shows no scaling penalties beyond normal DDP overhead, but it never ran a matched-parameter MLP control; the comparison is argued from first principles and the control experiment is left as specified future work (Section 5.5).
How much GPU memory does distributed KAN training use?
Peak GPU memory grew from 0.04 GB (2,726 parameters) to 0.18 GB (21,542 parameters) on A100s. Because fixed costs like the CUDA context and NCCL buffers dominate at small sizes, the parameter-per-GB ratio improves with model size, from ~74,000 to ~118,000 parameters/GB (Table 4).
Is the code for this KAN scalability study available?
Yes. The authors state that experimental scripts, logs, and analysis code are publicly available at github.com/anhushow/KAN_Scalability_DDP.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–7 appear only as captions in the extracted text; all quantitative checks in this review rely on Tables 1–4, which appear complete.
- The metadata-provided abstract is cut mid-sentence, but the full abstract and complete body (through references [1]–[18]) are present in the embedded text; no body truncation observed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.