BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Training
Bigyan Ghimire, Jon C. Calhoun
cs.DC · cs.LG
Quick verdict
Decomposes Ulysses' global all-to-all into per-batch subgroup collectives; 1.17-1.31x speedups on 8 A100s are plausible but small-scale and single-baseline.
Summary
The paper targets the all-to-all communication cost of DeepSpeed-Ulysses sequence parallelism during long-context LLM training. The authors observe that Ulysses performs a global N-way all-to-all during attention regardless of micro-batch size B. BASP exploits cases where B>1 and N=K·B: the N GPUs are partitioned into B disjoint groups of K=N/B GPUs, each owning one sequence and running an independent K-way all-to-all, cutting per-GPU communication peers from N−1 to K−1 while keeping BS/N tokens per GPU. Contiguous rank assignment confines communication to intra-node NVLink when K equals GPUs per node. The implementation modifies DeepSpeed with ZeRO-3 and mixed precision. On 2 nodes × 4 A100-40GB GPUs with 400Gbps InfiniBand, across six Llama/Qwen models (1B–8B) at 16K sequence, B=2, BASP achieves 1.17–1.31× end-to-end speedup and 2.23–3.10× all-to-all reduction (37.7%→16% on Qwen-1.5-1.8B). Speedup grows with batch size (1.10×→1.26× for B=2→8, 8K sequence, Llama-3.2-1B, with an 85× all-to-all reduction at B=8) and with sequence length (5.7% at 1K to 25.9% at 32K, Llama-3.2-3B, B=2). Accuracy evidence is an 800-iteration loss-overlap run on Llama-3.2-3B. Limitations: requires N divisible by B; evaluation limited to 8 GPUs and one topology; only Ulysses measured as baseline; no variance reported and no code release stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Standard Ulysses-SP performs a global N-way all-to-all during attention that is unnecessary when B>1 and N=K·B; BASP decomposes it into B independent K-way collectives with unchanged per-GPU memory (BS/N tokens) and workload.
- BASP improves end-to-end step time by 1.17–1.31× across six Llama/Qwen models (16K sequence, micro-batch 2, 8×A100) and cuts all-to-all time by 2.23–3.10×.
- Gains scale with micro-batch size (1.10×→1.26× for B=2→8 at 8K on Llama-3.2-1B, all-to-all down 85× at B=8) and with sequence length (5.7% at 1K → 25.9% at 32K on Llama-3.2-3B, B=2).
- Contiguous rank assignment confines all-to-all traffic to intra-node NVLink when K equals GPUs per node, explaining communication reductions beyond the ~2× participant-count prediction.
- BASP preserves accuracy, evidenced by overlapping loss curves with Ulysses-SP over 800 iterations on Llama-3.2-3B.
Strengths
Clean, provably workload-preserving decomposition
Section IV.C derives per-GPU work equality — (B/G)·H·(SG/P)·d = B·H·S/(P·d) — and the B=1 case in Fig. 8 (24.5% vs 24.9% all-to-all share) empirically confirms degeneracy to standard Ulysses.
Explicit positioning against the trivial alternative
Section IV.E and Table I carefully separate BASP from Ulysses with SP=N/B (which costs B²S/N tokens/GPU versus BASP's BS/N), making memory-preserving communication reduction the paper's sharpest technical claim.
Topology-aware design matches measured gains
Honest Amdahl accounting
Consistent trends across two model families
Weaknesses
Single baseline; the key foil is analyzed but never measured
Evaluation confined to 8 GPUs on one 2×4 topology
Thin accuracy validation
Narrow applicability plus polish issues
No artifact availability
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
BASP is a systems method paper targeting the all-to-all collective in DeepSpeed-Ulysses sequence parallelism during long-context LLM training (Sections I, III). The motivating observation (Fig. 1, Fig. 5 kernel profiling): Ulysses' attention-phase all-to-all spans all N GPUs regardless of micro-batch size B, growing from 24.5% to 33.8% of iteration time as B goes 1→8 (Llama-3.2-1B, 8K, N=8). The core claim is that when B>1 and N=K·B, the single N-way collective decomposes into B independent K-way all-to-alls — each group of K=N/B GPUs owns one sequence — without changing per-GPU workload or memory (BS/N tokens per GPU, Section IV.C). Secondary claims: contiguous rank assignment confines traffic to intra-node NVLink, and accuracy is preserved. Headline results: 1.17–1.31× end-to-end speedup and 2.23–3.10× all-to-all reduction on six Llama/Qwen models at 16K sequence, B=2, on 8×A100-40GB.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Batch-Aware Sequence Parallelism (BASP)?
BASP is a modification of DeepSpeed-Ulysses sequence parallelism for LLM training. When the micro-batch size B is greater than 1 and the GPU count N divides as N=K·B, it splits the N GPUs into B disjoint groups of K GPUs; each group handles one sequence and runs its own small K-way all-to-all during attention instead of one global N-way exchange. Per-GPU memory and compute stay unchanged (BS/N tokens per GPU).
How much speedup does BASP achieve?
On a cluster of 2 nodes × 4 NVIDIA A100-40GB GPUs, across six Llama and Qwen models (1B–8B) at 16K-token sequences with micro-batch size 2, BASP improves end-to-end step time by 1.17–1.31× over standard Ulysses-SP and cuts all-to-all communication time by 2.23–3.10×. Gains grow with batch size and sequence length, reaching 25.9% step-time reduction at 32K context on Llama-3.2-3B.
Does BASP change model accuracy or memory usage?
The authors claim no change: each GPU still holds BS/N tokens, and loss curves overlap with standard Ulysses-SP over 800 training iterations on Llama-3.2-3B. Independent verification is limited — no downstream evaluation, held-out perplexity, or long-horizon training is reported in the paper.
What are BASP's main limitations?
It requires the GPU count to be an integer multiple of the micro-batch size (N=K·B), falls back to data parallelism when B>N, and was evaluated only on 8 GPUs across 2 nodes against Ulysses-SP alone. Attention heads must also divide evenly across group members, and no code release is stated in the paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1, 7 and 8 were extracted as raw label/value sequences; the pairing of percentage labels to series (Ulysses vs BASP) relies on textual order, so a few label-to-series pairings carry minor interpretation risk.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.