SwarmBench: Can Large Language Models Act as Agent Swarm Orchestrators?
Jinshan Gao, Zhuoran Jin, Tianyi Men, Kang Liu, Jun Zhao
cs.CL
Quick verdict
SwarmBench: a new benchmark for LLM swarm orchestration, plus SwarmExp; sound multidimensional eval, but in-distribution experience reuse weakens gains claim.
Summary
The paper introduces SwarmBench, an 8-task, 400-sample benchmark for evaluating LLMs as orchestrators of dynamically created agent swarms, spanning three capability categories: task decomposition (Batch Download, Multi-Text Understanding, MATH), subagent creation/delegation (Multi-Perspective Analysis, Root Cause Analysis, Treasure Hunt), and result aggregation (Long-Text Generation, Wide Search). A shared lightweight swarm framework pins subagent backbones to a fixed six-model pool while the evaluated model acts as orchestrator; accuracy, dollar cost, parallelism rate/gain, and LLM-judged process quality are reported. Across ten orchestrators, GPT-5.4 leads (50.49 average accuracy, best on 6/8 tasks), open-source models lag and specialize, aggregation is the weakest process dimension, and cost-performance is non-linear, governed by orchestration structure rather than raw parallelism or spend. Building on these findings, SwarmExp extracts skills, tricks, and model cards from trajectories and reinjects them, improving GPT-5.4 by +0.25 to +6.46 on four weak tasks; experience transfers to 14/16 cross-model settings, and ablations attribute the largest share to model cards. Limitations include LLM-as-judge scoring for several tasks (human agreement 0.684-0.801 Pearson, single annotator), near-floor scores on RCA and Treasure Hunt, variance analysis covering only two models, and the fact that headline SwarmExp gains reuse experience extracted from the same benchmark tasks - an in-distribution setup that partly undermines claims of improved general orchestration.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing benchmarks, built for single-agent or general-purpose settings, cannot systematically evaluate Agent Swarm orchestration, which SwarmBench addresses with 8 tasks and 400 samples covering decomposition, delegation, and aggregation.
- Current models differ substantially as orchestrators: GPT-5.4 averages 50.49 accuracy and is best on 6/8 tasks, and cost-performance is governed by orchestration structure rather than by cost level or parallelism degree.
- Result aggregation is the orchestration capability that is hardest to stabilize, showing the lowest process scores across tasks and models.
- SwarmExp, which extracts skills, tricks, and model cards from trajectories and reinjects them into the orchestrator context, improves four weak tasks by +0.25 to +6.46, transfers to 14/16 cross-model settings, and loses most from model-card removal in ablation.
Strengths
Purpose-built orchestration benchmark with multi-axis metrics
8 tasks x 50 samples in three capability categories (Sec. 2.2), scored not only on accuracy but on USD cost (App. C.3), parallelism rate/gain (App. C.2) and LLM-judged process quality over decomposition, delegation and aggregation (Sec. 3.5) - a combination absent from single-agent or final-answer-only benchmarks.
Controlled harness isolates orchestration ability
The evaluated model is the sole orchestrator, barred from tool use and independent answering with part of its context hidden, while all orchestrators share one fixed six-model subagent pool (Sec. 2.3, App. B), so score differences are attributable to orchestration policy rather than subagent strength.
Rule-based evaluation for both new synthetic tasks
Honest cost-matched baseline
Component ablation and transfer analyses for SwarmExp
Weaknesses
In-distribution experience replay in headline SwarmExp results
Near-floor tasks weaken discriminative power
Thin statistical support for close rankings
LLM-as-judge centrality with opaque judge identity and single annotator
Consistency of SwarmExp overclaimed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SwarmBench (arXiv:2608.30661) targets an under-served evaluation niche: not whether LLM agents solve tasks, but whether the orchestrator model can decompose objectives, spawn heterogeneous subagents, and aggregate their outputs in a dynamic Agent Swarm. The benchmark assembles 8 tasks x 50 samples (400 total) in three capability categories: decomposition (Batch Download, Multi-Text Understanding, MATH), delegation (Multi-Perspective Analysis, Root Cause Analysis, Treasure Hunt), and aggregation (Long-Text Generation, Wide Search). Six tasks adapt Loong, Omni-MATH (difficulty >7), DeepReview, OpenRCA, LongInOutBench, and WideSearch; Batch Download and Treasure Hunt are new synthetic environments. The companion method SwarmExp claims that skills, tricks, and model cards mined from trajectories and replayed into the orchestrator's context consistently improve orchestration. Both claims are broadly supported, with the caveats below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SwarmBench?
SwarmBench is a benchmark of 8 tasks and 400 samples testing whether LLMs can act as orchestrators of dynamically created multi-agent 'swarms'. It scores accuracy, monetary cost, parallel execution gains, and the quality of the orchestration process (task decomposition, subagent delegation, result aggregation). Six tasks adapt public datasets (Loong, Omni-MATH, DeepReview, OpenRCA, LongInOutBench, WideSearch); Batch Download and Treasure Hunt are new synthetic environments with rule-based scoring.
Which LLM is the best swarm orchestrator according to the paper?
GPT-5.4, with 50.49 average accuracy and the best score on 6 of 8 tasks. Claude-Sonnet-4-6 follows at 43.51, while the strongest open-source orchestrators (Qwen3.5-397b at 40.62, Deepseek-v3.2 at 40.38, GLM-5.1 at 40.30) cluster around 40.
What is SwarmExp and how much does it help?
SwarmExp is a post-hoc method that mines swarm execution trajectories for reusable knowledge - workflow skills (SKILL.md files), short condition-action tricks, and empirical model cards - and injects them into the orchestrator's context. On GPT-5.4 it adds +0.25 to +6.46 points on four weak tasks, and experience extracted from GPT-5.4 improves 14 of 16 other-model settings. Ablations show removing model cards hurts most.
Do agent swarms actually beat a single agent in this study?
Mostly yes: the swarm framework beats single-agent baselines on most tasks and still wins MTU, LTG, and WS under a cost-matched comparison. The exception is Root Cause Analysis, where a cost-matched single agent (3.11) outperformed the Claude-Haiku-4-5 swarm (2.00), reported in the paper's Appendix D.1.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-4 are embedded as plot text; exact per-model parallelism gain/rate and process-quality scores beyond axis ranges (gain 1.0-1.6, rate 1.5-4.0) cannot be quoted numerically.
- Figure 1's illustrative UI dialogue is interleaved into Section 2.2 by PDF extraction, which locally garbles reading order but loses no content.
- Appendix B lists thinking-variant names (e.g., Claude-Sonnet-4-6-thinking, Gemini-3-flash-preview-thinking) that differ from the shorthand names in Table 1; the mapping is inferable but exact model-version pins are unstated.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.