Mo' Models, Mo' Problems: How to best select model pools when designing Multi-Agent Systems
Sara Vera Marjanović, Jiacheng Xu, Aleksandr Laptev, Grigor Nalbandyan, Erik Arakelyan, Evelina Bakhaturina
cs.MA · cs.AI
Quick verdict
Claims that growing heterogeneous model pools degrades MAS performance below its best single model; a systematic 23-model study, credible overall, but headline curves lack variance estimates.
Summary
The paper asks how the choice of candidate models affects multi-agent system (MAS) performance. Eight selection strategies - model size, architecture family, LLM recommendation, accuracy, correct-answer (IoU) diversity, error diversity, and two accuracy-diversity hybrids - order a pool of 23 open models (2B-1.6T parameters, six families, 2024-2026 releases), and subsets of size k in {3,5,10,15,20} are evaluated in three architectures: clustering-based routing, embedding-based majority voting (E5-Large-V2, cosine threshold 0.15), and LLM-as-a-judge (GenSelect with gpt-oss-120b), on HLE, GPQA-Diamond, and FrontierScience-Olympiad, with five generations per model per question. Calibration uses 15.5k questions whose per-model rankings correlate strongly with the test sets (r>0.9). Key results: oracle ceilings rise with k, but achieved performance of heterogeneous pools almost always falls below the best single member as k grows; only same-family subsets (notably Gemma-4, OLMo-3) occasionally improve over a standalone model; science specialists (cosmosage-v3.1, Chem-R-8B) do not outperform their generalist base Llama-3.1-8B even on their own domains; correct answers of accurate models correlate highly (Mantel r=0.931) while errors correlate only weakly (r=0.384). Main limitations: only simple MAS architectures and one routing backbone are tested, the judge is also a candidate model, GenSelect stratification may mechanically hurt large pools, and the headline curves lack variance estimates.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Expanding a heterogeneous candidate pool typically degrades actual MAS performance below that of the best single model in the pool, even as the oracle ceiling rises with pool size.
- Selecting candidates within a single model family is the only strategy that reliably approaches or exceeds standalone-model performance across MAS architectures.
- High oracle potential does not predict achieved MAS performance, so candidate models must be empirically evaluated rather than chosen from model-card signals.
- Science-domain specialists (cosmosage-v3.1, Chem-R-8B) do not outperform their generalist base Llama-3.1-8B even on their target domains, so specialization claims on model cards do not transfer.
- Accurate models answer highly overlapping question sets (Mantel r=0.931) while their errors are only weakly correlated (r=0.384), limiting the diversity available to heterogeneous pools.
Strengths
First systematic treatment of an overlooked MAS design axis
Classical ensemble literature made member selection central, but prior LLM-MAS work (routing surveys, MoA, LLM-Blender) mostly fixed or ignored pool composition; this paper crosses 8 selection strategies x 3 architectures x 3 benchmarks x 23 models (Section 3, Table 1), which is the right experimental grid for the question.
Judge quality is empirically validated and its bias direction disclosed
gpt-oss-120b judging was checked against 100 human-graded HLE answers: 93% agreement, Cohen's kappa 0.63 (Section 3 footnote 2 and A.2), and the authors state the judge is more lenient than the human grader, so reported accuracies are upper bounds.
Robustness analyses for the fragile components
Full subset disclosure enables auditing of selection effects
Useful negative result on specialists
Weaknesses
No variance or significance testing behind the headline claim
Judge is also a candidate, and GenSelect is structurally handicapped at high k
Majority vote is not literal voting
Family strategy is confounded by tiny subset sizes
Proprietary calibration data limits reproducibility
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of 'Mo' Models, Mo' Problems'?
Adding more heterogeneous LLMs to a multi-agent system usually lowers accuracy below the best single model already in the pool, even though the theoretical best-possible (oracle) score keeps rising with pool size. The most reliable way to pick agents is to take several models from one model family rather than mixing families or chasing diversity.
Which model-selection strategies does the paper compare?
Eight strategies for ordering candidate models: total parameter size, shared architecture family, recommendation by GPT-5 with Deep Research, measured accuracy, correct-answer diversity (IoU), error diversity, and two hybrids weighting accuracy together with IoU or error diversity. These are tested in three system types: a router, majority voting, and an LLM-as-a-judge setup.
Did specialized science models beat generalist models in the study?
No. Physics- and chemistry-fine-tuned models (cosmosage-v3.1 and Chem-R-8B, both built from Llama-3.1-8B) did not outperform their generalist base model, even on physics and chemistry questions, and their apparent gains were spread across unrelated domains. The authors warn against trusting domain-specialization claims on model cards.
Is code or data available for this paper?
The paper displays a repository marker (spaidataiga/mo-models) suggesting a code release, and it fully discloses every evaluated model subset in its appendix tables; however, the calibration set includes closed-source STEM questions from Turing, Scale, and Stack-Overflow, so full reproduction of the routing and selection pipeline may not be possible from public artifacts alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The main quantitative results live in figures (Figures 4-5, 13-21) that cannot be analyzed from the extracted text; only isolated inline numbers (29.4%/32.2%/36.5% on HLE; correlation values) are verifiable.
- The repository marker 'spaidataiga/mo-models' appears to be a mangled link in extraction; the actual code-release status could not be confirmed from the text.
- References to 2026 models and benchmarks (Qwen3.5, DeepSeek-v4, Gemma-4, FrontierScience) could not be cross-checked offline.
- Appendix figures 6-12 (threshold sensitivity, clustering heatmaps, specialist comparisons) are only available through their textual descriptions.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.