Scalability and Performance Evaluation of Federated Learning Frameworks: A Comparative Analysis
Bassel Soudan, Sohail Abbas, Ahmed Kubba, Manar Wasif Abu Talib, Qassim Nasir
cs.DC · cs.AI
Quick verdict
Claims OpenFL is most scalable of four FL frameworks on MNIST (2–100 clients); useful data, but unverifiable setup and inconsistent stats weaken trust.
Summary
This paper experimentally compares four open-source federated learning frameworks — Flower, FedML, Substra, and OpenFL — on MNIST, scaling simulated clients from 2 to 100 (the text elsewhere says 20 to 100, contradicting Table 2) on a single Windows workstation (Xeon W-2102, 32 GB RAM, RTX 4000 GPU), with 20 federated rounds of 15 local epochs; training time, loss, accuracy, and CPU/RAM usage were recorded via Psutil, excluding network delays. Reported results: Flower achieves the highest accuracy (98–99%) but an anomalously high loss (2.7–5.3); FedML reaches only 52–79% accuracy while consuming over 90% CPU; Substra is the most resource-efficient (~40% CPU, ~21% RAM) yet its training time grows from 180 to 13,630 seconds; OpenFL is judged most scalable, with stable accuracy (94.9–97.4%) and loss, though its RAM usage rises from 42.8% to 63.1%. Key limitations: model architecture, aggregation algorithm, framework versions, and repetition counts go unreported; no variance or statistical tests back the claimed 'statistical significance'; internal inconsistencies remain unexplained; no code is released; a single dataset and single-machine setup limit generalizability.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- OpenFL is the most scalable of the four frameworks, maintaining consistent accuracy, loss, and training time as clients scale from 2 to 100 (§6.2.4).
- Flower attains the highest accuracy (average ~98.75%) but an anomalously high loss (average ~4.247) across all client counts (§6.1.1).
- FedML achieves only 52–79% accuracy (Table 2; abstract says 66–79%) while consuming over 90–99% CPU, making it unsuitable for resource-constrained deployments (§6.2.2).
- Substra is the most resource-efficient framework (CPU ~35.8–42.9%, RAM ~20.6–23.2%) but its total training time grows from 180 s to 13,630 s, described as exponential (§6.2.3).
- No prior comparative work evaluated all four frameworks' scalability and resource efficiency, motivating this study's four stated contributions (§3).
Strengths
First same-protocol head-to-head across four major FL frameworks
Per the §3 survey, prior comparisons were survey-based [11], IoT-scoped [10], or literature-only [12]; Table 2 is a single quantitative table covering Flower, FedML, Substra, OpenFL under one dataset, hardware, and schedule (20 rounds × 15 epochs).
Multi-dimensional metric coverage with concrete numbers
Six client counts (2/15/30/60/80/100) crossed with four metric families — loss, accuracy (Table 2), total training time (Fig. 6), and CPU/RAM percentages (Figs. 7–8) — give practitioners a usable decision matrix, e.g., Substra at ~35.8–42.9% CPU vs FedML at 90.5–99.0%.
Actionable, data-tied deployment guidance
Explicit disclosure of measurement scope
Weaknesses
Irreproducible experimental configuration
No statistical reporting despite claiming significance
Internal inconsistencies in the paper's own data narrative
OpenFL 'most scalable' verdict overreaches its own table
Narrow external validity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.15681; cs.DC, cs.AI) targets a gap the authors identify in §3: earlier comparisons are survey-based [11], IoT-scoped [10], or literature-only [12], and none benchmark scalability or resource efficiency across frameworks. Four open-source FL frameworks — Flower, FedML, Substra, OpenFL — are benchmarked on MNIST at 2/15/30/60/80/100 clients, 20 federated rounds × 15 local epochs, measuring training time, loss, accuracy, and CPU/RAM usage (Table 2). Headline claims: Flower = best accuracy (avg ~98.75%) but anomalous loss (avg ~4.247); FedML = low accuracy with >96% CPU; Substra = most resource-efficient but exponential training-time growth (180→13,630 s); OpenFL = most scalable overall. Note the abstract's FedML range (66–79%) quietly drops Table 2's 52.12% low at 30 clients — an attribution slip worth flagging.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which federated learning framework does this paper find most scalable?
The authors conclude OpenFL is the most scalable: its accuracy (94.9–97.4%), loss, and training time stayed most consistent as simulated clients grew from 2 to 100 on MNIST. Caveats from its own data: OpenFL's RAM usage rose from ~43% to ~63% of the workstation, and FedML had lower absolute training times at every client count — so the verdict is about consistency, not raw speed.
Why does FedML score so poorly in this comparison?
The paper reports FedML reaching only 52.1–79.1% accuracy on MNIST (the abstract rounds this to 66–79%, which omits the 52% low point) while consuming over 90–99% CPU. No cause is investigated and no tuning is described, so it remains unclear whether this reflects the framework itself or a configuration/integration problem in the authors' setup.
What hardware and setup were used in the experiments?
All four frameworks ran on a single Windows 10 workstation (Intel Xeon W-2102, 4 cores at 2.9 GHz, 32 GB RAM, Nvidia RTX 4000), with all simulated clients and the aggregation server executed in parallel on that one machine. Experiments used MNIST with 20 federated rounds of 15 local epochs at 2/15/30/60/80/100 clients; timing and resource metrics came from the Python Psutil library, and network delays were excluded entirely.
Is the paper's code or data publicly available?
No. The paper does not mention releasing code, configurations, or raw measurements, and it omits the model architecture, optimizer, aggregation algorithm, and framework versions — so independent reproduction would require rebuilding the entire setup from scratch.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 4–10 appear in the extraction only as axis-label and value dumps; trend judgments (e.g., 'exponential' Substra growth) were checked against Table 2's numbers rather than the plotted curves.
- Figures 9–10 (per-framework CPU/RAM patterns, Substra training curves) contain no extractable data, so their supporting evidence could not be assessed.
- Framework versions and experiment dates are unstated; the findings may not reflect current releases of these fast-moving frameworks.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.