OpenMAS-GCom. A Diagnostic Benchmark for Graph-enhanced Multi-Agent Systems
Kairui Yang, Xunkai Li, Xunkai Li, Kaixiang Zhang, Minghao An, Zekai Chen, Zekai Chen, Yuxuan Ba, et al.
cs.LG
Quick verdict
Claims a controlled-intervention benchmark diagnosing G-MAS components across 29 datasets; rigorous design, but intervention evidence spans only five datasets.
Summary
The paper introduces OpenMAS-GCom, a benchmark that reframes evaluation of graph-enhanced multi-agent systems (G-MAS) around attribution rather than final-score comparison. Each system is expressed as an organization (collaboration units, directed communication edges, shared state, execution policy), and four paired intervention families are applied while tasks, models, prompts, and budgets stay fixed: degree-preserving edge rewiring (10-70%), role-unit removal (no-critic/no-specialist/expert-only), replacement of intermediate messages with format-valid incorrect candidates (0-80%), and worker failures (0-4). Seventeen configurations are run over 29 datasets in six domains, plus 400 new G-MAS-Complex multi-document tasks scored by strict exact match over values, rankings, and source identifiers. Reported results: rewiring drops the five-dataset mean from 81.88% to 74.99% at 70%; specialist removal costs 3.58 points versus 0.58 for critic removal; MAD and VeriMap, nearly tied at baseline (85.30/85.50), separate by 10.18 points at 80% corruption; GoAgent attains the highest G-MAS-Complex accuracy (50.25%) while AutoGen leads score-per-token (6.93), with only GoAgent and VeriMap beating the DeepSeek single-agent reference. Main limitations: intervention evidence covers five datasets and two method pairs; uncertainty is reported through heterogeneous sources whose run counts are unrecorded; no significance testing of headline contrasts; the reimplementation-validation protocol is specified but its numerical results are absent; and the released library is claimed without a visible URL in the text.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Degree-preserving rewiring of communication edges monotonically degrades performance (five-dataset mean 81.88% to 74.99% at 70% rewiring), so topology arrangement matters even when degrees, edge count, and budgets are fixed.
- Role removal effects are strongly asymmetric: removing specialist agents costs 3.58 percentage points versus 0.58 for critics (a 6.17x ratio) across five datasets.
- Systems with near-identical reference scores diverge sharply under perturbation: MAD vs VeriMap separates from 0.20 to 10.18 points at 80% message corruption, and GoAgent vs AutoGen from 0.40 to 3.00 points at four worker failures.
- On the 400-task G-MAS-Complex suite, the accuracy leader (GoAgent, 50.25% Strict EM) differs from the token-efficiency leader (AutoGen, 6.93 EM/K tokens), and only 2 of 12 graph-enhanced configurations beat the DeepSeek single-agent baseline (26.75%).
Strengths
Attribution-by-intervention design
Section 2.2 (Eqs. 4-6) compares paired score changes under shared controls (tasks, backend, prompts, parser, budgets, Table 6), with the complete modified execution as the unit of analysis; degree-preserving rewiring (Sec 3.3) holds in/out-degree sequences, edge count, and budget fixed — a clean answer to Q4 'does topology matter when degrees are fixed'.
Unusually transparent uncertainty taxonomy
Appendix A.8/Table 10 codes all 510 result cells by statistical source (s: three-sample SD; r: run-seed SD; e: composite estimate; b: binomial SE; m: backup), Table 17 attaches binomial SEs over N=400, and A.6 describes audit scripts excluding runs failing consistency checks before aggregation.
Broad coverage plus a targeted stress suite
Honest, decision-relevant findings
Weaknesses
Narrow intervention evidence base
No inferential statistics on headline contrasts
Baseline fidelity asserted, not demonstrated
Strict EM may over-weight output compliance
Open-source claim unverifiable
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
OpenMAS-GCom: Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is OpenMAS-GCom?
OpenMAS-GCom is a diagnostic benchmark for graph-enhanced multi-agent systems built on LLMs (G-MAS). Instead of only comparing final task scores, it applies controlled interventions — rewiring communication edges while preserving node degrees, removing specific role agents, replacing intermediate messages with incorrect but format-valid content, and disabling worker agents — while holding tasks, models, prompts, and compute budgets fixed, so that performance changes can be attributed to specific organizational components.
What are the main findings of the OpenMAS-GCom paper?
Across five datasets, removing specialist agents lowers the mean score by 3.58 percentage points versus 0.58 for critic removal, and 70% degree-preserving rewiring cuts the five-dataset mean from 81.88% to 74.99%. Systems with nearly identical baseline scores diverge under stress: MAD and VeriMap separate by 10.18 points at 80% message corruption, and GoAgent loses less accuracy than AutoGen as worker failures increase. On the 400-task G-MAS-Complex suite, GoAgent achieves the highest accuracy (50.25% strict exact match) while AutoGen achieves the best score per 1K tokens (6.93).
How many methods and datasets does OpenMAS-GCom evaluate?
It evaluates 17 configurations: 2 single-agent baselines (DeepSeek, Qwen), 3 ordinary multi-agent systems (AutoGen, MADebate, DyLAN), and 12 graph-enhanced systems (including GPTSwarm, G-Designer, GoAgent, VeriMap). These run on 29 datasets across six domains — mathematics, knowledge/commonsense, code, biomedical QA, finance, and tabular reasoning — plus a newly constructed suite of 400 multi-document G-MAS-Complex tasks in four difficulty tiers, scored by strict exact match requiring correct values, rankings, and source identifiers.
Is the OpenMAS-GCom code and data publicly available?
The paper claims an open-sourced benchmark library with reusable modules for task loading, method execution, scoring, and interventions, and describes detailed audit records (organization hashes, prompt hashes, intervention seeds, cost logs). However, no repository URL is visible in the paper text, so the availability and completeness of the release cannot be verified from the paper alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-5 were recovered only as embedded number streams with axis labels; panel curves and visual groupings were inferred from text, not from rendered images.
- No repository URL appears in the extracted text; the open-source claim could not be verified against an actual artifact.
- The abstract is truncated mid-sentence in the metadata, but the full body and appendices appear complete through the references.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.