MAGIC: Mixed-Granularity Agent Graphs via Incremental Construction with Dense-Reward Reinforcement Learning
Kairui Yang, Ziheng Yi, Xunkai Li, Minghao An, Zhanke Liu, Zekai Chen, Rong-Hua Li
cs.LG
Quick verdict
MAGIC learns per-role atomic-or-group MAS topologies via shaped RL, ranking first on 8 benchmarks; sound idea, but single-run results need caution.
Summary
Collaboration topology in LLM multi-agent systems is typically designed at a single granularity — individual agents or predefined groups. MAGIC instead lets a learned policy choose, for each functional role, whether to realize it as a single agent or a reusable fixed group, building a task-specific graph incrementally (select role, choose granularity, wire predecessors, STOP). Training is on-policy RL: returns from complete trajectories plus potential-based reward shaping that gives intermediate credit from probe-question utility gains, structural complexity, and role-repetition penalties; Proposition 1 shows shaping preserves the terminal task return. A frozen executor (DeepSeek V4 Flash for the main table) runs the graphs; only the small graph policy trains. Across eight benchmarks and 17 baselines (numbers imported from the OpenMAS-GCom benchmark), MAGIC ranks first everywhere, most visibly on MMLU-Pro (83.20 vs 73.60) and LiveCodeBench-v6 (36.57 vs 26.86). Ablations attribute gains to mixed granularity and dense feedback; budget-matched comparisons favor RL over Search-then-SFT. Protocol disclosure (seeds, sampling, scoring) is unusually transparent. Main limitations: all headline results are single runs with no variance estimates or significance tests; several wins sit within 1-2 points; baselines come from a third-party benchmark whose executor configuration may differ; exact prompts are external artifacts; and with at most three outer units the realized search space is far smaller than the framing implies.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Selecting granularity locally per role (atomic agent vs reusable group) outperforms both fixed-granularity endpoints, which remain special cases (Sec. 3 D1, Sec. 5.3 Table 2).
- Potential-based dense reward provides informative intermediate construction credit — raising non-zero advantage rates on intermediate additions from 32.81% to 83.13% on MMLU-Pro (D3) — while provably preserving the terminal task return (Proposition 1).
- Direct dense-reward RL beats Search-then-SFT under matched API cost budgets, winning 4 of 5 budget stages on both MMLU-Pro and TAT-QA (Sec. 5.4).
- MAGIC ranks first on all eight benchmarks against 17 baselines and sits on the performance-token Pareto frontier with the fewest tokens on three of four efficiency datasets (Sec. 5.2, 5.5).
Strengths
Well-motivated mixed-granularity formulation backed by a controlled diagnostic
D1 (200 queries x 8 atomic/group assignments of a fixed decomposer-solver-verifier skeleton) shows the preferred grouped role is task-dependent: GAA wins on MMLU-Pro (+4.5 pts over AAA, 81.5% vs 77.0%) while AAG wins on TAT-QA (+2.5 pts), cutting tokens 58.72% and 48.15% vs GGG (D.1, Table 13).
Correct, cleanly integrated reward-shaping theory
Proposition 1 and the B.2 telescoping proof correctly apply Ng et al. (1999) to graph construction: with eta=1 and settled endpoints, shaped return equals terminal task score, so probe/structural signals redistribute credit without changing the objective (Sec. 4.4, B.3).
Exceptional protocol transparency for this subfield
Budget-matched optimization comparison rather than headline-only
Consistent efficiency story
Weaknesses
No statistical robustness anywhere
Baselines imported from a third-party benchmark, not re-run
Missing control isolating topology learning from the execution harness
Realized search space is far smaller than the framing
Uneven diagnostics and dataset-role hygiene
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MAGIC targets topology generation for LLM-based multi-agent systems. The authors argue existing generators fix node granularity — atomic-agent methods (ARG-Designer, Li et al. 2025) versus predefined-group methods (GoAgent, Chen et al. 2026) — while subtasks within one query differ in collaboration needs. Three claims: (1) a mixed-granularity space where each role is locally realized as a single agent or a reusable fixed group; (2) on-policy RL with incremental construction (role, granularity, predecessors, STOP) plus potential-based shaping from probe utility, structural complexity, and role repetition; (3) empirical dominance — first place on eight benchmarks against 17 baselines and a performance-token Pareto frontier. The embedded text includes the full method, Appendices A-D, and complete protocol tables, so this review is high-confidence.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MAGIC in one sentence?
MAGIC is a framework that learns to build task-specific multi-agent graphs for LLMs where each functional role can be realized either as a single agent or as a reusable 3-worker group, trained with reinforcement learning that gives intermediate rewards from probe-question utility and graph structure.
How much does MAGIC improve over prior methods?
In the paper's Table 1 it ranks first on all eight benchmarks; the largest gaps are MMLU-Pro accuracy 83.20 vs 73.60 for the best baseline (DyLAN) and LiveCodeBench-v6 pass@1 36.57 vs 26.86 for Qwen Flash. Several other wins are small (under 1 point), and all results come from single runs without statistical significance testing.
Is code or data available for MAGIC?
The paper text contains no code-release statement. It describes exact seeds, sampling indices, reward weights, and scorer commits, but states that exact role prompts and dataset profiles are separate reproducibility artifacts, and both executors (DeepSeek V4 Flash, Qwen Flash) are paid APIs, so exact numbers require API spend.
What is the theoretical guarantee in MAGIC?
Proposition 1 shows the potential-based reward shaping preserves the total task return: with matching discount and zero endpoint potentials, the sum of shaped rewards equals the terminal task score, so intermediate signals redistribute credit without changing the optimization objective (a standard result from Ng et al., 1999, applied to graph construction).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-5 are available only as text summaries with embedded numbers; visual details (axes, point placements) cannot be independently verified.
- Baseline numbers originate from the OpenMAS-GCom benchmark and cannot be checked against this paper's own execution records.
- Exact prompts and dataset profiles are declared external artifacts, so full end-to-end reproducibility cannot be confirmed from the text, though the appendix appears complete through D.3.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.