Assembling the CREW: A Collaborative Multi-agent Reinforcement Learning Framework for Automated Related Work Generation
Hai-Dang Dang, Bao-Yen Pham, Bao Nguyen, Tran Thi Huong, Huynh Thi Thanh Binh
cs.LG
Quick verdict
CREW trains LLM agents via IPPO to coordinate related-work writing; citation gains strong but quality margins (+0.06) marginal, no significance tests.
Summary
The paper proposes CREW, a multi-agent reinforcement learning framework for automatic related work generation. Instead of a fixed pipeline where each LLM agent owns one step, CREW formulates the task as a Dec-POMDP in which homogeneous agents independently choose among four actions — Retrieve, Disseminate, Compose, Critique — guided by a single IPPO-trained policy over compact 20-dimensional observations, with delta-based rewards derived from heuristic quality measures aligned to the gold section. On OARelatedWork (1,829 training / 366 evaluation papers, Qwen 2.5 72B backbone), CREW reaches 3.82 overall LLM-judge score versus 3.76 for SRW and 3.80 for an architecture-matched centralized Orchestrator, while citation verification jumps to 99.33% from SRW's 80.40%; tokens fall modestly versus SRW (198.74K vs 210.37K) though inference is slower (755s vs 585s). The learned controller transfers across backbones (LLaMA 8B: 3.04→3.57) and to Multi-News without retraining. Ablations show the Critique action and social-activity observations are critical. Main limitations: no statistical significance or variance reporting for margins as small as 0.02–0.06 Likert points, an unexplained 12.90% token-reduction claim that does not match Table 1, full-system numbers differing between Tables 1 and 5, a 20-paper human study, and evaluation confined to LLM-as-a-judge plus citation-ID existence checks.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Replacing fixed multi-agent pipelines with an IPPO-learned decentralized action-selection policy improves RWG quality (3.82 overall vs SRW 3.76 and architecture-matched Orchestrator 3.80).
- CREW dramatically improves citation grounding over baselines (99.33% verified citations vs 80.40% for SRW and 48.61% for GPT-4o long-context).
- The Disseminate and Critique collaborative actions are essential: removing Critique collapses overall quality to 2.58 and verification to 10.00% (Table 5).
- The learned coordination policy transfers across LLM backbones (LLaMA 8B 3.04→3.57) and to Multi-News summarization without controller retraining.
- CREW reduces token usage relative to SRW and avoids the centralized coordinator as a single point of failure (812s vs 1012s under 5% API failure).
Strengths
Well-designed controls isolate the contribution of learned coordination
The Orchestrator baseline (§4.2) inherits CREW's shared workspace, local knowledge, and action space but swaps the learned policy for a centralized LLM coordinator; Table 2 further shows a naive HuggingGPT-style orchestrator scores only 3.40 versus 3.80 for the architecture-aware one, cleanly decomposing structure from policy.
Large, practical citation-grounding gain
Table 1: CREW's citation verification reaches 99.33% versus 80.40% for SRW (same Qwen 2.5 72B backbone) and 48.61% for GPT-4o long-context — directly addressing hallucinated references, the core reliability failure of RWG systems.
Evidence the learned controller is reusable
Informative ablations tied to failure modes
Compact, scalable formulation
Weaknesses
Headline margins are within plausible noise; no statistical treatment
Internal numeric inconsistencies
Efficiency claims contradict Table 1
Under-analyzed scalability and default choice
Evaluation breadth and metric depth
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CREW (arXiv:2609.15721) targets section-level abstractive Related Work Generation (RWG) over full texts. The authors argue that prior multi-agent LLM systems (e.g., SRW, Liu et al. 2025) hard-wire agent roles into sequential pipelines, and propose instead a Dec-POMDP in which M homogeneous agents independently choose among four actions — Retrieve, Disseminate, Compose, Critique — under a single IPPO-trained policy with parameter sharing. Three claims: (1) two new collaborative actions (Disseminate, Critique) improve coordination and draft quality; (2) learned decentralized action selection beats fixed pipelines; (3) the learned controller transfers across LLM backbones and to Multi-News summarization without retraining.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CREW and what problem does it solve?
CREW is a multi-agent reinforcement learning framework for automatically writing the Related Work section of a research paper. Instead of a fixed pipeline where each LLM agent has a permanent role, CREW lets agents autonomously choose among four actions — Retrieve (read a paper), Disseminate (share knowledge), Compose (draft), and Critique (review) — using a policy trained with Independent PPO on the OARelatedWork dataset.
How much better is CREW than previous methods?
On OARelatedWork with Qwen 2.5 72B, CREW scored 3.82 out of 5 from LLM judges versus 3.76 for the prior multi-agent system SRW and 3.80 for a centralized orchestrator — a small margin. Its clearest advantage is citation reliability: 99.33% of generated citations exist in the allowed reference pool versus 80.40% for SRW and 48.61% for GPT-4o with long context. It uses slightly fewer tokens than SRW but takes longer (755s vs 585s).
Does the trained policy work with other LLMs?
Yes. The controller trained with Qwen 2.5 72B was reused without retraining when the executing backbone was swapped to LLaMA 3.1 8B (overall score rose from 3.04 to 3.57), Gemini 2.0 Flash (3.28 to 3.71), and GPT-4o (3.82), and it also transferred to Multi-News multi-document summarization with only prompt changes.
Is the code and data available?
The paper links a public GitHub repository (YenPBao/CREW-Collaborative-MARL) and evaluates on the public OARelatedWork dataset (91,445 papers), using 1,829 papers for policy training and 366 for evaluation; hyperparameters and all action prompts are listed in the appendices. Exact reproduction depends on commercial LLM APIs (OpenRouter) with pinned model snapshots.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Full-text extraction truncated several blocks (start of §3.1, parts of §4.3–4.4 including raw Table 3/4 cells, and parts of Appendices D.2/F.2); judgments rely on prose-reported numbers where table cells were omitted.
- Figures 1–5 are not analyzable from the extracted text; Figure 2 failure descriptions come from the caption and body prose only.
- Equations were partially garbled in extraction (e.g., the piecewise definition in Eq. 4); feature interpretations follow the surrounding prose.
- Model snapshot identifiers (e.g., qwen-2.5-72b-september2024) are as printed; availability drift on OpenRouter may affect reproduction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.