Agensh: Scaling Organizational Intelligence to 1,024 Agents
Zhihao Zhan, Ting Song, Li Dong, Li Dong, Shaohan Huang, Shaohan Huang, Jianxun Lian, Yan Xia, et al.
cs.CL · cs.MA
Quick verdict
Reports orchestrator-free harness scaling to 1,024 LLM agents; +49% rel. gain at 128 agents on ProgramBench, but no variance, cost or orchestrator baseline.
Summary
This Microsoft Research technical report introduces Agensh, a multi-agent harness that removes the central orchestrator. Workers run a five-step asynchronous cooperation loop — gather context, claim sub-tasks, take action, verify results, merge progress — over three infrastructure components: a Git-based shared workspace, a Mattermost-style message interface, and a DeLM-inspired append-only shared-context board (OBSERVED/FACT/FAIL/CLAIM/PATCH_SUMMARY entries with grep over full history). Evaluation uses the five hardest ProgramBench tasks (FFmpeg, gromacs, pandoc, PHP-src, ctags; up to ~2.8M lines of code) with GPT-5.6-sol (high) on Copilot, 6-hour budget, no Internet. Scaling 1→8→32→128 agents raises mean final test-pass rate from 19.31% to 20.68%/26.52%/28.78% (~49% relative gain); larger organizations also reach given scores earlier (e.g., 128 agents exceed 30% on pandoc at 30 minutes). On pandoc only, 1,024 agents reach 55.06% versus 33.89% single-agent. Trajectory case studies suggest emergent specialization, review relationships, and standardized merge protocols at larger scales. Main limitations: apparent single runs with no variance estimates or significance tests; no orchestrator-based baseline at matched scale to support the central anti-orchestrator claim; no cost/token accounting for the 1,024-agent runs; component ablations (workspace vs. messages vs. shared context) absent; and emergent-cooperation evidence is anecdotal.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Scaling from 1 to 128 agents raises the mean final test-pass rate on the five hardest ProgramBench tasks from 19.31% to 28.78% (~49% relative) with model and harness fixed (§3.1, Fig. 5).
- Larger organizations reach a given performance level earlier; on pandoc, 128 agents exceed 30% test-pass rate at the 30-minute checkpoint vs 60/90 minutes for 32/8 agents (§3.1, Fig. 6).
- On pandoc only, scaling to 1,024 agents raises the final test-pass rate to 55.06%, from 33.89% (1 agent) and 50.94% (128 agents) (§3.1, Fig. 1).
- Distinct forms of self-organized cooperation — peer coordination, multi-worker integration, workflow standardization, role specialization — emerge progressively at 8/32/128/1,024 agents (§3.2).
- Agent count is a new scaling dimension for multi-agent organizations, complementing model and harness scaling (§1, §5).
Strengths
Unprecedented demonstrated scale
1,024 concurrent LLM agents (16 nodes × 64 agents, Appendix C) on a real long-horizon rebuild task (pandoc), far beyond prior public demonstrations such as Carlini's 16-agent Git coordination cited in §4.
Transparent system specification
§2 and Appendices A/B publish the verbatim worker prompt, event-driven router runtime, mid-turn delivery semantics via MCP tool-return wrappers, entry caps and idle detectors — unusually inspectable for a harness paper.
Consistent scaling trend plus latency benefit
Evaluation hygiene against cheating
Concrete emergence taxonomy
Weaknesses
No variance or statistical testing
Central anti-orchestrator claim is untested against the obvious baseline
No cost accounting
Missing component ablations
Anecdotal emergence evidence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Agensh (Microsoft Research; technical report, arXiv:2609.26781) is a multi-agent harness that removes the central orchestrator. Workers execute a five-step asynchronous cooperation loop — gather context, claim sub-task, take action, verify, merge progress — over three infrastructure components (§2.2): a Git-based shared workspace (Gitea), a message interface (Mattermost, with interrupting direct messages), and a DeLM-inspired append-only shared-context board carrying OBSERVED/FACT/FAIL/CLAIM/PATCH_SUMMARY entries with grep over full history. The paper claims: (1) agent count is a new scaling dimension; (2) scaling 1→128 agents lifts mean final test-pass rate on the five hardest ProgramBench tasks (FFmpeg, gromacs, pandoc, PHP-src, ctags — up to 2,812,757 LOC for PHP-src, Table 1) from 19.31% to 28.78% (~49% relative) with GPT-5.6-sol (high) on Copilot under a 6-hour offline budget; (3) on pandoc alone, 1,024 agents reach 55.06% vs 33.89% for one agent and 50.94% for 128; (4) cooperation forms (peer coordination → integration management → standardized workflows → specialized roles) emerge progressively with scale (§3.2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Agensh?
Agensh is a multi-agent harness from Microsoft Research that runs LLM coding agents without a central orchestrator. Every worker repeats a five-step loop (gather context, claim a sub-task, act, verify, merge progress) and coordinates through three shared components: a Git-based workspace, a messaging interface, and an append-only shared-context board. Code is published at github.com/microsoft/Agensh.
How many agents does Agensh scale to, and how much does it improve results?
The paper demonstrates up to 1,024 concurrent agents. On the five hardest ProgramBench software-rebuild tasks (6-hour offline budget, GPT-5.6-sol high reasoning), the mean final test-pass rate rose from 19.31% with 1 agent to 28.78% with 128 agents — about a 49% relative gain. On pandoc only, 1,024 agents reached 55.06% versus 33.89% for a single agent. These figures come from what appear to be single runs, with no variance estimates reported.
Do more agents always help in multi-agent LLM systems?
According to this paper, the trend is positive across 1/8/32/128 agents, but increments vary (+1.37, +5.84, +2.26 points), so gains are not uniform and returns diminish beyond 32 agents. The beyond-128 scaling was tested on only one task. Because the paper reports no repeated-run variance, it remains unknown how reliable each increment is; other research on scaling agent systems suggests benefits are task- and architecture-dependent.
Can I reproduce the Agensh results?
Partially. The authors release code (github.com/microsoft/Agensh), the project page (aka.ms/Agensh), the exact worker prompts and run protocols. However, exact replication requires access to GPT-5.6-sol (high) via a Copilot-based harness and, at 1,024 agents, a 16-node deployment — likely very expensive. The ProgramBench tasks and official evaluator are public, so smaller-scale reruns with open models are the practical route.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1, 5 and 6 could not be inspected from the extracted text; scaling-curve shapes and any plotted uncertainty are unverifiable.
- Extraction omitted portions of §2.3 (implementation), §4 (related work) and Appendix B, so some implementation details may be missed.
- The single-run assessment rests on the absence of any variance/repeat reporting in the extracted text, not on an explicit statement by the authors.
- The paper is an arXiv technical report; no peer-review signals are available.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.