Scaling Discovery through Test-Time Communication
Jongho Park, Vasilis Kontonis, Shivam Garg, Akshay Krishnamurthy, Dimitris Papailiopoulos
cs.LG · cs.AI · cs.CL
Quick verdict
Claims test-time communication beats independent agents (team@k≈best@4k); striking ARC-AGI-3/packing/MNIST gains, but few runs and no statistical uncertainty.
Summary
The paper asks whether LLM agents that communicate during test-time search outperform the same number of independent agents. Introducing team@k — k identical agents sharing an append-only log, score records, and a workspace with no roles or orchestrator — against best@k under matched per-agent budgets, the authors run experiments on ARC-AGI-3 with Claude Sonnet 4.6, Frontier-CS polyomino packing with Sonnet/Opus 4.6, MNIST classifier compression with GPT-5.6 Sol, and Terminal-Bench 2.0. On ARC-AGI-3, team@3 matches best@13 and team@5 matches best@33 in solve rate (4.6% vs 4.7%; 8.0% vs 8.1%), team@5 solves LP85 at 65% where 64 solo trials failed, and the average team member matches the best independent agent on the RHAE efficiency metric. The packing team reaches 0.945 versus a 0.894 prior best; the MNIST team yields a 1,957-byte 99.4%-accuracy classifier versus a 2,461-byte human benchmark. Gains require an accessible verifier and sufficient compute: at low budgets a coordination tax dominates, and on Terminal-Bench 2.0 team@2 (60.67%) does not beat pass@2 (62.36%). Main limitations: no confidence intervals or significance tests anywhere, a single 96-hour MNIST run, and headline scores drawn from the best of very few team runs.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On ARC-AGI-3, a team of k communicating agents matches the solve rate of roughly 4k independent agents (team@3 ≈ best@13 at 4.6% vs 4.7%; team@5 ≈ best@33 at 8.0% vs 8.1%), and the multiplier grows from 4.3x to 6.6x with team size.
- Communication unlocks tasks single agents cannot solve: LP85 goes from 0/64 single-agent successes to a 65% team@5 solve rate, and FT09 from 9.4% (single) to 90% (team@3).
- Communicating teams set new best-known results: 0.945 on Frontier-CS polyomino packing (prior best 0.894) and a 1,957-byte MNIST classifier at 99.4% accuracy (prior human reference 2,461 bytes; best@4 3,160 bytes).
- Communication benefits require an agent-accessible verifier and sufficient compute; at low budgets a coordination tax dominates and on Terminal-Bench 2.0 team@2 (60.67%) fails to beat pass@2 (62.36%).
Strengths
Compute-controlled comparison design
team@k vs best@k under matched per-agent resources; best@k computed exactly from the 64-trial solo pools via combinatorics (Section 2.1); token-matched curves (Figure 3: matching needs 3.8x/4.9x more output tokens) and action-budget ablations (Figure 4: team with native budgets beats a single agent given 5x budget, collapses at 0.2x) directly address the 'teams just spend more' confound.
Selection-artifact control via RHAE
Figure 5 shows the average team@5 member reaches 8.9% RHAE versus 8.8% for the best of five independent agents, and the best team agent hits 13.6% versus 8.8% — evidence the gain is not merely picking the best of more attempts.
Verifiable state-of-the-art artifacts with mechanistic case studies
Honest negative results and boundary conditions
Weaknesses
No uncertainty quantification anywhere
Under-powered headline results
Result concentration on ARC-AGI-3
Harness effects not separated from communication
Cross-configuration and baseline comparability
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This empirical study examines multi-agent "test-time communication": k identical LLM agents share an append-only log, score records, and a filesystem workspace with no roles or orchestrator. The central comparison is team@k (k communicating agents) versus best@k (best of k independent agents) under matched per-agent resources. Four claims are tested: (1) on ARC-AGI-3, team@3 matches best@13 and team@5 matches best@33 in solve rate (Table 2: 4.6% vs 4.7%; 8.0% vs 8.1%), with the multiplier growing from 4.3x to 6.6x; (2) communication unlocks tasks no single agent solves (LP85: 0/64 solo trials vs 65% for team@5, Figure 2b); (3) new best-known results on Frontier-CS polyomino packing (0.945 vs prior 0.894) and MNIST classifier compression (1,957 bytes at 99.4% vs the 2,461-byte human reference); (4) gains require an accessible verifier and sufficient compute, framed as "verified progress sharing" with a pedagogical separation model (Proposition 1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What do team@k and best@k mean in this paper?
team@k means k identical LLM agents work the same task while communicating through a shared workspace (an append-only log, shared score records, and files), with no assigned roles. best@k means k independent agents run separately and only the best final result is kept. The paper compares the two under the same per-agent compute budget.
Did AI agents beat the best human result on MNIST classifier compression?
According to the paper, yes: a team of four GPT-5.6 Sol agents produced a 1,957-byte classifier with 99.4% test accuracy over a 96-hour run, about 20% smaller than the 2,461-byte best-known human solution (the authors' reproduction of the open-source tiny_MNIST model under the same artifact format). The best of four independent agent runs reached only 3,160 bytes.
Does multi-agent communication always help LLM agents?
No. The paper itself shows communication loses in two regimes: when compute is scarce (a coordination tax — teams trail independent agents below roughly 400K output tokens per agent on ARC-AGI-3) and when feedback cannot reliably rank intermediate progress (on Terminal-Bench 2.0, team@2 scored 60.67% versus 62.36% for two independent attempts with oracle selection).
Is code available for this paper?
Yes. The paper links a repository (github.com/jerryjonghopark/test-time-communication) containing the tasks, communication prompts, and protocol. Exact replication nonetheless requires specific commercial model versions (Claude Sonnet/Opus 4.6, GPT-5.6 Sol via GitHub Copilot CLI) and multi-day compute.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures were parsed from embedded caption text only; curve shapes (Figures 3, 4, 6-10) could not be visually verified, though key endpoint numbers appear in captions and body text.
- Review is based on the arXiv v1 extracted text; the released repository's contents were not inspected.
- Model names (Sonnet 4.6, Opus 4.6, GPT-5.6 Sol, GPT-6 Astra) are as printed in the extraction and were not externally verifiable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.