Testing Interchangeability in LLM Agent Teams
Jianxin Gao, Tianyi Yu, Linna Deng, Runze Li, Zining Wang
cs.AI · cs.MA
Quick verdict
Claims LLM agent swaps cost little score but 16–63% more communication; placebo-controlled design is sound, though n=4 swap pairs and thin CIs cap confidence.
Summary
The paper asks whether an LLM agent that has worked in a team is interchangeable with an equally experienced stranger. Eight teams per setting are built from one base model (GPT-5.6 Luna) with the same role prompts; each agent keeps a private notebook split into task notes and partner notes across ten formation episodes. Teams then trade role-matched agents, and performance is measured on held-out tasks against a placebo condition that reproduces the disruption of a roster change without changing personnel. Across Collab-Overcooked (low and high coupling) and Hanabi, swaps cost little task score but raise communication per unit of progress by 16%, 51%, and 63% respectively; the partner-specific residue ρ = Π/V reaches 0.18, 0.26, and 0.44, and in Hanabi a swapped agent's coordination cost slightly exceeds that of an inexperienced replacement (ρC = 1.11). Seat-targeted swaps show the loss concentrates in the agenda-setting seat and is paid mostly by the incumbent. Ablations over base model, decoding temperature, and formation length move ρ in step with protocol divergence between independently formed teams; greedy decoding halves the penalty. Main limitations: only four swap pairs serve as independent units, confidence intervals are correspondingly coarse, the headline Hanabi cost claim (1.04 vs 1.00 hints per point) has no reported uncertainty, message-coding procedures are unspecified, and no code or data release is stated.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Swapping a role-matched, equally experienced agent between LLM teams costs little task score but raises communication per unit of progress by 16%, 51%, and 63% in low-coupling Overcooked, high-coupling Overcooked, and Hanabi respectively.
- In Hanabi, a swapped agent is more coordination-expensive than an inexperienced replacement (ρC = 1.11), consistent with interference from conventions learned with a former partner.
- The partner-specific residue ρ grows with required coupling (0.18, 0.26, 0.44), concentrates in the agenda-setting seat, and is paid mostly by the agent that stayed.
- Across base-model, decoding-temperature and formation-length ablations, ρ co-moves with protocol divergence between independently formed teams; greedy decoding halves the swap penalty (ρ 0.27→0.14) and doubling formation history nearly triples Π (0.053→0.083).
Strengths
Placebo control cleanly isolates identity from disruption
Section 4.2 measures the announcement effect alone at 0.9 progress-completeness points (both Overcooked settings) and 4–6% extra coordination cost — the same order as Π at low coupling — so taking every contrast against Placebo rather than Intact prevents misattributing generic roster-change disruption to the specific partner.
Notebook decomposition enables mechanistic localization
Splitting each agent's 200-token notebook into task notes (κ) and partner notes (π) makes the partner-specific state manipulable: W (deleting an intact team's own partner notes) reproduces 81% (high coupling) and 64% (Hanabi) of the swap penalty (Table 2), and Σ shows clearing an arriving agent's stale partner notes helps (+0.016, +0.043 normalized).
Converging, well-triangulated evidence for the score/cost dissociation
Unusually honest self-calibration
Cost and scale transparency
Weaknesses
n=4 independent units for every swap contrast
Headline Hanabi cost claim lacks reported uncertainty
Coding methodology for Tables 3 and 4 is unspecified
The ρ–d correlation is fragile
Closed-model, no-release setup
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper tests a specific operational assumption behind production multi-agent LLM systems: that an agent filling a role is interchangeable with any equally competent agent. The instrument is a swap test borrowed from multi-agent RL cross-play diagnostics (Hu et al. 2020; Bard et al. 2020). Eight teams per setting are formed independently from one frozen base model with identical role prompts; each agent rewrites a 200-token notebook split into task notes (κ) and partner notes (π) across 10 formation episodes; teams then trade role-matched agents and are evaluated on held-out tasks under six conditions (Intact, Placebo, Swap, Swap-cleared, Amnesia, Naive), all independent forks from the same post-formation state. The Placebo — remove and reinstate under the same announcement — is the design's key move, isolating the identity effect from the generic disruption of a roster change. Headline claims: swaps cost little task score but raise communication per unit of progress 16–63%; in Hanabi a swapped agent costs more than an inexperienced one; the partner-specific residue ρ grows with coupling and team history and co-moves with protocol divergence.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of 'Testing Interchangeability in LLM Agent Teams'?
That replacing a member of an LLM agent team with an equally experienced stranger from another team barely hurts task scores but raises the communication needed per unit of progress by 16–63% in their three settings, with the penalty growing when tasks demand more coordination and when the team has a longer shared history.
What benchmarks and models does the paper use?
Collab-Overcooked (levels 2 and 6, from Sun et al. 2025) and the two-player Hanabi setting of the LLM-Coordination benchmark (Agashe et al. 2025). The main base model is GPT-5.6 Luna in its low-cost tier; an ablation adds Gemini 3.7 Flash and Claude Sonnet 5. No weights are trained — adaptation lives entirely in text notebooks agents rewrite after each episode.
What does ρ (rho) mean in this paper?
ρ is the ratio of the partner-specific penalty (placebo score minus swap score) to the value of experience (placebo score minus naive-replacement score). ρ = 0 means a team's history is fully portable and members are interchangeable; ρ = 1 means all gains from working together are tied to the specific pairing; values above 1 mean an experienced stranger is worse than a novice.
Did the authors release code or data for this study?
No code or data release is mentioned in the paper. The experiments run on closed commercial APIs and cost a few hundred dollars of API calls in total, which limits exact reproduction but the design (about 4,190 episodes, described in detail) is intended to be cheap to re-run.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Per-episode values underlying Figures 2 and 3 are not recoverable from text extraction; only quantities quoted in captions and body text were used for this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.