Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learning
Oussama Hidaoui, Omer Ebead, Ulrich Armel Mbou Sob, Siddarth Singh, Juan Claude Formanek, Felix Chalumeau, Omayma Mahjoub, Sasha Abramowitz, et al.
cs.LG · cs.AI
Quick verdict
Claims task diversity—not dataset size—drives zero-shot generalisation in offline MARL (3.2x avg); solid large-scale evidence, but ratio metrics and checkpoint-selection ambiguity warrant scrutiny.
Summary
This InstaDeep/AIMS preprint studies zero-shot generalisation in offline multi-agent RL. The authors adapt three sequence models (Oryx, CQL-Sable, BC-Sable) to multi-task training via dynamic agent padding/shuffling/masking, task-balanced batching, and HL-Gauss value classification, all on a shared Sable backbone without task IDs. Using four JAX environments (LBF, Connector, RWARE, SMAX) with 28 train and 34 test task configurations, they find test performance improves near-monotonically with training-task count: a mean 3.2x gain over single-task models (5.4x RWARE, 2.9x Connector, 3.2x SMAX, 1.3x LBF), while enlarging datasets of fixed task diversity (RWARE, 32M-166M transitions) yields little test gain. Scaling models from 116k to 13.6M parameters improves both train and test performance, contrasting with prior single-task findings. A coverage-smoothness regret bound formalises the diversity claim, with an empirical proxy-coverage surrogate. Against Mediratta et al., the best offline method beats behaviour cloning per environment on mixed-quality replay data. Limitations: centralised architectures only, three seeds, no significance tests, 'maximum' ratio gains, and dataset release deferred until publication.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Scaling training-task diversity, not raw dataset size, is the dominant driver of zero-shot test performance in offline MARL sequence models (Figure 4 vs Figure 5a).
- Multi-task training yields a mean 3.2x test-task improvement over single-task models, per-environment maxima of 5.4x (RWARE), 2.9x (Connector), 3.2x (SMAX), 1.3x (LBF) (Figure 1).
- Increasing model capacity (116k to 13.6M parameters on RWARE) improves both training and held-out test performance, unlike in the single-task regime of Formanek et al. (2025) (Figure 5b).
- The regret of a multi-task offline policy is bounded by training error plus a coverage term, formally grounding the task-diversity finding (Theorem 4.4).
- On mixed-quality replay data, the best offline method per environment outperforms behaviour cloning, in contrast to Mediratta et al. (2024) (Appendix C, Table 2).
Strengths
Large, well-structured benchmark for an underexplored problem
Section 3.1 and Appendix H define 28 train and 34 test task splits across LBF, Connector, RWARE and SMAX with documented distributional shift (Figure 3); train/test partitions are fully enumerated in Table 3, an unusually high level of experimental transparency.
Convergent evidence across three algorithms and four environments
Figure 4 shows test-task performance improving near-monotonically with training-task count for MT Oryx, MT CQL-Sable and MT BC-Sable in all four environments, and Figure 5a separates the data-size effect (RWARE 32M-166M transitions: train up, test flat) from the task-diversity effect.
Honest, well-matched theory
Design choices are ablated, not just asserted
Careful treatment of the BC baseline
Weaknesses
Headline gains are 'maximum' near-zero-denominator ratios
Possible test-set selection
Overgeneralised BC comparison in the introduction
Thin statistics and thin theory
Environment-scoped hypotheses verified in the wrong place
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Review: Out-of-Distribution Generalisation with Sequence Models in Offline MARL (arXiv 2609.03667)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper on offline multi-agent RL generalisation?
That increasing the diversity of training tasks - not the sheer number of training transitions - is the main driver of zero-shot performance on unseen tasks. Across four environments (LBF, Connector, RWARE, SMAX), multi-task-trained sequence models improved held-out test performance by about 3.2x on average versus single-task models (up to 5.4x on RWARE), while enlarging a fixed-task dataset (32M to 166M transitions on RWARE) barely moved test performance.
How does this paper disagree with 'The Generalization Gap in Offline RL' (Mediratta et al., 2024)?
Mediratta et al. found, in single-agent settings with expert data, that simple behaviour cloning beats offline RL methods under task shift. This paper trains on mixed-quality replay data instead of expert data and finds the best offline method per environment (Oryx or CQL-Sable) beats behaviour cloning in all four environments - e.g. CQL-Sable reaches 0.620 vs BC's 0.415 normalised return on RWARE - though BC still beats CQL-Sable on LBF and Connector individually.
Is code and data available for this paper?
The paper states anonymised code is available now via its project site (sites.google.com/view/multi-task-marl), and that all code and datasets (collected from online Sable agents in Jumanji and JaxMARL environments) will be uploaded to a public HuggingFace repository upon publication - so the datasets are not yet public at preprint stage.
Does bigger model size help generalisation to unseen tasks?
Yes, in this paper's setting. On RWARE, scaling the Sable backbone from 116k to 13.6M parameters (embedding dimension 64 to 768) improved both training and held-out test performance - contrasting with the single-task result of Formanek et al. (2025), where an embedding dimension of just 64 was optimal. On easier environments (LBF, Connector, SMAX), gains saturate at much smaller model sizes.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure data points are not recoverable from the text extraction (only axis tick labels survive), so curve-level judgements rely on the paper's own narrative; per-point values were not invented.
- Table 2 is present in the extraction with numeric means and standard errors, but the bold 'largest mean' markers noted in its caption are not visible in the text layer.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.