Exploring Collaboration between a language and a non-language agent
Harini S, Somesh Singh, Yaman K Singla, Rajiv Ratn Shah, David Doermann, Balaji Krishnamurthy
cs.CL · cs.AI
Quick verdict
Claims internalizing engine latent state beats verbalized tool use; striking results, but judge-as-reward circularity and chess-only scope temper them.
Summary
The paper asks whether converting a non-language agent's internal state into text is a bottleneck for LLM-agent collaboration. Using chess, the authors introduce LLAMIA-Bench (six tasks: Elo-conditioned behavior cloning with OOD splits, puzzle difficulty/interest estimation, move annotation, and full-game commentary, plus a new Agadmator-2K dataset) and latent state internalization: 32 continuous tokens projected from the frozen Lc0-BT4 engine's penultimate layer into the LLM's token stream by a three-layer MLP, trained via supervised alignment then joint DAPO reinforcement learning. Against a training-matched verbalized control (LLAMIA-Verb) sharing backbone, engine, prompts, and recipe, LLAMIA-14B wins on all six tasks (puzzle-interest Spearman rho 0.52 vs 0.08; commentary G-eval 0.75 vs 0.40), the gap widens during training and persists from 4B to 14B. LLAMIA-14B approaches dedicated experts (53 vs Allie-Adaptive-Search 55 on in-distribution move-match) and exceeds frontier models with engine tool access; human studies (n=12) show bot-detection below chance (39%) versus 72% for LLAMIA-Verb and 72.2% commentary preference. A Go pilot with KataGo suggests transfer. Limitations: G-eval is both the RL reward and headline metric for commentary/annotation (BLEU-2 mitigates), several dataset sizes conflict across sections, and cross-table numbers disagree for three columns.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Verbalizing a non-language agent's state into text is a structural bottleneck: the internalization-verbalization gap widens during DAPO training and persists from 4B to 14B (Section 3.4, Table 18).
- A single 14B model with latent state internalization leads all six LLAMIA-Bench tasks, exceeding frontier models with engine tool access (GPT-5+Lc0: interest ρ 0.10 vs 0.52, difficulty 0.48 vs 0.71).
- The integration interface shapes learned collaboration: LLAMIA adopts task-adaptive strategies (65% engine-follow on gameplay, 40% counterfactual query on commentary) while LLAMIA-Verb collapses to engine-follow everywhere (62-76%, Figure 4).
- Internalization yields human-perceptible behavior: 39% bot detection (below chance) vs 72% for the verbalized control, and 72.2% preference for LLAMIA commentary (n=12, Section 3.7).
- The recipe transfers to Go with a frozen KataGo agent, matching rank-calibrated KataGo-HumanSL (48/50) and beating the verbalized control by ~10 points (Table 25).
Strengths
Single-variable control design
LLAMIA and LLAMIA-Verb share backbone, frozen Lc0-BT4 subagent, prompts, reward and DAPO hyperparameters (Section E.1); the Table 1 ladder (LLM-Only, LLM-ChessCLIP raw encoder, untrained tool use, latent-SFT) plus shuffled-token ablation (Table 23, monotone degradation toward LLAMIA-Verb) rules out capacity, token-count, and RL explanations for the gains.
Breadth and depth of evaluation
Six tasks with OOD splits (GM-25, Low-Time, ∆Elo), a 4B/8B/14B scale sweep (Table 18), five-engine agent ablation across three architectures (E.2), per-layer linear probes validating the layer-14 extraction point (Table 24), and a Go transfer pilot (Table 25).
Cleanest diagnostic task design
Honest cost accounting
Carefully screened human studies
Weaknesses
Metric–reward circularity for language-heavy tasks
Cross-table numerical inconsistencies
Dataset reporting conflicts
Abstract overclaims vs specialists
Chess-bound evidence for a general claim
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a real gap: LLM orchestrators collaborate with subagents only through text, yet the strongest domain agents (game engines, robot policies) are not language models. Four contributions are claimed: (1) latent state internalization — injecting k=32 projected tokens of a frozen engine's penultimate activations into the LLM's token stream; (2) LLAMIA, a Qwen3 backbone trained in two stages (projector alignment on 5M state-policy pairs, then joint DAPO RL) that 'matches or exceeds task specialists and frontier models including GPT-5.1'; (3) the first quantification of a 'verbalization debt' via LLAMIA-Verb, a control identical except for the integration channel; (4) LLAMIA-Bench, six chess tasks plus the new Agadmator-2K commentary dataset. The abstract's strongest wording needs qualification (see Risks).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the 'verbalization debt' in the LLAMIA paper?
It is the measured performance gap between two identically trained systems that differ only in how the chess engine's state reaches the LLM: as 32 projected latent tokens (LLAMIA) or as text summaries (LLAMIA-Verb). The paper reports the debt on all six tasks — e.g., puzzle-interest correlation 0.52 vs 0.08 and commentary G-eval 0.75 vs 0.40 at 14B — and shows it widens during RL training and persists from 4B to 14B parameters.
Is LLAMIA's code and data available?
The paper points to a project page (behavior-in-the-wild.github.io/llamia.html) for code, data and examples, including the new Agadmator-2K commentary dataset built from ~1,900 YouTube games. Full retraining is compute-heavy: RL used 32 A100 GPUs, though the appendix documents hyperparameters, prompts and the tool API in detail.
Does the latent-state approach work outside chess?
Only a preliminary Go experiment is reported: with frozen KataGo as the subagent, LLAMIA-Go-14B reaches 48/50 top-1 human move-match at rank 5k using 8k training positions, matching the rank-calibrated KataGo-HumanSL expert and beating the verbalized control by ~10 points. A full multi-task Go suite is left as future work, so cross-domain generality is suggested, not established.
How does LLAMIA compare to GPT-5.1 with engine tools?
When both access the same Lc0 engine as a verbalized tool, LLAMIA-14B leads on every LLAMIA-Bench task — e.g., puzzle difficulty ρ 0.71 vs 0.48 and interest ρ 0.52 vs 0.10 (Table 18) — and an 8B LLAMIA already beats GPT-5+Lc0 on all six tasks, so the advantage does not require the largest backbone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: early per-task main-text results (roughly §3.1-3.3), parts of Appendix I, and portions of the references fall inside '[...sampling gap...]' regions; all numbers cited in this review come from visible tables and sections.
- Table 17's GPT-5+Lc0 'Solved (%)' cell renders as 'XX.X' in the extraction — likely an artifact; treated as unreliable rather than a paper claim.
- Figures are evidenced only via captions and in-text numbers (e.g., Figure 3 curves, Figure 5 attention map); no figure imagery was inspected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.