Learning to Coach for Experiential Learning
Guanheng Chen, Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, Furu Wei
cs.CL
Quick verdict
L2C trains a coach model to distill a frozen actor's trajectory into guidance, with large text-game gains but thin, uncertified math margins and no confidence intervals.
Summary
Learning to Coach (L2C) trains a separate LLM-as-a-Coach to convert a frozen actor's own solution trajectory into a short natural-language note — experiential knowledge — which the actor then conditions on for a guided retry; only the coach is updated, so the actor keeps its original weights and general capabilities. Instead of human-written critiques, the coach is optimised with GRPO against a verifiable binary reward: whether the actor's guided response is correct. Two reward variants are studied, a same-instance reward scored on the source problem and a cross-instance reward scored on a disjoint probe set (eight problems for math, seven seeds for games). Eight candidate notes per trajectory form one GRPO advantage group; AdamW at a constant learning rate of 1e-6, 100 to 200 steps. On DAPO-Math-17K with Qwen3-1.7B/4B/8B and on TextArena's FrozenLake and Sokoban, L2C beats the base actor, Self-Refinement and an untrained coach: FrozenLake rises from 7.4 (base) and 23.4 (untrained coach) to 65.4 at K=2 and 73.0 at K=10; math improves more modestly, 66.1 to 67.8 (1.7B), 74.4 to 78.6 (4B) and 81.9 to 82.9 (8B). Ten coaching iterations beat doubling the decoding budget (Qwen3-4B: 82.6 versus 77.0 at 32k tokens). Cross-instance rewards yield transferable game rules but not transferable math knowledge. Main limits: no confidence intervals anywhere, a single model family, and out-of-distribution gains of only +1.5 to +2.9 points.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Training a dedicated LLM-as-a-Coach with GRPO on the correctness of a frozen actor's guided response improves downstream performance without modifying the actor.
- L2C outperforms the base actor, Self-Refinement and an untrained LLM-as-a-Coach on math and on interactive text-games, with the largest gains where trajectories expose latent environment rules (FrozenLake 7.4 to 65.4, Sokoban 3.8 to 23.6).
- The cross-instance reward elicits transferable, environment-general knowledge on text-games, whereas the same-instance reward elicits memorised, instance-specific notes.
- Scaling experiential-learning iterations is more effective test-time scaling than enlarging the actor's decoding budget: L2C at K=10 reaches 82.6 on DAPO for Qwen3-4B versus 77.0 with a 32k decoding budget.
- The trained coach produces progressively shorter experience snippets and actor responses, and transfers to out-of-distribution tasks including AIME 2026, text-games and IFEval.
Strengths
A clean division of labour that preserves the actor
Freezing pi_actor and training only pi_theta (Eq. 1-3, §2) is a real design idea, not a rebranding: the actor keeps its general capabilities and weights, the extracted knowledge is explicit and inspectable, and the same coach can in principle be paired with different actors. Appendix E actually tests this by fixing the actor at Qwen3-4B and varying the coach over 1.7B/4B/8B — and finds coach size barely matters, which supports the practical claim that a small coach can serve a large actor.
Two reward variants that produce qualitatively different, inspectable knowledge
rsame (Eq. 4) versus rcross (Eq. 5) is a well-constructed contrast, and Appendix A.4 shows the outputs really differ: the same-instance FrozenLake note (Fig. 13) memorises exact hole coordinates at (0,2)/(1,2) and a step-by-step path, while the cross-instance experience (Fig. 14) abstracts into grid rules and a down-then-right strategy. That is behavioural evidence the reward is doing what the authors claim, not just moving a number.
Gains are large and consistent where the task has latent structure
Honest negative result on where transfer does not work
Substantial reproducibility surface
Weaknesses
No uncertainty quantification anywhere in the paper
On math, most of the gain comes from the coaching pipeline, not from L2C training
The compute comparison in Table 2 is not a like-for-like accounting
Baselines are narrow and one of them is set up to lose
Key design choices are reported but never ablated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Learning to Coach (L2C) proposes a division of labour for experiential learning: keep the task-performing model frozen and train a second model, the LLM-as-a-Coach, to read the actor's trajectory and write a short natural-language note that the actor conditions on for a retry. The stated motivation (§1) is that raw trajectories are long, redundant and sometimes wrong, so re-feeding them — as self-refinement does — does not distinguish useful evidence from incorrect conclusions, while fine-tuning the actor is expensive and degrades general capability.
The paper makes five claims: the coach can be trained from verifiable rewards with no human critiques; L2C beats base, self-refinement and an untrained coach on math and text-games; the cross-instance reward elicits transferable knowledge; iteration is better test-time scaling than a larger decoding budget; and the trained coach is compact, actor-adapted and transfers out of distribution. Scope is three tasks (DAPO-Math-17K, FrozenLake, Sokoban from TextArena) and one model family (Qwen3-1.7B/4B/8B), with the coach always initialised from the same checkpoint as its actor.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Learning to Coach (L2C) and how does it differ from self-refinement?
L2C trains a separate language model, called an LLM-as-a-Coach, to read a frozen model's previous attempt and write a short note summarising what went wrong, which intermediate results to keep, and what to try next. The actor then retries with that note as guidance. Unlike self-refinement, where the model simply re-reads its own full previous answer, the coach compresses the trajectory into a compact, actionable snippet, and unlike fine-tuning, the actor's weights are never changed. The coach is trained with reinforcement learning so that its notes maximise the chance the actor's guided answer is correct.
How much does L2C actually improve accuracy?
It depends heavily on the task. On interactive text-games the gains are large: on FrozenLake with Qwen3-1.7B, accuracy rises from 7.4 percent for the base model and 23.4 percent with an untrained coach to 65.4 percent with L2C, and to 73.0 percent after ten iterations. On mathematics the gains are much smaller: Qwen3-1.7B goes from 66.1 to 67.8, Qwen3-4B from 74.4 to 78.6, and Qwen3-8B from 81.9 to 82.9. The paper reports no confidence intervals, so the math margins should be treated as provisional.
Does the experience learned by the coach transfer to other problems?
Only when problems share hidden structure. The paper's cross-instance reward produces general game rules that transfer across different text-game maps, and it beats the same-instance reward on both FrozenLake and Sokoban. On mathematics it fails: knowledge extracted from one problem applied to 250 other problems gives 51.3 percent for Qwen3-1.7B, well below the 64.4 percent the actor achieves with no experience at all. A fully specified word problem leaves little latent structure to reuse.
Is iterative coaching a better use of inference compute than simply letting the model think longer?
The paper argues yes. On DAPO math, giving Qwen3-4B a 32k decoding budget yields 77.0 percent, while two coaching iterations give 78.7 percent and ten iterations give 82.6 percent. Accuracy also rises while responses get shorter, because refined notes let the actor skip dead ends. One caveat: the comparison reports accuracy but not total compute, and each coaching iteration runs a second model that can generate thousands of tokens of notes.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1, 3, 4, 5, 6, 15 and 16 are not readable from the extracted text; all numbers cited come from Tables 1-6 and the body text, and the Figure 6 IFEval values are not tabulated anywhere.
- Appendix C, D and E results are reported in text and tables as extracted; per-checkpoint curve values are unavailable.
- Training compute, wall-clock cost and GRPO hyperparameters such as the KL coefficient and clipping range are not stated in the text, so cost comparisons cannot be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.