Experience Funnel: A State-Policy Alternating Loop for Self-Evolving Agents
Wenbo Gao, Zhaomou Song, Zhiyuan Ji, Renxi Liu, Xing Li, Xianzhi Yu, Xiaoguang Li, James Chung-wai Cheung, et al.
cs.CL
Quick verdict
Claims state–policy alternation beats state-only/internalization for self-evolving agents; gains small (≤1.4 avg pts), single-run, untested statistically.
Summary
The paper proposes Experience Funnel, a framework in which a self-evolving LLM agent alternates between two adaptation timescales. Trajectories collected by the current state–policy pair are aggregated via cross-trajectory reflection into candidate edits of an explicit textual state, accepted only when they improve held-out validation. A Transition-Aware Skill Distillation stage then compares state-free, previous-state, and updated-state rollouts, classifying trajectories as newly useful, persistently useful, regressive, or inactive; only the first two are distilled, weighted by token-level Jensen–Shannon divergence between state-conditioned and state-free predictions, combined with state-free reinforcement learning. On SearchQA with a Qwen3.5-4B policy and a 27B teacher, state-free accuracy rises from 58.1% to 61.3% and the full pair reaches 62.4%; across SearchQA, ALFWorld, and WebShop the method averages 57.6 versus 56.2 for SkillRL, the strongest co-evolution baseline. An ablation puts the full loop at 63.6% versus 62.8% policy-only. Limitations: nearly all results are single runs with 0.8–2.2-point margins and no statistical tests; the promised repeated-run uncertainty never appears; several numbers are internally inconsistent (63.6 vs 62.4; 56.9 vs 58.1); the conclusion cites coding experiments absent from the paper; and hyperparameters and code are not released.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 3
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Alternating state evolution and selective policy consolidation beats state-only evolution, policy internalization, and prior co-evolution baselines, averaging 57.6 across SearchQA, ALFWorld, and WebShop versus 56.2 for SkillRL.
- Consolidation progressively converts explicit experience into autonomous competence: state-free SearchQA accuracy rises from 58.1% to 61.3% across accepted rounds, and the full state–policy pair reaches 62.4%.
- Selecting trajectories by cross-version utility transitions (newly useful plus persistently useful) outperforms unfiltered consolidation, 63.0% versus 58.9% ACC after the second consolidation round.
- A larger 27B teacher improves the student only when conditioned on the evolved state; without state conditioning it degrades performance below the base policy (56.29% vs 58.14%).
- The benefit of explicit state becomes increasingly tied to its co-evolved policy: the co-evolved pair's advantage over the evolved-state-plus-initial-policy control widens from 0 to 2.2 points.
Strengths
Clean, well-specified two-level selection mechanism
Trajectory-level transitions over state-free/previous-state/updated-state rollouts (Eqs. 2–4) plus token-level JSD attribution (Eqs. 5–7) are directly ablated: selected (01+11) consolidation reaches 63.0 ACC vs 58.9 unfiltered after round 2 (§3.4.2).
Informative teacher-capacity control
Table 4 shows a 27B teacher without state conditioning degrades the student to 56.29, below the 58.14 base, while 27B+state yields 61.21 — ruling out generic big-teacher distillation as the source of gains.
Candor on selection semantics and budget limits
Strict data protocol with validation gates
Compatibility analysis tests the core thesis directly
Weaknesses
Statistically unsupported margins
Internal numeric inconsistencies
Baseline comparability unverified
Reproducibility gaps
Null residual-state result narrated as success
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a method paper on self-evolving LLM agents (Huawei + PolyU + RUC). The authors argue that explicit textual states (skills, harnesses) adapt fast but inflate context, while parametric policies are compact but slow to update, and propose *Experience Funnel* (also called the State–Policy Consolidation Loop): an alternating cycle in which trajectories are distilled into a validated textual state (§2.1), then selectively consolidated into the policy via Transition-Aware Skill Distillation (§2.2), after which the updated pair collects fresh trajectories. Central claims: (i) the loop beats state-only, policy-internalization, and prior co-evolution baselines (Table 1: 57.6 average vs 56.2 for SkillRL); (ii) consolidation progressively internalizes experience (state-free SearchQA accuracy 58.1→61.3, Table 5); (iii) selection by cross-version utility transitions is the key mechanism (§3.4.2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Experience Funnel framework?
It is a training loop for self-evolving LLM agents that alternates between (a) fast adaptation via an explicit textual 'state' — skills and instructions distilled from the agent's own interaction trajectories and accepted only if they improve held-out validation — and (b) slow consolidation, where only the state-enabled behavior that stays useful across state revisions is distilled into the model weights, combined with ordinary reinforcement learning.
How much does Experience Funnel improve over prior methods?
In single-run experiments it reaches 62.4% on SearchQA, 67.9% on ALFWorld, and 42.4% on WebShop (57.6 average), versus 56.2 average for SkillRL, the strongest prior state–policy co-evolution baseline. The margins are 0.8–2.2 points and no significance tests are reported.
Which models, benchmarks, and hardware were used?
Qwen3.5-4B serves as the evolving policy with a frozen Qwen3.5-27B as the distillation teacher, trained on Huawei Ascend 910B3 NPUs. Evaluation covers SearchQA (question answering), ALFWorld (embodied interaction), and WebShop (web navigation), over five state-evolution rounds of which two were accepted.
Is code or training data available for Experience Funnel?
The paper mentions no code or artifact release, and key hyperparameters (e.g., the α weight for persistently useful trajectories and the λ distillation coefficient) are not specified, so independent reproduction currently requires re-deriving the full training setup.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2 and 3 appear only as flattened axis labels and captions in the extraction; the per-round curve values could not be audited, so round-by-round trajectory claims rest on the authors' prose.
- The construction of the 'residual state' in Table 5 is not defined anywhere in the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.