A Bellman Optimality Equation for Plasticity
cs.LG
Quick verdict
First Bellman equation for directed-information plasticity; promising, but the convexity proof is flawed and solutions are suboptimal vs the definition.
Summary
This workshop paper takes a first step toward actively optimizing plasticity as recently defined by Abel et al. (2025)—the generalized directed information from an agent's observations to its actions. The authors derive a Bellman optimality equation (Eq. 6) whose per-step reward is the conditional mutual information I(S_{t+1}; A_{t+1} | S_t, A_t), mirroring Leibfried et al.'s (2019) Bellman formulation of empowerment. They prove the associated operator is a γ-contraction (Appendix A.1) and argue the inner maximization is convex, hence attained at deterministic policies (Appendix A.2); because the q marginal couples action choices across next states, they cast tabular optimization as a multi-weight knapsack dynamic program. Experiments on three small tabular environments (a 2-state Control-Gated MDP, a 5-state Road Environment, and a 35-state Two-Room gridworld) show monotone value convergence, and policy simulation shows the agent seeking the noisy room, where plasticity is highest. Key limitations: the convexity proof treats the q marginal as fixed although Eq. 7 defines it as a function of the policy; no proof connects the Bellman fixed point to the directed-information definition; and the paper itself reports the converged policy reaches about 0.92 bits of plasticity versus the 1-bit optimum found by enumeration in the Control-Gated MDP. No baselines or statistical tests are reported; code is released on GitHub.
Review complete
Assessment at a glance
4
risk flags identified
● high 2
● medium 2
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- There exists a Bellman optimality equation (Eq. 6) for maximizing plasticity within MDPs, analogous to prior Bellman formulations of empowerment (Leibfried et al. 2019).
- The Bellman operator associated with Eq. 6 is a γ-contraction, so value iteration converges to a unique fixed point (Theorem A.1).
- The inner optimization can be restricted to deterministic policies (Theorem A.2) and computed in tabular settings via a multi-weight knapsack dynamic program (Section 3.4, Algorithm 1).
- Value iteration under this equation converges monotonically in three tabular environments (Sections 5.1-5.3), with the agent in the Two-Room gridworld navigating to the noisy room to capture maximal plastic value.
- The converged policy does not fully maximize the directed-information definition of plasticity — a discrepancy the authors explicitly acknowledge (Section 5.1: ≈0.92 vs 1 bit).
Strengths
Fills a genuine, well-motivated gap
Section 1 correctly identifies that Abel et al. (2025) defined plasticity as generalized directed information I(O→A) but no prior work optimizes it, while empowerment has a rich optimization literature (Klyubin 2005; Mohamed & Rezende 2015; Leibfried 2019). The mirror-image derivation targeting the Leibfried-style Bellman equation is a natural, novel first step.
Convergence guarantee via contraction
Appendix A.1 proves the Bellman operator (Eq. 8) is a γ-contraction under the supremum norm, and Banach's fixed-point theorem then guarantees a unique V*. The conclusion is sound even though the write-up (Eqs. 10-11) loosely keeps one maximizer π for both value functions instead of applying |max f − max g| ≤ max |f − g| rigorously.
Creative algorithmic reduction
Honest negative observation
Strong reproducibility for the scale claimed
Weaknesses
Theorem A.2 (convexity) is invalid as proven
'Optimality for plasticity' is a surrogate claim, contradicted by the paper's own data
No baselines and minimal evaluation scale
No complexity analysis for the knapsack DP
Statistical reporting is descriptive only
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This workshop paper (Continual RL Workshop at RLC 2026) addresses the optimization side of the plasticity definition introduced by Abel et al. (2025): plasticity as generalized directed information I(O→A), the mirror of empowerment I(A→O). Section 1 correctly notes that while empowerment has a rich optimization literature (Klyubin et al. 2005; Mohamed & Rezende 2015; Leibfried et al. 2019), no prior work optimizes the new plasticity measure. Contributions: (1) a Bellman optimality equation for plasticity (Eq. 6) with step reward I(S_{t+1}; A_{t+1} | S_t, A_t) and values on history pairs (s_{t-1}, a_{t-1}); (2) a γ-contraction proof (Theorem A.1); (3) a reduction to deterministic policies (Theorem A.2); (4) a multi-weight knapsack dynamic program for tabular backups (Section 3.4, Algorithm 1); (5) experiments in three tabular environments. The abstract's framing as 'preliminary work' is accurate; the title's 'for plasticity' is less so (below).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is plasticity in this paper?
Following Abel et al. (2025), plasticity is defined information-theoretically as the generalized directed information flowing from an agent's observations to its actions — intuitively, how much the agent's behavior remains malleable to what it observes. It mirrors empowerment, which measures information from actions to observations (control over the future). This paper is the first attempt to actively optimize that plasticity measure inside Markov decision processes.
What does the paper actually prove?
It derives a Bellman optimality equation whose per-step reward is the conditional mutual information between the next observation and the chosen action, proves the associated operator is a γ-contraction (so value iteration has a unique fixed point), and argues the inner maximization is attained by deterministic policies, enabling a knapsack-style dynamic program. Notably, the paper's own experiments show the resulting policy does not fully maximize the directed-information definition — about 0.92 bits versus the 1-bit optimum in a 2-state benchmark.
Is code and data available for this paper?
Yes. Appendix C links a public GitHub repository (jeremy-lucas-mcgill/A-Bellman-Optimality-Equation-for-Plasticity) with source code for all three experiments, and Table 1 lists the hyperparameters (discount γ=0.9, 100 iterations, 100/100/10 trials per environment). All environments are small tabular MDPs fully specified in the text.
Does this paper solve the stability-plasticity problem in deep continual learning?
No. This is preliminary, workshop-scale work: it provides a tabular dynamic-programming formulation and evidence from environments of at most 35 states. The authors state (Section 6) that online sample-based algorithms with deep networks, and integration with empowerment maximization and external reward, are future work.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (e.g., the policy-enumeration plasticity-empowerment landscape in Figure 2 and the gridworld renderings) were extracted only as captions and embedded labels; graphical content could not be assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.