Policy Complexity, Reaction Time, and Bounded Rationality in Reinforcement Learning
cs.LG · cs.AI · cs.IT
Quick verdict
MI-SARSA adds MI policy compression to SARSA; tabular tests show reward-complexity, RT and robustness tradeoffs. Sound but incremental; RT is definitional.
Summary
The paper introduces MI-SARSA, an on-policy temporal-difference algorithm that augments SARSA with a mutual-information penalty on policy complexity. Complexity is measured as the KL divergence between the state-conditional policy and a marginal action prior learned online by exponential moving average (Eqs. 2–3), folded into the immediate reward (r̂ = r − β[log π − log p], Eq. 4), leaving the standard SARSA update otherwise intact (Algorithm 1). In two tabular Gymnasium domains — a modified 8×8 Frozen Lake (+10/−10/−1 rewards) and 4×12 Cliff Walking (−100 cliff) — β sweeps (0–5, 50 seeds) trace a reward-complexity frontier: higher complexity buys return. The authors map the state-wise KL cost to reaction time via RT(s) = RT0 + m·D_KL (RT0 = 10⁻³, m = 1, arbitrary), predicting slower decisions in hazardous, control-critical states; however, RT is simulated, never measured. Environment-shift experiments (20 maps per domain, 2000 episodes in 100-episode blocks) show a robustness-capacity tradeoff: larger β shrinks post-switch performance drops but monotonically lowers asymptotic return. Limitations: tabular scope only, no human data, no baselines (e.g., entropy-regularized SARSA or MI-soft-Q), no convergence analysis for the non-stationary shaped reward, and no code release; the authors position the work as a framework rather than a descriptive model.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Embedding a KL-to-learned-marginal policy-complexity penalty in SARSA's immediate reward (MI-SARSA) yields an empirical reward-complexity frontier in two tabular navigation domains.
- The same state-wise KL cost doubles as a trial-level reaction-time predictor, extending Hick's law to sequential RL, with slower decisions in control-critical states.
- Under environment shifts, stronger regularization reduces post-switch performance degradation but monotonically lowers asymptotic return, revealing a robustness-capacity tradeoff.
- The authors position MI-SARSA as a process-level framework for bounded sequential learning rather than a fitted descriptive model of human behavior.
Strengths
Principled RT bridge into sequential RL
Section 4.4 grounds the RT(s) = RT0 + m·D_KL(π(·|s)||p) mapping in Hick's law [22] and Zénon et al.'s KL cost of cognition [16], yielding spatially structured, state-resolved latency predictions (slower near holes and along the cliff path) rather than a single scalar.
Honest scoping and confound acknowledgment
Section 5 explicitly concedes tabular-only evaluation and absence of human data; Section 4.5 flags that positive Δ-performance under strong compression reflects within-block learning still improving (underfitting), not beneficial transfer — a nuance many papers would oversell.
Systematic sweep design
Complete, minimal algorithmic specification
Weaknesses
The RT result is circular
No baseline comparisons
Reproducibility gaps
No theoretical analysis of the shaped reward
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper proposes MI-SARSA, an on-policy temporal-difference algorithm that embeds a policy-complexity penalty — the KL divergence between the state-conditional policy π(·|s) and an exponentially averaged marginal action prior p(·) — directly into the immediate reward (Eq. 4, §3.2–3.3). The authors claim three linked results: (i) a reward-complexity frontier in two tabular Gymnasium domains (§4.3, Fig. 1); (ii) trial-level reaction-time predictions derived from the same state-wise cost, extending Hick's law into sequential RL (§4.4, Fig. 2); and (iii) a robustness-capacity tradeoff under environment shifts (§4.5, Fig. 3). The authors are candid that the objective itself is not new — it derives from Grau-Moya et al.'s MI-regularized soft Q-learning [5] and Lai & Gershman's policy-compression framework [7, 8] — and that the contribution is embedding it in SARSA and interpreting the per-state cost as cognitive latency. Section 5 correctly positions the work as "a framework rather than a detailed descriptive account."
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MI-SARSA?
MI-SARSA is a variant of the SARSA reinforcement-learning algorithm that adds a mutual-information penalty on policy complexity: the KL divergence between the state-specific policy and an online-learned marginal action prior is subtracted from the immediate reward, weighted by β. Everything else follows standard on-policy temporal-difference learning.
Does the paper test the reaction-time model on human data?
No. Reaction times are simulated via RT(s) = RT0 + m·D_KL(π(·|s)||p) with arbitrarily chosen parameters (RT0 = 10⁻³, m = 1). The authors state the model is not fit to human data and call for trial-by-trial empirical validation as future work.
What environments were used to evaluate MI-SARSA?
Two tabular Gymnasium navigation tasks: a modified 8×8 Frozen Lake (goal +10, hole −10, step −1) and the standard 4×12 Cliff Walking (step −1, cliff −100), plus 19 random variants of each domain for the environment-shift experiments.
Is code available for MI-SARSA?
The paper does not mention a code or data release. However, Algorithm 1 fully specifies the procedure and the environments are standard Gymnasium benchmarks, so independent reimplementation is feasible.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only through captions; curve shapes, variance magnitudes, and exact frontier values could not be verified from the extracted text.
- No supplementary material or hyperparameter tables appear in the text, so absolute performance levels cannot be cross-checked.
- The paper is a preprint (arXiv 2609.28737, dated September 25, 2026); peer-review status is unknown.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.