SUN: Reaching for Novelty in Reinforcement Learning
Wenyan Yang, Arsenii Mustafin, Dominik Baumann, Joni Pajarinen, Simone Parisi
cs.LG · cs.AI · cs.RO
Quick verdict
Proposes SUN, a novelty×reachability goal score for RL exploration; strong benchmarks and honest ablations, but proofs are oracle-scoped and baselines few.
Summary
The paper tackles exploration in reinforcement learning by selecting goals that are simultaneously novel and reachable within a goal-conditioned framework. Its core is SUN, a multiplicative indicator combining a successor value function (reachability) with an inverse-visit-count novelty term, pluggable into any off-policy algorithm (instantiated with DQN and TD3). Theoretical results, proved in an oracle setting with exact value functions, frozen counts, and deterministic dynamics, show SUN equals the value of a count-bonus reward, lower-bounds short-horizon hitting probability, and scores zero for unreachable goals; an appendix presents a UCB-style sample-complexity conjecture the authors explicitly leave incomplete. Methodologically, SUN adds an adaptive goal-selection rule that reselects mid-episode when the value estimate drops, and a replay-buffer pseudocount amortized into insertion for O(1) queries. Across 14 environments — three novel gridworlds with unreachable states, traps, and mazes; classic control; and GCRL locomotion/manipulation tasks — SUN outperforms AdaGoal, DISCOVER, and random exploration over 20 seeds on both coverage and normalized visitation entropy: mean coverage AUC improves 15.1% and 21.0% over the two baselines, entropy AUC 7.6% and 9.2%, and SUN runs roughly 3.5× faster on gridworlds and classic control. Limitations: count-based novelty is unsuited to image observations; candidate goals come only from the replay buffer; the pseudocount radius is sensitive in compact-support environments; no downstream task-reward evaluation is reported; and code is not yet released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SUN(g|s) = V^π(s,g)·(1/n_g) jointly scores novelty and reachability with no hand-tuned trade-off coefficient and plugs into any off-policy RL algorithm.
- Under oracle assumptions (exact SVF, frozen counts, deterministic dynamics), SUN provably equals the value of a count-bonus reward, lower-bounds short-horizon hitting probability (Pr[τ_g ≤ n] ≥ V^π − γ^(n+1)), and scores zero for unreachable goals.
- A replay-buffer pseudocount amortized into insertion yields O(1) novelty queries, avoiding KDE or critic-ensemble costs.
- Across 14 environments and 20 seeds, SUN outperforms AdaGoal, DISCOVER, and random exploration on both coverage and normalized visitation entropy, with mean AUC gains of +15.1%/+21.0% (coverage) and +7.6%/+9.2% (entropy) over the two learned baselines.
Strengths
Principled unification, not a heuristic product
Theorem 1 (Appendix A.2) shows SUN is exactly the value of a 1/n_g count-bonus reward, and the log-space decomposition (A.5, Eq. 18) shows the residual novelty/reachability trade-off is set by κ = −log γ — inherited from the discount — rather than a free coefficient, which the additive-vs-multiplicative ablation (Figure 7, Figure 8) then supports empirically.
Exemplary honesty in theoretical scoping
Assumption 1 declares the oracle setting (exact SVF, frozen counts, deterministic dynamics); Remark 1 concedes the online-count gap, Remark 2 concedes the 1/n_g exponent is a choice, and Appendix K openly labels the SUN-UCB sample-complexity result a conjecture, naming its two gaps (filter completeness, policy optimality) with a concrete counterexample MDP.
Thorough, decision-relevant ablations
Practical efficiency with quantified costs
Candid failure analysis
Weaknesses
Theory-practice gap
Narrow baseline set
Exploration-intrinsic metrics only
ρ sensitivity with a single tuned value
Camera-ready polish issues
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper proposes SUN (SUccessor-to-Novelty), a goal-selection indicator for reward-free exploration in goal-conditioned RL: SUN(g|s) = V^π(s,g) · (1/n_g) (Eq. 2), where V^π is a HER-trained successor value function (reachability) and n_g a replay-buffer pseudocount (novelty). Central claims: (1) the indicator integrates novelty and reachability without a hand-tuned trade-off coefficient; (2) it is provably the value of a count-bonus reward, lower-bounds short-horizon hitting probability, and rejects unreachable goals; (3) an adaptive reselection rule and an O(1)-query pseudocount make it practical; (4) it consistently beats AdaGoal, DISCOVER, and Random across 14 environments. The method is instantiated with DQN (discrete) and TD3 (continuous).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SUN indicator in reinforcement learning?
SUN (SUccessor-to-Novelty) is a goal-selection score for exploration: it multiplies a successor value function, which estimates how reachable a candidate goal is, by an inverse visit-count novelty term. Only goals that are both rarely visited and reachable get high scores. It can be plugged into off-policy algorithms such as DQN and TD3.
How does SUN differ from DISCOVER and AdaGoal?
Both prior methods score goals using disagreement among an ensemble of critics and commit to one goal per episode. SUN replaces the ensemble with a cheap replay-buffer pseudocount, multiplies novelty by reachability instead of adding them (avoiding a balancing coefficient), and reselects the goal mid-episode whenever its estimated value drops below the value at selection time.
Is code for the SUN paper available?
Not yet at submission time: the paper says source code will be available 'at link soon'. However, all environments are public (Gymnasium classic control, GCRL benchmarks of Bortkiewicz et al. 2025), the baselines build on the official DISCOVER implementation, and full hyperparameters are given in Appendix F.
What are the main limitations of SUN?
Four named in the paper: count-based novelty degrades on high-dimensional observations like images; candidate goals come only from the replay buffer, so unseen states are never proposed; goal selection is greedy and ignores the states traversed along the way; and the pseudocount radius is sensitive, especially in environments with compact reachable sets. The review also notes the absence of downstream task-reward evaluation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are present only as captions in the extraction; curve shapes and heatmaps were assessed from captions and in-text claims, not from image inspection.
- Mathematical notation is garbled by PDF extraction in places (e.g., exponents in Eq. 18, bin boundaries in Appendix D), though every cited statement remains reconstructable; there is no truncation — the text ends cleanly at Appendix K's final footnote.
- Table 1's duplicated 'AntMaze-H' row label appears in the extracted text itself; whether it is a paper typo or a table-extraction artifact could not be disambiguated from text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.