Admissable: Training Reinforcement Learning Agents against Adversarial Missingness
Paul Stahlhofen, Luca Hermes, Tim Kochs, Markus Vieth, Barbara Hammer
cs.LG
Quick verdict
Formalizes adversarial feature masking in RL and trains SAC against it; promising but thin evidence: 3 MuJoCo tasks, 1 baseline, 5 seeds, no sig. tests.
Summary
This paper introduces 'adversarial missingness' for reinforcement learning: an adversary permanently removes up to k features from the agent's state observation to minimize return, a threat model the authors argue is more realistic than ℓ∞-bounded perturbations for sensor-based agents. They formally extend the MDP with a fixed missing-feature subset, define a minimax robustness objective, and propose an adversarial training scheme built on SAC: every 5,000 steps, the value of each candidate missing feature is estimated via Monte-Carlo rollouts, and the sampling distribution over missing features is updated inversely to these scores, so the most damaging features are masked more often. On three MuJoCo environments (Hopper, Walker2d, Ant) with 11, 17, and 105 observation dimensions, training against this adaptive adversary yields better worst-case performance than a uniform-random-missingness baseline on all three tasks, with the gap growing with observation dimensionality; uniform training leaves a catastrophic weak spot on Ant. Limitations: only k=1 and three locomotion tasks, a single baseline, five seeds with no variance or significance reporting, an unspecified rollout budget N, no sensitivity analysis of the lower-bound hyperparameter l, and no code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Adversarial missingness — permanent adversarial removal of up to k state features — is a distinct and more realistic threat model for sensor-based RL agents than ℓ∞-bounded perturbations (Section 1).
- An SAC-based adversarial training that resamples the missing-feature distribution inversely to Monte-Carlo adversarial scores (Eqn. 8) optimizes the minimax robustness objective (Eqn. 7).
- Training against this adaptive adversary yields better worst-case performance than uniform-random-missingness training on Hopper, Walker2d, and Ant, with the advantage growing with observation dimensionality (Figure 1).
- Uniform-missingness training can leave a catastrophic single-feature weak spot (Ant) that evaluation averaging over missing features would fail to detect.
Strengths
Novel, practically motivated threat model
Section 1 argues that ℓ∞-bounded attacks model an attacker tweaking every sensor slightly, whereas adversarial missingness models permanent removal or destruction of chosen sensors — a distinction imported from causal structure learning (Koyuncu et al., 2024) and, to the authors' credit, not previously formalized for RL.
Clean formalization with explicit assumptions
Section 2 extends the MDP with the set U of admissible missing subsets (|u|≤k), a masking function (Eqn. 3), an occluding value function (Eqn. 4-5), and the minimax objective (Eqn. 7); the globally-fixed-u assumption (permanent sensor loss, not per-step attacks) is stated and justified rather than left implicit.
Simple, deployable training scheme
Dimension-scaling finding with practical bite
Weaknesses
Single baseline
No statistical reporting
Evaluation reuses the training signal
Hyperparameter l unexamined
Scope far narrower than the formalism
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.15297, cs.LG) formalizes **adversarial missingness for RL**: an adversary permanently removes up to k features from the agent's state (Eqn. 3, Section 2), in contrast to ℓ∞-bounded perturbations of every feature (the SA-MDP line) and to random missingness handled by imputation. The robustness objective is minimax: max_π min_{u∈U} v^π_u(s0) (Eqn. 7). Claims: (1) the threat model is new for RL and realistic (sensor destruction, Section 1); (2) a SAC-based adversarial training with inverse-score resampling (Eqn. 8) optimizes this objective; (3) it beats uniform-random-missingness training on Hopper, Walker2d, and Ant, with the advantage growing with observation dimension (11/17/105, Table 2); (4) uniform training leaves a catastrophic single-feature weak spot on Ant. The k=1 scope limit is honestly declared.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is adversarial missingness in reinforcement learning?
It is a threat model in which an adversary permanently removes a small number of features (e.g., one sensor reading) from the agent's state observation to minimize its performance — unlike classic adversarial attacks that slightly perturb every input feature within an ℓ∞ budget. The paper formalizes this for RL and trains agents to withstand it.
How does the paper's training method work?
During Soft Actor-Critic training, every 5,000 steps the authors estimate how much episodic return is lost when each individual feature is masked (zeroed, with a mask indicator appended), then sample which feature to mask with probability inversely proportional to these scores, so the most damaging features are masked more often. A baseline instead masks features uniformly at random.
What results does the paper report?
On three MuJoCo locomotion tasks (Hopper-v5, Walker2d-v5, Ant-v5 with 11, 17, and 105 observation dimensions), adversarial training beat uniform-random-missingness training in worst-case performance (minimum return over single-feature removals) on all three tasks, with the advantage growing with dimension; uniformly-trained policies showed a catastrophic weak spot on Ant.
Is code available for this paper?
The paper does not mention a code release. However, the method is simple to reimplement: SAC via Stable-Baselines3 on Gymnasium MuJoCo environments, with zero-masking plus an indicator vector and periodic resampling of the masked feature by inverse Monte-Carlo scores.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 — the sole quantitative evidence — is present only as captions in the extracted text; actual curve values, spread, and separation could not be inspected.
- The rollout budget N for adversarial score estimation is never stated, limiting precision of both the review and any reproduction.
- All experiments are k=1; the formalism covers general k, so empirical claims should not be read beyond single-feature removal.
- Training details beyond Table 1 (learning rates, network sizes, total steps, seeds) are absent from the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.