Specifying Reward Functions for RL Without Environment Sampling
Stephane Hatgis-Kessell, W. Bradley Knox, Emma Brunskill
cs.LG · cs.AI
Quick verdict
EARS learns rewards from preferences over imagined trajectories, zero env sampling; beats LLM prompting often, but tests only near-linear ground-truth rewards.
Summary
The paper addresses reward specification for RL when environment sampling is expensive or unsafe. It introduces EARS (Experience-Free Autonomous Reward Specification), a two-stage method using zero environment transitions. Stage 1 runs a structured conversation between stakeholder and facilitator LLMs to design a small set of executable Python reward features plus speculative trajectory-level ranges; Stage 2 samples 'imagined trajectories' uniformly in feature space, strategically selects pairs differing in at most two features (ENN-ensemble variance for Bradley-Terry preferences, linear-program disagreement for noiseless ones), and learns feature weights from preference labels. Using the Trajectory Alignment Coefficient (TAC) on three long-horizon domains from Pan et al. (2022)—pandemic lockdown, insulin dosing, highway control, each with two ground-truth rewards—EARS reaches .69–.96 under ground-truth preferences (Table 2), while direct LLM prompting often yields near-zero or negative alignment (pandemic r1: −.08 to −.37) even with privileged descriptions derived from the ground-truth reward. With LLM-labeled preferences over a written specification, EARS wins in nearly all settings (exception: traffic r1 with the realistic spec, −.03 vs .27). Against real-trajectory baselines, EARS is more data-efficient than Offline-RLHF and tends to match or beat LTRRT (Figure 2). Limitations: ground-truth rewards are effectively sums of a few interpretable terms, favoring the linear model; preferences are synthetic or LLM-labeled, not human; only Gemini 3 is tested; and the evaluation specifications were themselves generated from the ground-truth rewards.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Directly prompting an LLM to write a reward function often fails to achieve alignment, even when the prompt contains a privileged, highly detailed description generated from the ground-truth reward implementation (Table 2).
- EARS specifies reward functions with zero environment transitions while matching or exceeding the preference-label efficiency of baselines that learn from real trajectories (Figure 2).
- When preferences are labeled by an LLM conditioned on a natural-language specification, EARS produces more aligned reward functions than direct prompting in nearly all settings, with the exception of the traffic r1 realistic-specification case (Table 3).
- Sampling imagined trajectories from the LLM-designed feature space is essential; sampling from the raw observation space instead substantially underperforms (Table 4).
Strengths
Novel, practically motivated problem setting
EARS (Sections 1, 3) specifies reward functions with zero environment transitions — the paper notes it is 'strictly more environment-sample-efficient' than online RLHF, targeting domains where rollouts are unsafe or computationally prohibitive (Section 1).
The negative result on prompting is striking and well-controlled
Table 2 shows Direct Prompting fails even with a Privileged Reward Description generated from the ground-truth reward implementation (pandemic r1: −.14±.14 vs EARS .72–.73); full specification texts in Appendix F let readers audit what the prompting LLM actually saw.
Direct comparison against environment-sampling baselines answers the obvious objection
Decisive ablation isolating Stage 1
Hyperparameter discipline and honest reporting
Weaknesses
Benchmark rewards are near-linear, favoring EARS's hypothesis class
Evaluation infrastructure is entangled with the ground truth
Thin statistical treatment
Prompting baselines are one-shot only
Single LLM dependence and a minor internal inconsistency
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
EARS targets reward specification for RL agents when sampling the environment is expensive, slow, or unsafe (Section 1). The paper advances three claims: (1) directly prompting an LLM — even with a *privileged* natural-language description generated from the ground-truth reward implementation — often fails to produce an aligned reward function; (2) EARS learns reward functions from preferences over 'imagined trajectories' with zero environment transitions, matching the label efficiency of baselines that sample real trajectories; (3) given only a natural-language specification and an LLM preference labeler, EARS beats direct prompting. Alignment is measured by TAC (Section 4.1), the Kendall-tau correlation between trajectory rankings under the learned and ground-truth rewards, computed over 500 trajectories drawn from checkpoints of a policy trained on the ground-truth reward.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is EARS in reinforcement learning?
EARS (Experience-Free Autonomous Reward Specification) is a method for learning a reward function from preference labels without sampling any trajectories from the environment. An LLM dialogue first designs a small set of reward features from a task description and the observation space; then 'imagined trajectories' are sampled in feature space, preferences are collected over pairs, and linear feature weights are learned — so no policy rollouts, simulator calls, or real-world interactions are needed.
Does EARS beat directly prompting an LLM for reward design?
In the paper's three domains (pandemic lockdown, insulin dosing, highway control) with two ground-truth rewards each, yes in most settings: with ground-truth preferences EARS reached trajectory-alignment (TAC) values of .69–.96 while direct prompting often scored near zero or negative (pandemic r1: −.08 to −.37), even when the prompt included a highly detailed description derived from the ground-truth reward. One exception appears in the LLM-labeled setting: the traffic r1 case with the realistic specification, where prompting (.27) beat EARS (−.03).
Does EARS need a human in the loop?
Not necessarily. The paper's most deployable configuration (Section 4.5) uses an LLM to label preferences over imagined trajectories conditioned on a written specification of objectives, so no environment sampling and no interactive human labeling are needed. Labeling preferences by actual human stakeholders is explicitly left as future work, and the authors note it is unknown whether humans can reliably judge imagined trajectories that may be physically unrealizable.
What are the main limitations of EARS?
Per the paper and this review: preferences have only been tested with synthetic and LLM labelers, not humans; only Gemini 3 was used for all LLM stages; the three benchmark ground-truth rewards are effectively sums of a few interpretable terms, which suits EARS's linear reward model; and the natural-language specifications used in evaluation were themselves generated from the ground-truth rewards, so fully human-authored specifications remain untested.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2 (data-efficiency curves vs Offline-RLHF/LTRRT) is available only as a caption; exact crossover points and per-x values cannot be verified from the extracted text.
- The code-release URL (the word 'here' in Section 1) is not present in the extraction, so the repository could not be checked.
- Figure 1 (pipeline diagram) is not analyzable from text; math typography in Eqs. 1–4 is partially garbled but reconstructible from context.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.