Direct Diversity Optimization for Diverse Successful Trajectories in Preference Post-Training
Junwon Ko, Dong-Jae Lee, Minchan Kwon, Sunghyun Baek, Junmo Kim
cs.CL · cs.AI · cs.LG
Quick verdict
DDO pairs state-aligned branch data with reference-relative odds targets to preserve diverse successful agent strategies; strong gains, no significance tests.
Summary
The paper introduces Direct Diversity Optimization (DDO), an offline post-training method for LLM agents that aims to preserve multiple successful strategies rather than collapse onto a single winning trajectory. DDO combines Divergence-Tree Collection (DTC), which restores shared decision states along successful source trajectories, queries an expert model for alternative actions, and labels each branch by its rolled-out outcome; with the Reference-Relative Target-Odds objective (RTO), which fits the policy's log-odds between successful branches to a target distribution built from the reference model (interpolated with uniform via α=0.5), while success–failure pairs reduce to a DPO-style term. On BabyAI, BabaIsAI (via BALROG), and WebShop, with Qwen3-1.7B as the trained model and Qwen3.5-122B-A10B-FP8 as expert, DDO reaches the best joint task success (0.76 average) and coverage (H-ESD 0.35, ESD 0.40) among DPO, DivPO-style, and tie-aware baselines; the largest gain is on WebShop (success 0.26 to 0.36). DDO also shows the highest recovery after local action replacement (75.2% on BabyAI), beats successful-only imitation by 24–32pp in success, and outperforms temperature-increased DPO decoding, which gains coverage only by sacrificing success. Main limitations: coverage metrics rest on hand-designed trajectory-class equivalence relations, no statistical significance testing is reported, state restoration restricts applicability to reversible environments, and only one small target model family is tested.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Trajectory-level outcome labels in preference post-training provide no same-state supervision over multiple successful branches, causing the policy to collapse onto a single successful strategy.
- DDO (DTC + RTO) achieves the strongest joint task success and successful strategy coverage among the compared offline post-training methods on BabyAI, BabaIsAI, and WebShop (avg 0.76 success, 0.35 H-ESD, 0.40 ESD).
- Broader successful strategy coverage translates into the highest recovery rate (75.2% on BabyAI) after local action replacement.
- The gains require same-state outcome comparisons plus reference-relative targets: successful-only imitation at up to 7.3x exposure and temperature-increased DPO decoding both trail DDO in joint success and coverage.
Strengths
Precisely articulated gap with a targeted mechanism
Sec. 1/Fig. 1 identify that trajectory-level success labels give no same-state supervision over multiple successful branches; DTC (Sec. 3.2) and RTO (Sec. 3.3, Eqs. 2-6) are designed to fill exactly this hole rather than a generic diversity bonus.
Well-controlled comparison protocol
All preference methods (Table 2) start from the same task-SFT Reference, share the same DTC branch sets, LoRA config (rank 32, App. C.2), and decoding (T=0.6, top-p 0.95, App. C.4), isolating objective-side differences; Table 1 separately varies the data side in a 2x2 design.
Consistent gains with per-component attribution
Meaningful controls beyond main comparison
Collection efficiency and depth quantified
Weaknesses
No statistical significance or variance reporting
Coverage metrics depend on hand-crafted equivalence relations
Narrow model coverage
Expert-bounded diversity and missing online-RL comparison
Unanalyzed baseline anomalies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (KAIST; code released) targets a clearly articulated gap: trajectory-level success/failure labels in preference post-training carry no signal about how probability should be distributed among *multiple successful branches from the same decision state*, so policies collapse onto one winning trajectory (Sec. 1, Fig. 1). The proposed DDO combines Divergence-Tree Collection (DTC), which restores shared decision states along successful source trajectories and records outcome-labeled alternative branches (Sec. 3.2, Alg. 1), with the Reference-Relative Target-Odds objective (RTO), which matches the policy's reference-relative pairwise log-odds among successful branches to a target q interpolated (α=0.5) between uniform and the renormalized reference distribution; success-failure pairs reduce to a DPO term at p*=1 (Sec. 3.3, Eqs. 2-6). Claims are scoped as 'strongest among the compared post-training methods', and the paper largely respects that scope.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DDO (Direct Diversity Optimization) in one sentence?
DDO is an offline post-training method that keeps LLM agents capable of multiple different successful strategies for the same task by collecting outcome-labeled alternative action branches from shared decision states (Divergence-Tree Collection) and training the model to match reference-relative target odds among successful branches (RTO objective) instead of pushing one winner.
What results does DDO achieve?
On BabyAI, BabaIsAI (via BALROG), and WebShop with a Qwen3-1.7B agent, DDO reaches 0.76 average task success, 0.35 H-ESD, and 0.40 ESD, ahead of DPO (0.68/0.26/0.31) and other preference baselines; the largest gain is on WebShop (success 0.26 to 0.36), and it recovers from a forced wrong action 75.2% of the time on BabyAI vs 69.7% for DPO.
Is code or data available for DDO?
Yes. The paper links a public GitHub repository (github.com/koguma00/direct_diverse_optimization), and all components are public: the BabyAI and BabaIsAI environments via the BALROG text-action interface, WebShop, and the Qwen3-1.7B target and Qwen3.5-122B-A10B-FP8 expert models under Apache 2.0.
What are DDO's main limitations?
DTC requires environments that support exact state restoration and reversible execution, which excludes stochastic, partially observable, and real-world side-effect settings; coverage metrics rely on hand-designed trajectory equivalence rules; only a Qwen3-1.7B target model was tested; and the paper reports no statistical significance tests or error bars.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All tables are embedded as flattened inline text; values were legible but multi-column cell alignment (Tables 8-9) was reconstructed from the label ordering.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.