WHALE: A Simple Recipe for Joint Harness-Weight Optimization
Haechan Kim, Yoonho Lee, Gisang Lee, Chelsea Finn, Kangwook Lee
cs.LG · cs.AI
Quick verdict
Alternating weight and harness-code updates beats single-component and prompt-only joint training by 4.15-24.38pp; caveat: single runs, best-on-test selection.
Summary
The paper argues that an LLM agent's performance depends jointly on its weights and its executable harness, so optimizing either alone leaves the system bottlenecked by the frozen counterpart. It proposes WHALE, alternating online rejection-sampling fine-tuning (RSFT) of the model under the current harness with Meta-Harness search for a better harness under the updated model; switching uses fixed per-cycle budgets (E, I) or an adaptive patience rule driven only by training signals. With Qwen3.5-2B/4B agents on SearchQA, math with code execution, and chess puzzles, a single fixed schedule (0.6, 6) reaches 48.34%, 24.79%, and 29.83% best test mean@8, beating weight-only, harness-only, and a prompt-restricted Fast-Slow Training control by 7.67-24.38 and 4.15-13.00 points; adaptive WHALE reaches 52.82% (SearchQA) and 26.46% (Math). Analysis identifies harness-dominant and model-dominant regimes - harness search matches peak weight-only accuracy with 5.79% of the rollouts in SearchQA but yields 0.42% in Math - and shows small interleaved updates beat stagewise optimization by 5.32/9.16 points at 29%/49% of the rollouts. Main limitations: single training runs without error bars or significance tests; headline numbers are best values selected on the test set; harness acceptance uses a 256-example set with one rollout per candidate; and proposer LLM compute (Claude Opus 4.7) is excluded from efficiency accounting.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Alternating RSFT weight updates with Meta-Harness executable-harness search (WHALE) beats weight-only, harness-only, and a prompt-restricted FST control by 4.15-24.38 points of best test mean@8 across SearchQA, Math, and Chess (Figure 1b, Table 2).
- The optimization bottleneck is domain-dependent: harness search alone matches peak weight-only accuracy using 5.79% of the rollouts in harness-dominant SearchQA, but reaches only 0.42% accuracy in model-dominant Math until a small weight update unlocks it (Section 6.1).
- Small interleaved updates dominate stagewise weight-then-harness optimization, exceeding it by 5.32 (SearchQA) and 9.16 (Math) points at 29% and 49% of the stagewise rollout cost, due to conditional over-optimization in long phases (Section 6.2.1).
- A per-phase patience rule over training signals alone removes the fixed (E, I) budgets, achieving the best SearchQA result (52.82%) while trailing the best hand-tuned schedule in Math (26.46% vs 28.33%) (Section 6.2.3).
Strengths
Clean expressivity control via a prompt-restricted FST
The Fast-Slow Training control (Sections 2, 5.2, B.1) uses the identical RSFT update and WHALE schedule but confines harness search to system/user prompts, so the 4.15-13.00 point gap isolates executable-harness expressivity rather than algorithmic differences.
Budget-matched, mechanism-level bottleneck analysis
Figure 3 decomposes SearchQA into format/retrieval/extraction accuracy and Math into truncation/format rate; e.g., WHALE's cycle-1 harness search lifts Math format accuracy 0.83% to 4.38% in 4,608 rollouts versus harness-only's 46,080 rollouts for +0.63 points (Section 6.1), grounding the harness-dominant vs model-dominant regimes in measurable quantities.
Schedule ablation covering both failure extremes
Adaptive switching removes schedule hyperparameters
Above-average reproducibility disclosure
Weaknesses
Single-run evidence with no uncertainty quantification
Headline metric is best-so-far accuracy on the test set
Harness acceptance on a tiny, noisy signal
Efficiency claims exclude proposer compute
Baseline fidelity and scale limits
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
WHALE (arXiv 2609.00196; KRAFTON/KAIST/Stanford) targets a gap the authors argue prior joint-adaptation work leaves open: existing methods co-optimize weights with textual prompts only, while the executable harness - tool schemas, context management, parsing, termination - stays frozen. The proposed recipe alternates online rejection-sampling fine-tuning (RSFT, Section 3.1) with Meta-Harness search (Section 3.2), coordinated by fixed per-cycle budgets (E, I) or an adaptive patience rule on training signals (Section 4, Algorithm 2). Central claims: (1) WHALE beats weight-only, harness-only, and a prompt-restricted Fast-Slow Training (FST) control by 4.15-24.38 points of best test mean@8 across SearchQA, Math, and Chess (Figure 1b, Table 2); (2) bottleneck regimes are domain-dependent - SearchQA harness-dominant (harness search matches peak weight-only accuracy with 5.79% of the rollouts) and Math model-dominant (harness search alone reaches 0.42% vs 15.42% for weights, Section 6.1); (3) small interleaved updates beat stagewise optimization by 5.32/9.16 points at 29%/49% of its rollouts (Section 6.2.1); (4) adaptive switching recovers or exceeds tuned schedules (52.82% SearchQA; 26.46% vs 28.33% Math, Section 6.2.3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is WHALE in one sentence?
WHALE is a training recipe that alternates fine-tuning an LLM's weights (via online rejection-sampling fine-tuning) with searching for better executable agent harness code (via Meta-Harness), so the model and its harness co-adapt instead of one staying frozen while the other improves.
How much does WHALE improve over baselines?
In the paper's experiments with Qwen3.5-2B/4B agents, WHALE's best test mean@8 accuracy exceeds weight-only and harness-only training by 7.67-24.38 percentage points and a prompt-restricted Fast-Slow Training control by 4.15-13.00 points across SearchQA, math reasoning, and chess puzzle tasks; the best fixed schedule reaches 48.34%, 24.79%, and 29.83% in the three domains, and the adaptive variant reaches 52.82% on SearchQA.
What are the main weaknesses of the WHALE paper?
Results come from single training runs without error bars or significance tests; the headline numbers are the best values observed on the test set (checkpoint selection on test data); harness acceptance relies on a 256-example set with one rollout per candidate; the compute of the harness-proposer LLM (Claude Opus 4.7) is excluded from efficiency comparisons; and the Fast-Slow Training baseline is a re-implementation rather than the published algorithm.
Is the WHALE code available, and can it be reproduced?
The paper states code is available at github.com/krafton-ai/WHALE and gives detailed appendices (harness spaces, judge prompt, trajectory examples). Full reproduction, however, requires multi-GPU online training (roughly 0.3-1 million rollouts per run) plus paid API access to a Claude Opus 4.7 proposer and a GPT-5.4-mini judge, so independent replication is expensive and partially dependent on proprietary models.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure curves are not recoverable from the extraction: axis values and series labels appear as loose inline numbers, so trend judgments rely on the surrounding prose and on Tables 2-3, which are cleanly extracted.
- The abstract in the task metadata is cut mid-sentence ('stagewise weight-then-harn'), but the complete abstract appears at the top of the embedded full text, so no analysis was limited.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.