The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks
Wenbo Pan, Zhichao Liu, Shujie Liu, Jingying Zeng, Chin-Yew Lin, Xianfeng Tang, Yan Lu, Qi He, et al.
cs.AI
Quick verdict
Proposes Taste-Bench: 502 auto-mined decision forks labeled by hindsight outcomes; careful protocol and distillation gains, but label validity gaps remain.
Summary
The paper introduces 'taste' — an agent's ability to make good long-horizon decisions — and presents Taste-Bench, 502 binary decision-fork questions mined automatically from agent trajectories without human annotation. Forks come from two constructions: parallel attempts at the same task that diverge at an equivalent prefix, and detours where an agent abandons a failing direction mid-run. Recorded branch outcomes label the better candidate; a four-judge panel filters trivial and undecidable items, and a two-reviewer human check shows 98.8% label agreement (κ = 0.973). Evaluating 14 frontier models under a position-bias-aware both-orders-correct protocol (random = 25%), the best model, GPT-5.6 Sol, reaches 59.7%. Accuracy falls from 62.3% to 21.0% as the deciding evidence moves later in the trajectory, and higher reasoning budgets do not help. Taste-Bench correlates only partly with SWE-bench Verified (r = 0.63 over 11 models; engineering subset r = 0.37). Distilling a privileged teacher's reasoning into Qwen3.6-27B lifts held-out accuracy from 30.0% to 47.9%, and its advice raises a fixed executor's success on 41 held-out SWE-bench Pro tasks from 14.6% to 33.7% (39.0% with correct advice). Main concerns: outcome-attribution confounds in mined labels and the absence of a random-advice baseline in the end-to-end test.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Taste — choosing the better direction at a decision fork before outcomes are visible — can be measured automatically by mining forks from existing agent trajectories with hindsight labels.
- Frontier models have limited taste: the best (GPT-5.6 Sol) scores 59.7% on 502 questions under a both-orders-correct protocol where random guessing yields 25%.
- Accuracy falls from 62.3% to 21.0% as the deciding evidence moves later in the trajectory, and larger reasoning budgets do not improve accuracy.
- Distilling a privileged teacher's reasoning into Qwen3.6-27B raises held-out fork accuracy from 30.0% to 47.9% and lifts an executor's success on 41 held-out SWE-bench Pro tasks from 14.6% to 33.7%.
Strengths
Annotation-free labeling that scales with trajectory production
Section 2-3: forks are mined from 2,677 engineering rollouts and 1,132 METR MALT research runs; the recorded branch outcome supplies the label. Human review of 100 sampled questions shows 98.8% agreement with mined labels and Cohen's κ = 0.973 between reviewers (Appendix F).
Position-bias-aware evaluation protocol
Section 3.4: every question is asked in two candidate orders and counts as correct only if both are right, making random guessing 25% and constant-position answering 0%. This is a rigorous treatment of a well-known LLM-as-judge failure mode.
Strict, transparent filtering with mechanically verified evidence
Complete distillation study with leakage control and an upper-bound ablation
Exceptional reproducibility posture
Weaknesses
Label validity rests on an unproven attribution assumption
Missing random/inverted-advice baseline in the end-to-end experiment
Cross-benchmark correlation claims are statistically thin
Provenance and family bias cannot be excluded
Protocol conflates instruction-following with taste; horizon annotation has no reliability check
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper formalizes "taste" — an agent's ability to choose the better direction at a decision fork before outcomes are visible — and delivers three artifacts: (1) Taste-Bench, 502 binary questions mined from agent trajectories; (2) an evaluation of 14 frontier models; (3) a distillation recipe showing taste transfers to unseen tasks and improves end-to-end success. Central claims: the best model (GPT-5.6 Sol) reaches 59.7% under a both-orders-correct protocol where random guessing yields 25%; accuracy degrades with time horizon (mean 62.3% at in-prefix to 21.0% at more-work); reasoning budget does not help (−0.2pp for GPT-5.6 Sol, +2.2pp for Luna); distillation lifts Qwen3.6-27B from 30.0% to 47.9% on held-out questions and raises executor success on 41 held-out SWE-bench Pro tasks from 14.6% to 33.7% (correct-advice upper bound 39.0%).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Taste-Bench?
Taste-Bench is a benchmark of 502 multiple-choice questions, each showing an AI agent's task, its progress up to a decision fork, and two candidate next steps. One option led to a better recorded outcome in real agent runs; the evaluated model must pick it without seeing what happened afterwards. Code and data are public (github.com/wbopan/tastebench; huggingface.co/datasets/wenbopan/taste-bench).
What does 'taste' mean for AI agents?
In this paper, taste is an agent's ability to make good long-horizon decisions — e.g., which hypothesis to test or which implementation to build on — where the payoff only becomes clear much later in the task, after substantial budget has been spent.
How well do current models score on Taste-Bench?
The best model, GPT-5.6 Sol, answers 59.7% of questions correctly under a strict both-orders-correct protocol where random guessing scores 25%. Across 14 frontier models, scores range from 59.7% down to 15.7%.
Can an LLM's decision-making 'taste' be improved?
Per the paper, yes: distilling a teacher model that has seen the correct answer into Qwen3.6-27B raised accuracy on unseen fork questions from 30.0% to 47.9%, and using the student's judgment as task advice raised an executor agent's success rate on 41 held-out SWE-bench Pro tasks from 14.6% to 33.7%.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as embedded text renders; visual details (e.g., which points are highlighted in Figure 5, CI whisker shapes) were assessed from reported numbers, not the graphics.
- The abstract in the source is truncated mid-sentence ('teacher that has seen th...'); conclusions about the abstract's full wording are inferred from the body.
- Appendix A.4/D.1 tables were partially rendered; per-cell judge counts and some joint-outcome columns were read from dense text and could contain extraction artifacts.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.