When Better Turns Do Not Make Better Agents: Diagnosing the Gap Between Next-Turn Metrics and Workflow Success
Md Tahmid Rahman Laskar, Xue-Yong Fu, Xue-Yong Fu, Gundeep Singh, Karol Chang, Kevin Sanders, Shi Zong, Tania Habib, et al.
cs.CL
Quick verdict
Shows SFT lifts next-turn metrics but not workflow success (0/77 holistic); solid controlled design, yet proprietary data, tiny eval, no significance tests.
Summary
This empirical study from Dialpad asks whether improvements under gold-history next-turn evaluation predict autonomous multi-turn workflow success. The authors fine-tune Qwen3 (4B/14B) and Gemma 3 (4B/12B) models via full SFT on a proprietary dataset of 130 customer-support workflows, where synthetic conversations were generated by paired GPT-5 instances and retained only when both Claude-4.5-Opus and Gemini-2.5-Pro approved (1,027 of 1,800 conversations). Five protocols — ROUGE-1, LLM-judged gold-history turn success, strict tool-call matching, closed-loop replay with a deterministic user simulator, and holistic workflow judgment — are applied to 84 held-out conversations (542 next-action examples; 77 tool-requiring workflows). SFT raises average ROUGE-1 from 24.8 to 49.2 and text-turn success from 33.2% to 58.4%, but tool-turn success improves only from 17.8% to 23.7%, and Gemma 3-4B exact call accuracy drops to 0.0% from 3.0%. Closed-loop completion reaches at most 8/77 (10.4%, Qwen3-14B), and the holistic judge finds zero successful workflows for any model. The authors conclude that gold-history evaluation is an unreliable proxy for workflow capability and recommend separately reporting text quality, tool correctness, and end-to-end success. Main limitations: proprietary unreleased data, a small evaluation set with no significance testing, judge validity unvalidated against humans, and a single fixed SFT recipe without ablations.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SFT consistently improves gold-history next-turn metrics across all four model pairs, raising average ROUGE-1 by 24.4 points (24.8 to 49.2) and text-turn success by 25.2 points (33.2% to 58.4%).
- These turn-level gains do not transfer to autonomous execution: the best closed-loop completion is 8/77 (10.4%) by SFT Qwen3-14B, and holistic judgment yields 0/77 success for every model before and after SFT.
- Aggregate next-turn scores conceal tool-execution failures, exemplified by Gemma 3-4B's exact call accuracy dropping from 3.0% to 0.0% after SFT while its all-turn success rises from 20.1% to 41.1%.
- Gold-history next-turn evaluation is therefore not a reliable proxy for workflow success, and evaluations should separately report text quality, local action correctness, tool execution, and end-to-end completion.
Strengths
Controlled protocol isolation
The same four model pairs and the same 84 conversations are scored under five protocols that differ only in evaluation method (Sec 3.2, Table 1), cleanly attributing any change in conclusions to the protocol itself rather than to model or data differences.
Metric ladder exposes aggregate masking
Table 1 shows Gemma 3-4B's all-turn success rising 20.1% to 41.1% after SFT while its exact call accuracy falls from 3.0% to 0.0% — direct evidence that aggregate next-turn scores can hide tool-execution regressions (Sec 4.2).
Transparent, dual-filtered data pipeline
Honest negative result
Deterministic tool checking alongside LLM judging
Weaknesses
Fully proprietary, unreleased evaluation
No statistical treatment of small samples
Judge validity and potential circularity
Strict matching may inflate the failure gap
Single SFT recipe, no ablations
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow, well-posed question: do gains under gold-history next-turn evaluation predict success when models must autonomously execute multi-turn customer-support workflows? Dialpad authors fine-tune four open-weight models (Qwen3-4B/14B, Gemma-3-4B/12B) with one fixed five-epoch full-SFT recipe on 5,834 next-action examples derived from 1,027 GPT-5-simulated conversations over 130 proprietary workflows, then evaluate on 84 held-out conversations (542 next-action examples; 166 tool calls; 77 tool-requiring workflows) under five protocols of increasing strictness: ROUGE-1, LLM-judged gold-history turn success, strict tool matching, closed-loop replay, and holistic workflow judgment (Sec 3.2, Figure 1). Central claims (Sec 1, Sec 4): (1) SFT lifts every gold-history metric — average ROUGE-1 24.8 to 49.2, text-turn success 33.2% to 58.4%, all-turn success 28.5% to 47.7%; (2) gains do not transfer — tool-turn success rises only 17.8% to 23.7%, closed-loop completion peaks at 8/77 (10.4%, SFT Qwen3-14B), and holistic judgment yields 0/77 for all models; (3) aggregates mask regressions — Gemma 3-4B all-turn success rises 20.1% to 41.1% while its exact call accuracy drops 3.0% to 0.0%. The claims are internally consistent with Table 1 and appropriately scoped.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is next-turn evaluation and why can it mislead about agent ability?
In next-turn evaluation, the model predicts the next assistant action given the correct (gold) conversation history, and each prediction is scored against a reference. Because the history is always correct, early mistakes never propagate, so the score measures how well a model responds from a clean state — not whether it can build and maintain that state on its own during a real multi-turn task. This paper shows SFT models improving substantially under this protocol while still failing almost every autonomous workflow.
What is the best workflow success rate achieved by any model in this paper?
Under deterministic closed-loop replay, SFT Qwen3-14B completed 8 of 77 tool-requiring workflows, i.e., 10.4% — the highest of the four models studied. Under the stricter holistic LLM judgment, which also requires policy compliance and correct use of tool results, zero workflows succeeded for any model, before or after SFT.
Which models and data were used in this study?
Qwen3 (4B and 14B) and Gemma 3 (4B and 12B), each evaluated zero-shot and after a five-epoch full supervised fine-tune on a proprietary Dialpad dataset: 1,027 synthetic customer-support conversations spanning 130 workflows, generated by paired GPT-5 user/agent simulators and retained only when both Claude-4.5-Opus and Gemini-2.5-Pro approved them. Evaluation used 84 held-out conversations with 542 next-action examples, 77 of which involve tool-requiring workflows.
Is the dataset or code from this paper publicly available?
No. The workflows, policies, and user simulator are proprietary to Dialpad, and no code release is mentioned. The paper publishes only sanitized prompt templates for data generation and judging in its appendices, so the reported numbers cannot currently be reproduced outside the company.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (protocol overview) is referenced but its content cannot be analyzed from the extracted text.
- Possible supplementary analyses (e.g., failure-category distributions) may exist in appendix material not included in the extraction.
- The supplied metadata lists 8 authors while the full-text byline lists 15 with equal-contribution markers; this review relies on the full text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.