SAP: State-Guided Data Synthesis with Argument Provenance for Multi-Turn Tool Use
Zichen Tian, Jinpeng Chen, Cheng Gong, Suiyun Zhang, Rui Liu
cs.AI
Quick verdict
Provenance-tagged FSM synthesis for tool-use data; SAP-4B gains plausible but confounded by train/eval environment overlap and single-seed, untested numbers.
Summary
The paper proposes SAP, a pipeline that makes the causal source of every tool argument an explicit constraint during multi-turn tool-use data synthesis. AFSM converts tool documentation into a finite-state-machine skeleton whose edges carry per-argument provenance tags (initial state, previous output, self-created, previous user message, runtime fallback); Aplan fills arguments turn by turn, executes each call immediately in the target backends, and retries only failing calls; Amsg writes dialogue text after the call sequence is frozen. Two replay-guarded operators inject MissParam/MissFunc hard cases. SFT on ~9k synthesized trajectories yields SAP-4B (Qwen3-4B-Instruct-2507): BFCL v4 multi-turn average rises 22.1→30.4 (+8.3) and τ2-bench average 32.2→35.1 (+2.9), matching AWM-4B on BFCL (30.4 vs 30.3) and beating it on τ2 (35.1 vs 24.7), while still trailing 8B baselines on BFCL. Ablations attribute most of the gain to provenance tags (−6.4 without) and show tags are necessary to produce cross-turn dependencies at all (zero Po arguments without them). Key limitations: synthesis reuses the same executor backends and schemas used for evaluation, results come from single runs without statistical tests, the pipeline depends on proprietary Gemini-3 models, and the LLM judge lacks human-agreement validation. Code, data, and weights are stated to be publicly released.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing open-source multi-turn tool-use datasets have shallow cross-turn argument-dependency chains (mean ≤2.54, max ≤12) and few dependent arguments (6.8–20.5%).
- Making argument provenance an explicit synthesis-time constraint — FSM-declared per-argument tags checked against executed history — is the primary driver of downstream accuracy (A1: −6.4 BFCL v4 MT avg without tags).
- SFT on ~9k SAP trajectories lifts Qwen3-4B-Instruct-2507 from 22.1 to 30.4 BFCL v4 MT avg (+8.3) and from 32.2 to 35.1 τ2-bench avg (+2.9), without RL.
- The FSM skeleton and per-call retry are necessary for pipeline throughput (89% pass rate vs 50% without FSM, 54% without retry), and the tags are necessary to generate any cross-turn (Po) dependencies at all.
Strengths
Mechanism cleanly isolated by pipeline ablation
Table 6: the FSM+noTag variant keeps a 92% executor pass rate but produces zero Po arguments, showing the Provenance Tags — not the executability machinery — are what generate cross-turn dependencies; noRetry drops pass rate to 54% and noFSM+Tag to 50% with 15.75% fallback, quantifying each component's distinct role.
Executor-grounded synthesis with local retry
Every call is executed in the real backends at synthesis time (§4.3) and only the offending call is regenerated (App. A.3), yielding an 89% pass rate vs 54% without retry while avoiding whole-trajectory discard; the Pu dialogue cycle is solved by a remap-bind-restore protocol with closure checks (App. A.5).
Transparent cost and fallback accounting
Candid limitations and calibrated claims
No-regression check on single-turn capability
Weaknesses
Train/eval environment overlap is uncontrolled
Zero statistical rigor
Figure 1 diagnostic overstates its own data
Scale attribution without a scale control
No downstream AFSM or judge validation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SAP (arXiv 2609.06124, cs.AI) is a data-synthesis framework for multi-turn tool-use training trajectories. Its central claims: (1) existing open multi-turn tool-use datasets have shallow cross-turn argument-dependency chains (Fig. 1: mean chain length 0.93/1.71/2.54 for ToolACE/APIGen-MT/SAP; max 4/9/12); (2) making argument provenance an explicit synthesis-time constraint — a provenance-tagged FSM skeleton plus per-turn executor validation — yields high-accuracy trajectories cheaply (89% executor pass rate, ~$0.49/trajectory per B.1); (3) SAP-4B, an SFT of Qwen3-4B-Instruct-2507 on ~9k synthesized trajectories, improves BFCL v4 MT avg 22.1→30.4 (+8.3) and τ2-bench avg 32.2→35.1 (+2.9), matching AWM-4B on BFCL (30.4 vs 30.3) and beating it on τ2 (35.1 vs 24.7). The paper is explicit that it trails 8B baselines on BFCL (e.g., AWM-8B 40.1/37.4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SAP-4B and how good is it?
SAP-4B is Qwen3-4B-Instruct-2507 fine-tuned on about 9,000 synthetic multi-turn tool-use trajectories generated by the SAP pipeline. The paper reports 30.4 average on BFCL v4 multi-turn (up from 22.1 for the untuned backbone) and 35.1 average on τ2-bench (up from 32.2), matching the AWM-4B baseline on BFCL and clearly beating it on τ2-bench (35.1 vs 24.7), while remaining below 8B models on BFCL.
Is SAP's code, data, or model public?
The paper states that synthesis code, the ~9k synthesized trajectories, and the trained weights are available at github.com/Zichen1024/SAP. Note that re-running the synthesis pipeline itself requires access to proprietary models (Gemini-3/3.1 Pro and Qwen3-235B) as the generation backbones.
How is SAP different from existing tool-use data synthesis methods like APIGen-MT?
Intent-first methods such as APIGen-MT draft user intents, generate full dialogues, and validate afterwards, often discarding an entire trajectory when one check fails. SAP instead builds a provenance-tagged FSM skeleton first, fills and immediately executes each tool call with per-call retry, and only then writes the dialogue text — making argument dependencies an explicit, executor-checked constraint rather than a post-hoc filter.
Does training on multi-turn data hurt single-turn function calling?
The paper reports no regression: on BFCL v4 single-turn, SAP-4B scores 84.88 (non-live) and 75.95 (live) versus 84.36 and 75.41 for the untuned backbone — a small gain of roughly +0.5 points on both splits.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- AWM-8B's τ2-Retail cell in Table 2 is garbled by PDF extraction (rendered as '.1'); its τ2 Avg (37.4) and BFCL values are intact, and this review avoids citing the corrupted cell.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.