Debate-to-Skill: Capability-Bound Process Supervision for Industrial Query-to-Agent Annotation
Shiyu Zhang, Leisheng Cheng, Huifu Li
cs.AI
Quick verdict
Baidu claims process supervision of capability decisions lifts query-to-agent annotation (grey-zone F1 45.8→64.9); credible but closed-data, self-referential.
Summary
The paper argues that industrial query-to-agent annotation fails when topical relevance is mistaken for executable capability, and proposes capability-bound process supervision as the fix. Debate-to-Skill retrieves up to three decision principles from an evolving skill bank via lexical similarity, runs a two-round support-versus-veto debate with judge guidance over each query–agent pair, extracts a structured verdict, and recycles high-disagreement conflicts into new or refined skills. Traces supervise a Qwen2.5-14B-Instruct student through reasoning-SFT plus GRPO with a verifier reward (classification 1.0; section validity and skill grounding 0.1 each). On 170K proprietary industrial pairs (2025Q4) with 3,000 human-labeled test examples and a 312-pair grey-zone slice, Debate-to-Skill reaches 91.6% accuracy on Main Test versus 81.3% for direct-label SFT, lifts grey-zone F1 from 45.8 to 64.9, and cuts grey-zone over-acceptance from 33.7% to 16.9% while the positive miss rate also falls (9.8%→7.4%). A four-week online A/B test (10% traffic) reports relative lifts of +15.79% card impressions, +7.63% card CTR, and +6.12% redirect CTR (p<0.05). Main limitations: entirely closed data and pipeline, no external annotator baselines, no variance or significance reporting for the offline tables, and an unresolved appendix reference.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Industrial query-to-agent annotation failures are dominated by mistaking topical relevance for executable capability, so the supervision target should be the capability-critical decision process rather than a final relevance label.
- Debate-to-Skill's gains come from the support-vs-veto deliberation process itself, not the shared GRPO loop: removing the debate process drops Main Test accuracy from 91.6% to 88.6% while keeping the same SFT+GRPO training.
- Dynamic skill retrieval and disagreement-driven skill updates materially improve capability grounding: fixed seed skills collapse the skill grounding rate from 92.7% to 62.8% and grey-zone F1 from 64.9 to 59.8.
- Judge aggregation of the debate reduces grey-zone over-acceptance (16.9% vs 29.8% when the last debate round is used as the verdict).
- The offline gains transfer to production: a four-week A/B test shows relative lifts of +15.79% card impression rate, +7.63% card CTR, and +6.12% redirect CTR at p<0.05.
Strengths
Precise problem diagnosis with a label scheme that matches it
The ternary scheme (§2) explicitly isolates 'semantically related but capability-insufficient' pairs (raw-label-1), and the flight-rebooking example makes the relevance-vs-executability failure concrete — a framing absent from ToolBench/API-Bank-style benchmarks, as the authors note in §7.
Disciplined ablation chain with a proper closest control
w/o Debate keeps the same SFT+GRPO loop with one-pass prediction (88.6 vs 91.6 ACC), isolating the process object from the RL loop; w/o Judge (86.9 ACC, 29.8 over-accept) isolates aggregation; w/o Dynamic Skills (89.4 ACC) isolates the skill bank, and Table 4 shows its grounding rate collapsing to 62.8 vs 92.7 for the full method.
Error-mode analysis rather than aggregate metrics alone
Deployment evidence, not just offline benchmarks
Candid limitations section
Weaknesses
No external baselines — the evaluation is entirely self-referential
No uncertainty quantification on offline results
Closed pipeline with self-distillation circularity
Unclear firewall between skill-bank updates and evaluation data
Thin online disclosure
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This method paper from Baidu (arXiv 2609.11176) targets industrial query-to-agent (Q2A) annotation: deciding whether a candidate agent can actually execute a user request, as opposed to merely being topically related. The diagnosis (§1, §2) is that final-label supervision collapses semantic relatedness and executable capability, and that the costly errors are over-acceptances of grey-zone pairs — semantically plausible but capability-invalid (e.g., flight rebooking offered by a consultation-only assistant). The proposed fix is "capability-bound process supervision": annotate by deliberating over a capability-critical decision process, then train the routing model on those traces. Three claims are staked: (1) the supervision object, not generic reasoning, drives gains; (2) each structural component (debate, skills, judge) contributes measurably; (3) gains transfer to production traffic.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Debate-to-Skill?
Debate-to-Skill is an annotation framework in which a query–agent pair is judged by retrieving decision principles from an evolving skill bank, running a two-round support-versus-veto debate under judge guidance, extracting a structured verdict, and feeding recurring disagreements back into the skill bank. The traces then train a routing model via reasoning-SFT and GRPO.
Why is capability-bound supervision needed for query-to-agent matching?
The paper argues that query-to-agent matching fails when a query looks topically relevant but the candidate agent lacks the tools, authority, real-time access, or service boundary to actually execute (e.g., flight rebooking offered by a consultation-only assistant). It therefore supervises the capability-critical decision process — evidence decomposition, debate, verdict — rather than a final relevance label, and keeps a separate 'grey-zone' category for semantically related but capability-invalid pairs.
What results does Debate-to-Skill report?
On an industrial benchmark: 91.6% Main Test accuracy vs 81.3% for direct-label SFT; grey-zone F1 rises from 45.8 to 64.9 and grey-zone over-acceptance falls from 33.7% to 16.9% while positive miss rate also improves (9.8%→7.4%). A four-week online A/B test (10% traffic) reports relative lifts of +15.79% card impressions, +7.63% card CTR, and +6.12% redirect CTR, all at p<0.05.
Is the Debate-to-Skill code or dataset public?
No. All data are proprietary anonymized industrial traffic (170K training pairs, 3,000 human-labeled test examples), no code or repository is linked, and the authors note that no public benchmark contains their capability-invalid 'grey-zone' category. The paper's pseudocode appendix is referenced with a broken '??'.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 are embedded as flattened caption text in the extraction, so diagram-only details (arrow structure, box layout) are approximate.
- The provided text ends cleanly after the reference list — no truncation detected. The pseudocode gap is the paper's own unresolved 'Appendix ??' reference, not an extraction artifact.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.