SAGE: State-Grounded, Abstention-Aware Evaluation of Task-Oriented Dialogue Agents
Rayan Khoury, Shih-Yao Lin, Pratyush Mishra
cs.AI · cs.CL · cs.LG
Quick verdict
State-grounded symbolic/NLI cascade judges dialogue turns at zero LLM cost, tying GPT-4.1 judges; honest study, but failures are injected and partly circular.
Summary
SAGE is a framework for evaluating task-oriented dialogue agents by judging whether each turn correctly advances an underlying workflow state rather than scoring replies holistically. It compiles a workflow specification and the per-turn state diff into atomic, schema-grounded criteria covering four failure classes (ignored user value, wrong next field, premature confirmation, unsupported value), then routes each criterion through a cascade: deterministic symbolic rules (S1), a confidence-gated DeBERTa-v3 NLI/encoder verifier that abstains below a margin threshold (S2), and an optional focused LLM call (S3). The recommended SAGE-Core configuration (S1+S2) decides 81–91% of criteria at zero paid API cost. Across four injected-failure slices built from MultiWOZ 2.4, Schema-Guided Dialogue, and ABCD, no evaluated GPT-4.1-family judge—including a state-aware G-Eval and cheaper mini variants—significantly exceeds SAGE-Core; SAGE wins significantly only on SGD, ties on UV-rich, and its other wins are nominal and do not survive Holm–Bonferroni correction. A two-annotator human audit (n=200, κ=0.939) confirms label fidelity for premature confirmation and unsupported value, but the ignored-user-value class has weak human salience (1/30 endorsed), and on overall human-consensus F1 SAGE-Core trails the best judge (0.825 vs 0.904). Main limitations: all failures are injected and partly circular with the verifier's own rules, and a manually authored workflow specification is required.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SAGE-Core (compiler + symbolic S1 + NLI S2) decides 81–91% of compiled criteria with zero paid LLM calls and is not significantly exceeded by any evaluated GPT-4.1-family judge across four injected-failure slices.
- The only Holm-corrected significant win over the GPT-4.1 G-Eval judge is on SGD (ΔF1=+.075, p=.011), with a tie on UV-rich and nominal wins on MW-mixed and ABCD, at $0 versus $4.7–8.0 per 1,000 turns.
- A two-annotator audit (n=200, κ=0.939) validates injected labels for premature confirmation (1.00) and unsupported value (0.95), but shows ignored-user-value has weak human salience (1/30) and WNF is softer (0.73).
- On transcript-visible classes excluding IUV, SAGE-Core is statistically tied with the strongest LLM judge against the human consensus (0.909 vs 0.920, p=0.68).
- The optional focused-LLM tier S3 is not load-bearing: SAGE-LLM never improves F1 over SAGE-Core on any slice.
Strengths
Statistical discipline uncommon in LLM-judge papers
Paired bootstrap (10,000 resamples, seed 0) with within-resample re-selection of the strongest baseline, Holm–Bonferroni over the four pre-specified GPT-4.1 comparisons, 95% CIs in every caption, and a 7-point τ-sweep keeping UV-rich F1 within a 0.007 band (Appendix J).
Unusual self-scoping and honesty
The Discussion explicitly labels the symbolic-injection circularity ('largely inverting its own injector'), the audit exposes IUV's weak human salience (1/30) and WNF softness (0.73), and the paper distinguishes nominal from corrected significance instead of burying it.
Strong reproducibility infrastructure
Cost-quality evidence is mechanistic, not just end-to-end
The audit probes the benchmark itself
Weaknesses
All results rest on injected failures
Human-consensus deficit on the full audit
Narrow judge baseline family
Unmeasured specification burden
Small slices and per-class audit counts
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SAGE targets a narrow but important slice of dialogue evaluation: for task-oriented agents fronting structured workflows (booking, support, ticketing), it judges whether each turn correctly advances an underlying workflow state rather than scoring the reply holistically. The pipeline compiles a manually authored workflow spec W = (S, F, ≺, O) plus the per-turn state diff Δt into atomic, schema-grounded criteria spanning four failure classes — ignored user value (IUV), wrong next field (WNF), premature confirmation (PC), unsupported value (UV) — and routes each through a three-stage cascade: deterministic symbolic rules (S1), a confidence-gated DeBERTa-v3 NLI/MiniLM verifier that abstains when the entailment margin |m| < τ = 0.20 (S2), and an optional focused-LLM tier (S3). The claims are deliberately scoped: SAGE-Core (S1+S2) decides 81–91% of criteria at zero paid API cost, and no evaluated LLM-as-a-judge baseline — including a state-aware GPT-4.1 G-Eval and cheaper mini variants — significantly exceeds it on any of four slices. The paper explicitly disclaims coverage of fluency, safety, or open-ended quality.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SAGE and what problem does it solve?
SAGE is a framework for evaluating task-oriented dialogue agents turn by turn against a workflow specification, instead of scoring replies holistically with an LLM judge. It compiles the workflow spec and each turn's state diff into atomic checks (ignored user value, wrong next field, premature confirmation, unsupported value) and verifies them with deterministic rules and an NLI encoder, producing evidence-bearing verdicts that can abstain rather than guess.
Does SAGE require calling a paid LLM?
The recommended configuration, SAGE-Core, uses no paid API calls at all: it decides 81–91% of compiled criteria with deterministic symbolic rules and on-device encoders (DeBERTa-v3 NLI and MiniLM running on CPU). An optional focused-LLM tier (SAGE-LLM) exists for open-class criteria, but the paper shows it never improves F1 and recommends disabling it.
How does SAGE compare to GPT-4.1 LLM-as-a-judge baselines?
Across four injected-failure slices built from MultiWOZ 2.4, Schema-Guided Dialogue, and ABCD, no evaluated GPT-4.1-family judge significantly exceeded SAGE-Core. SAGE's only Holm-corrected significant win was on SGD (+0.075 F1, p=.011); it tied on the UV-rich slice and won nominally elsewhere, while paying $0 versus $4.7–8.0 per 1,000 turns for the G-Eval judge.
What are the paper's main limitations?
All failures are synthetically injected and partly circular with the verifier's own rules (only one natural failure was observed in the audit). A manual workflow specification is required. The ignored-user-value class has weak human agreement (1/30 audited turns endorsed), and on overall human-consensus F1 SAGE-Core trails the best LLM judge (0.825 vs 0.904), tying only when that class is excluded.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure graphics are not rendered in the text extraction; Figure 1–4 content is taken from their captions, which appear complete in the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.