Selection-Aware Stress Testing for Interactive Agents
Yang Xu, Chenang Li, Jiefu Zhang, Haixiang Sun, Zhou Li, Vaneet Aggarwal
cs.LG · stat.AP · stat.ML
Quick verdict
SASST: a discovery/confirmation protocol with validity proof for agent-eval stress testing; rigorous and candid, but its own agent studies confirm nothing.
Summary
The paper tackles double-dipping in agent evaluation: teams pick a winning workflow on a benchmark, then search the same outcomes for task types where its advantage weakens. SASST (Selection-Aware Semantic Stress Testing) splits tasks into discovery and confirmation clusters, learns a task-reweighting rule from pre-execution features on discovery data, freezes it, and re-evaluates the same paired workflow comparison on held-out confirmation tasks with joint simultaneous bounds over all planned claims. Rules must pass density, effective-sample-size, geometry-radius, and stability checks, and the protocol can return no claim. Theorem 1 proves conditional asymptotic validity of the confirmation bounds under cluster assumptions. Calibration studies show learned-rule coverage of 94.75-95.30% for M>=1000; a forty-cluster audit finds Gaussian undercoverage (90.55-93.50%) and conservative Bonferroni-t coverage (96.40-97.55%), and a 200-cluster positive control detects a planted fragility end-to-end in 64.6% of trials. In 480-episode tau-bench studies with Qwen3-8B and Qwen2.5-7B, however, zero claims were confirmed: a 3.75-point discovery gain for Plan+Verifier vanished on confirmation, and both models' stress rules were unstable (maximum stability 0.37 and 0.27). Main limitations: a single reused task pool, low power in the forty-cluster regime (3.9-11% power at ESS 20 in Table 7), kernel geometry failures, the same model serving as actor and user simulator, and no stated code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Evaluations that select a workflow winner and then search the same outcomes for fragility are doubly selection-contaminated, and a discovery/confirmation cluster split with a frozen pre-execution reweighting rule removes this contamination.
- Theorem 1: once discovery fixes the rules, confirmation bounds from a cluster Gaussian multiplier bootstrap are conditionally asymptotically jointly valid over all planned claims.
- At forty clusters the Gaussian multiplier bootstrap undercovers (familywise 90.55-93.50%) while one-sided Bonferroni-t is conservative (96.40-97.55%) across four audited null settings.
- In 480-episode tau-bench studies, a 3.75-point discovery gain for Plan+Verifier over ReAct (Qwen3-8B) vanished on confirmation, and no claim was confirmed for either model (max rule stability 0.37 and 0.27).
- The protocol's five statuses separate supported, underpowered, infeasible, and unstable findings, so non-confirmation is informative but is not equivalence.
Strengths
Clean formalization of a real double-dipping problem
Sec. 1-2 name the two selection events (workflow winner plus fragility search on the same outcomes) and solve them with a whole-cluster discovery/confirmation split where the reweighting rule is frozen before confirmation outcomes are read (Algorithm 1, Appendix C.1).
Real theory with complete proofs
Theorem 1 gives conditional asymptotic joint validity over the finite slot family; Appendix B proves a weighted-ratio expansion (Lemma 1), uniform cluster linearization including splitwise selection effects (Lemma 2), and the multiplier bootstrap via Cramer-Wold plus Chernozhukov et al. (2013) style arguments.
Unusually candid self-auditing
Small-sample behavior quantified rather than hidden
Operationally complete decision protocol
Weaknesses
Flagship experiments demonstrate no confirmed finding
Near-guaranteed non-confirmation by design power
Geometry machinery fails its own audit
Narrow empirical base
Safeguard and positive control lean on limited simulations
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a genuine methodological gap in LLM-agent evaluation: a team selects a winning workflow on a benchmark, then scans the *same* outcomes for task types where the advantage weakens — two selection events on one dataset. SASST (Selection-Aware Semantic Stress Testing, Sec. 2) formalizes the remedy: split whole task clusters into discovery and confirmation sets, learn a task-reweighting rule g from pre-execution features only, freeze it, and recompute the identical paired workflow contrast on confirmation clusters with joint simultaneous bounds over ten predeclared slots. Central claims: (i) Theorem 1 — conditional asymptotic joint validity of the confirmation bounds after discovery, proved in Appendix B via a weighted-ratio expansion (Lemma 1), uniform cluster linearization that accounts for both held-out evaluation and splitwise selection effects (Lemma 2), and a Gaussian multiplier bootstrap; (ii) a forty-cluster audit showing Gaussian undercoverage and conservative Bonferroni-t bounds; (iii) two 480-episode tau-bench studies in which a +3.75 pp discovery gain vanished on confirmation. The protocol explicitly returns one of five statuses, including no claim.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SASST in LLM agent evaluation?
SASST (Selection-Aware Semantic Stress Testing) is a statistical protocol for evaluating AI-agent workflows. It first learns a task-reweighting rule on a discovery split of a benchmark using only pre-execution task features, freezes that rule, and then re-tests the same paired workflow comparison on a separate confirmation split with joint confidence bounds over all planned claims, so conclusions are not contaminated by searching the same data twice.
What did the paper find when applying SASST to real agents?
In two 480-episode tau-bench studies (Qwen3-8B and Qwen2.5-7B, each running ReAct, Plan+Verifier, and Confirmation Gate workflows), no claim was confirmed. A 3.75-percentage-point discovery advantage for Plan+Verifier disappeared on the confirmation split, and the learned stress rules were too unstable to reuse (maximum stability 0.37 and 0.27 across the two models).
Why does the paper report mostly non-confirmations instead of findings?
The protocol is designed to return no claim rather than overstate. At the studied scale (40 confirmation task clusters, stressed effective sample size near 20), the authors' own power analysis shows only about 4-11% power, so effects of practical size could rarely be confirmed. They present this as an honest limitation and call for fresh, larger task pools with power plans based on stressed effective sample size.
Is code or data available for SASST?
The paper describes an internal audit with versioned corrected artifacts and provenance tracking (Appendix D.9), but in the provided text it does not state a public code, data, or trajectory release, so independent verification currently depends on reimplementing the protocol from Appendix C.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.