Beyond "AI Helps Humans": Decision-Targeted Evaluation Design for Human-Agent Teams in the Agentic Era
cs.AI · cs.HC
Quick verdict
Proposes TEAM-DESIGN, an optimal replay-allocation rule testing human-AI workflows against both baselines; theory solid, gains real but regime-dependent.
Summary
The paper asks when a deployed human–AI workflow is worth keeping: only if it beats both the human alone and the agent alone, yet neither baseline outcome is observed once the workflow runs. Under a fixed replay budget, TEAM-DESIGN assigns every task two replay probabilities, one per baseline, in closed form as the exact solution of a convex minimax program: replay more where the missing baseline outcome is hard to predict and where that comparison sits closer to its threshold, less where replay is expensive. Three guarantees cover first-order sample sizing, exact optimality of the square-root rule via KKT conditions, and asymptotic validity of the beats-both decision after pilot-based random replay, needing no multiplicity correction through the intersection–union structure. Evidence spans a reanalysis of 8,723 reads across 6 clinical settings in which no assisted clinician beats both alternatives; synthetic Bernoulli designs where the beats-both rate is 77.1% versus 65.1% for Neyman when one comparison is harder, but 39.1% versus 44.4% when balanced; a semi-synthetic chest X-ray design attaining the lowest RMSE on the harder gain (10.4 versus 15.7 ×10⁻³ at one-fifth of full replay); and a Gaussian study where it beats decision-targeted Bayesian and parameter-EIG designs (76.3% versus 75.2% and 59.5% power). Main limitations: the advantage vanishes or reverses when comparisons are balanced, the semi-synthetic setup violates the theorem's independence assumptions, and no task-level agentic coding data is analyzed.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 4 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Keeping a deployed human–AI workflow requires showing it beats both the human-only and agent-only baselines, and TEAM-DESIGN is claimed as the first evaluation design that allocates a fixed replay budget to target exactly this two-comparison decision.
- The square-root replay rule of Eq. (10) is the exact solution of a convex minimax program via KKT conditions (Theorem 4.2), not a heuristic.
- Drawing replays at random from pilot-recorded probabilities preserves asymptotic validity of the beats-both decision at level α with no multiplicity correction (Theorem 4.3, intersection–union test).
- Across 6 clinical settings and 8,723 reads, no assisted clinician beats both alternatives, while a coding benchmark shows the opposite ordering in aggregate.
- TEAM-DESIGN outperforms Neyman, uniform, decision-targeted Bayesian and parameter-EIG comparators only when one comparison is clearly harder, and underperforms Neyman when comparisons are balanced.
Strengths
Exact optimality, not heuristics
Theorem 4.2 proves via KKT conditions (Appendix A.3) that the square-root rule (10) is the exact solution of the convex minimax program (4), with complementary slackness characterizing which comparison receives budget — a rarity in evaluation-design work.
Validity under adaptive, pilot-chosen designs
Theorem 4.3 gives a conditional CLT given the pilot (Appendix A.4, Lindeberg–Feller for the design-dependent term), and the intersection–union structure removes multiplicity correction; the measured boundary-null false-claim rate is 3.6% against a nominal 5%, and 3.7% with a deliberately reversed pilot (Table 2).
Unusually candid negative results
Rigorous, cheap, reproducible simulation protocol
Real-data grounding with measured costs
Weaknesses
Narrow regime of benefit
External-validity gap with the framing
Theorem-to-evidence mismatch on the closest-to-real experiment
Sensitivity to tuning and pilot quality
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper formalizes a deployment decision: a human–AI workflow is kept only if it beats **both** the human-only baseline (∆H) and the agent-only baseline (∆A) beyond required improvements δj (Eq. 2). Once deployed, neither baseline outcome is observed, and replays cost expert time or compute; under a per-task budget B (Eq. 3), the design question is which tasks receive which replay. TEAM-DESIGN claims to be the first decision-targeted evaluation design for this problem: two replay probabilities per task, given in closed form as the exact solution of a convex minimax program (Eq. 4) whose objective is the larger of the two comparison difficulties. Three theoretical claims are made: a first-order sample-size argument (Prop. 4.1), exact optimality of the square-root rule (Thm. 4.2), and validity of the final beats-both decision under pilot-chosen random replay without multiplicity correction (Thm. 4.3, via intersection–union testing). Empirical claims rest on a clinical reanalysis, synthetic Bernoulli/Gaussian designs, a semi-synthetic chest X-ray design, and a Gaussian acquisition study.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is TEAM-DESIGN?
A replay-allocation rule for evaluating human–AI workflows: when only the joint workflow outcome is observed, it assigns each task two probabilities — one for a human-only replay, one for an agent-only replay — chosen in closed form so that, under a fixed budget, the harder of the two 'beats-both' comparisons is settled most efficiently. Replay probability rises where the missing baseline outcome is hard to predict and the comparison is close to its threshold, and falls where replay is expensive.
Did human–AI collaboration beat both humans and AI alone in the clinical studies?
No. Reanalyzing two reader studies (chest X-ray and melanoma) across 6 settings and 8,723 reads, the assisted clinician beat the clinician alone in every setting but lost to the model alone in every setting — 0 of 6 workflows met the beats-both standard.
When does TEAM-DESIGN beat Neyman allocation, and when does it lose?
It wins when one comparison clearly needs more evidence: 77.1% vs 65.1% beats-both decision rate in the paper's imbalanced synthetic scenario. It loses when the two comparisons are similarly difficult (39.1% vs 44.4%) and its advantage nearly closes at a 10:1 replay cost ratio (37.6% vs 37.0%).
Does selectively choosing which tasks to replay bias the conclusion?
The paper argues no: because replays are randomized from recorded probabilities and the pilot is independent of the evaluation pool, the gain estimator is unbiased regardless of predictor quality, and the false-claim rate stays at or below the nominal 5% (measured 3.6% at the boundary null, 3.7% with a deliberately misled pilot).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 appear in the extraction only as axis-label text; in-panel values could not be verified from the figure images, though all headline numbers cited here come from Tables 1–5 and body text, which are fully present.
- The abstract in the task metadata ends mid-sentence, but the embedded full text contains the complete abstract and all sections through Appendix C and references — the source text itself is not truncated.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.