From Task Success to Productive Success: Evaluating Human-AI Collaboration by Quality and Cost
Saki Imai, Mert İnan, Malihe Alikhani
cs.CL · cs.AI · cs.HC
Quick verdict
Introduces Pz=z(Q)−z(C) to score human-AI collaboration; correlational evidence across 4 tasks, robustness-checked; useful framework with judge-bias caveats.
Summary
The paper argues that task success alone hides interaction cost and defines collaborative productivity as Pz = z(Q) − z(C), where Q is LLM-as-judge rubric quality of the final artifact and C is weighted token cost (user tokens + 0.5×agent tokens), z-scored within task. Grounded in Clark's least collaborative effort principle and PARADISE, it is validated on two datasets spanning four tasks: 184 CoGym sessions (travel planning, tabular analysis, related work synthesis) and a new 42-session visualization study with GPT-5.1 over WildChat-1M. Findings: (1) same-quality sessions differ up to 70.6× in interaction cost (travel planning, 854 vs 60,324 tokens); (2) quality-cost Spearman correlations range from +0.55 (related work) to −0.48 (visualization), so no universal preference for shorter interaction; (3) of six subjective measures only CoGym satisfaction penalizes cost controlling for quality (partial ρ=−0.24), while visualization confidence rises with cost (+0.41); (4) top-quartile sessions show agent probing on 44% vs 27% of agent turns (d=+0.73) and less user repair (d=−0.58). Limitations: correlational design, token counts as effort proxy, four tasks only, shared judge/annotator model, and uncorrected multiple comparisons. Code and data are released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Sessions with identical quality ratings can differ by up to 70.6× in interaction cost (travel planning: 854 vs 60,324 tokens).
- The quality-cost relationship is task-dependent, ranging from ρ=+0.55 (related work) to ρ=−0.48 (visualization), so no universal preference for shorter interaction is justified.
- Subjective ratings are unreliable proxies for productivity: only CoGym satisfaction penalizes cost controlling for quality (partial ρ=−0.24), and visualization confidence positively correlates with cost (ρ=+0.41).
- Productive sessions shift grounding labor to the agent: agent probing occupies 44% vs 27% of agent turns in top vs bottom Pz quartiles (d=+0.73), while user repair, clarification, and probing are elevated in unproductive sessions.
- The Pz = z(Q) − z(C) index is robust to cost operationalization choices, with productivity rank correlations ≥.70 across all tested cost definitions.
Strengths
Theory-grounded, minimal framework
Pz = z(Q) − z(C) (§3.1) cleanly operationalizes least collaborative effort (Clark & Wilkes-Gibbs) and PARADISE (Walker et al. 1997); it is transparent, cheap to adopt, and deliberately decouples judge-based quality from user Likert ratings to avoid circularity (§3.2, citing Podsakoff et al.).
Unusual robustness testing for a small-n study
Appendix F shows productivity rankings are stable across five agent-token weights λ∈{0.5–2.0} (Spearman .77–1.00 CoGym, .93–1.00 visualization) and five alternative cost definitions (all ≥.70 CoGym, ≥.91 viz), with the default cost nearly identical to total tokens (ρ=.99).
Validated quality signal and fully published rubrics
New full-trajectory dataset
Task-dependence is the actionable contribution
Weaknesses
Correlational quartile contrasts with length confounds
Shared-model circularity
Multiplicity and interval reporting gaps
Sample-size inconsistencies
Promised weighting sensitivity missing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper (arXiv 2609.21117; cs.CL/cs.AI/cs.HC) targets a real gap: outcome-level evaluations of human-AI collaboration (task success, elapsed time, economic value) treat the interaction as a black box. It proposes evaluating sessions by collaborative productivity, operationalized as Pz(i) = z(Qi) − z(Ci) (§3.1), where Q is an LLM-as-judge 1–5 rubric score of the final artifact and C = user tokens + 0.5×agent tokens, both z-scored within task. The framing synthesizes the least collaborative effort principle (Clark & Wilkes-Gibbs 1986), Clark & Brennan's cost decomposition, and PARADISE (Walker et al. 1997). Four headline claims: (1) identical-quality sessions differ up to 70.6× in cost; (2) the quality-cost relationship is task-dependent; (3) subjective ratings do not track productivity; (4) productive sessions shift grounding labor to the agent. Claims are consistently scoped as correlational, and the Limitations section is candid.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Pz productivity metric in this paper?
Pz = z(Q) − z(C): the within-task z-scored outcome quality (an LLM judge's rubric score of the final artifact) minus the z-scored interaction cost (user tokens + 0.5×agent tokens). It ranks collaboration sessions by how much quality was achieved relative to the interaction effort the user had to spend.
Why can identical quality still be considered unproductive?
Because two sessions can earn the same rubric score while one took vastly more interaction — in the paper's travel planning data, top-quality sessions range from 854 to 60,324 tokens (a 70.6× difference). Task success alone hides that effort gap; the expensive session is a 'costly success' rather than a productive one.
Did users notice when an interaction was costly?
Mostly no. Of six subjective measures across two datasets, only CoGym satisfaction dropped as cost rose when quality was held fixed (partial correlation −0.24). Visualization confidence actually increased with cost (+0.41), suggesting users may read more interaction as more effort invested rather than as a worse experience.
What should agent designers take away from this paper?
Productive sessions had agents probing and clarifying early (44% vs 27% of agent turns in productive vs unproductive quartiles), with users confirming rather than repairing. The design implication is to front-load clarification in the agent instead of leaving repair, re-explanation, and probing work to the user — though the evidence is correlational, so an interventional test is still needed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2/3/5–7 could only be partially reconstructed from extracted axis labels; visual distributions (point spreads, error bars) could not be inspected directly.
- Numeric matrices inside Figures 6–7 appear garbled in the extraction; values were read from surrounding prose where possible, so individual correlation cells should be treated as approximate.
- The public code/data repository (footnote 1) could not be verified from the paper text alone; reproducibility assessment assumes the released data includes the session-level logs.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.