Human-AI-Powered Hypothesis Testing: Cost-Aware Selective AI Scoring and Sequential Human Escalation
Dae Woong, Ham, Xuejun Zhao, Stefanus Jasin, Fenghua Yang
cs.AI · cs.IT · stat.ME
Quick verdict
SCALE: cost-minimal AI+human hypothesis testing, finite-sample valid and first-order optimal; rigorous theory, evidence limited to synthetic simulations.
Summary
The paper studies hypothesis testing for a population proportion when an inexpensive but fallible AI judge (e.g., an LLM) can supplement costly human gold-standard labels. The decision maker buys a fixed pool of items upfront, then adaptively queries AI, queries humans directly, escalates AI-scored items to humans after seeing reports, or stops; cost is minimized in the worst case subject to type-I ≤ α and type-II ≤ β. The authors derive an information-theoretic lower bound built on an exact adaptive KL decomposition and a concave, report-dependent escalation frontier, then propose SCALE, an SPRT-style sequential policy with direction-dependent randomized sensing and a pre-committed Neyman–Pearson fallback that guarantees finite-sample validity (Theorem 3) and first-order cost optimality as errors vanish (Theorem 4). A guarded plug-in extension estimates the AI-output model from paired pilot data, retaining exact conditional validity (Theorem 5) and first-order optimality even with the pilot charged (Corollary 4). Simulations (p0=0.10, p1=0.20; binary AI report with 0.80 sensitivity/specificity; 10^6 Monte Carlo replications) show savings up to 36.37% (human-cost sweep at α=β=0.01) and 39.16% (error-target sweep at cH=10) over the cheaper single-source baseline. Main limitations: purely synthetic experiments with no real LLM data, simple binary hypotheses with known parameters, a finite report alphabet, i.i.d. items, and no released code.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Any feasible human–AI testing policy incurs worst-case expected cost at least LB(α,β), a computable bound combining a full-label sample-size floor with a convex allocation program over per-item KL quantities (Theorem 2).
- SCALE attains the prescribed type-I/II errors at every finite sample size (Theorem 3) and its worst-case expected cost matches the lower bound to first order as the errors vanish (Theorem 4).
- Estimating the AI-output model from a paired pilot with guardrails preserves exact conditional validity (Theorem 5) and first-order optimality even when the pilot's full cost is charged (Theorem 6, Corollary 4).
- In the paper's benchmark, SCALE cuts cost by up to 36.37% (varying cH ∈ {2,…,50} at α=β=0.01) and 39.16% (varying α=β ∈ {0.05, 0.01, 0.0001} at cH=10) relative to the cheaper of Human-only or AI-only baselines, with the largest gains in the intermediate regime.
Strengths
Exact finite-sample guarantees by construction
Theorem 3 (and Theorem 5 for the pilot-calibrated variant) delivers type-I/II control at every finite k via a martingale/optional-stopping argument for boundary routes (Lemma EC.5) plus a pre-committed Neyman–Pearson fallback on fixed index sets, so validity never hinges on asymptotics.
Novel, clean formalization of the nested AI→human information structure
Theorem 1's adaptive KL decomposition splits transcript divergence into AI-only, human-only, and report-conditional escalation terms d(h)(r)=kl(q1(r)‖q0(r)); the concave frontier Ψh (Lemma 4) gets an LP-dual interpretation as maximal extractable escalation information at rate s and is computed by a fractional-knapsack rule (Proposition 1) — the conceptual bridge between two-phase sampling and active testing.
Honest cost accounting with explicit rates
Unusually careful numerics for a theory paper
Weaknesses
Empirical evidence is one synthetic instance
Restrictive model
i.i.d. assumption is fragile in practice
Tuning and finite-sample gap unquantified
No code or data availability statement
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses a decision-theoretic question at the intersection of statistics and operations management: when the goal is a population-level hypothesis test (H0: p = p0 vs H1: p = p1 for the proportion of positive items), how should one combine cheap, noisy AI judgments with costly human gold-standard review to minimize cost while provably controlling type-I and type-II errors? The decision maker buys a fixed item pool upfront and then adaptively chooses among AI queries, direct human queries, post-report human escalation, and stopping. Four claims are made: (i) a computable information-theoretic lower bound LB(α,β) on any feasible policy's worst-case expected cost (Theorem 2); (ii) the SCALE policy is finite-sample feasible (Theorem 3) and first-order optimal as errors vanish (Theorem 4), with explicit rates (Corollary 1); (iii) a guarded pilot-calibrated plug-in retains exact conditional validity (Theorem 5, Corollary 2) and first-order optimality with the pilot charged (Theorem 6, Corollary 4); (iv) simulations show savings up to 36.37%/39.16% over single-source baselines (§5.4). All four are internally supported in the extracted text; the asymptotic claims are restricted to the balanced small-error regime (Assumption 3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SCALE method in this paper?
SCALE is a sequential decision policy for hypothesis testing that combines cheap AI judgments (e.g., from an LLM judge) with selective, costly human verification. It tracks a likelihood ratio as evidence accumulates, decides per item whether to query the AI, ask a human directly, or escalate an AI-scored item to a human based on its report, and stops when evidence is strong enough; a pre-committed Neyman–Pearson fallback guarantees valid type-I/type-II error control at any finite sample size.
How much cheaper is AI-assisted hypothesis testing than using only humans?
In the paper's synthetic benchmark (population proportion 0.10 vs 0.20, AI judge with 80% sensitivity and specificity), SCALE's total cost was up to 36.37% lower than the cheaper of Human-only or AI-only testing when human review cost varied from 2 to 50 (at α=β=0.01), and up to 39.16% lower when the error targets were tightened to 0.0001 (at cH=10). Savings are largest in the intermediate regime where AI is informative but human verification is still worth buying; they shrink when one source clearly dominates.
Was the method tested on real LLM data?
No. All experiments simulate the AI judge as a binary reporter with fixed 80% sensitivity/specificity under known probability models. The theory covers a known or pilot-estimated AI-output model, but the numerical section contains no real LLM outputs, continuous scores, or real-world datasets — a main limitation for practical adoption.
Is the statistical test guaranteed to be valid?
Yes, by construction: SCALE controls type-I and type-II errors at any finite sample size (Theorem 3), using martingale-based boundary analysis plus an exact Neyman–Pearson fallback if the item pool runs out. A version that estimates the AI's error model from paired pilot data remains exactly valid conditionally on a good pilot (Theorem 5); averaged over pilots, the error levels increase by at most a small slack δm (Corollary 2).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extraction omits several sections (parts of §3.4–§4.4, §5.1, §6.1–6.2, and portions of the appendix); judgments about those parts rest on statements repeated elsewhere in the text.
- Figures are available only as embedded axis-label values; visual details of Figures 1–2 could not be analyzed.
- Code/data availability was assessed from the extracted text only; a release statement could exist in unextracted portions.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.