Designing Agentic AI Workflow Portfolios under Imperfect Selection and Compute Cost
Mojtaba Abdolmaleki, Stefanus Jasin, Boyu Wang
cs.AI
Quick verdict
Portfolio-of-workflows design for agentic AI with sharp selector-strength caps and dual-guided generation; elegant OR theory and careful experiments, but modest ABCD gains with overlapping CIs.
Summary
The paper studies how a firm should decide how many AI workflow executions to run, how to allocate them across workflow types, and how to select a final answer when a post-hoc selector chooses among candidate outputs. Selector quality is summarized by an odds-lift index whose concave Luce-shaped envelope sharply caps the value of any portfolio relative to the best single workflow (Theorem 1), yielding pre-optimization screening rules. For finite workflow pools, fixed run-size problems reduce to concave coverage: exact integer programs, LP relaxations, randomized rounding with additive certificates (loss at most c_psi,k/e), and sparse geometric grids for the endogenous run size. For non-enumerable workflow classes, a finite-dimensional dual plus a batched pricing oracle and ellipsoid method yields high-probability near-optimality without enumeration; a stochastic extension adds two-stage fresh-sample estimation. Experiments on ABCD, Schema-Guided Dialogue, and HotpotQA, with 18 initial single-call workflows and ADAS/GPT-4o-mini generated compositions, show held-out actual selector accuracy gains of 6.6, 7.5, and 25.0 points over the best singleton; generation helps on ABCD (+3.5) and HotpotQA (+24.1) but not SGD. Limitations: ABCD gains are modest with overlapping Wilson CIs and no paired tests; the deployed ADAS pricing oracle cannot certify the theoretical non-existence condition; per-task success probabilities rest on only L=5 draws.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A finite selector odds-lift bound Lambda caps any portfolio's net value at max_k [h_Lambda(A_k) - gamma*C_k]; when compute is free, the gain over the best single workflow cannot exceed (sqrt(Lambda)-1)/(sqrt(Lambda)+1), regardless of how many workflows are available (Theorem 1, Corollary 1).
- For fixed run size, portfolio design is an exact concave-coverage integer program; LP relaxation plus with-replacement rounding loses at most c_psi,k/e in accuracy units, and sparse geometric grids certify a 1/rho-approximate run size (Theorems 2-3).
- Dual task and slot prices turn workflow generation into a separation problem solvable by an ellipsoid method with a batched pricing oracle, giving high-probability near-optimality over the entire implicit workflow class without enumeration (Theorems 4-5).
- On fresh held-out tasks, portfolio optimization plus dual-guided ADAS generation raises actual selector accuracy by 6.6 (ABCD), 7.5 (SGD), and 25.0 (HotpotQA) percentage points over the best single workflow, at positive total-system net value after compute and selector costs.
Strengths
Sharp and actionable selector-strength theory
Theorem 1's odds-lift envelope h_Lambda caps portfolio value independently of pool size, with tightness explicitly constructed in the proof (D.6); Corollary 1 gives closed-form screens ((sqrt(Lambda)-1)/(sqrt(Lambda)+1) max gain; uniform-cost rule (26)) usable before any optimization.
Correct handling of endogenous run size
Proposition 1 proves the objective is non-monotone and neither submodular nor supermodular; the paper responds with exact size-conditioned IPs (Prop 2), a dyadic-grid 1/2-approximation for run size (Theorem 2), and rounding certificates with an interpretable additive loss c_psi,k/e (Theorem 3).
Careful empirical protocol
Honest domain heterogeneity and recalibration
Two-stage fresh-sample discipline
Weaknesses
Modest and statistically under-supported ABCD gains
Guarantee-to-implementation gap
Plackett-Luce mis-specification acknowledged but unquantified
Stylized cost model
Headline HotpotQA result rests on truncated appendix
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a workflow portfolio in agentic AI?
It is a set of several AI workflow executions (for example, different prompting strategies or agent pipelines) run in parallel on the same task. A selector model then inspects all candidate outputs, together with their execution traces, and picks the final answer, instead of committing in advance to the single workflow with the best average accuracy.
Does running more AI workflows always improve accuracy?
No. Each extra execution costs compute and can add plausible wrong answers that confuse the final selector. The paper proves that if the selector's odds-lift is Lambda, the accuracy gain of any portfolio over the best single workflow is capped (at most (sqrt(Lambda)-1)/(sqrt(Lambda)+1) when compute is free), and its experiments show variety pays off in some domains (ABCD, HotpotQA) but adds nothing on Schema-Guided Dialogue.
How much did the proposed approach improve accuracy in the experiments?
On fresh held-out tasks, actual selector accuracy rose from 43.6% to 50.3% on ABCD customer-service routing, from 85.3% to 92.8% on Schema-Guided Dialogue, and from 30.3% to 55.3% on HotpotQA, compared with the best single workflow. Dual-guided workflow generation contributed 3.5 points on ABCD and 24.1 points on HotpotQA, and nothing on Schema-Guided Dialogue.
What is the odds-lift index for a selector?
It is a single number summarizing selector quality: the maximum, over possible fractions of correct candidates, of the ratio between the selector's odds of returning a correct answer and the odds under random picking. A value of 1 means the selector is no better than chance; larger values mean a stronger selector, and the index directly caps how much value any multi-workflow portfolio can add.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extracted text is truncated mid-proof in Appendix D.11; Appendices E, F (Algorithm 2), and G.1-G.2 (full SGD and HotpotQA experiments) are absent, so those proofs and secondary experiments could not be checked.
- Figures were extracted as text only; the visual fit of the Plackett-Luce curve in Figure 5 could not be independently assessed.
- The review relies solely on the embedded text; the arXiv record (2609.18126) could not be cross-checked, and no code or data availability statement appears in the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.