FinCUABuild: Can Agents Build Reliable Benchmarks for Dynamic Financial Computer Use?
Jingpu Yang, Fengxian Ji, Jinri Guo, Tianhao Li, Qian Jiang, Fan Zhang, Min Peng, Qianqian Xie, et al.
cs.AI
Quick verdict
Proposes agentic building of financial CUA benchmarks (31.3% vs 1.3-8.3% pass rate); promising, but no significance tests and gates may favor its own agent.
Summary
The paper asks whether LLM agents can autonomously construct evaluation tasks for dynamic financial computer-use agents (CUAs), where manual benchmark building such as Finch required 700+ expert hours. It contributes FinCUABuildBench: 576 preregistered construction slots spanning 24 financial workflow categories (192 WorkflowCores) under three runtime-dynamic profiles — state updates, tool failures, policy changes — with standardized inputs, budgets, and output contracts plus a strict qualification gate combining LLM-judge acceptance, traceability, execution/replay, verifier accuracy, dynamic validity, capability match, and non-duplication. It also contributes FinCUABuildAgent, an eleven-agent pipeline whose controller routes candidates through discovery, grounding, capability compilation, environment composition, an independent oracle, reference solving, and red-team testing with targeted repair. Under matched budgets, the system reaches 31.3% strict qualification on MiniMax-M3 (41.9% on Qwen3.5) versus 1.3–8.3% for adapted AutoBencher, BENCHAGENTS, and Benchmark Agent baselines; tool-augmented prompting reaches 17.5%. On 144 qualified tasks, downstream agents show near-saturated static completion (0.62–0.66 for native frameworks) but near-floor dynamic completion (0.004–0.043), and held-out transfer on 90 unseen requests is only 13.3%. Main limitations: modest absolute yields (human acceptance 33.4%), no statistical significance tests or confidence intervals, an undisclosed judge model for L-Acc/CapMatch, and qualification gates co-designed with the proposed system that may structurally disadvantage adapted baselines, which collapse on traceability (5.0–11.1%).
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- FinCUABuildBench is the first unified benchmark for evaluating financial CUA task construction, with 576 slots spanning 24 workflow categories × 8 WorkflowCores × 3 runtime-dynamic profiles.
- Under matched inputs, budgets, and outputs, FinCUABuildAgent achieves 31.3% strict qualification on MiniMax-M3 (41.9% on Qwen3.5) versus 1.3–8.3% for adapted benchmark-generation baselines.
- Tasks built by the system are diagnostic: evaluated agents show near-saturated static completion (0.62–0.66) but near-floor dynamic completion (0.004–0.043), isolating runtime-adaptation failure.
- Component ablations attribute performance to staged multi-agent coordination, the independent oracle, and red-team repair.
- Held-out transfer on 90 unseen requests reaches only 13.3% strict qualification, indicating partial, within-taxonomy generalization.
Strengths
Unusually formal qualification contract
Appendix metric definitions (Eqs. 8–27) specify every gate: mutation-based verifier accuracy (VMR/VAA, Eqs. 14–17), deterministic replay products in Exec (Eq. 11), structure-aware deduplication (Eqs. 22–24), and a minimum-dimension judge constraint (Eq. 9) that prevents a strong average from hiding a critical failure.
Honest disclosure of weak spots
Held-out strict qualification of 13.3% (Table 3), the P3 policy-change gap (46.7% Exec vs 6.7% Qual), and 'Dynamic < Static in 16 of 17 configurations' are reported rather than buried; the conclusion names qualification yield, transfer, licensing, and verifier maintenance as open problems.
Genuinely diagnostic downstream results
Broad matched-baseline and ablation coverage
Weaknesses
No statistical inference anywhere
Circularity between gates and proposed system
Undisclosed LLM-judge configuration
Modest absolute yield and unaccounted human cost
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper makes benchmark construction itself the object of evaluation: can LLM-driven agents autonomously produce complete, replayable evaluation episodes for dynamic financial computer-use agents (CUAs)? Motivated by manual cost (Finch's 700+ expert hours), it claims three contributions: **FinCUABuildBench**, 576 preregistered construction slots = 24 workflow categories × 8 WorkflowCores × 3 runtime-dynamic profiles (P1 state update, P2 tool failure, P3 policy change); a **strict qualification contract** (LLM-judge acceptance, traceability, execution/replay, mutation-tested verifier accuracy, dynamic validity, capability match, non-duplication — Eq. 25); and **FinCUABuildAgent**, an eleven-agent generate–audit–repair pipeline (Fig. 1) with an independent oracle, reference solver, and red-team agent. Headline claim: 31.3% strict qualification on MiniMax-M3 (41.9% on Qwen3.5) vs 1.3–8.3% for adapted AutoBencher/BENCHAGENTS/Benchmark Agent, plus downstream diagnostic value on 144 qualified tasks.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is FinCUABuildBench?
A benchmark with 576 construction requests that tests whether AI agents can automatically build evaluation tasks for dynamic financial computer-use agents. It covers 24 financial workflow categories × 8 workflow cores × 3 runtime-change types (state updates, tool failures, policy changes), and judges each generated task package with strict execution, verifier, and quality gates.
How well does FinCUABuildAgent perform?
Its generated task packages pass the strict qualification gate for 31.3% of slots with the MiniMax-M3 backbone (up to 41.9% with Qwen3.5), versus 1.3–8.3% for adapted benchmark-generation systems like AutoBencher. On 90 construction requests from unseen financial scenarios, the pass rate drops to 13.3%.
What do the generated tasks reveal about existing AI agents?
On 144 qualified tasks, mainstream agent frameworks complete the static deliverables well (0.62–0.66 completion) but almost completely fail when an event changes the task after execution starts (dynamic completion 0.004–0.043), showing runtime adaptation is the key weakness. Several financial agents also committed safety violations at 0.23–0.40 rates.
Is code or data available for this paper?
The paper links a GitHub repository (github.com/FengxianJi/FinCUABuild). Note that core evaluator components — the oracle, hidden verifiers, event controller, and scoring logic — are kept private to the evaluator by design, and episodes rely on versioned licensed financial sources, so full independent replication of the headline numbers requires the authors' evaluation stack.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 are embedded as jumbled layout text (agent/tool labels out of reading order), so figure-only details such as exact pipeline arrows could not be verified; all numbers cited in this review come from tables and body text, which extracted cleanly.
- Numeric gate thresholds (τH, τmin, τL, τV, τdup) and the count of construction runs are genuinely absent from the entire provided text including the appendix; related judgments reflect that absence, not extraction loss.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.