Propose, Don't Judge: An Anytime-Valid Referee for LLM Agents That Mine Investment Factors
Bo Qu, Mingguang Chen, Licheng Wang
cs.AI · q-fin.PM · q-fin.ST
Quick verdict
Splits agent factor-mining into proposing vs a frozen anytime-valid referee; evidence is convincing but headline gains are label-sensitive; rigorous, candid.
Summary
The paper asks which parts of an LLM-driven quantitative factor pipeline an agent should control. It proposes 'governed self-evolution': agents may propose candidates, diagnose retirements, and write their own probes, but a frozen statistical referee must judge. The referee scores each factor only on post-submission daily rank-IC via betting e-processes, admits through online e-BH over 2,000 slots, and retires via e-detectors, yielding FDR control at any stopping time for any proposer (Proposition 1). A 3-controller × 4-referee factorial with common random numbers runs on a planted-truth synthetic world, a probe-authoring environment, and a ten-year CSI 500 walk-forward (540 cells). The frozen referee admits 5–11× fewer sub-threshold factors than leaky referees under a scripted proposer (2.6–4.9× under an LLM); a legitimate fixed-horizon comparator narrows but does not close the gap on a nine-family library (41.4 vs 9.8 false). The LLM beats a script on yield in all six tested settings, roughly matches a UCB bandit, and uniquely authors diagnostic probes that cut intervention regret in 3 of 6 evaluable model families. Costs are candid: a ~500-day median admission wait and certified-book net Sharpe of 0.33–0.50 versus 0.59–0.77 ungated. Limitations: one market, replay-based LLM arms, label-sensitive headline ratios (2–3× under per-family break-even), and code available only on request.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A frozen referee scoring candidates only on post-submission data controls admission FDR at every stopping time for any proposal policy satisfying four stated conditions (Proposition 1).
- Who judges sets false admissions: the frozen referee admits 5–11× fewer sub-threshold factors than three leaky referees under a scripted proposer (2.6–4.9× under an LLM), and no proposer closes the gap.
- Who proposes sets yield: the LLM beats a round-robin script in all six family×library settings, roughly matches a UCB bandit, and uniquely writes diagnostic probes that cut intervention regret in 3 of 6 evaluable model families.
- The certificate costs about 500 trading days of median wait, so the certified portfolio's net Sharpe (0.33–0.50) trails an ungated book (0.59–0.77) on CSI 500.
Strengths
Validity argument is explicit and falsifiable
Proposition 1 (§4.5) names four conditions (Gt−1-measurable stakes, post-submission scoring only, frozen library per epoch, no proposer information about the post-submission stream) and one failure mode, foretelling; a unit test reproduces the Wang et al. (2025) counterexample (E[M]=1.25) and confirms the alignment invariant removes it.
The design separates the two effects the title claims
The 3-controller × 4-referee factorial (§7.1) with common random numbers lets 'who judges' and 'who proposes' be read independently: Table 8 shows controller-invariant false admissions (0.000 vs 0.26–0.85 per submission), and Table 2 shows the proposer matters only under leaky referees (a 2–2.5× shift).
Unusually honest accounting of its own weaknesses
A legitimate, non-strawman comparator
Weaknesses
Headline ratio is label-dependent
LLM real-data results are historical replays
Retirement guarantee is practically vacuous at campaign horizon
Single market, in-sample constants, no held-out period
Closed artifacts
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks which jobs in an agentic quantitative-research loop an LLM should keep, and answers: propose, diagnose, and build instruments — but never judge. 'Governed self-evolution' pairs a freely evolving agent surface (proposals, diagnosis, self-authored probes, memory) with a frozen trust kernel (referee, library, point-in-time data, append-only log, cost meter) that the agent cannot mount, patch or intercept (§4.5, Figure 1). Four claims are staked: (C1) an anytime-valid referee whose admission FDR holds at every stopping time for any proposer; (C2) a factorial that separates the judge effect from the proposer effect; (C3) a measurement of what the LLM adds — yield over script, parity with bandit, unique probe-authoring; (C4) a ten-year CSI 500 study pricing the certificate in days and Sharpe. The paper explicitly disclaims investability: the portfolio layer is a measuring instrument.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'anytime-valid referee' mean in this paper?
The referee treats evidence about a candidate factor as betting capital (an e-process) that can be read every trading day. Because the capital is a test supermartingale, error guarantees hold at any stopping time — so an agent that keeps resubmitting candidates after seeing verdicts cannot inflate false discoveries, unlike fixed-batch t-tests or deflated Sharpe checks.
What is the practical cost of the frozen referee?
An admitted true factor waits a median of about 500 trading days before certification, and the certified portfolio's net Sharpe (0.33–0.50) trails an ungated book (0.59–0.77) on the ten-year CSI 500 walk-forward, mostly because the book sits empty during the wait.
Did the LLM beat simpler proposers at mining factors?
On realized yield it beat a round-robin script in all six tested family×library settings and roughly matched a UCB bandit. Its unique contribution was authoring its own diagnostic probes, which significantly reduced intervention regret in 3 of 6 evaluable model families and never hurt.
Is the code or data for this paper public?
No. Code, run manifests, prompts and recorded model calls are available from the corresponding author on request, and the CSI 500 market data are licensed from a commercial vendor and cannot be redistributed. Only the synthetic-world and referee procedures are straightforwardly reimplementable.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2, 4–7 appear only as extracted axis labels and captions; visual patterns (cumulative-return bands, delay densities) could not be inspected.
- A few table values differ slightly across tables for the same arm (e.g., admitted 257.0 in Table 10 vs 254.8 in Table 12 for Frozen/LLM-Gemini), suggesting different seed pools; citations here follow the table cited in each section.
- Model versions are named only by family in the text; exact model snapshots live in run manifests that were not part of the extract.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.