Are LLMs Good Financial User Simulators? A Preliminary Study
Jiajie He, Jiangyuan Hong, Dongling Ni, Wenjin Liu, Xintong Chen
cs.AI · cs.CY · cs.HC
Quick verdict
Six frontier LLMs score below an always-hold baseline when simulating individual next-day paper trades; an honest, useful pilot with thin statistics.
Summary
This pilot study introduces AInvestor, a benchmark built from 120 volunteers who traded non-redeemable virtual funds for four months on a controlled paper-trading platform (230K+ interactions). Given only point-in-time information — an investor profile, portfolio state, long-term history, seven-day recent behavior, and optionally market context — an LLM must predict each participant's next-trading-day action, traded securities, and transaction quantity under a rolling protocol. Six frontier models (GPT-5.6 Sol/Luna, GPT-5.5, Gemini-3-Flash, DeepSeek-V4, GLM-5.2) were evaluated on action, ticker, and quantity metrics. The central finding is negative: every model scores below the 48.27% always-hold baseline (best: 47.53%), with sell recall between 0% and 6.57%. The authors diagnose systematic 'behavioral compression' — hold overproduction, single-ticker simplification of multi-transaction days, and widening cumulative inaction — and show that models largely ignore supplied market context, reproducing about a tenth of the real 14-point swing in trading rate across market fifths while emitting near-constant per-user predictions. A controlled ablation finds market context raises action accuracy (28.13% to 40.63%) and ticker exact match (16.67% to 20.00%) but worsens quantity error (MAPE 66.07% to 89.95%). Limitations include absent non-LLM baselines, conditional metrics without reported case counts, an unstated ablation sample size, and an output schema that itself cannot express multi-order trading days.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- All six evaluated frontier LLMs fall below the 48.27% always-hold baseline on next-day action prediction, indicating a systematic default to inaction (Table 1).
- Point-in-time market context improves action accuracy (28.13%→40.63%) and ticker exact match (16.67%→20.00%) but degrades transaction-quantity estimation (MAPE 66.07%→89.95%) in the controlled ablation (Figure 2a).
- LLM simulators exhibit systematic behavioral compression: hold overproduction, sell underprediction (sell recall 0-6.57%), and flattening of multi-security days into single-ticker outputs (§4.4.3).
- Models largely ignore supplied market context, reproducing only about one-tenth of the real ~14-point swing in trading rate across market fifths and emitting constant per-user predictions for roughly two-thirds of users (§4.4.1).
- Trajectory/return similarity cannot validate a user simulator because inaction can inflate apparent returns by omitting the user's losing trades (§4.4.3).
Strengths
Leakage-resistant, temporally aligned protocol
Every input component in Eq. 1 must precede the prediction cutoff, predictions roll forward per trading day, and the paper explicitly separates simulator fidelity from trading-agent profitability — a distinction most financial-agent work conflates (§1, §3).
Honest negative headline result
All six models in Table 1 score below the 48.27% always-hold baseline (best 47.53%), and §4.4.1 explicitly declines to rank models given the 4.93-point spread — an unusually candid framing for a benchmark paper.
Failure-mode analysis beyond aggregate scores
Multi-level evaluation protocol
Ethically constructed data
Weaknesses
No non-LLM baselines
Conditional metrics without case counts
Unstated ablation sample; numbers inconsistent with Table 1
Benchmark schema partly causes the headline bias
Under-specified, internally inconsistent manuscript
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Review: Are LLMs Good Financial User Simulators? A Preliminary Study
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the AInvestor benchmark?
AInvestor is a pilot benchmark built from 120 volunteers who traded with non-redeemable virtual funds for four months on a controlled paper-trading platform, producing over 230,000 logged interactions. It pairs each participant's de-identified behavior and virtual portfolio with point-in-time market data, and asks a large language model to predict the participant's next-trading-day action (buy, sell, or hold), traded securities, and transaction quantity.
Are LLMs good at simulating individual investors, according to this paper?
No. All six evaluated frontier models predicted next-day actions with accuracy below a trivial always-hold baseline of 48.27%, defaulted to 'hold' for the vast majority of days, almost never predicted sells (sell recall 0-6.57%), and largely ignored the market context they were given.
Did the study use real money or real brokerage data?
No. Participants used only non-redeemable virtual funds under real-time market conditions. The authors state that no real brokerage accounts, real-money positions, or real transaction records were accessed, and that released data is de-identified with LLM-assisted PII detection plus manual review.
Does giving the model market data help it simulate investors better?
Partly. In the controlled ablation, adding point-in-time market context raised action accuracy from 28.13% to 40.63% and ticker exact match from 16.67% to 20.00%, but worsened quantity estimation (MAPE 66.07% to 89.95%). Even with market data, models reproduced only about a tenth of how real trading activity responds to market moves.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 could not be analyzed from the text extraction; the Figure 2(b) cumulative-deviation claim rests on two end-point numbers (78.26%→60.87%).
- Table 1's promised valid-conditional-case counts column is absent from the extraction, limiting interpretation of the conditional ticker and quantity metrics.
- Evaluation configuration (prompts, decoding, rolling-day count, ablation subset) is not in the extracted text; Appendix A covers only data preprocessing.
- Author name order differs between the supplied metadata ('Jiangyuan Hong') and the extracted title block ('Yuanhong Jiang'); affiliations and contributions were assumed unchanged.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.