FinLifeBench: Exhaustive Life-Event History and Financial-State Reconstruction from Longitudinal Banking Dialogue
Hangyeul Lee, Juyoung Oh, Jaeyong Ko, Sunmin Kim, Jaeik Park, Hyunkyu Kim, Jungmin Son, Pilsung Kang
cs.AI · cs.CL
Quick verdict
Carefully built synthetic benchmark for exhaustive longitudinal banking reconstruction; rigorous metrics, honest results, but synthetic scope limits reach.
Summary
FinLifeBench is a synthetic benchmark of 6,000 eight-turn Korean banking dialogues (20 trajectories x 300 sessions) that tests whether LLMs can maintain exhaustive customer records over longitudinal interaction. Two tasks are posed over the same cumulative dialogue: reconstructing every life-event instance with its first-establishing session (24 event types), and reconstructing a complete 34-path financial state at 20 consecutive checkpoints, with deterministic gold and distractor sessions (hard negatives, stale recalls, cancellations). Eleven frontier LLMs were evaluated under a full-context condition using exact-match, change-accuracy, and evidence metrics with trajectory-cluster bootstrap CIs. Results are sobering: event-anchor recall falls from 0.591 at 15 sessions to 0.445 at 300, driven by omissions rather than mis-localization (conditional anchor accuracy 0.866); the best EA-F1 is 0.748 (Gemini 3.1 Pro). Financial-state reconstruction peaks at GCA@15 0.470 (Claude Opus 4.8), with superseded historical values recalled only 6.2% - most outdated facts are reported as current. The two tasks correlate weakly across models (Spearman rho=0.291, unstable under leave-one-out). Main limitations: synthetic single-domain data, LLM-generated and LLM-audited gold with limited human review, a static full-context protocol that excludes memory-management systems, and Korean-only dialogues.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- No prior benchmark requires generative, schema-complete reconstruction of both a grounded life-event history and the corresponding full structured financial state at repeated checkpoints (§1, Table 1).
- Event-anchor recall declines with dialogue depth (0.591 at 15 sessions to 0.445 at 300) primarily due to outright omission of established events, not mis-localization, since conditional anchor accuracy among recovered instances is 0.866 (§6.1).
- Financial-state reconstruction fails temporal validity: superseded historical and plausibly invalidated stale values are recalled at only 6.2% and 10.4%, with ~70% predicted as current; best GCA@15 is 0.470 (Claude Opus 4.8) against an Initial Copy baseline of 0.177 (§6.2, Table 3).
- Performance on the two reconstruction tasks is only weakly rank-associated across the eleven models (Spearman rho=0.291), though the authors' own leave-one-out analysis shows this association is carried by two influential points (§7.2).
Strengths
Deterministic state-first construction
§3.1 builds personas, initial 34-path states, and event trajectories via a finite-state transition graph (precedence, cooldowns, repeat limits) before dialogue generation, yielding exhaustive gold with provenance and revision histories - rare rigor for longitudinal benchmarks.
Distractor taxonomy validated empirically, not just asserted
§3.2 defines hard negatives, consequence follow-ups, stale-recall follow-ups, and cancellation evidence; §6.1 shows the design works: hard negatives cause only 33 false positives per 10,000 exposures and cancellation anchors were selected 3 times in 27,467 predicted pairs, so failures can be attributed to coverage rather than distractor confusion.
Metric decomposition with majority-class control
Above-average statistical transparency
Release commitment with deterministic scoring
Weaknesses
Small trajectory count bounds statistical resolution
Gold rests on LLM judgment with limited human audit and no human ceiling
Static full-context protocol excludes the systems the framing targets
Single-prompt, low-reasoning protocol without sensitivity analysis
Regularity leakage and single-language confound
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FinLifeBench (arXiv 2609.01198v2) targets a gap the authors argue is untested: generative, schema-complete reconstruction of both a grounded life-event history and a full structured financial state from long longitudinal dialogue (§1, Table 1). The benchmark comprises 6,000 eight-turn Korean banking sessions - 20 persona-conditioned trajectories × 300 sessions × 20 life-event instances over 24 event types and 34 state paths (§3, Table 2). Three headline claims: (1) event-anchor recall degrades with depth (0.591 at 15 sessions to 0.445 at 300) mainly through omissions, not mis-localization; (2) state reconstruction treats superseded facts as current (best GCA@15 0.470, Claude Opus 4.8); (3) the two tasks are only weakly rank-associated. Claims (1) and (2) are well supported by the reported evidence; (3) is fragile.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is FinLifeBench?
A benchmark of 6,000 synthetic eight-turn Korean banking dialogues (20 trajectories of 300 sessions each) that tests whether large language models can reconstruct a customer's complete life-event history - every event plus the session where it first became evident - and a full 34-field financial state at 20 successive checkpoints of the same cumulative conversation.
What were the main results across the eleven LLMs?
Best life-event EA-F1 was 0.748 (Gemini 3.1 Pro), with event-anchor recall falling from 0.591 at 15 sessions to 0.445 at 300 sessions, mostly because models omit established events. Best financial-state GCA@15 was 0.470 (Claude Opus 4.8); superseded values were reported as still-current about 70% of the time, and the two tasks correlated only weakly across models (Spearman rho = 0.291).
Is the FinLifeBench dataset or code available?
The paper states the frozen corpus, annotations, prompts, schema, and scoring code will be released under the Apache License 2.0; availability should be confirmed via the project's repository since the reviewed version only promises the release.
Why is the benchmark synthetic and in Korean?
Deterministic gold labels and evidence provenance require constructing states and event trajectories before writing the dialogues, which real banking transcripts cannot offer due to privacy and unknown ground truth. The domain targets Korean mobile/internet banking (personas drawn from NVIDIA's Korean Nemotron Personas, quotas informed by KakaoBank aggregates); the authors note cross-model gaps partly reflect Korean language proficiency.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure contents (Fig. 1-3 plots and the Fig. 2 dialogue screenshots) are available only through captions in the extraction; underlying visual details could not be verified.
- Table 2 notes that model-visible labels are Korean, but the Korean surface labels themselves are not present in the extracted text (only English identifiers).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.