Trustworthy FinAInce: Unpacking How AI-Mediated Financial Advice is Judged
Aryan Ramchandra Kapadia, Eshwar Chandrasekharan, Koustuv Saha
cs.HC · cs.AI · cs.CL · cs.CY
Quick verdict
Preregistered experiment (N=285): advice style outweighs source labels in judging financial advice — rigorous, but same-task overlap may inflate pathway R².
Summary
The paper reports a preregistered randomized vignette experiment with 285 U.S. adults (1,140 advice evaluations) on how people appraise personal financial advice. Eight scenarios spanned a 2×2×2 taxonomy (stakes, external uncertainty, verifiability); each was written in AI, Certified Financial Planner, and Online Community styles while holding facts and recommendation direction constant. Source labels were independently manipulated (correct, absent, incorrect). Guided by the Information Adoption Model, linear mixed-effects models show advice style most broadly shaped message and safety appraisals, an Expert label selectively raised perceived source knowledge, and decision context primarily shaped risk and safety appraisals. Models including the appraisal pathway explained 69.2% of Overall Quality, 75.9% of Trust, and 82.9% of Intended Reliance variance, versus 9.8–14.8% for upstream factors alone. A side-by-side ranking task replicated the Expert-style preference even without labels (47.8% first choice). Limitations: reliance was measured as intention only; appraisals and outcomes were collected in the same rating task, so pathway R² may partly reflect item overlap rather than a causal sequence; two appraisals were single items; stimuli were author-crafted; the sample was limited to Prolific U.S. adults.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- When the underlying recommendation is held constant, advice style — not displayed source labels — most broadly shapes message and safety appraisals of financial advice (e.g., Expert vs AI: β=+.334 Message Appraisal, β=−.245 Safety Concern).
- Source labels act selectively: an Expert label raises perceived Source Knowledge (β=+.225), and style and label effects are additive rather than interactive (all interaction LRTs p>.29).
- The appraisal pathway explains 69.2% of Overall Quality, 75.9% of Trust, and 82.9% of Intended Reliance variance versus 9.8–14.8% for upstream factors alone, with Quality→Trust (β=.607) and Trust→Reliance (β=.594) as the strongest links.
- Expert-style advice remains most preferred even without any source label (47.8% first choice), and participants overwhelmingly justify rankings via message-level criteria (clear reasoning 65.6%) rather than source (7.0%).
Strengths
Disentangles advice style from source labels
The design (§2.3, §3.4–3.5) occupies the gap between identical-text label studies and confounded naturalistic comparisons: facts, numbers, recommendation direction, and rationale are held constant across AI/Expert/OC styles while labels are independently correct, absent, or wrong — a genuinely useful methodological contribution.
Unusually thorough stimulus validation
§3.4.1: 10 of 12 linguistic features discriminate styles (FDR-corrected Friedman tests), CDI gradient 27.36→21.30→12.03 (χ²=14.25, p<.001), blind annotators identify intended style in 91.7% of judgments (Fleiss κ=.749), 100% top-1 within-scenario embedding retrieval, and a 72-text external reference corpus — with divergent cues (formality, hedging orderings) honestly reported rather than buried.
Preregistered, multi-layered randomization and factorial context
Robust, transparent statistics
Convergent evidence and honest interpretation
Weaknesses
Pathway R² likely inflated by same-battery measurement
Single-item constructs at the heart of the label effect
Style stimuli conflate authorship with style
No reported label-notice manipulation check
Intention-only reliance and weak ranking concordance
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks how U.S. adults appraise personal financial advice and how those appraisals relate to Overall Quality, Trust, and Intended Reliance (RQ1–RQ2). Method: a preregistered randomized vignette experiment, N=285 (labeled 100 / unlabeled 92 / mislabeled 93), yielding 1,140 evaluations of eight scenarios spanning a 2×2×2 taxonomy of stakes, external uncertainty, and verifiability (Table A1). Each scenario was rendered in AI, Certified Financial Planner, and Online Community (OC) styles with facts, numbers, recommendation direction, and rationale held constant; source labels were independently correct, absent, or incorrect. Central claims: advice style dominates message and safety appraisals; Expert labels selectively raise Source Knowledge; the appraisal pathway explains 69.2%/75.9%/82.9% of Quality/Trust/Reliance variance versus 9.8–14.8% for upstream factors alone; Expert-style advice remains preferred even unlabeled (47.8% first choice).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do people trust AI financial advice less than expert advice?
In this study, when the same recommendation was written in an AI, certified-planner, or online-community style, people judged message quality and safety mostly by style: expert-style advice rated best and AI-style worst, even with no source label shown. An 'expert' label mainly raised perceived source knowledge. Overall trust and willingness to follow advice were predicted far better by these appraisals (69–83% of variance) than by labels or decision context alone (about 10–15%).
How was the study conducted and how many participants took part?
285 U.S. adults recruited via Prolific completed a preregistered randomized vignette experiment, producing 1,140 advice evaluations. Eight financial scenarios varied stakes, external uncertainty, and verifiability; each appeared in three writing styles with identical facts and recommendation. Source labels were independently correct, missing, or wrong, and a final ranking task compared all three styles side by side.
Does labeling advice as coming from an expert make it more trusted?
Only selectively. An expert label increased perceived source knowledge and slightly improved message appraisal, but style and label effects were additive rather than interactive, and in the ranking task participants still preferred expert-style advice when no label was shown at all. The way advice is written carried much of the effect attributed to expertise.
What are the main limitations of this research?
Reliance was measured as an intention, not real behavior; appraisals and outcomes came from the same rating task, so the high explained-variance figures may partly reflect item overlap rather than a causal chain; two appraisals were single-item measures; stimuli were researcher-constructed; and the sample was limited to U.S. Prolific workers.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 are available only via captions; visual details (e.g., Fig. 2 linguistic profiles) could not be independently verified.
- Appendix tables were extracted from text; some cell alignment or symbols may be lossy.
- No code/data availability statement appears in the provided text; availability is unknown rather than confirmed absent.
- Preregistration is referenced via the authors' own arXiv preprint [38]; the registry content itself was not inspectable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.