Right Frame, Wrong Rule: Cultural Cues Expose the Financial Knowledge Gap They Were Meant to Close
Rania Elbadry, Ahmed Heakl, Saeed Almheiri, Fan Zhang, Muhra AlMahri, Xueqing Peng, Mohsinul Kabir, Shuyao Wang, et al.
cs.CL · cs.AI · cs.LG
Quick verdict
Four-choice benchmark shows cultural cues steer LLMs to Islamic-finance framing they answer wrongly; strong design, MCQ position bias limits some claims.
Summary
The paper formalizes normative pluralism — questions with valid answers under competing normative frameworks — and studies it in Islamic finance with a bilingual English/Arabic benchmark derived from the expert-validated SAHM corpus. A four-choice taxonomy crossing framework selection with within-framework correctness (CI/CW/II/IW) exposes the 'stereotype trap': a cultural cue steers a model toward a framework whose answers it then gets wrong. Across 12 models, 50 demographic signals, and 304 bilateral questions plus 64 Western-anchor and 41 Islamic-anchor controls, large open-weight models reach 96-97% Islamic-frame selection under a Shariah keyword, yet 57-66% of those selections are incorrect; frontier models keep the Islamic fake rate between 0.075 and 0.114. Control sets rule out general financial incompetence and signal conditioning, and activation patching localizes a cue-invariant commitment gate whose depth correlates with stereotype rate (r=-0.78) while baseline routing does not (r=-0.01). Benchmark construction is source-grounded and expert-validated (κ=0.71-0.85) with generator ablations. Key limitations: a single deterministic answer-order shuffle leaves residual position confounding (up to 52.5pp for Llama-3.1-8B in Arabic), evaluation is multiple-choice only, just 11 of 23 pre-registered signal conditions were implemented, and mechanistic analysis covers only open-weight models; the competence-conditioned routing hypothesis is motivated but untested.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- 文化线索改变了 LLM 在伊斯兰金融中的框架选择,同时降低了 12 个模型中 9 个模型的框架内正确性——这种失效模式在双选项评估中是不可见的。
- 在最强信号下,大型开放权重模型 97% 的时间选择伊斯兰框架,但这些选择中有 57-66% 在框架内是错误的。
- 该陷阱在两组控制集中都得以保留,具有方向特异性(非前沿 IFR 比同一项目上的 WFR 高 3-6 倍),且在 16 倍激活范围内部署不变。
- 一个不依赖于线索的承诺门局部化在开放权重模型的后半部分;其深度与刻板印象率相关(r=-0.78),而基线布线则不相关(r=-0.01)。
- 能力条件化布线假设——即模型偏向于它们更准确的框架——得到了激发,但未直接检验。
Strengths
真正新颖的评估分解
§1 和表 2 中的 CI/CW/II/IW 网格揭示了与 BBQ/CrowS-Pairs 风格测试正交的失效模式:在 KEYWORD_SHARIA 下,大型开放权重模型达到 0.964-0.970 的 pisl,而 IFR 为 0.570-0.661,因此双选项工具会误读为近乎完美的对齐。
基于来源的、经专家验证的构建
西方答案是在 12 个主要监管文件的上下文中生成的(表 1),带有源包含审计;专家验证 κ=0.71-0.85 贯穿始终;430 个候选者中有 78 个收敛和 48 个不准确的 CW 被排除;发布了自然化审计集以供直接检查。
精心设计的控制策略
罕见的自我审计诚实度
基于因果干预的机制工作
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 0 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
LLM 文化对齐中的“刻板印象陷阱”是什么?
这是一种失效模式,即文化线索(如穆斯林名字、海湾地区位置、Shariah 请求)引导模型趋向文化上所期望的框架——这里是伊斯兰金融——但模型随后在该框架内选择了事实错误的答案。双选项测试会将此评为成功的文化对齐;本文的四选项分类法(正确/错误 × 伊斯兰/西方)揭示,大型开放权重模型的多达 66% 的此类选择是错误的。
哪些模型在 WDB Islamic-finance benchmark 上表现最好和最差?
前沿模型最强:Claude Opus 4.5、Sonnet 4.5 和 Gemini 3 Flash 在最强线索下,将错误的伊斯兰框架回答比例(IFR)保持在 0.024 到 0.114 之间。最差的是小型开放权重模型:在 Shariah 关键词下,Gemma-3-4b 的 IFR 达到 0.794,Llama-3.1-8B 为 0.753;Qwen2.5-14B 为 0.661。以阿拉伯语为中心的模型(ALLaM-7B、Fanar-9B、SILMA-9B)陷入了与通用中型模型相同的陷阱。
这个 benchmark 是否证明模型知道哪些金融规则适用于哪个国家?
不是。评估提示词仅包含城市名称(例如 'I live in Tehran'),没有监管信息。位置线索效应衡量的是对地名的敏感度,而不是对伊朗或沙特阿拉伯银行法规的了解。作者明确指出,模型对司法管辖区的排序看似合理,但无法区分法定仅限伊斯兰教的系统(伊朗、苏丹)与双重系统(沙特阿拉伯、埃及)。
benchmark、数据或代码是否公开可用?
是的。三个数据集(WDB-Set-A-Base n=304、WDB-Set-A-Signal-Grid 和自然化审计集)在 Hugging Face 的 CulturalDefaultBias 组织下发布,提供阿拉伯语和英语版本。注释导出和细则已包含在内;论文未明确说明评估或激活修补代码是否已发布。
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- 提供的元数据列出了 8 位作者,但提取出的论文页眉列出了 15 位作者(包括 Yi Han、Saadeldine Eletter、Duzhen Zhang、Preslav Nakov、Yuxia Wang、Fajri Koto、Zhuohan Xie)——本次评审遵循提取出的页眉。
- 嵌入图表的文字(例如图 2 管道图和图 3 信号图)因 PDF 提取而出现乱码;所有定量陈述均取自表格和正文,这些内容是完整可读的。
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.
