FaithfulBench: Does AI Counsel Uphold or Undermine the User's Professed Faith?
M Waleed Kadous, Benjamin Olsen, Walter Scheirer, Daniel D. Slate, Alexander Arnold, DZ Kalman
cs.HC · cs.AI · cs.CL
Quick verdict
First cross-tradition benchmark of whether AI counsel upholds users' faith; large, candid, dual-judged grid — but LLM judges lack human validation.
Summary
FaithfulBench is, to the authors' knowledge, the first benchmark measuring whether an AI assistant's counsel upholds the user's professed faith, spanning eight traditions: Sunni Islam, Eastern Christianity, Roman Catholicism, Buddhism, Judaism, Taoism, Protestantism, and a secular control. Each of 555 scenarios is a first-person dilemma with per-scenario judge guidance binding judges to the tradition's own texts. Each runs as a two-round sitting — dilemma, then one of six pressures — under three framings: tradition unstated, named in one line, or accompanied by a one-page tradition guide. Five frontier models produced 49,950 sittings, scored at two scopes by two LLM judges from different providers (Gemini 3.6 Flash and Claude Opus 4.8), with every reported score their mean. Unstated, scores follow a three-tier normativity ordering, with the high-normativity tier (Catholicism, Sunni Islam) below zero; disclosure shrinks the cross-tradition spread from 0.55 to 0.23, which the authors attribute to recognition rather than capability. The one-line declaration wins correct first answers but not steadfastness under pressure; the guide secures both for the top models, though a high-normativity residual persists, largest in Sunni Islam. Limitations: LLM-only judging with no human raters, one judge also among the evaluated models, only three banks expert-audited (one by the corresponding author), and a post-hoc tier taxonomy confounded by unequal bank sizes.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Undeclared, frontier models counsel from a secular therapeutic default and every model fails some believers, with the high-normativity tier (Roman Catholicism, Sunni Islam) averaging below zero (−0.03, Table 1).
- The cross-tradition deficit is mostly recognition rather than capability: the best-minus-worst tradition spread falls from 0.55 (unstated) to 0.31 (stated) to 0.23 (guided).
- A one-line faith declaration wins faithful first answers but not steadfastness under pressure, whereas the tradition's one-page guide secures both for the top three models (steadfastness −0.02 to 0.00, Table 2).
- A high-normativity residual persists under full guidance — largest in Sunni Islam (top-two models +0.90 vs a +1.00 ceiling) — which the authors attribute to prescriptive vs formational counsel demands or possibly rubric strictness.
Strengths
Novel, well-positioned construct
First benchmark (§2 Positioning) scoring open-ended counsel against the user's own professed tradition rather than doctrine knowledge (IslamicMMLU) or fixed opinions (Santurkar et al.); per-scenario judge guidance (A.2) binds the judge to each tradition's proof texts, cleanly porting JaleesBench's single-tradition design across eight traditions.
Scale with unusual transparency
49,950 sittings and 204,195 judgments on a frozen, versioned corpus (4 Sep 2026); open-source harness/validator and browsable corpus; full cost table (~$5,100, Appendix E); serving-route changes bridged and disclosed (r=0.930, 0.949, D.3).
Serious judge-robustness work
Informative decomposition
Candid limitations section
Weaknesses
Validity rests on LLM self-consistency
Entirely internal expert audit
Guided framing risks supplying the answer
Post-hoc tiers and confounded banks
Inference gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FaithfulBench (arXiv 2609.13634) claims to be the first benchmark measuring whether AI counsel upholds the user's professed faith, positioned in §2 against knowledge benchmarks (IslamicMMLU, IslamicLegalBench, IslamTrust), opinion audits (Santurkar, Durmus), virtue-under-pressure work (VirtueBench), and the CEFE-AI omissive-bias line. Eight drop-in tradition modules — Sunni Islam (140 scenarios), Eastern Christianity (106), Roman Catholicism (76), Buddhism (52), secular sage (49), Judaism (48), Taoism (48), and a derived Protestant module (36) — total 555 scenarios. Central claims: (1) undeclared, models counsel from a 'secular therapeutic default' and the high-normativity tier (Catholicism, Sunni Islam) scores below zero (−0.03, Table 1); (2) the cross-tradition spread collapses 0.55→0.31→0.23 across unstated→stated→guided framings, so the deficit is recognition, not capability (§4.2); (3) the one-line declaration wins the first answer but not steadfastness; the guide secures both (§4.3, Table 2); (4) a high-normativity residual survives full guidance, largest in Sunni Islam (top-two models +0.90 vs a +1.00 ceiling).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is FaithfulBench?
A benchmark that measures whether an AI assistant's advice upholds the user's professed religious faith. It spans eight traditions (Sunni Islam, Eastern Christianity, Roman Catholicism, Buddhism, a secular philosophical control, Judaism, Taoism, and a consensus-derived Protestant module) and scores each conversation against the tradition's own texts via per-scenario judge guidance, including whether the model keeps its advice when the user pushes back.
Do AI chatbots give worse advice to religious users?
According to this paper, when the user's tradition is not stated, the five tested frontier models default to secular therapeutic-style counsel, and average scores for the most rule-bound traditions (Roman Catholicism and Sunni Islam) fall below zero on a −1…+1 scale, meaning counsel on balance runs against the user's own tradition. Naming the faith in one line mostly fixes the first answer, but only a fuller one-page tradition guide keeps answers steady under user pressure.
Which models were evaluated and which performed best?
Claude Sonnet 5, Inkling (Thinking Machines), GPT-5.6 Terra, Gemini 3.6 Flash, and Qwen3-235B. Claude Sonnet 5 and Inkling led in every framing; Qwen3-235B was last in all framings and scored below zero in all eight traditions when the faith was unstated. With the tradition guide, the top two models reached about +0.95 overall.
Is the FaithfulBench dataset and code public?
Yes. The corpus, harness, and validator are open source (github.com/faithfamilytechnologynetwork/multibench), the corpus is browsable online, and all results refer to a frozen versioned dataset published alongside the transcripts and judges' verdicts. The paper also discloses the full run cost (about $5,100).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1, 3–8 and 9–12 extracted as scattered axis/label text (typical PDF figure extraction); their quantitative content is nonetheless carried by Tables 1–12, which extracted cleanly. The text ends normally after Appendix F with no sign of truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.