IndicBankBench: Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking
Suvradip Paul, Chandra Bhushan, Harsh Sharma, Nitin Kukreja, Yatharth Dedhia, Keyur Doshi, Prashant Devadiga
cs.AI
Quick verdict
Shows at-least-once success overstates banking-agent reliability (60–74% vs 43.7–58.2% strict); rigorous staged benchmark, but judge audit is thin.
Summary
IndicBankBench is a 799-case benchmark from NPCI for evaluating language-model assistants in authenticated Indian retail banking sessions. Cases span five operational domains, a capability/refusal domain, and twenty primary axes, using deterministic mock tools, scripted customer turns, and adversarial context (stale balances, wrong claims, filter traps). Each transcript is graded in four ordered stages — safety, action/tool use, response adequacy, and advisory quality — with most checks deterministic; an LLM judge (GLM-5.2) handles only the semantic response gate, and a narrow resolver handles ambiguous confirmation-before-write checks. Eleven models run three times each at temperature 0.7; the headline metric is strict pass^3 (success on all trials). Strict reliability ranges from 43.7% (Gemma 4 E4B) to 58.2% (DeepSeek-V4-Pro), while at-least-once success reaches 60–74%, a 10.8–21.4-point gap showing that occasional success overstates dependable behavior. Paired bootstrap and McNemar tests find no reliable difference among the top five models. The wrong-information axis is hardest (no model above 40.1%); missing required tool calls (A1) and missed response requirements (R) dominate first failures. Main limitations: the judge audit covers only 40 transcripts (81.6% agreement), one candidate shares the judge's model family, case snapshots are not byte-identical across models, a single fixed system prompt is used, and English-only synthetic cases limit ecological validity. Code, cases, and the evaluation harness are released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- IndicBankBench is a 799-case, multi-turn, account-grounded benchmark with four-stage (S/A/R/Q) grading that records the stage at which each banking interaction fails.
- Under three repeated trials, strict pass^3 reliability (43.7–58.2%) trails at-least-once success (60–74%) by 10.8–21.4 points, so occasional-success metrics overstate dependable banking behavior.
- No pair among the five leading models shows a statistically reliable difference, while case-level diagnostics reveal distinct failure profiles behind similar aggregate scores.
- The wrong-information axis is the hardest task axis (no model above 40.1% of 142 cases), and missing required tool calls (A1) plus missed response requirements (R) are the dominant first-failure gates for all eleven models.
Strengths
Staged S/A/R/Q grading localizes failure rather than collapsing it into one score
§3.3 and Table 3 separate deterministic safety gates (S1–S4) and action gates (A1–A4) from an LLM-judged response gate and a non-gating advisory score; Table 10 then shows A1 (missing required call) and R dominate first failures for all eleven models, and §5.2's cross-stage analysis finds 12.9% of trajectories pass all safety/action gates but fail R versus 5.7% the reverse — evidence both check families are needed.
Repeated-trial protocol quantifies reliability, a dimension most benchmarks ignore
Three runs per case at fixed temperature 0.7 yield the strict pass^3 vs at-least-once gap of 10.8–21.4 points (Table 5, Fig. 3); GLM-5.3-Flash leads at-least-once (594/799, 74.3%) yet drops to 56.8% strict — a concrete demonstration that best-of metrics overstate dependable behavior.
Statistical transparency including an honestly reported null
Evaluator validation with documented ambiguity, and full prompt disclosure
Reproducibility-oriented release
Weaknesses
Judge validation is underpowered relative to the R gate's verdict weight
Potential judge–candidate family bias is acknowledged but not bounded
Cross-model comparability rests on unquantified snapshot drift
Single fixed prompt, single temperature, and non-comparable reasoning controls
Coverage skew and limited diagnostics resolution
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
IndicBankBench (NPCI) targets a gap the authors argue most tool-calling benchmarks miss: not whether a banking assistant succeeds, but where and how reliably it fails. The artifact is 799 authored cases — five operational domains (accounts/transactions, calculators, cards, customer service/catalog, deposits/loans), a 68-case capability/refusal domain, and 20 primary axes (Table 7) — driven by 32 deterministic mock tools (Table 6) and scripted customer turns grounded in synthetic Indian personas (Nemotron-Personas-India, CC BY 4.0). Central claims: (1) four-stage S/A/R/Q grading localizes failure; (2) strict pass^3 over three runs reveals a 10.8–21.4-point gap to at-least-once success (Table 5, Fig. 3), so best-of metrics overstate dependable behavior; (3) case-level diagnostics separate systems with similar aggregate scores. Within the released environment, all three claims are supported, and the paper is unusually explicit about what its numbers do not establish (§8).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is IndicBankBench?
A public 799-case benchmark from India's NPCI that evaluates language-model assistants on multi-turn, account-grounded Indian retail banking tasks using deterministic mock tools. Each transcript is graded in four stages — safety, action/tool use, response adequacy, and advisory quality — and most checks are deterministic programmatic rules.
What does strict pass^3 mean and why does it matter?
A case counts as passed only if the assistant succeeds in all three repeated runs at temperature 0.7. Across eleven models, strict pass^3 was 43.7–58.2% while at-least-once success was 60–74%, a 10.8–21.4-point gap showing that one-off or best-of success rates can substantially overstate how dependable a banking assistant actually is.
Which models performed best on IndicBankBench?
DeepSeek-V4-Pro-0813 (58.2% strict), Grok 4.6 (58.1%), Qwen3.8-2.4T-A95B (57.8%), Claude Opus 5 (57.7%) and GLM-5.3-Flash (56.8%) led the eleven-model evaluation, but paired bootstrap and McNemar tests found no statistically reliable difference among these five; the benchmark's value is differentiating their failure patterns instead.
Is the IndicBankBench data and code available?
Yes. Cases are released on Hugging Face under CC BY 4.0, the code and evaluation harness are on GitHub (npci/IndicBankBench) under the MIT license, and all evaluation prompts (system prompt, judge, confirmation resolver) are reproduced in the paper's appendix.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2–4 are embedded as raw number streams in the extracted text; figure-derived statements were cross-checked against Tables 5, 7, 10 and 11 instead of the rendered charts.
- The extraction includes the full appendices, but minor layout artifacts (footnote placement, figure captions) mean a few prose details were read with extra care; no table appeared truncated.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.