BENCHCOMPASS: From Scores to Signals for Training and Harness Decisions in Payment-Domain LLMs
Sijie Dong, Wei Ren, Xuanwei Hu, Jiawei Luo, Zifan Wang, Xiaoyun Feng, Hui Cai, Lyuxin Xue, et al.
cs.AI
Quick verdict
Solid payment-domain LLM benchmark (306 items) separating knowledge, context-use, attack robustness; small subsets and judge-family overlap temper claims.
Summary
The paper introduces BenchCompass, a construction pipeline and benchmark for evaluating LLMs on payment-domain work. Versioned payment documents are chunked with audit metadata, profiled, and linked into typed evidence packs; a generator (DeepSeek V4 Pro) pairs packs with 60 scenario frames and Bloom L1-L5 difficulty targets; LLM quality gates triage candidates; a separate stage creates answer-preserving task-input attacks; four payment-domain experts make final admission. The released Pro benchmark contains 306 items: 95 closed-book MCQs (Payment Knowledge), 96 open-ended Context-Grounded Reasoning items, and 115 Task-Input Attack items, with a larger lower-assurance Normal pool withheld from headline claims. Across 16 model variants under Closed/Open/Attacked-Open settings, closed-book payment knowledge is weak (44.2-80.0%) but rescued by supplying rules (open-book PK nearly saturated at 97.9-100%), context-grounded reasoning remains unsaturated (best frontier 89.6% vs 69.8% for a 32B open-weight model), and attacked inputs cut these to 81.7% and 42.6%. Diagnostic signals attribute model-case outcomes to knowledge, context-use, or harness-brittleness hypotheses. Author-acknowledged limitations: finite document snapshot, no retrieval evaluation, descriptive non-causal signal labels, judge/generator family overlap, and small L5 and low-count attack-mode subsets.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Supplying payment rules converts most knowledge-access failures into successes: PK-CB spans 44.2-80.0% while PK-OB is nearly saturated, and 19/20 audited Closed-to-Open rescues corrected substantive rule errors.
- Context-grounded payment reasoning is unsaturated: the best frontier model reaches 89.6% CGR-OB versus 69.8% for a representative 32B open-weight model.
- Robustness degrades sharply under answer-preserving task-input attacks, especially for smaller models (Qwen3 32B: 69.8% to 42.6%), and enforcing hierarchical rule applicability (Mode A, 49.8% mean) is the hardest failure mode.
- An auditable pipeline of evidence packs, Bloom-targeted scenario generation, LLM quality gates, and expert admission yields a 306-item Pro benchmark with 95.0% blind admission re-review agreement (kappa=0.900).
- Model-case diagnostic signals (Table 1) map failures to knowledge, context-use, or harness-brittleness repair families, offered as hypotheses rather than validated causal interventions.
Strengths
Diagnostic design with falsifiable framing
Table 1 signals are explicitly labeled hypotheses rather than validated causal interventions, and the Closed/Open/Attacked separation is backed by an audit: 19/20 sampled Gemini 3.1 Pro CGR-CB failures rescued in CGR-OB were substantive rule errors, not rubric-formulation artifacts (Sec 5.3.1).
Multi-layer quality control with documented reliability
Expert funnel of 506 finalists to 306 admitted (60.5%, Table 10), 60-item blind admission re-review at 57/60 agreement (kappa=0.900), attack-validity gates rejecting perturbations that change answers, and documented hold-out reasons (42.5% provider-specific rules, Table 11).
Judge validation beyond typical benchmark practice
Honest uncertainty reporting and anti-ranking stance
Actionability checks corroborate triage implications
Weaknesses
Internal inconsistency on the unsaturation claim
Cross-subset deltas conflate composition with difficulty
Attack-mode analysis is underpowered and confounded
Generator and judge family overlap in headline results
Contamination and provenance gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the BenchCompass benchmark?
BenchCompass is a payment-domain benchmark for large language models built by Ant International researchers. It contains 306 expert-reviewed Pro items in three subsets: 95 multiple-choice payment-knowledge questions, 96 open-ended questions requiring reasoning over supplied payment rules, and 115 robustness cases with answer-preserving perturbed inputs. It also ships the automated pipeline (evidence packs, scenario frames, LLM quality gates, expert admission) used to build it, plus a larger lower-assurance pool kept out of headline results.
What are BenchCompass's main findings about LLMs in payments?
Closed-book payment knowledge is weak (44.2%-80.0% across 16 models) but supplying the relevant rules nearly saturates performance, showing many failures are knowledge-access problems. Context-grounded reasoning remains unsolved: the best frontier model scores 89.6% and a 32B open-weight model 69.8%. Under attacked task inputs performance falls further, to 81.7% and 42.6% respectively, indicating models often follow plausible but invalid workflows even when the correct rule is in context.
Where can I download the BenchCompass dataset and code?
The paper points to the GitHub repository ant-intl/BenchCompass. The release includes benchmark cases, evidence snippets, source provenance, reference answers, grading rubrics, prompts, model outputs, judge decisions, human audit labels, and run configuration. Full original payment documents are redistributed only where licensing permits, so some source material may need to be obtained separately.
How reliable is the LLM-judge scoring in BenchCompass?
Open-ended answers are scored by a self-deployed DeepSeek V4 Flash judge producing structured verdicts. Human experts audited 255 judge decisions and accepted 98.4% of them. In a separate sensitivity analysis, two judge models from different families re-scored 921 fixed outputs and agreed with the headline judge on about 92%-93% of items, with correctness-rate shifts under about 2.2 percentage points. Repeated scorings by the same judge varied by at most 2.08 points.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 were extracted as flattened text; cell-level values in Figure 3 were read from embedded numeric labels rather than visual inspection, and Figure 4 groupings were reconstructed from run-on text.
- The extraction appears complete for Sections 1-6 and Appendices A-G, but any supplementary material outside the PDF (repository contents, licensing files) was not verifiable.
- Review is based on arXiv v1 (2609.18270, September 2026); no external reproduction of the reported scores was attempted.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.