Automated Regulatory Compliance Question Answering in Financial Services with Domain-Adapted Retrieval-Augmented Generation
Tobias Deußer, Abhishek Pillai, Aurelio F. Bariviera, Dhananjay Bhardwaj, Lorenz Sparrenberg, David Berghaus, Christian Bauckhage, Rafet Sifa
cs.CL · cs.AI
Quick verdict
Staged LegalBERT retriever shows real, well-ablated gains on re-split ObliQA, but RePASs generation gains are undercut by the authors' own closed-book control; significance untested.
Summary
This empirical study builds a retrieval-augmented QA pipeline for regulatory compliance over ADGM rulebooks (ObliQA) plus Open Australian LegalQA as an out-of-domain control. The retriever starts from LegalBERT and adds entailment tuning, contrastive tuning with in-batch negatives, and linear BM25 fusion, lifting Recall@10 on a leakage-free re-split of ObliQA from 0.256 to 0.774, ahead of BM25 (0.678) and E5-large-v2 (0.758, though MRR ties at 0.594–0.595). Five compact 4-bit generators (Qwen2.5-7B, Teuken-7B, Gemma-2-2B, Gemma-3-12B, DeepSeek-R1-Distill-8B) are compared under prompting versus RAFT-LoRA with GPT-4o-mini teacher labels. RAFT improves the RePASs score for the three models that adapted stably; two were excluded for unstable adaptation, and the best single score (0.735) belongs to unadapted DeepSeek zero-shot. ObliQA-trained adapters mostly fail to transfer to Australian case law. Crucially, a closed-book control scores within 0.011 RePASs of the full pipeline, showing the metric does not measure grounding. Limitations: 150-question evaluation samples with no significance testing, teacher-labeled targets, and all results reported on validation splits.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A three-stage adaptation of LegalBERT (entailment tuning, contrastive tuning, BM25 score fusion) raises ObliQA Recall@10 from 0.256 to 0.774 on a leakage-free re-split, beating BM25 (0.678) and E5-large-v2 (0.758).
- RAFT-LoRA improves the composite RePASs score for all three compact generators that could be adapted stably, with the largest gain on the smallest model (Gemma-2-2B: +0.072).
- Adapters trained on ADGM rulebook data do not meaningfully transfer to Australian case-law questions (only Teuken improves, 0.559→0.586).
- A closed-book model with no passages scores within 0.011 RePASs of the full retrieval pipeline, so RePASs alone cannot validate grounding in compliance settings.
- ObliQA's published splits suffer passage-level train–test leakage because source documents intersect across splits.
Strengths
Retrieval gain is large, fully ablated, and benchmarked
Table 3 isolates each stage (LegalBERT 0.256 → 0.495 ET → 0.732 CT → 0.774 hybrid R@10) and Table 4 places the result past BM25, BGE-M3, and E5-large-v2 with a BERT-base encoder (~1/3 of E5-large's parameters).
Leakage-free re-split with quantified cost
Section 3.2 shows ObliQA's published splits share source documents across train/eval (|Dtrain ∩ Deval| > 0), rebuilds splits at document level (Table 1), discards 989 multi-passage pairs, and audits residual overlap three ways (exact, fuzzy 85%, cosine 0.85).
The closed-book control is the paper's real contribution
Deliberately modest novelty claim and thorough component reporting
Negative results included, not buried
Weaknesses
Small evaluation samples, zero inferential statistics
Generation gain is measured by a metric the paper itself invalidates
Teacher-labeled RAFT targets create a stylistic confound
Results reported on the validation split; held-out test never used
Unablated, self-referential context assembly
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper builds a two-sided domain-adapted RAG pipeline for regulatory compliance QA and evaluates it on ObliQA (ADGM rulebooks) and Open Australian LegalQA (out-of-domain control). Claimed contributions (Section 1): (1) a three-stage retriever — entailment tuning, contrastive tuning with in-batch negatives, BM25 score fusion — lifting ObliQA Recall@10 from 0.256 to 0.774; (2) a leakage-free document-level re-split of ObliQA; (3) a controlled prompting-vs-RAFT-LoRA comparison across five compact generators; (4) two negative results: failed adapter transfer to Australian case law and a closed-book model scoring within 0.011 RePASs of the full pipeline.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How much does domain-adapted retrieval improve regulatory question answering?
In this paper, a LegalBERT encoder adapted in three stages (entailment tuning, contrastive tuning, BM25 fusion) raises Recall@10 on the ObliQA regulatory benchmark from 0.256 to 0.774, exceeding BM25 (0.678) and the general-purpose dense encoder E5-large-v2 (0.758), while using a much smaller model.
Can small on-premise LLMs replace frontier models for compliance QA?
The paper studies 2B–12B open models under 4-bit quantization. Fine-tuning with RAFT-LoRA improved their measured answer quality in-domain, but the best single score (0.735 RePASs) came from an unadapted DeepSeek-R1-Distill-8B, and gains did not transfer to a different legal domain.
Is the RePASs metric reliable for judging grounded regulatory answers?
The authors show it is not, on its own: a closed-book model that received no passages and cites nothing scored within 0.011 RePASs of the full retrieval pipeline, because generic statements find an entailing sentence and uncommitted answers incur little contradiction. They recommend reporting closed-book baselines and attribution-sensitive evaluation.
Does the RAFT-LoRA adapter trained on one rulebook work on other legal domains?
Mostly no. Adapters trained on Abu Dhabi Global Market (ADGM) rulebook data lost most of their benefit on Australian case-law questions (Qwen 0.642→0.581; Gemma-2 0.477→0.453); only Teuken improved. The authors advise building RAFT training data per regulatory regime.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Full text was provided including all tables; figures (Figure 2 axis detail) are recoverable from stated values but were not visually verified.
- arXiv ID and several references carry 2025–2026 dates that cannot be verified offline; reference [13] points to the RIRAG paper as the source of ObliQA/RePASs.
- No appendix was present, so prompts, teacher instructions, and selection details for α and thresholds could not be checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.