Efficient LLM Distillation for Bangladesh Legal Context: A Smartphone-Compatible Retrieval-Augmented Generation Model
MD. Nafis Kamal, Mahadi Hasan Fahim, Talha Ridwan, Nadifa Zaman, Fariha Roushon Florin, Farig Yousuf Sadeque, Saadat Rafid Ahmed
cs.CL
Quick verdict
Two-phase KD compresses Gemma-2 9B→2B for Bangladeshi legal RAG; solid ablations, but teacher-output references and n=50 benchmark cap correctness claims.
Summary
This BRAC University thesis-derived paper builds an offline legal assistant for Bangladesh. The authors crawl 1,083 acts plus 170 constitutional articles from the national legislation portal, split them into 36,029 passages, and distill a 9B Gemma-2 teacher into a 2B student using two-phase training: supervised fine-tuning on 9,429 quality-gated synthetic QA pairs (65% of 14,514 accepted), then sparse KL divergence against the teacher's top-50 per-token logits at temperature 4.0, both via QLoRA rank-32. At inference, hybrid retrieval (60% dense multilingual-e5, 40% BM25) with query expansion and cross-encoder re-ranking grounds every answer. On a 50-query English benchmark, the distilled model with RAG reaches ROUGE-L 0.4715 and BERTScore F1 0.5679 — a 103%/143% gain over the retrieval-only undistilled baseline (0.2323/0.2340) — and component ablation shows neither distillation nor RAG alone suffices (0.1120 vs 0.2323 vs 0.4715). The adapter quantises to a 1.6 GB GGUF file running 4–8 tokens/s offline on a Pixel 6. Bangla queries work through cross-lingual retrieval (ROUGE-L 0.4083) but answers stay in English. Main limitations: automated metrics score closeness to teacher outputs rather than legal correctness, the human evaluation involved one lawyer, the 50-query sets are synthetic with no significance testing, and no code or evaluation data is released.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Two-phase progressive distillation (CE SFT then sparse top-50 KL at τ=4.0, QLoRA r=32) lifts the Gemma-2 2B student from the ≈0.20 SFT plateau to ROUGE-L 0.4715 with RAG on a 50-query English benchmark.
- Distillation and RAG are complementary and neither suffices alone: distilled-no-RAG (0.1120) underperforms undistilled+RAG (0.2323), while the combination reaches 0.4715 (Table IX).
- The merged adapter quantises to a 1.6 GB GGUF Q4_K_M file and runs fully offline at 4–8 tokens/s on a Pixel 6, with ≤~2% metric degradation versus the server pipeline on a PC proxy benchmark (Table X).
- Cross-lingual retrieval lets Bangla queries work against the English corpus (ROUGE-L 0.4083, BERTScore 0.8133 with RAG), though responses remain in English.
- A single practising lawyer rated 50 responses at a weighted mean of 4.16/5, with 90% rated 4 or 5 (Table XII).
Strengths
Unusual transparency about failures
Reports the SFT plateau at ROUGE-L ≈0.20 across teacher sizes and formats (§IV-C), the abandoned Phase 3 (−6.8% regression), and the LLM-judge's bimodal faithfulness scores (35 scores of 1, 9 of 5, §V-A) with concrete numbers — failures most papers omit.
Informative three-way component isolation
Table IX shows distilled-no-RAG (0.1120) < undistilled+RAG (0.2323) < distilled+RAG (0.4715), cleanly establishing that distillation and RAG address different failure modes rather than asserting it.
Credible end-to-end offline deployment on consumer hardware
Legally-informed corpus curation with quantified chunking ablation
Incremental RAG design documented against named failure modes
Weaknesses
Teacher-as-reference circularity
No inferential statistics, tiny synthetic benchmark
Baseline configuration ambiguity
Evaluated system differs from deployed system
Single-rater human evaluation and shallow training-data gate
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.24177, BRAC University thesis work) builds a fully offline legal QA system for Bangladeshi statutory law and claims four results: (1) two-phase progressive distillation — SFT on 9,429 quality-gated QA pairs, then sparse top-50 KL divergence at τ=4.0 via QLoRA — lifts a Gemma-2 2B student from a ≈0.20 ROUGE-L SFT plateau to 0.4715 with RAG on a 50-query English benchmark; (2) distillation and RAG are complementary, neither suffices alone (Table IX: 0.1120 / 0.2323 / 0.4715); (3) the merged adapter quantises to a 1.6 GB GGUF Q4_K_M file running 4–8 tokens/s offline on a Pixel 6 with ≤2% metric loss (Table X); (4) Bangla queries work via cross-lingual retrieval (ROUGE-L 0.4083, BERTScore 0.8133), and a single practising lawyer rated outputs 4.16/5. Scope is explicitly statutory-only; case law and subordinate legislation are out of scope (§VII).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper do?
It compresses a 9-billion-parameter Gemma-2 model into a 2-billion-parameter student using two-phase knowledge distillation, so that questions about Bangladeshi statutory law can be answered entirely offline on an Android phone, with every answer grounded by hybrid retrieval over 36,029 passages from the Bangladesh Constitution and national legislation.
How accurate is the distilled Bangladesh legal model?
On a 50-query English benchmark with retrieval it scores ROUGE-L 0.4715 and BERTScore F1 0.5679 — 103% and 143% above a retrieval-only undistilled baseline (0.2323/0.2340). However, these metrics measure closeness to the teacher model's own answers, not verified legal correctness. The only human check was one practising lawyer, who rated 90% of answers 4 or 5 out of 5 (weighted mean 4.16/5).
Can it run on a phone without internet?
Yes, by the paper's account: a 1.6 GB quantised GGUF model runs at 4–8 tokens per second on a Pixel 6 CPU with no network access, alongside a 141 MB embedding index and 45 MB corpus stored on-device. Measured quality loss versus the server setup is small (−1.6% ROUGE-L on a PC proxy benchmark), though the app currently installs only via ADB, not the Play Store.
Does it support the Bangla language?
Partially. Bangla queries are handled through cross-lingual retrieval — on 50 Bangla queries with RAG it scores ROUGE-L 0.4083 and BERTScore 0.8133 — but the model always answers in English, which the authors flag as a key limitation for citizens who cannot read English.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–6 are referenced but not analysable from the extracted text; figure-only content (e.g., the Fig. 5 stage progression) was not independently verified.
- Tables were extracted as flat text; a few cells (e.g., Table XI's '∼44 words' teacher-latency row) look garbled and were avoided in this review.
- No code or data availability statement appears in the extracted text; codeAvailable=false reflects the absence of a release statement rather than a confirmed refusal.
- Exact wording and construction of the 50 English and 50 Bangla evaluation queries is not fully specified, limiting assessment of benchmark representativeness.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.