LEGO: Synergizing Expert GraphRAG and Expert Chain-of-Thought for Legal Reasoning
Qingjing Chen, Junkai Zhang, Shaochun Wang, Jiahao Ding, Siyuan Zheng, Yukun Yan, Zhi Zheng, Antonino Rotolo, et al.
cs.CL · cs.IR
Quick verdict
Expert-graph RAG + syllogistic CoT for Chinese civil law: strong same-backbone RAG gains, but claimed parity with larger models lacks statistical support.
Summary
LEGO is a two-module framework for complex legal reasoning over the PRC Civil Code. ExpertGraphRAG retrieves provisions from an expert-annotated graph — 318 concepts, 409 doctrinal rule cards, 1,211 of 1,260 articles, 4,152 support edges, built over seven months — using Normative Coverage Greedy, which scores candidate articles by relevance, newly covered expert rule units, and redundancy. ExpertCoT then generates a structured Provision–Fact–Conclusion syllogistic analysis under eleven prompt instructions refined through expert error analysis. On LawExamQA_Civil (723 Chinese judicial-exam multiple-choice items), a Qwen3-8B backbone with LEGO reaches 40.53% exact-match accuracy, exceeding five same-backbone RAG/GraphRAG baselines by +8.86 to +11.76 points (exact McNemar with Holm correction, p ≤ 4.4×10⁻⁶) and staying robust as question hop count rises; it also leads the evaluated baselines on the open-ended LexRAG_Civil and PLawBench_Civil benchmarks. Appendix L candidly reports that margins over Qwen3-30B-A3B, GPT-5 and DeepSeek-V3 671B are not statistically significant (confidence intervals span zero; the DeepSeek gap equals four items), and the ablation ExpertGraphRAG+IRAC-CoT is not significantly worse than the full system (p=0.067). Error analysis of 430 failures is dominated by wrong provision selection (36.74%) and prerequisite violations (20.70%). Limitations: single jurisdiction, expensive expert annotation, hop count only a proxy, and no evidence on explanation faithfulness.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Injecting expert-annotated normative relations into retrieval (ExpertGraphRAG with Normative Coverage Greedy) yields instance-specific provision subgraphs that significantly outperform five similarity-based RAG/GraphRAG baselines at the same Qwen3-8B backbone (+8.86 to +11.76 points, Holm-corrected p ≤ 4.4×10⁻⁶).
- With Qwen3-8B, LEGO reaches 40.53% exact match on LawExamQA_Civil and is not significantly different from GPT-5, DeepSeek-V3 671B or Qwen3-30B-A3B, which the authors frame as comparable performance from a far smaller model.
- ExpertCoT's structured Provision-Fact-Conclusion prompting plus the expert graph are individually and jointly beneficial, with full LEGO beating ten of eleven ablations significantly (the exception, ExpertGraphRAG+IRAC-CoT, p=0.067).
- LEGO's remaining errors concentrate in a small set of structural failure modes — incorrect provision selection (36.74%) and prerequisite violations (20.70%) cover 57.4% of 430 mis-answered items.
Strengths
Unusually rigorous and candid significance analysis
Appendix L runs two-sided exact McNemar tests on paired per-item predictions (n=723), 10,000-resample item-level percentile bootstrap CIs, and Holm correction within two pre-split families (Tables 24-25), and explicitly prints the non-significant rows rather than burying them.
Large, clean retrieval gains at fixed backbone
Table 24 shows LEGO beats all five same-backbone RAG baselines (Naive RAG, LightRAG, RAPTOR, HippoRAG 2, G-Retriever) by +8.86 to +11.76 accuracy points, adjusted p from 1.5×10⁻⁹ to 4.4×10⁻⁶ — the core retrieval thesis is solidly supported.
Principled retrieval objective with honest theory framing
Deep, mechanism-level error analysis
Transparent limitations reporting
Weaknesses
Equivalence overclaim versus larger models
The synergy claim is only partially supported by the ablation
Single jurisdiction built on a costly expert substrate
Open-ended wins risk judge-format circularity
Baseline comparisons are pipeline-level, not budget-matched
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
LEGO: Expert GraphRAG + Expert CoT for Legal Reasoning — Preliminary Review
**Paper:** arXiv:2609.27009 (cs.CL, cs.IR) · **Confidence:** medium — main tables truncated in extraction; statistical appendices fully present.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the LEGO framework for legal AI?
LEGO combines two modules: ExpertGraphRAG, which retrieves Chinese Civil Code articles from an expert-annotated knowledge graph using a normative-coverage greedy algorithm instead of pure similarity, and ExpertCoT, which makes the LLM produce a structured Provision-Fact-Conclusion (syllogistic) analysis under eleven prompt rules. Code is stated to be at github.com/BLK-WHT/LEGO.
How accurate is LEGO on legal exam questions?
With a Qwen3-8B backbone it answers 40.53% of LawExamQA_Civil (723 Chinese judicial-exam multiple-choice questions) exactly correct. It beats five same-backbone RAG baselines by 8.86-11.76 points with corrected statistical significance, but is not significantly different from GPT-5, DeepSeek-V3 671B or Qwen3-30B-A3B.
What knowledge graph does LEGO use and how was it built?
A graph over the PRC Civil Code with 318 legal concepts and 409 doctrinal rule cards, linking 1,211 of the code's 1,260 articles via 4,152 support edges. Trained legal experts built it over seven months; it encodes relations such as prerequisites, exceptions, priority and general-special links between provisions.
Can LEGO be used outside Chinese civil law?
Not directly: retrieval depends on an expert-annotated graph for the specific statute. The authors list cross-jurisdiction scalability of the annotation effort as an open limitation, and no other legal domain or country is tested in the paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Main result tables (Tables 1-3) were truncated in the extracted text; per-baseline accuracies beyond the deltas in Appendix L could not be independently verified.
- Figures 1, 2 and 8 are not analyzable from text; the multi-hop trend and error-type distribution are taken from captions and prose only.
- Appendix D (judge models and scoring protocol for the open-ended benchmarks) was not present in the extract, limiting assessment of judge dependence.
- The extract interleaves appendix case studies with main text; all cited numbers were used only where verbatim in the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.