GANDR: Claim Auditing for Verifiable Legal Answer Generation
Chen Qian, Yimeng Wang, Yu Chen, Lingfei Wu, Andreas Stathopoulos
cs.CL · cs.AI · cs.IR
Quick verdict
GANDR couples CREAC drafting with per-claim citation auditing, winning by 11.3 strict points; the gain may owe more to prompt design than the audit itself.
Summary
GANDR is a two-agent system for grounded legal answer generation. A Drafter writes answers in the CREAC legal-reasoning format (Conclusion-Rule-Explanation-Application-Conclusion) with inline {cite: rule_id} markers; a separate-context Critic, seeing only the question, retrieved passages and final draft, audits each claim against its cited source and emits a per-claim audit trace. The orchestrator commits on a deterministic regex protocol check, not the Critic's verdict. On a 185-item benchmark from LegalBench and LegalBench-RAG - six systems sharing one backbone (Nemotron-Cascade-2-30B), one BM25 retrieval surface, and identical citation instructions - GANDR reaches 70.8% strict accuracy (correct holding plus every citation resolving to a retrieved passage), leading the strongest baseline (B2, a CREAC-prompted single LLM) by 11.3 points (McNemar p=0.006). Reverting the commit rule costs 22.7 points, and the lead holds at +3.2 to +6.5 points on three further backbones in single-run checks. Against two law-trained annotators, the audit flags under-supported claims at F1 0.84, though its four-way labels are weakly calibrated (kappa=0.275). Main limitations: 98.6% of runs commit on round one, so the Critic adds no accuracy by construction; the B2 comparison removes B2's self-check rubric, an uncontrolled confound; per-domain confidence intervals overlap almost everywhere; code is only available upon request.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On a 185-item legal benchmark with one shared backbone, retrieval surface and citation instruction, GANDR ranks first on every primary metric, reaching 70.8% strict accuracy and leading the strongest baseline by 11.3 points (McNemar p=0.006).
- The protocol-anchored commit rule is load-bearing: reverting it to a 0.95 atomic-verification gate drops strict accuracy by 22.7 points (70.8% to 48.1%).
- The lead traces to the Drafter configuration and the commit rule rather than rewriting, since 98.6% of runs commit at Round 1 and no run ever passes via the rewrite loop.
- The per-claim audit trace detects under-supported claims at F1 0.84 as a binary flag against law-trained annotators, while its four-way verdict labels are advisory only (kappa=0.275).
- Multi-agent pipelines without auditing (CrewAI, LangGraph) degrade strict accuracy to 21.6%, below single-LLM baselines, because each extra call adds chances to fabricate citations.
Strengths
Honest attribution discipline
Appendix C shows 912/925 runs (98.6%) commit at Round 1 and no run ever reaches a passing answer through rewriting, so the paper explicitly refuses to credit the Critic or audit machinery with any accuracy gain (Appendix B); the only isolated architectural effect is the commit rule (Table 3: strict 70.8% to 48.1%, -22.7 points).
Controlled evaluation design
All six systems share one backbone (Nemotron-Cascade-2-30B, vLLM FP8, T=0.7), one BM25 index of 13,090 passages (top-8), and a verbatim shared citation-instruction block (Section 4, Appendix E); the strict criterion is computed post-hoc identically for all systems and never triggers a retry, preventing evaluation gaming.
Marker-normalization control
Statistical and judge care
Exceptional design documentation
Weaknesses
Uncontrolled prompt-vs-architecture confound in the headline
Two-agent framing outruns the mechanism
Strict criterion measures resolution and form, not semantic support
Small benchmark with per-domain sign flips
Fragile transfer and closed artifacts
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
GANDR targets a real gap: grounded-generation pipelines score answers in aggregate, so a correct legal disposition can ride on fabricated or loosely matched citations. The paper contributes (i) a two-agent system - a Drafter emitting CREAC-structured answers with inline {cite: rule_id} markers and a separate-context Critic auditing each claim against its cited source - and (ii) a deterministic strict criterion (correct holding plus every citation resolving to a retrieved passage) applied uniformly post-hoc. The headline claims are specific: first place on every primary metric of a 185-item LegalBench/LegalBench-RAG benchmark (Table 1: 71.9% lenient, 70.8% strict, S&C@5 0.460), +11.3 strict points over the strongest baseline B2 (McNemar p=0.006), a -22.7-point drop when the commit rule is reverted (Table 3), and +3.2 to +6.5 strict leads on three further backbones (Table 2). The Section 6 scope caveat is commendable: the benchmark is classification/NLI-weighted over supplied authority, so GANDR is validated as a citation-discipline layer, not open-corpus legal research.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is GANDR?
GANDR (Grounded ANswer DRafter) is a two-agent system for legal question answering: a Drafter writes a CREAC-structured answer (Conclusion-Rule-Explanation-Application-Conclusion) with inline citations, and a separate Critic - seeing only the question, retrieved passages and final draft - audits every claim against its cited source, shipping a per-claim audit trace with each answer. Commits are gated by a deterministic structural check, not by the Critic's verdict.
How much better is GANDR than existing approaches?
On a 185-item benchmark built from LegalBench and LegalBench-RAG, GANDR reaches 70.8% strict accuracy (correct conclusion plus every citation resolving to a retrieved passage) versus 59.5% for the strongest baseline, a single LLM given the same CREAC prompt - an 11.3-point lead (McNemar p=0.006). On three other model backbones the lead is smaller, +3.2 to +6.5 points, measured in single runs.
Does the Critic agent actually rewrite and fix wrong answers?
Almost never by rewriting: 98.6% of runs commit on round one and no run ever reaches a passing answer through the rewrite loop. The Critic's practical contribution is the audit trace - as a binary flag it detects under-supported claims at F1 0.84 against law-trained annotators, though its four-way verdict labels are only weakly calibrated (kappa=0.275) and are advisory.
Is GANDR's code or data publicly available?
The code is stated as available upon request, not publicly released. The underlying benchmarks are public (LegalBench, MIT license; LegalBench-RAG, CC-BY-4.0), and the paper reproduces all prompts, rubrics, pipelines and benchmark construction in its appendices, which would allow a faithful reimplementation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 and Figure 2 are extracted with interleaved layout lines; all numeric table cells and captions were readable, but figure-internal arrows and labels were reconstructed from jumbled text.
- The abstract as supplied in the task metadata is cut mid-word ('whi'), but the complete abstract is present at the top of the embedded full text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.