CiteGuard-RAG: A Validation-Centered AI System for Evidence-Grounded Question Answering
Sumit Barua, Guan Hong, Halil Dursunoglu, Charles Rodgers, Alvis Fong
cs.CL · cs.AI · cs.IR
Quick verdict
CiteGuard-RAG builds a sensible validation layer, but its 98.3 percent and 0.0 percent headline numbers are produced by that same layer and are close to circular.
Summary
CiteGuard-RAG is a retrieval-augmented question-answering pipeline for legal and policy documents built around a runtime validation layer rather than around generation. Documents are parsed page-aware, chunked paragraph-aware at 600 tokens with 120-token overlap, indexed with BM25 and BAAI/bge-base-en-v1.5 embeddings fused by min-max normalised scores at 0.65 semantic and 0.35 lexical, and queried at top-k = 8 with up to four chunks passed to a locally deployed instruction-tuned generator at temperature 0. The candidate answer is then split into sentences, each checked for citation validity plus lexical token overlap or semantic similarity against cited evidence, with thresholds of 0.10 and 0.40 set from pilot runs. A failure triggers exactly one regeneration pass with validation feedback; a second failure yields a canned refusal. On 150 author-built housing-law questions the system reports 99.1 percent retrieval accuracy, 98.3 percent grounded-answer accuracy, 98.3 percent citation validity and 0.0 percent hallucination, with Mistral and Qwen giving identical figures. The most informative result is Table 7: turning validation off leaves retrieval at 99.1 percent but drops grounded accuracy to 8.7 percent and citation validity to 12.3 percent. External evaluation on 200 PrivacyQA and 50 CUAD questions is much weaker, with 68.5 and 54.0 percent answer accuracy, 18.7 and 56.0 percent false refusal.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 3
● low 0
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- High retrieval accuracy alone does not guarantee grounded or citation-valid answers; disabling validation leaves retrieval unchanged at 99.1 percent while grounded answer accuracy falls to 8.7 percent.
- Sentence-level grounding validation should operate at runtime as a routing decision among accept, regenerate once and refuse, not as a post-hoc evaluation step.
- Hybrid semantic-lexical retrieval at 0.65/0.35 weighting beats either channel alone, and more semantic-heavy retrieval increases hallucination risk.
- Retrieval coverage and citation validity transfer across domain shift more robustly than answer-span selection and abstention calibration.
- Adding components can reduce reliability: a heuristic reranker lowered retrieval accuracy from 99.1 to 87.7 percent and was therefore disabled.
Strengths
The pipeline is specified in enough detail to rebuild
Section 2 gives concrete operational settings: 600-token chunks with 120-token overlap, top-k = 8 with at most four chunks forwarded to the generator, hybrid weights 0.65 semantic and 0.35 lexical over min-max normalised scores, BAAI/bge-base-en-v1.5 embeddings, temperature 0, one regeneration pass, and validation thresholds of 0.10 lexical overlap and 0.40 semantic similarity. Algorithm 1 states the accept, regenerate and refuse routing unambiguously.
The validation ablation isolates the actual contribution
Table 7 is the most informative result in the paper: with validation on versus off, retrieval accuracy is identical at 99.1 percent while grounded-answer accuracy moves from 98.3 to 8.7 percent, citation validity from 98.3 to 12.3 percent, and hallucination from 0.0 to 18.6 percent. Whatever one thinks of the metrics, this cleanly demonstrates that retrieval success and citation-grounded output are separate things.
Two error metrics are deliberately separated
Honest negative results on components that did not help
Weaknesses
The headline accuracy and hallucination numbers are scored by the same validator that gates delivery
Identical cross-model results with an unidentified primary generator
The support rule is permissive enough to reduce validation to citation formatting
External performance collapses the usefulness story
No comparison against any external RAG system or validation method
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CiteGuard-RAG addresses a real failure mode: RAG pipelines retrieve correct evidence and still produce answers that are uncited, wrongly cited or silently fabricated. The paper reframes grounding as a runtime control problem, inserting a sentence-level validator between candidate generation and delivery that routes each answer to acceptance, a single regeneration pass, or a standardised refusal. Five contributions are claimed, of which three are delivered: the architecture itself, the validation mechanism, and the multi-setting evaluation across 150 housing-law questions plus 200 PrivacyQA and 50 CUAD items. The claim that this yields 98.3 percent grounded accuracy and no hallucinations is where the paper outruns its evidence, because those two numbers are computed by the same gate that suppresses non-compliant answers.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CiteGuard-RAG?
CiteGuard-RAG is a retrieval-augmented question-answering system for legal and policy documents whose central component is a validation layer rather than a better generator. It retrieves evidence with a hybrid of BM25 lexical matching and dense embeddings (BAAI/bge-base-en-v1.5, weighted 0.35 and 0.65), asks a local instruction-tuned model to answer only from those sources with inline citations, then checks every generated sentence for a valid citation and for lexical or semantic support in the cited passage. A failed check triggers one regeneration using the validator's feedback; if that fails again the system returns a fixed refusal instead of an answer.
How accurate is CiteGuard-RAG?
On the authors' own 150-question housing-law set it reports 99.1 percent retrieval accuracy, 98.3 percent grounded-answer accuracy, 98.3 percent citation validity and 0.0 percent hallucination. These figures should be read with care: answers that fail validation twice are converted into refusals, so the answers that survive are the ones the validator already approved. On external benchmarks the picture is weaker: 68.5 percent answer accuracy with 18.7 percent false refusal on 200 PrivacyQA questions, and 54.0 percent answer accuracy with 56.0 percent false refusal on 50 CUAD questions.
Does removing the validation layer really hurt?
Yes, sharply, and this is the most useful result in the paper. With validation disabled, retrieval accuracy stays at 99.1 percent but grounded-answer accuracy drops from 98.3 to 8.7 percent, citation validity from 98.3 to 12.3 percent, and hallucination rises from 0.0 to 18.6 percent. The result cleanly separates evidence access from evidence use: finding the right passage does not mean the model will cite it correctly. The caveat is that the ablations use the same automatic checks for both routing and scoring.
Is the CiteGuard-RAG code and dataset publicly available?
No. The paper makes no code, model or data availability statement, and its controlled evaluation set is a purpose-built corpus of 150 questions over 18 housing-law documents drawn from multiple U.S. states and cities, 100 human-authored and 50 LLM-assisted. The external evaluations use publicly available PrivacyQA and CUAD subsets, which is the only part of the study a third party could reproduce today.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 are diagrams and are not present in the extracted text; their content was reconstructed from the captions and Section 2.
- The generator producing the headline Table 3 results is unnamed, and model sizes are not given, so the cross-model comparison cannot be fully assessed.
- No availability statement appears for the code or the controlled housing-law corpus, so nothing in the controlled evaluation can be independently re-run as reported.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.