EvidenT: Building Trustworthy Enterprise Assistants through Evidence Groundedness and Traceability
Anubha Kabra, Katie Jooyoung Kim, Colin Zhiwei Kou, Helene Sajer, Yimei Fan, Radomir Cisar, Heather Greenhalgh, Gabriel Martinez Vidiri
cs.AI
Quick verdict
Training-free n-gram verification lifts enterprise RAG groundedness (77-93% vs 47-73%) but traceability wins are partly by construction.
Summary
EvidenT is a training-free pipeline from Bloomberg that improves citation reliability in extractive enterprise question answering. After federated retrieval over internal sources, an LLM extracts candidate passages verbatim as structured JSON; a deterministic n-gram alignment module (n=5, overlap threshold 0.7) verifies each passage against its cited document — retaining, truncating, re-attributing drift to the correct retrieved source, or dropping it — before a second LLM call generates the cited answer. On roughly 500 real enterprise queries from a one-month pilot (55 users, ~4,000 queries), with LLaMA-3.1-8B and LLaMA-3.3-70B, groundedness against SME gold documents rises from 52% (direct prompting) and 47% (ground-every-sentence) to 77% on the 8B model, and from 73%/71% to 93% on the 70B model, with zero hallucinated citations and only ~2s added latency (12±3s vs 10±3s). Answer-to-source lexical coverage is near-saturated (AnsCov 0.98-0.999), though largely guaranteed by the copy-verbatim design. Semantic gains rely on raw, unnormalized cross-encoder scores from an unnamed model. A five-month production deployment (200+ users) reports +28% helpfulness feedback and 90% retention, without controls. Main limitations: proprietary data, no code release, hyperparameters tuned on the evaluation set, and a small 50-query, two-annotator human study.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Verifying LLM-extracted passages with deterministic n-gram alignment against retrieved documents eliminates hallucinated citations and corrects citation drift without any model retraining.
- Groundedness against SME gold documents rises from 52%/47% to 77% (LLaMA-3.1-8B) and from 73%/71% to 93% (LLaMA-3.3-70B) versus two prompting baselines on ~500 real enterprise queries.
- Answers achieve near-saturated answer-to-source lexical coverage (word-level AnsCov 0.998–0.999; Cov@10 0.979–0.986), enabling exact span-level traceability in the interface.
- The pipeline costs only ~2s of added latency (12±3s vs 10±3s), far less than verifier-based or fine-tuning alternatives.
- A five-month production deployment (200+ users) showed +28% helpfulness feedback, +19% user-rated contextual relevance, and 90% retention.
Strengths
Deployable, training-free design targeting observed failure modes
Pipeline is motivated by concrete pilot failures (Table 1: hallucinated links, citation drift, limited traceability); the deterministic n-gram check (§3.3) adds only ~2s latency (§8: 12±3s vs 10±3s) and no retraining, matching the enterprise constraints argued in Appendix H.
Large, statistically tested groundedness gains at two model scales
Table 2: groundedness 77% vs 52%/47% (LLaMA-3.1-8B) and 93% vs 73%/71% (LLaMA-3.3-70B), zero hallucinated citations with ≤0.6 upper bound, 95% binomial CIs and two-proportion z-tests at p<0.05.
Honest intermediate error analysis and ablation
Complete prompt documentation
Two-scale real deployment evidence, appropriately hedged
Weaknesses
Traceability metrics are largely guaranteed by construction
Headline derived percentages do not reconstruct from the tables
Tuning and evaluation share the same ~500 queries
Thin and internally inconsistent human evaluation
Uninterpretable semantics and low reproducibility
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
EvidenT (arXiv 2609.22537) is a Bloomberg system paper targeting three failure modes observed in a one-month pilot RAG assistant (55 users, ~4,000 queries): hallucinated URLs, citation drift, and limited span-level traceability (Table 1). The method is a training-free, post-retrieval pipeline: federated retrieval → LLM verbatim passage extraction (§3.2) → deterministic n-gram alignment/filtering (§3.3; n=5, threshold 0.7, with retain/truncate/re-attribute/drop actions) → answer generation from verified passages (§3.4) → cosmetic post-processing (§3.5). Central claims: (1) groundedness against SME gold documents improves an average of 29% over prompting baselines; (2) zero citations to non-retrieved URLs; (3) near-saturated answer-to-source lexical coverage; (4) only ~2s added latency.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is citation drift in RAG systems?
Citation drift is when a model cites a real retrieved document, but that document does not actually contain the text supporting the generated claim — the citation looks valid but is attached to the wrong source. EvidenT detects this via n-gram mismatch between the extracted passage and the cited document, then re-attributes the passage to the retrieved document that does contain the matching text.
Does EvidenT require fine-tuning or retraining?
No. It is a training-free pipeline layered on top of existing retrieval and generation components: one LLM call extracts verbatim passages, a deterministic n-gram check verifies them against the cited documents, and a second LLM call writes the cited answer.
How much latency does EvidenT add compared to standard RAG?
About 2 seconds: the paper reports 12±3s end-to-end versus 10±3s for a baseline single-call RAG system, because the verification step itself is a cheap lexical (n-gram) computation rather than extra model calls.
Is EvidenT's code or dataset publicly available?
No. The enterprise corpus, queries, and gold annotations are proprietary (Bloomberg internal data), and no code release is mentioned. The prompts are published in the paper's appendices, but the headline numbers cannot currently be independently re-derived.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 (pipeline and workflow diagrams) are referenced but not analyzable from the extracted text.
- Some tables (e.g., Table 6, Table 7) may contain extraction/spacing artifacts; values are quoted as extracted.
- The underlying data is proprietary, so gold-annotation quality and the 29% derivation could not be cross-checked beyond the paper's own tables.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.