Version- and Scope-Aware Question Answering over Normative Documents: A Deployed System and an End-to-End Evaluation at Production Scale
Liuyin Wang, Shuaipeng Jin, Jiwei Shi, Jensen Hsu
cs.AI
Quick verdict
Governed RAG beats Gemini File Search 97.7-88.1 on its own 200-question normative-doc benchmark; reproducible but whole-system, so governance isn't isolated.
Summary
The paper presents DeepKnown, a commercial Chinese-language question-answering system for normative documents (notices, service catalogs, fee schedules), and compares it end to end against Google Gemini File Search on the operator's production candidate corpus of roughly 73,000 documents. DeepKnown resolves document version, effective status, and applicability scope (jurisdiction, subject, time) at ingestion as typed fields enforced as hard predicates before ranking, and gates generation on an external evidence-sufficiency test. On 200 stratified-sampled questions, each with a gold source document, drawn from the authors' 460-item benchmark, DeepKnown scored 97.7 versus 88.1 (paired bootstrap 95% CI [5.7, 13.8]); the hosted service lagged furthest on partial-answer items (73.1 versus 92.3, n=13). Its 15 zero-score answers decompose into 6 empty returns, 2 uncited answers, and 7 cited-but-wrong answers. The question set, judge-supplied answer texts, scores, rationales, sampling script, and a reproduction script are public. Main limitations: the comparison is whole-system (governance, model, index, and template all differ) with no ablation isolating governance; benchmark and product share authors, with no external annotation or human-judge agreement; the LLM judge sees only the first 1,500 characters of each answer; deployment statistics are operator-reported and unaudited.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A governance layer that resolves document version, effective status, and applicability scope as typed predicates before ranking outperforms the hosted RAG default by 9.6 points (97.7 vs 88.1, 95% CI [5.7, 13.8]) on 200 questions over the ~73,000-document production corpus.
- The governed system's advantage holds for simple, complex, and partial-answer questions and is largest on partial-answer items, where the correct behavior is to mark what the corpus does not support (92.3 vs 73.1).
- Hard ingestion-time predicates, version lineage stored as fields, and an external evidence-sufficiency gate are architecturally preferable to similarity weighting, chunk-text provenance headers, and generator self-assessment for normative documents.
- The governed configuration is a production commercial system — live since January 2026, 1,126 registered users, roughly 100,000 calls per workday by April 2026 — so the evaluation reflects a deployed rather than prototype setting.
Strengths
Production-scale end-to-end evaluation
Both systems run on the ~73,000-document candidate corpus actually supplied to the deployment — including near-duplicates, superseded versions, and cross-jurisdiction texts — and the metric is the user-facing answer, not retrieval proxies (Why End-to-End Scores; Systems, Corpus, and Controls). This directly answers the buy-or-build question operators face, which the Related Work section argues prior hosted-service literature does not.
Unusual release of per-item evaluation artifacts
The released subset-200 includes gold source documents, the exact answer text supplied to the judge, citation counts, scores, and judge rationales for both systems, plus scripts/make_subset.py (seeded stratified sampling that reads no system outputs) and scripts/reproduce_200.py, which recomputes every cell of Tables 1-3 and exits nonzero on disagreement (Reproducibility; Question Set).
Disciplined scoring protocol
Honest failure decomposition and hedged claims
Weaknesses
Confounded headline attribution
Self-built benchmark, self-owned product
Judge validity untested
Unquantified corpus exclusions and admitted recall risk
Deployment evidence is unaudited
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What scores did DeepKnown and Gemini File Search achieve in this paper?
On 200 Chinese normative-document questions answered over roughly 73,000 candidate documents, the governed DeepKnown system scored 97.7 out of 100 on average, while Google Gemini File Search scored 88.1 — a 9.6-point gap with a 95% bootstrap confidence interval of [5.7, 13.8]. The advantage was largest on partial-answer questions (92.3 vs 73.1) and smallest on simple single-provision lookups (99.0 vs 91.1).
What is version- and scope-aware question answering?
It is question answering over regulations and similar documents where a correct answer requires more than finding similar text: the system must cite the currently effective version of the document (not a superseded or repealed one) and confirm the document applies to the asker's jurisdiction, applicant category, and date. Passage-level semantic similarity cannot determine these properties, because near-identical documents can differ exactly on the point that decides the case.
Is the benchmark and evaluation data from this paper publicly available?
According to the paper, yes: the 200-question subset with gold source documents and expected answer points, the exact answer text evaluated for both systems, per-question scores and judge rationales, the sampling script (scripts/make_subset.py), and a reproduction script (scripts/reproduce_200.py) are hosted in the GitHub repository ASI2030/normative-qa-benchmark under subset-200, alongside the original 460-item release.
Is DeepKnown a real deployed product or just a research prototype?
The paper reports it has operated commercially since January 2026 at yun.dknowc.cn, serving 1,126 registered users including named organizations Zhipu AI and Lecheng Health, with workday call volume reaching roughly 100,000 by April 2026. These operational figures are reported by the operator and are not independently audited in the paper; answer quality is measured separately on the 200-question benchmark.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 appears only as a caption in the extracted text; its visual content could not be reviewed.
- External artifacts (the GitHub repository, released per-question records, and the deployed service) could not be accessed from the provided text, so reproducibility judgments rest on the paper's descriptions.
- Judge prompts and individual per-question records are described but not embedded in the text, so scoring decisions, including the 8 rule-assigned zeros, could not be independently spot-checked.
- DeepKnown's base model, extraction-confidence threshold, and prompt/answer template are not specified in the text, limiting assessment of the whole-system confound.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.