Staged Linguistic Seeding: Grounded Query Expansion for Verified-Unit QA in AI Contact Centers
Hyeonseop Yoon, Jeong-Eun Park
cs.CL · cs.IR
Quick verdict
Honest application study: human-seeded LLM expansion lifts FAQ retrieval R@1 +0.27/+0.34 on private contact-center data; strong controls, in-distribution eval.
Summary
This application study deploys a generation-free QA system in an AI contact center that answers only from a closed set of human-verified QA units, routing weak-evidence queries to clarify, abstain, or handoff. Because callers phrase one need in many ways while each unit ships a single representative question, the authors propose staged linguistic seeding (SLS): a human authors a per-unit world-grounded slot recipe, gpt-4.1-mini renders it into candidate question variants, and a human gate filters them — all offline, so serving stays a single retrieval pass. On held-out recipe-reformulation queries from two proprietary Korean domains (AUTO, 90 units; ELEC, 229 units), SLS lifts hybrid (BM25+BGE-M3) R@1 from 0.609 to 0.881 (ELEC) and 0.588 to 0.930 (AUTO), with gains across all five retrievers tested. A matched-budget control attributes the gain to the seeded design rather than generated volume: budget-matched doc2query reaches only 0.685/0.611. A faithfulness study on 136 correct-retrieval queries finds free-form RAG unsupported in 7.4–13.2% of answers versus about 0% for verbatim verified-unit answers (McNemar p=0.012). Main limitations: evaluation queries are in-distribution recipe variants with a size-confounded cross-provenance control; the faithfulness judge is a single LLM without human adjudication; no gate-only ablation separates the human gate from the recipe; and the enterprise data is private, with code promised only upon acceptance.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Offline SLS expansion lifts hybrid (BM25+BGE-M3) R@1 from 0.609 to 0.881 (ELEC) and 0.588 to 0.930 (AUTO) on held-out recipe variants, with gains across all five retriever families tested.
- At the same generator and per-unit budget, SLS beats doc2query by +0.196/+0.319 R@1, attributing the gain to the world-grounded seeding design rather than the model or generated volume.
- Verified-unit answering removes free-form RAG's unsupported-content surface (7.4–13.2% vs approximately 0% on 136 correct-retrieval queries, single LLM judge, McNemar p=0.012/7.6e-5) at comparable gold coverage.
- The expansion gain concentrates on surface-disjoint reformulations (+0.59 on the 49% of AUTO held-out queries sharing no content word with the canonical question), consistent with a symbol-grounding account.
- Question-identity leakage inflates closed-set FAQ retrieval by up to roughly 8x for BM25 (2.6–5.5x across families) and is avoided by held-out-variant protocols.
Strengths
Matched-budget control isolates the mechanism
Table 2's doc2query N-matched arm holds the generator (gpt-4.1-mini), per-unit volume, and test set fixed; SLS still wins by +0.196/+0.319 hybrid R@1 with cluster-bootstrap CIs, cleanly supporting the 'seeding design, not volume' attribution.
Leakage audit with standalone methodological value
App. A/Table 4 shows question-identity self-retrieval inflates BM25 R@1 roughly 8x (0.863 leaky vs 0.103 answer-only) and 2.6–5.5x across families; the pitfall replicates on QQP (R@1 1.000 vs 0.81–0.94, App. C), a useful warning for any closed-set FAQ evaluation.
Statistical care beyond typical application papers
Honest negatives and bounded claims
Mechanism analysis lands where the account predicts
Weaknesses
In-distribution evaluation may inflate headline gains
Human gate and recipe contributions unseparated
Faithfulness study is thin
Small proprietary corpus limits external validity
Reproducibility gated on acceptance
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a self-described application study from a two-author independent team, anchored in a live AI contact center deployment: roughly 74,298 requests over two anonymized Korean domains (AUTO automotive live since early 2025, ELEC electronics since mid-2025), with a combined 89.0% containment rate (App. D). The serving system answers only from 319 human-verified QA units (AUTO 90, ELEC 229) or routes to clarify/abstain/handoff; nothing is generated at query time. The claim chain is: verified-unit answering removes free-form RAG's unsupported-content surface by construction (§3, Eqs. 1–2); removing the generator pushes the full linguistic burden onto retrieval; staged linguistic seeding (SLS) — a per-unit, human-authored world-grounded slot recipe rendered by gpt-4.1-mini under a light human gate, all offline — closes that burden; and a matched-budget control attributes the gain to the seeding design rather than the model or generated volume (§6, Table 2). All headline numbers come from held-out recipe-reformulation variants on proprietary logs; a Quora Question Pairs replication (App. C) extends the protocol-level findings to public data.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Staged Linguistic Seeding (SLS)?
An offline query-expansion method for FAQ-style retrieval: a human writes a per-question 'slot recipe' that grounds the question's key referents in world knowledge, an LLM (gpt-4.1-mini) renders the recipe into many candidate question variants, and a light human gate filters them. The accepted variants are indexed, so inference stays a single retrieval pass with no query-time generation.
How much does SLS improve retrieval accuracy?
On the paper's two proprietary Korean contact-center domains, hybrid BM25+BGE-M3 top-1 unit recall rose from 0.609 to 0.881 (electronics, 229 units) and from 0.588 to 0.930 (automotive, 90 units) on held-out rephrased queries, with gains across all five retrievers tested. A budget-matched doc2query baseline reached only 0.685/0.611.
Does returning verified answers verbatim eliminate hallucination?
It eliminates unsupported content by construction — the answer is always a human-verified text — but a wrongly retrieved unit still yields a wrong answer. The paper measures free-form RAG producing unsupported content in 7.4–13.2% of answers on identical retrieved evidence (LLM judge, n=136) versus about 0% for verbatim verified-unit answers.
Is the data or code available?
The enterprise logs and QA units are proprietary and access-controlled. The authors state they will release the public Quora Question Pairs replication code, the pipeline code, and a fine-tuned checkpoint upon acceptance; as of this writing no code or data is publicly available.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's embedded text is interleaved with body text by the PDF extraction (its labels appear inside the §6 metrics paragraph), so the figure could only be assessed via its caption; all tables and appendices extracted cleanly and the text ends completely at Appendix H with no truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.