Enforcing Narrative Reliability and Epistemic Pacing in LLM-Driven Detective Games via Structured Knowledge Trees
cs.AI · cs.CL
Quick verdict
Proposes SKT+tri-agent pipeline: critical hallucinations 17.80%→6.27%, premature confessions 10/33→0/33; promising but confounded evaluation, no code.
Summary
The paper presents a Structured Knowledge Tree (SKT) architecture and a tri-agent LLM pipeline—Analysis (node retrieval), Dialogue (generation), Detection (verification)—to control an LLM-driven interrogation game, The Interrogation of Adrian Gale. Authored clues live in a hierarchical JSON tree whose node states gate what the suspect may reveal; a contradiction-matching UI and a phase system hard-lock the confession behind four phases. In a within-subjects study (33 participants; two 20-minute sessions with different stories per version; randomized order; all agents using Llama 3.1 70B via Amazon Bedrock), author-coded gameplay logs showed critical hallucinations in 6.27% of SKT responses versus 17.80% for the LLM-Only baseline (a 64.78% relative reduction; 49 vs 152 events), with out-of-character system breaks eliminated. No participant extracted a confession in the SKT version (0/33 vs 10/33). However, all nine Likert survey items showed no significant differences after Bonferroni-corrected paired t-tests, so the claimed subjective sense of progression rests on directional (Q5) evidence only, and the SKT version scored worse on abrupt 'out-of-nowhere' reveals (Q2). Main limitations: the comparison bundles the SKT, tri-agent design, node board, phase gates, and different stories, so no component effect can be isolated; there are no ablations, no inter-rater reliability for hallucination coding, no reported p-values or effect sizes, and no code or data release.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The SKT + tri-agent architecture reduces critical hallucinations from 17.80% to 6.27% of agent responses (64.78% relative reduction) versus an LLM-Only baseline.
- The architecture entirely prevents premature narrative disclosure: 0/33 participants extracted a confession in the SKT version versus 10/33 in the LLM-Only baseline within 20-minute sessions.
- Strict mechanical enforcement causes 'forced information reveals' — the SKT version rated worse on information appearing 'out of nowhere' (Q2) — an acknowledged usability trade-off.
- Despite the friction, players maintained measurable structured progression (24/33 found at least one contradiction, averaging 6.79 prompts each) and a directionally clearer sense of progression (Q5).
Strengths
Well-motivated problem framing
The Introduction and Related Works crisply articulate why truth-maximizing RAG and roleplay-oriented state trackers conflict with the adversarial 'epistemic pacing' a mystery demands, positioning the work against Facade, FictionRAG, Dirigent, Her Story, and commercial titles (Vaudeville, Portopia AI preview).
Objective behavioral metrics, not just self-report
Hallucination measurement is grounded in full gameplay transcripts with a five-type taxonomy and explicit exclusion criteria (RQ1 section); headline outcomes are measured ratios (17.80% vs 6.27%; n=152 vs n=49 events) and confession counts (10/33 vs 0/33), not Likert ratings.
Honest reporting of negative and inconvenient results
Precise engineering specification
Learning-effect controls with a check
Weaknesses
Holistic confound with no ablations
Statistical reporting contradicts the abstract
Near-tautological pacing comparison
Opaque hallucination measurement
Narrow sample and truncated sessions
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a genuine, well-scoped problem: in LLM-driven detective games, truth-maximizing generation and retrieval are exactly wrong — the system must withhold information and pace its disclosure. The proposed solution couples a Structured Knowledge Tree (SKT) — a hierarchical JSON of authored clue nodes carrying reveal state, truth/lie flags, prerequisites, contradiction links, and children — to a tri-agent pipeline: an Analysis LLM that selects the single most relevant unrevealed node (or null), a Dialogue LLM that performs the suspect while constrained to that fact, and a Detection LLM that verifies the fact was actually conveyed before the node flips. Three research questions are tested on 'The Interrogation of Adrian Gale': hallucination reduction (RQ1), pacing enforcement (RQ2), and perceived progression (RQ3). The paper's most useful conceptual contribution is redefining 'hallucination' for adversarial fiction — where lying is a design feature — yielding a five-type taxonomy with explicit exclusion criteria (benign improvisation; premature disclosures counted as pacing failures rather than hallucinations).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a Structured Knowledge Tree (SKT) in LLM detective games?
In this paper, an SKT is a hierarchical JSON structure where each node is one pre-written clue with metadata: whether it has been revealed, whether it represents truth or a lie, prerequisite nodes for automatic deduction, contradiction links, and child nodes unlocked on reveal. It acts as the ground truth limiting what the LLM-controlled suspect may say, and is shown to players as a node board with unrevealed facts displayed as question marks.
How much did the system reduce hallucinations?
In a 33-participant user study, author-coded gameplay logs found critical hallucinations in 6.27% of suspect responses in the SKT version versus 17.80% in an unconstrained LLM-Only version — a 64.78% relative reduction (49 vs 152 hallucination events). Out-of-character system breaks dropped to zero in the SKT condition.
Could players still force a confession early?
No. In the SKT version, zero of 33 participants extracted a confession within 20-minute sessions, because the confession is hard-locked behind four narrative phases and a contradiction mechanic. In the LLM-Only baseline, 10 of 33 players forced a confession, typically via fabricated-evidence bluffs or roleplay hijacks such as impersonating the suspect's lawyer.
Did players prefer the stricter SKT system?
Mixed. After statistical correction, no survey item differed significantly between versions. The SKT version scored better on perceived logical progression (directionally, Q5) but worse on information seeming to appear 'out of nowhere' (Q2), because retrieved clues were sometimes forced into dialogue unnaturally; some roleplay-oriented players preferred the freer baseline.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are not analyzable from the extracted text: Figure 4's exact Likert distributions and Figure 3's architecture diagram are known only through captions and surrounding prose.
- Extraction artifacts (survey axis labels such as '20% 40% 60% 80% 100%' embedded mid-sentence before Q1) indicate figures bled into the text; some layout-level content may be imperfectly captured.
- No supplementary material (full survey instrument, agent prompts, story scripts) is included, so the coding scheme and prompt engineering cannot be verified from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.