Beyond Linear Context: Graph-Guided Evidence Navigation for Long-Novel Reasoning with a Local 9B Language Model
cs.AI · cs.CL
Quick verdict
Graph navigation for a 9B reader hits 53.85% vs 46.15% tail-window on DetectiveQA; no contrast survives Holm — honest but unproven exploratory design study.
Summary
The paper asks whether a frozen knowledge graph helps a small local language model answer questions about full detective novels. Using DetectiveQA (30 novels, 234 four-option multiple-choice questions with annotated clue/answer paragraphs), the author fixes one qwen3.5:9b reader and runs nine conditions: five graph navigation routes (three order permutations, a graph-only majority vote, and a fusion route combining keyword, dense and graph rankings), a last-50k-character tail window, hierarchical whole-book compression, ordinary vector retrieval, and a question-only control. The best graph route reaches 53.85% versus 46.15% for the tail window, 51.28% for compression, 51.71% for vector retrieval and 40.17% question-only; on the 140 questions unanswerable without the book, the graph route scores 42.86%. Crucially, the authors report that none of the fifteen graph–baseline contrasts survives Holm correction (best raw p=0.0505, adjusted 0.76), labeling everything exploratory because G4/G5 were selected while inspecting the corpus. The more durable contributions are an evidence audit — gold-overlap nodes are 2.35× enriched in the graph 2-core — and the finding that the two graph-builder pipelines map only 16% versus 73% of clue paragraphs into their graphs, so pooled accuracy confounds builder and method. Limitations: 234 questions, one reader, one translated corpus, no compute-matched baseline in this run, and a sealed confirmatory protocol with no results yet.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A frozen knowledge graph with navigation routes raises a fixed local qwen3.5:9b reader to 53.85% (126/234) on DetectiveQA versus 46.15% for a recent-window baseline, presented as exploratory because no graph-baseline contrast survives Holm correction.
- Annotated clue and answer evidence concentrates in the topological 2-core of the graphs, with 2.35× pooled enrichment of gold-overlap nodes.
- The two graph-building pipelines map 16.3% versus 73.2% of clue paragraphs into their graphs, so pooled accuracy measures builders as much as retrieval methods.
- Options-first oracle ceilings are inflated by option layout (79% on gold-D versus 22–53% on A–C); the position-neutral oracle gives 47–62%.
- 32 of 234 questions are answered only by a graph condition while all three non-graph baselines fail, and 29 are answered by no condition.
Strengths
Statistical honesty as a design principle
The paper foregrounds that no graph-baseline contrast survives Holm correction (best raw p=0.0505, adjusted 0.76, §6.3), discloses that G4/G5 were selected during corpus inspection (§4.2), labels all five graph conditions exploratory, and describes a pre-registered confirmatory protocol on 340 questions that remains sealed (§5).
Index-quality audit separates builder from method
§6.5 splits whether annotated evidence sits in the structurally central part of the graph (2.95× old20 / 1.61× new10 2-core enrichment with Haldane ORs 5.20/2.32, pooled 2.35×) from whether the pipeline maps the annotation into the graph at all (16.3% vs 73.2% clue mapping), showing pooling would average a build defect into a retrieval score.
Provenance and reproducibility machinery
Oracle bias analysis generalizes beyond this paper
Difficulty census gives concrete next steps
Weaknesses
The headline effect is statistically null
Cohort confound dominates the evaluation
No compute-matched comparison executed
Lexical gold-overlap audit
Narrow and internally inconsistent corpus framing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper tests a narrow, well-posed question: if a novel's evidence is reorganized into a frozen knowledge graph that a retrieval policy can walk, does a small local reader answer long-novel questions better than from a linear context of the same budget? The setting is DetectiveQA (§3): thirty detective novels, 234 four-option multiple-choice questions, paragraph-level gold annotations for clue and answer. One frozen qwen3.5:9b reader (Ollama, reasoning disabled) answers under nine conditions: five graph routes — G1–G3 (deterministic option-order permutations of the same evidence), G4 (graph-only majority vote), G5 (per-option queries fusing keyword search, dense retrieval and graph ranking via reciprocal rank, at most six passages) — plus B1 (final 50,000 characters), B2 (hierarchical whole-book compression), B3 (chunk-level vector retrieval) and Q0 (question-only). Four claims are staked: an exploratory accuracy gain for the graph; 2.35× enrichment of gold evidence in the graphs' 2-core; a 16%-vs-73% mapping-recall gap between the two builder pipelines; and inflated options-first oracle ceilings. All four are stated with unusual precision about what the statistics do and do not license.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What accuracy does the graph method achieve on DetectiveQA?
The strongest graph route (G5, a fusion of keyword search, dense retrieval and graph ranking feeding at most six passages) answers 126 of 234 questions (53.85%) with a local qwen3.5:9b model. A recent-window baseline gets 46.15%, whole-book compression 51.28%, vector retrieval 51.71% and a question-only control 40.17%. On the 140 questions no model can answer without the book, the graph route scores 42.86% versus 33.57% for the tail window.
Is the improvement statistically significant?
No. The best raw p-value is 0.0505 for the +7.69-point contrast against the tail-window baseline, and after Holm correction across fifteen planned comparisons the adjusted p is 0.76. The authors explicitly label all results exploratory — G4/G5 were selected while inspecting the corpus — and a pre-registered confirmatory run on 340 questions exists but remains sealed with no results.
What is the main finding beyond the accuracy numbers?
An evidence audit of the index itself: gold-annotated evidence concentrates in the graph's topological 2-core (2.35× pooled enrichment), but the two graph-building pipelines capture very different shares of annotated clue paragraphs (16% vs 73%), so a graph can look structurally healthy while missing most of the evidence a question needs. Pooling the two cohorts would have mixed a builder defect into the retrieval score.
What model and data were used?
One frozen qwen3.5:9b model served locally via Ollama with reasoning disabled, evaluated on DetectiveQA: 30 detective novels and 234 four-option multiple-choice questions with paragraph-level gold annotations for the clue and answer. The 30 knowledge graphs were built beforehand by two different pipelines (a legacy cohort of 20 novels and a newer Pass2-v4 cohort of 10) and were never rebuilt during evaluation; SHA-256 hashes were checked before and after each run.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only through captions with embedded numbers; visual patterns (e.g., the per-novel heatmap, Figure 6) could not be inspected directly.
- No explicit repository URL appears in the extracted text; the released artifacts (per-question CSV, JSON manifests, figure/table scripts) are named but their access path is unverified.
- Appendix-level details such as exact prompts, route hyperparameters and the sealed confirmatory protocol's full text are not in the extracted text.
- Minor internal inconsistency on corpus language (§3 'Chinese translation' vs §8 'English-translated Chinese corpus') could not be resolved from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.