Decoupling Error Attribution in Cloud-Native Graph-RAG: A Data Integrity Diagnostic Framework
Shuai Yan, Yuhang Wu, Xiaodong Huang, Ke Wang
cs.IR · cs.AI
Quick verdict
A three-layer Graph-RAG error audit finds data defects dominate and LLM memory masks them; promising, but tiny single-domain setup and unproven mechanism.
Summary
This paper proposes a three-layer diagnostic framework that decomposes Graph-RAG end-to-end error into reasoning loss, knowledge-graph defect error, and Cypher generation error via an additive identity across Oracle Evidence, Oracle Cypher, and End-to-End evaluation layers. Experiments use a small spatio-temporal ecological knowledge graph of Southeastern Tibet (1,172 nodes, 3,428 relationships), 100 manually authored questions, and eight defect types injected at a fixed 20% rate into cloned Neo4j databases, with Qianwen-Flash generating Cypher, Qianwen-Plus answering, and Qianwen-Max as LLM judge (2,700 calls). Clean-graph accuracy is 0.93 (Oracle Cypher) versus 0.94 (Oracle Evidence, p=0.706), so reasoning loss is negligible; structural defects drop Oracle Cypher accuracy to 0.39–0.45 (εKG up to 0.54, d=0.92–1.08, p<0.001) while numeric defects yield 0.58–0.62. Apparent Cypher generation error shrinks from 0.18 on the clean graph to 0.01–0.05 under severe structural defects (Masking Index 72.2% for Evidence Gap), which the authors name the Parametric Knowledge Masking Effect and attribute to LLM fallback on internal memory. Limitations: a single tiny domain, one injection rate, no direct evidence for the fallback mechanism, an orthogonality assumption contradicted by the observed interactions, no code or data release, and a cloud-native framing beyond the single-instance setup.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Data integrity rather than algorithmic reasoning is the dominant bottleneck: OC_clean 0.93 vs OE 0.94 (Wilcoxon p=0.706), while all eight defect types at 20% injection drop accuracy to 0.39–0.62.
- Structural (topology-breaking) defects cause roughly twice the damage of numeric defects (OC 0.39–0.45 vs 0.58–0.62; εKG up to 0.54 with d=0.92–1.08, p<0.001).
- Apparent Cypher generation error shrinks by over 70% under severe structural defects (0.18 → 0.01–0.05; Masking Index 72.2% under Evidence Gap), a Parametric Knowledge Masking Effect attributed to LLM fallback on parametric memory that creates false negatives for E2E-based monitoring.
- A three-layer (Oracle Evidence / Oracle Cypher / E2E) design orthogonally attributes Graph-RAG errors to reasoning, KG, and query-generation components.
Strengths
Clean control-variable design
Holding gold Cypher fixed in the Oracle Cypher layer lets εKG isolate pure storage damage; Table II defines each layer crisply, and Fig. 2/3 numbers reconcile exactly (e.g., Evidence Gap: OC 0.42, εKG 0.51, εcypher 0.05, summing back to the 0.93 clean baseline).
Above-average statistical hygiene
Wilcoxon signed-rank tests plus paired Cohen's d per defect type; the contrast between εKG (p<0.001 for all eight defects) and reasoning loss (p=0.706, d=−0.038) is reported consistently, with effect sizes stratifying structural (0.92–1.08) vs numeric (0.62–0.66) defects.
Actionable observability finding
Structural-vs-numeric prioritization rule
Honest scoping of the injection rate
Weaknesses
Orthogonality claim contradicted by its own results
PKME mechanism never directly tested
Tiny, single-domain evidence base
Single injection rate, no severity response
MI normalization and missing variance
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a real gap in Graph-RAG evaluation: end-to-end (E2E) scores cannot tell whether a failure stems from the LLM's reasoning, corrupted graph data, or malformed generated Cypher. The authors propose three evaluation layers — Oracle Evidence (gold Cypher on the clean graph with perfect textual evidence), Oracle Cypher (gold Cypher on defective graphs), and E2E (generated Cypher on defective graphs) — whose accuracies telescope into an additive decomposition: ε_total = ε_reason + ε_KG + ε_cypher (Eq. 1). Three claims are advanced: (1) data integrity, not reasoning, dominates (OC_clean 0.93 vs OE 0.94, p=0.706); (2) structural defects hurt roughly twice as much as numeric defects (OC 0.39–0.45 vs 0.58–0.62 at 20% injection); (3) a 'Parametric Knowledge Masking Effect' (PKME) makes apparent Cypher error collapse (0.18 → 0.01–0.05; Masking Index 72.2% under Evidence Gap), hiding storage corruption from E2E-based monitoring.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Parametric Knowledge Masking Effect (PKME)?
The paper's name for the observation that when graph retrieval paths break, the LLM appears to answer from internal memory, shrinking apparent query-generation errors by over 70% and hiding real data corruption from end-to-end monitoring. The authors present it as a hypothesis supported indirectly by error-component shrinkage, not by direct analysis of answer content.
How badly does data corruption hurt Graph-RAG accuracy in this study?
With a 20% defect injection rate on a 1,172-node ecological knowledge graph, accuracy fell from 0.93 on the clean graph to 0.39–0.62 depending on defect type; topology-breaking structural defects were roughly twice as damaging as numeric attribute defects (0.39–0.45 vs 0.58–0.62).
Is code or data available for this paper?
No release is stated. The paper describes a self-built spatio-temporal knowledge graph (1,172 nodes, 3,428 relationships), 100 manually written questions with gold Cypher, and Qianwen-family models, but provides no code, data, or prompt artifacts, which limits reproducibility.
Which models were used in the experiments?
Qianwen (Qwen) models throughout: Qianwen-Flash (temperature 0.2) generated Cypher queries, Qianwen-Plus answered questions with extended reasoning, and Qianwen-Max (temperature 0) served as the LLM-as-Judge scorer; 2,700 inference calls were executed across 9 databases, 100 questions, and 3 layers.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2–4 appear in the extraction as interleaved axis-label fragments; figure-level values quoted here were cross-checked against the prose of Sections V-A/V-B wherever possible, so figure-only details carry minor OCR-ordering risk.
- Fig. 1 (architecture diagram) is an image not rendered in the extracted text; the framework was assessed from the Section IV prose and Table II alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.