Beyond Vector Similarity: Hierarchical Context-Aware Graph RAG vs Standard RAG in Enterprise Code Migration
Nilesh Jaiswal, Aniket Agrawal, Arjit Shukla, Divya Malhotra, Saurabh Garg, Suchit Puri, Suddhasatwa Bhaumik
cs.AI
Quick verdict
Graph RAG cut API hallucinations (56.4%→16.2%) vs Standard RAG on one repo, but single-codebase evidence and inconsistent composite scores limit the claim.
Summary
Comparing Standard (vector) RAG against a proposed Hierarchical Context-Resident Graph (HCRG) pipeline, the authors automate Java-to-Python migration of the spring-petclinic-genai repository. HCRG parses code with tree-sitter, stores INHERITS/CALLS/IMPORTS edges in a Spanner property graph, serializes an 'architecture skeleton' into the Gemini (Vertex AI) Context Cache, and translates files in topological order, injecting generated parent classes into child prompts; the baseline chunks text recursively (800 chars, overlap 150, k=5 cosine retrieval). Both use gemini-2.5-pro at temperature 0. Seven AST-based SE metrics plus a global-lookup hallucination check yield: hallucination rate 56.4%→16.2%, Dependency Resolution Quality 34.8%→65.9%, Parent-Child Consistency 26.7%→45.5%, Type Hint Completion 51.4%→80.9%; meanwhile CodeBLEU stays ~91% for both, which the authors argue masks structural failure. Trade-offs: Cyclomatic Complexity Consistency falls 71.6%→46.7% (defensive over-engineering) and Docstring Preservation 67.0%→61.0% (attention dilution). Main limitations: a single small repository with apparently one run and no statistical testing; no code-aware chunking baseline or ablation isolating caching, topological ordering, and traversal; evaluation anchored to the authors' own symbol tables and whitelists; a closed-source stack with no prompts or code released; and per-file MQ scores (Tables 2-3) irreconcilable with the stated 40/30/20/10 weights.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- An HCRG Graph RAG pipeline cuts API hallucination from 56.4% to 16.2% versus Standard RAG on Java-to-Python migration of one repository (Table 1).
- Both pipelines score ~91% CodeBLEU (90.6% vs 91.1%), which the authors argue proves lexical metrics mask structural failure in code migration.
- Graph RAG's global context induces defensive over-engineering, degrading Cyclomatic Complexity Consistency from 71.6% to 46.7% and Docstring Preservation from 67.0% to 61.0%.
- The custom weighted Migration Quality score improves from 65.5% to 83.2% under the authors' seven-metric framework.
Strengths
Candid trade-off reporting
The paper openly reports where its method loses — CCC 71.6%→46.7% and DP 67.0%→61.0% (Table 1, §6.2) — and offers mechanistic explanations (defensive over-engineering, attention dilution) instead of claiming uniform superiority.
Compelling demonstration of lexical-metric blindness
Near-identical CodeBLEU (90.6% vs 91.1%) against a 40.2-point hallucination gap (Table 1) is a clean, memorable empirical argument that n-gram overlap cannot judge migration viability (§6.1).
Deterministic, executable evaluation design
Well-specified experimental parameters
Weaknesses
Single repository, apparently single run, no statistics
MQ composite irreconcilable with its own formula
Weak baseline and no ablation
Evaluation circularity
Closed stack, no artifacts, inconsistent repo naming
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This Google Cloud preprint (arXiv:2609.12464, cs.AI) is a comparative empirical study of Standard vector RAG versus a proposed Hierarchical Context-Resident Graph (HCRG) pipeline for automated Java-to-Python migration of a single repository, spring-petclinic-genai (§4.2). It advances four central claims: (1) HCRG decisively reduces structural failure — API hallucination falls from 56.4% to 16.2%, Dependency Resolution Quality (DRQ) rises from 34.8% to 65.9%, Parent-Child Consistency (PCC) from 26.7% to 45.5%, and Type Hint Completion (THC) from 51.4% to 80.9% (Table 1); (2) CodeBLEU is unfit for migration evaluation because both arms score near-identically (90.6% vs 91.1%), masking broken code behind lexical overlap; (3) Graph RAG pays for global context with defensive over-engineering — Cyclomatic Complexity Consistency (CCC) drops from 71.6% to 46.7% — and mild attention dilution, with Docstring Preservation (DP) falling from 67.0% to 61.0% (§6.2); (4) the weighted aggregate Migration Quality (MQ) improves from 65.5% to 83.2%.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is HCRG in this paper?
Hierarchical Context-Resident Graph: the authors' Graph RAG pipeline that parses code into ASTs with tree-sitter, stores inheritance/call/import edges in a Spanner property graph, caches a serialized 'architecture skeleton' in Gemini's Context Cache, and translates files parent-first in topological order so each child class is generated with its real parent code already in the prompt.
Why did Graph RAG and Standard RAG get the same CodeBLEU score?
Both scored ~91% (90.6% vs 91.1%) because CodeBLEU measures lexical n-gram overlap. Standard RAG produced fluent-looking but structurally broken Python (56.4% hallucinated APIs), so the paper argues n-gram metrics cannot judge code migration quality and proposes seven AST-based software-engineering metrics instead.
What are the downsides of Graph RAG found in this study?
On the paper's single-repository benchmark, Graph RAG lowered Cyclomatic Complexity Consistency from 71.6% to 46.7% (the model over-engineers when given global architectural context) and Docstring Preservation from 67.0% to 61.0% (prompt attention dilution).
Can I reproduce this paper's results?
Not directly: it relies on proprietary Gemini 2.5 Pro, Vertex AI Context Caching, and Spanner, and no prompts, code, or evaluation scripts are released. The metric suite (Python ast, pyflakes, lizard) could be reimplemented on the public petclinic repository, but the headline numbers would not transfer without the closed stack.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The abstract in the supplied metadata is cut mid-sentence ('whil'), but the provided full paper text is complete through the references — no content gap affects this review.
- PDF extraction flattened Tables 1-3 into per-row lists; all values cited here were read from those flattened rows and cross-checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.