Retrieval-Augmented Generation for Scientific Code Understanding
Aaron Nobile, Andreas Adelmann, Mohsen Sadr
cs.SE · cs.AI
Quick verdict
Local RAG agent for scientific C++: 9B model best (0.795), beats Claude Code baseline; retrieval pipeline unablated, GPT-judge circularity tempers conclusions.
Summary
The paper builds a fully local retrieval-augmented generation (RAG) system for answering questions about a scientific C++ codebase (IPPL). It separates an expensive offline ingestion stage — tree-sitter parsing of code and documentation, call/dependency graph construction, LLM-generated entity explanations at symbol, file, module, and call-chain levels, and nomic-embed-text embeddings stored in an 8013-vector FAISS index — from a lightweight online stage combining semantic retrieval, exact-match injection, query-intent routing, metadata reranking, and bounded structural expansion. Evaluation uses 100 GPT-5.4-generated questions across eleven categories, scored 1/0.5/0 by GPT-5.5 as judge. Across seven answering models, Qwen3.5 9B achieves the best average score (0.795 ± 0.034), beating Qwen3.5 32B (0.765) and Gemma4 31B (0.710); larger models were consistently slower. In a Claude Code comparison on three shared models, scores drop substantially (e.g., Qwen2.5 7B: 0.720 to 0.520), supporting the claim that prepared retrieval context matters more than the agentic loop for small models. Main limitations: no ablation isolating individual retrieval components; question generation and judging both rely on OpenAI models without human validation; category samples are small (three Numerical Meaning questions for the best model); and shared generation settings were not tuned per model, leaving the size-versus-score inversion unexplained. Code and benchmark are public.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A fully local RAG agent with offline ingestion enables small open-weight models to answer repository-specific questions about IPPL, with Qwen3.5 9B best at 0.795 ± 0.034 on a 100-question benchmark.
- For this task, model family and retrieval quality matter more than parameter count.
- The same small models score substantially lower when embedded in Claude Code's agentic retrieval architecture (e.g., Qwen2.5 7B: 0.720 vs 0.520).
- Front-loading code understanding into a codebase-specialized vector store compensates for the limited reasoning capacity of small models.
Strengths
Unusually transparent system documentation
Sections 3.2-4.1 publish the full pipeline: parser record schemas (Listings 3-5), reranker scoring formula (Listing 12), five selection constants (Table 2), nine intent classes with expansion profiles (Table 4), and complete runtime configuration (4.1), enabling faithful reimplementation.
An external agentic baseline on shared models
Table 11 runs three of the same small models inside Claude Code's retrieval architecture (0.520/0.560/0.665 vs 0.720/0.665 in the RAG pipeline), a rare direct architecture A/B that operationalizes the front-loading hypothesis.
Honest self-critique
Actionable category-level diagnostics
Latency trade-off reported alongside quality
Weaknesses
No component ablation of the retrieval pipeline
Judge and question generator entangled in one vendor
Statistical reporting cannot support the ranking claims
Claude Code baseline potentially unfair
Size inversion unexplained and confounded
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Retrieval-Augmented Generation for Scientific Code Understanding (arXiv 2609.12190)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main result of this paper?
The authors built a fully local RAG question-answering system over the IPPL C++ codebase. On a 100-question benchmark judged by GPT-5.5, a 9B model (Qwen3.5) achieved the best average score of 0.795 out of 1, beating larger models in the same pipeline (Qwen3.5 32B: 0.765; Gemma4 31B: 0.710) and the same small models run inside Claude Code's retrieval architecture (e.g., Qwen2.5 7B dropped from 0.720 to 0.520).
Does a 9B model really beat a 32B model for code question answering?
In this paper's setup, yes: Qwen3.5 9B scored 0.795 versus 0.765 for Qwen3.5 32B and 0.710 for Gemma4 31B (Table 10). The authors attribute this to newer model families being better aligned for code/RAG-style QA, and note the benchmark favors concise, grounded answers. Caveats: the gap is within about one standard error, generation settings were shared rather than tuned per model, and the judge may have stylistic biases.
Is the code and benchmark available?
Yes. The paper states that the implementation, evaluation material, and reproduction instructions are at github.com/nobileaaron/rag-scientific-code-agent; the 100-question benchmark and prompt templates are distributed as repository artifacts, and the indexed codebase (IPPL) is itself open source.
Why does the paper compare against Claude Code?
To test whether an agentic retrieval loop helps or hurts small local models. Three of the same small models scored substantially lower under the Claude Code architecture (0.520, 0.560, 0.665) than in the authors' RAG pipeline (0.720, 0.665 for overlapping models), supporting their claim that offline-prepared retrieval context matters more than retrieval-time agentic search when the answering model is small.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: [...sampling gap...] markers fall inside Section 3.3 (retrieval pipeline internals), between Table 6 and Table 8 in Section 5.1, and in the reference list; judgments about full evaluation setup (including the 'calibration baseline' referenced in 5.3) rest partly on unseen text.
- Figure 1 and other figures are captioned in the text but their visual content cannot be assessed from text extraction.
- Counts for five benchmark categories (Definition Location 15, File Purpose 13, Parallelism and Kokkos 9, Testing and Workflow 4, Numerical Meaning 3) are taken from Table 13 because Table 6 falls partly within a sampling gap; the counts are consistent with Table 12 totals.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.