AquiLLM: Evaluating Faithfulness in Open-Weight RAG-LLM Systems for Scientific Research
Bernie Boscoe, Srinath Saikrishnan, Vikram Seenivasan, Jack Stark, Andrew Lizarraga, Morgan Himes, Jonathan Soriano, PJ Allen, et al.
cs.AI
Quick verdict
Astronomers rate an open-weight RAG system's faithfulness; candid about design flaws, but self-rated responses and one unvetted model limit the evidence.
Summary
This paper presents AquiLLM, an open-weight, offline RAG-LLM platform for research groups, and a domain-expert evaluation of response faithfulness in an astronomy case study. Five astronomers from one research group built a 31-document collection (their own publications, HSC/Euclid/LSST survey documentation, unpublished material), posed ten questions across five query types, and rated 12-15 submissions per question on a five-point faithfulness scale without verbal anchors. The generation model was Qwen3.5-27B-Claude-4.6-OS, a third-party community fine-tune with no published independent evaluation. Factual Retrieval scored highest among multi-question categories (mean 4.17, n=60), Comparative Scientific Analysis lowest (2.97, n=39); single-question categories ranged from 3.62 to 4.73. Six failure modes are documented, dominated by credible-sounding overreach: hallucinated details (a 2.4x bias gap reported where sources support 1.2x), unsupported synthesis, retrieval mismatch, factual drift, overconfidence, and unresolved ambiguity. The authors are notably candid: because each participant rated only their own generated responses, inter-rater reliability is not computable, and rater severity is confounded with generation variability; no comparison system was used, justified by the unpublished corpus. Code is public but the evaluation dataset is withheld against contamination, available on request. Main limitations: one group, one domain, one unvetted model, and no inferential statistics.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- AquiLLM is a deployable, modular, open-weight offline RAG platform that small research groups can run on a single H100 within academic-allocation or modest commercial budgets (Section III, Appendix A, Table IV).
- Faithfulness of the evaluated system is highest for explicit retrieval-oriented questions (mean 4.17, n=60) and lowest for Comparative Scientific Analysis (2.97, n=39), i.e., failures concentrate in cross-source synthesis rather than being uniform (Table III).
- Six recurring failure modes characterize the errors, with credible-sounding overreach — not random error — as the dominant risk (Section V-C).
- Standard benchmark leaderboards fail to capture these domain-specific reliability failures, motivating reusable domain-expert evaluation methodology (Sections I, IV-D).
- Faithfulness evaluation of scientific RAG requires designs separating rater variation from generation variability, which this study's design deliberately cannot provide — a self-declared baseline limitation (Sections V-B, VII).
Strengths
Evaluation on a real group's private corpus, including unpublished material
Section IV-B: 31 documents (15 journal articles, 4 conference papers, 12 technical resources) spanning HSC/Euclid/LSST documentation and the group's own GalaxiesML work — most RAG faithfulness studies use public benchmarks; testing on material researchers actually depend on, including unpublished manuscripts, is rarer and more ecologically valid.
Exemplary methodological candor about the design's central flaw
Sections V-B and VII state explicitly that inter-rater reliability is 'not computable, not merely uncomputed' because no two ratings correspond to the same response text, and refuse causal interpretation of the consistency/failure co-occurrence. This transparency exceeds norms for system papers.
Actionable failure-mode taxonomy with quantified examples
Deployment reproducibility detail unusual for the genre
Citation enforcement active during the evaluation
Weaknesses
Self-rated, self-generated responses confound every category-level number
Single self-selected model generalizes to a plural title
Headline finding depends on fragile query labeling
Evaluation dataset withheld, blocking independent verification
Category imbalance and abstract-table tension
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper introduces AquiLLM (Section III), an open-weight, offline RAG-LLM platform for research groups, and reports a domain-expert faithfulness evaluation in an astronomy case study. Three claims anchor it: (1) the platform is adoptable on a single H100 within academic-allocation or modest commercial budgets (Appendix A, Table IV); (2) faithfulness is highest for explicit retrieval (mean 4.17) and degrades for cross-source comparative analysis (2.97); (3) six failure modes recur, dominated by credible-sounding overreach, which benchmark leaderboards do not capture. Note a scope mismatch: the abstract says faithfulness 'degrades for queries requiring synthesis or ambiguity resolution', but Table V classifies Scientific Interpretation — Synthesis=Yes, Reasoning=High — as the highest-scoring category (4.73, n=15). Section I's phrasing ('cross-source synthesis and comparative reasoning rather than uniformly across query types') is the accurate one; the abstract should adopt it.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AquiLLM?
AquiLLM is an open-source, open-weight retrieval-augmented generation (RAG) platform that research groups deploy locally and offline. Members create shared collections of their own documents (papers, unpublished manuscripts, notes, technical documentation) and query them in natural language, keeping unpublished research off external commercial AI services.
What did the evaluation of AquiLLM find?
Five astronomers rated the system's answers for 'faithfulness' (grounding in retrieved sources) on a 1-5 scale. Direct fact-lookup questions averaged 4.17, while questions requiring comparison across multiple papers averaged 2.97. Six failure modes recurred, the dominant one being plausible-sounding but unsupported claims — e.g., reporting a 2.4x effect where sources only support 1.2x.
Which large language model did AquiLLM use in the study?
A community fine-tune of Qwen3.5-27B called Qwen3.5-27B-Claude-4.6-OS, served locally on a single NVIDIA H100 GPU. The authors note this specific fine-tune, produced by a third party on Claude-derived examples, has no independent published evaluation — a stated limitation for generalizing the results.
Is the AquiLLM evaluation dataset public?
No. The platform code is public on GitHub, but the evaluation questions, reference answers, and ratings are withheld to prevent them from contaminating future model training corpora. Researchers can request the dataset from the authors under a no-training condition; public release is planned after follow-up studies.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Multiple sections are elided in the extraction ([...section omitted...] around system architecture, query-category definitions IV-D, the results narrative V-A, and part of the discussion); the review relies on the surviving tables and prose.
- A 'Table II' is implied by table numbering but absent from the extracted text, so per-question results beyond what Section V-B reports could not be checked.
- Figures, if any, did not survive extraction and could not be assessed; the reference list appears partially garbled (numbering vs. in-text citations may not align).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.