Empirical Evaluation of Open-Source Large Language Models for Retrieval-Augmented Generation in ESG Domain
Motaz Saad, Anna Borrelli, Ivan Gentile, Kianna Kazemi, Francesco Piccialli, Antonella Longo
cs.AI
Quick verdict
Seven open-source LLMs tested on one ESG-RAG pipeline; candid and reproducible, but no significance tests and judge circularity weaken model rankings.
Summary
The paper presents a controlled empirical evaluation of seven open-source LLMs (2B–30B: glm-4.7-flash, nemotron-3-nano:4b, qwen3:4b-instruct, gemma3:4b, gemma4:e4b, gemma4:e2b, ministral-3:8b) as generators in a shared RAG pipeline over 498 ESG reports from EU-listed companies (2010–2024). Using persona-driven synthetic QA pairs generated by qwen3:4b-instruct (284 pairs built; a deterministic 100-pair prefix evaluated) and seven RAGAS-family metrics judged by the same qwen3 model, with retrieval fixed to qwen3-embedding:4b at top-k=1, the study finds retrieval relatively strong (context recall 0.578–0.605, context precision 0.783–0.805, context relevance 0.965–0.985) while generation diverges: faithfulness 0.607–0.822 (glm-4.7-flash best, 0.822), answer relevancy 0.760–0.881 (ministral-3:8b best, 0.881), and factual correctness only 0.387–0.449 (qwen3 best, 0.449), which the authors read as motivation for domain fine-tuning. Key limitations are candidly acknowledged: generator, judge, and one evaluated model are all qwen3 (self-enhancement risk); results are single-run point estimates without significance testing on a small subset; decoding parameters were not normalized across models; top-k=1 imposes a retrieval ceiling, and no non-RAG baseline or hyperparameter ablation is included. A public GitHub resource (reports, QA pairs, scores, scripts) supports reproduction (~21.5 GPU-hours).
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under a shared top-k=1 ESG RAG pipeline, retrieval is strong but capped (context recall 0.578–0.605, context precision 0.783–0.805, context relevance 0.965–0.985, LLM-judged context precision with reference 0.830–0.878).
- Generation quality diverges most on faithfulness (0.607–0.822), with glm-4.7-flash:q4_K_M leading at 0.822.
- Factual correctness is the pipeline bottleneck for all models (0.387–0.449), motivating domain-specific fine-tuning and human-in-the-loop validation.
- No single model dominates: qwen3:4b-instruct leads factual correctness (0.449), ministral-3:8b leads answer relevancy (0.881), and gemma3:4b is the most balanced, so model choice should follow application priorities.
- The public ESG-RAG resource (498 processed reports, 284 synthetic QA pairs, scores, scripts) enables reproducible cross-model RAG comparison in the ESG domain.
Strengths
Genuinely controlled cross-model comparison
All seven models share one retriever (qwen3-embedding:4b, cosine, top-k=1 over 258,590 chunks), one prompt template (Section III-E3, Table II), and one judge, so the generator is the main experimental variable — exactly the gap the related-work section (III-C) identifies in ESGReveal, Climate Finance Bench, and SusGen-GPT.
Reproduction-grade documentation and a public resource
Section III-B/III-E and Table II specify chunking (1,000 chars, 200 overlap), model IDs with quantization (glm-4.7-flash:q4_K_M), hardware (RTX 6000 Ada, 48 GB), and runtime (~21.5 h for five models); the repo reportedly ships the Markdown corpus, 284 QA pairs, all scores, and scripts.
Unusually candid limitations section
Actionable metric-level diagnosis
Weaknesses
Circular evaluation: generator = judge = competitor
No inferential statistics on n=100 single runs
Decoding parameters left at per-model Ollama defaults
No baseline and no ablation under a retrieval ceiling
Draft inconsistencies and subset opacity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Structured Review: Empirical Evaluation of Open-Source LLMs for RAG in the ESG Domain
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which open-source LLM is best for RAG over ESG reports according to this paper?
It depends on which quality you prioritize, because no model dominated all metrics. In the paper's single retrieved-chunk setup: glm-4.7-flash scored highest on faithfulness (0.822, i.e., sticking to the retrieved evidence), qwen3:4b-instruct highest on factual correctness (0.449), ministral-3:8b highest on answer relevancy (0.881), and gemma3:4b was the most balanced. The authors caution that qwen3's lead may be inflated because the same model generated the test questions and served as the evaluation judge.
What is the ESG-RAG benchmark and where can I download it?
It is the public evaluation resource released with this paper: 498 ESG reports from EU-listed companies (2010–2024) converted to Markdown, 284 persona-based synthetic question-answer pairs grounded in those reports, per-model RAGAS evaluation scores for seven open-source models, and reproduction scripts. The authors host it on GitHub at motazsaad/ESG-RAG-benchmark-ragas.
Why is factual correctness so low (around 0.39–0.45) when retrieval looks strong?
The pipeline retrieves exactly one 1,000-character chunk per question (top-k=1). Context recall of roughly 0.58–0.61 means the reference answers often need evidence spread over multiple chunks, so the generator frequently answers from incomplete evidence. ESG KPIs also demand precise numbers, which amplifies small errors. The authors therefore recommend domain-specific fine-tuning and human validation rather than relying on any current model.
Is a RAGAS evaluation biased when the judge model is also the dataset generator and one of the competitors?
Potentially, yes. In this paper qwen3:4b-instruct generated the QA pairs, acted as the LLM-as-judge for all scoring, and was itself one of the seven evaluated models — a 'self-enhancement' risk the authors explicitly acknowledge, since stylistic alignment between question, answer, and judge can inflate qwen3's scores. They defer cross-validation with an independent judge model to future work.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The abstract in the provided extraction is truncated mid-sentence; conclusions were drawn from the body, Section V, and the conclusion section.
- Figures (RAG pipeline, RAG triad, heatmap) are available only as extracted text; the heatmap was cross-checked numerically against Table III but visual content was not independently verified.
- Model names and versions (e.g., glm-4.7-flash, gemma4, ministral-3, nemotron-3) could not be verified against external sources under the no-network constraint.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.