Can We Trust the Judges? Validation of Factuality Evaluation Methods via Answer Perturbation
Sarra Gharsallah, Adele Robaldo, Mariia Tokareva, Giovanni Gatti Pinheiro, Ilyana Guendouz, Raphaël Troncy, Paolo Papotti, Pietro Michiardi
cs.CL
Quick verdict
Controlled-perturbation meta-evaluation finds RAGAS-style pipelines track degradation better than LLM judges; sound but thin human validation, small NQ sample.
Summary
The paper introduces a meta-evaluation framework probing whether factuality evaluation metrics themselves are trustworthy, by applying controlled corruptions to gold-standard answers. A pipeline paraphrases each reference answer (A0), extracts factual components via dependency parsing, filters question terms, ranks them by importance, and uses an LLM to produce four progressively more erroneous but fluent variants (A1-A4). Pipeline validation used two human evaluators blindly comparing 20 Q&A pairs: a non-inferiority test against expert-written perturbations barely passes (margin delta=-0.1; mean composite score 0.125, 90% bootstrap CI [0.025, infinity]), with low inter-rater agreement (kappa=0.113) and 77% post hoc power. The main experiment generates 500 examples from 100 Natural Questions pairs: six open-weight LLM judges yield Pearson correlations of only -0.62 to -0.74, while RAGAS with gpt-4o-mini reaches -0.87 (Kendall -0.95), and an open LLM+NLI variant -0.82/-0.83 (Kendall -0.96/-0.94). The authors conclude that pipeline-based metrics track factual degradation better than LLM-as-judge, that model size is not decisive, and that LLM+NLI is a cost-effective alternative. Key limitations: perturbation severity uniformity is not guaranteed; fact-preserving manipulations are untested; and the cost-efficiency claim lacks measured cost or latency data.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Pipeline-based factuality metrics (RAGAS factual correctness with gpt-4o-mini: Pearson -0.87, Kendall -0.95) track controlled factual degradation substantially better than reference-based LLM-as-judge protocols (six open-weight judges: Pearson -0.62 to -0.74).
- An open-weight LLM+NLI variant of the RAGAS pipeline using nli-deberta-v3-large is statistically competitive (Pearson -0.82/-0.83, Kendall -0.96/-0.94) and claimed cost-efficient.
- Model size does not determine evaluator quality: several mid-sized open-weight models outperform larger judge-based LLMs (Figure 3b).
- The LLM-based perturbation pipeline is non-inferior to expert-written perturbations in a blind human comparison (Section 4), making the benchmark cheap to re-instantiate on new QA data.
Strengths
Targets a real, production-relevant meta-evaluation gap
Section 1-2 motivate testing whether routinely deployed metrics (RAGAS factual correctness, reference-based LLM-judge prompts) track graded degradation in open-ended QA, going beyond TRUE's binary labels, BUMP's minimal pairs, and FBI's detect-or-miss verdicts.
Ordered severity levels enable ranking analysis, not just detection
The A0-A4 design (Section 3) yields a totally ordered set per question, so sensitivity can be measured with Pearson/Kendall correlations (Table 2) instead of binary accuracy — a genuine protocol improvement over prior corruption-based benchmarks.
Pipeline is cheap to re-instantiate and validated against humans
Candid limitations and public artifacts
Practical negative result on scale
Weaknesses
Human validation of the pipeline is statistically thin
Narrow, non-random benchmark sample
No formal between-method statistical tests
Possible scale confound favoring pipelines
Cost-efficiency claimed but never measured
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This extended paper (building on an EvalLLM 2025 workshop version, with experiments unchanged) asks whether the factuality metrics now in production — RAGAS-style claim-decomposition pipelines and reference-based LLM-as-a-judge prompts — actually track *graded* factual degradation in open-domain QA, rather than merely separating consistent from inconsistent text. Contributions: (i) a controlled corruption pipeline producing five ordered answer levels (A0 faithful paraphrase to A4 extreme corruption, Section 3); (ii) a meta-evaluation protocol based on rank and sensitivity correlation rather than binary detection; (iii) a nine-configuration comparison (Table 2) plus a proposed open LLM+NLI variant. Central claims: pipeline metrics outperform judges, model size is not decisive, and the LLM+NLI variant is competitive and cost-efficient.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper conclude about RAGAS versus LLM-as-a-judge for factuality evaluation?
On a controlled-perturbation benchmark (500 corrupted-answer examples built from 100 Natural Questions pairs), the RAGAS factual correctness pipeline with gpt-4o-mini correlated with the intended degradation level at Pearson -0.87 and Kendall -0.95, while six open-weight LLM judges scored between -0.62 and -0.74 Pearson. The authors conclude that claim-decomposition pipelines track graded factual errors more reliably than holistic LLM judging.
How were the artificially corrupted answers validated as realistic?
Two human evaluators blindly compared pipeline-generated perturbations against perturbations written by a human expert on 20 question-answer pairs. About 82.5% of comparisons at error levels A1-A4 were ties, and a non-inferiority test (margin -0.1) passed with a mean composite score of 0.125, though inter-rater agreement was low (Cohen's kappa 0.113) and the study was small.
Is the code and dataset from this paper publicly available?
Yes. The perturbation pipeline and dataset are released as the 'truthbench' repository on GitHub (GiovanniGatti/truthbench) with a Zenodo archival DOI, and all LLM prompts used by the pipeline and the judge protocol are printed in Appendix B of the paper.
What are the main limitations of this factuality meta-evaluation?
The authors note that perturbation severity is not uniform across questions, that some corruptions miss the core fact or create internal inconsistencies, and that fact-preserving manipulations (which can game metrics) were not tested. The study also covers only one English Wikipedia-derived QA dataset, did not evaluate reasoning-augmented judges, and its cost-efficiency claims lack measured cost data.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-4 (contingency charts, score-vs-level curves, judge-size plot, UI screenshots) are referenced but not analyzable from the extracted text; quantitative statements about Figure 3 rely on the authors' in-text descriptions.
- The paper states experimental content is unchanged from a 2025 workshop version; this review covers the extended text as extracted.
- Bibliography formatting is partially garbled by extraction; this does not affect the scientific assessment.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.