Human-Anchored Factuality Evaluation with Strategic Annotation
Yu Wang, Craig Erickson, Kevin Small
cs.CL
Quick verdict
FSA-guided active statistical inference for human-anchored factuality eval; ESS gains (+40.3%/+27.1%) credible, but margin over confidence baseline is modest.
Summary
The paper addresses the bias of LLM-as-a-judge factuality metrics by combining judge predictions over a full dataset with a small, selectively chosen set of human labels, using Active Statistical Inference (ASI). Its contribution is a policy-design layer, failure-space analysis (FSA), that derives predictive features of judge-human disagreement from four signal families: judge-output artifacts, evidence/reference quality, task strata, and answer-rubric alignment. An XGBoost model trained on a 40% calibration split predicts residual magnitude and is converted into sampling probabilities via uniform mixing, with power tuning of the judge term. On AutoFA, an internal voice-assistant factuality system (N=2,338), and RAGTruth (N=2,937 GPT-4 responses, Claude Sonnet 4.6 as judge), judge FAR underestimates human FAR by 13.3 and 30.6 percentage points respectively. Aggregated over budgets 50-1000 with 300 Monte Carlo trials, FSA-guided ASI maintains ~0.95 coverage and improves effective sample size by 40.3% (AutoFA) and 27.1% (RAGTruth) over uniform human annotation, versus 17.0%/22.6% for confidence-only sampling. Notable analyses include the finding that more features can hurt (all-candidate models yield lower ESS due to aggressive mixing) and qualitative cases of confidently misaligned judges. Main limitations: one internal and one public benchmark, a single judge per dataset, no significance testing of the headline gains, unquantified feature-extraction cost, and reliance on stable residual structure.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLM-judge FAR is systematically biased downward relative to human FAR — by 13.3 percentage points on AutoFA and 30.6 points on RAGTruth (55.6% vs 86.2%) — with zero confidence-interval coverage.
- Judge-human disagreement concentrates in observable strata such as partially verified claims and weak evidence support, with residual lift up to 2.58x (Table 2).
- FSA-derived sampling policies raise effective sample size by 40.3% (AutoFA) and 27.1% (RAGTruth) over uniform human annotation while maintaining ~0.95 coverage.
- Parsimonious feature sets beat richer ones for policy design: a single ref-support-strength feature achieves +40.5% ESS versus +24.7% for all candidates on AutoFA (Table 5).
- Confidence-only sampling misses confidently misaligned judges, which FSA's evidence-quality features target (Appendix D records 168 and 5718).
Strengths
Human-anchored target with demonstrated bias correction
The judge-only estimator shows zero coverage on both datasets (Table 4) while all ASI variants hold 0.94-0.95 coverage across budgets 50-1000, cleanly validating the debiasing machinery against a 13.3pp (AutoFA) and 30.6pp (RAGTruth) judge bias.
Honest feature-parsimony analysis
Table 5 shows the all-candidates model on AutoFA (CV rho=0.52) yields only +24.7% ESS versus +40.5% for the single ref-support-strength feature, traced to extreme scores forcing tau=0.70 uniform mixing — a rare and useful negative result about richer policies.
Diagnostic grounding of the policy
Tuned estimator details fully specified
Above-average annotation documentation
Weaknesses
Largest gain sits on unreleasable internal data
Judge baseline may be unusually weak
No significance testing of headline comparisons
Cost of the method is asserted, not measured
Single split and unexplained correlation discrepancies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets reference-based factuality evaluation of short-form query-response pairs (Section 2.1), where an LLM judge scores all N items but only n_bgt can be human-labeled. The estimand is the human-defined Factual Accuracy Rate (FAR), theta = E[phi(X,Y)]. Three claims are central: (1) judge-predicted FAR is systematically biased — 13.3 percentage points below human FAR on AutoFA, and 55.6% versus 86.2% (a 30.6pp gap) on RAGTruth — with zero CI coverage (Table 4); (2) judge-human residuals concentrate in observable strata (Table 2: partial verification 2.58x lift on AutoFA, low self-confidence 2.24x on RAGTruth); (3) an FSA-derived sampling policy improves effective sample size by 40.3% (AutoFA) and 27.1% (RAGTruth) over uniform annotation while preserving ~0.95 coverage. All three are supported within the evaluated scope — one internal system plus one public benchmark, one judge each.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is failure-space analysis (FSA) in this paper?
FSA is a policy-design pipeline that organizes judge failure signals into four families — judge-output artifacts, evidence/reference quality, task and input strata, and answer-rubric alignment — extracts numeric features from them, and trains a model to predict where an LLM judge will disagree with humans, so that scarce human annotations are spent on the highest-risk examples.
How much does strategic annotation improve efficiency over random sampling?
In simulations over budgets of 50-1000 labels, the FSA-guided policy achieved effective-sample-size gains of 40.3% on the internal AutoFA dataset and 27.1% on the public RAGTruth benchmark versus uniform human annotation, while keeping about 0.95 confidence-interval coverage; a confidence-only baseline reached 17.0% and 22.6% respectively.
Is code or data available for this paper?
The paper does not announce a code release. AutoFA is an internal Amazon dataset, so external replication is limited to the public RAGTruth benchmark plus the judge and feature-extraction prompts reproduced in the appendices.
What is Active Statistical Inference (ASI)?
ASI combines machine predictions on every item with human labels on a small, policy-sampled subset, correcting the biased machine estimate with human residuals scaled by inverse inclusion probabilities, yielding an unbiased estimate of the human-defined metric with valid confidence intervals.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2, 4, 5 and 6 appear only as captions in the extraction; their plotted values (CI widths, tau-search curves, SHAP magnitudes) cannot be audited from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.