EAVer: Long-Form Factuality Verification as an End-to-End Agentic Policy
Kening Zheng, Aoying Zheng, Zhigang Chang, Yazhi Guo, Miaotian Guo, Qingwei Zong, Xianhai Xie, Weiqiang Jin, et al.
cs.CL · cs.LG
Quick verdict
Trains an 8B agentic verifier with privileged-teacher trajectories: ~80% fewer searches, top Macro-F1; solid method, but no code and no variance reported.
Summary
EAVer reframes long-form factuality verification as a single learned agentic policy rather than a static decompose-search-verify pipeline. A privileged teacher (Claude Opus 4.8), given hidden gold claims and labels, synthesizes executable multi-turn tool-interaction trajectories with live search over a local E5/FAISS index; quality-control filters (structure, label alignment, tool protocol, budget, leakage) retain 1,447 trajectories. The policy groups claims by entity, verifies high-confidence groups directly, searches only for uncertain groups, and stores evidence as compact in-context memos reused across claims. A second stage adds 794 same-trajectory preference pairs differing only in verdict labels, optimized by DPO masked to decision tokens. With Qwen3-8B, EAVer attains 73.43 Macro-F1 on in-domain VeriFastScore and 74.05 on out-of-distribution FaStFact-Bench, beating the strongest search-based baselines (VeriFastScore 70.55; FIRE 69.32) with roughly 80% fewer searches (3.14 vs 15.62 per sample); for Qwen3-8B the false support rate falls from ~89 (untrained) to 41.6 in-domain and 35.9 OOD, and gains hold across six 4B-32B backbones. Main limitations: the search-free teacher itself scores 72.21 on FaStFact-Bench, only ~2 points below the student; training and evaluation use different search backends; results are three-run means without variance or significance tests; and no code or data release is mentioned.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Treating response-level verification as one learned agentic policy (entity grouping, confidence-routed search, memo-based evidence reuse) beats the strongest search-based pipelines on Macro-F1 while using ~80% fewer searches (3.14 vs 15.62 per sample).
- A privileged-teacher pipeline converting gold claim annotations into executable multi-turn tool-interaction trajectories with live search yields 1,447 QC-passed trajectories that cut an 8B model's false support rate from ~89% to 35.9-41.6%.
- Decision-focused DPO over 794 same-trajectory preference pairs, with the loss masked to factuality-decision tokens, lowers FSR while preserving or modestly improving Macro-F1 across backbones.
- The learned policy transfers without backbone-specific synthesis across six models spanning Qwen and Llama families from 4B to 32B parameters.
- Prior per-claim pipelines exhibit 23.5-66.5% cross-claim link redundancy (35.9-103.7 searches/sample), which string-level caching does not remove.
Strengths
Quantified diagnosis of prior-pipeline waste
Figure 1 measures existing systems at 35.9-103.7 searches/sample with 23.5-66.5% repeated-link redundancy on an independent VeriFastScore validation split, versus 3.42 searches and 0.3% redundancy for EAVer — a concrete, measured motivation.
Methodologically clean preference construction
The 794 DPO pairs (405 false-support-to-unsupported, 389 reverse) are same-trajectory counterfactuals, byte-identical except synchronized verdict labels, and the loss (Eqs. 1-2) is masked to decision tokens only — isolating the factuality decision from search and formatting behavior.
Disciplined data-quality and leakage controls
Broad, consistent generalization evidence
Honest reporting of unfavorable trade-offs
Weaknesses
Teacher-proximity ceiling
Train/eval search-backend mismatch never ablated
Thin statistical reporting
Missing ablation isolating the synthesis approach
Ambiguous DPO status of headline numbers
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
EAVer targets long-form factuality verification: decompose an LLM response into atomic claims and judge each supported/unsupported. The empirical premise (Figure 1) is that decompose-search-verify pipelines are wasteful — on an independent VeriFastScore validation split, prior systems issue 35.9-103.7 searches per sample with 23.5-66.5% repeated-link redundancy, versus 3.42 searches and 0.3% redundancy for EAVer. The thesis: the entire response-level workflow (claim grouping, confidence-routed search, evidence reuse, verdicts) should be learned as one policy rather than hand-orchestrated. Headline results (Table 2): with Qwen3-8B, 73.43 Macro-F1 on in-domain VeriFastScore and 74.05 on out-of-distribution FaStFact-Bench — +2.88 over the strongest search-based baseline in-domain (VeriFastScore, 70.55) and +4.73 OOD (FIRE, 69.32) — at 3.14 searches/sample versus 15.62 for the most search-efficient baseline. Metrics are Macro-F1 (primary), False Support Rate (FSR), and claim-extraction F1 under greedy 0.5-threshold SequenceMatcher matching.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is EAVer and how does it cut search costs in factuality verification?
EAVer is a fine-tuned open-weight LLM (e.g., Qwen3-8B) trained to run the entire long-form factuality verification workflow — claim decomposition, grouping by entity, confidence-based routing between direct verification and search, and evidence reuse via in-context memos — in a single agentic trajectory. Because related claims share one search and high-confidence groups need none, it averages about 3 searches per response versus 15.6 for the most search-efficient prior pipeline (roughly 80% fewer), while achieving higher Macro-F1.
How were EAVer's training trajectories created without new human annotation?
A privileged teacher (Claude Opus 4.8) was secretly given the gold atomic claims and their binary labels for each training example and produced realistic multi-turn tool-using verification trajectories that reach those labels through genuine live search. Automated checks for structure, label alignment, tool protocol, search budget, and label leakage kept 1,447 trajectories, and 20 human experts spot-checked them with a 98% pass rate. The student model never sees the gold claims or labels at inference.
What benchmarks and headline results does EAVer report?
On an author-constructed in-domain test set from VeriFastScore (22,532 claims) and the out-of-distribution human-annotated FaStFact-Bench (6,954 claims), EAVer with Qwen3-8B reaches 73.43 and 74.05 Macro-F1 respectively, beating the strongest search-based baselines (VeriFastScore 70.55, FIRE 69.32) with about 3.1 searches per sample. False support rate drops from about 89% for the untrained backbone to 41.6% in-domain and 35.9% OOD, and all six tested backbones (4B-32B) improve.
Is EAVer's code, data, or model publicly available?
No code, weights, or training-data release is mentioned in the paper. Training draws on the public VeriFastScore training pool and FaStFact-Bench but relies on a proprietary teacher (Claude Opus 4.8) with only prompt excerpts published, so exact replication would require reimplementing the synthesis and training pipeline from the paper's detailed but incomplete specification.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 4's per-backbone DPO values are embedded only as axis labels and legend text; data-point coordinates are not recoverable from the extraction, so DPO effects could be individually verified only for Qwen2.5-7B, whose numbers appear in the prose (FSR 38.51 to 34.16, Macro-F1 71.77 to 72.31).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.