What Changes When Fact-Verification Scores Improve? Evidence and Answer Accounting Across Trained Verifiers and LLMs
cs.CL
Quick verdict
Exact 2×2 recombination shows most of a 9.61-pt joint gain is evidence, not answers; rigorous stats, but scoped to DCUF-trained verifiers; LLM effects mixed.
Summary
The paper asks how much of an improvement in a joint fact-verification score—credit requires both a correct answer and eligible submitted evidence—survives when answers are held fixed. It introduces an exact 2×2 decomposition: answers generated under each of two sources are scored against both sources' evidence submissions, yielding two order-dependent splits whose difference is an interaction; averaging orders gives a Shapley-style allocation. Study 1 scores four DeBERTa-large checkpoints on 7,890 FEVEROUS development claims with DCUF versus UnifEE evidence: strict score rises 9.61 points [8.77, 10.43], of which 7.92–9.08 points persist with answers fixed, versus a 1.96-point accuracy gain. Study 2 generates 470,400 responses from Qwen3-8B and Llama-3.1-8B across FEVER, FEVEROUS and SciFact under two answer formats and two context budgets with BM25 versus BGE orderings. Only the FEVEROUS context effect is robust (3.84 and 3.10 points for Qwen and Llama); the prespecified cross-dataset replication criterion fails, so the recorded outcome is 'restricted/mixed'. Post-hoc analyses show aggregate accuracy and coverage rates can miss the claim-level pattern, especially for FEVER/Qwen where a covariance term supplies most of the context effect. Limitations: Study 1 checkpoints were trained on DCUF-derived inputs and never predict NEI; trained weights are excluded from the supplement; the protocol was preregistered only internally; the FEVEROUS LLM panel carries historical exposure.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On FEVEROUS, swapping DCUF for UnifEE evidence raises the strict score of four DCUF-trained DeBERTa verifiers by 9.61 points [8.77, 10.43], of which 7.92-9.08 points persist with answers held fixed, versus only a 1.96-point accuracy gain.
- The two decomposition orders differ by a 1.15-point interaction, so score attribution depends on order and marginal rates (accuracy times eligibility change) can misestimate the evidence component.
- For two frozen 8B LLMs, the fixed-answer BM25-to-BGE evidence gain on FEVEROUS rises by 3.84 (Qwen) and 3.10 (Llama) points when context grows from 256 to 2,048 tokens, with all five seed signs positive.
- The prespecified cross-dataset replication criterion is not met and not all intervals fit inside the ±2-point small-effect band, so the recorded outcome is 'restricted/mixed' evidence.
- Aggregate accuracy and evidence-coverage rates can miss the claim-level pattern: on FEVER/Qwen a covariance term supplies 1.055 of the 1.165-point context effect (Eq. 4, Table 10).
Strengths
Exact, assumption-light decomposition identity
Section 3's 2×2 recombination makes off-diagonal cells exact recombinations of saved answer-correctness and evidence-eligibility indicators for FEVEROUS strict accuracy, and the identity holds for corpus metrics generally. Both orders plus the interaction are reported (Eqs. 1-3; Tables 2-3): Study 1 splits 9.61 points as E0=7.92/A1=1.69 or A0=0.53/E1=9.08 with interaction 1.15 [0.86, 1.45].
Statistical discipline unusual for NLP evaluation
Twelve prespecified contrasts with Bonferroni adjustment; separate 60- and 42-comparison families with explicit non-combinability statements; claim-family bootstrap with independent seed resampling (50k/1M draws); Monte Carlo precision audited to per-batch percentile endpoints (Appendix K); and an explicit 'interval includes zero' vs 'effect is small' distinction citing Card et al. 2020.
Radical transparency about failures and history
Scale and determinism of Study 2
Actionable reporting guidance
Weaknesses
Study 1 confound limits the headline split
Zero NEI recall narrows the population
Cross-dataset criterion tested unequal interventions
Internal-only preregistration with post-hoc plan after peeking
Inference-level reproduction impossible
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is an evaluation-methodology study for joint fact-verification scoring. The target quantity is how much of a joint-score gain is attributable to evidence rather than answers. Two empirical claims carry the paper. First, on FEVEROUS, four DCUF-trained DeBERTa checkpoints gain 9.61 strict-score points [8.77, 10.43] when DCUF evidence is replaced by UnifEE evidence, and 7.92-9.08 points survive with answers held fixed, against a 1.96-point accuracy gain (Tables 2-3). Second, in frozen 8B LLMs, the fixed-answer evidence gain responds to context budget on FEVEROUS (+3.84 Qwen, +3.10 Llama) while the prespecified cross-dataset replication fails, recorded as 'restricted/mixed' (Section 6.1, Table 15). A methodological claim accompanies both: endpoint-only reporting cannot reconstruct attribution; saving answers and submitted evidence under both conditions plus two extra scoring passes suffices (Section 8).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the FEVEROUS strict score used in this paper?
The percentage of claims for which the system returns a correct answer AND includes the complete annotated evidence group among its submitted evidence. A claim only scores if both hold, so score gains can come from better answers, better evidence, or both - decomposing which is exactly this paper's question.
What is the paper's main finding?
For four DeBERTa verifiers on FEVEROUS, 7.92-9.08 of a 9.61-point strict-score gain from replacing DCUF with UnifEE evidence remains when answers are held fixed, while answer accuracy rises only 1.96 points. For two 8B LLMs, the evidence gain depends on context budget on FEVEROUS (+3.84/+3.10 points at 2,048 vs 256 tokens), but the effect does not replicate across FEVER and SciFact under the paper's prespecified criterion.
Are the code and data available?
Partly. Per the supplement, prompt-building functions, mappings, scorer code, all eight Study 1 prediction files, scoring flags and all raw LLM responses are public. Trained checkpoint weights, fully instantiated prompts and the source corpora are excluded (Appendix M), so scoring can be replayed exactly but the models cannot be re-run as configured.
Why is the LLM result called 'restricted/mixed evidence'?
It is the paper's own prespecified label: the FEVEROUS context effect passes local magnitude, interval and sign criteria for both models, but the rule required the same effect on two or more datasets including SciFact, which did not happen, and not all twelve adjusted intervals fit inside the ±2-point small-effect band.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 are not visually analyzable from the extraction; their values were recovered from Tables 15 and 26 and the Section 7 text.
- The extracted text appears complete through Appendix M, but the supplementary CSV/code archives could not be inspected, so availability claims are as stated by the authors.
- The footnote states ChatGPT, Claude and Codex assisted implementation, checking and drafting; claims rest on archived outputs that cannot be independently re-executed from this text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.