Judging by the Cover: Cleaning LLM Truthfulness Benchmarks to Avoid Surface-Level Feature Leakage
Foad Namjoo, Remy Ogasawara, Amirali Abdullah, Cullen Anderson, Narmeen Fatimah Oozeer, Jeff M. Phillips
cs.CL · cs.LG
Quick verdict
Detects surface leakage in TruthfulQA (6-feature probe AUC 0.715), releases Audit-Prune cleaner and TruthfulQA-476; rigorous, well-supported audit work.
Summary
The paper audits binary-choice TruthfulQA and 13 other benchmarks for surface-level feature leakage: systematic differences between correct and incorrect answers that let models exceed chance without the intended reasoning. Using SURFACE6 — six interpretable features (negation lead/count, hedging, length, token statistics) — with logistic regression and grouped cross-validation, a question-blind probe separates TruthfulQA's correct from incorrect answers at 0.689 accuracy / 0.715 AUC (negation dominant). HaluEval QA (0.973) and MedHallu (0.821) leak even more; BoolQ and PIQA sit near chance. The authors propose Audit-Prune, a greedy removal-and-add-back procedure targeting a pre-declared residual AUC threshold, and release TruthfulQA-476 (476/790 pairs, AUC 0.528, p=0.048), which preserves model rankings over 14 open-weight models (Spearman ρ=0.915) and leaves less residual leakage than AFLite at matched sizes (0.528 vs 0.603 at N=476). A surface-inversion test (n=131 adversarial pairs generated with knowledge of SURFACE6, plus a natural control) shows classifiers trained on the cleaned set are substantially more robust (e.g., SmolLM2-1.7B 0.427→0.634). Limitations: narrow hand-designed feature family, English-only lexicons, greedy non-optimal pruning, LLM-generated and partially circular evaluation cohorts, open-weight-only model panel, and shifted category coverage (e.g., Politics retains 1/10 pairs).
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The 2025 binary-choice reformulation of TruthfulQA still contains exploitable surface-form leakage: a question-blind six-feature logistic probe reaches 0.689 grouped-CV accuracy / 0.715 AUC on 790 pairs, driven mainly by negation.
- Surface leakage is widespread across benchmarks — HaluEval QA (AUC 0.973) and MedHallu (0.821) exceed TruthfulQA, and HaluBench inherits the HaluEval length artifact (AUC 0.971 on its verbatim 10,000-example slice).
- Audit-Prune reduces leakage to a pre-declared AUC threshold while retaining more pairs than fixed-prefix removal or AFLite at matched residual leakage, yielding the released TruthfulQA-476 (476/790 pairs, AUC 0.528, p=0.048, model-ranking Spearman ρ=0.915).
- Classifiers trained on cleaned TruthfulQA are markedly more robust on surface-inverted adversarial items (e.g., SmolLM2-1.7B 0.427→0.634, McNemar p<0.001) while holding accuracy on a natural control set.
- IRT-selected tinyBenchmarks anchors do not remove the confound: the 95 relevant anchors audit at AUC 0.7018, comparable to a random 100-pair subset.
Strengths
Interpretable probe with a sound, leakage-aware evaluation protocol
SURFACE6 uses six answer-only features (neg_lead, neg_cnt, hedge_rate, word_count, avg_token_len, type_token) with StandardScaler + l2 logistic regression under grouped 5-fold CV that keeps each question pair in one fold. Ablations localize the signal: dropping NEG_CNT costs −0.074 AUC (Table 1), removing the negation group drops 0.715→0.592 (Table 7), and an extended negation lexicon changes AUC by <0.0002, showing the narrow lexicon is not missing signal.
Direct comparison against the established cleaner at matched sizes
Section 3.3 and Table 2 compare Audit-Prune with an adapted AFLite (m=64 bge-large classifiers, six runs with sd) at N ∈ {408, 476, 536, 588}: at N=476 residual audit AUC is 0.528 vs AFLite's 0.603 with comparable ranking fidelity, and Table 14 shows the fixed-prefix baseline cannot reach any θ ≤ 0.58 (minimum attainable AUC 0.5826) while Audit-Prune reaches θ=0.51 at N=401.
Downstream consequence demonstrated, not just asserted
Exemplary artifact and reproducibility package
Candid limitations treatment
Weaknesses
Headline cleaned-set claim rests on borderline significance with post-hoc threshold selection
Circularity risk in the generalization test
Model-ranking fidelity restricted to small open-weight models
Rank fidelity against the leaky full benchmark is a contested fidelity target
Material category-coverage shift after pruning
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow but consequential question: does the 2025 binary-choice reformulation of TruthfulQA still admit answer-form shortcuts? Four claims are made. (1) Leakage is detectable: a question-blind probe built on six hand-crafted features (SURFACE6: negation lead, negation count, hedge rate, word count, average token length, type-token ratio) reaches 0.689 grouped-CV accuracy / 0.715 AUC on 790 pairs (1,580 answers). (2) The problem is broad: HaluEval QA audits at AUC 0.973 (a length artifact — hallucinated answers average 11.2 vs 2.3 words), MedHallu at 0.821, MultiNLI at 0.641, while BoolQ (0.525) and PIQA (0.509) sit near chance. (3) A remedy exists: Audit-Prune, a greedy removal-and-add-back algorithm targeting a pre-declared AUC threshold θ, producing the released TruthfulQA-476. (4) The leakage matters downstream: classifiers trained on the leaky set are fooled by surface-inverted items; cleaned-set training mitigates this. The scope is honestly framed as a partial-input, feature-family-bounded audit.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is surface-level feature leakage in TruthfulQA?
It means the correct and incorrect answer texts differ in style rather than substance — for example, correct answers more often start with No or contain negation words — so a classifier that never reads the question can pick the true answer from style alone. This paper measures 0.689 accuracy (chance is 0.5) on the binary version of TruthfulQA using just six such style features.
What is TruthfulQA-476?
A publicly released cleaned subset of the binary-choice TruthfulQA benchmark containing 476 of the original 790 question pairs. A six-feature style probe barely separates its true and false answers (AUC 0.528, versus 0.715 on the full set), while model rankings largely agree with the full benchmark (Spearman 0.915). It is offered as a drop-in, leakage-controlled replacement for the binary TruthfulQA evaluation.
How does the Audit-Prune algorithm work?
Starting from the full dataset, it repeatedly removes the question pair whose two answers most reinforce the style asymmetry the probe exploits — scored by classifier weights times the pair's standardized feature gaps — refitting the classifier after every removal until cross-validated AUC falls under a chosen threshold; it then re-adds any removed pairs that keep AUC under the threshold, maximizing the amount of retained data.
Is Audit-Prune better than AFLite for cleaning benchmarks?
On TruthfulQA at matched retained sizes, yes on the targeted leakage: at 476 retained pairs the residual style-probe AUC is 0.528 versus AFLite's 0.603, with comparable model-ranking fidelity. The advantage comes from targeting six interpretable surface cues directly, whereas AFLite removes whatever an embedding-based classifier ensemble finds most predictable, for whatever reason.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction is essentially complete (text runs through Appendix F and Table 14 with no mid-sentence cut), so no truncation caveat applies; Figure 1's diagram text is interleaved/garbled by PDF extraction, but its numbers were cross-checked against Tables 1 and 14 and all cited figures appear in the extracted tables.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.