Fallacy Benchmarks Measure Scheme Recognition, Not Fallacy Detection
Navyansh Singh, Animesh Pathak, Aarav Singh
cs.CL · cs.LG
Quick verdict
Claims fallacy-benchmark FPRs are artifacts of valid-class construction; controlled evidence models track scheme identity, not validity, is largely convincing.
Summary
The paper argues that fallacy-detection benchmarks report artificially low false-positive rates because their single 'valid'/'none' negative class is a construction residue containing almost no scheme-matched negatives: valid arguments instantiating the same reasoning scheme as a fallacy type. Auditing four benchmarks (CoCoLoFa, Reddit, Argotario, MAFALDA), two annotators found only 3 scheme-matched items among 183 sampled 'none' cases. The authors generate scheme-matched and wrong-scheme valid negatives from one Gemini 2.5 Flash pipeline, filtered by a cross-lineage DeepSeek judge and validated by blind annotators (93.1%/77.5% precision). A ModernBERT classifier trained on CoCoLoFa flags 58.9% of matched negatives versus 16.6% of its native valid class (Reddit within-corpus: 62.0% vs 5.7%); the crucial control shows matched negatives are labeled as the source fallacy type 40.9 points more often than wrong-scheme ones, which instead receive their actual scheme (85.9% vs 0.4%). The dissociation survives within-corpus retraining, a different encoder, and three zero-shot LLMs, and attenuates on the deliberately built Touche 2026 negative class. Limitations: raw rates are generator- and prompt-dependent (sign reversals under a binary prompt), the audit rests on 3/183 items, the natural-text check has only 13 confirmed items, and the wrong-scheme condition is 77.5% certified valid.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The 'valid'/'none' classes of four fallacy benchmarks contain almost no scheme-matched valid arguments (3 of 183 audited items), so reported false-positive rates average over an easy population.
- On constructed scheme-matched negatives, false-positive rates jump from 16.6% to 58.9% (CoCoLoFa) and from 5.7% to 62.0% (Reddit), showing reported rates understate over-flagging where detection actually matters.
- Classifiers label matched negatives as their source fallacy type 40.9 points more often than wrong-scheme negatives and assign wrong-scheme items their actual scheme 85.9% vs 0.4% — they learned which scheme an argument uses, not whether it uses it correctly.
- The same dissociation appears in three zero-shot LLM detectors and attenuates on a deliberately built negative class (Touché 2026), so the artifact is a property of benchmark construction, not of one model family.
- A reported false-positive rate is uninterpretable until the valid class is audited for scheme-matched coverage; the authors release the Scheme Foils challenge set and a five-step audit procedure to enable this.
Strengths
Causally clean wrong-scheme control
Matched and wrong-scheme negatives come from one Gemini pipeline differing only in scheme identity (§4), so generation quality, topic, length and register cancel; the type-specific statistic (+40.9±1.9pp) and the 85.9%-vs-0.4% scheme-labeling asymmetry (§6.3) were fixed as predictions before measurement.
Convergent evidence across independent axes
The dissociation survives within-corpus Reddit retraining (gap grows 26.6→56.3pp, ruling out domain shift), RoBERTa (§6.3), three zero-shot LLMs (§6.4), naturally occurring scheme-matched text (7/13 flagged, §6.6), and discriminates a deliberately built negative class (Touché 26.2% vs 42.9% on matched negatives, §6.5).
Rigorous validation chain for constructed data
The failure is visible in the unmodified benchmark
Honest, actionable deliverable
Weaknesses
Prevalence claim rests on 3 items
Headline raw rates are generator- and prompt-fragile
Wrong-scheme columns are not FPRs and the type-specific contrast has a residual confound
Per-type heterogeneity tracks scheme-mapping quality
Thin validation coverage in places
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a scheme-matched negative in fallacy detection?
A valid argument that uses the same reasoning pattern (argumentation scheme, e.g., appeal to expert opinion) as a fallacy type while answering that scheme's critical questions — for example, citing a genuine, relevant domain expert. Its surface form resembles the fallacy but the reasoning is sound, making it the hardest negative case for a fallacy detector. This paper shows such items are nearly absent from benchmark valid classes and that classifiers heavily over-flag them.
What are the key numbers in this paper?
A ModernBERT classifier trained on CoCoLoFa flags 58.9% of scheme-matched valid arguments versus 16.6% of the benchmark's native valid class; trained from scratch on Reddit, 5.7% becomes 62.0%. Matched negatives are labeled as the source fallacy type 40.9 percentage points more often than equally careful wrong-scheme negatives, which are instead labeled with the scheme they actually use (85.9% vs 0.4% for the source type). The same pattern appears in three zero-shot LLMs.
What is Scheme Foils and where can I get it?
Scheme Foils is the authors' released challenge set: 1,042 scheme-matched and 1,127 wrong-scheme valid arguments across the CoCoLoFa and Reddit corpora, each carrying a human-validated precision estimate for its split (93.1% for matched items, 77.5% for wrong-scheme items). The intended use is to score an already-trained detector on the matched split against its own reported false-positive rate, and to check scheme specificity with the wrong-scheme split. It is hosted on GitHub (fine2006/the-concealment-hypothesis).
Does this mean fallacy detectors are useless?
No. The paper shows that the low false-positive rates reported on current benchmarks do not measure the ability to separate fallacious from valid arguments, because the valid classes rarely contain hard, scheme-matched cases. Models — both fine-tuned encoders and zero-shot LLMs — reliably recognize which argumentation scheme a text uses, but not whether the scheme is used correctly. The authors argue benchmarks need audited or deliberately constructed valid classes (as Touché 2026 attempted) before false-positive rates mean anything.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Tables 1-2 arrived as interleaved raw text; cell values were reconstructed from body prose, so per-cell alignment (seed counts, WS columns) is inferred rather than read directly.
- Figures and any graphical content were not analyzable; all numbers cited here come from prose or inline tables.
- The extraction appears complete through Appendix D, but if supplementary per-type tables were dropped, judgments touching per-type CIs inherit that uncertainty.
- arXiv metadata, affiliations, and the released repository contents could not be verified (no network access).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.