Solving versus Verifying: Catching Contradictions in Tax Reasoning Systems
Albert Sadowski, Jarosław A. Chudziak
cs.CL · cs.AI
Quick verdict
Claims top LLMs compute through contradictions yet self-verify them; small, well-controlled SARA study; credible cheap gate; confounded, narrow scope.
Summary
The paper asks whether LLMs that solve statutory tax cases accurately also refuse defective inputs, and whether the same models detect defects when asked to verify rather than solve. From 91 SARA (LegalBench) tax cases the authors build 78 redacted variants, with a required numeric fact removed, and 108 contradicted variants, where a rule-generated conflicting sentence about a categorical fact is inserted verbatim and manually reviewed, then evaluate six models: GPT-5 mini, GPT-5.2, Claude Sonnet 4.6, Qwen3.7-Plus, Kimi K2.5 and Gemini 2.5 Flash. The four high-accuracy models (89-97% within ±10% on clean inputs) abstain on redactions (clean-answer preservation at most 1.3%) yet return the clean answer on 63-76% of contradictions with only 0-15% abstention. Asked instead to verify the same inputs, they flag 75-83% of those contradictions (95-97% for filing-status conflicts, 30-59% for dependent counts, many defeasible). A contradiction gate — one extra verification call that overrides the answer with null when a conflict is reported — raises contradiction abstention to 70-83% across all six models at a clean-accuracy cost of 0-5.1 percentage points; a self-consistency resampling baseline detects only 11-35%. Main limitations: one benchmark and legal domain, defect type confounded with fact category, small samples over three runs without significance tests, and two direct-answer models scoring very low under the single-number schema.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On SARA-derived tax cases, the four high-accuracy models abstain when a numeric fact is redacted (clean-answer preservation ≤1.3%) but compute through injected categorical contradictions, returning the clean answer 63-76% of the time with 0-15% abstention.
- Asked to verify rather than solve, the same models flag 75-83% of the identical contradictions, with detection varying by type (filing status 95-97%, marital status 78-83%, dependent count 30-59%).
- A one-call contradiction gate that overrides the answer on a reported conflict raises contradiction abstention to 70-83% across all six models at 0-5.1 percentage points of clean-accuracy cost, with no training or external tooling.
- Self-consistency resampling of the solve task detects only 11-35% of contradictions, so the recovered detection is attributable to the task change rather than to sampling, with a temperature-0 caveat.
- High accuracy on well-formed statutory inputs is therefore an incomplete measure of reliability for legal AI deployment.
Strengths
Clean solve-vs-verify contrast isolates the phenomenon
Holding weights and inputs fixed, solve-time abstention on contradictions is 0-15% for the four high-accuracy models (Table 3) while verify-time detection of the identical cases is 75-83% (Table 4) — the failure is task-dependent, not a capability ceiling.
Perturbation hygiene above genre norm
Contradictory sentences come from a fixed rule table, inserted verbatim at least one sentence from the original fact; automated integrity checks plus two manual review passes, the second removing 8 unsound cases (consistent restatements, narrative-entailed statuses, non-conflicts) per §3.2.
Selective-failure control and resampling baseline
Gate cost measured per model, not asserted
Candid scope bounding
Weaknesses
Defect type × fact category confound
Small samples, no inferential tests
Handicapped direct-answer arms
Verifier is not a validity judge
Same-model correlated errors, informal lower bound
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a deployment-shaped question: when statutory tax-reasoning systems receive defective inputs, do they abstain — and if they compute anyway, can the same model catch the defect when the task changes from solving to verifying? From 91 SARA tax-liability cases (via LegalBench), the authors generate 78 redacted variants (a required numeric fact removed) and 108 contradicted variants (a verbatim, rule-table-generated conflicting sentence about a categorical fact), each passing automated integrity checks and manual review (§3.2). Three claims carry the paper: (1) high-accuracy models abstain on missing facts but compute through contradictions, returning the clean answer 63-76% of the time at 0-15% abstention (Table 3); (2) asked to verify, the same models flag 75-83% of those identical cases (Table 4); (3) a one-call contradiction gate converts detection into abstention at 0-5.1 percentage points of clean-accuracy cost across all six models. Solver and verifier are the same weights by design, so the comparison isolates task framing. Claims are scoped carefully and the artifact story is strong.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the 'contradiction gate' in this paper?
A single extra LLM call in which the same model that solves the tax case is asked to verify whether the input facts are sufficient and internally consistent. If the verifier answers 'contradiction', the system overrides the solver's numeric answer with an abstention (null). It needs no training or external tools and cost 0-5.1 percentage points of clean-input accuracy across six models in the study.
Why do LLMs miss contradictions but catch missing facts in tax problems?
The paper's evidence: when a required fact is missing, strong models abstain or at least stop reproducing the clean answer (preservation under 2%), but when two stated facts conflict, models still have values to compute with and resolve the conflict silently, returning the clean answer 63-76% of the time. Asked explicitly to verify the input, the same models flag 75-83% of the same conflicts, so the failure is tied to the task, not to the model's knowledge.
What dataset and models were used?
91 tax-liability cases from the SARA statutory reasoning task (included in LegalBench), perturbed into 78 redacted variants (a numeric fact removed) and 108 contradicted variants (a categorical fact contradicted by an inserted sentence). Six models were evaluated: GPT-5 mini, GPT-5.2, Claude Sonnet 4.6, Qwen3.7-Plus, Kimi K2.5 and Gemini 2.5 Flash. Data, scripts and prompts are on Zenodo.
Does the finding transfer beyond US tax law?
Unknown. The authors scope the result to SARA-derived cases and a single legal domain (US federal income tax), and list other legal settings, external/cross-model verifiers and defeasible conflicts as open questions. Detection is also uneven within tax: filing-status conflicts are caught 95-97% of the time but dependent-count conflicts only 30-59%.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Exact model identifiers, full prompts and per-run outputs are externalized to the project repository/Zenodo (footnotes 3-4) and do not appear in the paper text; this review could not inspect them.
- Figures 1-2 appear only via captions in the extracted text; their internal values were not readable, so figure-derived claims here come from captions and body text only.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.