Can We Do Interpretable NLI with Graphs Based on Atomic Propositions?
Younes Boufouss, Luc Pommeret, Thomas Gerald, Patrick Paroubek, Sophie Rosset
cs.AI · cs.IR
Quick verdict
Claims graph-only NLI costs 1.9 pts on SNLI and 9-14 on ANLI vs identical text models; rigorous and honest, but the headline SNLI gap sits within seed noise.
Summary
This paper asks whether natural language inference can rest entirely on interpretable structures. Sentences are decomposed into atomic propositions (MPropositionneur-V2-large), converted into ConceptNet triples by Qwen3.5-9B under a constrained JSON schema with a 28-relation enum, and rendered as three graphs per pair — premise, hypothesis, and a retrieved ConceptNet subgraph — serialised in a fixed order for a fine-tuned Qwen3.5-0.8B-Base classifier that never sees raw text. On SNLI the pipeline reaches 89.7% (three-seed mean ± 0.006), 1.9 points below an identically trained text model; combining both modalities gives 92.1%. On ANLI (R1–R3, with SNLI replay) it trails its text counterpart by 9–14 points, though it matches published RoBERTa-large on R2 and R3. A scaling experiment — adding roughly 600,000 MNLI and FEVER-NLI training pairs yields only +0.6 points on ANLI — supports the claim that the gap is representational rather than a data problem. Ablations show the retrieved knowledge is neutral on SNLI. Main limitations: single-seed text baselines and ANLI runs, two interrupted graph seeds (the 1.9-point gap sits within run-to-run variation), no human validation of interpretability, no direct evaluation of graph quality, and the no-K ablation untested on ANLI.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A graph-only NLI pipeline in which the classifier never sees raw text reaches 89.7% on SNLI, within 1.9 points of an identically trained text model of the same backbone.
- On ANLI the graph model trails its own text counterpart by 9–14 points, a quantity the authors define and measure as the price of interpretability.
- The ANLI gap stems from representational limits, not data scarcity: adding ~600,000 MNLI and FEVER-NLI training pairs improves ANLI by only ~0.6 points.
- Graphs and text are complementary — combining both raises SNLI accuracy to 92.1% versus 91.6% for text alone.
- The retrieved ConceptNet subgraph K contributes nothing on SNLI (0.895 without K vs 0.892 with).
Strengths
Strict ceteris paribus design
Text and graph classifiers share the Qwen3.5-0.8B-Base backbone, optimiser and epoch budget (Section 5.2), and zero-shot controls show the base model is at chance on both modalities (0.343 SNLI, 0.334–0.335 ANLI, Table 1/2), cleanly attributing all competence to fine-tuning rather than the backbone.
Representation-vs-data bottleneck isolated experimentally
Section 6.4 directly tests the under-training objection: adding ~600,000 MNLI + FEVER-NLI pairs moves ANLI by at most +0.6 (0.568/0.479/0.461 vs 0.565/0.462/0.463), far less than the 9–14 point gap, making the representational conclusion robust to seed noise.
Unusually candid limitations section
Full reproducibility package
Transparent negative result on external knowledge
Weaknesses
Headline SNLI gap sits within seed noise
RoBERTa-large parity claim is apples-to-oranges
Interpretability, the titular claim, is never validated
Graph quality never evaluated directly
K ablation missing exactly where retrieval should matter
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Can We Do Interpretable NLI with Graphs Based on Atomic Propositions?
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the 'price of interpretability' in this paper?
It is the accuracy drop between a fine-tuned Qwen3.5-0.8B model reading raw premise/hypothesis text and the identical model reading only ConceptNet-style triple graphs: −1.9 points on SNLI and −9 to −14 points across the ANLI rounds.
How accurate is the graph-based NLI system compared to text models?
On SNLI it reaches 89.7% (three-seed mean), about 1.9 points under the same-backbone text model (91.6%) and 2.4 points under text+graphs (92.1%). On ANLI it scores 0.565/0.462/0.463 on R1–R3 with SNLI replay, 9–14 points behind its text counterpart, though matching the published RoBERTa-large numbers on R2 and R3.
Does the ConceptNet knowledge graph actually help?
On SNLI, no: removing the retrieved subgraph K changes accuracy only from 0.892 to 0.895 (Table 4). The authors did not run the no-K ablation on ANLI, where the retrieval mechanism was designed to matter most due to low lexical overlap.
Is the code and data for this paper available?
Yes — the pipeline code is on GitHub (l-pommeret/graph-nli) and the atomic-proposition datasets (Younes2E/atomic-snli and Younes2E/atomic-anli) are on the HuggingFace Hub; Appendix B gives end-to-end reproduction commands.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review is based on the embedded v2 full text including appendices; Figures 1–7 were assessed from their text transcriptions only, so any visual detail (layout, omitted curves) was not verifiable.
- Some claims (e.g., seed-level details, interrupted-training specifics) rely on the authors' self-reporting in the Limitations section and could not be independently checked from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.
