arXiv paper· natural language processing· method paper

Can We Do Interpretable NLI with Graphs Based on Atomic Propositions?

Younes Boufouss, Luc Pommeret, Thomas Gerald, Patrick Paroubek, Sophie Rosset

cs.AI · cs.IR

Quick verdict

Claims graph-only NLI costs 1.9 pts on SNLI and 9-14 on ANLI vs identical text models; rigorous and honest, but the headline SNLI gap sits within seed noise.

Summary

This paper asks whether natural language inference can rest entirely on interpretable structures. Sentences are decomposed into atomic propositions (MPropositionneur-V2-large), converted into ConceptNet triples by Qwen3.5-9B under a constrained JSON schema with a 28-relation enum, and rendered as three graphs per pair — premise, hypothesis, and a retrieved ConceptNet subgraph — serialised in a fixed order for a fine-tuned Qwen3.5-0.8B-Base classifier that never sees raw text. On SNLI the pipeline reaches 89.7% (three-seed mean ± 0.006), 1.9 points below an identically trained text model; combining both modalities gives 92.1%. On ANLI (R1–R3, with SNLI replay) it trails its text counterpart by 9–14 points, though it matches published RoBERTa-large on R2 and R3. A scaling experiment — adding roughly 600,000 MNLI and FEVER-NLI training pairs yields only +0.6 points on ANLI — supports the claim that the gap is representational rather than a data problem. Ablations show the retrieved knowledge is neutral on SNLI. Main limitations: single-seed text baselines and ANLI runs, two interrupted graph seeds (the 1.9-point gap sits within run-to-run variation), no human validation of interpretability, no direct evaluation of graph quality, and the no-K ablation untested on ANLI.

Review complete

Assessment at a glance

natural language processing

6

risk flags identified

● high 0

● medium 4

● low 2

Reproducibility readinesshigh
Code & data availableyes
Review confidencehigh
AI text signal0/20 passages flagged
Deep research viableCode-reproducibleNeeds GPU

5 strengths · 5 weaknesses · 5 author questions

AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

  • A graph-only NLI pipeline in which the classifier never sees raw text reaches 89.7% on SNLI, within 1.9 points of an identically trained text model of the same backbone.
  • On ANLI the graph model trails its own text counterpart by 9–14 points, a quantity the authors define and measure as the price of interpretability.
  • The ANLI gap stems from representational limits, not data scarcity: adding ~600,000 MNLI and FEVER-NLI training pairs improves ANLI by only ~0.6 points.
  • Graphs and text are complementary — combining both raises SNLI accuracy to 92.1% versus 91.6% for text alone.
  • The retrieved ConceptNet subgraph K contributes nothing on SNLI (0.895 without K vs 0.892 with).

Strengths

  • Strict ceteris paribus design

    Text and graph classifiers share the Qwen3.5-0.8B-Base backbone, optimiser and epoch budget (Section 5.2), and zero-shot controls show the base model is at chance on both modalities (0.343 SNLI, 0.334–0.335 ANLI, Table 1/2), cleanly attributing all competence to fine-tuning rather than the backbone.

  • Representation-vs-data bottleneck isolated experimentally

    Section 6.4 directly tests the under-training objection: adding ~600,000 MNLI + FEVER-NLI pairs moves ANLI by at most +0.6 (0.568/0.479/0.461 vs 0.565/0.462/0.463), far less than the 9–14 point gap, making the representational conclusion robust to seed noise.

  • Unusually candid limitations section

  • Full reproducibility package

  • Transparent negative result on external knowledge

Weaknesses

  • Headline SNLI gap sits within seed noise

  • RoBERTa-large parity claim is apples-to-oranges

  • Interpretability, the titular claim, is never validated

  • Graph quality never evaluated directly

  • K ablation missing exactly where retrieval should matter

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

Report preview

Preliminary Review: Can We Do Interpretable NLI with Graphs Based on Atomic Propositions?

The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.

  • All 5 strengths & 5 weaknesses, analyzed in detail
  • 6 risk flags with severity, section & reviewer questions
  • Methods & statistical evidence assessment
  • 5 questions a peer reviewer would ask
  • Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

What is the 'price of interpretability' in this paper?

It is the accuracy drop between a fine-tuned Qwen3.5-0.8B model reading raw premise/hypothesis text and the identical model reading only ConceptNet-style triple graphs: −1.9 points on SNLI and −9 to −14 points across the ANLI rounds.

How accurate is the graph-based NLI system compared to text models?

On SNLI it reaches 89.7% (three-seed mean), about 1.9 points under the same-backbone text model (91.6%) and 2.4 points under text+graphs (92.1%). On ANLI it scores 0.565/0.462/0.463 on R1–R3 with SNLI replay, 9–14 points behind its text counterpart, though matching the published RoBERTa-large numbers on R2 and R3.

Does the ConceptNet knowledge graph actually help?

On SNLI, no: removing the retrieved subgraph K changes accuracy only from 0.892 to 0.895 (Table 4). The authors did not run the no-K ablation on ANLI, where the retrieval mechanism was designed to matter most due to low lexical overlap.

Is the code and data for this paper available?

Yes — the pipeline code is on GitHub (l-pommeret/graph-nli) and the atomic-proposition datasets (Younes2E/atomic-snli and Younes2E/atomic-anli) are on the HuggingFace Hub; Appendix B gives end-to-end reproduction commands.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

  • Review is based on the embedded v2 full text including appendices; Figures 1–7 were assessed from their text transcriptions only, so any visual detail (layout, omitted curves) was not verifiable.
  • Some claims (e.g., seed-level details, interrupted-training specifics) rely on the authors' self-reporting in the Limitations section and could not be independently checked from the text.

Go deeper on this paper

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper
Can We Do Interpretable NLI with Graphs Based on Atomic Propositions? | Evolve.Pub