Linguistic Features for Interpretable Textual Entailment
David Torres-Moreno, Jorge Hermosillo-Valadez, Jorge Hermosillo-Valadez, Asela Reig-Alamillo
cs.CL · cs.SC
Quick verdict
SLITE: 17 interpretable features reach 96% SICK-CE and 83% SICK, near RoBERTa; strong hybrid result but SICK-only eval and no significance tests weaken it.
Summary
The paper introduces SLITE, a hybrid, inherently interpretable model for Recognizing Textual Entailment combining two feature layers: a Structural-Relational layer based on ConceptNet-derived semantic compatibility/incompatibility between premise-hypothesis entities plus a semantically extended Jaro similarity with negation-sensitive polarity matching, and a Distributional-Informational layer computing alignment, entropy, and transfer-entropy statistics over submatrices of embedding similarity matrices. A logistic regression over the 17 features reaches 96% accuracy on binary SICK-CE and 0.828 on three-class SICK, surpassing the interpretable IsoLex baseline by 4 percentage points and trailing fine-tuned RoBERTa (98%) by about 2. Ablations show the Structural-Relational layer drives entailment/contradiction detection (contradiction F1 falls from 0.812 to 0.661 without it) while Distributional features chiefly aid neutral-class detection. Cross-annotation-shift experiments suggest better robustness than BERT (50.8% vs 29.4% in one condition). SHAP analyses with four worked cases give instance-level transparency, candidly exposing a world-knowledge failure. Main limitations: evaluation restricted to the SICK dataset family, hand-tuned constants (E8's 0.1 weight, Jaro penalties 0.01/0.6/0.8) tuned on a 500-pair dev set, no significance testing or confidence intervals, and questionable applicability of transfer entropy to non-temporal similarity matrices.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A logistic regression on 17 interpretable features achieves 96% accuracy on SICK-CE and 0.828 on three-class SICK, exceeding IsoLex by 4 points and trailing fine-tuned RoBERTa by about 2 points on SICK-CE.
- Structural-Relational features are the primary classification drivers (removing them drops contradiction F1 from 0.812 to 0.661), while Distributional-Informational features mainly support neutral-class detection.
- SLITE generalizes across annotation shifts better than fine-tuned BERT (50.8% vs 29.4% accuracy when tested on original SICK labels after Logic-annotation training, Table 5).
- The model's principal failure mode is missing world knowledge, specifically event-level mutual exclusivity relations absent from ConceptNet (Case 2, Section 5.4).
Strengths
Inherent interpretability backed by genuine instance-level analysis
Logistic regression on 17 linguistically defined features plus per-case SHAP waterfalls (Section 5.4), including a candid high-confidence failure (Case 2) traced to a spurious ConceptNet synonymy — rare transparency about error mechanisms in this genre.
Evaluation breadth within the SICK ecosystem
Original SICK, SICK-CE, MonaLog-corrected SICK, and the Logic/Commonsense re-annotations (Tables 2-6), plus a cross-annotation-shift probe where SLITE (50.8%) beats BERT (29.4%) (Table 5) — a clever test of dataset-bias reliance.
Systematic two-level ablation evidence
Extreme efficiency with competitive accuracy
Transparent mathematical specification
Weaknesses
No inferential statistics
Hand-tuned constants with overfitting risk
External validity untested
Conceptually underdefended transfer entropy
Baseline comparability and knowledge-resource noise
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SLITE targets Recognizing Textual Entailment (RTE/NLI) with an inherently interpretable pipeline: 17 engineered linguistic features feeding a logistic regression. The paper claims (Abstract, Section 5): 96% accuracy on binary SICK-CE and 0.828 on three-class SICK (Tables 2 and 4); a 4-point win over the interpretable IsoLex baseline and a ~2-point gap to fine-tuned RoBERTa on SICK-CE (Table 3); SR-layer dominance with DI-layer support for the neutral class (Section 5.6); and better-than-BERT robustness under annotation shift (Table 5). The framing — interpretability without competitive sacrifice — is well supported *within the evaluated scope*, which is exclusively the SICK dataset family (original, CE, MonaLog-corrected, Logic/Commonsense re-annotations).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SLITE and how does it detect textual entailment?
SLITE is an interpretable hybrid model for Recognizing Textual Entailment — deciding whether a hypothesis sentence follows from a premise. It extracts entities from both sentences, classifies their relationships via the ConceptNet knowledge graph (equivalence/generalization, opposition, specificity, or no relation), and combines these with alignment, entropy, and transfer-entropy statistics over word-embedding similarity matrices into 17 features. A logistic regression then outputs entailment, contradiction, or neutral, with every decision traceable to specific feature contributions.
How accurate is SLITE compared to RoBERTa?
On the SICK-CE binary benchmark the paper reports 96% accuracy for SLITE versus 98% for fine-tuned RoBERTa — a 2-point gap — while beating the interpretable IsoLex baseline by 4 points (92% to 96%). On the full three-class SICK dataset SLITE reaches 0.828 accuracy. SLITE uses a linear classifier on 17 hand-designed features, a tiny fraction of RoBERTa's roughly 355 million parameters.
What datasets was SLITE evaluated on?
Only the SICK family: original three-class SICK (9,927 pairs), its binary entailment-contradiction subset SICK-CE (4,245 pairs), the MonaLog-corrected SICK re-annotation, and the SICK Logic and Commonsense re-annotations. Larger benchmarks such as SNLI, MultiNLI, or adversarial sets (ANLI, HANS) were not evaluated and are listed as future work.
What are the main limitations of SLITE?
Its main failure mode is missing world knowledge — for example, it cannot tell that 'putting away an instrument' and 'playing an instrument' are incompatible, because ConceptNet wrongly treats them as related. It is also validated only on SICK-derived data, several tuned constants risk development-set overfitting, and the paper reports no statistical significance tests for its comparisons.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 (pipeline diagram, SHAP waterfall and summary plots) could not be analyzed from the extracted text; per-feature values cited come from the surrounding prose only.
- Table 5's train/test condition mapping is partially garbled in extraction; cross-annotation-shift numbers were cross-read against the Section 5.6 narrative.
- This is an arXiv v1 preprint; no peer-review verdict exists at review time.
- Two cited works are dated 2026, including the foundational SKA paper by the same authors; their content could not be verified from this text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.