E-CONAN (Entailment, CONtradition And Neutral) Benchmarks: Arabic Textual Entailment and Natural Inference Datasets
Khloud AL Jallad, Nada Ghneim, Ghaida Rebdawi
cs.CL · cs.AI · cs.LG
Quick verdict
Introduces two multi-source Arabic NLI/RTE benchmarks with 9 encoders + 5 LLMs zero-shot; diverse and useful, but accounting slips, unspecified eval models and uncontrolled overlap weaken the transfer claims.
Summary
The paper introduces E-CONAN, two Arabic inference benchmarks assembled from eight source datasets spanning four construction paradigms: ArNLI (itself including ArbTEDS, SICK, PHEME and Stanford contradiction pairs), human-translated Arabic SNLI, Arabic XNLI, the stance-derived AnsStance set, plus news-headline ArEntail for the RTE variant. E-CONAN-2 contains 24,875 binary pairs (37% entail); E-CONAN-3 contains 18,875 triples (about 32/34/34% across entailment, contradiction, neutral). Nine multilingual encoders were evaluated zero-shot alongside an Arabic-specific MARBERT-XNLI-tuned baseline, plus five LLMs with a fixed prompt at temperature 0. mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 leads encoders with 0.71 accuracy on E-CONAN-3 and 0.86 on Arabic XNLI (test+validation); Gemma3 leads LLMs with 0.68 on E-CONAN-3, rising to 0.95 under 2-way reformulation, with Qwen2.5 at 0.94. Error analysis finds pretrained models misled by lexical overlap and LLMs by topical familiarity, with most 3-way errors between neutral and contradiction. Main limitations: per-source statistics do not reconcile cleanly with totals, the exact Gemma3/Qwen2.5/ALLaM variants and decoding stack are unspecified, machine-translation artifacts are inherited, inter-annotator agreement was not re-validated after merging, and no code is released.
Review complete
Assessment at a glance
6
risk flags identified
● high 3
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- E-CONAN-2 (24,875 pairs) and E-CONAN-3 (18,875 pairs) unify four data-construction paradigms into class-balanced Arabic RTE and NLI benchmarks that are more diverse than XNLI or ArNLI alone.
- mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 outperforms all other evaluated encoders zero-shot, scoring 0.71 accuracy on E-CONAN-3 and 0.86 on Arabic XNLI (test+validation).
- Gemma3 is the best of five evaluated LLMs with 0.68 accuracy on E-CONAN-3, rising to 0.95 (Qwen2.5 0.94) when the benchmark is reformulated as 2-way, because most 3-way errors confuse neutral with contradiction.
- Pretrained models fail mainly through lexical-overlap bias while LLMs fail through topical familiarity, with both showing an entailment-class bias.
- E-CONAN's mixed machine-translated, human-validated, hand-crafted and news-headline composition makes it a better probe of model generalization than single-paradigm predecessors.
Strengths
Genuinely multi-source design
Eight source datasets and four construction paradigms (Section III) give E-CONAN-3 a near-balanced 32/34/34% class split versus ArNLI's 53%-neutral skew (Table 5), and Figure 4 reports a concrete domain mix: News 35%, Simple Everyday 30%, Social Media 20%, Multi-Genre 15%.
Broad, mostly fair evaluation
Nine multilingual encoders plus an Arabic-specific MARBERT baseline (Section IV.a) and five LLMs under a shared temperature-0 prompt (Section IV.c), all zero-shot across ArNLI, ArXNLI, ArSNLI, AnsStance and E-CONAN, with accuracy plus macro precision/recall/F1 (Tables 6-10, Figures 9-13).
Transparent quality reporting of inherited data
Actionable error analysis
Statistical significance check
Weaknesses
Internal accounting and versioning inconsistencies
Item overlap between compared datasets is uncontrolled
Annotation validity of merged sources is thin
Reproducibility gaps for the LLM evaluation
Uneven documentation and minor overclaiming
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
E-CONAN Benchmarks — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What are the E-CONAN datasets and how large are they?
Two Arabic inference benchmarks from eight sources (ArNLI, Arabic SNLI, Arabic XNLI, AnsStance, plus ArEntail for the RTE set). E-CONAN-2 is a 2-way (entail / not-entail) set of 24,875 pairs; E-CONAN-3 is a 3-way (entailment / contradiction / neutral) set of 18,875 pairs with roughly balanced classes.
Which models perform best on E-CONAN?
Among zero-shot pretrained encoders, mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 is best with 0.71 accuracy on E-CONAN-3 (0.84 on E-CONAN-2). Among the five evaluated LLMs, Gemma3 leads with 0.68 on the 3-way task and 0.95 on the 2-way reformulation, with Qwen2.5 at 0.94.
Where can I download E-CONAN?
The paper's Data Availability section points to Hugging Face at huggingface.co/datasets/KhloudJ/E-CONAN. The datasets are released, but no evaluation code is mentioned.
What are the main limitations of the E-CONAN paper?
Per-source statistics do not fully reconcile with totals, the exact LLM variants and decoding settings are unspecified, machine-translation artifacts are inherited from XNLI/SNLI sources, inter-annotator agreement was not re-validated after merging, and only Modern Standard Arabic is covered (no dialects).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Arabic example strings (Table 11, word cloud) have disrupted character ordering from PDF extraction, so the linguistic content of examples was judged from the paper's own English glosses rather than the Arabic text itself.
- Figures 9-15 survive only as embedded text sequences; numeric judgments are anchored to the tables and running text, which are complete for all cited values.
- Whether the Hugging Face dataset link currently resolves was not verified in this session; availability is reported as the paper states it.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.