Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understanding
Fan Zhang, Yankai Chen, Zhuohan Xie, Yixi Zhou, Sijia Peng, Lei Fan, Xinhua Ji, Cunyuan Zheng, et al.
cs.CL
Quick verdict
Claims cost-latency-correctness trade-offs for Jev vs 9 LLMs on ContractNLI; rigorous paired design, but 30-anchor panel keeps stability claims exploratory.
Summary
The paper benchmarks Jev 1.13.0, a 'decision-native' system that holds the contract as shared state and answers each hypothesis as a structured Choice question, against nine language models on ContractNLI classification. It evaluates four dimensions: inference cost, response time, baseline accuracy on the official 123-contract test split (2,091 judgments), and persistent correctness in a 30-contract stability panel where one anchor hypothesis is queried under four request conditions (varying visible hypotheses, requested outputs, and output order) with three repeats. Jev attains the lowest cost ($0.000228 per contract) and median latency (1.24 s), but its baseline accuracy (77.38%) trails all seven hosted LLMs (78.96-83.21%); rankings by all-twelve-correct answers diverge from accuracy rankings (Jev 23/30, Sonnet 5 24/30). Paired analyses show equal aggregate accuracy can hide compensating corrections and regressions, and agreement can preserve stable errors. Uncertainty uses contract-level bootstrap intervals, unadjusted for multiple comparisons. Key limitations: the 30-anchor panel yields very wide intervals, three comparators were added after results were inspected, and interface, provider, and timing differences confound cost and latency comparisons. Code is released on GitHub.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Jev 1.13.0 has the lowest per-contract cost ($0.000228) and the lowest median baseline response time (1.24 s) among the ten evaluated configurations.
- All seven hosted LLM comparators exceed Jev's 77.38% baseline accuracy (range 78.96-83.21%), though the Luna-minus-Jev paired interval includes zero.
- Rankings by baseline accuracy differ from rankings by persistent (all-twelve-correct) correctness across the compared models.
- Equal aggregate accuracy can conceal compensating corrections and regressions (e.g., Qwen4B: A and B accuracy identical at 75.6% in the Figure 1 dev panel while 9/90 paired judgments change).
- Agreement across repeats does not imply correctness: consistent labels can be consistently wrong (Jev five, Astra six stable errors).
Strengths
Decision-level accounting beyond aggregate accuracy
The paired decomposition of changes into corrections, regressions, and wrong-to-wrong switches (Table 6; Section 4.3), plus the stable-error vs stable-correct distinction (Section 4.2), is the paper's real contribution: Jev and Astra repeat labels on 28/30 anchors, but five and six of those are wrong throughout, versus one for Sonnet.
Same-target comparison isolates the mean/persistence gap
Table 2 holds target composition fixed: Jev 79.44% (286/360) vs Qwen9B 79.17% (285/360) mean correctness, yet 23 vs 18 anchors correct in all twelve responses, because Qwen9B's errors disperse (8 changing targets, 2 invalid) while Jev's concentrate (5 stable errors).
Strict validity accounting with truncation bounds
Controlled condition ladder and anti-overclaim discipline
Unusually transparent protocol
Weaknesses
Stability panel is too small for its headline use
Configuration confounds pervade cost and latency claims
No multiplicity control on ~20+ paired intervals
Staged, partly post-hoc model selection
The central comparator is under-described
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper benchmarks Jev 1.13.0 - described as a 'decision-native' system that holds the contract as shared state and poses each hypothesis as a structured Choice question - against nine language models (Qwen3.5-4B/9B, Gemini 3.5 Flash-Lite, Gemini 3.1 Pro Preview, GPT-5.6 Luna/Terra, GPT-6 Astra, Claude Sonnet 5, Claude Haiku 4.5) on ContractNLI's classification task. It claims (RQ1-RQ3) that Jev offers the lowest cost and median latency, that hosted LLMs win on baseline accuracy, that accuracy rankings and persistent-correctness rankings diverge, and that aggregate accuracy hides compensating decision changes. Evidence comes from the official 123-contract test split (2,091 judgments) plus a 30-contract stability panel with four request conditions (A: anchor only; B: 17 hypotheses visible, anchor requested; C/D: all 17 requested, anchor first/last) and three repeats per condition.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Jev in this paper?
Jev (version 1.13.0) is described as a 'decision-native' model: instead of generating free text, it receives the contract as shared state and answers each hypothesis as a structured multiple-choice question (entailment / contradiction / not_mentioned). The paper does not detail its architecture, size, or training data.
Which model was most accurate on ContractNLI in this study?
Gemini 3.1 Pro Preview had the highest baseline accuracy at 83.21% on the official 123-contract test split, followed by Gemini 3.5 Flash-Lite at 82.54%. Jev scored 77.38% but was the cheapest ($0.000228 per contract) and fastest (1.24 s median) configuration.
Did Jev beat the language models?
Not on accuracy - all seven hosted LLMs scored higher (78.96-83.21% vs 77.38%). Jev stayed competitive on 'persistent correctness' (23/30 anchors correct in all twelve responses, vs Sonnet 5's 24/30), but the study explicitly says the small 30-anchor panel cannot establish a general stability advantage.
Is the code and data available?
Yes. The paper links a GitHub repository (ZF-Utokyo/Jev-Benchmark) and states that retained inputs, individual predictions, and scoring code support independent recomputation. ContractNLI itself is public under CC BY 4.0. Exact re-runs of hosted models may not be possible due to provider-side model drift.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Several passages were elided in extraction (parts of Section 3.2 and Appendices C, F, L); judgments rest on the retained tables and narrative.
- Figures 1-6 are available only as caption text; figure-based claims (frontier shape, heatmap cells) were cross-checked against Tables 1-6 and 23 where possible.
- Jev's architecture, size, and developer affiliation are absent from the extracted text, so the central comparator cannot be independently assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.