SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics
cs.AI · cs.SE
Quick verdict
Claims systematic LLM blind spots in version-constraint semantics via a 240-item oracle-labeled benchmark; credible, though mitigation evidence is single-run.
Summary
SemVerBench is a 240-item benchmark probing whether LLMs correctly decide if a version satisfies a constraint in three package ecosystems: npm, PEP 440 (Python), and Cargo. Items are Boolean, machine-checkable, and drawn from four balanced sources — each ecosystem's official resolver test suite plus proposals from three frontier LLMs — with labels fixed by two independent resolver implementations per ecosystem rather than by humans. Evaluating six models at temperature 0 over three runs, Claude-Opus leads at 90.6% and Sonnet at 86.8%, with GPT-5.1, GPT-4.1, GPT-4o, and Gemini-2.5-Pro between 80% and 83% (McNemar tests significant). Errors concentrate in specific mechanisms: a partial-comparator carry rule drops every model to roughly 60% on Cargo, and on zero-pad/post-release prefix-match corner cases GPT-5.1 scores 0/26 on a separate 67-item oracle-validated set while Claude reaches 97–100%. Injecting the relevant rule recovers most errors and light correct hints fix failed items, whereas interval decomposition does not help; the authors read this as an activation/application gap while conceding their design cannot separate it from supplying missing knowledge. Tool delegation to official resolvers reaches about 100%. Limitations include single-run mitigation experiments, small per-mechanism buckets, an adversarial rather than natural item distribution, and possible pretraining contamination of official-suite items.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Frontier LLMs exhibit systematic, per-mechanism blind spots in version-constraint resolution despite at-ceiling performance on the basic forms of the same rules (all six models fall to 59-63% on Cargo partial-comparator carry, vs a 56% bucket floor).
- PEP 440 prefix-match corner cases are a sharp vendor-specific blind spot: GPT-5.1 scores 0/26 on zero-pad/post-release 'true' items of the 67-item oracle-validated set while Claude reaches 25-26/26.
- Claude-Opus (90.6%) and Claude-Sonnet (86.8%) significantly outperform the OpenAI models and Gemini by McNemar testing, and no model shows significant self-favoritism on its own vendor's items (only Gemini scores significantly lower on its own, p=0.02).
- The failures reflect an activation/application gap rather than a knowledge gap: rule injection recovers most errors (E1 lifts GPT-5.1 from 80.4% to 91%), light correct hints fix previously failed items, and interval decomposition does not help.
- Because a free deterministic resolver exists, tool delegation reaches about 100% (E3, with only 2/1440 tool-output overrides), so coding agents should gate version decisions through resolvers rather than reason in-head.
Strengths
Fully verifiable task with a non-circular labeling oracle
Every item is a Boolean membership query with a unique answer; labels require agreement of two independent implementations per ecosystem (node-semver 7.8.1 x semantic_version 2.10.0; packaging 26.2 x pep440_rs 0.6.5; Rust semver 1.0.28 x semantic_version 2.10.0), plus a prerelease-policy robustness filter for PEP 440 (Section III-E).
Author-neutrality is engineered and empirically audited, not just asserted
Four balanced 20-item sources per ecosystem with proposer identity recorded; Table IV stratifies accuracy by source and finds no significant self-favoritism (largest gap GPT-4.1 +10, p~0.10); the only significant effect is anti-favoritism (Gemini 70.0 own vs 83.9 other, p=0.02); headline blind spots hold on official-suite items no panel model authored.
Statistical reporting above benchmark norms
The headline failure is validated beyond the small main bucket
Unusually candid limitations section
Weaknesses
Single-run mitigation and false-premise conditions undercut causal claims
The activation/application-gap diagnosis is confounded by design
External validity of the Cargo partial-comparator trap is unexamined
Small buckets and adversarial distribution limit generalization
Below-floor performance on ordered-exclusion is mischaracterized
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SemVerBench (arXiv 2609.11180, accepted at ICTAI 2026) claims to be the first direct benchmark of LLM comprehension of version-constraint resolution semantics. The task is deliberately narrow: given ecosystem E (npm, PEP 440, Cargo), version V, and constraint C, decide whether V satisfies C. Every item has a unique machine-checkable Boolean answer, so errors cannot be attributed to formatting or taste. The paper claims (1) systematic per-mechanism blind spots, (2) a cross-vendor gap favoring Claude, (3) an activation/application-gap diagnosis, and (4) that delegating resolution to real resolvers reaches ~100% accuracy. The narrowness is a genuine strength for attribution, and the paper is explicit that it measures isolated membership decisions, not end-to-end agent failures.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SemVerBench?
A benchmark of 240 machine-checkable yes/no questions testing whether large language models correctly decide if a version (e.g., 2) satisfies a dependency constraint (e.g., ==2.0.* or ^1.2.3) across three packaging ecosystems: npm, PEP 440 (Python), and Cargo. Every label comes from two independent resolver implementations agreeing, not from human annotators.
Which LLMs fail at version-constraint semantics?
All six tested models (Claude-Opus/Sonnet, GPT-5.1/4.1/4o, Gemini-2.5-Pro) fail the partial-comparator carry rule on Cargo, scoring roughly 60% against a 56% guessing floor. The sharpest single failure is GPT-5.1 on PEP 440 zero-pad and post-release prefix cases (0/26 on a 67-item validation set), while Claude models score 97-100%. Claude-Opus leads overall at 90.6%.
Should coding agents resolve version constraints themselves?
According to this paper, no. Giving the model a tool that calls the official ecosystem resolver reaches about 100% accuracy versus 80-91% for in-head reasoning. When no resolver is reachable, the second-best mitigation is injecting the relevant versioning rule into the prompt.
Is the benchmark biased because LLMs helped write the items?
The authors stratify accuracy by item source and find no statistically significant self-favoritism; the only significant author effect is Gemini scoring about 14 points lower on items proposed by its own family. The headline blind spots also appear on official test-suite items that no panel model authored.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The metadata-supplied abstract is cut off mid-sentence ('tool delegation reaches ~100%: co'), but the complete abstract appears at the start of the embedded full text, so there is no impact on this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.