ContractEval: Query-Conditioned Execution Matching for Procedural Instruction Conformance
Praphul Singh, Shanu Kumar, Akshat Agarwal, Ganesh Kumar
cs.AI
Quick verdict
Obligation-graph matching makes LLM procedural conformance auditable; gold-graph 1.000 is by construction, but the judge-failure gap looks real.
Summary
ContractEval is a diagnostic framework that evaluates whether an LLM's response or an agent trace conforms to the procedural obligations an instruction artifact (prompt, SOP, policy) makes active for a specific query. The method compiles the artifact into a declared hierarchical task network, prunes it into a query-conditioned expected graph, extracts an observed execution graph from the response or trace with an LLM that is blind to the answer key, and aligns the two graphs via maximum-weight bipartite matching, reporting six metrics (required node recall, observed precision, edge conformance, branch correctness, invariant and output scores). On a self-authored suite of 10 SOP contracts, 200 clean executions, and 1,400 paired cases across six perturbation families plus a clean control, output-only LLM judges detect on average only 0.495 of injected failures and trace-aware judges 0.570 (the best trace-aware judge, GPT-5.4, still misses 29.0%); gold-graph ContractEval scores 1.000 across detection and localization as a construct-validity check, and LLM-backed extraction retains 0.944-1.000 detection with localization 0.660-0.916. Main limitations: contracts and audits are author-produced with only a 2-contract re-audit, perturbations inject one failure at a time into the same representation the metric scores, no significance or run-variance reporting, and matching uses fixed token overlap.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Output-only and trace-aware LLM judges miss a large share of injected structural failures (average detection 0.495 and 0.570; the best trace-aware judge, GPT-5.4, still misses 29.0% of perturbed cases).
- Under gold expected and observed graphs, ContractEval detects and localizes all controlled perturbations, which the authors frame as construct validity of the metric family.
- LLM-backed observed-graph extraction preserves most of the signal (detection 0.944-1.000; localization 0.660-0.916), with extractor calibration - especially clean false positives - as the scaling bottleneck.
- Static declared-HTN compilation is the least automatable pipeline component (GED similarity 0.242-0.409), while query-conditioned expected-subgraph extraction is near-saturated (node F1 0.965-0.986).
Strengths
Explicit denominator framing
Section 3 makes query-active obligations the evaluation target, cleanly separating what an artifact declares from what a query activates - a sharper formulation than holistic judging, and the basis for falsifiable, localizable failure reports.
Paired hidden-failure protocol
Section 5 holds contract, query, and base evidence fixed while injecting one known failure per case; the metric-to-family decision rule (Appendix A) is fixed before evaluation with no per-model tuning, a disciplined validity design.
Honest staging of claims
Component study locates the automation bottleneck
Disaggregated error analysis
Weaknesses
Oracle result is near-tautological
Small, self-authored benchmark
No statistical uncertainty
Brittle lexical matching
Extractor false alarms underexplored
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ContractEval (arXiv 2609.09458; Oracle Health AI + MBZUAI) argues that LLM procedural failures - omitted checks, wrong branches, reversed dependencies, extra actions, invariant and output-contract breaches - should be evaluated against the obligations an instruction artifact makes active for a given query, not against answer plausibility or raw trace activity. Central claims: (1) output-only and trace-aware LLM judges miss many injected structural failures (average detection 0.495 and 0.570; best trace-aware judge GPT-5.4 misses 29.0%); (2) under gold expected and observed graphs the framework detects and localizes all controlled perturbations; (3) LLM-backed observed-graph extraction preserves much of the signal (detection 0.944-1.000, localization 0.660-0.916, Table 10), making extractor calibration the bottleneck. The authors carefully frame the gold-graph result as construct validity, not automation, and disclaim compliance-guarantee status.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ContractEval?
A diagnostic framework that checks whether an LLM's response or an agent trace satisfied the procedural obligations an instruction (prompt, SOP, policy) makes active for the given query. It builds a query-conditioned expected graph of obligations and matches it against an observed graph extracted from the response or trace, reporting omissions, ordering errors, wrong branches, extra actions, invariant breaches, and output-contract violations as separate failure types.
Why do LLM judges miss procedural failures?
Because they judge whether an answer or trace looks plausible without an explicit denominator of which obligations applied. In the paper's controlled suite, output-only judges detected on average only 0.495 of injected structural failures and trace-aware judges 0.570; even the best trace-aware judge (GPT-5.4) missed 29.0%, mainly omissions and order swaps that leave the final answer intact.
Is ContractEval fully automated or a compliance guarantee?
No. The perfect 1.000 detection/localization results use gold audited graphs and are a construct-validity check. With LLM-extracted observed graphs, detection stays high (0.944-1.000) but localization drops to 0.660-0.916 and clean false positives appear (up to 0.395 for the weakest extractor). The authors explicitly state it is not a compliance guarantee.
What data is the benchmark built on?
A self-authored suite of 10 SOP-style contracts (customer service, aircraft inspection, patient intake, referral-abuse and traffic-spoofing detection, etc.) with 20 queries each, 200 clean executions, and six perturbation families plus a clean control, totaling 1,400 evaluation cases. The authors state they will release the framework, contracts, prompts, and benchmark artifacts.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 appear only as caption text in the extraction; diagram-internal details could not be verified against the visuals, though all cited numbers come from extracted tables and body text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.