Symbolic Temporal Supervision of LLM Agents Using Contracts
cs.AI · cs.LO
Quick verdict
Compiles LTLf contracts into DFAs that gate LLM-agent tool calls online and grade traces offline; strong results, mild contract-benchmark circularity concern.
Summary
The paper introduces ContrAgent, a framework that supervises tool-calling LLM agents using assume-guarantee contracts written in arithmetic LTL over finite traces (ALTLf) over a fixed vocabulary of interaction predicates (Call, ArgHas, Cnt, numeric thresholds). Each contract's assumption and guarantee compile to deterministic finite automata; at runtime the automata advance on every tool-call event, serving two roles: online enforcement (blocking/redirecting/escalating at the first violating call) and offline evaluation of recorded traces with three-valued per-contract verdicts. The compiled contract library is model-agnostic and checked for consistency and conflicts via unsat cores. Evaluation on four benchmarks: on SOPBench, enforcement raises safety from 32% to 98% while keeping success at base level (~90%) with +0.135s median overhead; on AgentDojo it lowers prompt-injection ASR from 47.7% to 11.1% (0.79% with contracts mined from attack traces) at 0.16ms per call; on R-Judge it reaches 91.8% F1 (GPT-4o judge: 74.4%); on tau2-bench it exposes procedural violations behind outcome success (Claude 3.7: pass4 25-60% but joint4 0%). Main limitations: supervision covers only what tool-call structure exposes, contract libraries are derived from the benchmarks' own policy documents or attack traces, and automatic policy autoformalization is left to future work.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A single compiled contract artifact (DFA checkers over ALTLf) can both shield an agent's tool calls online and deterministically grade recorded traces offline with per-contract attribution (Secs. 1, 5.2).
- On SOPBench, contract enforcement raises safety from 32% to 98% while preserving base-level success (~90%) at +0.135s median overhead, unlike prompting or LLM guarding which over-block (Table 2).
- ContrAgent reduces AgentDojo prompt-injection ASR from 47.7% to 11.1% without a detector and to 0.79% with trace-learned contracts, at 0.16ms per call versus 50-500ms for model-based defenses (Table 3).
- On R-Judge, contract-based offline evaluation reaches 91.8% F1, exceeding GPT-4o judging (74.4%) and the reported human baseline (~89%) (Sec. 6.3).
- Outcome-only metrics conceal pervasive procedural violations: Claude 3.7 achieves 25-60% pass4 on tau2-bench but 0% joint4 in all three domains (Sec. 6.4, Table 9).
Strengths
Unified dual-role artifact fills a genuine gap
The same compiled DFA checkers serve online gating (map sigma, Problem 1) and offline trace scoring (map mu) with per-contract violation attribution naming the contract and triggering event (Sec. 5.2, Fig. 3), bridging post-hoc stochastic LLM judges and per-call-only guardrails that the paper correctly identifies as disjoint (Sec. 1, Sec. 2).
Strong quantitative results across four benchmarks
SOPBench safety 32%->98% with success held at ~90% (Table 2); AgentDojo ASR 47.7%->11.1% no-detector and 0.79% trace-learned at 0.16ms/call vs ~50-500ms baselines (Table 3); R-Judge F1 91.8% vs GPT-4o's 74.4% and ~89% human baseline (Sec. 6.3); tau2-bench proc-clean/joint gap exposed per model (Table 9).
Determinism demonstrably robust to backbone degradation
Trajectory-level expressiveness demonstrated, not just asserted
Engineering discipline around specification quality
Weaknesses
Trace-learned AgentDojo contracts are fit to the attacks being evaluated
Supervisor enjoys oracle access to benchmark specifications
Self-referential violation ground truth in tau2-bench
Coverage gap quantified but unresolved
No significance testing and mixed baseline provenance
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ContrAgent?
ContrAgent is a framework from arXiv paper 2609.18128 that supervises tool-calling LLM agents using assume-guarantee contracts expressed in linear temporal logic over finite traces. Each contract compiles into a deterministic finite automaton that checks the agent's sequence of tool calls. The same compiled artifact plays two roles: it blocks unsafe tool calls in real time while the agent runs, and it deterministically scores recorded agent traces afterwards, always attributing any verdict to a specific contract and event.
How does ContrAgent compare to LLM-as-judge safety evaluators and guardrails?
Unlike LLM judges, whose verdicts are stochastic and hard to reproduce, ContrAgent is fully deterministic: the same trace always yields the same verdict. On the paper's benchmarks it matches or beats strong baselines — SOPBench safety rose from 32% to 98% without hurting task success, AgentDojo prompt-injection attack success dropped from 47.7% to 11.1% (0.79% with contracts learned from attack traces) at 0.16ms per call versus 50-500ms for classifier or LLM-filter defenses, and R-Judge detection F1 was 91.8% versus GPT-4o's 74.4%.
What are the main limitations of ContrAgent?
It can only enforce properties visible in the tool-call structure, so free-form semantic harms (e.g., a plausible but wrong text answer) need separate content-level checks. The paper's contract libraries were built from each benchmark's own policy documents or from recorded attack traces, which is more favorable than real-world policy authoring; fully automatic translation of natural-language policies into contracts is left as future work. It also trusts the framework's own event-reporting channel, which an adversarial environment could in principle subvert.
Is the ContrAgent code and data available?
Yes. The paper links a GitHub repository (github.com/yfxiao16/ContrAgent) and evaluates exclusively on public benchmarks (SOPBench, AgentDojo, R-Judge, tau2-bench). Appendices provide exact per-domain and per-model numbers, per-bin scaling curves, a latency breakdown, and a grounding ablation, which makes independent reproduction comparatively straightforward.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 4-6 are only available as captions plus surrounding prose; the review relies on the corresponding appendix tables (7, 9) that were included in the extraction.
- Several baseline numbers (AgentDojo defenses, R-Judge GPT-4o, tau2-bench native metrics) are quoted from published runs in the paper and could not be independently verified from the text.
- The paper cites multiple 2026-dated venues and preprints whose publication status cannot be checked from the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.