Ajar: Measuring Open Privilege in Agent Defenses
Reshabh K Sharma, Linxi Jiang, Shuo Chen, Zhiqiang Lin
cs.CR · cs.AI · cs.SE
Quick verdict
Ajar adds a third axis—harm-weighted open privilege left by agent defenses—to AgentDojo; convincing and honest, but 73% of oracle labels are self-asserted.
Summary
Ajar is a measurement tool that evaluates LLM-agent defenses on a third axis the authors argue current benchmarks miss: the privilege a defense leaves open on paths no benchmark run ever took. Attached to AgentDojo v1.2.2 (97 benign tasks), it generates 10,471 labeled candidate tool calls at decision points along each task's reference solution, labels each as required or excess via a construction-based oracle, and queries five defenses (Progent, CaMeL, AC4A, Permission Assistant, Claude Code's Auto mode) through a uniform decide interface under two deciding models (Sonnet-5, Haiku-4.5). It reports harm-weighted over-privilege leakage (OPL) beside sufficiency: measured defenses fall between the tool-name allowlist (0.3551) and the argument-exact oracle (0.0020), with Permission Assistant tightest (0.0810 under Sonnet-5, dropping to 0.0217 with a personalized history) and Claude Code's Auto mode loosest (0.3009). OPL proves unrecoverable from attack success or benign utility, and sufficiency does not predict task cost. Robustness checks cover six harm weightings (Kendall's τ ≥ 0.67), test-budget subsampling, prompt paraphrasing, and run-to-run variance (3 runs; SD up to 0.0203). Main limitations, largely self-acknowledged: only 27% of oracle labels are validated by AgentDojo's own predicates (73% rest on the authors' construction), no own attack-success measurements (the benchmark's attacks fail even undefended), confidence intervals for adjacent defenses overlap, and Claude Code's Auto mode is scored on a smaller, incomparable agent-issued subset.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
6 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Attack success rate and benign utility are computed on executed runs and therefore cannot measure the privilege a defense leaves open on unexecuted paths; Ajar measures it directly by reusing an existing benchmark's tasks, schemas, reference solutions and goal states.
- Every measured defense's over-privilege leakage lies between the tool-name allowlist (0.3551) and the argument-exact oracle (0.0020), with the tightest defenses still leaking over 40 times the oracle's rate.
- Over-privilege leakage is not recoverable from attack success or benign utility: Progent and AC4A under Sonnet-5 differ by less than 0.009 in OPL yet 37 points of defended benign-task completion, and the tighter one completes more tasks.
- A defense's OPL is a property of its whole decision procedure: it moves with the deciding model and run-to-run model variance, but the defense ordering survives changes to harm weighting, test budget and task wording.
Strengths
Genuinely new measurement target
Shifts the unit of analysis from the agent model (FORTIS, ToolPrivBench, GrantBox, §8) to the enforcement layer, scoring privilege left open on paths no run took — a property ASR and utility provably cannot express (§1, §3).
Construction-label coupling avoids the oracle problem
The rule that builds each candidate also fixes its ALLOW/DENY label, with a fixed precedence for family collisions fixed a priori (§4.2, §4.4); labels are checked against the host's required-tool set and utility predicate, not post-hoc judgment.
Exemplary label-provenance honesty
Robustness section goes beyond token checks
Correct uncertainty methodology
Informative baselines
Weaknesses
Metric validity rests on self-asserted labels
Fine-grained ordering is statistically unsupported
No demonstrated attack-prevention linkage
Claude Code Auto mode is not an apples-to-apples row
Single host benchmark
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Ajar targets an evaluation gap: agent-security benchmarks (AgentDojo, InjecAgent, Agent Security Bench) judge defenses only on attack success rate and benign utility, both read off runs that actually happened. A defense can look clean while standing ready to allow a funds transfer no task needed. The paper claims this open privilege is a measurable third axis. Ajar attaches to AgentDojo v1.2.2, reuses its 97 benign tasks, tool schemas, reference solutions and utility predicates, and queries five defenses — Progent, CaMeL, AC4A, Permission Assistant, and Claude Code's Auto mode — through one decide interface with candidate calls the runs never made. Two scores result: harm-weighted over-privilege leakage (OPL) and sufficiency, plus over-restriction and an attack-admission count. The claims are carefully scoped and mostly delivered; the weakest link is that 73% of oracle labels (7649/10471, §7.1) rest on the authors' construction rather than benchmark validation.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 6 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is over-privilege leakage in LLM agent defenses?
It is the harm-weighted share of tool calls a task does not need that a defense would nonetheless allow. The Ajar tool generates candidate calls from AgentDojo's benign tasks — calls the benchmark runs never made — presents them to a defense, and scores how many excess ones pass, weighted by a rubric from reversible reads (weight 1) up to irreversible third-party or funds actions (weight 27).
Which agent defense is least over-privileged according to the paper?
On AgentDojo with Sonnet-5 as the deciding model, Permission Assistant is tightest among the stock configurations (OPL 0.0810, dropping to 0.0217 when given a personalized permission history), followed by CaMeL (0.0895), Progent (0.1026) and AC4A (0.1113); Claude Code's Auto mode is loosest at 0.3009. However, the bootstrap confidence intervals of the four academic defenses overlap, so the fine-grained ranking is not statistically settled.
Does Ajar show agent defenses failing real prompt-injection attacks?
No. Ajar reports no attack success rates of its own: AgentDojo's important-instructions attack did not succeed even against the undefended agent in their setup. Instead it reports 'attack admission' — whether a defense grants the sink calls that the benchmark's shipped injections aim at — as a proxy connecting open privilege to attack risk (e.g., 25/609 admissions for Permission Assistant vs 74/581 for Claude Code's Auto mode).
Is Ajar's code and data available for reproduction?
The paper points to https://github.com/reSHARMA/Ajar. The generator, oracle and scoring depend only on the Python standard library and attach to the public AgentDojo v1.2.2, so test generation and the deterministic baselines are re-runnable locally. Re-scoring the five defenses requires API access to Sonnet-5/Haiku-4.5 at 10,471-test scale, and Claude Code's Auto mode can only be measured inside Claude Code itself.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are extracted as fragmentary text; only their captions and in-text descriptions were used, so plot-level details (e.g., Fig. 2 frontier shape) were not analyzed visually.
- Tables were extracted as flattened text; row/column alignment was reconstructed from captions and cross-checked against surrounding prose, but a few cell-level readings (e.g., Table 4's variant columns) rely on that reconstruction.
- The arXiv ID (2609.26900), the evaluated models (Sonnet-5, Haiku-4.5) and several cited works are dated 2026; this review relies solely on the provided text and no external verification was performed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.