Policy as Code: A Coroutine-Bridge Harness for Fast-Reasoning Reliability on CAR-bench
cs.AI · cs.CL
Quick verdict
Competition report: a coroutine-bridge harness wins CAR-bench Track 2 at 60.0% Pass^3 via policy-as-code; n=30 hidden set limits statistical strength.
Summary
The paper presents a harness for CAR-bench, a tool-using agent benchmark where the evaluator executes all tools and each tool-result exchange is a separate agent round-trip. Instead of a conventional next-action agent that spends one model call per round of results, the model's only action is to emit a Python program; a coroutine bridge suspends the program at each tool call, ships the official call to the evaluator, and resumes the same stack frame when results arrive, so one model invocation drives many sequential and parallel tool round-trips (median 2 model calls vs 7 agent turns per task; Table 1). Deterministic domain policies are compiled into guarded tool-layer implementations rather than prompt rules, enforcing compliance without reasoning cost and boosting trial-to-trial consistency. On the 30-task hidden set the harness won Track 2 (Cerebras gpt-oss-120b) with 60.0% Pass^3 versus 50.0% for the runner-up and 13.33% for the organizer baseline, at the lowest estimated cost and fastest median latency (3.14 s); the identical harness scored 60.0% Pass^3 on GPT-5.5 in the Open track. On the 125-task public split it reports 93.3% Pass^1 and 87.3% Pass^3. Limitations: 40% of hidden tasks still fail at least one trial, Pass@3 trails the runner-up (66.7% vs 76.7%), worker state does not survive restarts, and no significance tests, ablations, or code release are reported for the 90-trial hidden comparison.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 18 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A coroutine-bridge harness decouples model invocations from evaluator tool round-trips, achieving a median of 2 model calls against 7 agent turns per task (Table 1).
- Encoding deterministic CAR-bench policies as guarded tool-layer code enforces compliance at zero reasoning cost and converts single-task success into trial-to-trial consistency, which is what Pass^3 rewards.
- The harness won Track 2 of the official hidden evaluation with 60.0% Pass^3, 4.5× the organizer baseline, at the lowest estimated cost and fastest median task latency (3.14 s) among entries above baseline.
- The unchanged harness reproduced 60.0% Pass^3 on GPT-5.5 in the Open track, trading coverage (higher Pass@3) for consistency (lower conversion).
- A byte-identical static prompt with tail-appended state serves 78% of input tokens from cache (86.6% warm tail), compounding the few-call design into a small fraction of nominal input compute.
Strengths
Architecture with a quantified causal effect
The coroutine bridge (§2) is precisely specified — persistent per-context Python worker, blocking tool wrappers, batch([...]) batching — and its effect is measured, not asserted: Table 1 shows median 2 model calls carrying 7 agent turns, 1.8 s median model latency, p90 of 4 calls; §5 corroborates with 899 s agent LLM time out of 5,351 s wall for 375 evaluations.
The consistency decomposition is the paper's best evidence
Table 2 shows our entry and the runner-up converted an identical 57/90 successful trials, yet scored 60.0% vs 50.0% Pass^3 because success consistency differs (92.5% vs 73.9%). This cleanly isolates the mechanism Pass^3 rewards and links it to policy-as-code enforcement rather than model ability.
Cross-model transfer without modification
Candid limitations and unusual accounting transparency
Development discipline against overfitting
Weaknesses
No inferential statistics on the decisive comparison
No component ablation
Development on the evaluation data
Reproducibility gaps and thin external support
Ambiguous competition framing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a short competition/system report (arXiv, cs.AI/cs.CL) describing the harness that won Track 2 of CAR-bench [Kirmayr et al., 2026], a tool-using agent benchmark in an in-car assistant domain (58 tools, 19 policies, 254 public tasks) whose tools are executed by the evaluator, making every tool-result exchange a separate agent round-trip. Five claims are made: (1) a coroutine-bridge harness lets one model call drive many tool round-trips — median 2 model calls vs 7 agent turns per task (Table 1); (2) deterministic policies are enforced as guarded tool-layer code at zero reasoning cost (§3); (3) the harness won Track 2 with 60.0% Pass^3, 4.5× the organizer baseline, at lowest estimated cost and fastest median latency (3.14 s, Table 2); (4) the unchanged harness scored an identical 60.0% Pass^3 on GPT-5.5 in the Open track; (5) a byte-identical static prompt serves 78% of input tokens from cache (86.6% warm tail), compounding the few-call design.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Pass^3 in CAR-bench?
Pass^3 credits a task only when it passes all three independent evaluation trials, so it measures trial-to-trial consistency rather than best-case ability. The related metric Pass@3 credits a task solved at least once in three tries; a system can score higher on Pass@3 yet lower on Pass^3.
How does the coroutine-bridge harness work?
The LLM's only action is to emit a Python program instead of a single tool call. When the program calls a tool, a bridge suspends it, sends the official tool call to the evaluator, and resumes the exact same stack frame when results return — so one model invocation can drive many sequential and parallel tool round-trips without further model calls.
What results did the harness achieve on CAR-bench?
It won Track 2 on the 30-task hidden set with 60.0% Pass^3 (runner-up 50.0%, organizer baseline 13.33%) at the lowest estimated cost and fastest median latency (3.14 s); the unchanged harness scored 60.0% Pass^3 with GPT-5.5 in the Open track, and on the 125-task public split it reports 93.3% Pass^1 and 87.3% Pass^3.
Is the harness code available?
The paper does not mention a code release. The architecture is described in enough detail to reimplement against the public CAR-bench split, but the frozen prompt, guarded tool implementations, and repair logic are not included, and hidden-set numbers are organizer-scored.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 is present only as caption text in the extracted text; its visual content was not independently analyzed.
- Hidden-set results are organizer-scored and author-reported; they cannot be independently verified from the paper.
- The paper is a short competition-style report; the frozen prompt and guarded tool implementations are absent, limiting method-level review depth.
- Table 2's caption 'first of 13 Track 2 submissions' is ambiguous (rank vs chronology) in the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.