DAREBench: Deployment-Aware and Reliable Evaluation of Models as Agents
Yu Liu, Zhilin Liu, Zhiwei Yang, Shaojie Zhang, Zheyuan Deng, Tingwei Huang, Zhenbo Luo, Lei Jiang, et al.
cs.AI
Quick verdict
Agent benchmark unifying 233 tasks and 35 models in a shared OpenClaw runtime with evidence-audited scoring; credible, but single-run results lack uncertainty.
Summary
DAREBench is an agent benchmark that evaluates LLMs as general-purpose agents under a shared OpenClaw runtime. It curates 233 tasks from 22 public benchmarks via a three-stage filter (feasibility, difficulty, representativeness) and organizes them into a 2×3 workload matrix crossing input modality (text/multimodal) and execution form (single-step, multi-step, multi-step+tools). Tasks are formalized as contracts (instruction, initial workspace, tools, artifact predicates, scorer, time budget) and scored by automated checks, an LLM judge (Qwen3.5-VL-Plus), or hybrids, with an evidence-based audit: deterministic pre-filters flag suspicious positives for a meta-judge (Claude-Opus-4.6), which removed 143 unsupported positive scores from 3,340 judge/hybrid calls. The study evaluates 23 API and 12 locally served open-weight models over 7,587 model–task runs, reporting accuracy, token usage, and reference costs. Key findings: rankings are workload-dependent (Claude-Opus-4.8 leads Text-SS/Text-MT/MM-MT; GPT-5.5 Text-MS/MM-SS; Gemini-3.1-Pro MM-MS); text and multimodal accuracy–cost frontiers differ; the best local model, Qwen3.6-27B (Overall 59.8), trails frontier APIs by roughly 15 points. Main limitations: one run per model–task with no confidence intervals or significance tests despite sub-point frontier claims; workload labels assigned by source benchmark rather than observed behavior; possible train–test contamination from public source tasks is not discussed; the meta-judge is itself an evaluated model.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- No single model dominates all six workload groups; leaders differ by group (Claude-Opus-4.8 on Text-SS/Text-MT/MM-MT, GPT-5.5 on Text-MS/MM-SS, Gemini-3.1-Pro on MM-MS).
- Text and multimodal workloads exhibit distinct accuracy–cost Pareto frontiers, with the sub-$0.02 text range already near saturation.
- 143 of 3,340 judge/hybrid positive scores lacked execution evidence and were removed, and hallucination patterns split by deployment mode (deadlock loops and pseudo-tool-calls exclusively local; timeout-with-credit skewed to API).
- Locally deployed open-weight models (best: Qwen3.6-27B, Overall 59.8) are competitive in several groups but trail frontier commercial APIs by roughly 15 percentage points under a single-GPU 32K profile.
- The evidence-based audit brings automated scores closer to human experts (MAE 0.049 vs 0.098 on 100 audited trajectories).
Strengths
Shared-runtime design removes scaffold confounds
All 35 models run the same OpenClaw agent loop, identical tool/skill suite and permission boundaries (§A.2), directly addressing the attribution problem the paper raises; the task-contract formalization (Eqs. 1–3) makes artifact requirements mechanically checkable rather than inferred from final text.
Evidence-based audit with human validation
Five judge-hallucination patterns are characterized (Table 3: deadlock loop 86, rubric leakage 26, pseudo-tool-call 20, output instability 7, timeout-with-credit 4); deterministic pre-filters flagged 182 of 3,340 judge/hybrid calls and the meta-judge removed 143; on 100 expert-audited trajectories the corrected scores reach MAE 0.049 vs 0.098 for the raw judge (§C).
Deployment-profile analysis is genuinely informative
Internally consistent numbers and cache-aware cost accounting
Honest scope statements
Weaknesses
No uncertainty quantification
Judge conflicts of interest unexamined
Contamination not discussed
Workload labels are provenance-based, not behavioral
Audit zeroing lacks sensitivity analysis; minor inconsistencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
DAREBench (arXiv 2609.06059) is a benchmark paper for evaluating LLMs as agents. It re-packages 233 tasks from 22 public benchmarks (GPQA, SimpleQA, Terminal-Bench, OSWorld, HLE, CharXiv, MMSearch, etc.; §F.3, Table 9) into a 2×3 workload matrix — {text, multimodal} × {single-step, multi-step, multi-step+tools}, with group sizes 20/87/55/20/26/25 — and runs everything in one shared OpenClaw environment so scaffold differences cannot confound model comparisons. Tasks are formalized as contracts (instruction, initial workspace, tool subset, artifact predicates, scorer, time budget; Eqs. 1–3). Central claims: (1) rankings are workload-dependent — no model dominates all six groups; (2) text and multimodal workloads have distinct accuracy–cost frontiers; (3) 143 of 3,340 LLM-judge/hybrid positive scores were unsupported by execution evidence; (4) local open-weight models trail frontier APIs by ~15 points under a realistic single-GPU profile.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DAREBench?
DAREBench is a benchmark for evaluating LLMs as agents: 233 tasks curated from 22 public benchmarks, all executed in one shared OpenClaw agent environment and organized into a 2×3 workload matrix by input modality (text/multimodal) and execution form (single-step, multi-step, multi-step+tools). It scores tasks via automated checks, LLM judging, or hybrids, and audits judge scores against execution evidence.
Which model performs best on DAREBench?
It depends on the workload group: Claude-Opus-4.8 leads Text-Single-Step (98.8), Text-Multi+Tools (65.3) and MM-Multi+Tools (61.8); GPT-5.5 leads Text-Multi-Step (83.6) and MM-Single-Step (90.0); Gemini-3.1-Pro leads MM-Multi-Step (88.5). The strongest locally deployed open-weight model is Qwen3.6-27B (59.8 overall), roughly 15 points behind frontier APIs.
How does DAREBench make LLM-judge scores trustworthy?
Deterministic post-hoc rules flag judge/hybrid positive scores that lack execution evidence (e.g., empty outputs, no file writes, tool-like text that was never executed, timed-out runs). A meta-judge from a different model family reviews flagged cases; 143 of 3,340 such scores were zeroed, and on 100 human-audited trajectories the corrected scores' mean absolute error is 0.049 versus 0.098 for the raw judge.
Is DAREBench code and data available?
The paper points to a public GitHub repository (github.com/SeerRay-Lab/DAREBench), and its appendix reproduces judge/meta-judge prompts, per-model pricing, and per-benchmark task lists. Fully replicating all 7,587 runs would require paid access to 23 commercial APIs plus a 96GB GPU for the 12 locally served models.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–7 appear in the extraction only as text fragments (captions, axis labels, trace listings); visual content such as exact Pareto curves or case-study screenshots could not be inspected.
- The supplied metadata author list omits Yanbing Liu and Pei Fu, who appear in the extracted title block; the extracted text was treated as authoritative for authorship.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.