CoArena: Evaluating Computer-Use and Multi-Agent Systems in Real Time
cs.LG
Quick verdict
Defines real-time agent evaluation via five measurable properties and full Bradley-Terry machinery; careful math, but no deployed-system measurements.
Summary
The paper specifies CoArena, an arena in which real users submit computer-use tasks; two entrants (single models or multi-agent pipelines behind one fixed tool interface) execute the same task concurrently in identical sandboxed desktops; blind judges pick the better outcome; and a public leaderboard is periodically refit from the votes. Its central contribution is a formal definition of 'real-time' evaluation as five measurable properties — continuous task arrival, live concurrent execution, online rating updates, freshness with contamination resistance, and bounded feedback latency from failed runs to reusable training environments — each with equations, worked examples, and time-horizon tiers (Table 1). The rating methodology is complete: ridge-penalized (r=10^-3) Bradley-Terry maximum likelihood, covariance projection for the unidentified shift, judge-clustered sandwich intervals, seeded bootstrap rank bands, and a streaming Elo update derived as a first-order stochastic-gradient step on the same likelihood. A five-agent, 211-vote worked example (Table 2) carries the pipeline from vote matrix to ratings (A: 1140, CI 1073-1206) and rank bands; two seeded simulations illustrate convergence and throughput. Critically, the authors state that no number measures a deployed system — the duplicate rate is literally '[to be measured]' — so the work is a verifiable specification whose real-world effectiveness remains unvalidated.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Static computer-use benchmarks have three structural weaknesses — task-set aging, leakage into training corpora, and author-encoded success checkers — which a live human-preference arena addresses by measuring use directly.
- 'Real-time evaluation' is precisely definable as five measurable properties (continuous arrival, live concurrent execution, online rating updates, freshness with contamination resistance, bounded feedback latency), each gradeable at time-horizon tiers.
- A penalized Bradley-Terry batch refit with judge-clustered sandwich intervals and seeded bootstrap rank bands — plus a streaming Elo update that is a first-order step on the same likelihood — yields honest published rating uncertainty.
- A failed agent run can be converted into a reusable, validated training environment within a bounded latency via a capture-redact-synthesize-validate-publish pipeline.
Strengths
Statistically careful rating machinery
Ridge penalty (r=10^-3) guarantees estimator existence where Ford's condition fails (§5.3); the projection P = I - 11'/m removes the unidentified shift whose penalized variance would be a spurious 1/(mr)=200 in theta units at m=5 (§5.6); judge-clustered sandwich with an explicit publication gate when G<=m; per-agent seeded bootstrap so adding an agent cannot perturb others' rank bands (Algorithm 3).
Radical transparency about non-empiricism
The 'note on numbers' (§1) states every figure is derived from stated inputs or seeded simulation and none measures a deployed system; the duplicate rate is written as D(0.55) = '[to be measured]' (§4.5) and model-card metrics are '[measured]' placeholders (Fig 8) rather than fabricated.
Fully checkable arithmetic
Honest separation of machine vs human latency
The five-property definition is a usable vocabulary
Weaknesses
Zero empirical validation
Cluster-robust machinery never exercised
Contamination argument is partially circular and thin
Undisclosed commercial conflict and thin governance detail
Unvalidated trust heuristics and tie approximation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CoArena (arXiv 2609.14239, September 2026) is a system-design and formalization paper from the makers of CoArena.ai. The proposed arena works as follows: real users submit computer-use tasks; two entrants — single models or multi-agent pipelines behind one fixed tool interface — execute the task concurrently in identical sandboxed desktops; blind judges pick the better outcome; a public leaderboard is periodically refit from the votes. The paper makes four claims: (1) static benchmarks age, leak into training corpora, and encode author-defined checkers (§1); (2) 'real-time' evaluation is exactly five measurable properties with tier bounds (§4, Definition 4.1, Table 1); (3) a penalized Bradley-Terry refit with judge-clustered intervals and bootstrap rank bands, plus a streaming Elo update shown to be a first-order step on the same likelihood, publishes honest uncertainty (§5); (4) failed runs convert to validated reusable training environments within bounded latency (§4.6, Algorithm 4). Crucially, the authors disclaim all empirical content: 'none is a measurement of a deployed system.'
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CoArena and how does it evaluate computer-use agents?
CoArena is an arena where real users submit desktop/browser tasks; two AI entrants (a single model or a multi-agent pipeline behind the same tool interface) execute each task at the same time in identical sandboxed desktops; human judges see both anonymized trajectories and pick the better result (or declare a tie, both-unacceptable, or abstain); a public leaderboard is refit from these blind votes using a Bradley-Terry model with Elo-scaled ratings, confidence intervals, and rank bands.
Does the CoArena paper include real experimental results?
No. The authors state explicitly that every number is either derived from stated illustrative inputs or produced by stated simulations with given seeds — none is a measurement of the deployed CoArena.ai system. The five-agent, 211-vote example (Table 2) and both simulations are constructed examples, and the duplicate rate is left as '[to be measured]'.
How is CoArena different from Chatbot Arena?
Chatbot Arena compares text responses, one generation per comparison. CoArena compares complete screen-action trajectories executed in isolated virtual desktops: each comparison costs minutes of sandboxed execution rather than a single generation, which drives its concurrency, queueing, freshness, and failure-to-training-environment analysis.
What does 'real-time' mean in this paper?
Five measurable properties, each with an equation and a time-horizon tier: continuous task arrival (no release epochs), live concurrent execution (identical inputs, simultaneous lanes), online rating updates (bounded staleness of the published leaderboard), freshness with contamination resistance (tasks executed minutes after being written, before any training cutoff could have seen them), and bounded feedback latency (a failed run becomes a validated, reusable training environment within a stated horizon).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-9 are available only as text descriptions in the extraction; visual content was not independently inspectable.
- The paper itself contains no empirical measurements, so any judgment about real-world performance is necessarily about design intent, not results.
- Numerical checks in this review were verified by hand recomputation from stated inputs (Table 2, §6.4, §C.6), not by executing code.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.