WorkWorlds: An Infrastructure for Evaluating AI Agents on Workplace Tasks
cs.AI
Quick verdict
Org-state-first agent eval infra; 192 matched runs show task curation lifts pass 8.7pp via evidence access; rigorous but single synthetic world, proprietary.
Summary
WorkWorlds is an evaluation infrastructure for workplace AI agents that separates organizational state from task specification: a frozen company revision, date, and employee seat first materialize the full role-visible workspace (775 files per seat in the PharmaCo pharmaceutical world), and tasks are added only afterward. In a matched experiment spanning 8 tasks, 6 seats, and 3 frontier agents (Claude Opus 5, Grok 4.6, GPT-5.6 Sol), 192 runs and 1,728 criterion executions compared the full projection against task-curated packets (median 9 sources, about 1% of files) selected by two blinded domain practitioners. Curation raised sufficient-evidence access by 17.6 points (72.8% to 90.4%) and criterion pass by 8.7 points (68.0% to 76.7%, 95% CI 1.0–15.6), while pass given access stayed nearly unchanged (84.2% vs 82.7%); most of the gap arose before agents reached sufficient evidence. Construction checks (SHA-256 rebuild identity, seeded violations) and blinded grader audits (κ=0.85) support validity. Main limitations: results come from one synthetic, proprietary world at a single date; the pass confidence interval barely excludes zero; task-level effects are heterogeneous with one reversal; and neither the measured corpus nor the production code is released.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Task-first benchmark construction encodes task information into the evaluation environment and pre-completes part of the information-localization work that workplace performance normally requires.
- WorkWorlds makes the workplace the reusable unit of evaluation: a frozen revision, date, and employee seat deterministically materialize a role-visible projection, with rubric and held-out evidence sealed on the verifier side.
- In matched PharmaCo runs (192 runs, 1,728 criterion executions), task-level curation raised sufficient-evidence access by 17.6 points (72.8%→90.4%) and criterion pass by 8.7 points (68.0%→76.7%), while pass conditional on access stayed nearly unchanged, so the measured difference arises before sufficient evidence is reached.
- Because two independent expert curators chose different source sets for 5 of 8 tasks and produced different pass rates, task-level context selection is itself an evaluation decision that most benchmarks leave implicit.
Strengths
Clean identification of the curation effect
The matched experiment (§4, Fig. 2) varies only the information projection while world, task, seat, agent, grader, and compute budget stay fixed; runs were interleaved, and per-task-agent cells were collected in blocks, making the +8.7 pp pass and +17.6 pp access contrasts causally interpretable within this world.
Construction validation is enforced mechanically, not procedurally
SHA-256 digest manifests, pinned rendering, staged-workspace verification, sealed candidate/verifier packages, and replayable append-only event histories (§3.2, App. A/D) make the ordering claims mechanically enforced: 3/3 identical rebuilds, 18/18 identical seat materializations, 6/6 distinct seat fingerprints, 4/4 seeded violations caught (Table D1).
Statistical treatment exceeds benchmark norms
Multi-layered human audits
The curator comparison turns a nuisance into a finding
Weaknesses
External validity limited to one near-maximal contrast in one world
Core environment is proprietary and unverifiable
The validation-defense anomaly is acknowledged but unexplained
No human baseline; absolute scores are grader-protocol-bound
Agent-level comparisons are harness-confounded
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
WorkWorlds is positioned as an infrastructure contribution: an organizational world (employees, permissions, files, records, event history) is frozen as a revision; a date and employee seat then materialize the full role-visible projection (775 files per measured MSAT/QC seat in PharmaCo); only afterward is a task overlaid, with rubric and held-out evidence sealed on the verifier side. The authors claim (1) task-first benchmark construction (they cite GDPval, FORTE) leaks task information into the environment and pre-completes information localization; (2) the construction contract makes cross-task workplace identity directly checkable; (3) in a matched experiment, curation raises evidence access +17.6 pp and criterion pass +8.7 pp while pass-given-access stays flat, locating the gap before sufficient evidence is reached. Claims are mostly well-scoped to the PharmaCo panel (8 tasks, 6 seats, 72 criteria, 3 agents, one revision and date), though the abstract's framing invites broader generalization than a single-world design licenses.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is WorkWorlds?
WorkWorlds is an AI-agent evaluation infrastructure that builds the workplace before the task. A frozen company revision, a date, and an employee seat first materialize everything that role can see (e.g., 775 files per seat in its synthetic pharmaceutical company PharmaCo); the task is added only afterward, and rubrics plus held-out evidence are sealed on the grader side. This prevents benchmark tasks from pre-selecting their own context.
How much does giving an AI agent only the relevant files improve its benchmark scores?
In the paper's matched experiment (8 tasks, 3 models, 192 runs, 1,728 criterion executions in a synthetic pharma company), giving agents curated task-relevant packets instead of the full 775-file workspace raised evidence access from 72.8% to 90.4% (+17.6 points) and criterion pass from 68.0% to 76.7% (+8.7 points). Performance after reaching sufficient evidence was nearly unchanged, so the gain came almost entirely from easier information finding.
Is the WorkWorlds benchmark or dataset publicly available?
Partially. The GitHub repo (agent-evalscience/workworld) contains a simplified illustrative implementation, de-identified result ledgers, and statistical analysis code. However, the measured PharmaCo corpus, the production infrastructure, and raw agent trajectories are not released, so the reported experiment cannot be re-run independently.
Which AI models were evaluated in the WorkWorlds paper?
Three frontier models: Claude Opus 5, Grok 4.6, and GPT-5.6 Sol. All three passed more criteria with curated packets than with the full workspace (+6.2, +12.8, and +6.9 percentage points respectively), with rank order preserved across conditions.
Do curated context packets make agent benchmarks easier, and by how much?
In a matched experiment within the synthetic pharmaceutical world PharmaCo, task-level curation raised sufficient-evidence access from 72.8% to 90.4% (+17.6 points) and criterion pass from 68.0% to 76.7% (+8.7 points, 95% CI 1.0–15.6) across 192 runs and 1,728 criterion executions. Pass conditional on access stayed nearly unchanged (84.2% vs 82.7%), meaning most of the difference arose before agents reached sufficient evidence. Limitations: results come from one synthetic proprietary world at a single date, the pass confidence interval barely excludes zero, effects are task-heterogeneous with one reversal, and neither the measured corpus nor the production code is released.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 are referenced but not analyzable from the extracted text; all cited numbers come from the prose and Tables 1, D1, D2, E1–E3, G1, which are included in the extraction.
- This is arXiv v2 (23 Sep 2026), a non-peer-reviewed preprint.
- The released artifact's contents (de-identified ledgers, analysis code, reference world) are asserted by the paper and were not independently verified.
- Appendix G's comparison of related benchmarks is partially characterized by the authors; the related-work positioning was not independently checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.