WorldBench: Culturally Grounded Benchmark for Multilingual Agents
Leonardo Ranaldi, Sherrie Shen, Jushi Kai, Alexandra Birch
cs.AI · cs.CL
Quick verdict
Multilingual agent benchmark adds state-preservation metric CTS; top model 49.2%, but single-run results and table inconsistencies weaken its model rankings.
Summary
WorldBench is a benchmark of 1,600 persona-grounded, file-based agentic tasks spanning seven languages and eight locales (EN-US, EN-UK, IT, PT, ES, FR, DE, ZH). Tasks are generated from human-written persona/scenario/constraint seeds, expanded automatically, and audited by annotators born in each target country (25% double-annotated), then balanced to 200 tasks and 50 personas per setting, with 15-20 distractor artefacts per testbed. Unlike translated benchmarks, tasks are constructed natively in each locale, with localized currencies, documents, and conventions. The headline metric, Constrained Task Success (CTS), requires both that all task-specific evaluators pass and that every pre-existing non-target file stays byte-identical. Evaluators are mostly deterministic (containment, cell values, file existence, calendar non-overlap); the two LLM-judged functions are validated against human references (Cohen's kappa 0.76-0.84). Nine agents run once per task: Gemini-3.1-Pro leads at 49.2% CTS, EuroLLM-9B trails at 10.8%, and pass rates exceed CTS by 10-17 points, quantifying collateral edits. English leads and Chinese trails for nearly all models; failure analysis attributes 42.4% of failures to preservation-type errors (distractor capture, shell over-reach), and locale mismatch rises from 6.8% (EN) to 21.4% (non-EN). Main limitations: single run per model-task with no significance testing, internal inconsistencies (Gemini-2.5 models appear in Table 4 and several figures but not the declared nine-model list), one unrendered figure, and non-parallel per-language task sets that confound the language gradient.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Frontier LLM agents reach at most 49.2% CTS on WorldBench, with the weakest at 10.8% (Table 3).
- Pass rate exceeds CTS by 10.1–16.8 points for every model, showing agents routinely complete tasks while corrupting unrelated workspace files.
- A stable language gradient exists — English settings lead and Chinese trails across nearly all models — driven largely by locale-convention mismatch errors, which rise from 6.8% (EN) to 21.4% (non-EN) of failures.
- Constrained Task Success, pairing task-correctness with byte-level preservation of non-target files, is a necessary headline metric for file-based agents and complements trajectory-level metrics.
- Culturally grounded multilingual tasks can be constructed natively (without translation) at 200-task scale per locale via seed-driven generation plus in-country human audit.
Strengths
Preservation-aware headline metric with a crisp definition
CTS (§4.1) = pass(t) ∧ preserve(t), where all pre-existing non-target files must be byte-identical. Table 3 quantifies pass-to-CTS gaps of 10.1 (Gemini-3.1-Pro, 59.3→49.2) to 16.8 (Llama-3.3-70B, 40.9→24.1), exposing collateral edits that correctness-only scoring misses.
Native, audited multilingual construction at scale
§3.5 and Appendix F: 640 human-written seed tasks (20 personas × 4 tasks × 8 settings), automated augmentation, then audit by annotators born in and resident of the target country (25% double-annotated, free-text revision loop), balanced to 200 tasks/50 personas per setting — no translation-based shortcuts.
Judge influence is bounded and validated
Objections pre-empted by ablations
Unusually deep harness documentation
Weaknesses
Single run per model-task, no uncertainty anywhere
Internal model-set inconsistencies suggest mixed result versions
Broken figure in the compiled PDF
Audit workforce partially unreported
Language-gradient confound acknowledged but unquantified
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
WorldBench (arXiv 2609.01056) targets three gaps the authors identify in agentic evaluation: contextual grounding, state preservation, and multilingual coverage. The benchmark delivers 1,600 file-based, persona-grounded tasks over seven languages and eight locales (EN-US, EN-UK, IT, PT, ES, FR, DE, ZH), built natively per locale rather than translated. The headline contribution is Constrained Task Success (CTS, §4.1): a task counts only if every task-specific evaluator passes AND every pre-existing non-target file remains byte-identical. Central claims: (1) frontier agents top out at 49.2% CTS (Gemini-3.1-Pro, Table 3); (2) pass rate exceeds CTS by 10.1–16.8 points across all nine models, exposing collateral edits invisible to correctness-only scoring; (3) an English-leading/Chinese-trailing gradient holds for nearly all models (Table 4); (4) the failure taxonomy (Appendix J) mechanistically explains the gaps — distractor capture (24.6%) plus shell over-reach (17.8%) account for 42.4% of 400 manually inspected failures.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Constrained Task Success (CTS) in WorldBench?
CTS is the paper's headline metric: a task counts as successful only if (1) every task-specific evaluation function passes on the final sandbox state (e.g., correct spreadsheet cell values, required text present) AND (2) every pre-existing file that is not a task target remains byte-identical to its initial state. This separates correct completion from completion that accidentally edits, overwrites, or deletes unrelated workspace files.
How well do LLM agents perform on WorldBench?
In the paper's experiments, Gemini-3.1-Pro leads at 49.2% CTS, followed by GPT-5 (48.8%) and Qwen-3-32B (48.0%); the weakest tested model, EuroLLM-9B, reaches 10.8%. Every model's plain pass rate is 10–17 percentage points higher than its CTS, meaning many nominally correct solutions also damaged unrelated files.
Is WorldBench just an English benchmark translated into other languages?
No. Tasks are generated natively within each of eight language-locale settings (US English, UK English, Italian, Portuguese, Spanish, French, German, Chinese) from human-written persona and scenario seeds, with localized names, currencies, document conventions, and cultural context. Annotators born in each target country audited realism, cultural appropriateness, and solvability. The trade-off is that task sets are not direct translations, so cross-language comparisons are setting-level rather than item-level.
What kinds of tasks are in WorldBench and how are they evaluated?
Each of the 1,600 tasks places an agent in a sandboxed file environment (spreadsheets, Word documents, PDFs, mailboxes, calendars, shell) with 15–20 distractor files, acting through structured JSON actions under a 30-step cap. Evaluation combines five deterministic checks (required/forbidden text, spreadsheet cell values, file existence, calendar non-overlap) with two LLM-judged checks for generated emails and notes, which were validated against human annotations with Cohen's kappa of 0.76–0.84.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Footnote 1's code-and-data link is garbled in the extraction (renders as '§ WorldBench instance'), so the actual repository URL could not be verified from the provided text; code availability is asserted but not locatable here.
- The extracted audit sentence reads 'The audit involved annotators, with 40 assigned to each language–locale setting' — the total annotator headcount is absent; this could be either a source typo or an extraction loss, so it is raised as a reviewer question rather than asserted as a source defect.
- The provided text ends mid-sentence in the Figure 15 caption ('...underlying the language.'), so at most a few final words of Appendix L are cut; everything cited in this review precedes that point and all references/appendices A–L are otherwise present.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.