ShowTellArena: Evaluating Business Workflow Understanding from Demonstrations
David Garg, Ritobrata Sarkar, Ehsan Azarnasab, Siddhartha Borah
cs.AI
Quick verdict
Benchmark of agent comprehension after narrated business demos; dataset unusually transparent, but pilot favors the authors' own product—exploratory only.
Summary
ShowTellArena is a benchmark protocol and public dataset (v1.0: 50 business workflow tasks, 502 questions, 438 narration segments, 7,182 screenshots) for evaluating what an agent comprehends after a narrated business demonstration delivered through each product's native teaching interface (Brackett Show & Tell, Claude's teaching flow, Codex Record & Replay). Tasks span finance, hiring, procurement, customer decisions, inventory, and logistics across five fixture applications (ERPNext, Fleetbase, ONLYOFFICE, Roundcube, Twenty). Questions test operational rules, boundaries, exceptions, and errors in proposed automations, with 857 evidence links tying answers to recording intervals, screenshots, or source records. An exploratory pilot of 218 selected attempts across 39 cases (28 attempted by all three systems) reports, on the shared set: Brackett 83.8%, Codex 55.4%, Claude 16.7% — with severely uneven grading coverage (83/83 vs 32/32 vs 12/56 graded) and asymmetric exclusions. The authors disclose substantial gaps: 16 of 857 evidence hashes disagree and 68 lack hashes, human positive and question-only controls are incomplete, grading provenance is mixed, and they develop Brackett, the top-scoring system; the pilot is explicitly not a controlled ranking. The inspectable, evidence-linked task format and assessment workflow are credible contributions; the comparative numbers are not interpretable as product standings.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 3
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ShowTellArena contributes a task format that links each comprehension question to teaching evidence (recording intervals, screenshots, source records), making answerability inspectable.
- The public v1.0 release comprises 50 business workflow tasks with 502 questions (253 closed-answer, 249 rubric), 438 narration segments and 7,182 screenshots across five fixture applications.
- In an exploratory pilot of 218 selected attempts across 39 cases, the 28 shared cases yield 83.8% (Brackett), 55.4% (Codex), and 16.7% (Claude), exposing both answer errors and failures to complete the teaching experience.
- The pilot's uneven coverage, asymmetric exclusions, and mixed grading provenance mean it is explicitly not a controlled product ranking.
- The release contains disclosed verification gaps — 16 of 857 evidence hash disagreements and 68 unhashed references — with human and question-only controls still pending.
Strengths
Novel, well-motivated evaluation target
Section 2 positions comprehension-after-teaching as complementary to execution benchmarks (WorkArena, OSWorld, τ-bench, TheAgentCompany): 'Correctly explaining a refund rule does not establish that the agent can carry out the refund safely.' Native Teach–Comprehend adapters (§3.1) test capture + memory as part of the system, which no cited precedent does.
Evidence-linked, auditable question design
§3.3 requires every expected claim to cite a recording interval, screenshot, or source record; 857 question-evidence references are published, with a structural audit reporting 773/857 hashes verified, 16 disagreeing, 68 missing — failures disclosed rather than hidden (§4.2), plus a separately versioned corrected returns quiz with a change record.
Exceptional methodological candor
Question design that separates recall from rule application
Reproducibility scaffolding for the aggregation
Weaknesses
Direct conflict of interest on the headline comparison
Pilot selection and grading are non-comparable by construction
Unvalidated measurement instrument
Dataset integrity gaps beyond the patched example
No statistical treatment and permanent contamination exposure
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ShowTellArena (arXiv 2609.25467, Brackett Labs) proposes a benchmark protocol plus public dataset for a question execution benchmarks ignore: what does an agent *comprehend* after being taught a business workflow through a narrated demonstration? The claimed contributions are (1) a task format linking each quiz question to teaching evidence, (2) a v1.0 release of 50 workflow tasks with 502 questions (253 closed-answer, 249 rubric), 438 narration segments, 7,182 screenshots, and five fixture applications (ERPNext, Fleetbase, ONLYOFFICE, Roundcube, Twenty), and (3) an exploratory analysis of 218 selected pilot attempts across 39 cases. From the abstract onward, the authors repeatedly disclaim ranking status — a disarming but structurally necessary move, because Table 2 compares their own commercial product against two competitors.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ShowTellArena?
ShowTellArena is a benchmark protocol and public dataset for testing whether an AI agent understood a narrated business workflow demonstration. A product is taught a scripted lesson (clicks plus narration) through its own teaching interface, then answers a fixed quiz on the demonstrated operational rules, boundaries, exceptions, and errors in proposed automations.
Which systems were evaluated and what were the scores?
An exploratory pilot compared Brackett Show & Tell, Claude's teaching flow, and Codex Record & Replay. On the 28 cases attempted by all three, scores were 83.8% (Brackett), 55.4% (Codex), and 16.7% (Claude), with very different numbers of graded attempts (83/83, 32/32, 12/56). The authors state this is not a controlled product ranking.
Did Brackett win because the authors built it?
Brackett scored highest in the pilot, and its authors developed it. The paper discloses this conflict and attributes the gap partly to favorable attempt selection and full grading coverage for Brackett versus mostly ungraded, failed teaching flows for Claude — so the numbers should not be read as a product ranking.
Is the dataset public and usable as a hidden test?
Yes — v1.0 is on Hugging Face with 50 tasks and 502 questions, MIT-licensed code, and CC BY 4.0 benchmark content. However, because answer keys are public, the authors themselves say it is unsuitable as a hidden test against training-data contamination.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Text extraction appears complete (abstract through Appendix B), but Figure 1 is only available via its caption; visual details were not analyzed.
- Screenshots, recordings, and the Hugging Face bundles could not be inspected, so answerability and hash-level claims rest on the paper's own audit.
- Pilot aggregates derive from a private workbook; only a sanitized row export is described, not independently verified here.
- Appendix-embedded numbers (e.g., exclusion counts) are taken as printed; no supplementary files were available.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.