The Era by Eon Benchmark: A Generated Enterprise Estate with Exact Ground Truth for Benchmarking LLM Agents
Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary
cs.AI
Quick verdict
Seed-generated fictional companies with exact answer keys for agent benchmarks; sound validation, but headline eval is one small estate and code is unreleased.
Summary
The paper introduces the Era by Eon Benchmark, which generates the entire evaluation environment for LLM agents operating enterprise systems of record. From industry, size, business model, application portfolio, and a seed, a deterministic generator builds one fictional company as an entity graph, projected into simulators of 66 products (Salesforce, Zendesk, Slack, Gong, among others) and supplemented by internal databases whose schemas and records are generated from author-supplied questions. Every expected answer is computed from final records, so grading is exact; a reachability audit re-reads answers through real product APIs and MCP tools, and a consistency gate ties internal databases to shared graph entities. Validation pairs a five-axis realism scorecard (marginal, joint, temporal, structural, content) with an adversarial detector: across 23 generated companies, mean realism rose from 61.8 to 97.0 and flagged synthetic records fell from 55.2% to zero, with the baseline recomputed under the final scorecard. A reported comparison of nine models on one estate (33 questions, three repeats, 891 answers) spans 42.4%–76.8% accuracy, with wide 95% intervals and only three of 36 pairwise differences supported after correction; simple filtering reached 92.6% while multi-hop scored 3.7%. Main limitations: the headline evaluation covers one small estate, the generator and estates are proprietary, internal-database and document tracks lack model results, and realism targets partly derive from the authors' own operational data.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A seeded, deterministic generator can produce a complete fictional enterprise estate — product simulators plus internal databases — where every benchmark answer is computed exactly from the records agents read.
- A question-conditioned generator can turn business questions into internal-database schemas, planted records with near-misses, and answer keys that pass design-adherence, anti-shortcut, answer-key-fairness, and dependency checks while staying consistent with the shared entity graph.
- Realism can be measured without a paired real dataset: across 23 generated companies, mean scorecard realism rose from 61.8 to 97.0 and the share of records flagged synthetic fell from 55.2% to zero, with the baseline recomputed under the final scorecard.
- On one estate, nine models over 33 questions with three repeats score between 42.4% and 76.8%, and only three of 36 pairwise differences remain supported after correction, so most model comparisons are inconclusive.
- Current models handle simple filtering well (92.6%) but fail multi-hop (3.7%), superlative (6.2%), temporal (7.4%), and ranking (11.1%) questions over enterprise records.
Strengths
Exact, self-consistent ground truth by construction
Questions come from code templates and every answer key is computed from the final records (Sec. 4.1), so key–data disagreement is impossible; a CI test regenerates the committed corpus byte-identically, and no answer is ever hand-written or hand-maintained.
Reachability audit protects agents from impossible questions
Sec. 4.2 re-reads every audited fact through native product APIs and MCP tools; the paper reports a real caught defect (a call–contact join correct in stored data but impossible through the product interface, since only participant email is visible on both sides).
Anti-shortcut instrumentation for question quality
Unusually honest statistical reporting
Before/after realism protocol with overfitting safeguards
Weaknesses
Headline evaluation is statistically underpowered
Realism validation is partly self-referential
Closed artifact undermines the central claims
Narrow reported coverage versus the full construct
Template-generated text has an admitted ceiling
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper introduces the Era by Eon Benchmark, a generated evaluation environment for LLM agents operating enterprise systems of record. From five inputs — industry, company size, business model, application portfolio, and a random seed — a deterministic generator produces a complete fictional company: one entity graph (workforce, customers, pipeline, tickets, calls, chats, documents) projected into simulators of 66 real products (Salesforce, Zendesk, Slack, Gong, S3, Google Drive, among others) and extended by internal databases whose schemas and records are generated from author-supplied business questions (Sections 3, 4.4). Two derived claims: (a) realism can be measured without a paired real dataset, via a five-axis scorecard and an adversarial detector (mean over 23 companies rising 61.8 → 97.0, later passes to 98.6; flagged synthetic records 55.2% → 0); and (b) agents can be graded exactly because every answer key is computed from final records. The evaluation claim is deliberately modest: on one frozen estate (segment_fintech_small), 33 questions, three repeats, nine models, accuracy spans 42.4%–76.8% and "most apparent differences between models remain inconclusive" (Section 6.2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Era by Eon Benchmark?
It is a benchmark environment for evaluating LLM agents that work with enterprise software (CRMs, support desks, call recorders, file stores). Instead of hand-built test data, it deterministically generates a complete fictional company — including simulated Salesforce, Zendesk, Slack, Gong, and other products plus internal databases — from industry, size, business model, application portfolio, and a random seed. Every benchmark question's correct answer is computed from the generated records, so grading is exact and repeatable.
Which AI models performed best on the Era by Eon Benchmark?
In the paper's reported comparison, nine models answered 33 questions three times each on one generated company. Claude Opus 4.8 scored highest at 76.8% accuracy and GPT-5.4 Mini lowest at 42.4%. However, because the question set is small, confidence intervals are roughly 30 points wide, and after statistical correction only three of 36 pairwise differences were supported — all involving the weakest model — so most model rankings remain inconclusive.
How does the benchmark prove its generated company data is realistic?
Two instruments validate the data. A five-axis realism scorecard (marginal distributions, field correlations, calendar/temporal patterns, heavy-tailed structure, text coherence) compares generated companies against reference statistics from Eon's operational data and published sources; over 23 generated companies the mean score rose from 61.8 to 97.0 during generator development. A separate adversarial detector scans for synthetic tells (numbered names, flat histograms, uniform timestamps); flagged records fell from 55.2% to zero.
Is the Era by Eon Benchmark open source or publicly available?
The paper does not announce any public release of code, generated estates, or answer keys. It explicitly describes the internal-database generator as an internal development tool — clients receive finished estates and answer keys, not the generator itself — so independent verification of the reported numbers is currently not possible from the paper alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.