Era by Eon: Benchmarking Enterprise Agents on Hidden Knowledge
Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary
cs.SE · cs.AI
Quick verdict
Adds 8 deterministic hidden-knowledge questions to Era by Eon; elegant construct, honest limits, but single-seed eval and internal count errors weaken claims.
Summary
The paper extends Era by Eon, a benchmark that deterministically generates a fictional company and serves its data through simulators of Salesforce, HubSpot, Zendesk, Jira, Gong, Slack, and S3, with exact answers computed by code rather than a language model. Because the original computable questions barely separate strong agents once code execution is available — the four strongest of six models answer 22-25 of 27 (Table 1) — the authors add eight question templates that depend on 'hidden knowledge': facts no question or document states, contradicted by the record the question names, and implied only by other served data, such as a service credit promised verbally on a recorded call. Code fills the templates and computes each key; four checks guarantee unique, exact answers. Twelve agents (six models x two programs, one with a Python sandbox) answered eight questions three times each. The best agent reached 18 of 24 runs; four of six models managed at most 6; two 'which record is right' questions were solved once in 72 runs (the abstract inconsistently says 84). Code execution did not help here. Limitations: one company, one seed, Wilson intervals up to 37 points wide, no human baseline, an identical injected remark across companies inviting memorization, and logs that omit actual answers, blocking failure analysis.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- When agents can run code, the four strongest of six models answer 22-25 of 27 computable Era by Eon questions, so that question set no longer separates them (Table 1).
- Eight new templates whose answers depend on hidden knowledge discriminate sharply: the best agent answers 18 of 24 runs and four of six models answer at most 6 (Table 2).
- Code execution does not improve hidden-knowledge performance (Claude Fable 5.1: 15 with sandbox vs 18 without; GPT-6 Astra: 12 vs 13).
- Questions requiring selection of one right record among similar ones are nearly unsolved: one correct run in 72 attempts across all twelve agents (the abstract misstates 84).
- Questions and exact keys are generated deterministically from company data with no language model involvement, and any question failing four validity checks is dropped.
Strengths
LLM-free, deterministic ground truth pipeline
Section 5 traces the service-credit template end-to-end: entity selection by a fixed identifier order, one injected sentence, key computed in pure code from served data, and four self-checks (including recomputing the key from what simulators serve) with the question dropped on failure. No model noise enters questions or keys, and the seed-43 walkthrough shows seed sensitivity.
Precisely specified, reusable construct
Section 4 defines hidden facts via three properties, distinguishes three kinds of hidden knowledge, and states four conditions (evidence served, unique answer, obvious answer wrong, exact multi-field answer) that keep every question machine-gradable — a taxonomy other benchmark builders can adopt.
Clean, decision-relevant comparative finding
Candid limitations and useful behavioral signal
Strict grading and honest scoring of failures
Weaknesses
Evaluation rests on one company and one seed
Internal count inconsistencies in reported results
Contamination-by-construction of the call-remark cue
No human baseline and no failure telemetry
Weak computable-question baseline arm
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This benchmark paper extends the authors' Era by Eon environment (ref [4]) with eight question templates whose answers depend on what the authors define as *hidden knowledge*: a fact (i) that the key depends on, (ii) that no question or document states and that the record the question names contradicts, and (iii) for which other served data supplies evidence (Section 4.1). Three kinds are distinguished: a rule spoken on a recorded call (four templates), one-right-record-among-several (three templates), and a computation method recoverable only from an old report (one template) (Sections 4.2, 6). Four conditions — evidence served, unique answer, obvious answer wrong, exact multi-field answer (Section 4.3) — keep every question machine-gradable. The central claims: computable questions saturate when agents can run code (22-25 of 27 for the four strongest models, Table 1); hidden-knowledge questions discriminate sharply (best agent 18/24; four of six models at most 6/24, Table 2); code execution, decisive on computable questions, does not help here; and the 'several records' kind is nearly unsolved (one correct run in 72 attempts on two questions).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'hidden knowledge' in the Era by Eon benchmark?
A fact that an agent's correct answer depends on but that no question or document states: the record the question names even shows something else, and the only evidence is other served data — for example, a service credit promised verbally on a recorded sales call. The benchmark adds eight such questions with exact, code-computed answers.
How well do AI agents perform on these hidden-knowledge questions?
On one generated company, the best agent (Claude Fable 5.1 without code execution) answered 18 of 24 runs correctly, while four of six models managed at most 6 of 24. Two questions about which of several similar records the company actually relies on were solved only once in 72 total attempts across all agents.
Does letting agents run code help on hidden-knowledge questions?
No. Code execution lifted scores by 16-21 questions on ordinary computable questions, but on the hidden-knowledge set the strongest models scored slightly lower with the sandbox (Claude Fable 5.1: 15 vs 18; GPT-6 Astra: 12 vs 13). Reading more records does not reveal which record the company relies on.
Is the Era by Eon benchmark publicly available?
The generated companies are available at console.era.eon.io. Generation is deterministic from a specification and seed, and all question answers are computed by code without any language model. The agent harness code and per-run result logs are not stated to be released in this paper.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review based on the extracted text only; Figure 1's template table is rendered as prose but was fully readable, and both result tables were present.
- Per-run answers and tool traces are absent from the paper itself (authors state logs omit them), so Section 9's behavioral narrative cannot be independently verified from the text.
- The internal count inconsistencies (84 vs 72; per-kind sums vs Table 2) mean some reported figures could not be cross-checked and may change in a revision.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.