Continual Enterprise World Model Discovery in Dynamic Systems
Shambhavi Mishra, David Vazquez, Perouz Taslakian, Marco Pedersoli, Jose Dolz, Issam H. Laradji
cs.AI
Quick verdict
Claims a ServiceNow benchmark and agent beating per-question rule lookup by up to 8.98 IoU; solid, but single-run gaps and readable rules qualify the claim.
Summary
The paper introduces continual enterprise world model discovery: an agent with no prior knowledge of an organization's ServiceNow business rules interacts with records, maintains an explicit world model of trigger-effect rules, and revises it as rules change. The EnterpriseWorldShift benchmark runs on a live ServiceNow instance with nine tables, 25 hidden rules and 600 evaluation actions, presenting four versions of one world in which a single rule is modified (World B), added (World C), then removed (World D). The Continual Discovery Agent (CDA) is compared against the Discovery Agent (DA) of prior work, which retrieves rule definitions at question time, and against a programmatic Simulator executing the same discovered model. Across three LLM backbones, CDA leads DA by 6.99-8.98 IoU points, with the advantage coming from precision rather than recall, and answers without live queries after a 91-480-action discovery phase. Continual maintenance beats rebuilding in Worlds B and C but loses in World D, and no backbone retires the stale rule despite observing disconfirming evidence, an instructive negative result. Main limitations: all reported conditions can read rule definitions during discovery, so the setting is not fully opaque; headline gaps rest on single runs, and a three-seed study narrows the Gemma gap to 4.83; the benchmark covers one lineage with one rule changing at a time; and the rule representation cannot express 5 of the 25 rules.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- CDA predicts the effects of hidden business rules more accurately than per-question rule lookup (DA) by 6.99-8.98 IoU points across three backbones, without issuing any live queries.
- CDA's advantage is precision-driven (mean 91.15 vs 82.47) at roughly equal recall, meaning the accumulated model is more selective than runtime retrieval.
- Maintaining one world model continually beats rebuilding per world in Worlds B and C but loses in World D, where reset wins on every backbone.
- Retirement of a removed rule succeeds on no backbone, even though every backbone observed disconfirming evidence (edits, empty effects, rule lists lacking the rule).
- EnterpriseWorldShift is the first benchmark presenting a changing enterprise world with controlled single-rule modifications, scored against per-action-validated ground truth from a reference implementation.
Strengths
First benchmark scoring a changing enterprise world model
EnterpriseWorldShift (§3) isolates discovery, revision, extension and retirement with single-rule deltas across four worlds (r03 modified 99→50, r26 added, r02 deactivated), and ground truth comes from a reference implementation validated against the live ServiceNow instance on all 600 evaluation actions at collection time (§3.3, B.1) — a per-action certificate, not a held-out assumption.
Honest, well-diagnosed negative result
Retirement succeeds on 0/3 backbones (Table 4), and §5.4 with Table 6 shows each backbone edited the trigger fields, observed the empty effect, and read rule lists lacking the removed rule — localizing the failure to the absence of any prediction-versus-observation comparison in the loop, not to missing evidence.
Mechanism-level ablations
Transparent artifacts and cost accounting
Careful scoring design
Weaknesses
Readable rule definitions dilute the discovery claim
Headline margins rest on single runs
Simulator gap undermines the auditable-world-model framing
Internal numeric inconsistency and modest scale
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper defines continual enterprise world model discovery: an agent without knowledge of an organization's ServiceNow business rules interacts with records, assembles an explicit world model whose components are inferred trigger-effect rules, and carries it across environment revisions. Two artifacts support the claims: the EnterpriseWorldShift benchmark (§3) — a live ServiceNow instance with nine tables, 50 fields, 252 seeded records, 25 hidden deterministic rules, and 600 evaluation actions split 150 per world — and the Continual Discovery Agent (§4), which maintains a belief list of rules and may add, refine, or retire them each turn. The four worlds isolate one model operation each: World A discovery, World B revision (rule r03's written value changes 99→50), World C extension (r26 added), World D retirement (r02 deactivated, unannounced). Central claims: CDA beats per-question rule lookup (DA, from Sethumadhavan Nair et al. 2026) by 6.99-8.98 IoU on three backbones without live queries; the gain is precision-driven; continual maintenance beats rebuilding in Worlds B/C but loses in D; retirement fails on every backbone.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is EnterpriseWorldShift?
A benchmark built on a live ServiceNow instance with nine tables, 25 hidden business rules and 600 evaluation actions. It presents four versions of the same enterprise world — modifying, then adding, then removing one rule — to test whether an AI agent can discover the rules through interaction and keep its world model correct as they change.
What is the Continual Discovery Agent (CDA)?
An agent that experiments with records in the system, writes down the business rules it infers as an explicit trigger-effect world model, carries that model across environment versions, and revises it when evidence contradicts it. After discovery it answers questions from its own model without querying the live system.
How much better is CDA than prior work?
On three LLM backbones it beats the prior Discovery Agent by 6.99 to 8.98 IoU points on change prediction, mainly through higher precision rather than broader recall, and it needs no live queries. On the one backbone where multiple seeds were run, the average gap narrows to 4.83 points.
What is the paper's main negative finding?
When a rule is removed from the environment (World D), none of the three backbones retires the stale rule from its world model, even though each observed the effect disappearing and read rule lists without the rule. The discovery loop never compares what the model predicts against what is observed, so an expected effect that fails to appear goes unnoticed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 are available only as caption text; visual content (plots, diagrams) could not be assessed directly.
- No code or data URL appears in the extracted text; a 'release' of prompts and world models is referenced (C.2, C.3) but its location is unstated.
- Extraction appears complete through Appendices A-C, but visual artifacts such as the shaded IoU column in Table 2 cannot be verified from text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.