Symbolic Separation: Grounding Deep Agents in Knowledge Graphs for Trustworthy Operational Data Analytics
Baibek Davletiyarov, Junaid Ahmed Khan, Andrea Bartolini
cs.AI
Quick verdict
Ontology-gated deep agent lifts analytics success from 43% to 86% on 14 tasks, yet its central validator is never ablated; interesting, needs stronger evidence.
Summary
This method/systems paper proposes "symbolic separation": a deep LLM agent reasons freely but may touch data only through an ontology-constrained Virtual Knowledge Graph with deterministic pre-execution validation. Instantiated as a Neurosymbolic Deep Analyst (coordinator plus EXASAGE Reflect Agent retrieval subagent plus sandboxed code subagent, connected via MCP), it is evaluated on Marconi100 supercomputer telemetry (M100 ExaData, 49.9 TB). Three backends sharing the same coordinator, skills and LLM were compared on 14 author-designed end-to-end queries requiring retrieval plus analysis: the best proposed configuration (Qwen3.6-35B-A3B) reached 86% success versus 43% for the non-symbolic Datalake ablation (with GPT-OSS-120B) and 7% for the prior EXASAGE Query Tool, whose collapse from its originally reported 93.6% the authors attribute to rigid template design. Token accounting reports 2.4x lower median tokens per successful query (300k vs 713k) and zero hallucinated answers, whereas the ablation produced silent factual errors. On 75 retrieval-only queries drawn from HPC studies, accuracy was 88% versus 56% and 48%. Main weaknesses: the tiny evaluation sets, no statistical testing, no ablation isolating the deterministic validator from the new LLM extraction, reflection loop and RAG prompting, over-stated "out-of-design-samples" validation (the 75-query set includes all ten EXASAGE archetypes), author-constructed test sets, inconsistent counting across tables, and no user evaluation of the clarify interaction. A strong engineering contribution whose headline mechanism still needs independent validation.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 4
● low 0
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Symbolic separation — routing all data access through an ontology-constrained VKG with deterministic pre-execution validation — raises end-to-end analytics success from 43% (best non-symbolic configuration) to 86% (proposed, Qwen3.6-35B-A3B) and eliminates hallucinated answers.
- Materialising cross-source relations as validated graph edges converts complex questions into single validated traversals, preventing the LLM from inventing relations, unlike tool-API interface contracts.
- The EXASAGE Reflect Agent cuts median tokens per successful query by 2.4× versus the best non-symbolic configuration and retries rarely, avoiding wasted computation.
- The 93.6% previously reported for EXASAGE Query Tool does not transfer: on the new 75-query set it reaches only 56%, and on the 14 end-to-end queries only ~7%, because of its rigid single-shot template design.
- The revised ODA ontology (308 axioms, 36 classes, 23 object, 55 data properties) is itself a contribution enabling validator-enforced, broader-coverage analytics.
Strengths
Well-controlled three-way ablation isolating the data-access layer
Section IV fixes the coordinator, MetricsExpert skill, LLM and hardware across configurations, swapping only the retrieval backend (EXASAGE Reflect Agent vs EXASAGE Query Tool vs Datalake Query Tool). The 14%→88% retrieval spread and 43%→86% end-to-end spread are therefore attributable to the backend design rather than orchestration.
Diagnostic attribution of the prior system's collapse
Section IV-B plus Table IV decomposes EXASAGE Query Tool's out-of-template failures: entity extraction 63.4%, SPARQL generation 46.3%, VKG generation 59.8%, final answer 42.7% on the 75-query set — evidence that its regex-based extraction and template coverage, not graph grounding, are the bottleneck.
Trustworthiness accounting beyond accuracy
Discovery of a genuine sandbox-escape failure mode
Weaknesses
Very small, author-constructed evaluation sets
The central contribution is never ablated
Inconsistent outcome counting across Table I, Table II, and the text
Cross-model headline comparison; model-sensitivity unexplored
Token-efficiency framing is median-selective
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review — Symbolic Separation (arXiv 2609.17107)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is "symbolic separation" in this paper?
An agent design principle where the LLM reasons freely in natural language, but any access to data must pass through a knowledge-graph layer constrained by a domain ontology, where every generated SPARQL query is checked against the ontology before execution. The claim is that pre-connected, ontology-validated relations prevent the model from hallucinating how data sources relate — not just which fields exist.
How much does the proposed system improve over baselines?
On 14 end-to-end data-analysis queries over supercomputer telemetry, the best configuration (with Qwen3.6-35B-A3B) completed 12/14 (86%) versus 6/14 (43%) for a non-symbolic agent querying the raw data lake and 1/14 (7%) for the authors' prior EXASAGE Query Tool. On 75 retrieval-only queries it answered 66/75 (88%) versus 42/75 (56%) and 36/75 (48%).
What data was used for the evaluation?
The public M100 ExaData corpus: about 49.9 TB of telemetry from the Marconi100 supercomputer (980+ nodes, 2.5 years), partitioned into nine plugins (IPMI power, Ganglia GPU metrics, SLURM jobs, weather, and others).
What are the main limitations of the study?
Small evaluation sets (14 end-to-end and 75 retrieval queries, both constructed by the authors), no statistical tests or repeated runs, no ablation isolating the deterministic validator from other pipeline upgrades, a cross-model headline comparison, inconsistent outcome counts between tables and text, and no user evaluation of the clarification interaction.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 (architecture, ontology, pipeline, state machine) were not analyzable from extracted text; judgments rely on the prose descriptions.
- Table II per-query token/status cells were extracted linearly and may misalign columns; the count discrepancies flagged could partly stem from extraction artifacts.
- Appendix B output plots are described only via captions; visual quality of artifacts was not assessable.
- Reference [62] carries an inline 'VERIFY exact variant' note, suggesting the model identity (Qwen3.6-35B-A3B) was unverified in the manuscript itself.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.