From Document Silos to Process Intelligence: A Multi-Layer Knowledge Graph for CMC Process Development
Reza Amirmoshiri, Faryad Sahneh, Yasser Jangjou
cs.AI · cs.MA
Quick verdict
Sanofi's dual-layer CMC knowledge graph shows rigorous lexical-layer eval (95% T1/85% T2, 505 Qs), but the domain layer and router remain under-evaluated.
Summary
A Sanofi team presents an agentic-AI platform that converts heterogeneous CMC process-development documents into a dual-layer knowledge graph: a lossless lexical layer (Document→Section→Chunk trees in Neo4j, built with Docling plus hybrid BM25/vector retrieval) and an ontology-aligned domain graph whose 537 canonical entities are provenance-anchored to source chunks via HAS_MENTION edges. DSPy ReAct agents query both layers under a two-stage router. On a proprietary corpus of 38 documents (~52,000 words) from a discontinued small-molecule program, the authors introduce a three-tier benchmark: 505 LLM-generated multiple-choice questions across four difficulty cohorts, free-form answering scored by an LLM judge, and SME review of divergences. The lexical agent achieves 95.0% Tier-1 accuracy and 84.7% Tier-2 pass rate versus 88.5%/65.5% for a vector-RAG baseline; among 74 divergent items, question-quality issues (43%) and extraction errors (36%) dominate the failure taxonomy. The domain graph is demonstrated only on a small head-to-head question set, the router and SME scoring are unblinded, entity extraction is non-deterministic even at temperature zero, and no code or data is released. A transparent industry case study whose headline architecture is evaluated less rigorously than its lexical foundation.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A dual-layer knowledge graph - a lossless lexical layer plus a provenance-anchored ontology domain graph - converts heterogeneous CMC documents into queryable institutional memory.
- The three-tier protocol shows Tier-1 MCQ accuracy (95.0%) systematically overestimates reliability relative to Tier-2 free-form LLM-judge pass rate (84.7%) on 505 proprietary questions.
- Lexical_ReAct outperforms a vector-RAG baseline by 6.5 points at Tier-1 and 19.2 points at Tier-2 on the same question bank.
- Lexical graphs serve precise, locally scoped questions while the provenance-anchored domain graph serves cross-document aggregation, overlap, and connectivity questions, with a router selecting between them.
- Most Tier-1/Tier-2 divergence stems from question-quality issues (43%) and extraction errors (36%) rather than agent reasoning failures (3%).
Strengths
Three-tier evaluation that quantifies what MCQ accuracy hides
Section 2.5/3.3 and Table 1: full T1×T2 contingency per cohort with Cohen's kappa (0.17-0.32, pooled 0.22), Wilson CIs on both headline rates, and a judge×embedding cross-tab exposing 22 paraphrase-trap cases (Table 3). The 95.0% vs 84.7% gap is decomposed rather than just reported.
Provenance anchoring designed for auditable, regulated settings
Section 2.3: 22,688 HAS_MENTION edges with an explicit account of unanchored entities and 98.9% effective one-hop coverage; the paper argues traceable wrong answers are correctable in GxP contexts - an argument most GraphRAG papers omit.
Realistic proprietary testbed targeting version proliferation
Unusually candid limitations section
Weaknesses
The intelligence layer and router are asserted, not measured
LLM-generated question bank with material quality problems
Confounded baseline comparison
Reproducibility is low and judge bias unevaluated
Pipeline blind to figures forces benchmark exclusions
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This industry paper (Sanofi CMC Synthetics) presents a modular agentic-AI platform that turns a heterogeneous CMC process-development corpus into a dual-layer knowledge graph: a lossless *lexical* layer (Document→Section→Chunk trees in Neo4j, §2.2) and an ontology-aligned *domain* layer whose 537 canonical entities are anchored to source chunks via 22,688 HAS_MENTION edges (§2.3, 'provenance anchoring,' 98.9% effective one-hop coverage). Three DSPy ReAct agents (lexical, domain, router) run on Claude Sonnet 4.5 at temperature 0.2. The claims: (1) the architecture answers both keyword-precise and corpus-wide questions; (2) a novel three-tier protocol measures 'deployment fidelity' on proprietary data, complementing competence benchmarks like ChemRAG-Bench; (3) Tier-1 MCQ accuracy (95.0%) systematically overstates reliability versus Tier-2 free-form judging (84.7%). Results concern one discontinued small-molecule program (oditrasertib; 38 documents, ~52k words).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What accuracy does the Sanofi CMC knowledge graph system achieve?
On 505 proprietary questions from 38 internal development reports, the lexical graph agent scored 95.0% on four-option multiple-choice questions (Tier-1) and 84.7% on the stricter free-form test scored by an LLM judge (Tier-2), versus 88.5% and 65.5% for a vector-RAG baseline on the same questions.
What is a dual-layer knowledge graph in this paper?
It combines (1) a lexical layer that losslessly stores every document as Document, Section, and Chunk nodes in a Neo4j graph with keyword and vector search, and (2) an ontology-aligned domain layer whose canonical entities (chemicals, process steps, equipment) are linked back to the exact source text passages via HAS_MENTION edges so answers remain traceable and auditable.
Can I reproduce this benchmark on my own documents?
Not directly. The corpus is proprietary Sanofi data and no code is released. However, the protocol is described in enough detail to reimplement: LLM-generated multiple-choice questions in four difficulty cohorts, free-form re-answering without options, LLM-judge scoring on a 0-5 rubric, and human review of disagreements with a failure taxonomy.
What caused most of the system's failures?
Among the 74 questions where the multiple-choice and free-form evaluations disagreed, subject-matter-expert review attributed 43% to ambiguous or inconsistent questions, 36% to document-extraction errors, 11% to answer-letter parsing, 5% to judge scoring mistakes, and only 3% to genuine agent reasoning failures.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 4's panel layout is jumbled in the PDF extraction (axis and legend labels interleaved with body text); all quantitative values cited in this review come from the body prose, Table 1, and Table 3, which extract cleanly and reconcile arithmetically.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.