SAGE: Governed Artifact Generation from Enterprise Guidelines
Mohammadreza Sediqin, Shivali Dalmia, Sumukha Thoppanahalli, Srinivasa Karthikeya Reddy Kovvuri, Abhishek Mukherji
cs.AI
Quick verdict
SAGE cuts guideline-to-artifact work from days to ~1h at 3.2% hallucination via a governed LLM pipeline; sound design, but evaluation rests on proprietary data and proxy metrics.
Summary
SAGE is an industrial system from Centific that converts enterprise annotation-guideline documents (PDF/DOCX/PPTX mixing text, tables, and images) into structured work artifacts such as annotator instructions and statements of work. A deterministic PyMuPDF parsing path is combined with Qwen2.5-VL-32B for visual content; candidate rules are normalized to a 26-field schema and stored in a versioned rule store keyed by stable rule_ids, with Pydantic schema contracts between stages. Quality control layers comprise L1 deterministic structural validation, L2 LLM-as-judge semantic scoring with minimum-dimension routing, and a consistency module (embedding similarity plus NLI) for deduplication, contradiction detection, and gap analysis; only flagged or uncertain items reach persona-routed human review. On 120 proprietary documents, SAGE extracts 3,896 rules and produces 812 artifacts, reporting 3.2% hallucination versus 15.7% for an ungoverned one-pass baseline, 96% document-level success, and turnaround reduced from an estimated 2-3 days to 20-100 minutes. The L2 judge shows F1 0.957 and kappa 0.813 against 300 blind human labels. Main limitations: all data is NDA-protected with no code release mentioned; hallucination and grounding are measured by embedding cosine thresholds with human inspection of only 10-15 documents; thresholds were tuned on the full corpus with no held-out split; and the manual-effort baseline is an expert estimate rather than a controlled measurement.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A governed pipeline built on a versioned rule store, schema-validated inter-stage contracts, and selective HITL routing converts enterprise guideline documents into validated artifacts, cutting turnaround from an estimated 2-3 days to 20-100 minutes.
- Governance layers reduce hallucination from 15.7% to 3.2% and duplication from 10.3% to 3.0% relative to an ungoverned one-pass baseline using identical models (Table 6).
- Only 28.6% of the 3,896 extracted rules require human review because threshold-based auto-approval is reliable, with the L2 judge matching 300 blind expert labels at F1 0.957 and kappa 0.813.
- The system surfaces specification problems manual workflows miss: 1,040 gaps, 117 duplicates, and 113 contradictions across the 115-document corpus.
Strengths
Complete governance architecture, not just extraction
Shared versioned rule store with stable rule_ids, Pydantic schema contracts between all stages (Sec. 3), and provenance tracing every artifact back to source rules — addressing a gap the paper convincingly argues extraction-only systems (Docling, UIE, Donut) leave open.
Judge validated against blind human labels
The L2 judge is calibrated on 300 independently labeled rules with precision 0.941, recall 0.974, F1 0.957, and Cohen's kappa 0.813 (Sec. 4.3) — substantive agreement statistics rarely reported for LLM-as-judge pipelines.
Layer-wise ablation and baseline sharing models with SAGE
Routing design prevents score masking and self-evaluation bias
Honest scoping of the headline efficiency claim
Weaknesses
Entirely unverifiable evaluation
Proxy metrics carry the abstract's central claim
No held-out data; thresholds tuned on the evaluation corpus
Residual human burden underexplored
No statistical inference and promised stratification missing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the SAGE paper do?
SAGE (Centific Research, arXiv 2609.17775) is a system that automatically converts enterprise annotation-guideline documents — which mix narrative text, tables, and images — into structured work artifacts such as annotator instructions, QA rubrics, and statements of work. It validates every extracted rule structurally and semantically, checks for duplicates and contradictions, traces each artifact back to its source rules, and sends only uncertain items to human reviewers.
How much time does SAGE save compared to manual work?
The paper reports manual conversion takes an estimated 2-3 working days per document, while SAGE completes the same work in 20-100 minutes total, of which roughly 23-31 minutes is human review time. The authors note the manual baseline is an expert estimate rather than a controlled measurement, so this is an order-of-magnitude claim.
How accurate is SAGE — what is its hallucination rate?
On 120 enterprise documents, SAGE reports a 3.2% hallucination rate for extracted rules versus 15.7% for the same models without governance layers, a 96% document-level success rate, and 812 generated artifacts. Caveat: hallucination is measured by embedding cosine-similarity thresholds, and human verification covered only a small sample (10-15 documents).
Is the SAGE code or dataset publicly available?
No. The paper does not mention any code release, and all 120 evaluation documents are client data under non-disclosure agreements, processed only with self-hostable models (Qwen2.5-VL-32B, Qwen3-32B, LLaVA-13B) for confidentiality. The evaluation therefore cannot currently be reproduced independently.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (pipeline overview) is referenced but its content cannot be verified from the extracted text.
- Minor extraction artifacts exist in the source text (e.g., a duplicated 'as as'); section structure, all six tables, references, and Appendix A.1 otherwise appear complete.
- arXiv preprint (2609.17775, Sep 2026) with no peer-review status verifiable; only self-reported results are available for all quantitative claims.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.