Schematize: An Agentic System for Generating and Refining Information-Extraction Schemas for Legal Research
Albert Sawczyn, Jakub Binkowski, Kamil Tagowski, Łukasz Augustyniak, Berenika Kaczmarek-Templin, Tomasz Kajdanowicz
cs.CL · cs.LG
Quick verdict
Agentic system turns legal research questions into extraction schemas; best or second on 6/7 LLM backbones, yet ablations show its own loops add little.
Summary
The paper presents Schematize, an open-source multi-agent system that converts a natural-language legal research problem into an extraction schema validated against real documents before large-scale extraction. The pipeline couples a problem-definition-helper dialogue that elicits implicit expert intent, query generation, schema drafting, a document-blind criteria-based refinement loop, a data-grounded refinement loop that tests the schema against retrieved judgments, and chat-based post-editing. Evaluation uses a novel protocol: seven Polish legal experts (three per problem, three problems) independently wrote roughly 15–30 atomic questions each without seeing generated schemas, and an LLM-as-judge (gpt-5.4-mini) scores what fraction each schema covers. Across seven backbones (four closed-, three open-weight), Schematize is best or second-best in six of seven configurations — e.g., 79.4% coverage with gpt-5.4 versus 64.3% for the strongest baseline — and exceeds the average inter-expert coverage (53.8%) on four backbones. Costs are disclosed honestly: about $3.79 per problem with gpt-5.4, roughly 160x a single LLM call. Ablations complicate the story: removing either refinement loop individually shifts coverage only +1.5/−1.0 points, while removing the clarification stage costs −14.3 points. Key limitations: one jurisdiction and language, an unvalidated LLM judge, coverage measuring only recall, and small samples with standard deviations up to ±35 points.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Schematize turns a natural-language research problem into a document-grounded extraction schema via a clarification dialogue plus criteria-based and data-grounded refinement loops.
- Across 7 LLM backbones, schematize achieves best or second-best expert-question coverage in 6 of 7 configurations (e.g., 79.4% with gpt-5.4 vs 64.3% for the strongest baseline).
- On 4 of 7 backbones, schematize exceeds the average inter-expert coverage benchmark of 53.8% (Table 2).
- The expert-question coverage protocol is a novel, bias-limiting evaluation methodology that avoids hindsight and confirmation bias and is repeatable without re-annotation.
- The three pipeline components (PDH, CBR, DGR) are complementary and the full system performs best (contested by the paper's own Table 4 single-removal deltas).
Strengths
Novel, bias-limiting evaluation protocol
Experts author 15–30 atomic questions per problem blind to generated schemas (Section 4.1), and inter-expert coverage (53.8% mean ordered-pair, Table 2) computed with the same judge provides a calibration anchor rarely seen in schema-generation work.
Comprehensive and fair baseline design
Every baseline is run both with and without the same fixed PDH exchange (Section 4.2), isolating the pipeline's contribution from the extra input; 7 backbones spanning closed and open weights, plus ablations (Table 4), a hyperparameter grid (Fig 4) and full cost accounting (Table 5).
Honest failure and cost reporting
Engineering and release quality
Deliberate anti-overfitting architecture
Weaknesses
Ablation contradicts the architectural narrative
Statistical evidence too thin for superiority claims
A single unvalidated judge underpins everything
Human-in-the-loop stages unevaluated
Narrow external validity vs broad framing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Schematize (arXiv 2609.22209, cs.CL) is a systems paper with a bundled evaluation methodology and empirical study. It targets the upstream bottleneck of empirical legal research: turning a natural-language research problem into an extraction schema before any large-scale extraction. Four claims anchor the paper: (1) a multi-agent pipeline (clarification dialogue → query generation → schema drafting → criteria-based refinement (CBR) → data-grounded refinement (DGR) → summary/chat) produces schemas grounded in both expert intent and real documents; (2) a novel expert-question coverage protocol limits evaluation bias; (3) the system is best or second-best in 6 of 7 backbone configurations (Table 1); (4) everything is released (pip package, MIT license, docs, Hugging Face dataset pl-court-raw).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does Schematize do?
It is an open-source multi-agent system that takes a natural-language legal research question and, through a clarifying dialogue plus automated refinement loops tested against real court documents, produces a validated information-extraction schema ready for large-scale structured data extraction.
How well does Schematize perform compared to baselines?
Across seven LLM backbones it achieved the best or second-best expert-question coverage in six of seven configurations — e.g., 79.4% coverage with gpt-5.4 versus 64.3% for the strongest baseline (ScheMatiQ given the same clarification dialogue) — and beat the average inter-expert coverage benchmark of 53.8% on four backbones.
Is the Schematize code and data publicly available?
Yes. The paper reports a pip-installable Python package on PyPI, source code and documentation under the MIT license, and the Polish court judgment dataset (pl-court-raw) on Hugging Face under CC BY 4.0.
What are the main limitations of the Schematize paper?
Evaluation covers only Polish case law with 3 research problems and 7 experts; the coverage metric is scored by an LLM judge that was never checked against human decisions; coverage measures only whether fields exist (recall), not end-to-end extraction accuracy; and ablations show the refinement loops contribute little individually — most of the gain comes from the clarification stage.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2–4 are graphics in the PDF; the extraction preserves only axis/legend labels as jumbled text, so figure-level values (iteration curves, grid-search cells) could not be read — judgments about them rely on the surrounding prose.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.