Data Quality Rule Generation with LLMs
Anna-Christina Glock, Thomas Hütter, Johannes Fürnkranz, Wolfram Wöß, Christine Dominka-Kiss, Lisa Ehrlinger
cs.LG
Quick verdict
LeDQeR generates Great Expectations rules from dirty tuples via LLM + four filters; sound framework and eval, but no baselines and a no-op coverage filter.
Summary
The paper presents LeDQeR, an LLM-based approach that automatically extends the rule set of a rule-based data quality tool (Great Expectations 1.9.1) when a dirty tuple passes validation unnoticed. Given a dirty tuple, a prompt template, rule-syntax guidelines, and a dataset description, an LLM generates candidate GX rules, which four filters then refine: a syntactic filter (executability), a correctness filter (the rule must flag its trigger tuple), a coverage filter (recall on same-error-type tuples; with no threshold enforced it retains all rules), and a redundancy filter (greedy precision-1 covering on a held-out test set). Experiments span five datasets (beers, flights, hospital, tax, and a synthetic nine-error-type PCI set built from real Austrian addresses), three local LLMs (GLM-4.7-Flash, Qwen3-Coder, Gemma-4), three input variants (Dirty & Clean, Dirty & Type, Just Dirty), and two syntax-guide variants. Key results: 87.13-100% executability with parameter information; mean F1 0.66 (best ebv 0.87, worst con 0.4); providing error type plus column lifts F1 by 0.33 over Just Dirty; mean recall 0.88; final sets of 2-136 rules reach at least 50% dirty-tuple coverage at precision 1 on four of five test sets, with tax reaching 99.58% given six false positives. Limitations: no non-LLM baselines or filter ablations, internal numeric inconsistencies (recall 0.58 attributed to dov in Section 4.4 but dmv in Section 4.6), and unstated LLM inference parameters.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLMs reliably generate executable Great Expectations rules from dirty tuples, with 87.13%-100% executability when prompts include parameter information (Table 3, Section 4.2).
- The filter pipeline compresses tens of thousands of candidate rules into compact final sets of 2-136 rules that generalize to unseen data, achieving at least 50% dirty-tuple coverage at precision 1 on four of five test sets (Sections 4.5-4.6).
- Providing error type and column information improves F1 by an average of 0.33 over dirty-data-only input, reducing the need for clean reference data (Section 4.3).
- LeDQeR differs from prior LLM rule generators (Xie et al., Schneider) by incrementally deriving rules from observed errors in real dirty tuples rather than one-shot from requirements or templates (Table 4).
Strengths
Reusable problem formalization and honest positioning
Section 3.1 formalizes a generate-filter framework and Table 4 maps prior LLM rule-generation work (Xie et al., Schneider) onto it by input/generation mode/filtering/output, giving the community a vocabulary beyond this one system.
Unusually broad generation experiments
Table 3 spans 3 LLMs x 5 datasets x 3 input variants x 2 syntax-guide variants (90 cells), with concrete rule volumes (e.g., 27,745 Qwen3-Coder rules for PCI Just Dirty), enabling per-condition statements like GLM-4.7-Flash dropping to 28% executability on beers (Dirty & Clean, no parameters).
Practitioner-relevant operating-point analysis
Above-average artifact availability
Transparent worked examples
Weaknesses
No baselines and no filter ablation
Internal numeric inconsistencies
The coverage filter is a no-op as evaluated
Descriptive-only statistics and unreported sampling configuration
Realistic-setting performance is modest and partly manual
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
LeDQeR (Sections 1, 3) targets gap-closing in rule-based data quality tooling: when a dirty tuple passes an existing Great Expectations (GX) rule set unnoticed (Figure 1), the system generates new GX 1.9.1 expectations from that tuple. Central claims: (1) a generalizable generate-filter formalization (Section 3.1); (2) an LLM-based instantiation with four filters ensuring executability, correctness, generalizability, and non-redundancy; (3) empirical evidence across five datasets and up to nine error types that the approach yields effective and compact rule sets. Claims are scoped to GX syntax and single-table validation; the framework is advertised as tool-agnostic but only instantiated for GX.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is LeDQeR?
LeDQeR is an LLM-based framework that automatically extends the rule set of the open-source data quality tool Great Expectations. When a dirty record slips past existing validation rules, a large language model generates candidate validation rules from that record, and four filters (syntax, correctness, coverage, redundancy) reduce them to a small deployable rule set.
How well does LeDQeR perform?
In the paper's experiments on five datasets, generated rules were executable 87.13%-100% of the time when prompts included rule parameter information. Mean F1 across nine error types was 0.66, final rule sets contained only 2-136 rules, and at a zero-false-positive setting they covered at least 50% of dirty tuples on four of five test datasets (99.58% on the tax dataset after accepting six false positives).
Does LeDQeR need clean data or labeled examples?
No clean data is required: the method works from dirty records alone, though this realistic mode yields the lowest quality (implied mean F1 around 0.33). Providing just the error type and affected column improves F1 by about 0.33 on average; the redundancy filter additionally benefits from a curated labeled test set to estimate precision and recall.
Is the code and data for the paper available?
Yes. The authors provide public GitHub repositories containing the LeDQeR implementation, the generator for the synthetic PCI dataset, and the polluted versions of the beers, flights, hospital, and tax datasets; the three quantized local models (GLM-4.7-Flash, Qwen3-Coder, Gemma-4) are available on HuggingFace.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 4-7 are images in the PDF; only the aggregate values quoted in the prose (e.g., F1 0.66, recall 0.88 with std 0.16, tax coverage 99.58%) could be verified — per-error-type and per-model breakdowns inside the figures could not be checked against the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.