MedDeID enables locally governed clinical-text de-identification from real or synthetic training data
Stig Hellemans, Tom Stroobants, Elyne Scheurwegs, Pieter Meysman, Philippe G. Jorens, Kris Laukens
cs.CL · cs.LG
Quick verdict
MedDeID: synthetic-only training rivals hospital-trained Dutch clinical de-identification; solid open framework, but gold standards partly author-annotated.
Summary
The paper presents MedDeID, an on-premises workflow for de-identifying Dutch clinical free text that covers annotation, synthetic-note generation, compact transformer training (a RobBERT-2023 dual-head span/label architecture), inference, pseudonymisation and evaluation under one shared schema. On a 300-note held-out, adjudicated UZA hospital benchmark, the hospital-trained model reaches 98.9% core PII recall with 0.24% non-PII redaction, while a model trained exclusively on synthetic notes reaches 96.1% — above all tested external comparators, including Belgian DEDUCE (88.0%), deidentify (86.4%) and a locally deployed Qwen3-8B (84.2%). On 100 primary-care notes the synthetic-trained model outperforms the hospital-trained one (90.3% vs 87.0%) and is more robust to formatting and value perturbations. An English model trained only on synthetic text achieves 99.7% (Technetium-I) and 98.9% (ASQ-PHI) annotation-character recall, the highest of seven systems. A pseudonymisation layer shifts dates per patient while preserving intervals and reduces age granularity age-dependently, with gold-span transformation failures of 0.0–3.0%. Main limitations: the gold standards were annotated or adjudicated largely by the authors themselves; benchmarks come from a single hospital and a single practice; the English results rest entirely on synthetic data; and the Qwen3-8B comparator used deliberately light prompt engineering.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On a 300-note independently annotated, adjudicated Dutch hospital benchmark, the hospital-trained compact transformer detected 98.9% of core PII characters with 0.24% non-PII redaction, exceeding all tested external systems (best comparator: Belgian DEDUCE at 88.0%).
- A model trained exclusively on synthetic Dutch notes reached 96.1% on the hospital benchmark and outperformed the hospital-trained model on 100 primary-care notes (90.3% vs 87.0%) with greater robustness to formatting and value perturbations.
- An English MedDeID trained only on synthetic text achieved the highest annotation-character recall of seven systems on two external synthetic benchmarks (99.730% on Technetium-I, 98.9% on ASQ-PHI), which the authors explicitly state does not establish performance on real English clinical text.
- MedDeID's pseudonymisation layer shifts each patient's dates by a common offset preserving clinical intervals and reduces age granularity age-dependently, with gold-span transformation failure rates of 0.0%–3.0% across the three Dutch benchmarks.
Strengths
Comprehensive, fairly engineered comparison
Nine systems on the Dutch benchmarks and seven on English, all pushed through the same post-processing/healing layer with optional metadata injection (Fig. 1, Table 2), so differences reflect detection quality rather than output-format quirks. Per-category and sub-annotation breakdowns (Table S5, Figs S2–S3) expose where each system fails, e.g. OpenAI filter at 12.6% recall on healthcare organisations.
Character-level sub-annotation and core-PII metric
Excluding formatting, titles, clinical content and clock times from the recall denominator (Eqs. 1–2, Table S3) is a genuine methodological contribution that makes span-boundary-unfair comparisons across systems with different taxonomy conventions meaningful; the authors note it is new for Dutch clinical text.
Utility and deployment are measured, not asserted
Strong openness and unusually honest scoping
The synthetic-only result is practically consequential
Weaknesses
Author-constructed gold standards
Narrow validation geography and scale
Weak LLM comparator
Deployment-relevant failure modes under-analysed
English evidence is entirely synthetic
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MedDeID (arXiv:2609.10049) is an on-premises workflow for clinical-text de-identification spanning annotation, synthetic-note generation, compact transformer training, inference, pseudonymisation and evaluation under a shared taxonomy and JSONL contract (Fig. 4). Three claims anchor the paper: (1) a hospital-trained compact transformer attains near-annotator performance on a 300-note adjudicated UZA hospital benchmark — 98.9% core PII recall, 0.24% non-PII redaction (Table 2), versus 98.8%/98.5% for the two physician annotators; (2) training exclusively on 6,493 synthetic Dutch notes yields 96.1% on the same benchmark and 90.3% on 100 primary-care notes, surpassing the hospital-trained model (87.0%) there and under perturbation; (3) the workflow transfers to English, where a synthetic-only model scores 99.730% (Technetium-I) and 98.9% (ASQ-PHI) annotation-character recall, best of seven systems (Table 3). Scoping is unusually careful: the abstract and Discussion state repeatedly that English results demonstrate workflow transfer, not clinical English performance.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MedDeID and what problem does it solve?
MedDeID is an open, on-premises framework for removing personally identifiable information from clinical notes. It covers annotation tools, synthetic training-data generation, model training, local inference, pseudonymisation (patient-level date shifting, age-dependent age reduction) and evaluation under one shared schema — so a hospital can de-identify text without sending it or the trained model weights to external services. It is demonstrated on Dutch hospital and primary-care text and on synthetic English benchmarks.
How accurate is MedDeID?
On a 300-note adjudicated Dutch hospital benchmark, the hospital-trained model detected 98.9% of core identifying characters while removing only 0.24% of non-identifying text; a model trained purely on synthetic data reached 96.1%, still above all external comparators tested (e.g. Belgian DEDUCE 88.0%, Qwen3-8B 84.2%). The English model reached 99.7% and 98.9% character recall on two public synthetic English benchmarks, but this does not establish performance on real English clinical notes.
Can I use or train MedDeID without real patient data?
Yes. The synthetic-trained Dutch model (96.1% recall on held-out real hospital notes) and the English model were trained exclusively on generated notes. The models are on Hugging Face, the synthetic corpora and benchmarks on Zenodo, and the code on GitHub, so institutions can start from synthetic data and fine-tune locally on their own annotations.
Is MedDeID validated on real English clinical notes?
No. The English model was evaluated only on synthetic benchmarks (Technetium-I and ASQ-PHI). The authors explicitly state these results demonstrate that the workflow transfers to another language, not clinical English performance — validation on authentic English clinical text in a real health system remains to be done.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with explicitly marked [...sampling gap...] regions (parts of the Discussion, the tail of the Methods stability subsection, and supplementary guideline pages). All numbers cited in this review were verified present in the visible text; no claim of missing content is made beyond these marked gaps.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.