Meddies-PII: A Multilingual Framework for Personally Identifiable Information Extraction in Clinical De-identification
Linh Uyen Le, Christian Hoang, Huy Hoang Ha
cs.CL · cs.AI
Quick verdict
Synthetic 1M-doc, 17-language clinical PII dataset; its model tops 16 benchmarks (0.827 vs 0.658 mean F1) but ablations and real-record validation are absent.
Summary
The paper introduces Meddies-PII-Dataset, one million synthetic clinical documents in seventeen languages (English 25.55%, Vietnamese 11.55%, the other fifteen languages 4.19% each), annotated with a nine-label PII ontology informed by ISO/IEC 29100:2024. Documents are produced by an attribute-conditioned pipeline that samples six attribute groups — language, required labels, twelve medical document types, eleven text formats, five rule groups, twelve scenarios — prompts one of nine LLMs, and validates each draft through thirteen deterministic gates with a conservative regex repair step. A 5,100-document benchmark (300 per language: 100 standard, 200 adversarial via thirteen perturbation techniques) accompanies it. Fine-tuning LFM2.5-350M as a 37-class BIOES token classifier with constrained Viterbi decoding yields Meddies-PII-Model, which achieves the best exact-match entity-level micro-F1 on all sixteen reported benchmarks: an unweighted external mean of 0.827 versus 0.658 for OpenAI Privacy Filter, the strongest of four baselines. Margins are largest on Vietnamese, Chinese, and Korean OpenPII subsets and on the authors' benchmark (0.878 versus 0.540). Main limitations: no ablations attributing the gains to the framework itself, undisclosed label-mapping details that condition all baseline scores, no statistical uncertainty estimates, evaluation exclusively on synthetic data, and release of all assets deferred until acceptance.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Meddies-PII-Dataset provides one million synthetic clinical documents across seventeen languages with nine PII labels, produced by attribute-conditioned prompting over a nine-LLM pool and validated by thirteen deterministic gates.
- The synthesis framework offers more explicit and reproducible controls than prior synthetic PII datasets, which the authors argue are under-documented or rely on simple prompting.
- Meddies-PII-Model, a 37-class BIOES fine-tune of LFM2.5-350M with constrained Viterbi decoding, achieves the best exact-match entity-level micro-F1 on all sixteen evaluated benchmarks (external mean 0.827 vs 0.658 for the strongest baseline).
- Gains extend beyond English to Southeast Asian languages, with the largest margins on Vietnamese, Chinese, and Korean OpenPII subsets.
Strengths
Scale and language coverage beyond existing PII corpora
One million documents across seventeen languages (§3.1, Fig. 2) including low-resource Southeast Asian languages (Lao, Burmese, Tamil, Filipino) that manually annotated clinical corpora [2-4,9,10] and prior synthetic sets [5-7] do not cover; per-language counts are explicit (41,934 samples for each of fifteen languages).
Unusually transparent synthesis pipeline
§3.2 and Fig. 3 expose the concrete prompt template, six attribute groups, the nine-model generator pool, and an explicit accept/repair/reject policy (repair only unambiguous boundary defects; reject missing labels, invalid offsets, templated phones, absent native script, duplicates via SHA-256) — directly addressing the reproducibility gap the paper attributes to [5-7].
Consistent empirical dominance in Table 1
Honest evaluation hygiene
Weaknesses
No ablations or alternative-data control
Undisclosed label mapping conditions every baseline score
Benchmark circularity
No statistical uncertainty
No privacy audit of the synthetic corpus
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Meddies-PII (arXiv 2609.12544, submitted to PSB 2026) is a resource paper from Meddies AI claiming three artifacts: (1) Meddies-PII-Dataset — one million synthetic clinical documents across seventeen languages with a nine-label PII ontology (address, company_name, date, email_address, human_name, id_number, phone_number, private_url, secret; §3.1); (2) a controlled multilingual synthesis framework built on attribute-conditioned prompting plus thirteen deterministic validation gates (§3.2); and (3) Meddies-PII-Model, a BIOES token classifier shown to rank first on all sixteen evaluated benchmarks (§4.2). The headline number is an unweighted mean exact-match entity-level micro-F1 of 0.827 over fifteen external benchmarks versus 0.658 for OpenAI Privacy Filter, the strongest of four baselines (Table 1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Meddies-PII-Dataset?
A synthetic corpus of one million clinical-style documents in seventeen languages, each annotated with nine personally identifiable information (PII) labels such as names, dates, addresses, phone numbers, and secrets. It was generated by prompting one of nine different LLMs per document with controlled attributes, and every document passed thirteen deterministic validation checks before inclusion.
How well does Meddies-PII-Model perform?
It is a 350M-parameter BIOES token classifier fine-tuned on the synthetic dataset. In the paper's evaluation it achieves the best exact-match entity-level F1 on all sixteen reported benchmarks, with a mean F1 of 0.827 across fifteen external benchmarks versus 0.658 for the strongest compared system, OpenAI Privacy Filter.
Is the Meddies-PII dataset publicly available?
Not yet. The authors state that the dataset, benchmark, model, generation framework, and evaluation code will be publicly released only upon acceptance of the paper.
Does Meddies-PII work on real clinical records?
That is untested. All reported evaluations use synthetic benchmarks; the authors explicitly list evaluation on appropriately governed real clinical records as future work, and note the validation gates check structure rather than semantic naturalness.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 is an image with no content in the extracted text; pipeline understanding relies on the §3.2 prose, which appears complete.
- Table 1 was extracted as linearized rows; all quoted values were cross-checked against the in-text numbers and are consistent, so misreading risk is low.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.