PersianAnonymizer: Evaluating LLM-Labeled Training for Efficient NER-based Anonymization in Persian
Mohammad Hossein Shalchian, Mostafa Amiri, Amir Mahdi Sadeghzadeh
cs.CL
Quick verdict
Three LLM labelers compared for training a compact Persian anonymization NER; OSS_ZeroShot wins on LLM-made gold (macro-F1 0.851), human validation absent.
Summary
The paper targets privacy-preserving anonymization of Persian industrial customer chats by comparing three instruction-tuned LLMs — DeepSeek-V3-0324, GPT-OSS-120B, and Qwen3-235B-A22B-Instruct-2507 — as automatic labelers producing span annotations under a shared JSON protocol over a confidential 265K-message corpus (225K train / 40K test). Four LLM-labeled corpora result (OSS_ZeroShot, Qwen_ZeroShot, Qwen_FewShot, DeepSeek_FewShot), each used to fine-tune an identical MatinaRoberta token classifier. The OSS_ZeroShot-trained NER performs best on its own labeler's test annotations: macro-F1 0.851 and Label Coverage Recall 90.04%, versus 0.762/0.756 for the two Qwen variants and 0.733/80.04% for DeepSeek_FewShot (Table 3). Token-level Venn analysis shows only 5.46% of test tokens receive non-O labels from all three labelers (Table 6). On the human-annotated PEYMA benchmark, NER_OSS (macro-F1 0.8356) trails ParsBERT-NER (0.9324), while dominating on its own OSS-labeled test set (0.8334 vs 0.3320). The trained NER labels the 40K-message test set in ~2 minutes on one RTX 3090 versus multi-GPU H200 minutes for the LLMs. Main limitations: the evaluation gold standard is itself LLM-generated with no human adjudication, each model is scored against its own labeler's annotations (confounding the comparison), sparse classes (CREDIT_CARD, IBAN) are statistically unreliable, and neither data nor full hyperparameters are released.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Supervision from GPT-OSS-120B zero-shot yields the most learnable Persian anonymization NER, achieving macro-F1 0.851 and LCR 90.04%, ahead of the Qwen (0.762/0.756) and DeepSeek (0.733) variants (Table 3).
- Cross-labeler agreement is only partial: just 5.46% of test tokens receive non-O labels from all three LLMs, with 2.34% labeled exclusively by GPT-OSS (Table 6).
- The fine-tuned NER labels a 40K-message test set in ~2 minutes on a single RTX 3090, replacing minutes-long multi-GPU H200 LLM annotation at deployment (§5.4, Table 1).
- On the human-annotated PEYMA benchmark with shared labels, NER_OSS reaches macro-F1 0.8356 versus ParsBERT-NER's 0.9324, while dominating on its own OSS-labeled test set (0.8334 vs 0.3320) (Tables 4-5).
Strengths
Complete cost/latency accounting on both pipeline stages
Table 1 logs per-labeler annotation time on identical 8×H200 nodes (DeepSeek 92/22 min, Qwen 36/6.5 min, OSS 16/4 min for train/test), and §5.4 shows the trained NER labeling the 40K-message test set in ~2 min on one RTX 3090 — the deployment economics motivating the paper are measured, not asserted.
Controlled training-side comparison
All four corpora use the same backbone (MatinaRoberta), identical hyperparameters (Table 2), the same 225K/40K splits, and a shared JSON prompt/parse contract (§3.3), making the training comparison as fair as the design allows.
Multi-angle evaluation design
Candid limitation reporting
Documented alignment engineering
Weaknesses
LLM-gold circularity confounds the central comparison
Human-anchored evidence contradicts the headline
Model identity and prompting regime are entangled
No statistical rigor
Irreproducible in substance
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper (arXiv:2609.00958, cs.CL) targets anonymization of Persian industrial customer chats by training a compact NER model on LLM-generated supervision. Three instruction-tuned labelers — DeepSeek-V3-0324, GPT-OSS-120B, and Qwen3-235B-A22B-Instruct-2507 — annotate a confidential corpus of 265,000 messages (225K train / 40K test) under one shared JSON span protocol with 14 PII-oriented entity types (COST, CREDIT_CARD, DATETIME, EMAIL, IBAN, IP_ADDRESS, LOCATION, NUMBER, ORGANIZATION, PASSWORD, PERSON, PHONENUMBER, URL, USERNAME). Usable model×regime combinations yield four corpora (OSS_ZeroShot, Qwen_ZeroShot, Qwen_FewShot, DeepSeek_FewShot), each fine-tuning an identical MatinaRoberta token classifier. Central claims: (1) OSS_ZeroShot supervision yields the most learnable NER — macro-F1 0.851 and LCR 90.04% (Table 3); (2) cross-labeler agreement is only partial — 5.46% of tokens get non-O labels from all three LLMs (Table 6); (3) the deployed NER is dramatically cheaper — 40K test messages in ~2 minutes on one RTX 3090 versus minutes on 8×H200 nodes (§5.4, Table 1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PersianAnonymizer?
A pipeline from a 2026 arXiv paper (2609.00958) that trains a compact Persian named-entity-recognition model to anonymize customer chats. Instead of running a large LLM at inference time, the authors use LLMs (DeepSeek-V3-0324, GPT-OSS-120B, Qwen3-235B) only to generate training labels, then fine-tune a MatinaRoberta token classifier that runs on a single consumer GPU.
How fast is the trained Persian anonymization model?
The paper reports the trained NER labels a 40,000-message test set in roughly 2 minutes on one RTX 3090 (24 GB). By comparison, producing the same annotations with the source LLMs took several minutes on nodes with eight H200 GPUs.
Which LLM produced the best training labels for Persian NER in this study?
According to the paper, GPT-OSS-120B in zero-shot mode: the NER trained on its labels reached macro-F1 0.851 and 90.04% Label Coverage Recall, ahead of the Qwen3 variants (0.762/0.756) and DeepSeek-V3 (0.733). Important caveat: the evaluation gold standard was itself LLM-generated, and no human-adjudicated test subset was used.
Is the PersianAnonymizer dataset or code publicly available?
No. The 265,000-message industrial chat corpus is confidential and cannot be released, per the paper's ethics statement. Code release is not mentioned; the authors only say they may release the trained NER checkpoints.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2's per-class F1 values extract as an unordered number sequence, so specific per-class numbers could not be attributed to individual models; per-class statements here rely on §5.1 prose instead.
- Persian examples extract in Arabic presentation forms; quoted entity spans are faithful in content but not byte-exact.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.