PII-TRACE: A Benchmark for Context-Aware PII Detection in Multi-Turn LLM Conversations
Kaiyuan Zhang, Chuan Wang, Joey Zhong, Paul Fryzel, Kyle Polley, Jerry Ma, Ninghui Li
cs.CL · cs.AI
Quick verdict
First cross-turn PII benchmark (13k multilingual dialogues) + 0.6B detector leading consistent detection; LLM-made gold and in-domain training temper wins.
Summary
The paper introduces PII-TRACE, a benchmark of 13,148 synthetic multi-turn user–assistant dialogues (13 languages; 37,431 PII mentions across nine identifier types) with character-level spans and entity clusters linking recurring mentions, enabling evaluation of cross-turn consistent detection — whether every mention of a recurring identifier is found. The corpus is synthesized from production assistant traffic via an LLM-labeling, masking, paraphrase and surrogate-replacement pipeline with rule-based alignment gates and a second-LLM audit. Eleven baselines (Presidio, OpenAI Privacy Filter, GLiNER variants, Piiranha, OpenMed, four frontier LLMs) plus the authors' PII-Tracer, a 0.6B Qwen3-based bidirectional encoder with 37-label BIOES tagging, are evaluated on a 1,922-document test split. PII-Tracer achieves the best character F1 (0.629) and consistent detection (CD 0.853; multi-mention 0.794; cross-turn 0.776), while public specialized detectors flag most PII-free conversations (FP0 0.557–0.972) and frontier LLMs lose coverage as mention counts grow (Opus 4.8: 0.840 to 0.045 from 1 to 6–10 mentions). Sliding-window decoding lifts recall to 0.965. PII-Tracer also beats the OpenAI Privacy Filter on five external benchmarks (ai4privacy 0.950 vs 0.907). Main limitations: gold labels are LLM-generated with only partial human audit; training data shares production-traffic provenance with the benchmark; no significance testing; baselines use a single inference configuration; some non-English text retains untagged residual names.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- PII-TRACE is the first PII benchmark with explicit cross-turn mention chains for multi-turn conversations (Table 1).
- No evaluated public detector achieves strong consistent detection without flagging most PII-free conversations (FP0 0.557–0.972, Table 7).
- Single-pass reading loses about a third of gold characters on long dialogues (recall 0.687 at ≥10k characters, Table 5).
- PII-Tracer, a 0.6B detector, attains the highest entity-level coverage among twelve evaluated systems (CD 0.853, CD-m 0.794, xCD 0.776, Table 7).
- PII-Tracer transfers to standard single-record benchmarks, outperforming the OpenAI Privacy Filter on all five tested (Figure 4).
Strengths
Genuinely new evaluation axis
Cross-turn consistent detection via entity clusters (§3.1, Table 1) is absent from all listed prior benchmarks; even concurrent REDACT and RedactionBench lack mention chains, making the 'first' claim defensible.
Scale, diversity, and negatives
13,148 dialogues in 13 languages, lengths <1k to >100k characters, mixed-medium text (code/tables), and 7,503 PII-free conversations enabling FP0 measurement — most prior corpora lack clean negatives entirely.
Verification-gated synthesis
Instructive negative results
Transparent scoping
Weaknesses
Machine-generated gold labels
In-domain training advantage unquantified
No inferential statistics
Config-sensitive baseline conclusions
Incomplete detector coverage in key analysis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PII-TRACE (arXiv 2609.22200; Perplexity/Rutgers/Purdue) contributes two artifacts: a benchmark of 13,148 synthetic multi-turn dialogues (13 languages; 9,202/2,024/1,922 train/val/test; 5,645 PII-bearing, 7,503 PII-free) with character-level spans over nine identifier types (37,431 mentions) and entity clusters linking recurring mentions; and PII-Tracer, a 0.6B detector. Central claims: (1) it is the first benchmark with explicit cross-turn mention chains (Table 1); (2) no evaluated public detector combines strong consistent detection with low false positives on PII-free dialogues (FP0 0.557–0.972, Table 7); (3) single-pass reading drops roughly a third of gold characters on long dialogues (recall 0.687 at ≥10k chars, Table 5); (4) PII-Tracer leads all twelve systems on entity-level coverage (CD 0.853, CD-m 0.794, xCD 0.776, Table 7).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PII-TRACE?
A public benchmark of 13,148 synthetic multi-turn user–assistant conversations in 13 languages, with character-level PII spans over nine identifier types and clusters linking every mention of a recurring identifier. It is built to test whether PII detectors catch all occurrences of an identifier across conversation turns, not just isolated records.
What does 'consistent detection' mean in PII-TRACE?
An entity-level metric: a recurring identifier (e.g., a name or phone number appearing in several turns) counts as covered only if every one of its mentions in the conversation is detected. Missing even a single mention leaves that occurrence unprotected, so consistent detection is stricter than ordinary recall.
How does PII-Tracer compare with frontier LLMs like GPT-5.6?
On PII-TRACE's test split, the compact 0.6B PII-Tracer reaches character F1 0.629 and consistent detection 0.853, versus 0.611 and 0.675 for prompted GPT-5.6-sol. Frontier LLMs are more precise but miss far more mentions of recurring identifiers — Claude Opus 4.8's coverage drops from 0.840 to 0.045 as mention counts grow from 1 to 6–10.
Does PII-TRACE contain real user data?
No. Every released conversation is synthetic: real values were masked, the dialogue paraphrased, and fabricated format-matched surrogate values inserted, with rule-based leak checks and a second-model audit. Only the synthetic data is released; no source conversation is published.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.