Extracting Forgotten Prompts from Targeted Unlearned Models
Au Ashley Hoi-Ting, Meghdad Kurmanji, William F. Shen, Nicholas D. Lane, Ligang He
cs.LG
Quick verdict
Claims unlearned prompts are discoverable by black-box refusal probing (100% entity ID, ~95% reconstruction); novel, broad eval, but headline rests on lenient judging and single-target setups.
Summary
This machine-unlearning security paper argues that the *prompts* targeted by unlearning can themselves be recovered, not just their answers. It formalizes forget-prompt discovery: an attacker with only black-box text access and the retained dataset must identify forgotten entities and reconstruct their prompts from refusal footprints. The proposed attack, TAS (Sec. 3), builds candidates from retained prompts (entities plus canonical templates), scores refusals lexically and semantically, tracks Beta posteriors over entities and templates, and alternates warm-up coverage, Thompson sampling, and a confirmation phase under a fixed budget. Across DPO/NPO/LUNAR unlearning of Llama2-7B, Llama3-8B, and Gemma-7B on DUSK, TOFU, and PISTOL, TAS attains 100% entity-identification accuracy, reconstructs up to ~95% of prompts, and uses up to 99.7% fewer queries than brute force (0.30% of TOFU's exhaustive space). It also shows conventional forget metrics (e.g., Forget ROUGE-1) fail to predict discoverability. Limitations: scoring uses Llama3-8B as judge with a lenient topical criterion, each run targets a single randomly chosen forget target, no empirical defense comparison, and the multi-entity extension shows degrading cardinality estimates.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 2
● low 4
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Forgotten prompts themselves — not just their answers — can be extracted from refusal-aligned unlearned LLMs using retained data and black-box text-only access (TAS).
- TAS achieves 100% entity-identification accuracy and reconstructs up to ~95% of forgotten prompts while using up to 99.7% fewer queries than brute-force probing (0.30% of TOFU's 804,138-query space).
- Conventional forget-set metrics (Forget ROUGE-1, Forget probability) do not predict black-box discoverability; answer suppression and target leakage are distinct failure modes.
- Different unlearning objectives leave characteristically different footprints: DPO/NPO give high recall/lower precision (0.966/0.586 and 0.943/0.530), LUNAR narrower but more precise (0.780/0.799) (Table 4).
- TAS extends to two simultaneously forgotten entities with perfect ranking (1.000 MAP) but degraded cardinality estimation (e.g., TOFU pair A: 0.833 entity recall).
Strengths
Genuinely new problem framing
Sec. 1-2 position forget-prompt discovery against prior attacks that presuppose knowledge of the forgotten prompt (membership tests, known-question answer recovery, or candidate-pool selection), and the threat model explicitly excludes forget set, pre-unlearning checkpoint, logits, and unlearning implementation.
Broad, systematic evaluation grid
Nine LLM×unlearning-method combinations per dataset across DUSK (Qspace 1,278), PISTOL (18,216), and TOFU (804,138), three seeds, with four baselines (Brute-force, Random, Greedy, UCB); UCB, the strongest, still reaches only 0.926/0.852/0.875 match accuracy while TAS reports 1.000 with zero variance (Table 3).
Ablations isolate each claimed component
Actionable negative finding for unlearning evaluation
Weaknesses
Headline 100% rests on a favorable, small setup
Prompt reconstruction scored by a lenient LLM judge
Efficiency headline maximized against a broken baseline
Defenses remain conceptual
Hand-curated refusal regexes are a brittleness risk
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Extracting Forgotten Prompts from Targeted Unlearned Models
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the TAS attack on unlearned LLMs?
TAS (Targeted Active Search) is a black-box attack that discovers which prompts were removed by LLM unlearning. It queries the model through its normal text API, builds candidate entities and question templates from retained data, and uses refusal-like responses as a signal to guide an adaptive search (warm-up, Thompson sampling, confirmation) that identifies forgotten entities and reconstructs the forgotten prompts.
How accurate is TAS and how many queries does it need?
In the paper's experiments across three unlearning methods (DPO, NPO, LUNAR), three models (Llama2-7B, Llama3-8B, Gemma-7B), and three benchmarks (DUSK, TOFU, PISTOL), TAS identified the forgotten entity in 100% of runs and reconstructed up to about 95% of forgotten prompts. With early stopping it used as few as 2,488 queries on TOFU — 0.30% of the 804,138-query exhaustive space, i.e., 99.7% fewer than brute-force probing.
Why does discovering forgotten prompts matter for privacy?
Because the fact that something was unlearned can itself be sensitive. If a clinical assistant was unlearned to hide an association between a patient and a medical treatment, an anomalous refusal on exactly the right entity discloses that association — even if the model never reveals the suppressed answer. The paper shows standard unlearning metrics, which only check whether answers are suppressed, do not detect this leakage.
Does unlearning really remove knowledge from a model?
This paper adds evidence that preference-based unlearning (DPO, NPO) and activation-redirection (LUNAR) suppress answers rather than erase knowledge: the refusal behavior they induce leaves a measurable footprint that lets an attacker recover what was targeted. NPO on PISTOL scored 0.000 Forget ROUGE-1 (answers fully suppressed) yet 96.1% of its forget prompts were still reconstructable.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review based on a sampled ~80k-character excerpt with three '[...sampling gap...]' regions: part of Sec. 3.3 (posterior details, covered by Appendix A.1 summary), part of Sec. 5.2 (Table 3/4 headers and some rows), and portions of Related Work and Appendix A/D — judgments avoid relying on gap-hidden content.
- Code or data availability is not stated anywhere in the provided text; codeAvailable=false reflects that absence, not a confirmed lack in the full paper.
- The strict-judge reconstruction results are not in the provided text (only the topical judge's prompts and numbers); strict-judge results may exist in a sampling gap.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.