Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy

Venue: arXiv:2403.01218, v1 2 Mar 2024, v3 21 May 2024. No journal-ref — not peer-reviewed. All authors at Google DeepMind. Verification: verified_fetched — arXiv abs page and the PDF.

Read status: pp. 2–7 read from page images — contributions, §2 Background, §2.1 Terminology, §2.2 Unlearning goals, §3 How should we evaluate privacy, Definition 3.1, §3.1 Population vs per-example MIAs, LiRA (Eqs. 1–3), §3.2 Population U-MIAs. §3.3 (their per-example U-MIA construction), §4 experiments and all results not read.

⚠️ Read the author list before reading anything else

Jamie Hayes and Eleni Triantafillou are two of SCRUB's four authors. This paper is SCRUB's own authors, two years later, arguing that the class of MIA used in SCRUB substantially overestimates the privacy protection that unlearning provides — and naming Kurmanji et al. explicitly among the algorithms affected.

This does not invalidate our reproduction. Tier A reproduces what SCRUB reported, using SCRUB's metrics. But it decisively shapes what may be concluded from those numbers, and it belongs in docs/REPRODUCTION.md as a stated limitation rather than being discovered by a panel member who happens to know this paper.

1. The core argument

Efficacy is measured by whether an adversary can distinguish an unlearned model from a retrained one. Adapted membership inference attacks are called U-MIAs.

"If the adversary mounting the U-MIA fails to distinguish between settings A and B, then the approximation achieved by the unlearning technique is a good one … However, in practice most of the U-MIAs considered in the literature are sub-optimal. We argue that for this reason, machine unlearning algorithms have, historically, a false sense of privacy due to benchmarking against sub-optimal U-MIAs. If the attack is far from optimal, the failure of the attack has little to say on the success of the machine unlearning approximation."

The mechanism they identify: "inexact machine unlearning algorithms implicitly or explicitly optimize for some average-case metric of unlearning. Hence, we obtain stronger U-MIAs by devising an attacker that creates decision rules on a sample by sample basis." Existing algorithms optimising average-case metrics "leave a large set of datapoints vulnerable to per-sample U-MIAs."

Demonstrated on ResNet-18 / CIFAR-10 and PaLM 2 — i.e. on our exact vision setting.

2. Definition 3.1 — the membership inference game on machine unlearning

Standard MIA game, with the unlearning modifications:

1. Challenger samples D_{\text{train}} \sim \mathcal{D}, trains \theta^o = A(D_{\text{train}}). Then chooses D_f \subset D_{\text{train}} and unlearns: \theta^f = U(\theta^o, D_f).
2. Challenger flips a fair coin b. If b=0, sample z \notin D_{\text{train}} from \mathcal{D}; if b=1, sample z \in D_{\text{train}} — specifically from D_f in the unlearning setting. Send z and \theta^o — \theta^f in the unlearning setting — to the adversary.
3. Adversary builds a decision rule h and outputs h(z), with access to \mathcal{D}, A and U. Wins if h(z) = b.

Note the adversary is given the unlearning algorithm U, not merely the model.

3. The categorisation that matters

Population MIAs (Shokri et al.). Train shadow models on subsets of the distribution; collect their outputs on training and test inputs labelled 1 and 0; train one binary classifier h on that pooled data. "a 'population MIA' in the sense that the binary classifier is trained on outputs produced across different examples from the data distribution."

Per-example MIAs (Ye et al.; Carlini et al.). "instantiating a dedicated attacker for each datapoint, with the aim of distinguishing the outputs of that datapoint when it was included or excluded from training… constructing an attack classifier h for each sample."

LiRA poses membership inference as hypothesis testing between two worlds; by the Neyman–Pearson lemma the uniformly most powerful test thresholds

with \mathbb{Q}_1 = \{\theta \leftarrow A(D_{\text{train}} \cup \{z\})\} and \mathbb{Q}_0 = \{\theta \leftarrow A(D_{\text{train}} \setminus \{z\})\}. Made tractable by (i) replacing the worlds with k shadow models each, (ii) replacing parameters with the model's output probability at z, logit-rescaled by \phi(w) = \log\frac{w}{1-w} (Eq. 2), and (iii) fitting Gaussians to the in/out shadow distributions:

Their justification for using an unrealistically strong attacker: "privacy is generally a worst-case property. To avoid a false sense of privacy, evaluations should instantiate attacks that approximate as closely as possible the optimal adversary."

4. Where SCRUB's MIA sits — stated explicitly

§3.2, in the Population U-MIAs paragraph:

"Kurmanji et al. [24] design a U-MIA that uses no shadow models. Specifically, let f(\cdot;\theta^f) be a model that trained on D_{\text{train}} … and then unlearned D_f … The test set D_{\text{test}} and forget set D_f are partitioned into subsets … such that all subsets have the same number of elements and there is no overlap …"

That matches SCRUB's own "Basic attack" description exactly — a binary classifier trained on the unlearned model's losses on forget versus test examples, evaluated on held-out losses.

So SCRUB's basic MIA is classified here as a population U-MIA, and a shadow-model-free one at that — the weakest category in the taxonomy. And the paper's headline claim is that "weaker population U-MIAs commonly used in the unlearning literature substantially overestimate the privacy protection of unlearning algorithms compared to a per-example U-MIA."

SCRUB does also contribute a LiRA-for-unlearning adaptation, which is a per-example attack and therefore in the stronger category. The criticism lands on the basic MIA, which is what Tables 2 and 21–28 report — including Table 23, our Phase 2 target.

5. Three findings that bear directly on Stage 6

(a) Unlearning can make the retain set less private.

"we show that many inexact machine unlearning algorithms increase the potential for privacy leakage of datapoints remaining in the training set after unlearning has concluded. For example, on the two aforementioned state-of-the-art unlearning algorithms [24, 21], there is an increase in privacy leakage after unlearning on over half of the training datapoints not selected for unlearning."

Reference [24] is Kurmanji et al. — SCRUB. [21] is Jia et al., the sparsity paper already in our database.

Our Stage 6 harness currently measures MIA vulnerability on the forget set only. If unlearning degrades privacy for retain-set points, a forget-set-only harness cannot see it. This is a concrete, cheap addition — run the same attack against retain-set examples — and it is worth doing before the harness freezes. Recorded as a Stage 6 candidate, not yet implemented.

(b) Per-example vulnerability varies enormously, and methods trade one example off against another. "Different unlearning algorithms lead to reducing the vulnerability of some (but not all) forget set examples, sometimes at the expense of increasing it for other examples." An aggregate MIA number can therefore look good while individual examples remain fully exposed. Any single-number forgetting metric hides this by construction.

(c) A single global stopping criterion cannot protect all examples — which is what SCRUB's rewinding is.

"We observe that different examples are unlearned at different rates. Therefore, lacking a procedure for selecting per-example hyperparameters and stopping criteria for the unlearning procedure, it is fundamentally challenging to equally protect all examples from membership inference attacks. We demonstrate empirically that a naive attempt at achieving this fails at improving privacy protection."

SCRUB+R rewinds to a single checkpoint chosen by one global forget-error criterion. On this argument that cannot equalise protection across examples, whatever it does to the average. This is a substantive, citable limitation of the very mechanism SCRUB introduces for the UP application — and it comes from SCRUB's own co-authors.

6. Their recommendation to the field

"we encourage the unlearning community to adopt more formal adversary definitions and threat models, so one can more clearly understand when U-MIAs are sufficiently powerful in the threat model of interest … one concrete direction that the community could consider is to derive formal bounds on the success of U-MIAs, as done for MIAs."

Worth pairing in the review with Machine Unlearning via Information Theoretic Regularization, whose Definition 2.5 bounds I(\hat Y_{\text{margin}};Z) — i.e. an information-theoretic bound on exactly this inference. Two papers, one asking for formal bounds on attack success and one supplying a candidate, is a coherent thread for the "what transfers?" section.

7. Consequences for this project — decisions, not just observations

1. docs/REPRODUCTION.md must state this limitation. Tier A reproduces SCRUB's basic MIA faithfully; that number should not be read as an unqualified privacy claim. The citation is SCRUB's own co-authors.
2. The LiRA deferral becomes more consequential. We deferred full LiRA on compute grounds (~128 GPU-hours/setting). This paper says the cheap attacks we are keeping are exactly the ones that overestimate protection. The deferral stands — the compute is real — but it must be reported as a known weakness of our evaluation, not a neutral scoping choice.
3. Add retain-set MIA to the Stage 6 candidate list (finding (a)). Cheap, and it detects a failure mode the current design is blind to.
4. Do not claim SCRUB "defends MIA" on the basis of reproduced basic-MIA numbers. The defensible claim is narrower: we reproduce the basic-MIA numbers SCRUB reports.

8. What the paper does not do (from sections read)

• §3.3, where their per-example U-MIA is actually constructed, is unread; no numbers from §4 are recorded anywhere.
• Not peer-reviewed. Cite as an arXiv preprint. Given the author overlap with SCRUB and the DeepMind affiliation this is credible, but the panel constraint distinction stands.
• They study ResNet-18/CIFAR-10 and PaLM 2 — the vision half matches our Phase 2 setting, which makes the findings unusually transferable for a critique paper.

9. Links

• Formalises and criticises the evaluation used by Towards Unbounded Machine Unlearning, by two of its own authors; also names Model Sparsity Can Simplify Machine Unlearning.
• Builds on Membership Inference Attacks From First Principles (LiRA) as the per-example exemplar, and on Shokri et al. for the population category.
• Converges with Towards Adversarial Evaluations for Inexact Machine Unlearning, Machine Unlearning via Information Theoretic Regularization and On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning on the theme that current evaluation practice is too weak — but from the attack-strength direction rather than the definitional one.
• The call for formal bounds on U-MIA success is answered in kind by Machine Unlearning via Information Theoretic Regularization's Definition 2.5.

@misc{hayes2024inexact,
  title         = {Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy},
  author        = {Jamie Hayes and Ilia Shumailov and Eleni Triantafillou and Amr Khalifa and Nicolas Papernot},
  year          = {2024},
  howpublished  = {arXiv preprint (no journal-ref as of verification)},
  eprint        = {2403.01218},
  archiveprefix = {arXiv},
  url           = {https://arxiv.org/abs/2403.01218},
  note          = {Not peer-reviewed}
}