When Machine Unlearning Jeopardizes Privacy

Venue: ACM SIGSAC Conference on Computer and Communications Security (CCS) 2021 · arXiv: 2005.02205 (v2, 14 Sep 2021) Affiliations: CISPA Helmholtz Center for Information Security; CMU; UVA; U. Lausanne. Verification: dblp for venue; PDF fetched.

Read status: abstract and pp. 3–4 read — §2.3 Threat Model, §3.1 Attack Pipeline, §3.2 Attack Model Training, and the opening of §3.3 Feature Construction. §4–§7 (feature methods in detail, experiments, four defences) not read.

1. The central claim

Unlearning creates two models — the original \mathcal{M}_o and the unlearned \mathcal{M}_u — and the discrepancy between them is itself a leak.

"While originally designed to protect the privacy of the data owner, we argue that machine unlearning may leave some imprint of the data in the ML model and thus create unintended privacy risks… we show that our attack in multiple cases outperforms the classical membership inference attack on the original ML model, which indicates that machine unlearning can have counterproductive effects on privacy."

And the sting:

"the privacy degradation is especially significant for well-generalized ML models where classical membership inference does not perform well."

So the models on which ordinary MIA fails — the ones a practitioner would call safe — are where unlearning does the most damage.

2. Threat model — and why it does not match ours

Adversary's knowledge. Black-box access to both \mathcal{M}_o and \mathcal{M}_u. Justified as realistic for MLaaS: "the target black-box model can be queried at any time, such as in the setting of MLaaS, and all of the query results can be stored locally by the adversary."

Adversary's goal. Given target sample x, infer whether x is in the original model's training set and out of the unlearned model's — the \langle \text{in}, \text{out}\rangle case. Three other cases are treated as negatives or handled by classical MIA: \langle \text{out},\text{out}\rangle, \langle \text{in},\text{in}\rangle, \langle \text{out},\text{in}\rangle (the last arises under incremental learning).

An elegant detail: "when there are no changes on the target sample's outputs from two consecutive queries, then the unlearning did not happen and the adversary does not need to launch the attack." The attack is triggered by observing that something changed.

This is a different threat model from everything else in our evaluation stack

Our Stage 6 harness scores the unlearned model alone. Membership Inference Attacks From First Principles and Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy also attack a single model. Chen et al. attack the pair, and report it can beat classical MIA on the original.

Consequence: a good result under our harness does not rule out this leak, because our harness cannot see it by construction. This is a scope limitation of the evaluation to state in docs/specs/EVALUATION.md, not a defect to fix — implementing a two-model attack with shadow models is out of scope for Phase 1/2, and it is a different research question from reproducing SCRUB.

It does, though, strengthen the case for storing both the original and unlearned checkpoints with full provenance, which our registry already does. A future two-model attack would then be runnable against archived artifacts without re-running anything.

3. Attack pipeline (§3.1)

Three phases:

1. Posteriors generation. Query \mathcal{M}_o and \mathcal{M}_u with target x, obtaining confidence vectors \mathbb{P}_o and \mathbb{P}_u.
2. Feature construction. Aggregate the two posteriors into a feature vector \mathcal{F}. The straightforward choice is concatenation \mathbb{P}_o \| \mathbb{P}_u; five representative methods are compared in their §3.3 (unread).
3. Inference. A binary classifier \mathcal{M}_A maps \mathcal{F} to a membership decision.

Attack model training (§3.2). The adversary holds a shadow dataset \mathcal{D}^s, possibly from a different distribution than the target's. Partition it into shadow negative \mathcal{D}^s_n and shadow positive \mathcal{D}^s_p. Train the shadow original \mathcal{M}^s_o on \mathcal{D}^s_p; for deletion requests \mathcal{R}_p = \{x^1_p,\dots,x^m_p\}, train shadow unlearned models \mathcal{M}^{s,i}_u on \mathcal{D}^s_p \setminus x^i_p.

Positive cases are the samples in \mathcal{R}_p — member of the shadow original, non-member of the shadow unlearned. Negative cases \mathcal{R}_n are drawn from \mathcal{D}^s_n, balanced so |\mathcal{R}_p| = |\mathcal{R}_n|. Diversity is improved by training multiple shadow originals on random subsets. Attack models: logistic regression, decision tree, random forest, multi-layer perceptron.

Note the shadow-model cost: one shadow original plus one shadow unlearned model per deletion request. This is the same order of expense as LiRA, and part of why a two-model attack is out of scope for us.

4. Relation to SISA

Their §2.3 describes SISA explicitly as the ensemble-style efficient unlearning method they evaluate against, using posterior average as the aggregation strategy in their experiments. So this paper attacks Machine Unlearning among others — relevant because SISA is exact unlearning, and the attack still applies: the leak is in the difference between two released models, not in any approximation error.

That is a genuinely interesting point for the review's theory section. Exactness of unlearning does not by itself defeat this attack, because the attack does not exploit imperfect forgetting — it exploits the observable fact that forgetting occurred.

5. §5 — the setup, and two results that change how this paper reads here

The experimental frame. Each dataset is "evenly split into disjoint target dataset D_t and shadow dataset D_s"; the attack model is trained on positive and negative feature sets using "four standard and widely used classifiers … Logistic regression, decision tree, random forest, and multilayer perceptron". Three metrics: AUC, plus their two proposed degradation measures, DegCount and DegRate.

Targets. Categorical datasets against simple models, image datasets against convolutional networks — "the SimpleCNN model for MNIST … the ResNet50 and DenseNet models for CIFAR10". Insta-NY and STL10 appear only in the data-transferring experiment of §5.6.

Numbers are deliberately not recorded here. Only two sets of experimental results are transcribed anywhere in this project, and this is not one of them. Every result below is the authors' qualitative statement.

Result 1 — the attack is worst where classical membership inference is weakest. The counter-intuitive finding, and the one that makes the paper matter: privacy degradation "is especially significant for well-generalized ML models where classical membership inference does not perform well". An attack that only works on overfitted models is a curiosity; one that works precisely where the standard attack fails is a real threat model.

Result 2 — deleting a group blunts the attack. §5.5 studies deleting a group of samples together rather than one:

"the attack performance of group deletion is slightly worse than single sample deletion … The reason is that a single sample could be hidden among the group of deleted samples, thereby preserving its membership information. This result reveals that conducting group deletion could mitigate, to some extent, the impact of our attack."

This sharpens §2's conclusion rather than repeating it

§2 records that the threat model does not match this project's, because Chen et al. assume an adversary with access to two model versions — before and after — and this project releases one. That argument stands on the interface.

The group-deletion result adds a second, independent reason, and it is about the setting rather than the interface: this project's forget sets are groups, not single samples, and the attack's power is reported to fall as the deleted group grows. So even an adversary who did obtain both versions would be attacking the configuration the paper itself identifies as its weakest.

The two arguments should be kept apart in the review. One says the adversary cannot get what the attack needs; the other says the attack is weaker against what we would be deleting. Only the first is structural — an implementation that ever published intermediate models would lose it, and then the second is all that remains.

A third scenario worth noting for Gap 2. §5.4 studies "multiple intermediate unlearned models", sweeping the number of released intermediates over \{1, 10, 50, 100, 250\}. That is a release-over-time setting, adjacent to the sequential-request question, and it is a second instance — with SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation's iterative experiment — of the literature touching the sequential case from a direction that is not method evaluation.

6. Defences — and the one that fails

Four mechanisms are proposed and evaluated: "(1) publishing only the top-k confidence values of the posterior vector, (2) publishing only the predicted label, (3) temperature scaling, and (4) differential privacy." They divide cleanly: "the former two mechanisms reduce the information accessible to the adversary … the latter two eliminate the impact of a single sample on the output of the ML models."

The result is asymmetric, and the failure is the interesting half:

"our attack is robust to the top-k defense, even when the model owner only releases the top-1 confidence value. On the other hand, publishing only the predicted label, temperature scaling, and differential privacy can effectively prevent our attack."

Why the top-k failure matters. Truncating the posterior is the cheap, obvious mitigation — and it does not work, down to a single confidence value. What works is either removing the confidence entirely or changing what the model computes. That is a sharper statement than "withhold the posterior", and it is worth carrying into any discussion of what an unlearning service should expose: the attack lives in the value of the top confidence, not in the tail of the distribution.

7. What the paper does not do (from sections read)

• §4–§6 experiments unread; no claimed numbers recorded.
• The five feature-construction methods are named but not read.
• The four defences are not characterised.

8. Links

• Attacks the two-model discrepancy, a threat model orthogonal to Membership Inference Attacks From First Principles and Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy, both of which attack a single model.
• Evaluates against Machine Unlearning; the attack survives exact unlearning.
• Cited by Towards Unbounded Machine Unlearning in its LiRA-for-unlearning discussion, for the idea of producing shadow unlearned models by applying the unlearning algorithm to shadow models.
• Its "well-generalized models leak more under unlearning" finding pairs with Membership Inference Attacks From First Principles's insistence on evaluating attacks against well-generalising models rather than overfitted ones.

@inproceedings{chen2021jeopardizes,
  title         = {When Machine Unlearning Jeopardizes Privacy},
  author        = {Min Chen and Zhikun Zhang and Tianhao Wang and Michael Backes and Mathias Humbert and Yang Zhang},
  year          = {2021},
  booktitle     = {ACM Conference on Computer and Communications Security (CCS) 2021},
  eprint        = {2005.02205},
  archiveprefix = {arXiv},
  url           = {https://arxiv.org/abs/2005.02205}
}