Fast Yet Effective Machine Unlearning

Venue: IEEE Transactions on Neural Networks and Learning Systems (TNNLS) 2024 · arXiv: 2111.08947 Verification: dblp for venue and year; PDF fetched. (Supervisor's notes had TNNLS 2023; dblp records the journal article as 2024 — see VERIFICATION_REPORT.md §1.)

Read status: printed pp. 4 and 9 read from page images / text — §IV Error-Maximizing Noise based Unlearning (Eq. 1), §IV-A, §IV-B UNSIR impair/repair, §V experimental setup, §VI scope limitations, §VII Conclusion opening. §I–III, §V results tables and §VI body not reads.

Project status: Priority 2, gated. But see §1 — it may not be applicable to our settings at all.

1. ⚠️ UNSIR excludes our Phase 2 setting, by the authors' own statement

"Unlearning a random cohort of data is beyond the scope of this work. Although, in theory an error-maximizing noise matrix can be generated corresponding to the random samples. But this would hurt the zero-glance assumption and thus, unlearning random samples, or only a subset of a class is out of the scope of this paper."

Our Phase 2 target is selective unlearning — 100 examples of class 5, which is precisely "only a subset of a class". And Phase 1's small-scale setting forgets 25 examples of class 0, also a subset of a class.

UNSIR does not apply to either target setting

It is a class-unlearning method only — "unlearn a single or multiple class(es) of data". Class unlearning sits outside the default scope of this project's second phase, so as the plan stands there is no setting in which UNSIR could be run.

What follows from that. Three positions are available. UNSIR can be dropped for structural non-applicability, as SISA was. It can be retained conditionally, entering scope only if class unlearning does. Or the generalisation its authors gesture at ("in theory an error-maximizing noise matrix can be generated corresponding to the random samples") can be implemented — but that is new method development rather than reproduction, and it would break the zero-glance property the paper is built around.

This project takes the second position. It is the only one that neither discards a paper from the reading list on grounds the paper cannot answer, nor attributes to UNSIR a method its authors did not publish.

2. The method

Zero-glance is the organising constraint. UNSIR never looks at the forget samples. It needs only the forget class label and a small subset of retain data. That is why the noise exists at all — it is a stand-in for data the method is forbidden to see.

Equation 1 — error-maximizing noise:

where \mathcal{L}(f,y) is "the classification loss corresponding to the class to unlearn", f the trained model, w_{noise} the noise parameters ("can be interpreted as pixel values in terms of an image"), and \lambda trades the two terms off. "The optimization problem finds the L_p-norm bounded noise that maximizes the model's classification loss. In our method, we use a Cross-Entropy loss function \mathcal{L} with L_2 normalization."

• \mathcal{N} has the same size as the model input, initialised from \mathcal{N}(0,1).
• The pretrained model's weights are frozen during noise generation.
• Goal: "create a correlation between \mathcal{N} and the unlearning class label."
• For multiple classes, "we learn the noise matrix \mathcal{N} for each class separately."

Why the regulariser is not optional — a nice failure mode to record: "The second term \lambda\|w_{noise}\| … is proposed to regularize the overall loss by preventing the values in \mathcal{N} from becoming too large. Without this regularization of \mathcal{N}, the model will start believing that images with higher values belong to the unlearn class."

§IV-B — impair then repair:

• Impair: train on \mathcal{D}_{r\_sub} \cup \mathcal{N} for 1 epoch at a high learning rate. "it corrupts those weights in the network which are responsible for recognition of the data in forget class(es) … almost always only a single epoch of impair is sufficient."
• Repair: train for 1 epoch on \mathcal{D}_{r\_sub} only, to restore retain-class performance. "on rare occasions, more epochs may be required."

\mathcal{D}_{r\_sub} is "a small subset of samples drawn from the retain dataset" — so UNSIR needs neither the full retain set nor any of the forget set.

They also claim a relearn-time criterion: "The final updated model has high relearn time, i.e., it takes substantial number of epochs for the network to relearn the forget samples. This is one of the important criteria for effective unlearning." That is a distinct evaluation axis from anything in our Stage 6 plan, and worth recording in taxonomy.md.

3. Setup, and a fifth VGGFace data point

Networks: ResNet-18, AllCNN, MobileNetv2, Vision Transformers. Datasets: CIFAR-10, CIFAR-100, and VGGFace-100 — "100 face IDs collected from the VGGFaces2 [52]".

This adds a fifth attribution to the VGG-Faces tangle, and it conflicts with Foster's:

Source	Says the face data is
Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks	VGGFaces2
Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations	VGG-Faces
UNSIR (here)	VGGFaces2
Fast Machine Unlearning without Retraining through Selective Synaptic Dampening	VGG-Face (Parkhi, Vedaldi & Zisserman 2015) — and attributes this to Golatkar 2020a, which actually says VGGFaces2
Towards Unbounded Machine Unlearning	"[Cao and Yang, 2015]" — the unlearning paper

So Foster appears to mis-state what Golatkar 2020a used. Recorded in VERIFICATION_REPORT.md §5b as further evidence that the secondary literature is collectively unreliable on this point, and the review should say exactly that rather than pick a winner. Adjudicating it would require fetching Parkhi 2015 and Cao 2018 directly, which has not been done.

Efficiency claim: "the efficiency gain in the proposed method is 154× and 1875× more than the retrain and Fisher method, respectively. However, this efficiency gain comes at the cost of decreased accuracy in comparison to the retrain method." Creditably explicit about the trade-off.

Also out of scope, by their own statement: "the analysis of adaptive adversaries, that have exact knowledge about the proposed algorithm, is out of scope as well."

4. Authorship overlap with Bad-T

The same four authors — Tarun, Chundawat, Mandal, Kankanhalli — appear on this and on Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks Using an Incompetent Teacher, in different orders. Verified from the recorded author lists. Same caveat as SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation/Model Sparsity Can Simplify Machine Unlearning: their mutual comparisons are not independent external validation.

5. What the paper does not do (from sections read)

• §I–III, §V results tables and most of §VI unread; no reported accuracy is recorded beyond the qualitative efficiency claim quoted above.
• Random-sample and subset-of-class unlearning explicitly out of scope (§1 above).
• Adaptive adversaries explicitly out of scope.
• \lambda has no stated value in the sections read; impair/repair learning rates vary by dataset (e.g. 0.02/0.01 for one setting, 0.01/0.001 for VGGFace-100), so any spec must pin them per-setting rather than assume one pair.

6. Links

• Same four authors as Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks Using an Incompetent Teacher; UNSIR is the earlier method and Bad-T the later one from the same group.
• Its zero-glance constraint puts it closer to When to Forget? Complexity Trade-offs in Machine Unlearning's retain-only algorithm class than SCRUB, NegGrad+ or Bad-T, all of which use the forget set directly.
• Compared against Fisher Forgetting (Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks) for efficiency.
• Its relearn-time criterion is an evaluation axis absent from our Stage 6 plan.

@article{tarun2024unsir,
  title         = {Fast Yet Effective Machine Unlearning},
  author        = {Ayush K. Tarun and Vikram S. Chundawat and Murari Mandal and Mohan S. Kankanhalli},
  year          = {2024},
  journal       = {IEEE Transactions on Neural Networks and Learning Systems (TNNLS)},
  eprint        = {2111.08947},
  archiveprefix = {arXiv},
  url           = {https://arxiv.org/abs/2111.08947}
}