Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks

Venue: CVPR 2020 · arXiv: 1911.04933 Verification: dblp for venue; PDF fetched.

Read status: pp. 3–6 read from page images — §1.4, §2 Definition and Testing of Forgetting (Def. 1, Prop. 1, Lemma 1, Example 1, Eq. 4), §2.1 (Prop. 2, Cor. 1), §3 Optimal Quadratic Scrubbing Algorithm (Props. 3–4, Eqs. 6–7), §3.3 Hessian/FIM, §4 Deep Network Scrubbing, §5–5.2 experimental setup. Results and appendices not read.

This is SCRUB's Fisher baseline. It is also the source of the Lacuna datasets, the All-CNN variant, and the pretraining recipe that SCRUB inherits.

1. The definition of forgetting

Definition 1 (Optimal scrubbing). Given a readout function f, an optimal scrubbing function for f is S(w; \mathcal{D}_f) — written S(w) — such that there is another function S_0(w) that does not depend on \mathcal{D}_f for which

Their footnote is worth keeping — it is the same degenerate-solution problem that recurs everywhere in this literature: "If S_0 could depend on \mathcal{D}_f, we could take S(w) = w to be the identity, and let S_0(w) ignore w and obtain new weights by training from scratch… This brings the KL to zero, but does not scrub any information."

Proposition 1 connects this to membership inference. With Y an attribute depending on \mathcal{D}_f:

"if that quantity is zero then… we cannot predict with better-than-chance accuracy whether the model w' = S(w) was trained with or without the data. In other words, after forgetting, membership attacks will fail."

Lemma 1 removes the dependence on the attacker's readout function: for any f, \mathrm{KL}(P(f(S(w))|\mathcal{D}) \| P(f(S_0(w))|\mathcal{D}_r)) \le \mathrm{KL}(P(S(w)|\mathcal{D}) \| P(S_0(w)|\mathcal{D}_r)), so it suffices to minimise the weight-space KL (Eq. 3) — which "guarantees robustness to any readout function."

Example 1 shows pure noise works and why it is useless: S(w) = w + \sigma n drives the KL to 0 as \sigma \to \infty, but "throws away the baby along with the bath water, rendering the model useless."

Equation 4 — the Forgetting Lagrangian, which is the whole paper in one line:

Two stated assumptions (§2.1), both of which matter for whether any of this transfers: (1) \mathcal{D}_f is a small portion of \mathcal{D}, "lest one is better-off re-training than forgetting"; (2) the training process A(\mathcal{D},\epsilon) is stable — if \mathcal{D} and \mathcal{D}' differ by a few samples, the outcomes are close. SCRUB's §1 attacks exactly this stability assumption as violated by deep networks.

Corollary 1 (Gaussian forgetting). With S(w) = h(w)+n, S_0(w) = w+n', n,n' \sim \mathcal{N}(0,\Sigma):

giving three routes to forget: move w toward w', add noise along h(w)-w', or both.

2. The quadratic theory

Proposition 3 (Optimal quadratic scrubbing algorithm). Assuming L_\mathcal{D} and L_{\mathcal{D}_r} are quadratic and optimisation is gradient flow, the scrubbing function

with A = \nabla^2 L_\mathcal{D}(w), B = \nabla^2 L_{\mathcal{D}_r}(w), d = A^{-1}\nabla_w L_\mathcal{D}, d_r = B^{-1}\nabla_w L_{\mathcal{D}_r}, satisfies h(A_t(\mathcal{D},\epsilon)) = A_t(\mathcal{D}_r,\epsilon) for all seeds and times, and scrubs the model clean: \mathrm{KL}(P(S(w)|\mathcal{D},\epsilon)\|P(w|\mathcal{D}_r,\epsilon)) = 0.

As t \to \infty this reduces to the Newton update S_\infty(w) = w - B^{-1}\nabla L_{\mathcal{D}_r}(w).

Proposition 4 (Robust scrubbing) adds noise optimally: with h(w)-w' \sim \mathcal{N}(0,\Sigma_h), the Lagrangian-minimising noise satisfies \Sigma B \Sigma = \lambda\Sigma_h; isotropic error \Sigma_h = \sigma_h^2 I gives \Sigma = \sqrt{\lambda\sigma_h^2}\,B^{-1/2}. Combining:

\lambda trades residual information against retain accuracy; \sigma_h "reflect[s] the error in approximating the SGD behavior with a continuous gradient flow."

§3.3 — Hessian → Fisher. The Hessian is "too expensive to compute for a DNN… we cannot even ensure it is positive definite", so they use the Levenberg–Marquardt semi-positive-definite approximation

which "coincides with the Fisher Information Matrix (FIM)… this approximation is exact for some problems, such as linear (logistic) regression."

3. What is actually implemented — and it is much simpler than the theory

§4, "Fisher forgetting" — this is the algorithm to implement for Stage 7, and the gap between it and §3 is large enough that implementing from §3 would be wrong:

"In our experiments, we find that the diagonal is not a good enough approximation of B for a full Newton step h(w) = w - B^{-1}\nabla L_{\mathcal{D}_r}(w) in eq. (7). However, the diagonal is still a good approximation for the purpose of adding noise. Therefore, we simplify the procedure and take h(w) = w, while we still use the approximation of the FIM as the covariance of the noise. This results in the simplified scrubbing procedure:

S(w) = w + (\lambda\sigma_h^2)^{1/4}F^{-1/4}

where F is the FIM (eq. 8) computed at the point w for the dataset \mathcal{D}_r."

So Fisher Forgetting, as run, is noise addition with FIM-shaped covariance and no Newton step at all. They are explicit that this "relies on w and w' already being close, which hinges on the stability of SGD" — precisely the assumption SCRUB argues is violated. Interpretation offered: "adding noise to destroy the weights that may have been informative about \mathcal{D}_f but not \mathcal{D}_r."

⚠️ AMBIGUITY F1 — the printed equation is missing its noise sample

S(w) = w + (\lambda\sigma_h^2)^{1/4}F^{-1/4} has no random vector on the right-hand side; as written it is a deterministic shift by a matrix power, which is dimensionally incoherent as a weight update and contradicts the surrounding text ("the FIM as the covariance of the noise") and Eq. 7, which does carry \epsilon. The intended reading is almost certainly

with F diagonal, so F^{-1/4} is elementwise. Settled against the authors' public code, which SCRUB says it used for Fisher "for correctness" — and the reading was right: the noise vector is present in both renderings of the paper and in the implementation, which scrubs with p.data = mu + var.sqrt() * torch.empty_like(p.data0).normal_(). Only the later simplified line omits \epsilon.

A real discrepancy came out of the same reading. The code applies the square of the paper's Fisher dependence: the printed noise scale is (\lambda\sigma_h^2)^{1/4}F^{-1/4}, the implemented one is \sqrt{\alpha/F}. For reproduction the repository wins — an implementation of the printed exponent would not produce SCRUB's Fisher row. Verified in two independent renderings before being recorded, because a false accusation against a paper we are reproducing is worse than no finding. See docs/REPRODUCTION.md §3a.

Other ambiguities for the spec

• F2 — FIM computed on which data? The text says "for the dataset \mathcal{D}_r". Confirmed by the code: hessian(...) is called three times, always as hessian(retain_loader.dataset, ...). The retain set, as the paper says. Resolved.
• F3 — diagonal vs Kronecker-factored. They mention K-FAC as "a better" option but report using the diagonal. Diagonal is what to implement. Resolved from the paper.
• F4 — \lambda and \sigma_h. \lambda is the forgetting/accuracy trade-off knob; no value is given in the pages read. Settled against the code, and they never appear separately: the two collapse into a single knob alpha, which is the sweep axis — 3e-6 as the signature default, 1e-6 for the notebook's Fisher model, 1e-9 for the second variant. Resolved.
• F5 — numerical floor on F. F^{-1/4} diverges wherever a Fisher diagonal entry is zero, which happens for dead units. Settled against the code: a floor and three clamps, none of which appear in the paper — beginning with var = 1.0 / (p.grad2_acc + 1e-8). A reimplementation from the paper alone would omit all of them. Resolved; see docs/REPRODUCTION.md §3a.

Note what F4 and F5 have in common: neither is a case of the paper being unclear. In both, the implementation contains machinery the paper does not mention at all, which is a different kind of reproduction hazard and one that only reading the code can surface.

Variational forgetting is the paper's second variant: instead of the FIM, optimise the noise covariance directly via the proxy Lagrangian \mathcal{L}(\Sigma) = \mathbb{E}_{n\sim\mathcal{N}(0,\Sigma)}[L_{\mathcal{D}_r}(w+n)] - \lambda\log|\Sigma|, using the local reparametrisation trick. Not in SCRUB's table — out of scope.

4. Setup — which SCRUB inherits wholesale

Datasets. MNIST, CIFAR-10, and Lacuna-10 / Lacuna-100, which this paper introduces: "faces of 10 and 100 different celebrities from VGGFaces2 [6]". Forget either an entire class or a hundred images of a class.

🔎 This resolves the VGG-Faces citation finding

Golatkar — whom SCRUB explicitly follows for Lacuna — attributes the source to VGGFaces2, i.e. Cao, Shen, Xie, Parkhi & Zisserman (2018), whose first author is Qiong Cao. SCRUB's bibliography contains only Yinzhi Cao & Junfeng Yang (2015), the unlearning paper. So SCRUB's "[Cao and Yang, 2015]" for VGG-Faces is a collision between two different Caos — the citation key resolved to the wrong entry.

This upgrades the earlier finding from "cites the wrong paper" to a specific, explicable mechanism, and it is now supported by a second source rather than only by SCRUB's own bibliography. Updated in VERIFICATION_REPORT.md.

Models and training. "a small All-CNN (reducing the number of layers), to which we add batch normalization before each non-linearity." Pretrain on Lacuna-100/CIFAR-100 for 15 epochs, SGD, fixed lr 0.1, momentum 0.9, weight decay 0.0005. Fine-tune on Lacuna-10/CIFAR-10 with lr 0.01.

A BatchNorm detail that may propagate to SCRUB. "To simplify the analysis, during fine-tuning we do not update the running mean and variance of batch normalization, and rather reuse the pre-trained ones." SCRUB says it uses "the same models with the same architectural modifications in [Golatkar et al. 2020a,b]" — but says nothing about BN statistics. Whether SCRUB also freezes BN running stats is ambiguity A7 in kurmanji2023scrub.md. It was the first evidence on it, and the SCRUB repository has since confirmed the training-time half: originals and oracles do freeze BN running statistics. During SCRUB's own unlearning steps they update.

Note SCRUB pretrains for 30 epochs where Golatkar uses 15; the recipes are close but not identical, so do not substitute one for the other.

Baselines (§5.2). Fine-tune ("akin to catastrophic forgetting… more so because of the larger learning rate") and Negative Gradient — the plain NegGrad that SCRUB later describes as the weaker predecessor of NegGrad+. Two further baselines follow in unread text.

5. What the paper does not test (from sections read)

• Results tables unread; no claimed numbers recorded.
• The quadratic/gradient-flow theory of §3 is not what §4 implements; the gap is acknowledged but its cost is not quantified in the pages read.
• Stability of SGD is assumed, not verified — the assumption SCRUB attacks.
• \mathcal{D}_f assumed small; large-forget-set behaviour is out of scope by assumption.

6. Links

• SCRUB's Fisher baseline; SCRUB used the authors' public code, so our registry entry must carry implementation: authors_code if we do the same.
• Introduces Lacuna, which this project drops, and the All-CNN + pretraining recipe SCRUB inherits.
• Companion paper Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations (NTK), also in SCRUB's table.
• Its stability assumption is the target of Towards Unbounded Machine Unlearning's critique.
• Independently reported as infeasible at CIFAR scale by Towards Adversarial Evaluations for Inexact Machine Unlearning.

@inproceedings{golatkar2020eternal,
  title         = {Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks},
  author        = {Aditya Golatkar and Alessandro Achille and Stefano Soatto},
  year          = {2020},
  booktitle     = {CVPR 2020},
  eprint        = {1911.04933},
  archiveprefix = {arXiv},
  url           = {https://arxiv.org/abs/1911.04933}
}