Fast Machine Unlearning without Retraining through Selective Synaptic Dampening

Venue: AAAI 2024 · arXiv: 2308.07707 Verification: dblp for venue; arXiv API for the ID; PDF fetched.

Read status: Abstract, §Introduction, §Related Work, §Preliminaries, §Proposed Method (Eqs. 1–4, Algorithm 1) and §Experimental setup read. §Results and the appendices not read.

Extraction caveat. In the pinned copy, pdftotext drops the Greek symbols: \alpha and \lambda come out as empty strings. Any search for their values by text extraction returns nothing, and a reading that relies on it cannot settle them. This is why they remain open below — an extraction limit, recorded as one, not an absence in the paper.

Project status: Priority 2, gated to next semester. Notes are spec-grade because SSD shares Fisher machinery with the Priority 1 Fisher Forgetting method, so building one informs the other.

1. The problem the paper sets itself

SSD is positioned by a two-sided complaint about the field, stated in the abstract and sharpened in §1. Retrain-based methods work but carry a hidden requirement:

"they typically require some level of retraining over the retained data, in order to protect or restore model performance. This adds computational overhead and mandates that the training data remain available and accessible, which may not be feasible."

And the existing retrain-free alternatives fail on the other axis: they "are prohibitively computationally expensive and do not perform on par with their retrain-based counterparts."

SSD's claim is to occupy neither corner: "a novel two-step, post hoc, retrain-free approach to machine unlearning which is fast, performant, and does not require long-term storage of the training data."

The premise underneath the method is memorisation. §1 and §Preliminaries argue that overparameterised networks "memorize specific training examples and that parameters in later layers are highly specialized to specific features", so the model's information divides into generalized and specialized, and only the second is a privacy problem:

"The remaining information in the model is generalized and therefore not violating individual privacy (e.g., the ability to detect the shape of a person versus detecting Jane Smith)."

That is the whole argument for selectivity: if forgetting can be confined to specialised parameters, retain performance need not be repaired afterwards, and the retain set need never be touched.

The three stated contributions: a retrain-free selective method competitive with retrain-based ones; the framing of unlearning as selective, "in which only a small number of parameters should be modified to preserve model consistency"; and the storage property —

"SSD only needs access to the training data once to compute the FIM and can discard it afterwards, reducing storage requirements compared to retraining-based methods."

This puts a method paper on the theory track's storage axis

assumptions.yaml compares the theory papers on what an unlearner may retain between training and the deletion request: Remember What You Want to Forget: Algorithms for Machine Unlearning excludes storing the training set, Near-Optimal Pure Machine Unlearning for Smooth Strongly Convex Losses grants full storage, and the difference is one reason their rates do not compose. SSD is the method-side instance of exactly that axis. It stores a single diagonal FIM over \mathcal{D} and discards the data, which is neither of the two theory positions: more than nothing, far less than everything, and fixed in size rather than growing with |\mathcal{D}|.

That is a concrete answer to a question the theory track poses abstractly, and it is the kind of correspondence §5 of the review is looking for: not a bound that transfers, but a design axis that both literatures independently found load-bearing.

2. What the related-work section reveals

Two of its characterisations bear directly on records this project already holds.

(a) A fourth instance of the Ginart definition drifting from exact to approximate. SSD writes:

"Ginart et al. (2019) introducing a probabilistic definition of unlearning that requires the output distribution of a model that has unlearnt data to be similar to the output distribution of a model that was never trained on that data."

ginart2019deletion.md records that Ginart's Definition 3.1 is equality in distribution (=_d) — exact deletion, not similarity. The slippage here is small and precise: "similar to" where the source says equal. It is the same drift already documented from other secondary characterisations, now in a peer-reviewed AAAI paper, and it is further evidence for this project's rule that a definition is taken from the paper that states it and never from a paper that cites it.

(b) Two direct criticisms of methods in this corpus, both worth recording as edges rather than as background:

• Of Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks's Fisher Forgetting: "This is computationally very expensive and updates the whole model which causes significant degradation to the accuracy on the retain dataset, as shown in experiments of Tarun et al. (2023b)." Note the evidence is Fast Yet Effective Machine Unlearning's benchmarks, not SSD's own — so the criticism is inherited, and SSD says so.
• Of Amnesiac Machine Learning: storing "a list of samples and parameter updates for every training batch … adds significant overhead and cannot be performed post-hoc." This is a structural objection, not a performance one, and it is the same property that makes Amnesiac Unlearning awkward for a harness that starts from a frozen w^o.

SSD also inherits its design vocabulary from the survey in this corpus, which in turn inherits it from the field's first paper: it cites Nguyen et al. for completeness, timeliness and light-weightiness, and completeness/timeliness are Towards Making Systems Forget with Machine Unlearning's original two metrics. The 2015 framing is still doing work in 2024.

3. ⚠️ An independent confirmation that Lacuna is unobtainable

Buried in their experimental setup, and directly relevant to a decision this project already made:

"Golatkar, Achille, and Soatto (2020a) also use the VGG-Face dataset (Parkhi, Vedaldi, and Zisserman 2015), however, this is no longer accessible so we substitute it with the PinsFaceRecognition dataset (Burak 2020), which consists of 17,534 faces of 105 celebrities collected from Pinterest."

Three things follow.

(a) Our Phase 1/2 decision to drop Lacuna is independently vindicated by a peer-reviewed AAAI 2024 paper that hit the same wall and had to substitute. This is much stronger than our own inference from URL rot, and it belongs in docs/CHARTER.md's risk register and in REPRODUCTION.md as the justification for the deviation.

(b) There is a known substitute — PinsFaceRecognition, 17,534 faces of 105 celebrities — should a face dataset ever be wanted. Recorded, not adopted; CIFAR-only remains the plan.

(c) It adds a third data point to the VGG-Faces attribution puzzle, and the picture is genuinely tangled. Recorded as evidence, not resolved:

So even the two Golatkar papers differ from each other. Foster's attribution to Parkhi, Vedaldi & Zisserman (2015) is the first explicit author list any of these gives, and it is consistent with my earlier hypothesis — but a hypothesis confirmed by one secondary source is not a verified fact. VERIFICATION_REPORT.md should record the disagreement itself as the finding, and no claim about VGG-Face's authorship should enter the review until that paper is fetched directly and added to papers.yaml.

4. The method

Eq. 1 — Fisher information matrix, used as the importance measure:

Eq. 2 — naive pruning, rejected: zero every parameter with non-zero forget-set importance. "it is highly likely that [\;]_{\mathcal{D}_f} is greater than zero for a majority of parameters", so this destroys retain performance.

Eq. 3 — stricter selection, still binary:

Eq. 4 — the actual method: dampening, not pruning. The binary rule "limits the forgetting-performance trade-off"; a parameter slightly over the threshold is treated the same as one vastly more important to the forget set. So:

"if \lambda = 1 then \beta < 1 for all parameters that are specialized towards \mathcal{D}_f. Therefore, \beta \to 0 as a parameter becomes more specialized for \mathcal{D}_f. Since \beta scales this update, this dampening factor is given an upper bound of 1 to prevent large \lambda values from causing parameters to grow."

Algorithm 1: Selective Synaptic Dampening
Input: θ, D, D_f;  optional to skip step 1: []_D
Parameter: α, λ
 1: Calculate and store []_D once. Discard D.
 2: Calculate []_{D_f}
 3: for i in range |θ| do
 4:   if []_{D_f,i} > α·[]_{D,i} then
 5:     θ_i = min( λ·[]_{D,i}/[]_{D_f,i} · θ_i , θ_i )
 6:   end if
 7: end for
 8: return θ

Two hyperparameters: \alpha (selection threshold / protectiveness) and \lambda (dampening strength).

A deliberate substitution worth noting for the spec. Eq. 3 is written against [\;]_{\mathcal{D}_r} but Eq. 4 and Algorithm 1 use [\;]_{\mathcal{D}}:

"[\;]_{\mathcal{D}_r} is substituted with [\;]_\mathcal{D} in the selection step. This is because [\;]_{\mathcal{D}_r} must be recalculated for every new forget request. [\;]_\mathcal{D} can be calculated at any point after training before unlearning and only needs to be computed once, allowing for the training set to be discarded and only [\;]_\mathcal{D} stored… the solution remains accurate as typically |\mathcal{D}_f| \ll |\mathcal{D}|."

So SSD never needs the retain set at unlearning time — only the forget set and a precomputed full-data FIM. That makes it unusually cheap and is why the authors claim it is "post hoc, fast, retrain-free, selective in the parameters to be manipulated, and not reliant on additional models."

Implementation consequence for us: SSD and Fisher Forgetting both need a diagonal FIM, so a single tested unlearn/utils/fisher.py serves both — but note they compute it over different data. Golatkar's Fisher noise uses the FIM over \mathcal{D}_r; SSD needs FIMs over both \mathcal{D} and \mathcal{D}_f. The utility must therefore take the dataset as an argument rather than assuming one, which is an easy thing to get wrong when building it for Fisher first.

5. Setup

Models: ResNet-18 and Vision Transformer, following Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks Using an Incompetent Teacher. Datasets: CIFAR-20 (superclass), CIFAR-100, PinsFaceRecognition. Hardware: NVIDIA RTX 4090, Intel Xeon. Python 3, PyTorch, Ubuntu 20.04.6. Training: early stopping, multi-step LR scheduler from lr = 0.1, Adam. Three tasks: single-class forgetting, sub-class forgetting, and random-observation forgetting — the same three regimes our SplitSpec already supports. MIA: "the logistic regression MIA implementation from Chundawat et al. (2023a)" — i.e. the same family as SCRUB's basic MIA, and subject to the same Membership Inference Attacks From First Principles and Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy critiques.

6. What the paper does not do, and what could not be resolved

• The §Results section and all appendices are unread; no reported number recorded.
• \alpha and \lambda remain unresolved. They are the two hyperparameters the method cannot run without, and this reading could not settle them: the symbols do not survive text extraction from the pinned PDF (see the caveat above), Optuna appears in the bibliography but no search space is stated in the sections read, and unlike the other Priority 2 methods SSD's code is not pinned in external/MANIFEST.yaml. Settling them needs either page images of the results section or a pinned repository, and a spec cannot be written before that.
• The [\;]_{\mathcal{D}_r} \to [\;]_\mathcal{D} substitution is justified by |\mathcal{D}_f| \ll |\mathcal{D}|, which holds for our selective setting (0.25%) but is weaker for class unlearning (10%). Worth checking if class unlearning ever comes into scope.
• Their claim that the approach extends to repeated forget requests is explicitly a hypothesis: "We hypothesise that this is a valid approach for repeated forget requests."

7. Links

• Shares diagonal-FIM machinery with Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks, but computes it over different data.
• Follows Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks Using an Incompetent Teacher for models, forgetting tasks and the MIA implementation.
• Its dampening-not-pruning argument parallels the granularity concern in Are we making progress in unlearning? Findings from the first NeurIPS unlearning competition's erase phase.
• Criticises Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks on cost and retain degradation, on the evidence of Fast Yet Effective Machine Unlearning's benchmarks rather than its own.
• Criticises Amnesiac Machine Learning structurally: per-batch logging cannot be done post hoc.
• Mischaracterises Making AI Forget You: Data Deletion in Machine Learning's Definition 3.1 as approximate; recorded above.
• Inherits completeness/timeliness from A Survey of Machine Unlearning, which traces to Towards Making Systems Forget with Machine Unlearning.
• Its once-then-discard storage profile sits between Remember What You Want to Forget: Algorithms for Machine Unlearning and Near-Optimal Pure Machine Unlearning for Smooth Strongly Convex Losses on the storage axis in assumptions.yaml.
• Independently confirms the VGG-Face inaccessibility that motivated dropping Lacuna.

@inproceedings{foster2024ssd,
  title         = {Fast Machine Unlearning without Retraining through Selective Synaptic Dampening},
  author        = {Jack Foster and Stefan Schoepf and Alexandra Brintrup},
  year          = {2024},
  booktitle     = {AAAI 2024},
  eprint        = {2308.07707},
  archiveprefix = {arXiv},
  url           = {https://arxiv.org/abs/2308.07707}
}