Model Sparsity Can Simplify Machine Unlearning

Venue: NeurIPS 2023 · arXiv: 2304.04934 Code: https://github.com/OPTML-Group/Unlearn-Sparse Verification: dblp for venue; arXiv API for the ID; PDF fetched. Pagination: verified 1:1 — extraction page N = printed page N, confirmed from the page-number footers. (Unlike SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation, which is offset by +2.)

Read status: printed pp. 5–6 read from page images — Proposition 2 and its discussion, §4 Sparsity-Aided Machine Unlearning, pruning-method selection, "Sparsity-aware unlearning" (Eq. 3), and Table 2. §1–3, §5 experiments and appendices not read.

Project status: medium relevance, context only. Not a Tier A target — it is NeurIPS 2023, concurrent with SCRUB, and does not appear in SCRUB's comparison table.

1. ✅ Verified: SalUn's critique of \ell_1-sparse is fair, on Jia's own numbers

SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation criticises this paper by name:

"\ell_1-sparse may appear to be the strongest baseline in the 10% random data forgetting scenario when considering UA alone. However, this apparent strength comes at the cost of sacrificing RA and TA."

I said I would check that rather than repeat it. Jia's own Table 2 — random data forgetting, 10% across all classes, ResNet-18 / CIFAR-10, gaps against Retrain in parentheses:

The best scheduler achieves a UA gap of 0.06 and an MIA gap of 0.41 — essentially matching Retrain on the forgetting metrics — while carrying an RA gap of 2.61 and a TA gap of 3.16. Retrain reaches RA 100.00 / TA 94.42; \ell_1-sparse reaches 97.39 / 91.26.

Verdict: SalUn's critique is substantiated by Jia's own reported numbers. Near-perfect forgetting metrics, a ~2.6-point retain-accuracy cost and a ~3.2-point test-accuracy cost.

Two caveats recorded so the review states this precisely:

• SalUn's Table 1 and Jia's Table 2 are different experiments — SalUn re-ran the baselines. What is verified here is that the pattern SalUn describes appears in Jia's own numbers, not that SalUn's specific figures match Jia's.
• The trade-off is not hidden by Jia. Their RTE column shows 2.53 min versus Retrain's 42.15 — roughly a 17× speed-up. The paper is transparent that this is an efficiency/fidelity exchange; SalUn's point is about single-metric reading, not about concealment.

This is a useful worked example for the review's evaluation section: the same method looks state-of-the-art or mediocre depending on which column you read, and both papers are honest. It is the most concrete available argument for the multi-metric, gap-to-oracle reporting our Stage 6 harness plans.

2. The two contributions

(a) "Prune first, then unlearn." Sparsify the trained model, then apply an existing unlearning method. Motivated by Proposition 2, which bounds the unlearning error of gradient ascent:

with \sigma(\mathbf{m}) the largest singular value of the training-loss Hessian among unmasked dimensions. "the unlearning error reduces as the model sparsity in \mathbf{m} increases. By contrast, the unlearning error derived in Thudi et al. [8] for a dense model (i.e. \mathbf{m}=\mathbf{1}) is proportional to the dense model distance \|\theta_t-\theta_0\|_2."

They immediately qualify it: "in a high-sparsity regime, the model's generalization could decrease. Consequently, it is crucial to select the model sparsity to strike a balance between generalization and unlearning performance."

Pruning-method selection uses three criteria: ① least dependence on \mathcal{D}_f, ② lossless generalization, ③ pruning efficiency. They adopt OMP (one-shot magnitude pruning) by default, and reject IMP explicitly:

"IMP (iterative magnitude pruning) is not suitable for MU, despite being widely used to find the most accurate sparse models … IMP heavily relies on the training set including forgetting data points … the unlearning metrics get worse for IMP with increasing sparsity."

(b) \ell_1-sparse MU (Eq. 3) — sparsity-aware unlearning, doing both at once:

"where we specify L_u by the fine-tuning objective, and \gamma>0 is a regularization parameter that controls the penalty level of the \ell_1 norm."

So \ell_1-sparse MU is simply Finetune-on-retain plus an \ell_1 penalty. That is a very small increment over the Finetune baseline our Stage 5 already builds. If it is ever wanted, it is close to free — though it is not a Tier A target and should not be added to Phase 1/2 scope on cheapness alone.

\gamma scheduling matters and is their own finding: three schemes — constant, linearly growing, linearly decaying. Table 2 shows the choice moves the UA gap from 1.61 to 0.06, so a reimplementation that fixed \gamma constant would substantially understate the method.

3. Applicability note

Unlike Machine Unlearning, "prune first, then unlearn" is applicable post hoc: OMP is one-shot magnitude pruning performed on the already-trained \theta_o. So it does not require changing how the original model was trained, and it would be compatible with our frozen originals. It does, however, change the model being evaluated, which makes a like-for-like comparison against unpruned methods awkward — a fairness point rather than a feasibility one.

4. What the paper does not do (from sections read)

• §1–3, §5 and appendices unread; no number beyond Table 2 is recorded.
• Proposition 2 is stated for gradient ascent specifically; the extension to other methods is the empirical Figure 3, not the theory.
• The sparsity/generalisation trade-off is acknowledged but the selection of sparsity is empirical.
• Table 2's setting is 10% random data forgetting on ResNet-18/CIFAR-10 — a different forget spec from our Phase 2 target (0.25% selective), so its numbers are not directly comparable to anything we will produce.

5. Links

• Criticised by name in SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation, whose critique this file verifies against Jia's own Table 2.
• Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy names this paper alongside Towards Unbounded Machine Unlearning as the two state-of-the-art methods where retain-set privacy worsens after unlearning.
• Proposition 2 builds on the dense-model unlearning error of Unrolling SGD: Understanding Factors Influencing Machine Unlearning.
• Overlapping authorship with SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation, which is worth knowing when reading their mutual comparisons. Verified: Jiancheng Liu and Sijia Liu appear on both, and both release code under the same GitHub organisation (OPTML-Group/Unlearn-Sparse and OPTML-Group/Unlearn-Saliency). So SalUn's critique of \ell_1-sparse is a group criticising its own prior work — which makes it more credible as a limitation, not less, but the review should not present it as independent external validation.

@inproceedings{jia2023sparsity,
  title         = {Model Sparsity Can Simplify Machine Unlearning},
  author        = {Jinghan Jia and Jiancheng Liu and Parikshit Ram and Yuguang Yao and Gaowen Liu and Yang Liu and Pranay Sharma and Sijia Liu},
  year          = {2023},
  booktitle     = {NeurIPS 2023},
  eprint        = {2304.04934},
  archiveprefix = {arXiv},
  url           = {https://arxiv.org/abs/2304.04934}
}