Machine unlearning for image classifiers: two literatures that do not compose
A literature review of 27 papers, 2015–2026.
Supporting material: taxonomy.md fixes the vocabulary and the three organising axes;
timeline.md gives the chronology; gaps.md ranks the openings by whether this project
could act on them; papers.yaml and notes/ hold the per-paper record, including which
sections of each paper were actually read; VERIFICATION_REPORT.md records what has been
checked against primary sources and what has not.
Two conventions govern what follows. Citations are [@citekey] and every one resolves
to references.bib, which is generated from papers.yaml rather than typed. And no
comparative performance number between methods appears anywhere in this review. Nothing
has been reproduced yet, and only two sets of experimental results were read in full during
this survey; asymptotic rates taken from sections that were read are used freely, since
those are claims the papers prove rather than measure.
1. Introduction and thesis
Machine unlearning asks a question that sounds administrative and turns out to be deep: a model was trained on a dataset, someone withdraws part of that dataset, and we must produce a model that is — in some sense to be specified — as if the withdrawn data had never been there. The obvious answer, retrain without it, is correct and usually unaffordable. Ten years of work has gone into finding something cheaper that is still defensible.
That work has produced two literatures.
One is theoretical. It defines unlearning by indistinguishability from retraining, borrows its machinery from differential privacy and convex optimisation, and proves statements about strongly convex losses under explicit constraints on what an unlearner may store and touch. Its results are correct, sharp, and in the most recent cases matched by lower bounds.
The other is empirical and vision-facing. It takes an overparameterised non-convex network, applies a post-hoc procedure that uses the forget set directly, and measures whether some attack can still tell that the forgotten data was there. Its methods work well enough to be deployed and are, by their own authors' admission, heuristic.
These two literatures cite each other. The claim of this review is that they do not compose:
The bridge between them — the second-order, influence-function programme that both sides once expected to connect them — collapses at three identifiable and independent points: convexity, dimension, and a failure mode specific to the atypical examples that deletion requests actually concern. And the one place the two literatures genuinely meet, evaluation, has been shown inadequate along three orthogonal axes — in several cases by the authors of the very methods being evaluated.
Three properties make this defensible rather than merely rhetorical. It is specific:
each collapse point is a named assumption in a named section, recorded in
assumptions.yaml. It was not imposed: the second-order thread emerged from reading
Sekhari et al. 2021 next to Golatkar et al. 2020a next to Regehr et al. 2026, and
no paper in the corpus states it. And it sets up a contribution: if the barrier is
dimension, and the dimension in every bound is the ambient parameter count while the
method literature's own analysis says most of that count is inert, then asking whether the
governing dimension is effective rather than ambient is the natural next move.
Three things this review deliberately does not argue. It does not claim any method is better
than another. It does not claim the theory is wrong — it is correct within its
assumptions, and the claim is about transfer, which is a different and more careful
thing. And it makes no claim about how papers motivate themselves, because most of the
introductions in this corpus were not read (VERIFICATION_REPORT.md, correction 9).
A note on method. A recent survey of the area exists Nguyen et al. 2025, and its own
framing is that the field "has not been able to completely solve the problem due to the lack
of common frameworks and resources". This review does not organise itself around it. The
axes in taxonomy.md were derived from primary sources so that each is traceable to a
section actually read, which is also what makes the thesis above falsifiable: every claim
resolves to a paper and a section rather than to a summary of one.
2. What it means to have forgotten
The field's definitions moved through three stages, and the movement is the interesting part.
Stage one: outputs. Cao et al. 2015 coined the term and defined success operationally through two metrics, completeness ("how completely they can forget data") and timeliness ("how quickly they can do so"). Their mechanism converts learning into a summation form — a small number of sums of efficiently computable transforms of the training data, so that forgetting means updating sums and recomputing — with generality claimed via statistical query learning. Two features of this founding paper were later lost. It treated feature selection as something requiring unlearning, where every subsequent paper in this corpus assumes a fixed architecture and unlearns only weights. And it named three motivations — privacy, security (training-data pollution), usability — of which the field kept one.
Stage two: distributions over models. Ginart et al. 2019 Definition 3.1 requires the deletion operation's output and A(D_{-i}) to be equal in distribution. This is a real advance: two models with different parameters may have been trained on the same data, and two models trained on different data may have identical parameters, so comparing outputs or parameters pointwise is the wrong move. Bourtoule et al. 2021 Definition III.1 makes the same distributional commitment and spells out the reasoning.
Ginart's second contribution is a framing that set the field's agenda: "Every learning algorithm A supports a trivial data deletion operation corresponding to simply retraining … Because of this, the challenge of data deletion is computational." Their Remark 3.1 supplies the matching complexity statement, an \Omega(n/m) amortised lower bound for m deletion requests in the online setting.
A lineage correction. It is common to cite Ginart et al. 2019 as the origin of an (\varepsilon,\delta)-approximate definition. It is not: Definition 3.1 is exact, and the paper says so. The relaxation is deferred to an appendix. The correct lineage is Cao et al. 2015 → Ginart et al. 2019 (exact, distributional) → Guo et al. 2020 (genuinely (\varepsilon,\delta)) → Sekhari et al. 2021 and Neel et al. 2021. This is a small point that a theory-oriented reader would catch immediately.
Stage three: differential-privacy-shaped bounds. Guo et al. 2020 defines (\varepsilon,\delta)-certified removal by bounding the max-divergence between the mechanism's output and retraining without the point. Sekhari et al. 2021 Definition 2 states an analogous condition and adds two things the earlier work lacked. First, deletion capacity: the maximum number of points that can be unlearned while keeping excess population risk below a threshold. Their Theorem 1 shows that "if we only care about minimizing the empirical loss, then we can delete up to n-1 sample points… However, the deletion capacity is inherently limited if we want to retain test loss guarantees" — exact empirical deletion is easy and nearly unbounded; deletion that preserves generalisation is capped. Second, a storage constraint: the unlearner may use statistics T(S) but not S, "aiming for memory requirements which are independent of the training data size n", which they note "precludes strategies which involve storing and reusing the entire training set, or aggressive model checkpointing."
Neel et al. 2021 adds the dimension everyone else drops: an ongoing sequence of additions and deletions. Its Definition 2.4 routes the model through a publishing function before release, so an unlearner may keep secret state that would betray the forget set if published, while perfect unlearning may not. Definition 2.8 then separates strong unlearning (accuracy independent of sequence length, and cost growing at most logarithmically in the number of requests handled) from weak.
Where does privacy sit in all this? Ginart et al. 2019 gives the sharpest separation: "Even an operation that outputs the entire dataset in the clear could support data deletion, whereas such an operation is certainly not private. In this sense, the challenge of data deletion only arises in the presence of computational limitations." Cao et al. 2015 had made the qualitative version of the point a decade earlier — DP protects "equally and invariably", forgetting systems "restore privacy on select data" — and Sekhari et al. 2021 later made it quantitative, showing a \Theta(d^{1/4}) separation in deletion capacity between what DP guarantees and what unlearning can achieve.
Running against all of it, Chen et al. 2021 observes that unlearning produces two models, and the discrepancy between them is itself attackable. Deletion pursued without regard to privacy is not merely unambitious; it can create the leak it was meant to close.
3. Exact unlearning and its price
Bourtoule et al. 2021 is the canonical exact method and the cleanest illustration of what exactness costs.
Its definition includes a clause that matters later: unlearning "does not necessarily require that the owner retrain the model M' from scratch … as long as they are able to provide evidence that model M' could plausibly have been trained from scratch on \mathcal{D}'\setminus d_u. In our work, this evidence takes the form of a training algorithm."
SISA is Sharded, Isolated, Sliced, Aggregated. Partition the data into S disjoint shards and train one constituent per shard with no gradient exchange whatsoever, so "the influence of a shard … is restricted to the model that is being trained using it". Split each shard into R slices trained incrementally with a checkpoint saved after each. Aggregate by label-based majority vote — chosen over learned aggregation specifically because "the aggregation strategy should not involve the training data (otherwise the aggregation mechanism itself would have to be unlearned in some cases)". A deletion touches one slice of one shard, giving a best-case retraining speed-up of \frac{(R+1)S}{2}\times for a single request.
The costs are structural, and the paper names them. Each constituent sees 1/S of the data and so "could be weak learners… this effect is more profound in complex learning tasks." Storage is the whole dataset plus R checkpoints per shard. And the intervention is at training time: SISA cannot be applied to a model that already exists.
That last point has three consequences worth stating together, because they are usually treated as separate facts. It is why SISA is absent from Kurmanji et al. 2023's comparison table — it is not a competitor but a different paradigm. It is why this project cannot use SISA as a baseline: every Stage 7 method takes a frozen w^o and modifies it, and comparing forget quality would be vacuous anyway, since SISA's is exact by construction. And it is precisely the property that exempts SISA from Van Waerebeke et al. 2025's impossibility result, whose authors note that "most existing exact MU algorithms rely on modifications of the training set and are thus not included in our framework."
The prehistory is worth recording: SISA is Ginart et al. 2019's modularity principle, published two years earlier, applied to deep networks. Ginart's other three principles — linearity, laziness, quantization — map most of the rest of the field. Linearity became the second-order programme; quantization became the stability assumption; laziness has no deep-network descendant in this corpus at all, which is a fact about what the field chose rather than about what is possible.
4. The second-order programme
The programme that both literatures expected to bridge them runs as follows. If the loss is convex with a unique minimiser, removing a training point moves that minimiser by a predictable, computable amount — a one-step Newton correction, classically the influence function of the removed point. Compute it, apply it, and you have unlearned without retraining.
The idea is Ginart et al. 2019's linearity design principle, stated in 2019 as the observation that "linear computation allows for simple post-processing to undo the influence of a single datapoint". Its appeal is that it promises everything at once: a closed form, a cost independent of n, and — because the correction is derived rather than fitted — the possibility of a guarantee. Nothing else in the field offers all three.
What follows is best read as a sequence of patches, because the patches are what reveal the programme's shape. Each paper finds the previous version leaks, or accumulates, or does not scale, and repairs it in a way that narrows the claim.
Guo et al. 2020 gives the canonical form. For a regularised linear model with unique minimiser \mathbf{w}^\star, the removal is \mathbf{w}^- := \mathbf{w}^\star + H_{\mathbf{w}^\star}^{-1}\Delta, and H^{-1}\Delta "is also known as the influence function of the training point". Three refinements follow, and each is instructive.
A small gradient residual is not certification. Their Theorem 1 bounds the residual norm, but "the direction of the gradient residual may leak information about the training sample that was 'removed'", so the mechanism adds loss perturbation at training time to mask it (Theorem 3). Closeness in parameters is not enough — a point made with a counterexample in which "an approximate removal algorithm that leaves w_i small but non-zero still reveals that (\mathbf{e}_i, i) appeared during training."
The guarantee is a depleting budget. The worst-case residual grows linearly in the number of removals T, and their Theorem 4 shows the batch bound grows quadratically in batch size m. Algorithm 2 therefore carries an explicit fallback: when the budget is exhausted, retrain from scratch. Certified removal is not an unbounded capability. It is the algorithmic counterpart of Sekhari et al. 2021's deletion capacity, arriving from the optimisation side rather than the statistical one.
And it does not certify the network. Guo et al. 2020 §3.3 is explicit that in deep learning the mechanism applies to "the linear model that is applied to the output of the feature extractor", with the extractor either frozen and pretrained on public data, or differentially private. This is decisive and under-appreciated: certified removal never certifies the representation. It is not that the bound is loose for a network trained end to end; the object it certifies is a different object. Bourtoule et al. 2021 §III.D rejects the certified line on exactly these grounds, and that rejection is fair rather than uncharitable.
Sekhari et al. 2021 carries the same skeleton into the statistical setting and changes what is being protected. Its Assumption 1 requires the per-example loss to be \lambda-strongly convex, L-Lipschitz and M-Hessian-Lipschitz; its stored statistic T(S) is the Hessian at the output point, costing O(d^2) space; and Gaussian noise is sized for the (\varepsilon,\delta) guarantee. The shift is from empirical to population risk, and it is what makes the deletion-capacity result of §2 possible: the programme is no longer trying to reproduce a particular retrained model but to preserve generalisation while deleting.
Its title records the resulting awkwardness precisely. The unlearner uses the points in U during the unlearning phase, and the authors name it: "This paradox of storing and using information in order to delete it". That is not an incidental cost. It places the whole programme at a specific point on the access axis — one where an unlearner must retain what it was asked to forget, at least until it has finished forgetting it.
Izzo et al. 2021 attacks the remaining cost. Its Projective Residual Update uses a classical leave-one-out residual identity and the hat matrix to compute what the deleted model would predict on the removed points without knowing that model, then takes a gradient step fitting those synthetic targets. The result is exactly the projection of the true update onto the span of the deleted points, which the paper shows is provably the best any gradient-based update using only those points can do — an optimality statement about a restricted class, and a rare instance of a method paper proving it has reached a ceiling.
Read as a sequence, the patches trace an outline. Guo adds noise because parameter closeness leaks; Guo adds a retraining fallback because the budget accumulates; Guo confines the mechanism to a linear head because O(d^3) does not scale; Sekhari adds a storage bound and moves to population risk; Izzo projects to reduce cost and proves the projection is optimal. Every patch narrows the claim, and none of them touches convexity, which is assumed throughout.
Then the programme reaches deep networks, and something revealing happens: it is derived and abandoned in the same papers.
Golatkar et al. 2020a defines forgetting through a Forgetting Lagrangian, derives an optimal scrubbing function under a quadratic-loss and gradient-flow assumption, observes it reduces to a Newton update at convergence, and approximates the Hessian by the Fisher Information Matrix. What is actually implemented is far simpler than what is derived. Its companion Golatkar et al. 2020b explains why, in a passage that is the most theoretically interesting in the method literature:
"if the matrix is wide (more parameters than samples in the network, as is often the case in Deep Learning), the Hessian is not invertible … when the model is over-parametrized there is a large null-space of weights that do not change the final activations or the loss. The degenerate Hessian is not informative of where the network will converge in this null-space, while the NTK matrix gives the exact point."
Fisher and NTK forgetting are the same procedure when there are more samples than parameters, and diverge exactly when there are not — which is the regime every network in this corpus inhabits.
5. Theory and practice: what transfers?
This is the review's central question, and the honest answer is layered: less than the citation graph suggests, for reasons that differ per paper, and one of the three theory papers is not in the position the others are.
5.1 What the three theory papers actually assume
| Van Waerebeke et al. 2025 | Regehr et al. 2026 | Xu et al. 2025 | |
|---|---|---|---|
| Convexity | strongly convex class \mathcal{F}_{sc}; unique minimiser | per-example \mu-strongly convex, \beta-smooth, L-Lipschitz | none assumed |
| Forget-set access | unlearner class may only sample \mathcal{D}_r | full storage, T(Z) = Z | not the object of study |
| Dimension | appears as d in the ratio | appears as e^{-2\varepsilon/(d+2)} | bound described as "dimension- and output-space-agnostic" |
| Quantity bounded | computation time to target excess risk | excess population risk | mutual information, with a utility budget |
The differences are not cosmetic. Regehr et al. 2026 explicitly drops Sekhari et al. 2021's storage constraint, granting the unlearner the whole dataset, and also weakens the Hessian-Lipschitz requirement — "Unlike [SAKS21], we do not require that Hessian of the loss is Lipschitz." Van Waerebeke et al. 2025 goes the other way, restricting the unlearner to sampling the retain set only. Rates proved under different access models are not comparable, and a review that tabulates them side by side without saying so is misleading its reader.
5.2 What the bounds actually guarantee
Three different currencies, routinely conflated:
- Van Waerebeke et al. 2025 bounds the unlearning complexity ratio T^U_e / T^S_e — the minimax computation time to reach excess risk e by unlearning, against reaching it by retraining — producing a three-regime phase diagram: Trivial (adding Gaussian noise suffices), Inefficient (unlearning cannot asymptotically beat retraining), and Efficient.
- Regehr et al. 2026 bounds excess population risk under a pure (\delta = 0) \varepsilon-unlearning constraint, with matching upper and lower bounds agreeing up to the condition number.
- Xu et al. 2025 bounds a mutual information between the model's marginal outputs and a latent source label, paired with a utility budget.
Nothing converts between them. Neither has any counterpart to forget error or membership inference advantage, which is what the vision literature reports. When Kurmanji et al. 2023 says methods with guarantees "either aren't applicable to deep networks, or …", the first disjunct is doing more work than a reader might assume — it is not only that the assumptions fail, but that the guaranteed quantity is not the measured one.
5.3 What breaks for a ResNet on CIFAR
Three independent strands. Independence is the point: fixing one would not fix the others.
Convexity. Neel et al. 2021 Lemma 2.12 is the load-bearing step of the whole family: if f_z is L-Lipschitz and m-strongly convex, then \|\theta^\star_\mathcal{D} - \theta^\star_{\mathcal{D}\circ u}\|_2 \le \frac{2L}{mn}. Strong convexity is what makes deleting one point move the optimum only O(1/n). Regehr et al. 2026 uses the same stability bound. Remove strong convexity and there is no guarantee the retrained optimum is anywhere near the original — and a ResNet's loss surface has many minima, so the assumption is not approximately true, it is structurally false. Kurmanji et al. 2023 makes precisely this objection to the prior literature: it rests on assumptions — SGD stability, "small enough" forget sets — that deep networks violate.
Dimension. This is where the sharpest available statement lives. Regehr et al. 2026's matching bounds carry the factor e^{-2\varepsilon/(d+2)}, and the authors state the regime plainly: the algorithm "gives an exponential improvement over the retrain-from-scratch rate of \Theta((m/n)^2) in the regime \varepsilon = \Omega(d)." Read the factor in the other direction. When \varepsilon \lesssim d it tends to 1 and the bound collapses to \frac1n + (\frac mn)^2 — exactly the retrain-from-scratch rate. For ResNet-18, d \approx 1.1\times 10^7, so any gain requires \varepsilon \gg 10^7, whereas meaningful privacy parameters are single digits. In the overparameterised regime, this theorem says retraining from scratch is optimal — and it says so with a matching lower bound, not a loose upper one.
Two related statements agree. Guo et al. 2020's Newton step is O(d^3) at removal time, which for the same d is around 10^{21} operations, and even the offline O(d^2 n) Hessian is unreachable. And Van Waerebeke et al. 2025 note that if the forget ratio r_f \propto 1/n then the complexity ratio is of order d/n — the wrong direction for overparameterised models.
The outlier failure mode. Izzo et al. 2021 §4.1, Theorem 5: as a training point is scaled up, the influence-updated model converges back to the un-updated one. In their words, "when we try to delete points with large norm, the influence method will barely update the parameters at all, with the update shrinking as the size of the removed features increases" Since the influence/Newton correction is the shared skeleton of Guo et al. 2020, Sekhari et al. 2021 and Golatkar et al. 2020a, this is a weakness of the whole programme — and it is orthogonal to convexity and dimension, since it appears in the well-conditioned linear case where those two are satisfied. Its force comes from a fact outside the paper: deletion requests concentrate on atypical records, and those are precisely the examples Carlini et al. 2022 and Hayes et al. 2024 identify as most vulnerable to membership inference. The programme may degrade exactly on the population it is asked to protect.
This last point is stated for linear regression only, and this review presents it as suggestive rather than established for deep networks.
5.4 A correction: Xu et al. 2025 is not a convexity paper
The premise this project began with — that the theory track is a convexity track whose barrier to deep networks is the convexity assumption — is true of two of the three papers and false of the third, and getting this right changes what the section has to argue.
Xu et al. 2025 assumes no convexity. Its Theorem 2.1 is described by its authors as "dimension- and output-space-agnostic", and Definition 2.5 is said to be "implementation friendly and scalable in practice, hence particularly applicable in deep learning", with a §6 that implements the criterion on image classification directly. Its argument is not about optimisation at all: it is that the field's dominant anchor-based principle — indistinguishable from a model retrained on \mathcal{D}_r — is inauditable, because it depends on an unknown retrained model in parameter space. It replaces the anchor with a condition stated purely in output space: no observer should be able to infer, from the model's outputs, whether the training data came from the retain distribution or from the mixture that also includes the unlearn source. Definition 2.5 makes this I(\hat Y_{\text{margin}}; Z) \le \mu together with L(f) - L^\star \le \rho.
So for this paper the question is not whether it transfers but whether its criterion is estimable at our scale. Mutual information estimated from few samples is badly biased, and Phase 1's CIFAR-5 setting has 500 training images. That concern is ours, not the paper's — finite-sample estimation is not addressed in the sections we read, and its §6 is unread, so this review makes no claim about its empirical behaviour.
It is worth registering that its conclusion converges with Thudi et al. 2022b's from a completely different direction. One argues the anchor is unknowable; the other proves that for minibatch SGD, forging makes unlearning "not well-defined over the logs (and thus the models) of finite datasets", so it "cannot be defined on the parameter space" at all. Two independent routes to stop measuring in parameter space.
5.5 Which methods descend from the theory, and which do not
The descent is traceable and short. Fisher forgetting Golatkar et al. 2020a and NTK forgetting Golatkar et al. 2020b are direct descendants — one approximates the Newton update's Hessian by the FIM, the other replaces the degenerate Hessian with the NTK matrix. SSD Foster et al. 2024 is the interesting hybrid: it uses the same diagonal Fisher information, but as a selection rule for which parameters to dampen rather than as a certified update. The machinery survives; the guarantee does not.
Everything else is avowedly heuristic. Kurmanji et al. 2023 §8 states the absence of theoretical guarantees as its primary limitation. Chundawat et al. 2023, Fan et al. 2024, Graves et al. 2021 and Tarun et al. 2024 make no formal claim at all. That is not a criticism — it is the honest position for a method that works on objects the theory cannot describe — but it does mean the field's best-performing methods and its strongest theorems have no formal relationship whatsoever.
6. Methods for deep networks
Organised by the erase/repair shape of Triantafillou et al. 2024 §5, with their own caveat attached: "unlearning phases cannot always be neatly categorized as one of the two: for instance, finetuning on the retain set still has an indirect erasing effect on the forget set due to catastrophic forgetting."
Repair-only methods rely on that indirect effect. Fine-tuning on \mathcal{D}_r is the simplest. Goel et al. 2022 contributes CF-k (freeze the first k layers, fine-tune the rest on \mathcal{D}_r) and EU-k (freeze the first k, re-initialise and retrain the rest) — deliberately trivial baselines that exist to test whether more elaborate methods earn their complexity. Graves et al. 2021's "Naive Retraining" is the same idea under a different name. Jia et al. 2023 adds a structural observation: sparsifying the model first, or adding an \ell_1 penalty during fine-tuning, improves unlearning, motivated by a bound showing gradient-ascent unlearning error shrinks with sparsity.
Erase-only methods modify weights directly and accept the utility damage. Golatkar et al. 2020a adds FIM-shaped noise; Golatkar et al. 2020b applies a closed-form NTK jump; Foster et al. 2024 dampens parameters disproportionately important to the forget set rather than pruning them, and is notable on the access axis for needing no retain set at unlearning time — its selection uses a precomputed Fisher over the full dataset, because a retain-set Fisher "must be recalculated for every new forget request."
Erase-and-repair is where the field converged. Graves et al. 2021's relabelling variant assigns random incorrect labels to the sensitive data and retrains briefly. Fan et al. 2024 restricts any such update to salient weights, identified by hard-thresholding the magnitude of the forget-set loss gradient at the original weights. Tarun et al. 2024 is the outlier on the access axis: a zero-glance method that never looks at the forget samples, only the forget class label, learning an error-maximising noise matrix, corrupting with an IMPAIR epoch and restoring with a REPAIR epoch. Its authors state that "unlearning random samples, or only a subset of a class is out of the scope of this paper", which excludes it from this project's settings entirely.
The two distillation methods are worth contrasting because they look similar and are structurally opposite. Chundawat et al. 2023 uses two teachers — the original model as the competent teacher, a randomly initialised network as the incompetent one — and a per-example switch pulling the student toward the bad teacher on forget samples. Kurmanji et al. 2023 uses one teacher and has the student selectively disobey it: MAX steps push forget-set predictions away from the teacher, MIN steps keep retain-set predictions close, alternating. Bad-T minimises a KL toward a bad target; SCRUB maximises a KL away from the only target. Both descend; only one has an anti-target.
Kurmanji et al. 2023 also reframes the objective, and this is its more durable contribution. Rather than one notion of forgetting, three applications with incompatible desiderata: RB (removing biases) wants maximal forget error; RC (resolving confusion) wants interclass confusion eliminated; UP (user privacy) wants forget error only as high as the oracle's, because more is itself a leak. SCRUB targets the first; SCRUB+R adds a rewinding step that selects the epoch whose forget error best matches a reference, to hit the third.
The RB/UP tension is genuine and it recovers something old. RC is recovery from label pollution — Cao et al. 2015's security motivation. RB is their usability motivation. SCRUB's claim to be first in arguing that forgetting is application-dependent is fair with respect to the evaluation literature, but the breadth it restores was present in the field's founding paper and lost in between. That arc is only visible from the 2015 paper, which a reproduction of SCRUB has no particular reason to open.
7. Evaluation, and why it is the weakest link
If the two literatures meet anywhere, it is here — and this is where the corpus is most damning. Three orthogonal attacks, none of which subsumes another.
Axis one: the oracle is the wrong target. Goel et al. 2022 Theorem 1 shows matching a particular retrained model in weight space is not sufficient for forgetting, and observes it is not necessary either, since a randomly initialised network contains no forget-set information while being far from the retrained model. Thudi et al. 2022a adds a measurement objection: "comparison with respect to any one retrained model leaves the metric itself noisy", partly because of "numerical instabilities introduced by back-end randomness in floating point computations". Xu et al. 2025 adds the definitional one: the anchor is unknown, so the criterion is unauditable.
One of those objections dissolves under engineering discipline, and it is worth saying so because the field treats oracle noise as irreducible. Under a fully deterministic pipeline, repeated runs at a fixed seed are bitwise identical within a backend — measured, not assumed, in
docs/DETERMINISM.md. The surviving objection, that any one oracle draw is a noisy reference, is addressable by reporting the oracle's own seed distribution rather than a single draw. Neither is a new criterion; both make the existing one honest.
Axis two: the attacks are too weak. Carlini et al. 2022 argues that membership inference should be judged by true-positive rate at low false-positive rates on a log-scale ROC, not by average-case accuracy or AUC, because privacy is a worst-case property — and introduces LiRA, a likelihood-ratio attack calibrated per example. Hayes et al. 2024 applies this to unlearning specifically, separating population U-MIAs (one attacker trained on outputs pooled across examples) from per-example U-MIAs, and finding the weaker population attacks "substantially overestimate the privacy protection of unlearning algorithms". Two of its four authors wrote SCRUB. Two findings bear directly on how anything is measured: different examples are unlearned at different rates, so "it is fundamentally challenging to equally protect all examples" with a single global stopping criterion — which is exactly what SCRUB+R's rewinding is — and methods reduce "the vulnerability of some (but not all) forget set examples, sometimes at the expense of increasing it for other examples", so an aggregate number can look good while individual examples remain exposed.
Triantafillou et al. 2024 turns this into a scoring framework: estimate a per-example \hat\varepsilon from the error rates of the strongest attacker in a swept family of threshold attacks, aggregate by binning, and multiply by retain- and test-accuracy ratios against the oracle. Their stated principle is the right one — "the failure of a poor attacker at distinguishing the two distributions does not say much about the quality of unlearning". Their compute requirement is the sobering part: N = 512 samples per distribution in their cheapest setup, with no confidence intervals.
Axis three: the artifact cannot be audited at all. Thudi et al. 2022b constructs a forging map that rewrites a training log so the same parameter trajectory is explained by a different dataset. If one exists between \mathcal{D} and \mathcal{D}\setminus x^*, then then "they do not need to do anything and can claim the unlearning is done". Their Corollary 2 is the strongest negative result in this corpus, and it applies specifically to minibatch SGD. Their prescription — verify the process, not the artifact — is, as it happens, what a config-hash/git-SHA/split-hash/seed provenance record implements, though this project built one for reproducibility rather than for auditing.
And unlearning itself leaks. Chen et al. 2021 attacks the pair of models, concatenating posteriors from the original and unlearned models into a feature vector for a binary attack classifier. Anyone releasing an updated model after a deletion request has published a differential.
Two observations about this literature as a body. First, the authorship overlap is
striking: three of SCRUB's four authors appear on Triantafillou et al. 2024, and
Hayes et al. 2024 comes from the same group. A field whose strongest self-criticism comes
from the authors of the methods being criticised is correcting itself — and those critiques
are not independent evidence, which is recorded in VERIFICATION_REPORT.md. Second, and
awkwardly for a panel-facing project, the two most recent and most relevant critiques have
no peer-reviewed venue at all. Sorting the corpus by venue makes a related pattern visible
that no individual paper mentions: not one of the ten deep-network method papers appeared at
a security venue, while all four peer-reviewed evaluation critiques did — CCS, IEEE S&P,
USENIX Security, EuroS&P. The field was founded at IEEE S&P, the methods left for
machine-learning conferences, and the criticism stayed behind.
8. Gaps and where a contribution might sit
Ranked by whether this project could act, not by interest. gaps.md carries the full
evidence for each.
1. The governing dimension may be effective, not ambient. Every bound uses ambient d, and the method literature's own analysis says most of d is inert: Golatkar et al. 2020b §4.1 identifies "a large null-space of weights that do not change the final activations or the loss". Two literatures point at the same structural fact and neither cites the other's version. Actionable, because the Fisher machinery is needed for Stage 7 regardless, and its diagonal spectrum is a by-product. The first honest question is empirical — does unlearning difficulty track the effective spectrum better than it tracks d? — and answering it establishes whether a bound's premise is plausible, which is the right first move.
2. Sequential requests are studied in theory and barely measured in practice. Neel et al. 2021 formalises update sequences and Guo et al. 2020 Theorem 4 quantifies budget depletion. On the method side exactly one paper in this corpus reports it: Fan et al. 2024 forgets ten per cent per round over five rounds against Retrain and fine-tuning, on one architecture, one dataset and one request size. So the gap is not that nobody looks, but that a single unreplicated experiment is the whole of the practical evidence, and no two papers measure it the same way. Actionable but outside the reproduction scope: it is new evaluation, not reproduction.
3. Every reported forgetting number is an average over examples that behave differently. Established by Hayes et al. 2024, by SCRUB's own authors. The cheap half has a sequencing constraint: storing per-example attack scores is trivial before the evaluation harness is fixed and expensive afterwards, and Gap 4 is untestable without it. The expensive half — Triantafillou et al. 2024's \mathcal{F} — is out of reach at 5 seeds against their N = 512, and implementing it anyway under that name would be a different statistic wearing a borrowed one.
4. The methods may fail precisely on the points people ask to delete. Izzo et al. 2021 Theorem 5, marked suggestive rather than established because it is proved for linear regression. Testable as a reporting change — stratify the forget set by an atypicality proxy and report per stratum — once Gap 3's per-example scores exist.
5. No criterion is both auditable and estimable. Xu et al. 2025 Definition 2.5 is auditable and needs no anchor, but its estimator is unevaluated at our sample sizes. Recorded as the gap this project cannot close.
The plausible thesis contribution is Gap 1, approached through Gap 3's instrumentation. It is the only one where a theoretical question, an empirical observation from a different literature, and machinery this project must build anyway all point at the same object.
9. Conclusion
Machine unlearning is ten years old and has two well-developed literatures that address the same words with different objects. The theory line has, in the last two years, become genuinely sharp — Regehr et al. 2026 proves matching upper and lower bounds, and Van Waerebeke et al. 2025 gives a phase diagram — and precisely because it is sharp, it can be read as saying that in the overparameterised regime, retraining from scratch is optimal. The vision line has, over the same period, produced methods that work well enough to matter, on models the theory does not describe, measured by criteria that three separate lines of work have shown to be inadequate.
The bridge that was supposed to connect them, the second-order programme, fails in three independent places, and this review's contribution is to show that they are independent: convexity is not the only obstacle, and removing it would leave the dimension and outlier problems untouched.
Two things follow for the work that comes next. First, the reproduction target is well chosen: Kurmanji et al. 2023 is the strongest method in the class the theory cannot describe, and reproducing it faithfully is a precondition for saying anything about that class. Second, the most promising opening is not a new method but a measurement: whether the dimension governing the unlearning penalty is the ambient parameter count or something much smaller. Both literatures point at that question. Neither has asked it.