Reading path · 6 papers
The collapse of the second-order programme
Influence functions and Newton corrections dominate the theory of unlearning. Why does none of that machinery reach a ResNet, and at exactly which step does it break?
-
1Certified Data Removal from Machine Learning Models
Establishes the shape everything downstream inherits: a one-step Newton update on the removed point's gradient influence, plus calibrated noise, because a small gradient residual alone does not certify -- its DIRECTION leaks. Costs O(d^3) at removal time. Section 3.3 already concedes the decisive limit: for deep models the mechanism is applied only to a LINEAR HEAD over a frozen or DP-trained feature extractor. Certified removal never certifies the representation.
-
2Remember What You Want to Forget: Algorithms for Machine Unlearning
Pushes the same correction further and gives it a budget -- deletion capacity, the number of points removable while excess POPULATION risk stays bounded. Its Theorem 1 is the sharp conceptual result: exact EMPIRICAL deletion is nearly unbounded, but deletion preserving generalisation is capped. The price is storage of the Hessian, O(d^2), which at d ~ 1.1e7 is roughly half a petabyte.
-
3Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks
Where the programme meets practice and loses. The theory derives a Newton update; Section 4 states plainly that the FIM diagonal "is not a good enough approximation of B for a full Newton step", sets h(w) = w, and keeps ONLY the noise. The correction is abandoned because the curvature estimate is unaffordable -- and the remaining method leans on SGD stability, exactly the assumption SCRUB attacks.
-
4Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations
Explains structurally why the Hessian fails here. Section 4.1: Fisher and NTK forgetting COINCIDE when there are more samples than parameters, and diverge when d > n, because the Hessian becomes non-invertible and "the degenerate Hessian is not informative of where the network will converge in this null-space". The overparameterised regime is not a hard case for the method; it is a different case.
-
5Approximate Data Deletion from Machine Learning Models
Adds a failure mode orthogonal to convexity and dimension. Theorem 5: the influence method barely updates parameters at all for high-norm points. The points most likely to be deletion targets are atypical ones -- which are also the points membership inference finds most vulnerable. Proved only for linear regression, so suggestive rather than established for deep networks.
-
6Near-Optimal Pure Machine Unlearning for Smooth Strongly Convex Losses
Closes the argument from the theory side, with MATCHING upper and lower bounds. The gain over retraining carries a factor e^{-2 eps/(d+2)}, which only helps when eps >> d. At d ~ 1.1e7 that demands a vacuous privacy parameter; otherwise the bound collapses to exactly the retrain-from-scratch rate. In the overparameterised regime, retraining is optimal -- and that is a lower bound, not a loose upper one.
What this leaves open
Every step above measures d as the AMBIENT parameter count. Golatkar's own null-space argument says most of that dimension does not affect the loss, and trained-network Hessians are famously low-rank. Nobody in this corpus has asked whether the dimension governing the unlearning penalty is effective rather than ambient. If it is, the regime boundary moves by orders of magnitude -- and the Fisher machinery Stage 7 must build anyway is the natural instrument for asking.