Gaps: what is missing, and which of it we could act on
Five gaps, ranked by whether this project could address them, not by how interesting they are. A gap nobody can act on is a paragraph in a review; a gap we can act on is a thesis.
Each entry states three things separately, because conflating them is how a review talks itself into a contribution it cannot deliver:
- The gap — what is missing.
- The evidence — the sections that establish it is missing, with the paper and section named. Where the evidence is suggestive rather than established, it says so.
- Actionability — what this project could do, on this compute, within its timeframe.
A standing constraint on all five. Nothing here rests on a results section we did not
read, and no gap is claimed on the basis of a method comparison, because nothing has been
reproduced. Every "no paper does X" below means no paper in this 27-paper corpus, in the
sections recorded as read in papers.yaml.
Gap 1 — The governing dimension may be effective, not ambient
Rank: most actionable. This is the one a contribution could plausibly sit on.
The gap
Every bound in the theory track is stated in the ambient parameter count d. Nothing in the corpus asks whether the quantity that actually governs the difficulty of unlearning is the effective dimension — the rank of the subspace the training data moved the weights through — even though the empirical literature says most of d is inert.
The evidence
Two papers point at this from opposite sides, and neither cites the other's version of it.
From the practice side, Golatkar et al. 2020b §4.1 is explicit. Under an L_2 loss in the NTK regime, the Fisher and NTK scrubbing procedures are the same procedure when \nabla f_0(\mathcal{D}_r) is tall, and diverge when it is wide:
"if the matrix is wide (more parameters than samples in the network, as is often the case in Deep Learning), the Hessian is not invertible … In general, when the model is over-parametrized there is a large null-space of weights that do not change the final activations or the loss. The degenerate Hessian is not informative of where the network will converge in this null-space, while the NTK matrix gives the exact point."
So the ambient count is misleading by the method literature's own analysis, and there is a concrete object — the NTK matrix — that behaves correctly where the Hessian degenerates.
From the theory side, the same regime is where the bounds get bad: Regehr et al. 2026's rate carries an e^{-2\varepsilon/(d+2)} factor, and Van Waerebeke et al. 2025's complexity ratio is of order d/n when the forget ratio r_f \propto 1/n. Both degrade in d; neither distinguishes ambient from effective d.
Actionability
Yes, and cheaply. Stage 7 must build diagonal-Fisher machinery regardless — both Golatkar et al. 2020a and Foster et al. 2024 need it — and the eigenvalue spectrum of that diagonal is a by-product, not extra work. A first, honest question is purely empirical and answerable within Phase 1: what fraction of the Fisher diagonal's mass sits in how many coordinates, and does the unlearning difficulty of a forget set track that number better than it tracks d?
That would not prove a bound. It would establish whether the premise of a bound is plausible, which is the right first move and the one this project is equipped for.
Scope discipline. This is a later-phase thread. It is recorded here so the Stage 7 Fisher utility is written to expose its spectrum rather than hide it, which costs nothing now and is expensive to retrofit.
Gap 2 — Nobody reports what happens after the second request
The gap
Deletion requests arrive in a stream. Every vision method in this corpus is specified, tuned and evaluated for a single request.
The evidence
The theory side treats sequences as the object of study:
- Neel et al. 2021 models "an ongoing stream of additions and deletions" and distinguishes strong from weak unlearning by how cost scales with the number of requests handled. It is the only formal treatment of many sequential requests in this corpus.
- Guo et al. 2020 Theorem 4 shows the accumulated gradient-residual bound grows linearly in the number of removals T and quadratically in batch size m, so their Algorithm 2 falls back to retraining when the budget depletes.
- Ginart et al. 2019 Remark 3.1 gives an \Omega(n/m) amortised lower bound online.
- Sekhari et al. 2021's deletion capacity is the same idea from the statistical side.
The vision side does not. Across all ten vision_heuristic notes, sequential-request
behaviour appears nowhere. Even Bourtoule et al. 2021, which is architecturally about
handling many requests, defers the batched and sequential setting to its §VII — a section we
did not read, so this is stated as a gap in our reading of it, not as an absence in the
paper.
Actionability
Yes in principle, not within this phase. Running SCRUB over a stream of ten sequential 25-example forget requests and reporting utility decay is a small experiment on our hardware, and Fan et al. 2024's Appendix C shows the setting is studiable: it forgets ten per cent per round over five rounds against Retrain and fine-tuning. That single experiment is also the whole of the practical evidence, on one architecture, one dataset and one request size. But it is new method evaluation, not reproduction of Kurmanji et al. 2023, and Phase 2 is the deliverable. Recorded as the strongest later extension that requires no new theory.
Gap 3 — Every reported forgetting number is an average over examples that behave differently
The gap
Forgetting quality is reported as one number per method per setting. The evidence says that number hides a distribution wide enough to invert the conclusion.
The evidence
Hayes et al. 2024 states it directly, and the authorship matters — two of the four are SCRUB's own authors:
"Different unlearning algorithms lead to reducing the vulnerability of some (but not all) forget set examples, sometimes at the expense of increasing it for other examples."
And the consequence for stopping criteria:
"We observe that different examples are unlearned at different rates. Therefore, lacking a procedure for selecting per-example hyperparameters and stopping criteria … it is fundamentally challenging to equally protect all examples … We demonstrate empirically that a naive attempt at achieving this fails at improving privacy protection."
That lands squarely on SCRUB+R, which rewinds to a single checkpoint chosen by one global forget-error criterion. Triantafillou et al. 2024 operationalises the same concern into a per-example \hat\varepsilon aggregated by binning — and needed N = 512 samples per distribution to do it, in their cheapest setup with no confidence intervals.
Actionability
Partly, and the cheap half must be settled before the evaluation harness is fixed.
- Affordable: store per-example attack scores rather than only the aggregate, and
report TPR at low FPR as Carlini et al. 2022 argues. This needs no extra models — only
storage — and it is what makes Gaps 3 and 4 testable at all. It is carried as an open
decision in
project.yaml. - Not affordable: the competition's \mathcal{F} itself. Phase 1 budgets 5 seeds against their N = 512. Implementing \mathcal{F} on 5 samples and reporting it under that name would be a differently-defined statistic wearing a borrowed one.
Gap 4 — The methods may fail precisely on the points people ask to delete
The gap
Deletion requests are not uniform draws. They concentrate on atypical records — the unusual case, the distinctive face, the memorised outlier. There is a result saying the influence-function family degrades exactly there, and no evaluation in the corpus conditions on it.
The evidence
Izzo et al. 2021 §4.1, Theorem 5: with D^{\text{full}} = \{(\lambda x_1, \lambda y_1)\} \cup D^{\setminus 1}, the influence-updated model \theta^{\text{inf}} \to \theta^{\text{full}} as \lambda \to \infty. In words:
"when we try to delete points with large norm, the influence method will barely update the parameters at all, with the update shrinking as the size of the removed features increases."
Two facts turn that from a linear-algebra curiosity into a gap. First, the influence/Newton correction is the shared skeleton of Guo et al. 2020, Sekhari et al. 2021 and Golatkar et al. 2020a — so it is a weakness of the whole second-order programme, and one that is independent of the convexity and dimension arguments. Second, the points it fails on are the same population Carlini et al. 2022 and Hayes et al. 2024 identify as most vulnerable to membership inference.
⚠️ Stated for linear regression only. Theorem 5 is proved in the linear setting. For deep networks this is suggestive, not established, and the review says so in those words. Presenting it as established would be exactly the overreach
SYNTHESIS_PLAN.md§4 warns about.
Actionability
Testable with our harness, once per-example scores exist. Stratify the forget set by an atypicality proxy — gradient norm at the original weights, or original-model loss — and report forget quality per stratum instead of pooled. That is a reporting change, not a new experiment, and it costs one extra column. Whether Fisher-based unlearning underperforms specifically on high-influence examples becomes an answerable question rather than a speculation.
Its dependency is Gap 3's cheap half, which is why the two are adjacent in the ranking.
Gap 5 — No criterion is both auditable and estimable
Rank: least actionable. Included because it is the honest terminus of the argument.
The gap
The field's dominant criterion is anchor-based: indistinguishable from a model retrained on \mathcal{D}_r. There is a serious argument that this is not auditable, and a proposed replacement that is — but whose estimator we cannot evaluate at our sample sizes.
The evidence
Against the anchor. Xu et al. 2025 argues the anchor-based principle is inauditable because it depends on an unknown retrained model in parameter space. Thudi et al. 2022b reaches a compatible conclusion by a different route: for minibatch SGD, forging means unlearning "is not well-defined over the logs (and thus the models) of finite datasets", so it "cannot be defined on the parameter space" at all. Thudi et al. 2022a adds the measurement-noise objection — "comparison with respect to any one retrained model leaves the metric itself noisy."
The proposed replacement. Xu et al. 2025 Definition 2.5 states the criterion in output space: I(\hat Y_{\text{margin}}; Z) \le \mu with a utility budget L(f) - L^\star \le \rho. It needs only samples of (X_{\text{margin}}, Z, \hat Y_{\text{margin}}) — no anchor — and the authors call it "implementation friendly and scalable in practice, hence particularly applicable in deep learning."
Why we cannot use it yet. Estimating a mutual information from few samples is badly biased, and Phase 1's CIFAR-5 setting has 500 training images. The paper does not address finite-sample estimation in the sections we read — that concern is ours, not theirs, and is recorded as such — and its §6, which implements the criterion on image classification, is unread. Characterising its empirical behaviour before reading §6 would be assertion, not evidence.
Actionability
No. Recorded honestly as the gap this project cannot close.
Two things follow that do not require closing it. First, Thudi et al. 2022a's three
objections to oracle-based evaluation are not equally strong, and one of them dissolves
under a deterministic pipeline. Their objection is that "numerical instabilities introduced
by back-end randomness in floating point computations" make the metric noisy;
docs/DETERMINISM.md records repeated runs at a fixed seed as bitwise identical within a
backend — 0.000\mathrm{e}{+}00 across three runs on CPU and again across three runs on
CUDA. So on a fixed machine that source of noise is removable, not irreducible. (Across
backends it is not: CPU versus CUDA at the same seed differs by 9.360\mathrm{e}{-}03, which
is why every run records its device.) Second, the surviving objection — that any one retrained model is a noisy
reference — is addressable by reporting the oracle's own seed distribution rather than a
single draw. Neither is a new criterion; both are cheap and make the existing one honest.
Summary
| # | Gap | Evidence strength | Us? | When |
|---|---|---|---|---|
| 1 | Effective vs ambient dimension | established from two sides, unconnected | yes | next semester; instrument now |
| 2 | Sequential deletion requests | established in theory, absent in vision | yes, but new work | next semester |
| 3 | Per-example forgetting hidden by averages | established, by SCRUB's own authors | cheap half yes | settle before the harness is fixed |
| 4 | Failure on the points actually deleted | suggestive only (linear regression) | yes, as a reporting change | Phase 2, if Gap 3 lands |
| 5 | Auditable and estimable criterion | established as a problem; no usable answer | no | reported as a limitation |
The one with a deadline is Gap 3. Per-example score storage is trivial before the harness freezes and expensive afterwards, and Gap 4 is untestable without it. That is the single decision in this file with a date attached.