Why SCRUB's privacy numbers are weaker evidence than they look

The claim under examination. Its basic MIA trains a logistic-regression attacker on the unlearned model's losses for forget versus test examples, with 5-fold cross-validation, and reports mean accuracy; 50% reads as a perfect defence. This populates Tables 2 and 21-28, including Table 23, our Phase 2 target.

Attacks the ATTACK. Categorises U-MIAs into population and per-example, shows the latter is far stronger, and names SCRUB's as a population attack using no shadow models -- the weakest category. Also reports that retain-set privacy can WORSEN after unlearning, on over half the untouched points, for SCRUB specifically. Two of its five authors are SCRUB authors.

Attacks the METRIC. Balanced accuracy is symmetric and average-case, and is shown to be uncorrelated with low-FPR success. Their worked example: on a well-generalising CIFAR-10 ResNet the LOSS attack reaches 60% balanced accuracy yet is correct only 48% of the time on the points it is most confident about -- worse than chance. It is a strong NON-membership attack.

Attacks the THREAT MODEL. Assumes access to both the original and unlearned model and exploits the discrepancy, reporting that this can beat classical MIA on the original, with the worst degradation on well-generalised models where classical MIA fails. Notably the attack survives EXACT unlearning: the leak is in the fact that forgetting occurred.

Shows what a defensible measurement costs. Estimates a per-example epsilon from the strongest attacker in a swept family and bins it into a forgetting score. The cheapest setup used N = 512 samples per distribution with no confidence intervals. Our plan is 5 seeds, so this framework is out of reach -- which is a fact about our compute, not a reason to approximate it.