Membership Inference Attacks From First Principles

Venue: IEEE Symposium on Security and Privacy (S&P) 2022 · arXiv: 2112.03570 Affiliations: Google Research; University of Massachusetts Amherst. Authors ordered alphabetically. Code: https://github.com/tensorflow/privacy/tree/master/research/mi_lira_2021 Verification: dblp for venue; PDF fetched.

Read status: pp. 1–3 read from page images — abstract, §I Introduction, §II Background, §III–III.A Definitions (Def. 1, Def. 2, the LOSS attack), §III.B Evaluating membership inference attacks. §IV (the LiRA construction itself) and all results not read — LiRA's mechanics are recorded second-hand in hayes2024inexact.md and must be read here directly before any implementation.

Source of the LiRA attack SCRUB adapts for its UP application. But the part that matters most for this project is not the attack — it is the argument that the metric almost everyone reports, including SCRUB, is the wrong one.

1. The security game

Definition 1 (Membership inference security game). Between challenger \mathcal{C} and adversary \mathcal{A}:

1. \mathcal{C} samples D \leftarrow \mathbb{D} and trains f_\theta \leftarrow \mathcal{T}(D).
2. \mathcal{C} flips a bit b; if b=0 samples a fresh challenge point (x,y) \leftarrow \mathbb{D} with (x,y) \notin D, otherwise selects (x,y) \xleftarrow{\$} D.
3. \mathcal{C} sends (x,y) to \mathcal{A}.
4. \mathcal{A} gets query access to \mathbb{D} and f_\theta, and outputs \hat b \leftarrow \mathcal{A}^{\mathbb{D},f}(x,y).
5. Output 1 if \hat b = b.

Attacks output a continuous confidence score thresholded to a decision: \mathcal{A}(x,y) = \mathbb{1}[\mathcal{A}'(x,y) > \tau].

The LOSS attack (Yeom et al.), which is the "basic loss-threshold attack" in our Stage 6 plan:

2. Why balanced accuracy is the wrong metric

Definition 2 (Balanced attack accuracy). \Pr_{x,y,f,b}\big[\mathcal{A}^{\mathbb{D},f}(x,y) = b\big].

Two objections:

1. It is symmetric. "the metric assigns equal cost to false-positives and to false-negatives. However, in practice, adversaries often only care about one of these two sources of errors."
2. It is average-case, and security is not. Their example: "Attack A perfectly targets a known subset of 0.1% of users, but succeeds with a random 50% chance on the rest. Attack B succeeds with 50.05% probability on any given user. On average, these two attacks have the same attack success rate (and thus the same balanced accuracy). However, the second attack is practically useless, while the first attack is exceptionally potent."

And aggregate metrics are not merely coarse — they are uncorrelated with what matters: "the attack of Yeom et al. has a high accuracy (59.5%) yet fails completely at low FPRs, and the attack of Long et al. has a much lower accuracy (53.5%) but achieves higher success rates at low FPRs."

The illustration that should change how we report

On a standard CIFAR-10 ResNet trained to 92% test accuracy — with weight decay, augmentation and early stopping, so only an 8% train–test gap:

"this attack's balanced accuracy is 60%. This is (much) better than random guessing, and so one might reasonably conclude that the attack is useful and practically worrying.

However, this attack completely fails at confidently identifying any members! … the 1% of samples … with lowest losses … are the samples where the attack is most confident that they are members. Yet, on this subset, the attack is only correct 48% of the time (worse than random guessing). In contrast, for the 1% samples with highest loss (confident non-members), the attack is correct 100% of the time. Thus, the LOSS attack is actually a strong non-membership inference attack, and is practically useless at inferring membership."

Their prescription: evaluate by true-positive rate at low false-positive rates (e.g. \le 0.1\%), and plot log-scale ROC curves. "If a membership inference attack can reliably violate the privacy of even just a few users in a sensitive dataset, it has succeeded. And conversely, an attack that only unreliably achieves high aggregate attack success rate should not be considered successful."

One further methodological note worth recording: "all models we train use state-of-the-art generalization-enhancing techniques. This makes our analysis much more realistic than prior work, which often uses models with 2–5× higher error rates than our models." An attack evaluated on an overfitted model flatters itself.

3. Consequences for this project — the metric, not just the attack

SCRUB reports its basic MIA as an accuracy. Verified in its own words: "For a perfect defense, the attacker's accuracy is 50%, indicating that it is unable to separate the two sets." That is precisely Definition 2, the metric Carlini argues is "inherently inadequate."

⚠️ Be fair to SCRUB here — it is not unaware of this argument

An earlier draft of this note implied SCRUB ignores Carlini's critique. It does not. SCRUB's Figure 4 reports log-scale ROC curves and AUC for its LiRA-for-unlearning attack, and its §11 says, citing this paper directly:

"A successful defense is associated with a smaller Area Under the Curve (AUC) … Carlini et al. [2022] however advocate that the AUC is not a [sufficient metric, directing attention to the] region of the ROC curve associated with very small False Positive Rates … We are primarily interested in the area of small False Positive Rate [Carlini et al., 2022]."

So the accurate statement is narrower and more interesting: SCRUB applies Carlini's methodology to its strong attack (LiRA) but not to its basic attack, and it is the basic attack that populates Tables 2 and 21–28 — including Table 23, our Phase 2 target.

The gap we can close is therefore modest and specific: apply the reporting standard SCRUB already endorses to the metric it did not apply it to.

This compounds the Hayes critique along an orthogonal axis:

Both land on the same numbers — SCRUB's Table 23, our Phase 2 target. And note the LOSS attack in Carlini's illustration is exactly the "basic loss-threshold attack" our Stage 6 plan specifies. On a well-generalising model it is a strong non-membership attack.

Stage 6 recommendation: reproduce the metric, add the better one

Tier A must report MIA accuracy, because that is what SCRUB reports and reproduction means matching it. But the underlying per-example scores are the same object either way, so TPR at low FPR and a log-scale ROC cost essentially nothing extra to compute — it is a different aggregation of numbers we will already have.

Proposal: the harness computes and stores per-example attack scores, not just the summary accuracy, and the report generator emits accuracy (for Tier A comparability) and TPR@0.1%FPR / TPR@1%FPR / log-ROC AUC (for the actual privacy claim). Storing per-example scores also makes the per-example variance Hayes et al. describe visible, which a summary statistic hides by construction.

This is a design decision to take before the harness freezes, and it is cheap. It does not require LiRA or shadow models.

5. What the paper does not do, and what is still unread

§IV has now been read directly and is recorded in §4 above; the earlier secondhand account, taken from Hayes et al.'s description, has been replaced by the source. Results, ablations and the defence discussion (§V–§VII) remain unread, so no reported number is recorded beyond the shadow-model counts in the experimental setup, which are configuration rather than outcome.

Still open: the four defences and the DP-SGD evaluation, which would matter if this project ever needed to argue about mitigations rather than about measurement.

4. §IV — the attack itself

The note above covers why the metric is wrong. This section covers what Carlini et al. put in its place, because the construction is what determines whether this project can afford it.

Membership inference is a hypothesis test. The game in Definition 1 asks the adversary to separate two worlds, so §IV-A formalises it as exactly that. Define two distributions over models,

and ask which one the observed model was drawn from. The choice of test is not a heuristic; it is forced:

"We perform this hypothesis test according to the Neyman-Pearson lemma, which states that the best hypothesis test at a fixed false positive rate is obtained by thresholding the Likelihood-ratio Test between the two hypotheses."

Equation 2 — the ideal test, over model parameters:

Why this settles the metric argument. §III-B says balanced accuracy is the wrong summary; §IV-A says what the right one is and why. If the optimal test at a fixed false positive rate is a likelihood ratio, then the quantity that characterises an attack is its whole ROC curve in the low-FPR regime, and a single balanced accuracy is a lossy projection of it. The two halves of the paper are one argument, and citing only the first half understates it.

Making it computable. Equation 2 is intractable — \mathbb{Q}_{\text{in}} and \mathbb{Q}_{\text{out}} "are not analytically known". Two reductions follow, and both matter for anyone implementing it:

1. Reduce to one dimension. Replace the distributions over models with \tilde{\mathbb{Q}}_{\text{in/out}}, the distributions of losses on (x,y) for models trained or not trained on it. "This is now a likelihood test for a one-dimensional statistic, which can be efficiently computed with query access to f."
2. Assume the one-dimensional distributions are Gaussian, "reducing our attack to estimating just four parameters: the mean and variance of each distribution."

The logit scaling, and why it is not cosmetic. The Gaussian assumption does not hold on the natural scales, and §IV-B says so explicitly: the cross-entropy loss "is on a logarithmic scale", while its negative exponent — the model confidence f(x)_y — is "bounded in the interval [0,1] and thus not normally distributed either (i.e., the confidences for outliers and inliers concentrate, respectively, around 0 and 1)". Hence

applied to the confidence before fitting. This is the transform Are we making progress in unlearning? Findings from the first NeurIPS unlearning competition adopts for its per-example scalar, and reading it here confirms the connection at the source rather than by inference.

Algorithm 1, the online attack. Train N shadow models on random samples from \mathbb{D}, arranged so half contain the target point and half do not; fit one Gaussian to the IN confidences and one to the OUT confidences in logit scale; query the target model and output the parametric likelihood-ratio test. It amortises across targets: "we train shadow models on N subsets of D, chosen so that each target (x,y) \in D appears in N/2 subsets. The same N shadow models can then be used to estimate the Likelihood-ratio test for all examples in D."

The offline attack, and the reason it exists — a usability limitation they name themselves. The offline variant "trains shadow models on randomly sampled datasets ahead of time, and never trains shadow models on the target points", estimating only \mu_{\text{out}} and \sigma^2_{\text{out}} and replacing the two-sided test with a one-sided one: the probability of observing a confidence as high as the target model's under the null hypothesis that the point is a non-member.

What this costs, and what it means for this project

The shadow-model counts are stated in the experimental setup: N = 64 for ImageNet and N = 256 otherwise for the online attack; the offline results use 128 OUT models (32 for ImageNet). That is the concrete price of the critique this project has been repeating.

Saying "SCRUB's MIA is the weakest category" is cheap. Replacing it is not: a faithful LiRA means training on the order of a hundred additional models per dataset, against a project budget that is arguing about whether 5 seeds or 10 is affordable. The offline variant is the only version worth costing, because it is the only one whose shadow models can be trained once and reused across every method and every forget request.

This does not weaken the critique — the attack's weakness is still real, and Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy shows what it hides. It fixes what the project can honestly say about it: that the stronger attack is specified, affordable in its offline form only, and out of budget in its online form, which is a statement about resources rather than about validity.

One assumption to note if it is ever implemented. Their shadow models are trained on subsets of the same dataset as the target: "the training sets of individual shadow models and the target model may partially overlap. This is a strong assumption, which we make here mainly due to the small size of some of the datasets we consider." They report in §VI-D that the attack works with fully disjoint shadow training sets, which is the configuration a reproduction would want.

6. Links

• Adapted to unlearning by Towards Unbounded Machine Unlearning as its LiRA-for-unlearning attack, and used as the per-example exemplar by Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy.
• Its metric critique and Hayes et al.'s attack-strength critique are independent and both apply to SCRUB's basic-MIA tables.
• Relevant to When Machine Unlearning Jeopardizes Privacy, which attacks unlearning itself rather than training.

@inproceedings{carlini2022lira,
  title         = {Membership Inference Attacks From First Principles},
  author        = {Nicholas Carlini and Steve Chien and Milad Nasr and Shuang Song and Andreas Terzis and Florian Tramer},
  year          = {2022},
  booktitle     = {IEEE Symposium on Security and Privacy (S\&P) 2022},
  eprint        = {2112.03570},
  archiveprefix = {arXiv},
  url           = {https://arxiv.org/abs/2112.03570}
}