Skip to main content

DPSGD Subgroup Disparity

Analyzing Differentially Private Stochastic Gradient Descent (DPSGD) Under Class Imbalance and Subgroup Disparity

Author

Kacper Lisowicz, Yunzhe Zheng, Daniel Lin

Training a model with differential privacy (DP) guarantees that no single person’s record changes the model much. The standard method, DP-SGD, does this by capping each example’s influence (clipping) and adding random noise. The guarantee is defined per record, but the cost in accuracy is felt per group. Overall accuracy can hide that a few groups pay most of it.

This project reproduces and analyses Esipova et al., Disparate Impact in Differential Privacy from Gradient Misalignment (ICLR 2023). We ask two questions:

  1. Does DP-SGD’s accuracy cost fall unevenly when a group is rare (MNIST) and when it is just harder to learn (Adult)? And how does that change on larger, modern census data as the training set grows (Folktables, our extension)?
  2. Does the paper’s mitigation, DPSGD-Global-Adapt, reduce that unevenness?

How we measure it

For each group \(k\), the privacy cost is \(\pi_k\) = accuracy without privacy − accuracy with privacy, in percentage points. The privacy-cost gap \(\lvert \pi_a - \pi_b \rvert\) is how much more one group pays than the other. The gap is not the plain accuracy difference, because groups can differ in accuracy before any privacy is added.

Results

MNIST: Class Imbalance

Figure 1: Privacy-cost gap \(\lvert \pi_2 - \pi_8 \rvert\): ours vs Esipova et al. (2023), Table 2. Mean ± SE over 5 seeds.

The digit 8 is made rare on purpose (~1% of training data).

  • DP-SGD: the 8 loses about 61 points of accuracy, the 2 about 9.
  • Global-Adapt: shrinks the gap from about 53 to about 13 points, but does not close it.

Adult: Subgroup Disparity

Figure 2: Privacy-cost gap \(\lvert \pi_{\text{Male}} - \pi_{\text{Female}} \rvert\): ours vs Esipova et al. (2023), Table 4. Mean ± SE over 5 seeds.

Census income prediction, with men and women balanced to equal size.

  • DP-SGD: costs the harder group (men) about 11 points, women about 4.
  • Global-Adapt: nearly closes the gap (about 0.1 ± 0.2 points).

Folktables: Census-Scale Subgroup Disparity

Figure 3: Privacy-cost gap \(\lvert \pi_{\text{Male}} - \pi_{\text{Female}} \rvert\) vs training-set size, CA natural ratio, \(\varepsilon = 3.41\), same test set at every \(n\) (ours; mean ± SE).

2018 California census data (195,665 people) at a fixed \(\varepsilon = 3.41\), varying training-set size.

  • DP-SGD: costs men about 10 points more than women at 5–10k examples, under 1 point from 50k on.
  • Global-Adapt: keeps the gap at or below about 0.6 points at every size.

MNIST and Adult closely track the paper’s published numbers (hatched bars: Esipova et al.; solid: ours). Folktables has no published DP-SGD target: all its numbers are ours, and it is an extension, not a reproduction.

Why it Happens

  • Large gradients get clipped most. Examples the model fits poorly produce large gradients, and clipping shrinks those the most.
  • Clipping changes direction, not just size. That tilts each update toward the well-fit majority. Esipova et al. call this gradient misalignment.
  • The bias accumulates. The noise is random and largely averages out over training; the clipping bias is systematic.
  • Global-Adapt’s fix: it scales all gradients by one shared factor, which preserves their relative sizes and the update’s direction. It never uses group labels.

Our results so far are consistent with this explanation. Testing it directly is the next step. On Folktables, the large gaps appear when DP-SGD falls back to the majority answer, which happens with small training sets. There, the cost comes almost entirely from clipping rather than noise: the gap barely changes between \(\varepsilon = 0.5\) and 8. Because every run trains for 20 epochs, data size and training length also grow together in that sweep.

Next Steps

  • Mechanism experiments with per-group clipping and gradient angles.
  • Smaller checks: Adult at its natural sex ratio, and a matched-learning-rate check of Global-Adapt vs DP-SGD. On Folktables, the non-private baseline has already been checked at Global-Adapt’s learning rate; the gap conclusions hold.
  • Folktables follow-up, with a size sweep at a fixed number of training steps, to separate data size from training length.