Machine Learning in Montpellier, Theory & Practice

In this talk, we present some technical details of privacy accounting for differentially private mechanisms. We focus on privacy accounting for Differentially Private Stochastic Gradient Descent (DP-SGD). First, we introduce various techniques used to analyze the privacy guarantees of DP-SGD. Then, we discuss two different batch generation methods. Most privacy accounting techniques assume that batches for DP-SGD are generated using Poisson subsampling. However, Poisson subsampling is often not feasible in practice when training models on large datasets. We can instead implement a different technique such as sampling without replacement. The choice of sampling scheme has typically been regarded as a minor technical detail. We demonstrate that the sampling scheme significantly impacts the privacy guarantees for some (realistic) DP-SGD hyperparameters.

Map /Briefcase/RF-Inria_logo_signat.png" target="_blank" rel="noopener">Online