v2.1

Releasing two DP pre-trained VisionTransformer models (ViT; ~86M param). The models are trained following Pre-training Differentially Private Models with Limited Public Data in NeurIPS 2024. The pre-training uses ImageNet-1k with AdamW (1k classes, 1 million images) and ImageNet-11k with DP-AdamW (11k classes, 11 million images). Note the DP guarantee is only valid on ImageNet-11k excluding ImageNet-1k.

v2.0

1. Adding support to DeepSpeed and FSDP through DP-ZeRO on multi-GPU
2. Adding a second approach to compute private gradient. This approach re-writes and extends the torch layers' back-propagation. New approach does not need ghost differentiation, may be slower (but improvable), and is much more generally applicable.
3. Removing param.summed_clipped_grad and replacing with param.private_grad
4. Adding ZeRO examples for image classification and GPT
5. Adding mixed precision training (fp16 and bf16)

v1.1

1. re-design hooks so that the 'privacy_engine.attach()' function is removed, making DP training script more similar to the standard non-private one;
2. adding per-layer and group-wise clipping, which enjoy the same training speed as flat/all-layer clipping.

v1.0

Fast differentially private optimization for large-scale deep learning. The training speed is about 0.9 times of standard non-private training, and the memory overhead is about 1% than standard training.

Examples include image classification (CIFAR10/CIFAR100/CelebA), text classification (SST2/QNLI/QQP/MNLI), and text generation (E2E/DART). This library supports multiple per-sample clipping functions (especially automatic clipping), clipping modes, fine-tuning methods (especially DP-BiTFiT). The user-interface is designed to be as close to standard training as possible, without virtual_step or per-sample losses and with explicit backward().