Opacus

https://opacus.ai/
Admin
Opacus is a PyTorch library that makes it easy to train neural networks with differential privacy...
Tokens:132,862
Snippets:804
Trust Score:8.9
Update:6 months ago
Show doc for...
Tokens:
Raw
### Install Opacus from Source

Source: https://opacus.ai/

Follow these steps to clone the Opacus repository and install it from source.

```bash
git clone https://github.com/pytorch/opacus.git
cd opacus
pip install -e .
```

--------------------------------

### Initialize DPOptimizer

Source: https://opacus.ai/api/_modules/opacus/optimizers/optimizer.html

Example of wrapping a standard PyTorch optimizer with DPOptimizer for differentially private training.

```python
>>> module = MyCustomModel()
>>> optimizer = torch.optim.SGD(module.parameters(), lr=0.1)
>>> dp_optimizer = DPOptimizer(
...     optimizer=optimizer,
...     noise_multiplier=1.0,
...     max_grad_norm=1.0,
...     expected_batch_size=4,
... )
```

--------------------------------

### Install Opacus via Pip

Source: https://opacus.ai/

Use this command to install the Opacus library using pip.

```bash
pip install opacus
```

--------------------------------

### Install Opacus via Conda

Source: https://opacus.ai/

Use this command to install the Opacus library using conda.

```bash
conda install -c conda-forge opacus
```

--------------------------------

### Launch Training Loop

Source: https://opacus.ai/tutorials/ddp_tutorial

Launches the distributed training process, including model initialization, loss calculation, backpropagation, and optimizer steps. This function is intended to be called for each process in a distributed setup.

```python
import numpy as np

def test(model, device):
    test_loader = DataLoader(
        mnist_test_ds,
        batch_size=BATCH_SIZE,
    )

    model.eval()
    correct = 0
    with torch.no_grad():
        for data, target in test_loader:
            data, target = data.to(device), target.to(device)
            output = model(data)
            pred = output.argmax(
                dim=1, keepdim=True
            )
            correct += pred.eq(target.view_as(pred)).sum().item()

    model.train()
    return correct / len(mnist_test_ds)

def launch(rank, world_size, epochs):
    setup(rank, world_size)
    criterion = nn.CrossEntropyLoss()

    model, optimizer, data_loader, privacy_engine = init_training(rank)
    model.to(rank)
    model.train()


    for e in range(epochs):
        losses = []
        correct = 0
        total = 0

        for data, target in data_loader:
            data, target = data.to(rank), target.to(rank)
            optimizer.zero_grad()
            output = model(data)

            pred = output.argmax(dim=1, keepdim=True)
            correct += pred.eq(target.view_as(pred)).sum().item()
            total += len(data)

            loss = criterion(output, target)
            loss.backward()
            optimizer.step()

```

--------------------------------

### Compute DP-SGD Privacy Command-Line Example

Source: https://opacus.ai/api/compute_dp_sgd_privacy.html

Use this command to estimate the privacy budget of a model trained with DP-SGD. Specify the number of epochs, target delta, sample rate, noise multiplier, and a list of RDP orders (alphas).

```bash
python -m opacus.scripts.compute_dp_sgd_privacy –epochs=3 –delta=1e-5 –sample-rate 0.01 –noise-multiplier 1.0 –alphas 2 5 10 20 100
```

--------------------------------

### Initialize PrivacyEngine with Ghost Clipping

Source: https://opacus.ai/tutorials/building_text_classifier

Sets up the PrivacyEngine with grad_sample_mode='ghost' and executes the training loop.

```python
device = torch.device("cuda:0")
os.environ["CUDA_LAUNCH_BLOCKING"] = "1"

optimizer = torch.optim.AdamW(model.parameters(), lr=5e-4, eps=1e-8)
model = model.train()

privacy_engine = PrivacyEngine()
criterion = nn.CrossEntropyLoss(reduction="mean")

model_gc, optimizer_gc, criterion_gc, train_dataloader = (
    privacy_engine.make_private_with_epsilon(
        module=model,
        optimizer=optimizer,
        data_loader=train_dataloader,
        criterion=criterion,
        target_delta=DELTA,
        target_epsilon=EPSILON,
        epochs=EPOCHS,
        max_grad_norm=MAX_GRAD_NORM,
        grad_sample_mode="ghost",
    )
)

model_gc = model_gc.to(device)
model_gc = model_gc.train()

for epoch in range(1, EPOCHS + 1):
    losses = []
    for step, batch in enumerate(tqdm(train_dataloader)):
        optimizer_gc.zero_grad()
        batch = tuple(t.to(device) for t in batch)
        inputs = {
            "input_ids": batch[0],
            "attention_mask": batch[1],
            "token_type_ids": batch[2],
            "labels": batch[3],
        }
        outputs = model_gc(**inputs)  # output = loss, logits, hidden_states, attentions
        loss = criterion_gc(outputs[1], batch[3])
        loss.backward()
        optimizer_gc.step()
        losses.append(loss.item())

        if step > 0 and step % LOGGING_INTERVAL == 0:
            train_loss = np.mean(losses)
            eval_loss, eval_accuracy = evaluate(model_gc)
            eps = privacy_engine.get_epsilon(DELTA)
            print(
                f"Epoch: {epoch} | "
                f"Step: {step} | "
                f"Train loss: {train_loss:.3f} | "
                f"Eval loss: {eval_loss:.3f} | "
                f"Eval accuracy: {eval_accuracy:.3f} | "
                f"ɛ: {eps:.2f}"
            )
```

--------------------------------

### Initialize DPOptimizerFastGradientClipping

Source: https://opacus.ai/api/optim/dp_optimizer_fast_gradient_clipping.html

Demonstrates how to wrap a standard PyTorch optimizer with the DPOptimizerFastGradientClipping class.

```python
>>> module = MyCustomModel()
>>> optimizer = torch.optim.SGD(module.parameters(), lr=0.1)
>>> dp_optimizer = DPOptimizerFastGradientClipping(
...     optimizer=optimizer,
...     noise_multiplier=1.0,
...     max_grad_norm=1.0,
...     expected_batch_size=4,
... )
```

--------------------------------

### Initialize DPOptimizerFastGradientClipping

Source: https://opacus.ai/api/_modules/opacus/optimizers/optimizer_fast_gradient_clipping.html

Instantiate DPOptimizerFastGradientClipping with a base optimizer, noise multiplier, max grad norm, and expected batch size. Other parameters like loss reduction, generator, and secure mode are optional.

```python
dp_optimizer = DPOptimizerFastGradientClipping(
    optimizer=optimizer,
    noise_multiplier=1.0,
    max_grad_norm=1.0,
    expected_batch_size=4,
)
```

--------------------------------

### disable_hooks

Source: https://opacus.ai/api/grad_sample_module.html

Globally disables all hooks installed by the Opacus library.

```APIDOC
## disable_hooks

### Description
Globally disables all hooks installed by the Opacus library. This is a workaround for a known issue in PyTorch's Autograd where removing hooks might not function as expected if the computation graph has already been constructed.

### Return Type
`None`
```

--------------------------------

### Configure Model and Optimizer

Source: https://opacus.ai/tutorials/building_text_classifier

Prepare the model for training by moving it to the appropriate device and initializing the AdamW optimizer.

```python
# Move the model to appropriate device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)

# Set the model to train mode (HuggingFace models load in eval mode)
model = model.train()
# Define optimizer
optimizer = torch.optim.AdamW(model.parameters(), lr=5e-4, eps=1e-8)
```

--------------------------------

### Initialize Optimizer and Loss Function

Source: https://opacus.ai/tutorials/building_image_classifier

Sets up a standard PyTorch loss function and optimizer compatible with Opacus.

```python
import torch.nn as nn
import torch.optim as optim

criterion = nn.CrossEntropyLoss()
optimizer = optim.RMSprop(model.parameters(), lr=LR)
```

--------------------------------

### Get Submodule by Path

Source: https://opacus.ai/api/_modules/opacus/utils/module_utils.html

Retrieves a submodule by its dot-separated path string. Throws an AttributeError if the path is invalid.

```python
def get_submodule(module: nn.Module, target: str) -> nn.Module:
    """
    Returns the submodule given by target if it exists, otherwise throws an error.

    This is copy-pasta of Pytorch 1.9's ``get_submodule()`` implementation; and is
    included here to also support Pytorch 1.8. This function can be removed in favour
    of ``module.get_submodule()`` once Opacus abandons support for torch 1.8.

    See more details at https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=get_submodule#torch.nn.Module.get_submodule

    Args:
        module: module
        target: submodule string

    Returns:
        The submodule given by target if it exists

    Raises:
        AttributeError
            If submodule doesn't exist
    """

    if target == "":
        return module

    atoms: List[str] = target.split(".")
    mod: nn.Module = module

    for item in atoms:
        if not hasattr(mod, item):
            raise AttributeError(
                mod._get_name() + " has no " "attribute `" + item + "`"
            )
        mod = getattr(mod, item)
        if not isinstance(mod, torch.nn.Module):
            raise AttributeError("`" + item + "` is not " "an nn.Module")
    return mod
```

--------------------------------

### Get Accounting Mechanism Name - RDPAccountant

Source: https://opacus.ai/api/accounting/rdp.html

The mechanism class method returns the name of the accounting mechanism used by RDPAccountant.

```python
classmethod mechanism()
```

--------------------------------

### Initialize Distributed Environment with Opacus and DDP

Source: https://opacus.ai/tutorials/ddp_tutorial

Sets up the distributed environment for DDP training using PyTorch. Ensure 'gloo' is available or use 'nccl' for GPU-accelerated communication.

```python
%%writefile opacus_ddp_demo.py
import os
import torch.distributed as dist
import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'

    # initialize the process group
    dist.init_process_group("gloo", rank=rank, world_size=world_size)

def cleanup():
    dist.destroy_process_group()
```

--------------------------------

### Define CharByteEncoder Module

Source: https://opacus.ai/tutorials/building_lstm_name_classifier

A PyTorch module that encodes UTF-8 strings into byte-based tensors with start, end, and padding tokens.

```python
import torch
import torch.nn as nn

class CharByteEncoder(nn.Module):
    """
    This encoder takes a UTF-8 string and encodes its bytes into a Tensor. It can also
    perform the opposite operation to check a result.
    Examples:
    >>> encoder = CharByteEncoder()
    >>> t = encoder('Ślusàrski')  # returns tensor([256, 197, 154, 108, 117, 115, 195, 160, 114, 115, 107, 105, 257])
    >>> encoder.decode(t)  # returns "<s>Ślusàrski</s>"
    """

    def __init__(self):
        super().__init__()
        self.start_token = "<s>"
        self.end_token = "</s>"
        self.pad_token = "<pad>"

        self.start_idx = 256
        self.end_idx = 257
        self.pad_idx = 258

    def forward(self, s: str, pad_to=0) -> torch.LongTensor:
        """
        Encodes a string. It will append a start token <s> (id=self.start_idx) and an end token </s>
        (id=self.end_idx).
        Args:
            s: The string to encode.
            pad_to: If not zero, pad by appending self.pad_idx until string is of length `pad_to`.
                Defaults to 0.
        Returns:
            The encoded LongTensor of indices.
        """
        encoded = s.encode()
        n_pad = pad_to - len(encoded) if pad_to > len(encoded) else 0
        return torch.LongTensor(
            [self.start_idx]
            + [c for c in encoded]  # noqa
            + [self.end_idx]
            + [self.pad_idx for _ in range(n_pad)]
        )

    def decode(self, char_ids_tensor: torch.LongTensor) -> str:
        """
        The inverse of `forward`. Keeps the start, end, and pad indices.
        """
        char_ids = char_ids_tensor.cpu().detach().tolist()

        out = []
        buf = []
        for c in char_ids:
            if c < 256:
                buf.append(c)
            else:
                if buf:
                    out.append(bytes(buf).decode())
                    buf = []
                if c == self.start_idx:
                    out.append(self.start_token)
                elif c == self.end_idx:
                    out.append(self.end_token)
                elif c == self.pad_idx:
                    out.append(self.pad_token)

        if buf:  # in case some are left
            out.append(bytes(buf).decode())
        return "".join(out)

    def __len__(self):
        """
        The length of our encoder space. This is fixed to 256 (one byte) + 3 special chars
        (start, end, pad).
        Returns:
            259
        """
        return 259
```

--------------------------------

### Instantiate Model and Device

Source: https://opacus.ai/tutorials/building_lstm_name_classifier

Configures the device and initializes the CharNNClassifier with specific embedding and hidden layer dimensions.

```python
# Set the device to run on a GPU
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Define classifier parameters
embedding_size = 64
hidden_size = 128  # Number of neurons in hidden layer after LSTM
n_lstm_layers = 1
bidirectional_lstm = False

model = CharNNClassifier(
    embedding_size,
    hidden_size,
    len(ds.labels),
    n_lstm_layers,
    bidirectional_lstm,
).to(device)
```

--------------------------------

### GET opacus.accountants.utils.get_noise_multiplier

Source: https://opacus.ai/api/accounting/utils.html

Computes the noise level sigma required to reach a total privacy budget of (target_epsilon, target_delta) given specific training parameters.

```APIDOC
## get_noise_multiplier

### Description
Computes the noise level sigma to reach a total budget of (target_epsilon, target_delta) at the end of epochs, with a given sample_rate.

### Parameters
- **target_epsilon** (float) - Required - The privacy budget's epsilon.
- **target_delta** (float) - Required - The privacy budget's delta.
- **sample_rate** (float) - Required - The sampling rate (usually batch_size / n_data).
- **epochs** (int) - Optional - The number of epochs to run.
- **steps** (int) - Optional - Number of steps to run.
- **accountant** (str) - Optional - Accounting mechanism used to estimate epsilon (default: 'rdp').
- **epsilon_tolerance** (float) - Optional - Precision for the binary search (default: 0.01).

### Response
- **Return Type** (float) - The noise level sigma to ensure privacy budget of (target_epsilon, target_delta).
```

--------------------------------

### Initialize and execute DP-SGD training

Source: https://opacus.ai/tutorials/building_text_classifier

Sets up the PrivacyEngine, makes the model private with Ghost Clipping, and runs the training loop.

```python
device = torch.device("cuda:0")
os.environ["CUDA_LAUNCH_BLOCKING"] = "1"

optimizer = torch.optim.AdamW(model_with_lora.parameters(), lr=5e-4, eps=1e-8)
model_with_lora = model_with_lora.train()

privacy_engine = PrivacyEngine()
criterion = nn.CrossEntropyLoss(reduction="mean")

model_lora, optimizer_lora, criterion_lora, train_dataloader = (
    privacy_engine.make_private_with_epsilon(
        module=model_with_lora,
        optimizer=optimizer,
        data_loader=train_dataloader,
        criterion=criterion,
        target_delta=DELTA,
        target_epsilon=EPSILON,
        epochs=EPOCHS,
        max_grad_norm=MAX_GRAD_NORM,
        grad_sample_mode="ghost",
    )
)

model_lora = model_lora.to(device)
model_lora = model_lora.train()

for epoch in range(1, EPOCHS + 1):
    losses = []
    for step, batch in enumerate(tqdm(train_dataloader)):
        optimizer_lora.zero_grad()
        batch = tuple(t.to(device) for t in batch)
        inputs = {
            "input_ids": batch[0],
            "attention_mask": batch[1],
            "token_type_ids": batch[2],
            "labels": batch[3],
        }
        outputs = model_lora(**inputs)  # output = loss, logits, hidden_states, attentions
        loss = criterion_lora(outputs[1], batch[3])
        loss.backward()
        optimizer_lora.step()
        losses.append(loss.item())

        if step > 0 and step % LOGGING_INTERVAL == 0:
            train_loss = np.mean(losses)
            eval_loss, eval_accuracy = evaluate(model_lora)
            eps = privacy_engine.get_epsilon(DELTA)
            print(
                f"Epoch: {epoch} | "
                f"Step: {step} | "
                f"Train loss: {train_loss:.3f} | "
                f"Eval loss: {eval_loss:.3f} | "
                f"Eval accuracy: {eval_accuracy:.3f} | "
                f"ɛ: {eps:.2f}"
            )
```

--------------------------------

### Get Epsilon from RDPAccountant

Source: https://opacus.ai/api/_modules/opacus/accountants/rdp.html

A convenience method to retrieve only the epsilon value from the privacy spent calculation. It uses the RDP mechanism to determine the privacy budget.

```python
eps = accountant.get_epsilon(delta=1e-5)
```

--------------------------------

### DistributedDPOptimizer

Source: https://opacus.ai/api/_modules/opacus/optimizers/ddpoptimizer.html

The DistributedDPOptimizer class provides privacy-preserving optimization for distributed training setups. It handles noise addition on a single worker and gradient reduction across all workers.

```APIDOC
## DistributedDPOptimizer

### Description

This class extends `DPOptimizer` to support distributed data processing. It ensures that noise is added only on the primary worker (rank 0) and gradients are correctly reduced across all distributed workers.

### Method

`__init__`

### Parameters

- **optimizer** (Optimizer) - Required - The original PyTorch optimizer to wrap.
- **noise_multiplier** (float) - Required - The noise multiplier for differential privacy.
- **max_grad_norm** (float) - Required - The maximum gradient norm for clipping.
- **expected_batch_size** (Optional[int]) - Optional - The expected batch size for privacy accounting.
- **loss_reduction** (str) - Optional - The method for reducing loss, defaults to 'mean'.
- **generator** (Optional) - Optional - A random number generator.
- **secure_mode** (bool) - Optional - Whether to enable secure mode, defaults to False.

### Methods

#### `add_noise()`

Adds noise to the gradients. This operation is performed only on the worker with rank 0.

#### `reduce_gradients()`

Reduces the gradients across all distributed workers using `torch.distributed.all_reduce`. If `loss_reduction` is 'mean', gradients are divided by the world size.

#### `step(closure: Optional[Callable[[], float]] = None)`

Performs a single optimization step. It first calls `pre_step()`, then `reduce_gradients()`, and finally steps the original optimizer. Returns `None` if `pre_step()` returns `False`.

### Request Example

```python
# Example initialization (assuming distributed environment is set up)
import torch
from torch.optim import SGD
from opacus.optimizers.ddpoptimizer import DistributedDPOptimizer

# Assume model parameters are defined
params = model.parameters()
original_optimizer = SGD(params, lr=0.01)

ddp_optimizer = DistributedDPOptimizer(
    optimizer=original_optimizer,
    noise_multiplier=1.0,
    max_grad_norm=1.0,
    expected_batch_size=64
)
```

### Response

#### Success Response (200)

- **Optional[torch.Tensor]** - The result of the optimizer step, or None if the step was skipped.
```

--------------------------------

### Initialize DPTensorFastGradientClipping

Source: https://opacus.ai/api/_modules/opacus/utils/fast_gradient_clipping_utils.html

Initializes the DPTensorFastGradientClipping class, which packages the training loop for Fast Gradient and Ghost Clipping into loss.backward(). Requires the module, optimizer, and per-sample loss tensor.

```python
class DPTensorFastGradientClipping:
    """
    Packages the training loop for Fast Gradient and Ghost Clipping into loss.backward().
    """

    def __init__(
        self,
        module: GradSampleModuleFastGradientClipping,
        optimizer: DPOptimizerFastGradientClipping,
        loss_per_sample: torch.Tensor,
        loss_reduction: str = "mean",
    ):
        """
        Args:
            module: the module to train
            optimizer: the optimizer used to train the module
            loss_per_sample: loss on each sample in the mini-batch of size [batch_size, 1]
        """
        self.module = module
        self.optimizer = optimizer
        self.loss_per_sample = loss_per_sample
        self.loss_reduction = loss_reduction
```

--------------------------------

### Instantiate ResNet18 Model

Source: https://opacus.ai/tutorials/building_image_classifier

This snippet shows how to instantiate a ResNet18 model with 10 output classes using torchvision.models. This is a common starting point for image classification tasks.

```python
from torchvision import models

model = models.resnet18(num_classes=10)
```

--------------------------------

### Initialize GradSampleModuleFastGradientClipping

Source: https://opacus.ai/api/grad_sample_module_fast_gradient_clipping.html

Wraps an nn.Module to enable efficient per-sample gradient norm computation.

```python
class opacus.grad_sample.grad_sample_module_fast_gradient_clipping.GradSampleModuleFastGradientClipping(_m_ , _*_ , _batch_first =True_, _loss_reduction ='mean'_, _strict =True_, _force_functorch =False_, _max_grad_norm =1_, _use_ghost_clipping =True_)[source]
```

--------------------------------

### Disable Hooks Globally in Opacus

Source: https://opacus.ai/api/_modules/opacus/grad_sample/grad_sample_module.html

Globally disables all hooks installed by the Opacus library. This is a workaround for a PyTorch bug where removing hooks might not function correctly if the graph is already constructed.

```python
def disable_hooks(self) -> None:
        r"""
        Globally disable all hooks installed by this library.
        Why is this needed? As per https://github.com/pytorch/pytorch/issues/25723, there is
        a bug in Autograd that makes removing hooks do nothing if the graph was already
        constructed. For this reason, we have this method to at least turn them off.
        """
        self.hooks_enabled = False
```

--------------------------------

### Get Privacy Spent with RDPAccountant

Source: https://opacus.ai/api/_modules/opacus/accountants/rdp.html

Calculates the total privacy spent (epsilon) and the best alpha for a given delta. It aggregates RDP values from the history and converts them to (epsilon, delta)-DP.

```python
eps, best_alpha = accountant.get_privacy_spent(delta=1e-5)
```

--------------------------------

### DPOptimizerFastGradientClipping Initialization

Source: https://opacus.ai/api/_modules/opacus/optimizers/optimizer_fast_gradient_clipping.html

Initializes the DPOptimizerFastGradientClipping with a base optimizer and privacy-specific parameters.

```APIDOC
## DPOptimizerFastGradientClipping

### Description
``torch.optim.Optimizer`` wrapper to implement Fast Gradient and Ghost Clipping -- modifies DPOptimizer to only add noise to the average gradient, without clipping.

Can be used with any ``torch.optim.Optimizer`` subclass as an underlying optimizer. ``DPOptimizerFastGradientClipping`` assumes that parameters over which it performs optimization belong to GradSampleModuleFastGradientClipping and therefore have the ``grad_sample`` attribute.

On a high level ``DPOptimizerFastGradientClipping``'s step looks like this:
1) Add Gaussian noise to ``p.grad`` calibrated to a given noise multiplier and
max grad norm limit (``std = noise_multiplier * max_grad_norm``).
2) Call underlying optimizer to perform optimization step

### Method
__init__

### Parameters
#### Arguments
- **optimizer** (Optimizer) - Required - The base optimizer to wrap.
- **noise_multiplier** (float) - Required - The noise multiplier for differential privacy.
- **max_grad_norm** (float) - Required - The maximum gradient norm for clipping.
- **expected_batch_size** (Optional[int]) - Required - The expected batch size for gradient averaging. Required if ``loss_reduction='mean'``, ignored if ``loss_reduction='sum'``.
- **loss_reduction** (str) - Optional - Indicates if the loss reduction is a sum or mean operation. Defaults to "mean".
- **generator** (torch.Generator) - Optional - A PyTorch generator object for randomness.
- **secure_mode** (bool) - Optional - If ``True``, uses a noise generation approach robust to floating point arithmetic attacks. Defaults to ``False``.

### Request Example
```python
module = MyCustomModel()
optimizer = torch.optim.SGD(module.parameters(), lr=0.1)
dp_optimizer = DPOptimizerFastGradientClipping(
    optimizer=optimizer,
    noise_multiplier=1.0,
    max_grad_norm=1.0,
    expected_batch_size=4,
)
```
```

--------------------------------

### Get Epsilon from GaussianAccountant

Source: https://opacus.ai/api/_modules/opacus/accountants/gdp.html

Calculates and returns the privacy budget (epsilon) expended so far, based on the target delta and whether Poisson sampling was used. It utilizes helper functions from opacus.analysis.gdp.

```python
def get_epsilon(self, delta: float, poisson: bool = True, **kwargs) -> float:
        """
        Return privacy budget (epsilon) expended so far.

        Args:
            delta: target delta
            poisson: ``True`` is input batches was sampled via Poisson sampling,
                ``False`` otherwise
        """

        compute_eps = (
            privacy_analysis.compute_eps_poisson
            if poisson
            else privacy_analysis.compute_eps_uniform
        )
        noise_multiplier, sample_rate, num_steps = self.history[-1]
        return compute_eps(
            steps=num_steps,
            noise_multiplier=noise_multiplier,
            sample_rate=sample_rate,
            delta=delta,
        )
```

--------------------------------

### Initialize PrivacyEngine

Source: https://opacus.ai/api/_modules/opacus/privacy_engine.html

Instantiate PrivacyEngine with options for the accounting mechanism and secure mode. Secure mode requires `torchcsprng` for cryptographic guarantees.

```python
privacy_engine = PrivacyEngine()

```

```python
privacy_engine = PrivacyEngine(accountant="rdp", secure_mode=True)

```

--------------------------------

### Initialize Distributed Training Components

Source: https://opacus.ai/tutorials/ddp_tutorial

Sets up the model with GradSampleModule and DPDDP, and configures the DistributedDPOptimizer for privacy-preserving training.

```python
%%writefile opacus_sync_demo.py

import sys
sys.path.append('/data/home/shilov/opacus')

from opacus_ddp_demo import setup, cleanup, mnist_train_ds, SampleConvNet
import logging
from torch.utils.data import DataLoader
import torch.optim as optim
from opacus.data_loader import DPDataLoader
from opacus import GradSampleModule
from opacus.distributed import DifferentiallyPrivateDistributedDataParallel as DPDDP
from opacus.optimizers import DistributedDPOptimizer
from torch.nn.parallel import DistributedDataParallel as DDP

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

BATCH_SIZE = 64
LR = 64

def init_training(rank, world_size):
    model = SampleConvNet()
    optimizer = optim.SGD(model.parameters(), lr=LR, momentum=0)

    model = GradSampleModule(model)
    model = DPDDP(model)

    optimizer = DistributedDPOptimizer(
        optimizer=optimizer,
        noise_multiplier=0.,
        max_grad_norm=100.,
        expected_batch_size=BATCH_SIZE//world_size,
    )

    data_loader = DPDataLoader.from_data_loader(
        data_loader=DataLoader(
            mnist_train_ds,
            batch_size=BATCH_SIZE,
        ),
        distributed=True,
    )


    return model, optimizer, data_loader
```

--------------------------------

### Wrapping Modules with GradSampleModule

Source: https://opacus.ai/tutorials/guide_to_grad_sampler

Shows how to wrap an existing PyTorch module to enable per-sample gradient tracking.

```python
import torch.nn as nn
from opacus.grad_sample import GradSampleModule

lin_mod = nn.Linear(42,2)
print(f"Before wrapping: {lin_mod}")

gs_lin_mod = GradSampleModule(lin_mod)
print(f"After wrapping : {gs_lin_mod}")
```

--------------------------------

### Get Privacy Budget (Epsilon) - RDPAccountant

Source: https://opacus.ai/api/accounting/rdp.html

Use get_epsilon to retrieve the privacy budget (epsilon) expended so far. It accepts a target delta and optionally a list of RDP orders (alphas) for optimal conversion.

```python
get_epsilon(_delta_ , _alphas =None_, _** kwargs_)
```

--------------------------------

### DPOptimizer Initialization

Source: https://opacus.ai/api/_modules/opacus/optimizers/optimizer.html

Initializes the DPOptimizer wrapper to enable per-sample gradient clipping and noise addition.

```APIDOC
## DPOptimizer.__init__

### Description
Initializes a new DPOptimizer instance that wraps an existing PyTorch optimizer to perform differentially private gradient updates.

### Parameters
#### Request Body
- **optimizer** (Optimizer) - Required - The underlying PyTorch optimizer to wrap.
- **noise_multiplier** (float) - Required - The ratio of the standard deviation of the Gaussian noise to the L2-sensitivity of the function.
- **max_grad_norm** (float) - Required - The maximum L2 norm of the per-sample gradients for clipping.
- **expected_batch_size** (int) - Optional - The batch size used for averaging gradients; required if loss_reduction is 'mean'.
- **loss_reduction** (str) - Optional - Specifies the reduction to apply to the output: 'mean' or 'sum'. Default is 'mean'.
- **generator** (torch.Generator) - Optional - Random number generator for noise generation.
- **secure_mode** (bool) - Optional - If True, uses a noise generation approach robust to floating-point arithmetic attacks.
```

--------------------------------

### CollateFnWithEmpty Usage Example

Source: https://opacus.ai/api/_modules/opacus/data_loader.html

Demonstrates how CollateFnWithEmpty handles batches. A non-empty batch defines the structure, and subsequent empty batches are transformed to match this structure with zero-length batch dimensions. This is crucial for maintaining DP guarantees.

```python
collator_fn = CollateFnWithEmpty(default_collate)
# First batch: [{"x": tensor([1, 2]), "y": tensor([3, 4])}]
# Empty batch: [] -> {"x": tensor([]), "y": tensor([])}
```

--------------------------------

### Execute Distributed Demo

Source: https://opacus.ai/tutorials/ddp_tutorial

Runs the generated Python script to observe the synchronization behavior.

```bash
!python -W ignore opacus_sync_demo.py
```

--------------------------------

### Class: GradSampleModuleFastGradientClipping

Source: https://opacus.ai/api/grad_sample_module_fast_gradient_clipping.html

Initializes the module wrapper for efficient per-sample gradient norm computation.

```APIDOC
## Class: GradSampleModuleFastGradientClipping

### Description
Hooks-based implementation of GradSampleModule with Fast Gradient and Ghost Clipping. Computes norms of gradients without gradient instantiation.

### Parameters
- **m** (Module) - Required - nn.Module to be wrapped
- **batch_first** (bool) - Optional - Flag to indicate if the input tensor has the first dimension representing the batch. Default: True
- **loss_reduction** (str) - Optional - Indicates if the loss reduction is a sum or a mean operation. Default: 'mean'
- **max_grad_norm** (float) - Optional - The value at which gradients are to be clipped. Default: 1
- **strict** (bool) - Optional - If True, validates that the module does not have buffers. Default: True
- **force_functorch** (bool) - Optional - If True, uses functorch to compute all per sample gradients. Default: False
- **use_ghost_clipping** (bool) - Optional - If True, uses Ghost Clipping for supported layers. Default: True
```

--------------------------------

### DPOptimizerFastGradientClipping Methods

Source: https://opacus.ai/api/_modules/opacus/optimizers/optimizer_fast_gradient_clipping.html

Details on methods available for gradient accumulation and zeroing.

```APIDOC
## DPOptimizerFastGradientClipping Methods

### accumulated_iterations

#### Description
Returns the number of batches currently accumulated and not yet processed.

In other words ``accumulated_iterations`` tracks the number of forward/backward passed done in between two optimizer steps. The value would typically be 1, but there are possible exceptions.

Used by privacy accountants to calculate real sampling rate.

### accumulate

#### Description
Performs gradient accumulation. Stores aggregated gradients into `p.summed_grad`.

### zero_grad

#### Description
Clears gradients. Clears ``p.grad`` and ``p.summed_grad`` for all of its parameters.

Notes:
``set_to_none`` argument only affects ``p.grad`` and ``p.summed_grad`` is never zeroed out and always set to None.
Normal grads can do this, because their shape is always the same.
Grad samples do not behave like this, as we accumulate gradients from different batches in a list.

#### Arguments
- **set_to_none** (bool) - Optional - Instead of setting to zero, set the grads to None. (only affects regular gradients. Per sample gradients are always set to None). Defaults to ``False``.
```

--------------------------------

### Initialize DPDataLoader

Source: https://opacus.ai/api/_modules/opacus/data_loader.html

Initializes a DPDataLoader instance. It handles distributed environments by selecting the appropriate sampler. Note that `drop_last` is ignored and a warning is logged if set.

```python
def __init__(
        self,
        dataset: Dataset,
        sample_rate: float,
        distributed: bool = False,
        generator=None,
        batch_first: bool = True,
        rand_on_empty: bool = False,
        **kwargs,
    ):
        """
        Initializes DPDataLoader.

        Args:
            dataset: Dataset to be iterated over.
            sample_rate: The probability of an element to be sampled.
            distributed: set ``True`` if you'll be using DPDataLoader in a DDP environment
                Selects between ``DistributedUniformWithReplacementSampler`` and
                ``UniformWithReplacementSampler`` sampler implementations
            rand_on_empty: set ``True`` to return a batch containing random numbers when encountering
                empty batches rather than tensors with zero-length batch dimensions
        """

        self.sample_rate = sample_rate
        self.distributed = distributed

        if distributed:
            batch_sampler = DistributedUniformWithReplacementSampler(
                total_size=len(dataset),  # type: ignore[assignment, arg-type]
                sample_rate=sample_rate,
                generator=generator,
            )
        else:
            batch_sampler = UniformWithReplacementSampler(
                num_samples=len(dataset),  # type: ignore[assignment, arg-type]
                sample_rate=sample_rate,
                generator=generator,
            )
        sample_empty_shapes = [(0, *shape_safe(x)) for x in dataset[0]]
        dtypes = [dtype_safe(x) for x in dataset[0]]

        if collate_fn is None:
            collate_fn = default_collate

        if drop_last:
            logger.warning(
                "Ignoring drop_last as it is not compatible with DPDataLoader."
            )

        super().__init__(
            dataset=dataset,
            batch_sampler=batch_sampler,
            collate_fn=wrap_collate_with_empty(
                collate_fn=collate_fn,
                batch_first=batch_first,
                rand_on_empty=rand_on_empty,
                sample_empty_shapes=sample_empty_shapes,
                dtypes=dtypes,
            ),
            generator=generator,
            **kwargs,
        )
```

--------------------------------

### Configure Optimizer and Loss Criterion

Source: https://opacus.ai/tutorials/building_lstm_name_classifier

Sets up the standard SGD optimizer and CrossEntropyLoss for the model.

```python
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
```

--------------------------------

### Initialize Data Loaders

Source: https://opacus.ai/tutorials/building_text_classifier

Configure PyTorch DataLoaders for training and testing datasets.

```python
from torch.utils.data import DataLoader, RandomSampler, SequentialSampler
from opacus.utils.uniform_sampler import UniformWithReplacementSampler

train_dataloader = DataLoader(train_dataset, batch_size=BATCH_SIZE)
test_dataloader = DataLoader(test_dataset, sampler=SequentialSampler(test_dataset), batch_size=BATCH_SIZE)
```

--------------------------------

### Split Dataset and Configure Loaders

Source: https://opacus.ai/tutorials/building_lstm_name_classifier

Splits the dataset into training and testing sets and initializes DataLoader instances.

```python
secure_mode = False
train_split = 0.8
test_every = 5
batch_size = 800

ds = NamesDataset(names_folder)
train_len = int(train_split * len(ds))
test_len = len(ds) - train_len

print(f"{train_len} samples for training, {test_len} for testing")

train_ds, test_ds = torch.utils.data.random_split(ds, [train_len, test_len])
```

```python
from torch.utils.data import DataLoader

train_loader = DataLoader(
    train_ds,
    batch_size=batch_size,
    pin_memory=True,
    collate_fn=padded_collate,
)

test_loader = DataLoader(
    test_ds,
    batch_size=2 * batch_size,
    shuffle=False,
    pin_memory=True,
    collate_fn=padded_collate,
)
```

--------------------------------

### PrivacyEngine Initialization

Source: https://opacus.ai/api/privacy_engine.html

Initializes the PrivacyEngine to manage privacy state and accounting.

```APIDOC
## PrivacyEngine Initialization

### Description
Creates a new PrivacyEngine instance to track privacy budget and configure the accounting mechanism.

### Parameters
- **accountant** (str) - Optional - Accounting mechanism (rdp, gdp, or prv).
- **secure_mode** (bool) - Optional - If True, uses cryptographically strong random number generation.
```

--------------------------------

### Simple Distributed Per-Layer Optimizer Initialization

Source: https://opacus.ai/api/_modules/opacus/optimizers/ddp_perlayeroptimizer.html

Initializes the optimizer for distributed training with per-layer clipping support.

```python
class SimpleDistributedPerLayerOptimizer(DPPerLayerOptimizer, DistributedDPOptimizer):
    def __init__(
        self,
        optimizer: Optimizer,
        *,
        noise_multiplier: float,
        max_grad_norm: float,
        expected_batch_size: Optional[int],
        loss_reduction: str = "mean",
        generator=None,
        secure_mode: bool = False,
        **kwargs,
    ):
        self.rank = torch.distributed.get_rank()
        self.world_size = torch.distributed.get_world_size()

        super().__init__(
            optimizer,
            noise_multiplier=noise_multiplier,
            max_grad_norm=max_grad_norm,
            expected_batch_size=expected_batch_size,
            loss_reduction=loss_reduction,
            generator=generator,
            secure_mode=secure_mode,
        )
```

--------------------------------

### Initialize PrivacyEngine

Source: https://opacus.ai/tutorials/building_lstm_name_classifier

Wraps the model and optimizer with the PrivacyEngine to enforce differential privacy during training.

```python
from opacus import PrivacyEngine
privacy_engine = PrivacyEngine(secure_mode=secure_mode)

model, optimizer, train_loader = privacy_engine.make_private_with_epsilon(
    module=model,
    optimizer=optimizer,
    data_loader=train_loader,
    max_grad_norm=max_per_sample_grad_norm,
    target_delta=delta,
    target_epsilon=epsilon,
    epochs=epochs,
)
```

--------------------------------

### Spawn Distributed Processes

Source: https://opacus.ai/tutorials/ddp_tutorial

Uses torch.multiprocessing to spawn the training processes across available devices.

```python
%%writefile -a opacus_sync_demo.py

import torch.multiprocessing as mp
import torch

world_size = torch.cuda.device_count()

if __name__ == '__main__':
    mp.spawn(
        launch,
        args=(world_size,),
        nprocs=world_size,
        join=True
    )
```

--------------------------------

### Launch Distributed Training Loop

Source: https://opacus.ai/tutorials/ddp_tutorial

Executes the training loop, demonstrating that gradients remain unsynchronized until the optimizer step is called.

```python
%%writefile -a opacus_sync_demo.py

import torch.nn as nn
import numpy as np

def launch(rank, world_size):
    setup(rank, world_size)
    criterion = nn.CrossEntropyLoss()

    model, optimizer, data_loader = init_training(rank, world_size)
    model.to(rank)
    model.train()

    for data, target in data_loader:
        data = data
        target = torch.tensor(target)

        data, target = data.to(rank), target.to(rank)
        optimizer.zero_grad()

        output = model(data)
        loss = criterion(output, target)
        loss.backward()

        flat_grad = torch.cat([p.grad_sample.sum(dim=0).view(-1) for p in model.parameters()]).cpu().numpy() / optimizer.expected_batch_size
        logger.info(
            f"(rank={rank}) Gradient norm before optimizer.step(): {np.linalg.norm(flat_grad):.4f}"
        )
        logger.info(
            f"(rank={rank}) Gradient sample before optimizer.step(): {flat_grad[:3]}"
        )

        optimizer.step()

        flat_grad = torch.cat([p.grad.view(-1) for p in model.parameters()]).cpu().numpy()
        logger.info(
            f"(rank={rank}) Gradient norm after optimizer.step(): {np.linalg.norm(flat_grad):.4f}"
        )
        logger.info(
            f"(rank={rank}) Gradient sample after optimizer.step(): {flat_grad[:3]}"
        )

        break

    cleanup()
```

--------------------------------

### Configure Device for Training

Source: https://opacus.ai/tutorials/building_image_classifier

Detects available CUDA devices and moves the model to the appropriate hardware.

```python
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = model.to(device)
```

--------------------------------

### DPOptimizer pre_step Method

Source: https://opacus.ai/api/_modules/opacus/optimizers/optimizer_fast_gradient_clipping.html

Performs actions before the optimizer step, such as accumulating gradients, adding noise, and scaling gradients. Handles skipping steps if necessary.

```python
def pre_step(
        self,
        closure: Optional[Callable[[], float]] = None
    ) -> Optional[float]:
        """
        Perform actions specific to ``DPOptimizer`` before calling
        underlying  ``optimizer.step()``

        Args:
            closure: A closure that reevaluates the model and
                returns the loss. Optional for most optimizers.
        """
        # The corner case when the optimizer has no trainable parameters.
        # Essentially the DPOptimizer act as a normal optimizer

        self.accumulate()
        if self._check_skip_next_step():
            self._is_last_step_skipped = True
            return False

        self.add_noise()
        self.scale_grad()

        if self.step_hook:
            self.step_hook(self)

        self._is_last_step_skipped = False
        return True
```

--------------------------------

### DPOptimizer pre_step Method

Source: https://opacus.ai/api/optim/dp_optimizer.html

Performs optimizer-specific actions before the underlying optimizer's step. Use this when custom pre-step logic is needed.

```python
pre_step(_closure =None_)
    
    Perform actions specific to `DPOptimizer` before calling underlying `optimizer.step()` 

Parameters:
    
**closure** (`Optional`[`Callable`[[], `float`]]) – A closure that reevaluates the model and returns the loss. Optional for most optimizers. 

Return type:
    
`Optional`[`float`]
```

--------------------------------

### DPOptimizer pre_step method

Source: https://opacus.ai/api/_modules/opacus/optimizers/optimizer_fast_gradient_clipping.html

Handles actions before calling the underlying optimizer.step() for DPOptimizer.

```APIDOC
## DPOptimizer.pre_step

### Description
Perform actions specific to ``DPOptimizer`` before calling underlying ``optimizer.step()``.

### Method
`pre_step`

### Parameters
#### Arguments
- **closure** (Callable[[], float], optional) - A closure that reevaluates the model and returns the loss. Optional for most optimizers.

### Returns
- Optional[float] - Returns `True` if the step should proceed, `False` if skipped.
```

--------------------------------

### Initialize Training Objects

Source: https://opacus.ai/tutorials/intro_to_advanced_features

Re-initialize the model, optimizer, and data loader before manual wrapping.

```python
model = SampleNet()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
data_loader = DataLoader(DATASET, batch_size=10)
```

--------------------------------

### DPOptimizer Initialization

Source: https://opacus.ai/api/optim/dp_optimizer.html

Initializes a new DPOptimizer instance to wrap an existing PyTorch optimizer.

```APIDOC
## DPOptimizer Initialization

### Description
Wraps a torch.optim.Optimizer to add per-sample gradient clipping and Gaussian noise.

### Parameters
- **optimizer** (Optimizer) - Required - The wrapped optimizer.
- **noise_multiplier** (float) - Required - Noise multiplier for Gaussian noise.
- **max_grad_norm** (float) - Required - Max gradient norm for clipping.
- **expected_batch_size** (int) - Optional - Batch size for averaging gradients.
- **loss_reduction** (str) - Optional - 'sum' or 'mean'.
- **generator** (torch.Generator) - Optional - Randomness source.
- **secure_mode** (bool) - Optional - Enable robust noise generation.
```

--------------------------------

### Initialize DistributedDPOptimizer

Source: https://opacus.ai/api/optim/dp_ddp_optimizer.html

Instantiate DistributedDPOptimizer with necessary parameters for distributed training. Ensure `expected_batch_size` is provided if `loss_reduction` is 'mean'.

```python
class opacus.optimizers.ddpoptimizer.DistributedDPOptimizer(_optimizer_ , _*_ , _noise_multiplier_ , _max_grad_norm_ , _expected_batch_size_ , _loss_reduction ='mean'_, _generator =None_, _secure_mode =False_, _** kwargs_)[source]¶
```

--------------------------------

### Manual Per-Sample Gradient Training Loop

Source: https://opacus.ai/tutorials/guide_to_grad_sampler

Demonstrates the inefficient approach of training with a batch size of 1 to access individual sample gradients.

```python
optimizer = torch.optim.SGD(lr=0.01)
for x, y i DataLoader(train_dataset, batch_size=128):
  # Run samples one-by-one to get per-sample gradients
  for x_i, y_i in zip(x, y):
    y_hat_i = model(x_i)
    loss = criterion(y_hat_i, y_i)
    loss.backward()
  
    # Clip each parameter's per-sample gradient
    for p in model.parameters():
      per_sample_grad = p.grad.detach().clone()
      torch.nn.utils.clip_grad_norm(per_sample_grad, max_norm=1.0)
      p.accumulated_grads.append(per_sample_grad)
    model.zero_grad(). # p.grad is accumulative, so we need to manually reset
  
  # Aggregate clipped gradients of all samples in a batch, and add DP noise
  for p in model.parameters():
    p.grad = accumulate_and_noise(p.accumulated_grads, dp_paramters)
  
  optimizer.step()
  optimizer.zero_grad()
```

--------------------------------

### Log Module Gradient Sample Mode

Source: https://opacus.ai/api/_modules/opacus/grad_sample/grad_sample_module_fast_gradient_clipping.html

Logs the gradient sample mode for modules, identifying Ghost Clipping or Fast Gradient Clipping usage.

```python
    def log_module_gradient_sample_mode(
        self, module: nn.Module, *, force_functorch=False, use_ghost_clipping=True
    ):
        """
        Add logs to track gradient sample mode for each part of the module, including 1) Ghost Clipping, 2) Fast Gradient Clipping (hook mode), and 3) Fast Gradient Clipping (functorch mode).

        Args:
            module: nn.Module to be checked
            force_functorch: If set to ``True``, will use functorch to compute
                all per sample gradients. Otherwise, functorch will be used only
                for layers without registered grad sampler methods.
            use_ghost_clipping: If set to ``True``, Ghost Clipping
                will be used for clipping gradients of supported layers. If ``False``, Fast
                Gradient Clipping will be used for all layers.
        """
        for m_name, m in trainable_modules(module):
            if type(m) in [DPRNN, DPLSTM, DPGRU]:
                logger.info(
                    f"Module name: {m_name}, module type: {type(m)}. No hook or functorch is added."
                )

            elif use_ghost_clipping and type(m) in self.NORM_SAMPLERS:
                logger.info(
```

--------------------------------

### Initialize DPOptimizer

Source: https://opacus.ai/tutorials/intro_to_advanced_features

Configure DPOptimizer with essential privacy parameters like noise_multiplier, max_grad_norm, and expected_batch_size. The expected_batch_size is derived from the dataset size and sample rate.

```python
from opacus.optimizers import DPOptimizer

sample_rate = 1 / len(data_loader)
expected_batch_size = int(len(data_loader.dataset) * sample_rate)

optimizer = DPOptimizer(
    optimizer=optimizer,
    noise_multiplier=1.0,
    max_grad_norm=1.0,
    expected_batch_size=expected_batch_size,
)
```
Tokens:
Raw