### Install Libraries for HKR Multiclass Example
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/notebooks/wasserstein_classification_fashionMNIST.ipynb
Installs the necessary Python libraries, deel-torchlip and foolbox, required for running the HKR multiclass and fooling example. Uncomment the line to execute the installation.
```python
# Install the required libraries deel-torchlip and foolbox (uncomment below if needed)
# %pip install -qqq deel-torchlip foolbox
```
--------------------------------
### Setup Development Environment with Make
Source: https://github.com/deel-ai/deel-torchlip/blob/master/CONTRIBUTING.md
These commands clone the repository, set up a virtual environment, install development dependencies, and install the library in editable mode.
```bash
git clone https://github.com/deel-ai/deel-torchlip.git
cd deel-torchlip
make prepare-dev && source torchlip_dev_env/bin/activate
pip install -e .
```
--------------------------------
### Install Libraries for Deel-Torchlip and Foolbox
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/source/wasserstein_classification_fashionMNIST.md
Installs the necessary Python libraries, deel-torchlip for Lipschitz networks and foolbox for adversarial attacks. This is a prerequisite for running the subsequent code examples.
```ipython3
# Install the required libraries deel-torchlip and foolbox (uncomment below if needed)
# %pip install -qqq deel-torchlip foolbox
```
--------------------------------
### Prepare Development Environment
Source: https://github.com/deel-ai/deel-torchlip/blob/master/README.md
This command installs all necessary dependencies for contributing to the deel-torchlip project. It ensures the development environment is correctly set up.
```shell
$ make prepare-dev
```
--------------------------------
### Install Deel-Torchlip
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/source/index.md
Installs the deel-torchlip library using pip. This command requires a Python package manager to be installed. Ensure you have a compatible PyTorch version installed separately.
```bash
pip install deel-torchlip
```
--------------------------------
### Install Deel-Torchlip Library
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/notebooks/wasserstein_toy.ipynb
This snippet shows how to install the deel-torchlip library using pip. It is a prerequisite for running the Wasserstein distance estimation examples.
```python
# Install the required library deel-torchlip (uncomment line below)
# %pip install -qqq deel-torchlip
```
--------------------------------
### Set Up DataLoaders for Training and Testing
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/notebooks/wasserstein_classification_MNIST08.ipynb
Creates PyTorch DataLoader instances for training and testing datasets. These are essential for iterating over data in batches during the training and evaluation phases.
```python
train_loader = torch.utils.data.DataLoader(train, batch_size=batch_size, shuffle=True)
test_loader = torch.utils.data.DataLoader(test, batch_size=32, shuffle=False)
```
--------------------------------
### Generate Adversarial Certificates and Run Attacks using Foolbox in Python
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/notebooks/wasserstein_classification_fashionMNIST.ipynb
This snippet demonstrates how to generate Lipschitz certificates for a given image sample and perform adversarial attacks using the foolbox library. It involves selecting data, computing certificates based on model outputs, and running the L2 Carlini & Wagner attack to find adversarial examples.
```python
import numpy as np
# Select only the first batch from the test set
sub_data, sub_targets = next(iter(test_loader))
sub_data, sub_targets = sub_data.to(device), sub_targets.to(device)
# Drop misclassified elements
output = vanilla_model(sub_data)
well_classified_mask = output.argmax(dim=-1) == sub_targets
sub_data = sub_data[well_classified_mask]
sub_targets = sub_targets[well_classified_mask]
# Retrieve one image per class
images_list, targets_list = [], []
for i in range(10):
# Select the elements of the i-th label and keep the first one
label_mask = sub_targets == i
x = sub_data[label_mask][0]
y = sub_targets[label_mask][0]
images_list.append(x)
targets_list.append(y)
images = torch.stack(images_list)
targets = torch.stack(targets_list)
```
```python
import foolbox as fb
# Compute certificates
values, _ = vanilla_model(images).topk(k=2)
#The factor is 2.0 when using disjoint_neurons==True
certificates = (values[:, 0] - values[:, 1]) / 2.
# Run Carlini & Wagner attack
fmodel = fb.PyTorchModel(vanilla_model, bounds=(0.0, 1.0), device=device)
attack = fb.attacks.L2CarliniWagnerAttack(binary_search_steps=6, steps=8000)
_, advs, success = attack(fmodel, images, targets, epsilons=None)
dist_to_adv = (images - advs).square().sum(dim=(1, 2, 3)).sqrt()
```
--------------------------------
### Visualize Adversarial Examples
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/notebooks/wasserstein_classification_fashionMNIST.ipynb
This function visualizes original images, their adversarial counterparts, difference maps, and associated certificates/distances. It requires a model, images, adversarial images, and class names. The output is a Matplotlib figure.
```python
import matplotlib.pyplot as plt
import numpy as np
def adversarial_viz(model, images, advs, class_names):
"""
This functions shows for each image sample:
- the original image
- the adversarial image
- the difference map
- the certificate and the observed distance to adversarial
"""
scale = 1.5
nb_imgs = images.shape[0]
# Compute certificates
values, _ = model(images).topk(k=2)
certificates = (values[:, 0] - values[:, 1]) / np.sqrt(2)
# Compute distance between image and its adversarial
dist_to_adv = (images - advs).square().sum(dim=(1, 2, 3)).sqrt()
# Find predicted classes for images and their adversarials
orig_classes = [class_names[i] for i in model(images).argmax(dim=-1)]
advs_classes = [class_names[i] for i in model(advs).argmax(dim=-1)]
# Compute difference maps
advs = advs.detach().cpu()
images = images.detach().cpu()
diff_pos = np.clip(advs - images, 0, 1.0)
diff_neg = np.clip(images - advs, 0, 1.0)
diff_map = np.concatenate(
[diff_neg, diff_pos, np.zeros_like(diff_neg)], axis=1
).transpose((0, 2, 3, 1))
# Create plot
def _set_ax(ax, title):
ax.set_title(title)
ax.set_xticks([])
ax.set_yticks([])
ax.axis("off")
figsize = (3 * scale, nb_imgs * scale)
_, axes = plt.subplots(
ncols=3, nrows=nb_imgs, figsize=figsize, squeeze=False, constrained_layout=True
)
for i in range(nb_imgs):
_set_ax(axes[i][0], orig_classes[i])
axes[i][0].imshow(images[i].squeeze(), cmap="gray")
_set_ax(axes[i][1], advs_classes[i])
axes[i][1].imshow(advs[i].squeeze(), cmap="gray")
_set_ax(axes[i][2], f"certif: {certificates[i]:.2f}, obs: {dist_to_adv[i]:.2f}")
axes[i][2].imshow(diff_map[i] / diff_map[i].max())
adversarial_viz(vanilla_model, images, advs, test_set.classes)
```
--------------------------------
### Sequential
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/source/deel.torchlip.md
An equivalent of torch.Sequential that allows setting a global k-lip factor, supporting condensation and vanilla exportation. Currently implements constant repartition where each layer gets n_sqrt(k_lip_factor).
```APIDOC
## class deel.torchlip.Sequential
### Description
Equivalent of torch.Sequential but allows setting k-lip factor globally. Also supports condensation and vanilla exportation. Currently, constant repartition is implemented (each layer gets n_sqrt(k_lip_factor), where n is the number of layers). In the future, other repartition functions may be implemented.
* **Parameters:**
* **layers** – list of layers to add to the model.
* **name** – name of the model, can be None
* **k_coef_lip** – the Lipschitz coefficient to ensure globally on the model.
```
--------------------------------
### Define Loss Parameters and Initialize Model Components
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/notebooks/wasserstein_classification_MNIST08.ipynb
Initializes loss parameters like min_margin and alpha, and sets up various loss functions (KRLoss, HKRLoss, HingeMarginLoss) and an optimizer (Adam). This is a prerequisite for the training process.
```python
min_margin = 1
alpha = 0.98
kr_loss = KRLoss()
hkr_loss = HKRLoss(alpha=alpha, min_margin=min_margin)
hinge_margin_loss =HingeMarginLoss(min_margin=min_margin)
optimizer = torch.optim.Adam(lr=0.001, params=wass.parameters())
```
--------------------------------
### Define Lipschitz Network Architecture
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/notebooks/wasserstein_classification_fashionMNIST.ipynb
Illustrates the setup for a Lipschitz network architecture using deel.torchlip. It highlights the use of FrobeniusLinear with disjoint_neurons=True for enforcing 1-Lipschitz constraints on the output heads in a multiclass setting.
```python
from deel import torchlip
# Sequential has the same properties as any Lipschitz layer. It only acts as a
# container, with features specific to Lipschitz functions (condensation,
# ...)
```
--------------------------------
### Initialize and Train with HKR Loss in PyTorch
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/source/wasserstein_classification_fashionMNIST.md
Demonstrates how to initialize different HKR loss variants (HKRMulticlassLoss, SoftHKRMulticlassLoss, LseHKRMulticlassLoss) and integrate them into a PyTorch training loop. It includes optimizer setup, forward/backward passes, metric calculation, and validation. Requires the 'torchlip' library and a PyTorch model, data loaders, and device to be defined.
```ipython3
loss_choice = "LseHKRMulticlassLoss" # "HKRMulticlassLoss" or "SoftHKRMulticlassLoss"or "LseHKRMulticlassLoss"
epochs = 10
optimizer = torch.optim.Adam(lr=1e-3, params=model.parameters())
hkr_loss = None
if loss_choice == "HKRMulticlassLoss":
hkr_loss = torchlip.HKRMulticlassLoss(alpha=0.99, min_margin=0.25) #Robust
#hkr_loss = torchlip.HKRMulticlassLoss(alpha=0.999, min_margin=0.10) #Accurate
if loss_choice == "SoftHKRMulticlassLoss":
hkr_loss = torchlip.SoftHKRMulticlassLoss(alpha=0.995, min_margin=0.10, temperature=50.0)
if loss_choice == "LseHKRMulticlassLoss":
hkr_loss = torchlip.LseHKRMulticlassLoss(alpha=0.9, min_margin=1.0, temperature=10.0)
assert hkr_loss is not None, "Please choose a valid loss function"
kr_multiclass_loss = torchlip.KRMulticlassLoss()
for epoch in range(epochs):
m_kr, m_acc = 0, 0
for step, (data, target) in enumerate(train_loader):
# For multiclass HKR loss, the targets must be one-hot encoded
target = torch.nn.functional.one_hot(target, num_classes=10)
data, target = data.to(device), target.to(device)
# Forward + backward pass
optimizer.zero_grad()
output = model(data)
loss = hkr_loss(output, target)
loss.backward()
optimizer.step()
# Compute metrics on batch
m_kr += kr_multiclass_loss(output, target)
m_acc += (output.argmax(dim=1) == target.argmax(dim=1)).sum() / len(target)
# Train metrics for the current epoch
metrics = [
f"{k}: {v:.04f}"
for k, v in {
"loss": loss,
"acc": m_acc / (step + 1),
"KR": m_kr / (step + 1),
}.items()
]
# Compute validation loss for the current epoch
test_output, test_targets = [], []
for data, target in test_loader:
data, target = data.to(device), target.to(device)
test_output.append(model(data).detach().cpu())
test_targets.append(
torch.nn.functional.one_hot(target, num_classes=10).detach().cpu()
)
test_output = torch.cat(test_output)
test_targets = torch.cat(test_targets)
val_loss = hkr_loss(test_output, test_targets)
val_kr = kr_multiclass_loss(test_output, test_targets)
val_acc = (test_output.argmax(dim=1) == test_targets.argmax(dim=1)).float().mean()
# Validation metrics for the current epoch
metrics += [
f"val_{k}: {v:.04f}"
for k, v in {
"loss": hkr_loss(test_output, test_targets),
"acc": (test_output.argmax(dim=1) == test_targets.argmax(dim=1))
.float()
.mean(),
"KR": kr_multiclass_loss(test_output, test_targets),
}.items()
]
print(f"Epoch {epoch + 1}/{epochs}")
print(" - ".join(metrics))
```
--------------------------------
### Building a 1-Lipschitz Network with deel-torchlip
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/source/basic_example.md
This code demonstrates how to construct a 1-Lipschitz neural network using spectral convolutional and linear layers from the deel-torchlip library. It shows integration within a `torch.nn.Sequential` model and its setup for training with a custom loss function (HKRLoss) and an Adam optimizer.
```python
import torch
from deel import torchlip
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# deel-torchlip layers can be used like any torch.nn layers in
# Sequential or other types of container modules.
model = torch.nn.Sequential(
torchlip.SpectralConv2d(1, 32, (3, 3), padding=1),
torchlip.SpectralConv2d(32, 32, (3, 3), padding=1),
torch.nn.MaxPool2d(kernel_size=(2, 2)),
torchlip.SpectralConv2d(32, 32, (3, 3), padding=1),
torchlip.SpectralConv2d(32, 32, (3, 3), padding=1),
torch.nn.MaxPool2d(kernel_size=(2, 2)),
torch.nn.Flatten(),
torchlip.SpectralLinear(1568, 256),
torchlip.SpectralLinear(256, 1)
).to(device)
# Training can be done as usual, except that we are doing
# binary classification with -1 and +1 labels to the target
# must be fixed from the dataset.
optimizer = torch.optim.Adam(lr=0.01, params=model.parameters())
hkr_loss = HKRLoss(alpha=10, min_margin=1)
for data, target in mnist_08:
data, target = data.to(device), target.to(device)
optimizer.zero_grad()
output = model(data)
loss = hkr_loss(output, target)
loss.backward()
optimizer.step()
```
--------------------------------
### Build Lipschitz Model with SpectralLinear and FullSort
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/notebooks/wasserstein_classification_MNIST08.ipynb
Constructs a sequential neural network model using `deel.torchlip` components for Lipschitz continuity. It includes `SpectralLinear` layers with `FullSort` for weight normalization and a final `FrobeniusLinear` layer. The model is moved to the appropriate device (GPU or CPU).
```python
import torch
from deel import torchlip
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
ninputs = 28 * 28
wass = torchlip.Sequential(
torch.nn.Flatten(),
torchlip.SpectralLinear(ninputs, 128),
torchlip.FullSort(),
torchlip.SpectralLinear(128, 64),
torchlip.FullSort(),
torchlip.SpectralLinear(64, 32),
torchlip.FullSort(),
torchlip.FrobeniusLinear(32, 1),
).to(device)
wass
```
--------------------------------
### Evaluate Lipschitz Constant using torchlip
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/notebooks/wasserstein_classification_MNIST08.ipynb
Demonstrates how to use the `evaluate_lip_const` function from `deel.torchlip.utils` to compute the empirical Lipschitz constant of a model. It shows different evaluation methods and their potential impact on computation time.
```python
from deel.torchlip.utils import evaluate_lip_const
x,y = next(iter(test_loader))
evaluate_lip_const(wass, x.to(device), evaluation_type="all", disjoint_neurons=False, double_attack=True)
```
--------------------------------
### Export and Analyze Model with Vanilla Export (Python)
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/notebooks/wasserstein_classification_MNIST08.ipynb
Demonstrates how to export a model using `vanilla_export` and then analyze the singular values of its linear layers' weights. This process involves creating a new model instance, loading state dict, performing a forward pass for initialization, and then exporting. The analysis helps verify the model's properties post-export.
```python
wexport = wass.vanilla_export()
print("=== After export ===")
layers = list(wexport.children())
for layer in layers:
if hasattr(layer, "weight"):
w = layer.weight
u, s, v = torch.svd(w)
print(f"{type(layer)}, min={s.min()}, max={s.max()}")
```
--------------------------------
### Implement Training Loop with Metric Computation
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/notebooks/wasserstein_classification_MNIST08.ipynb
Defines a standard PyTorch training loop that iterates over epochs and batches. It includes forward and backward passes, optimizer steps, and computation of training metrics (loss, KR, accuracy).
```python
for epoch in range(epochs):
m_kr, m_hm, m_acc = 0, 0, 0
wass.train()
for step, (data, target) in enumerate(train_loader):
data, target = data.to(device), target.to(device)
optimizer.zero_grad()
output = wass(data)
loss = hkr_loss(output, target)
loss.backward()
optimizer.step()
# Compute metrics on batch
m_kr += kr_loss(output, target)
m_hm += hinge_margin_loss(output, target)
m_acc += (torch.sign(output).flatten() == torch.sign(target)).sum() / len(target)
# Train metrics for the current epoch
metrics = [
f"{k}: {v:.04f}"
for k, v in {
"loss": loss,
"KR": m_kr / (step + 1),
"acc": m_acc / (step + 1),
}.items()
]
# Compute test loss for the current epoch
wass.eval()
testo = []
for data, target in test_loader:
data, target = data.to(device), target.to(device)
testo.append(wass(data).detach().cpu())
testo = torch.cat(testo).flatten()
# Validation metrics for the current epoch
metrics += [
f"val_{k}: {v:.04f}"
for k, v in {
"loss": hkr_loss(
testo, test.tensors[1]
),
"KR": kr_loss(testo.flatten(), test.tensors[1]),
"acc": (torch.sign(testo).flatten() == torch.sign(test.tensors[1]))
.float()
.mean(),
}.items()
]
print(f"Epoch {epoch + 1}/{epochs}")
print(" - ".join(metrics))
```
--------------------------------
### Prepare MNIST Data for Binary Classification
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/notebooks/wasserstein_classification_MNIST08.ipynb
Prepares the MNIST dataset for binary classification by selecting two specified classes, converting labels to {-1, 1}, and normalizing pixel values to the range [-1, 1]. It returns a TensorDataset suitable for training.
```python
import torch
from torchvision import datasets
# First we select the two classes
selected_classes = [0, 8] # must be two classes as we perform binary classification
def prepare_data(dataset, class_a=0, class_b=8):
"""
This function converts the MNIST data to make it suitable for our binary
classification setup.
"""
x = dataset.data
y = dataset.targets
# select items from the two selected classes
mask = (y == class_a) + (
y == class_b
) # mask to select only items from class_a or class_b
x = x[mask]
y = y[mask]
# convert from range int[0,255] to float32[-1,1]
x = x.float() / 255
x = x.reshape((-1, 28, 28, 1))
# change label to binary classification {-1,1}
y_ = torch.zeros_like(y).float()
y_[y == class_a] = 1.0
y_[y == class_b] = -1.0
return torch.utils.data.TensorDataset(x, y_)
train = datasets.MNIST("./data", train=True, download=True)
test = datasets.MNIST("./data", train=False, download=True)
# Prepare the data
train = prepare_data(train, selected_classes[0], selected_classes[1])
test = prepare_data(test, selected_classes[0], selected_classes[1])
# Display infos about dataset
print(
f"Train set size: {len(train)} samples, classes proportions: "
f"{100 * (train.tensors[1] == 1).numpy().mean():.2f} %"
)
print(
f"Test set size: {len(test)} samples, classes proportions: "
f"{100 * (test.tensors[1] == 1).numpy().mean():.2f} %"
)
```
--------------------------------
### Export Lipschitz-constrained Models to Vanilla PyTorch (PyTorch)
Source: https://context7.com/deel-ai/deel-torchlip/llms.txt
Demonstrates how to convert Lipschitz-constrained models built with `torchlip` into standard PyTorch models suitable for deployment. The `vanilla_export()` method removes Lipschitz constraints while preserving the learned weights, resulting in faster inference. The example includes training a `torchlip.Sequential` model and then exporting it, along with an individual layer.
```python
from deel import torchlip
import torch
# Build Lipschitz model
lip_model = torchlip.Sequential(
torchlip.SpectralConv2d(3, 32, 3, padding=1),
torchlip.GroupSort2(),
torchlip.SpectralLinear(32 * 32 * 32, 10),
k_coef_lip=1.0
)
# Train the model
optimizer = torch.optim.Adam(lip_model.parameters(), lr=0.001)
loss_fn = torchlip.HKRMulticlassLoss(alpha=0.9)
# Assuming dataloader is defined
# for x, y in dataloader:
# optimizer.zero_grad()
# output = lip_model(x)
# loss = loss_fn(output, y)
# loss.backward()
# optimizer.step()
# Export to vanilla PyTorch (removes hooks, keeps weights)
vanilla_model = lip_model.vanilla_export()
# Vanilla model has no Lipschitz constraints (faster inference)
x = torch.randn(1, 3, 32, 32)
with torch.no_grad():
lip_output = lip_model(x)
vanilla_output = vanilla_model(x)
# Outputs are identical
print(torch.allclose(lip_output, vanilla_output, atol=1e-6)) # True
# Save vanilla model for deployment
torch.save(vanilla_model.state_dict(), "model_deployment.pth")
# Individual layer export
layer = torchlip.SpectralLinear(128, 64)
vanilla_layer = layer.vanilla_export() # Returns torch.nn.Linear
```
--------------------------------
### Build Sequential Model with Torchlip Layers (Python)
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/notebooks/wasserstein_classification_fashionMNIST.ipynb
Constructs a sequential neural network model using various Lipschitz-constrained layers from the torchlip library. This includes convolutional, pooling, and linear layers, along with GroupSort2 for regularization. The model is then moved to a CUDA-enabled device if available, otherwise to the CPU.
```python
import torch
import torchlip
# Define the sequential model with torchlip layers
model = torchlip.Sequential(
# Lipschitz layers preserve the API of their superclass (here Conv2d). An optional
# argument is available, k_coef_lip, which controls the Lipschitz constant of the
# layer
torchlip.SpectralConv2d(
in_channels=1, out_channels=16, kernel_size=(3, 3), padding="same"
),
torchlip.GroupSort2(),
# Usual pooling layer are implemented (avg, max), but new pooling layers are also
# available
torchlip.ScaledL2NormPool2d(kernel_size=(2, 2)),
torchlip.SpectralConv2d(
in_channels=16, out_channels=32, kernel_size=(3, 3), padding="same"
),
torchlip.GroupSort2(),
torchlip.ScaledL2NormPool2d(kernel_size=(2, 2)),
# Our layers are fully interoperable with existing PyTorch layers
torch.nn.Flatten(),
torchlip.SpectralLinear(1568, 64),
torchlip.GroupSort2(),
torchlip.FrobeniusLinear(64, 10, bias=True, disjoint_neurons=True),
# Similarly, model has a parameter to set the Lipschitz constant that automatically
# sets the constant of each layer.
k_coef_lip=1.0,
)
# Move the model to the appropriate device (GPU or CPU)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
```
--------------------------------
### Define ResNet-style Architecture with Lipschitz Residuals (PyTorch)
Source: https://context7.com/deel-ai/deel-torchlip/llms.txt
Demonstrates how to define a ResNet-style neural network block using Lipschitz-constrained layers from the `torchlip` library. The `LipResBlock` incorporates `SpectralConv2d`, `GroupSort2`, and `LipResidual` for Lipschitz continuity. The example also shows how to stack these blocks within a `torch.nn.Sequential` model.
```python
import torchlip
import torch
class LipResBlock(torch.nn.Module):
def __init__(self, channels):
super().__init__()
self.conv1 = torchlip.SpectralConv2d(channels, channels, 3, padding=1)
self.act1 = torchlip.GroupSort2()
self.conv2 = torchlip.SpectralConv2d(channels, channels, 3, padding=1)
self.residual = torchlip.LipResidual()
def forward(self, x):
y = self.conv1(x)
y = self.act1(y)
y = self.conv2(y)
return self.residual(x, y) # Learnable α*x + (1-α)*y
# Stack multiple residual blocks
model = torch.nn.Sequential(
torchlip.SpectralConv2d(3, 64, 3, padding=1),
LipResBlock(64),
LipResBlock(64),
LipResBlock(64),
torch.nn.Flatten(),
torchlip.SpectralLinear(64 * 32 * 32, 10)
)
```
--------------------------------
### L2 Attack and Certificate Calculation using PyTorch and Foolbox
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/source/wasserstein_classification_fashionMNIST.md
This snippet demonstrates how to compute robustness certificates and perform L2 attacks on a PyTorch model using the foolbox library. It calculates certificates based on the top-2 model outputs and then executes the Carlini & Wagner attack to find adversarial examples, comparing the distance to the original images.
```ipython3
import foolbox as fb
# Compute certificates
values, _ = vanilla_model(images).topk(k=2)
#The factor is 2.0 when using disjoint_neurons==True
certificates = (values[:, 0] - values[:, 1]) / 2.
# Run Carlini & Wagner attack
fmodel = fb.PyTorchModel(vanilla_model, bounds=(0.0, 1.0), device=device)
attack = fb.attacks.L2CarliniWagnerAttack(binary_search_steps=6, steps=8000)
_, advs, success = attack(fmodel, images, targets, epsilons=None)
dist_to_adv = (images - advs).square().sum(dim=(1, 2, 3)).sqrt()
```
--------------------------------
### Spectral and Björck Normalization for Custom Layers (PyTorch)
Source: https://context7.com/deel-ai/deel-torchlip/llms.txt
Illustrates the use of low-level normalization functions from `deel.torchlip.normalizers` for custom implementations. It shows how to apply spectral normalization to reduce the largest singular value and Björck normalization to enforce orthogonality. The example also demonstrates applying these normalizations as parametrization hooks to standard PyTorch layers.
```python
from deel.torchlip import normalizers
import torch
# Spectral normalization: reduces largest singular value to 1
weight = torch.randn(64, 128) # Output x Input
u_init = torch.randn(64) # Initial singular vector
normalized_weight, u_vec, sigma = normalizers.spectral_normalization(
kernel=weight,
u=u_init,
eps=1e-3,
maxiter=10
)
print(f"Largest singular value: {sigma.item():.6f}") # ~1.0
# Björck normalization: pushes all singular values toward 1
# (requires spectral normalization first)
orthogonal_weight = normalizers.bjorck_normalization(
w=normalized_weight,
eps=1e-3,
beta=0.5,
maxiter=15
)
# Verify orthogonality: W @ W^T ≈ I (for square matrices)
if weight.size(0) == weight.size(1):
product = orthogonal_weight @ orthogonal_weight.T
identity = torch.eye(64)
error = (product - identity).abs().max()
print(f"Orthogonality error: {error.item():.6f}")
# Manual layer construction (equivalent to SpectralLinear)
layer = torch.nn.Linear(128, 64)
torch.nn.init.orthogonal_(layer.weight)
layer.bias.data.fill_(0.0)
# Apply as parametrization hooks
torch.nn.utils.parametrizations.spectral_norm(
layer,
name="weight",
eps=1e-3
)
from deel.torchlip.utils import bjorck_norm
bjorck_norm(layer, name="weight", eps=1e-3)
# Now layer.weight is automatically normalized on each forward pass
```
--------------------------------
### Training with Margin Maximization using HKRLoss
Source: https://context7.com/deel-ai/deel-torchlip/llms.txt
This snippet demonstrates training a model with the HKRMulticlassLoss from deel-torchlip, which emphasizes margin maximization for certified robustness. It includes standard PyTorch training loop elements like optimizer setup, data loading, one-hot encoding, forward and backward passes, and optimizer steps. It also evaluates the average certified radius on a test set after each epoch.
```python
import torch
import torchlip
# Assume model, dataloader, test_loader are defined elsewhere
# model = ...
# dataloader = ...
# test_loader = ...
loss_fn = torchlip.HKRMulticlassLoss(
alpha=0.9, # High alpha emphasizes margin
min_margin=2.0 # Target margin of 2.0 -> certified radius of 1.0
)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
for epoch in range(10):
for x_batch, y_batch in dataloader:
y_onehot = torch.nn.functional.one_hot(y_batch, 10).float()
optimizer.zero_grad()
output = model(x_batch)
loss = loss_fn(output, y_onehot)
loss.backward()
optimizer.step()
# Evaluate average certified radius on test set
with torch.no_grad():
all_radii = []
for x_test, y_test in test_loader:
logits = model(x_test)
top2 = logits.topk(2, dim=1).values
radii = (top2[:, 0] - top2[:, 1]) / 2.0
all_radii.append(radii)
avg_radius = torch.cat(all_radii).mean()
print(f"Epoch {epoch}: Avg certified radius = {avg_radius.item():.4f}")
```
--------------------------------
### Build a Lipschitz Network in Python
Source: https://github.com/deel-ai/deel-torchlip/blob/master/README.md
Demonstrates building a Lipschitz network using deel-torchlip's Sequential and layer classes. This model can be integrated into a standard PyTorch training loop. It utilizes SpectralConv2d and SpectralLinear layers.
```python
import torch
from deel import torchlip
# Build a Lipschitz network with 4 layers, that can be used in a training loop,
# like any torch.nn.Sequential network
model = torchlip.Sequential(
torchlip.SpectralConv2d(
in_channels=3, out_channels=16, kernel_size=(3, 3), padding="same"
),
torchlip.GroupSort2(),
torch.nn.Flatten(),
torchlip.SpectralLinear(15544, 64)
)
```
--------------------------------
### Check All Code Quality and Hooks
Source: https://github.com/deel-ai/deel-torchlip/blob/master/README.md
This command runs all pre-commit hooks and checks to ensure code changes are compliant and will be accepted. It's recommended to run this before submitting contributions.
```shell
$ make check_all
```
--------------------------------
### GroupSort2 and FullSort: Lipschitz Activations
Source: https://context7.com/deel-ai/deel-torchlip/llms.txt
Demonstrates the use of `torchlip.GroupSort2` and `torchlip.FullSort` which are non-linear activation functions that preserve the Lipschitz constant. `GroupSort2` sorts adjacent pairs of elements, while `FullSort` sorts all elements within a tensor. The examples show how to apply these activations and their effect on input tensors, including an example of `GroupSort` with a custom group size.
```python
from deel import torchlip
# GroupSort2: sorts adjacent pairs of elements
activation = torchlip.GroupSort2(k_coef_lip=1.0)
x = torch.tensor([[0.5, -0.3, 0.8, -0.1]])
y = activation(x)
print(y) # tensor([[-0.3, 0.5, -0.1, 0.8]]) # pairs sorted
# GroupSort with custom group size
activation_4 = torchlip.GroupSort(group_size=4, k_coef_lip=1.0)
x = torch.randn(2, 8)
y = activation_4(x) # Sorts groups of 4 elements
# FullSort: sorts all elements
full_sort = torchlip.FullSort(k_coef_lip=1.0)
x = torch.tensor([[3.0, 1.0, 2.0, 4.0]])
y = full_sort(x)
print(y) # tensor([[1.0, 2.0, 3.0, 4.0]])
```
--------------------------------
### Comparison of PyTorch nn Modules and deel-torchlip Equivalents for Lipschitz Networks
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/source/basic_example.md
This table outlines PyTorch's standard neural network modules and their 1-Lipschitz equivalents provided by deel-torchlip. It specifies when a module is not inherently 1-Lipschitz and lists the compatible deel-torchlip alternatives, along with relevant comments regarding their functionality.
```markdown
| `torch.nn` | 1-Lipschitz? | `deel-torchlip` equivalent | comments |
| [`torch.nn.Linear`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Linear.html#torch.nn.Linear) | no | [`SpectralLinear`](deel.torchlip.md#deel.torchlip.SpectralLinear)
[`FrobeniusLinear`](deel.torchlip.md#deel.torchlip.FrobeniusLinear) | [`SpectralLinear`](deel.torchlip.md#deel.torchlip.SpectralLinear) and [`FrobeniusLinear`](deel.torchlip.md#deel.torchlip.FrobeniusLinear) are similar when there is a single output. |
| [`torch.nn.Conv2d`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Conv2d.html#torch.nn.Conv2d) | no | [`SpectralConv2d`](deel.torchlip.md#deel.torchlip.SpectralConv2d)
[`FrobeniusConv2d`](deel.torchlip.md#deel.torchlip.FrobeniusConv2d) | [`SpectralConv2d`](deel.torchlip.md#deel.torchlip.SpectralConv2d) also implements Björck normalization. |
| [`torch.nn.Conv1d`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Conv1d.html#torch.nn.Conv1d) | no | [`SpectralConv1d`](deel.torchlip.md#deel.torchlip.SpectralConv1d) | [`SpectralConv1d`](deel.torchlip.md#deel.torchlip.SpectralConv1d) also implements Björck normalization. |
```
--------------------------------
### Compute Certified Adversarial Robustness Radius (PyTorch)
Source: https://context7.com/deel-ai/deel-torchlip/llms.txt
Shows how to compute the certified adversarial robustness radius for a given model using its Lipschitz constant. A 1-Lipschitz classifier is constructed using `torchlip.Sequential` with `k_coef_lip=1.0`. The example calculates the margin between the top two predicted logits and then uses this margin to determine the certified radius, guaranteeing no adversarial example exists within that L2-ball.
```python
from deel import torchlip
import torch
# Build 1-Lipschitz classifier
model = torchlip.Sequential(
torchlip.SpectralConv2d(1, 32, 3, padding=1),
torchlip.GroupSort2(),
torch.nn.Flatten(),
torchlip.SpectralLinear(32 * 28 * 28, 10),
k_coef_lip=1.0 # Global K=1 guarantees 1-Lipschitz
)
# Get predictions
x = torch.randn(16, 1, 28, 28)
logits = model(x)
pred_class = logits.argmax(dim=1)
# Compute margin: difference between top logit and second-best
top2_logits, top2_indices = logits.topk(2, dim=1)
margin = top2_logits[:, 0] - top2_logits[:, 1]
# Certified radius = margin / (2 * K)
# For K=1, certified_radius = margin / 2
certified_radius = margin / 2.0
print(f"Sample certified radii:")
for i in range(5):
print(f" Sample {i}: radius = {certified_radius[i].item():.4f}")
print(f" Predicted class {pred_class[i].item()}")
print(f" Guarantee: No adversarial example within L2-ball of radius {certified_radius[i].item():.4f}")
```
--------------------------------
### deel.torchlip.functional.hinge_multiclass_loss
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/source/deel.torchlip.functional.md
Calculates the hinge loss in a multiclass setup, computing the elementwise hinge term. It requires input, target (one-hot encoded), and an optional min_margin.
```APIDOC
## deel.torchlip.functional.hinge_multiclass_loss
### Description
Loss to estimate the Hinge loss in a multiclass setup. It compute the elementwise hinge term. Note that this formulation differs from the one commonly found in tensorflow/pytorch (with maximise the difference between the two largest logits). This formulation is consistent with the binary classification loss used in a multiclass fashion.
### Parameters
- **input** (Tensor) - Tensor of arbitrary shape.
- **target** (Tensor) - Tensor of the same shape as input containing one hot encoding target labels (0 and +1).
- **min_margin** (float) - Optional. The minimal margin to enforce. Defaults to 1.
### Note
target should be one hot encoded. labels in (1,0)
### Returns
- **Tensor** - The hinge margin multiclass loss.
```
--------------------------------
### Initialize PyTorch and deel-torchlip
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/source/wasserstein_toy_classification.md
Initializes PyTorch and imports the necessary torchlip module from the deel library. It also sets the device to CUDA if available, otherwise CPU.
```ipython3
import torch
from deel import torchlip
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
```
--------------------------------
### Define HKR Classifier Training Parameters
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/notebooks/wasserstein_classification_MNIST08.ipynb
Sets the training parameters for the HKR classifier. This includes the number of epochs for training and the batch size to be used during the training process.
```python
from deel.torchlip import KRLoss, HKRLoss, HingeMarginLoss
# training parameters
epochs = 10
batch_size = 128
```
--------------------------------
### Export Model for Inference
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/notebooks/wasserstein_classification_MNIST08.ipynb
Illustrates the process of exporting a trained model for inference using the `vanilla_export()` method. This method converts torchlip-specific layers into standard PyTorch layers, optimizing the model for deployment.
```python
wass.vanilla_export()
```
--------------------------------
### Initialize and Normalize Linear Layer with Spectral and Björck Norm
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/source/basic_example.md
This snippet demonstrates how to initialize a PyTorch Linear layer with orthogonal weights and zero biases, then apply spectral normalization and Björck normalization to its weight. It's equivalent to using SpectralLinear(16, 32).
```python
import torch
import torchlip
m = torch.nn.Linear(16, 32)
torch.nn.init.orthogonal_(m.weight)
m.bias.data.fill_(0.0)
torch.nn.utils.spectral_norm(m, "weight", eps=1e-3)
torclip.utils.bjorck_norm(m, "weight", eps=1e-3)
```
--------------------------------
### Lipschitz PReLU Activation with PyTorch
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/source/deel.torchlip.md
Applies element-wise PReLU activation with a Lipschitz constraint. It learns a parameter 'a' that is clipped by k_coef_lip to ensure the Lipschitz continuity.
```python
import torch
import deel.torchlip as torchlip
x = torch.randn(5)
# Example with 1 learnable parameter
lprelu_layer = torchlip.LPReLU(num_parameters=1, init=0.25, k_coef_lip=1.0)
output = lprelu_layer(x)
# Example with channel-wise parameters (assuming input has channels)
# x_channels = torch.randn(1, 3, 5, 5) # (N, C, H, W)
# lprelu_channels = torchlip.LPReLU(num_parameters=3, init=0.25, k_coef_lip=1.0)
# output_channels = lprelu_channels(x_channels)
```
--------------------------------
### Model Export Procedure (Conceptual)
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/source/wasserstein_classification_MNIST08.md
This illustrates the steps required to export a Deel-Torchlip model for inference while preserving the original model. It involves creating a new model instance, copying state dictionaries, performing a forward pass for initialization, and then calling vanilla_export().
```python
# Build e new mode for instance with torchlip.Sequential(
# torchlip.SpectralConv2d(…), …)
wexport = ()
# Copy the parameters from the reference t the new model
wexport.load_state_dict(wass.state_dict())
# one forward required to initialize pamatrizations
vanilla_model(one_input)
# vanilla_export the new model
wexport = wexport.vanilla_export()
```
--------------------------------
### LPReLU Activation
Source: https://github.com/deel-ai/deel-torchlip/blob/master/docs/source/deel.torchlip.md
Applies element-wise PReLU activation with a Lipschitz constraint, ensuring the learnable parameter 'a' is bounded.
```APIDOC
## LPReLU Activation
### Description
Applies element-wise PReLU activation with Lipschitz constraint.
LPReLU(x) = max(0, x) + a' * min(0, x)
where a' = max(min(a, k), -k), and a is a learnable parameter.
See also [`torch.nn.PReLU`](https://docs.pytorch.org/docs/stable/generated/torch.nn.PReLU.html#torch.nn.PReLU) and [`functional.lipschitz_prelu()`](deel.torchlip.functional.md#deel.torchlip.functional.lipschitz_prelu).
### Parameters
* **num_parameters** (int) - Number of 'a' to learn. Can be 1 or the number of channels at input.
* **init** (float) - The initial value of 'a'.
* **k_coef_lip** (float) - The lipschitz coefficient to enforce.
### Input/Output Shape
* Input: (N, *) where * means any number of additional dimensions
* Output: (N, *), same shape as the input.
### Example
```python
import torch
import deel.torchlip as torchlip
x = torch.randn(5)
prelu = torchlip.LPReLU(num_parameters=1, init=0.25, k_coef_lip=1.0)
output = prelu(x)
```
### Request Example
```json
{
"input_tensor": "torch.randn(5)",
"num_parameters": 1,
"init": 0.25,
"k_coef_lip": 1.0
}
```
### Response
#### Success Response (200)
* **output_tensor** (torch.Tensor) - The tensor after applying LPReLU activation.
```
--------------------------------
### Check Code Formatting with Make
Source: https://github.com/deel-ai/deel-torchlip/blob/master/CONTRIBUTING.md
Use `make check_all` to ensure all files adhere to the project's coding style and formatting conventions before submitting changes.
```bash
make check_all
```