### Training Step Example

Source: https://context7.com/lucidrains/zorro-pytorch/llms.txt

Shows a simple training step using CrossEntropyLoss with the model's output and generated labels.

```python
# Training step example
labels = torch.randint(0, 10, (batch_size,))
loss = nn.CrossEntropyLoss()(logits, labels)
print(f"Loss: {loss.item():.4f}")
```

--------------------------------

### Install zorro-pytorch

Source: https://github.com/lucidrains/zorro-pytorch/blob/main/README.md

Install the zorro-pytorch library using pip.

```bash
pip install zorro-pytorch
```

--------------------------------

### Multimodal Classification Example

Source: https://context7.com/lucidrains/zorro-pytorch/llms.txt

A complete example demonstrating how to use Zorro for multimodal video-audio classification by adding a classification head on top of the fusion tokens. This setup returns only the fusion token for classification.

```python
import torch
import torch.nn as nn
from zorro_pytorch import Zorro, TokenTypes as T

class MultimodalClassifier(nn.Module):
    def __init__(self, num_classes, dim=512, depth=6):
        super().__init__()
        self.zorro = Zorro(
            dim=dim,
            depth=depth,
            dim_head=64,
            heads=8,
            num_fusion_tokens=16,
            audio_patch_size=16,
            video_patch_size=16,
            video_temporal_patch_size=2,
            video_channels=3,
            return_token_types=(T.FUSION,)  # Only return fusion token
        )
        self.classifier = nn.Sequential(
            nn.LayerNorm(dim),
            nn.Linear(dim, num_classes)
        )

    def forward(self, audio, video):
        # Get fusion token representation
        fusion_token = self.zorro(audio=audio, video=video)
        fusion_token = fusion_token.squeeze(1)  # (batch, dim)
        return self.classifier(fusion_token)

# Initialize classifier
model = MultimodalClassifier(num_classes=10)
```

--------------------------------

### Contrastive Learning Setup with Zorro

Source: https://context7.com/lucidrains/zorro-pytorch/llms.txt

Configures Zorro for contrastive learning by returning modality-specific tokens and computes a contrastive loss using InfoNCE.

```python
import torch
import torch.nn.functional as F
from zorro_pytorch import Zorro, TokenTypes as T

# Model configured for contrastive learning
model = Zorro(
    dim=512,
    depth=6,
    dim_head=64,
    heads=8,
    num_fusion_tokens=16,
    audio_patch_size=16,
    video_patch_size=16,
    video_temporal_patch_size=2,
    return_token_types=(
        T.AUDIO,   # Index 0: audio representation
        T.VIDEO,   # Index 1: video representation
    )
)

# Batch of audio-video pairs
batch_size = 8
video = torch.randn(batch_size, 3, 8, 32, 32)
audio = torch.randn(batch_size, 10240)

# Get audio and video embeddings
tokens = model(audio=audio, video=video)
audio_emb = tokens[:, 0, :]  # (batch, dim)
video_emb = tokens[:, 1, :]  # (batch, dim)

# Normalize embeddings
audio_emb = F.normalize(audio_emb, dim=-1)
video_emb = F.normalize(video_emb, dim=-1)

# Compute similarity matrix for InfoNCE loss
temperature = 0.07
similarity = torch.einsum('id,jd->ij', audio_emb, video_emb) / temperature

# Contrastive loss (diagonal pairs are positives)
labels = torch.arange(batch_size)
loss_a2v = F.cross_entropy(similarity, labels)
loss_v2a = F.cross_entropy(similarity.T, labels)
contrastive_loss = (loss_a2v + loss_v2a) / 2

print(f"Contrastive Loss: {contrastive_loss.item():.4f}")
```

--------------------------------

### Custom Video Patch Configuration

Source: https://context7.com/lucidrains/zorro-pytorch/llms.txt

Configure video patchification with separate temporal, height, and width patch sizes to control the granularity of video tokenization. Video dimensions must be divisible by their respective patch sizes. This example shows custom `video_patch_size` and `video_temporal_patch_size`.

```python
import torch
from zorro_pytorch import Zorro, TokenTypes as T

# Custom video patch configuration
model = Zorro(
    dim=256,
    depth=4,
    heads=4,
    dim_head=64,
    num_fusion_tokens=8,
    audio_patch_size=16,
    video_patch_size=(8, 8),        # Spatial patches: 8x8 pixels
    video_temporal_patch_size=4,    # Temporal patches: 4 frames
    video_channels=3,               # RGB video
    return_token_types=(T.VIDEO, T.FUSION)
)

# Video must have dimensions divisible by patch sizes
# Time: 16 frames / 4 temporal_patch = 4 temporal tokens
# Height: 64 pixels / 8 spatial_patch = 8 height tokens
# Width: 64 pixels / 8 spatial_patch = 8 width tokens
# Total video tokens: 4 * 8 * 8 = 256 tokens

video = torch.randn(1, 3, 16, 64, 64)  # (batch, channels, time, height, width)
audio = torch.randn(1, 8192)

output = model(audio=audio, video=video)
print(output.shape)  # torch.Size([1, 2, 256])
```

--------------------------------

### Initialize Zorro Model and Process Data

Source: https://github.com/lucidrains/zorro-pytorch/blob/main/README.md

Initialize the Zorro model with specified dimensions and process audio and video data. The return_token_types parameter controls which token types are returned.

```python
import torch
from zorro_pytorch import Zorro, TokenTypes as T

model = Zorro(
    dim = 512,
    depth = 6,
    dim_head = 64,
    heads = 8,
    ff_mult = 4,
    num_fusion_tokens = 16,
    audio_patch_size = 16,
    video_patch_size = 16,
    video_temporal_patch_size = 2,
    video_channels = 3,
    return_token_types = (
        T.AUDIO,
        T.AUDIO,
        T.FUSION,
        T.GLOBAL,
        T.VIDEO,
        T.VIDEO,
        T.VIDEO,
    ) # say you want to return 2 tokens for audio, 1 token for fusion, 3 for video - for whatever self-supervised learning, supervised learning, etc etc
)

video = torch.randn(2, 3, 8, 32, 32) # (batch, channels, time, height, width)
audio = torch.randn(2, 1024 * 10)    # (batch, time)

return_tokens = model(audio = audio, video = video) # (2, 6, 512) - all 6 tokes as indicated above is returned
```

--------------------------------

### Forward Pass with Audio and Video

Source: https://context7.com/lucidrains/zorro-pytorch/llms.txt

The forward method processes raw audio waveforms and video tensors, automatically converting audio to spectrograms and patchifying both modalities before transformer processing. Returns pooled tokens based on the configured return_token_types.

```python
import torch
from zorro_pytorch import Zorro, TokenTypes as T

model = Zorro(
    dim=512,
    depth=6,
    dim_head=64,
    heads=8,
    ff_mult=4,
    num_fusion_tokens=16,
    audio_patch_size=16,
    video_patch_size=16,
    video_temporal_patch_size=2,
    video_channels=3,
    return_token_types=(
        T.AUDIO,
        T.AUDIO,
        T.FUSION,
        T.GLOBAL,
        T.VIDEO,
        T.VIDEO,
        T.VIDEO,
    )
)

# Prepare input tensors
batch_size = 2
video = torch.randn(batch_size, 3, 8, 32, 32)  # (batch, channels, time, height, width)
audio = torch.randn(batch_size, 1024 * 10)     # (batch, time) - raw waveform

# Forward pass returns all configured token types
return_tokens = model(audio=audio, video=video)
print(return_tokens.shape)
# Output: torch.Size([2, 7, 512]) - 7 tokens as configured, each with dim=512
```

--------------------------------

### Process Data for Contrastive Learning

Source: https://github.com/lucidrains/zorro-pytorch/blob/main/README.md

Process audio and video data to extract specific token types (e.g., one audio and one video token) for contrastive learning. The return_token_indices parameter specifies which tokens to return.

```python
audio_token, video_token = model(audio = audio, video = video, return_token_indices = (0, 3)).unbind(dim = -2) # (2, 512), (2, 512)
```

--------------------------------

### Initialize Zorro Multimodal Transformer

Source: https://context7.com/lucidrains/zorro-pytorch/llms.txt

Initialize the Zorro multimodal transformer with configurable dimensions, depth, attention parameters, and patch sizes for audio and video modalities. Fusion tokens enable cross-modal information exchange.

```python
import torch
from zorro_pytorch import Zorro, TokenTypes as T

# Initialize the Zorro multimodal transformer
model = Zorro(
    dim=512,                        # Model embedding dimension
    depth=6,                        # Number of transformer layers
    dim_head=64,                    # Dimension per attention head
    heads=8,                        # Number of attention heads
    ff_mult=4,                      # Feedforward expansion multiplier
    num_fusion_tokens=16,           # Number of learnable fusion tokens
    audio_patch_size=16,            # Audio spectrogram patch size (can be tuple)
    video_patch_size=16,            # Video spatial patch size (can be tuple)
    video_temporal_patch_size=2,    # Video temporal patch size
    video_channels=3,               # Number of video channels (RGB=3)
    return_token_types=(
        T.AUDIO,
        T.VIDEO,
        T.FUSION,
    )
)

print(f"Model parameters: {sum(p.numel() for p in model.parameters()):,}")
# Output: Model parameters: 12,345,678
```

--------------------------------

### Basic Model Forward Pass

Source: https://context7.com/lucidrains/zorro-pytorch/llms.txt

Demonstrates a basic forward pass of the Zorro model with audio and video inputs, printing the output shape.

```python
batch_size = 4
video = torch.randn(batch_size, 3, 8, 32, 32)
audio = torch.randn(batch_size, 10240)

# Forward pass
logits = model(audio, video)
print(logits.shape)  # torch.Size([4, 10])
```

--------------------------------

### Selective Token Return with Indices

Source: https://context7.com/lucidrains/zorro-pytorch/llms.txt

Use the `return_token_indices` parameter to selectively return only specific tokens from the configured `return_token_types`. This is useful for tasks like contrastive learning where only audio and video representations are needed.

```python
import torch
from zorro_pytorch import Zorro, TokenTypes as T

model = Zorro(
    dim=512,
    depth=6,
    dim_head=64,
    heads=8,
    num_fusion_tokens=16,
    audio_patch_size=16,
    video_patch_size=16,
    video_temporal_patch_size=2,
    video_channels=3,
    return_token_types=(
        T.AUDIO,    # Index 0
        T.AUDIO,    # Index 1
        T.FUSION,   # Index 2
        T.VIDEO,    # Index 3
        T.VIDEO,    # Index 4
    )
)

video = torch.randn(2, 3, 8, 32, 32)
audio = torch.randn(2, 1024 * 10)

# Return only first audio token (index 0) and first video token (index 3)
selected_tokens = model(audio=audio, video=video, return_token_indices=(0, 3))
audio_token, video_token = selected_tokens.unbind(dim=-2)

print(audio_token.shape)  # torch.Size([2, 512])
print(video_token.shape)  # torch.Size([2, 512])

# Use for contrastive learning
similarity = torch.einsum('bd,bd->b', audio_token, video_token)
print(similarity.shape)   # torch.Size([2])
```

--------------------------------

### Spectrogram Configuration for Audio Input

Source: https://context7.com/lucidrains/zorro-pytorch/llms.txt

The Zorro model includes built-in spectrogram conversion for audio inputs with configurable FFT parameters. Audio waveforms are automatically converted to spectrograms and patchified for transformer processing. Configure spectrogram parameters like `spec_n_fft`, `spec_power`, `spec_win_length`, and `spec_hop_length` for audio processing.

```python
import torch
from zorro_pytorch import Zorro, TokenTypes as T

# Configure spectrogram parameters for audio processing
model = Zorro(
    dim=512,
    depth=6,
    num_fusion_tokens=16,
    audio_patch_size=(16, 16),      # Height x width patches for spectrogram
    video_patch_size=16,
    video_temporal_patch_size=2,
    # Spectrogram configuration
    spec_n_fft=128,                 # FFT size
    spec_power=2,                   # Power for spectrogram
    spec_win_length=24,             # Window length
    spec_hop_length=None,           # Hop length (None = win_length // 4)
    spec_pad=0,                     # Padding
    spec_center=True,               # Center the signal
    spec_pad_mode='reflect',        # Padding mode
    return_token_types=(T.AUDIO, T.VIDEO, T.FUSION)
)

# Raw audio waveform input (automatically converted to spectrogram)
audio = torch.randn(2, 10240)       # 10240 samples
video = torch.randn(2, 3, 8, 32, 32)

output = model(audio=audio, video=video)
print(output.shape)  # torch.Size([2, 3, 512])
```

--------------------------------

### TokenTypes Enum Usage

Source: https://context7.com/lucidrains/zorro-pytorch/llms.txt

The TokenTypes enum defines the four types of tokens used in the Zorro architecture: AUDIO, VIDEO, FUSION, and GLOBAL. Configure the model to return specific token types.

```python
from zorro_pytorch import TokenTypes as T

# Available token types for return_token_types configuration
print(T.AUDIO.value)   # 0 - Audio modality tokens
print(T.VIDEO.value)   # 1 - Video modality tokens
print(T.FUSION.value)  # 2 - Cross-modal fusion tokens
print(T.GLOBAL.value)  # 3 - Global tokens (attend to all)

# Configure model to return specific token types
model = Zorro(
    dim=512,
    depth=6,
    return_token_types=(
        T.AUDIO,    # First return token attends to audio
        T.AUDIO,    # Second return token attends to audio
        T.FUSION,   # Third return token attends to all (fusion)
        T.GLOBAL,   # Fourth return token attends to all (global)
        T.VIDEO,    # Fifth return token attends to video
        T.VIDEO,    # Sixth return token attends to video
        T.VIDEO,    # Seventh return token attends to video
    )
)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.