### Install Nyström Attention

Source: https://github.com/lucidrains/nystrom-attention/blob/main/README.md

Install the nystrom-attention package using pip.

```bash
pip install nystrom-attention
```

--------------------------------

### Import Nyströmer Alias

Source: https://github.com/lucidrains/nystrom-attention/blob/main/README.md

Demonstrates importing the Nystromer alias for the NystromAttention class.

```python
from nystrom_attention import Nystromer
```

--------------------------------

### Initialize and Use Nystromformer

Source: https://context7.com/lucidrains/nystrom-attention/llms.txt

Build a complete transformer encoder and process sequences, including handling variable-length batches.

```python
import torch
from nystrom_attention import Nystromformer

# Initialize Nystromformer with 6 layers
model = Nystromformer(
    dim=512,                           # model dimension
    depth=6,                           # number of transformer layers
    dim_head=64,                       # dimension per attention head
    heads=8,                           # number of attention heads
    num_landmarks=256,                 # landmarks for Nyström approximation
    pinv_iterations=6,                 # Moore-Penrose iterations
    attn_values_residual=True,         # residual connection on attention values
    attn_values_residual_conv_kernel=33,  # kernel size for residual conv
    attn_dropout=0.1,                  # attention dropout
    ff_dropout=0.1                     # feed-forward dropout
)

# Process long sequences efficiently
x = torch.randn(1, 16384, 512)  # batch=1, seq_len=16384, dim=512
mask = torch.ones(1, 16384).bool()

# Forward pass through all transformer layers
output = model(x, mask=mask)  # Returns: (1, 16384, 512)

# Example: Processing variable-length sequences in a batch
batch_size = 4
max_seq_len = 8192
x_batch = torch.randn(batch_size, max_seq_len, 512)

# Create mask for variable lengths (e.g., sequences of length 5000, 6000, 7000, 8192)
lengths = [5000, 6000, 7000, 8192]
mask_batch = torch.zeros(batch_size, max_seq_len).bool()
for i, length in enumerate(lengths):
    mask_batch[i, :length] = True

output_batch = model(x_batch, mask=mask_batch)
```

--------------------------------

### Initialize and Use Nyström Attention

Source: https://github.com/lucidrains/nystrom-attention/blob/main/README.md

Initialize NystromAttention with specified dimensions and parameters. Use this for applying Nyström attention to input tensors with an optional mask.

```python
import torch
from nystrom_attention import NystromAttention

attn = NystromAttention(
    dim = 512,
    dim_head = 64,
    heads = 8,
    num_landmarks = 256,    # number of landmarks
    pinv_iterations = 6,    # number of moore-penrose iterations for approximating pinverse. 6 was recommended by the paper
    residual = True         # whether to do an extra residual with the value or not. supposedly faster convergence if turned on
)

x = torch.randn(1, 16384, 512)
mask = torch.ones(1, 16384).bool()

attn(x, mask = mask) # (1, 16384, 512)
```

--------------------------------

### Initialize and Use Nystromformer Model

Source: https://github.com/lucidrains/nystrom-attention/blob/main/README.md

Initialize the Nystromformer model with specified dimensions, attention heads, and depth. This model applies Nyström attention layers to input tensors with an optional mask.

```python
import torch
from nystrom_attention import Nystromformer

model = Nystromformer(
    dim = 512,
    dim_head = 64,
    heads = 8,
    depth = 6,
    num_landmarks = 256,
    pinv_iterations = 6
)

x = torch.randn(1, 16384, 512)
mask = torch.ones(1, 16384).bool()

model(x, mask = mask) # (1, 16384, 512)
```

--------------------------------

### Use Nystromer Alias

Source: https://context7.com/lucidrains/nystrom-attention/llms.txt

Use the Nystromer alias as an alternative import for the Nystromformer class.

```python
from nystrom_attention import Nystromer

# Nystromer is identical to Nystromformer
model = Nystromer(
    dim=512,
    depth=6,
    dim_head=64,
    heads=8,
    num_landmarks=256,
    pinv_iterations=6
)

x = torch.randn(1, 4096, 512)
output = model(x)  # Returns: (1, 4096, 512)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.