### Install Linformer

Source: https://github.com/lucidrains/linformer/blob/master/README.md

Install the linformer package using pip.

```bash
pip install linformer
```

--------------------------------

### Initialize LinformerSelfAttention

Source: https://context7.com/lucidrains/linformer/llms.txt

Import the standalone self-attention layer for custom architectures.

```python
import torch
from linformer import LinformerSelfAttention
```

--------------------------------

### Initialize and use LinformerSelfAttention

Source: https://context7.com/lucidrains/linformer/llms.txt

Demonstrates self-attention and cross-attention configurations for the Linformer layer.

```python
attn = LinformerSelfAttention(
    dim=512,               # input dimension
    seq_len=4096,          # maximum sequence length
    heads=8,               # number of attention heads
    k=256,                 # projection dimension for linear complexity
    dim_head=64,           # dimension per head (optional)
    one_kv_head=True,      # share key/value across heads
    share_kv=True,         # share key and value projections
    dropout=0.1            # attention dropout
)

# Self-attention on input
x = torch.randn(1, 4096, 512)  # (batch_size, seq_len, dim)
output = attn(x)  # Output: (1, 4096, 512)

# Cross-attention with external context
# Useful for encoder-decoder architectures
attn_with_context = LinformerSelfAttention(
    dim=512,
    seq_len=8192,          # seq_len refers to context length
    heads=8,
    k=256,
    one_kv_head=True,
    share_kv=True
)

query = torch.randn(1, 2048, 512)      # shorter query sequence
context = torch.randn(1, 8192, 512)    # longer context for keys/values
output = attn_with_context(query, context)  # Output: (1, 2048, 512)
```

--------------------------------

### Initialize and Use LinformerLM

Source: https://context7.com/lucidrains/linformer/llms.txt

Create a full language model with embeddings and perform forward passes or training. Supports variable sequence lengths during inference.

```python
import torch
from linformer import LinformerLM

# Initialize a language model for vocabulary of 20000 tokens
model = LinformerLM(
    num_tokens=20000,      # vocabulary size
    dim=512,               # embedding dimension
    seq_len=4096,          # maximum sequence length
    depth=12,              # number of transformer layers
    heads=8,               # number of attention heads
    dim_head=128,          # dimension per attention head (optional, defaults to dim // heads)
    k=256,                 # projection dimension for keys/values (lower = more efficient)
    one_kv_head=True,      # share one key/value head across all attention heads
    share_kv=False,        # whether keys and values share the same projection
    reversible=True,       # enable reversible layers for memory efficiency
    dropout=0.1            # dropout rate
)

# Forward pass with token indices
x = torch.randint(0, 20000, (1, 4096))  # (batch_size, seq_len)
logits = model(x)  # Output: (1, 4096, 20000) - logits over vocabulary

# Training example
model.train()
target = torch.randint(0, 20000, (1, 4096))
loss = torch.nn.functional.cross_entropy(
    logits.view(-1, 20000),
    target.view(-1)
)
loss.backward()

# Inference with shorter sequences (variable length supported)
short_input = torch.randint(0, 20000, (4, 512))  # batch of 4, length 512
output = model(short_input)  # Output: (4, 512, 20000)
```

--------------------------------

### Build a custom transformer block

Source: https://context7.com/lucidrains/linformer/llms.txt

Shows how to integrate LinformerSelfAttention into a custom PyTorch module.

```python
class CustomTransformerBlock(torch.nn.Module):
    def __init__(self, dim, seq_len, heads, k):
        super().__init__()
        self.attn = LinformerSelfAttention(
            dim=dim, seq_len=seq_len, heads=heads, k=k,
            one_kv_head=True, share_kv=True
        )
        self.norm1 = torch.nn.LayerNorm(dim)
        self.norm2 = torch.nn.LayerNorm(dim)
        self.ff = torch.nn.Sequential(
            torch.nn.Linear(dim, dim * 4),
            torch.nn.GELU(),
            torch.nn.Linear(dim * 4, dim)
        )

    def forward(self, x):
        x = x + self.attn(self.norm1(x))
        x = x + self.ff(self.norm2(x))
        return x

block = CustomTransformerBlock(dim=512, seq_len=4096, heads=8, k=256)
x = torch.randn(2, 4096, 512)
output = block(x)  # Output: (2, 4096, 512)
```

--------------------------------

### Configure FeedForward network

Source: https://context7.com/lucidrains/linformer/llms.txt

Standard and GLU-enabled feed-forward network configurations.

```python
import torch
from linformer.linformer import FeedForward

# Standard feed-forward network
ff = FeedForward(
    dim=512,               # input/output dimension
    mult=4,                # hidden dimension multiplier (hidden = dim * mult)
    dropout=0.1,           # dropout probability
    glu=False              # use standard GELU activation
)

x = torch.randn(1, 4096, 512)
output = ff(x)  # Output: (1, 4096, 512)

# GLU variant (Gated Linear Unit) for potentially better performance
ff_glu = FeedForward(
    dim=512,
    mult=4,
    dropout=0.1,
    glu=True               # enables gating mechanism
)

output_glu = ff_glu(x)  # Output: (1, 4096, 512)
```

--------------------------------

### Linformer Model

Source: https://github.com/lucidrains/linformer/blob/master/README.md

Instantiate and use the general Linformer model. The input tensor should have dimensions (batch_size, sequence_length, dimension).

```python
import torch
from linformer import Linformer

model = Linformer(
    dim = 512,
    seq_len = 4096,
    depth = 12,
    heads = 8,
    k = 256,
    one_kv_head = True,
    share_kv = True
)

x = torch.randn(1, 4096, 512)
model(x) # (1, 4096, 512)
```

--------------------------------

### Linformer Language Model

Source: https://github.com/lucidrains/linformer/blob/master/README.md

Instantiate and use the Linformer language model. Ensure the input tensor matches the specified sequence length and token count.

```python
import torch
from linformer import LinformerLM

model = LinformerLM(
    num_tokens = 20000,
    dim = 512,
    seq_len = 4096,
    depth = 12,
    heads = 8,
    dim_head = 128,        # be able to set the dimension of each head in multi-head attention
    k = 256,               # this is the k that the key/values are projected to along the sequence dimension
    one_kv_head = True,    # share one key/value head across all heads
    share_kv = False,      # share the same projection for keys and values
    reversible = True      # make network reversible, like Reformer
)

x = torch.randint(0, 20000, (1, 4096))
model(x) # (1, 4096, 20000)
```

--------------------------------

### Initialize and Use Linformer Encoder

Source: https://context7.com/lucidrains/linformer/llms.txt

Use the transformer encoder for pre-embedded sequences. Reversible layers can be enabled for memory-efficient training on very long sequences.

```python
import torch
from linformer import Linformer

# Initialize transformer encoder
model = Linformer(
    dim=512,               # input/output dimension
    seq_len=4096,          # maximum sequence length
    depth=12,              # number of transformer layers
    k=256,                 # projection dimension (controls efficiency vs quality tradeoff)
    heads=8,               # number of attention heads
    dim_head=64,           # dimension per head (optional)
    one_kv_head=True,      # use single key/value head (more parameter efficient)
    share_kv=True,         # share key and value projections (even more efficient)
    reversible=False,      # set True to reduce memory usage during training
    dropout=0.1            # dropout probability
)

# Process pre-embedded sequences
x = torch.randn(1, 4096, 512)  # (batch_size, seq_len, dim)
output = model(x)  # Output: (1, 4096, 512)

# Batch processing
batch = torch.randn(32, 4096, 512)
batch_output = model(batch)  # Output: (32, 4096, 512)

# Memory-efficient training with reversible layers
efficient_model = Linformer(
    dim=512,
    seq_len=8192,          # very long sequences
    depth=24,              # deep network
    k=128,                 # aggressive projection for efficiency
    heads=8,
    reversible=True        # saves memory by recomputing activations
)

long_sequence = torch.randn(4, 8192, 512)
result = efficient_model(long_sequence)  # Output: (4, 8192, 512)
```

--------------------------------

### Implement ReversibleSequence

Source: https://context7.com/lucidrains/linformer/llms.txt

Reduces memory usage by recomputing activations during the backward pass.

```python
import torch
import torch.nn as nn
from linformer.reversible import ReversibleSequence

# Create pairs of functions for reversible layers
# Each layer needs (f, g) pair for the reversible formulation
layers = nn.ModuleList([
    nn.ModuleList([
        nn.Linear(256, 256),  # f function
        nn.Linear(256, 256)   # g function
    ]) for _ in range(6)
])

# Wrap in reversible sequence
rev_net = ReversibleSequence(layers)

# Forward pass - input dimension is doubled internally
x = torch.randn(4, 100, 256)
output = rev_net(x)  # Output: (4, 100, 256)

# During backward pass, activations are recomputed rather than stored
# This significantly reduces memory for deep networks
```

--------------------------------

### Linformer Self-Attention with Context

Source: https://github.com/lucidrains/linformer/blob/master/README.md

Use the Linformer self-attention layer with contextual keys. The sequence length is validated against the length of the context tensor.

```python
import torch
from linformer import LinformerSelfAttention

attn = LinformerSelfAttention(
    dim = 512,
    seq_len = 8192,
    heads = 8,
    k = 256,
    one_kv_head = True,
    share_kv = True
)

x = torch.randn(1, 2048, 512)
context = torch.randn(1, 8192, 512)
attn(x, context) # (1, 2048, 512)
```

--------------------------------

### Linformer Self-Attention Layer

Source: https://github.com/lucidrains/linformer/blob/master/README.md

Instantiate and use a single Linformer self-attention layer. The input tensor should have dimensions (batch_size, sequence_length, dimension).

```python
import torch
from linformer import LinformerSelfAttention

attn = LinformerSelfAttention(
    dim = 512,
    seq_len = 4096,
    heads = 8,
    k = 256,
    one_kv_head = True,
    share_kv = True
)

x = torch.randn(1, 4096, 512)
attn(x) # (1, 4096, 512)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.