### Run Training Example

Source: https://github.com/lucidrains/deep-cross-attention/blob/main/README.md

Install example dependencies and execute the training script.

```bash
$ pip install .[examples]
```

```bash
$ python train.py
```

--------------------------------

### Install Deep Cross Attention with Examples

Source: https://context7.com/lucidrains/deep-cross-attention/llms.txt

Installs the library with example dependencies. Ensure the enwik8 dataset is downloaded and placed in the ./data/ directory.

```bash
pip install deep-cross-attention[examples]
```

--------------------------------

### Full Training Configuration Example

Source: https://context7.com/lucidrains/deep-cross-attention/llms.txt

Provides a comprehensive configuration for training, including CPU usage, batch size, learning rate, and generation parameters.

```bash
python train.py \
    --cpu=False \
    --num_batches=100000 \
    --batch_size=4 \
    --grad_accum_every=4 \
    --learning_rate=1e-4 \
    --validate_every=100 \
    --generate_every=500 \
    --generate_length=512 \
    --seq_len=512 \
    --use_dca=True
```

--------------------------------

### Install Deep Cross Attention

Source: https://github.com/lucidrains/deep-cross-attention/blob/main/README.md

Install the package via pip.

```bash
$ pip install deep-cross-attention
```

--------------------------------

### Complete Training Loop with DCAGPT and Accelerate

Source: https://context7.com/lucidrains/deep-cross-attention/llms.txt

Demonstrates an end-to-end training loop for DCAGPT using Accelerate for distributed training. Includes custom dataset, model initialization, optimizer setup, and text generation.

```python
import torch
from torch.optim import Adam
from torch.utils.data import DataLoader, Dataset
from accelerate import Accelerator
from deep_cross_attention import DCAGPT

# Custom dataset for character-level modeling
class TextDataset(Dataset):
    def __init__(self, data, seq_len):
        self.data = data
        self.seq_len = seq_len

    def __len__(self):
        return len(self.data) // self.seq_len

    def __getitem__(self, idx):
        start = torch.randint(0, len(self.data) - self.seq_len - 1, (1,)).item()
        return torch.tensor(self.data[start:start + self.seq_len + 1], dtype=torch.long)

# Initialize accelerator and model
accelerator = Accelerator()
model = DCAGPT(
    num_tokens=256,
    dim=512,
    depth=6,
    past_layers_k=2
)

# Prepare data (example with random data)
train_data = torch.randint(0, 256, (100000,)).tolist()
dataset = TextDataset(train_data, seq_len=512)
loader = DataLoader(dataset, batch_size=4, shuffle=True)

# Setup optimizer
optimizer = Adam(model.parameters(), lr=1e-4)
model, optimizer, loader = accelerator.prepare(model, optimizer, loader)

# Training loop
model.train()
for batch_idx, batch in enumerate(loader):
    optimizer.zero_grad()
    loss = model(batch, return_loss=True)
    accelerator.backward(loss)
    accelerator.clip_grad_norm_(model.parameters(), 0.5)
    optimizer.step()

    if batch_idx % 100 == 0:
        print(f"Step {batch_idx}, Loss: {loss.item():.4f}")

    if batch_idx >= 1000:
        break

# Text generation helper
def generate(model, prompt_ids, max_length=100, temperature=1.0):
    model.eval()
    generated = prompt_ids.clone()
    with torch.no_grad():
        for _ in range(max_length):
            logits = model(generated)[:, -1, :]
            probs = torch.softmax(logits / temperature, dim=-1)
            next_token = torch.multinomial(probs, 1)
            generated = torch.cat([generated, next_token], dim=1)
    return generated

# Generate text
prompt = torch.randint(0, 256, (1, 32)).to(accelerator.device)
output = generate(model, prompt, max_length=100)
print(f"Generated sequence shape: {output.shape}")
```

--------------------------------

### Initialize and use baseline GPT model

Source: https://context7.com/lucidrains/deep-cross-attention/llms.txt

Standard GPT implementation for comparison, utilizing traditional residual connections.

```python
import torch
from deep_cross_attention import GPT

# Initialize baseline GPT model
baseline_model = GPT(
    num_tokens=256,        # Vocabulary size
    dim=512,               # Model dimension
    depth=6,               # Number of transformer layers
    dim_head=64,           # Dimension per attention head
    heads=8,               # Number of attention heads
    ff_expansion_factor=4. # Feedforward expansion factor
)

# Forward pass for inference
ids = torch.randint(0, 256, (2, 1024))
logits = baseline_model(ids)  # Returns: (2, 1024, 256)

# Forward pass for training
loss = baseline_model(ids, return_loss=True)
print(f"Baseline loss: {loss.item():.4f}")
```

--------------------------------

### Initialize and Run DCAGPT

Source: https://github.com/lucidrains/deep-cross-attention/blob/main/README.md

Instantiate the DCAGPT model and perform a forward pass with input token IDs.

```python
import torch
from deep_cross_attention import DCAGPT

gpt = DCAGPT(
    num_tokens = 256,
    dim = 512,
    depth = 6,
    heads = 8,
    dim_head = 64,
    past_layers_k = 2
)

ids = torch.randint(0, 256, (2, 4096))

logits = gpt(ids) # (2, 4096, 256)
```

--------------------------------

### Train Baseline GPT Model

Source: https://context7.com/lucidrains/deep-cross-attention/llms.txt

Runs training with a baseline GPT model for comparison. Adjust batch size and learning rate as needed.

```bash
python train.py --use_dca=False --batch_size=4 --learning_rate=1e-4
```

--------------------------------

### Initialize and train DCAGPT model

Source: https://context7.com/lucidrains/deep-cross-attention/llms.txt

Demonstrates model instantiation, inference, and a standard training step using the Deep Cross Attention GPT model.

```python
import torch
from deep_cross_attention import DCAGPT

# Initialize the DCAGPT model
model = DCAGPT(
    num_tokens=256,        # Vocabulary size
    dim=512,               # Model dimension
    depth=6,               # Number of transformer layers
    past_layers_k=2,       # Number of past layers to consider (k value from paper)
    dim_head=64,           # Dimension per attention head
    heads=8,               # Number of attention heads
    ff_expansion_factor=4. # Feedforward expansion factor
)

# Forward pass for inference
ids = torch.randint(0, 256, (2, 4096))  # (batch_size, sequence_length)
logits = model(ids)  # Returns: (2, 4096, 256)

# Forward pass for training with cross-entropy loss
ids = torch.randint(0, 256, (2, 4096))
loss = model(ids, return_loss=True)  # Returns: scalar loss tensor
loss.backward()

# Example: Complete training step
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
optimizer.zero_grad()
loss = model(ids, return_loss=True)
loss.backward()
optimizer.step()
print(f"Training loss: {loss.item():.4f}")
```

--------------------------------

### Initialize and use DCABlock

Source: https://context7.com/lucidrains/deep-cross-attention/llms.txt

Transformer block using GRN-based routing. Expects stacked tokens from multiple layers as input.

```python
import torch
from deep_cross_attention import DCABlock

# Initialize a DCA block
dca_block = DCABlock(
    dim=512,                    # Model dimension
    grn_num_layers=4,           # Number of input layers for GRN
    dim_head=64,                # Dimension per attention head
    heads=8,                    # Number of attention heads
    ff_expansion_factor=4.,     # Feedforward expansion factor
    grn_activation=torch.nn.ReLU()  # GRN activation function
)

# Forward pass with stacked tokens from multiple layers
# Input shape: (num_layers, batch_size, sequence_length, dim)
tokens_across_depth = torch.randn(4, 2, 512, 512)  # 4 past layers
output = dca_block(tokens_across_depth)  # Returns: (2, 512, 512)

# Example: Building custom architecture with DCABlock
batch_size, seq_len, dim = 2, 256, 512
layer_outputs = [torch.randn(batch_size, seq_len, dim) for _ in range(4)]
stacked = torch.stack(layer_outputs)  # (4, 2, 256, 512)
next_layer_output = dca_block(stacked)
print(f"Output shape: {next_layer_output.shape}")  # (2, 256, 512)
```

--------------------------------

### GRN Module Initialization and Usage

Source: https://context7.com/lucidrains/deep-cross-attention/llms.txt

Initializes and uses the GRN module for aggregating token representations across different layers. This is useful for preparing Q, K, V inputs for attention mechanisms.

```python
tokens_across_depth = torch.randn(4, 2, 256, 512)
aggregated = grn_single(tokens_across_depth)  # Returns: (2, 256, 512)

grn_qkv = GRN(
    dim=512,
    num_layers=4,
    num_outputs=3,  # Separate outputs for Q, K, V
    activation=torch.nn.ReLU()
)

# Forward pass producing Q, K, V inputs
q_input, k_input, v_input = grn_qkv(tokens_across_depth)
print(f"Q shape: {q_input.shape}")  # (2, 256, 512)
print(f"K shape: {k_input.shape}")  # (2, 256, 512)
print(f"V shape: {v_input.shape}")  # (2, 256, 512)
```

--------------------------------

### Initialize Gated Residual Network

Source: https://context7.com/lucidrains/deep-cross-attention/llms.txt

Core mechanism for dynamic layer aggregation. Requires definition of input layers and output dimensions.

```python
import torch
from deep_cross_attention import GRN

# Initialize GRN for single output (e.g., final aggregation)
grn_single = GRN(
    dim=512,              # Model dimension
    num_layers=4,         # Number of input layers to aggregate
    num_outputs=1,        # Single aggregated output
    activation=torch.nn.ReLU()
)

# Forward pass with stacked layer outputs
```

--------------------------------

### Train DCAGPT Model

Source: https://context7.com/lucidrains/deep-cross-attention/llms.txt

Runs training with the Deep Cross Attention model enabled. Adjust batch size and learning rate as needed.

```bash
python train.py --use_dca=True --batch_size=4 --learning_rate=1e-4
```

--------------------------------

### BibTeX Citation

Source: https://github.com/lucidrains/deep-cross-attention/blob/main/README.md

Citation for the DeepCrossAttention paper.

```bibtex
@inproceedings{Heddes2025DeepCrossAttentionST,
    title   = {DeepCrossAttention: Supercharging Transformer Residual Connections},
    author  = {Mike Heddes and Adel Javanmard and Kyriakos Axiotis and Gang Fu and MohammadHossein Bateni and Vahab S. Mirrokni},
    year    = {2025},
    url     = {https://api.semanticscholar.org/CorpusID:276250576}
}
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.