### Install H-Transformer-1D

Source: https://github.com/lucidrains/h-transformer-1d/blob/main/README.md

Install the package via pip.

```bash
$ pip install h-transformer-1d
```

--------------------------------

### Text Generation with H-Transformer-1D

Source: https://context7.com/lucidrains/h-transformer-1d/llms.txt

Shows how to perform text generation using a trained H-Transformer-1D model in evaluation mode. It generates a sequence of tokens starting from random initial tokens.

```python
# Text generation
model.eval()
start_tokens = torch.randint(0, 256, (1, 64)).cuda()

with torch.no_grad():
    generated = model.generate(
        start_tokens=start_tokens,
        seq_len=512,
        temperature=0.9,
        filter_thres=0.9
    )

print(f"Generated text: {decode_tokens(generated[0])}")
```

--------------------------------

### Initialize and Run H-Transformer-1D

Source: https://github.com/lucidrains/h-transformer-1d/blob/main/README.md

Configure the model architecture and perform a forward pass with input tokens and masks.

```python
import torch
from h_transformer_1d import HTransformer1D

model = HTransformer1D(
    num_tokens = 256,          # number of tokens
    dim = 512,                 # dimension
    depth = 12,                # depth
    causal = False,            # autoregressive or not
    max_seq_len = 8192,        # maximum sequence length
    heads = 8,                 # heads
    dim_head = 64,             # dimension per head
    block_size = 128,          # block size
    reversible = True,         # use reversibility, to save on memory with increased depth
    shift_tokens = True        # whether to shift half the feature space by one along the sequence dimension, for faster convergence (experimental feature)
)

x = torch.randint(0, 256, (1, 8000))   # variable sequence length
mask = torch.ones((1, 8000)).bool()    # variable mask length

# network will automatically pad to power of 2, do hierarchical attention, etc

logits = model(x, mask = mask) # (1, 8000, 256)
```

--------------------------------

### Training Loop with Gradient Accumulation and Clipping

Source: https://context7.com/lucidrains/h-transformer-1d/llms.txt

Demonstrates a typical training loop for a sequence model. It includes gradient accumulation to simulate larger batch sizes and gradient clipping to stabilize training.

```python
# Example: create synthetic data (replace with actual data loading)
data_train = torch.randint(0, 256, (int(1e6),))
train_dataset = TextSamplerDataset(data_train, SEQ_LEN)
train_loader = cycle(DataLoader(train_dataset, batch_size=BATCH_SIZE))

# Optimizer
optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)

# Training loop
for i in range(100):
    model.train()

    # Gradient accumulation
    for _ in range(GRADIENT_ACCUMULATE_EVERY):
        loss = model(next(train_loader))
        loss.backward()

    # Gradient clipping and optimizer step
    torch.nn.utils.clip_grad_norm_(model.parameters(), 0.25)
    optimizer.step()
    optimizer.zero_grad()

    if i % 10 == 0:
        print(f"Step {i}, Loss: {loss.item():.4f}")
```

--------------------------------

### Configure HTransformer1D for Training

Source: https://context7.com/lucidrains/h-transformer-1d/llms.txt

Sets up a full HTransformer1D model with an autoregressive wrapper for language modeling. Enabling reversible mode is recommended for long sequences.

```python
import torch
import torch.optim as optim
from torch.utils.data import DataLoader, Dataset
from h_transformer_1d import HTransformer1D
from h_transformer_1d.autoregressive_wrapper import AutoregressiveWrapper

# Hyperparameters
SEQ_LEN = 4096
BATCH_SIZE = 4
LEARNING_RATE = 2e-4
GRADIENT_ACCUMULATE_EVERY = 4

# Create model
model = HTransformer1D(
    num_tokens=256,           # byte-level vocabulary
    dim=512,
    max_seq_len=SEQ_LEN,
    depth=8,
    heads=8,
    causal=True,
    reversible=True           # memory efficient for long sequences
)

model = AutoregressiveWrapper(model)
model.cuda()
```

--------------------------------

### Infinite Data Loader

Source: https://context7.com/lucidrains/h-transformer-1d/llms.txt

Creates an infinite data loader by cycling through a given PyTorch DataLoader. This is useful for training loops where data needs to be continuously provided.

```python
def cycle(loader):
    while True:
        for data in loader:
            yield data
```

--------------------------------

### Initialize HTransformer1D (Causal Decoder)

Source: https://context7.com/lucidrains/h-transformer-1d/llms.txt

Use this for causal (autoregressive) decoder models for language modeling. The model handles sequence padding and masking internally.

```python
# Causal (autoregressive) decoder model for language modeling
decoder_model = HTransformer1D(
    num_tokens=256,
    dim=512,
    depth=8,
    causal=True,             # causal attention for autoregressive generation
    max_seq_len=4096,
    heads=8,
    dim_head=64,
    block_size=128,
    reversible=True
)

# Language modeling forward pass
x_causal = torch.randint(0, 256, (2, 2048))  # batch of 2
logits_causal = decoder_model(x_causal)       # output: (2, 2048, 256)
print(f"Causal output shape: {logits_causal.shape}")
```

--------------------------------

### Initialize Non-Causal HAttention1D

Source: https://context7.com/lucidrains/h-transformer-1d/llms.txt

Instantiates a bidirectional hierarchical attention layer with O(N log N) complexity. Requires rotary positional embeddings for optimal performance.

```python
import torch
from h_transformer_1d.h_transformer_1d import HAttention1D
from rotary_embedding_torch import RotaryEmbedding

# Create hierarchical attention layer
pos_emb = RotaryEmbedding(dim=64)  # rotary positional embedding

h_attention = HAttention1D(
    dim=512,                  # input dimension
    heads=8,                  # number of attention heads
    dim_head=64,              # dimension per head
    block_size=16,            # hierarchical block size
    pos_emb=pos_emb,          # optional positional embedding
    eps=1e-8                  # numerical stability epsilon
)

# Forward pass with embeddings (not tokens)
batch_size, seq_len, dim = 2, 1024, 512
x = torch.randn(batch_size, seq_len, dim)
mask = torch.ones(batch_size, seq_len).bool()

# Apply hierarchical attention
output = h_attention(x, mask=mask)
print(f"Input: {x.shape}, Output: {output.shape}")  # both (2, 1024, 512)
```

--------------------------------

### Initialize CausalHAttention1D

Source: https://context7.com/lucidrains/h-transformer-1d/llms.txt

Instantiates an autoregressive hierarchical attention layer. The max_seq_len parameter is mandatory for precomputing causal masks.

```python
import torch
from h_transformer_1d.h_transformer_1d import CausalHAttention1D
from rotary_embedding_torch import RotaryEmbedding

# Create causal hierarchical attention layer
pos_emb = RotaryEmbedding(dim=64)

causal_attention = CausalHAttention1D(
    dim=512,
    max_seq_len=2048,         # required for precomputing causal masks
    heads=8,
    dim_head=64,
    block_size=16,
    pos_emb=pos_emb,
    eps=1e-8
)

# Forward pass - each position only attends to previous positions
batch_size, seq_len, dim = 2, 1024, 512
x = torch.randn(batch_size, seq_len, dim)

output = causal_attention(x)
print(f"Causal attention output: {output.shape}")  # (2, 1024, 512)
```

--------------------------------

### Implement ReversibleSequence for Memory Efficiency

Source: https://context7.com/lucidrains/h-transformer-1d/llms.txt

Uses reversible residual connections to recompute activations during backpropagation. This reduces memory footprint for deep models at the expense of extra computation.

```python
import torch
import torch.nn as nn
from h_transformer_1d.reversible import ReversibleSequence, SequentialSequence

# Define attention and feedforward layers
class SimpleAttention(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.proj = nn.Linear(dim, dim)

    def forward(self, x, **kwargs):
        return self.proj(x)

class FeedForward(nn.Module):
    def __init__(self, dim, mult=4):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(dim, dim * mult),
            nn.GELU(),
            nn.Linear(dim * mult, dim)
        )

    def forward(self, x):
        return self.net(x)

dim = 256
depth = 4

# Create layer pairs (attention, feedforward)
layers = nn.ModuleList([
    nn.ModuleList([SimpleAttention(dim), FeedForward(dim)])
    for _ in range(depth)
])

# Reversible sequence for memory efficiency
reversible_model = ReversibleSequence(
    blocks=layers,
    args_route={},            # argument routing for layers
    layer_dropout=0.0         # optional layer dropout during training
)

# Standard sequential for comparison
sequential_model = SequentialSequence(
    layers=layers,
    args_route={},
    layer_dropout=0.0
)

# Forward pass
x = torch.randn(2, 128, dim)

out_reversible = reversible_model(x)
out_sequential = sequential_model(x)

print(f"Reversible output: {out_reversible.shape}")
print(f"Sequential output: {out_sequential.shape}")
```

--------------------------------

### AutoregressiveWrapper for Text Generation

Source: https://context7.com/lucidrains/h-transformer-1d/llms.txt

Wraps a causal HTransformer1D model to enable autoregressive text generation. Handles training with cross-entropy loss and provides a 'generate' method for sampling. Ensure to set ignore_index and pad_value appropriately.

```python
import torch
from h_transformer_1d import HTransformer1D
from h_transformer_1d.autoregressive_wrapper import AutoregressiveWrapper

# Create base causal model
base_model = HTransformer1D(
    num_tokens=256,
    dim=512,
    max_seq_len=4096,
    depth=8,
    heads=8,
    causal=True,
    reversible=True
)

# Wrap with autoregressive capabilities
model = AutoregressiveWrapper(
    net=base_model,
    ignore_index=-100,       # index to ignore in loss calculation
    pad_value=0              # padding token value
)

# Training: compute cross-entropy loss
# Input is shifted internally: predicts x[1:] from x[:-1]
training_sequence = torch.randint(0, 256, (4, 1024))  # batch of 4
loss = model(training_sequence)
print(f"Training loss: {loss.item():.4f}")

# Backpropagation
loss.backward()

# Generation: sample new tokens autoregressively
model.eval()
start_tokens = torch.randint(0, 256, (1, 64))  # prime sequence

generated = model.generate(
    start_tokens=start_tokens,
    seq_len=256,              # number of tokens to generate
    temperature=1.0,          # sampling temperature (higher = more random)
    filter_thres=0.9,         # top-k filtering threshold
    eos_token=None            # optional early stopping token
)

print(f"Start tokens shape: {start_tokens.shape}")
print(f"Generated tokens shape: {generated.shape}")  # (1, 256)
```

--------------------------------

### Initialize HTransformer1D (Non-Causal Encoder)

Source: https://context7.com/lucidrains/h-transformer-1d/llms.txt

Use this for non-causal (bidirectional) encoder models for classification or encoding tasks. Ensure max_seq_len is divisible by block_size. The model automatically handles padding to powers of 2.

```python
import torch
from h_transformer_1d import HTransformer1D

# Non-causal (bidirectional) encoder model for classification/encoding tasks
encoder_model = HTransformer1D(
    num_tokens=256,          # vocabulary size
    dim=512,                 # model dimension
    depth=12,                # number of transformer layers
    causal=False,            # bidirectional attention
    max_seq_len=8192,        # maximum sequence length (must be divisible by block_size)
    heads=8,                 # number of attention heads
    dim_head=64,             # dimension per attention head
    block_size=128,          # hierarchical block size (Nr in paper)
    reversible=True,         # use reversibility for memory efficiency
    shift_tokens=True        # shift feature space for faster convergence
)

# Process variable-length input with masking
x = torch.randint(0, 256, (1, 8000))       # batch of 1, sequence length 8000
mask = torch.ones((1, 8000)).bool()         # attention mask

# Forward pass - automatically pads to power of 2, applies hierarchical attention
logits = encoder_model(x, mask=mask)        # output: (1, 8000, 256)
print(f"Input shape: {x.shape}, Output shape: {logits.shape}")
```

--------------------------------

### Custom Dataset for Text Data

Source: https://context7.com/lucidrains/h-transformer-1d/llms.txt

Defines a custom PyTorch Dataset for text data, suitable for training sequence models. It samples random sequences of a specified length from the provided data.

```python
class TextSamplerDataset(Dataset):
    def __init__(self, data, seq_len):
        super().__init__()
        self.data = data
        self.seq_len = seq_len

    def __getitem__(self, index):
        rand_start = torch.randint(0, self.data.size(0) - self.seq_len, (1,))
        full_seq = self.data[rand_start: rand_start + self.seq_len + 1].long()
        return full_seq.cuda()

    def __len__(self):
        return self.data.size(0) // self.seq_len
```

--------------------------------

### Cite H-Transformer-1D

Source: https://github.com/lucidrains/h-transformer-1d/blob/main/README.md

BibTeX citations for the H-Transformer-1D paper and related software.

```bibtex
@misc{zhu2021htransformer1d,
    title   = {H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences}, 
    author  = {Zhenhai Zhu and Radu Soricut},
    year    = {2021},
    eprint  = {2107.11906},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG}
}
```

```bibtex
@software{peng_bo_2021_5196578,
    author       = {PENG Bo},
    title        = {BlinkDL/RWKV-LM: 0.01},
    month        = {aug},
    year         = {2021},
    publisher    = {Zenodo},
    version      = {0.01},
    doi          = {10.5281/zenodo.5196578},
    url          = {https://doi.org/10.5281/zenodo.5196578}
}
```

--------------------------------

### Decode Tokens to Text

Source: https://context7.com/lucidrains/h-transformer-1d/llms.txt

Helper function to decode a tensor of token IDs back into a human-readable string. It converts token IDs to their corresponding ASCII characters, ensuring printable characters.

```python
def decode_tokens(tokens):
    return ''.join([chr(max(32, t)) for t in tokens.tolist()])
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.