### Install Example Dependencies

Source: https://github.com/lucidrains/h-net-dynamic-chunking/blob/main/README.md

Install the necessary dependencies for running examples, including optional dependencies for the 'examples' package.

```shell
$ pip install '.[examples]'
```

--------------------------------

### Run Training Example

Source: https://github.com/lucidrains/h-net-dynamic-chunking/blob/main/README.md

Execute the training script for the Enwik8 example with 2 hierarchies. This command initiates the training process using the provided example configuration.

```shell
$ python train.py
```

--------------------------------

### Install h-net-dynamic-chunking

Source: https://github.com/lucidrains/h-net-dynamic-chunking/blob/main/README.md

Install the library using pip. This command is used to add the package to your Python environment.

```shell
$ pip install h-net-dynamic-chunking
```

--------------------------------

### Initialize and Use DynamicSequenceChunker

Source: https://context7.com/lucidrains/h-net-dynamic-chunking/llms.txt

Initialize the DynamicSequenceChunker module for downsampling and upsampling token sequences. Use the forward pass to get downsampled tokens, an upsample function, and an auxiliary loss. The upsample function reconstructs the original sequence length. Auxiliary loss should be included in the training objective. Set `return_only_chunk_lens=True` to get only chunk lengths.

```python
import torch
from h_net_dynamic_chunking import DynamicSequenceChunker

# Initialize the dynamic sequence chunker
# dim: input/output dimension of tokens
# boundary_threshold: threshold for boundary detection (default 0.5)
# target_avg_token_length: target average chunk length for auxiliary loss
# ratio_loss_weight: weight for the auxiliary ratio loss
downsampler = DynamicSequenceChunker(
    dim=512,
    boundary_threshold=0.5,
    target_avg_token_length=6.0,
    ratio_loss_weight=3e-2,
    handle_residual_proj=False,  # Set True to handle residual projections automatically
)

# Input: batch of token sequences
# Shape: (batch_size, sequence_length, dimension)
tokens = torch.randn(3, 1024, 512).requires_grad_()

# Forward pass returns a namedtuple with:
# - downsampled: variable-length downsampled tokens (padded)
# - upsample_fn: function to reconstruct original sequence length
# - weighted_aux_ratio_loss: auxiliary loss for training
downsampled, upsample_fn, aux_loss = downsampler(tokens)

# Upsample back to original sequence length
# The upsample function restores the sequence to original shape
upsampled = upsample_fn(downsampled)

# Verify shape preservation
assert upsampled.shape == tokens.shape  # (3, 1024, 512)

# Include auxiliary loss in training objective
total_loss = your_main_loss + aux_loss
total_loss.backward()

# Get only chunk lengths (useful for inference/tokenization)
chunk_lens = downsampler(tokens, return_only_chunk_lens=True)
```

--------------------------------

### Initialize Nested HNet Architecture

Source: https://context7.com/lucidrains/h-net-dynamic-chunking/llms.txt

Construct a hierarchical network with nested HNet layers and identity encoders/decoders.

```python
# Dimensions: 512 (outer) -> 1024 (middle) -> 2048 (innermost)
net = HNet(
    encoder=nn.Identity(),      # Outer encoder (process before downsampling)
    network=HNet(               # Middle hierarchy
        encoder=nn.Identity(),  # Middle encoder
        network=HNet(           # Innermost hierarchy
            encoder=nn.Identity(),
            network=nn.Identity(),  # Innermost network
            decoder=nn.Identity(),
            dim=2048
        ),
        decoder=nn.Identity(),  # Middle decoder
        dim=1024,
        dim_inner=2048
    ),
    decoder=nn.Identity(),      # Outer decoder (process after upsampling)
    dim=512,
    dim_inner=1024,
)

# Input tokens
tokens = torch.randn(1, 1024, 512)

# Forward pass returns output and accumulated auxiliary loss
output, aux_loss = net(tokens)
# output shape: (1, 1024, 512)
# aux_loss shape: (1,) - accumulated from all hierarchy levels

# Get intermediates for debugging/analysis
output, aux_loss, intermediates = net(tokens, return_intermediates=True)
# intermediates is a tuple of Intermediates namedtuples from each level
```

--------------------------------

### Basic Dynamic Sequence Chunking

Source: https://github.com/lucidrains/h-net-dynamic-chunking/blob/main/README.md

Initialize and use the DynamicSequenceChunker for downsampling and upsampling sequences. Ensure the input tensor requires gradients for backpropagation.

```python
import torch
from h_net_dynamic_chunking import DynamicSequenceChunker

downsampler = DynamicSequenceChunker(512)

tokens = torch.randn(3, 1024, 512).requires_grad_()

downsampled, upsample_fn, *_ = downsampler(tokens)

assert upsample_fn(downsampled).shape == tokens.shape
```

--------------------------------

### Implement MultiHeadDynamicSequenceChunker

Source: https://context7.com/lucidrains/h-net-dynamic-chunking/llms.txt

Configure a multi-head chunker to learn multiple tokenization patterns simultaneously, merging heads into the batch dimension.

```python
import torch
from h_net_dynamic_chunking import MultiHeadDynamicSequenceChunker

# Create multi-head chunker with 4 heads
# Each head learns independent boundary detection patterns
downsampler = MultiHeadDynamicSequenceChunker(
    dim=512,
    heads=4,
    heads_merged_with_batch=True,  # Merge heads with batch dimension
    concat_heads=True,              # Concatenate and project heads (vs sum pooling)
    boundary_threshold=0.5,
    target_avg_token_length=6.0,
)

# Input tokens
tokens = torch.randn(3, 1024, 512).requires_grad_()

# Forward pass - heads are merged with batch
downsampled, upsample_fn, aux_loss = downsampler(tokens)
# downsampled shape: (heads * batch, variable_seq_len, dim)

# Upsample reconstructs original shape with head combination
upsampled = upsample_fn(downsampled)
assert upsampled.shape == tokens.shape  # (3, 1024, 512)
```

--------------------------------

### 3-Layer Hierarchical Network

Source: https://github.com/lucidrains/h-net-dynamic-chunking/blob/main/README.md

Demonstrates creating a 3-layer hierarchical network using DynamicSequenceChunker. Each layer downsamples and upsamples the sequence, with auxiliary losses computed at each step. The final assertion checks if the reconstituted sequence matches the original shape.

```python
import torch
from h_net_dynamic_chunking import DynamicSequenceChunker

downsampler1 = DynamicSequenceChunker(512)
downsampler2 = DynamicSequenceChunker(512)
downsampler3 = DynamicSequenceChunker(512)

tokens = torch.randn(3, 1024, 512).requires_grad_()

downsampled1, upsample_fn1, aux_loss1 = downsampler1(tokens)

# hierarchical network 1 ...

downsampled2, upsample_fn2, aux_loss2 = downsampler2(downsampled1)

# hierarchical network 2 ...

downsampled3, upsample_fn3, aux_loss3 = downsampler3(downsampled2)

# inner most network

# reconstituting

assert upsample_fn1(upsample_fn2(upsample_fn3(downsampled3))).shape == tokens.shape
```

--------------------------------

### Implement HNet Language Model

Source: https://context7.com/lucidrains/h-net-dynamic-chunking/llms.txt

Construct a hierarchical language model using HNet, local attention transformers, and dynamic chunking for character-level prediction.

```python
import torch
from torch import nn
from torch.nn import Module
import torch.nn.functional as F
from torch.optim import Adam
from einops import rearrange

from h_net_dynamic_chunking import HNet
from local_attention import LocalTransformer

class HNetLanguageModel(Module):
    def __init__(
        self,
        num_tokens=256,      # Vocabulary size (e.g., 256 for byte-level)
        dim=256,             # Outer dimension
        dim_inner=512,       # Inner hierarchy dimension
        enc_depth=3,         # Encoder depth
        depth=2,             # Inner network depth
        dec_depth=3,         # Decoder depth
        max_seq_len=512,
        window_size=32,
    ):
        super().__init__()
        self.token_emb = nn.Embedding(num_tokens, dim)
        self.pos_emb = nn.Embedding(max_seq_len, dim)

        # Encoder: Local attention transformer
        encoder = LocalTransformer(
            num_tokens=None,
            dim=dim,
            depth=enc_depth,
            local_attn_window_size=window_size,
        )

        # Inner network: Any sequence model
        inner_network = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(d_model=dim_inner, nhead=8),
            num_layers=depth
        )

        # Decoder: Local attention transformer
        decoder = LocalTransformer(
            num_tokens=None,
            dim=dim,
            depth=dec_depth,
            local_attn_window_size=window_size,
        )

        # Wrap in HNet for automatic chunking
        self.hnet = HNet(
            encoder,
            inner_network,
            decoder,
            dim=dim,
            dim_inner=dim_inner
        )

        self.to_logits = nn.Sequential(
            nn.RMSNorm(dim),
            nn.Linear(dim, num_tokens, bias=False)
        )

    def forward(self, x, return_loss=False):
        if return_loss:
            x, target = x[:, :-1], x[:, 1:]

        seq_len, device = x.shape[-1], x.device

        # Embed tokens with positional encoding
        tokens = self.token_emb(x)
        pos_emb = self.pos_emb(torch.arange(seq_len, device=device))
        tokens = tokens + pos_emb

        # HNet forward: automatic chunking + processing + reconstruction
        embed, aux_loss = self.hnet(tokens)

        logits = self.to_logits(embed)

        if not return_loss:
            return logits

        ar_loss = F.cross_entropy(
            rearrange(logits, 'b n l -> b l n'),
            target
        )

        return ar_loss + aux_loss, (ar_loss, aux_loss)

# Initialize model
model = HNetLanguageModel(
    num_tokens=256,
    dim=256,
    dim_inner=512,
    enc_depth=3,
    depth=2,
    dec_depth=3,
).cuda()

# Training loop example
optimizer = Adam(model.parameters(), lr=1e-4)

# Assume data is byte-level text: shape (batch, seq_len)
data = torch.randint(0, 256, (4, 513)).cuda()

loss, (ar_loss, aux_loss) = model(data, return_loss=True)
loss.backward()
optimizer.step()
optimizer.zero_grad()

print(f"AR Loss: {ar_loss.item():.4f}, Aux Loss: {aux_loss.item():.4f}")
```

--------------------------------

### Multi-Layer Hierarchical Chunking

Source: https://context7.com/lucidrains/h-net-dynamic-chunking/llms.txt

Stack multiple DynamicSequenceChunker modules to create a multi-layer hierarchical system. Each layer independently learns boundary patterns and progressively compresses the sequence. Upsample functions are called in reverse order to reconstruct the original sequence length. Combine auxiliary losses from each layer for training.

```python
import torch
from h_net_dynamic_chunking import DynamicSequenceChunker

# Create a 3-layer hierarchical chunking system
# Each layer learns independent boundary patterns
downsampler1 = DynamicSequenceChunker(dim=512)
downsampler2 = DynamicSequenceChunker(dim=512)
downsampler3 = DynamicSequenceChunker(dim=512)

# Input tokens
tokens = torch.randn(3, 1024, 512).requires_grad_()

# Layer 1: First level downsampling
downsampled1, upsample_fn1, aux_loss1 = downsampler1(tokens)
# downsampled1 shape: variable, typically smaller than (3, 1024, 512)

# Apply hierarchical network 1 here (e.g., transformer layer)
# processed1 = network1(downsampled1)

# Layer 2: Second level downsampling
downsampled2, upsample_fn2, aux_loss2 = downsampler2(downsampled1)

# Apply hierarchical network 2 here
# processed2 = network2(downsampled2)

# Layer 3: Third level downsampling (innermost)
downsampled3, upsample_fn3, aux_loss3 = downsampler3(downsampled2)

# Process at innermost level
# innermost_output = inner_network(downsampled3)

# Reconstruct by upsampling in reverse order
upsampled3 = upsample_fn3(downsampled3)  # or upsample_fn3(innermost_output)
upsampled2 = upsample_fn2(upsampled3)
upsampled1 = upsample_fn1(upsampled2)

# Verify complete reconstruction
assert upsampled1.shape == tokens.shape  # (3, 1024, 512)

# Combine auxiliary losses for training
total_aux_loss = aux_loss1 + aux_loss2 + aux_loss3
```

--------------------------------

### Configure Hierarchical Autoregressive Loss

Source: https://context7.com/lucidrains/h-net-dynamic-chunking/llms.txt

Enable hierarchical AR loss to provide training signals at intermediate levels, optionally detaching targets to control gradient flow.

```python
import torch
from h_net_dynamic_chunking import MultiHeadDynamicSequenceChunker

# Create chunkers - enable hier_ar_loss on middle layer
downsampler1 = MultiHeadDynamicSequenceChunker(dim=512, heads=2)
downsampler2 = MultiHeadDynamicSequenceChunker(
    dim=512,
    heads=2,
    add_hier_ar_loss=True,      # Enable hierarchical AR loss
    detach_hier_target=True,    # Detach targets to prevent gradient flow
)
downsampler3 = MultiHeadDynamicSequenceChunker(dim=512, heads=2)

# Input tokens
tokens = torch.randn(3, 1024, 512).requires_grad_()

# Forward through all layers
downsampled1, upsample_fn1, aux_loss1 = downsampler1(tokens)
downsampled2, upsample_fn2, aux_loss2 = downsampler2(downsampled1)
downsampled3, upsample_fn3, aux_loss3 = downsampler3(downsampled2)

# Process at innermost level (your network here)
# processed = inner_network(downsampled3)

# Upsample - layer with hier_ar_loss returns tuple (output, ar_loss)
upsampled3 = upsample_fn3(downsampled3)

# upsample_fn2 returns (upsampled, hierarchical_ar_loss)
upsampled2, hier_ar_loss = upsample_fn2(upsampled3)

upsampled1 = upsample_fn1(upsampled2)
```

--------------------------------

### HNet Wrapper for Hierarchical Networks

Source: https://github.com/lucidrains/h-net-dynamic-chunking/blob/main/README.md

Utilizes the HNet wrapper to construct a hierarchical network with specified dimensions. The wrapper simplifies the creation of nested HNet layers. The output includes the processed tokens and auxiliary loss.

```python
import torch
from torch import nn
from h_net_dynamic_chunking.h_net import HNet

# 3 hierarchies, from 512 -> 1024 -> 2048 inner

net = HNet(
    nn.Identity(),
    HNet(
        nn.Identity(),
        HNet(
            nn.Identity(),
            nn.Identity(),
            nn.Identity(),
            dim = 2048
        ),
        nn.Identity(),
        dim = 1024,
        dim_inner = 2048
    ),
    nn.Identity(),
    dim = 512,
    dim_inner = 1024,
)

tokens = torch.randn(1, 1024, 512)

out, aux_loss = net(tokens) # (1, 1024, 512), (1,)
```

--------------------------------

### HNet Wrapper for Hierarchical Architectures

Source: https://context7.com/lucidrains/h-net-dynamic-chunking/llms.txt

Utilize the HNet wrapper for a convenience solution that combines encoder, decoder, and inner network modules with automatic dynamic chunking. HNet manages the full hierarchical architecture, including residual projections and dimension transformations between hierarchy levels.

```python
import torch
from torch import nn
from h_net_dynamic_chunking.h_net import HNet

# Create a 3-level nested HNet hierarchy

```

--------------------------------

### Access Intermediates for Debugging

Source: https://context7.com/lucidrains/h-net-dynamic-chunking/llms.txt

Retrieve internal state like boundary masks and chunk lengths from the chunking process for visualization or custom processing.

```python
import torch
from h_net_dynamic_chunking import DynamicSequenceChunker
from h_net_dynamic_chunking.h_net_dynamic_chunking import Intermediates

# Create chunker
downsampler = DynamicSequenceChunker(dim=512)

# Get outputs with intermediates
tokens = torch.randn(2, 256, 512).requires_grad_()

outputs, intermediates = downsampler(tokens, return_intermediates=True)

# Access intermediate values
print(f"Mask shape: {intermediates.mask.shape}")
print(f"Boundary probs shape: {intermediates.probs.shape}")
print(f"Chunk lengths: {intermediates.chunk_lens}")
print(f"Boundary mask: {intermediates.boundary_mask}")
print(f"Number of chunks per batch: {intermediates.boundary_mask.sum(dim=-1)}")

# Manually call upsample with intermediates
upsampled = downsampler.upsample(
    outputs.downsampled,
    intermediates,
    apply_scale=True  # Apply confidence scaling for gradients
)
assert upsampled.shape == tokens.shape
```

--------------------------------

### Include hierarchical AR loss

Source: https://context7.com/lucidrains/h-net-dynamic-chunking/llms.txt

Calculate the total loss by summing the main autoregressive loss, auxiliary losses, and hierarchical AR loss before backpropagation.

```python
total_loss = main_loss + aux_loss1 + aux_loss2 + aux_loss3 + hier_ar_loss
total_loss.backward()
```

--------------------------------

### Process Multi-Head Chunks Separately

Source: https://context7.com/lucidrains/h-net-dynamic-chunking/llms.txt

Disable batch merging to process individual attention heads with distinct network architectures like GRU or LSTM.

```python
import torch
from torch.nn import GRU, LSTM
from h_net_dynamic_chunking import MultiHeadDynamicSequenceChunker

# Create multi-head chunker with heads separated from batch
downsampler = MultiHeadDynamicSequenceChunker(
    dim=512,
    heads=2,
    heads_merged_with_batch=False,  # Keep heads separate for individual processing
)

# Input tokens
tokens = torch.randn(3, 1024, 512).requires_grad_()

# Forward pass returns downsampled with shape (heads, batch, seq, dim)
downsampled, upsample_fn, aux_loss = downsampler(tokens)

# Create different networks for each head
network1 = GRU(512, 512, batch_first=True)
network2 = LSTM(512, 512, batch_first=True)

# Process each head with its dedicated network
first_head, second_head = downsampled  # Unpack heads

out1, _ = network1(first_head)   # GRU processes first head
out2, _ = network2(second_head)  # LSTM processes second head

# Reconstruct by passing list of processed heads
upsampled = upsample_fn([out1, out2])
assert upsampled.shape == tokens.shape  # (3, 1024, 512)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.