### minLM Language Model Autoregressive Generation Setup

Source: https://context7.com/lucidrains/mingru-pytorch/llms.txt

Sets the minLM model to evaluation mode and initializes a prompt for autoregressive generation. This is the setup before starting the generation loop.

```python
# Autoregressive generation with hidden state caching
model.eval()
prompt = torch.randint(0, 256, (1, 32))  # Initial prompt
```

--------------------------------

### Initialize and Use minLSTM for Parallel Processing

Source: https://context7.com/lucidrains/mingru-pytorch/llms.txt

Demonstrates initializing minLSTM and processing an entire sequence in parallel, suitable for training. Includes an example with an expansion factor.

```python
import torch
from minGRU_pytorch.minLSTM import minLSTM

# Initialize minLSTM
min_lstm = minLSTM(dim=512)

# Parallel processing
x = torch.randn(2, 1024, 512)  # (batch, seq_len, dim)
out = min_lstm(x)
print(out.shape)  # torch.Size([2, 1024, 512])

# With expansion factor
min_lstm_expanded = minLSTM(dim=512, expansion_factor=1.5)
x = torch.randn(1, 512, 512)
out = min_lstm_expanded(x)
print(out.shape)  # torch.Size([1, 512, 512])
```

--------------------------------

### Install minGRU-pytorch

Source: https://github.com/lucidrains/mingru-pytorch/blob/main/README.md

Install the package using pip.

```bash
$ pip install minGRU-pytorch
```

--------------------------------

### Create and Use minLM Language Model for Forward Pass

Source: https://context7.com/lucidrains/mingru-pytorch/llms.txt

Shows how to initialize a minLM language model with various configurations, including vocabulary size, dimensions, depth, and enabling/disabling LSTM, convolutions, and dropout. Demonstrates a forward pass to get logits.

```python
import torch
from minGRU_pytorch.minLM import minLM

# Create a character-level language model
model = minLM(
    num_tokens=256,          # Vocabulary size (256 for byte-level)
    dim=512,                 # Model dimension
    depth=6,                 # Number of layers
    ff_mult=4,               # Feedforward expansion multiplier
    expansion=1.5,           # RNN hidden dimension expansion
    conv_kernel_size=3,      # Causal conv kernel size (if enabled)
    use_lstm=False,          # Use minLSTM instead of minGRU
    enable_conv=False,       # Enable causal convolutions
    dropout=0.1              # Dropout rate
)

# Forward pass - get logits
x = torch.randint(0, 256, (2, 512))  # (batch, seq_len) token ids
logits = model(x)
print(logits.shape)  # torch.Size([2, 512, 256])
```

--------------------------------

### Initialize and Train minLM Model on enwik8

Source: https://context7.com/lucidrains/mingru-pytorch/llms.txt

Sets up the minLM model, optimizer, and data loaders for training on the enwik8 dataset. Includes a simplified training loop with gradient accumulation and clipping.

```python
# Create model
model = minLM(
    num_tokens=256,
    dim=512,
    depth=6,
    use_lstm=False  # Set True for minLSTM variant
)

# Setup training
train_dataset = TextSamplerDataset(data_train, SEQ_LEN)
train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE)
optim = Adam(model.parameters(), lr=LEARNING_RATE)

# Training loop (simplified)
model.train()
for batch_idx, data in enumerate(train_loader):
    loss = model(data, return_loss=True)
    (loss / GRAD_ACCUM_EVERY).backward()

    if (batch_idx + 1) % GRAD_ACCUM_EVERY == 0:
        torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)
        optim.step()
        optim.zero_grad()
        print(f"Batch {batch_idx}, Loss: {loss.item():.4f}")

    if batch_idx >= 100:  # Demo: stop after 100 batches
        break
```

--------------------------------

### Initialize and Use minGRU for Parallel Processing

Source: https://context7.com/lucidrains/mingru-pytorch/llms.txt

Demonstrates initializing minGRU and processing an entire sequence at once, suitable for training. Supports an expansion factor for larger hidden dimensions.

```python
import torch
from minGRU_pytorch import minGRU

# Initialize minGRU with input dimension
min_gru = minGRU(dim=512)

# Parallel processing - process entire sequence at once (training mode)
x = torch.randn(2, 1024, 512)  # (batch, seq_len, dim)
out = min_gru(x)
print(out.shape)  # torch.Size([2, 1024, 512])

# With expansion factor for larger hidden dimension
min_gru_expanded = minGRU(dim=512, expansion_factor=1.5)
x = torch.randn(1, 2048, 512)
out = min_gru_expanded(x)
print(out.shape)  # torch.Size([1, 2048, 512])
```

--------------------------------

### Run Training Test

Source: https://github.com/lucidrains/mingru-pytorch/blob/main/README.md

Execute the training script for the enwik8 dataset.

```bash
$ python train.py
```

--------------------------------

### Initialize minGRUAttnHybrid Module

Source: https://context7.com/lucidrains/mingru-pytorch/llms.txt

Creates an instance of the minGRUAttnHybrid module. This module combines minGRU with multi-head attention. Set `learned_mix=True` to learn per-token mixing weights.

```python
import torch
from minGRU_pytorch.hybrid import minGRUAttnHybrid

# Create hybrid module
hybrid = minGRUAttnHybrid(
    dim=512,           # Input/output dimension
    dim_head=64,       # Dimension per attention head
    heads=8,           # Number of attention heads
    learned_mix=True   # Learn per-token mixing weights
)

# Forward pass
x = torch.randn(2, 256, 512)  # (batch, seq_len, dim)
out = hybrid(x)
print(out.shape)  # torch.Size([2, 256, 512])
```

--------------------------------

### Initialize hybridLM Language Model

Source: https://context7.com/lucidrains/mingru-pytorch/llms.txt

Instantiates the hybridLM, a language model using a hybrid minGRU-attention architecture. Supports vocabulary sizes, model dimensions, depth, and attention configurations.

```python
import torch
from minGRU_pytorch.hybridLM import hybridLM

# Create hybrid language model
model = hybridLM(
    num_tokens=256,    # Vocabulary size
    dim=512,           # Model dimension
    depth=6,           # Number of layers
    ff_mult=4,         # Feedforward expansion
    heads=8,           # Attention heads
    dim_head=64        # Dimension per head
)

# Forward pass for logits
x = torch.randint(0, 256, (2, 512))  # (batch, seq_len)
logits = model(x)
print(logits.shape)  # torch.Size([2, 512, 256])

# Training with loss computation
x = torch.randint(0, 256, (2, 513))
loss = model(x, return_loss=True)
print(f"Training loss: {loss.item():.4f}")

# Note: hybridLM does not support hidden state caching
# due to the attention component requiring full context
```

--------------------------------

### minLM Language Model Training Loss Calculation

Source: https://context7.com/lucidrains/mingru-pytorch/llms.txt

Demonstrates how to compute the training loss for the minLM language model by providing an extra token for labels and setting `return_loss=True`. Prints the calculated training loss.

```python
# Training mode - compute cross-entropy loss directly
x = torch.randint(0, 256, (2, 513))  # Need extra token for labels
loss = model(x, return_loss=True)
print(f"Training loss: {loss.item():.4f}")
```

--------------------------------

### Basic minGRU Usage

Source: https://github.com/lucidrains/mingru-pytorch/blob/main/README.md

Initialize the minGRU model and process a tensor input.

```python
import torch
from minGRU_pytorch import minGRU

min_gru = minGRU(512)

x = torch.randn(2, 1024, 512)

out = min_gru(x)

assert x.shape == out.shape
```

--------------------------------

### Load and Prepare enwik8 Dataset

Source: https://context7.com/lucidrains/mingru-pytorch/llms.txt

Loads the enwik8 dataset and prepares it into PyTorch tensors for training. Defines a custom Dataset class for sampling sequences.

```python
import gzip
import numpy as np
import torch
from torch.optim import Adam
from torch.utils.data import DataLoader, Dataset
from minGRU_pytorch.minLM import minLM

# Hyperparameters
BATCH_SIZE = 4
GRAD_ACCUM_EVERY = 4
LEARNING_RATE = 1e-4
SEQ_LEN = 512

# Load enwik8 data
with gzip.open("./data/enwik8.gz") as file:
    data = np.frombuffer(file.read(int(95e6)), dtype=np.uint8).copy()
    np_train, np_valid = np.split(data, [int(90e6)])
    data_train = torch.from_numpy(np_train)
    data_val = torch.from_numpy(np_valid)

# Dataset class
class TextSamplerDataset(Dataset):
    def __init__(self, data, seq_len):
        self.data = data
        self.seq_len = seq_len

    def __len__(self):
        return self.data.size(0) // self.seq_len

    def __getitem__(self, index):
        rand_start = torch.randint(0, self.data.size(0) - self.seq_len, (1,))
        full_seq = self.data[rand_start : rand_start + self.seq_len + 1].long()
        return full_seq
```

--------------------------------

### minLSTM Sequential Processing with State Caching

Source: https://context7.com/lucidrains/mingru-pytorch/llms.txt

Illustrates sequential processing with minLSTM using hidden state caching for inference. Outputs the shape of the final sequential output.

```python
# Sequential processing with state caching
prev_hidden = None
for token in x.unbind(dim=1):
    out, prev_hidden = min_lstm(
        token[:, None, :],
        prev_hidden,
        return_next_prev_hidden=True
    )
print(f"Final output shape: {out.shape}")  # torch.Size([1, 1, 512])
```

--------------------------------

### minGRU Sanity Check

Source: https://github.com/lucidrains/mingru-pytorch/blob/main/README.md

Verify that parallel and sequential processing modes produce equivalent outputs.

```python
import torch
from minGRU_pytorch import minGRU

min_gru = minGRU(dim = 512, expansion_factor = 1.5)

x = torch.randn(1, 2048, 512)

# parallel

parallel_out = min_gru(x)[:, -1:]

# sequential

prev_hidden = None
for token in x.unbind(dim = 1):
    sequential_out, prev_hidden = min_gru(token[:, None, :], prev_hidden, return_next_prev_hidden = True)

assert torch.allclose(parallel_out, sequential_out, atol = 1e-4)
```

--------------------------------

### minGRU Sequential Processing with State Caching

Source: https://context7.com/lucidrains/mingru-pytorch/llms.txt

Shows how to use minGRU for sequential processing token-by-token with hidden state caching, optimized for autoregressive inference. Verifies output consistency with parallel processing.

```python
# Sequential processing with hidden state caching (inference mode)
min_gru = minGRU(dim=512, expansion_factor=1.5)
x = torch.randn(1, 2048, 512)

# First, get parallel output for comparison
parallel_out = min_gru(x)[:, -1:]

# Sequential token-by-token processing
prev_hidden = None
for token in x.unbind(dim=1):
    sequential_out, prev_hidden = min_gru(
        token[:, None, :],  # Single token: (batch, 1, dim)
        prev_hidden,
        return_next_prev_hidden=True
    )

# Verify outputs match
assert torch.allclose(parallel_out, sequential_out, atol=1e-4)
print("Parallel and sequential outputs match!")
```

--------------------------------

### Generate Tokens One at a Time with Caching

Source: https://context7.com/lucidrains/mingru-pytorch/llms.txt

Generates new tokens sequentially using a model with caching of previous hidden states. Suitable for iterative text generation.

```python
prev_hiddens = None
generated = prompt.clone()

for _ in range(100):  # Generate 100 new tokens
    logits, prev_hiddens = model(
        generated,
        return_prev_hiddens=True,
        prev_hiddens=prev_hiddens
    )
    # Sample next token
    next_token = logits[:, -1:].argmax(dim=-1)
    generated = torch.cat([generated, next_token], dim=1)

print(f"Generated sequence length: {generated.shape[1]}")
```

--------------------------------

### Text Generation Utilities

Source: https://context7.com/lucidrains/mingru-pytorch/llms.txt

Provides functions for log, Gumbel noise sampling, top-k filtering, and autoregressive text generation with hidden state caching. Use for controlled text generation with temperature and filtering.

```python
import math
import torch
from minGRU_pytorch.minLM import minLM

def log(t, eps=1e-20):
    return torch.log(t.clamp(min=eps))

def gumbel_noise(t):
    noise = torch.zeros_like(t).uniform_(0, 1)
    return -log(-log(noise))

def gumbel_sample(t, temperature=1., dim=-1, keepdim=True):
    return ((t / max(temperature, 1e-10)) + gumbel_noise(t)).argmax(dim=dim, keepdim=keepdim)

def top_k(logits, thres=0.9):
    k = math.ceil((1 - thres) * logits.shape[-1])
    val, ind = torch.topk(logits, k)
    probs = torch.full_like(logits, float('-inf'))
    probs.scatter_(-1, ind, val)
    return probs

def generate(model, prompt, seq_len, temperature=1., filter_thres=0.9):
    """Generate text autoregressively with caching."""
    prompt_len = prompt.shape[-1]
    out = prompt.clone()
    prev_hiddens = None

    for _ in range(max(0, seq_len - prompt_len)):
        logits, prev_hiddens = model(
            out,
            return_prev_hiddens=True,
            prev_hiddens=prev_hiddens if model.can_cache else None
        )
        logits = top_k(logits[:, -1], thres=filter_thres)
        sample = gumbel_sample(logits, temperature=temperature, dim=-1)
        out = torch.cat((out, sample), dim=-1)

    return out[..., prompt_len:]
```

```python
# Usage example
model = minLM(num_tokens=256, dim=512, depth=6)
model.eval()

# Generate from prompt
prompt = torch.randint(0, 256, (1, 64))  # Random prompt
generated = generate(model, prompt, seq_len=256, temperature=0.8)

# Decode to text
def decode_tokens(tokens):
    return "".join(chr(max(32, t)) for t in tokens.tolist())

print(decode_tokens(generated[0]))
```

--------------------------------

### Citations

Source: https://github.com/lucidrains/mingru-pytorch/blob/main/README.md

BibTeX citations for the relevant research papers.

```bibtex
@inproceedings{Feng2024WereRA,
    title   = {Were RNNs All We Needed?},
    author  = {Leo Feng and Frederick Tung and Mohamed Osama Ahmed and Yoshua Bengio and Hossein Hajimirsadegh},
    year    = {2024},
    url     = {https://api.semanticscholar.org/CorpusID:273025630}
}
```

```bibtex
@inproceedings{anonymous2024hymba,
    title   = {Hymba: A Hybrid-head Architecture for Small Language Models},
    author  = {Anonymous},
    booktitle = {Submitted to The Thirteenth International Conference on Learning Representations},
    year    = {2024},
    url     = {https://openreview.net/forum?id=A1ztozypga},
    note    = {under review}
}
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.