### Install Light Recurrent Unit Pytorch

Source: https://github.com/lucidrains/light-recurrent-unit-pytorch/blob/main/README.md

Install the package via pip.

```bash
$ pip install light-recurrent-unit-pytorch
```

--------------------------------

### Train LRU Language Model

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

Example of a single training step for the LRU-based language model. It involves generating input tokens, calculating the loss, and performing backpropagation.

```python
# Initialize model
model = LRULM(
    num_tokens=256,    # byte-level vocabulary
    dim=384,
    depth=6
).cuda()

# Training step
x = torch.randint(0, 256, (4, 128)).cuda()  # (batch, seq_len)
loss = model(x, return_loss=True)
loss.backward()
```

--------------------------------

### Generate Text with Sampling and Decode

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

Example of using the `base_decoding` function to generate text with specified temperature and top-k filtering, followed by decoding the generated tokens into a human-readable string.

```python
import math
import torch
from torch import Tensor

# Assuming LRULM model and helper functions (log, gumbel_noise, gumbel_sample, top_k) are defined above

# Usage example
model.eval()
prompt = torch.randint(0, 256, (1, 64)).cuda()

# Generate 128 new tokens with temperature sampling
generated = base_decoding(
    model,
    prompt,
    seq_len=192,          # total length (prompt + generated)
    temperature=0.8,
    filter_thres=0.9      # top-k filtering threshold
)

# Decode to text
def decode_tokens(tokens):
    return "".join([chr(max(32, t)) for t in tokens])

output_text = decode_tokens(generated[0].cpu().tolist())
print(output_text)
```

--------------------------------

### Initialize and Use LightRecurrentUnitLayer

Source: https://github.com/lucidrains/light-recurrent-unit-pytorch/blob/main/README.md

Use the layer-level module for processing sequences.

```python
import torch
from light_recurrent_unit_pytorch import LightRecurrentUnitLayer

layer = LightRecurrentUnitLayer(256)

x = torch.randn(2, 1024, 256)

out = layer(x) # (2, 1024, 256)

assert out.shape == x.shape
```

--------------------------------

### Initialize and Use Stacked LightRecurrentUnit

Source: https://github.com/lucidrains/light-recurrent-unit-pytorch/blob/main/README.md

Use the stacked module for multi-layer recurrent architectures.

```python
import torch
from light_recurrent_unit_pytorch import LightRecurrentUnit

lru = LightRecurrentUnit(256, depth = 4)

x = torch.randn(2, 1024, 256)

out, layer_hiddens = lru(x) # (2, 1024, 256), List[(2, 256)]

assert out.shape == x.shape
```

--------------------------------

### Instantiate Basic LRU Block

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

Demonstrates how to instantiate a basic LightRecurrentUnitBlock with normalization and residual connections. Use `has_ff_block=False` for a pure LRU block.

```python
import torch
from light_recurrent_unit_pytorch import LightRecurrentUnitBlock

block = LightRecurrentUnitBlock(
    dim=256,
    depth=2,
    learned_init_hidden=True,
    depth_gated_lru=True,         # use GatedLightRecurrentUnit
    has_ff_block=False            # no feedforward block
)

x = torch.randn(2, 512, 256)
out = block(x)                    # (2, 512, 256)
```

--------------------------------

### Initialize and use Stacked LightRecurrentUnit

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

Initialize a stacked LRU architecture with multiple layers for deeper sequence modeling. Supports per-layer projection control and continuation from previous states.

```python
import torch
from light_recurrent_unit_pytorch import LightRecurrentUnit

# Initialize stacked LRU with 4 layers
lru = LightRecurrentUnit(
    dim=256,
    depth=4,
    proj_input=True,              # can be tuple for per-layer control
    learned_init_hidden=False
)

# Process sequence
x = torch.randn(2, 1024, 256)     # (batch, seq_len, dim)

# Returns output and list of final hidden states per layer
out, layer_hiddens = lru(x)       # (2, 1024, 256), List[(2, 256)]

assert out.shape == x.shape
assert len(layer_hiddens) == 4

# With per-layer projection control
lru_custom = LightRecurrentUnit(
    dim=256,
    depth=4,
    proj_input=(True, True, False, False)  # first 2 layers project, last 2 don't
)

# Continue from previous hidden states (for generation/streaming)
out2, new_hiddens = lru(x, hiddens=layer_hiddens)
```

--------------------------------

### Initialize and Use LightRecurrentUnitCell

Source: https://github.com/lucidrains/light-recurrent-unit-pytorch/blob/main/README.md

Use the cell-level module for step-by-step recurrent processing.

```python
import torch
from light_recurrent_unit_pytorch import LightRecurrentUnitCell

cell = LightRecurrentUnitCell(256)

x = torch.randn(2, 256)
hidden = torch.randn(2, 256)

next_hidden = cell(x, hidden) # (2, 256)
```

--------------------------------

### Instantiate LRU Block with Feedforward

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

Shows how to instantiate a LightRecurrentUnitBlock that includes a feedforward block, similar to transformer architectures. The feedforward hidden dimension can be controlled with `ff_expansion_factor`.

```python
import torch
from light_recurrent_unit_pytorch import LightRecurrentUnitBlock

block_with_ff = LightRecurrentUnitBlock(
    dim=256,
    depth=2,
    learned_init_hidden=True,
    depth_gated_lru=True,
    has_ff_block=True,
    ff_expansion_factor=4         # feedforward hidden dim = dim * 4
)

x = torch.randn(2, 512, 256)
out = block_with_ff(x)            # (2, 512, 256)
```

--------------------------------

### Initialize LightRecurrentUnitBlock

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

Initialize a transformer-style block combining LRU with RMSNorm and optional feedforward layers.

```python
import torch
from light_recurrent_unit_pytorch import LightRecurrentUnitBlock
```

--------------------------------

### Run Enwik8 Training Test

Source: https://github.com/lucidrains/light-recurrent-unit-pytorch/blob/main/README.md

Execute the training script for the Enwik8 dataset.

```bash
$ python train.py
```

--------------------------------

### Initialize and use GatedLightRecurrentUnit

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

Initialize an advanced gated LRU variant with residual connections gated by an additional LRU cell. Supports continuation from previous hidden states.

```python
import torch
from light_recurrent_unit_pytorch import GatedLightRecurrentUnit

# Initialize gated LRU
gated_lru = GatedLightRecurrentUnit(
    dim=256,
    depth=3,                      # number of gated blocks
    learned_init_hidden=False,
    num_layers_per_depth=2        # LRU layers within each block
)

# Process sequence
x = torch.randn(2, 512, 256)      # (batch, seq_len, dim)

# Returns output and nested hidden states
out, hiddens = gated_lru(x)       # (2, 512, 256), nested list of hiddens

assert out.shape == x.shape

# Continue from previous hidden states
out2, new_hiddens = gated_lru(x, hiddens=hiddens)
```

--------------------------------

### Initialize and use LightRecurrentUnitLayer

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

Initialize a complete LRU layer for processing entire sequences. It wraps LightRecurrentUnitCell for batch-first sequence processing.

```python
import torch
from light_recurrent_unit_pytorch import LightRecurrentUnitLayer

# Initialize layer
layer = LightRecurrentUnitLayer(
    dim=256,
    dim_hidden=None,
    proj_input=True,
    learned_init_hidden=False
)

# Process sequence (batch_first format)
x = torch.randn(2, 1024, 256)     # (batch, seq_len, dim)

# Forward pass returns output for all timesteps
out = layer(x)                     # (2, 1024, 256)

assert out.shape == x.shape

# With initial hidden state
initial_hidden = torch.randn(2, 256)
out = layer(x, hidden=initial_hidden)  # (2, 1024, 256)
```

--------------------------------

### Generate Text with LRU LM (Greedy Decoding)

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

Demonstrates text generation using the trained LRU language model with greedy decoding. It takes a prompt and generates the next token based on the highest probability.

```python
# Generation (inference)
model.eval()
prompt = torch.randint(0, 256, (1, 64)).cuda()
logits = model(prompt)                       # (1, 64, 256)
next_token = logits[:, -1].argmax(dim=-1)    # greedy decoding
```

--------------------------------

### Initialize and use LightRecurrentUnitCell

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

Initialize the fundamental LRU cell for single timestep processing. Supports optional input projection and learned initial hidden states.

```python
import torch
from light_recurrent_unit_pytorch import LightRecurrentUnitCell

# Initialize cell with dimension 256
cell = LightRecurrentUnitCell(
    dim=256,
    dim_hidden=None,          # defaults to dim if not specified
    proj_input=True,          # project input through linear + tanh
    learned_init_hidden=False # use zeros or learned initial hidden
)

# Single step processing
x = torch.randn(2, 256)           # (batch_size, dim)
hidden = torch.randn(2, 256)      # (batch_size, dim_hidden)

# Compute next hidden state
next_hidden = cell(x, hidden)     # (2, 256)

# Without providing hidden state (uses init_hidden)
next_hidden = cell(x)             # (2, 256)

# With learned initial hidden state
cell_learned = LightRecurrentUnitCell(
    dim=256,
    learned_init_hidden=True
)
next_hidden = cell_learned(x)     # Uses learned initial hidden
```

--------------------------------

### LightRecurrentUnit (Stacked)

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

A stacked architecture with multiple LRU layers for deeper sequence modeling, returning both the output sequence and the final hidden states from each layer.

```APIDOC
## LightRecurrentUnit (Stacked)

### Description
A stacked architecture with multiple LRU layers for deeper sequence modeling, returning both the output sequence and the final hidden states from each layer.

### Parameters
- **dim** (int) - Required - Dimension of the input.
- **depth** (int) - Required - Number of LRU layers.
- **proj_input** (bool/tuple) - Optional - Projection control for layers.

### Request Example
```python
lru = LightRecurrentUnit(dim=256, depth=4)
out, layer_hiddens = lru(x)
```
```

--------------------------------

### Gumbel Sampling for Text Generation

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

Provides helper functions for Gumbel noise generation and sampling. `gumbel_sample` is used with temperature and top-k filtering to produce diverse text outputs.

```python
import math
import torch
from torch import Tensor

def log(t, eps=1e-20):
    return torch.log(t.clamp(min=eps))

def gumbel_noise(t):
    noise = torch.zeros_like(t).uniform_(0, 1)
    return -log(-log(noise))

def gumbel_sample(t, temperature=1., dim=-1, keepdim=True):
    return ((t / max(temperature, 1e-10)) + gumbel_noise(t)).argmax(dim=dim, keepdim=keepdim)
```

--------------------------------

### Build LRU-based Language Model

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

Constructs a character-level language model using LRU blocks. It includes token embeddings, multiple LRU layers, and a final linear layer for logits. The model supports calculating cross-entropy loss during training.

```python
import torch
from torch import nn
import torch.nn.functional as F
from light_recurrent_unit_pytorch import LightRecurrentUnitBlock

class LRULM(nn.Module):
    def __init__(self, num_tokens, dim, depth):
        super().__init__()
        self.token_emb = nn.Embedding(num_tokens, dim)
        self.layers = nn.ModuleList([
            LightRecurrentUnitBlock(dim=dim, has_ff_block=True)
            for _ in range(depth)
        ])
        self.to_logits = nn.Sequential(
            RMSNorm(dim),
            nn.Linear(dim, num_tokens, bias=False)
        )

    def forward(self, x, return_loss=False):
        if return_loss:
            x, labels = x[:, :-1], x[:, 1:]

        x = self.token_emb(x)
        for block in self.layers:
            x = block(x)
        logits = self.to_logits(x)

        if not return_loss:
            return logits
        return F.cross_entropy(logits.transpose(1, 2), labels)
```

--------------------------------

### LightRecurrentUnitLayer

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

A complete layer that processes an entire sequence by iterating over timesteps, wrapping LightRecurrentUnitCell for convenient batch-first sequence processing.

```APIDOC
## LightRecurrentUnitLayer

### Description
A complete layer that processes an entire sequence by iterating over timesteps, wrapping LightRecurrentUnitCell for convenient batch-first sequence processing.

### Parameters
- **dim** (int) - Required - Dimension of the input.
- **dim_hidden** (int) - Optional - Dimension of the hidden state.
- **proj_input** (bool) - Optional - Whether to project input.
- **learned_init_hidden** (bool) - Optional - Whether to use a learned initial hidden state.

### Request Example
```python
layer = LightRecurrentUnitLayer(dim=256)
out = layer(x, hidden=initial_hidden)
```
```

--------------------------------

### LRU Citations

Source: https://github.com/lucidrains/light-recurrent-unit-pytorch/blob/main/README.md

BibTeX citations for the Light Recurrent Unit and related research.

```bibtex
@Article{electronics13163204,
    AUTHOR  = {Ye, Hong and Zhang, Yibing and Liu, Huizhou and Li, Xuannong and Chang, Jiaming and Zheng, Hui},
    TITLE   = {Light Recurrent Unit: Towards an Interpretable Recurrent Neural Network for Modeling Long-Range Dependency},
    JOURNAL = {Electronics},
    VOLUME  = {13},
    YEAR    = {2024},
    NUMBER  = {16},
    ARTICLE-NUMBER = {3204},
    URL     = {https://www.mdpi.com/2079-9292/13/16/3204},
    ISSN    = {2079-9292},
    DOI     = {10.3390/electronics13163204}
}
```

```bibtex
@article{Merrill2024TheIO,
    title   = {The Illusion of State in State-Space Models},
    author  = {William Merrill and Jackson Petty and Ashish Sabharwal},
    journal = {ArXiv},
    year    = {2024},
    volume  = {abs/2404.08819},
    url     = {https://api.semanticscholar.org/CorpusID:269149086}
}
```

--------------------------------

### Top-K Filtering for Logits

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

Implements top-k filtering to restrict sampling to the `k` most likely next tokens, where `k` is determined by the `thres`hold. This helps in controlling the randomness of generated text.

```python
import math
import torch
from torch import Tensor

def top_k(logits, thres=0.9):
    k = math.ceil((1 - thres) * logits.shape[-1])
    val, ind = torch.topk(logits, k)
    probs = torch.full_like(logits, float('-inf'))
    probs.scatter_(-1, ind, val)
    return probs
```

--------------------------------

### GatedLightRecurrentUnit

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

An advanced variant where stacked LRU layers have residual connections gated by an additional LRU cell, enabling deeper networks with better gradient flow.

```APIDOC
## GatedLightRecurrentUnit

### Description
An advanced variant where stacked LRU layers have residual connections gated by an additional LRU cell, enabling deeper networks with better gradient flow.

### Parameters
- **dim** (int) - Required - Dimension of the input.
- **depth** (int) - Required - Number of gated blocks.
- **num_layers_per_depth** (int) - Optional - LRU layers within each block.

### Request Example
```python
gated_lru = GatedLightRecurrentUnit(dim=256, depth=3)
out, hiddens = gated_lru(x)
```
```

--------------------------------

### LightRecurrentUnitCell

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

The fundamental building block that processes a single timestep, computing the next hidden state from the current input and previous hidden state using a learnable forget gate mechanism.

```APIDOC
## LightRecurrentUnitCell

### Description
The fundamental building block that processes a single timestep, computing the next hidden state from the current input and previous hidden state using a learnable forget gate mechanism.

### Parameters
- **dim** (int) - Required - Dimension of the input.
- **dim_hidden** (int) - Optional - Dimension of the hidden state (defaults to dim).
- **proj_input** (bool) - Optional - Whether to project input through linear + tanh.
- **learned_init_hidden** (bool) - Optional - Whether to use a learned initial hidden state.

### Request Example
```python
cell = LightRecurrentUnitCell(dim=256, proj_input=True)
next_hidden = cell(x, hidden)
```
```

--------------------------------

### Base Decoding Function for Text Generation

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

A core function for text generation that iteratively samples tokens using Gumbel sampling with temperature and top-k filtering. It extends a given prompt to a specified sequence length.

```python
import math
import torch
from torch import Tensor

def base_decoding(net, prompt: Tensor, seq_len: int, temperature=1., filter_thres=0.9):
    prompt_seq_len, out = prompt.shape[-1], prompt.clone()
    sample_num_times = max(0, seq_len - prompt_seq_len)

    for _ in range(sample_num_times):
        logits = net(out)
        logits = logits[:, -1]
        logits = top_k(logits, thres=filter_thres)
        sample = gumbel_sample(logits, temperature=temperature, dim=-1)
        out = torch.cat((out, sample), dim=-1)

    return out[..., prompt_seq_len:]
```

--------------------------------

### Define RMSNorm Layer

Source: https://context7.com/lucidrains/light-recurrent-unit-pytorch/llms.txt

Defines a Root Mean Square Normalization (RMSNorm) layer, commonly used in modern neural network architectures. It normalizes inputs and applies a learned scaling factor.

```python
import torch
from torch import nn
import torch.nn.functional as F

class RMSNorm(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.scale = dim ** 0.5
        self.gamma = nn.Parameter(torch.zeros(dim))

    def forward(self, x):
        return F.normalize(x, dim=-1) * self.scale * (self.gamma + 1.)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.