### Setup Docker environment for StripedHyena

Source: https://context7.com/togethercomputer/stripedhyena/llms.txt

Build and run the Docker container with GPU support and verify FlashAttention installation.

```bash
# Build the Docker image
docker build --tag sh:test .

# Run container with GPU support
docker run -it --gpus all \
    --network="host" \
    --shm-size 900G \
    -v=/path/to/stripedhyena:/mnt:rw \
    --rm sh:test

# Inside container, verify installation
python -c "import flash_attn; print(f'FlashAttention version: {flash_attn.__version__}')"

# Run generation
cd /mnt
python generate.py \
    --config_path ./configs/7b-sh-32k-v1.yml \
    --checkpoint_path /path/to/checkpoint.bin \
    --cached_generation \
    --prompt_file ./test_prompt.txt
```

--------------------------------

### Expected Generation Output

Source: https://github.com/togethercomputer/stripedhyena/blob/main/README.md

Sample output generated by the test prompt after successful installation.

```text
The four species of hyenas are the striped hyena (Hyaena hyaena), the brown hyena (Parahyaena brunnea), the spotted hyena (Crocuta crocuta), and the aardwolf (Proteles cristata).

The striped hyena is the most widespread species, occurring in Africa, the Middle East, and Asia.
```

--------------------------------

### Initialize and Run StripedHyena Model

Source: https://context7.com/togethercomputer/stripedhyena/llms.txt

Demonstrates loading a model configuration, weights, and performing both stateless and stateful forward passes.

```python
import torch
import yaml
from stripedhyena.model import StripedHyena
from stripedhyena.utils import dotdict

# Load configuration from YAML
config = dotdict(yaml.load(open('./configs/7b-sh-32k-v1.yml'), Loader=yaml.FullLoader))

# Initialize the model
model = StripedHyena(config)

# Load pretrained weights
device = torch.device("cuda")
model.load_state_dict(torch.load('pytorch-model.bin', map_location=device), strict=False)
model = model.to(device)

# Convert to mixed precision (keep poles/residues in float32 for numerical stability)
model.to_bfloat16_except_poles_residues()

# Example forward pass (stateless)
input_ids = torch.randint(0, 32000, (1, 128)).to(device)  # batch_size=1, seq_len=128
with torch.inference_mode():
    logits, _ = model(input_ids)  # Output shape: (1, 128, vocab_size)

print(f"Output logits shape: {logits.shape}")  # torch.Size([1, 128, 32000])

# Stateful forward pass with caching for generation
inference_params = model.initialize_inference_params()
with torch.inference_mode():
    logits, inference_params_out = model(input_ids, inference_params_dict=inference_params)
```

--------------------------------

### Build and Run StripedHyena Docker Container

Source: https://github.com/togethercomputer/stripedhyena/blob/main/README.md

Commands to build the environment image and launch an interactive container with GPU support.

```bash
docker build --tag sh:test .
```

```bash
docker run -it --gpus all --network="host" --shm-size 900G -v=<path_to_this_repo>:/mnt:rw --rm sh:test
```

--------------------------------

### Run Command-Line Generation

Source: https://context7.com/togethercomputer/stripedhyena/llms.txt

Execute text generation using standalone checkpoints via the command line.

```bash
# Standalone generation with local checkpoint
python generate.py \
    --config_path ./configs/7b-sh-32k-v1.yml \
    --checkpoint_path /path/to/pytorch-model.bin \
    --cached_generation \
    --prompt_file ./test_prompt.txt \
    --num_tokens 100
```

--------------------------------

### Generate Text with StripedHyena Models

Source: https://github.com/togethercomputer/stripedhyena/blob/main/README.md

Use this command to generate text with StripedHyena models hosted on HuggingFace. Replace `<model_id>` with either 'togethercomputer/StripedHyena-Hessian-7B' or 'togethercomputer/StripedHyena-Nous-7B'. Ensure you have the 'test_prompt.txt' file in the current directory.

```bash
python generate_transformers.py --model-name <model_id> --input-file ./test_prompt.txt
```

--------------------------------

### Load and Generate with StripedHyena

Source: https://context7.com/togethercomputer/stripedhyena/llms.txt

Demonstrates loading a model via HuggingFace AutoClasses and performing text generation with both streaming and standard output.

```python
model_name = "togethercomputer/StripedHyena-Nous-7B"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    trust_remote_code=True
)
tokenizer.pad_token = tokenizer.eos_token

config = AutoConfig.from_pretrained(model_name, trust_remote_code=True)
config.use_cache = True  # Enable caching for efficient generation

device = torch.device("cuda")
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    config=config,
    trust_remote_code=True
).to(device)

# Chat model prompt format
prompt = "### Instruction:\nExplain what neural networks are in simple terms.\n\n### Response:\n"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)

# Generate with streaming output
streamer = TextStreamer(tokenizer)
output = model.generate(
    input_ids,
    max_new_tokens=256,
    temperature=0.7,
    top_k=50,
    top_p=0.9,
    repetition_penalty=1.1,
    do_sample=True,
    streamer=streamer,
    eos_token_id=tokenizer.eos_token_id
)

# Full generation without streaming
output_ids = model.generate(
    input_ids,
    max_new_tokens=128,
    temperature=None,  # Greedy decoding when temperature is None
    do_sample=False
)
response = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print(response)
```

--------------------------------

### Configure StripedHyena model parameters

Source: https://context7.com/togethercomputer/stripedhyena/llms.txt

Define model architecture, layer assignments, and kernel settings in a YAML configuration file.

```yaml
model_name: sh-7b-32k-v1
vocab_size: 32000
hidden_size: 4096
num_filters: 4096
num_layers: 32
num_attention_heads: 32

# Layer assignment (0-indexed)
attn_layer_idxs: [1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31]
hyena_layer_idxs: [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32]

# Hyena-specific settings
state_size: 2
short_filter_length: 3
short_filter_bias: true
hyena_filter_groups: 1

# Attention settings
rotary_emb_base: 500000
proj_groups: 4  # GQA groups
qkv_proj_bias: false
mha_out_proj_bias: false

# MLP settings
inner_mlp_size: 14336

# Sequence settings
max_seqlen: 32768
max_batch_size: 1

# Precision settings
eps: 0.00001
tie_embeddings: false

# Kernel options (FlashAttention required, others optional)
use_flash_attn: true
use_flash_rmsnorm: true
use_flash_depthwise: false
use_flashfft: false

# Generation settings
prefill_style: fft  # Options: "fft" (faster) or "recurrence" (lower memory)
tokenizer_type: HFAutoTokenizer
vocab_file: tokenizer/tokenizer.json
```

--------------------------------

### Generate text with HuggingFace models

Source: https://context7.com/togethercomputer/stripedhyena/llms.txt

Run inference using the generate_transformers.py script with various sampling and input configurations.

```bash
python generate_transformers.py \
    --model-name togethercomputer/StripedHyena-Hessian-7B \
    --input-file ./test_prompt.txt \
    --max-new-tokens 128
```

```bash
python generate_transformers.py \
    --model-name togethercomputer/StripedHyena-Nous-7B \
    --input-file ./prompt.txt \
    --max-new-tokens 256 \
    --temperature 0.7 \
    --top-k 50 \
    --top-p 0.9 \
    --repetition-penalty 1.1
```

```bash
python generate_transformers.py \
    --model-name togethercomputer/StripedHyena-Nous-7B \
    --max-new-tokens 100 \
    --temperature 0.8
```

--------------------------------

### Generate Text with StripedHyena

Source: https://github.com/togethercomputer/stripedhyena/blob/main/README.md

Execute text generation using a specified configuration and checkpoint path.

```bash
python generate.py --config_path ./configs/7b-sh-32k-v1.yml \
--checkpoint_path <path_to_ckpt> --cached_generation \
--prompt_file ./test_prompt.txt
```

--------------------------------

### Integrate with HuggingFace Transformers

Source: https://context7.com/togethercomputer/stripedhyena/llms.txt

Shows the necessary imports for loading StripedHyena models via the HuggingFace ecosystem.

```python
import torch
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer, TextStreamer

# Available models:
# - Base: togethercomputer/StripedHyena-Hessian-7B
```

--------------------------------

### Configure StripedHyena Inference Parameters

Source: https://context7.com/togethercomputer/stripedhyena/llms.txt

Manage inference parameters for MHA and Hyena layers, including sequence offsets and state resets.

```python
# Combined inference params dictionary (used by StripedHyena model)
inference_params_dict = {
    "mha": mha_params,
    "hyena": hyena_params
}

# Using model's built-in initialization (recommended)
config = {"max_seqlen": 8192, "max_batch_size": 1, "short_filter_length": 3, "state_size": 2}
# model = StripedHyena(config)
# inference_params = model.initialize_inference_params()

# Updating sequence offset during generation
inference_params_dict["mha"].seqlen_offset += 1
inference_params_dict["hyena"].seqlen_offset += 1

# Resetting parameters for new sequence
mha_params.reset(max_seqlen=4096, max_batch_size=2)
hyena_params.reset()

# Access cached states
print(f"KV memory dict keys: {mha_params.key_value_memory_dict.keys()}")
print(f"FIR state dict keys: {hyena_params.fir_state_dict.keys()}")
print(f"IIR state dict keys: {hyena_params.state_dict.keys()}")
```

--------------------------------

### Sampling Strategies and Logit Filtering

Source: https://context7.com/togethercomputer/stripedhyena/llms.txt

Implements top-k and top-p sampling strategies and provides utilities for in-place logit filtering.

```python
import torch
from stripedhyena.sample import sample, modify_logits_for_top_k_filtering, modify_logits_for_top_p_filtering

# Example logits from model output
batch_size, vocab_size = 1, 32000
logits = torch.randn(batch_size, vocab_size)

# Greedy decoding (top_k=1)
greedy_token = sample(logits.clone(), top_k=1)
print(f"Greedy token: {greedy_token}")

# Top-k sampling (sample from top 50 tokens)
topk_token = sample(logits.clone(), top_k=50, temperature=0.8)
print(f"Top-k sampled token: {topk_token}")

# Top-p (nucleus) sampling
topp_token = sample(logits.clone(), top_k=0, top_p=0.9, temperature=1.0)
print(f"Top-p sampled token: {topp_token}")

# Combined top-k and top-p with temperature
combined_token = sample(
    logits.clone(),
    top_k=50,
    top_p=0.95,
    temperature=0.7
)
print(f"Combined sampling token: {combined_token}")

# Direct logit filtering for custom sampling
test_logits = torch.randn(1, 100)

# Apply top-k filtering (modifies in-place)
modify_logits_for_top_k_filtering(test_logits, top_k=10)
print(f"After top-k: {(test_logits > float('-inf')).sum()} tokens remain")

# Apply top-p filtering (modifies in-place)
test_logits2 = torch.randn(1, 100)
modify_logits_for_top_p_filtering(test_logits2, top_p=0.9)
```

--------------------------------

### Inference State Management

Source: https://context7.com/togethercomputer/stripedhyena/llms.txt

Configures inference parameters for attention KV caching and Hyena layer recurrent states.

```python
from stripedhyena.cache import InferenceParams, RecurrentInferenceParams
from stripedhyena.model import StripedHyena
import torch

# InferenceParams for attention layers (KV cache)
mha_params = InferenceParams(
    max_seqlen=8192,       # Maximum sequence length
    max_batch_size=4,       # Maximum batch size
    seqlen_offset=0         # Starting position in sequence
)

# RecurrentInferenceParams for Hyena layers (recurrent state)
hyena_params = RecurrentInferenceParams(
    fir_filter_length=3,    # Short FIR filter length
    state_dim=16,           # State dimension per channel
    seqlen_offset=0
)
```

--------------------------------

### Perform Hyena Inference Engine Operations

Source: https://context7.com/togethercomputer/stripedhyena/llms.txt

Execute parallel FIR filtering and sequential autoregressive generation steps using the HyenaInferenceEngine.

```python
from stripedhyena.engine import HyenaInferenceEngine
import torch

# Initialize engine with prefill style
engine = HyenaInferenceEngine(
    layer_idx=0,
    iir_prefill_style="modal-fft"  # Options: "recurrence", "modal-fft", "hybrid-modal-recurrence"
)

# Available prefill modes (trade-off between memory and speed):
# - "recurrence": Most memory efficient, slower (good for very long sequences)
# - "modal-fft": Fast parallel computation using FFT (default)
# - "canonical-fft": FFT with companion form SSM

# Parallel FIR computation (during prefill)
batch_size, seq_len, hidden_size = 1, 1024, 4096
u = torch.randn(batch_size, hidden_size, seq_len, device="cuda")  # Input (B, D, L)
weight = torch.randn(3 * hidden_size, 1, 3, device="cuda")  # Filter weights
bias = torch.randn(3 * hidden_size, device="cuda")

z_pre, fir_state = engine.parallel_fir(
    fir_fn=torch.nn.functional.conv1d,
    u=u,
    weight=weight,
    bias=bias,
    L=seq_len,
    fir_length=3
)
print(f"FIR output shape: {z_pre.shape}")  # (B, 3*D, L)

# Step FIR for autoregressive generation
u_step = torch.randn(batch_size, 3 * hidden_size, device="cuda")  # Single token
fir_state = torch.randn(batch_size, 3 * hidden_size, 2, device="cuda")  # State buffer

y, new_fir_state = engine.step_fir(
    u=u_step,
    fir_state=fir_state,
    weight=weight,
    bias=bias
)
print(f"Step FIR output shape: {y.shape}")  # (B, 3*D)

# Step IIR for recurrent generation
state_dim = 2
x2 = torch.randn(batch_size, hidden_size, device="cuda")
x1 = torch.randn(batch_size, hidden_size, device="cuda")
v = torch.randn(batch_size, hidden_size, device="cuda")
D = torch.randn(hidden_size, device="cuda")
residues = torch.randn(hidden_size, state_dim, 1, 2, device="cuda")
poles = torch.randn(hidden_size, state_dim, 1, 2, device="cuda")
iir_state = torch.zeros(batch_size, hidden_size, state_dim, dtype=torch.complex64, device="cuda")

y, new_iir_state = engine.step_iir(
    x2=x2, x1=x1, v=v,
    D=D, residues=residues, poles=poles,
    iir_state=iir_state
)
print(f"Step IIR output shape: {y.shape}")  # (B, D)
```

--------------------------------

### StripedHyena Citation

Source: https://github.com/togethercomputer/stripedhyena/blob/main/README.md

If you find the StripedHyena pretrained models or architecture useful for your research or application, please cite the project using the provided BibTeX entry.

```bibtex
@software{stripedhyena,
  title        = {{StripedHyena: Moving Beyond Transformers with Hybrid Signal Processing Models}},
  author       = { Poli, Michael and Wang, Jue and Massaroli, Stefano and Quesnelle, Jeffrey and Carlow, Ryan and Nguyen, Eric and Thomas, Armin},
  month        = 12,
  year         = 2023,
  url          = { https://github.com/togethercomputer/stripedhyena },
  doi          = { 10.57967/hf/1595 },
}
```

--------------------------------

### Tokenization with HFAutoTokenizer

Source: https://context7.com/togethercomputer/stripedhyena/llms.txt

Provides methods for tokenizing and detokenizing text, including batch operations and access to special tokens.

```python
from stripedhyena.tokenizer import HFAutoTokenizer
import torch

# Initialize tokenizer from vocabulary file
tokenizer = HFAutoTokenizer('./tokenizer/tokenizer.json')

# Tokenize text to tensor
text = "Hello, how are you today?"
token_ids = tokenizer.tokenize(text)
print(f"Token IDs: {token_ids}")  # tensor([15043, 28725, 910, 460, 368, 3154, 28804])

# Detokenize back to text
decoded_text = tokenizer.detokenize(token_ids.tolist())
print(f"Decoded: {decoded_text}")  # Hello, how are you today?

# Batch tokenization
texts = ["First sentence.", "Second sentence.", "Third sentence."]
batch_tokens = tokenizer.tokenize_batch(texts)

# Batch detokenization
token_batches = torch.tensor([
    [15043, 28725, 910, 460],
    [1014, 478, 460, 3154]
])
decoded_batch = tokenizer.detokenize_batch(token_batches, skip_special_tokens=True)
print(decoded_batch)

# Access special tokens
print(f"EOS token: {tokenizer.eos}")       # </s>
print(f"BOS token: {tokenizer.bos}")       # <s>
print(f"Vocab size: {tokenizer.vocab_size}")  # 32000
```

--------------------------------

### Tokenize Sequences with CharLevelTokenizer

Source: https://context7.com/togethercomputer/stripedhyena/llms.txt

Perform character-level tokenization and detokenization for byte-level sequence modeling.

```python
from stripedhyena.tokenizer import CharLevelTokenizer
import torch

# Initialize character-level tokenizer
tokenizer = CharLevelTokenizer(vocab_size=256)

# Tokenize text (converts to byte values)
text = "ATCGATCG"  # DNA sequence example
token_ids = tokenizer.tokenize(text)
print(f"Token IDs: {token_ids}")  # [65, 84, 67, 71, 65, 84, 67, 71]

# Detokenize back to string
decoded = tokenizer.detokenize(token_ids)
print(f"Decoded: {decoded}")  # ATCGATCG

# Batch tokenization
sequences = ["ATCG", "GCTA", "TTAA"]
batch_tokens = tokenizer.tokenize_batch(sequences)
print(f"Batch tokens: {batch_tokens}")

# Batch detokenization from tensor
token_tensor = torch.tensor([[65, 84, 67, 71], [71, 67, 84, 65]])
decoded_batch = tokenizer.detokenize_batch(token_tensor)
print(f"Decoded batch: {decoded_batch}")  # ['ATCG', 'GCTA']

# Special tokens
print(f"EOS ID: {tokenizer.eos_id}")  # 0
print(f"PAD ID: {tokenizer.pad_id}")  # 1
print(f"Vocab size: {tokenizer.vocab_size}")  # 256
```

--------------------------------

### Generate Text with Generator Class

Source: https://context7.com/togethercomputer/stripedhyena/llms.txt

Uses the Generator class for high-level text generation with sampling parameters and caching support.

```python
import torch
import yaml
from stripedhyena.model import StripedHyena
from stripedhyena.generation import Generator
from stripedhyena.tokenizer import HFAutoTokenizer
from stripedhyena.utils import dotdict

# Setup
config = dotdict(yaml.load(open('./configs/7b-sh-32k-v1.yml'), Loader=yaml.FullLoader))
tokenizer = HFAutoTokenizer(config.vocab_file)
device = torch.device("cuda")

model = StripedHyena(config)
model.load_state_dict(torch.load('pytorch-model.bin', map_location=device), strict=False)
model = model.to(device)
model.to_bfloat16_except_poles_residues()

# Initialize generator with sampling parameters
generator = Generator(
    model=model,
    tokenizer=tokenizer,
    top_k=50,      # Consider top 50 tokens
    top_p=0.7,     # Nucleus sampling threshold
    temperature=1  # Sampling temperature (1.0 = no scaling)
)

# Generate text with caching (recommended for efficiency)
prompt = "The four species of hyenas are"
with torch.inference_mode():
    generation, scores = generator.generate(
        device=device,
        input_string=prompt,
        num_tokens=100,              # Number of tokens to generate
        cached_generation=True,      # Enable KV and Hyena state caching
        print_generation=True,       # Print tokens as they're generated
        verbose=True,                # Show memory usage and debug info
        max_seqlen=8192,            # Maximum sequence length
        stop_at_eos=True            # Stop when EOS token is generated
    )

# Decode the generated tokens
output_text = tokenizer.detokenize(generation[0].tolist())
print(f"Generated: {output_text}")
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.