### Command Line Setup for StreamingLLM (Bash)

Source: https://context7.com/mit-han-lab/streaming-llm/llms.txt

Provides commands to set up a Python environment and install necessary dependencies for using StreamingLLM from the command line. This includes creating a conda environment, activating it, and installing PyTorch, transformers, and other required libraries.

```Bash
# Environment setup
conda create -yn streaming python=3.8
conda activate streaming
pip install torch torchvision torchaudio
pip install transformers==4.33.0 accelerate datasets evaluate wandb scikit-learn scipy sentencepiece
python setup.py develop
```

--------------------------------

### Perplexity Evaluation with StreamingLLM (Python)

Source: https://context7.com/mit-han-lab/streaming-llm/llms.txt

Evaluates model perplexity on long sequences using StreamingLLM. This example demonstrates processing long documents token-by-token while maintaining bounded memory usage by applying KV cache truncation. Dependencies include PyTorch, datasets, and streaming_llm.

```Python
import torch
from torch.nn import CrossEntropyLoss
from tqdm import tqdm
from datasets import load_dataset
from streaming_llm.utils import load
from streaming_llm.kv_cache import StartRecentKVCache
from streaming_llm.pos_shift.modify_llama import enable_llama_pos_shift_attention

# Load model and enable streaming
model, tokenizer = load("meta-llama/Llama-2-7b-hf")
enable_llama_pos_shift_attention(model)

kv_cache = StartRecentKVCache(
    start_size=4,
    recent_size=2044,
    k_seq_dim=2,
    v_seq_dim=2,
)

# Load evaluation dataset
data = load_dataset("wikitext", "wikitext-2-raw-v1", split="test")

loss_fn = CrossEntropyLoss(reduction="none")
nlls = []
past_key_values = None

for text in data["text"][:10]:  # Evaluate on first 10 samples
    encodings = tokenizer(text, return_tensors="pt")
    seq_len = encodings.input_ids.size(1)

    for idx in tqdm(range(seq_len - 1)):
        input_ids = encodings.input_ids[:, idx:idx+1].to(model.device)

        with torch.no_grad():
            outputs = model(
                input_ids,
                past_key_values=past_key_values,
                use_cache=True,
            )
            logits = outputs.logits.view(-1, model.config.vocab_size)
            past_key_values = outputs.past_key_values

            # Apply KV cache truncation
            past_key_values = kv_cache(past_key_values)

            label = encodings.input_ids[:, idx+1:idx+2].to(logits.device).view(-1)
            nll = loss_fn(logits, label)
            nlls.append(nll)

# Calculate perplexity
ppl = torch.exp(torch.stack(nlls).mean())
print(f"Perplexity: {ppl.item():.2f}")
```

--------------------------------

### Load Model and Tokenizer for Streaming

Source: https://context7.com/mit-han-lab/streaming-llm/llms.txt

Provides a utility function to load models and tokenizers with optimized settings for streaming, including float16 precision and automatic device mapping. It also demonstrates the manual equivalent for custom configurations.

```python
from streaming_llm.utils import load

model, tokenizer = load("lmsys/vicuna-13b-v1.3")

# Manual loading equivalent
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("lmsys/vicuna-13b-v1.3", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "lmsys/vicuna-13b-v1.3",
    device_map="auto",
    torch_dtype=torch.float16,
    trust_remote_code=True,
)
if tokenizer.pad_token_id is None:
    tokenizer.pad_token_id = tokenizer.eos_token_id or 0
model.eval()
```

--------------------------------

### Manage KV Cache with StartRecentKVCache

Source: https://context7.com/mit-han-lab/streaming-llm/llms.txt

Implements the core cache eviction logic. This class handles tensor slicing to keep only the initial sink tokens and the most recent tokens, supporting various model architectures.

```python
import torch
from streaming_llm.kv_cache import StartRecentKVCache

kv_cache = StartRecentKVCache(
    start_size=4,
    recent_size=512,
    k_seq_dim=2,
    v_seq_dim=2,
)

past_key_values = model_outputs.past_key_values
truncated_cache = kv_cache(past_key_values)
past_key_values = kv_cache.evict_for_space(past_key_values, num_coming=10)
past_key_values = kv_cache.evict_range(past_key_values, start=10, end=50)
```

--------------------------------

### Streaming Inference for Continuous Dialogue (Python)

Source: https://context7.com/mit-han-lab/streaming-llm/llms.txt

Demonstrates the full inference loop for continuous multi-turn dialogue using StreamingLLM. It includes KV cache management for unlimited conversation length and uses greedy decoding for token generation. Dependencies include PyTorch and the streaming_llm library.

```Python
import torch
from streaming_llm.utils import load
from streaming_llm.enable_streaming_llm import enable_streaming_llm

@torch.no_grad()
def greedy_generate(model, tokenizer, input_ids, past_key_values, max_gen_len):
    """Generate tokens using greedy decoding with KV caching."""
    outputs = model(
        input_ids=input_ids,
        past_key_values=past_key_values,
        use_cache=True,
    )
    past_key_values = outputs.past_key_values
    pred_token_idx = outputs.logits[:, -1, :].argmax(dim=-1).unsqueeze(1)
    generated_ids = [pred_token_idx.item()]

    for _ in range(max_gen_len - 1):
        outputs = model(
            input_ids=pred_token_idx,
            past_key_values=past_key_values,
            use_cache=True,
        )
        past_key_values = outputs.past_key_values
        pred_token_idx = outputs.logits[:, -1, :].argmax(dim=-1).unsqueeze(1)
        generated_ids.append(pred_token_idx.item())

        if pred_token_idx == tokenizer.eos_token_id:
            break

    response = tokenizer.decode(generated_ids, skip_special_tokens=True)
    return past_key_values, response

@torch.no_grad()
def streaming_chat(model, tokenizer, prompts, kv_cache, max_gen_len=1000):
    """Run multi-turn conversation with streaming KV cache management."""
    past_key_values = None

    for prompt in prompts:
        formatted_prompt = f"USER: {prompt}\n\nASSISTANT: "
        input_ids = tokenizer(formatted_prompt, return_tensors="pt").input_ids
        input_ids = input_ids.to(model.device)

        # Evict old tokens to make space for new input + generation
        if kv_cache is not None:
            space_needed = input_ids.shape[1] + max_gen_len
            past_key_values = kv_cache.evict_for_space(past_key_values, space_needed)

        past_key_values, response = greedy_generate(
            model, tokenizer, input_ids, past_key_values, max_gen_len
        )
        print(f"User: {prompt}")
        print(f"Assistant: {response}\n")

# Main usage
model, tokenizer = load("lmsys/vicuna-13b-v1.3")
kv_cache = enable_streaming_llm(model, start_size=4, recent_size=2000)

# Run unlimited conversation
prompts = [
    "What is the capital of France?",
    "Tell me more about its history.",
    "What are some famous landmarks there?",
    # ... can continue indefinitely
]
streaming_chat(model, tokenizer, prompts, kv_cache)
```

--------------------------------

### Enable StreamingLLM

Source: https://context7.com/mit-han-lab/streaming-llm/llms.txt

This function enables StreamingLLM on supported model architectures by applying attention modifications and the StartRecentKVCache mechanism.

```APIDOC
## enable_streaming_llm

### Description
The main entry point for enabling StreamingLLM on supported model architectures. This function automatically detects the model type (Llama, MPT, GPT-NeoX, or Falcon) and applies the appropriate position shift attention modifications along with the StartRecentKVCache mechanism to enable infinite-length streaming.

### Method
Python function call

### Endpoint
N/A (Python function)

### Parameters
#### Function Parameters
- **model** (object) - Required - The Hugging Face model object to enable StreamingLLM on.
- **start_size** (int) - Optional - Number of initial "sink" tokens to always keep (default: 4).
- **recent_size** (int) - Optional - Number of most recent tokens to keep in cache (default: 2000).

### Request Example
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from streaming_llm.enable_streaming_llm import enable_streaming_llm

# Load a model (supports Llama-2, MPT, Falcon, Pythia/GPT-NeoX)
model_name = "lmsys/vicuna-13b-v1.3"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype=torch.float16,
    trust_remote_code=True,
)
model.eval()

# Enable StreamingLLM with attention sinks
kv_cache = enable_streaming_llm(
    model,
    start_size=4,       # Keep first 4 tokens as attention sinks
    recent_size=2000    # Keep most recent 2000 tokens
)
# Total cache size = start_size + recent_size = 2004 tokens
```

### Response
#### Success Response
- **kv_cache** (object) - A kv_cache object to use during inference.

#### Response Example
```
StartRecentKVCache: 4, 2000
```
```

--------------------------------

### StartRecentKVCache Class

Source: https://context7.com/mit-han-lab/streaming-llm/llms.txt

The core KV cache management class that implements the attention sink mechanism, maintaining a fixed-size cache by keeping initial "sink" tokens and the most recent tokens.

```APIDOC
## StartRecentKVCache

### Description
The core KV cache management class that implements the attention sink mechanism. It maintains a fixed-size cache by keeping initial "sink" tokens and the most recent tokens while evicting intermediate tokens. This class handles the tensor slicing across different model architectures with varying KV dimension layouts.

### Method
Python class instantiation and methods

### Endpoint
N/A (Python class)

### Parameters
#### Class Parameters
- **start_size** (int) - Required - Number of initial tokens to keep.
- **recent_size** (int) - Required - Number of recent tokens to keep.
- **k_seq_dim** (int) - Required - Sequence dimension index in the key tensor (e.g., 2 for Llama/GPT-NeoX, 3 for MPT, 1 for Falcon).
- **v_seq_dim** (int) - Required - Sequence dimension index in the value tensor (e.g., 2 for Llama/GPT-NeoX, 2 for MPT, 1 for Falcon).

### Request Example (Instantiation)
```python
from streaming_llm.kv_cache import StartRecentKVCache

# Create KV cache manager for Llama-2
kv_cache = StartRecentKVCache(
    start_size=4,       # Number of initial tokens to keep
    recent_size=512,    # Number of recent tokens to keep
    k_seq_dim=2,        # Dimension of sequence in key tensor
    v_seq_dim=2,        # Dimension of sequence in value tensor
)
# Total cache size = start_size + recent_size = 516 tokens
```

### Usage Examples
#### Usage 1: Truncate existing cache when it exceeds capacity
```python
# past_key_values is a list of (key, value) tuples per layer
past_key_values = model_outputs.past_key_values  # From model forward pass
truncated_cache = kv_cache(past_key_values)      # Truncates to cache_size if needed
```

#### Usage 2: Evict tokens to make space for incoming tokens
```python
# Useful before processing new input to ensure sufficient space
space_needed = input_seq_len + max_gen_len  # Total tokens needed
past_key_values = kv_cache.evict_for_space(past_key_values, num_coming=space_needed)
```

#### Usage 3: Evict a specific range of tokens from cache
```python
# Removes tokens from index 'start' to 'end' (exclusive)
past_key_values = kv_cache.evict_range(past_key_values, start=10, end=50)
```

### Response
#### Success Response (Methods)
- **truncated_cache** (list) - The KV cache after truncation.
- **past_key_values** (list) - The KV cache after evicting tokens.

#### Response Example (Method Output)
```
# Output is the modified past_key_values list
```
```

--------------------------------

### Enable StreamingLLM for Supported Models

Source: https://context7.com/mit-han-lab/streaming-llm/llms.txt

Initializes StreamingLLM on a pre-trained causal language model. It automatically detects the architecture and configures the StartRecentKVCache to manage memory during long-context inference.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from streaming_llm.enable_streaming_llm import enable_streaming_llm

model_name = "lmsys/vicuna-13b-v1.3"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype=torch.float16,
    trust_remote_code=True,
)
model.eval()

kv_cache = enable_streaming_llm(
    model,
    start_size=4,
    recent_size=2000
)
```

--------------------------------

### Enable Position Shift Attention for GPT-NeoX

Source: https://context7.com/mit-han-lab/streaming-llm/llms.txt

Enables streaming inference for GPT-NeoX and Pythia architectures. This involves modifying attention layers and setting up a KV cache with sequence dimensions of 2.

```python
from streaming_llm.pos_shift.modify_gpt_neox import enable_gpt_neox_pos_shift_attention
from streaming_llm.kv_cache import StartRecentKVCache

enable_gpt_neox_pos_shift_attention(model)
kv_cache = StartRecentKVCache(
    start_size=4,
    recent_size=2044,
    k_seq_dim=2,
    v_seq_dim=2,
)
```

--------------------------------

### Enable Position Shift Attention for Llama

Source: https://context7.com/mit-han-lab/streaming-llm/llms.txt

Modifies Llama attention layers in-place to support position shifting. It initializes a StartRecentKVCache with specific sequence dimensions for Llama-2 context windows.

```python
from streaming_llm.pos_shift.modify_llama import enable_llama_pos_shift_attention
from streaming_llm.kv_cache import StartRecentKVCache

enable_llama_pos_shift_attention(model)
kv_cache = StartRecentKVCache(
    start_size=4,
    recent_size=4092,
    k_seq_dim=2,
    v_seq_dim=2,
)
```

--------------------------------

### Enable Position Shift Attention for Falcon

Source: https://context7.com/mit-han-lab/streaming-llm/llms.txt

Configures Falcon models for streaming inference by modifying attention layers. It accounts for Falcon's unique key/value dimension layout (dim 1) compared to standard architectures.

```python
from streaming_llm.pos_shift.modify_falcon import enable_falcon_pos_shift_attention
from streaming_llm.kv_cache import StartRecentKVCache

enable_falcon_pos_shift_attention(model)
kv_cache = StartRecentKVCache(
    start_size=4,
    recent_size=2044,
    k_seq_dim=1,
    v_seq_dim=1,
)
```

--------------------------------

### Enable Llama Position Shift Attention

Source: https://context7.com/mit-han-lab/streaming-llm/llms.txt

Modifies Llama model attention layers to support position shifting for streaming inference by patching the forward method to correctly apply rotary position embeddings.

```APIDOC
## enable_llama_pos_shift_attention

### Description
Modifies Llama model attention layers to support position shifting for streaming inference. This function patches the forward method of all LlamaAttention modules to apply rotary position embeddings correctly when the cache contains non-contiguous position IDs (due to evicted middle tokens).

### Method
Python function call

### Endpoint
N/A (Python function)

### Parameters
#### Function Parameters
- **model** (object) - Required - The Llama-based Hugging Face model object.

### Request Example
```python
import torch
from transformers import AutoModelForCausalLM
from streaming_llm.pos_shift.modify_llama import enable_llama_pos_shift_attention
from streaming_llm.kv_cache import StartRecentKVCache

# Load Llama-based model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-chat-hf",
    device_map="auto",
    torch_dtype=torch.float16,
)

# Enable position shift attention for Llama models
enable_llama_pos_shift_attention(model)
```

### Response
#### Success Response
- The model's attention layers are modified in-place.

#### Response Example
```
# No explicit return value, model is modified directly.
```
```

--------------------------------

### Cite StreamingLLM Research Paper

Source: https://github.com/mit-han-lab/streaming-llm/blob/main/README.md

BibTeX citation format for referencing the StreamingLLM research paper in academic or technical documentation.

```bibtex
@article{xiao2023streamingllm,
        title={Efficient Streaming Language Models with Attention Sinks},
        author={Xiao, Guangxuan and Tian, Yuandong and Chen, Beidi and Han, Song and Lewis, Mike},
        journal={arXiv},
        year={2023}
        }
```

--------------------------------

### Patch Llama Attention for Position Shifting

Source: https://context7.com/mit-han-lab/streaming-llm/llms.txt

Modifies Llama attention layers to support correct rotary position embeddings when the KV cache contains non-contiguous position IDs due to token eviction.

```python
import torch
from transformers import AutoModelForCausalLM
from streaming_llm.pos_shift.modify_llama import enable_llama_pos_shift_attention

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-chat-hf",
    device_map="auto",
    torch_dtype=torch.float16,
)
enable_llama_pos_shift_attention(model)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.