### Scheduler Example Flow

Source: https://context7.com/odysseusq/vlcache/llms.txt

Conceptual example of using the Scheduler to manage inference requests, including prefill and decode phases, and postprocessing.

```python
from nanovllm.engine.scheduler import Scheduler
from nanovllm.engine.sequence import Sequence, SequenceStatus
from nanovllm.config import Config
from nanovllm.sampling_params import SamplingParams

# Create config (normally loaded from model)
# config = Config("/path/to/model")

# Scheduler manages waiting and running queues
# scheduler = Scheduler(config)

# Example scheduling flow (conceptual):
"""
# Add requests to the scheduler
sampling_params = SamplingParams(temperature=0.6, max_tokens=100)
seq1 = Sequence([1, 2, 3, 4, 5], sampling_params)
seq2 = Sequence([6, 7, 8, 9, 10], sampling_params)

scheduler.add(seq1)
scheduler.add(seq2)

# Schedule returns sequences for current step
while not scheduler.is_finished():
    seqs, is_prefill = scheduler.schedule()

    if is_prefill:
        # Prefill phase: process all prompt tokens
        print(f"Prefill {len(seqs)} sequences")
        # For partial recompute sequences, scheduler.block_manager
        # checks if image KV can be reused
    else:
        # Decode phase: generate one token per sequence
        print(f"Decode {len(seqs)} sequences")

    # After model forward pass, postprocess with generated tokens
    generated_tokens = [42] * len(seqs)  # Example tokens
    scheduler.postprocess(seqs, generated_tokens)

    # Check for finished sequences
    for seq in seqs:
        if seq.is_finished:
            print(f"Sequence {seq.seq_id} finished")
"""

```

--------------------------------

### LLM Initialization with Config Options

Source: https://context7.com/odysseusq/vlcache/llms.txt

Initialize the LLM class, passing configuration options as keyword arguments. This example demonstrates enabling eager execution, setting tensor parallel size, max model length, and enabling cache reuse via recompute_ratio.

```python
from nanovllm import LLM
llm = LLM(
    "/path/to/qwen2.5-vl-3b",
    enforce_eager=True,
    tensor_parallel_size=2,       # Use 2 GPUs
    max_model_len=8192,
    recompute_ratio=0.1           # Enable 90% cache reuse
)
```

--------------------------------

### Sequence Creation with Image Hashes

Source: https://context7.com/odysseusq/vlcache/llms.txt

Create a Sequence object, which represents a generation task. This example includes token IDs, sampling parameters, and image hashes, indicating that the sequence involves multimodal input.

```python
# Create a sequence
sampling_params = SamplingParams(temperature=0.6, max_tokens=128)
token_ids = [1, 2, 3, 151655, 151655, 151655, 4, 5]  # 151655 = image token
image_hashes = [123456789]  # Content hash of the image

seq = Sequence(token_ids, sampling_params, mm_inputs=None, image_hashes=image_hashes)
```

--------------------------------

### Initialize LLM and Generate Text Completions

Source: https://context7.com/odysseusq/vlcache/llms.txt

Initializes the LLM with a Qwen3 model and configures sampling parameters for text generation. Use this for standard text-based inference tasks.

```python
import os
from nanovllm import LLM, SamplingParams
from transformers import AutoTokenizer

# Initialize the LLM with a Qwen3 model
path = os.path.expanduser("~/huggingface/Qwen3-0.6B/")
tokenizer = AutoTokenizer.from_pretrained(path)
llm = LLM(
    path,
    enforce_eager=True,           # Disable CUDA graphs for debugging
    tensor_parallel_size=1,       # Number of GPUs for tensor parallelism
    max_model_len=4096,           # Maximum sequence length
    gpu_memory_utilization=0.9,   # Fraction of GPU memory to use
    recompute_ratio=0.0           # Ratio of image tokens to recompute (0.0 = full recompute)
)

# Configure sampling parameters
sampling_params = SamplingParams(
    temperature=0.6,    # Sampling temperature (must be > 0)
    max_tokens=256,     # Maximum tokens to generate
    ignore_eos=False    # Whether to ignore EOS token
)

# Prepare prompts using chat template
prompts = ["introduce yourself", "list all prime numbers within 100"]
prompts = [
    tokenizer.apply_chat_template(
        [{"role": "user", "content": prompt}],
        tokenize=False,
        add_generation_prompt=True,
    )
    for prompt in prompts
]

# Generate completions
outputs = llm.generate(prompts, sampling_params)

# Process outputs
for prompt, output in zip(prompts, outputs):
    print(f"Prompt: {prompt!r}")
    print(f"Completion: {output['text']!r}")
    print(f"Token IDs: {output['token_ids']}")
    print(f"TTFT: {output['ttft']:.3f}s")
```

--------------------------------

### Initialize VL Model and Perform Multimodal Inference

Source: https://context7.com/odysseusq/vlcache/llms.txt

Initializes a Vision-Language model (Qwen2.5-VL) and prepares multimodal inputs for inference. This snippet demonstrates how to process images and text together for tasks like image description.

```python
import os
from nanovllm import LLM, SamplingParams
from transformers import AutoProcessor
from qwen_vl_utils import process_vision_info

# Initialize VL model
path = os.path.expanduser("~/huggingface/Qwen2.5-VL-3B-Instruct/")
processor = AutoProcessor.from_pretrained(path)
llm = LLM(
    path,
    enforce_eager=True,
    tensor_parallel_size=1,
    recompute_ratio=0.1  # Enable partial recompute: compute 10%, reuse 90%
)

sampling_params = SamplingParams(temperature=0.6, max_tokens=256)

# Prepare multimodal message
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

# Process inputs through the processor
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)

# Prepare multimodal inputs for the engine
prompts = inputs.input_ids.tolist()
mm_inputs = [{
    "pixel_values": inputs.pixel_values,
    "image_grid_thw": inputs.image_grid_thw,
}]

# Generate with multimodal inputs
outputs = llm.generate(prompts, sampling_params, mm_inputs=mm_inputs)

for output in outputs:
    print(f"Output: {output['text']}")
    print(f"VIT time: {output['vit_time']*1000:.1f}ms")
    print(f"TTFT: {output['ttft']*1000:.1f}ms")
```

--------------------------------

### Run Partial Recompute Benchmark

Source: https://context7.com/odysseusq/vlcache/llms.txt

Executes the benchmark script for partial KV cache reuse with vision tokens. Requires the Qwen2.5-VL-3B model.

```bash
# Run the benchmark (requires Qwen2.5-VL-3B model)
python test_partial_recompute.py

# Expected output:
# ============================================================
# Phase 1: Baseline  (recompute_ratio=0.0)
# ============================================================
#   Image tokens: 10752, R: 0, Reused: 0, Prompt1: 10812, Prompt2: 10807
#   After req1: image_kv_cache=1 entries
#   [Req 1 cold] TTFT:   850.2 ms  VIT: 45.3 ms
#   [Req 2 warm] TTFT:   820.1 ms  VIT: 44.8 ms
#
# ============================================================
# Phase 2: Partial recompute  (recompute_ratio=0.1)

```

--------------------------------

### Allocate and Store Image KV Cache

Source: https://context7.com/odysseusq/vlcache/llms.txt

Demonstrates allocating blocks for a sequence, attempting partial recompute by reusing image KV, and storing image KV for future requests.

```python
if block_manager.can_allocate(seq):
    block_manager.allocate(seq)
    print(f"Allocated {len(seq.block_table)} blocks for sequence")

# Try partial recompute (checks if image KV can be reused)
image_token_id = 151655
can_reuse = block_manager.try_partial_recompute(seq, image_token_id)
if can_reuse:
    print(f"Partial recompute enabled: {seq.num_recompute_tokens} tokens to recompute")
    print(f"Image token range: {seq.image_token_range}")

# Store image KV for future reuse after prefill
seq.find_image_token_range(image_token_id)
block_manager.store_image_kv(seq)
print(f"Image KV cache entries: {len(block_manager.image_kv_cache)}")

# Later requests with same image can retrieve cached KV
entry = block_manager.get_image_kv_entry(image_hashes[0])
if entry:
    print(f"Found cached KV: blocks={entry.block_table[:3]}..., 
          f"range=[{entry.img_start}, {entry.img_end})")
```

--------------------------------

### Config Class Initialization

Source: https://context7.com/odysseusq/vlcache/llms.txt

Instantiate the Config dataclass to set up the LLM engine's parameters, including memory, parallelism, and cache settings. The model path is a required argument.

```python
from nanovllm.config import Config

# Create configuration directly (usually done internally by LLM)
config = Config(
    model="/path/to/model",           # Path to model directory (required)
    max_num_batched_tokens=16384,     # Max tokens per batch
    max_num_seqs=512,                 # Max concurrent sequences
    max_model_len=4096,               # Max sequence length
    gpu_memory_utilization=0.9,       # GPU memory fraction to use
    tensor_parallel_size=1,           # Number of GPUs for TP
    enforce_eager=False,              # Disable CUDA graphs if True
    kvcache_block_size=256,           # Block size for KV cache (must be multiple of 256)
    encoder_cache_ratio=0.2,          # Fraction of available memory for encoder cache
    recompute_ratio=0.0               # Image token recompute ratio (0.0-1.0)
)
```

--------------------------------

### Benchmark Sampling Parameters

Source: https://context7.com/odysseusq/vlcache/llms.txt

Set up sampling parameters for benchmarking, specifically ignoring the End-of-Sequence (EOS) token for longer generations.

```python
benchmark_params = SamplingParams(
    temperature=0.8,
    max_tokens=1024,
    ignore_eos=True  # Continue generating even after EOS
)
```

--------------------------------

### Standard Sampling Parameters

Source: https://context7.com/odysseusq/vlcache/llms.txt

Configure standard sampling parameters like temperature, max tokens, and EOS token handling for text generation.

```python
params = SamplingParams(
    temperature=0.6,    # Sampling temperature (must be > 1e-10, no greedy)
    max_tokens=128,     # Maximum tokens to generate
    ignore_eos=False    # Stop at EOS token
)
```

--------------------------------

### Benchmark Partial KV Cache Recomputation

Source: https://github.com/odysseusq/vlcache/blob/main/README.md

This script benchmarks the Time To First Token (TTFT) improvement when reusing image KV cache across requests with the same image but different text prompts. It compares a baseline of full recomputation against a partial recomputation strategy.

```bash
python test_partial_recompute.py
```

--------------------------------

### Create Multimodal Sequence with Image

Source: https://context7.com/odysseusq/vlcache/llms.txt

Initializes a Sequence object for multimodal inference, including image data and associated hashes, and finds the image token range.

```python
from nanovllm.engine.sequence import Sequence, SequenceStatus
from nanovllm.sampling_params import SamplingParams

# Create a multimodal sequence with image
mm_inputs = {
    "pixel_values": None,  # Would contain actual pixel data
    "image_grid_thw": None  # Would contain grid dimensions
}
image_hashes = [987654321]  # Content-based hash of the image

vl_seq = Sequence(
    token_ids=[1, 151655, 151655, 151655, 2],  # 151655 = image token
    sampling_params=sampling_params,
    mm_inputs=mm_inputs,
    image_hashes=image_hashes
)

# Find image token positions
image_token_id = 151655
vl_seq.find_image_token_range(image_token_id)
print(f"Image token range: {vl_seq.image_token_range}")  # (1, 4)

# Simulate token generation
vl_seq.append_token(100)
vl_seq.append_token(101)
print(f"After generation - Total: {len(vl_seq)}, Completion: {vl_seq.num_completion_tokens}")

# Check completion status
vl_seq.status = SequenceStatus.FINISHED
print(f"Is finished: {vl_seq.is_finished}")
print(f"Completion token IDs: {vl_seq.completion_token_ids}")
```

--------------------------------

### BlockManager Initialization for KV Cache

Source: https://context7.com/odysseusq/vlcache/llms.txt

Initialize the BlockManager for managing KV cache blocks, including allocation and storage for partial recomputation. Key parameters are the total number of blocks, block size, and recompute ratio.

```python
from nanovllm.engine.block_manager import BlockManager, ImageKVEntry
from nanovllm.engine.sequence import Sequence
from nanovllm.sampling_params import SamplingParams

# Initialize block manager
block_manager = BlockManager(
    num_blocks=1000,          # Total number of KV cache blocks
    block_size=256,           # Tokens per block
    recompute_ratio=0.1       # Ratio of image tokens to recompute
)
```

--------------------------------

### Configure VLCache Recompute Ratio

Source: https://github.com/odysseusq/vlcache/blob/main/README.md

Adjust the recompute_ratio in nanovllm/config.py to control the balance between image token recomputation and cache reuse.

```text
Phase 1: Baseline  (recompute_ratio=0.0)
  [Req 1 cold] TTFT:   XXX.X ms  VIT: XX.X ms
  [Req 2 warm] TTFT:   XXX.X ms  VIT: XX.X ms

Phase 2: Partial recompute  (recompute_ratio=0.1)
  [Req 1 cold] TTFT:   XXX.X ms  VIT: XX.X ms
  [Req 2 warm] TTFT:    XX.X ms  VIT: XX.X ms

Summary
  Req2 warm speedup: X.XXx  (+XXX.X ms)
  PASS: Partial recompute significantly reduces warm TTFT
```

--------------------------------

### Benchmark Throughput

Source: https://context7.com/odysseusq/vlcache/llms.txt

Calculates tokens per second by running concurrent requests with variable input and output lengths.

```python
import os
import time
from random import randint, seed
from nanovllm import LLM, SamplingParams

def benchmark_throughput():
    seed(0)
    num_seqs = 256
    max_input_len = 1024
    max_output_len = 1024

    path = os.path.expanduser("~/huggingface/Qwen3-0.6B/")
    llm = LLM(path, enforce_eager=False, max_model_len=4096)

    # Generate random prompts with variable lengths
    prompt_token_ids = [
        [randint(0, 10000) for _ in range(randint(100, max_input_len))]
        for _ in range(num_seqs)
    ]

    # Variable output lengths per request
    sampling_params = [
        SamplingParams(
            temperature=0.6,
            ignore_eos=True,  # For consistent benchmarking
            max_tokens=randint(100, max_output_len)
        )
        for _ in range(num_seqs)
    ]

    # Warmup run
    llm.generate(["Benchmark warmup: "], SamplingParams())

    # Timed benchmark
    t = time.time()
    llm.generate(prompt_token_ids, sampling_params, use_tqdm=False)
    elapsed = time.time() - t

    total_tokens = sum(sp.max_tokens for sp in sampling_params)
    throughput = total_tokens / elapsed

    print(f"Total: {total_tokens} tokens")
    print(f"Time: {elapsed:.2f}s")
    print(f"Throughput: {throughput:.2f} tok/s")

    return throughput

# benchmark_throughput()
```

--------------------------------

### Configure Sampling Parameters for Text Generation

Source: https://context7.com/odysseusq/vlcache/llms.txt

Defines the `SamplingParams` dataclass for controlling text generation behavior. Adjust parameters like temperature, max_tokens, and ignore_eos to fine-tune the output.

```python
from nanovllm import SamplingParams
```

--------------------------------

### Benchmark Recompute Ratio

Source: https://context7.com/odysseusq/vlcache/llms.txt

Measures Time-To-First-Token (TTFT) for cold and warm requests to evaluate the efficiency of partial image KV cache recomputation.

```python
# Programmatic benchmarking
import os
from nanovllm import LLM, SamplingParams
from transformers import AutoProcessor
from qwen_vl_utils import process_vision_info

MODEL_PATH = "/path/to/Qwen2.5-VL-3B-Instruct"
IMAGE_PATH = "/path/to/test_image.png"

def benchmark_recompute_ratio(recompute_ratio: float):
    """Benchmark TTFT with specified recompute ratio."""
    processor = AutoProcessor.from_pretrained(MODEL_PATH)
    llm = LLM(MODEL_PATH, enforce_eager=True, recompute_ratio=recompute_ratio)

    def build_inputs(prompt_text):
        messages = [{"role": "user", "content": [
            {"type": "image", "image": IMAGE_PATH},
            {"type": "text", "text": prompt_text},
        ]}]
        text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
        image_inputs, video_inputs = process_vision_info(messages)
        inputs = processor(text=[text], images=image_inputs, videos=video_inputs,
                          padding=True, return_tensors="pt")
        return inputs.input_ids[0].tolist(), {
            "pixel_values": inputs.pixel_values,
            "image_grid_thw": inputs.image_grid_thw
        }

    sampling_params = SamplingParams(temperature=0.6, max_tokens=128)

    # Request 1 (cold): populates image KV cache
    prompt1, mm1 = build_inputs("Describe this image in detail.")
    out1 = llm.generate([prompt1], sampling_params, mm_inputs=[mm1], use_tqdm=False)

    # Request 2 (warm): reuses cached image KV
    prompt2, mm2 = build_inputs("What colors are in this image?")
    out2 = llm.generate([prompt2], sampling_params, mm_inputs=[mm2], use_tqdm=False)

    return {
        "recompute_ratio": recompute_ratio,
        "cold_ttft": out1[0]["ttft"],
        "warm_ttft": out2[0]["ttft"],
        "cold_vit": out1[0]["vit_time"],
        "warm_vit": out2[0]["vit_time"],
    }

# Compare baseline vs partial recompute
# baseline = benchmark_recompute_ratio(0.0)
# partial = benchmark_recompute_ratio(0.1)
# speedup = baseline["warm_ttft"] / partial["warm_ttft"]
# print(f"Warm TTFT speedup: {speedup:.2f}x")
```

--------------------------------

### Create Text-Only Sequence

Source: https://context7.com/odysseusq/vlcache/llms.txt

Initializes a Sequence object for standard text-only inference, setting up token IDs and sampling parameters.

```python
from nanovllm.engine.sequence import Sequence, SequenceStatus
from nanovllm.sampling_params import SamplingParams

# Create a sequence for text-only inference
token_ids = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
sampling_params = SamplingParams(temperature=0.6, max_tokens=100)
seq = Sequence(token_ids, sampling_params)

print(f"Sequence ID: {seq.seq_id}")
print(f"Status: {seq.status}")  # SequenceStatus.WAITING
print(f"Total tokens: {len(seq)}")
print(f"Prompt tokens: {seq.num_prompt_tokens}")
print(f"Completion tokens: {seq.num_completion_tokens}")  # 0 initially
```

--------------------------------

### EncoderCacheManager Initialization

Source: https://context7.com/odysseusq/vlcache/llms.txt

Initialize the EncoderCacheManager for caching vision encoder outputs. Key parameters include the number of blocks, block size, hidden dimension, data type, and device.

```python
import torch
from nanovllm.engine.encoder_cache_manager import EncoderCacheManager

# Initialize encoder cache manager
cache_manager = EncoderCacheManager(
    num_blocks=100,           # Number of cache blocks
    block_size=256,           # Tokens per block
    hidden_size=3584,         # Model hidden dimension
    dtype=torch.bfloat16,     # Data type for cache
    device="cuda"             # Device for cache storage
)
```

--------------------------------

### Variable Sampling Parameters per Request

Source: https://context7.com/odysseusq/vlcache/llms.txt

Generate a list of SamplingParams, each with a random max_tokens value, for use in multi-request generation scenarios.

```python
from random import randint, seed

seed(0)
num_seqs = 256
sampling_params_list = [
    SamplingParams(
        temperature=0.6,
        ignore_eos=True,
        max_tokens=randint(100, 1024)
    )
    for _ in range(num_seqs)
]
```

--------------------------------

### Cite VLCache Paper

Source: https://github.com/odysseusq/vlcache/blob/main/README.md

Use this BibTeX entry to cite the VLCache research paper in academic work.

```bibtex
@article{qin2025vlcache,
  title={VLCache: Computing 2% Vision Tokens and Reusing 98% for Vision-Language Inference},
  author={Qin, Shengling and Yu, Hao and Wu, Chenxin and Li, Zheng and Cao, Yizhong and Zhuge, Zhengyang and Zhou, Yuxin and Yao, Wentao and Zhang, Yi and Wang, Zhengheng and Bai, Shuai and Zhang, Jianwei and Lin, Junyang},
  journal={arXiv preprint arXiv:2512.12977},
  year={2025}
}
```

--------------------------------

### Compute Image Hash for Encoder Cache

Source: https://context7.com/odysseusq/vlcache/llms.txt

Calculate a content hash for an image using its pixel values and grid information. This hash is used to identify cached image embeddings.

```python
# Compute hash for an image (pixel values + grid info)
pixel_values = torch.randn(1024, 1176, device="cuda")  # Example image tokens
grid_thw = torch.tensor([[1, 32, 32]], device="cuda")  # temporal, height, width
image_hash = EncoderCacheManager.compute_hash(pixel_values, grid_thw[0])
```

--------------------------------

### Encoder Cache Write Operation

Source: https://context7.com/odysseusq/vlcache/llms.txt

Handle a cache miss by allocating blocks for new image embeddings, computing them (from a visual encoder), and writing them to the cache. This ensures that newly computed embeddings are stored for future use.

```python
else:
    # Cache miss - compute embeddings and store
    num_tokens = 1024
    if cache_manager.can_allocate(num_tokens):
        block_ids = cache_manager.allocate(image_hash, num_tokens)
        # embeddings would come from visual encoder
        embeddings = torch.randn(num_tokens, 3584, device="cuda", dtype=torch.bfloat16)
        cache_manager.write(block_ids, embeddings)
        print(f"Stored {num_tokens} embeddings in {len(block_ids)} blocks")
```

--------------------------------

### Encoder Cache Read Operation

Source: https://context7.com/odysseusq/vlcache/llms.txt

Check if image embeddings are cached using their hash. If a cache hit occurs, retrieve the embeddings from the cache using the block IDs and number of tokens.

```python
# Check if image embeddings are cached
block_ids = cache_manager.get_block_ids(image_hash)
if block_ids is not None:
    # Cache hit - read embeddings from cache
    num_tokens = 1024
    cached_embeds = cache_manager.read(block_ids, num_tokens)
    print(f"Cache hit! Retrieved {cached_embeds.shape} embeddings")
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.