### Run VibeServe Quickstart

Source: https://github.com/uw-syfi/vibe-serve/blob/main/README.md

Launches VibeServe with specified configurations for a quick experiment. This example uses the issue-tracker outer loop, Codex CLI, Docker on local CUDA, and runs for 4 rounds.

```bash
vibe-serve \
  --ref examples/moonshine-streaming/reference \
  --acc-checker examples/moonshine-streaming/accuracy_checker \
  --bench examples/moonshine-streaming/benchmark \
  --exp-name my-experiment \
  --docker \
  --agent-backend cli --cli-provider codex \
  --max-rounds 4 \
  --modality speech_to_text
```

--------------------------------

### FlashAttention Installation and Import

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/backends/flashattention.md

Installation commands and import statements for different versions of FlashAttention.

```bash
pip install flash-attn
```

```bash
cd hopper && python setup.py install
```

```bash
pip install flash-attn-4
```

```python
from flash_attn import ...
```

```python
import flash_attn_interface
```

```python
from flash_attn.cute import ...
```

--------------------------------

### Install Dependencies

Source: https://github.com/uw-syfi/vibe-serve/blob/main/README.md

Installs project dependencies and copies configuration files. Ensure provider keys are set in the .env file.

```bash
uv sync
cp .env.example .env       # provider keys (Anthropic / OpenAI / Vertex / …)
cp agent.toml.example agent.toml
```

--------------------------------

### Reference Repo Path Convention Example

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/CLAUDE.md

Defines the base path for reference repositories and provides an example grep recipe for searching within the vLLM repository.

```bash
$SERVE_REPOS = <vibe-serve-root>/skills/serving-systems/repos

rg "register.*backend" $SERVE_REPOS/vllm/vllm/v1/attention/backends/
```

--------------------------------

### SKILL.md Frontmatter Example

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/CLAUDE.md

Example of the required YAML frontmatter for the SKILL.md file, including name and description. The description should be concise and include trigger keywords.

```yaml
---
name: serving-systems
description: >-
  ~250-400 characters. Lead with what this covers, then list trigger
  keywords. Hard cap is 1024 chars per the agentskills spec, but stay
  near 100 tokens — this field is in the always-loaded metadata pool.
---
```

--------------------------------

### FlashInfer Setup

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/backends/flashinfer.md

Basic import statement for using FlashInfer and its paging module.

```python
import flashinfer
import flashinfer.page

```

--------------------------------

### Prefill Example for `get_batch_indices_positions`

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/backends/flashinfer.md

Example demonstrating `get_batch_indices_positions` for a single request during prefill. `batch_indices` will contain all zeros, and `positions` will be sequential.

```python
qo_indptr = torch.tensor([0, 20], dtype=torch.int32, device=device)
seq_lens = torch.tensor([20], dtype=torch.int32, device=device)
batch_indices, positions = flashinfer.page.get_batch_indices_positions(qo_indptr, seq_lens, 20)
# batch_indices = [0, 0, 0, ..., 0]  (20 zeros)
# positions = [0, 1, 2, ..., 19]
```

--------------------------------

### Example Output Directory Structure

Source: https://github.com/uw-syfi/vibe-serve/blob/main/README.md

Shows the directory structure created for each run, containing the workspace, logs, and a snapshot of the reference implementation.

```text
exp_env/<run>/
├── workspace/                # the unified, git-tracked workspace (each round = one commit)
├── logs/
│   ├── run-*.log             # top-level run log
│   ├── run-*-roundNNN.log    # per-round agent log (agent loop)
│   ├── progress.md           # long-term memory file the Orchestrator reads/edits
│   ├── rounds.json           # per-round audit
│   ├── state.json            # cursor (plain loop)
│   ├── issues.json           # IssueBoard (plain loop)
│   ├── population.json       # Individual list (evolve loop)
│   └── docker.log
└── reference/                # snapshot of --ref at start
```

--------------------------------

### Setup TensorRT-LLM Repository Path

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/engines/trtllm.md

Sets the SERVE_REPOS environment variable to point to the TensorRT-LLM repository. This is a prerequisite for accessing the source code.

```bash
export SERVE_REPOS=<vibe-serve-root>/skills/serving-systems/repos
```

--------------------------------

### Algorithm Compatibility Matrix Example

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/CLAUDE.md

Presents the compatibility matrix format found in algorithm reference files, detailing implementations, engines, backends, and hardware support.

```markdown
## Compatibility

| Implementation | Engine | Backend / library | Hardware |
|:--|:--|:--|:--|
| FlashInfer paged KV attention | SGLang, vLLM | flashinfer | NVIDIA (sm_80+) |
| FA3 variable-length | vLLM v1 | flashattention | NVIDIA Hopper+
```

--------------------------------

### Example Target Directory Structure

Source: https://github.com/uw-syfi/vibe-serve/blob/main/README.md

Illustrates the standard directory layout for each evaluation target, including objective files, reference implementations, accuracy checkers, benchmarks, and READMEs.

```text
examples/<name>/
├── OBJECTIVE.md          # free-form deployment goal (model + hardware + workload + interface)
├── reference/            # reference HuggingFace Transformers implementation
│   ├── reference.py
│   ├── config.json
│   └── meta.json         # model id + revision
├── accuracy_checker/     # checker.py + tests/data — the correctness gate
├── benchmark/            # benchmark.py + load levels — emits the metric to optimize
└── README.md             # human-readable description
```

--------------------------------

### vLLM Benchmark Command

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/tooling/serving-benchmark.md

Example command to run a benchmark using vLLM. Requires specifying the model, dataset, number of prompts, and request rate.

```bash
python benchmarks/benchmark_serving.py \
    --model <model> \
    --dataset-name sharegpt \
    --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
    --num-prompts 500 \
    --request-rate 4.0
```

--------------------------------

### SGLang Open-Loop Benchmark Command

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/tooling/serving-benchmark.md

Example command to run an open-loop benchmark using SGLang. Specifies dataset, sequence lengths, number of prompts, and request rate.

```bash
python -m sglang.bench_serving \
    --backend sglang \
    --dataset-name random \
    --random-input-len 1024 --random-output-len 256 \
    --num-prompts 500 \
    --request-rate 4.0 \
    --host localhost --port 30000
```

--------------------------------

### Example Batch Arrays for Paged KV Cache

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/backends/flashinfer.md

Illustrates the expected output for batch arrays given a specific state of active requests and page size.

```text
kv_indptr        = [0, 2, 3, 6]
kv_indices       = [0, 1, 2, 3, 4, 5]
kv_last_page_len = [4, 10, 3]

```

--------------------------------

### Engine Reference Table Example

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/CLAUDE.md

Illustrates the 'Where's X' table format used in engine reference files to map needs to paths within specific engine repositories.

```markdown
| Need | Path in repos/<engine>/ |
|:-----|:------------------------|
| Attention backends | python/sglang/srt/layers/attention/ |
| Scheduler | python/sglang/srt/managers/scheduler.py |
```

--------------------------------

### Image Preprocessing for LLaVA/LLaVA-NeXT

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/tooling/io-handling.md

Example of key image preprocessing operations: loading, resizing, normalizing, and patchifying. These steps are crucial for models like LLaVA and LLaVA-NeXT. Avoid slow PIL paths in hot serving.

```python
# Load
img = PIL.Image.open(path).convert("RGB")
# Resize (model-specific)
img = img.resize((H, W), resample=PIL.Image.BICUBIC)
# Normalize
arr = (np.asarray(img, dtype=np.float32) / 255.0 - mean) / std
# Patchify: (H, W, 3) → (H/P, W/P, P*P*3)
patches = arr.reshape(H//P, P, W//P, P, 3).transpose(0, 2, 1, 3, 4).reshape(H//P * W//P, P*P*3)
```

--------------------------------

### Tokenizer Info Setup for Structured Output

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/algorithms/structured-output.md

Initializes TokenizerInfo, ensuring the correct vocabulary size is used. Always pass the model's vocab_size (lm_head.out_features), not the tokenizer's length, to prevent incorrect logit masking.

```python
tokenizer_info = xgrammar.TokenizerInfo.from_huggingface(
    tokenizer, vocab_size=model.config.vocab_size, stop_token_ids=[eos_id],
)
```

--------------------------------

### Video Deletion Response Example

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/tooling/openai-api.md

Example of the JSON response when a video generation job is successfully deleted.

```json
{"id": "...", "object": "video", "deleted": true}
```

--------------------------------

### Realtime Session Creation Request Example

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/tooling/openai-api.md

Example of a JSON request to create a realtime session. It allows specifying the model, modalities, and audio formats.

```json
{
  "model": "...",
  "modalities": ["text", "audio"],
  "instructions": "You are a helpful assistant.",
  "voice": "alloy",
  "input_audio_format": "pcm16",
  "output_audio_format": "pcm16"
}
```

--------------------------------

### Decode Example for `get_batch_indices_positions`

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/backends/flashinfer.md

Example demonstrating `get_batch_indices_positions` for multiple requests during decoding. `batch_indices` map tokens to requests, and `positions` indicate the end of the sequence for each request.

```python
append_indptr = torch.tensor([0, 1, 2, 3], dtype=torch.int32, device=device)  # 1 token each
seq_lens = torch.tensor([21, 11, 36], dtype=torch.int32, device=device)
batch_indices, positions = flashinfer.page.get_batch_indices_positions(append_indptr, seq_lens, 3)
# batch_indices = [0, 1, 2]
# positions = [20, 10, 35]   (= seq_len - 1 for each)
```

--------------------------------

### Speech-to-Text Streaming Response Example

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/tooling/openai-api.md

Example of Server-Sent Events (SSE) for streaming transcription results. It shows incremental text updates and a final completion message.

```text
data: {"type":"transcript.text.delta","delta":"partial "}

data: {"type":"transcript.text.done","text":"full."}

data: [DONE]

```

--------------------------------

### Registering a New Compute Backend Fragment Implementation

Source: https://github.com/uw-syfi/vibe-serve/blob/main/src/vibe_serve/templates/_backend/README.md

Shows how to define a concrete subclass for a new backend's fragments and register it.

```python
class RocmComputeBackendFragment(ComputeBackendFragment):
    backend = ComputeBackend.ROCM
```

--------------------------------

### Start VibeServe Issue MCP Server

Source: https://github.com/uw-syfi/vibe-serve/blob/main/README.md

Starts the issue MCP server, which is used by the plain outer loop. This is a separate entry point from the main vibe-serve command.

```bash
vibe-serve-issue-mcp                         # serves issues.json over MCP
```

--------------------------------

### Speech Synthesis Request Example

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/tooling/openai-api.md

Example of a JSON request to synthesize speech. Specify the model, input text, and desired voice. Optional parameters include response format and speed.

```json
{
  "model": "tts-1",
  "input": "Hello world",
  "voice": "alloy"
}
```

--------------------------------

### Prefill Wrapper `run()` Method

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/backends/flashinfer.md

Executes the prefill attention computation once per layer. The plan metadata persists across `run()` calls for different layers.

```python
# q shape: (total_tokens, num_qo_heads, head_dim)
attn_output = prefill_wrapper.run(q, kv_cache_for_this_layer)
# attn_output shape: (total_tokens, num_qo_heads, head_dim)
```

--------------------------------

### Grep for Engine and Launcher Entrypoints

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/engines/sglang.md

Search for the main entry points of the SGLang engine and server launcher, including Engine class definitions and launch functions.

```bash
rg "class Engine|def launch_engine|launch_server" \
   $SERVE_REPOS/sglang/python/sglang/srt/entrypoints/engine.py \
   $SERVE_REPOS/sglang/python/sglang/launch_server.py
```

--------------------------------

### VoxServe Serving Abstraction

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/models/speech-generation.md

Illustrates the consistent adapter pattern used in VoxServe for speech generation models, detailing the roles of preprocess, forward, sampling, and postprocess methods.

```python
preprocess()  — text / audio → token frames + input features + masks
forward()     — streaming generation via KV cache (+ depth cache for hierarchical)
sampling()    — logits → codec tokens (with repetition penalty, etc.)
postprocess() — codec tokens → audio via model-specific decoder
```

--------------------------------

### FA2 Aliased Unpadded Function

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/backends/flashattention.md

This is an alias for the FA1 `flash_attn_unpadded_func` for backward compatibility. Ensure `flash-attn` is installed.

```python
from flash_attn import flash_attn_unpadded_func

# Example usage (for back-compat)
# out = flash_attn_unpadded_func(q, k, v, ...)
```

--------------------------------

### Prefill Wrapper `plan()` Method

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/backends/flashinfer.md

Plans the prefill operation once per batch step before the layer loop. It configures attention parameters like number of heads, head dimension, page size, and causality.

```python
prefill_wrapper.plan(
    qo_indptr=qo_indptr,                     # int32: [0, prompt_len] for single request
    paged_kv_indptr=kv_indptr,                # from build_batch_arrays
    paged_kv_indices=kv_indices,              # from build_batch_arrays
    paged_kv_last_page_len=kv_last_page_len,  # from build_batch_arrays
    num_qo_heads=num_qo_heads,                # e.g. 32
    num_kv_heads=num_kv_heads,                # e.g. 8 (GQA)
    head_dim_qk=head_dim,                     # e.g. 64
    page_size=page_size,                      # e.g. 16
    causal=True,                              # standard causal attention
    q_data_type=dtype,                        # e.g. torch.float16
)
```

--------------------------------

### FA2 Fused-KV Variants

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/backends/flashattention.md

Use these functions when K and V share storage. Ensure `flash-attn` is installed.

```python
from flash_attn import flash_attn_kvpacked_func, flash_attn_varlen_kvpacked_func

# Example usage for packed KV
# kv_packed = torch.cat([k, v], dim=-1)
# out = flash_attn_kvpacked_func(q, kv_packed, ...)
# out_varlen = flash_attn_varlen_kvpacked_func(q, kv_packed, ...)
```

--------------------------------

### FA2 Fused-QKV Variants

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/backends/flashattention.md

These functions are for scenarios where the engine keeps Q/K/V in a single packed tensor. Ensure `flash-attn` is installed.

```python
from flash_attn import flash_attn_qkvpacked_func, flash_attn_varlen_qkvpacked_func

# Example usage for packed QKV
# qkv_packed = torch.cat([q, k, v], dim=-1)
# out = flash_attn_qkvpacked_func(qkv_packed, ...)
# out_varlen = flash_attn_varlen_qkvpacked_func(qkv_packed, ...)
```

--------------------------------

### Initialize and Fetch SGLang Repository

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/engines/sglang.md

Initialize a local sglang repository and fetch a specific pinned commit. This ensures that the repository state matches the version expected by the documentation.

```bash
mkdir -p "$SERVE_REPOS/sglang" && cd "$SERVE_REPOS/sglang"
git init -q
git remote add origin https://github.com/sgl-project/sglang.git
git fetch --depth 1 origin 04b1caf75b3c6f043a979ddce21d43ed07c217a6
git checkout -q FETCH_HEAD
```

--------------------------------

### Realtime WebSocket Connection URL

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/tooling/openai-api.md

Example of how to connect to the realtime WebSocket endpoint. The model name is passed as a query parameter.

```text
wss://api.openai.com/v1/realtime?model=<name>
```

--------------------------------

### JSON Completion Example

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/tooling/accuracy-checker.md

Tests JSON completion with punctuation-heavy input, including quotes and braces, to ensure correct tokenizer splits.

```python
('{\"name\": \"Alice\", \"age\":', 10, "JSON completion")
```

--------------------------------

### KV Cache Retention Configuration Example

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/algorithms/radix-prefix-caching.md

Demonstrates how to configure KV cache retention priorities and TTLs using `KvCacheRetentionConfig` and `TokenRangeRetentionConfig` in TensorRT-LLM.

```python
from tensorrt_llm.kv_cache_manager import KvCacheRetentionConfig, TokenRangeRetentionConfig

# Example: High priority for system prompt (tokens 0-1024) for 60 seconds
retention_config = KvCacheRetentionConfig(
    decode_retention_policy=TokenRangeRetentionConfig(priority=35, duration_ms=None), # Default for generated tokens
    input_retention_policy=TokenRangeRetentionConfig(priority=90, duration_ms=60000), # High priority for input tokens 0-1024
    secondary_offload_min_priority=50 # Only offload blocks with priority >= 50 to CPU
)

# Apply this config to your KV cache manager
```

--------------------------------

### Running the Qwen3-32B Code Edit Benchmark

Source: https://github.com/uw-syfi/vibe-serve/blob/main/examples/qwen3-32b-code-edit/README.md

This bash command launches the benchmark script. It specifies the server URL, model name, number of samples, and the output file for results. Ensure the server is running before execution.

```bash
uv run python inputs/qwen3-32b-code-edit/benchmark/benchmark.py \
    --url http://localhost:8000 --model qwen3-32b \
    --num-samples 100 \
    --output-json /tmp/code_edit_baseline.json
```

--------------------------------

### FA2 flash_attn_func for Dense Tensors

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/backends/flashattention.md

Use this API for dense attention kernels when dealing with equal-length batched tensors. Ensure `flash-attn` is installed and prerequisites are met.

```python
from flash_attn import flash_attn_func

# Example usage (q, k, v are torch.Tensors)
# out = flash_attn_func(q, k, v, ...)
```

--------------------------------

### Resume a Vibe Serve Run

Source: https://github.com/uw-syfi/vibe-serve/blob/main/README.md

Demonstrates how to resume a previous vibe-serve run using the --resume flag, either for the latest run or a specific run directory.

```bash
vibe-serve --resume                  # newest run
vibe-serve --resume 20260507-...     # specific dir
```

--------------------------------

### Replay Method - Plan, Replay, and Slice

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/backends/cuda-graph.md

Executes the plan for the decode wrapper, replays the CUDA graph, and returns the static logits for the actual batch size.

```python
self.decode_wrappers[padded_bs].plan(...)  # OUTSIDE graph
self.graphs[padded_bs].replay()
return self.static_logits[padded_bs][:actual_bs]
```

--------------------------------

### Video Generation Job Submission Response

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/tooling/openai-api.md

Example of the JSON response received after submitting a video generation job. It includes the job ID, status, and other metadata.

```json
{
  "id": "video_<hex>", 
  "object": "video", 
  "created_at": 1700000000,
  "status": "queued", 
  "model": "...", 
  "progress": 0,
  "seconds": 4, 
  "size": "1280x720"
}
```

--------------------------------

### Run Show-o2 1.5B HQ Benchmark

Source: https://github.com/uw-syfi/vibe-serve/blob/main/examples/show-o2-1.5B-HQ/benchmark/README.md

Executes the benchmark script with specified parameters for running and collecting data. Use this to test model performance and output.

```bash
python benchmark.py \
  --url http://localhost:8000 \
  --warmup-requests 1 \
  --closed-loop \
  --collect-server-timings \
  --save-images-dir /tmp/show_o2_images \
  --postprocess-mode native \
  --prompt "a small red robot holding a handwritten sign that says VibeServe" \
  --request-seed 1234 \
  --fixed-request-seed \
  --num-requests 5 \
  --steps 4
```

--------------------------------

### Get FlashAttention Kernel with HuggingFace Kernels

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/backends/flashattention.md

Use this snippet to obtain a FlashAttention kernel instance from the HuggingFace kernels library. This is convenient for notebooks and rapid iteration.

```python
from kernels import get_kernel
fa = get_kernel("kernels-community/flash-attn2")      # or "flash-attn3"
```

--------------------------------

### Run Olmo-Hybrid-7B Prefix-Caching Benchmark

Source: https://github.com/uw-syfi/vibe-serve/blob/main/examples/olmo-hybrid-prefix-caching/benchmark/README.md

Execute the benchmark script for the Olmo-Hybrid-7B prefix-caching workload against a local server.

```bash
python benchmark.py --url http://localhost:8000
```

--------------------------------

### Apply Chat Template for Tokenization

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/tooling/io-handling.md

Use `tokenizer.apply_chat_template` to format chat messages exactly once. This example shows how to apply the template and tokenize the resulting prompt for input IDs.

```python
prompt = tokenizer.apply_chat_template(
    messages=[
        {"role": "system", "content": "..."},
        {"role": "user",   "content": "..."},
    ],
    tokenize=False,
    add_generation_prompt=True,
)
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
```

--------------------------------

### Audio Preprocessing for Whisper-style Models

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/tooling/io-handling.md

Demonstrates audio preprocessing steps including loading, padding/trimming, and computing log-mel spectrograms. This is typical for Whisper-style models.

```python
# Load audio
waveform, sr = librosa.load(path, sr=16000, mono=True)
# Pad or trim to 30s chunks
# Compute log-mel spectrogram (80 or 128 mel bins)
mel = librosa.feature.melspectrogram(y=waveform, sr=16000, n_fft=400, hop_length=160, n_mels=128)
log_mel = np.log(np.maximum(mel, 1e-10))
```

--------------------------------

### Backend Reference Out of Scope Example

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/CLAUDE.md

Shows the 'Out of scope' section for kernel implementation in backend reference files, directing users to agent-gpu-skills for new kernel writing.

```markdown
## Out of scope — kernel implementation

For writing new kernels (not using this library): see agent-gpu-skills's
triton-skill / cutlass-skill / cuda-skill.
```

--------------------------------

### Local HTTP Smoke Test for Show-o2

Source: https://github.com/uw-syfi/vibe-serve/blob/main/examples/show-o2-1.5B-HQ-h100/README.md

Run a local HTTP server with mock weights for a quick smoke test. This command does not download model weights.

```bash
uv run python playground/show_o2_local/serve.py --mock --port 8000
```

--------------------------------

### Initialize Git Submodules

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/README.md

Initialize the git submodules for the reference engines. This command is necessary to fetch the source code for vLLM, SGLang, and TensorRT-LLM.

```bash
git submodule update --init skills/serving-systems/repos
```

--------------------------------

### Loading and Generating Text with mlx-lm

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/frameworks/mlx.md

Demonstrates the basic usage of the `mlx-lm` library to load a pre-quantized model and generate text based on a prompt.

```python
from mlx_lm import load, generate
model, tokenizer = load("mlx-community/Llama-3.2-3B-Instruct-4bit")
response = generate(model, tokenizer, prompt="Hello", max_tokens=256)
```

--------------------------------

### Get Padded Batch Size Function

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/backends/cuda-graph.md

Determines the smallest captured batch size that is greater than or equal to the actual batch size. Returns None for eager fallback.

```python
def get_padded_batch_size(self, actual_bs):
    for bs in self.batch_sizes:  # sorted ascending
        if bs >= actual_bs:
            return bs
    return None
```

--------------------------------

### FA3 Import and Function Call

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/backends/flashattention.md

This demonstrates importing and calling the attention function from FA3, which is optimized for Hopper hardware and supports FP8. Ensure FA3 is installed correctly from the source.

```python
import flash_attn_interface

# Example usage (q, k, v are torch.Tensors)
# out = flash_attn_interface.flash_attn_func(q, k, v, ...)
```

--------------------------------

### Autotune Context Manager for FlashInfer Kernels

Source: https://github.com/uw-syfi/vibe-serve/blob/main/resources/skills/serving-systems/references/backends/flashinfer.md

Shows how to use the autotune context manager to gate autotuning for kernels like GEMM, MoE, and FP4, ensuring it does not run inside CUDA graph capture.

```python
with flashinfer.autotune():
    warmup_pass(dummy_inputs)   # autotune runs
    # persist with flashinfer.autotune().save() or equivalent
# Autotune disabled for subsequent captured regions
```