### Running Candle-VLLM Examples

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/rust_crate.md

Command to run the release build of candle-vllm examples with CUDA support enabled.

```bash
cargo run --release --features cuda --example simple_gen
```

--------------------------------

### Start Qwen3-VL Server

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/multimodal.md

Launches the candle-vllm server with the Qwen3-VL model. Ensure CUDA is enabled if using GPU.

```bash
cargo run --release --features cuda -- \
  --m Qwen/Qwen3-VL-8B-Instruct \
  --p 8000
```

--------------------------------

### Run OpenCode

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/opencode.md

Starts the OpenCode application using the configured settings.

```bash
opencode
```

--------------------------------

### Install xbot CLI

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/xbot.md

Installs the xbot command-line interface. Choose either the Cargo or npm installation method.

```bash
cargo install xbot
```

```bash
npm install -g @trusted-ai/xbot
```

--------------------------------

### Install OpenCode

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/opencode.md

Installs the OpenCode CLI tool. Use either the curl script for direct installation or npm for package management.

```bash
curl -fsSL https://opencode.ai/install | bash
```

```bash
npm i -g opencode-ai
```

--------------------------------

### Install Candle-vLLM via Script

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Use the provided shell script for a quick installation on supported systems.

```bash
curl -sSL https://ericlbuehler.github.io/candle-vllm/install.sh | bash
```

--------------------------------

### Install Kilo Code CLI

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/kilocode.md

Installs the Kilo Code command-line interface globally using npm.

```bash
npm install -g @kilocode/cli
```

--------------------------------

### CLI Configuration for Single MCP Server

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/mcp_tool_calling.md

This command-line example shows how to configure a single stdio MCP server using `npx` to run the `@modelcontextprotocol/server-filesystem`.

```bash
cargo run --release -- --p 8000 \
  --mcp-command npx \
  --mcp-args "-y @modelcontextprotocol/server-filesystem /tmp"
```

--------------------------------

### Start Candle-vLLM Server

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/xbot.md

Launches the Candle-vLLM server with specified features and model. Ensure CUDA and NCCL are enabled if using GPU.

```bash
cargo run --release --features cuda,nccl,flashinfer,cutlass -- \
  --m Qwen/Qwen3.6-27B-FP8 \
  --p 8000 \
  --kv-fraction 0.6
```

--------------------------------

### Start Candle-vLLM Server

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/opencode.md

Launches the Candle-vLLM server with specified model and features. Ensure CUDA and NCCL are enabled if using GPU.

```bash
cargo run --release --features cuda,nccl,flashinfer,cutlass -- \
  --m Qwen/Qwen3.6-27B-FP8 \
  --d 0 \
  --p 8000 \
  --kv-fraction 0.6 \
  --enforce-parser qwen_coder
```

--------------------------------

### Configure Server Binding

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Examples of various ways to bind the candle-vllm server using different address formats and ports.

```bash
candle-vllm --h 127.0.0.1 --p 8000 --m Qwen/Qwen3.6-27B-FP8
candle-vllm --h 127.0.0.1:8000 --m Qwen/Qwen3.6-27B-FP8
candle-vllm --h '[::1]:8000' --m Qwen/Qwen3.6-27B-FP8
candle-vllm --h unix:///tmp/candle-vllm.sock --m Qwen/Qwen3.6-27B-FP8
```

--------------------------------

### Start Gemma3 Vision Server

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/multimodal.md

Launches the candle-vllm server with a Gemma3 vision model. CUDA is required for GPU acceleration.

```bash
cargo run --release --features cuda -- \
  --m google/gemma-3-4b-it \
  --p 8000
```

--------------------------------

### Start candle-vllm Server with Metal

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/embedding.md

Launches the candle-vllm server using Metal acceleration. Ensure the model path and port are correctly specified.

```bash
cargo run --release --features metal -- --m google/gemma-3-4b-it --p 8000
```

--------------------------------

### Start candle-vllm Server with CUDA

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/embedding.md

Launches the candle-vllm server using CUDA acceleration. Ensure the model path and port are correctly specified.

```bash
cargo run --release --features cuda -- --m Qwen/Qwen3-0.6B --p 8000
```

--------------------------------

### Rust API Example for Embeddings

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/embedding.md

Programmatic generation of embeddings using the candle-vllm Rust API. This example initializes the engine, creates an embedding request, and prints the number of embeddings generated.

```rust
use candle_vllm::api::{EngineBuilder, ModelRepo};
use candle_vllm::openai::requests::{EmbeddingInput, EmbeddingRequest};

#[tokio::main]
async fn main() -> candle_core::Result<()> {
    let engine = EngineBuilder::new(ModelRepo::ModelID(("Qwen/Qwen3-0.6B", None)))
        .build_async()
        .await?;

    let request = EmbeddingRequest {
        model: Some("default".to_string()),
        input: EmbeddingInput::String("hello world".to_string()),
        encoding_format: Default::default(),
        embedding_type: Default::default(),
    };

    let response = engine.embed(request)?;
    println!("embeddings: {}", response.data.len());
    engine.shutdown();
    Ok(())
}
```

--------------------------------

### OpenAI-style Tool Calling Request Example

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/mcp_tool_calling.md

This JSON structure represents a user's request to the model, including a message and a list of available tools the model can choose to call.

```json
{
  "model": "default",
  "messages": [
    {"role": "user", "content": "List files in the current directory"}
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "list_files",
        "description": "List files in a directory",
        "parameters": {
          "type": "object",
          "properties": {
            "path": {"type": "string"}
          },
          "required": ["path"]
        }
      }
    }
  ]
}
```

--------------------------------

### Request Base64 Embeddings with Last-Token Pooling

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/embedding.md

Example cURL request to the /v1/embeddings endpoint for base64 encoded embeddings using last-token pooling. This example demonstrates processing multiple inputs.

```bash
curl -X POST http://localhost:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input":["hello","hola"],"embedding_type":"last","encoding_format":"base64"}'
```

--------------------------------

### Configure xbot for Candle-vLLM

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/xbot.md

Guides through the interactive configuration of xbot to use a custom provider pointing to the Candle-vLLM API. It prompts for a provider name, API base URL, and default model.

```shell
> Select provider to configure: custom
> Enter a unique name for this custom provider: candle-vllm
> Enter API Base URL (e.g. https://api.yourprovider.com/v1): http://localhost:8000/v1/
> Enter API Key? (Optional for local/custom) No
Fetching available models...
> Select default model: Qwen3.6-27B-FP8
Using contextWindowTokens from model metadata: 262144
> Configure subagent provider/model now? No

Configuration saved successfully!
Config file: /root/.xbot/config.json

Final Provider Config:
{
  "api_key": "",
  "api_base": "http://localhost:8000/v1/",
  "extra_headers": {},
  "reasoning_effort": null
}
```

--------------------------------

### Text Generation with Candle-VLLM

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/rust_crate.md

Example of performing text generation using the candle-vllm Rust API. This requires setting up the engine and constructing a chat completion request.

```rust
use candle_vllm::api::{EngineBuilder, ModelRepo};
use candle_vllm::openai::requests::{ChatCompletionRequest, Messages};

#[tokio::main]
async fn main() -> candle_core::Result<()> {
    let engine = EngineBuilder::new(ModelRepo::ModelID(("Qwen/Qwen3-0.6B", None)))
        .build_async()
        .await?;

    let request = ChatCompletionRequest {
        model: Some("default".to_string()),
        messages: Messages::Map(vec![std::collections::HashMap::from([
            ("role".to_string(), "user".to_string()),
            ("content".to_string(), "Say hello from the Rust API.".to_string()),
        ])]),
        max_tokens: Some(64),
        ..Default::default()
    };

    let response = engine.generate_request(request).await?;
    println!("{response:?}");
    engine.shutdown();
    Ok(())
}
```

--------------------------------

### Embeddings Generation with Candle-VLLM

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/rust_crate.md

Example of generating embeddings using the candle-vllm Rust API. This involves creating an embedding request and passing it to the engine.

```rust
use candle_vllm::openai::requests::{EmbeddingInput, EmbeddingRequest};

let request = EmbeddingRequest {
    model: Some("default".to_string()),
    input: EmbeddingInput::String("hello world".to_string()),
    encoding_format: Default::default(),
    embedding_type: Default::default(),
};

let response = engine.embed(request)?;
```

--------------------------------

### Multimodal Request with Image URL

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/multimodal.md

Example JSON payload for a chat completion request including text and an image URL. The model must be multimodal.

```json
{
  "model": "default",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "Describe this image"},
        {"type": "image_url", "image_url": "https://example.com/cat.png"}
      ]
    }
  ]
}
```

--------------------------------

### Multimodal Request with Base64 Image

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/multimodal.md

Example JSON payload for a chat completion request with text and a base64 encoded image. Ensure the base64 string is correctly formatted.

```json
{
  "model": "default",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "Describe this image"},
        {"type": "image_base64", "image_base64": "data:image/png;base64,..."}
      ]
    }
  ]
}
```

--------------------------------

### Prefix Cache Usage Reporting Example

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/prefix_cache.md

This JSON structure shows how prefix cache and reasoning token details are reported in OpenAI-compatible chat responses when they are non-zero. `cached_tokens` indicates reused prompt tokens, and `reasoning_tokens` indicates generated tokens within reasoning blocks.

```json
{
  "usage": {
    "prompt_tokens": 128,
    "completion_tokens": 64,
    "total_tokens": 192,
    "prompt_time_costs": 4,
    "completion_time_costs": 250,
    "prompt_tokens_details": {
      "cached_tokens": 64
    },
    "completion_tokens_details": {
      "reasoning_tokens": 32
    }
  }
}
```

--------------------------------

### Request Float Embeddings with Mean Pooling

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/embedding.md

Example cURL request to the /v1/embeddings endpoint for float embeddings using mean pooling. The input can be a single string or an array of strings.

```bash
curl -X POST http://localhost:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input":"hello world","model":"default","embedding_type":"mean","encoding_format":"float"}'
```

--------------------------------

### Build Candle-vLLM from Source

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Compile the project from source for specific hardware configurations like CUDA or Metal.

```bash
git clone git@github.com:EricLBuehler/candle-vllm.git
cd candle-vllm

# CUDA (11+, 12+, 13.0) — remove flashinfer,cutlass for sm_70/sm_75
cargo install --features cuda,nccl,flashinfer,cutlass --path .

# macOS/Metal
cargo install --features metal --path .
```

--------------------------------

### Run Models via CLI

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Basic commands to launch models with different quantization and hardware configurations.

```bash
# FP8 model + Web UI
candle-vllm --m Qwen/Qwen3.8-27B-FP8 --ui-server

# GLM-5.2 FP8 model
candle-vllm --d 0,1,2,3,4,5,6,7 --m zai-org/GLM-5.2-FP8 --ui-server

# Unquantized Safetensors (multi-GPU)
candle-vllm --d 0,1 --w /home/Qwen3-30B-A3B-Instruct-2507/

# ISQ on-the-fly quantization
candle-vllm --m Qwen/Qwen3.6-27B --isq q4k

# FP4 Model
candle-vllm --m GadflyII/GLM-4.7-Flash-NVFP4 --ui-server

# GGUF model
candle-vllm --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server

# Manual YaRN scaling
candle-vllm --m Qwen/Qwen3.6-35B-A3B --yarn-scaling-factor 4.0 --ui-server
```

--------------------------------

### Running with MCP Config File

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/mcp_tool_calling.md

This command demonstrates how to run `candle-vllm` using a specified MCP configuration file.

```bash
cargo run --release -- --p 8000 --mcp-config mcp_config.json
```

--------------------------------

### Run xbot Tasks

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/xbot.md

Executes xbot for one-shot tasks or interactive sessions. The 'chat /init' command scans the project and initializes an agent file.

```bash
cd YOUR PROJECT
# scan project and init agent file
xbot chat /init
# real task
xbot chat "find bugs in this project."
```

```bash
cd YOUR PROJECT
# you may add .xbot workspace folder to .gitignore
xbot repl
```

--------------------------------

### Build Candle-vLLM Docker Image

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Build a Docker container with custom CUDA and SM version parameters.

```bash
# Pass custom SM version and CUDA version: ./build_docker.sh "cuda,nccl,flashinfer,cutlass" sm_90 13.0.0
./build_docker.sh "cuda,nccl,flashinfer,cutlass"
```

--------------------------------

### Configure OpenCode for Candle-vLLM

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/opencode.md

Sets up the OpenCode configuration file to use a local Candle-vLLM instance. Specify the baseURL and the model name matching the served model.

```json
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "local-candle-vllm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Candle-vLLM Local",
      "options": {
        "baseURL": "http://localhost:8000/v1"
      },
      "models": {
        "qwen3-coder": {
          "name": "Qwen/Qwen3.6-27B-FP8"
        }
      }
    }
  },
  "model": "local-candle-vllm/qwen3-coder"
}
```

--------------------------------

### Run FP8 and FP4 Models

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Commands for running high-precision and quantized models, including specific hardware optimizations.

```bash
# FP8 Model (block-wise quant, build with cutlass feature)
candle-vllm --m Qwen/Qwen3.8-27B-FP8 --ui-server

# Faster GDN prefill on Hopper with slight precision loss
SM90_LOWER_PRECISION_GDN_PREFILL=1 candle-vllm --m Qwen/Qwen3.5-35B-A3B-FP8

# Qwen3.5 MTP speculative decoding (2 draft tokens per step)
candle-vllm --w /data/Qwen3.5-35B-A3B-FP8/ --mtp 2 --ui-server

# GLM-5.2 FP8 Model
candle-vllm --d 0,1,2,3,4,5,6,7 --m zai-org/GLM-5.2-FP8 --ui-server

# FP8 on MacOS/Metal (Dense)
candle-vllm --m Qwen/Qwen3-4B-Instruct-2507-FP8 --ui-server

# FP4 Model (MXFP4/NVFP4, MLX quantized format not supported)
candle-vllm --m GadflyII/GLM-4.7-Flash-NVFP4 --ui-server

# MXFP4
candle-vllm --m nm-testing/Qwen3-30B-A3B-MXFP4A16 --ui-server
```

--------------------------------

### Run GPTQ, AWQ, and Marlin Models

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Commands for running compressed models and converting them to Marlin-compatible formats.

```bash
# Marlin-compatible GPTQ (4-bit, 128-group, desc_act=False)
candle-vllm --m thesven/Llama-3-8B-GPTQ-4bit

# Convert uncompressed model to Marlin-compatible format
python3 examples/convert_marlin.py --src /home/DeepSeek-R1-Distill-Qwen-14B/ --dst /home/DeepSeek-R1-Distill-Qwen-14B-GPTQ_4bit-128g
candle-vllm --w /home/DeepSeek-R1-Distill-Qwen-14B-GPTQ_4bit-128g

# Convert AWQ to Marlin-compatible format
python3 examples/convert_awq_marlin.py --src /home/Meta-Llama-3.1-8B-Instruct-AWQ-INT4/ --dst /home/Meta-Llama-3.1-8B-Instruct-AWQ-INT4-Marlin/ --bits 4 --method awq --group 128 --nk False
candle-vllm --d 0 --w /home/Meta-Llama-3.1-8B-Instruct-AWQ-INT4-Marlin/

# AWQ MoE (pack-quantized), e.g. Qwen3-Coder-Next
candle-vllm --m cyankiwi/Qwen3-Coder-Next-AWQ-4bit

# Direct Marlin-format model
candle-vllm --w /home/DeepSeek-R1-Distill-Qwen-14B-GPTQ-Marlin/
```

--------------------------------

### Run ISQ Quantization

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Apply in-situ quantization to unquantized models using the --isq parameter.

```bash
candle-vllm --m Qwen/Qwen3.6-27B --isq q4k
```

--------------------------------

### Enable Reasoning Content Split

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/tool_parsing.md

Set this environment variable to 1 to emit reasoning as OpenAI-style `reasoning_content` instead of keeping it within `content` with reasoning markers. Values like '0', 'false', or 'no' disable this split. The default is enabled.

```bash
export CANDLE_VLLM_STREAM_AS_REASONING_CONTENT=1
```

--------------------------------

### Run Kilo Code

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/kilocode.md

Executes the Kilo Code CLI, which will use the configuration to connect to the local Candle-vLLM instance.

```bash
kilo
```

--------------------------------

### Run GGUF Models

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Various methods for loading GGUF models from local paths or HuggingFace repositories.

```bash
# Local GGUF file via --m (recommended)
candle-vllm --m /home/data/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server

# Local GGUF file via --f (legacy)
candle-vllm --f /home/data/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server

# Local directory containing GGUF (auto-detected, mmproj loaded on demand)
candle-vllm --m /home/data/Qwen3.5-35B-A3B-GGUF/ --ui-server

# From HuggingFace (exact file)
candle-vllm --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server

# From HuggingFace (subfolder — downloads all GGUF files in the remote path)
candle-vllm --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_S --d 0,1 --ui-server

# Local multi-shard GGUF (split files auto-discovered from local path)
candle-vllm --m /home/data/model-00001-of-00003.gguf --d 0,1 --ui-server

# GGUF on Apple Silicon
candle-vllm --m /home/qwq-32b-q4_k_m.gguf --ui-server
candle-vllm --m Qwen/QwQ-32B-GGUF --f qwq-32b-q4_k_m.gguf --ui-server
```

--------------------------------

### Configure Multi-Node Inference

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Distribute inference across multiple machines using TCP-based NCCL bootstrap. Model weights must be present locally on all nodes.

```bash
# On master node (192.168.1.100):
candle-vllm --d 0,1,2,3,4,5,6,7 --w /data/DeepSeek-R1-AWQ-Marlin/ \
  --num-nodes 2 --node-rank 0 --master-addr 192.168.1.100 --master-port 29500

# On worker node (192.168.1.101):
candle-vllm --d 0,1,2,3,4,5,6,7 --w /data/DeepSeek-R1-AWQ-Marlin/ \
  --num-nodes 2 --node-rank 1 --master-addr 192.168.1.100 --master-port 29500
```

--------------------------------

### MCP Config File for Multiple Servers

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/mcp_tool_calling.md

A JSON configuration file defining MCP servers, in this case, a 'filesystem' server using `npx` with specific arguments and a workspace path.

```json
{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/home/user/workspace"]
    }
  }
}
```

--------------------------------

### Add Candle-VLLM Dependency

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/rust_crate.md

Add the candle-vllm crate to your Cargo.toml file. Ensure to include backend features like 'cuda' if needed.

```toml
[dependencies]
candle-vllm = { path = "../candle-vllm", features = ["cuda"] }
tokio = { version = "1", features = ["full"] }
```

--------------------------------

### Run Models from Local Paths

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Execute inference using local safetensors directories or GGUF model files.

```bash
# Local safetensors directory
candle-vllm --d 0,1,2,3,4,5,6,7 --m /home/data/GLM-5.2-FP8/ --ui-server

# Local GGUF file (single or split-shard)
candle-vllm --d 0,1 --m /home/data/model-Q4_K_M.gguf --ui-server

# Local directory containing GGUF files (auto-detected)
candle-vllm --d 0,1 --m /home/data/Qwen3.5-35B-A3B-GGUF/ --ui-server
```

--------------------------------

### Configure Candle-vLLM Reasoning Routing

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/opencode.md

Controls how reasoning content is streamed in Candle-vLLM. Set to 1 to stream reasoning separately, or 0 to keep it within the 'content' field.

```bash
export CANDLE_VLLM_STREAM_AS_REASONING_CONTENT=1
```

```bash
export CANDLE_VLLM_STREAM_AS_REASONING_CONTENT=0
```

--------------------------------

### Apply NUMA Binding

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Optimize performance by binding processes to specific NUMA nodes. Ensure MAP_NUMA_NODE matches the numactl binding.

```bash
sudo apt-get install numactl

# 8 GPUs, 2 NUMA nodes
MAP_NUMA_NODE=0,0,0,0,1,1,1,1 numactl --cpunodebind=0 --membind=0 candle-vllm --d 0,1,2,3,4,5,6,7 --w /home/data/DeepSeek-V2-Chat-AWQ-Marlin

# 4 GPUs
MAP_NUMA_NODE=0,0,0,0 numactl --cpunodebind=0 --membind=0 candle-vllm --d 0,1,2,3 --w /home/data/DeepSeek-V2-Chat-AWQ-Marlin
```

--------------------------------

### Enable Candle-vLLM Chat Logger

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/opencode.md

Activates detailed logging for chat interactions within Candle-vLLM, useful for debugging.

```bash
export CANDLE_VLLM_CHAT_LOGGER=1
```

--------------------------------

### Discover Served Model Name

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/opencode.md

Retrieves the list of models served by Candle-vLLM to identify the correct model ID for OpenCode configuration.

```bash
curl http://localhost:8000/v1/models
```

--------------------------------

### Configure TurboQuant KV Cache

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Select KV cache compression modes using the --kvcache-dtype flag. TurboQuant is incompatible with MLA models.

```bash
# Turbo4 (4-bit KV cache, ~3.7x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo4

# Turbo8 (FP8 K + 4-bit V, ~2.6x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo8

# Turbo3 (3-bit K + 4-bit V, ~4.7x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo3

# FP8 KV Cache
candle-vllm --w /data/Qwen3.5-35B-A3B-FP8/ --kvcache-dtype fp8
```

--------------------------------

### Run Models via HuggingFace ID

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Execute inference using a model identifier from the HuggingFace Hub.

```bash
candle-vllm --m Qwen/Qwen3.6-27B-FP8 --ui-server

candle-vllm --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_S --d 0,1 --ui-server

candle-vllm --m zai-org/GLM-5.2-FP8 --d 0,1,2,3,4,5,6,7 --ui-server
```

--------------------------------

### Model Response with Tool Call

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/mcp_tool_calling.md

When the model decides to use a tool, its response will contain a `tool_calls` array, specifying the tool to be invoked and its arguments. The `finish_reason` will be `tool_calls`.

```json
{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "tool_calls": [
          {
            "id": "call_123",
            "type": "function",
            "function": {
              "name": "list_files",
              "arguments": "{\"path\":\".\"}"
            }
          }
        ]
      },
      "finish_reason": "tool_calls"
    }
  ]
}
```

--------------------------------

### Set Hybrid Mamba Snapshot Stride

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/prefix_cache.md

Controls sparse decode-time snapshot capture for hybrid Mamba models to ensure compatible snapshot boundaries for prefix reuse. A larger stride size can be beneficial for systems with limited GPU memory.

```bash
export CANDLE_VLLM_MAMBA_SNAPSHOT_STRIDE_BLOCKS=1
```

--------------------------------

### Client Sending Tool Result Back

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/mcp_tool_calling.md

After executing the tool, the client must send the result back to the model as a `role="tool"` message, including the `tool_call_id` to match the original call.

```json
{
  "role": "tool",
  "tool_call_id": "call_123",
  "content": "file1\nfile2\nfile3"
}
```

--------------------------------

### Run Multi-GPU Inference

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Execute inference across multiple GPUs using multi-process mode. The number of GPUs must be a power of 2.

```bash
# QwQ-32B BF16 on two GPUs
candle-vllm --d 0,1 --w /home/QwQ-32B/

# QwQ-32B 4-bit AWQ on two GPUs
python3 examples/convert_awq_marlin.py --src /home/QwQ-32B-AWQ/ --dst /home/QwQ-32B-AWQ-Marlin/ --bits 4 --method awq --group 128 --nk False
candle-vllm --d 0,1 --w /home/QwQ-32B-AWQ-Marlin/
```

--------------------------------

### Debug Multi-threaded Inference

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Use multi-threaded mode for debugging purposes. Disable P2P if memory access errors occur.

```bash
# Add --multithread parameter
candle-vllm --multithread --d 0,1 --w /home/QwQ-32B/

# Troubleshooting
export NCCL_P2P_DISABLE=1  # disable P2P if encountering illegal memory access
```

--------------------------------

### POST /v1/embeddings

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/embedding.md

Generates embeddings for the given input text using an OpenAI-compatible API. Supports different pooling strategies and encoding formats.

```APIDOC
## POST /v1/embeddings

### Description
This endpoint generates embeddings for text inputs. It supports various models and configuration options for embedding generation.

### Method
POST

### Endpoint
/v1/embeddings

### Parameters
#### Request Body
- **input** (string or string array) - Required - The text or array of texts to generate embeddings for.
- **model** (string) - Optional - The model to use for embedding. Defaults to the loaded model if not specified.
- **embedding_type** (string) - Optional - The type of embedding to generate. Options are `mean` (averages token hidden states) or `last` (uses the final token hidden state). Defaults to `mean`.
- **encoding_format** (string) - Optional - The format for encoding the embeddings. Options are `float` or `base64`. Defaults to `float`.
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.