### Install xbot CLI

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/xbot.md

Installs the xbot command-line interface. Choose either the Cargo or npm installation method.

```bash
cargo install xbot
```

```bash
npm install -g @trusted-ai/xbot
```

--------------------------------

### Running Candle-VLLM Examples

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/rust_crate.md

Command to run the release build of candle-vllm examples with CUDA support enabled.

```bash
cargo run --release --features cuda --example simple_gen
```

--------------------------------

### Start Qwen3-VL Server

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/multimodal.md

Launches the candle-vllm server with the Qwen3-VL model. Ensure CUDA is enabled if using GPU.

```bash
cargo run --release --features cuda -- \
  --m Qwen/Qwen3-VL-8B-Instruct \
  --p 8000
```

--------------------------------

### Run OpenCode

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/opencode.md

Starts the OpenCode application using the configured settings.

```bash
opencode
```

--------------------------------

### Install OpenCode

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/opencode.md

Installs the OpenCode CLI tool. Use either the curl script for direct installation or npm for package management.

```bash
curl -fsSL https://opencode.ai/install | bash
```

```bash
npm i -g opencode-ai
```

--------------------------------

### Install Candle-vLLM with One-line Script

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Use this command for a quick installation of Candle-vLLM on systems with DEB package support or for binary installations.

```bash
curl -sSL https://ericlbuehler.github.io/candle-vllm/install.sh | bash
```

--------------------------------

### Start Candle-vLLM Server

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/opencode.md

Launches the Candle-vLLM server with specified model and features. Ensure CUDA and NCCL are enabled if using GPU.

```bash
cargo run --release --features cuda,nccl,flashinfer,cutlass -- \
  --m Qwen/Qwen3.6-27B-FP8 \
  --d 0 \
  --p 8000 \
  --kv-fraction 0.6 \
  --enforce-parser qwen_coder
```

--------------------------------

### Start Gemma3 Vision Server

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/multimodal.md

Launches the candle-vllm server with a Gemma3 vision model. CUDA is required for GPU acceleration.

```bash
cargo run --release --features cuda -- \
  --m google/gemma-3-4b-it \
  --p 8000
```

--------------------------------

### Install Kilo Code CLI

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/kilocode.md

Installs the Kilo Code command-line interface globally using npm.

```bash
npm install -g @kilocode/cli
```

--------------------------------

### CLI Configuration for Single MCP Server

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/mcp_tool_calling.md

This command-line example shows how to configure a single stdio MCP server using `npx` to run the `@modelcontextprotocol/server-filesystem`.

```bash
cargo run --release -- --p 8000 \
  --mcp-command npx \
  --mcp-args "-y @modelcontextprotocol/server-filesystem /tmp"
```

--------------------------------

### Install candle-vllm using curl

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/pages/index.html

Installs candle-vllm by piping the install script to bash. Ensure you trust the source before executing.

```bash
$ curl -sSL https://__PAGES_HOST__/candle-vllm/install.sh | bashCopy
```

--------------------------------

### Start candle-vllm Server with Metal

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/embedding.md

Launches the candle-vllm server using Metal acceleration. Ensure the model path and port are correctly specified.

```bash
cargo run --release --features metal -- --m google/gemma-3-4b-it --p 8000
```

--------------------------------

### Start candle-vllm Server with CUDA

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/embedding.md

Launches the candle-vllm server using CUDA acceleration. Ensure the model path and port are correctly specified.

```bash
cargo run --release --features cuda -- --m Qwen/Qwen3-0.6B --p 8000
```

--------------------------------

### OpenAI-style Tool Calling Request Example

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/mcp_tool_calling.md

This JSON structure represents a user's request to the model, including a message and a list of available tools the model can choose to call.

```json
{
  "model": "default",
  "messages": [
    {"role": "user", "content": "List files in the current directory"}
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "list_files",
        "description": "List files in a directory",
        "parameters": {
          "type": "object",
          "properties": {
            "path": {"type": "string"}
          },
          "required": ["path"]
        }
      }
    }
  ]
}
```

--------------------------------

### Rust API Example for Embeddings

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/embedding.md

Programmatic generation of embeddings using the candle-vllm Rust API. This example initializes the engine, creates an embedding request, and prints the number of embeddings generated.

```rust
use candle_vllm::api::{EngineBuilder, ModelRepo};
use candle_vllm::openai::requests::{EmbeddingInput, EmbeddingRequest};

#[tokio::main]
async fn main() -> candle_core::Result<()> {
    let engine = EngineBuilder::new(ModelRepo::ModelID(("Qwen/Qwen3-0.6B", None)))
        .build_async()
        .await?;

    let request = EmbeddingRequest {
        model: Some("default".to_string()),
        input: EmbeddingInput::String("hello world".to_string()),
        encoding_format: Default::default(),
        embedding_type: Default::default(),
    };

    let response = engine.embed(request)?;
    println!("embeddings: {}", response.data.len());
    engine.shutdown();
    Ok(())
}
```

--------------------------------

### Configure xbot for Candle-vLLM

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/xbot.md

Guides through the interactive configuration of xbot to use a custom provider pointing to the Candle-vLLM API. It prompts for a provider name, API base URL, and default model.

```shell
> Select provider to configure: custom
> Enter a unique name for this custom provider: candle-vllm
> Enter API Base URL (e.g. https://api.yourprovider.com/v1): http://localhost:8000/v1/
> Enter API Key? (Optional for local/custom) No
Fetching available models...
> Select default model: Qwen3.6-27B-FP8
Using contextWindowTokens from model metadata: 262144
> Configure subagent provider/model now? No

Configuration saved successfully!
Config file: /root/.xbot/config.json

Final Provider Config:
{
  "api_key": "",
  "api_base": "http://localhost:8000/v1/",
  "extra_headers": {},
  "reasoning_effort": null
}
```

--------------------------------

### Text Generation with Candle-VLLM

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/rust_crate.md

Example of performing text generation using the candle-vllm Rust API. This requires setting up the engine and constructing a chat completion request.

```rust
use candle_vllm::api::{EngineBuilder, ModelRepo};
use candle_vllm::openai::requests::{ChatCompletionRequest, Messages};

#[tokio::main]
async fn main() -> candle_core::Result<()> {
    let engine = EngineBuilder::new(ModelRepo::ModelID(("Qwen/Qwen3-0.6B", None)))
        .build_async()
        .await?;

    let request = ChatCompletionRequest {
        model: Some("default".to_string()),
        messages: Messages::Map(vec![std::collections::HashMap::from([
            ("role".to_string(), "user".to_string()),
            ("content".to_string(), "Say hello from the Rust API.".to_string()),
        ])]),
        max_tokens: Some(64),
        ..Default::default()
    };

    let response = engine.generate_request(request).await?;
    println!("{response:?}");
    engine.shutdown();
    Ok(())
}
```

--------------------------------

### Request Base64 Embeddings with Last-Token Pooling

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/embedding.md

Example cURL request to the /v1/embeddings endpoint for base64 encoded embeddings using last-token pooling. This example demonstrates processing multiple inputs.

```bash
curl -X POST http://localhost:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input":["hello","hola"],"embedding_type":"last","encoding_format":"base64"}'
```

--------------------------------

### NUMA Binding Configuration

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Optimize inference performance by binding to specific NUMA nodes using `numactl`. Install `numactl` and use `MAP_NUMA_NODE` and `numactl` commands to specify CPU and memory binding.

```bash
sudo apt-get install numactl

# 8 GPUs, 2 NUMA nodes
MAP_NUMA_NODE=0,0,0,0,1,1,1,1 numactl --cpunodebind=0 --membind=0 candle-vllm --d 0,1,2,3,4,5,6,7 --w /home/data/DeepSeek-V2-Chat-AWQ-Marlin

# 4 GPUs
MAP_NUMA_NODE=0,0,0,0 numactl --cpunodebind=0 --membind=0 candle-vllm --d 0,1,2,3 --w /home/data/DeepSeek-V2-Chat-AWQ-Marlin
```

--------------------------------

### Embeddings Generation with Candle-VLLM

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/rust_crate.md

Example of generating embeddings using the candle-vllm Rust API. This involves creating an embedding request and passing it to the engine.

```rust
use candle_vllm::openai::requests::{EmbeddingInput, EmbeddingRequest};

let request = EmbeddingRequest {
    model: Some("default".to_string()),
    input: EmbeddingInput::String("hello world".to_string()),
    encoding_format: Default::default(),
    embedding_type: Default::default(),
};

let response = engine.embed(request)?;
```

--------------------------------

### Build Candle-vLLM from Source (CUDA)

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Build Candle-vLLM from source for CUDA-enabled systems. Ensure you have CUDA 11+ installed. Features like flashinfer and cutlass are included, but may need to be removed for older GPU architectures (sm_70/sm_75).

```bash
git clone git@github.com:EricLBuehler/candle-vllm.git
cd candle-vllm

# CUDA (11+, 12+, 13.0) 
cargo install --features cuda,nccl,flashinfer,cutlass --path .
```

--------------------------------

### Multimodal Request with Image URL

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/multimodal.md

Example JSON payload for a chat completion request including text and an image URL. The model must be multimodal.

```json
{
  "model": "default",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "Describe this image"},
        {"type": "image_url", "image_url": "https://example.com/cat.png"}
      ]
    }
  ]
}
```

--------------------------------

### Multimodal Request with Base64 Image

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/multimodal.md

Example JSON payload for a chat completion request with text and a base64 encoded image. Ensure the base64 string is correctly formatted.

```json
{
  "model": "default",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "Describe this image"},
        {"type": "image_base64", "image_base64": "data:image/png;base64,..."}
      ]
    }
  ]
}
```

--------------------------------

### Multi-GPU Inference (Multi-process)

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Run inference across multiple GPUs using the `--d` flag to specify GPU IDs. The number of GPUs must be a power of 2. This mode is recommended for multi-GPU setups.

```bash
# QwQ-32B BF16 on two GPUs
candle-vllm --d 0,1 --w /home/QwQ-32B/

# QwQ-32B 4-bit AWQ on two GPUs
python3 examples/convert_awq_marlin.py --src /home/QwQ-32B-AWQ/ --dst /home/QwQ-32B-AWQ-Marlin/ --bits 4 --method awq --group 128 --nk False
candle-vllm --d 0,1 --w /home/QwQ-32B-AWQ-Marlin/
```

--------------------------------

### Prefix Cache Usage Reporting Example

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/prefix_cache.md

This JSON structure shows how prefix cache and reasoning token details are reported in OpenAI-compatible chat responses when they are non-zero. `cached_tokens` indicates reused prompt tokens, and `reasoning_tokens` indicates generated tokens within reasoning blocks.

```json
{
  "usage": {
    "prompt_tokens": 128,
    "completion_tokens": 64,
    "total_tokens": 192,
    "prompt_time_costs": 4,
    "completion_time_costs": 250,
    "prompt_tokens_details": {
      "cached_tokens": 64
    },
    "completion_tokens_details": {
      "reasoning_tokens": 32
    }
  }
}
```

--------------------------------

### Request Float Embeddings with Mean Pooling

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/embedding.md

Example cURL request to the /v1/embeddings endpoint for float embeddings using mean pooling. The input can be a single string or an array of strings.

```bash
curl -X POST http://localhost:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input":"hello world","model":"default","embedding_type":"mean","encoding_format":"float"}'
```

--------------------------------

### Run Candle-vLLM with Local Model Path (Safetensors)

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Serve a local LLM using safetensors format by specifying the directory path. The `--ui-server` flag launches the web UI.

```bash
# Local safetensors directory
candle-vllm --d 0,1 --m /home/Qwen3-30B-A3B-Instruct-2507/ --ui-server
```

--------------------------------

### Build Candle-vLLM from Source (macOS/Metal)

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Build Candle-vLLM from source for macOS systems utilizing the Metal GPU acceleration.

```bash
git clone git@github.com:EricLBuehler/candle-vllm.git
cd candle-vllm

# macOS/Metal
cargo install --features metal --path .
```

--------------------------------

### Run FP8 Model with Web UI

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Launches the API server and the built-in Web UI for an FP8 model. Ensure the model path is correct.

```bash
candle-vllm --m Qwen/Qwen3.6-27B-FP8 --ui-server
```

--------------------------------

### Run Candle-vLLM with Local Model Path (GGUF)

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Run Candle-vLLM with a local GGUF model file, which can be a single file or split into shards. The `--ui-server` option activates the web UI.

```bash
# Local GGUF file (single or split-shard)
candle-vllm --d 0,1 --m /home/data/model-Q4_K_M.gguf --ui-server
```

--------------------------------

### Running with MCP Config File

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/mcp_tool_calling.md

This command demonstrates how to run `candle-vllm` using a specified MCP configuration file.

```bash
cargo run --release -- --p 8000 --mcp-config mcp_config.json
```

--------------------------------

### Run Candle-vLLM with Local Directory of GGUF Files

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Serve models from a local directory containing GGUF files. Candle-vLLM will auto-detect the models. Use `--ui-server` to enable the web UI.

```bash
# Local directory containing GGUF files (auto-detected)
candle-vllm --d 0,1 --m /home/data/Qwen3.5-35B-A3B-GGUF/ --ui-server
```

--------------------------------

### Run xbot Tasks

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/xbot.md

Executes xbot for one-shot tasks or interactive sessions. The 'chat /init' command scans the project and initializes an agent file.

```bash
cd YOUR PROJECT
# scan project and init agent file
xbot chat /init
# real task
xbot chat "find bugs in this project."
```

```bash
cd YOUR PROJECT
# you may add .xbot workspace folder to .gitignore
xbot repl
```

--------------------------------

### Run Direct Marlin-Format Model

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Loads and serves a model that is already in the direct Marlin format.

```bash
candle-vllm --w /home/DeepSeek-R1-Distill-Qwen-14B-GPTQ-Marlin/
```

--------------------------------

### Run FP4 Model with Web UI

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Serves an FP4 model with the integrated Web UI. Ensure the model path points to a compatible FP4 format.

```bash
candle-vllm --m GadflyII/GLM-4.7-Flash-NVFP4 --ui-server
```

--------------------------------

### Run Candle-vLLM with HuggingFace Model ID

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Launch Candle-vLLM using a model identifier from HuggingFace. Add `--ui-server` to enable the built-in ChatGPT-style Web UI.

```bash
candle-vllm --m Qwen/Qwen3.6-27B-FP8 --ui-server
```

```bash
candle-vllm --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_S --d 0,1 --ui-server
```

--------------------------------

### Run GGUF Model with Web UI

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Loads and serves a GGUF model, optionally with the Web UI. The `--f` flag can specify the exact GGUF file if needed.

```bash
candle-vllm --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server
```

--------------------------------

### Configure OpenCode for Candle-vLLM

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/opencode.md

Sets up the OpenCode configuration file to use a local Candle-vLLM instance. Specify the baseURL and the model name matching the served model.

```json
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "local-candle-vllm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Candle-vLLM Local",
      "options": {
        "baseURL": "http://localhost:8000/v1"
      },
      "models": {
        "qwen3-coder": {
          "name": "Qwen/Qwen3.6-27B-FP8"
        }
      }
    }
  },
  "model": "local-candle-vllm/qwen3-coder"
}
```

--------------------------------

### Load GGUF from HuggingFace (Subfolder)

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Downloads and loads all GGUF files from a specified subfolder on HuggingFace. Supports multi-GPU inference.

```bash
candle-vllm --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_S --d 0,1 --ui-server
```

--------------------------------

### Build Candle-vLLM Docker Image

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Build a Docker image for Candle-vLLM. You can pass custom CUDA and SM versions to the build script for specific environments.

```bash
# Pass custom SM version and CUDA version: ./build_docker.sh "cuda,nccl,flashinfer,cutlass" sm_90 13.0.0
./build_docker.sh "cuda,nccl,flashinfer,cutlass"
```

--------------------------------

### Run Model with In-Situ Quantization (ISQ)

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Performs on-the-fly quantization to a specified format (e.g., q4k) when running an unquantized model. Available options include q4_0, q4_1, q5_0, q5_1, q8_0, q2k, q3k, q4k, q5k, q6k.

```bash
candle-vllm --m Qwen/Qwen3.6-27B --isq q4k
```

--------------------------------

### JavaScript for rendering download cards

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/pages/index.html

This JavaScript code dynamically generates download links for pre-built candle-vllm packages based on selected filters (e.g., deb package, specific GPU architecture). It handles different architectures including NVIDIA GPUs and Apple Silicon.

```javascript
(function(){
  'use strict';
  var CANDLE_VLLM_VER='0.8.6';
  var DL='https://github.com/__REPO__/releases/download/v'+CANDLE_VLLM_VER+'/';
  var archs=[
    {sm:'sm70',name:'SM70',gpu:'V100',cuda:'CUDA 12',arch:'x64'},
    {sm:'sm75',name:'SM75',gpu:'T4 / Turing',cuda:'CUDA 12',arch:'x64'},
    {sm:'sm80',name:'SM80',gpu:'A100 / A10',cuda:'CUDA 13',arch:'x64'},
    {sm:'sm89',name:'SM89',gpu:'RTX 4090 / L40',cuda:'CUDA 13',arch:'x64'},
    {sm:'sm90',name:'SM90',gpu:'H800 / H200',cuda:'CUDA 13',arch:'x64'},
    {sm:'sm100',name:'SM100',gpu:'B100 / B200',cuda:'CUDA 13',arch:'x64'},
    {sm:'sm120',name:'SM120',gpu:'RTX 5090 / RTX PRO 6000',cuda:'CUDA 13',arch:'x64'},
    {sm:'sm121',name:'SM121',gpu:'DGX Spark / GB10',cuda:'CUDA 13',arch:'arm64'}
  ];
  var metalA={sm:'metal',name:'Metal',gpu:'macOS Apple Silicon',cuda:''};
  window.renderCards=function(type,btn){
    document.querySelectorAll('.tabs button').forEach(function(b){b.classList.remove('on')});
    if(btn)btn.classList.add('on');
    var el=document.getElementById('cards');
    el.innerHTML='';
    var items=archs.slice();
    items.push(metalA);
    items.forEach(function(a){
      var fname,cpuArch=a.arch||'x64';
      if(a.sm==='metal'){
        fname='candle-vllm-'+CANDLE_VLLM_VER+'-darwin-arm64-metal.tar.gz';
      }else if(type==='deb'){
        fname='candle-vllm-linux-'+cpuArch+'-'+a.sm+'.deb';
      }else{
        fname='candle-vllm-'+CANDLE_VLLM_VER+'-linux-'+cpuArch+'-'+a.sm+'.tar.gz';
      }
      var c=document.createElement('a');
      c.href=DL+fname;
      c.target='_blank';
      c.className='card';
      var archLabel=cpuArch==='arm64'?' (ARM64)':'';
      c.innerHTML='<h3>'+a.name+' — '+a.gpu+'</h3>' +(a.cuda?'<p>'+a.cuda+archLabel+'</p>':'<p>Apple Silicon (tarball)</p>') +'<span class="file">'+fname+'</span>' +'<span class="dl">Download &rarr;</span>';
      el.appendChild(c);
    });
  };
  renderCards('deb',null);
 })();
```

--------------------------------

### Run FP8 Model on MacOS/Metal (Dense)

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Infers an FP8 model on MacOS using Metal acceleration. Ensure the model path is correct.

```bash
candle-vllm --m Qwen/Qwen3-4B-Instruct-2507-FP8 --ui-server
```

--------------------------------

### Load Local GGUF File via --m

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Loads a GGUF model directly from a local file path using the `--m` argument. This is the recommended method.

```bash
candle-vllm --m /home/data/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server
```

--------------------------------

### Bind Candle-VLLM to a specific IP and Port

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Use the `--h` and `--p` flags to specify the host and port for the server. The `--m` flag is required to specify the model.

```bash
candle-vllm --h 127.0.0.1 --p 8000 --m Qwen/Qwen3.6-27B-FP8
```

--------------------------------

### Enable Reasoning Content Split

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/tool_parsing.md

Set this environment variable to 1 to emit reasoning as OpenAI-style `reasoning_content` instead of keeping it within `content` with reasoning markers. Values like '0', 'false', or 'no' disable this split. The default is enabled.

```bash
export CANDLE_VLLM_STREAM_AS_REASONING_CONTENT=1
```

--------------------------------

### Run Kilo Code

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/kilocode.md

Executes the Kilo Code CLI, which will use the configuration to connect to the local Candle-vLLM instance.

```bash
kilo
```

--------------------------------

### Run MXFP4 Model

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Loads and serves a model quantized using the MXFP4 format.

```bash
candle-vllm --m nm-testing/Qwen3-30B-A3B-MXFP4A16 --ui-server
```

--------------------------------

### Load Local GGUF File via --f (Legacy)

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Loads a GGUF model from a local file path using the legacy `--f` argument. Prefer using `--m` for GGUF files.

```bash
candle-vllm --f /home/data/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server
```

--------------------------------

### Multi-Node Inference Configuration

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Distribute inference across multiple machines using TCP-based NCCL bootstrap. Ensure all nodes have model weights locally and are TCP-reachable on the master port.

```bash
# On master node (192.168.1.100):
candle-vllm --d 0,1,2,3,4,5,6,7 --w /data/DeepSeek-R1-AWQ-Marlin/ \
  --num-nodes 2 --node-rank 0 --master-addr 192.168.1.100 --master-port 29500

# On worker node (192.168.1.101):
candle-vllm --d 0,1,2,3,4,5,6,7 --w /data/DeepSeek-R1-AWQ-Marlin/ \
  --num-nodes 2 --node-rank 1 --master-addr 192.168.1.100 --master-port 29500
```

--------------------------------

### Run Converted AWQ-Marlin Model

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Loads and serves a model converted from AWQ to Marlin format. Supports multi-GPU inference.

```bash
candle-vllm --d 0 --w /home/Meta-Llama-3.1-8B-Instruct-AWQ-INT4-Marlin/
```

--------------------------------

### Bind Candle-VLLM using a Unix Socket

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Allows binding to a Unix domain socket for local communication. The `--m` flag is required to specify the model.

```bash
candle-vllm --h unix:///tmp/candle-vllm.sock --m Qwen/Qwen3.6-27B-FP8
```

--------------------------------

### Run GGUF on Apple Silicon

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Infers a GGUF model on Apple Silicon. Both local file paths and HuggingFace model identifiers are supported.

```bash
candle-vllm --m /home/qwq-32b-q4_k_m.gguf --ui-server
```

```bash
candle-vllm --m Qwen/QwQ-32B-GGUF --f qwq-32b-q4_k_m.gguf --ui-server
```

--------------------------------

### Run Converted Marlin Model

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Loads and serves a model that has been converted to the Marlin-compatible format.

```bash
candle-vllm --w /home/DeepSeek-R1-Distill-Qwen-14B-GPTQ_4bit-128g
```

--------------------------------

### Run Marlin-compatible GPTQ Model

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Loads and serves a GPTQ model compatible with the Marlin format (4-bit, 128-group, desc_act=False).

```bash
candle-vllm --m thesven/Llama-3-8B-GPTQ-4bit
```

--------------------------------

### MCP Config File for Multiple Servers

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/mcp_tool_calling.md

A JSON configuration file defining MCP servers, in this case, a 'filesystem' server using `npx` with specific arguments and a workspace path.

```json
{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/home/user/workspace"]
    }
  }
}
```

--------------------------------

### Add Candle-VLLM Dependency

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/rust_crate.md

Add the candle-vllm crate to your Cargo.toml file. Ensure to include backend features like 'cuda' if needed.

```toml
[dependencies]
candle-vllm = { path = "../candle-vllm", features = ["cuda"] }
tokio = { version = "1", features = ["full"] }
```

--------------------------------

### Run Unquantized Safetensors Model on Multi-GPU

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Infers using an unquantized Safetensors model across specified GPUs. Adjust the device list and model path as needed.

```bash
candle-vllm --d 0,1 --w /home/Qwen3-30B-A3B-Instruct-2507/
```

--------------------------------

### Configure Candle-vLLM Reasoning Routing

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/opencode.md

Controls how reasoning content is streamed in Candle-vLLM. Set to 1 to stream reasoning separately, or 0 to keep it within the 'content' field.

```bash
export CANDLE_VLLM_STREAM_AS_REASONING_CONTENT=1
```

```bash
export CANDLE_VLLM_STREAM_AS_REASONING_CONTENT=0
```

--------------------------------

### Load Local Multi-shard GGUF

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Loads a multi-shard GGUF model from a local path. Split files are automatically discovered from the same directory. Supports multi-GPU inference.

```bash
candle-vllm --m /home/data/model-00001-of-00003.gguf --d 0,1 --ui-server
```

--------------------------------

### Bind Candle-VLLM with Host and Port in Address

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

The `--h` flag can directly include the port number, making the `--p` flag redundant. The `--m` flag is required to specify the model.

```bash
candle-vllm --h 127.0.0.1:8000 --m Qwen/Qwen3.6-27B-FP8
```

--------------------------------

### Enable Candle-vLLM Chat Logger

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/opencode.md

Activates detailed logging for chat interactions within Candle-vLLM, useful for debugging.

```bash
export CANDLE_VLLM_CHAT_LOGGER=1
```

--------------------------------

### Configure TurboQuant KV Cache Compression

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Use the `--kvcache-dtype` flag to select different TurboQuant compression modes for KV cache. Supported modes include turbo8, turbo4, and turbo3, offering varying levels of compression and quality.

```bash
# Turbo4 (4-bit KV cache, ~3.7x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo4

# Turbo8 (FP8 K + 4-bit V, ~2.6x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo8

# Turbo3 (3-bit K + 4-bit V, ~4.7x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo3

# FP8 KV Cache
candle-vllm --w /data/Qwen3.5-35B-A3B-FP8/ --kvcache-dtype fp8
```

--------------------------------

### Load Local Directory Containing GGUF

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Loads a GGUF model from a local directory. The model is auto-detected, and vision tower auxiliary files (`mmproj*.gguf`) are loaded on demand for multimodal models.

```bash
candle-vllm --m /home/data/Qwen3.5-35B-A3B-GGUF/ --ui-server
```

--------------------------------

### Discover Served Model Name

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/opencode.md

Retrieves the list of models served by Candle-vLLM to identify the correct model ID for OpenCode configuration.

```bash
curl http://localhost:8000/v1/models
```

--------------------------------

### Convert Uncompressed Model to Marlin Format

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Converts an uncompressed model to a Marlin-compatible format using a Python script. The `--src` and `--dst` arguments specify the source and destination paths.

```python
python3 examples/convert_marlin.py --src /home/DeepSeek-R1-Distill-Qwen-14B/ --dst /home/DeepSeek-R1-Distill-Qwen-14B-GPTQ_4bit-128g
```

--------------------------------

### Multi-GPU Inference (Multi-threaded)

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Enable multi-threaded inference using the `--multithread` parameter. This mode is primarily for debugging purposes. Use `export NCCL_P2P_DISABLE=1` to disable P2P if encountering memory access issues.

```bash
# Add --multithread parameter
candle-vllm --multithread --d 0,1 --w /home/QwQ-32B/

# Troubleshooting
export NCCL_P2P_DISABLE=1  # disable P2P if encountering illegal memory access
```

--------------------------------

### Bind Candle-VLLM to an IPv6 Address

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Supports binding to IPv6 addresses by enclosing the address in square brackets and specifying the port. The `--m` flag is required to specify the model.

```bash
candle-vllm --h '[::1]:8000' --m Qwen/Qwen3.6-27B-FP8
```

--------------------------------

### Model Response with Tool Call

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/mcp_tool_calling.md

When the model decides to use a tool, its response will contain a `tool_calls` array, specifying the tool to be invoked and its arguments. The `finish_reason` will be `tool_calls`.

```json
{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "tool_calls": [
          {
            "id": "call_123",
            "type": "function",
            "function": {
              "name": "list_files",
              "arguments": "{\"path\":\".\"}"
            }
          }
        ]
      },
      "finish_reason": "tool_calls"
    }
  ]
}
```

--------------------------------

### Run Model with Manual YaRN Scaling

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Overrides the YaRN RoPE scaling factor for a model. Adjust the scaling factor as required.

```bash
candle-vllm --m Qwen/Qwen3.6-35B-A3B --yarn-scaling-factor 4.0 --ui-server
```

--------------------------------

### Set Hybrid Mamba Snapshot Stride

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/prefix_cache.md

Controls sparse decode-time snapshot capture for hybrid Mamba models to ensure compatible snapshot boundaries for prefix reuse. A larger stride size can be beneficial for systems with limited GPU memory.

```bash
export CANDLE_VLLM_MAMBA_SNAPSHOT_STRIDE_BLOCKS=1
```

--------------------------------

### Client Sending Tool Result Back

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/mcp_tool_calling.md

After executing the tool, the client must send the result back to the model as a `role="tool"` message, including the `tool_call_id` to match the original call.

```json
{
  "role": "tool",
  "tool_call_id": "call_123",
  "content": "file1\nfile2\nfile3"
}
```

--------------------------------

### Convert AWQ to Marlin Format

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/README.md

Converts an AWQ model to a Marlin-compatible format using a Python script. Specify bits, method, group size, and `nk` parameter.

```python
python3 examples/convert_awq_marlin.py --src /home/Meta-Llama-3.1-8B-Instruct-AWQ-INT4/ --dst /home/Meta-Llama-3.1-8B-Instruct-AWQ-INT4-Marlin/ --bits 4 --method awq --group 128 --nk False
```

--------------------------------

### POST /v1/embeddings

Source: https://github.com/ericlbuehler/candle-vllm/blob/master/docs/embedding.md

Generates embeddings for the given input text using an OpenAI-compatible API. Supports different pooling strategies and encoding formats.

```APIDOC
## POST /v1/embeddings

### Description
This endpoint generates embeddings for text inputs. It supports various models and configuration options for embedding generation.

### Method
POST

### Endpoint
/v1/embeddings

### Parameters
#### Request Body
- **input** (string or string array) - Required - The text or array of texts to generate embeddings for.
- **model** (string) - Optional - The model to use for embedding. Defaults to the loaded model if not specified.
- **embedding_type** (string) - Optional - The type of embedding to generate. Options are `mean` (averages token hidden states) or `last` (uses the final token hidden state). Defaults to `mean`.
- **encoding_format** (string) - Optional - The format for encoding the embeddings. Options are `float` or `base64`. Defaults to `float`.
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.