vLLM

https://docs.vllm.ai/en/stable/
Admin
vLLM is a fast and easy-to-use library for LLM inference and serving with state-of-the-art...
Tokens:12,686,210
Snippets:76,846
Trust Score:9.5
Update:2 days ago
Show doc for...
Tokens:
Raw
### Install dstack client and start server

Source: https://docs.vllm.ai/en/stable/deployment/frameworks/dstack/

Install the dstack client with all extras and start the dstack server. Run these commands to set up dstack before configuring a project.

```bash
pip install dstack[all]
dstack server
```

--------------------------------

### Start vLLM server with Qwen3-0.6B-Chat model

Source: https://docs.vllm.ai/en/stable/deployment/frameworks/open-webui/

Use this command to start the vLLM server with a supported chat completion model. Ensure Docker is installed first.

```bash
vllm serve Qwen/Qwen3-0.6B-Chat
```

--------------------------------

### Start vLLM server with Mistral-7B-Instruct-v0.2

Source: https://docs.vllm.ai/en/stable/deployment/frameworks/autogen/

Launch the vLLM server with a supported chat completion model. This example uses mistralai/Mistral-7B-Instruct-v0.2.

```bash
vllm serve mistralai/Mistral-7B-Instruct-v0.2
```

--------------------------------

### Install AutoAWQ

Source: https://docs.vllm.ai/en/stable/features/quantization/auto_awq/

Install the AutoAWQ library to quantize models. Required before running the quantization example.

```bash
pip install autoawq
```

--------------------------------

### Serve Qwen2-VL-7B-Instruct

Source: https://docs.vllm.ai/en/stable/benchmarking/cli/

Start a vLLM server for the Qwen2-VL-7B-Instruct model, required before running benchmarks that use this model.

```bash
vllm serve Qwen/Qwen2-VL-7B-Instruct
```

--------------------------------

### Install vLLM Production Stack with Helm

Source: https://docs.vllm.ai/en/stable/deployment/integrations/production-stack/

Adds the vLLM Helm repository and installs the vLLM stack using a minimal example values file. Requires Helm installed on the GPU server.

```bash
sudo helm repo add vllm https://vllm-project.github.io/production-stack
sudo helm install vllm vllm/vllm-stack -f tutorials/assets/values-01-minimal-example.yaml
```

--------------------------------

### _post_init_setup method

Source: https://docs.vllm.ai/en/stable/api/vllm/model_executor/layers/fused_moe/modular_kernel/

Resolves leftover setup dependencies between prepare_finalize and fused_experts. Asserts that activation formats match.

```python
def _post_init_setup(self):
    """Resolve any leftover setup dependencies between self.prepare_finalize
    and self.fused_experts here.
    """
    self.prepare_finalize.post_init_setup(self.impl.fused_experts)
    assert (
        self.prepare_finalize.activation_format
        == self.fused_experts.activation_format()
    )
```

--------------------------------

### Example --serve-params JSON for startup sweep

Source: https://docs.vllm.ai/en/stable/benchmarking/sweeps/

Defines engine settings to vary across startup benchmarks. Each object includes a _benchmark_name and parameters like model, tensor_parallel_size, and gpu_memory_utilization.

```json
[
    {
        "_benchmark_name": "tp1",
        "model": "Qwen/Qwen3-0.6B",
        "tensor_parallel_size": 1,
        "gpu_memory_utilization": 0.9
    },
    {
        "_benchmark_name": "tp2",
        "model": "Qwen/Qwen3-0.6B",
        "tensor_parallel_size": 2,
        "gpu_memory_utilization": 0.9
    }
]
```

--------------------------------

### Start vLLM server with Qwen2.5-1.5B-Instruct

Source: https://docs.vllm.ai/en/stable/getting_started/quickstart/

Launches the vLLM OpenAI-compatible server on localhost:8000. Use --host and --port to change the address.

```bash
vllm serve Qwen/Qwen2.5-1.5B-Instruct
```

--------------------------------

### Install Volcano via Helm

Source: https://docs.vllm.ai/en/stable/deployment/integrations/kthena/

Adds the Volcano Helm repository, updates it, and installs Volcano in the volcano-system namespace. Volcano provides gang scheduling for the example.

```bash
helm repo add volcano-sh https://volcano-sh.github.io/helm-charts
helm repo update
helm install volcano volcano-sh/volcano -n volcano-system --create-namespace
```

--------------------------------

### Start vLLM server with Qwen2.5-3B-Instruct

Source: https://docs.vllm.ai/en/stable/examples/features/structured_outputs/

Start a vLLM server with any model of your choice. This command serves the Qwen/Qwen2.5-3B-Instruct model.

```bash
vllm serve Qwen/Qwen2.5-3B-Instruct
```

--------------------------------

### Start vLLM server with reasoning parser

Source: https://docs.vllm.ai/en/stable/examples/reasoning/openai_chat_completion_with_reasoning_streaming/

Start the vLLM server with the DeepSeek-R1 reasoning parser to enable reasoning output. This is required before running the streaming client example.

```bash
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
     --reasoning-parser deepseek_r1
```

--------------------------------

### Run prefill and decode examples sequentially with run.sh

Source: https://docs.vllm.ai/en/stable/examples/disaggregated/example_connector/

Cleans up local_storage and output.txt, then runs prefill_example.py and decode_example.py with VLLM_ENABLE_V1_MULTIPROCESSING=0 and CUDA_VISIBLE_DEVICES=0. Must be executed from the examples/disaggregated/example_connector directory.

```bash
rm -rf local_storage/

if [ -f "output.txt" ]; then
    rm output.txt
fi

# The directory of current script
SCRIPT_DIR=$(dirname "$(readlink -f "$0")")

VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=0 python3 "$SCRIPT_DIR/prefill_example.py"
VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=0 python3 "$SCRIPT_DIR/decode_example.py"
```

--------------------------------

### Start API server on 4 GPUs with --tensor-parallel-size

Source: https://docs.vllm.ai/en/stable/serving/parallelism_scaling/

Include --tensor-parallel-size when starting the server for multi-GPU serving. This example runs the API server on 4 GPUs.

```bash
vllm serve facebook/opt-13b \
     --tensor-parallel-size 4
```

--------------------------------

### Run all allowed_tools examples in main

Source: https://docs.vllm.ai/en/stable/examples/tool_calling/openai_responses_client_with_mcp_tools/

Main entry point that runs the no-filter, wildcard, specific-tools, and object-format examples and prints a summary of allowed_tools behavior. Note that allowed_tools filters sub-tools within an MCP server.

```python
def main():
    """Run all examples."""
    print("\n" + "=" * 60)
    print("MCP Tools with allowed_tools Examples")
    print("=" * 60 + "\n")

    # Run all examples
    example_no_filter()
    example_wildcard()
    example_specific_tools()
    example_object_format()

    print("=" * 60)
    print("Summary:")
    print("  - No filter or '*' → All tools available from server")
    print("  - Specific list → Only those sub-tools available")
    print("  - Object format → More control with tool_names field")
    print("")
    print("Note: allowed_tools filters SUB-TOOLS within an MCP server:")
    print("  - code_interpreter (python): No sub-tools to filter")
    print("  - web_search_preview (browser): Has 'search', 'open', 'find'")
    print("=" * 60)


if __name__ == "__main__":
    main()
```

--------------------------------

### Install vLLM in editable mode with precompiled wheels

Source: https://docs.vllm.ai/en/stable/contributing/incremental_build/

Use this to set up an editable vLLM installation quickly. It creates a virtual environment, activates it, and installs vLLM with VLLM_USE_PRECOMPILED=1 to use precompiled wheels, which speeds up the initial setup before using the CMake incremental workflow.

```bash
uv venv --python 3.12 --seed
source .venv/bin/activate
VLLM_USE_PRECOMPILED=1 uv pip install -U -e . --torch-backend=auto
```

--------------------------------

### Start vLLM server with deepseek_r1 reasoning parser

Source: https://docs.vllm.ai/en/stable/examples/reasoning/openai_chat_completion_with_reasoning/

Run this command to start the vLLM server with the reasoning parser required for the example. The server must be running before executing the Python client code.

```bash
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
    --reasoning-parser deepseek_r1
```

--------------------------------

### Install FlexKV from source

Source: https://docs.vllm.ai/en/stable/api/vllm/distributed/kv_transfer/kv_connector/v1/flexkv_connector/

Clone the FlexKV repository and run the build script to install FlexKV. Required before using FlexKVConnectorV1.

```bash
git clone [email protected]:taco-project/FlexKV.git
cd FlexKV && bash build.sh
```

--------------------------------

### Example pod status output

Source: https://docs.vllm.ai/en/stable/deployment/frameworks/lws/

Shows the expected output of kubectl get pods after successful deployment.

```text
NAME       READY   STATUS    RESTARTS   AGE
vllm-0     1/1     Running   0          2s
vllm-0-1   1/1     Running   0          2s
```

--------------------------------

### Example --startup-params JSON for startup sweep

Source: https://docs.vllm.ai/en/stable/benchmarking/sweeps/

Defines startup-specific options such as iteration counts. Each object includes a _benchmark_name and parameters like num_iters_cold, num_iters_warmup, and num_iters_warm.

```json
[
    {
        "_benchmark_name": "qwen3-0.6",
        "num_iters_cold": 2,
        "num_iters_warmup": 1,
        "num_iters_warm": 2
    }
]
```

--------------------------------

### Setup HF3FS storage clients

Source: https://docs.vllm.ai/en/stable/api/vllm/distributed/kv_transfer/kv_connector/v1/hf3fs/hf3fs_connector/

Creates the storage directory, initializes HF3FS clients for each job, and sets up the metadata client. Requires the storage path and file size to be configured; logs and re-raises any initialization errors.

```python
def _setup_storage_clients(self):
    os.makedirs(self._storage_path, exist_ok=True)

    self._rank = get_tensor_model_parallel_rank()
    file_path = os.path.join(
        self._storage_path, f"hf3fs_vllm_data_file_{self._rank}"
    )

    try:
        # Initialize HF3FS clients
        self._ac = AtomicCounter(self._numjobs)
        assert Hf3fsClient is not None
        self._clients = [
            Hf3fsClient(
                path=file_path,
                size=self._file_size,
                bytes_per_page=self._bytes_per_page,
                entries=DEFAULT_MAX_IO_ENTRIES,
            )
            for _ in range(self._numjobs)
        ]

        # Initialize metadata client
        num_pages = self._file_size // self._bytes_per_page
        self._metadata_client = Hf3fsMetadataClient()
        self._metadata_client.initialize(self._rank, num_pages, role="worker")
    except Exception as e:
        logger.error("HF3FS client initialization failed: %s", e)
        raise
```

--------------------------------

### LLM.get_world_size

Source: https://docs.vllm.ai/en/stable/api/vllm/

Get the world size from the parallel config. This method returns the number of processes in the distributed setup.

```APIDOC
## LLM.get_world_size

### Description
Get the world size from the parallel config. This method returns the number of processes in the distributed setup.

### Method
`get_world_size(include_dp=True)`

### Parameters
- **include_dp** (bool, optional) - Whether to include data parallelism in the world size. Defaults to True.

### Response
Returns an integer representing the world size.
```

--------------------------------

### Start Gradio Webserver for vLLM

Source: https://docs.vllm.ai/en/stable/examples/applications/chatbot/

Run this script after starting the vLLM API server to launch a Gradio web interface. Requires gradio installed and the vLLM API server running at the specified model URL.

```python
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Example for starting a Gradio Webserver
Start vLLM API server:
    python examples/applications/api_server/server.py \
        --model meta-llama/Llama-2-7b-chat-hf

Start Webserver:
    python examples/applications/chatbot/gradio_webserver.py

Note that `pip install --upgrade gradio` is needed to run this example.
More details: https://github.com/gradio-app/gradio

If your antivirus software blocks the download of frpc for gradio,
you can install it manually by following these steps:

1. Download this file: https://cdn-media.huggingface.co/frpc-gradio-0.3/frpc_linux_amd64
2. Rename the downloaded file to: frpc_linux_amd64_v0.3
3. Move the file to this location: /home/user/.cache/huggingface/gradio/frpc
"""

import argparse
import json

import gradio as gr
import requests


def http_bot(prompt):
    headers = {"User-Agent": "vLLM Client"}
    pload = {
        "prompt": prompt,
        "stream": True,
        "max_tokens": 128,
    }
    response = requests.post(args.model_url, headers=headers, json=pload, stream=True)

    for chunk in response.iter_lines(
        chunk_size=8192, decode_unicode=False, delimiter=b"\n"
    ):
        if chunk:
            data = json.loads(chunk.decode("utf-8"))
            output = data["text"][0]
            yield output


def build_demo():
    with gr.Blocks() as demo:
        gr.Markdown("# vLLM text completion demo\n")
        inputbox = gr.Textbox(label="Input", placeholder="Enter text and press ENTER")
        outputbox = gr.Textbox(
            label="Output", placeholder="Generated result from the model"
        )
        inputbox.submit(http_bot, [inputbox], [outputbox])
    return demo


def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument("--host", type=str, default=None)
    parser.add_argument("--port", type=int, default=8001)
    parser.add_argument(
        "--model-url", type=str, default="http://localhost:8000/generate"
    )
    return parser.parse_args()


def main(args):
    demo = build_demo()
    demo.queue().launch(server_name=args.host, server_port=args.port, share=True)


if __name__ == "__main__":
    args = parse_args()
    main(args)
```

--------------------------------

### setup_mooncake_store(store, config, local_hostname)

Source: https://docs.vllm.ai/en/stable/api/vllm/distributed/mooncake_store/

Initializes a native Mooncake Store, preserving explicit tenant configuration. Builds setup keyword arguments from the provided MooncakeStoreConfig and calls store.setup with the local hostname, metadata server, segment sizes, protocol, device name, and master server address.

```APIDOC
## setup_mooncake_store(store, config, local_hostname)

### Description
Initialize a native Store, preserving explicit tenant configuration. If the config's tenant_id differs from the default tenant ID, it is passed as a tenant_id setup keyword argument.

### Method
setup_mooncake_store(store: Any, config: MooncakeStoreConfig, local_hostname: str) -> None

### Parameters
- **store** (Any) - Required - The native store object whose setup method is invoked.
- **config** (MooncakeStoreConfig) - Required - Configuration providing tenant_id, metadata_server, global_segment_size, local_buffer_size, protocol, device_name, and master_server_address.
- **local_hostname** (str) - Required - The local hostname passed to store.setup.

### Response
- **None** - Returns nothing on success.

### Errors
- **RuntimeError** - Raised with message "Initialize MooncakeDistributedStore failed." when store.setup returns a non-zero value.
```

--------------------------------

### Install and run vllm-router proxy manually

Source: https://docs.vllm.ai/en/stable/features/moriio_connector_usage/

Install vllm-router via pip and start it with PD disaggregation and MoRIIO connector. The discovery address port 36367 must match the proxy_ping_port configured on each vLLM instance.

```bash
pip install vllm-router
vllm-router \
  --vllm-pd-disaggregation \
  --kv-connector moriio \
  --vllm-discovery-address "0.0.0.0:36367"
```

--------------------------------

### Launch Prefill+Decode Worker with ECExampleConnector

Source: https://docs.vllm.ai/en/stable/examples/disaggregated/disaggregated_encoder/

Starts a vLLM server with encoder consumer role using ECExampleConnector and shared storage path. Requires environment variables like MODEL, GPU_PD, etc.

```bash
env "$DEVICE_AFFINITY_ENV=$GPU_PD" vllm serve "$MODEL" \
    --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION_PD" \
    --port "$PREFILL_DECODE_PORT" \
    --enforce-eager \
    --enable-request-id-headers \
    --max-num-seqs "$MAX_NUM_SEQS" \
    --max-model-len "$MAX_MODEL_LEN" \
    --allowed-local-media-path "${GIT_ROOT}"/tests/v1/ec_connector/integration \
    --ec-transfer-config '{
        "ec_connector": "ECExampleConnector",
        "ec_role": "ec_consumer",
        "ec_connector_extra_config": {
            "shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'"
        }
    }' \
    >"${PD_LOG}" 2>&1 &

PIDS+=($!)

# Wait for workers
wait_for_server "$ENCODE_PORT"
wait_for_server "$PREFILL_DECODE_PORT"
```

--------------------------------

### Worker.get_kv_connector_handshake_metadata

Source: https://docs.vllm.ai/en/stable/api/vllm/v1/worker/gpu_worker/?q=

Gets KV connector metadata from this worker if available. This metadata is used for handshaking between workers in distributed setups.

```APIDOC
## Worker.get_kv_connector_handshake_metadata

### Description
Gets KV connector metadata from this worker if available. This metadata is used for handshaking between workers in distributed setups.

### Method
`get_kv_connector_handshake_metadata()`

### Parameters
None.

### Returns
- **metadata** (dict or None) - KV connector metadata if available, otherwise None.

### Example
```python
metadata = worker.get_kv_connector_handshake_metadata()
```
```

--------------------------------

### Initialize PrefetchOffloader and Start Initial Prefetches

Source: https://docs.vllm.ai/en/stable/api/vllm/model_executor/offloader/

Logs initialization details including number of modules, GPU memory saved, buffer pool size, and configuration parameters. Then starts prefetching for the first prefetch_step modules.

```python
logger.info_once(
    f"[PrefetchOffloader] Initialized {len(self.module_offloaders)} modules. "
    f"Total GPU memory saved: {self.total_offloaded_bytes / 1e9:.4f} GB, "
    f"Static buffer pool: {self.buffer_pool.total_bytes / 1e9:.4f} GB "
    f"(group_size={self.group_size}, num_in_group={self.num_in_group}, "
    f"prefetch_step={self.prefetch_step}, mode={self.mode})"
)

# Start initial prefetches
for i in range(min(self.prefetch_step, len(self.module_offloaders))):
    self.module_offloaders[i].start_onload_to_static()
```

--------------------------------

### Install PyTorch nightly for ROCm 7.0 using pip

Source: https://docs.vllm.ai/en/stable/getting_started/installation/gpu/

Installs PyTorch and torchvision from the ROCm 7.0 nightly wheel index. Use this as an alternative to starting from a ROCm PyTorch Docker image when preparing for a full build.

```bash
# Install PyTorch
pip uninstall torch -y
pip install --no-cache-dir torch torchvision --index-url https://download.pytorch.org/whl/nightly/rocm7.0
```

--------------------------------

### Serve documentation with MkDocs

Source: https://docs.vllm.ai/en/stable/contributing/

Starts the MkDocs development server. The first command includes API reference and takes about 10 minutes; the second excludes API reference for a faster build (~15 seconds). By default the server runs on 127.0.0.1:8000; use -a to specify a different host/port.

```bash
mkdocs serve                           # with API ref (~10 minutes)
API_AUTONAV_EXCLUDE=vllm mkdocs serve  # API ref off (~15 seconds)
```

--------------------------------

### Start vLLM server for Qwen2.5-VL-3B-Instruct

Source: https://docs.vllm.ai/en/stable/benchmarking/cli/

Launch the server with bfloat16 dtype, 16384 max model length, and limit multimodal items per prompt to 3 images and 0 videos. The mm-processor-kwargs sets max_pixels to 1003520.

```bash
vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
  --dtype bfloat16 \
  --max-model-len 16384 \
  --limit-mm-per-prompt '{"image": 3, "video": 0}' \
  --mm-processor-kwargs max_pixels=1003520
```

--------------------------------

### Get layers start and end indices for pipeline parallelism

Source: https://docs.vllm.ai/en/stable/api/vllm/config/

Computes the start and end layer indices for the current pipeline parallel rank. Uses get_pp_indices from vllm.distributed.utils based on total hidden layers, pp_rank, and pp_size.

```python
def get_layers_start_end_indices(
    self, parallel_config: ParallelConfig
) -> tuple[int, int]:
    from vllm.distributed.utils import get_pp_indices

    total_num_hidden_layers = self.get_total_num_hidden_layers()

    # the layout order is: DP x PP x TP
    pp_rank = (
        parallel_config.rank // parallel_config.tensor_parallel_size
    ) % parallel_config.pipeline_parallel_size
    pp_size = parallel_config.pipeline_parallel_size
    start, end = get_pp_indices(total_num_hidden_layers, pp_rank, pp_size)
    return start, end
```

--------------------------------

### Start vLLM server for ShareGPT4V images

Source: https://docs.vllm.ai/en/stable/benchmarking/cli/

Starts a vLLM server with Qwen2.5-VL-7B-Instruct, limiting to one image per prompt and allowing local media from the ShareGPT4V images directory.

```bash
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
  --dtype bfloat16 \
  --limit-mm-per-prompt '{"image": 1}' \
  --allowed-local-media-path /path/to/sharegpt4v/images
```

--------------------------------

### Start vLLM server for ShareGPT4Video videos

Source: https://docs.vllm.ai/en/stable/benchmarking/cli/

Starts a vLLM server with Qwen2.5-VL-7B-Instruct, limiting to one video per prompt and allowing local media from the ShareGPT4Video videos directory.

```bash
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
  --dtype bfloat16 \
  --limit-mm-per-prompt '{"video": 1}' \
  --allowed-local-media-path /path/to/sharegpt4video/videos
```

--------------------------------

### Install and serve with BGE-M3 IO processor plugin

Source: https://docs.vllm.ai/en/stable/models/pooling_models/specific_models/

Install the reference plugin from a source checkout and start vLLM with --io-processor-plugin bge_m3_sparse_plugin. The plugin selects the internal embed&token_classify task to compute dense and lexical outputs together.

```bash
uv pip install ./tests/plugins/bge_m3_sparse_plugin

vllm serve BAAI/bge-m3 \
  --runner pooling \
  --hf-overrides '{"architectures": ["BgeM3EmbeddingModel"]}' \
  --io-processor-plugin bge_m3_sparse_plugin
```

--------------------------------

### OpenAI client with xLAM-2 tool calling example

Source: https://docs.vllm.ai/en/stable/examples/tool_calling/openai_chat_completion_client_with_tools_xlam/

Full Python example using the OpenAI client to interact with a vLLM server, defining tools and processing tool calls. Requires the server to be started with the xLAM tool call parser.

```python
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# ruff: noqa: E501
"""Set up this example by starting a vLLM OpenAI-compatible server with tool call
options enabled for xLAM-2 models:

vllm serve --model Salesforce/Llama-xLAM-2-8b-fc-r --enable-auto-tool-choice --tool-call-parser xlam

OR

vllm serve --model Salesforce/xLAM-2-3b-fc-r --enable-auto-tool-choice --tool-call-parser xlam
"""

import json
import time

from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "empty"
openai_api_base = "http://localhost:8000/v1"


# Define tool functions
def get_weather(location: str, unit: str):
    return f"Weather in {location} is 22 degrees {unit}."


def calculate_expression(expression: str):
    try:
        result = eval(expression)
        return f"The result of {expression} is {result}"
    except Exception as e:
        return f"Could not calculate {expression}: {e}"


def translate_text(text: str, target_language: str):
    return f"Translation of '{text}' to {target_language}: [translated content]"


# Define tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "City and state, e.g., 'San Francisco, CA'",
                    },
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location", "unit"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "calculate_expression",
            "description": "Calculate a mathematical expression",
            "parameters": {
                "type": "object",
                "properties": {
                    "expression": {
                        "type": "string",
                        "description": "Mathematical expression to evaluate, needs to be a valid python expression",
                    }
                },
                "required": ["expression"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "translate_text",
            "description": "Translate text to another language",
            "parameters": {
                "type": "object",
                "properties": {
                    "text": {"type": "string", "description": "Text to translate"},
                    "target_language": {
                        "type": "string",
                        "description": "Target language for translation",
                    },
                },
                "required": ["text", "target_language"],
            },
        },
    },
]

# Map of function names to implementations
tool_functions = {
    "get_weather": get_weather,
    "calculate_expression": calculate_expression,
    "translate_text": translate_text,
}


def process_response(response, tool_functions, original_query):
    """Process a non-streaming response with possible tool calls."""
    print("\n--- Response Output ---")

    # Check if the response has content
    if response.choices[0].message.content:
        print(f"Content: {response.choices[0].message.content}")

    # Check if the response has tool calls
    if response.choices[0].message.tool_calls:
        print("--------------------------------")
        print(f"Tool calls: {response.choices[0].message.tool_calls}")
        print("--------------------------------")

        # Collect all tool calls and results before making follow-up request
        tool_results = []
        assistant_message = {"role": "assistant"}

        if response.choices[0].message.content:
            assistant_message["content"] = response.choices[0].message.content

        assistant_tool_calls = []

        # Process each tool call
        for tool_call in response.choices[0].message.tool_calls:
            function_name = tool_call.function.name
            function_args = tool_call.function.arguments
            function_id = tool_call.id
```

--------------------------------

### Run disaggregated multimodal render-to-generate round-trip with example_mm_serve.py

Source: https://docs.vllm.ai/en/stable/examples/scale_out/example_mm_serve/

Launch the server first with vllm serve Qwen/Qwen3-VL-2B-Instruct --dtype bfloat16 --max-model-len 4096 --enforce-eager --enable-scale-out, then run this script. It renders a 224x224 red test image through /v1/chat/completions/render and passes the response directly to /inference/v1/generate with only sampling_params added. The script warns if no kwargs_data appears in the render response or if 'red' is not found in the generated text.

```python
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Disaggregated multimodal serving: render → generate round-trip.

Demonstrates the two-phase disaggregated flow:
  1. /v1/chat/completions/render  – preprocesses a multimodal chat request
     into token IDs and serialized tensor features.
  2. /inference/v1/generate       – runs inference on the preprocessed tokens.

The render response is passed *directly* to generate with only
``sampling_params`` added, showing that the two endpoints compose with
zero client-side transformation.

Launch the server first:

    vllm serve Qwen/Qwen3-VL-2B-Instruct \
        --dtype bfloat16 --max-model-len 4096 --enforce-eager \
        --enable-scale-out

Then run this script:

    python example_mm_serve.py
"""

import io

import pybase64 as base64
import requests
from PIL import Image
from transformers import AutoTokenizer

BASE_URL = "http://localhost:8000"
MODEL_NAME = "Qwen/Qwen3-VL-2B-Instruct"


def make_data_url(image: Image.Image) -> str:
    """Encode a PIL image as a base64 data URL."""
    buf = io.BytesIO()
    image.save(buf, format="PNG")
    b64 = base64.b64encode(buf.getvalue()).decode()
    return f"data:image/png;base64,{b64}"


def main():
    # -- Step 1: Create a test image (solid red) -------------------------
    image = Image.new("RGB", (224, 224), color=(255, 0, 0))
    data_url = make_data_url(image)
    print("Created 224x224 red test image")

    # -- Step 2: Render (preprocess) -------------------------------------
    render_payload = {
        "model": MODEL_NAME,
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "image_url", "image_url": {"url": data_url}},
                    {
                        "type": "text",
                        "text": "What color is this image? Answer in one word.",
                    },
                ],
            }
        ],
    }

    print("\n--- Render ---")
    render_resp = requests.post(
        f"{BASE_URL}/v1/chat/completions/render", json=render_payload
    )
    render_resp.raise_for_status()
    render_data = render_resp.json()

    print(f"Response keys: {list(render_data.keys())}")
    print(f"Number of token_ids: {len(render_data['token_ids'])}")

    features = render_data.get("features")
    if features and features.get("kwargs_data"):
        print(f"kwargs_data modalities: {list(features['kwargs_data'].keys())}")
        for modality, items in features["kwargs_data"].items():
            print(
                f"  {modality}: {len(items)} item(s), "
                f"first item type: {type(items[0])} length: {len(items[0])}"
                if items
                else "First item: (empty)"
            )
    else:
        print("WARNING: no kwargs_data in render response")

    # -- Step 3: Generate (inference) ------------------------------------
    # Pass the render output directly — only add sampling_params.
    generate_payload = render_data
    generate_payload["sampling_params"] = {
        "max_tokens": 20,
        "temperature": 0.0,
    }

    print("\n--- Generate ---")
    gen_resp = requests.post(f"{BASE_URL}/inference/v1/generate", json=generate_payload)
    gen_resp.raise_for_status()
    gen_data = gen_resp.json()

    # -- Step 4: Decode & print ------------------------------------------
    output_ids = gen_data["choices"][0]["token_ids"]
    tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
    text = tokenizer.decode(output_ids, skip_special_tokens=True)

    print(f"Output token count: {len(output_ids)}")
    print(f"Generated text: {text!r}")

    if "red" in text.lower():
        print("\nModel correctly identified the red image.")
    else:
        print(f"\nWARNING: Expected 'red' in output, got: {text!r}")


if __name__ == "__main__":
    main()
```

--------------------------------

### Run disagg_encoder_proxy.py for E + P + D setup

Source: https://docs.vllm.ai/en/stable/examples/disaggregated/disaggregated_encoder/

Example invocation of the encoder proxy with separate encode, prefill, and decode server URLs.

```bash
python disagg_encoder_proxy.py \
      --encode-servers-urls "http://e1:8001,http://e2:8001" \
      --prefill-servers-urls "http://p1:8003,http://p2:8004" \
      --decode-servers-urls "http://d1:8005,http://d2:8006"
```

--------------------------------

### Set Up pre-commit Hooks

Source: https://docs.vllm.ai/en/stable/contributing/

Install pre-commit and set up the hooks to run automatically on every commit.

```bash
uv pip install pre-commit>=4.5.1
pre-commit install
```

--------------------------------

### Get Torch Root Directory

Source: https://docs.vllm.ai/en/stable/api/vllm/env_override/

Locates the installed torch package directory without importing torch. Returns None if torch is not found.

```python
def _get_torch_root():
    """Locate the installed torch package without importing it."""
    spec = importlib.util.find_spec("torch")
    if not spec:
        return None
    if spec.origin:
        return os.path.dirname(spec.origin)
    if spec.submodule_search_locations:
        return spec.submodule_search_locations[0]
    return None
```
Tokens:
Raw