### E + P + D Setup Example

Source: https://docs.vllm.ai/en/stable/examples/disaggregated/disaggregated_encoder

Example usage for a full Encoder + Prefill + Decode (E+P+D) setup. This configuration uses dedicated servers for each phase.

```bash
$ python disagg_encoder_proxy.py \
      --encode-servers-urls "http://e1:8001,http://e2:8001" \
      --prefill-servers-urls "http://p1:8003,http://p2:8004" \
      --decode-servers-urls "http://d1:8005,http://d2:8006"

```

--------------------------------

### E + PD Setup Example

Source: https://docs.vllm.ai/en/stable/examples/disaggregated/disaggregated_encoder

Example usage for an Encoder + Prefill/Decode (E+PD) setup. This configuration skips the dedicated prefill phase.

```bash
$ python disagg_encoder_proxy.py \
      --encode-servers-urls "http://e1:8001,http://e2:8002" \
      --prefill-servers-urls "disable" \
      --decode-servers-urls "http://pd1:8003,http://pd2:8004"

```

--------------------------------

### vLLM Server Setup Example

Source: https://docs.vllm.ai/en/stable/examples/pooling/score?q=

Shows how to start a vLLM server with the pooling runner for the Vision Score API. This includes specifying model, max model length, and Hugging Face overrides.

```bash
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# ruff: noqa: E501

"""
Example online usage of Score API.

Run `vllm serve <model> --runner pooling` to start up the server in vLLM.

e.g.
    vllm serve jinaai/jina-reranker-m0 --runner pooling

    vllm serve Qwen/Qwen3-VL-Reranker-2B \
        --runner pooling \
        --max-model-len 4096 \
        --hf_overrides '{"architectures": ["Qwen3VLForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}' \
        --chat-template examples/pooling/score/template/qwen3_vl_reranker.jinja
"""

```

--------------------------------

### Start vLLM Server with Reasoning Parser

Source: https://docs.vllm.ai/en/stable/examples/reasoning/openai_chat_completion_with_reasoning

To run the example, start the vLLM server with the appropriate reasoning parser. This command specifies the model and the reasoning parser to be used.

```bash
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
    --reasoning-parser deepseek_r1
```

--------------------------------

### Disaggregated Prefill Example Setup

Source: https://docs.vllm.ai/en/stable/examples/disaggregated/lmcache

Sets up environment variables and imports necessary modules for disaggregated prefilling with LMCache. This example involves launching multiple vLLM instances and an LMCache server.

```python
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""
This file demonstrates the example usage of disaggregated prefilling
with LMCache.
We will launch 2 vllm instances (GPU 0 for prefill and GPU 1 for decode),
and launch an additional LMCache server.
KV cache is transferred in the following manner:
vLLM prefill node -> LMCache server -> vLLM decode node.

Note that `pip install lmcache` is needed to run this example.
Learn more about LMCache in https://github.com/LMCache/LMCache.
"""

import os
import subprocess
import time
from multiprocessing import Event, Process

from lmcache.experimental.cache_engine import LMCacheEngineBuilder
from lmcache.integration.vllm.utils import ENGINE_NAME

from vllm import LLM, SamplingParams
from vllm.config import KVTransferConfig

# LMCache-related environment variables

```

--------------------------------

### Start vLLM Server with Reasoning and Tool Choice

Source: https://docs.vllm.ai/en/stable/examples/reasoning/openai_chat_completion_tool_calls_with_reasoning?q=

This command starts the vLLM server, enabling both reasoning parsing with 'deepseek_r1' and automatic tool choice with the 'hermes' tool call parser. This setup is necessary for the example to function.

```bash
vllm serve Qwen/QwQ-32B \
     --reasoning-parser deepseek_r1 \
     --enable-auto-tool-choice --tool-call-parser hermes
```

--------------------------------

### Start Proxy Server for 3P1D Run

Source: https://docs.vllm.ai/en/stable/design/p2p_nccl_connector?q=

Starts the disaggregated proxy server for a 3 Prefill, 1 Decode setup. This is identical to the proxy setup for the 1 Prefill, 3 Decode run.

```bash
cd {your vllm directory}/examples/disaggregated/p2p_nccl_xpyd/
python3 disagg_proxy_p2p_nccl_xpyd.py &

```

--------------------------------

### Starting vLLM Server with Tool Calling

Source: https://docs.vllm.ai/en/stable/examples/tool_calling/openai_chat_completion_client_with_tools

Example commands to start a vLLM OpenAI-compatible server with tool calling enabled. Ensure you use the correct chat templates and parsers for your chosen model.

```bash
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""
Set up this example by starting a vLLM OpenAI-compatible server with tool call
options enabled. For example:

IMPORTANT: for mistral, you must use one of the provided mistral tool call
templates, or your own - the model default doesn't work for tool calls with vLLM
See the vLLM docs on OpenAI server & tool calling for more details.

vllm serve mistralai/Mistral-7B-Instruct-v0.3 \
            --chat-template examples/tool_chat_template_mistral.jinja \
            --enable-auto-tool-choice --tool-call-parser mistral

OR
vllm serve NousResearch/Hermes-2-Pro-Llama-3-8B \
            --chat-template examples/tool_chat_template_hermes.jinja \
            --enable-auto-tool-choice --tool-call-parser hermes
"""

import json
from typing import Any

from openai import OpenAI

```

--------------------------------

### Install GDRcopy Script Example

Source: https://docs.vllm.ai/en/stable/serving/expert_parallel_deployment

Example command for installing GDRcopy, a dependency for disaggregated serving with Expert Parallelism. Ensure to replace placeholder with your specific OS version.

```bash
install_gdrcopy.sh "${GDRCOPY_OS_VERSION}" "12.8" "x64"
```

--------------------------------

### Custom Configuration Examples

Source: https://docs.vllm.ai/en/stable/examples/disaggregated/disaggregated_encoder?q=

Demonstrates various ways to customize the setup of disaggregated encoder scripts using environment variables.

```bash
# Use specific GPUs
GPU_E=0 GPU_PD=1 GPU_P=1 GPU_D=2 bash disagg_1e1p1d_example.sh

# Use specific ports
ENDPOINT_PORT=10001 bash disagg_1e1p1d_example.sh

# Use specific model
MODEL="Qwen/Qwen2.5-VL-3B-Instruct" bash disagg_1e1p1d_example.sh

# Use specific storage path
EC_SHARED_STORAGE_PATH="/tmp/my_ec_cache" bash disagg_1e1p1d_example.sh

# Run on XPU; scripts switch from CUDA_VISIBLE_DEVICES to ZE_AFFINITY_MASK
DEVICE_PLATFORM=xpu GPU_E=0 GPU_PD=1 bash disagg_1e1pd_example.sh
```

--------------------------------

### Start vLLM Server with Reasoning and Tool Calling

Source: https://docs.vllm.ai/en/stable/examples/reasoning/openai_chat_completion_tool_calls_with_reasoning

To run this example, start the vLLM server with the reasoning parser and tool calling enabled. This command specifies the model, the reasoning parser, and enables auto-tool-choice with the Hermes tool call parser.

```bash
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""
An example demonstrates how to use tool calling with reasoning models 
like QwQ-32B. The reasoning will not be parsed by the tool 
calling process; only the final output will be parsed.

To run this example, you need to start the vLLM server with both 
the reasoning parser and tool calling enabled.

```bash
vllm serve Qwen/QwQ-32B \
     --reasoning-parser deepseek_r1 \
     --enable-auto-tool-choice --tool-call-parser hermes

```

"""

from openai import OpenAI


# Now, simulate a tool call
def get_current_weather(city: str, state: str, unit: "str"):
    return (
        "The weather in Dallas, Texas is 85 degrees fahrenheit. It is "
        "partly cloudly, with highs in the 90's."
    )


available_tools = {"get_current_weather": get_current_weather}

```

--------------------------------

### Serve a Base Model

Source: https://docs.vllm.ai/en/stable/examples/features/structured_outputs?q=

Start a vLLM server with a chosen model. This is the basic command to get a server running.

```bash
vllm serve Qwen/Qwen2.5-3B-Instruct

```

--------------------------------

### Setup pre-commit Hooks

Source: https://docs.vllm.ai/en/stable/contributing

Install pre-commit and set up the hooks to run automatically on every commit for linting and formatting the codebase.

```bash
uv pip install pre-commit>=4.5.1
pre-commit install

```

--------------------------------

### RAG Example with Langchain and vLLM

Source: https://docs.vllm.ai/en/stable/examples/applications/rag

This snippet shows the basic setup for a RAG application using Langchain's QA chain with vLLM as the LLM backend. Ensure Langchain and vLLM are installed.

```python
# SPDX-License-Identifier: Apache-2.0
from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI
from langchain.document_loaders import PyPDFLoader
from langchain.embeddings import OpenAIEmbeddings
from langchain.llms import OpenAI
from langchain.memory import ConversationBufferMemory
from langchain.vectorstores import Chroma

# Example usage:
# 1. Load documents
loader = PyPDFLoader("example.pdf")
documents = loader.load()

# 2. Create embeddings and vector store
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(documents, embeddings)
retriever = vectorstore.as_retriever()

# 3. Initialize LLM (using vLLM via Langchain's OpenAI wrapper)
# Make sure to set the OPENAI_API_BASE environment variable to your vLLM endpoint
# e.g., export OPENAI_API_BASE="http://localhost:8000/v1"
llm = OpenAI(temperature=0)

# 4. Create QA chain
qa_chain = RetrievalQA.from_chain_type(
    llm,
    retriever=retriever,
    memory=ConversationBufferMemory(
        memory_key="chat_history", return_messages=True
    ),
)

# 5. Ask a question
question = "What is the main topic of the document?"
result = qa_chain({"query": question})
print(result["result"])

```

--------------------------------

### Initialize Qwen3NextConfig and Model

Source: https://docs.vllm.ai/en/stable/api/vllm/transformers_utils/configs/qwen3_next?q=

Demonstrates how to initialize a Qwen3Next configuration and then a Qwen3NextModel using that configuration. Also shows how to access the model's configuration after initialization.

```python
>>> from transformers import Qwen3NextModel, Qwen3NextConfig

>>> # Initializing a Qwen3Next style configuration
>>> configuration =  Qwen3NextConfig()

>>> # Initializing a model from the Qwen-3-Next-80B-A3B style configuration
>>> model = Qwen3NextModel(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config
```

--------------------------------

### Start vLLM Server with Tool Calling

Source: https://docs.vllm.ai/en/stable/serving/integrations/codex

Launch the vLLM server with a model that supports tool calling. This example uses Qwen/Qwen3.6-27B and specifies necessary flags for tool calling and reasoning.

```bash
vllm serve Qwen/Qwen3.6-27B --port 8000 --tensor-parallel-size 8 --max-model-len 262144 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder

```

--------------------------------

### vLLM GUI Setup and Run Configuration

Source: https://docs.vllm.ai/en/stable/deployment/frameworks/skypilot?q=

This YAML configuration defines environment variables, resource requirements, setup commands (conda environment, pip installs), and the run command to start a Gradio server for a vLLM model. It clones the vLLM repository and launches the chatbot web server, directing requests to the specified vLLM API endpoint.

```yaml
envs:
  MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
  ENDPOINT: x.x.x.x:3031 # Address of the API server running vllm.

resources:
  cpus: 2

setup: |
  conda create -n vllm python=3.10 -y
  conda activate vllm

  # Install Gradio for web UI.
  pip install gradio openai

run: |
  conda activate vllm
  export PATH=$PATH:/sbin

  echo 'Starting gradio server...'
  git clone https://github.com/vllm-project/vllm.git || true
  python vllm/examples/applications/api_client/gradio_openai_chatbot_webserver.py \
    -m $MODEL_NAME \
    --port 8811 \
    --model-url http://$ENDPOINT/v1 \
    --stop-token-ids 128009,128001 | tee ~/gradio.log

```

--------------------------------

### Perform Post Initialization Setup

Source: https://docs.vllm.ai/en/stable/api/vllm/model_executor/layers/fused_moe/modular_kernel

Completes the setup by resolving dependencies between prepare_finalize and fused_experts, and asserting consistent activation formats.

```python
def _post_init_setup(self):
    """
    Resolve any leftover setup dependencies between self.prepare_finalize
    and self.fused_experts here.
    """
    self.prepare_finalize.post_init_setup(self.impl.fused_experts)
    assert (
        self.prepare_finalize.activation_format
        == self.fused_experts.activation_format()
    )

```

--------------------------------

### Async LLM Streaming Example

Source: https://docs.vllm.ai/en/stable/examples/deployment/async_llm_streaming

This script demonstrates streaming offline inference with vLLM's AsyncLLM engine. It shows how to get token-by-token output using DELTA mode streaming. Ensure you have vLLM installed and a compatible model downloaded.

```python
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""
Simple example demonstrating streaming offline inference with AsyncLLM (V1 engine).

This script shows the core functionality of vLLM's AsyncLLM engine for streaming
token-by-token output in offline inference scenarios. It demonstrates DELTA mode
streaming where you receive new tokens as they are generated.

Usage:
    python examples/deployment/async_llm_streaming.py
"""

import asyncio

from vllm import SamplingParams
from vllm.engine.arg_utils import AsyncEngineArgs
from vllm.sampling_params import RequestOutputKind
from vllm.v1.engine.async_llm import AsyncLLM


async def stream_response(engine: AsyncLLM, prompt: str, request_id: str) -> None:
    """
    Stream response from AsyncLLM and display tokens as they arrive.

    This function demonstrates the core streaming pattern:
    1. Create SamplingParams with DELTA output kind
    2. Call engine.generate() and iterate over the async generator
    3. Print new tokens as they arrive
    4. Handle the finished flag to know when generation is complete
    """
    print(f"\n🚀 Prompt: {prompt!r}")
    print("💬 Response: ", end="", flush=True)

    # Configure sampling parameters for streaming
    sampling_params = SamplingParams(
        max_tokens=100,
        temperature=0.8,
        top_p=0.95,
        seed=42,  # For reproducible results
        output_kind=RequestOutputKind.DELTA,  # Get only new tokens each iteration
    )

    try:
        # Stream tokens from AsyncLLM
        async for output in engine.generate(
            request_id=request_id, prompt=prompt, sampling_params=sampling_params
        ):
            # Process each completion in the output
            for completion in output.outputs:
                # In DELTA mode, we get only new tokens generated since last iteration
                new_text = completion.text
                if new_text:
                    print(new_text, end="", flush=True)

            # Check if generation is finished
            if output.finished:
                print("\n✅ Generation complete!")
                break

    except Exception as e:
        print(f"\n❌ Error during streaming: {e}")
        raise


async def main():
    print("🔧 Initializing AsyncLLM...")

    # Create AsyncLLM engine with simple configuration
    engine_args = AsyncEngineArgs(
        model="meta-llama/Llama-3.2-1B-Instruct",
        enforce_eager=True,  # Faster startup for examples
    )
    engine = AsyncLLM.from_engine_args(engine_args)

    try:
        # Example prompts to demonstrate streaming
        prompts = [
            "The future of artificial intelligence is",
            "In a galaxy far, far away",
            "The key to happiness is",
        ]

        print(f"🎯 Running {len(prompts)} streaming examples...")

        # Process each prompt
        for i, prompt in enumerate(prompts, 1):
            print(f"\n{'=' * 60}")
            print(f"Example {i}/{len(prompts)}")
            print(f"{'=' * 60}")

            request_id = f"stream-example-{i}"
            await stream_response(engine, prompt, request_id)

            # Brief pause between examples
            if i < len(prompts):
                await asyncio.sleep(0.5)

        print("\n🎉 All streaming examples completed!")

    finally:
        # Always clean up the engine
        print("🔧 Shutting down engine...")
        engine.shutdown()


if __name__ == "__main__":
    try:
        asyncio.run(main())
    except KeyboardInterrupt:
        print("\n🛑 Interrupted by user")

```

--------------------------------

### Shell Script for Disaggregated Prefill Setup

Source: https://docs.vllm.ai/en/stable/examples/disaggregated/lmcache

This script checks for necessary environment variables (HF_TOKEN), verifies GPU availability, ensures required Python libraries are installed, and launches the prefiller, decoder, and proxy servers. It then waits for servers to become available before starting a benchmark.

```bash
#!/bin/bash

# Switch to the directory of the current script
cd "$(dirname "${BASH_SOURCE[0]}")"

check_hf_token() {
    if [ -z "$HF_TOKEN" ]; then
        echo "HF_TOKEN is not set. Please set it to your Hugging Face token."
        exit 1
    fi
    if [[ "$HF_TOKEN" != hf_* ]]; then
        echo "HF_TOKEN is not a valid Hugging Face token. Please set it to your Hugging Face token."
        exit 1
    fi
    echo "HF_TOKEN is set and valid."
}

check_num_gpus() {
    # can you check if the number of GPUs are >=2 via nvidia-smi/rocm-smi?
    if ! which rocm-smi > /dev/null 2>&1;
    then
    num_gpus=$(nvidia-smi --query-gpu=name --format=csv,noheader | wc -l)
    else
    num_gpus=$(rocm-smi --showid | grep -c Instinct)
    fi

    if [ "$num_gpus" -lt 2 ]; then
        echo "You need at least 2 GPUs to run disaggregated prefill."
        exit 1
    else
        echo "Found $num_gpus GPUs."
    fi
}

ensure_python_library_installed() {
    echo "Checking if $1 is installed..."
    if ! python3 -c "import $1" > /dev/null 2>&1;
    then
        if [ "$1" == "nixl" ]; then
            echo "$1 is not installed. Please refer to https://github.com/ai-dynamo/nixl for installation."
        else
            echo "$1 is not installed. Please install it via pip install $1."
        fi
        exit 1
    else
        echo "$1 is installed."
    fi
}

cleanup() {
    echo "Stopping everything…"
    trap - INT TERM        # prevent re-entrancy
    kill -- -$$            # negative PID  ==  “this whole process-group”
    wait                   # reap children so we don't leave zombies
    exit 0
}

wait_for_server() {
  local port=$1
  local timeout_seconds=1200
  local start_time=$(date +%s)

  echo "Waiting for server on port $port..."

  while true; do
    if curl -s "localhost:${port}/v1/completions" > /dev/null;
    then
      return 0
    fi

    local now=$(date +%s)
    if (( now - start_time >= timeout_seconds )); then
      echo "Timeout waiting for server"
      return 1
    fi

    sleep 1
  done
}


main() {
    check_hf_token
    check_num_gpus
    ensure_python_library_installed lmcache
    ensure_python_library_installed nixl
    ensure_python_library_installed pandas
    ensure_python_library_installed datasets
    ensure_python_library_installed vllm

    trap cleanup INT
    trap cleanup USR1
    trap cleanup TERM

    echo "Launching prefiller, decoder and proxy..."
    echo "Please check prefiller.log, decoder.log and proxy.log for logs."

    bash disagg_vllm_launcher.sh prefiller \
        > >(tee prefiller.log) 2>&1 &
    prefiller_pid=$!
    PIDS+=("$prefiller_pid")

    bash disagg_vllm_launcher.sh decoder  \
        > >(tee decoder.log)  2>&1 &
    decoder_pid=$!
    PIDS+=("$decoder_pid")

    python3 disagg_proxy_server.py \
        --host localhost \
        --port 9000 \
        --prefiller-host localhost \
        --prefiller-port 8100 \
        --decoder-host localhost \
        --decoder-port 8200  \
        > >(tee proxy.log)    2>&1 &
    proxy_pid=$!
    PIDS+=("$proxy_pid")

    wait_for_server 8100
    wait_for_server 8200
    wait_for_server 9000

    echo "All servers are up. Starting benchmark..."

    # begin benchmark
    cd ../../../../benchmarks/
    vllm bench serve --port 9000 --seed "$(date +%s)" \
        --model meta-llama/Llama-3.1-8B-Instruct \
        --dataset-name random --random-input-len 7500 --random-output-len 200 \
        --num-prompts 200 --burstiness 100 --request-rate 3.6 | tee benchmark.log

    echo "Benchmarking done. Cleaning up..."

    cleanup

}

main

```

--------------------------------

### Initialize ExampleSecondaryTierManager

Source: https://docs.vllm.ai/en/stable/api/vllm/v1/kv_offload/tiering/example/manager

Initializes the example secondary tier manager. Accepts a custom integer parameter for demonstration.

```python
def __init__(
    self,
    offloading_spec: "OffloadingSpec",
    primary_kv_view: memoryview,
    tier_type: str,
    custom_param: int = 0,
):
    """
    Initialize the example secondary tier.

    Args:
        custom_param: Dummy parameter demonstrating custom args.
    """
    super().__init__(
        offloading_spec=offloading_spec,
        primary_kv_view=primary_kv_view,
        tier_type=tier_type,
    )

    logger.info(
        "ExampleSecondaryTierManager initialized with custom_param=%d", custom_param
    )

    # key -> True (only care about presence)
    self.blocks: dict[OffloadKey, bool] = {}

    # Completed jobs waiting to be retrieved by get_finished()
    self.completed_jobs: list[JobResult] = []

```

--------------------------------

### Run Disaggregated Prefill Example (vLLM v1)

Source: https://docs.vllm.ai/en/stable/examples/disaggregated/lmcache

This script launches the main example for disaggregated prefill using NIXL on a single node with vLLM v1. Ensure prerequisites like LMCache, NIXL, and a Hugging Face token are met.

```bash
disagg_example_nixl.sh
```

--------------------------------

### Install InstantTensor

Source: https://docs.vllm.ai/en/stable/models/extensions/instanttensor

Install the InstantTensor library using pip. This is the first step before using InstantTensor.

```bash
pip install instanttensor

```

--------------------------------

### assert_ray_available

Source: https://docs.vllm.ai/en/stable/api/vllm/v1/executor/ray_utils

Checks if the Ray library is available and raises a ValueError if it is not installed or importable, guiding the user to install it.

```APIDOC
## assert_ray_available

### Description
Raises an exception if Ray is not available. This is a prerequisite check before using Ray functionalities.

### Method
None (Function call)

### Endpoint
None (Function call)

### Parameters
None

### Response
#### Success Response
None. The function returns if Ray is available.

#### Error Response
- **ValueError**: Raised if Ray is not imported successfully, with instructions to install it.
```

--------------------------------

### Define Startup Parameters for Startup Benchmark

Source: https://docs.vllm.ai/en/stable/benchmarking/sweeps?q=

Example JSON structure for `--startup-params` to vary startup-specific options like iteration counts for cold and warm startups.

```json
[
    {
        "_benchmark_name": "qwen3-0.6",
        "num_iters_cold": 2,
        "num_iters_warmup": 1,
        "num_iters_warm": 2
    }
]

```

--------------------------------

### Install NIXL Library

Source: https://docs.vllm.ai/en/stable/features/nixl_connector_usage?q=

Installs the NIXL library using pip. This is a quick start for Nvidia platforms.

```bash
uv pip install nixl
```

--------------------------------

### Start Gradio Chatbot Webserver

Source: https://docs.vllm.ai/en/stable/examples/applications/chatbot

This Python script starts a Gradio web server for a vLLM chatbot. It requires the vLLM API server to be running separately. Ensure Gradio is installed (`pip install --upgrade gradio`).

```python
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

import argparse
import json

import gradio as gr
import requests


def http_bot(prompt):
    headers = {"User-Agent": "vLLM Client"}
    pload = {
        "prompt": prompt,
        "stream": True,
        "max_tokens": 128,
    }
    response = requests.post(args.model_url, headers=headers, json=pload, stream=True)

    for chunk in response.iter_lines(
        chunk_size=8192, decode_unicode=False, delimiter=b"\n"
    ):
        if chunk:
            data = json.loads(chunk.decode("utf-8"))
            output = data["text"][0]
            yield output


def build_demo():
    with gr.Blocks() as demo:
        gr.Markdown("# vLLM text completion demo\n")
        inputbox = gr.Textbox(label="Input", placeholder="Enter text and press ENTER")
        outputbox = gr.Textbox(
            label="Output", placeholder="Generated result from the model"
        )
        inputbox.submit(http_bot, [inputbox], [outputbox])
    return demo

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument("--host", type=str, default=None)
    parser.add_argument("--port", type=int, default=8001)
    parser.add_argument(
        "--model-url", type=str, default="http://localhost:8000/generate"
    )
    return parser.parse_args()

def main(args):
    demo = build_demo()
    demo.queue().launch(server_name=args.host, server_port=args.port, share=True)


if __name__ == "__main__":
    args = parse_args()
    main(args)

```

--------------------------------

### Method: post_init_setup

Source: https://docs.vllm.ai/en/stable/api/vllm/model_executor/layers/fused_moe/modular_kernel?q=

Initializes settings that depend on the FusedMoEExperts object. Implementations can override this for custom setup.

```python
def post_init_setup(self, fused_experts: "FusedMoEExperts"):
    """
    Initialize FusedMoEPrepareAndFinalizeModular settings that depend on
    FusedMoEExpertsModular experts object.
    The FusedMoEPrepareAndFinalizeModular implementations that have such
    dependencies may choose to override this function.
    """
    return
```

--------------------------------

### Build vLLM with Existing PyTorch Installation

Source: https://docs.vllm.ai/en/stable/getting_started/installation/gpu?q=

Build vLLM using an existing PyTorch installation, which is useful when PyTorch cannot be installed with uv or when using non-default PyTorch builds. This involves cloning the repository, running a setup script, and installing requirements.

```bash
# install PyTorch first, either from PyPI or from source
git clone https://github.com/vllm-project/vllm.git
cd vllm
python use_existing_torch.py
uv pip install -r requirements/build/cuda.txt
uv pip install --no-build-isolation -e .

```

--------------------------------

### Main Function for Distributed Setup

Source: https://docs.vllm.ai/en/stable/examples/rl/rlhf_nccl_fsdp_ep

Initializes Ray, downloads model weights, sets up FSDP workers, creates a vLLM AsyncLLMEngine with expert parallelism, performs initial generation, and initializes NCCL weight transfer.

```python
async def main():
    ray.init()

    # Download model weights to local/shared disk once.
    local_model_path = snapshot_download(MODEL_NAME)
    print(f"[init] Model downloaded to {local_model_path}")

    # FSDP rendezvous address (single-node)
    fsdp_master_addr = get_ip()
    fsdp_master_port = get_open_port()

    # Launch 4 FSDP training workers.
    # Ray allocates 1 GPU per worker; AsyncLLMEngine's internal DP
    # placement groups will land on the remaining 4 GPUs.
    fsdp_workers = [
        FSDPTrainWorker.remote(
            local_model_path,
            rank,
            FSDP_WORLD_SIZE,
            fsdp_master_addr,
            fsdp_master_port,
        )
        for rank in range(FSDP_WORLD_SIZE)
    ]
    ray.get([w.get_rank.remote() for w in fsdp_workers])
    print(f"[init] {FSDP_WORLD_SIZE} FSDP training workers ready.")

    # Launch vLLM with expert parallelism + data parallelism.
    # AsyncLLMEngine with data_parallel_backend="ray" creates its own
    # placement groups internally — no manual placement group needed.
    print("[engine] Creating AsyncLLMEngine...")
    engine = create_async_engine(
        model=local_model_path,
        enforce_eager=True,
        tensor_parallel_size=INFERENCE_TP_SIZE,
        data_parallel_size=INFERENCE_DP_SIZE,
        enable_expert_parallel=True,
        distributed_executor_backend="ray",
        data_parallel_backend="ray",
        weight_transfer_config=WeightTransferConfig(backend="nccl"),
        load_format="dummy",
        gpu_memory_utilization=0.7,
    )
    print("[engine] AsyncLLMEngine created.")

    prompts = [
        "Hello, my name is",
        "The president of the United States is",
        "The capital of France is",
        "The future of AI is",
    ]
    sampling_params = SamplingParams(temperature=0)

    # Generate with dummy weights — expect gibberish.
    print("[generate] Starting generation with dummy weights...")
    outputs = await generate_batch(engine, prompts, sampling_params)
    print("[generate] Generation complete.")

    print("-" * 60)
    print("BEFORE weight sync (dummy weights):")
    print("-" * 60)
    for output in outputs:
        print(f"Prompt: {output.prompt!r}")
        print(f"Generated: {output.outputs[0].text!r}")
        print("-" * 60)

    # --- Weight-transfer setup ---
    print("[transfer] Setting up weight-transfer endpoint...")
    transfer_addr, transfer_port = ray.get(
        fsdp_workers[0].setup_transfer_endpoint.remote()
    )
    print(f"[transfer] Endpoint ready at {transfer_addr}:{transfer_port}")

    transfer_world_size = INFERENCE_TP_SIZE * INFERENCE_DP_SIZE + 1
    print(
        f"[transfer] World size: {transfer_world_size} "
        f"(1 trainer + {INFERENCE_TP_SIZE * INFERENCE_DP_SIZE} vLLM workers)"
    )

    print("[transfer] Initializing NCCL groups...")
    train_handle = fsdp_workers[0].init_weight_transfer_group.remote(
        transfer_world_size
    )
    await engine.init_weight_transfer_engine(
        WeightTransferInitRequest(
            init_info=asdict(
                NCCLWeightTransferInitInfo(
                    master_address=transfer_addr,
                    master_port=transfer_port,
                    rank_offset=1,
                    world_size=transfer_world_size,
                )
            )
        )
    )
    ray.get(train_handle)
    print("[transfer] NCCL groups initialized.")

    # --- Pause, transfer weights, resume ---
    print("[sync] Pausing generation...")
```

--------------------------------

### Get Layer Start and End Indices

Source: https://docs.vllm.ai/en/stable/api/vllm/config/model

Calculates the start and end indices of the layers assigned to the current GPU based on pipeline parallelism configuration.

```python
from vllm.distributed.utils import get_pp_indices

total_num_hidden_layers = self.get_total_num_hidden_layers()

# the layout order is: DP x PP x TP
pp_rank = (
    parallel_config.rank // parallel_config.tensor_parallel_size
) % parallel_config.pipeline_parallel_size
pp_size = parallel_config.pipeline_parallel_size
start, end = get_pp_indices(total_num_hidden_layers, pp_rank, pp_size)
return start, end
```

--------------------------------

### Define Serve Parameters for Startup Benchmark

Source: https://docs.vllm.ai/en/stable/benchmarking/sweeps?q=

Example JSON structure for `--serve-params` to vary engine settings like tensor parallel size and GPU memory utilization for startup benchmarks.

```json
[
    {
        "_benchmark_name": "tp1",
        "model": "Qwen/Qwen3-0.6B",
        "tensor_parallel_size": 1,
        "gpu_memory_utilization": 0.9
    },
    {
        "_benchmark_name": "tp2",
        "model": "Qwen/Qwen3-0.6B",
        "tensor_parallel_size": 2,
        "gpu_memory_utilization": 0.9
    }
]

```

--------------------------------

### Build and Install vLLM from Source with uv

Source: https://docs.vllm.ai/en/stable/getting_started/installation/cpu?q=

Build the wheel and install vLLM using uv, specifying CPU target and extra index URL for PyTorch.

```bash
uv pip install -v \
        --extra-index-url https://download.pytorch.org/whl/cpu \
        --torch-backend auto \
        -r requirements/build/cpu.txt \
        -r requirements/cpu.txt \
    VLLM_TARGET_DEVICE=cpu python setup.py bdist_wheel && \
        uv pip install dist/*.whl
```

--------------------------------

### Get Decoder Start Token ID

Source: https://docs.vllm.ai/en/stable/api/vllm/renderers/base?q=

Retrieves the decoder start token ID for encoder-decoder models. Raises a RuntimeError if neither the decoder start token ID nor the beginning-of-sequence (BOS) token ID can be found.

```python
def get_dec_start_token_id(self) -> int:
    """
    Obtain the decoder start token id employed by an encoder/decoder model,
    raising an error if it is not available.
    """
    dec_start_token_id = getattr(
        self.model_config.hf_config, "decoder_start_token_id", None
    )

    if dec_start_token_id is None:
        logger.warning_once(
            "Falling back on <BOS> for decoder start token id "
            "because decoder start token id is not available."
        )
        dec_start_token_id = self.get_bos_token_id()

    if dec_start_token_id is None:
        raise RuntimeError("Cannot find decoder start token id or <BOS>")

    return dec_start_token_id
```

--------------------------------

### Initialize Qwen3NextConfig and Model

Source: https://docs.vllm.ai/en/stable/api/vllm/transformers_utils/configs/qwen3_next

Demonstrates how to initialize a Qwen3Next configuration and then a Qwen3NextModel using that configuration. This is useful for setting up the model with default or custom parameters.

```python
>>> from transformers import Qwen3NextModel, Qwen3NextConfig

>>> # Initializing a Qwen3-Next style configuration
>>> configuration =  Qwen3NextConfig()

>>> # Initializing a model from the Qwen3-Next-80B-A3B style configuration
>>> model = Qwen3NextModel(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config
```

--------------------------------

### Start P2P NCCL Proxy

Source: https://docs.vllm.ai/en/stable/design/p2p_nccl_connector

Starts the disaggregated proxy service. Ensure 'quart' is installed on the proxy node. This script listens on ports 10001 and 30001.

```bash
cd {your vllm directory}/examples/disaggregated/p2p_nccl_xpyd/
python3 disagg_proxy_p2p_nccl_xpyd.py &
```

--------------------------------

### Initialize Qwen3ASRTextConfig and Model

Source: https://docs.vllm.ai/en/stable/api/vllm/transformers_utils/configs/qwen3_asr

Demonstrates how to initialize a Qwen3ASRTextConfig with default settings and then use it to create a Qwen3ASRTextModel. It also shows how to access the model's configuration after initialization.

```python
from transformers import Qwen3ASRTextModel, Qwen3ASRTextConfig

# Initializing a Qwen3ASR style configuration
configuration = Qwen3ASRTextConfig()

# Initializing a model from the Qwen3-VL-7B style configuration
model = Qwen3ASRTextModel(configuration)

# Accessing the model configuration
configuration = model.config

```

--------------------------------

### Build and Install AITER

Source: https://docs.vllm.ai/en/stable/getting_started/installation/gpu?q=

Build and install AITER from source, allowing for the use of specific branches or commits. This involves cloning the repository, updating submodules, and running the setup script.

```bash
python3 -m pip uninstall -y aiter
git clone --recursive https://github.com/ROCm/aiter.git
cd aiter
git checkout $AITER_BRANCH_OR_COMMIT
git submodule sync; git submodule update --init --recursive
python3 setup.py develop
```

--------------------------------

### Get Decoder Start Token ID in vLLM

Source: https://docs.vllm.ai/en/stable/api/vllm/renderers/base?q=

Obtains the decoder start token ID for encoder-decoder models. Falls back to the BOS token ID if the specific decoder start token ID is not available, raising an error if neither can be found.

```python
def get_dec_start_token_id(self) -> int:
        """
        Obtain the decoder start token id employed by an encoder/decoder model,
        raising an error if it is not available.
        """
        dec_start_token_id = getattr(
            self.model_config.hf_config, "decoder_start_token_id", None
        )

        if dec_start_token_id is None:
            logger.warning_once(
                "Falling back on <BOS> for decoder start token id "
                "because decoder start token id is not available."
            )
            dec_start_token_id = self.get_bos_token_id()

        if dec_start_token_id is None:
            raise RuntimeError("Cannot find decoder start token id or <BOS>")

        return dec_start_token_id
```

--------------------------------

### Initialize dstack Project

Source: https://docs.vllm.ai/en/stable/deployment/frameworks/dstack?q=

Create and navigate to a new dstack project directory. Then, initialize the project configuration.

```bash
mkdir -p vllm-dstack
cd vllm-dstack
dstack init
```

--------------------------------

### ApertusToolParser Example Usage

Source: https://docs.vllm.ai/en/stable/api/vllm/tool_parsers/apertus_tool_parser?q=

Demonstrates how to initialize and use the ApertusToolParser to extract tool calls from a given output string. This example shows how to get the content and details of the extracted tool calls.

```python
>>> tokenizer = ...  # Mock tokenizer
>>> parser = ApertusToolParser(tokenizer)
>>> output = 'I will check. <|tools_prefix|>[{"get_weather": 
            '{"city": "Paris"}}]<|tools_suffix|>'
>>> request = ChatCompletionRequest(...)
>>> info = parser.extract_tool_calls(output, request)
>>> info.content
"I will check."
>>> info.tool_calls[0].function.name
"get_weather"
>>> info.tool_calls[0].function.arguments
'{"city": "Paris"}'
```

--------------------------------

### Initialize Qwen3NextConfig and Model

Source: https://docs.vllm.ai/en/stable/api/vllm/transformers_utils/configs/qwen3_next

Demonstrates how to initialize a Qwen3NextConfig with default parameters and then create a Qwen3NextModel from this configuration. It also shows how to access the model's configuration object.

```python
from transformers import Qwen3NextModel, Qwen3NextConfig

# Initializing a Qwen3Next style configuration
configuration = Qwen3NextConfig()

# Initializing a model from the Qwen3-Next-80B-A3B style configuration
model = Qwen3NextModel(configuration)

# Accessing the model configuration
configuration = model.config
```

--------------------------------

### Install vLLM Editable with Pre-compiled Wheels

Source: https://docs.vllm.ai/en/stable/contributing/incremental_build

Installs vLLM in editable mode, leveraging pre-compiled wheels for the initial setup to speed up the process. This is a prerequisite for the CMake incremental build workflow.

```bash
uv venv --python 3.12 --seed
source .venv/bin/activate
VLLM_USE_PRECOMPILED=1 uv pip install -U -e . --torch-backend=auto

```

--------------------------------

### Initialize Qwen3ASRTextConfig and Model

Source: https://docs.vllm.ai/en/stable/api/vllm/transformers_utils/configs/qwen3_asr

Demonstrates how to initialize a Qwen3ASRTextConfig object and then use it to create a Qwen3ASRTextModel. Shows how to access the model's configuration after initialization.

```python
>>> from transformers import Qwen3ASRTextModel, Qwen3ASRTextConfig

>>> # Initializing a Qwen3ASR style configuration
>>> configuration = Qwen3ASRTextConfig()

>>> # Initializing a model from the Qwen3-VL-7B style configuration
>>> model = Qwen3ASRTextModel(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config
```

--------------------------------

### Multi-Node Expert Parallel Deployment Example

Source: https://docs.vllm.ai/en/stable/serving/expert_parallel_deployment?q=

Example command for deploying a model across two nodes using the `deepep_low_latency` communication backend for Expert Parallelism. This setup is optimized for decode-dominated workloads.

```bash
# Multi-node EP deployment with deepep_low_latency
vllm serve deepseek-ai/DeepSeek-V3-0324 \
    --tensor-parallel-size 1 \
    --data-parallel-size 8 \
    --enable-expert-parallel \
    --all2all-backend deepep_low_latency

```

--------------------------------

### Start Serving a Model with vLLM

Source: https://docs.vllm.ai/en/stable/benchmarking/cli

Before running benchmarks, start serving your model using the vLLm serve command. This command loads the specified model and makes it available for requests.

```bash
vllm serve NousResearch/Hermes-3-Llama-3.1-8B
```