### Install dstack Client and Server

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/deployment/frameworks/dstack.md

Install the dstack client with all extras and start the dstack server. This is the initial setup for using dstack.

```console
pip install "dstack[all]"
dstack server
```

--------------------------------

### Build and Install vLLM CPU Backend

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/installation/cpu-x86.md

Builds and installs the vLLM CPU backend by setting the target device and running the setup script.

```bash
VLLM_TARGET_DEVICE=cpu python setup.py install
```

--------------------------------

### Serve Model with vLLM and Performance Settings

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/installation/cpu-x86.md

Starts vLLM serving with specified KV cache space and CPU thread binding. This example reserves CPU cores 30 and 31 for the framework.

```bash
export VLLM_CPU_KVCACHE_SPACE=40
export VLLM_CPU_OMP_THREADS_BIND=0-29
vllm serve facebook/opt-125m
```

--------------------------------

### Build vLLM for TPU

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/installation/tpu.md

Run the setup script to build and install vLLM with TPU support. Set the `VLLM_TARGET_DEVICE` environment variable to 'tpu'.

```bash
VLLM_TARGET_DEVICE="tpu" python setup.py develop
```

--------------------------------

### Start vLLM Server with Tool Calling Enabled

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/features/tool_calling.md

Enable tool calling features when starting the vLLM server. This example uses a specific chat template for Llama 3.1.

```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --enable-auto-tool-choice \
    --tool-call-parser llama3_json \
    --chat-template examples/tool_chat_template_llama3.1_json.jinja
```

--------------------------------

### Registering a Custom Model with vLLM Plugin

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/design/plugin_system.md

Example of a Python plugin setup using setuptools to register a custom model. This involves defining entry points in `setup.py` and implementing a registration function in a separate module.

```python
# inside `setup.py` file
from setuptools import setup

setup(name='vllm_add_dummy_model',
      version='0.1',
      packages=['vllm_add_dummy_model'],
      entry_points={
          'vllm.general_plugins':
          ["register_dummy_model = vllm_add_dummy_model:register"]
      })

# inside `vllm_add_dummy_model.py` file
def register():
    from vllm import ModelRegistry

    if "MyLlava" not in ModelRegistry.get_supported_archs():
        ModelRegistry.register_model("MyLlava",
                                        "vllm_add_dummy_model.my_llava:MyLlava")
```

--------------------------------

### Build and Run OpenVINO Environment with Docker

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/installation/openvino.md

Build a Docker image for the vLLM OpenVINO environment and run it interactively. This is a quick way to get started.

```console
$ docker build -f Dockerfile.openvino -t vllm-openvino-env .
$ docker run -it --rm vllm-openvino-env
```

--------------------------------

### Setup and Launch Disaggregated Prefill Instances

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/examples/disaggregated_prefill.md

This script sets up two vLLM instances for disaggregated prefilling. One instance handles prefilling and acts as a KV producer, while the other handles decoding and acts as a KV consumer. It includes installation of 'quart', server readiness checks, and trap for cleanup.

```bash
#!/bin/bash
# This file demonstrates the example usage of disaggregated prefilling
# We will launch 2 vllm instances (1 for prefill and 1 for decode),
# and then transfer the KV cache between them.

echo "🚧🚧 Warning: The usage of disaggregated prefill is experimental and subject to change 🚧🚧"
sleep 1

# Trap the SIGINT signal (triggered by Ctrl+C)
trap 'cleanup' INT

# Cleanup function
cleanup() {
    echo "Caught Ctrl+C, cleaning up..."
    # Cleanup commands
    pgrep python | xargs kill -9
    pkill -f python
    echo "Cleanup complete. Exiting."
    exit 0
}

export VLLM_HOST_IP=$(hostname -I | awk '{print $1}')

# install quart first -- required for disagg prefill proxy serve
if python3 -c "import quart" &> /dev/null; then
    echo "Quart is already installed."
else
    echo "Quart is not installed. Installing..."
    python3 -m pip install quart
fi 

# a function that waits vLLM server to start
wait_for_server() {
  local port=$1
  timeout 1200 bash -c "
    until curl -s localhost:${port}/v1/completions > /dev/null; do
      sleep 1
    done" && return 0 || return 1
}


# You can also adjust --kv-ip and --kv-port for distributed inference.

# prefilling instance, which is the KV producer
CUDA_VISIBLE_DEVICES=0 vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
    --port 8100 \
    --max-model-len 100 \
    --gpu-memory-utilization 0.8 \
    --kv-transfer-config \
    '{"kv_connector":"PyNcclConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2}' &

# decoding instance, which is the KV consumer
CUDA_VISIBLE_DEVICES=1 vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
    --port 8200 \
    --max-model-len 100 \
    --gpu-memory-utilization 0.8 \
    --kv-transfer-config \
    '{"kv_connector":"PyNcclConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2}' &

# wait until prefill and decode instances are ready
wait_for_server 8100
wait_for_server 8200

# launch a proxy server that opens the service at port 8000
# the workflow of this proxy:
# - send the request to prefill vLLM instance (port 8100), change max_tokens 
#   to 1
# - after the prefill vLLM finishes prefill, send the request to decode vLLM 
#   instance
# NOTE: the usage of this API is subject to change --- in the future we will 
# introduce "vllm connect" to connect between prefill and decode instances
python3 ../benchmarks/disagg_benchmarks/disagg_prefill_proxy_server.py &
sleep 1

# serve two example requests
output1=$(curl -X POST -s http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
"prompt": "San Francisco is a",
"max_tokens": 10,
"temperature": 0
}')

output2=$(curl -X POST -s http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
"prompt": "Santa Clara is a",
"max_tokens": 10,
"temperature": 0
}')


# Cleanup commands
pgrep python | xargs kill -9
pkill -f python

echo ""

sleep 1

# Print the outputs of the curl requests
echo ""
echo "Output of first request: $output1"
echo "Output of second request: $output2"

echo "🎉🎉 Successfully finished 2 test requests! 🎉🎉"
echo ""

```

--------------------------------

### Build and Install vLLM from Source

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/installation/hpu-gaudi.md

Clone the vLLM repository and install it in development mode using setup.py.

```bash
git clone https://github.com/vllm-project/vllm.git
cd vllm
python setup.py develop
```

--------------------------------

### Install Dependencies and Build Docs

Source: https://github.com/vonchenplus/vllm/blob/main/docs/README.md

Installs necessary Python packages and then builds the HTML documentation.

```bash
pip install -r requirements-docs.txt
```

```bash
make clean
make html
```

--------------------------------

### Download Example Batch File

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/examples/offline_inference_openai.md

Use wget to download the example batch file for local inference.

```bash
wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/offline_inference/offline_inference_openai/openai_example_batch.jsonl
```

--------------------------------

### Setup and Basic Chat Completion with Tools

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/examples/openai_chat_completion_client_with_tools.md

Configure the OpenAI client to connect to a vLLM server and perform a basic chat completion request with defined tools. Ensure your vLLM server is started with the appropriate flags for tool calling.

```python
import json

from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    # defaults to os.environ.get("OPENAI_API_KEY")
    api_key=openai_api_key,
    base_url=openai_api_base,
)

models = client.models.list()
model = models.data[0].id

tools = [{
    "type": "function",
    "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {
                    "type":
                    "string",
                    "description":
                    "The city to find the weather for, e.g. 'San Francisco'"
                },
                "state": {
                    "type":
                    "string",
                    "description":
                    "the two-letter abbreviation for the state that the city is"
                    " in, e.g. 'CA' which would mean 'California'"
                },
                "unit": {
                    "type": "string",
                    "description": "The unit to fetch the temperature in",
                    "enum": ["celsius", "fahrenheit"]
                }
            },
            "required": ["city", "state", "unit"]
        }
    }
}]

messages = [{
    "role": "user",
    "content": "Hi! How are you doing today?"
}, {
    "role": "assistant",
    "content": "I'm doing well! How can I help you?"
}, {
    "role":
    "user",
    "content":
    "Can you tell me what the temperate will be in Dallas, in fahrenheit?"
}]

chat_completion = client.chat.completions.create(messages=messages,
                                                 model=model,
                                                 tools=tools)

print("Chat completion results:")
print(chat_completion)
print("\n\n")

```

--------------------------------

### Install LangChain and Langchain-Community

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/serving/integrations/langchain.md

Install the necessary LangChain packages to use vLLM. The `-q` flag suppresses verbose output during installation.

```console
pip install langchain langchain_community -q
```

--------------------------------

### Start OpenAI-Compatible Server

Source: https://context7.com/vonchenplus/vllm/llms.txt

Command to start the vLLM server with a specified model, host, and port.

```bash
vllm serve Qwen/Qwen2.5-1.5B-Instruct --host 0.0.0.0 --port 8000
```

--------------------------------

### Install OpenTelemetry Packages

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/examples/opentelemetry.md

Install the necessary OpenTelemetry SDK, API, OTLP exporter, and semantic conventions for AI.

```bash
pip install \
  'opentelemetry-sdk>=1.26.0,<1.27.0' \
  'opentelemetry-api>=1.26.0,<1.27.0' \
  'opentelemetry-exporter-otlp>=1.26.0,<1.27.0' \
  'opentelemetry-semantic-conventions-ai>=0.4.1,<0.5.0'
```

--------------------------------

### Run OpenAI Pooling Client Example

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/examples/openai_pooling_client.md

This script demonstrates how to send requests to a vLLM server running the Pooling API. It requires the server to be started with `vllm serve <model> --task <embed|classify|reward|score>`. The client can send prompts in formats compatible with both the Completions API and the Chat API.

```python
import argparse
import pprint

import requests


def post_http_request(prompt: dict, api_url: str) -> requests.Response:
    headers = {"User-Agent": "Test Client"}
    response = requests.post(api_url, headers=headers, json=prompt)
    return response


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--host", type=str, default="localhost")
    parser.add_argument("--port", type=int, default=8000)
    parser.add_argument("--model",
                        type=str,
                        default="jason9693/Qwen2.5-1.5B-apeach")

    args = parser.parse_args()
    api_url = f"http://{args.host}:{args.port}/pooling"
    model_name = args.model

    # Input like Completions API
    prompt = {"model": model_name, "input": "vLLM is great!"}
    pooling_response = post_http_request(prompt=prompt, api_url=api_url)
    print("Pooling Response:")
    pprint.pprint(pooling_response.json())

    # Input like Chat API
    prompt = {
        "model":
        model_name,
        "messages": [{"role": "user", "content": [{"type": "text", "text": "vLLM is great!"}]}]
    }
    pooling_response = post_http_request(prompt=prompt, api_url=api_url)
    print("Pooling Response:")
    pprint.pprint(pooling_response.json())
```

--------------------------------

### Install vLLM from PyPI

Source: https://context7.com/vonchenplus/vllm/llms.txt

Install vLLM using uv (recommended) or pip. Ensure you are in a Python virtual environment.

```bash
# Using uv (recommended)
uv venv myenv --python 3.12 --seed
source myenv/bin/activate
uv pip install vllm

# Or using pip
pip install vllm
```

--------------------------------

### Clone and Install vLLM from Source on macOS

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/installation/cpu-apple.md

Use these commands to clone the vLLM repository, install CPU-specific requirements, and then install vLLM in editable mode. This is the standard procedure for building from source on macOS.

```bash
$ git clone https://github.com/vllm-project/vllm.git
$ cd vllm
$ pip install -r requirements-cpu.txt
$ pip install -e .
```

--------------------------------

### Install AutoFP8 Library

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/features/quantization/fp8.md

Clone the AutoFP8 repository and install it in editable mode to use its quantization features.

```bash
git clone https://github.com/neuralmagic/AutoFP8.git
pip install -e AutoFP8
```

--------------------------------

### Install and Login to Cerebrium Client

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/deployment/frameworks/cerebrium.md

Install the Cerebrium client and log in to your account to manage deployments.

```bash
$ pip install cerebrium
$ cerebrium login
```

--------------------------------

### Install vLLM from Source for Neuron

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/installation/neuron.md

Clone the vLLM repository, install Neuron-specific requirements, and then install vLLM with the target device set to 'neuron'. This process ensures that vLLM is compiled with Neuron support.

```bash
$ git clone https://github.com/vllm-project/vllm.git
$ cd vllm
$ pip install -U -r requirements-neuron.txt
$ VLLM_TARGET_DEVICE="neuron" pip install .
```

--------------------------------

### Build and Install HabanaAI vLLM-fork

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/installation/hpu-gaudi.md

Clone the HabanaAI vLLM-fork repository, checkout the habana_main branch, and install it in development mode.

```bash
git clone https://github.com/HabanaAI/vllm-fork.git
cd vllm-fork
git checkout habana_main
python setup.py develop
```

--------------------------------

### Install lm-evaluation-harness

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/features/quantization/fp8.md

Install the lm-evaluation-harness library, version 0.4.4, which is used for evaluating the accuracy of quantized models.

```console
$ pip install vllm lm-eval==0.4.4
```

--------------------------------

### Serve Documentation Locally

Source: https://github.com/vonchenplus/vllm/blob/main/docs/README.md

Starts a local HTTP server to view the built documentation in a browser.

```bash
python -m http.server -d build/html/
```

--------------------------------

### Offline Inference with Choice Guided Decoding

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/features/structured_outputs.md

Configure guided decoding for offline inference using the `choice` parameter within `GuidedDecodingParams`. This example demonstrates classifying sentiment by restricting output to 'Positive' or 'Negative'.

```python
from vllm import LLM, SamplingParams
from vllm.sampling_params import GuidedDecodingParams

llm = LLM(model="HuggingFaceTB/SmolLM2-1.7B-Instruct")

guided_decoding_params = GuidedDecodingParams(choice=["Positive", "Negative"])
sampling_params = SamplingParams(guided_decoding=guided_decoding_params)
outputs = llm.generate(
    prompts="Classify this sentiment: vLLM is wonderful!",
    sampling_params=sampling_params,
)
print(outputs[0].outputs[0].text)
```

--------------------------------

### Structured Outputs - Guided Choice

Source: https://context7.com/vonchenplus/vllm/llms.txt

Python example using 'guided_choice' in extra_body for constrained output classification.

```python
from enum import Enum
from pydantic import BaseModel
from openai import OpenAI

client = OpenAI(api_key="-", base_url="http://localhost:8000/v1")
MODEL = "Qwen/Qwen2.5-3B-Instruct"

# Choice constraint
resp = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "Classify: vLLM is wonderful!"}],
    extra_body={"guided_choice": ["positive", "negative"]},
)
print(resp.choices[0].message.content)   # "positive"
```

--------------------------------

### Launch Prometheus and Grafana

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/examples/prometheus_grafana.md

Start Prometheus and Grafana servers using Docker Compose. Ensure Docker and Docker Compose are installed.

```bash
docker compose up
```

--------------------------------

### Initialize and Run LLMEngine

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/examples/llm_engine_example.md

This script demonstrates initializing the LLMEngine with arguments parsed from the command line, creating test prompts with different sampling parameters, and continuously processing these requests until all are finished.

```python
import argparse
from typing import List, Tuple

from vllm import EngineArgs, LLMEngine, RequestOutput, SamplingParams
from vllm.utils import FlexibleArgumentParser


def create_test_prompts() -> List[Tuple[str, SamplingParams]]:
    """Create a list of test prompts with their sampling parameters."""
    return [
        ("A robot may not injure a human being",
         SamplingParams(temperature=0.0, logprobs=1, prompt_logprobs=1)),
        ("To be or not to be,",
         SamplingParams(temperature=0.8, top_k=5, presence_penalty=0.2)),
        ("What is the meaning of life?",
         SamplingParams(n=2,
                        best_of=5,
                        temperature=0.8,
                        top_p=0.95,
                        frequency_penalty=0.1)),
    ]


def process_requests(engine: LLMEngine,
                     test_prompts: List[Tuple[str, SamplingParams]]):
    """Continuously process a list of prompts and handle the outputs."""
    request_id = 0

    while test_prompts or engine.has_unfinished_requests():
        if test_prompts:
            prompt, sampling_params = test_prompts.pop(0)
            engine.add_request(str(request_id), prompt, sampling_params)
            request_id += 1

        request_outputs: List[RequestOutput] = engine.step()

        for request_output in request_outputs:
            if request_output.finished:
                print(request_output)


def initialize_engine(args: argparse.Namespace) -> LLMEngine:
    """Initialize the LLMEngine from the command line arguments."""
    engine_args = EngineArgs.from_cli_args(args)
    return LLMEngine.from_engine_args(engine_args)


def main(args: argparse.Namespace):
    """Main function that sets up and runs the prompt processing."""
    engine = initialize_engine(args)
    test_prompts = create_test_prompts()
    process_requests(engine, test_prompts)


if __name__ == '__main__':
    parser = FlexibleArgumentParser(
        description='Demo on using the LLMEngine class directly')
    parser = EngineArgs.add_cli_args(parser)
    args = parser.parse_args()
    main(args)

```

--------------------------------

### Structured Outputs - Guided Regex

Source: https://context7.com/vonchenplus/vllm/llms.txt

Python example using 'guided_regex' and 'stop' in extra_body for generating output matching a specific pattern.

```python
from enum import Enum
from pydantic import BaseModel
from openai import OpenAI

client = OpenAI(api_key="-", base_url="http://localhost:8000/v1")
MODEL = "Qwen/Qwen2.5-3B-Instruct"

# Regex constraint
resp = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "Generate an email for alan@enigma.com format:"}],
    extra_body={"guided_regex": r"\w+@\w+\.com", "stop": ["\n"]},
)
print(resp.choices[0].message.content)
```

--------------------------------

### Initialize dstack Project

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/deployment/frameworks/dstack.md

Create a new dstack project directory and navigate into it. This prepares the environment for dstack configurations.

```console
mkdir -p vllm-dstack
cd vllm-dstack
dstack init
```

--------------------------------

### Starting the vLLM Server

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/serving/openai_compatible_server.md

You can start the vLLM server using the `vllm serve` command, specifying the model and optional parameters like data type and API key. Alternatively, deployment can be done via Docker.

```APIDOC
## Start Server

### Command
```bash
vllm serve NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123
```

### Description
Starts the vLLM inference server with the specified model. `--dtype auto` automatically selects the data type, and `--api-key` sets an authentication token. Deployment can also be managed via Docker.
```

--------------------------------

### Start OpenAI-Compatible Server

Source: https://context7.com/vonchenplus/vllm/llms.txt

Launch the vLLM server with a specified model. Custom host, port, and API key can also be configured.

```bash
vllm serve Qwen/Qwen2.5-1.5B-Instruct
# Custom host/port and API key
vllm serve meta-llama/Meta-Llama-3-8B-Instruct --host 0.0.0.0 --port 8080 --api-key my-secret-key
```

--------------------------------

### Get Nested Structured Output with Pydantic Models

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/features/structured_outputs.md

Handle complex, step-by-step structured data by defining nested Pydantic models. This example demonstrates parsing a math solution with multiple steps.

```python
from typing import List
from pydantic import BaseModel
from openai import OpenAI


class Step(BaseModel):
    explanation: str
    output: str


class MathResponse(BaseModel):
    steps: List[Step]
    final_answer: str


client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="dummy")
completion = client.beta.chat.completions.parse(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful expert math tutor."},
        {"role": "user", "content": "Solve 8x + 31 = 2."},
    ],
    response_format=MathResponse,
    extra_body=dict(guided_decoding_backend="outlines"),
)

message = completion.choices[0].message
print(message)
assert message.parsed
for i, step in enumerate(message.parsed.steps):
    print(f"Step #{i}:", step)
print("Answer:", message.parsed.final_answer)
```

--------------------------------

### Distributed Inference with Ray Data

Source: https://context7.com/vonchenplus/vllm/llms.txt

Demonstrates distributed inference on a multi-node cluster using vLLM and Ray Data. This setup is suitable for large-scale batch workloads. Ensure Ray is installed and configured for distributed execution.

```python
import numpy as np
import ray
from ray.util.scheduling_strategies import PlacementGroupSchedulingStrategy
from vllm import LLM, SamplingParams

sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=128)
tensor_parallel_size = 1
num_instances = 2

class LLMPredictor:
    def __init__(self):
        self.llm = LLM(
            model="meta-llama/Llama-2-7b-chat-hf",
            tensor_parallel_size=tensor_parallel_size,
        )

    def __call__(self, batch: dict) -> dict:
        outputs = self.llm.generate(batch["text"].tolist(), sampling_params)
        return {
            "prompt":         [o.prompt for o in outputs],
            "generated_text": [o.outputs[0].text for o in outputs],
        }

# Read prompts from S3 / local files
ds = ray.data.read_text("s3://anonymous@air-example-data/prompts.txt")

ds = ds.map_batches(
    LLMPredictor,
    concurrency=num_instances,
    num_gpus=1,
    batch_size=32,
)

# Inspect first 10 results
for row in ds.take(limit=10):
    print(f"Prompt: {row['prompt']!r} -> {row['generated_text']!r}")

# Write full output to S3
# ds.write_parquet("s3://my-bucket/inference-output/")
```

--------------------------------

### Install vLLM with OpenVINO Backend from Source

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/installation/openvino.md

Installs the vLLM library with the OpenVINO backend from source. Sets the target device and PyTorch wheel URL.

```console
$ PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu" VLLM_TARGET_DEVICE=openvino python -m pip install -v .
```

--------------------------------

### Score API Request using curl

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/serving/openai_compatible_server.md

Example of sending a POST request to the vLLM Score API using curl to get similarity scores between two sentences. This demonstrates the basic request structure and expected JSON payload.

```bash
curl -X 'POST' \
  'http://127.0.0.1:8000/score' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{ \
  "model": "BAAI/bge-reranker-v2-m3", \
  "encoding_format": "float", \
  "text_1": "What is the capital of France?", \
  "text_2": "The capital of France is Paris."
}'
```

--------------------------------

### OpenAI Completion Client Example

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/examples/openai_completion_client.md

This example shows how to set up the OpenAI client to communicate with a vLLM API server. It demonstrates listing models and performing a text completion request, with options for streaming or non-streaming output. Ensure your vLLM API server is running before executing.

```python
from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    # defaults to os.environ.get("OPENAI_API_KEY")
    api_key=openai_api_key,
    base_url=openai_api_base,
)

models = client.models.list()
model = models.data[0].id

# Completion API
stream = False
completion = client.completions.create(
    model=model,
    prompt="A robot may not injure a human being",
    echo=False,
    n=2,
    stream=stream,
    logprobs=3)

print("Completion results:")
if stream:
    for c in completion:
        print(c)
else:
    print(completion)
```

--------------------------------

### vLLM Gradio GUI YAML Configuration

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/deployment/frameworks/skypilot.md

Configure the environment, resources, setup, and run commands for deploying a vLLM service with a Gradio web UI. This YAML file specifies model details, endpoint addresses, and installation steps for Gradio and OpenAI libraries.

```yaml
envs:
  MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
  ENDPOINT: x.x.x.x:3031 # Address of the API server running vllm.

resources:
  cpus: 2

setup: |
  conda create -n vllm python=3.10 -y
  conda activate vllm

  # Install Gradio for web UI.
  pip install gradio openai

run: |
  conda activate vllm
  export PATH=$PATH:/sbin

  echo 'Starting gradio server...'
  git clone https://github.com/vllm-project/vllm.git || true
  python vllm/examples/online_serving/gradio_openai_chatbot_webserver.py \
    -m $MODEL_NAME \
    --port 8811 \
    --model-url http://$ENDPOINT/v1 \
    --stop-token-ids 128009,128001 | tee ~/gradio.log
```

--------------------------------

### Pythonic Tool Call Example

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/features/tool_calling.md

Demonstrates how models can output a Python list to represent tool calls, inherently supporting parallel calls and avoiding JSON schema ambiguities. This format is useful when models do not emit specific tokens to start or end tool calls.

```python
[get_weather(city='San Francisco', metric='celsius'), get_weather(city='Seattle', metric='celsius')]
```

--------------------------------

### Install vLLM OpenVINO Prerequisites

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/installation/openvino.md

Installs pip and the necessary build requirements for the vLLM OpenVINO backend. Ensure you have `requirements-build.txt` available.

```console
$ pip install --upgrade pip
$ pip install -r requirements-build.txt --extra-index-url https://download.pytorch.org/whl/cpu
```

--------------------------------

### Install vLLM with ROCm Support

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/installation/gpu-rocm.md

Follow these steps to install vLLM for ROCm 6.2. This includes upgrading pip, installing a specific PyTorch version for ROCm, AMD SMI, and necessary dependencies. Note that `pip install .` is not supported for ROCm installations.

```bash
pip install --upgrade pip

# Install PyTorch
pip uninstall torch -y
pip install --no-cache-dir --pre torch==2.6.0.dev20241024 --index-url https://download.pytorch.org/whl/nightly/rocm6.2

# Build & install AMD SMI
pip install /opt/rocm/share/amd_smi

# Install dependencies
pip install --upgrade numba scipy huggingface-hub[cli]
pip install "numpy<2"
pip install -r requirements-rocm.txt

# Build vLLM for MI210/MI250/MI300.
export PYTORCH_ROCM_ARCH="gfx90a;gfx942"
python3 setup.py develop
```

--------------------------------

### Install vLLM with conda

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/quickstart.md

Installs vLLM in a new Python environment using conda. Ensure conda is installed first.

```bash
$ conda create -n myenv python=3.12 -y
$ conda activate myenv
$ pip install vllm
```

--------------------------------

### LoRA-augmented Request Example

Source: https://context7.com/vonchenplus/vllm/llms.txt

Demonstrates how to load a LoRA adapter on demand for a specific request. Ensure the LoRA adapter is correctly specified with its name, ID, and path.

```python
from vllm import LLM, SamplingParams, LoRARequest

lora_request = LoRARequest(
    lora_name="sql-lora",     # unique name
    lora_int_id=1,            # integer ID for caching
    lora_path="/path/to/sql-lora-adapter",
)
llm = LLM()
sampling_params = SamplingParams(temperature=0.8, max_tokens=128)
lora_out = llm.generate(
    "[user] Write a SQL query for table 'orders' to find total revenue [/user] [assistant]",
    sampling_params,
    lora_request=lora_request,
)
print(lora_out[0].outputs[0].text)
```

--------------------------------

### Install vLLM with pip

Source: https://github.com/vonchenplus/vllm/blob/main/README.md

Use this command to install the vLLM library using pip. Ensure you have Python and pip installed.

```bash
pip install vllm
```

--------------------------------

### Install SkyPilot and Check Environment

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/deployment/frameworks/skypilot.md

Install the SkyPilot nightly build and verify that your cloud or Kubernetes environment is correctly configured.

```bash
pip install skypilot-nightly
sky check
```

--------------------------------

### Install LlamaIndex vLLM Integration

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/serving/integrations/llamaindex.md

Install the necessary package to use vLLM with LlamaIndex. Use the `-q` flag for a quiet installation.

```console
pip install llama-index-llms-vllm -q
```

--------------------------------

### Bucketing Example with Ramp-up

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/installation/hpu-gaudi.md

Illustrates how bucketing ranges are determined with ramp-up when min is 2, step is 32, and max is 64. The ramp-up phase includes powers of two until the step is reached, followed by stable buckets.

```text
min = 2, step = 32, max = 64
=> ramp_up = (2, 4, 8, 16)
=> stable = (32, 64)
=> buckets = ramp_up + stable => (2, 4, 8, 16, 32, 64)
```

--------------------------------

### Serve Model with Infiniband and NCCL Debug

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/serving/distributed_serving.md

Example of serving a model with Infiniband enabled for efficient cross-node communication and NCCL debugging enabled to trace communication performance.

```bash
NCCL_DEBUG=TRACE vllm serve /path/to/the/model/in/the/container \
     --tensor-parallel-size 8 \
     --pipeline-parallel-size 2
```

--------------------------------

### Install vLLM with uv

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/quickstart.md

Installs vLLM in a new Python environment using the uv package manager. Ensure uv is installed first.

```bash
$ uv venv myenv --python 3.12 --seed
$ source myenv/bin/activate
$ uv pip install vllm
```

--------------------------------

### Prevent Dependency Installation

Source: https://github.com/vonchenplus/vllm/blob/main/CMakeLists.txt

Configures CMake to prevent the installation of dependencies like 'cutlass' by default. This can be controlled via CMake install components.

```cmake
install(CODE "set(CMAKE_INSTALL_LOCAL_ONLY TRUE)" ALL_COMPONENTS)
```

--------------------------------

### Install Triton Dependencies

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/installation/gpu-rocm.md

Install necessary build tools like ninja, cmake, and wheel before installing Triton. This is a prerequisite for building Triton from source.

```bash
$ python3 -m pip install ninja cmake wheel pybind11
```

--------------------------------

### Start OpenAI-Compatible API Server

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/design/arch_overview.md

Start the vLLM OpenAI-compatible API server using the `vllm serve` command followed by the model name. This command serves models for online inference.

```bash
vllm serve <model>
```

--------------------------------

### Install python-json-logger

Source: https://github.com/vonchenplus/vllm/blob/main/docs/source/getting_started/examples/logging_configuration.md

Install the necessary package for JSON log formatting.

```bash
pip install python-json-logger
```