### Minimal Configuration (OpenChat)

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/configuration.md

Sets the model name, quantization method, and GPU memory utilization for a minimal setup using OpenChat.

```bash
MODEL_NAME=openchat/openchat_3.5-0106
QUANTIZATION=awq
GPU_MEMORY_UTILIZATION=0.80
```

--------------------------------

### Example Invalid Config: LoRA Adapter

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/errors.md

Shows an incorrectly formatted JSON string for `LORA_MODULES` where the 'path' key is missing.

```bash
LORA_MODULES='{"name": "adapter1"}'  # Missing "path"
```

--------------------------------

### Python OpenAI SDK Example

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/endpoints.md

This Python snippet demonstrates how to use the OpenAI SDK to interact with RunPod's vLLM endpoints for chat completions.

```python
from openai import OpenAI

client = OpenAI(
    api_key="your-runpod-key",
    base_url="https://api.runpod.io/v2/{ENDPOINT_ID}/openai/v1"
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hi!"}],
    temperature=0.7,
    max_tokens=100,
    stream=False
)
print(response.choices[0].message.content)
```

--------------------------------

### vLLM Configuration File Example

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/configuration.md

Use this YAML format to configure vLLM parameters. Mount the file at `/vllm_config.yaml` or specify its path with `VLLM_CONFIG_FILE`. Environment variables take precedence over config file values.

```yaml
model: meta-llama/Llama-3.1-8B-Instruct
dtype: bfloat16
max-model-len: 8192
gpu-memory-utilization: 0.90
quantization: awq
tensor-parallel-size: 2
```

--------------------------------

### vLLM Worker Initialization Sequence

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/architecture.md

Outlines the step-by-step process from Docker container start to the first request arrival, detailing the initialization of vLLM and OpenAI engines.

```text
Docker Container Start
  ↓
__main__ block in handler.py (if MainProcess)
  ├─ Import vLLMEngine, OpenAIvLLMEngine
  ├─ Initialize vllm_engine = vLLMEngine()
  │  ├─ get_engine_args() → AsyncEngineArgs
  │  ├─ Load env vars, config file, defaults
  │  ├─ Initialize AsyncLLMEngine.from_engine_args()
  │  │  └─ CUDA init, model loading (slow step)
  │  └─ Create TokenizerWrapper
  ├─ Initialize openai_engine = OpenAIvLLMEngine(vllm_engine)
  │  ├─ Parse LORA_MODULES if set
  │  └─ Defer serving engine init to first request
  └─ Call runpod.serverless.start({"handler": handler, ...})
     └─ RunPod now accepts requests

First Request Arrives
  ↓
handler(job) called
  ├─ Create JobInput
  ├─ Select engine
  └─ Call engine.generate()
     └─ For OpenAI routes: first call triggers
        _ensure_engines_initialized()
        ├─ Initialize OpenAIServingChat, etc.
        └─ Warmup if supported
```

--------------------------------

### Worker Won't Start Debugging Checklist

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/errors.md

A checklist for diagnosing why the RunPod worker might not be starting. It covers checking logs, environment variables, GPU status, and model cache.

```bash
1. Check worker logs in RunPod console
2. Verify GPU available: nvidia-smi
3. Check env vars: docker exec ... env | grep -i model
4. Test model download locally: huggingface-cli scan-cache
5. Check vLLM version compatibility
```

--------------------------------

### Get Engine Arguments

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/api-reference.md

Build final AsyncEngineArgs from all configuration sources, prioritizing environment variables. This function retrieves arguments for vLLM initialization.

```python
from engine_args import get_engine_args

def get_engine_args() -> AsyncEngineArgs:
    pass
```

```python
from engine_args import get_engine_args

args = get_engine_args()
print(args.model)
print(args.tensor_parallel_size)
```

--------------------------------

### Text Completion Request Body Example

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/endpoints.md

This is an example of the JSON request body for text completion. It includes parameters like the model to use, the prompt, and generation settings.

```json
{
  "model": "meta-llama/Llama-3.1-8B",
  "prompt": "Write a poem about cats:",
  "temperature": 0.8,
  "max_tokens": 150,
  "stream": false
}
```

--------------------------------

### Error Response Example (400 Bad Request)

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/endpoints.md

Example of an error response when the request is malformed or invalid. This specific example shows a 'BadRequestError'.

```json
{
  "error": {
    "message": "Invalid model specified",
    "type": "BadRequestError",
    "code": 400
  }
}
```

--------------------------------

### Generate with OpenAIvLLMEngine for Chat Completions

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/api-reference.md

Make requests to the OpenAI-compatible API using the generate method of OpenAIvLLMEngine. This example demonstrates a chat completion request.

```python
from utils import JobInput

# Chat completion request
chat_request = {
    "openai_route": "/v1/chat/completions",
    "openai_input": {
        "model": "meta-llama/Llama-3.1-8B-Instruct",
        "messages": [{"role": "user", "content": "Hello"}],
        "temperature": 0.7,
        "max_tokens": 100
    }
}
request = JobInput(chat_request)

async for response in openai_engine.generate(request):
    print(response)
```

--------------------------------

### Chat Completions Streaming Chunk Example

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/endpoints.md

This is an example of a streaming chunk for chat completions, containing a partial message or token.

```json
{
  "choices": [
    {
      "index": 0,
      "delta": {
        "role": "assistant",
        "content": "token"
      },
      "finish_reason": null
    }
  ]
}
```

--------------------------------

### Error Response Example (500 Internal Server Error)

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/endpoints.md

Example of an error response indicating an issue on the server side during inference. This shows an 'InternalServerError'.

```json
{
  "error": {
    "message": "Internal server error during inference",
    "type": "InternalServerError",
    "code": 500
  }
}
```

--------------------------------

### Chat Completions Request Body Example

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/endpoints.md

This is an example of the JSON payload for the Chat Completions endpoint. It includes model selection, conversation messages, and generation parameters.

```json
{
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": "What is the capital of France?"
    }
  ],
  "temperature": 0.7,
  "max_tokens": 100,
  "stream": false
}
```

--------------------------------

### Responses API Request Body Example

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/endpoints.md

Example of a request body for the /v1/responses endpoint, which implements the OpenAI Responses API. It specifies the model and the input prompt.

```json
{
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "input": "Tell me a joke about programming."
}
```

--------------------------------

### Standard Request Generation

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/README.md

Send a standard text generation request to the vLLM engine. This example demonstrates streaming tokens back as they are generated.

```python
from utils import JobInput

request = {
    "prompt": "What is AI?",
    "sampling_params": {
        "temperature": 0.7,
        "max_tokens": 100
    }
}
job_input = JobInput(request)

async for batch in vllm_engine.generate(job_input):
    for choice in batch["choices"]:
        for token in choice["tokens"]:
            print(token, end="")
```

--------------------------------

### Using DummyRequest with Chat Completion

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/types.md

Example of how the DummyRequest object is used when calling the create_chat_completion method, passing it as the raw_request argument.

```python
dummy_request = DummyRequest()
response = await self.chat_engine.create_chat_completion(
    request_obj,
    raw_request=dummy_request
)
```

--------------------------------

### Multi-GPU Configuration (Llama 70B)

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/configuration.md

Configures a multi-GPU setup for a large model like Llama 70B, specifying tensor parallelism, GPU memory, and model length.

```bash
MODEL_NAME=meta-llama/Llama-2-70b-chat-hf
TENSOR_PARALLEL_SIZE=4
GPU_MEMORY_UTILIZATION=0.90
MAX_MODEL_LEN=4096
```

--------------------------------

### OpenAI Chat Completion Request

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/README.md

Make an OpenAI-compatible chat completion request using the OpenAIvLLMEngine wrapper. This example specifies the route and input parameters compatible with the OpenAI API.

```python
request = {
    "openai_route": "/v1/chat/completions",
    "openai_input": {
        "model": "meta-llama/Llama-3.1-8B-Instruct",
        "messages": [{"role": "user", "content": "Hello!"}],
        "temperature": 0.7,
        "max_tokens": 100
    }
}
job_input = JobInput(request)

async for response in openai_engine.generate(job_input):
    print(response)
```

--------------------------------

### Example Invalid Request: Sampling Parameters

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/errors.md

Demonstrates invalid sampling parameters that would trigger a BadRequestError. Ensure temperature, top_p, and max_tokens are within their valid ranges.

```json
{
  "sampling_params": {
    "temperature": -0.5,  // Invalid: must be >= 0
    "top_p": 1.5,         // Invalid: must be <= 1.0
    "max_tokens": -10     // Invalid: must be > 0
  }
}
```

--------------------------------

### vLLM Engine Args via Environment Variables

Source: https://github.com/runpod-workers/worker-vllm/blob/main/README.md

Configure vLLM engine arguments by setting environment variables. Any valid AsyncEngineArgs field name, when uppercased, will be automatically applied. For example, 'MAX_MODEL_LEN' maps to 'max_model_len'.

```bash
export MAX_MODEL_LEN=4096
export ENFORCE_EAGER=true
export ENABLE_CHUNKED_PREFILL=true
```

--------------------------------

### Handler Function Example

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/README.md

The main handler function called per request. It determines the appropriate engine based on the job input and yields batches of generated output.

```python
async def handler(job):
    """Called per request by RunPod"""
    job_input = JobInput(job["input"])
    engine = openai_engine if job_input.openai_route else vllm_engine
    async for batch in engine.generate(job_input):
        yield batch
```

--------------------------------

### Example Model Not Found Request Structure

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/errors.md

This Python dictionary illustrates a potential request structure where a model name might be misspelled or incorrect, leading to a 'Model Not Found' error. Verify the model ID against HuggingFace.

```python
request = {
    "model": "meta-llama/Llama3-8b",  # Typo: should be Llama-3.1-8B
    ...
}
```

--------------------------------

### Responses API Server-Sent Events Example

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/endpoints.md

This illustrates the server-sent events format used for responses from the /v1/responses API. It shows the structure for content block start, delta, and stop events.

```text
event: content_block_start
data: {"type": "content_block_start", "content_block": {"type": "text"}}

event: content_block_delta
data: {"type": "content_block_delta", "delta": {"type": "text_delta", "text": "token"}}

event: content_block_stop
data: {"type": "content_block_stop"}

event: message_stop
data: {"type": "message_stop"}
```

--------------------------------

### Initialize vLLMEngine

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/api-reference.md

Instantiate the vLLMEngine. It can be initialized with default environment configurations or by reusing an existing AsyncLLMEngine instance.

```python
from vllm import AsyncLLMEngine

# Initialize with defaults from environment
engine = vLLMEngine()

# Or reuse an existing async engine
args = AsyncEngineArgs()
existing_engine = AsyncLLMEngine.from_engine_args(args)
reused_engine = vLLMEngine(existing_engine)
```

--------------------------------

### Initialize vLLM Engines

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/README.md

Instantiate the standard vLLM engine or the OpenAI-compatible wrapper. Ensure the base vLLM engine is initialized before the wrapper.

```python
from engine import vLLMEngine, OpenAIvLLMEngine

# Standard vLLM
vllm_engine = vLLMEngine()

# OpenAI-compatible
openai_engine = OpenAIvLLMEngine(vllm_engine)
```

--------------------------------

### Initialize OpenAIvLLMEngine

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/api-reference.md

Wrap a base vLLMEngine instance with OpenAIvLLMEngine to provide an OpenAI-compatible API. This is useful for integrating with tools expecting OpenAI endpoints.

```python
base_engine = vLLMEngine()
openai_engine = OpenAIvLLMEngine(base_engine)
```

--------------------------------

### Example Invalid Chat Completion Request

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/errors.md

This example demonstrates an invalid chat completion request where the 'messages' field is a string instead of a list of dictionaries. This will trigger a BadRequestError.

```json
{
  "model": "llama",
  "messages": "Say hello"  // Should be list of dicts
}
```

--------------------------------

### Messages API Request Body Example

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/endpoints.md

An example of the JSON request body for the /v1/messages endpoint, which is compatible with the Anthropic Messages API. It includes the model, messages array, and system prompt.

```json
{
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "max_tokens": 256,
  "messages": [
    {
      "role": "user",
      "content": "Hello, how are you?"
    }
  ],
  "system": "You are a helpful assistant."
}
```

--------------------------------

### Build Image with OpenChat-3.5

Source: https://github.com/runpod-workers/worker-vllm/blob/main/README.md

Build a Docker image specifying the model name and base path for storage.

```bash
docker build -t username/image:tag --build-arg MODEL_NAME="openchat/openchat_3.5" --build-arg BASE_PATH="/models" .
```

--------------------------------

### Load and Update Configuration

Source: https://github.com/runpod-workers/worker-vllm/blob/main/docs/conventions.md

Loads default arguments, overrides them with environment variables, and then with local model-specific arguments. Finally, it validates these arguments against vLLM requirements.

```python
# Standard pattern for new configuration options
def get_engine_args():
    args = DEFAULT_ARGS
    args.update(os.environ)  # Environment override
    args.update(get_local_args())  # Baked model override
    return match_vllm_args(args)  # Validate against vLLM
```

--------------------------------

### OpenAIvLLMEngine Constructor

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/api-reference.md

Initializes the OpenAIvLLMEngine, wrapping a base vLLMEngine instance to provide OpenAI-compatible API endpoints.

```APIDOC
## OpenAIvLLMEngine Constructor

### Description
Initializes the OpenAIvLLMEngine, wrapping a base vLLMEngine instance to provide OpenAI-compatible API endpoints.

### Signature
```python
OpenAIvLLMEngine(vllm_engine: vLLMEngine)
```

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters Table
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| vllm_engine | vLLMEngine | Yes | Base vLLM engine instance to wrap with OpenAI compatibility. |

### Returns
OpenAIvLLMEngine instance

### Raises
Exception if LoRA adapters cannot be loaded

### Request Example
```python
base_engine = vLLMEngine()
openai_engine = OpenAIvLLMEngine(base_engine)
```

### Response
#### Success Response (200)
None

#### Response Example
None
```

--------------------------------

### OpenAIvLLMEngine.served_model_name

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/api-reference.md

Gets the model name exposed in OpenAI API responses.

```APIDOC
## OpenAIvLLMEngine.served_model_name

### Description
Gets the model name exposed in OpenAI API responses.

### Signature
```python
served_model_name: str
```

### Details
Model name exposed in OpenAI API responses. Set from environment variable `OPENAI_SERVED_MODEL_NAME_OVERRIDE` or engine args.

### Request Example
```python
# Accessing the property
model_name = openai_engine.served_model_name
print(f"Served model: {model_name}")
```
```

--------------------------------

### Initialize BatchSize for Dynamic Batching

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/api-reference.md

Set up a BatchSize manager with maximum, minimum, and growth factor parameters. This controls how batch sizes dynamically adjust during streaming responses.

```python
from utils import BatchSize

batch = BatchSize(max_batch_size=100, min_batch_size=10, batch_size_growth_factor=2)
while tokens_buffered < batch.current_batch_size:
    # collect tokens...
    pass
yield buffered_batch
batch.update()  # grows from 10 -> 20 -> 40 -> 80 -> 100
```

--------------------------------

### Example Error: Chat Template Missing

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/errors.md

This JSON structure represents a BadRequestError when a chat template is missing for a model.

```json
{
  "error": {
    "message": "Chat template does not exist for this model, you must provide a single string input instead of a list of messages",
    "type": "BadRequestError",
    "code": 400
  }
}
```

--------------------------------

### vLLMEngine.engine_args

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/api-reference.md

Exposes the complete configuration object for the vLLM engine.

```APIDOC
## vLLMEngine.engine_args

### Description
Exposes the complete configuration object for the vLLM engine.

### Signature
```python
engine_args: AsyncEngineArgs
```

### Details
Complete engine configuration object. Contains all vLLM engine parameters (model, dtype, quantization, tensor_parallel_size, etc.).

**Source**: `src/engine_args.py:445-606`

### Request Example
```python
# Accessing the property
config = engine.engine_args
print(f"Model: {config.model}")
```
```

--------------------------------

### Initialize OpenAI Client for Runpod

Source: https://github.com/runpod-workers/worker-vllm/blob/main/README.md

Initialize the OpenAI client using your Runpod API key and endpoint URL. Ensure the RUNPOD_API_KEY environment variable is set.

```python
from openai import OpenAI
import os

# Initialize the OpenAI Client with your Runpod API Key and Endpoint URL
client = OpenAI(
    api_key=os.environ.get("RUNPOD_API_KEY"),
    base_url="https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1",
)
```

--------------------------------

### Troubleshooting: Check Worker Logs

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/README.md

Use 'docker logs' to inspect the container's output for errors or status messages. This is a primary step for diagnosing startup issues.

```bash
# Check logs
docker logs container-id
```

--------------------------------

### vLLMEngine Constructor

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/api-reference.md

Initializes the vLLMEngine for inference operations. It can either create a new engine based on environment configuration or reuse an existing AsyncLLMEngine instance.

```APIDOC
## vLLMEngine Constructor

### Description
Initializes the vLLMEngine for inference operations. It can either create a new engine based on environment configuration or reuse an existing AsyncLLMEngine instance.

### Signature
```python
vLLMEngine(engine: AsyncLLMEngine | None = None)
```

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters Table
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| engine | AsyncLLMEngine \| None | No | None | Existing AsyncLLMEngine instance to reuse. If None, initializes a new engine from environment configuration. |

### Returns
vLLMEngine instance

### Raises
Exception if vLLM engine initialization fails

### Request Example
```python
# Initialize with defaults from environment
engine = vLLMEngine()

# Or reuse an existing async engine
from vllm import AsyncLLMEngine
existing_engine = AsyncLLMEngine.from_engine_args(args)
reused_engine = vLLMEngine(existing_engine)
```

### Response
#### Success Response (200)
None

#### Response Example
None
```

--------------------------------

### Correct Config: LoRA Adapter

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/errors.md

Provides the correct JSON format for `LORA_MODULES`, including the 'name' and 'path' for each adapter.

```bash
LORA_MODULES='[{"name": "adapter1", "path": "/path/to/adapter"}]'
```

--------------------------------

### Add LoRA Adapters from Hugging Face

Source: https://github.com/runpod-workers/worker-vllm/blob/main/docs/configuration.md

Configure LORA_MODULES with a list of dictionaries, each specifying the adapter 'name', 'path', and 'base_model_name' from Hugging Face.

```bash
LORA_MODULES='[{"name": "my-lora-adapter", "path": "./my-lora-weights", "base_model_name": "meta-llama/Llama-2-7b-hf"}]'
```

--------------------------------

### Node.js OpenAI Client for RunPod vLLM

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/endpoints.md

Initialize an OpenAI client to interact with a RunPod vLLM endpoint. Ensure the RUNPOD_API_KEY and ENDPOINT_ID environment variables are set.

```javascript
const OpenAI = require('openai');

const client = new OpenAI({
  apiKey: process.env.RUNPOD_API_KEY,
  baseURL: `https://api.runpod.io/v2/${process.env.ENDPOINT_ID}/openai/v1`,
});

const response = await client.chat.completions.create({
  model: 'meta-llama/Llama-3.1-8B-Instruct',
  messages: [{ role: 'user', content: 'Hello!' }],
  max_tokens: 100,
});
console.log(response.choices[0].message.content);
```

--------------------------------

### Configure Speculative Decoding with JSON

Source: https://github.com/runpod-workers/worker-vllm/blob/main/docs/configuration.md

Set SPECULATIVE_CONFIG to a JSON string containing the full speculative decoding configuration. This example uses the 'ngram' method.

```bash
SPECULATIVE_CONFIG='{"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 4}'
```

--------------------------------

### Build Image with vLLM Nightly and Specific Model

Source: https://github.com/runpod-workers/worker-vllm/blob/main/README.md

Combine VLLM_NIGHTLY with specific model and base path arguments for building the Docker image.

```bash
docker build -t username/image:tag --build-arg VLLM_NIGHTLY=true --build-arg MODEL_NAME="meta-llama/Llama-3.1-8B-Instruct" --build-arg BASE_PATH="/models" .
```

--------------------------------

### cURL Chat Completion Request

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/README.md

Example of making a chat completion request using cURL. This includes setting the endpoint, headers, and the JSON payload.

```bash
curl -X POST \
  https://api.runpod.io/v2/{ENDPOINT_ID}/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer {API_KEY}" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'
```

--------------------------------

### Initialize JobInput with vLLM or OpenAI format

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/api-reference.md

Instantiate JobInput with either a standard vLLM prompt and sampling parameters, or an OpenAI-compatible request structure. This normalizes diverse job payloads.

```python
from utils import JobInput

# Standard vLLM format
vllm_job = {
    "prompt": "Answer the question: what is 2+2?",
    "sampling_params": {
        "temperature": 0.0,
        "max_tokens": 10
    }
}
input_obj = JobInput(vllm_job)

# OpenAI format
openai_job = {
    "openai_route": "/v1/chat/completions",
    "openai_input": {
        "model": "gpt-3.5-turbo",
        "messages": [{"role": "user", "content": "Hello"}]
    }
}
input_obj = JobInput(openai_job)
```

--------------------------------

### Anthropic Messages API HTTP Request

Source: https://github.com/runpod-workers/worker-vllm/blob/main/README.md

Example cURL command to interact with the Anthropic Messages API on Runpod. Requires your endpoint ID and API key.

```bash
curl https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1/messages \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <YOUR RUNPOD API KEY>" \
  -d '{
    "model": "<YOUR DEPLOYED MODEL REPO/NAME>",
    "max_tokens": 256,
    "messages": [
      {"role": "user", "content": "Hello!"}
    ]
  }'
```

--------------------------------

### Build Image with vLLM Nightly

Source: https://github.com/runpod-workers/worker-vllm/blob/main/README.md

Build a Docker image using the latest unreleased vLLM build by setting VLLM_NIGHTLY to true.

```bash
docker build -t username/image:tag --build-arg VLLM_NIGHTLY=true .
```

--------------------------------

### OpenAI Responses API HTTP Request

Source: https://github.com/runpod-workers/worker-vllm/blob/main/README.md

Example cURL command to interact with the OpenAI Responses API on Runpod. Requires your endpoint ID and API key.

```bash
curl https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <YOUR RUNPOD API KEY>" \
  -d '{
    "model": "<YOUR DEPLOYED MODEL REPO/NAME>",
    "input": "Tell me a joke."
  }'
```

--------------------------------

### Error Response Example (500 Internal Server Error)

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/endpoints.md

This JSON structure is returned when the server encounters an internal error, such as a CUDA error or engine crash.

```json
{
  "error": {
    "type": "internal_error",
    "message": "Failed to generate messages"
  }
}
```

--------------------------------

### vLLM Async Engine Arguments

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/types.md

Configuration for the AsyncLLMEngine, automatically discovered from environment variables. Specifies model, quantization, parallelism, and memory utilization.

```python
class AsyncEngineArgs:
    # Model and Tokenizer
    model: str = "facebook/opt-125m"
    tokenizer: str | None = model
    revision: str = "main"
    dtype: str = "auto"
    quantization: str | None = None
    load_format: str = "auto"
    
    # Parallelism
    tensor_parallel_size: int = get_gpu_memory_utilization() # auto-detected
    pipeline_parallel_size: int = 1
    
    # KV Cache
    block_size: int = 16
    kv_cache_dtype: str = "auto"
    enable_prefix_caching: bool = False
    
    # Generation constraints
    max_model_len: int | None = None
    max_num_seqs: int = 256
    max_num_batched_tokens: int | None = max_model_len
    
    # GPU Memory
    gpu_memory_utilization: float = 0.95
    
    # LoRA
    enable_lora: bool = False
    max_loras: int = 1
    max_lora_rank: int = 16
    
    # CPU Offload
    swap_space: int = 4
    cpu_offload_gb: int = 0
    
    # Misc
    seed: int = 0
    trust_remote_code: bool = False
    worker_use_ray: bool = False
```

--------------------------------

### Models List API Response Example

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/endpoints.md

This JSON structure represents the response when querying the /v1/models endpoint. It lists the available models with their identifiers and ownership details.

```json
{
  "object": "list",
  "data": [
    {
      "id": "meta-llama/Llama-3.1-8B-Instruct",
      "object": "model",
      "owned_by": "runpod",
      "permission": [],
      "created": 1234567890
    }
  ]
}
```

--------------------------------

### get_engine_args

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/api-reference.md

Builds the final AsyncEngineArgs from all configuration sources, prioritizing environment variables. This function is crucial for initializing the vLLM engine with the correct settings.

```APIDOC
## get_engine_args

### Description
Build final AsyncEngineArgs from all configuration sources (defaults → config file → env vars → local args). Environment variables take highest precedence.

### Method
get_engine_args

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Returns
AsyncEngineArgs dataclass instance ready for vLLM initialization

### Priority Order (highest to lowest):
1. Environment variable aliases (MODEL_NAME → model)
2. Auto-discovered env vars (UPPERCASED field names)
3. Config file values (/vllm_config.yaml)
4. Baked-in model args (/local_model_args.json)
5. Worker defaults (DEFAULT_ARGS)
6. vLLM defaults

### Request Example
```python
from engine_args import get_engine_args

args = get_engine_args()
print(args.model)
print(args.tensor_parallel_size)
```
```

--------------------------------

### Chat Completions API Request

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/README.md

Example of a POST request to the chat completions endpoint using the OpenAI format. Ensure the model and messages are correctly specified.

```http
POST /v1/chat/completions
Content-Type: application/json

{
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "messages": [{"role": "user", "content": "Hi"}],
  "temperature": 0.7,
  "max_tokens": 100,
  "stream": false
}
```

--------------------------------

### Create Error Response Utility

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/types.md

Example of using a utility function to create an error response object. This is useful for standardizing error generation within the application.

```python
from utils import create_error_response

error = create_error_response("Invalid model", err_type="BadRequestError")
yield error.model_dump()
```

--------------------------------

### OpenAI Chat Completion Request Handling

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/architecture.md

Demonstrates processing an OpenAI-formatted chat completion request. Shows how the worker identifies the route, prepares input for the OpenAI engine, and initiates the generation process.

```python
# Request
job = {
  "input": {
    "openai_route": "/v1/chat/completions",
    "openai_input": {
      "model": "llama-3.1-8b",
      "messages": [{"role": "user", "content": "Hi"}],
      "temperature": 0.7
    }
  }
}

# Processing
job_input = JobInput(job["input"])
# → openai_route = "/v1/chat/completions"
# → openai_input = {...}

# Engine selection
engine = openai_engine  # since openai_route is set

# Routing
await openai_engine._ensure_engines_initialized()
# → Creates chat_engine, completion_engine, etc.

async for response in openai_engine._handle_chat_or_completion_request(job_input):
  # → Creates ChatCompletionRequest from openai_input
  # → Calls chat_engine.create_chat_completion()
  # → Returns OpenAI-formatted response
  yield response
```

--------------------------------

### Streaming Response with Dynamic Batching Configuration

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/architecture.md

Shows the configuration for streaming responses with dynamic batching. It details how batch size parameters are set and how the system accumulates tokens and updates batch size during generation.

```python
# Configuration
job = {
  "input": {
    "prompt": "Write 1000 words about...",
    "stream": True,
    "max_batch_size": 50,
    "min_batch_size": 10,
    "batch_size_growth_factor": 2
  }
}

# Processing
batch_size = BatchSize(50, 10, 2)
# → current_batch_size = 10 (starts at min)
# → is_dynamic = True

async for request_output in vllm_engine.generate(...):
  # Accumulate tokens
  token_counters["total"] += 1
  
  # When reaching batch threshold
  if token_counters["batch"] >= batch_size.current_batch_size:
    yield batch_dict
    token_counters["batch"] = 0
    batch_size.update()  # 10 → 20 → 40 → 50
```

--------------------------------

### OpenAIvLLMEngine.generate

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/api-reference.md

Handles requests for OpenAI-compatible API routes, processing various OpenAI and Anthropic API calls.

```APIDOC
## OpenAIvLLMEngine.generate

### Description
Handles requests for OpenAI-compatible API routes, processing various OpenAI and Anthropic API calls.

### Signature
```python
async def generate(self, openai_request: JobInput) -> AsyncGenerator[dict | str, None]
```

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters Table
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| openai_request | JobInput | Yes | Request with `openai_route` and `openai_input` fields specifying the API endpoint and payload. |

### Returns
- For `/v1/models`: dict with model list
- For `/v1/chat/completions` or `/v1/completions`: dict or streaming chunk
- For `/v1/responses`: dict or SSE event string
- For `/v1/messages`: dict or async chunk

### Raises
Yields error responses on validation failure

### Supported Routes
- `/v1/models` - List available models
- `/v1/chat/completions` - OpenAI chat completion API
- `/v1/completions` - OpenAI text completion API
- `/v1/responses` - OpenAI Responses API
- `/v1/messages` - Anthropic Messages API

### Request Example
```python
# Chat completion request
chat_request = {
    "openai_route": "/v1/chat/completions",
    "openai_input": {
        "model": "meta-llama/Llama-3.1-8B-Instruct",
        "messages": [{"role": "user", "content": "Hello"}],
        "temperature": 0.7,
        "max_tokens": 100
    }
}
request = JobInput(chat_request)

async for response in openai_engine.generate(request):
    print(response)
```

### Response
#### Success Response (200)
None

#### Response Example
None
```

--------------------------------

### Initialize OpenAI Client with Runpod Worker

Source: https://github.com/runpod-workers/worker-vllm/blob/main/README.md

Update the OpenAI client initialization to use your Runpod API key and Serverless Endpoint URL. Ensure your RUNPOD_API_KEY environment variable is set.

```python
from openai import OpenAI

client = OpenAI(
    api_key=os.environ.get("RUNPOD_API_KEY"),
    base_url="https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1",
)
```

--------------------------------

### Configure LoRA Adapters

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/architecture.md

Configure LoRA adapters by setting the LORA_MODULES environment variable with a JSON array of adapter configurations. These adapters are loaded in OpenAIvLLMEngine._load_lora_adapters().

```bash
LORA_MODULES='[{"name": "...", "path": "..."}]'
```

--------------------------------

### Chat Completions API Response (Non-Streaming)

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/README.md

Example of a non-streaming JSON response from the chat completions endpoint. The 'content' field within 'message' contains the assistant's reply.

```json
{
  "choices": [{
    "message": {"role": "assistant", "content": "..."},
    "finish_reason": "stop"
  }],
  "usage": {"prompt_tokens": 10, "completion_tokens": 20}
}
```

--------------------------------

### BatchSize Constructor

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/api-reference.md

Initializes a BatchSize manager for dynamic batching of streaming responses. It requires maximum and minimum batch sizes, along with a growth factor.

```APIDOC
## BatchSize Constructor

### Description
Initializes a BatchSize manager for dynamic batching of streaming responses. It requires maximum and minimum batch sizes, along with a growth factor.

### Constructor
```python
BatchSize(
    max_batch_size: int,
    min_batch_size: int,
    batch_size_growth_factor: int
)
```

### Parameters
#### Path Parameters
N/A

#### Query Parameters
N/A

#### Request Body
- **max_batch_size** (int) - Required - Maximum batch size to grow to.
- **min_batch_size** (int) - Required - Starting batch size.
- **batch_size_growth_factor** (int) - Required - Multiplier per update (e.g., 2 doubles size each time).

### Request Example
```python
from utils import BatchSize

batch = BatchSize(max_batch_size=100, min_batch_size=10, batch_size_growth_factor=2)
while tokens_buffered < batch.current_batch_size:
    # collect tokens...
    pass
yield buffered_batch
batch.update()  # grows from 10 -> 20 -> 40 -> 80 -> 100
```

### Response
#### Success Response (200)
N/A

#### Response Example
N/A
```

--------------------------------

### Messages API Response Example

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/endpoints.md

This JSON represents a successful response from the /v1/messages endpoint. It includes message ID, role, content, model used, stop reason, and token usage.

```json
{
  "id": "msg-xxxx",
  "type": "message",
  "role": "assistant",
  "content": [
    {
      "type": "text",
      "text": "Hello! I'm doing well, thank you for asking."
    }
  ],
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "stop_reason": "end_turn",
  "stop_sequence": null,
  "usage": {
    "input_tokens": 15,
    "output_tokens": 12
  }
}
```

--------------------------------

### Enable Docker BuildKit

Source: https://github.com/runpod-workers/worker-vllm/blob/main/README.md

Enable Docker BuildKit, which is required for using Docker secrets.

```bash
export DOCKER_BUILDKIT=1
```

--------------------------------

### Missing Baked-in Model Log Output

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/errors.md

This log message indicates that a model name was specified during image build (MODEL_NAME) but the corresponding model details are missing from the /local_model_args.json file. This prevents the worker from starting correctly.

```log
Model name not found in /local_model_args.json
```

--------------------------------

### OpenAIvLLMEngine - OpenAI Chat Completion

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/README.md

The `OpenAIvLLMEngine` wraps the `vLLMEngine` to provide an OpenAI-compatible API. The `generate` method handles requests for chat completions and other OpenAI-like endpoints.

```APIDOC
## OpenAIvLLMEngine.generate (OpenAI Chat Completions)

### Description
Handles OpenAI-compatible API requests, specifically for chat completions.

### Method
`generate(openai_request: JobInput) -> AsyncGenerator[dict|str, None]`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None (Input is provided via the `openai_request` parameter, which is a `JobInput` object)

### Request Example
```python
from engine import vLLMEngine, OpenAIvLLMEngine
from utils import JobInput

request_data = {
    "openai_route": "/v1/chat/completions",
    "openai_input": {
        "model": "meta-llama/Llama-3.1-8B-Instruct",
        "messages": [{"role": "user", "content": "Hello!"}],
        "temperature": 0.7,
        "max_tokens": 100
    }
}
vllm_engine = vLLMEngine()
openai_engine = OpenAIvLLMEngine(vllm_engine)
job_input = JobInput(request_data)

async for response in openai_engine.generate(job_input):
    print(response)
```

### Response
#### Success Response
An asynchronous generator yielding dictionaries or strings representing the chat completion response.

#### Response Example
```json
{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "created": 1700000000,
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello there! How can I help you today?"
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 20,
    "total_tokens": 30
  }
}
```
```

--------------------------------

### Configure Multimodal Support Limits

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/architecture.md

Limit multimodal content per prompt by setting the LIMIT_MM_PER_PROMPT environment variable. The format specifies the maximum number of images and videos allowed. This is parsed in engine_args.convert_limit_mm_per_prompt().

```bash
LIMIT_MM_PER_PROMPT="image=5,video=2"
```

--------------------------------

### Troubleshooting: Out of Memory - Enable Quantization

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/README.md

Enabling quantization, such as 'awq', can significantly reduce the memory footprint of the model, helping to resolve OOM errors.

```bash
# 4. SET QUANTIZATION=awq
```

--------------------------------

### Access OpenAIvLLMEngine Properties

Source: https://github.com/runpod-workers/worker-vllm/blob/main/_autodocs/README.md

Examine properties of the OpenAIvLLMEngine wrapper, including the served model name and whether raw OpenAI output is enabled.

```python
engine = OpenAIvLLMEngine(vllm_engine)
engine.served_model_name  # str
engine.raw_openai_output  # bool
```