### Start LLM Server for Benchmarking

Source: https://context7.com/alibabapai/llumnix/llms.txt

Starts the LLM API server with specified configurations for benchmarking. Ensure Ray cluster is launched and instance information is logged.

```bash
export HEAD_NODE_IP='127.0.0.1'
HEAD_NODE=1 python -m llumnix.entrypoints.vllm.api_server \
    --config-file configs/vllm.yml \
    --host 127.0.0.1 --port 8000 \
    --initial-instances 4 \
    --launch-ray-cluster \
    --model /path/to/Qwen2.5-7B \
    --worker-use-ray \
    --migration-backend rayrpc \
    --log-instance-info
```

--------------------------------

### Install Llumnix from PyPI or Source

Source: https://context7.com/alibabapai/llumnix/llms.txt

Install Llumnix using pip or build from source. Optional packages for migration backends like NCCL and Gloo can also be installed.

```bash
pip install llumnix
```

```bash
git clone https://github.com/AlibabaPAI/llumnix.git
cd llumnix
make vllm_install
```

```bash
# Optional: install NCCL migration backend support
make cupy-cuda
```

```bash
# Optional: install Gloo migration backend support (also requires cupy-cuda)
# Install Bazel >= 5.1.0 first, then:
make pygloo
```

--------------------------------

### Install Llumnix from PyPI

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Quickstart.md

Install Llumnix using pip. This is the recommended method for most users.

```bash
pip install llumnix
```

--------------------------------

### Centralized Launch Entrypoint for vLLM Serve

Source: https://context7.com/alibabapai/llumnix/llms.txt

Deploys all servers and instances using a pre-existing Ray cluster. Useful for job submission APIs and automated setups.

```bash
# Assumes a Ray cluster is already running.
# Deploy 4 total instances (2 prefill + 2 decode) with PD disaggregation.
python -m llumnix.entrypoints.vllm.serve \
    --host $(hostname -i) \
    --port 8000 \
    --model /path/to/Qwen2-7B \
    --worker-use-ray \
    --trust-remote-code \
    --max-model-len 4096 \
    --config-file configs/vllm.yml \
    --enable-pd-disagg \
    --pd-ratio 1:1 \
    --max-units 4
```

```bash
# Standard serve without PD disaggregation, auto-scaling enabled
python -m llumnix.entrypoints.vllm.serve \
    --config-file configs/vllm.yml \
    --model /path/to/Llama3-8B \
    --worker-use-ray \
    --max-units 8 \
    --enable-scaling \
    --min-units 2 \
    --scale-up-threshold 10 \
    --scale-down-threshold 60
```

--------------------------------

### Environment Setup for Multi-Node Deployment

Source: https://context7.com/alibabapai/llumnix/llms.txt

Configure environment variables on all nodes to enable Ray cluster formation. Set HEAD_NODE_IP on all nodes and HEAD_NODE=1 only on the head node.

```bash
# Set on ALL nodes — IP address of the head node
export HEAD_NODE_IP=192.168.1.10

# Set ONLY on the head node
export HEAD_NODE=1
```

--------------------------------

### Centralized Launch Entrypoint

Source: https://context7.com/alibabapai/llumnix/llms.txt

The `llumnix.entrypoints.vllm.serve` module provides a centralized entrypoint for deploying all servers and instances at once, assuming a pre-existing Ray cluster. This is particularly useful for Ray job submission APIs and automated cluster setups.

```APIDOC
## `python -m llumnix.entrypoints.vllm.serve`

### Description
Deploy all servers and instances at once using a pre-existing Ray cluster. Useful for Ray job submission APIs and automated cluster setups.

### Usage Examples

**Deploy 4 total instances (2 prefill + 2 decode) with PD disaggregation:**
```bash
# Assumes a Ray cluster is already running.
python -m llumnix.entrypoints.vllm.serve \
    --host $(hostname -i) \
    --port 8000 \
    --model /path/to/Qwen2-7B \
    --worker-use-ray \
    --trust-remote-code \
    --max-model-len 4096 \
    --config-file configs/vllm.yml \
    --enable-pd-disagg \
    --pd-ratio 1:1 \
    --max-units 4
```

**Standard serve without PD disaggregation, auto-scaling enabled:**
```bash
python -m llumnix.entrypoints.vllm.serve \
    --config-file configs/vllm.yml \
    --model /path/to/Llama3-8B \
    --worker-use-ray \
    --max-units 8 \
    --enable-scaling \
    --min-units 2 \
    --scale-up-threshold 10 \
    --scale-down-threshold 60
```
```

--------------------------------

### Install PyGloo for Migration

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Quickstart.md

Install PyGloo for Gloo migration backend. This requires installing CuPy-CUDA first and ensuring Bazel is installed.

```bash
make pygloo
```

--------------------------------

### GET /is_ready

Source: https://context7.com/alibabapai/llumnix/llms.txt

Checks if the Llumnix cluster and backend engine instances are fully initialized.

```APIDOC
## GET /is_ready

### Description
Returns `true` once the Llumnix cluster and backend engine instances are fully initialized.

### Method
GET

### Endpoint
http://localhost:8000/is_ready

### Response
#### Success Response (200)
- **true** (boolean) - Indicates the instance is ready.
```

--------------------------------

### Launch Llumnix API Server

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Quickstart.md

Replace the vLLM server entrypoint with Llumnix's to deploy multiple instances. This command starts the Llumnix API server, which will manage multiple vLLM instances.

```python
python -m llumnix.entrypoints.vllm.api_server \
    --config-file $CONFIG_PATH \
    # vLLM arguments ...
    # Llumnix arguments ...
    ...
```

--------------------------------

### GET /is_ready - Instance Readiness Check

Source: https://context7.com/alibabapai/llumnix/llms.txt

This endpoint returns 'true' when the Llumnix cluster and backend engine instances are fully initialized. It can be used in deployment scripts to wait for readiness.

```bash
curl http://localhost:8000/is_ready
```

```bash
until curl -sf http://localhost:8000/is_ready; do
  echo "Waiting for Llumnix to be ready..."
  sleep 5
done
echo "Llumnix is ready!"
```

--------------------------------

### vLLM API Server Entrypoint

Source: https://context7.com/alibabapai/llumnix/llms.txt

The `llumnix.entrypoints.vllm.api_server` module serves as a drop-in replacement for the vLLM API server. It automatically bootstraps the Ray cluster, starts Llumnix Manager and Llumlet actors, and launches vLLM engine instances. It accepts all standard vLLM engine arguments along with Llumnix-specific configurations.

```APIDOC
## `python -m llumnix.entrypoints.vllm.api_server`

### Description
Drop-in replacement for the vLLM API server. Automatically bootstraps the Ray cluster, starts Llumnix Manager/Llumlet actors, and launches vLLM engine instances. Accepts all vLLM engine arguments plus Llumnix-specific arguments.

### Usage Examples

**Single-node deployment with 2 instances, using Gloo migration backend:**
```bash
export HEAD_NODE_IP='127.0.0.1'
HEAD_NODE=1 python -m llumnix.entrypoints.vllm.api_server \
    --config-file configs/vllm.yml \
    --host 0.0.0.0 \
    --port 8000 \
    --model /path/to/Qwen2-7B \
    --worker-use-ray \
    --trust-remote-code \
    --max-model-len 4096 \
    --initial-instances 2 \
    --launch-ray-cluster \
    --migration-backend gloo \
    --dispatch-policy load \
    --enable-routine-migration
```

**Multi-node: head node (starts Ray head + 1 instance):**
```bash
HEAD_NODE=1 python -m llumnix.entrypoints.vllm.api_server \
    --config-file configs/vllm.yml \
    --host 0.0.0.0 --port 8000 \
    --model /path/to/Llama2-13B \
    --worker-use-ray --initial-instances 1 \
    --launch-ray-cluster
```

**Multi-node: worker node (joins Ray cluster + 1 instance):**
```bash
python -m llumnix.entrypoints.vllm.api_server \
    --config-file configs/vllm.yml \
    --host 0.0.0.0 --port 8001 \
    --model /path/to/Llama2-13B \
    --worker-use-ray --initial-instances 1
```
```

--------------------------------

### Install CuPy-CUDA for Migration

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Quickstart.md

Install CuPy-CUDA manually if you intend to use NCCL as the migration backend. Ensure your CUDA version is compatible.

```bash
make cupy-cuda
```

--------------------------------

### launch_ray_cluster(port)

Source: https://context7.com/alibabapai/llumnix/llms.txt

Programmatically starts or joins a Ray cluster.

```APIDOC
## launch_ray_cluster(port)

### Description
Starts or joins a Ray cluster programmatically. Stops any existing Ray process on the node, then starts a head or worker node based on environment variables.

### Parameters
- **port** (integer) - The port to use for the Ray cluster.

### Usage
```python
import os
from llumnix import launch_ray_cluster, connect_to_ray_cluster

# Configure environment for head node
os.environ['HEAD_NODE'] = '1'
os.environ['HEAD_NODE_IP'] = '127.0.0.1'

# Start the Ray head node on port 6379
launch_ray_cluster(port=6379)

# Connect the current Python process to the cluster
connect_to_ray_cluster(port=6379)
# Or connect with explicit head node IP:
connect_to_ray_cluster(head_node_ip='127.0.0.1', port=6379, namespace='llumnix')
```
```

--------------------------------

### Launch vLLM API Server with Llumnix

Source: https://context7.com/alibabapai/llumnix/llms.txt

Configure instance restart behavior and polling interval when launching the vLLM API server with Llumnix. Ensure Ray is installed for fault-tolerance mechanisms.

```bash
python -m llumnix.entrypoints.vllm.api_server \
    --config-file configs/vllm.yml \
    --model /path/to/model \
    --worker-use-ray \
    --initial-instances 4 \
    --launch-ray-cluster \
    --polling-interval 0.05   # check instance health every 50ms (default)
```

--------------------------------

### Launch Llumnix Server with Ray Cluster

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Quickstart.md

Use this command to start the Llumnix server and automatically build a Ray cluster. Ensure HEAD_NODE_IP is set to your server's IP address. This command will overwrite any existing Ray cluster.

```bash
export HEAD_NODE_IP='127.0.0.1'

HEAD_NODE=1 python -m llumnix.entrypoints.vllm.api_server \
                --config-file $CONFIG_PATH \
                --host $HOST \
                --port $PORT \
                --initial-instances $INITIAL_INSTANCES \
                --launch-ray-cluster \
                --model $MODEL_PATH \
                --worker-use-ray \
                --migration-backend rayrpc \

```

--------------------------------

### vLLM Backend Configuration (YAML)

Source: https://context7.com/alibabapai/llumnix/llms.txt

Example YAML configuration for the vLLM backend. This file defines server, manager, and instance settings, including queue types, dispatch policies, and migration options. CLI arguments override these settings.

```yaml
# configs/vllm.yml — example for vLLM backend

SERVER:
  HOST: '0.0.0.0'
  PORT: 8000
  REQUEST_OUTPUT_QUEUE_TYPE: "zmq"  # or "rayqueue"
  RAY_CLUSTER_PORT: 6379
  LAUNCH_RAY_CLUSTER: False         # set True on head node, or use --launch-ray-cluster

MANAGER:
  INITIAL_INSTANCES: 2
  DISPATCH_POLICY: 'load'           # 'load' | 'balanced' | 'queue' | 'rr'
  ENABLE_ROUTINE_MIGRATION: True
  ENABLE_SCALING: False
  DISPATCH_LOAD_METRIC: 'remaining_steps'        # 'remaining_steps' | 'kv_blocks_ratio'
  DISPATCH_PREFILL_LOAD_METRIC: 'kv_blocks_ratio'
  DISPATCH_DECODE_LOAD_METRIC: 'remaining_steps'

INSTANCE:
  MIGRATION_LOAD_METRIC: 'remaining_steps'
  ENABLE_DEFRAG: True
  MIGRATION_BACKEND: 'gloo'         # 'gloo' | 'nccl' | 'rayrpc' | 'grpc' | 'kvtransfer'
  MIGRATION_BUFFER_BLOCKS: 512
  REQUEST_MIGRATION_POLICY: 'SR'    # 'LCR' | 'SR' | 'LR' | 'FCW' | 'FCWSR'

```

--------------------------------

### BladeLLM Backend Configuration (YAML)

Source: https://context7.com/alibabapai/llumnix/llms.txt

Example YAML configuration for the BladeLLM backend. This file specifies server, manager, and instance parameters, including queue types, migration policies, and backend-specific settings for KV transfer.

```yaml
# configs/bladellm.yml — example for BladeLLM backend

SERVER:
  HOST: '127.0.0.1'
  PORT: 1234
  REQUEST_OUTPUT_QUEUE_TYPE: "zmq"
  RAY_CLUSTER_PORT: 6379
  LAUNCH_RAY_CLUSTER: False

MANAGER:
  ENABLE_ROUTINE_MIGRATION: False
  MIGRATE_OUT_THRESHOLD: -3.0
  DISPATCH_POLICY: 'load'
  PAIR_MIGRATION_POLICY: 'defrag'

INSTANCE:
  DISPATCH_LOAD_METRIC: 'remaining_steps'
  MIGRATION_LOAD_METRIC: 'remaining_steps'
  ENABLE_DEFRAG: True
  REQUEST_MIGRATION_POLICY: 'SR'
  MIGRATION_BACKEND: 'grpc'         # BladeLLM supports 'grpc' and 'kvtransfer'
  MIGRATION_BUFFER_BLOCKS: 512
  KVTRANSFER_MIGRATION_BACKEND_TRANSFER_TYPE: 'rdma'
  KVTRANSFER_MIGRATION_BACKEND_NAMING_URL: "file:/tmp/llumnix/naming/"

```

--------------------------------

### GET /health

Source: https://context7.com/alibabapai/llumnix/llms.txt

Checks the health status of the Llumnix server.

```APIDOC
## GET /health

### Description
Returns HTTP 200 when the server is running.

### Method
GET

### Endpoint
http://localhost:8000/health

### Response
#### Success Response (200)
Indicates the server is running.
```

--------------------------------

### Offline Batch Inference with Python API

Source: https://context7.com/alibabapai/llumnix/llms.txt

Use Llumnix programmatically for offline batch inference. This example demonstrates initializing the engine, setting up prompts and sampling parameters, and collecting outputs asynchronously.

```python
import asyncio, uuid, os, time
import ray
from vllm.engine.arg_utils import EngineArgs
from vllm.sampling_params import SamplingParams
from llumnix import (
    init_scaler, get_llumnix_actor_handle, LlumnixActor,
    launch_ray_cluster, connect_to_ray_cluster,
    ManagerArgs, InstanceArgs, EntrypointsArgs, LaunchArgs,
    RayQueueServer, QueueType, BackendType, LaunchMode,
    LlumnixRequestOuput, RequestProcessingContext,
)
from llumnix.entrypoints.vllm.arg_utils import VLLMEngineArgs

os.environ['HEAD_NODE'] = '1'
os.environ['HEAD_NODE_IP'] = '127.0.0.1'
launch_ray_cluster(port=6379)
connect_to_ray_cluster(port=6379)

prompts = ["Hello, my name is", "The capital of France is", "The future of AI is"]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

engine_args = EngineArgs(model="facebook/opt-125m", worker_use_ray=True,
                         max_model_len=370, enforce_eager=True)
launch_args = LaunchArgs(launch_mode=LaunchMode.LOCAL, backend_type=BackendType.VLLM)
vllm_engine_args = VLLMEngineArgs(engine_args=engine_args)

scaler = init_scaler(ManagerArgs(), InstanceArgs(), EntrypointsArgs(), engine_args, launch_args)
ray.get(scaler.is_ready.remote())
manager = get_llumnix_actor_handle(LlumnixActor.MANAGER)

# Initialize instances
node_id = ray.get_runtime_context().get_node_id()
instance_ids, instances = ray.get(
    scaler.init_instances.remote(QueueType("rayqueue"), InstanceArgs(), vllm_engine_args, node_id)
)
while ray.get(manager.scale_up.remote([], [], [])) == 0:
    time.sleep(1.0)

# Set up output queue
server_id = str(uuid.uuid4().hex)
request_output_queue = RayQueueServer()
request_processing_context = RequestProcessingContext(
    server_id, QueueType("rayqueue"), request_output_queue.queue, None, None
)

async def collect_outputs(num_tasks):
    finished = 0
    while finished < num_tasks:
        outputs = await request_output_queue.get()
        for out in outputs:
            req_out = out.get_engine_output()
            if req_out.finished:
                finished += 1
                print(f"Prompt: {req_out.prompt!r}, Output: {req_out.outputs[0].text!r}")
    request_output_queue.cleanup()

async def main():
    asyncio.create_task(request_output_queue.run_server_loop())
    output_task = asyncio.create_task(collect_outputs(len(prompts)))
    for prompt in prompts:
        await manager.generate.remote(
            request_id=str(uuid.uuid4().hex),
            request_processing_context=request_processing_context,
            prompt=prompt,
            params=sampling_params,
        )
    await output_task

asyncio.run(main())
ray.shutdown()

```

--------------------------------

### Text Generation Endpoint (Non-Streaming)

Source: https://context7.com/alibabapai/llumnix/llms.txt

Generates text completions using a POST request to the /generate endpoint. Accepts prompt and SamplingParams. This example shows a non-streaming response.

```bash
# Non-streaming generation
curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "The capital of France is",
    "n": 1,
    "temperature": 0.8,
    "top_p": 0.95,
    "max_tokens": 100,
    "stream": false
  }'
# Response: {"text": ["The capital of France is Paris, a city known for..."]}
```

--------------------------------

### GET /health - Health Check

Source: https://context7.com/alibabapai/llumnix/llms.txt

Use this endpoint to check if the server is running. It returns an HTTP 200 OK status. This is commonly used in Kubernetes readiness probes.

```bash
curl http://localhost:8000/health
```

```yaml
livenessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 30
  periodSeconds: 10
```

--------------------------------

### Launch Prefill Instance for P-D Disaggregation

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Prefill-decode_Disaggregation.md

Launch a prefill instance for Prefill-Decode disaggregation. Ensure Ray cluster is launched and specify '--instance-type prefill'.

```bash
python -m llumnix.entrypoints.vllm.api_server --host `hostname -i` --port $PORT --model $MODEL_PATH --worker-use-ray --trust-remote-code --max-model-len 4096 --config-file configs/vllm.yml --enable-pd-disagg --instance-type prefill --initial-instances 1
```

--------------------------------

### Launch Prefill Instance with Ray

Source: https://context7.com/alibabapai/llumnix/llms.txt

Launches a prefill instance of the LLM API server using Ray. Ensure the model path and configuration file are correctly specified.

```bash
python -m llumnix.entrypoints.vllm.api_server \
    --host $(hostname -i) --port 8000 \
    --model /path/to/model \
    --worker-use-ray --trust-remote-code --max-model-len 4096 \
    --config-file configs/vllm.yml \
    --enable-pd-disagg \
    --instance-type prefill \
    --initial-instances 1 \
    --launch-ray-cluster
```

--------------------------------

### Launch vLLM API Server (Single-Node)

Source: https://context7.com/alibabapai/llumnix/llms.txt

Launches a single-node vLLM API server with multiple instances and Gloo migration backend. Requires setting HEAD_NODE_IP and HEAD_NODE.

```bash
# Single-node deployment with 2 instances, using Gloo migration backend
export HEAD_NODE_IP='127.0.0.1'
HEAD_NODE=1 python -m llumnix.entrypoints.vllm.api_server \
    --config-file configs/vllm.yml \
    --host 0.0.0.0 \
    --port 8000 \
    --model /path/to/Qwen2-7B \
    --worker-use-ray \
    --trust-remote-code \
    --max-model-len 4096 \
    --initial-instances 2 \
    --launch-ray-cluster \
    --migration-backend gloo \
    --dispatch-policy load \
    --enable-routine-migration
```

--------------------------------

### Build Llumnix from Source

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Quickstart.md

Clone the repository and build Llumnix from source. This method is useful for development or if you need specific build configurations.

```bash
git clone https://github.com/AlibabaPAI/llumnix.git
cd llumnix
make vllm_install
```

--------------------------------

### Launch Decode Instance

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Prefill-decode_Disaggregation.md

Use this command to launch a decode instance. Ensure `--instance-type` is specified and `--initial-instances` controls the number of instances launched. Requests may hang if no decode instances are available.

```bash
python -m llumnix.entrypoints.vllm.api_server --host `hostname -i` --port $PORT --model $MODEL_PATH --worker-use-ray --trust-remote-code --max-model-len 4096 --config-file configs/vllm.yml --enable-pd-disagg --instance-type decode --initial-instances 1
```

--------------------------------

### Run Serving Performance Benchmark

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Quickstart.md

Execute this script after launching the Llumnix server to evaluate serving performance. Configure IP_PORTS with the server's address and port, and specify model, dataset, and performance parameters.

```bash
cd benchmark

python benchmark_serving.py \
    --ip_ports ${IP_PORTS[@]} \
    --tokenizer $MODEL_PATH \
    --random_prompt_count $NUM_PROMPTS \
    --dataset_type "sharegpt" \
    --dataset_path $DATASET_PATH \
    --qps $QPS \
    --distribution "poisson" \
    --log_latencies \
    --fail_on_response_failure \

```

--------------------------------

### Launch vLLM API Server (Multi-Node)

Source: https://context7.com/alibabapai/llumnix/llms.txt

Launches vLLM API servers for multi-node deployments. The first command is for the head node, and the second is for worker nodes.

```bash
# Multi-node: head node (starts Ray head + 1 instance)
HEAD_NODE=1 python -m llumnix.entrypoints.vllm.api_server \
    --config-file configs/vllm.yml \
    --host 0.0.0.0 --port 8000 \
    --model /path/to/Llama2-13B \
    --worker-use-ray --initial-instances 1 \
    --launch-ray-cluster
```

```bash
# Multi-node: worker node (joins Ray cluster + 1 instance)
python -m llumnix.entrypoints.vllm.api_server \
    --config-file configs/vllm.yml \
    --host 0.0.0.0 --port 8001 \
    --model /path/to/Llama2-13B \
    --worker-use-ray --initial-instances 1
```

--------------------------------

### Run Serving Benchmark with Poisson Distribution

Source: https://context7.com/alibabapai/llumnix/llms.txt

Executes a serving benchmark using a Poisson distribution for traffic. Specify IP:port, tokenizer path, dataset details, and desired QPS.

```bash
cd benchmark
python benchmark_serving.py \
    --ip_ports 127.0.0.1:8000 \
    --tokenizer /path/to/Qwen2.5-7B \
    --random_prompt_count 500 \
    --dataset_type sharegpt \
    --dataset_path /path/to/sharegpt.jsonl \
    --qps 4.0 \
    --distribution poisson \
    --log_latencies \
    --fail_on_response_failure \
    --log_filename results/bench.log
```

--------------------------------

### Run Serving in Simulator Mode

Source: https://context7.com/alibabapai/llumnix/llms.txt

Launches the LLM API server in simulator mode using profiled latency data. This mode is useful for offline scheduling research and policy evaluation without requiring GPUs.

```bash
python -m llumnix.entrypoints.vllm.api_server \
    --config-file configs/vllm.yml \
    --host 127.0.0.1 --port 8000 \
    --model /path/to/model \
    --initial-instances 4 \
    --launch-ray-cluster \
    --simulator-mode \
    --profiling-result-file-path profiling_result.pkl \
    --gpu-type A100
```

--------------------------------

### Llumnix Initial Instances Argument

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Arguments.md

Sets the number of instances to create at initialization. Defaults to 1.

```bash
--initial-instances INITIAL_INSTANCES
```

--------------------------------

### Centralized Launch of Llumnix Servers

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Quickstart.md

Use the centralized launch script to deploy multiple servers and instances at once. This assumes a Ray cluster is already running.

```python
python -m llumnix.entrypoints.vllm.serve \
    --config-file $CONFIG_PATH \
    # vLLM arguments ...
    # Llumnix arguments ...
    ...
```

--------------------------------

### init_scaler()

Source: https://context7.com/alibabapai/llumnix/llms.txt

Initializes the Llumnix Scaler Ray actor.

```APIDOC
## init_scaler()

### Description
Creates or retrieves the Llumnix `Scaler` Ray actor, which manages the `Manager`, instance lifecycle, and auto-scaling. Returns a `Scaler` handle.

### Parameters
- **manager_args** (ManagerArgs) - Arguments for the Manager.
- **instance_args** (InstanceArgs) - Arguments for the instances.
- **entrypoints_args** (EntrypointsArgs) - Arguments for the entrypoints.
- **engine_args** (EngineArgs) - Arguments for the backend engine (e.g., vLLM).
- **launch_args** (LaunchArgs) - Arguments for launching the backend.

### Usage
```python
import ray
from llumnix import (
    init_scaler, ManagerArgs, InstanceArgs, EntrypointsArgs,
    LaunchArgs, BackendType, LaunchMode,
)
from llumnix.entrypoints.vllm.arg_utils import VLLMEngineArgs
from vllm.engine.arg_utils import EngineArgs

# Configure arguments
manager_args = ManagerArgs()           # uses defaults from config/default.py
instance_args = InstanceArgs()
entrypoints_args = EntrypointsArgs()
engine_args = EngineArgs(
    model="facebook/opt-125m",
    worker_use_ray=True,
    max_model_len=512,
    enforce_eager=True,
)
launch_args = LaunchArgs(launch_mode=LaunchMode.LOCAL, backend_type=BackendType.VLLM)
vllm_engine_args = VLLMEngineArgs(engine_args=engine_args)

# Create Scaler (or get existing one if already created by another server)
scaler = init_scaler(manager_args, instance_args, entrypoints_args, engine_args, launch_args)
ray.get(scaler.is_ready.remote())  # wait until Scaler and Manager are ready

print("Scaler initialized:", scaler)
```
```

--------------------------------

### Centralized Launch with Prefill/Decode Ratio

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Prefill-decode_Disaggregation.md

This command is used for centralized launch via Ray job submission. The `--pd-ratio` argument sets the prefill to decode unit ratio, and `--max-units` defines the total number of units. Llumnix ensures at least one prefill and one decode unit are launched.

```bash
python -m llumnix.entrypoints.vllm.serve --host `hostname -i` --port $PORT --model $MODEL_PATH --trust-remote-code --worker-use-ray --max-model-len 4096 --config-file configs/vllm.yml --enable-pd-disagg --pd-ratio 1:1 --max-units 2
```

--------------------------------

### Configure Environment Variables for Multi-Instance Deployment

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Quickstart.md

Set environment variables on all nodes to configure Llumnix for multi-instance deployment using Ray. HEAD_NODE_IP should be set to the IP address of the head node.

```bash
# Configure on all nodes.
export HEAD_NODE_IP=$HEAD_NODE_IP_ADDRESS

# Configure on head node.
export HEAD_NODE=1
```

--------------------------------

### Centralized Launch with Automatic Ratio Assignment

Source: https://context7.com/alibabapai/llumnix/llms.txt

Launches LLM serving instances centrally, automatically assigning ratios for prefill and decode. Use `--pd-ratio` to control the prefill:decode ratio and `--max-units` for the total number of instances.

```bash
python -m llumnix.entrypoints.vllm.serve \
    --host $(hostname -i) --port 8000 \
    --model /path/to/model \
    --worker-use-ray --trust-remote-code --max-model-len 4096 \
    --config-file configs/vllm.yml \
    --enable-pd-disagg \
    --pd-ratio 1:1 \
    --max-units 4
```

--------------------------------

### Multi-Server Benchmark with Gamma Distribution

Source: https://context7.com/alibabapai/llumnix/llms.txt

Runs a benchmark across multiple servers using a gamma distribution for traffic. Configure IP:port for each server, tokenizer, dataset, QPS, and coefficient of variation.

```bash
python benchmark_serving.py \
    --ip_ports 192.168.1.10:8000 192.168.1.11:8001 \
    --tokenizer /path/to/model \
    --random_prompt_count 1000 \
    --dataset_type sharegpt \
    --dataset_path /path/to/sharegpt.jsonl \
    --qps 8.0 \
    --distribution gamma \
    --coefficient_variation 0.5 \
    --log_latencies \
    --backend vLLM
```

--------------------------------

### Llumnix Config File Argument

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Arguments.md

Specifies the path to a configuration file for arguments. Defaults to None.

```bash
--config-file CONFIG_FILE
```

--------------------------------

### Llumnix SSL Keyfile Argument

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Arguments.md

Specifies the path to the SSL key file for secure connections. Defaults to None.

```bash
--ssl-keyfile SSL_KEYFILE
```

--------------------------------

### POST /generate_benchmark - Benchmark Generation

Source: https://context7.com/alibabapai/llumnix/llms.txt

This endpoint is for benchmarking generation. It returns per-token latency data for performance analysis. Ensure 'stream' is set to false for benchmarking.

```bash
curl -X POST http://localhost:8000/generate_benchmark \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "The future of artificial intelligence is",
    "n": 1,
    "best_of": 1,
    "temperature": 0.0,
    "top_k": 1,
    "max_tokens": 128,
    "ignore_eos": true,
    "stream": false
  }'
```

--------------------------------

### Launch Decode Instance with Ray

Source: https://context7.com/alibabapai/llumnix/llms.txt

Launches a decode instance of the LLM API server using Ray. This is typically run on a different node or the same node as the prefill instance.

```bash
python -m llumnix.entrypoints.vllm.api_server \
    --host $(hostname -i) --port 8001 \
    --model /path/to/model \
    --worker-use-ray --trust-remote-code --max-model-len 4096 \
    --config-file configs/vllm.yml \
    --enable-pd-disagg \
    --instance-type decode \
    --initial-instances 1
```

--------------------------------

### Initialize Llumnix Scaler Actor

Source: https://context7.com/alibabapai/llumnix/llms.txt

Creates or retrieves the Llumnix Scaler Ray actor, which manages instance lifecycle and auto-scaling. Requires configuration arguments for Manager, Instance, Entrypoints, Engine, and Launch. Waits until the Scaler and Manager are ready.

```python
import ray
from llumnix import (
    init_scaler, ManagerArgs, InstanceArgs, EntrypointsArgs,
    LaunchArgs, BackendType, LaunchMode,
)
from llumnix.entrypoints.vllm.arg_utils import VLLMEngineArgs
from vllm.engine.arg_utils import EngineArgs

# Configure arguments
manager_args = ManagerArgs()           # uses defaults from config/default.py
instance_args = InstanceArgs()
entrypoints_args = EntrypointsArgs()
engine_args = EngineArgs(
    model="facebook/opt-125m",
    worker_use_ray=True,
    max_model_len=512,
    enforce_eager=True,
)
launch_args = LaunchArgs(launch_mode=LaunchMode.LOCAL, backend_type=BackendType.VLLM)
vllm_engine_args = VLLMEngineArgs(engine_args=engine_args)

# Create Scaler (or get existing one if already created by another server)
scaler = init_scaler(manager_args, instance_args, entrypoints_args, engine_args, launch_args)
ray.get(scaler.is_ready.remote())  # wait until Scaler and Manager are ready

print("Scaler initialized:", scaler)
```

--------------------------------

### Configure Auto-Scaling for LLM Instances

Source: https://context7.com/alibabapai/llumnix/llms.txt

Enables automatic scaling for LLM engine instances based on load metrics. Configure minimum and maximum units, scaling interval, and thresholds for scaling up and down.

```bash
python -m llumnix.entrypoints.vllm.api_server \
    --config-file configs/vllm.yml \
    --model /path/to/model \
    --worker-use-ray \
    --initial-instances 2 \
    --launch-ray-cluster \
    --enable-scaling \
    --min-units 1 \
    --max-units 8 \
    --scaling-interval 10 \
    --scaling-policy avg_load \
    --scale-up-threshold 10 \
    --scale-down-threshold 60
```

--------------------------------

### Llumnix Dispatch Prefill as Decode Load Metric Argument

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Arguments.md

An experimental argument for the load metric of prefill instances during adaptive prefill-decode disaggregation. Possible choices include 'remaining_steps', 'kv_blocks_ratio', and 'adaptive_decode'. Defaults to 'adaptive_decode'.

```bash
--dispatch-prefill-as-decode-load-metric {remaining_steps,kv_blocks_ratio,adaptive_decode}
```

--------------------------------

### Llumnix SSL Certfile Argument

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Arguments.md

Specifies the path to the SSL certificate file for secure connections. Defaults to None.

```bash
--ssl-certfile SSL_CERTFILE
```

--------------------------------

### Llumnix Profiling Backend Usage

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Simulator.md

Command-line arguments for the Llumnix profiling backend. Use this to generate profiling results from logs.

```bash
usage: -m llumnix.backends.profiling [-h]
            [--database PROFILING_RESULT_FILE_PATH]
            [--log-csv-path CSV_FILE_PATH]
            [--model MODEL_NAME]
            [--tp TENSOR_PARALLEL_SIZE]
            [--pp PIPELINE_PARALLEL_SIZE]
            [--gpu-memory-utilization GPU_MEMORY_UTILIZATION]
            [--block-size BLOCK_SIZE]
            [--max-num-batched-tokens MAX_NUM_BATCHED_TOKENS]
            [--num-gpu-blocks NUM_GPU_BLOCKS]
            [--new-data]
```

--------------------------------

### Profile Real Deployment for Latency Data

Source: https://context7.com/alibabapai/llumnix/llms.txt

Profiles a real LLM deployment to gather latency data for simulator mode. Specify database path, log file, model details, and GPU utilization.

```bash
python llumnix/backends/profiling.py \
    --database profiling_result.pkl \
    --log-csv-path real_benchmark.log.csv \
    --model Qwen2.5-7B \
    --tp 1 \
    --gpu-memory-utilization 0.9 \
    --block-size 16 \
    --num-gpu-blocks 1024 \
    --new-data
```

--------------------------------

### Llumnix Host Argument

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Arguments.md

Configures the hostname for the Llumnix server. Defaults to 'localhost' if not specified.

```bash
--host HOST
```

--------------------------------

### POST /generate

Source: https://context7.com/alibabapai/llumnix/llms.txt

Generates text based on a prompt. Supports streaming output and includes optional tracing for latency breakdown.

```APIDOC
## POST /generate

### Description
Generates text based on a prompt. Supports streaming output and includes optional tracing for latency breakdown.

### Method
POST

### Endpoint
http://localhost:8000/generate

### Parameters
#### Request Body
- **prompt** (string) - Required - The input prompt for text generation.
- **temperature** (number) - Optional - Controls randomness. Lower values make output more deterministic.
- **max_tokens** (integer) - Optional - The maximum number of tokens to generate.
- **stream** (boolean) - Optional - If true, returns output in chunks.

### Request Example
```json
{
  "prompt": "Explain quantum computing in simple terms:",
  "temperature": 0.7,
  "max_tokens": 200,
  "stream": true
}
```

### Response
#### Success Response (200)
- **text** (string) - The generated text chunk (when stream is true).
- **llumnix_trace_info** (object) - Contains per-token latency breakdowns (when x-llumnix-trace header is true).
```

--------------------------------

### Llumnix Dispatch Prefill Load Metric Argument

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Arguments.md

Specifies the load metric for prefill instances during prefill-decode disaggregation. Possible choices are 'remaining_steps' and 'kv_blocks_ratio'. Defaults to 'kv_blocks_ratio'.

```bash
--dispatch-prefill-load-metric {remaining_steps,kv_blocks_ratio}
```

--------------------------------

### POST /generate_benchmark

Source: https://context7.com/alibabapai/llumnix/llms.txt

Benchmarks text generation by returning per-token latency data.

```APIDOC
## POST /generate_benchmark

### Description
Benchmarks text generation by returning per-token latency data for performance analysis.

### Method
POST

### Endpoint
http://localhost:8000/generate_benchmark

### Parameters
#### Request Body
- **prompt** (string) - Required - The input prompt for text generation.
- **n** (integer) - Optional - Number of completions to generate.
- **best_of** (integer) - Optional - Number of completions to generate and return.
- **temperature** (number) - Optional - Controls randomness.
- **top_k** (integer) - Optional - Samples from the top K most likely next tokens.
- **max_tokens** (integer) - Optional - Maximum number of tokens to generate.
- **ignore_eos** (boolean) - Optional - If true, generation stops only at max_tokens.
- **stream** (boolean) - Optional - If true, returns output in chunks.

### Request Example
```json
{
  "prompt": "The future of artificial intelligence is",
  "n": 1,
  "best_of": 1,
  "temperature": 0.0,
  "top_k": 1,
  "max_tokens": 128,
  "ignore_eos": true,
  "stream": false
}
```

### Response
#### Success Response (200)
- **request_id** (string) - Unique identifier for the request.
- **generated_text** (string) - The generated text.
- **num_output_tokens_cf** (integer) - Number of output tokens generated.
- **per_token_latency** (array) - Array of [timestamp, latency] pairs for each token.
```

--------------------------------

### Programmatic Ray Cluster Launch

Source: https://context7.com/alibabapai/llumnix/llms.txt

Launches or joins a Ray cluster programmatically. Configure 'HEAD_NODE' and 'HEAD_NODE_IP' environment variables before calling. Connects the current Python process to the cluster.

```python
import os
from llumnix import launch_ray_cluster, connect_to_ray_cluster

# Configure environment
os.environ['HEAD_NODE'] = '1'
os.environ['HEAD_NODE_IP'] = '127.0.0.1'

# Start the Ray head node on port 6379
launch_ray_cluster(port=6379)

# Connect the current Python process to the cluster
connect_to_ray_cluster(port=6379)
# Or connect with explicit head node IP:
connect_to_ray_cluster(head_node_ip='127.0.0.1', port=6379, namespace='llumnix')
```

--------------------------------

### POST /generate - Streaming Generation

Source: https://context7.com/alibabapai/llumnix/llms.txt

Use this endpoint for streaming text generation. The response is newline-delimited JSON chunks. Enable 'x-llumnix-trace' header for per-token latency breakdowns.

```bash
curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Explain quantum computing in simple terms:",
    "temperature": 0.7,
    "max_tokens": 200,
    "stream": true
  }'
```

```bash
curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -H "x-llumnix-trace: true" \
  -d '{"prompt": "Hello world", "max_tokens": 50}'
```

--------------------------------

### Deploy Llumnix API Server

Source: https://github.com/alibabapai/llumnix/blob/main/README.md

Replace your existing vLLM serving command with this to enable Llumnix as the request scheduling layer. This command configures Llumnix to automatically manage multiple vLLM engine instances.

```bash
python -m llumnix.entrypoints.vllm.api_server \
    --host $HOST \
    --port $PORT \
    ...
```

--------------------------------

### Llumnix vLLM API Server Usage

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Arguments.md

This is the main usage string for the Llumnix vLLM API server, detailing all available command-line arguments. Use this to understand the full range of configuration options.

```bash
usage: -m llumnix.entrypoints.vllm.api_server [-h]
            [--host HOST]
            [--port PORT]
            [--ssl-keyfile SSL_KEYFILE]
            [--ssl-certfile SSL_CERTFILE]
            [--log-level {debug,info,warning,error}]
            [--launch-ray-cluster]
            [--ray-cluster-port RAY_CLUSTER_PORT]
            [--disable-log-to-driver]
            [--request-output-queue-type {rayqueue,zmq}]
            [--disable-log-requests-server]
            [--config-file CONFIG_FILE]
            [--initial-instances INITIAL_INSTANCES]
            [--dispatch-load-metric {remaining_steps,kv_blocks_ratio}]
            [--dispatch-prefill-load-metric {remaining_steps,kv_blocks_ratio}]
            [--dispatch-prefill-as-decode-load-metric {remaining_steps,kv_blocks_ratio,adaptive_decode}]
            [--dispatch-decode-load-metric {remaining_steps,kv_blocks_ratio}]
            [--dispatch-decode-as-prefill-load-metric {remaining_steps,kv_blocks_ratio}]
            [--migration-load-metric {remaining_steps,kv_blocks_ratio}]
            [--scaling-load-metric {remaining_steps,kv_blocks_ratio}]
            [--polling-interval POLLING_INTERVAL]
            [--dispatch-policy {balanced,load,queue,rr}]
            [--topk-random-dispatch TOPK_RANDOM_DISPATCH]
            [--enable-routine-migration]
            [--enable-engine-migration-interface]
            [--enable-defrag]
            [--pair-migration-frequency PAIR_MIGRATION_FREQUENCY]
            [--pair-migration-policy {balanced,defrag}]
            [--migrate-out-threshold MIGRATE_OUT_THRESHOLD]
            [--max-migration-concurrency MAX_MIGRATION_CONCURRENCY]
            [--request-migration-policy {LCR,SR,LR,FCW,FCWSR}]
            [--enable-scaling]
            [--min-units MIN_UNITS]
            [--max-units MAX_UNITS]
            [--scaling-interval SCALING_INTERVAL]
            [--scaling-policy {max_load,avg_load}]
            [--scale-up-threshold SCALE_UP_THRESHOLD]
            [--scale-down-threshold SCALE_DOWN_THRESHOLD]
            [--log-instance-info]
            [--log-filename LOG_FILENAME]
            [--simulator-mode]
            [--profiling-result-file-path PROFILING_RESULT_FILE_PATH]
            [--gpu-type GPU_TYPE]
            [--migration-backend {gloo,nccl,rayrpc,grpc,kvtransfer}]
            [--migration-buffer-blocks MIGRATION_BUFFER_BLOCKS]
            [--migration-num-layers MIGRATION_NUM_LAYERS]
            [--migration-backend-init-timeout MIGRATION_BACKEND_INIT_TIMEOUT]
            [--kvtransfer-migration-backend-transfer-type {ipc,rdma}]
            [--kvtransfer-migration-backend-naming-url KVTRANSFER_MIGRATION_BACKEND_NAMING_URL]
            [--migration-max-stages MIGRATION_MAX_STAGES]
            [--migration-last-stage-max-blocks MIGRATION_LAST_STAGE_MAX_BLOCKS]
            [--enable-adaptive-pd]
            [--enable-pd-disagg]
            [--enable-vllm-v1-engine-pd-disagg]
            [--enable-bladellm-engine-pd-disagg]
            [--pd-ratio PD_RATIO]
            [--load-registered-service]
            [--load-registered-service-path]
            [--enable-port-increment]
            [--enable-port-offset-store]
            [--instance-type INSTANCE_TYPE]
            [--engine-disagg-inst-id-env-var]
            [--request-output-token-forwarding-mode]

```

--------------------------------

### Llumnix Dispatch Load Metric Argument

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Arguments.md

Defines the load metric used for instance dispatching. Possible choices are 'remaining_steps' and 'kv_blocks_ratio'. Defaults to 'remaining_steps'.

```bash
--dispatch-load-metric {remaining_steps,kv_blocks_ratio}
```

--------------------------------

### Llumnix Launch Ray Cluster Argument

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Arguments.md

A flag to indicate whether to launch a Ray cluster. No default value is specified, implying it's a boolean flag.

```bash
--launch-ray-cluster
```

--------------------------------

### Llumnix Request Output Queue Type Argument

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Arguments.md

Configures the type of queue used for request output. Possible choices are 'rayqueue' and 'zmq'. Defaults to 'zmq'.

```bash
--request-output-queue-type {rayqueue,zmq}
```

--------------------------------

### Text Generation Endpoint

Source: https://context7.com/alibabapai/llumnix/llms.txt

The `/generate` endpoint allows for text completion generation. It accepts a prompt and standard vLLM `SamplingParams` fields, supporting both streaming and non-streaming responses.

```APIDOC
## `POST /generate`

### Description
Generate text completions. Accepts a prompt and standard vLLM `SamplingParams` fields. Supports both streaming and non-streaming responses.

### Method
POST

### Endpoint
`/generate`

### Request Body
- **prompt** (string) - Required - The input text prompt for generation.
- **n** (integer) - Optional - The number of completions to generate.
- **temperature** (float) - Optional - Controls randomness. Lower values make output more deterministic.
- **top_p** (float) - Optional - Nucleus sampling parameter.
- **max_tokens** (integer) - Optional - The maximum number of tokens to generate.
- **stream** (boolean) - Optional - Whether to stream the response.

### Request Example
```json
{
  "prompt": "The capital of France is",
  "n": 1,
  "temperature": 0.8,
  "top_p": 0.95,
  "max_tokens": 100,
  "stream": false
}
```

### Response
#### Success Response (200)
- **text** (array of strings) - A list of generated text completions.
```

--------------------------------

### Llumnix Dispatch Decode Load Metric Argument

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Arguments.md

Defines the load metric for decode instances. The possible choices are 'remaining_steps' and 'kv_blocks_ratio'.

```bash
--dispatch-decode-load-metric {remaining_steps,kv_blocks_ratio}
```

--------------------------------

### Llumnix Log Level Argument

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Arguments.md

Determines the verbosity of server logs. Possible choices are debug, info, warning, and error. Defaults to 'info'.

```bash
--log-level {debug,info,warning,error}
```

--------------------------------

### Llumnix Ray Cluster Port Argument

Source: https://github.com/alibabapai/llumnix/blob/main/docs/Arguments.md

Specifies the port for the Ray cluster. Defaults to 6379.

```bash
--ray-cluster-port RAY_CLUSTER_PORT
```