### Start vLLM Load Balancer

Source: https://github.com/intel/llm-scaler/blob/main/vllm/README.md

Use this setup as a drop-in alternative to DP. It provides slightly better performance and does not require DP-specific configuration. Navigate to the load balancer directory and start the services.

```bash
cd vllm/docker-compose/load_balancer
docker compose up -d
```

--------------------------------

### Launch vLLM Server with INT4 Quantization

Source: https://github.com/intel/llm-scaler/blob/main/vllm/README.md

Start the vLLM server with symmetric INT4 quantization enabled. This example also configures various serving parameters like model path, dtype, and port.

```bash
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
vllm serve \
    --model /llm/models/DeepSeek-R1-Distill-Qwen-7B \
    --dtype=float16 \
    --enforce-eager \
    --port 8000 \
    --host 0.0.0.0 \
    --trust-remote-code \
    --disable-sliding-window \
    --gpu-memory-util=0.9 \
    --no-enable-prefix-caching \
    --max-num-batched-tokens=8192 \
    --disable-log-requests \
    --max-model-len=8192 \
    --block-size 64 \
    --quantization sym_int4 \
    -tp=1 \
    2>&1 | tee /llm/vllm.log > /proc/1/fd/1 &
```

--------------------------------

### Start LLM Scaler Server

Source: https://github.com/intel/llm-scaler/blob/main/omni/README.md

Instructions to start the LLM scaler server using the sglang command or a provided script. Includes a cURL example for image generation.

```APIDOC
## Start server
sglang serve --model-path /llm/models/Z-Image-Turbo/ \
    --vae-cpu-offload --pin-cpu-memory \
    --num-gpus 1 --port 30010

Or use the provided script:

bash /llm/entrypoints/start_sgl_diffusion.sh

**cURL example:**

```bash
curl http://localhost:30010/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Z-Image-Turbo",
    "prompt": "A beautiful sunset over the ocean",
    "size": "1024x1024"
  }'
```

**Python example (OpenAI SDK):**

```python
from openai import OpenAI
import base64

client = OpenAI(base_url="http://localhost:30010/v1", api_key="EMPTY")

response = client.images.generate(
    model="Z-Image-Turbo",
    prompt="A beautiful sunset over the ocean",
    size="1024x1024",
)

# Save image from base64 response
with open("output.png", "wb") as f:
    f.write(base64.b64decode(response.data[0].b64_json))
```
```

--------------------------------

### Start SGLang Diffusion Server

Source: https://github.com/intel/llm-scaler/blob/main/omni/docs/SGLang_Diffusion_Guide.md

Start an HTTP server with OpenAI-compatible endpoints for image/video generation. Configure proxy settings if necessary. This example starts the server with Z-Image-Turbo model.

```bash
# Configure proxy if needed
export http_proxy=<your_http_proxy>
export https_proxy=<your_https_proxy>
export no_proxy=localhost,127.0.0.1

# Start server with Z-Image-Turbo
sglang serve --model-path /llm/models/Z-Image-Turbo/ \
    --vae-cpu-offload --pin-cpu-memory \
    --num-gpus 1 --port 30010
```

```bash
bash /llm/entrypoints/start_sgl_diffusion.sh
```

--------------------------------

### Install Bare Metal Environment

Source: https://github.com/intel/llm-scaler/blob/main/vllm/README.md

Execute the installation script for the Intel Multi-ARC base platform. Ensure you are in the correct directory and have extracted the installer. A system reboot is required after successful installation.

```bash
sudo su -
cd the_path_of_multi-arc-bmg-offline-installer-x.x.x.x 
./installer.sh
```

--------------------------------

### Setup Gradio Environment and Launch Interface

Source: https://github.com/intel/llm-scaler/blob/main/vllm/webui/multi-modal-gradio/README.md

Create a new Conda environment, install necessary Python packages, and run the Gradio main script. The model URL is optional and should point to the running vLLM service.

```bash
conda create -n qwen_gradio python=3.11
```

```bash
conda activate qwen_gradio
```

```bash
pip install gradio openai opencv-python
```

```bash
python /llm-scaler/vllm/webui/multi-modal-gradio/main.py --model yourmodelname [--model-url 'http://localhost:8000/v1']
```

--------------------------------

### Start ComfyUI Server

Source: https://github.com/intel/llm-scaler/blob/main/omni/docs/SGLang_Diffusion_ComfyUI_Guide.md

Starts the ComfyUI web server. Access it via your browser after starting.

```bash
cd /llm/ComfyUI
python3 main.py --listen 0.0.0.0
```

--------------------------------

### Start vLLM Server for Reranker Task

Source: https://github.com/intel/llm-scaler/blob/main/vllm/README.md

Launch the vLLM server configured for reranking tasks. This example sets the task type to 'score' and specifies a reranker model.

```bash
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
vllm serve \
    --model /llm/models/bge-reranker-base \
    --served-model-name bge-reranker-base \
    --task score \
    --dtype=float16 \
    --enforce-eager \
    --port 8000 \
    --host 0.0.0.0 \
    --trust-remote-code \
    --disable-sliding-window \
    --gpu-memory-util=0.9 \
    --no-enable-prefix-caching \
    --max-num-batched-tokens=2048 \
    --disable-log-requests \
    --max-model-len=2048 \
    --block-size 64 \
    -tp=1 \
    2>&1 | tee /llm/vllm.log > /proc/1/fd/1 &
```

--------------------------------

### Installation with oneAPI Environment

Source: https://github.com/intel/llm-scaler/blob/main/omni/omni_xpu_kernel/README.md

Sets up the oneAPI environment variables before installing the omni_xpu_kernel package. Allows specifying the target GPU device.

```bash
source /opt/intel/oneapi/setvars.sh

# Default: builds for Arc B580 (bmg)
pip install -e . --no-build-isolation

# Specify GPU target:
OMNI_XPU_DEVICE=bmg pip install -e . --no-build-isolation   # Arc B580/B770
OMNI_XPU_DEVICE=pvc pip install -e . --no-build-isolation   # Data Center GPU Max
```

--------------------------------

### Clone Repository and Navigate

Source: https://github.com/intel/llm-scaler/blob/main/omni/comfyui_windows_setup/README.md

Download the LLM Scaler repository and change the directory to the ComfyUI setup script location.

```cmd
git clone https://github.com/intel/llm-scaler.git
cd llm-scaler\omni\comfyui_windows_setup
```

--------------------------------

### Start Xinference Server

Source: https://github.com/intel/llm-scaler/blob/main/omni/README.md

Instructions to start the Xinference server locally. This server supports various models including text-to-speech, speech-to-text, and text-to-image.

```APIDOC
## XInference

```bash
xinference-local --host 0.0.0.0 --port 9997
```

Supported models:
- Stable Diffusion 3.5 Medium
- Kokoro 82M
- whisper large v3

### WebUI Usage

#### 1. Access Xinference Web UI
![xinference_launch](./assets/xinference_launch.png)

#### 2. Select model and configure `model_path`
![xinference_model](./assets/xinference_configure.png)

#### 3. Find running model and launch Gradio UI for this model
![xinference_gradio](./assets/xinference_gradio.png)

#### 4. Generate within Gradio UI
![xinference_example](./assets/xinference_sd.png)

### OpenAI API Usage

> Visit http://127.0.0.1:9997/docs to inspect the API docs.

#### 1. Launch API service
You can select model and launch service via WebUI (refer to [here](#1-access-xinference-web-ui)) or by command:

```bash
xinference-local --host 0.0.0.0 --port 9997

xinference launch --model-name sd3.5-medium --model-type image --model-path /llm/models/stable-diffusion-3.5-medium/ --gpu-idx 0
```

#### 2. Post request in OpenAI API format

For TTS model (`Kokoro 82M` for example):
```bash
curl http://localhost:9997/v1/audio/speech   -H "Content-Type: application/json"   -d '{
    "model": "Kokoro-82M",
    "input": "kokoro, hello, I am kokoro." 
  }'   --output output.wav
```

For STT models (`whisper large v3` for example):
```bash
AUDIO_FILE_PATH=<your_audio_file_path>

curl -X 'POST' \
  "http://localhost:9997/v1/audio/translations" \
  -H 'accept: application/json' \
  -F "model=whisper-large-v3" \
  -F "file=@${AUDIO_FILE_PATH}"

{"text":" Cacaro's hello, I am Cacaro."}
```

For text-to-image models (`Stable Diffusion 3.5 Medium` for example):
```bash
curl http://localhost:9997/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "sd3.5-medium",
    "prompt": "A Shiba Inu chasing butterflies on a sunny grassy field, cartoon style, with vibrant colors.",
    "n": 1,
    "size": "1024x1024",
    "quality": "standard",
    "response_format": "url"
  }'
```
```

--------------------------------

### Launch MinerU Gradio Demo

Source: https://github.com/intel/llm-scaler/blob/main/vllm/OCR/README.md

Start the MinerU Gradio interface for interactive testing.

```bash
mineru-gradio --server-name 0.0.0.0 --server-port 8002
```

--------------------------------

### Start SGLang Diffusion Server

Source: https://github.com/intel/llm-scaler/blob/main/omni/docs/SGLang_Diffusion_ComfyUI_Guide.md

Starts the SGLang Diffusion server. Ensure this is running before connecting ComfyUI.

```bash
bash /llm/entrypoints/start_sgl_diffusion.sh
```

--------------------------------

### Install ComfyUI and Intel-Optimized PyTorch

Source: https://github.com/intel/llm-scaler/blob/main/vllm/webui/multi-modal-comfyui/README.md

Clones the ComfyUI repository, installs Intel-optimized PyTorch for XPU acceleration, and then installs the remaining ComfyUI dependencies. Use the nightly build URL for potential performance improvements.

```bash
git clone https://github.com/comfyanonymous/ComfyUI.git
cd ./ComfyUI

pip install torch torchvision torchaudio \
    --index-url https://download.pytorch.org/whl/xpu

# For nightly builds with potential performance improvements:
# pip install --pre torch torchvision torchaudio \
#     --index-url https://download.pytorch.org/whl/nightly/xpu

pip install -r requirements.txt
```

--------------------------------

### Start SGLang Diffusion OpenAI-compatible Image Generation Server

Source: https://context7.com/intel/llm-scaler/llms.txt

Launches a persistent HTTP server for image generation with OpenAI-compatible endpoints. Supports advanced controls and can be configured for single or multi-GPU setups.

```bash
# Start single-GPU server
sglang serve --model-path /llm/models/Z-Image-Turbo/ \
    --vae-cpu-offload --pin-cpu-memory \
    --num-gpus 1 --port 30010 \
    --attention-backend torch_sdpa

# Or use multi-GPU with sequence parallelism (FLUX.1-dev on 2 GPUs)
sglang serve --model-path /llm/models/FLUX.1-dev/ \
    --vae-cpu-offload --pin-cpu-memory \
    --num-gpus 2 --ulysses-degree 2 \
    --port 30010
```

--------------------------------

### Verify Bare Metal Installation

Source: https://github.com/intel/llm-scaler/blob/main/vllm/README.md

This output indicates a successful installation of the Intel Multi-ARC base platform. Tools like gemm, 1ccl, and xpu-smi are installed in /usr/bin. A reboot is necessary to apply the changes.

```bash
[INFO] Intel Multi-ARC base platform installation complete.
[INFO] Please reboot the system to apply changes.

Tools installed: gemm / 1ccl / xpu-smi in /usr/bin
level-zero-tests: ./tools/level-zero-tests
Support scripts: ./scripts
Installation log: ./install_log_20260129_164959.log
```

--------------------------------

### Start OpenAI API Server

Source: https://github.com/intel/llm-scaler/blob/main/omni/docs/SGLang_Diffusion_Guide.md

Start an HTTP server that exposes OpenAI-compatible endpoints for image/video generation.

```APIDOC
## Starting the Server

Start an HTTP server that exposes OpenAI-compatible endpoints for image/video generation.

```bash
# Configure proxy if needed
export http_proxy=<your_http_proxy>
export https_proxy=<your_https_proxy>
export no_proxy=localhost,127.0.0.1

# Start server with Z-Image-Turbo
sglang serve --model-path /llm/models/Z-Image-Turbo/ \
    --vae-cpu-offload --pin-cpu-memory \
    --num-gpus 1 --port 30010
```

Or use the provided entrypoint script:

```bash
bash /llm/entrypoints/start_sgl_diffusion.sh
```
```

--------------------------------

### Start Paddlex Serving Service

Source: https://github.com/intel/llm-scaler/blob/main/vllm/OCR/README.md

Launch the Paddlex serving service using the configured pipeline.

```bash
paddlex --serve --pipeline ./PaddleOCR-VL.yaml
```

--------------------------------

### Start LLM Scaler Server

Source: https://github.com/intel/llm-scaler/blob/main/omni/README.md

Use this command to start the LLM Scaler server with specified model path and configuration. Ensure the model path is correct and the port is available.

```bash
sglang serve --model-path /llm/models/Z-Image-Turbo/ \
    --vae-cpu-offload --pin-cpu-memory \
    --num-gpus 1 --port 30010
```

--------------------------------

### Install PaddleOCR Dependencies

Source: https://github.com/intel/llm-scaler/blob/main/vllm/OCR/README.md

Install PaddleOCR and its serving components.

```bash
pip install "paddleocr[doc-parser]" paddlepaddle
paddlex --install serving
```

--------------------------------

### Launch dots.ocr Gradio Demo

Source: https://github.com/intel/llm-scaler/blob/main/vllm/OCR/README.md

Start the Gradio interface for testing the dots.ocr model.

```bash
python demo/demo_gradio.py 9000
```

--------------------------------

### Start ComfyUI Server on Windows

Source: https://github.com/intel/llm-scaler/blob/main/omni/README.md

After initializing the environment, navigate to the ComfyUI directory, activate the conda environment, set proxy variables if necessary, and start the ComfyUI server.

```powershell
cd ComfyUI
conda activate omni_env
$env:HTTP_PROXY = <your_proxy>
$env:HTTPS_PROXY = <your_proxy>
python .\main.py --listen 0.0.0.0
```

--------------------------------

### Start MinerU Service with vLLM

Source: https://github.com/intel/llm-scaler/blob/main/vllm/OCR/README.md

Launch the vLLM API server for the MinerU 2.6 model, configuring environment variables and server parameters.

```bash
export MODEL_NAME="/llm/models/MinerU2.5-2509-1.2B/"
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1

python3 -m vllm.entrypoints.openai.api_server \
  --model $MODEL_NAME \
  --dtype float16 \
  --enforce-eager \
  --port 8000 \
  --host 0.0.0.0 \
  --trust-remote-code \
  --gpu-memory-util 0.85 \
  --no-enable-prefix-caching \
  --max-num-batched-tokens=32768 \
  --max-model-len=32768 \
  --block-size 64 \
  --max-num-seqs 256 \
  --served-model-name MinerU \
  --tensor-parallel-size 1 \
  --pipeline-parallel-size 1 \
  --logits-processors mineru_vl_utils:MinerULogitsProcessor
```

--------------------------------

### Start Xinference Local Server

Source: https://context7.com/intel/llm-scaler/llms.txt

Initializes the Xinference server, making its WebUI and OpenAI-compatible API endpoints available for various AI tasks.

```bash
# Start Xinference server
xinference-local --host 0.0.0.0 --port 9997
```

--------------------------------

### Install System Dependencies

Source: https://github.com/intel/llm-scaler/blob/main/vllm/webui/multi-modal-comfyui/README.md

Installs essential system packages including git, python3-pip, python3-venv, and OpenCL development headers. Ensure your system is up-to-date before running.

```bash
sudo apt update && sudo apt install -y \
    git python3-pip python3-venv \
    ocl-icd-opencl-dev
```

--------------------------------

### Start Prometheus and Grafana for vLLM monitoring

Source: https://context7.com/intel/llm-scaler/llms.txt

Starts Prometheus and Grafana using Docker Compose for visualizing vLLM metrics. Access the Grafana UI and import the provided dashboard.

```bash
# Start Prometheus + Grafana via Docker Compose
cd vllm/monitor
docker compose up

# Access metrics endpoint directly from the running vLLM server:
curl http://localhost:8000/metrics

# Grafana UI: http://localhost:3000  (admin / admin)
# Add Prometheus datasource URL: http://prometheus:9090
# Import dashboard: upload vllm/monitor/grafana.json
```

--------------------------------

### Run Docker Container

Source: https://github.com/intel/llm-scaler/blob/main/omni/standalone_examples/Hunyuan3D-2.1/README.md

Starts a Docker container for Hunyuan3D-2.1. This command mounts necessary devices and sets up the environment. Replace placeholders as needed.

```bash
export DOCKER_IMAGE=llm-scaler-omni:latest-hunyuan3d2.1
export CONTAINER_NAME=hunyuan3d-2.1
sudo docker run -itd \
        --privileged \
        --net=host \
        --device=/dev/dri \
        -e no_proxy=localhost,127.0.0.1 \
        --name=$CONTAINER_NAME \
        --shm-size="16g" \
        --entrypoint=/bin/bash \
        $DOCKER_IMAGE 
```

--------------------------------

### Run FunASR Inference

Source: https://github.com/intel/llm-scaler/blob/main/omni/standalone_examples/FunAudioLLM/README.md

Navigate to the FunASR directory within the running container and execute the example script for FunASR inference.

```bash
cd /audio/FunASR
python Fun_ASR_Nano_2512_example.py
```

--------------------------------

### Launch ComfyUI

Source: https://github.com/intel/llm-scaler/blob/main/vllm/webui/multi-modal-comfyui/README.md

Command to start the ComfyUI web server. Access the interface via the provided localhost URL.

```bash
python main.py
```

--------------------------------

### Qwen-Omni Video Input Example

Source: https://github.com/intel/llm-scaler/blob/main/vllm/README.md

Shows how to send video input along with text prompts for analysis using vLLM. This requires a model capable of processing video data.

```bash
curl -sS http://localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "Qwen3-Omni-30B-A3B-Instruct",
    "temperature": 0,
    "max_tokens": 1024,
    "messages": [{"role": "user",
      "content": [
        { "type": "text", "text": "Please describe the video comprehensively as much as possible." },
        { "type": "video_url", "video_url": { "url": "https://raw.githubusercontent.com/EvolvingLMMs-Lab/sglang/dev/onevision_local/assets/jobs.mp4" } }
      ]
    }]
  }'
```

--------------------------------

### ComfyUI Windows (conda) installation and run

Source: https://context7.com/intel/llm-scaler/llms.txt

Installs and runs ComfyUI natively on Windows using conda. Clones the repository, initializes the conda environment, activates it, and starts the Python script.

```powershell
git clone https://github.com/intel/llm-scaler.git
cd llm-scaler\omni\ 
.
init_conda_env.bat   # installs conda env 'omni_env'

cd ComfyUI
conda activate omni_env
$env:HTTP_PROXY = "<your_proxy>"
$env:HTTPS_PROXY = "<your_proxy>"
python .\main.py --listen 0.0.0.0
# ComfyUI accessible at http://localhost:8188/
```

--------------------------------

### Start Multi-Modal Service (V1 Engine)

Source: https://github.com/intel/llm-scaler/blob/main/vllm/README.md

Launches a vLLM service with support for multi-modal models like Qwen2.5-VL-7B-Instruct. Ensure the model path and test media path are correctly set. Logs are redirected to a file and stdout.

```bash
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
vllm serve \
    --model /llm/models/Qwen2.5-VL-7B-Instruct \
    --served-model-name Qwen2.5-VL-7B-Instruct \
    --allowed-local-media-path /llm/models/test \
    --dtype=float16 \
    --enforce-eager \
    --port 8000 \
    --host 0.0.0.0 \
    --trust-remote-code \
    --gpu-memory-util=0.9 \
    --no-enable-prefix-caching \
    --max-num-batched-tokens=5120 \
    --disable-log-requests \
    --max-model-len=5120 \
    --block-size 64 \
    --quantization fp8 \
    -tp=1 \
    2>&1 | tee /llm/vllm.log > /proc/1/fd/1 &
```

```bash
tail -f /llm/vllm.log
```

--------------------------------

### Start vLLM serving service (FP16 and FP8 quantization)

Source: https://context7.com/intel/llm-scaler/llms.txt

Launches the OpenAI-compatible vLLM server with options for FP8 or INT4 quantization. This command is recommended for most models and includes settings for optimal performance on Intel XPU.

```APIDOC
## Start vLLM serving service (FP16 and FP8 quantization)

### Description
Launches the OpenAI-compatible vLLM server with options for FP8 or INT4 quantization. This command is recommended for most models and includes settings for optimal performance on Intel XPU.

### Usage
```bash
# FP8 online quantization serving (recommended for most models)
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
vllm serve \
    --model /llm/models/DeepSeek-R1-Distill-Qwen-7B \
    --served-model-name DeepSeek-R1-Distill-Qwen-7B \
    --dtype=float16 \
    --enforce-eager \
    --port 8000 \
    --host 0.0.0.0 \
    --trust-remote-code \
    --disable-sliding-window \
    --gpu-memory-util=0.9 \
    --max-num-batched-tokens=8192 \
    --disable-log-requests \
    --max-model-len=8192 \
    --block-size 64 \
    --quantization fp8 \
    -tp=1 \
    2>&1 | tee /llm/vllm.log > /proc/1/fd/1 &
tail -f /llm/vllm.log

# INT4 online quantization — replace --quantization flag:
# --quantization sym_int4

# For large models at risk of OOM during quantization, set before launch:
export VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1

# Avoid memory fragmentation for long-running services:
export PYTORCH_ALLOC_CONF="expandable_segments:True"
```
```

--------------------------------

### Start vLLM Server for Embedding Task

Source: https://github.com/intel/llm-scaler/blob/main/vllm/README.md

Launch the vLLM server specifically configured for embedding tasks. This example sets the task type to 'embed' and specifies a model suitable for embeddings.

```bash
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
vllm serve \
    --model /llm/models/bge-m3 \
    --served-model-name bge-m3 \
    --task embed \
    --dtype=float16 \
    --enforce-eager \
    --port 8000 \
    --host 0.0.0.0 \
    --trust-remote-code \
    --disable-sliding-window \
    --gpu-memory-util=0.9 \
    --no-enable-prefix-caching \
    --max-num-batched-tokens=2048 \
    --disable-log-requests \
    --max-model-len=2048 \
    --block-size 64 \
    -tp=1
```

--------------------------------

### Start Omni Service (V1 Engine)

Source: https://github.com/intel/llm-scaler/blob/main/vllm/README.md

Launches a vLLM service with support for Omni models like Qwen2.5-Omni-7B. This configuration is similar to the multi-modal service startup, with specific model paths and parameters.

```bash
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
vllm serve \
    --model /llm/models/Qwen2.5-Omni-7B \
    --served-model-name Qwen2.5-Omni-7B \
    --allowed-local-media-path /llm/models/test \
    --dtype=float16 \
    --enforce-eager \
    --port 8000 \
    --host 0.0.0.0 \
    --trust-remote-code \
    --gpu-memory-util=0.9 \
    --no-enable-prefix-caching \
    --max-num-batched-tokens=5120 \
    --disable-log-requests \
    --max-model-len=5120 \
    --block-size 64 \
    --quantization fp8 \
    -tp=1 \
    2>&1 | tee /llm/vllm.log > /proc/1/fd/1 &
```

```bash
tail -f /llm/vllm.log
```

--------------------------------

### Qwen-Omni Multi-Modal Input Example

Source: https://github.com/intel/llm-scaler/blob/main/vllm/README.md

Demonstrates how to send multi-modal inputs (image, audio, text) to a vLLM-served model using the OpenAI API format. Ensure the model supports multi-modal inputs.

```bash
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "https://modelscope.oss-cn-beijing.aliyuncs.com/resource/qwen.png"}},
        {"type": "audio_url", "audio_url": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2.5-Omni/cough.wav"}},
        {"type": "text", "text": "What is the text in the illustration, and what is the sound in the audio?"}
    ]}
    ]
    }'
```

--------------------------------

### Multi-node distributed deployment (PP + TP via Ray)

Source: https://context7.com/intel/llm-scaler/llms.txt

Scales inference across multiple machines using Ray for coordination and Docker Swarm for networking. This example shows the setup for Node-1 (head).

```APIDOC
## Multi-node distributed deployment (PP + TP via Ray)

Scales inference across multiple machines using Ray for coordination and Docker Swarm for networking.

```bash
# --- Node-1 (head) ---
docker swarm init --advertise-addr <MachineA_IP>
docker network create --driver overlay --attachable my-overlay

sudo docker run -td --privileged --network=my-overlay \
    --device=/dev/dri --name=node-1 \
    -v /model_path:/llm/models/ --shm-size="32g" \
    --entrypoint /bin/bash intel/llm-scaler-vllm:<VERSION>

docker exec -it node-1 bash
```
```

--------------------------------

### Run Qwen-image Demo on Single GPU

Source: https://github.com/intel/llm-scaler/blob/main/omni/standalone_examples/Qwen-image/README.md

Executes the Qwen-image demo script on a single GPU. Ensure models are downloaded before running.

```bash
python3 qwen_image_example.py
```

--------------------------------

### Enable BPE-Qwen tokenizer acceleration

Source: https://context7.com/intel/llm-scaler/llms.txt

Installs the bpe-qwen package and configures vLLM to use it for faster tokenization with Qwen models. Ensure the model path and served model name are correct.

```bash
pip install bpe-qwen

# Add to vllm serve command:
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
vllm serve \
    --model /llm/models/Qwen3-14B \
    --served-model-name Qwen3-14B \
    --tokenizer-mode bpe-qwen \
    --dtype=float16 \
    --enforce-eager \
    --port 8000 \
    --host 0.0.0.0 \
    --trust-remote-code \
    --gpu-memory-util=0.9 \
    --quantization fp8 \
    -tp=1
```

--------------------------------

### Install omni_xpu_kernel Wheel

Source: https://github.com/intel/llm-scaler/blob/main/omni/omni_xpu_kernel/WHL_BUILD_INSTALL.md

Installs the newly built omni_xpu_kernel wheel into the ComfyUI embedded Python environment. Use --force-reinstall and --no-deps for a clean installation.

```batch
"%EMBED_PYTHON_DIR%\python.exe" -m pip install --force-reinstall --no-deps "%OUTPUT_DIR%\omni_xpu_kernel-0.1.0-cp312-cp312-win_amd64.whl"
```

--------------------------------

### cURL Example: Text-to-Image

Source: https://github.com/intel/llm-scaler/blob/main/omni/docs/SGLang_Diffusion_Guide.md

Example of generating an image using cURL with the OpenAI-compatible API.

```APIDOC
### cURL Examples

**Text-to-Image:**

```bash
curl http://localhost:30010/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Z-Image-Turbo",
    "prompt": "A beautiful sunset over the ocean",
    "size": "1024x1024"
  }'
```
```

--------------------------------

### Launch Qwen2.5-Omni Multimodal Service

Source: https://context7.com/intel/llm-scaler/llms.txt

Launches a vLLM service for multimodal omni models. Ensure audio dependencies are installed first. The service accepts simultaneous image, audio, and video inputs alongside text.

```bash
pip install librosa soundfile

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
vllm serve \
    --model /llm/models/Qwen2.5-Omni-7B \
    --served-model-name Qwen2.5-Omni-7B \
    --allowed-local-media-path /llm/models/test \
    --dtype=float16 \
    --enforce-eager \
    --port 8000 \
    --host 0.0.0.0 \
    --trust-remote-code \
    --gpu-memory-util=0.9 \
    --no-enable-prefix-caching \
    --max-num-batched-tokens=5120 \
    --max-model-len=5120 \
    --block-size 64 \
    --quantization fp8 \
    -tp=1 2>&1 | tee /llm/vllm.log &
```

```json
{
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "https://modelscope.oss-cn-beijing.aliyuncs.com/resource/qwen.png"}},
        {"type": "audio_url", "audio_url": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2.5-Omni/cough.wav"}},
        {"type": "text", "text": "What is the text in the image and what is the sound in the audio?"}
      ]}
    ]
  }
```

--------------------------------

### Install Transformers Version

Source: https://github.com/intel/llm-scaler/blob/main/vllm/README.md

Install a specific version of the transformers library required for certain models.

```bash
pip install transformers==4.56.1
```

--------------------------------

### Install dots.ocr Dependencies

Source: https://github.com/intel/llm-scaler/blob/main/vllm/OCR/README.md

Install the necessary Python dependencies for dots.ocr, including Gradio and related libraries.

```bash
pip install -e . --no-deps
pip install gradio gradio_image_annotation PyMuPDF qwen_vl_utils
```

--------------------------------

### Verify Git Installation

Source: https://github.com/intel/llm-scaler/blob/main/omni/comfyui_windows_setup/README.md

Check if Git for Windows is installed correctly by running this command in the Command Prompt.

```cmd
git --version
```

--------------------------------

### cURL Example: Text-to-Image with Advanced Parameters

Source: https://github.com/intel/llm-scaler/blob/main/omni/docs/SGLang_Diffusion_Guide.md

Example of generating an image with advanced parameters using cURL.

```APIDOC
**Text-to-Image with advanced parameters:**

```bash
curl http://localhost:30010/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Z-Image-Turbo",
    "prompt": "A futuristic city skyline at night, neon lights, cyberpunk",
    "negative_prompt": "blurry, ugly, bad quality",
    "size": "1024x1024",
    "num_inference_steps": 9,
    "guidance_scale": 6.0,
    "seed": 42
  }'
```
```

--------------------------------

### Verify omni_xpu_kernel Installation

Source: https://github.com/intel/llm-scaler/blob/main/omni/omni_xpu_kernel/WHL_BUILD_INSTALL.md

Verifies the installation of omni_xpu_kernel by importing it and checking its version using the embedded Python interpreter.

```python
"%EMBED_PYTHON_DIR%\python.exe" -c "import omni_xpu_kernel as ok, importlib.metadata as im; print('omni_xpu_kernel:', ok); print('version:', im.version('omni-xpu-kernel'))"
```

--------------------------------

### Start vLLM Serving with FP8 Quantization

Source: https://context7.com/intel/llm-scaler/llms.txt

Launches the OpenAI-compatible vLLM server with FP8 online quantization. This is recommended for most models. Ensure VLLM_WORKER_MULTIPROC_METHOD is set to 'spawn' for Intel XPU.

```bash
# FP8 online quantization serving (recommended for most models)
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
vllm serve \
    --model /llm/models/DeepSeek-R1-Distill-Qwen-7B \
    --served-model-name DeepSeek-R1-Distill-Qwen-7B \
    --dtype=float16 \
    --enforce-eager \
    --port 8000 \
    --host 0.0.0.0 \
    --trust-remote-code \
    --disable-sliding-window \
    --gpu-memory-util=0.9 \
    --max-num-batched-tokens=8192 \
    --disable-log-requests \
    --max-model-len=8192 \
    --block-size 64 \
    --quantization fp8 \
    -tp=1 \
    2>&1 | tee /llm/vllm.log > /proc/1/fd/1 &
tail -f /llm/vllm.log

# INT4 online quantization — replace --quantization flag:
# --quantization sym_int4

# For large models at risk of OOM during quantization, set before launch:
export VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1

# Avoid memory fragmentation for long-running services:
export PYTORCH_ALLOC_CONF="expandable_segments:True"
```

--------------------------------

### Install Omni Dependencies

Source: https://github.com/intel/llm-scaler/blob/main/vllm/README.md

Installs the necessary Python packages for Omni model support, including librosa and soundfile.

```bash
pip install librosa soundfile
```

--------------------------------

### Serve Diffusion Model (Video Generation)

Source: https://github.com/intel/llm-scaler/blob/main/omni/docs/SGLang_Diffusion_Guide.md

Launches a diffusion model server configured for video generation, with text encoder offloading to CPU and CPU memory pinning enabled.

```bash
sglang serve --model-path /llm/models/Wan2.1-T2V-1.3B-Diffusers \
    --text-encoder-cpu-offload --pin-cpu-memory \
    --num-gpus 1 --port 30010
```

--------------------------------

### Install Audio Dependencies

Source: https://github.com/intel/llm-scaler/blob/main/vllm/README.md

Installs the necessary Python packages for audio model support, including transformers and librosa.

```bash
pip install transformers==4.52.4 librosa
```

--------------------------------

### Verify ComfyUI Installation with XPU

Source: https://github.com/intel/llm-scaler/blob/main/omni/comfyui_windows_setup/README.md

After installation, confirm that PyTorch can detect and utilize the Intel XPU. This command checks for XPU availability.

```python
python_embeded\python.exe -c "import torch; print(f'XPU available: {torch.xpu.is_available()}')"
```

--------------------------------

### Run Wan 2.1 Demo on Single GPU

Source: https://github.com/intel/llm-scaler/blob/main/omni/standalone_examples/Wan2.1/README.md

Executes the text-to-video generation demo for Wan 2.1 on a single GPU. Ensure the model checkpoint is available at the specified path.

```bash
python3.10 generate.py  --task t2v-1.3B --size 832*480 --ckpt_dir /llm/models/Wan2.1-T2V-1.3B  --offload_model True --t5_cpu --sample_shift 8 --sample_guide_scale 6 --frame_num 33 --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."
```

--------------------------------

### Start Monitoring Services

Source: https://github.com/intel/llm-scaler/blob/main/vllm/monitor/README.md

Use Docker Compose to start the Prometheus and Grafana services for monitoring. Ensure you are in the correct directory before running the command.

```bash
cd llm-scaler/vllm/monitor
docker compose up #docker-compose.yaml
```

--------------------------------

### Add XPU Link Libraries (setup.py)

Source: https://github.com/intel/llm-scaler/blob/main/omni/omni_xpu_kernel/WHL_BUILD_INSTALL.md

Specifies necessary link libraries for XPU symbol resolution in the setup.py file. This is crucial for avoiding linker errors on Windows and Linux.

```python
# Add on Windows:
#   torch_xpu.lib
#   c10_xpu.lib

# Add on Linux:
#   -ltorch_xpu
#   -lc10_xpu
```

--------------------------------

### List Files in Multi-ARC Directory

Source: https://github.com/intel/llm-scaler/blob/main/vllm/README.md

This command lists the contents of the multi-arc-bmg-offline-installer directory after installation. It shows installed scripts, configuration files, and subdirectories for tools and scripts.

```bash
(base) root@intel:~/multi-arc-bmg-offline-installer-26.5.6.1# ll
total 48
drwxrwxr-x  6 intel intel 4096 Jan 26 16:16 ./
drwx------ 19 root  root  4096 Jan 29 16:55 ../
-rwxrwxr-x  1 intel intel 5769 Jan 23 11:21 install*
-rwxrwxr-x  1 intel intel 3439 Jan 23 11:31 installer.sh*
-rw-rw-r--  1 intel intel  818 Jan 23 11:31 README.md
drwxrwxr-x  2 intel intel 4096 Jan 20 19:58 results/
drwxrwxr-x  7 intel intel 4096 Jan 23 11:21 scripts/
drwxrwxr-x  3 intel intel 4096 Jan 20 19:58 tools/
drwxrwxr-x  9 intel intel 4096 Jan 19 15:31 ubuntu-25.04-desktop/
-rwxrwxr-x  1 intel intel  409 Jan 23 11:21 uninstall*
-rw-rw-r--  1 intel intel  587 Jan 23 11:31 VERSION
```

--------------------------------

### GLM-TTS Inference Example

Source: https://github.com/intel/llm-scaler/blob/main/omni/standalone_examples/GLM-TTS/README.md

Perform standard GLM-TTS inference within the running container. This command requires the model to be mounted correctly and uses example data.

```bash
cd /audio/GLM-TTS
python3 glmtts_inference.py \
    --data=example_zh \
    --exp_name=_test \
    --use_cache
```

--------------------------------

### Start Xinference Local Server

Source: https://github.com/intel/llm-scaler/blob/main/omni/README.md

Command to launch the Xinference local server. This server supports various models including image generation and speech processing.

```bash
xinference-local --host 0.0.0.0 --port 9997
```

--------------------------------

### GLM-ASR Inference Example

Source: https://github.com/intel/llm-scaler/blob/main/omni/standalone_examples/GLM-ASR/README.md

Perform standard GLM-ASR inference using a pre-trained model and an example audio file. Ensure you are in the correct directory within the container.

```bash
cd /audio/GLM-ASR
python inference.py --checkpoint_dir ./GLM-ASR-Nano-2512 --audio examples/example_zh.wav
```