### Start Example Load Balance Proxy Server

Source: https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/large_scale_ep.html

Enables an example proxy server for distributed DP. Configure prefiller and decoder node hosts, ports, and repetition counts.

```bash
python load_balance_proxy_server_example.py \
  --port 8000 \
  --host 0.0.0.0 \
  --prefiller-hosts \
    192.0.0.1 \
    192.0.0.2 \
    192.0.0.3 \
    192.0.0.4 \
  --prefiller-hosts-num \
    2 2 2 2 \
  --prefiller-ports \
    9000 9000 9000 9000 \
  --prefiller-ports-inc \
    2 2 2 2\
  --decoder-hosts \
    192.0.0.5 \
    192.0.0.6 \
    192.0.0.7 \
    192.0.0.8 \
  --decoder-hosts-num \
    16 16 16 16 \
  --decoder-ports  \
    9000 9000 9000 9000 \
  --decoder-ports-inc \
    16 16 16 16 \

```

--------------------------------

### Setup for Multi-NPU Serving with jemalloc

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen-VL-Dense.html

This script prepares the environment for multi-NPU serving by updating the OS and installing jemalloc, then setting the LD_PRELOAD environment variable. It includes conditional logic for Ubuntu and openEuler.

```bash
#!/bin/sh
# if os is Ubuntu
apt update
apt install libjemalloc2 
# if os is openEuler
yum update
yum install jemalloc
# Add the LD_PRELOAD environment variable
if [ -f /usr/lib/aarch64-linux-gnu/libjemalloc.so.2 ]; then
    # On Ubuntu, first install with `apt install libjemalloc2`
    export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
elif [ -f /usr/lib64/libjemalloc.so.2 ]; then
    # On openEuler, first install with `yum install jemalloc`
    export LD_PRELOAD=/usr/lib64/libjemalloc.so.2:$LD_PRELOAD
fi
```

--------------------------------

### Execute Proxy Script

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/DeepSeek-V3.1.html

Navigates to the example directory and executes the proxy.sh script to start the proxy server. Ensure you are in the correct directory before running.

```bash
cd vllm-ascend/examples/disaggregated_prefill_v1/
bash proxy.sh

```

--------------------------------

### Benchmark Serving Throughput with vLLM

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3-Omni-30B-A3B-Thinking.html

Set up environment variables and install necessary dependencies to benchmark the serving throughput of Qwen3-Omni-30B-A3B-Thinking using vLLM. This involves starting the OpenAI API server and then running the vLLM benchmark tool.

```bash
export VLLM_USE_MODELSCOPE=True
export MODEL=Qwen/Qwen3-Omni-30B-A3B-Thinking
python3 -m vllm.entrypoints.openai.api_server --model $MODEL --tensor-parallel-size 2 --swap-space 16 --disable-log-stats --disable-log-request --load-format dummy

pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
pip install -r vllm-ascend/benchmarks/requirements-bench.txt

vllm bench serve --model $MODEL --dataset-name random --random-input 200 --num-prompts 200 --request-rate 1 --save-result --result-dir ./.

```

--------------------------------

### Manual CANN Installation and Environment Setup

Source: https://docs.vllm.ai/projects/ascend/en/latest/installation.html

This script installs CANN and NNAL manually, along with required Python packages. It includes downloading, running installers, and sourcing environment setup scripts. Ensure you have the correct wget headers and file permissions.

```bash
# Create a virtual environment.
python -m venv vllm-ascend-env
source vllm-ascend-env/bin/activate

# Install required Python packages.
python -m pip install --upgrade pip
pip3 install attrs numpy decorator sympy cffi pyyaml pathlib2 psutil protobuf scipy requests absl-py wheel typing_extensions

# Download and install the CANN package.
wget --header="Referer: https://www.hiascend.com/" https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%209.0.0/Ascend-cann-toolkit_9.0.0_linux-"$(uname -i)".run
chmod +x ./Ascend-cann-toolkit_9.0.0_linux-"$(uname -i)".run
./Ascend-cann-toolkit_9.0.0_linux-"$(uname -i)".run --full
source /usr/local/Ascend/ascend-toolkit/set_env.sh

wget --header="Referer: https://www.hiascend.com/" https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%209.0.0/Ascend-cann-910b-ops_9.0.0_linux-"$(uname -i)".run
chmod +x ./Ascend-cann-910b-ops_9.0.0_linux-"$(uname -i)".run
./Ascend-cann-910b-ops_9.0.0_linux-"$(uname -i)".run --install

wget --header="Referer: https://www.hiascend.com/" https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%209.0.0/Ascend-cann-nnal_9.0.0_linux-"$(uname -i)".run
chmod +x ./Ascend-cann-nnal_9.0.0_linux-"$(uname -i)".run
./Ascend-cann-nnal_9.0.0_linux-"$(uname -i)".run --install

source /usr/local/Ascend/nnal/atb/set_env.sh

```

--------------------------------

### Environment Setup for vLLM on Ascend

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Kimi-K2.5.html

Sets up environment variables for performance optimization, including jemalloc, CPU governor, swappiness, and vLLM specific configurations for Ascend hardware. Ensure libjemalloc.so is installed for performance benefits.

```bash
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
kernel.sched_migration_cost_ns=50000
export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000

export HCCL_OP_EXPANSION_MODE="AIV"
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export TASK_QUEUE_ENABLE=1
export ASCEND_BUFFER_POOL=4:8
export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/mooncake:$LD_LIBRARY_PATH
```

--------------------------------

### Start vLLM Server with Docker

Source: https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/evaluation/using_lm_eval.html

Run a Docker container to start the vLLM server on a single NPU. Ensure to update the DEVICE and IMAGE environment variables according to your setup.

```bash
# Update DEVICE according to your device (/dev/davinci[0-7])
export DEVICE=/dev/davinci7
# Update the vllm-ascend image
export IMAGE=quay.io/ascend/vllm-ascend:v0.19.1rc1
docker run --rm \
--name vllm-ascend \
--shm-size=1g \
--device $DEVICE \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-p 8000:8000 \
-e VLLM_USE_MODELSCOPE=True \
-e PYTORCH_NPU_ALLOC_CONF=max_split_size_mb:256 \
-it $IMAGE \
/bin/bash
vllm serve Qwen/Qwen2.5-0.5B-Instruct --max_model_len 4096 &
```

--------------------------------

### Run Qwen3 Embedding Serving Benchmark

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3_embedding.html

Execute this command to start a serving benchmark for the Qwen3-Embedding-8B model. Ensure vLLM is installed and the model is accessible. This command configures the benchmark to use OpenAI embeddings, a random dataset, and specifies host, port, and endpoint details.

```bash
vllm bench serve --model Qwen3-Embedding-8B --backend openai-embeddings --dataset-name random --host 127.0.0.1 --port 8888 --endpoint /v1/embeddings --tokenizer /root/.cache/Qwen3-Embedding-8B --random-input 200 --save-result --result-dir ./
```

--------------------------------

### Install qwen_vl_utils for CPU

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen-VL-Dense.html

Install the qwen_vl_utils package with CPU support. This is a prerequisite for running offline inference.

```bash
pip install qwen_vl_utils --extra-index-url https://download.pytorch.org/whl/cpu/
```

--------------------------------

### Verify Ascend NPU Firmware and Driver Installation

Source: https://docs.vllm.ai/projects/ascend/en/latest/installation.html

Run this command to check if the Ascend NPU firmware and driver are installed correctly. Refer to the Ascend Environment Setup Guide for more details.

```bash
npu-smi info

```

--------------------------------

### Single Node Startup Configuration for MiniMax-M2.5

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/MiniMax-M2.5.html

Recommended environment variables and vLLM serve command for short-context performance on a single node. Adjust `--max-model-len` for specific needs and `VLLM_ASCEND_BALANCE_SCHEDULING` for enhanced scheduling.

```bash
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_BUFFSIZE=1024
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export OMP_NUM_THREADS=1
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
kernel.sched_migration_cost_ns=50000
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
export TASK_QUEUE_ENABLE=1

export VLLM_ASCEND_ENABLE_FUSED_MC2=1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
export VLLM_ASCEND_BALANCE_SCHEDULING=1

vllm serve /path/to/weight/MiniMax-M2.5-w8a8-QuaRot \
    --served-model-name "MiniMax-M2.5" \
    --host 0.0.0.0 \
    --port 8000 \
    --trust-remote-code \
    --quantization ascend \
    --async-scheduling \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --additional-config '{"enable_cpu_binding":true}' \
    --enable-expert-parallel \
    --tensor-parallel-size 4 \
    --data-parallel-size 4 \
    --max-num-seqs 48 \
    --max-model-len 40690 \
    --max-num-batched-tokens 16384 \
    --gpu-memory-utilization 0.85 \
    --speculative_config '{"enforce_eager": true, "method": "eagle3", "model": "/path/to/weight/Eagle3/", "num_speculative_tokens": 3}' \

```

--------------------------------

### Set Up Test Environment (Local CPU)

Source: https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/contribution/testing.html

Use the main branch's container image to quickly set up a local test environment for running unit tests on CPUs. This includes configuring mirrors for faster downloads and installing necessary packages.

```bash
cd ~/vllm-project/
# ls
# vllm  vllm-ascend

# Use mirror to speed up download
# docker pull m.daocloud.io/quay.io/ascend/cann:9.0.0-910b-ubuntu22.04-py3.11
export IMAGE=quay.io/ascend/cann:9.0.0-910b-ubuntu22.04-py3.11
docker run --rm --name vllm-ascend-ut \
    -v $(pwd):/vllm-project \
    -v ~/.cache:/root/.cache \
    -ti $IMAGE bash

# (Optional) Configure mirror to speed up download
sed -i 's|ports.ubuntu.com|mirrors.huaweicloud.com|g' /etc/apt/sources.list
pip config set global.index-url https://mirrors.huaweicloud.com/repository/pypi/simple/

# For torch-npu dev version or x86 machine
export PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu/ https://mirrors.huaweicloud.com/ascend/repos/pypi"

# src path
export SRC_WORKSPACE=/vllm-workspace
mkdir -p $SRC_WORKSPACE
cd $SRC_WORKSPACE

apt-get update -y
apt-get install -y python3-pip git vim wget net-tools gcc g++ cmake libnuma-dev curl gnupg2

git clone -b v0.19.1rc1 --depth 1 https://github.com/vllm-project/vllm-ascend.git
git clone --depth 1 https://github.com/vllm-project/vllm.git

# vllm
cd $SRC_WORKSPACE/vllm
VLLM_TARGET_DEVICE=empty python3 -m pip install .
python3 -m pip uninstall -y triton

# vllm-ascend
cd $SRC_WORKSPACE/vllm-ascend
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/Ascend/ascend-toolkit/latest/$(uname -m)-linux/devlib
# For cpu environment, set SOC_VERSION for different chips.
# See https://github.com/vllm-project/vllm-ascend/blob/3cb0af0bcf3299089ca7e72159fa36e825a470f8/setup.py#L132 for detail.
export SOC_VERSION="ascend910b1"
python3 -m pip install .
python3 -m pip install -r requirements-dev.txt

```

--------------------------------

### Serve Qwen3-VL-8B-Instruct on Single-NPU

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen-VL-Dense.html

Use this command to start a vLLM server for the Qwen3-VL-8B-Instruct model on a single NPU. Adjust `--max_model_len` based on your NPU's on-chip memory.

```bash
vllm serve Qwen/Qwen3-VL-8B-Instruct \
--dtype bfloat16 \
--max_model_len 16384 \
--max-num-batched-tokens 16384
```

--------------------------------

### Start PaddlePaddle CANN Docker Container

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/PaddleOCR-VL.html

Starts a Docker container with PaddlePaddle and CANN support, mounting local directories and setting environment variables for NPU access. Ensure to adjust volume mounts and environment variables as per your setup.

```bash
docker run -it --name paddle-npu-dev -v $(pwd):/work \
    --privileged --network=host --shm-size=128G -w=/work \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -e ASCEND_RT_VISIBLE_DEVICES="0,1,2,3,4,5,6,7" \
    ccr-2vdh3abv-pub.cnc.bj.baidubce.com/device/paddle-npu:cann800-ubuntu20-npu-910b-base-$(uname -m)-gcc84 /bin/bash
```

--------------------------------

### Set Up Development Environment and Run Lint Locally

Source: https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/contribution/index.html

Clone the repository, set up a virtual environment, install linting requirements, and run the linting script. This is a prerequisite for running CI locally.

```bash
cd ~/vllm-project/
python3 -m venv .venv
source ./.venv/bin/activate
git clone https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
pip install -r requirements-lint.txt
bash format.sh
```

--------------------------------

### Launch vLLM Server with Mooncake Disaggregation

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/features/pd_disaggregation_mooncake_multi_node.html

Command to start the vLLM serving engine with specific configurations for multi-node, data-parallel, and tensor-parallel setups, including Mooncake disaggregation and quantization. Adjust parameters like model path, ports, and parallel sizes according to your cluster setup.

```bash
vllm serve /path_to_weight/DeepSeek-r1_w8a8_mtp \
  --host 0.0.0.0 \
  --port $2 \
  --data-parallel-size $3 \
  --data-parallel-rank $4 \
  --data-parallel-address $5 \
  --data-parallel-rpc-port $6 \
  --tensor-parallel-size $7 \
  --enable-expert-parallel \
  --seed 1024 \
  --served-model-name ds_r1 \
  --max-model-len 40000 \
  --max-num-batched-tokens 256 \
  --max-num-seqs 40 \
  --trust-remote-code \
  --gpu-memory-utilization 0.94  \
  --quantization ascend \
  --no-enable-prefix-caching \
  --speculative-config '{"num_speculative_tokens": 1, "method":"deepseek_mtp"}' \
  --additional-config '{"recompute_scheduler_enable":true,"multistream_overlap_shared_expert": true,"finegrained_tp_config": {"lmhead_tensor_parallel_size":16}}' \
  --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
  --kv-transfer-config \
  '{"kv_connector": "MooncakeConnectorV1",
  "kv_role": "kv_consumer",
  "kv_port": "36200",
  "engine_id": "2",
  "kv_connector_extra_config": {
            "prefill": {
                    "dp_size": 2,
                    "tp_size": 8
             },
             "decode": {
                    "dp_size": 32,
                    "tp_size": 1
             }
      }
  }'
```

--------------------------------

### Start vLLM Server for Qwen2.5-VL-3B-Instruct on NPU

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/hardwares/310p.html

Launches the vLLM server for the Qwen2.5-VL-3B-Instruct model on a single NPU card. Explicitly set `--max-model-len` to avoid Out-of-Memory errors.

```bash
vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
    --tensor-parallel-size 1 \
    --max-model-len 4096 \
    --enforce-eager \
    --dtype float16

```

--------------------------------

### PD Proxy Server Configuration

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3-235B-A22B.html

Example command to start a load balancing proxy server for the prefill and decode nodes. Specify the ports and host IPs for both prefillers and decoders.

```bash
python load_balance_proxy_server_example.py --port 12347 --prefiller-hosts prefill_node_1_ip --prefiller-port 8000 --decoder-hosts decode_node_1_ip --decoder-ports 8000

```

--------------------------------

### Start vLLM Server for Qwen2.5-7B-Instruct on NPU

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/hardwares/310p.html

Launches the vLLM server for the Qwen2.5-7B-Instruct model across two NPU cards. Explicitly set `--max-model-len` to avoid Out-of-Memory errors.

```bash
vllm serve Qwen/Qwen2.5-7B-Instruct \
    --tensor-parallel-size 2 \
    --max-model-len 4096 \
    --enforce-eager \
    --dtype float16

```

--------------------------------

### Node1 (secondary) startup script for MiniMax-M2.5

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/MiniMax-M2.5.html

Configure environment variables and launch the vLLM service for the secondary node in a dual-node setup. Ensure to replace placeholders like {SecondaryNodeIP}, {PrimaryNodeIP}, and {NIC}.

```bash
# Secondary node (node1)
export HCCL_IF_IP={SecondaryNodeIP}
export GLOO_SOCKET_IFNAME="{NIC}"
export TP_SOCKET_IFNAME="{NIC}"
export HCCL_SOCKET_IFNAME="{NIC}"
export HCCL_BUFFSIZE=1024
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

export HCCL_OP_EXPANSION_MODE="AIV"
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True

export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1

export HCCL_INTRA_PCIE_ENABLE=1
export HCCL_INTRA_ROCE_ENABLE=0

# profiling (optional)
export VLLM_TORCH_PROFILER_WITH_STACK=0
export VLLM_TORCH_PROFILER_DIR="{profiling_dir}"

vllm serve /opt/data/verification/models/MiniMax-M2.5/ \
  --served-model-name "minimax25" \
  --host {SecondaryNodeIP} \
  --port 20004 \
  --headless \
  --tensor-parallel-size 8 \
  --data-parallel-size 2 \
  --data-parallel-size-local 1 \
  --data-parallel-start-rank 1 \
  --data-parallel-address {PrimaryNodeIP} \
  --data-parallel-rpc-port 2347 \
  --max-num-seqs 128 \
  --max-num-batched-tokens 65536 \
  --gpu-memory-utilization 0.92 \
  --enable-expert-parallel \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser minimax_m2 \
  --reasoning-parser minimax_m2_append_think \
  --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
  --mm_processor_cache_type="shm" \
  --async-scheduling \
  --additional-config '{"enable_cpu_binding":true}'

```

--------------------------------

### Query vLLM Server

Source: https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/evaluation/using_opencompass.html

Sends a request to the running vLLM server to get a text completion. This example uses `curl` to send a JSON payload with the prompt and generation parameters.

```bash
curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen2.5-7B-Instruct",
        "prompt": "The future of AI is",
        "max_completion_tokens": 7,
        "temperature": 0
    }'

```

--------------------------------

### Serve Model with Dynamic Batching Enabled

Source: https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/dynamic_batch.html

Example command to start vLLM inference server with dynamic batching enabled. Adjust the SLO_LIMIT and other parameters based on model and service requirements.

```bash
SLO_LIMIT=50
vllm serve Qwen/Qwen2.5-14B-Instruct \
    --additional_config '{"SLO_limits_for_dynamic_batch":'${SLO_LIMIT}'}' \
    --max-num-seqs 256 \
    --block-size 128 \
    --tensor_parallel_size 8 \
    --load_format dummy \
    --max_num_batched_tokens 1024 \
    --max-model-len 9000 \
    --host localhost \
    --port 12091 \
    --gpu-memory-utilization 0.9 \
    --trust-remote-code

```

--------------------------------

### Run Load Balance Proxy Server Example

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Kimi-K2.5.html

Execute the proxy script to set up a load balancing proxy for prefiller and decoder services. Ensure you are in the correct directory.

```python
python load_balance_proxy_server_example.py \
  --port 1999 \
  --host 141.xx.xx.1 \
  --prefiller-hosts \
    141.xx.xx.1 \
    141.xx.xx.1 \
    141.xx.xx.2 \
    141.xx.xx.2 \
  --prefiller-ports \
    7100 7101 7100 7101 \
  --decoder-hosts \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
  --decoder-ports \
    7100 7101 7102 7103 7104 7105 7106 7107 7108 7109 7110 7111 7112 7113 7114 7115 \
    7100 7101 7102 7103 7104 7105 7106 7107 7108 7109 7110 7111 7112 7113 7114 7115
```

--------------------------------

### Set Up Test Environment (Multi Cards)

Source: https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/contribution/testing.html

Configure and run a Docker container for testing on multiple NPU devices. Mount all specified device files, manager, SVM, and HDC devices. Ensure the correct vLLM Ascend image is used and necessary host directories are mounted.

```bash
# Update the vllm-ascend image
export IMAGE=quay.io/ascend/vllm-ascend:main
docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /root/.cache:/root/.cache \
    -p 8000:8000 \
    -it $IMAGE bash

```

```bash
cd /vllm-workspace/vllm-ascend/

# Prepare
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple

# Install required packages
pip install -r requirements-dev.txt

```

--------------------------------

### Verify Hunyuan Model with cURL

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Hunyuan-A13B-Instruct.html

Sends a POST request to the deployed vLLM server to get a chat completion from the Hunyuan model. This example demonstrates how to format the request with messages, max tokens, and temperature.

```bash
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Hunyuan",
        "messages": [{"role": "user", "content": "Give me a short introduction to large language models."}],
        "max_tokens": 100,
        "temperature": 0.7
    }'

```

--------------------------------

### Start vLLM Server for Qwen3-0.6B on NPU

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/hardwares/310p.html

Launches the vLLM server for the Qwen3-0.6B model on a single NPU card. Explicitly set `--max-model-len` to avoid Out-of-Memory errors.

```bash
vllm serve Qwen/Qwen3-0.6B \
    --tensor-parallel-size 1 \
    --max-model-len 4096 \
    --enforce-eager \
    --dtype float16

```

--------------------------------

### vLLM Server Startup Logs

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3-VL-235B-A22B-Instruct.html

Example log output indicating a successful vLLM server startup on node 0. This confirms the application is running and ready to accept requests.

```log
INFO:     Started server process [44610]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Started server process [44611]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

```

--------------------------------

### Run Decode Service with Mooncake

Source: https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/ucm_deployment.html

Starts the Decode service, configured to consume KV cache via Mooncake. Requires similar environment variable setup as the Prefill service. Note the different `kv_role` and `kv_connector` configuration.

```bash
export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:$LD_LIBRARY_PATH
export PYTHONHASHSEED=0
export PYTHONPATH=$PYTHONPATH:/vllm-workspace/vllm
export MOONCAKE_CONFIG_PATH="./mooncake.json"
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3

vllm serve /models/QwQ-32B \
    --host 0.0.0.0 \
    --port 9000 \
    --data-parallel-size 1 \
    --tensor-parallel-size 4 \
    --seed 1024 \
    --max-model-len 17000 \
    --max-num-batched-tokens 8000 \
    --trust-remote-code \
    --max-num-seqs 4 \
    --gpu-memory-utilization 0.92 \
    --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
    --kv-transfer-config \
    '{ 
        "kv_connector": "MooncakeConnectorV1",
        "kv_role": "kv_consumer",
        "kv_port": 20001,
        "kv_connector_extra_config": {
            "prefill": {"dp_size": 1, "tp_size": 4},
            "decode": {"dp_size": 1, "tp_size": 4}
        }
    }'

```

--------------------------------

### Set Up Test Environment (Single Card)

Source: https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/contribution/testing.html

Configure and run a Docker container for testing on a single NPU device. Ensure the DEVICE environment variable is set correctly and the correct vLLM Ascend image is used. Mount necessary directories and ports.

```bash
# Update DEVICE according to your device (/dev/davinci[0-7])
export DEVICE=/dev/davinci0
# Update the vllm-ascend image
export IMAGE=quay.io/ascend/vllm-ascend:main
docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --device $DEVICE \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /root/.cache:/root/.cache \
    -p 8000:8000 \
    -it $IMAGE bash

```

```bash
# Prepare
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple

# Switch to the /vllm-workspace/vllm-ascend directory
cd /vllm-workspace/vllm-ascend/

# Install required packages
pip install -r requirements-dev.txt

```

--------------------------------

### Start vLLM Server for Online Inference

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3-30B-A3B.html

Initiate the vLLM server for online inference on Multi-NPU. Adjust `tensor-parallel-size` based on available NPU memory. For example, use at least 2 for 64GB cards and 4 for 32GB cards.

```bash
vllm serve Qwen/Qwen3-30B-A3B --tensor-parallel-size 4 --enable-expert-parallel

```

--------------------------------

### vLLM Serving Command for Qwen3.5-397B-A17B

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3.5-397B-A17B.html

This command initiates the vLLM server for the specified model on Ascend hardware. Key parameters control parallelism, quantization, model configuration, and resource utilization.

```bash
vllm serve Eco-Tech/Qwen3.5-397B-A17B-w8a8-mtp \
--host 0.0.0.0 \
--port 8000 \
--data-parallel-size 1 \
--tensor-parallel-size 16 \
--enable-expert-parallel \
--seed 1024 \
--quantization ascend \
--served-model-name qwen3.5 \
--max-num-seqs 128 \
--max-model-len 133000 \
--max-num-batched-tokens 16384 \
--trust-remote-code \
--gpu-memory-utilization 0.90 \
--enable-prefix-caching \
--speculative_config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3, "enforce_eager": true}' \
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
--additional-config '{"enable_cpu_binding":true}' \
--async-scheduling
```

--------------------------------

### Online Inference with Quantized Model

Source: https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/quantization.html

Start an API server for online inference with a quantized model. This command corresponds to the offline inference setup. For ModelSlim-quantized models, the `--quantization ascend` parameter is implicitly handled by vLLM Ascend when the model path is recognized.

```bash
# Corresponding to offline inference
python -m vllm.entrypoints.api_server \
    --model /path/to/your/quantized_model \
    --max-model-len 4096 \
    --port 8000 \
    --tensor-parallel-size 2 \
    --data-parallel-size 1 \
    --served-model-name quantized_model \
    --trust-remote-code 
```

--------------------------------

### Start vLLM Server for Video Inputs

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3-VL-30B-A3B-Instruct.html

Run this command to start the vLLM server configured for video inputs. It enables expert parallelism and specifies an allowed local media path for video files.

```bash
vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
--tensor-parallel-size 2 \
--enable-expert-parallel \
--max-model-len 128000 \
--allowed-local-media-path /media
```

--------------------------------

### Run vLLM Server with Docker

Source: https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/evaluation/using_opencompass.html

Starts a vLLM server on a single NPU using a Docker container. Ensure to update DEVICE and IMAGE environment variables according to your setup. This command mounts necessary directories and exposes port 8000 for API access.

```bash
# Update DEVICE according to your device (/dev/davinci[0-7])
export DEVICE=/dev/davinci7
# Update the vllm-ascend image
export IMAGE=quay.io/ascend/vllm-ascend:v0.19.1rc1
docker run --rm \
--name vllm-ascend \
--shm-size=1g \
--device $DEVICE \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-p 8000:8000 \
-e VLLM_USE_MODELSCOPE=True \
-e PYTORCH_NPU_ALLOC_CONF=max_split_size_mb:256 \
-it $IMAGE \
vllm serve Qwen/Qwen2.5-7B-Instruct --max_model_len 26240

```

--------------------------------

### Start Memcache MetaService via Executable

Source: https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/kv_pool.html

Execute the Memcache meta service binary after setting environment variables, including the configuration path and library path. This method is an alternative to starting via Python.

```shell
source /usr/local/memcache_hybrid/set_env.sh
source /usr/local/memfabric_hybrid/set_env.sh
export MMC_META_CONFIG_PATH=/home/memcache/shell/mmc-meta.conf # Set it to the path of your own configuration file.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/python3.11.10/lib/
/usr/local/memcache_hybrid/latest/aarch64-linux/bin/mmc_meta_service
```

--------------------------------

### Start vLLM Server with Ascend Quantization

Source: https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/contribution/multi_node_test.html

This command starts the vLLM serving process with specific configurations for multi-node testing on Ascend hardware. It includes settings for data and tensor parallelism, quantization, and KV cache transfer.

```bash
vllm serve vllm-ascend/DeepSeek-V3-W8A8 --host 0.0.0.0 --port 8080 --data-parallel-size 2 --data-parallel-size-local 2 --tensor-parallel-size 8 --seed 1024 --enforce-eager --enable-expert-parallel --max-num-seqs 16 --max-model-len 8192 --max-num-batched-tokens 8192 --quantization ascend --trust-remote-code --no-enable-prefix-caching --gpu-memory-utilization 0.9 --kv-transfer-config {"kv_connector": "MooncakeConnectorV1", "kv_role": "kv_producer", "kv_port": "30000", "engine_id": "0", "kv_connector_extra_config": {
        "prefill": {
                "dp_size": 2,
                "tp_size": 8
        },
        "decode": {
                "dp_size": 2,
                "tp_size": 8
        }
    }
}
```

--------------------------------

### Configure Distributed DP Server Parameters (Decoder Node)

Source: https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/large_scale_ep.html

This Python script sets up the environment for the decoder nodes in a distributed DP setup. It defines the total number of DP engines, the number of DP engines per node, and the starting DP rank for the current node.

```python
import multiprocessing
import os
import sys
dp_size = 64 # total number of DP engines for decode/prefill
dp_size_local = 16 # number of DP engines on the current node
dp_rank_start = 0 # starting DP rank for the current node. e.g. 0/16/32/48

```

--------------------------------

### Install uv-wheelnext

Source: https://docs.vllm.ai/projects/ascend/en/latest/installation.html

Installs uv-wheelnext to enable incremental wheel downloads. This is part of the uv-wheelnext installation method.

```bash
# install uv-wheelnext
curl -LsSf https://astral.sh/uv/install.sh | sed 's/verify_checksum "$_file"/true/' | INSTALLER_DOWNLOAD_URL=https://wheelnext.astral.sh sh
source $HOME/.local/bin/env
```

--------------------------------

### Install vLLM and vLLM-Ascend using uv-wheelnext

Source: https://docs.vllm.ai/projects/ascend/en/latest/installation.html

Installs vLLM and vLLM-Ascend using uv-wheelnext for smaller download sizes. This method supports incremental wheels.

```bash
# Install vllm-project/vllm. The newest supported version is v0.19.1.
pip install vllm==0.19.1
```

```bash
# Install vllm-project/vllm-ascend from wheelnext index.
uv pip install --system \
--extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi/variant   \
vllm-ascend==0.19.1rc1
```

--------------------------------

### Install LMCache Package

Source: https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/lmcache_ascend_deployment.html

Install the LMCache package with CUDA extensions disabled. This is part of the manual installation process.

```bash
NO_CUDA_EXT=1 pip install lmcache==0.3.12

```

--------------------------------

### Install Pandas

Source: https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/dynamic_batch.html

Install the pandas library if it is not already installed. This is required for loading the dynamic batching lookup table.

```bash
pip install pandas 
```

--------------------------------

### Serve Kimi-K2.5 Model with vLLM

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Kimi-K2.5.html

Launches the vLLM serving process for the Kimi-K2.5 model, specifying various parameters for distributed training, quantization, and performance tuning. This command is intended for use with Ascend hardware.

```bash
vllm serve Eco-Tech/Kimi-K2.5-W4A8 \
  --host 0.0.0.0 \
  --port $2 \
  --data-parallel-size $3 \
  --data-parallel-rank $4 \
  --data-parallel-address $5 \
  --data-parallel-rpc-port $6 \
  --tensor-parallel-size $7 \
  --enable-expert-parallel \
  --seed 1024 \
  --quantization ascend \
  --served-model-name kimi_k25 \
  --trust-remote-code \
  --max-num-seqs 48 \
  --max-model-len 32768 \
  --max-num-batched-tokens 256 \
  --no-enable-prefix-caching \
  --gpu-memory-utilization 0.95 \
  --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[4,8,16,32,48,64,80,96,112,128,144,160]}' \
  --additional-config '{"recompute_scheduler_enable":true,"multistream_overlap_shared_expert": false}' \
  --speculative-config '{"method": "eagle3", "model":"lightseekorg/kimi-k2.5-eagle3", "num_speculative_tokens": 3}' \
  --kv-transfer-config \
  '{"kv_connector": "MooncakeConnectorV1",
  "kv_role": "kv_consumer",
  "kv_port": "30200",
  "engine_id": "2",
  "kv_connector_extra_config": {
            "prefill": {
                    "dp_size": 2,
                    "tp_size": 8
            },
            "decode": {
                    "dp_size": 32,
                    "tp_size": 1
            }
      }
  }'
```

--------------------------------

### Install Mooncake Dependencies

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/features/pd_colocated_mooncake_multi_instance.html

Run the provided script to install necessary dependencies for Mooncake. Go installation is not required.

```bash
bash dependencies.sh -y

```

--------------------------------

### vLLM-Ascend Startup Script for DeepSeek-V3.1

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/features/dynamic_chunked_pipeline_parallel.html

Configures environment variables and starts the vLLM server with Dynamic Chunked Pipeline Parallelism enabled for DeepSeek-V3.1.

```bash
#!/bin/sh
unset https_proxy
unset http_proxy

export OMP_PROC_BIND=false
export PYTORCH_NPU_ALLOC_CONF="expandable_segments:True"
export OMP_NUM_THREADS=1
export HCCL_BUFFSIZE=2048
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
export HCCL_OP_EXPANSION_MODE="AIV"
export VLLM_USE_V1=1
export TASK_QUEUE_ENABLE=1
export ASCEND_LAUNCH_BLOCKING=0
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
export VLLM_ASCEND_ENABLE_FUSED_MC2=1
export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=204
export HCCL_CONNECT_TIMEOUT=120

vllm serve /mnt/weight/DeepSeek-V3.1-w8a8 \
  --host 0.0.0.0 \
  --port 8003 \
  --served-model-name model \
  --data-parallel-size 1 \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2 \
  --enable-expert-parallel \
  --max-num-seqs 32 \
  --max-model-len 131072 \
  --max-num-batched-tokens 32768 \
  --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill \
  --no-enable-prefix-caching \
  --trust-remote-code \
  --quantization ascend \
  --additional-config '{ \
    "profiling_chunk_config":{"enabled":true, "smooth_factor":1.0, "min_chunk":4096}
  }'

```

--------------------------------

### Install AISBench from Source

Source: https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/evaluation/using_ais_bench.html

Clone the AISBench repository and install it using pip. This command installs AISBench and its dependencies.

```bash
git clone https://github.com/AISBench/benchmark.git
cd benchmark/
pip3 install -e ./ --use-pep517
```

--------------------------------

### Start vLLM Server for Image Inputs

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3-VL-30B-A3B-Instruct.html

Run this command to start the vLLM server optimized for image inputs. It enables expert parallelism and sets specific limits for multimedia prompts and model length.

```bash
vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
--tensor-parallel-size 2 \
--enable-expert-parallel \
--limit-mm-per-prompt.video 0 \
--max-model-len 128000
```