vLLM Ascend Plugin

https://docs.vllm.ai/projects/ascend/en/latest
Admin
vLLM Ascend Plugin is a hardware plugin that enables running vLLM on Ascend NPU, supporting popular...
Tokens:375,217
Snippets:3,900
Trust Score:9.5
Update:3 months ago
Show doc for...
Tokens:
Raw
### Server startup logs

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen-VL-Dense.html

Example output indicating the server has successfully started.

```text
INFO:     Started server process [2736]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
```

--------------------------------

### Configure and Start MiniMax-M2 Service

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/MiniMax-M2.html

Environment setup and vLLM serve command for short-context scenarios on A3 nodes.

```bash
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_BUFFSIZE=1024
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export OMP_NUM_THREADS=1
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl kernel.sched_migration_cost_ns=50000
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
export TASK_QUEUE_ENABLE=1

export VLLM_ASCEND_BALANCE_SCHEDULING=0

vllm serve /path/to/weight/MiniMax-M2.7-w8a8-QuaRot \
    --served-model-name "MiniMax-M2.7" \
    --host 0.0.0.0 \
    --port 8000 \
    --trust-remote-code \
    --quantization ascend \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --async-scheduling \
    --additional-config '{"enable_cpu_binding":true,
                          "enable_fused_mc2":true,
                          "enable_flashcomm1":true,
                          "weight_nz_mode":true}' \
    --enable-expert-parallel \
    --tensor-parallel-size 4 \
    --data-parallel-size 4 \
    --max-num-seqs 48 \
    --max-model-len 40690 \
    --max-num-batched-tokens 16384 \
    --gpu-memory-utilization 0.85 \
    --speculative_config '{"enforce_eager": true, "method": "eagle3", "model": "/path/to/weight/Eagle3/", "num_speculative_tokens": 3}'
```

--------------------------------

### Run vLLM Serve Benchmark

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3-Omni-30B-A3B-Thinking.html

Configures the environment, starts the API server, installs dependencies, and executes the serving benchmark.

```bash
export VLLM_USE_MODELSCOPE=True
export MODEL=Qwen/Qwen3-Omni-30B-A3B-Thinking
python3 -m vllm.entrypoints.openai.api_server --model $MODEL --tensor-parallel-size 2 --swap-space 16 --disable-log-stats --disable-log-request --load-format dummy

pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
pip install -r vllm-ascend/benchmarks/requirements-bench.txt

vllm bench serve --model $MODEL --dataset-name random --random-input 200 --num-prompts 200 --request-rate 1 --save-result --result-dir ./
```

--------------------------------

### Install vLLM from Source

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3-Next.html?q=

Clone and install the core vLLM repository from source.

```bash
git clone https://github.com/vllm-project/vllm.git
cd vllm
pip install -e .
```

--------------------------------

### Install AISBench

Source: https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/evaluation/using_ais_bench.html?q=

Clone and install the AISBench framework from source.

```bash
git clone https://github.com/AISBench/benchmark.git
cd benchmark/
pip3 install -e ./ --use-pep517
```

--------------------------------

### Start etcd Cluster

Source: https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/kv_pool.html?q=

Download, install, and start a single-node etcd cluster for service discovery.

```bash
ETCD_VERSION="v3.5.12"
ETCD_IP="127.0.0.1"
if [ "$(uname -m)" = "aarch64" ]; then
  ETCD_ARCH="linux-arm64"
else
  ETCD_ARCH="linux-amd64"
fi
wget https://github.com/etcd-io/etcd/releases/download/${ETCD_VERSION}/etcd-${ETCD_VERSION}-${ETCD_ARCH}.tar.gz
tar -xvf etcd-${ETCD_VERSION}-${ETCD_ARCH}.tar.gz
cd etcd-${ETCD_VERSION}-${ETCD_ARCH}
sudo cp etcd etcdctl /usr/local/bin/

etcd \
  --name etcd-single \
  --data-dir /tmp/etcd-data \
  --listen-client-urls http://0.0.0.0:2379 \
  --advertise-client-urls http://${ETCD_IP}:2379 \
  --listen-peer-urls http://0.0.0.0:2380 \
  --initial-advertise-peer-urls http://${ETCD_IP}:2380 \
  --initial-cluster etcd-single=http://${ETCD_IP}:2380 &

etcdctl --endpoints "${ETCD_IP}:2379" put key "value"
etcdctl --endpoints "${ETCD_IP}:2379" get key
```

--------------------------------

### Start Docker Container for Qwen3-Next

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3-Next.html

Launches the vllm-ascend Docker container with necessary device mappings and volume mounts for A3 series hardware.

```bash
#!/bin/sh
# Update the vllm-ascend image
# For Atlas A2 machines:
# export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1
# For Atlas A3 machines:
export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1-a3
docker run --rm \
--shm-size=1g \
--name vllm-ascend-qwen3 \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci4 \
--device /dev/davinci5 \
--device /dev/davinci6 \
--device /dev/davinci7 \
--device /dev/davinci8 \
--device /dev/davinci9 \
--device /dev/davinci10 \
--device /dev/davinci11 \
--device /dev/davinci12 \
--device /dev/davinci13 \
--device /dev/davinci14 \
--device /dev/davinci15 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-p 8000:8000 \
-it $IMAGE bash
```

--------------------------------

### Run vLLM Ascend Container on Ubuntu

Source: https://docs.vllm.ai/projects/ascend/en/latest/quick_start.html?q=

Configures and starts the vLLM Ascend container on Ubuntu, including the installation of curl.

```bash
# Update DEVICE according to your device (/dev/davinci[0-7])
export DEVICE=/dev/davinci0
# Update the vllm-ascend image
# Atlas A2:
# export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1
# Atlas A3:
# export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1-a3
export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1
docker run --rm \
--name vllm-ascend \
--shm-size=1g \
--device $DEVICE \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-p 8000:8000 \
-it $IMAGE bash
# Install curl
apt-get update -y && apt-get install -y curl
```

--------------------------------

### Set up local linting environment

Source: https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/contribution/index.html

Commands to initialize a virtual environment, clone the repository, and install linting dependencies.

```bash
# Choose a base dir (~/vllm-project/) and set up venv
cd ~/vllm-project/
python3 -m venv .venv
source ./.venv/bin/activate

# Clone vllm-ascend and install
git clone https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend

# Install lint requirement and enable pre-commit hook
pip install -r requirements-lint.txt

# Run lint (You need to install pre-commits deps via proxy network at first time)
bash format.sh
```

--------------------------------

### Configure and Start Decode Node

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/DeepSeek-V4-Flash.html

Environment variables and vllm serve command for initializing a decode node in a PD separation setup.

```bash
unset ftp_proxy
unset https_proxy
unset http_proxy
rm -rf ~/ascend/log

nic_name="xxxxxx" #eg."enp67s0f0np0"
local_ip=`hostname -I|awk -F " " '{print$1}'`

export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
export HCCL_OP_EXPANSION_MODE="AIV"
export TASK_QUEUE_ENABLE=1
export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=204
export HCCL_CONNECT_TIMEOUT=1200

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=1024

export ASCEND_RT_VISIBLE_DEVICES=$1

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Flash-w8a8-mtp \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name dsv4 \
    --max-model-len 135000 \
    --max-num-batched-tokens 60 \
    --max-num-seqs 30 \
    --async-scheduling \
    --block-size 128 \
    --no-disable-hybrid-kv-cache-manager \
    --no-enable-prefix-caching \
    --trust-remote-code \
    --gpu-memory-utilization 0.9 \
    --quantization ascend \
    --safetensors-load-strategy 'prefetch' \
    --model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
    --tokenizer-mode deepseek_v4 \
    --tool-call-parser deepseek_v4 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v4 \
    --speculative-config '{"num_speculative_tokens": 1, "method": "mtp","enforce_eager": true}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --kv-transfer-config \
    '{"kv_connector": "MooncakeHybridConnector",
    "kv_role": "kv_consumer",
    "kv_port": "30400",
    "engine_id": "4",
    "kv_connector_extra_config": {
                "prefill": {
                    "dp_size": 8,
                    "tp_size": 1
                },
                "decode": {
                    "dp_size": 32,
                    "tp_size": 1
                }
        }
    }' \
    --additional-config '{
        "ascend_compilation_config":{
              "enable_npugraph_ex":true,
              "enable_static_kernel":false
        },
       "enable_cpu_binding":true,
       "multistream_overlap_shared_expert":true,
       "recompute_scheduler_enable":true
    }'
```

--------------------------------

### Start vLLM Online Serving

Source: https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/performance_and_debug/performance_benchmark.html

Initializes the model server for online benchmarking.

```bash
export VLLM_USE_MODELSCOPE=True
vllm serve Qwen/Qwen3-8B
```

--------------------------------

### Run vLLM Ascend Container on openEuler

Source: https://docs.vllm.ai/projects/ascend/en/latest/quick_start.html?q=

Configures and starts the vLLM Ascend container on openEuler, including the installation of curl via yum.

```bash
# Update DEVICE according to your device (/dev/davinci[0-7])
export DEVICE=/dev/davinci0
# Update the vllm-ascend image
# Atlas A2:
# export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1-openeuler
# Atlas A3:
# export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1-a3-openeuler
export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1-openeuler
docker run --rm \
--name vllm-ascend \
--shm-size=1g \
--device $DEVICE \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-p 8000:8000 \
-it $IMAGE bash
# Install curl
yum update -y && yum install -y curl
```

--------------------------------

### Deploy Kimi-K2.6-W4A8 with vLLM

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Kimi-K2.6.html?q=

Startup script for configuring environment variables and launching the vLLM server for the Kimi-K2.6-W4A8 model.

```bash
#!/bin/sh
export HCCL_OP_EXPANSION_MODE="AIV"
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export TASK_QUEUE_ENABLE=1
export VLLM_ASCEND_ENABLE_MLAPO=1

# [Optional] jemalloc
# jemalloc is for better performance, if `libjemalloc.so` is installed on your machine, you can turn it on.
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

export HCCL_BUFFSIZE=600
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
export VLLM_ASCEND_BALANCE_SCHEDULING=0

vllm serve Eco-Tech/Kimi-K2.6-W4A8 \
    --quantization ascend \
    --tool-call-parser kimi_k2 \
    --reasoning-parser kimi_k2 \
    --allowed-local-media-path / \
    --served-model-name kimi_k26 \
    --trust-remote-code \
    --tensor-parallel-size 4 \
    --data-parallel-size 4 \
    --no-enable-prefix-caching \
    --enable-expert-parallel \
    --port 8089 \
    --max-num-seqs 4 \
    --max-model-len 34816 \
    --max-num-batched-tokens 16384 \
    --gpu-memory-utilization 0.87 \
    --seed 42 \
    --async-scheduling \
    --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
    --profiler-config '{"profiler": "torch", "torch_profiler_dir": "./vllm_profile", "torch_profiler_with_stack": true}' \
    --mm-processor-cache-gb 0 \
    --mm-encoder-tp-mode data \
    --additional-config '{"enable_balance_scheduling": true}' \
    --speculative-config '{"method": "dflash","model": "z-lab/Kimi-K2.5-DFlash", "num_speculative_tokens": 15}'
```

--------------------------------

### Build and preview documentation site

Source: https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/contribution/doc_writing.html

Commands to install dependencies, clean previous builds, compile the site, and start a local preview server.

```bash
# Install documentation build dependencies
python3 -m pip install -r docs/requirements-docs.txt

# (Optional) Clean previous builds
make -C docs clean

# Build the English site
make -C docs html

# (Optional) Build the Chinese site
make -C docs intl

# Preview locally
python3 -m http.server -d docs/_build/html 8000

# Then open in a browser:
# http://localhost:8000
```

--------------------------------

### Define Multi-Node Cluster Configuration

Source: https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/contribution/multi_node_test.html

Example YAML configuration for a 2-node disaggregated prefill setup. Use this structure to define environment variables, node roles, and server commands for each node in the cluster.

```yaml
test_name: "test DeepSeek-V3 disaggregated_prefill"
# the model being tested
model: "vllm-ascend/DeepSeek-V3-W8A8"
# how large the cluster is
num_nodes: 2
npu_per_node: 16
# All env vars you need should add it here
env_common: &env_common
  VLLM_USE_MODELSCOPE: true
  OMP_PROC_BIND: false
  OMP_NUM_THREADS: 100
  HCCL_BUFFSIZE: 1024
  SERVER_PORT: 8080
disaggregated_prefill:
  enabled: true
  # node index(a list) which meet all the conditions:
  #  - prefiller
  #  - no headless(have api server)
  prefiller_host_index: [0]
  # node index(a list) which meet all the conditions:
  #  - decoder
  decoder_host_index: [1]

# Add each node's vllm serve cli command just like you run locally
# Add each node's individual envs like follow
deployment:
- name: prefiller node # optional: just for description, not used in code
  envs:
    <<: *env_common
    VLLM_ASCEND_ENABLE_FLASHCOMM1: 1
    # Continue to add other envs if needed
  server_cmd: >
    vllm serve ...
- name: decoder node # optional: just for description, not used in code
  envs:
    <<: *env_common
    VLLM_ASCEND_ENABLE_FLASHCOMM1: 1
    # Continue to add other envs if needed
  server_cmd: >
    vllm serve ...
benchmarks:
  perf:
    # fill with performance test kwargs
  acc:
    # fill with accuracy test kwargs
```

--------------------------------

### Deploy Qwen 3.5-27B-w8a8 with MTP

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3.5-27B-Qwen3.6-27B.html

Startup script for deploying the quantized Qwen 3.5 model with speculative decoding enabled.

```bash
#!/bin/sh
# Load model from ModelScope to speed up download
export VLLM_USE_MODELSCOPE=True
# To reduce memory fragmentation and avoid out of memory
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
# Size of the shared buffer (in MB) used by HCCL for NPU-to-NPU collective communication
export HCCL_BUFFSIZE=512
# Whether OpenMP threads are bound to specific CPU cores
export OMP_PROC_BIND=false
# Number of OpenMP threads available for parallel regions
export OMP_NUM_THREADS=1
# Enables the Ascend task queue for asynchronous operator dispatch
export TASK_QUEUE_ENABLE=1

# Model weight path; can be a ModelScope model id (e.g., Eco-Tech/Qwen3.5-27B-w8a8-mtp) or a local directory path
export MODEL_PATH=Eco-Tech/Qwen3.5-27B-w8a8-mtp

vllm serve $MODEL_PATH \
--host 0.0.0.0 \
--port 8000 \
--data-parallel-size 1 \
--tensor-parallel-size 2 \
--seed 1024 \
--quantization ascend \
--served-model-name qwen3.5 \
--max-num-seqs 32 \
--max-model-len 133000 \
--max-num-batched-tokens 8096 \
--trust-remote-code \
--gpu-memory-utilization 0.90 \
--no-enable-prefix-caching \
--speculative-config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3, "enforce_eager": true}' \
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
--additional-config '{"enable_cpu_binding":true}' \
--async-scheduling
```

--------------------------------

### Verify Mooncake Installation

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/features/pd_colocated_mooncake_multi_instance.html

Checks the installation by printing the path to the installed mooncake package.

```bash
python -c "import mooncake; print(mooncake.__file__)"
# Expected output path:
# /usr/local/Ascend/ascend-toolkit/latest/python/
# site-packages/mooncake/__init__.py
```

--------------------------------

### Start the RFork Planner

Source: https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/rfork.html

Launch the planner service using the provided script.

```bash
python rfork_planner.py \
  --host 0.0.0.0 \
  --port <planner_port>
```

--------------------------------

### Verify Server Startup

Source: https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/evaluation/using_ais_bench.html?q=

Expected log output indicating the server has successfully started.

```text
INFO:     Started server process [9446]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
```

--------------------------------

### Install triton-ascend

Source: https://docs.vllm.ai/projects/ascend/en/latest/installation.html?q=

Installs the triton-ascend package, required to be installed last when using uv.

```bash
pip install triton-ascend==3.2.1 --extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi
```

--------------------------------

### Install LLM-Compressor

Source: https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/quantization.html?q=

Install the LLM-Compressor library via pip.

```bash
pip install llmcompressor
```

--------------------------------

### Start the vLLM Server for Reranking

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3-VL-Reranker.html

Launch the vLLM server with specific overrides required for the Qwen3-VL-Reranker architecture.

```bash
vllm serve Qwen/Qwen3-VL-Reranker-8B \
    --runner pooling \
    --max-model-len 4096 \
    --hf_overrides '{"architectures": ["Qwen3VLForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}' \
    --chat-template ./qwen3_vl_reranker.jinja
```

--------------------------------

### Install Mooncake Dependencies

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/features/pd_colocated_mooncake_multi_instance.html

Runs the dependency installation script.

```bash
bash dependencies.sh -y
```

--------------------------------

### Start Qwen3-Reranker server with vLLM

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3_reranker.html?q=

Command to launch the vLLM server for the Qwen3-Reranker-8B model with necessary architecture overrides.

```bash
vllm serve Qwen/Qwen3-Reranker-8B --host 127.0.0.1 --port 8888 --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'
```

--------------------------------

### Deployment Configurations for Qwen3-Dense

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3-Dense.html

Environment variables and vLLM serve commands tailored for specific performance scenarios on Ascend NPUs.

```bash
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1
export HCCL_OP_EXPANSION_MODE="AIV"

vllm serve your_model_path \
  --served-model-name qwen3 \
  --trust-remote-code \
  --distributed-executor-backend mp \
  --tensor-parallel-size 4 \
  --max-model-len 5500 \
  --max-num-batched-tokens 40960 \
  --no-enable-prefix-caching \
  --async-scheduling \
  --quantization ascend \
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[4,8,64,72,76,80,96,100,120,140,144,160,192,216,240,252,288,320,336,360,384,400,408,416,420,432,480,540,576,600]}' \
  --additional-config '{"weight_prefetch_config":{"enabled":true}, "enable_flashcomm1": true, "pa_shape_list":[32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256]}' \
  --host <host_ip> \
  --port <port> \
  --block-size 128 \
  --gpu-memory-utilization 0.9
```

```bash
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1
export HCCL_OP_EXPANSION_MODE="AIV"

vllm serve your_model_path \
  --host <host_ip> \
  --port <port> \
  --served-model-name qwen3 \
  --trust-remote-code \
  --seed 1024 \
  --max-model-len 135000 \
  --max-num-batched-tokens 40960 \
  --tensor-parallel-size 4 \
  --distributed-executor-backend "mp" \
  --async-scheduling \
  --no-enable-prefix-caching \
  --speculative-config '{"method": "eagle3", "model":"your_eagle3_model_path", "num_speculative_tokens": 3}' \
  --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
  --hf-overrides '{"rope_parameters": {"rope_type":"yarn","rope_theta":1000000,"factor":4,"original_max_position_embeddings":131072}}' \
  --gpu-memory-utilization 0.9 \
  --quantization ascend \
  --additional-config '{"enable_flashcomm1": true}'
```

```bash
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1
export HCCL_OP_EXPANSION_MODE="AIV"

vllm serve your_model_path \
  --served-model-name qwen3 \
  --trust-remote-code \
  --distributed-executor-backend mp \
  --tensor-parallel-size 8 \
  --max-model-len 5500 \
  --max-num-batched-tokens 40960 \
  --no-enable-prefix-caching \
  --async-scheduling \
  --quantization ascend \
  --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2,4,8,16,32,64,72,76,80,96,100,120,140,144,160,192,216,240,252,288,320,336,360,384,400,408,416,420,432,480,540,576,600]}' \
  --speculative-config '{"method": "eagle3", "model":"your_eagle3_model_path", "enforce_eager": true, "num_speculative_tokens": 3}' \
  --port <port> \
  --block-size 128 \
  --gpu-memory-utilization 0.9
```

--------------------------------

### Configure and Run Prefill Node 0

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/GLM5.html

Sets up environment variables for Ascend NPU communication and launches the vLLM server with specific KV transfer and model configurations.

```shell
nic_name="xxxx" # change to your own nic name
local_ip="xxxx" # change to your own ip

export HCCL_OP_EXPANSION_MODE="AIV"

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name

export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=256

export ASCEND_AGGREGATE_ENABLE=1
export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
export ASCEND_A3_ENABLE=1
# Timeout (in seconds) for automatically releasing the prefiller’s KV cache for a particular request.
export VLLM_MOONCAKE_ABORT_REQUEST_TIMEOUT=480

export ASCEND_RT_VISIBLE_DEVICES=$1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1

export VLLM_ASCEND_ENABLE_FUSED_MC2=1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/GLM5-w8a8 \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --speculative-config '{"num_speculative_tokens": 3, "method":"deepseek_mtp"}' \
    --profiler-config \
    '{"profiler": "torch",
    "torch_profiler_dir": "./vllm_profile",
    "torch_profiler_with_stack": false}' \
    --seed 1024 \
    --served-model-name glm-5 \
    --max-model-len 131072 \
    --additional-config '{"fuse_muls_add": true, "multistream_overlap_shared_expert": true, "recompute_scheduler_enable": true, "enable_dsa_cp": true}' \
    --max-num-batched-tokens 4096 \
    --trust-remote-code \
    --max-num-seqs 64 \
    --enable-chunked-prefill \
    --quantization ascend \
    --gpu-memory-utilization 0.95 \
    --enforce-eager \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm45 \
    --kv-transfer-config \
    '{"kv_connector": "MooncakeConnectorV1",
    "kv_role": "kv_producer",
    "kv_port": "30000",
    "kv_connector_extra_config": {
                "use_ascend_direct": true,
                "prefill": {
                        "dp_size": 2,
                        "tp_size": 16
                },
                "decode": {
                        "dp_size": 16,
                        "tp_size": 4
                }
        }
    }'
```

--------------------------------

### Install YuanRong TransferEngine

Source: https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/rfork.html

Install the required package via pip.

```bash
pip install openyuanrong-transfer-engine
```

--------------------------------

### Install Xlite

Source: https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/graph_mode.html?q=

Required installation command for enabling XliteGraph functionality.

```bash
pip install xlite
```

--------------------------------

### Start Load-Balance Proxy Server

Source: https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/external_dp.html

Launch the proxy server to distribute requests across the running vLLM DP instances.

```bash
# For example, we have already started two DP instances in single node:
# python launch_online_dp.py --dp-size 2 --tp-size 8 --dp-size-local 2 --dp-rank-start 0 --dp-address x.x.x.x --dp-rpc-port 12342
# By default, launch_online_dp.py will launch vLLM instances from starting port 9000,
# so the vLLM ports for DP0 and DP1 are 9000 and 9001 separately.
# Then you can start the load-balance proxy server by:
cd examples/external_online_dp
python dp_load_balance_proxy_server.py \
    --host 0.0.0.0 --port 8000 \
    --dp-hosts 127.0.0.1 127.0.0.1 \
    --dp-ports 9000 9001 \
```

--------------------------------

### Configure and Start Prefill-Decode Disaggregation

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/GLM5.2.html

Environment variables and server startup command for initializing the vLLM server with specific Ascend and KV transfer configurations.

```shell
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=500
export ASCEND_AGGREGATE_ENABLE=1
export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
export ASCEND_A3_ENABLE=1
export VLLM_VERSION=0.21.0
export TASK_QUEUE_ENABLE=1
export ASCEND_RT_VISIBLE_DEVICES=$1
export DYNAMIC_EPLB=1
export VLLM_ASCEND_ENABLE_FUSED_MC2=1
export VLLM_ASCEND_ENABLE_MLAPO=1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/GLM-5.2-w8a8 \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name glm-52 \
    --max-model-len 135000 \
    --max-num-batched-tokens 164 \
    --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
    --speculative-config '{"num_speculative_tokens": 5, "method":"deepseek_mtp"}' \
    --additional-config '{"enable_sparse_c8":false,"fuse_muls_add": true, "multistream_overlap_shared_expert": true, "recompute_scheduler_enable": true}' \
    --trust-remote-code \
    --max-num-seqs 48 \
    --gpu-memory-utilization 0.92 \
    --async-scheduling \
    --quantization ascend \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm45 \
    --kv-transfer-config \
    '{"kv_connector": "MooncakeConnectorV1",
    "kv_role": "kv_consumer",
    "kv_port": "30100",
    "engine_id": "1",
    "kv_connector_extra_config": {
                "use_ascend_direct": true,
                "prefill": {
                        "dp_size": 2,
                        "tp_size": 16
                },
                "decode": {
                        "dp_size": 8,
                        "tp_size": 4
                }
        }
    }'
```

```shell
nic_name="xxxx" # change to your own nic name local_ip="xxxx" # change to your own ip
export HCCL_OP_EXPANSION_MODE="AIV" export HCCL_IF_IP=\(local_ip export GLOO_SOCKET_IFNAME=\)nic_name export TP_SOCKET_IFNAME=\(nic_name export HCCL_SOCKET_IFNAME=\)nic_name export OMP_PROC_BIND=false export OMP_NUM_THREADS=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export HCCL_BUFFSIZE=500 export ASCEND_AGGREGATE_ENABLE=1 export ASCEND_TRANSPORT_PRINT=1 export ACL_OP_INIT_MODE=1 export ASCEND_A3_ENABLE=1 export TASK_QUEUE_ENABLE=1 export VLLM_VERSION=0.21.0 export ASCEND_RT_VISIBLE_DEVICES=\(1 export DYNAMIC_EPLB=1 export VLLM_ASCEND_ENABLE_FUSED_MC2=1 export VLLM_ASCEND_ENABLE_MLAPO=1 export LD_LIBRARY_PATH=\)LD_LIBRARY_PATH:/usr/local/lib
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/GLM-5.2-w8a8 \ --host 0.0.0.0 \ --port $2 \ --data-parallel-size $3 \ --data-parallel-rank $4 \ --data-parallel-address $5 \ --data-parallel-rpc-port $6 \ --tensor-parallel-size $7 \ --enable-expert-parallel \ --seed 1024 \ --served-model-name glm-52 \ --max-model-len 135000 \ --max-num-batched-tokens 164 \ --speculative-config '{"num_speculative_tokens": 5, "method":"deepseek_mtp"}' \ --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \ --additional-config '{"enable_sparse_c8":false,"fuse_muls_add": true, "multistream_overlap_shared_expert": true, "recompute_scheduler_enable": true}' \ --trust-remote-code \ --max-num-seqs 48 \ --gpu-memory-utilization 0.92 \ --async-scheduling \ --quantization ascend \ --enable-auto-tool-choice \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --kv-transfer-config \ '{"kv_connector": "MooncakeConnectorV1", "kv_role": "kv_consumer", "kv_port": "30100", "engine_id": "1", "kv_connector_extra_config": { "use_ascend_direct": true, "prefill": { "dp_size": 2, "tp_size": 16 }, "decode": { "dp_size": 8, "tp_size": 4 } } }'
```

--------------------------------

### Verify source installation

Source: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Gemma4.html

Check the installed version of vLLM Ascend.

```bash
python -c "import vllm_ascend; print(vllm_ascend.__version__)"
```

--------------------------------

### Install AISBench

Source: https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/contribution/multi_node_test.html

Clone and install AISBench on the primary node (LWS_WORKER_INDEX=0).

```bash
export AIS_BENCH_TAG="v3.1-20260330-master"
export AIS_BENCH_URL="https://github.com/AISBench/benchmark.git"
export BENCHMARK_HOME=/vllm-workspace/vllm-ascend/benchmark

git clone -b ${AIS_BENCH_TAG} --depth 1 ${AIS_BENCH_URL} $BENCHMARK_HOME
cd $BENCHMARK_HOME
pip install -e . -r requirements/api.txt -r requirements/extra.txt
```

--------------------------------

### Start the Proxy Server

Source: https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/kv_pool.html?q=

Execute this command to launch the load balance proxy server, replacing localhost with your actual IP address.

```bash
python vllm-ascend/examples/disaggregated_prefill_v1/load_balance_proxy_server_example.py \
    --host localhost \
    --prefiller-hosts localhost \
    --prefiller-ports 8100 \
    --decoder-hosts localhost \
    --decoder-ports 8200 \
```

--------------------------------

### Install AISBench

Source: https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/evaluation/using_ais_bench.html

Install the AISBench framework and its required dependencies from source.

```bash
git clone https://github.com/AISBench/benchmark.git
cd benchmark/
pip3 install -e ./ --use-pep517
```

```bash
pip3 install -r requirements/api.txt
pip3 install -r requirements/extra.txt
```
Tokens:
Raw