SGLang

https://docs.sglang.io/
Admin
SGLang is a high-performance serving framework for large language models and multimodal models,...
Tokens:1,592,245
Snippets:9,053
Trust Score:4.3
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Setup for Qwen2.5-VL Example

Source: https://docs.sglang.io/docs/advanced_features/vlm_query

Sets up the environment for running the Qwen2.5-VL example. It applies nest_asyncio, imports a documentation patch, and defines the model path, chat template, and example image URL used by subsequent snippets.

```python
import nest_asyncio

nest_asyncio.apply()

import sglang.test.doc_patch  # noqa: F401

model_path = "Qwen/Qwen2.5-VL-3B-Instruct"
chat_template = "qwen2-vl"
example_image_url = "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"

```

--------------------------------

### Start prefill server on Intel XPU

Source: https://docs.sglang.io/docs/hardware-platforms/xpu

Starts the prefill server on GPU 0 with the NIXL KV-transfer backend. Requires `pip install nixl sglang-router` and the `UCX_POSIX_USE_PROC_LINK=n` environment variable to avoid UCX shared-memory transport issues on Intel XPU.

```bash
ZE_AFFINITY_MASK=0 UCX_POSIX_USE_PROC_LINK=n python -m sglang.launch_server \
    --model-path Qwen/Qwen3-0.6B --trust-remote-code --device xpu \
    --disaggregation-mode prefill --disaggregation-transfer-backend nixl \
    --disaggregation-bootstrap-port 12335 --host 0.0.0.0 --port 30000

```

--------------------------------

### Navigate to examples/monitoring

Source: https://docs.sglang.io/docs/references/production_metrics

Moves into the monitoring example directory, which contains the Prometheus and Grafana stack referenced by the setup guide. This step precedes starting the monitoring stack with docker compose up -d.

```bash
cd examples/monitoring

```

--------------------------------

### Start SGLang server for LingBot Video MoE

Source: https://docs.sglang.io/cookbook/diffusion/LingBot-Video/LingBot-Video-MoE

Starts the SGLang server with the Hugging Face model ID `robbyant/lingbot-video-moe-30b-a3b` on port 30010. Run inside a Docker container or your Python environment after following the SGLang Diffusion installation guide.

```bash
sglang serve \
  --model-path robbyant/lingbot-video-moe-30b-a3b \
  --port 30010

```

--------------------------------

### Install Go bindings

Source: https://docs.sglang.io/docs/advanced_features/sgl_model_gateway

Install the Go bindings. First build the FFI library using make, then fetch the Go SDK. Requires Go 1.24+ and a Rust toolchain.

```bash
# Build the FFI library first
cd sgl-model-gateway/bindings/golang
make build && make lib

# Then use in your Go project
go get github.com/sgl-project/sgl-go-sdk

```

--------------------------------

### Pull SGLang Docker images for Ascend NPU

Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K3

Pull Docker images for Ascend NPU platforms (A3 and 950PR/DT Series). Requires host and platform setup per the NPU installation guide.

```bash
docker pull quay.io/ascend/sglang:main-cann9.0.0-a3   # Ascend A3 Series
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-950-B070   # Ascend 950PR/DT Series

```

--------------------------------

### Write setup.py

Source: https://docs.sglang.io/docs/supported-models/support_new_models

Write this setup.py file in the project root to make the package installable.

```python
# sglang_custom_project/setup.py

from setuptools import setup, find_packages
setup(
    name="sglang-custom-plugins",
    version="0.1",
    packages=find_packages(),
)
```

--------------------------------

### Install Nunchaku runtime dependency

Source: https://docs.sglang.io/docs/sglang-diffusion/quantization

Install the Nunchaku runtime dependency before using SGLang with Nunchaku checkpoints. For platform-specific installation methods and troubleshooting, see the Nunchaku installation guide.

```bash
pip install nunchaku

```

--------------------------------

### Quick start: launch SGLang server and benchmark with --backend sglang

Source: https://docs.sglang.io/docs/developer_guide/bench_serving

Start an SGLang server and run a basic benchmark against its /generate endpoint. The server command must be run first, then the bench_serving command targets it via --host and --port.

```bash
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct

```

```bash
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 --port 30000 \
  --num-prompts 1000 \
  --model meta-llama/Llama-3.1-8B-Instruct

```

--------------------------------

### Install and Run General Plugin

Source: https://docs.sglang.io/docs/hardware-platforms/plugin

Install the plugin in editable mode and run the server. Look for the printed message in the logs.

```bash
pip install -e my_general_plugin/
sglang serve --model-path <model> [options]
# Look for "Scheduler initialized!" in logs

```

--------------------------------

### Verify Docker installation

Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/getting-started/quick_start

Check that Docker is installed and the daemon is running before proceeding with the container setup.

```bash
docker --version && docker info

```

--------------------------------

### Launch multi-modal server with Qwen2.5-VL-7B-Instruct

Source: https://docs.sglang.io/docs/references/frontend/frontend_tutorial

Launch a server for multi-modal generation using the Qwen2.5-VL-7B-Instruct model. It starts the server, waits for it to be ready, and prints the server URL.

```python
server_process, port = launch_server_cmd(
    "python -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct --host 0.0.0.0 --log-level warning"
)

wait_for_server(f"http://localhost:{port}", process=server_process)
print(f"Server started on http://localhost:{port}")

```

--------------------------------

### Install OpenAI Python client with uv

Source: https://docs.sglang.io/docs/get-started/examples

Install the OpenAI Python client in your activated environment. Required before running the examples on this page.

```bash
uv pip install openai

```

--------------------------------

### Pip: Install SGLang nightly version

Source: https://docs.sglang.io/cookbook/autoregressive/Xiaomi/MiMo-V2-Flash

Installs the required SGLang version via pip. Starts a nightly container, uninstalls any existing SGLang, then installs the specific pre-release version from the provided index.

```bash
# On a machine with SGLang dependencies installed or inside a SGLang nightly container
# Start an SGLang nightly container
docker run -it --gpus all \
  --shm-size=32g \
  --ipc=host \
  --network=host \
  lmsysorg/sglang:latest bash

# If you already have SGLang installed, uninstall the current SGLang version
pip uninstall sglang -y

# Install the PyPI Package
pip install sglang==0.5.6.post2.dev8005+pr.15207.g39d5bd57a \
  --extra-index-url https://sgl-project.github.io/whl/pr/

```

--------------------------------

### Run Go binding examples

Source: https://docs.sglang.io/docs/advanced_features/sgl_model_gateway

Run the Go binding examples: simple non-streaming chat completion, streaming chat completion with SSE, and a full OpenAI-compatible HTTP server.

```bash
# Run examples
cd sgl-model-gateway/bindings/golang/examples/simple && ./run.sh
cd sgl-model-gateway/bindings/golang/examples/streaming && ./run.sh
cd sgl-model-gateway/bindings/golang/examples/oai_server && ./run.sh

```

--------------------------------

### Deploy Qwen3-Next-80B-A3B-Instruct with sglang.launch_server

Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct

Deploys Qwen3-Next-80B-A3B-Instruct on Ascend A3 with W8A8 INT8 quantization, 1 card, PD Mixed mode, and 20ms TPOT. Requires updating MODEL_PATH, DRAFT_MODEL_PATH, and network interface names before running. Sets environment variables for HCCL, ZBCCL, and SGLANG performance tuning.

```bash
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_HCCL_BUFFSIZE=2000
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048
export DEEPEP_NORMAL_LONG_SEQ_ROUND=10
export FORCE_DRAFT_MODEL_NON_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=400
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
export SGLANG_NPU_USE_MULTI_STREAM=0
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_WARMUP_TIMEOUT=3600
export STREAMS_PER_DEVICE=32
export TASK_QUEUE_ENABLE=1
export ZBCCL_BOOTSTRAP_URL=tcp://127.0.0.1:24669
export ZBCCL_ENABLE_GRAPH=1
export ZBCCL_LOCAL_MEM_SIZE=60416
export ZBCCL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --page-size 128 \
    --tp-size 2 \
    --watchdog-timeout 9000 \
    --mem-fraction-static 0.85 \
    --disable-radix-cache \
    --max-prefill-tokens 28672 \
    --context-length 26384 \
    --max-total-tokens 122304 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --chunked-prefill-size -1 \
    --max-running-requests 2 \
    --cuda-graph-bs-decode 2 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder

```

--------------------------------

### Start server for benchmark

Source: https://docs.sglang.io/cookbook/diffusion/SenseNova/SenseNova-U1.5-8B-MoT

Starts the SGLang server without dynamic batching for the benchmark run. Uses the same model and port as the dynamic batching example.

```bash
sglang serve \
  --model-path sensenova/SenseNova-U1.5-8B-MoT \
  --port 30000
```

--------------------------------

### Hy3 instant mode example (reasoning_effort=no_think)

Source: https://docs.sglang.io/cookbook/autoregressive/Tencent/Hy3

Python example using reasoning_effort=no_think to skip thinking and get an instant response. Prints only the content field.

```python
response = client.chat.completions.create(
    model="tencent/Hy3",
    messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}],
    reasoning_effort="no_think",
    max_tokens=256,
)

print("Content:", response.choices[0].message.content)

```

```text
Content: Relativity is Einstein's theory that space, time, mass, and gravity are interconnected and relative, not fixed, fundamentally changing our understanding of the universe.

```

--------------------------------

### Launch SGLang server with Qwen2.5-VL-7B-Instruct

Source: https://docs.sglang.io/docs/basic_usage/openai_api_vision

Starts the SGLang server with the Qwen2.5-VL-7B-Instruct model and waits for it to initialize. Uses launch_server_cmd and wait_for_server from sglang.test and sglang.utils.

```python
from sglang.test.doc_patch import launch_server_cmd
from sglang.utils import wait_for_server, print_highlight, terminate_process

example_image_url = "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"
logo_image_url = (
    "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"
)

vision_process, port = launch_server_cmd("""
python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct --log-level warning
""")

wait_for_server(f"http://localhost:{port}", process=vision_process)

```

--------------------------------

### Launch SGLang server with Qwen2.5-7B-Instruct

Source: https://docs.sglang.io/docs/references/frontend/frontend_tutorial

Launches a local SGLang server with the Qwen2.5-7B-Instruct model and waits for it to initialize. The server process and port are returned for later use.

```python
from sglang import assistant_begin, assistant_end
from sglang import assistant, function, gen, system, user
from sglang import image
from sglang import RuntimeEndpoint
from sglang.lang.api import set_default_backend
from sglang.srt.utils import load_image
from sglang.test.doc_patch import launch_server_cmd
from sglang.utils import print_highlight, terminate_process, wait_for_server

server_process, port = launch_server_cmd(
    "python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0 --log-level warning"
)

wait_for_server(f"http://localhost:{port}", process=server_process)
print(f"Server started on http://localhost:{port}")
```

--------------------------------

### Start SRT enhancer server

Source: https://docs.sglang.io/docs/sglang-diffusion/models_with_pe

Starts a text-only SRT server on GPU 0 with the Qwen model on port 31000. Requires SGLang installed and a CUDA GPU.

```bash
CUDA_VISIBLE_DEVICES=0 sglang serve \
  --model-path Qwen/Qwen3-4B-Instruct-2507 \
  --port 31000

```

--------------------------------

### Launch Qwen3-VL-235B-A22B-Instruct with full multimodal optimizations

Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3-VL

Example launch command with full multimodal optimizations: enables CUDA IPC transport, disables VLM cache, sets attention backends to fa3, and tunes memory and request limits. Includes metrics and cache reporting.

```bash
SGLANG_USE_CUDA_IPC_TRANSPORT=1 \
SGLANG_VLM_CACHE_SIZE_MB=0 \
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-235B-A22B-Instruct \
  --host 0.0.0.0 \
  --port 30000 \
  --trust-remote-code \
  --tp-size 8 \
  --enable-cache-report \
  --log-level info \
  --max-running-requests 64 \
  --mem-fraction-static 0.65 \
  --chunked-prefill-size 8192 \
  --attention-backend fa3 \
  --mm-attention-backend fa3 \
  --enable-metrics
```

--------------------------------

### Install uv and create virtual environment

Source: https://docs.sglang.io/docs/hardware-platforms/cpu_server

Installs the uv package manager, creates a Python 3.12 virtual environment, and activates it. The example uses /opt as the environment folder.

```bash
# Taking '/opt' as the example uv env folder, feel free to change it as needed
cd /opt
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv venv --python 3.12
source .venv/bin/activate

```

--------------------------------

### Install system dependencies and setuptools

Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/getting-started/installation

Installs system dependencies libGL and libglib2.0-0, and ensures setuptools contains the pkg_resources module by installing a version below 80.

```bash
# libGL
apt update
apt install libgl1 libglib2.0-0

# ensure setuptools contains pkg_resources module
pip install "setuptools<80"

```

--------------------------------

### Get model info with requests.get

Source: https://docs.sglang.io/docs/basic_usage/native_api

Sends a GET request to /get_model_info and asserts the response fields. The example expects the model path 'qwen/qwen2.5-0.5b-instruct' and that preferred_sampling_params is None because it was not configured in server args.

```python
url = f"http://localhost:{port}/get_model_info"

response = requests.get(url)
response_json = response.json()
print_highlight(response_json)
assert response_json["model_path"] == "qwen/qwen2.5-0.5b-instruct"
assert response_json["is_generation"] is True
assert response_json["tokenizer_path"] == "qwen/qwen2.5-0.5b-instruct"
assert response_json["preferred_sampling_params"] is None
assert response_json.keys() == {
    "model_path",
    "is_generation",
    "tokenizer_path",
    "preferred_sampling_params",
    "weight_version",
    "has_image_understanding",
    "has_audio_understanding",
    "model_type",
    "architectures",
    "embedding",
}
```

--------------------------------

### Start language-only server for GLM-5.3-Flash

Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.3-Flash

Starts the language-only TP4 process on port 30000 for encoder disaggregation. Must be started after the encoder server. Keep --mem-fraction-static 0.78 so the encoder retains room for vision workspaces. Install torchcodec for video support.

```bash
sglang serve \
  --model-path zai-org/GLM-5.3-Flash \
  --tp-size 4 \
  --attention-backend dsa \
  --dsa-prefill-backend tilelang \
  --dsa-decode-backend tilelang \
  --linear-attn-backend triton \
  --kv-cache-dtype bfloat16 \
  --quantization fp8 \
  --moe-runner-backend flashinfer_trtllm \
  --max-running-requests 64 \
  --chunked-prefill-size 8192 \
  --max-prefill-tokens 8192 \
  --disable-prefill-cuda-graph \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --language-only \
  --encoder-urls http://localhost:30001 \
  --mem-fraction-static 0.78 \
  --host 0.0.0.0 \
  --port 30000

```

--------------------------------

### Deploy Qwen3.6-27B on Ascend with sglang.launch_server

Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b

Sets up the environment and launches the SGLang server for Qwen3.6-27B on Ascend A3. Requires updating MODEL_PATH and network interface variables. Uses PD mixed mode, BF16 quantization, and NEXTN speculative decoding.

```bash
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_VIT_ENABLE_CUDA_GRAPH=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 82688 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 38 \
    --max-mamba-cache-size 38 \
    --mem-fraction-static 0.7 \
    --cuda-graph-bs-decode 1 2 4 8 10 12 16 20 24 28 30 32 35 38 \
    --enable-prefill-delayer \
    --prefill-delayer-queue-min-ratio 0.45 \
    --prefill-delayer-max-delay-ms 5500 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder

```

--------------------------------

### Install SGLang from main with uv

Source: https://docs.sglang.io/cookbook/autoregressive/Poolside/Laguna-M.1

Install SGLang from main to get Laguna-M.1 support (PRs #28400, #28604, and #28649 for FP8) since it is not yet in a tagged release. Creates a Python 3.12 virtual environment and installs the serving runtime from the base dependencies.

```bash
pip install -U uv
uv venv --python 3.12 && source .venv/bin/activate

# Laguna-M.1 support is on SGLang main (PRs #28400 + #28604, plus #28649 for FP8), not yet in a
# tagged release — install from main. The serving runtime is in the base dependencies, no extra needed:
git clone https://github.com/sgl-project/sglang.git
cd sglang
uv pip install --prerelease=allow -e python

```

--------------------------------

### Deploy Qwen3-Next-80B-A3B-Instruct with sglang.launch_server on Ascend

Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct

Deploys the Qwen3-Next-80B-A3B-Instruct model on Ascend A3 Series with W8A8 INT8 quantization and NEXTN speculative decoding. Requires setting MODEL_PATH, DRAFT_MODEL_PATH, and network interface variables before running. Uses sglang.launch_server with --device npu, --quantization modelslim, and a 2P PD-mixed configuration.

```bash
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_HCCL_BUFFSIZE=2000
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048
export DEEPEP_NORMAL_LONG_SEQ_ROUND=10
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export FORCE_DRAFT_MODEL_NON_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=400
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
export SGLANG_NPU_USE_MULTI_STREAM=0
export SGLANG_WARMUP_TIMEOUT=3600
export STREAMS_PER_DEVICE=32
export TASK_QUEUE_ENABLE=1
export ZBCCL_BOOTSTRAP_URL=tcp://127.0.0.1:24669
export ZBCCL_ENABLE_GRAPH=1
export ZBCCL_LOCAL_MEM_SIZE=60416
export ZBCCL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --page-size 128 \
    --tp-size 4 \
    --watchdog-timeout 9000 \
    --mem-fraction-static 0.85 \
    --disable-radix-cache \
    --max-prefill-tokens 28672 \
    --context-length 81920 \
    --max-total-tokens 122304 \
    --attn-dp-size 2 \
    --enable-dp-lm-head \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --chunked-prefill-size -1 \
    --max-running-requests 16 \
    --cuda-graph-bs-decode 2 4 8 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```
Tokens:
Raw