Qwen

https://qwen.readthedocs.io/en/latest/
Admin
Qwen is a series of large language and multimodal models by Alibaba Group capable of natural...
Tokens:79,591
Snippets:516
Trust Score:8.3
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Start OpenLLM server for qwen2.5:7b

Source: https://qwen.readthedocs.io/en/latest/deployment/openllm.html

Starts a model server for the specified Qwen2.5 model. By default, the server starts at http://localhost:3000/.

```bash
openllm serve qwen2.5:7b
```

--------------------------------

### Install Unsloth

Source: https://qwen.readthedocs.io/en/latest/training/unsloth.html

Installs the Unsloth library locally. Linux is recommended for local installation.

```bash
pip install unsloth

```

--------------------------------

### dstack server startup output with admin token

Source: https://qwen.readthedocs.io/en/latest/deployment/dstack.html

Example output shown when starting the dstack server, displaying the auto-generated admin token and server URL. Use the token for Authorization header in API requests.

```text
The admin token is "bbae0f28-d3dd-4820-bf61-8f4bb40815da"
The server is running at http://127.0.0.1:3000/

```

--------------------------------

### Download Qwen3-8B GGUF example

Source: https://qwen.readthedocs.io/en/latest/run_locally/llama.cpp.html

Example command to download the Qwen3-8B model in GGUF format quantized with the Q4_K_M scheme.

```bash
huggingface-cli download Qwen/Qwen3-8B-GGUF qwen3-8b-q4_k_m.gguf --local-dir .

```

--------------------------------

### Initialize dstack workspace

Source: https://qwen.readthedocs.io/en/latest/deployment/dstack.html

Initializes a dstack workspace in a new directory named dstack-qwen-deploy. Run after installing dstack and starting the dstack server.

```bash
mkdir dstack-qwen-deploy && cd dstack-qwen-deploy
dstack init

```

--------------------------------

### Start LM Studio's local API server with lms server start

Source: https://qwen.readthedocs.io/en/latest/run_locally/lmstudio.html

Starts LM Studio's local API server so you can use Qwen models programmatically via REST APIs. You must run LM Studio at least once before using the lms CLI.

```bash
lms server start

```

--------------------------------

### Install Qwen-Agent with optional extras

Source: https://qwen.readthedocs.io/en/latest/framework/qwen_agent.html

Install the stable version of Qwen-Agent from PyPI. The optional extras in double brackets enable GUI, RAG, Code Interpreter, and MCP support; omit them for a minimal installation.

```bash
pip install -U "qwen-agent[gui,rag,code_interpreter,mcp]"
# Or use `pip install -U qwen-agent` for the minimal requirements.
# The optional requirements, specified in double brackets, are:
#   [gui] for Gradio-based GUI support;
#   [rag] for RAG support;
#   [code_interpreter] for Code Interpreter support;
#   [mcp] for MCP support.

```

--------------------------------

### Install optimum>=1.20.0 for GPTQ support

Source: https://qwen.readthedocs.io/en/latest/quantization/gptq.html

Install the required optimum version to use official Qwen2.5 GPTQ models with transformers. Ensure compatible versions of transformers and auto_gptq are also installed.

```bash
pip install -U "optimum>=1.20.0"

```

--------------------------------

### Install qwen-agent library

Source: https://qwen.readthedocs.io/en/latest/framework/function_call.html

Install or upgrade the qwen-agent library, which provides the canonical implementation of function calling for Qwen3 and exposes it through an OpenAI-compatible API.

```bash
pip install -U qwen-agent
```

--------------------------------

### Install AutoGPTQ from source

Source: https://qwen.readthedocs.io/en/latest/quantization/gptq.html

Clone the AutoGPTQ repository and install the package in editable mode. This ensures you have the latest version.

```bash
git clone https://github.com/AutoGPTQ/AutoGPTQ
cd AutoGPTQ
pip install -e .

```

--------------------------------

### Install langchain and faiss-gpu

Source: https://qwen.readthedocs.io/en/latest/framework/Langchain.html

Install the required packages for the Langchain-based Q&A application. The first command installs langchain version 0.0.174, and the second installs faiss-gpu for vector storage.

```bash
pip install langchain==0.0.174
pip install faiss-gpu

```

--------------------------------

### Install AutoAWQ

Source: https://qwen.readthedocs.io/en/latest/quantization/awq.html

Install the required AutoAWQ library version.

```bash
pip install "autoawq<0.2.7"
```

--------------------------------

### Install ms-swift and dependencies

Source: https://qwen.readthedocs.io/en/latest/training/ms_swift.html

Install ms-swift and its dependencies. Includes optional packages for multi-GPU training (deepspeed), GPU memory savings (liger-kernel), and flash attention (flash-attn).

```bash
pip install ms-swift -U
# Install from source
pip install git+https://github.com/modelscope/ms-swift.git

pip install transformers -U

# Optional packages
pip install deepspeed # multi-GPU training
pip install liger-kernel # save GPU memory resources
pip install flash-attn --no-build-isolation
```

--------------------------------

### Start vLLM OpenAI-compatible server

Source: https://qwen.readthedocs.io/en/latest/framework/function_call.html

Starts an OpenAI-compatible API server with vLLM for Qwen3-8B, enabling automatic tool choice with the Hermes parser and the DeepSeek R1 reasoning parser. Requires vllm >= v0.8.5.

```bash
vllm serve Qwen/Qwen3-8B --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser deepseek_r1

```

--------------------------------

### Install math_verify and vllm for GRPO

Source: https://qwen.readthedocs.io/en/latest/training/ms_swift.html

Install the additional dependencies required for GRPO training with ms-swift: math_verify version 0.5.2 and vllm.

```bash
pip install "math_verify==0.5.2"
pip install vllm

```

--------------------------------

### Run Qwen2.5 with Ollama

Source: https://qwen.readthedocs.io/en/latest/run_locally/ollama.html

Runs the default Qwen2.5 model locally with Ollama. This is the simplest way to start using Qwen2.5 on macOS, Linux, or Windows.

```bash
ollama run qwen2.5

```

--------------------------------

### Install mlx-lm with pip

Source: https://qwen.readthedocs.io/en/latest/run_locally/mlx-lm.html

Install the mlx-lm package using pip, the easiest way to get started on macOS.

```bash
pip install mlx-lm
```

--------------------------------

### Launch Qwen2.5-72B-Instruct with SkyPilot

Source: https://qwen.readthedocs.io/en/latest/deployment/skypilot.html

Starts serving Qwen2.5-72B-Instruct on a single instance with any available GPU from the list in serve-72b.yaml, using a vLLM-powered OpenAI-compatible endpoint. Before launching, change Qwen/Qwen2-72B-Instruct to Qwen/Qwen2.5-72B-Instruct in the YAML file.

```bash
sky launch -c qwen serve-72b.yaml

```

--------------------------------

### Launch SGLang server with Qwen3-8B

Source: https://qwen.readthedocs.io/en/latest/deployment/sglang.html

Starts an OpenAI-compatible API server at http://localhost:30000. Use --host and --port to change the address. Downloads the model from Hugging Face Hub if the path is not a local directory.

```bash
python -m sglang.launch_server --model-path Qwen/Qwen3-8B
```

--------------------------------

### Serve Qwen3-Instruct-2507 with vLLM

Source: https://qwen.readthedocs.io/en/latest/getting_started/quickstart.html

Start an OpenAI-compatible API server for Qwen3-Instruct-2507 using vLLM. vllm>=0.9.0 is recommended. Adjust the context length according to available GPU memory.

```bash
vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507 --port 8000 --tensor-parallel-size 8 --max-model-len 262144

```

--------------------------------

### Launch SGLang server with Qwen25 tool call parser

Source: https://qwen.readthedocs.io/en/latest/deployment/sglang.html

Starts an SGLang server for Qwen/Qwen3-8B with the qwen25 tool call parser to parse tool calling content into structured messages. Refer to the Function Calling guide for more details.

```bash
python -m sglang.launch_server --model-path Qwen/Qwen3-8B --tool-call-parser qwen25
```

--------------------------------

### Serve Qwen3-Instruct-2507 with SGLang

Source: https://qwen.readthedocs.io/en/latest/getting_started/quickstart.html

Start an OpenAI-compatible API server for Qwen3-Instruct-2507 using SGLang. Requires sglang>=0.4.6.post1. Adjust the context length according to available GPU memory.

```bash
python -m sglang.launch_server --model-path Qwen/Qwen3-235B-A22B-Instruct-2507 --port 8000 --tp 8 --context-length 262144

```

--------------------------------

### Serve Qwen3-8B with vLLM

Source: https://qwen.readthedocs.io/en/latest/getting_started/quickstart.html

Start an OpenAI-compatible API server for Qwen3-8B using vLLM. vllm>=0.9.0 is recommended. The --enable-reasoning and --reasoning-parser qwen3 flags are required.

```bash
vllm serve Qwen/Qwen3-8B --port 8000 --enable-reasoning --reasoning-parser qwen3

```

--------------------------------

### Install OpenLLM with pip

Source: https://qwen.readthedocs.io/en/latest/deployment/openllm.html

Installs OpenLLM using pip.

```bash
pip install openllm
```

--------------------------------

### Install vLLM with pip

Source: https://qwen.readthedocs.io/en/latest/deployment/vllm.html

Installs vLLM with pip in a clean environment. The prebuilt vllm has strict dependencies on torch and its CUDA versions; check the official installation note for help.

```bash
pip install "vllm>=0.8.5"
```

--------------------------------

### Serve Qwen model with vLLM

Source: https://qwen.readthedocs.io/en/latest/deployment/vllm.html

Starts an OpenAI-compatible API server at http://localhost:8000 by default. If the model does not point to a valid local directory, it downloads the model files from the Hugging Face Hub.

```bash
vllm serve Qwen/Qwen3-8B
```

--------------------------------

### Check for C++ compiler and CMake

Source: https://qwen.readthedocs.io/en/latest/run_locally/llama.cpp.html

Run these commands in a terminal to verify that a C++ compiler and CMake are installed. If installed, the build configuration is printed; if errors are raised, install the tools first.

```bash
cc --version
```

```bash
cmake --version
```

--------------------------------

### Start TGI with EETQ quantization

Source: https://qwen.readthedocs.io/en/latest/deployment/tgi.html

Command to run TGI with EETQ quantization on the original model (e.g., Qwen/Qwen2.5-7B-Instruct). Uses --quantize eetq and shares a volume to avoid re-downloading weights.

```bash
model=Qwen/Qwen2.5-7B-Instruct
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run

docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --quantize eetq

```

--------------------------------

### Install mlx-lm with conda

Source: https://qwen.readthedocs.io/en/latest/run_locally/mlx-lm.html

Install the mlx-lm package using conda from the conda-forge channel.

```bash
conda install -c conda-forge mlx-lm
```

--------------------------------

### Run calibration with model.quantize

Source: https://qwen.readthedocs.io/en/latest/quantization/gptq.html

Configure logging and run the quantization calibration process on the prepared data. The cache_examples_on_gpu parameter is set to False.

```python
import logging

logging.basicConfig(
    format="%(asctime)s %(levelname)s [%(name)s] %(message)s", level=logging.INFO, datefmt="%Y-%m-%d %H:%M:%S"
)
model.quantize(data, cache_examples_on_gpu=False)

```

--------------------------------

### Install LlamaIndex packages

Source: https://qwen.readthedocs.io/en/latest/framework/LlamaIndex.html

Install the required LlamaIndex packages: llama-index, llama-index-llms-huggingface, and llama-index-readers-web.

```bash
pip install llama-index
pip install llama-index-llms-huggingface
pip install llama-index-readers-web

```

--------------------------------

### Run Qwen3 with llama-cli

Source: https://qwen.readthedocs.io/en/latest/run_locally/llama.cpp.html

Run Qwen3-8B with llama.cpp using llama-cli. The command loads the model from Hugging Face hub, enables the chat template with --jinja, colors output, offloads all layers to GPU, and sets sampling and context parameters. Use -m for a local path or -mu for a remote URL instead of -hf.

```bash
./llama-cli -hf Qwen/Qwen3-8B-GGUF:Q8_0 --jinja --color -ngl 99 -fa -sm row --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 -c 40960 -n 32768 --no-context-shift

```

--------------------------------

### Install ModelScope

Source: https://qwen.readthedocs.io/en/latest/training/unsloth.html

Installs the ModelScope library quietly to enable loading models directly from ModelScope.

```bash
pip install modelscope -qqq

```

--------------------------------

### Launch OpenAI-compatible API service with vLLM

Source: https://qwen.readthedocs.io/en/latest/quantization/gptq.html

Run this shell command to start an OpenAI-compatible API service using the Qwen2.5-7B-Instruct-GPTQ-Int4 model. The service will be available at http://localhost:8000.

```shell
vllm serve Qwen2.5-7B-Instruct-GPTQ-Int4

```

--------------------------------

### Install SGLang with pip

Source: https://qwen.readthedocs.io/en/latest/deployment/sglang.html

Installs SGLang with all dependencies in a clean environment. Requires Python and pip.

```bash
pip install "sglang[all]>=0.4.6.post1"
```

--------------------------------

### Install llama.cpp with Homebrew

Source: https://qwen.readthedocs.io/en/latest/run_locally/llama.cpp.html

Install llama-cli and llama-server pre-built binaries on macOS or Linux using Homebrew. Note that the installed binaries might not be built with optimal compile options for your hardware, which can lead to poor performance, and they don't support GPU on Linux systems.

```bash
brew install llama.cpp

```

--------------------------------

### Output of openllm model list --tag qwen2.5

Source: https://qwen.readthedocs.io/en/latest/deployment/openllm.html

Example output from `openllm model list --tag qwen2.5`, showing model versions, required GPU RAM, and supported platforms.

```text
model    version                repo     required GPU RAM    platforms
-------  ---------------------  -------  ------------------  -----------
qwen2.5  qwen2.5:0.5b           default  12G                 linux
         qwen2.5:1.5b           default  12G                 linux
         qwen2.5:3b             default  12G                 linux
         qwen2.5:7b             default  24G                 linux
         qwen2.5:14b            default  80G                 linux
         qwen2.5:14b-ggml-q4    default                      macos
         qwen2.5:14b-ggml-q8    default                      macos
         qwen2.5:32b            default  80G                 linux
         qwen2.5:32b-ggml-fp16  default                      macos
         qwen2.5:72b            default  80Gx2               linux
         qwen2.5:72b-ggml-q4    default                      macos
```

--------------------------------

### Train Qwen3-1.7B with verl using FSDP and vLLM (GRPO)

Source: https://qwen.readthedocs.io/en/latest/training/verl.html

Starts GRPO training of Qwen3-1.7B on GSM8K using FSDP for the actor and vLLM for rollout. Requires a single 80GB GPU and more than 64GB memory. Configures hyperparameters, KL loss, and logging.

```bash
python3 -m verl.trainer.main_ppo \
    algorithm.adv_estimator=grpo \
    data.train_files=$HOME/data/gsm8k/train.parquet \
    data.val_files=$HOME/data/gsm8k/test.parquet \
    data.train_batch_size=1024 \
    data.max_prompt_length=512 \
    data.max_response_length=1024 \
    data.filter_overlong_prompts=True \
    data.truncation='error' \
    actor_rollout_ref.model.path=Qwen/Qwen3-1.7B \
    actor_rollout_ref.actor.optim.lr=1e-6 \
    actor_rollout_ref.model.use_remove_padding=True \
    actor_rollout_ref.actor.ppo_mini_batch_size=80 \
    actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=20 \
    actor_rollout_ref.actor.use_kl_loss=True \
    actor_rollout_ref.actor.kl_loss_coef=0.001 \
    actor_rollout_ref.actor.kl_loss_type=low_var_kl \
    actor_rollout_ref.actor.entropy_coeff=0 \
    actor_rollout_ref.model.enable_gradient_checkpointing=True \
    actor_rollout_ref.actor.fsdp_config.param_offload=False \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \
    actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=20 \
    actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
    actor_rollout_ref.rollout.name=vllm \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.6 \
    actor_rollout_ref.rollout.n=3 \
    actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=20 \
    actor_rollout_ref.ref.fsdp_config.param_offload=True \
    algorithm.use_kl_in_reward=False \
    trainer.critic_warmup=0 \
    trainer.logger=['console'] \
    trainer.project_name='verl_grpo_example_gsm8k' \
    trainer.experiment_name='qwen3_1_7b_function_rm' \
    trainer.n_gpus_per_node=1 \
    trainer.nnodes=1 \
    trainer.save_freq=-1 \
    trainer.test_freq=5 \
    trainer.total_epochs=15 $@
```

--------------------------------

### Install build tools on Ubuntu

Source: https://qwen.readthedocs.io/en/latest/run_locally/llama.cpp.html

Installs the essential build packages (gcc and cmake) on Ubuntu. For other Linux distributions, the command may vary.

```bash
sudo apt install build-essential
```

--------------------------------

### Install build tools on macOS

Source: https://qwen.readthedocs.io/en/latest/run_locally/llama.cpp.html

Installs the required C++ compiler and build tools on macOS using the Xcode Command Line Tools.

```bash
xcode-select --install
```

--------------------------------

### Launch SGLang server with Qwen3 reasoning parser

Source: https://qwen.readthedocs.io/en/latest/deployment/sglang.html

Starts an SGLang server for Qwen/Qwen3-8B with the qwen3 reasoning parser. The response will include a reasoning_content field. This feature is not OpenAI API compatible and may conflict with enable_thinking=False.

```bash
python -m sglang.launch_server --model-path Qwen/Qwen3-8B --reasoning-parser qwen3
```

--------------------------------

### Install SkyPilot with pip

Source: https://qwen.readthedocs.io/en/latest/deployment/skypilot.html

Installs SkyPilot nightly with AWS and GCP cloud support. Replace the cloud list in the extras with the clouds you have access to.

```bash
# You can use any of the following clouds that you have access to:
# aws, gcp, azure, oci, lamabda, runpod, fluidstack, paperspace,
# cudo, ibm, scp, vsphere, kubernetes
pip install "skypilot-nightly[aws,gcp]"
```
Tokens:
Raw