Qwen

https://qwen.readthedocs.io
Admin
Qwen is a series of large language and multimodal models developed by Alibaba Group. It is capable...
Tokens:126,904
Snippets:881
Trust Score:8.3
Update:1 week ago
Show doc for...
Tokens:
Raw
### Install Unsloth

Source: https://qwen.readthedocs.io/en/latest/_sources/training/unsloth.md.txt

Install Unsloth locally. Linux is recommended.

```bash
pip install unsloth
```

--------------------------------

### Start OpenLLM server with Qwen2.5 model

Source: https://qwen.readthedocs.io/en/latest/_sources/deployment/openllm.rst.txt

Start a server with a Qwen2.5 model using `openllm serve qwen2.5:7b`. By default, the server starts at http://localhost:3000/.

```bash
openllm serve qwen2.5:7b
```

--------------------------------

### Preparation Code for Function Calling Example

Source: https://qwen.readthedocs.io/en/latest/framework/function_call.html

Defines two temperature-related tools (get_current_temperature and get_temperature_date), a helper to retrieve them by name, a TOOLS list in OpenAI function format, and an initial MESSAGES list. This setup is used for the function calling inference examples with Qwen-Agent and vLLM.

```python
import json

def get_current_temperature(location: str, unit: str = "celsius"):
    """Get current temperature at a location.

    Args:
        location: The location to get the temperature for, in the format "City, State, Country".
        unit: The unit to return the temperature in. Defaults to "celsius". (choices: ["celsius", "fahrenheit"])

    Returns:
        the temperature, the location, and the unit in a dict
    """
    return {
        "temperature": 26.1,
        "location": location,
        "unit": unit,
    }


def get_temperature_date(location: str, date: str, unit: str = "celsius"):
    """Get temperature at a location and date.

    Args:
        location: The location to get the temperature for, in the format "City, State, Country".
        date: The date to get the temperature for, in the format "Year-Month-Day".
        unit: The unit to return the temperature in. Defaults to "celsius". (choices: ["celsius", "fahrenheit"])

    Returns:
        the temperature, the location, the date and the unit in a dict
    """
    return {
        "temperature": 25.9,
        "location": location,
        "date": date,
        "unit": unit,
    }


def get_function_by_name(name):
    if name == "get_current_temperature":
        return get_current_temperature
    if name == "get_temperature_date":
        return get_temperature_date

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "get_current_temperature",
            "description": "Get current temperature at a location.",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": 'The location to get the temperature for, in the format "City, State, Country".',
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": 'The unit to return the temperature in. Defaults to "celsius".',
                    },
                },
                "required": ["location"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "get_temperature_date",
            "description": "Get temperature at a location and date.",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": 'The location to get the temperature for, in the format "City, State, Country".',
                    },
                    "date": {
                        "type": "string",
                        "description": 'The date to get the temperature for, in the format "Year-Month-Day".',
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": 'The unit to return the temperature in. Defaults to "celsius".',
                    },
                },
                "required": ["location", "date"],
            },
        },
    },
]
MESSAGES = [
    {"role": "user",  "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."},
]

```

--------------------------------

### Start OpenAI-compatible API service with vLLM

Source: https://qwen.readthedocs.io/en/latest/_sources/quantization/awq.md.txt

Run this shell command to launch an OpenAI-compatible API service using the AWQ model Qwen/Qwen2.5-7B-Instruct-AWQ. Requires vLLM installed and vllm>=0.6.1 recommended for performance.

```bash
vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ
```

--------------------------------

### Install AutoGPTQ from source

Source: https://qwen.readthedocs.io/en/latest/_sources/quantization/gptq.md.txt

Clone the AutoGPTQ repository and install the package in editable mode to get the latest version.

```bash
git clone https://github.com/AutoGPTQ/AutoGPTQ
cd AutoGPTQ
pip install -e .
```

--------------------------------

### Install ms-swift and dependencies

Source: https://qwen.readthedocs.io/en/latest/_sources/training/ms_swift.md.txt

Install ms-swift, transformers, and optional packages for multi-GPU training, memory savings, and flash attention. Use -U to upgrade, and install from source via git if needed.

```shell
pip install ms-swift -U
# Install from source
pip install git+https://github.com/modelscope/ms-swift.git

pip install transformers -U

# Optional packages
pip install deepspeed # multi-GPU training
pip install liger-kernel # save GPU memory resources
pip install flash-attn --no-build-isolation
```

--------------------------------

### Run Qwen2.5 with Ollama

Source: https://qwen.readthedocs.io/en/latest/_sources/run_locally/ollama.md.txt

Starts a chat with the default Qwen2.5 model using Ollama. Requires Ollama installed and the model pulled.

```bash
ollama run qwen2.5
```

--------------------------------

### Launch SGLang server with tool-call parser

Source: https://qwen.readthedocs.io/en/latest/deployment/sglang.html

Starts an SGLang server for Qwen3-8B with the qwen25 tool-call parser to structure tool calling content. See the Function Calling guide for details.

```bash
python -m sglang.launch_server --model-path Qwen/Qwen3-8B --tool-call-parser qwen25

```

--------------------------------

### Verify OpenLLM installation and show help

Source: https://qwen.readthedocs.io/en/latest/deployment/openllm.html

Display OpenLLM help information to verify the installation.

```bash
openllm --help

```

--------------------------------

### Install Qwen-Agent with optional dependencies

Source: https://qwen.readthedocs.io/en/latest/_sources/framework/qwen_agent.rst.txt

Install the stable version from PyPI. The command includes optional requirements for GUI, RAG, code interpreter, and MCP support. Use `pip install -U qwen-agent` for minimal requirements.

```bash
pip install -U "qwen-agent[gui,rag,code_interpreter,mcp]"
# Or use `pip install -U qwen-agent` for the minimal requirements.
# The optional requirements, specified in double brackets, are:
#   [gui] for Gradio-based GUI support;
#   [rag] for RAG support;
#   [code_interpreter] for Code Interpreter support;
#   [mcp] for MCP support.
```

--------------------------------

### Install Qwen-Agent

Source: https://qwen.readthedocs.io/en/latest/_sources/framework/function_call.md.txt

Installs the latest version of the Qwen-Agent library using pip.

```bash
pip install -U qwen-agent
```

--------------------------------

### Install mlx-lm with pip

Source: https://qwen.readthedocs.io/en/latest/_sources/run_locally/mlx-lm.md.txt

Install the mlx-lm package using pip. This is the easiest way to get started.

```bash
pip install mlx-lm
```

--------------------------------

### Launch Qwen2.5-72B-Instruct service with sky launch

Source: https://qwen.readthedocs.io/en/latest/_sources/deployment/skypilot.rst.txt

Start serving Qwen2.5-72B-Instruct on a single instance using the serve-72b.yaml file. Before launching, change Qwen/Qwen2-72B-Instruct to Qwen/Qwen2.5-72B-Instruct in the YAML file.

```bash
sky launch -c qwen serve-72b.yaml
```

--------------------------------

### Start RL training with FSDP and vLLM

Source: https://qwen.readthedocs.io/en/latest/_sources/training/verl.md.txt

Launches GRPO training for Qwen3-1.7B using FSDP as the training framework and vLLM as the inference engine. Requires a single 80GB GPU and more than 64GB of memory.

```bash
python3 -m verl.trainer.main_ppo \
    algorithm.adv_estimator=grpo \
    data.train_files=$HOME/data/gsm8k/train.parquet \
    data.val_files=$HOME/data/gsm8k/test.parquet \
    data.train_batch_size=1024 \
    data.max_prompt_length=512 \
    data.max_response_length=1024 \
    data.filter_overlong_prompts=True \
    data.truncation='error' \
    actor_rollout_ref.model.path=Qwen/Qwen3-1.7B \
    actor_rollout_ref.actor.optim.lr=1e-6 \
    actor_rollout_ref.model.use_remove_padding=True \
    actor_rollout_ref.actor.ppo_mini_batch_size=80 \
    actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=20 \
    actor_rollout_ref.actor.use_kl_loss=True \
    actor_rollout_ref.actor.kl_loss_coef=0.001 \
    actor_rollout_ref.actor.kl_loss_type=low_var_kl \
    actor_rollout_ref.actor.entropy_coeff=0 \
    actor_rollout_ref.model.enable_gradient_checkpointing=True \
    actor_rollout_ref.actor.fsdp_config.param_offload=False \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \
    actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=20 \
    actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
    actor_rollout_ref.rollout.name=vllm \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.6 \
    actor_rollout_ref.rollout.n=3 \
    actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=20 \
    actor_rollout_ref.ref.fsdp_config.param_offload=True \
    algorithm.use_kl_in_reward=False \
    trainer.critic_warmup=0 \
    trainer.logger=['console'] \
    trainer.project_name='verl_grpo_example_gsm8k' \
    trainer.experiment_name='qwen3_1_7b_function_rm' \
    trainer.n_gpus_per_node=1 \
    trainer.nnodes=1 \
    trainer.save_freq=-1 \
    trainer.test_freq=5 \
    trainer.total_epochs=15 $@
```

--------------------------------

### Example tool call results in no_think mode

Source: https://qwen.readthedocs.io/en/latest/_sources/framework/function_call.md.txt

Shows the assistant messages with function_call objects for two tool calls: getting current temperature and getting temperature for a specific date.

```python
[
    {"role": "assistant", "content": "", "function_call": {"name": "get_current_temperature", "arguments": "{\"location\": \"San Francisco, California, United States\", \"unit\": \"celsius\"}"}},
    {"role": "assistant", "content": "", "function_call": {"name": "get_temperature_date", "arguments": "{\"location\": \"San Francisco, California, United States\", \"date\": \"2024-10-01\", \"unit\": \"celsius\"}"}},
]
```

--------------------------------

### Start LM Studio's local API server with lms CLI

Source: https://qwen.readthedocs.io/en/latest/_sources/run_locally/lmstudio.md.txt

Run lms server start to enable LM Studio's local API server, allowing programmatic access to Qwen models via REST APIs. LM Studio must be run at least once before using lms.

```bash
lms server start
```

--------------------------------

### Initialize dstack workspace

Source: https://qwen.readthedocs.io/en/latest/_sources/deployment/dstack.rst.txt

Creates a new directory and initializes the dstack workspace. Run after installing dstack and starting the server.

```bash
mkdir dstack-qwen-deploy && cd dstack-qwen-deploy
dstack init
```

--------------------------------

### Deploy Qwen2.5 with TGI using Docker

Source: https://qwen.readthedocs.io/en/latest/_sources/deployment/tgi.rst.txt

Run this command to start a TGI server for a Qwen2.5 model. Replace 'model' with the model ID and 'volume' with your local data directory. The volume is shared to avoid re-downloading weights.

```bash
model=Qwen/Qwen2.5-7B-Instruct
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run

docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model
```

--------------------------------

### Start vLLM server with Qwen3 and tool call parser

Source: https://qwen.readthedocs.io/en/latest/_sources/framework/function_call.md.txt

Command to start an OpenAI-compatible API server with vLLM for Qwen3, enabling auto tool choice with the Hermes parser and the deepseek_r1 reasoning parser.

```bash
vllm serve Qwen/Qwen3-8B --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser deepseek_r1
```

--------------------------------

### Run TGI with EETQ quantization

Source: https://qwen.readthedocs.io/en/latest/_sources/deployment/tgi.rst.txt

Starts a TGI Docker container with data-agnostic EETQ quantization. Pass the original model with --quantize eetq. The example sets model and volume variables first.

```bash
model=Qwen/Qwen2.5-7B-Instruct
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run

docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --quantize eetq
```

--------------------------------

### Start Qwen3-Instruct-2507 API server with vLLM

Source: https://qwen.readthedocs.io/en/latest/_sources/getting_started/quickstart.md.txt

Launches an OpenAI-compatible API server for Qwen3-235B-A22B-Instruct-2507 using vLLM. Requires vllm>=0.9.0. Adjust context length based on GPU memory.

```shell
vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507 --port 8000 --tensor-parallel-size 8 --max-model-len 262144
```

--------------------------------

### Function calling messages in think mode

Source: https://qwen.readthedocs.io/en/latest/_sources/framework/function_call.md.txt

Example of a message list for function calling in 'think' mode, where the assistant includes reasoning_content and function_call fields. The model calls two functions to get current and tomorrow's temperature.

```python
[
    {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."},
    {"role": "assistant", "content": "", "reasoning_content": "Okay, the user is asking for the current temperature in San Francisco and the temperature for tomorrow. Let me check the available tools.\n\nFirst, there's the get_current_temperature function. It requires the location and optionally the unit. Since the user didn't specify the unit, I'll default to celsius. The location should be \"San Francisco, State, Country\". Wait, the example format is \"City, State, Country\", but San Francisco is a city in California, USA. So the location parameter would be \"San Francisco, California, United States\".\n\nThen, for tomorrow's temperature, the user mentioned the current date is 2024-09-30, so tomorrow would be 2024-10-01. The get_temperature_date function requires location, date, and unit. Again, using the same location and default unit. I need to format the date as \"Year-Month-Day\", which is 2024-10-01.\n\nWait, the current date given is 2024-09-30. If today is September 30, then tomorrow is October 1st. So the date parameter for the second function call should be \"2024-10-01\".\n\nI should make two separate function calls: one for the current temperature and another for tomorrow's date. Let me structure the JSON for both tool calls accordingly."},
    {"role": "assistant", "content": "", "function_call": {"name": "get_current_temperature", "arguments": "{\"location\": \"San Francisco, California, United States\", \"unit\": \"celsius\"}"}},
    {"role": "assistant", "content": "", "function_call": {"name": "get_temperature_date", "arguments": "{\"location\": \"San Francisco, California, United States\", \"date\": \"2024-10-01\", \"unit\": \"celsius\"}"}},
    {"role": "function", "name": "get_current_temperature", "content": '{"temperature": 26.1, "location": "San Francisco, California, United States", "unit": "celsius"}'},
    {"role": "function", "name": "get_temperature_date", "content": '{"temperature": 25.9, "location": "San Francisco, California, United States", "date": "2024-10-01", "unit": "celsius"}'},
]
```

--------------------------------

### Set up RetrievalQA chain with Qwen and FAISSWrapper

Source: https://qwen.readthedocs.io/en/latest/framework/Langchain.html

Shows the main execution block: loads documents from a file, creates a FAISSWrapper index, defines a prompt template, and builds a RetrievalQA chain using the Qwen LLM and HuggingFace embeddings. Requires a file path, embedding model path, and Qwen instance to be configured.

```python
if __name__ == '__main__':
    # load docs (pdf file or txt file)
    filepath = 'your file path'
    # Embedding model name
    EMBEDDING_MODEL = 'text2vec'
    PROMPT_TEMPLATE = """Known information:
    {context_str}
    Based on the above known information, respond to the user's question concisely and professionally. If an answer cannot be derived from it, say 'The question cannot be answered with the given information' or 'Not enough relevant information has been provided,' and do not include fabricated details in the answer. Please respond in English. The question is {question}"""
    # Embedding running device
    EMBEDDING_DEVICE = "cuda"
    # return top-k text chunk from vector store
    VECTOR_SEARCH_TOP_K = 3
    CHAIN_TYPE = 'stuff'
    embedding_model_dict = {
        "text2vec": "your text2vec model path",
    }
    llm = Qwen()
    embeddings = HuggingFaceEmbeddings(model_name=embedding_model_dict[EMBEDDING_MODEL],model_kwargs={'device': EMBEDDING_DEVICE})

    docs = load_file(filepath)

    docsearch = FAISSWrapper.from_documents(docs, embeddings)

    prompt = PromptTemplate(
        template=PROMPT_TEMPLATE, input_variables=["context_str", "question"]
    )

    chain_type_kwargs = {"prompt": prompt, "document_variable_name": "context_str"}
    qa = RetrievalQA.from_chain_type(
        llm=llm,
        chain_type=CHAIN_TYPE,
        retriever=docsearch.as_retriever(search_kwargs={"k": VECTOR_SEARCH_TOP_K}),
        chain_type_kwargs=chain_type_kwargs)

    query = "Give me a short introduction to large language models."
    print(qa.run(query))
```

--------------------------------

### Install OpenLLM with pip

Source: https://qwen.readthedocs.io/en/latest/_sources/deployment/openllm.rst.txt

Install OpenLLM using pip. After installation, verify with `openllm --help`.

```bash
pip install openllm
```

```bash
openllm --help
```

--------------------------------

### Output of openllm model list --tag qwen2.5

Source: https://qwen.readthedocs.io/en/latest/deployment/openllm.html

Example output showing the available Qwen2.5 models, their versions, required GPU RAM, and supported platforms.

```text
model    version                repo     required GPU RAM    platforms
-------  ---------------------  -------  ------------------  -----------
qwen2.5  qwen2.5:0.5b           default  12G                 linux
         qwen2.5:1.5b           default  12G                 linux
         qwen2.5:3b             default  12G                 linux
         qwen2.5:7b             default  24G                 linux
         qwen2.5:14b            default  80G                 linux
         qwen2.5:14b-ggml-q4    default                      macos
         qwen2.5:14b-ggml-q8    default                      macos
         qwen2.5:32b            default  80G                 linux
         qwen2.5:32b-ggml-fp16  default                      macos
         qwen2.5:72b            default  80Gx2               linux
         qwen2.5:72b-ggml-q4    default                      macos

```

--------------------------------

### Install optimum for GPTQ models

Source: https://qwen.readthedocs.io/en/latest/_sources/quantization/gptq.md.txt

Install the required version of the optimum library to use official Qwen2.5 GPTQ models with transformers. Ensure optimum>=1.20.0 is installed.

```bash
pip install -U "optimum>=1.20.0"
```

--------------------------------

### Launch SGLang server for Qwen/Qwen3-8B

Source: https://qwen.readthedocs.io/en/latest/_sources/deployment/sglang.md.txt

Starts an OpenAI-compatible API server at http://localhost:30000 by default. Use --host and --port to change the address. If --model-path is not a local directory, the model is downloaded from Hugging Face Hub.

```shell
python -m sglang.launch_server --model-path Qwen/Qwen3-8B
```

--------------------------------

### Install ModelScope

Source: https://qwen.readthedocs.io/en/latest/_sources/training/unsloth.md.txt

Install the modelscope package to use models from ModelScope.

```bash
pip install modelscope -qqq
```

--------------------------------

### Deploy Qwen2.5-7B-Instruct-GPTQ-Int4 with TGI

Source: https://qwen.readthedocs.io/en/latest/_sources/deployment/tgi.rst.txt

Command to start TGI with a GPTQ quantized model. Replace 'volume' with your local data directory.

```bash
model=Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
```

--------------------------------

### Install AutoAWQ

Source: https://qwen.readthedocs.io/en/latest/_sources/quantization/awq.md.txt

Install AutoAWQ version below 0.2.7 using pip.

```bash
pip install "autoawq<0.2.7"
```

--------------------------------

### Start an OpenAI-compatible API service with vLLM

Source: https://qwen.readthedocs.io/en/latest/_sources/quantization/gptq.md.txt

Run this command in a shell to start an OpenAI-compatible API service using the Qwen2.5-7B-Instruct-GPTQ-Int4 model. The service will be available at http://localhost:8000.

```bash
vllm serve Qwen2.5-7B-Instruct-GPTQ-Int4
```

--------------------------------

### Install vLLM with pip

Source: https://qwen.readthedocs.io/en/latest/_sources/deployment/vllm.md.txt

Install vLLM with pip in a clean environment. The prebuilt vllm has strict dependencies on torch and its CUDA versions; check the official installation notes for help.

```shell
pip install "vllm>=0.8.5"
```

--------------------------------

### Apply the service configuration

Source: https://qwen.readthedocs.io/en/latest/_sources/deployment/dstack.rst.txt

Deploys the service defined in serve-30b.dstack.yml. Run from the directory containing the file.

```bash
dstack apply -f serve-30b.dstack.yml
```
Tokens:
Raw