### Install Dependencies for Qwen Fine-tuning

Source: https://github.com/qwenlm/qwen/blob/main/recipes/applications/domain_finetune/qwen_domain_finetune.ipynb

Installs all necessary Python packages for fine-tuning the Qwen model by referencing the requirements.txt file.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/qwenlm/qwen/blob/main/recipes/inference/hf_modelscope/README.md

Installs the necessary Python packages for the Qwen project from a requirements file. It also provides instructions for optionally installing flash-attention for improved performance.

```bash
pip install -r Qwen/requirements.txt
git clone https://github.com/Dao-AILab/flash-attention
cd flash-attention && pip install .
# pip install csrc/layer_norm
# pip install csrc/rotary
```

--------------------------------

### Install DashScope SDK

Source: https://github.com/qwenlm/qwen/blob/main/README.md

Installation commands for the DashScope SDK using Python pip or Java Maven dependencies.

```bash
pip install dashscope
```

```xml
<dependency>
    <groupId>com.alibaba</groupId>
    <artifactId>dashscope-sdk-java</artifactId>
    <version>the-latest-version</version>
</dependency>
```

--------------------------------

### Install ms-swift and Dependencies (Shell)

Source: https://github.com/qwenlm/qwen/blob/main/recipes/finetune/swift/README.md

Installs the ms-swift package and optional dependencies like deepspeed, auto_gptq, or bitsandbytes for QLoRA training. It also includes commands for environment alignment.

```shell
pip config set global.index-url https://mirrors.aliyun.com/pypi/simple/
git clone https://github.com/modelscope/swift.git
cd swift
pip install -e .[llm]

pip install deepspeed -U

pip install auto_gptq -U

pip install bitsandbytes -U

pip install -r requirements/framework.txt  -U
pip install -r requirements/llm.txt  -U
```

--------------------------------

### Configure Docker for Qwen

Source: https://github.com/qwenlm/qwen/blob/main/README.md

Provides commands to install and configure Docker and the NVIDIA Container Toolkit, essential for running Qwen Docker images. Includes tests to verify correct installation.

```bash
# configure docker
sudo systemctl start docker
# test if docker is correctly installed
sudo docker run hello-world

# configure nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
# test if nvidia-container-toolkit is correctly installed
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
```

--------------------------------

### Qwen Model Fine-tuning Examples (PyTorch)

Source: https://github.com/qwenlm/qwen/blob/main/recipes/finetune/swift/README.md

A collection of examples for fine-tuning various Qwen models using PyTorch. These scripts cover different fine-tuning techniques such as LoRA, QLoRA, and full fine-tuning, often with distributed training strategies like DDP (Distributed Data Parallel) and MP (Model Parallel). Each example points to a specific script directory within the Swift repository.

```shell
Examples for qwen_14b:
[lora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b/lora_ddp_ds)
[qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b/qlora)
[qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b/qlora_ddp_ds)
```

```shell
Examples for qwen_14b_chat:
[lora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat/lora_ddp_ds)
[qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat/qlora)
[qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat/qlora_ddp_ds)
```

```shell
Examples for qwen_14b_chat_int4:
[qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat_int4/qlora)
[qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat_int4/qlora_ddp_ds)
```

```shell
Examples for qwen_14b_chat_int8:
[qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat_int8/qlora)
[qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat_int8/qlora_ddp_ds)
```

```shell
Examples for qwen_1_8b_chat:
[full](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_1_8b_chat/full)
[full_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_1_8b_chat/full_ddp)
```

```shell
Examples for qwen_72b_chat:
[lora_mp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat/lora_mp)
[lora_mp_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat/lora_mp_ddp)
[qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat/qlora)
```

```shell
Examples for qwen_72b_chat_int4:
[qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat_int4/qlora_ddp_ds)
```

```shell
Examples for qwen_72b_chat_int8:
[qlora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat_int8/qlora_ddp_ds)
```

```shell
Examples for qwen_7b:
[lora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b/lora_ddp_ds)
[qlora_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b/qlora_ddp)
```

```shell
Examples for qwen_7b_chat:
[full](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/full)
[full_freeze_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/full_freeze_ddp)
[full_mp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/full_mp)
[full_mp_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/full_mp_ddp)
[lora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/lora)
[lora_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/lora_ddp)
[lora_ddp_ds](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/lora_ddp_ds)
[lora_mp_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/lora_mp_ddp)
[qlora](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/qlora)
[qlora_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/qlora_ddp)
```

--------------------------------

### Deploy Qwen with vLLM and FastChat

Source: https://context7.com/qwenlm/qwen/llms.txt

Deploys a production-ready Qwen service using the vLLM backend and FastChat web interface. This involves installing necessary Python packages, starting the FastChat controller, model worker (with options for single/multi-GPU and quantized models), and finally launching the web UI or OpenAI-compatible API. Dependencies include 'vllm' and 'fschat'.

```bash
# Install dependencies
pip install vllm
pip install "fschat[model_worker,webui]"

# Start controller
python -m fastchat.serve.controller

# Start model worker (single GPU)
python -m fastchat.serve.vllm_worker \
    --model-path Qwen/Qwen-7B-Chat \
    --trust-remote-code \
    --dtype bfloat16

# For multi-GPU with tensor parallelism
python -m fastchat.serve.vllm_worker \
    --model-path Qwen/Qwen-72B-Chat \
    --trust-remote-code \
    --tensor-parallel-size 4 \
    --dtype bfloat16

# For quantized models
python -m fastchat.serve.vllm_worker \
    --model-path Qwen/Qwen-7B-Chat-Int4 \
    --trust-remote-code \
    --dtype float16

# Launch Web UI
python -m fastchat.serve.gradio_web_server

# Or launch OpenAI-compatible API
python -m fastchat.serve.openai_api_server --host localhost --port 8000
```

--------------------------------

### Launch Web and CLI Demos

Source: https://github.com/qwenlm/qwen/blob/main/README.md

Instructions for running the provided standalone web and CLI demonstration scripts. Requires installing the specific requirements file first.

```bash
pip install -r requirements_web_demo.txt
python web_demo.py
python cli_demo.py
```

--------------------------------

### Call Qwen API using DashScope SDK

Source: https://context7.com/qwenlm/qwen/llms.txt

Demonstrates how to use the DashScope SDK to interact with Qwen models hosted on Alibaba Cloud. It includes installation instructions, API key setup, and a Python function to send messages and receive responses. The function supports different model options like 'qwen_turbo' and 'qwen_plus'.

```bash
# Install SDK
pip install dashscope

# Set API key
export DASHSCOPE_API_KEY="YOUR_DASHSCOPE_API_KEY"
```

```python
import random
from http import HTTPStatus
from dashscope import Generation

def call_qwen_api():
    messages = [
        {'role': 'system', 'content': 'You are a helpful assistant.'},
        {'role': 'user', 'content': 'How do I make scrambled eggs?'}
    ]

    gen = Generation()
    response = gen.call(
        Generation.Models.qwen_turbo,  # Options: qwen_turbo (fast), qwen_plus (better)
        messages=messages,
        seed=random.randint(1, 10000),  # Random seed for reproducibility
        result_format='message',
    )
    return response

response = call_qwen_api()
if response.status_code == HTTPStatus.OK:
    print(response.output.text)
else:
    print(f'Error: {response.code} - {response.message}')
```

--------------------------------

### Launch Qwen Web Demo

Source: https://github.com/qwenlm/qwen/blob/main/recipes/applications/chatbot/qwen_chatbot.ipynb

Executes the web_demo.py script to start the Gradio web interface, pointing it to the downloaded model directory.

```bash
!python ../../../web_demo.py -c /tmp/models/qwen/Qwen-7B-Chat
```

--------------------------------

### Deploy Qwen with FastChat and vLLM

Source: https://github.com/qwenlm/qwen/blob/main/README.md

Commands to set up FastChat for serving Qwen models. This includes installing dependencies, launching the controller, starting the model worker with tensor parallelism, and exposing a web UI or OpenAI-compatible API.

```bash
pip install "fschat[model_worker,webui]"
python -m fastchat.serve.controller
python -m fastchat.serve.vllm_worker --model-path $model_path --trust-remote-code --tensor-parallel-size 4 --dtype bfloat16
python -m fastchat.serve.gradio_web_server
python -m fastchat.serve.openai_api_server --host localhost --port 8000
```

--------------------------------

### Install ModelScope and Dependencies

Source: https://github.com/qwenlm/qwen/blob/main/recipes/applications/chatbot/qwen_chatbot.ipynb

Installs the ModelScope library required for downloading the model, as well as the project-specific requirements and Gradio for the web interface.

```bash
!pip install modelscope
!pip install -r ../../../requirements.txt
!pip install gradio==3.37.0 mdtex2html
```

--------------------------------

### LoRA Fine-tuning Qwen Models

Source: https://context7.com/qwenlm/qwen/llms.txt

Provides instructions and scripts for performing LoRA (Low-Rank Adaptation) fine-tuning on Qwen models. It covers dependency installation, data preparation format, and commands for both single-GPU and distributed fine-tuning using DeepSpeed. Also includes an example of loading a LoRA-finetuned model for inference.

```bash
# Install dependencies
pip install "peft<0.8.0" deepspeed

# Prepare training data (example.json format):
# [
#   {
#     "id": "identity_0",
#     "conversations": [
#       {"from": "user", "value": "Hello"},
#       {"from": "assistant", "value": "Hi! How can I help you?"}
#     ]
#   }
# ]

# Single GPU LoRA finetuning
bash finetune/finetune_lora_single_gpu.sh -m Qwen/Qwen-7B-Chat -d /path/to/data.json

# Distributed LoRA finetuning with DeepSpeed
bash finetune/finetune_lora_ds.sh -m Qwen/Qwen-7B-Chat -d /path/to/data.json
```

```python
# Load finetuned LoRA model for inference
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

model = AutoPeftModelForCausalLM.from_pretrained(
    "output_qwen",  # Path to finetuned adapter
    device_map="auto",
    trust_remote_code=True
).eval()

tokenizer = AutoTokenizer.from_pretrained("output_qwen", trust_remote_code=True)
response, history = model.chat(tokenizer, "Hello", history=None)
print(response)
```

--------------------------------

### Start SWIFT Web UI (Shell)

Source: https://github.com/qwenlm/qwen/blob/main/recipes/finetune/swift/README.md

Launches the SWIFT web-based user interface, enabling model training and inference through a graphical interface.

```shell
swift web-ui
```

--------------------------------

### Install Fine-tuning Dependencies

Source: https://github.com/qwenlm/qwen/blob/main/README.md

Install the necessary Python packages, specifically Peft and DeepSpeed, required to run the official Qwen fine-tuning scripts.

```bash
pip install "peft<0.8.0" deepspeed
```

--------------------------------

### Deploy Web UI Demo using Docker

Source: https://github.com/qwenlm/qwen/blob/main/README.md

Starts a web-based demo for the Qwen model using Docker. This script configures the Docker image, checkpoint path, and host port, then launches the web UI in the background. The service is set to auto-restart.

```bash
IMAGE_NAME=qwenllm/qwen:cu117
PORT=8901
CHECKPOINT_PATH=/path/to/Qwen-7B-Chat   # Path to downloaded model checkpoints and codes
bash docker/docker_web_demo.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH} --port ${PORT}
```

--------------------------------

### Install LangChain and Qwen Dependencies

Source: https://github.com/qwenlm/qwen/blob/main/recipes/applications/retrieval/retrieval.ipynb

Installs essential libraries for LangChain, DashScope, sentence processing, and vectorization, including specific versions required for the project. These are critical for the Q&A pipeline.

```python
!pip install langchain==0.0.187 dashscope==1.0.4 sentencepiece==0.1.99 cpm_kernels==1.0.11 nltk==3.8.1 sentence_transformers==2.2.2 unstructured==0.6.5 faiss-cpu==1.7.4 icetk==0.0.7
```

--------------------------------

### Download Qwen-7B-Chat Model with Modelscope

Source: https://github.com/qwenlm/qwen/blob/main/recipes/applications/retrieval/retrieval.ipynb

Installs the modelscope library and downloads the Qwen-7B-Chat model to a specified cache directory. This is the first step in setting up the local LLM.

```python
!pip install modelscope
from modelscope.hub.snapshot_download import snapshot_download
snapshot_download("Qwen/Qwen-7B-Chat",cache_dir='/tmp/models')
```

--------------------------------

### Deploy Qwen Models with vLLM OpenAI-like API Server

Source: https://github.com/qwenlm/qwen/blob/main/recipes/inference/vllm/README.md

This section provides instructions and code examples for deploying Qwen models using vLLM's standalone OpenAI API server. It requires downloading a chat template file and launching the server via a command-line interface. The Python code demonstrates how to interact with this API for both streaming and non-streaming responses, setting necessary parameters like `stop_token_ids`.

```bash
python -m vllm.entrypoints.openai.api_server --model $model_path --trust-remote-code --chat-template template_chatml.jinja

# run int4 model or GPUs with compute capability less than 8.0
# python -m vllm.entrypoints.openai.api_server --model $model_path --trust-remote-code --dtype float16 --chat-template template_chatml.jinja
```

```python
import openai
openai.api_base = "http://localhost:8000/v1"
openai.api_key = "none"

# create a request activating streaming response
for chunk in openai.ChatCompletion.create(
    model="Qwen",
    messages=[
        {"role": "user", "content": "你好"}
    ],
    stream=True, 
    stop_token_ids=[151645]
):
    if hasattr(chunk.choices[0].delta, "content"):
        print(chunk.choices[0].delta.content, end="", flush=True)

# create a request not activating streaming response
response = openai.ChatCompletion.create(
    model="Qwen",
    messages=[
        {"role": "user", "content": "你好"}
    ],
    stream=False,
    stop_token_ids=[151645]
)
print(response.choices[0].message.content)
```

--------------------------------

### Install Flash Attention for Performance

Source: https://github.com/qwenlm/qwen/blob/main/recipes/quickstart/qwen.ipynb

Installs the flash-attention library to accelerate model training and inference. This is an optional step for users seeking performance improvements.

```bash
!git clone https://github.com/Dao-AILab/flash-attention
!cd flash-attention && pip install .
```

--------------------------------

### Launch Docker Container for Ascend NPU

Source: https://github.com/qwenlm/qwen/blob/main/recipes/finetune/ascend/README.md

Starts a Docker container with the necessary Ascend NPU device mappings and volume mounts for training.

```bash
ImageID=pai-image-manage-registry.cn-wulanchabu.cr.aliyuncs.com/pai/llm-inference:qwen_v23.0.rc3
docker run -it -u root --ipc=host \
--device=/dev/davinci0 \
--device=/dev/davinci1 \
--device=/dev/davinci2 \
--device=/dev/davinci3 \
--device=/dev/davinci4 \
--device=/dev/davinci5 \
--device=/dev/davinci6 \
--device=/dev/davinci7 \
--device=/dev/davinci_manager \
--device=/dev/devmm_svm \
--device=/dev/hisi_hdc \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/add-ons/:/usr/local/Ascend/add-ons/ \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /var/log/npu/:/usr/slog \
-v /etc/hccn.conf:/etc/hccn.conf \
${ImageID} /bin/bash
```

--------------------------------

### Install Required Python Dependencies

Source: https://github.com/qwenlm/qwen/blob/main/recipes/quickstart/qwen.ipynb

Installs the necessary Python packages including transformers, accelerate, and modelscope to run Qwen models.

```bash
!pip install transformers>=4.32.0 accelerate tiktoken einops scipy transformers_stream_generator==0.0.4 peft deepspeed modelscope
```

--------------------------------

### Install Flash Attention

Source: https://github.com/qwenlm/qwen/blob/main/README.md

Clones and installs the Flash Attention library to improve inference efficiency and reduce memory usage on supported hardware.

```bash
git clone https://github.com/Dao-AILab/flash-attention
cd flash-attention && pip install .
```

--------------------------------

### Fine-tune Qwen Model with SWIFT (Python)

Source: https://github.com/qwenlm/qwen/blob/main/recipes/finetune/swift/README.md

Demonstrates how to fine-tune the Qwen model using the SWIFT framework. Supports using pre-defined datasets or custom JSONL files, and includes configurations for single-GPU, DDP, and multi-machine multi-GPU setups.

```python
CUDA_VISIBLE_DEVICES=0 \
swift sft \
    --model_id_or_path qwen/Qwen-7B-Chat \
    --dataset blossom-math-zh \
    --output_dir output \

```

```python
CUDA_VISIBLE_DEVICES=0 \
swift sft \
    --model_id_or_path qwen/Qwen-7B-Chat \
    --custom_train_dataset_path chatml.jsonl \
    --output_dir output \

```

```python
CUDA_VISIBLE_DEVICES=0,1 \
NPROC_PER_NODE=2 \
swift sft \
    --model_id_or_path qwen/Qwen-7B-Chat \
    --dataset blossom-math-zh \
    --output_dir output \

```

```python
CUDA_VISIBLE_DEVICES=0,1,2,3 \
NNODES=2 \
NODE_RANK=0 \
MASTER_ADDR=127.0.0.1 \
NPROC_PER_NODE=4 \
swift sft \
    --model_id_or_path qwen/Qwen-7B-Chat \
    --dataset blossom-math-zh \
    --output_dir output \

```

--------------------------------

### Using Qwen API with OpenAI Client (Streaming and Non-Streaming)

Source: https://context7.com/qwenlm/qwen/llms.txt

Python code demonstrating how to configure the OpenAI client to interact with a local Qwen API server. Includes examples for both streaming and non-streaming responses, and custom stop words.

```python
import openai

# Configure client to use local Qwen server
openai.api_base = "http://localhost:8000/v1"
openai.api_key = "none"  # Not required for local server

# Streaming response
for chunk in openai.ChatCompletion.create(
    model="Qwen",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is Python?"}
    ],
    stream=True
):
    if hasattr(chunk.choices[0].delta, "content"):
        print(chunk.choices[0].delta.content, end="", flush=True)

# Non-streaming response with stop words
response = openai.ChatCompletion.create(
    model="Qwen",
    messages=[
        {"role": "user", "content": "Hello, how are you?"}
    ],
    stream=False,
    stop=["Observation:"]  # Custom stop words for ReAct prompting
)
print(response.choices[0].message.content)
```

--------------------------------

### Start Single-GPU LoRA Finetuning using Docker

Source: https://github.com/qwenlm/qwen/blob/main/README.md

Initiates a single-GPU LoRA finetuning process for the Qwen model using a Docker image. This script requires specifying the Docker image, the path to model checkpoints and code, the path to the finetuning data, and the desired output path for the finetuned model.

```bash
IMAGE_NAME=qwenllm/qwen:cu117
CHECKPOINT_PATH=/path/to/Qwen-7B                # Path to downloaded model checkpoints and codes
#CHECKPOINT_PATH=/path/to/Qwen-7B-Chat-Int4     # Path to downloaded model checkpoints and codes (Q-LoRA)
DATA_PATH=/path/to/data/root                    # Prepare finetune data at ${DATA_PATH}/example.json
OUTPUT_PATH=/path/to/output/checkpoint          # Path to finetune outputs

# Use all host devices by default
DEVICE=all
```

--------------------------------

### Function Calling with Tools using Qwen

Source: https://context7.com/qwenlm/qwen/llms.txt

Example of enabling Qwen to call external functions or tools. It defines available tools (Google Search, Image Generator) and demonstrates making requests with function calling capabilities.

```python
import openai
import json

openai.api_base = 'http://localhost:8000/v1'
openai.api_key = 'none'

# Define available tools
functions = [
    {
        'name_for_human': 'Google Search',
        'name_for_model': 'google_search',
        'description_for_model': 'A search engine for accessing the internet and querying information. Format the arguments as a JSON object.',
        'parameters': [{
            'name': 'search_query',
            'description': 'Search keywords or phrase',
            'required': True,
            'schema': {'type': 'string'},
        }],
    },
    {
        'name_for_human': 'Image Generator',
        'name_for_model': 'image_gen',
        'description_for_model': 'AI image generation service. Input text description, returns image URL. Format the arguments as a JSON object.',
        'parameters': [{
            'name': 'prompt',
            'description': 'English keywords describing the desired image content',
            'required': True,
            'schema': {'type': 'string'},
        }],
    },
]

# Make request with function calling
messages = [{'role': 'user', 'content': 'Search for information about GPT-4'}]
response = openai.ChatCompletion.create(
    model='Qwen',
    messages=messages,
    functions=functions
)

result = response.choices[0]['message']
print(f"Response: {result.get('content', '')}")
if result.get('function_call'):
    print(f"Function: {result['function_call']['name']}")
    print(f"Arguments: {result['function_call']['arguments']}")
    # Output:
    # Function: google_search
    # Arguments: {"search_query": "GPT-4"}

# Provide function result and continue conversation
messages.append({
    'role': 'assistant',
    'content': 'I should use Google Search to find information.',
    'function_call': {'name': 'google_search', 'arguments': '{"search_query": "GPT-4"}'},
})
messages.append({
    'role': 'function',
    'name': 'google_search',
    'content': 'GPT-4 is a large multimodal model by OpenAI.',
})

response = openai.ChatCompletion.create(model='Qwen', messages=messages, functions=functions)
print(response.choices[0]['message']['content'])
```

--------------------------------

### Install AutoGPTQ Dependencies

Source: https://github.com/qwenlm/qwen/blob/main/recipes/inference/quantization/README.md

Install the necessary Python packages to enable GPTQ quantization support. Ensure your environment meets the specific torch and transformer version requirements.

```bash
pip install auto-gptq optimum
```

--------------------------------

### vLLM High-Performance Inference

Source: https://context7.com/qwenlm/qwen/llms.txt

This example shows how to initialize and use the vLLMWrapper for high-performance inference with Qwen models, including support for tensor parallelism and GPU memory utilization configuration. It also provides an example for quantized models.

```python
from vllm_wrapper import vLLMWrapper

# Initialize model with vLLM backend
model = vLLMWrapper(
    'Qwen/Qwen-7B-Chat',
    tensor_parallel_size=1,         # Set to number of GPUs for parallelism
    gpu_memory_utilization=0.98,    # GPU memory fraction to use
    dtype="bfloat16"                # Options: "bfloat16", "float16", "float32"
)

# For quantized models:
# model = vLLMWrapper('Qwen/Qwen-7B-Chat-Int4', tensor_parallel_size=1, dtype="float16")
```

--------------------------------

### Launch FastChat Web UI Server

Source: https://github.com/qwenlm/qwen/blob/main/recipes/inference/vllm/README.md

Starts the Gradio-based web UI server for interacting with the deployed Qwen model through FastChat. This allows for a visual interface to test the model.

```bash
python -m fastchat.serve.gradio_web_server
```

--------------------------------

### Load and Run Qwen-7B using Hugging Face

Source: https://github.com/qwenlm/qwen/blob/main/README.md

Shows how to load the base Qwen-7B model using Hugging Face Transformers and perform text generation. It includes examples of different precision settings and how to move inputs to the model's device.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import GenerationConfig

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen-7B", trust_remote_code=True)

# Load model with auto precision
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen-7B",
    device_map="auto",
    trust_remote_code=True
).eval()

# Example text generation
inputs = tokenizer('蒙古国的首都是乌兰巴托（Ulaanbaatar）\n冰岛的首都是雷克雅未克（Reykjavik）\n埃塞俄比亚的首都是', return_tensors='pt')
inputs = inputs.to(model.device)
pred = model.generate(**inputs)
print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))
```

--------------------------------

### Test Qwen Model with Python

Source: https://github.com/qwenlm/qwen/blob/main/recipes/applications/domain_finetune/qwen_domain_finetune.ipynb

This Python script demonstrates how to load a pre-trained Qwen model and tokenizer from a local directory. It then uses the model to generate a response to a given prompt through a chat interface. Ensure the 'transformers' library is installed and the model weights are correctly merged in the specified directory.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import GenerationConfig

tokenizer = AutoTokenizer.from_pretrained("output_qwen_merged", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "output_qwen_merged",
    device_map="auto",
    trust_remote_code=True
).eval()

response, history = model.chat(tokenizer, "什么是VDAC1？", history=None)
print(response)
```

--------------------------------

### Qwen Model Inference Example

Source: https://github.com/qwenlm/qwen/blob/main/recipes/finetune/swift/README.md

This code snippet demonstrates how to load a Qwen model and tokenizer, set up the appropriate template, and perform inference for both single-turn and multi-turn conversations. It includes setting CUDA devices, generation configurations, and handling query-response cycles.

```APIDOC
## Qwen Model Inference

### Description
This section provides a Python code example for performing inference with Qwen models using the `swift.llm` library. It covers model and tokenizer loading, template configuration, and interactive inference.

### Method
N/A (This is a script example, not a direct API endpoint)

### Endpoint
N/A

### Parameters
N/A

### Request Example
```python
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'

from swift.llm import (
    get_model_tokenizer,
    get_template,
    inference,
    ModelType,
    get_default_template_type,
)
from swift.utils import seed_everything

# Specify the model type
model_type = ModelType.qwen_7b_chat
template_type = get_default_template_type(model_type)
print(f'template_type: {template_type}')

kwargs = {}
# Optional: uncomment to use flash_attn if available
# kwargs['use_flash_attn'] = True

# Load model and tokenizer
model, tokenizer = get_model_tokenizer(model_type, model_kwargs={'device_map': 'auto'}, **kwargs)

# Modify generation configuration (e.g., max_new_tokens)
model.generation_config.max_new_tokens = 128

# Get the template
template = get_template(template_type, tokenizer)

# Set random seed for reproducibility
seed_everything(42)

# First query (single-turn)
query1 = 'What is the provincial capital of Zhejiang?'
response1, history1 = inference(model, template, query1)
print(f'query: {query1}')
print(f'response: {response1}')

# Second query (multi-turn, using history)
query2 = 'What delicious food can be found here?'
response2, history2 = inference(model, template, query2, history=history1)
print(f'query: {query2}')
print(f'response: {response2}')
print(f'history: {history2}')
```

### Response
#### Success Response (Output)
```
query: What is the provincial capital of Zhejiang?
response: The provincial capital of Zhejiang is Hangzhou.
query: What delicious food can be found here?
response: Hangzhou has many famous delicacies, such as West Lake Vinegar Fish, Longjing Shrimp, Sweet and Sour Spare Ribs, and Maodu. Additionally, there are unique Hangzhou-style pastries like Osmanthus Cake, Lotus Paste Pastry, and Aiwo Steamed Rice Cakes.
history: [(\'What is the provincial capital of Zhejiang?\', \'The provincial capital of Zhejiang is Hangzhou.\'), (\'What delicious food can be found here?\', \'Hangzhou has many famous delicacies, such as West Lake Vinegar Fish, Longjing Shrimp, Sweet and Sour Spare Ribs, and Maodu. Additionally, there are unique Hangzhou-style pastries like Osmanthus Cake, Lotus Paste Pastry, and Aiwo Steamed Rice Cakes.\')]
```

#### Response Example
N/A (Output is shown above)

### Error Handling
N/A (Standard Python exceptions will be raised for errors.)
```

--------------------------------

### Deploy CLI Demo using Docker

Source: https://github.com/qwenlm/qwen/blob/main/README.md

Launches a command-line interface (CLI) demo for the Qwen model within a Docker container. This script requires specifying the Docker image and the path to the model checkpoints and code.

```bash
IMAGE_NAME=qwenllm/qwen:cu117
CHECKPOINT_PATH=/path/to/Qwen-7B-Chat   # Path to downloaded model checkpoints and codes
bash docker/docker_cli_demo.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH}
```

--------------------------------

### Initializing Qwen Tokenizer with Extra Vocabulary (Python)

Source: https://github.com/qwenlm/qwen/blob/main/examples/tokenizer_showcase.ipynb

Shows how to load the Qwen tokenizer from a pre-trained model and specify an additional vocabulary file for extended token support. This allows for custom token handling.

```python
tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen-7B', trust_remote_code=True, extra_vocab_file="qwen_extra.tiktoken")
print(len(tokenizer))
```

--------------------------------

### Install Fine-tuning Dependencies

Source: https://github.com/qwenlm/qwen/blob/main/recipes/finetune/deepspeed/readme.md

Installs the necessary libraries for fine-tuning, specifically PEFT and DeepSpeed. Ensure pydantic version is less than 2.0 to avoid conflicts.

```bash
pip install peft deepspeed
```

--------------------------------

### Initialize Qwen Model with Transformers

Source: https://github.com/qwenlm/qwen/blob/main/README.md

Demonstrates how to load the Qwen-Chat model and tokenizer using the Hugging Face Transformers library.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import GenerationConfig

# Model names: "Qwen/Qwen-7B-Chat", "Qwen/Qwen-14B-Chat"
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen-7B-Chat", trust_remote_code=True)
```

--------------------------------

### Install DashScope Python SDK

Source: https://github.com/qwenlm/qwen/blob/main/recipes/inference/dashscope/README.md

This command installs the DashScope SDK for Python using pip. This SDK is required to interact with the DashScope API from a Python application.

```bash
pip install dashscope
```

--------------------------------

### Run Qwen Models using ModelScope

Source: https://github.com/qwenlm/qwen/blob/main/README.md

Demonstrates how to use Qwen models directly with ModelScope, including loading the tokenizer and model, and performing chat-based interactions. It also shows how to configure generation parameters.

```python
from modelscope import AutoModelForCausalLM, AutoTokenizer
from modelscope import GenerationConfig

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("qwen/Qwen-7B-Chat", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("qwen/Qwen-7B-Chat", device_map="auto", trust_remote_code=True, fp16=True).eval()
model.generation_config = GenerationConfig.from_pretrained("qwen/Qwen-7B-Chat", trust_remote_code=True) # Can specify different generation length, top_p, etc.

# Perform chat interactions
response, history = model.chat(tokenizer, "你好", history=None)
print(response)
response, history = model.chat(tokenizer, "浙江的省会在哪里？", history=history)
print(response)
response, history = model.chat(tokenizer, "它有什么好玩的景点", history=history)
print(response)
```

--------------------------------

### Install vLLM

Source: https://github.com/qwenlm/qwen/blob/main/recipes/inference/vllm/README.md

Installs the vLLM library, a high-throughput and memory-efficient inference engine for large language models. This command is suitable for users with CUDA 12.1 and PyTorch 2.1.

```bash
pip install vllm
```

--------------------------------

### Install Dependencies for OpenAI API

Source: https://github.com/qwenlm/qwen/blob/main/README.md

Installs the necessary Python packages to deploy a local API compatible with the OpenAI API. This includes libraries for web serving, API interaction, and data validation.

```bash
pip install fastapi uvicorn "openai<1.0" pydantic sse_starlette
```

--------------------------------

### Perform Inference with Qwen Models using SWIFT

Source: https://github.com/qwenlm/qwen/blob/main/recipes/finetune/swift/README.md

This script demonstrates how to initialize a Qwen model and tokenizer, set up a chat template, and perform multi-turn inference. It uses the SWIFT library to manage model loading and generation configurations.

```python
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'

from swift.llm import (
    get_model_tokenizer, get_template, inference, ModelType, get_default_template_type,
)
from swift.utils import seed_everything

model_type = ModelType.qwen_7b_chat
template_type = get_default_template_type(model_type)
print(f'template_type: {template_type}')

kwargs = {}
model, tokenizer = get_model_tokenizer(model_type, model_kwargs={'device_map': 'auto'}, **kwargs)
model.generation_config.max_new_tokens = 128

template = get_template(template_type, tokenizer)
seed_everything(42)
query = 'What is the provincial capital of Zhejiang?'
response, history = inference(model, template, query)
print(f'query: {query}')
print(f'response: {response}')

query = 'What delicious food can be found here?'
response, history = inference(model, template, query, history)
print(f'query: {query}')
print(f'response: {response}')
print(f'history: {history}')
```

--------------------------------

### Install FastChat for vLLM Integration

Source: https://github.com/qwenlm/qwen/blob/main/recipes/inference/vllm/README.md

Installs FastChat, a platform for training, serving, and evaluating large language models, along with necessary dependencies for integrating with vLLM for model workers and web UI.

```bash
pip install "fschat[model_worker,webui]=0.2.33" "openai<1.0"
```

--------------------------------

### Deploying Qwen as an OpenAI-Compatible API Server

Source: https://context7.com/qwenlm/qwen/llms.txt

Instructions and code for deploying Qwen as an OpenAI-compatible API server. This allows Qwen to be used as a drop-in replacement for OpenAI API calls.

```bash
# Install dependencies
pip install fastapi uvicorn "openai<1.0" pydantic sse_starlette

# Start the API server
python openai_api.py -c Qwen/Qwen-7B-Chat --server-port 8000

# For CPU-only deployment:
# python openai_api.py -c Qwen/Qwen-7B-Chat --server-port 8000 --cpu-only
```

--------------------------------

### Get Specific Token from ID

Source: https://github.com/qwenlm/qwen/blob/main/examples/tokenizer_showcase.ipynb

Retrieves the token string for a given token ID, including special tokens.

```APIDOC
## POST /api/tokenizer/id_to_token

### Description
Retrieves the token string for a given token ID, including special tokens.

### Method
POST

### Endpoint
/api/tokenizer/id_to_token

### Parameters
#### Request Body
- **token_id** (int) - Required - The token ID to get the token string for.

### Request Example
```json
{
  "token_id": 151850
}
```

### Response
#### Success Response (200)
- **token** (string) - The token string corresponding to the ID.

#### Response Example
```json
{
  "token": "<|extra_204|>"
}
```
```

--------------------------------

### Get Specific Token ID

Source: https://github.com/qwenlm/qwen/blob/main/examples/tokenizer_showcase.ipynb

Retrieves the token ID for a given token string, including special tokens.

```APIDOC
## POST /api/tokenizer/token_to_id

### Description
Retrieves the token ID for a given token string, including special tokens.

### Method
POST

### Endpoint
/api/tokenizer/token_to_id

### Parameters
#### Request Body
- **token** (string) - Required - The token string to get the ID for.

### Request Example
```json
{
  "token": "<|extra_204|>"
}
```

### Response
#### Success Response (200)
- **token_id** (int) - The ID of the token.

#### Response Example
```json
{
  "token_id": 151850
}
```
```

--------------------------------

### Fine-tune Qwen-1.8B-Chat with QLoRA and DeepSpeed

Source: https://github.com/qwenlm/qwen/blob/main/recipes/finetune/deepspeed/finetune_qlora_multi_gpu.ipynb

This command initiates the QLoRA fine-tuning process for the Qwen-1.8B-Chat model using DeepSpeed for multi-GPU training. It specifies model paths, training parameters, and DeepSpeed configuration. Ensure 'torchrun' and the 'finetune.py' script are available.

```bash
!torchrun --nproc_per_node 2 --nnodes 1 --node_rank 0 --master_addr localhost --master_port 6601 ../../finetune.py \
    --model_name_or_path "Qwen/Qwen-1_8B-Chat-Int4/" \
    --data_path "Belle_sampled_qwen.json" \
    --bf16 True \
    --output_dir "output_qwen" \
    --num_train_epochs 5 \
    --per_device_train_batch_size 1 \
    --per_device_eval_batch_size 1 \
    --gradient_accumulation_steps 16 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 1000 \
    --save_total_limit 10 \
    --learning_rate 1e-5 \
    --weight_decay 0.1 \
    --adam_beta2 0.95 \
    --warmup_ratio 0.01 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --report_to "none" \
    --model_max_length 512 \
    --gradient_checkpointing True \
    --lazy_preprocess True \
    --deepspeed "../../finetune/ds_config_zero2.json" \
    --use_lora \
    --q_lora
```

--------------------------------

### Get Tokenizer Vocabulary Size

Source: https://github.com/qwenlm/qwen/blob/main/examples/tokenizer_showcase.ipynb

Retrieves the total number of tokens in the tokenizer's vocabulary, including special tokens.

```APIDOC
## GET /api/tokenizer/vocab_size

### Description
Retrieves the total number of tokens in the tokenizer's vocabulary, including special tokens.

### Method
GET

### Endpoint
/api/tokenizer/vocab_size

### Response
#### Success Response (200)
- **vocab_size** (int) - The total size of the tokenizer's vocabulary.

#### Response Example
```json
{
  "vocab_size": 151857
}
```
```

--------------------------------

### Define Multi-turn Training Data

Source: https://github.com/qwenlm/qwen/blob/main/recipes/finetune/deepspeed/readme.md

Example of a multi-turn conversation format for training, where multiple user-assistant exchanges are included in the conversation list.

```json
[
  {
    "id": "identity_0",
    "conversations": [
      {
        "from": "user",
        "value": "你好"
      },
      {
        "from": "assistant",
        "value": "你好！我是一名AI助手，我叫通义千问，有需要请告诉我。"
      },
      {
        "from": "user",
        "value": "你都能做什么"
      },
      {
        "from": "assistant",
        "value": "我能做很多事情，包括但不限于回答各种领域的问题、提供实用建议和指导、进行多轮对话交流、文本生成等。"
      }
    ]
  }
]
```

--------------------------------

### Prepare Training Data for Qwen

Source: https://github.com/qwenlm/qwen/blob/main/recipes/finetune/ascend/README.md

Downloads sample data, converts it to the required conversation format, and generates Mindrecord files for training.

```bash
wget -c https://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/alpaca_data_min.json
python research/qwen/alpaca_converter.py \
    --data_path alpaca_data_min.json \
    --output_path alpaca-data-conversation_min.json
python research/qwen/qwen_preprocess.py \
    --input_glob alpaca-data-conversation_min.json \
    --model_file Qwen/Qwen-7B-Chat/qwen.tiktoken \
    --seq_length 1024 \
    --output_file alpaca_min.mindrecord
```

--------------------------------

### Deploy Local OpenAI API

Source: https://github.com/qwenlm/qwen/blob/main/README.md

Launches a local API server using Python, compatible with the OpenAI API. This allows interaction with the Qwen model through standard OpenAI client libraries. Various arguments can be used to customize deployment, such as specifying the checkpoint or enabling CPU-only mode.

```bash
python openai_api.py
```

--------------------------------

### Load Qwen Model on CPU using Transformers

Source: https://github.com/qwenlm/qwen/blob/main/recipes/inference/hf_modelscope/README.md

This snippet demonstrates how to load a Qwen model directly onto the CPU using the Transformers library. It specifies 'cpu' as the device map. Note that CPU inference can be significantly slower than GPU inference.

```python
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-7B-Chat", device_map="cpu", trust_remote_code=True).eval()
```

--------------------------------

### Launch FastChat Controller

Source: https://github.com/qwenlm/qwen/blob/main/recipes/inference/vllm/README.md

Starts the FastChat controller, which manages the distributed model workers and routes requests. This is a prerequisite for launching model workers and API servers.

```bash
python -m fastchat.serve.controller
```

--------------------------------

### Tokenization Example: Default Behavior

Source: https://github.com/qwenlm/qwen/blob/main/tokenization_note.md

Demonstrates the default tokenization behavior where special token surface forms are treated as regular text. This is the behavior before enabling injection prevention.

```python
print("<|endoftext|>")
# Expected tokenization: ids:[1350, 9639, 91, 8691, 723, 427, 91, 82598]
# tokens: [b'print', b'("<', b'|', b'endo', b'ft', b'ext', b'|', b'>")']
```

--------------------------------

### Run Qwen Inference with vLLM Wrapper

Source: https://github.com/qwenlm/qwen/blob/main/README.md

Demonstrates how to use the vLLMWrapper to load a Qwen model and perform multi-turn dialogue interactions. It supports chat-based queries and maintains conversation history.

```python
from vllm_wrapper import vLLMWrapper

model = vLLMWrapper('Qwen/Qwen-7B-Chat', tensor_parallel_size=1)
response, history = model.chat(query="你好", history=None)
print(response)
response, history = model.chat(query="给我讲一个年轻人奋斗创业最终取得成功的故事。", history=history)
print(response)
response, history = model.chat(query="给这个故事起一个标题", history=history)
print(response)
```