### Launch Qwen3-Instruct API with SGLang Source: https://qwen.readthedocs.io/en/latest/_sources/getting_started/quickstart.md.txt Use this command to start the API server for Qwen3-Instruct-2507 with SGLang. Ensure sglang is installed and the model path is correct. Adjust context length based on GPU memory. ```shell python -m sglang.launch_server --model-path Qwen/Qwen3-235B-A22B-Instruct-2507 --port 8000 --tp 8 --context-length 262144 ``` -------------------------------- ### Multi-GPU Training Setup Source: https://qwen.readthedocs.io/en/latest/training/ms_swift.html Demonstrates setting up multi-GPU training for large models. This example uses 4 GPUs and requires specifying dataset, batch sizes, learning rate, and other training parameters. It also includes options for packing and attention implementation. ```bash # 4 * 60GB # You can run the experiment by setting `--dataset AI-ModelScope/alpaca-gpt4-data-en`. # Note: If you want to specify `--packing true`, you must additionally set `--attn_impl flash_attn`. NPROC_PER_NODE=4 \ CUDA_VISIBLE_DEVICES=0,1,2,3 \ swift sft \ --model Qwen/Qwen3-8B \ --train_type full \ --dataset '' \ --torch_dtype bfloat16 \ --per_device_train_batch_size 1 \ --per_device_eval_batch_size 1 \ --learning_rate 1e-5 \ --gradient_accumulation_steps 4 \ --packing true \ --eval_steps 100 \ --save_steps 100 \ --logging_steps 5 \ --max_length 8192 \ --warmup_ratio 0.05 \ --dataloader_num_workers 8 \ --dataset_num_proc 8 \ --save_total_limit 2 \ --save_only_model true \ --output_dir output \ --deepspeed zero3 \ --use_liger_kernel true \ --attn_impl flash_attn ``` -------------------------------- ### Run llama-server with Reasoning Source: https://qwen.readthedocs.io/en/latest/_sources/run_locally/llama.cpp.md.txt Start the llama.cpp HTTP server for LLM interaction via REST APIs. This example loads a model, enables Jinja templating, and specifies reasoning format and performance options. ```shell ./llama-server -hf Qwen/Qwen3-8B-GGUF:Q8_0 --jinja --reasoning-format deepseek -ngl 99 -fa -sm row --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 -c 40960 -n 32768 --no-context-shift ``` -------------------------------- ### Install AutoGPTQ from Source Source: https://qwen.readthedocs.io/en/latest/_sources/quantization/gptq.md.txt Install the latest version of AutoGPTQ by cloning the repository and installing it in editable mode. This is recommended for quantizing your own models. ```bash git clone https://github.com/AutoGPTQ/AutoGPTQ cd AutoGPTQ pip install -e . ``` -------------------------------- ### Train Qwen3 Models with FSDP and vLLM using GRPO Source: https://qwen.readthedocs.io/en/latest/training/verl.html Example command to start training Qwen3 models using verl with FSDP and vLLM frameworks, employing the GRPO algorithm. This configuration is suitable for a single 80GB GPU and machines with over 64GB of memory. ```bash python3 -m verl.trainer.main_ppo \ algorithm.adv_estimator=grpo \ data.train_files=$HOME/data/gsm8k/train.parquet \ data.val_files=$HOME/data/gsm8k/test.parquet \ data.train_batch_size=1024 \ data.max_prompt_length=512 \ data.max_response_length=1024 \ data.filter_overlong_prompts=True \ data.truncation='error' \ actor_rollout_ref.model.path=Qwen/Qwen3-1.7B \ actor_rollout_ref.actor.optim.lr=1e-6 \ actor_rollout_ref.model.use_remove_padding=True \ actor_rollout_ref.actor.ppo_mini_batch_size=80 \ actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=20 \ actor_rollout_ref.actor.use_kl_loss=True \ actor_rollout_ref.actor.kl_loss_coef=0.001 \ actor_rollout_ref.actor.kl_loss_type=low_var_kl \ actor_rollout_ref.actor.entropy_coeff=0 \ actor_rollout_ref.model.enable_gradient_checkpointing=True \ actor_rollout_ref.actor.fsdp_config.param_offload=False \ actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \ actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=20 \ actor_rollout_ref.rollout.tensor_model_parallel_size=1 \ actor_rollout_ref.rollout.name=vllm \ actor_rollout_ref.rollout.gpu_memory_utilization=0.6 \ actor_rollout_ref.rollout.n=3 \ actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=20 \ actor_rollout_ref.ref.fsdp_config.param_offload=True \ algorithm.use_kl_in_reward=False \ trainer.critic_warmup=0 \ trainer.logger=['console'] \ trainer.project_name='verl_grpo_example_gsm8k' \ trainer.experiment_name='qwen3_1_7b_function_rm' \ trainer.n_gpus_per_node=1 \ trainer.nnodes=1 \ trainer.save_freq=-1 \ trainer.test_freq=5 \ trainer.total_epochs=15 \ $@ ``` -------------------------------- ### Install OpenLLM Source: https://qwen.readthedocs.io/en/latest/deployment/openllm.html Install the OpenLLM package using pip. This is the first step to get started with OpenLLM. ```bash pip install openllm ``` -------------------------------- ### Install Build Tools on Ubuntu Source: https://qwen.readthedocs.io/en/latest/_sources/run_locally/llama.cpp.md.txt Installs essential build tools on Ubuntu using apt. ```bash sudo apt install build-essential ``` -------------------------------- ### Run Default Qwen2.5 Model with Ollama Source: https://qwen.readthedocs.io/en/latest/_sources/run_locally/ollama.md.txt Use this command to quickly start running the default Qwen2.5 model locally after installing Ollama. ```bash ollama run qwen2.5 ``` -------------------------------- ### Launch Megatron-SWIFT SFT Training Source: https://qwen.readthedocs.io/en/latest/training/ms_swift.html Use this command to start a supervised fine-tuning job with Megatron-SWIFT. Ensure weight-saving paths are identical across nodes. This example utilizes specific parallelism settings and optimization techniques for large models. ```bash # https://help.aliyun.com/zh/pai/user-guide/general-environment-variables # Ensure that the weight-saving paths on the two nodes are identical. NNODES=$WORLD_SIZE \ NODE_RANK=$RANK \ megatron sft \ --load Qwen3-30B-A3B-Base-mcore \ --dataset 'liucong/Chinese-DeepSeek-R1-Distill-data-110k-SFT' \ --tensor_model_parallel_size 2 \ --expert_model_parallel_size 8 \ --moe_grouped_gemm true \ --moe_shared_expert_overlap true \ --moe_aux_loss_coeff 0.01 \ --micro_batch_size 1 \ --global_batch_size 16 \ --packing true \ --recompute_granularity full \ --recompute_method uniform \ --recompute_num_layers 1 \ --train_iters 2000 \ --eval_iters 50 \ --finetune true \ --cross_entropy_loss_fusion true \ --lr 1e-5 \ --lr_warmup_iters 100 \ --min_lr 1e-6 \ --save megatron_output/Qwen3-30B-A3B-Base \ --eval_interval 200 \ --save_interval 200 \ --max_length 8192 \ --num_workers 8 \ --dataset_num_proc 8 \ --no_save_optim true \ --no_save_rng true \ --sequence_parallel true \ --use_flash_attn true ``` -------------------------------- ### Preparation Code for Function Call Example Source: https://qwen.readthedocs.io/en/latest/_sources/framework/function_call.md.txt Defines functions for getting temperature, a helper to retrieve functions by name, and the tool definitions with JSON schema. This code is used to set up the environment for the example. ```python import json def get_current_temperature(location: str, unit: str = "celsius"): """Get current temperature at a location. Args: location: The location to get the temperature for, in the format "City, State, Country". unit: The unit to return the temperature in. Defaults to "celsius". (choices: ["celsius", "fahrenheit"]) Returns: the temperature, the location, and the unit in a dict """ return { "temperature": 26.1, "location": location, "unit": unit, } def get_temperature_date(location: str, date: str, unit: str = "celsius"): """Get temperature at a location and date. Args: location: The location to get the temperature for, in the format "City, State, Country". date: The date to get the temperature for, in the format "Year-Month-Day". unit: The unit to return the temperature in. Defaults to "celsius". (choices: ["celsius", "fahrenheit"]) Returns: the temperature, the location, the date and the unit in a dict """ return { "temperature": 25.9, "location": location, "date": date, "unit": unit, } def get_function_by_name(name): if name == "get_current_temperature": return get_current_temperature if name == "get_temperature_date": return get_temperature_date TOOLS = [ { "type": "function", "function": { "name": "get_current_temperature", "description": "Get current temperature at a location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": 'The location to get the temperature for, in the format "City, State, Country".', }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": 'The unit to return the temperature in. Defaults to "celsius".', }, }, "required": ["location"], }, }, }, { "type": "function", "function": { "name": "get_temperature_date", "description": "Get temperature at a location and date.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": 'The location to get the temperature for, in the format "City, State, Country".', }, "date": { "type": "string", "description": 'The date to get the temperature for, in the format "Year-Month-Day".', }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": 'The unit to return the temperature in. Defaults to "celsius".', }, }, "required": ["location", "date"], }, }, }, ] MESSAGES = [ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}, ] ``` -------------------------------- ### Install SGLang Source: https://qwen.readthedocs.io/en/latest/deployment/sglang.html Install SGLang with all optional dependencies for a complete setup. Ensure you are using version 0.4.6.post1 or later. ```bash pip install "sglang[all]>=0.4.6.post1" ``` -------------------------------- ### Install Qwen-Agent Source: https://qwen.readthedocs.io/en/latest/_sources/framework/function_call.md.txt Install the latest version of the qwen-agent library using pip. ```bash pip install -U qwen-agent ``` -------------------------------- ### Install Unsloth Locally Source: https://qwen.readthedocs.io/en/latest/_sources/training/unsloth.md.txt Install the Unsloth library for local model training. Linux is recommended for optimal performance. ```bash pip install unsloth ``` -------------------------------- ### Full ChatML Example with Roles Source: https://qwen.readthedocs.io/en/latest/getting_started/concepts.html A complete example demonstrating the ChatML format with system, user, and assistant roles, including a meta message for the system role. ```text <|im_start|>system You are a cat.<|im_end|> <|im_start|>user hello<|im_end|> <|im_start|>assistant *Meow~* Hello there! The sun is shining so brightly today, and I'm feeling extra fluffy. Did you bring me a treat? 🐾<|im_end|> ``` -------------------------------- ### Start vLLM Server for Tool Use Source: https://qwen.readthedocs.io/en/latest/framework/function_call.html Command to start a vLLM server with specific configurations for enabling auto tool choice and parsing reasoning and tool calls. ```bash vllm serve Qwen/Qwen3-8B --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser deepseek_r1 ``` -------------------------------- ### Install OpenLLM Source: https://qwen.readthedocs.io/en/latest/_sources/deployment/openllm.rst.txt Install OpenLLM using pip. This command installs the necessary packages to run Qwen2.5 models. ```bash pip install openllm ``` -------------------------------- ### Install Langchain and FAISS Source: https://qwen.readthedocs.io/en/latest/_sources/framework/Langchain.rst.txt Installs the necessary Langchain and FAISS libraries for building the Q&A application. Ensure you have the correct versions. ```bash pip install langchain==0.0.174 pip install faiss-gpu ``` -------------------------------- ### Starting vLLM Server for Tool Calling Source: https://qwen.readthedocs.io/en/latest/_sources/framework/function_call.md.txt Command to start the vLLM server with specific configurations for auto tool choice and parsing. Ensure vLLM version is at least v0.8.5. ```bash vllm serve Qwen/Qwen3-8B --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser deepseek_r1 ``` -------------------------------- ### Launch OpenAI-API compatible service with vLLM Source: https://qwen.readthedocs.io/en/latest/_sources/quantization/awq.md.txt Run this command in your shell to start an OpenAI-compatible API service using a vLLM backend with an AWQ model. Ensure vLLM is installed and updated to at least v0.6.1 for optimal performance. ```bash vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ ``` -------------------------------- ### Install AutoAWQ Source: https://qwen.readthedocs.io/en/latest/_sources/quantization/awq.md.txt Install the AutoAWQ library, which is recommended for quantizing your own models. Note the version constraint for compatibility. ```bash pip install "autoawq<0.2.7" ``` -------------------------------- ### Serve Qwen2.5 Model Source: https://qwen.readthedocs.io/en/latest/_sources/deployment/openllm.rst.txt Start a local server to serve a specified Qwen2.5 model. The server will be accessible at http://localhost:3000/ by default. ```bash openllm serve qwen2.5:7b ``` -------------------------------- ### Qwen Chat Template Example Source: https://qwen.readthedocs.io/en/latest/_sources/getting_started/concepts.md.txt Demonstrates the Qwen chat template format using control tokens for system, user, and assistant roles, including a full conversation example. ```text <|im_start|>system You are a cat.<|im_end|> <|im_start|>user hello<|im_end|> <|im_start|>assistant *Meow~* Hello there! The sun is shining so brightly today, and I'm feeling extra fluffy. Did you bring me a treat? 🐾<|endoftext|> <|im_start|>user Explain large language models like I'm 5.<|im_end|> <|im_start|>assistant *Paws at a toy, then looks up with curious eyes* Hey there! 🐾 Imagine you have a super-smart robot friend who loves to talk and play. This robot has *gigantic* brainpower (like a million puzzle pieces all stuck together!) and knows *everything* about stories, animals, and even how to make up new words. When you ask it a question, like “What’s a rainbow?” it uses its brain to find the answer and then *tells you* it in a way that makes sense. It can even help you write a story or solve a puzzle! But here’s the magic: it’s not just a robot—it’s like a *super-duper* smart helper that learns more every day. It’s like having a friend who’s always curious and wants to help you explore the world! 🌟 *Meow~* Want to ask it something fun? 😺<|endoftext|> ``` -------------------------------- ### Install Qwen-Agent with Optional Features Source: https://qwen.readthedocs.io/en/latest/_sources/framework/qwen_agent.rst.txt Install the Qwen-Agent package from PyPI. Use optional dependencies like [gui], [rag], [code_interpreter], and [mcp] for extended functionality. ```bash pip install -U "qwen-agent[gui,rag,code_interpreter,mcp]" # Or use `pip install -U qwen-agent` for the minimal requirements. # The optional requirements, specified in double brackets, are: # [gui] for Gradio-based GUI support; # [rag] for RAG support; # [code_interpreter] for Code Interpreter support; # [mcp] for MCP support. ``` -------------------------------- ### Launch Qwen3 API with vLLM Source: https://qwen.readthedocs.io/en/latest/_sources/getting_started/quickstart.md.txt Start the API server for Qwen3-8B using vLLM, enabling reasoning and specifying the Qwen3 parser. ```shell vllm serve Qwen/Qwen3-8B --port 8000 --enable-reasoning --reasoning-parser qwen3 ``` -------------------------------- ### Install MS-SWIFT and Dependencies Source: https://qwen.readthedocs.io/en/latest/training/ms_swift.html Install the ms-swift package and its core dependencies. Optional packages for enhanced training performance and memory efficiency are also listed. ```bash pip install ms-swift -U # Install from source pip install git+https://github.com/modelscope/ms-swift.git pip install transformers -U # Optional packages pip install deepspeed # multi-GPU training pip install liger-kernel # save GPU memory resources pip install flash-attn --no-build-isolation ``` -------------------------------- ### Install Qwen-Agent Source: https://qwen.readthedocs.io/en/latest/framework/qwen_agent.html Install the stable version of Qwen-Agent from PyPI. Optional requirements like GUI, RAG, Code Interpreter, and MCP can be included. ```bash pip install -U "qwen-agent[gui,rag,code_interpreter,mcp]" # Or use `pip install -U qwen-agent` for the minimal requirements. # The optional requirements, specified in double brackets, are: # [gui] for Gradio-based GUI support; # [rag] for RAG support; # [code_interpreter] for Code Interpreter support; # [mcp] for MCP support. ``` -------------------------------- ### ChatML Example with User and Assistant Roles Source: https://qwen.readthedocs.io/en/latest/getting_started/concepts.html An example showing a user's request and the assistant's response formatted using ChatML, illustrating a conversational turn. ```text <|im_start|>user Explain large language models like I'm 5.<|im_end|> <|im_start|>assistant *Paws at a toy, then looks up with curious eyes* Hey there! 🐾 Imagine you have a super-smart robot friend who loves to talk and play. This robot has *gigantic* brainpower (like a million puzzle pieces all stuck together!) and knows *everything* about stories, animals, and even how to make up new words. When you ask it a question, like “What’s a rainbow?” it uses its brain to find the answer and then *tells you* it in a way that makes sense. It can even help you write a story or solve a puzzle! But here’s the magic: it’s not just a robot—it’s like a *super-duper* smart helper that learns more every day. It’s like having a friend who’s always curious and wants to help you explore the world! 🌟 *Meow~* Want to ask it something fun? 😺<|im_end|><|endoftext|> ``` -------------------------------- ### Install SkyPilot with Cloud Support Source: https://qwen.readthedocs.io/en/latest/deployment/skypilot.html Install the SkyPilot nightly build with support for AWS and GCP. Ensure you have pip installed. ```bash pip install "skypilot-nightly[aws,gcp]" ``` -------------------------------- ### Install Dependencies Source: https://qwen.readthedocs.io/en/latest/_sources/training/llama_factory.md.txt Install necessary packages for training, including deepspeed and flash-attn. ```bash pip install deepspeed pip install flash-attn --no-build-isolation ``` -------------------------------- ### Install AutoAWQ Source: https://qwen.readthedocs.io/en/latest/quantization/awq.html Install the AutoAWQ library. Ensure you use a version less than 0.2.7. ```bash pip install "autoawq<0.2.7" ``` -------------------------------- ### Install ModelScope Source: https://qwen.readthedocs.io/en/latest/training/unsloth.html Install the ModelScope library, which is required for loading models directly from ModelScope. ```bash pip install modelscope -qqq ``` -------------------------------- ### Install Optimum for Transformers Source: https://qwen.readthedocs.io/en/latest/quantization/gptq.html Ensure you have the necessary version of Optimum installed to use official Qwen2.5 GPTQ models with transformers. ```bash pip install -U "optimum>=1.20.0" ``` -------------------------------- ### Launch Qwen3 API with SGLang Source: https://qwen.readthedocs.io/en/latest/_sources/getting_started/quickstart.md.txt Start the API server for Qwen3-8B using SGLang, enabling the Qwen3 reasoning parser. ```shell python -m sglang.launch_server --model-path Qwen/Qwen3-8B --port 8000 --reasoning-parser qwen3 ``` -------------------------------- ### Install LlamaIndex Packages Source: https://qwen.readthedocs.io/en/latest/_sources/framework/LlamaIndex.rst.txt Install the core LlamaIndex library and necessary integrations for HuggingFace LLMs and web readers. ```bash pip install llama-index pip install llama-index-llms-huggingface pip install llama-index-readers-web ``` -------------------------------- ### Example Tool Definitions for Function Calling Source: https://qwen.readthedocs.io/en/latest/framework/function_call.html Defines two example tools, 'get_current_temperature' and 'get_temperature_date', using JSON Schema for parameters. These tools can be used by AI models to fetch temperature information. ```json [ { "type": "function", "function": { "name": "get_current_temperature", "description": "Get current temperature at a location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The location to get the temperature for, in the format \"City, State, Country\"." }, "unit": { "type": "string", "enum": [ "celsius", "fahrenheit" ], "description": "The unit to return the temperature in. Defaults to \"celsius\"." } }, "required": [ "location" ] } } }, { "type": "function", "function": { "name": "get_temperature_date", "description": "Get temperature at a location and date.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The location to get the temperature for, in the format \"City, State, Country\"." }, "date": { "type": "string", "description": "The date to get the temperature for, in the format \"Year-Month-Day\"." }, "unit": { "type": "string", "enum": [ "celsius", "fahrenheit" ], "description": "The unit to return the temperature in. Defaults to \"celsius\"." } }, "required": [ "location", "date" ] } } } ] ``` -------------------------------- ### Launch Single Instance for Qwen2.5-72B-Instruct Source: https://qwen.readthedocs.io/en/latest/deployment/skypilot.html Launches a single instance to serve Qwen2.5-72B-Instruct with a vLLM-powered OpenAI-compatible endpoint. Ensure the YAML file is updated with the correct model name before launching. ```bash sky launch -c qwen serve-72b.yaml ``` -------------------------------- ### Launch Qwen3-Instruct API with vLLM Source: https://qwen.readthedocs.io/en/latest/_sources/getting_started/quickstart.md.txt Use this command to start the API server for Qwen3-Instruct-2507 with vLLM. vLLM version 0.9.0 or higher is recommended. Adjust context length based on GPU memory. ```shell vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507 --port 8000 --tensor-parallel-size 8 --max-model-len 262144 ``` -------------------------------- ### Call OpenAI-API Service with curl Source: https://qwen.readthedocs.io/en/latest/_sources/quantization/gptq.md.txt Example of how to call the OpenAI-API compatible service started by vLLM using curl. ```bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen2.5-7B-Instruct-GPTQ-Int4", "messages": [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} ], "temperature": 0.7, "top_p": 0.8, "repetition_penalty": 1.05, "max_tokens": 512 }' ``` -------------------------------- ### Call vLLM API with curl Source: https://qwen.readthedocs.io/en/latest/_sources/quantization/awq.md.txt Example of how to call the OpenAI-compatible API service started by vLLM using curl. This demonstrates sending a chat completion request. ```bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen2.5-7B-Instruct-AWQ", "messages": [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} ], "temperature": 0.7, "top_p": 0.8, "repetition_penalty": 1.05, "max_tokens": 512 }' ``` -------------------------------- ### Start RL Training with verl (FSDP and vLLM) Source: https://qwen.readthedocs.io/en/latest/_sources/training/verl.md.txt Initiate RL training for Qwen3 models using FSDP and vLLM frameworks. This command configures various training parameters, including data paths, batch sizes, model paths, learning rates, and KL divergence settings. ```bash python3 -m verl.trainer.main_ppo \ algorithm.adv_estimator=grpo \ data.train_files=$HOME/data/gsm8k/train.parquet \ data.val_files=$HOME/data/gsm8k/test.parquet \ data.train_batch_size=1024 \ data.max_prompt_length=512 \ data.max_response_length=1024 \ data.filter_overlong_prompts=True \ data.truncation='error' \ actor_rollout_ref.model.path=Qwen/Qwen3-1.7B \ actor_rollout_ref.actor.optim.lr=1e-6 \ actor_rollout_ref.model.use_remove_padding=True \ actor_rollout_ref.actor.ppo_mini_batch_size=80 \ actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=20 \ actor_rollout_ref.actor.use_kl_loss=True \ actor_rollout_ref.actor.kl_loss_coef=0.001 \ actor_rollout_ref.actor.kl_loss_type=low_var_kl \ actor_rollout_ref.actor.entropy_coeff=0 \ actor_rollout_ref.model.enable_gradient_checkpointing=True \ actor_rollout_ref.actor.fsdp_config.param_offload=False \ actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \ actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=20 \ actor_rollout_ref.rollout.tensor_model_parallel_size=1 \ actor_rollout_ref.rollout.name=vllm \ actor_rollout_ref.rollout.gpu_memory_utilization=0.6 \ actor_rollout_ref.rollout.n=3 \ actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=20 \ actor_rollout_ref.ref.fsdp_config.param_offload=True \ algorithm.use_kl_in_reward=False \ trainer.critic_warmup=0 \ trainer.logger=['console'] \ trainer.project_name='verl_grpo_example_gsm8k' \ trainer.experiment_name='qwen3_1_7b_function_rm' \ trainer.n_gpus_per_node=1 \ trainer.nnodes=1 \ trainer.save_freq=-1 \ trainer.test_freq=5 \ trainer.total_epochs=15 $@ ``` -------------------------------- ### Clone verl Repository and Preprocess Data Source: https://qwen.readthedocs.io/en/latest/training/verl.html Clone the verl GitHub repository and preprocess the GSM8K dataset for training. Ensure you have the necessary environment configuration by following the installation guide. ```bash git clone https://github.com/volcengine/verl.git cd verl python3 examples/data_preprocess/gsm8k.py --local_dir ~/data/gsm8k ``` -------------------------------- ### Start vLLM API Service with AWQ Model Source: https://qwen.readthedocs.io/en/latest/quantization/awq.html Run this command in a shell to launch an OpenAI-compatible API service using a specified AWQ model. Ensure vLLM is installed and the model path is correct. ```shell vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ ``` -------------------------------- ### Preparation Code for Function Calling Example Source: https://qwen.readthedocs.io/en/latest/framework/function_call.html Defines functions for getting temperature information and sets up the tool definitions and initial messages for a function calling scenario. This code is used to demonstrate how a model can utilize tools to answer user queries. ```python import json def get_current_temperature(location: str, unit: str = "celsius"): """Get current temperature at a location. Args: location: The location to get the temperature for, in the format "City, State, Country". unit: The unit to return the temperature in. Defaults to "celsius". (choices: ["celsius", "fahrenheit"]) Returns: the temperature, the location, and the unit in a dict """ return { "temperature": 26.1, "location": location, "unit": unit, } def get_temperature_date(location: str, date: str, unit: str = "celsius"): """Get temperature at a location and date. Args: location: The location to get the temperature for, in the format "City, State, Country". date: The date to get the temperature for, in the format "Year-Month-Day". unit: The unit to return the temperature in. Defaults to "celsius". (choices: ["celsius", "fahrenheit"]) Returns: the temperature, the location, the date and the unit in a dict """ return { "temperature": 25.9, "location": location, "date": date, "unit": unit, } def get_function_by_name(name): if name == "get_current_temperature": return get_current_temperature if name == "get_temperature_date": return get_temperature_date TOOLS = [ { "type": "function", "function": { "name": "get_current_temperature", "description": "Get current temperature at a location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": 'The location to get the temperature for, in the format "City, State, Country".', }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": 'The unit to return the temperature in. Defaults to "celsius".', }, }, "required": ["location"], }, }, }, { "type": "function", "function": { "name": "get_temperature_date", "description": "Get temperature at a location and date.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": 'The location to get the temperature for, in the format "City, State, Country".', }, "date": { "type": "string", "description": 'The date to get the temperature for, in the format "Year-Month-Day".', }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": 'The unit to return the temperature in. Defaults to "celsius".', }, }, "required": ["location", "date"], }, }, }, ] MESSAGES = [ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}, ] ``` -------------------------------- ### Run Quantization Process Source: https://qwen.readthedocs.io/en/latest/_sources/quantization/gptq.md.txt Execute the quantization process using the prepared data. Set `cache_examples_on_gpu` based on your needs. ```python import logging logging.basicConfig( format="%(asctime)s %(levelname)s [%(name)s] %(message)s", level=logging.INFO, datefmt="%Y-%m-%d %H:%M:%S" ) model.quantize(data, cache_examples_on_gpu=False) ``` -------------------------------- ### Install MS-SWIFT Source: https://qwen.readthedocs.io/en/latest/_sources/training/ms_swift.md.txt Install the MS-SWIFT package using pip. You can install the latest stable version or from the source repository. ```shell pip install ms-swift -U ``` ```shell pip install git+https://github.com/modelscope/ms-swift.git ``` ```shell pip install transformers -U ``` ```shell pip install deepspeed # multi-GPU training ``` ```shell pip install liger-kernel # save GPU memory resources ``` ```shell pip install flash-attn --no-build-isolation ``` -------------------------------- ### Setting up a RetrievalQA Chain with Langchain Source: https://qwen.readthedocs.io/en/latest/framework/Langchain.html Demonstrates how to set up a question-answering chain using Langchain, integrating document loading, FAISS vector search, and a Qwen language model. Ensure you have the necessary libraries and models installed. ```python if __name__ == '__main__': # load docs (pdf file or txt file) filepath = 'your file path' # Embedding model name EMBEDDING_MODEL = 'text2vec' PROMPT_TEMPLATE = """Known information: {context_str} Based on the above known information, respond to the user's question concisely and professionally. If an answer cannot be derived from it, say 'The question cannot be answered with the given information' or 'Not enough relevant information has been provided,' and do not include fabricated details in the answer. Please respond in English. The question is {question}""" # Embedding running device EMBEDDING_DEVICE = "cuda" # return top-k text chunk from vector store VECTOR_SEARCH_TOP_K = 3 CHAIN_TYPE = 'stuff' embedding_model_dict = { "text2vec": "your text2vec model path", } llm = Qwen() embeddings = HuggingFaceEmbeddings(model_name=embedding_model_dict[EMBEDDING_MODEL],model_kwargs={'device': EMBEDDING_DEVICE}) docs = load_file(filepath) docsearch = FAISSWrapper.from_documents(docs, embeddings) prompt = PromptTemplate( template=PROMPT_TEMPLATE, input_variables=["context_str", "question"] ) chain_type_kwargs = {"prompt": prompt, "document_variable_name": "context_str"} qa = RetrievalQA.from_chain_type( llm=llm, chain_type=CHAIN_TYPE, retriever=docsearch.as_retriever(search_kwargs={"k": VECTOR_SEARCH_TOP_K}), chain_type_kwargs=chain_type_kwargs) query = "Give me a short introduction to large language models." print(qa.run(query)) ``` -------------------------------- ### Verify OpenLLM Installation Source: https://qwen.readthedocs.io/en/latest/_sources/deployment/openllm.rst.txt Verify the installation of OpenLLM and display its help information. This is useful for checking if the installation was successful and exploring available commands. ```bash openllm --help ``` -------------------------------- ### Develop a Qwen-Agent with Custom OpenAI-Compatible Endpoint Source: https://qwen.readthedocs.io/en/latest/framework/qwen_agent.html This example demonstrates how to configure and run a Qwen-Agent using a custom endpoint compatible with the OpenAI API, such as those provided by vLLM or SGLang. It includes defining the LLM configuration and available tools. ```python import os from qwen_agent.agents import Assistant # Define LLM llm_cfg = { # Use a custom endpoint compatible with OpenAI API by vLLM/SGLang: 'model': 'Qwen/Qwen3-32B', 'model_server': 'http://localhost:8000/v1', # api_base 'api_key': 'EMPTY', # 'generate_cfg': { # # When using vLLM/SGLang OAI API, pass the parameter of whether to enable thinking mode in this way # 'extra_body': { # 'chat_template_kwargs': {'enable_thinking': False} # }, # # # Add: When the content is `this is the thoughtthis is the answer` # # Do not add: When the response has been separated by reasoning_content and content # # This parameter will affect the parsing strategy of tool call # # 'thought_in_content': True, # }, } # llm_cfg = { # # Use the model service provided by DashScope: # 'model': 'qwen3-235b-a22b', # 'model_type': 'qwen_dashscope', # # # 'generate_cfg': { # # # When using the Dash Scope API, pass the parameter of whether to enable thinking mode in this way # # 'enable_thinking': False, # # }, # } # llm_cfg = { # # Use the OpenAI-compatible model service provided by DashScope: # 'model': 'qwen3-235b-a22b', # 'model_server': 'https://dashscope.aliyuncs.com/compatible-mode/v1', # 'api_key': os.getenv('DASHSCOPE_API_KEY'), # # # 'generate_cfg': { # # # When using Dash Scope OAI API, pass the parameter of whether to enable thinking mode in this way # # 'extra_body': { # # 'enable_thinking': False # # }, # # }, # } # Define Tools tools = [ {'mcpServers': { # You can specify the MCP configuration file 'time': { 'command': 'uvx', 'args': ['mcp-server-time', '--local-timezone=Asia/Shanghai'] }, "fetch": { "command": "uvx", "args": ["mcp-server-fetch"] } } }, 'code_interpreter', # Built-in tools ] # Define Agent bot = Assistant(llm=llm_cfg, function_list=tools) # Streaming generation messages = [{'role': 'user', 'content': 'https://qwenlm.github.io/blog/ Introduce the latest developments of Qwen'}] for responses in bot.run(messages=messages): pass print(responses) ``` -------------------------------- ### Install mlx-lm with Pip Source: https://qwen.readthedocs.io/en/latest/_sources/run_locally/mlx-lm.md.txt Install the mlx-lm package using pip for easy integration. ```bash pip install mlx-lm ``` -------------------------------- ### Install SGLang Source: https://qwen.readthedocs.io/en/latest/_sources/deployment/sglang.md.txt Install sglang with all features using pip. Ensure you are in a clean environment. ```shell pip install "sglang[all]>=0.4.6.post1" ``` -------------------------------- ### Serve Qwen3-Instruct-2507 Source: https://qwen.readthedocs.io/en/latest/_sources/getting_started/quickstart.md.txt Instructions to serve the Qwen3-235B-A22B-Instruct-2507 model using SGLang or vLLM. ```APIDOC ## Serve Qwen3-Instruct-2507 ### Description Instructions to serve the Qwen3-235B-A22B-Instruct-2507 model using SGLang or vLLM. ### SGLang ```shell python -m sglang.launch_server --model-path Qwen/Qwen3-235B-A22B-Instruct-2507 --port 8000 --tp 8 --context-length 262144 ``` ### vLLM ```shell vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507 --port 8000 --tensor-parallel-size 8 --max-model-len 262144 ``` ``` -------------------------------- ### Install mlx-lm with Conda Source: https://qwen.readthedocs.io/en/latest/_sources/run_locally/mlx-lm.md.txt Install the mlx-lm package using conda for environments managed by conda. ```bash conda install -c conda-forge mlx-lm ``` -------------------------------- ### Install vLLM Source: https://qwen.readthedocs.io/en/latest/_sources/deployment/vllm.md.txt Install vLLM using pip. Ensure you have a compatible torch and CUDA version. ```shell pip install "vllm>=0.8.5" ``` -------------------------------- ### Langchain Initialization and Document Loading Source: https://qwen.readthedocs.io/en/latest/_sources/framework/Langchain.rst.txt Sets up an LLM, embeddings, and a FAISS vector store from local documents. Configures prompt templates and search parameters for question answering. ```python if __name__ == '__main__': # load docs (pdf file or txt file) filepath = 'your file path' # Embedding model name EMBEDDING_MODEL = 'text2vec' PROMPT_TEMPLATE = """Known information: {context_str} Based on the above known information, respond to the user's question concisely and professionally. If an answer cannot be derived from it, say 'The question cannot be answered with the given information' or 'Not enough relevant information has been provided,' and do not include fabricated details in the answer. Please respond in English. The question is {question}""" # Embedding running device EMBEDDING_DEVICE = "cuda" # return top-k text chunk from vector store VECTOR_SEARCH_TOP_K = 3 CHAIN_TYPE = 'stuff' embedding_model_dict = { "text2vec": "your text2vec model path", } llm = Qwen() embeddings = HuggingFaceEmbeddings(model_name=embedding_model_dict[EMBEDDING_MODEL],model_kwargs={'device': EMBEDDING_DEVICE}) docs = load_file(filepath) docsearch = FAISSWrapper.from_documents(docs, embeddings) prompt = PromptTemplate( ``` -------------------------------- ### Develop a Qwen-Agent with Custom Tools Source: https://qwen.readthedocs.io/en/latest/_sources/framework/qwen_agent.rst.txt Define an LLM configuration and available tools to create a Qwen-Agent. This example shows how to use a custom OpenAI-compatible endpoint and integrate MCP tools along with the built-in code interpreter. ```python import os from qwen_agent.agents import Assistant # Define LLM llm_cfg = { # Use a custom endpoint compatible with OpenAI API by vLLM/SGLang: 'model': 'Qwen/Qwen3-32B', 'model_server': 'http://localhost:8000/v1', # api_base 'api_key': 'EMPTY', # 'generate_cfg': { # # When using vLLM/SGLang OAI API, pass the parameter of whether to enable thinking mode in this way # 'extra_body': { # 'chat_template_kwargs': {'enable_thinking': False} # }, # # # Add: When the content is `this is the thoughtthis is the answer` # # Do not add: When the response has been separated by reasoning_content and content # # This parameter will affect the parsing strategy of tool call # # 'thought_in_content': True, # }, } # llm_cfg = { # # Use the model service provided by DashScope: # 'model': 'qwen3-235b-a22b', # 'model_type': 'qwen_dashscope', # # # 'generate_cfg': { # # # When using the Dash Scope API, pass the parameter of whether to enable thinking mode in this way # # 'enable_thinking': False, # # }, # } # llm_cfg = { # # Use the OpenAI-compatible model service provided by DashScope: # 'model': 'qwen3-235b-a22b', # 'model_server': 'https://dashscope.aliyuncs.com/compatible-mode/v1', # 'api_key': os.getenv('DASHSCOPE_API_KEY'), # # # 'generate_cfg': { # # # When using Dash Scope OAI API, pass the parameter of whether to enable thinking mode in this way # # 'extra_body': { # # 'enable_thinking': False # # }, # # }, # } # Define Tools tools = [ {'mcpServers': { # You can specify the MCP configuration file 'time': { 'command': 'uvx', 'args': ['mcp-server-time', '--local-timezone=Asia/Shanghai'] }, "fetch": { "command": "uvx", "args": ["mcp-server-fetch"] } } }, 'code_interpreter', # Built-in tools ] # Define Agent bot = Assistant(llm=llm_cfg, function_list=tools) # Streaming generation messages = [{'role': 'user', 'content': 'https://qwenlm.github.io/blog/ Introduce the latest developments of Qwen'}] for responses in bot.run(messages=messages): pass print(responses) ``` -------------------------------- ### Install llama.cpp with Homebrew Source: https://qwen.readthedocs.io/en/latest/_sources/run_locally/llama.cpp.md.txt Installs llama-cli and llama-server using the Homebrew package manager on macOS and Linux. ```bash brew install llama.cpp ``` -------------------------------- ### Install Build Tools on macOS Source: https://qwen.readthedocs.io/en/latest/_sources/run_locally/llama.cpp.md.txt Installs necessary build tools on macOS using the xcode-select command. ```bash xcode-select --install ``` -------------------------------- ### Initializing OpenAI Client for vLLM API Source: https://qwen.readthedocs.io/en/latest/_sources/framework/function_call.md.txt Sets up the OpenAI client to connect to a locally running vLLM server. Requires specifying the API base URL and an empty API key. ```python from openai import OpenAI openai_api_key = "EMPTY" openai_api_base = "http://localhost:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) model_name = "Qwen/Qwen3-8B" ```