### Qwen2.5-Omni Quick Start and Cookbooks

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README_CN.md

Guides for getting started with Qwen2.5-Omni, including usage instructions for Transformers, ModelScope, GPTQ-Int4/AWQ, and pointers to more advanced use cases and cookbooks for further exploration.

```Markdown
## 快速开始

- [Transformers 使用方法](#--transformers-使用方法)
- [ModelScope 使用方法](#-modelscope-使用方法)
- [GPTQ-Int4 和 AWQ 使用方法](#gptq-int4-和-awq-使用方法)
- [使用提示](#使用提示)
- [更多使用用例的 Cookbooks](#更多使用用例的-cookbooks)
- [API 推理](#api-推理)
- [自定义模型设定](#自定义模型设定)

This section provides a roadmap for users to quickly start using Qwen2.5-Omni, covering various integration methods and resources for deeper dives into specific functionalities and use cases.
```

--------------------------------

### Install Qwen-Omni-Utils

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README.md

Installs the Qwen-Omni utility toolkit for handling various audio and visual inputs (base64, URLs, interleaved media). It's recommended to install with the `[decord]` feature for faster video loading, provided `ffmpeg` is installed.

```Bash
# It's highly recommended to use `[decord]` feature for faster video loading.
pip install qwen-omni-utils[decord] -U
```

--------------------------------

### Install qwen-omni-utils

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/qwen-omni-utils/README.md

Installs the qwen-omni-utils package using pip. This is the first step to use the library's functionalities.

```bash
pip install qwen-omni-utils
```

--------------------------------

### Install Dependencies

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/cookbooks/screen_recording_interaction.ipynb

Installs necessary libraries for Qwen2.5-Omni interaction, including transformers, qwen-omni-utils, openai, and flash-attn.

```python
!pip install git+https://github.com/huggingface/transformers@v4.51.3-Qwen2.5-Omni-preview
!pip install qwen-omni-utils
!pip install openai
!pip install flash-attn --no-build-isolation
```

--------------------------------

### Install Web Demo Dependencies

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README.md

Installs the necessary Python packages required to run the Qwen2.5-Omni web UI demo.

```bash
pip install -r requirements_web_demo.txt
```

--------------------------------

### Install and Run Qwen2.5-Omni with AWQ

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README_CN.md

This snippet outlines the installation steps and execution command for using the Qwen2.5-Omni-7B-AWQ model with the autoawq library. It covers installing transformers, accelerate, and autoawq, cloning the repository, changing to the low-VRAM-mode directory, and running the AWQ demo script.

```bash
pip install transformers==4.52.3
pip install accelerate
pip install autoawq==0.2.9

git clone https://github.com/QwenLM/Qwen2.5-Omni.git

cd Qwen2.5-Omni/low-VRAM-mode/

CUDA_VISIBLE_DEVICES=0 python3 low_VRAM_demo_awq.py
```

--------------------------------

### Install Web Demo Dependencies

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README_CN.md

This bash command installs the necessary Python dependencies required to run the local web-based demonstration UI for the Qwen2.5-Omni model. Ensure you have the `requirements_web_demo.txt` file in your current directory.

```bash
pip install -r requirements_web_demo.txt
```

--------------------------------

### Install and Run Qwen2.5-Omni with GPTQ-Int4

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README_CN.md

This snippet details the necessary installations and commands to run the Qwen2.5-Omni-7B-GPTQ-Int4 model using the gptqmodel library. It includes installing transformers, accelerate, gptqmodel, and numpy, cloning the repository, navigating to the correct directory, and executing the demo script.

```bash
pip install transformers==4.52.3
pip install accelerate
pip install gptqmodel==2.0.0
pip install numpy==2.0.0

git clone https://github.com/QwenLM/Qwen2.5-Omni.git

cd Qwen2.5-Omni/low-VRAM-mode/

CUDA_VISIBLE_DEVICES=0 python3 low_VRAM_demo_gptq.py
```

--------------------------------

### vLLM Installation for Qwen2.5-Omni

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README.md

Installs the vLLM library from a specific branch to ensure compatibility and support for Qwen2.5-Omni. This includes cloning the repository, checking out a specific commit, and installing dependencies.

```bash
git clone -b qwen2_omni_public https://github.com/fyabc/vllm.git
cd vllm
git checkout de8f43fbe9428b14d31ac5ec45d065cd3e5c3ee0
pip install setuptools_scm torchdiffeq resampy x_transformers qwen-omni-utils accelerate
pip install -r requirements/cuda.txt
pip install --upgrade setuptools wheel
pip install .
pip install transformers==4.52.3
```

--------------------------------

### vLLM Installation for Qwen2.5-Omni

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README_CN.md

This sequence of bash commands installs the vLLM library with specific support for Qwen2.5-Omni. It involves cloning a custom branch of vLLM, checking out a specific commit, installing necessary utilities and dependencies, and finally installing vLLM itself along with a specific version of the transformers library.

```bash
git clone -b qwen2_omni_public https://github.com/fyabc/vllm.git
cd vllm
git checkout de8f43fbe9428b14d31ac5ec45d065cd3e5c3ee0
pip install setuptools_scm torchdiffeq resampy x_transformers qwen-omni-utils accelerate
pip install -r requirements/cuda.txt
pip install --upgrade setuptools wheel
pip install .
pip install transformers==4.52.3
```

--------------------------------

### vLLM Local Inference Example

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README.md

Provides an example command to perform local inference with Qwen2.5-Omni using vLLM. This specific example is for text output on a single GPU and utilizes a particular prompt configuration.

```bash
# git clone -b qwen2_omni_public https://github.com/fyabc/vllm.git
# cd vllm
# git checkout de8f43fbe9428b14d31ac5ec45d065cd3e5c3ee0
# cd examples/offline_inference/qwen2_5_omni/

# only text output for single GPU
python end2end.py --model Qwen/Qwen2.5-Omni-7B --prompt audio-in-video-v2 --enforce-eager --thinker-only
```

--------------------------------

### Install OpenAI Library

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README_CN.md

Installs the OpenAI Python client library, which is used for interacting with the Qwen2.5-Omni API.

```bash
pip install openai
```

--------------------------------

### Install Dependencies

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/cookbooks/video_information_extracting.ipynb

Installs necessary libraries for using Qwen2.5-Omni, including transformers, qwen-omni-utils, openai, and flash-attn.

```python
!pip install git+https://github.com/huggingface/transformers@v4.51.3-Qwen2.5-Omni-preview
!pip install qwen-omni-utils
!pip install openai
!pip install flash-attn --no-build-isolation
```

--------------------------------

### Install Qwen2.5-Omni Dependencies

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README_CN.md

Installs the necessary Hugging Face Transformers library and accelerate for efficient model loading and inference. It also provides a command to install the qwen-omni-utils package with optional support for faster video processing using decord.

```bash
pip install transformers==4.52.3
pip install accelerate
```

```bash
# Very recommended to use `[decord]` feature for faster video reading speed
pip install qwen-omni-utils[decord] -U
```

```bash
pip install qwen-omni-utils -U
```

--------------------------------

### vLLM Serving for Qwen2.5-Omni

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README_CN.md

Starts a vLLM server for the Qwen2.5-Omni model, configured for multi-GPU usage (example with 4 GPUs). Specifies the model path, port, host, data type, and tensor parallelism.

```bash
vllm serve /path/to/Qwen2.5-Omni-7B/ --port 8000 --host 127.0.0.1 --dtype bfloat16 -tp 4
```

--------------------------------

### Install Flash Attention 2

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README.md

Installs the latest version of Flash Attention 2, a prerequisite for using it to speed up model generation. Requires compatible hardware and PyTorch versions.

```bash
pip install -U flash-attn --no-build-isolation
```

--------------------------------

### vLLM Offline Inference Examples

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README_CN.md

These bash commands demonstrate end-to-end offline inference using vLLM with Qwen2.5-Omni. They cover scenarios for text-only output on single and multi-GPU setups, and audio output with specified voice types and output directories on single and multi-GPU configurations. The `--enforce-eager` flag is used for eager execution.

```bash
# only text output for single GPU
python end2end.py --model Qwen/Qwen2.5-Omni-7B --prompt audio-in-video-v2 --enforce-eager --thinker-only
```

```bash
# only text output for multi GPUs (example in 4 GPUs)
python end2end.py --model Qwen/Qwen2.5-Omni-7B --prompt audio-in-video-v2 --enforce-eager --thinker-only --thinker-devices [0,1,2,3] --thinker-gpu-memory-utilization 0.9
```

```bash
# audio output for single GPU
python end2end.py --model Qwen/Qwen2.5-Omni-7B --prompt audio-in-video-v2 --enforce-eager --do-wave --voice-type Chelsie --warmup-voice-type Chelsie --output-dir output_wav
```

```bash
# audio output for multi GPUs (example in 4 GPUs)
python end2end.py --model Qwen/Qwen2.5-Omni-7B --prompt audio-in-video-v2 --enforce-eager --do-wave --voice-type Chelsie --warmup-voice-type Chelsie --thinker-devices [0,1] --talker-devices [2] --code2wav-devices [3] --thinker-gpu-memory-utilization 0.9 --talker-gpu-memory-utilization 0.9 --output-dir output_wav
```

--------------------------------

### Multimodal Input Examples for Qwen2.5 Omni

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/qwen-omni-utils/README.md

Demonstrates how to structure messages for multimodal inputs, including images and videos. Supports local file paths, URLs, base64 encoded data, PIL Image objects, and dynamic resizing/frame rate adjustments.

```python
messages = [
    # Image
    ## Local file path
    [{"role": "user", "content": [{"type": "image", "image": "file:///path/to/your/image.jpg"}, {"type": "text", "text": "Describe this image."}]}],
    ## Image URL
    [{"role": "user", "content": [{"type": "image", "image": "http://path/to/your/image.jpg"}, {"type": "text", "text": "Describe this image."}]}],
    ## Base64 encoded image
    [{"role": "user", "content": [{"type": "image", "image": "data:image;base64,/9j/..."}, {"type": "text", "text": "Describe this image."}]}],
    ## PIL.Image.Image
    [{"role": "user", "content": [{"type": "image", "image": pil_image}, {"type": "text", "text": "Describe this image."}]}],
    ## Model dynamically adjusts image size, specify dimensions if required.
    [{"role": "user", "content": [{"type": "image", "image": "file:///path/to/your/image.jpg", "resized_height": 280, "resized_width": 420}, {"type": "text", "text": "Describe this image."}]}],
    # Video
    ## Local video path
    [{"role": "user", "content": [{"type": "video", "video": "file:///path/to/video1.mp4"}, {"type": "text", "text": "Describe this video."}]}],
    ## Local video frames
    [{"role": "user", "content": [{"type": "video", "video": ["file:///path/to/extracted_frame1.jpg", "file:///path/to/extracted_frame2.jpg", "file:///path/to/extracted_frame3.jpg"],}, {"type": "text", "text": "Describe this video."},],}],
    ## Model dynamically adjusts video nframes, video height and width. specify args if required.
    [{"role": "user", "content": [{"type": "video", "video": "file:///path/to/video1.mp4", "fps": 2.0, "resized_height": 280, "resized_width": 280}, {"type": "text", "text": "Describe this video."}]}],
]
```

--------------------------------

### Install Flash Attention 2

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README_CN.md

Installs the latest version of Flash Attention 2, a requirement for accelerating model generation. Ensure your hardware is compatible and refer to the official documentation for more details.

```bash
pip install -U flash-attn --no-build-isolation
```

--------------------------------

### Qwen2.5-Omni End-to-End Script Examples

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README.md

Examples of using the end2end.py script for Qwen2.5-Omni. Demonstrates configurations for multi-GPU text-only output, single-GPU audio output, and multi-GPU audio output with specific device assignments.

```python
python end2end.py --model Qwen/Qwen2.5-Omni-7B --prompt audio-in-video-v2 --enforce-eager --thinker-only --thinker-devices [0,1,2,3] --thinker-gpu-memory-utilization 0.9
```

```python
python end2end.py --model Qwen/Qwen2.5-Omni-7B --prompt audio-in-video-v2 --enforce-eager --do-wave --voice-type Chelsie --warmup-voice-type Chelsie --output-dir output_wav
```

```python
python end2end.py --model Qwen/Qwen2.5-Omni-7B --prompt audio-in-video-v2 --enforce-eager --do-wave --voice-type Chelsie --warmup-voice-type Chelsie --thinker-devices [0,1] --talker-devices [2] --code2wav-devices [3] --thinker-gpu-memory-utilization 0.9 --talker-gpu-memory-utilization 0.9 --output-dir output_wav
```

--------------------------------

### Qwen2.5-Omni API Interaction Example

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README_CN.md

Demonstrates how to interact with the Qwen2.5-Omni model using the OpenAI API. This example shows how to send a video URL and receive text and audio responses. It requires the `openai` library and specifies audio output parameters.

```python
import base64
import numpy as np
import soundfile as sf

from openai import OpenAI

client = OpenAI(
    api_key="your_api_key",
    base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
)

messages = [
    {
        "role": "system",
        "content": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech.",
    },
    {
        "role": "user",
        "content": [
            {"type": "video_url", "video_url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2.5-Omni/draw.mp4"},
        ],
    },
]

# Qwen-Omni only supports stream mode
completion = client.chat.completions.create(
    model="qwen-omni-turbo",
    messages=messages,
    modalities=["text", "audio"],
    audio={
        "voice": "Cherry", # Cherry, Ethan, Serena, Chelsie is available
        "format": "wav"
    },
    stream=True,
    stream_options={"include_usage": True}
)

text = []
audio_string = ""
for chunk in completion:
    if chunk.choices:
        if hasattr(chunk.choices[0].delta, "audio"):
            try:
                audio_string += chunk.choices[0].delta.audio["data"]
            except Exception as e:
                text.append(chunk.choices[0].delta.audio["transcript"])
    else:
        print(chunk.usage)

print("".join(text))
wav_bytes = base64.b64decode(audio_string)
wav_array = np.frombuffer(wav_bytes, dtype=np.int16)
sf.write("output.wav", wav_array, samplerate=24000)
```

--------------------------------

### vLLM Serve for Qwen2.5-Omni

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README.md

Instructions for starting the vLLM server for Qwen2.5-Omni models. Supports single and multi-GPU configurations. Note that vLLM serve for Qwen2.5-Omni currently only supports 'thinker' and text output.

```bash
# for single GPU
vllm serve /path/to/Qwen2.5-Omni-7B/ --port 8000 --host 127.0.0.1 --dtype bfloat16
```

```bash
# for multi GPUs (example in 4 GPUs)
vllm serve /path/to/Qwen2.5-Omni-7B/ --port 8000 --host 127.0.0.1 --dtype bfloat16 -tp 4
```

--------------------------------

### Batch Inference Examples

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README.md

Provides examples of preparing conversations for batch inference, including scenarios with video only, audio only, and pure text inputs. This helps in understanding how to structure different types of multimodal data for processing.

```Python
# Sample messages for batch inference

# Conversation with video only
conversation1 = [
    {
        "role": "system",
        "content": [
            {"type": "text", "text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."}
        ],
    },
    {
        "role": "user",
        "content": [
            {"type": "video", "video": "/path/to/video.mp4"},
        ]
    }
]

# Conversation with audio only
conversation2 = [
    {
        "role": "system",
        "content": [
            {"type": "text", "text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."}
        ],
    },
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": "/path/to/audio.wav"},
        ]
    }
]

# Conversation with pure text
conversation3 = [
    {
        "role": "system",
        "content": [
            {"type": "text", "text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."}
        ],
    },
    {
        "role": "user",
        "content": "who are you?"
    }
]
```

--------------------------------

### Install Dependencies for AWQ

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README.md

Installs the required Python libraries and clones the Qwen2.5-Omni repository for using the AWQ quantized model. Specifies versions for transformers and autoawq.

```shell
pip install transformers==4.52.3
pip install accelerate
pip install autoawq==0.2.9

git clone https://github.com/QwenLM/Qwen2.5-Omni.git

cd Qwen2.5-Omni/low-VRAM-mode/
```

--------------------------------

### Qwen2.5-Omni API Inference Example

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README.md

Demonstrates how to use the OpenAI API to interact with Qwen2.5-Omni for multimodal tasks. This example shows how to send a video URL and receive text and audio output, utilizing streaming for real-time processing. It requires the 'openai' and 'soundfile' libraries.

```bash
pip install openai
```

```python
import base64
import numpy as np
import soundfile as sf

from openai import OpenAI

client = OpenAI(
    api_key="your_api_key",
    base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
)

messages = [
    {
        "role": "system",
        "content": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech.",
    },
    {
        "role": "user",
        "content": [
            {"type": "video_url", "video_url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2.5-Omni/draw.mp4"},
        ],
    },
]

# Qwen-Omni only supports stream mode
completion = client.chat.completions.create(
    model="qwen-omni-turbo",
    messages=messages,
    modalities=["text", "audio"],
    audio={
        "voice": "Cherry", # Cherry, Ethan, Serena, Chelsie is available
        "format": "wav"
    },
    stream=True,
    stream_options={"include_usage": True}
)

text = []
audio_string = ""
for chunk in completion:
    if chunk.choices:
        if hasattr(chunk.choices[0].delta, "audio"):
            try:
                audio_string += chunk.choices[0].delta.audio["data"]
            except Exception as e:
                text.append(chunk.choices[0].delta.audio["transcript"])
    else:
        print(chunk.usage)

print("".join(text))
wav_bytes = base64.b64decode(audio_string)
wav_array = np.frombuffer(wav_bytes, dtype=np.int16)
sf.write("output.wav", wav_array, samplerate=24000)
```

--------------------------------

### Install Transformers and Accelerate

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README.md

Installs the necessary Hugging Face libraries for using Qwen2.5-Omni models. It's recommended to use specific versions to avoid potential errors like 'KeyError: 'qwen2_5_omni''.

```Bash
pip install transformers==4.52.3
pip install accelerate
```

--------------------------------

### Qwen2.5-Omni: OCR on Video

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/cookbooks/screen_recording_interaction.ipynb

Shows how to perform Optical Character Recognition (OCR) on a video using Qwen2.5-Omni. The example prompts the model to identify authors from a paper shown in the video.

```python
video_path = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2.5-Omni/screen.mp4"
prompt = "Who is the authors of this paper?"

display(Video(video_path, width=640, height=360))

## Use a local HuggingFace model to inference.
response = inference(video_path, prompt=prompt, sys_prompt="You are a helpful assistant.")
print(response[0])
```

--------------------------------

### Docker Run Command

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README.md

Command to run the Qwen2.5-Omni Docker image. This command mounts all GPUs, uses host IPC and network, and starts an interactive bash session within the container.

```bash
docker run --gpus all --ipc=host --network=host --rm --name qwen2.5-omni -it qwenllm/qwen-omni:2.5-cu121 bash
```

--------------------------------

### vLLM Serve API Service

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README_CN.md

This bash command starts an API service using vLLM for Qwen2.5-Omni. It specifies the model path, port, host, and data type (bfloat16). Note that vLLM serve currently only supports text output.

```bash
# for single GPU
vllm serve /path/to/Qwen2.5-Omni-7B/ --port 8000 --host 127.0.0.1 --dtype bfloat16
```

--------------------------------

### Transformers Usage

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README_CN.md

Guide for using Qwen2.5-Omni with the Transformers library. This section provides necessary steps and code examples for integrating the model into your projects.

```Python
# Example usage with Transformers (assuming model is downloaded)
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Qwen/Qwen2.5-Omni-7B"

token = "YOUR_HF_TOKEN" # Replace with your Hugging Face token if needed

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True, token=token)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, token=token)

# Prepare input (example with text and image)
# Note: Actual image handling would involve PIL or similar libraries
inputs = tokenizer.apply_chat_template([
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "Describe this image: <image>"}
], tokenize=False, add_generation_prompt=True)

# For image input, you would typically pass it as a separate argument or encoded string
# This is a placeholder for how you might structure the input
# For actual implementation, refer to the Transformers documentation for multimodal inputs

# Example of generating text response (simplified)
# messages = [
#     {"role": "user", "content": "Hello, who are you?"}
# ]
# prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
# inputs = tokenizer(prompt, return_tensors="pt")

# outputs = model.generate(**inputs, max_new_tokens=200)
# response = tokenizer.decode(outputs[0], skip_special_tokens=True)
# print(response)

```

--------------------------------

### Docker Deployment for Qwen2.5-Omni

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README_CN.md

Instructions for deploying Qwen2.5-Omni using Docker. Includes commands to run the pre-built environment and launch the web demo, with options for enabling FlashAttention-2.

```bash
docker run --gpus all --ipc=host --network=host --rm --name qwen2.5-omni -it qwenllm/qwen-omni:2.5-cu121 bash
```

```bash
bash docker/docker_web_demo.sh --checkpoint /path/to/Qwen2.5-Omni-7B
```

```bash
bash docker/docker_web_demo.sh --checkpoint /path/to/Qwen2.5-Omni-7B --flash-attn2
```

--------------------------------

### Launch Web Demo with FlashAttention-2

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README_CN.md

These bash commands launch the local web demo for Qwen2.5-Omni, with an option to enable FlashAttention-2 for improved performance, especially with large amounts of image and video data. The first command is for the 7B model, and the second specifies the 3B model.

```bash
# default for Qwen2.5-Omni-7B
python web_demo.py --flash-attn2
```

```bash
# for Qwen2.5-Omni-3B
python web_demo.py --flash-attn2 -c Qwen/Qwen2.5-Omni-3B
```

--------------------------------

### Docker Web Demo Launch

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README.md

Commands to launch the web demo using the provided Docker script. Includes options for specifying the checkpoint path and enabling FlashAttention-2.

```bash
bash docker/docker_web_demo.sh --checkpoint /path/to/Qwen2.5-Omni-7B
```

```bash
bash docker/docker_web_demo.sh --checkpoint /path/to/Qwen2.5-Omni-7B --flash-attn2
```

--------------------------------

### Multimodal Input Preparation for Qwen2.5 Omni

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/qwen-omni-utils/README.md

Shows how to structure message lists for the Qwen2.5 Omni model, including various ways to input images (local path, URL, base64, PIL Image) and videos (local path, extracted frames). It also demonstrates how to specify dimensions and frame rates for media.

```python
messages = [
    # Image
    ## Local file path
    [{"role": "user", "content": [{"type": "image", "image": "file:///path/to/your/image.jpg"}, {"type": "text", "text": "Describe this image."}]}],
    ## Image URL
    [{"role": "user", "content": [{"type": "image", "image": "http://path/to/your/image.jpg"}, {"type": "text", "text": "Describe this image."}]}],
    ## Base64 encoded image
    [{"role": "user", "content": [{"type": "image", "image": "data:image;base64,/9j/..."}, {"type": "text", "text": "Describe this image."}]}],
    ## PIL.Image.Image
    [{"role": "user", "content": [{"type": "image", "image": pil_image}, {"type": "text", "text": "Describe this image."}]}],
    ## Model dynamically adjusts image size, specify dimensions if required.
    [{"role": "user", "content": [{"type": "image", "image": "file:///path/to/your/image.jpg", "resized_height": 280, "resized_width": 420}, {"type": "text", "text": "Describe this image."}]}],
    # Video
    ## Local video path
    [{"role": "user", "content": [{"type": "video", "video": "file:///path/to/video1.mp4"}, {"type": "text", "text": "Describe this video."}]}],
    ## Local video frames
    [{"role": "user", "content": [{"type": "video", "video": ["file:///path/to/extracted_frame1.jpg", "file:///path/to/extracted_frame2.jpg", "file:///path/to/extracted_frame3.jpg"],}, {"type": "text", "text": "Describe this video."},],}],
    ## Model dynamically adjusts video nframes, video height and width. specify args if required.
    [{"role": "user", "content": [{"type": "video", "video": "file:///path/to/video1.mp4", "fps": 2.0, "resized_height": 280, "resized_width": 280}, {"type": "text", "text": "Describe this video."}]}],
]

```

--------------------------------

### Run Web Demo (Default)

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README.md

Launches the Qwen2.5-Omni web UI demo using the standard attention implementation. This is the default behavior if FlashAttention-2 is not specified.

```bash
# default for Qwen2.5-Omni-7B
python web_demo.py
```

```bash
# for Qwen2.5-Omni-3B
python web_demo.py -c Qwen/Qwen2.5-Omni-3B
```

--------------------------------

### Launch Local Web UI Demo

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README.md

Instructions for launching a local web-based user interface to interact with the Qwen2.5-Omni model. This typically involves running a script in a terminal.

```shell
# Example command to launch the local web UI
# python web_demo.py --model "Qwen/Qwen2.5-Omni-1.1B"
```

--------------------------------

### Run Web Demo with FlashAttention-2

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README.md

Launches the Qwen2.5-Omni web UI demo with FlashAttention-2 enabled for enhanced performance, particularly for multi-image and video processing. This command is recommended for optimal efficiency.

```bash
# default for Qwen2.5-Omni-7B
python web_demo.py --flash-attn2
```

```bash
# for Qwen2.5-Omni-3B
python web_demo.py --flash-attn2 -c Qwen/Qwen2.5-Omni-3B
```

--------------------------------

### Qwen2.5 Omni Model Initialization and Inference

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/qwen-omni-utils/README.md

Initializes the Qwen2.5 VL model and processor, prepares input messages, processes vision information, and performs inference. Includes setting environment variables for video processing.

```python
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_omni_utils import process_vision_info


# You can set the maximum tokens for a video through the environment variable VIDEO_MAX_PIXELS
# based on the maximum tokens that the model can accept.
# export VIDEO_MAX_PIXELS = 32000 * 28 * 28 * 0.9

model_path = "/path/to/your/qwen2.5-omni/model"
processor = AutoProcessor.from_pretrained(model_path)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(model_path, torch_dtype="auto", device_map="auto")

# Assuming 'messages' is defined as in the previous example
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
images, videos = process_vision_info(messages)
inputs = processor(text=text, images=images, videos=videos, padding=True, return_tensors="pt")

print(inputs)
generated_ids = model.generate(**inputs)
print(generated_ids)
```

--------------------------------

### Interacting with Qwen2.5-Omni

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README_CN.md

Information on how to interact with Qwen2.5-Omni, including options for online demos, launching local web demos, and real-time interaction experiences.

```Markdown
## 和 Qwen2.5-Omni 对话

- [在线演示](#在线演示)
- [启动本地网页演示](#启动本地网页演示)
- [实时交互](#实时交互)

This section guides users on how to engage with Qwen2.5-Omni through various interactive platforms, including live online demos and local setups.
```

--------------------------------

### Install Dependencies for GPTQ-Int4

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README.md

Installs the necessary Python libraries and clones the Qwen2.5-Omni repository for using the GPTQ-Int4 quantized model. Ensures compatibility with specific versions of transformers and gptqmodel.

```shell
pip install transformers==4.52.3
pip install accelerate
pip install gptqmodel==2.0.0
pip install numpy==2.0.0

git clone https://github.com/QwenLM/Qwen2.5-Omni.git

cd Qwen2.5-Omni/low-VRAM-mode/
```

--------------------------------

### vLLM Chat API Example (curl)

Source: https://github.com/qwenlm/qwen2.5-omni/blob/main/README.md

Example of how to interact with the vLLM chat API using curl. Demonstrates sending a request with both image and audio inputs, along with text.

```json
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "https://modelscope.oss-cn-beijing.aliyuncs.com/resource/qwen.png"}},
        {"type": "audio_url", "audio_url": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2.5-Omni/cough.wav"}},
        {"type": "text", "text": "What is the text in the illustrate ans what it the sound in the audio?"}
    ]}
    ]
    }'
```