### Installation Rules for Executables and Libraries

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/multimodal_model_demo/deploy/CMakeLists.txt

Configures the installation prefix based on the source directory and target system architecture. It then installs the built executables ('imgenc', demo executable) and runtime libraries ('librknnrt.so', 'librkllmrt.so') to their respective destinations.

```cmake
# Install the executable file to the specified directory
set(CMAKE_INSTALL_PREFIX ${CMAKE_SOURCE_DIR}/install/demo_${CMAKE_SYSTEM_NAME}_${TARGET_LIB_ARCH})
install(TARGETS imgenc ${PROJECT_NAME} DESTINATION ./) 
install(PROGRAMS ${LIBRKNNRT} ${LIBRKLLMRT} DESTINATION lib)
install(PROGRAMS ../data/demo.jpg DESTINATION ./) 
```

--------------------------------

### RKLLMClient Quick Start for Function Calling

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_server_demo/README.md

Demonstrates using RKLLMClient for function calling. This involves parsing tool calls, executing them, and synthesizing a final answer.

```python
from chat_api_flask import RKLLMClient, TOOLS, parse_tool_calls, execute_tool_calls

client = RKLLMClient(base_url="http://x.x.x.x:8080")

messages = [
    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\nCurrent Date: 2024-09-30"},
    {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow?"},
]

# Step 1: Model returns which tools to call and with what arguments
resp = client.chat(messages=messages, tools=TOOLS, stream=False)
tool_calls = parse_tool_calls(resp["choices"][0]["message"]["content"])

# Step 2: Execute tool calls and append results to messages
assistant_msg, tool_msgs = execute_tool_calls(tool_calls)
messages.append(assistant_msg)
messages.extend(tool_msgs)

# Step 3: Model synthesizes the final answer from tool results
resp = client.chat(messages=messages, tools=None, stream=False)
print("A:", resp["choices"][0]["message"]["content"])
```

--------------------------------

### Compile and Build C++ Demo

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/multimodal_model_demo/README.md

Scripts to compile the C++ example code for Linux and Android. Ensure the cross-compiler path is correctly set. This generates an executable demo in the deploy directory.

```bash
cd deploy
# for linux
./build-linux.sh
# for android
./build-android.sh
# push install dir to device
adb push ./install/demo_Linux_aarch64 /data
# push model file to device
adb push qwen2-vl-vision_rk3588.rknn /data/models
adb push qwen2-vl-llm_rk3588.rkllm /data/models
```

--------------------------------

### LLM Demo Interaction Example

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_api_demo/README.md

Example of interacting with the LLM demo by providing a question number or custom input.

```text
user:
```
example 1 （DeepSeek-R1-Distill-Qwen-1.5B_W8A8_RK3588.rkllm）

```
user: 0
现有一笼子，里面有鸡和兔子若干只，数一数，共有头14个，腿38条，求鸡和兔子各有多少只？
robot: <think>
首先，设鸡的数量为x，兔子的数量为y。

根据题目中的条件，我们知道：

1. 鸡和兔子的总数是14，因此有方程：
   x + y = 14

2. 鸡有两条腿，兔子有四条腿，总腿数是38，所以有另一个方程：
   2x + 4y = 38

接下来，通过代入法或消元法来解这两个方程。假设我们用代入法：

从第一个方程中，可以得到：
x = 14 - y

将这个表达式代入第二个方程：
2(14 - y) + 4y = 38
展开计算后得到：
28 - 2y + 4y = 38
合并同类项：
2y = 10
解得：
y = 5

然后，将y的值代入x = 14 - y中：
x = 14 - 5 = 9

因此，鸡有9只，兔子有5只。
</think>

要解决这个问题，我们可以设鸡的数量为 \( x \)，兔子的数量为 \( y \)。根据题目给出的条件：

1. **头的总数**：每只鸡和兔子都有一个头，所以：
   \[
   x + y = 14
   \]

2. **腿的总数**：鸡有两条腿，兔子有四条腿，总腿数为38条，因此：
   \[
   2x + 4y = 38
   \]

接下来，我们可以通过解这两个方程来找到 \( x \) 和 \( y \) 的值。

**步骤一：简化第二个方程**

将第二个方程两边同时除以2：
\[
x + 2y = 19
\]

现在，我们有两个方程：
\[
\begin{cases}
x + y = 14 \\
x + 2y = 19
\end{cases}
\]

**步骤二：消元法**

用第二个方程减去第一个方程：
\[
(x + 2y) - (x + y) = 19 - 14 \\
y = 5
\]

**步骤三：代入求 \( x \)**

将 \( y = 5 \) 代入第一个方程：
\[
x + 5 = 14 \\
x = 14 - 5 \\
x = 9
\]

因此，鸡的数量是 **9只**，兔子的数量是 **5只**。

**最终答案：**

鸡有 \(\boxed{9}\) 只，兔子有 \(\boxed{5}\) 只。
```

example 2 （DeepSeek-R1-Distill-Qwen-1.5B_W8A8_RK3588.rkllm）

```

user: 1
有28位小朋友排成一行,从左边开始数第10位是学豆,从右边开始数他是第几位?
robot: <think>
首先，总共有28位小朋友。

从左边开始数，第10位是学豆的位置。

因此，从右边开始数，学豆的位置是从右边数的第(28 - 10 + 1) = 第19位。
</think>

**解答：**

我们有28位小朋友排成一行。题目要求确定从右边开始数时，第10位是学豆的位置。

**步骤如下：**

1. **总人数**：共有28位小朋友。
2. **左边数的顺序**：从左边开始数，第10位是学豆。
3. **右边数的计算**：
   - 从右边数时，第1位对应左边数的第28位。
   - 因此，第n位在左边对应的是第(28 - n + 1)位在右边。

4. **具体计算**：
   \[
   第10位在左边 = 第(28 - 10 + 1) = 第19位在右边
   \]

**最终答案：**

\boxed{19}
```
```

--------------------------------

### InternVL3 Image Token Configuration Example

Source: https://github.com/airockchip/rknn-llm/blob/main/README.md

This Python code snippet shows how to define image start, end, and context tokens within the chat function for the InternVL3 model.

```python
def chat(self, tokenizer, pixel_values, question, generation_config, history=None, return_history=False,
            num_patches_list=None, IMG_START_TOKEN='<img>', IMG_END_TOKEN='</img>', IMG_CONTEXT_TOKEN='<IMG_CONTEXT>',
            verbose=False):
   
```

--------------------------------

### Compile and Build C++ Demo

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_api_demo/Readme.md

Scripts to compile the C++ LLM demo for Linux or Android. After compilation, the install directory will contain executables and libraries.

```bash
cd deploy
# for linux
./build-linux.sh
# for android
./build-android.sh
```

--------------------------------

### Example 1: Chicken and Rabbit Problem

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_api_demo/Readme.md

Solves a classic word problem involving chickens and rabbits with a given number of heads and legs. This example demonstrates basic arithmetic and algebraic equation solving.

```python
user:
```
example 1 （DeepSeek-R1-Distill-Qwen-1.5B_W8A8_RK3588.rkllm）

```
user: 0
现有一笼子，里面有鸡和兔子若干只，数一数，共有头14个，腿38条，求鸡和兔子各有多少只？
robot: <think>
首先，设鸡的数量为x，兔子的数量为y。

根据题目中的条件，我们知道：

1. 鸡和兔子的总数是14，因此有方程：
   x + y = 14

2. 鸡有两条腿，兔子有四条腿，总腿数是38，所以有另一个方程：
   2x + 4y = 38

接下来，通过代入法或消元法来解这两个方程。假设我们用代入法：

从第一个方程中，可以得到：
x = 14 - y

将这个表达式代入第二个方程：
2(14 - y) + 4y = 38
展开计算后得到：
28 - 2y + 4y = 38
合并同类项：
2y = 10
解得：
y = 5

然后，将y的值代入x = 14 - y中：
x = 14 - 5 = 9

因此，鸡有9只，兔子有5只。
</think>

要解决这个问题，我们可以设鸡的数量为 \( x \)，兔子的数量为 \( y \)。根据题目给出的条件：

1. **头的总数**：每只鸡和兔子都有一个头，所以：
   [ \nx + y = 14 \n]

2. **腿的总数**：鸡有两条腿，兔子有四条腿，总腿数为38条，因此：
   [ \n2x + 4y = 38 \n]

接下来，我们可以通过解这两个方程来找到 \( x \) 和 \( y \) 的值。

**步骤一：简化第二个方程**

将第二个方程两边同时除以2：
[ \nx + 2y = 19 \n]

现在，我们有两个方程：
[ \n\begin{cases}
   x + y = 14 \\
   x + 2y = 19
\end{cases}
]

**步骤二：消元法**

用第二个方程减去第一个方程：
[ \n(x + 2y) - (x + y) = 19 - 14 \\
   y = 5
]

**步骤三：代入求 \( x \)**

将 \( y = 5 \) 代入第一个方程：
[ \nx + 5 = 14 \\
   x = 14 - 5 \\
   x = 9
]

因此，鸡的数量是 **9只**，兔子的数量是 **5只**。

**最终答案：**

鸡有 \(\boxed{9}\) 只，兔子有 \(\boxed{5}\) 只。

```

--------------------------------

### Basic CMake Project Setup

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/multimodal_model_demo/deploy/CMakeLists.txt

Sets the minimum CMake version, project name, and defines C++ compiler flags for the project. It also includes conditional link options for specific CMake versions and Linux systems.

```cmake
cmake_minimum_required(VERSION 3.15.1)

project(demo)

set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wl,--allow-shlib-undefined")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11 -Wl,--allow-shlib-undefined")

if(${CMAKE_VERSION} VERSION_GREATER "3.15.0" AND CMAKE_SYSTEM_NAME STREQUAL "Linux")
  add_link_options("-Wl,-Bsymbolic")
endif()
```

--------------------------------

### RKLLMClient Quick Start for Regular Chat

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_server_demo/README.md

Use the RKLLMClient wrapper for simple chat interactions. Supports both non-streaming and streaming responses.

```python
from chat_api_flask import RKLLMClient

client = RKLLMClient(base_url="http://x.x.x.x:8080")

# Non-streaming
reply = client.chat_simple("Hello, introduce yourself please.")
print(reply)

# Streaming
for chunk in client.chat(messages=[{"role": "user", "content": "Hello"}], stream=True):
    print(chunk["content"], end="", flush=True)
```

--------------------------------

### CMakeLists.txt Configuration for RKLLM API Demo

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_api_demo/deploy/CMakeLists.txt

This snippet shows the main CMakeLists.txt file for the RKLLM API demo. It sets up the project, C++ standard, determines the target architecture based on the system, and configures build and installation paths.

```cmake
cmake_minimum_required(VERSION 3.10)
project(rkllm_demo)

set(CMAKE_CXX_STANDARD 11)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

if (CMAKE_SYSTEM_NAME STREQUAL "Android")
    set (TARGET_LIB_ARCH ${CMAKE_ANDROID_ARCH_ABI})
else()
    if(CMAKE_SIZEOF_VOID_P EQUAL 8)
        set (TARGET_LIB_ARCH aarch64)
    else()
        set (TARGET_LIB_ARCH armhf)
    endif()
    if (CMAKE_C_COMPILER MATCHES "uclibc")
        set (TARGET_LIB_ARCH ${TARGET_LIB_ARCH}_uclibc)
    endif()
endif()

set(SOURCE_FILES_1 src/llm_demo.cpp)
add_executable(llm_demo ${SOURCE_FILES_1})

set(RKLLM_API_PATH "${CMAKE_SOURCE_DIR}/../../../rkllm-runtime/${CMAKE_SYSTEM_NAME}/librkllm_api")
include_directories(${RKLLM_API_PATH}/include)
if(CMAKE_SYSTEM_NAME STREQUAL "Android")
    set(RKLLM_RT_LIB ${RKLLM_API_PATH}/${CMAKE_ANDROID_ARCH_ABI}/librkllmrt.so)
    find_package(OpenMP REQUIRED)
    target_link_libraries(llm_demo  ${RKLLM_RT_LIB} log OpenMP::OpenMP_CXX)
elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux")
    set(RKLLM_RT_LIB ${RKLLM_API_PATH}/${CMAKE_SYSTEM_PROCESSOR}/librkllmrt.so)
    target_link_libraries(llm_demo  ${RKLLM_RT_LIB})
endif()

# Install the executable file to the specified directory
set(CMAKE_INSTALL_PREFIX ${CMAKE_SOURCE_DIR}/install/demo_${CMAKE_SYSTEM_NAME}_${TARGET_LIB_ARCH})
install(TARGETS llm_demo DESTINATION ./) 
install(PROGRAMS ${RKLLM_RT_LIB} DESTINATION lib)
```

--------------------------------

### Convert Qwen3-VL Vision Model to ONNX and RKNN

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/multimodal_model_demo/README.md

Steps to convert the Qwen3-VL vision model. This involves installing a specific transformers version, exporting to ONNX format, and then converting the ONNX model to RKNN format.

```bash
1、pip install transformers==4.57.0
2、python export_vision.py --path=/path/to/Qwen3-VL --model_name=qwen3-vl --height=448 --width=448
3、python export_vision_rknn.py --path=./onnx/qwen3-vl_vision.onnx --model_name=qwen3-vl --height=448 --width=448
```

--------------------------------

### Set Build CUDA Extension to 0 for Python 3.12

Source: https://github.com/airockchip/rknn-llm/blob/main/README.md

Before installing the RKLLM package in a Python 3.12 environment, set the BUILD_CUDA_EXT environment variable to 0.

```bash
export BUILD_CUDA_EXT=0
```

--------------------------------

### Example 2: Positional Counting Problem

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_api_demo/Readme.md

Determines the position of an individual from the right end of a line, given their position from the left end and the total number of people. This demonstrates basic counting and positional logic.

```python
example 2 （DeepSeek-R1-Distill-Qwen-1.5B_W8A8_RK3588.rkllm）

```

user: 1
有28位小朋友排成一行,从左边开始数第10位是学豆,从右边开始数他是第几位?
robot: <think>
首先，总共有28位小朋友。

从左边开始数，第10位是学豆的位置。

因此，从右边开始数，学豆的位置是从右边数的第(28 - 10 + 1) = 第19位。
</think>

**解答：**

我们有28位小朋友排成一行。题目要求确定从右边开始数时，第10位是学豆的位置。

**步骤如下：**

1. **总人数**：共有28位小朋友。
2. **左边数的顺序**：从左边开始数，第10位是学豆。
3. **右边数的计算**：
   - 从右边数时，第1位对应左边数的第28位。
   - 因此，第n位在左边对应的是第(28 - n + 1)位在右边。

4. **具体计算**：
   [ \n第10位在左边 = 第(28 - 10 + 1) = 第19位在右边
]

**最终答案：**

\boxed{19}

```

--------------------------------

### Run Demo with Qwen3-VL Model

Source: https://github.com/airockchip/rknn-llm/blob/main/README.md

Execute the demo with the Qwen3-VL model, specifying image path, RKNN encoder model, RKLLM language model, and tokenization parameters.

```bash
./demo demo.jpg ./qwen3-vl-2b_vision_rk3588.rknn ./qwen3-vl-2b-instruct_w8a8_rk3588.rkllm 2048 4096 3 rk3588 "<|vision_start|>" "<|vision_end|>" "<|image_pad|>"
```

--------------------------------

### Deploy and Run C++ Demo

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_api_demo/Readme.md

Commands to push the compiled demo to a device, set environment variables, and run the LLM inference executable. Adjust model path and memory parameters as needed.

```bash
adb push install/demo_Linux_aarch64 /data
# push model file to device
adb push DeepSeek-R1-Distill-Qwen-1.5B.rkllm /data/demo_Linux_aarch64
# push the appropriate fixed-frequency script to the device
adb push ../../../scripts/fix_freq_rk3588.sh /data/demo_Linux_aarch64

adb shell
cd /data/demo_Linux_aarch64
# export lib path
export LD_LIBRARY_PATH=./lib
# Execute the fixed-frequency script
sh fix_freq_rk3588.sh
# Set the logging level for performance analysis
export RKLLM_LOG_LEVEL=1
./llm_demo /path/to/your/rkllm/model 2048 4096
```

--------------------------------

### Second Call: Get Final Answer

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_server_demo/README.md

Send the updated messages, including the tool execution results, back to the RKLLM server to get the final answer from the model.

```python
data["messages"] = messages
resp = session.post(server_url, json=data, headers=headers)
print("A:", resp.json()["choices"][0]["message"]["content"])

# Example output:
# A: The current temperature in San Francisco is 26.1°C.
#     Tomorrow, the temperature is expected to be 25.9°C.
```

--------------------------------

### Run Demo with Qwen2.5-VL Model

Source: https://github.com/airockchip/rknn-llm/blob/main/README.md

Execute the demo with the Qwen2.5-VL model, specifying image path, RKNN encoder model, RKLLM language model, and tokenization parameters.

```bash
./demo demo.jpg ./qwen2_5_vl_3b_vision_rk3588.rknn ./qwen2.5-vl-3b-w8a8_level1_rk3588.rkllm 2048 4096 3 rk3588 "<|vision_start|>" "<|vision_end|>" "<|image_pad|>"
```

--------------------------------

### Run Demo with Qwen3.5 Model

Source: https://github.com/airockchip/rknn-llm/blob/main/README.md

Execute the demo with the Qwen3.5 model, specifying image path, RKNN encoder model, RKLLM language model, and tokenization parameters.

```bash
./demo demo.jpg ./Qwen3.5-0.8B_vision_rk3588.rknn ./Qwen3.5-0.8B_w8a8_rk3588.rkllm 2048 4096 3 rk3588 "<|vision_start|>" "<|vision_end|>" "<|image_pad|>"
```

--------------------------------

### Run C++ LLM Demo

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_api_demo/README.md

Execute the compiled LLM demo on the device. Set environment variables for library paths and logging, then run the demo with the model path and context/generation lengths.

```bash
adb shell
cd /data/demo_Linux_aarch64
# export lib path
export LD_LIBRARY_PATH=./lib
# Execute the fixed-frequency script
sh fix_freq_rk3588.sh
# Set the logging level for performance analysis
export RKLLM_LOG_LEVEL=1
./llm_demo /path/to/your/rkllm/model 2048 4096
```

--------------------------------

### Run Demo with InternVL3 Model

Source: https://github.com/airockchip/rknn-llm/blob/main/README.md

Execute the demo with the InternVL3 model, specifying image path, RKNN encoder model, RKLLM language model, and specific image token placeholders.

```bash
./demo demo.jpg ./internvl3-1b_vision_fp16_rk3588.rknn ./internvl3-1b_w8a8_rk3588.rkllm 2048 4096 3 rk3588 "<img>" "</img>" "<IMG_CONTEXT>"
```

--------------------------------

### Push Demo and Model Files to Device

Source: https://github.com/airockchip/rknn-llm/blob/main/README.md

Use adb to push the demo executable and model files to the device's /data directory. Ensure you have the necessary model files and the demo executable.

```bash
adb push ./demo_Linux_aarch64 /data
adb push model.rkllm /data/demo_Linux_aarch64
adb push model.rknn /data/demo_Linux_aarch64
```

--------------------------------

### Run Interactive Chat Demo

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_server_demo/README.md

Launches the built-in interactive chat demo for the server. Replace <board-ip> with the actual IP address.

```bash
# Interactive chat (Demo 1)
python chat_api_flask.py --server http://<board-ip>:8080

# Disable streaming
python chat_api_flask.py --server http://<board-ip>:8080 --no-stream
```

--------------------------------

### Set Up Environment Variables on Device

Source: https://github.com/airockchip/rknn-llm/blob/main/README.md

Connect to the device via adb shell, navigate to the demo directory, and set the LD_LIBRARY_PATH environment variable to include the local lib directory.

```bash
adb shell
cd /data/demo_Linux_aarch64
export LD_LIBRARY_PATH=./lib
```

--------------------------------

### First Call: Model Returns Tool Invocation Instructions

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_server_demo/README.md

Send an initial request to the RKLLM server with user messages and tool definitions. The model will respond with instructions to call specific tools.

```python
messages = [
    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\nCurrent Date: 2024-09-30"},
    {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow?"},
]

data = {
    "model": "rkllm",
    "messages": messages,
    "stream": False,
    "tools": TOOLS,
}

resp = session.post(server_url, json=data, headers=headers)
server_answer = resp.json()["choices"][0]["message"]["content"]

# Example model output:
# <tool_call>
# {"name": "get_current_temperature", "arguments": {"location": "San Francisco"}}
# </tool_call>
# <tool_call>
# {"name": "get_temperature_date", "arguments": {"location": "San Francisco", "date": "2024-10-01"}}
# </tool_call>
```

--------------------------------

### Build Gradio Server for RKLLM

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_server_demo/README.md

Builds and deploys the Gradio server for a web UI. Ensure the model path and platform are correctly specified.

```bash
# Usage: ./build_rkllm_server_gradio.sh --workshop [Working Path] --model_path [Absolute Path of RKLLM Model on Board] --platform [Target Platform: rk3588/rk3576] [--lora_model_path [Lora Model Path]] [--prompt_cache_path [Prompt Cache File Path]] [--adb_device [ADB Device Serial]]
./build_rkllm_server_gradio.sh --workshop /userdata --model_path /userdata/model.rkllm --platform rk3588 --adb_device 1234567890abcdef
```

--------------------------------

### Compile and Build C++ Demo

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_api_demo/README.md

Scripts to compile the C++ LLM demo for Linux or Android. After compilation, push the necessary files to the target device.

```bash
cd deploy
# for linux
./build-linux.sh
# for android
./build-android.sh
# push install dir to device
adb push install/demo_Linux_aarch64 /data
# push model file to device
adb push DeepSeek-R1-Distill-Qwen-1.5B.rkllm /data/demo_Linux_aarch64
# push the appropriate fixed-frequency script to the device
adb push ../../../scripts/fix_freq_rk3588.sh /data/demo_Linux_aarch4
```

--------------------------------

### Convert DeepSeekOCR Vision Model to ONNX and RKNN

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/multimodal_model_demo/README.md

Steps to convert the DeepSeekOCR vision model. This involves installing a specific ONNX version, exporting to ONNX format, and then converting the ONNX model to RKNN format. Note the requirement to set 'antialias' to False for F.interpolate calls in deepencoder.py.

```bash
1、pip install onnx==1.18.0
2、python export_vision.py --path=/path/to/DeepSeek-OCR --model_name=deepseekocr --height=448 --width=448
3、python export_vision_rknn.py --path=./onnx/deepseekocr_vision.onnx --model_name=deepseekocr --height=448 --width=448
```

--------------------------------

### Run HuggingFace Multimodal Model Demo

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/multimodal_model_demo/README.md

Execute the inference script for the Qwen2-VL demo. Ensure the model path in 'infer.py' is correctly set before running.

```bash
cd examples/Qwen2-VL_Demo
python infer.py
# expect results:
["The image depicts an astronaut in a white spacesuit, reclining on a green chair with his feet up. He is holding a green beer bottle in his right hand. The astronaut is on a moon-like surface, with the Earth visible in the background. The scene is set against a backdrop of stars and the moon's surface, creating a surreal and whimsical atmosphere."]
```

--------------------------------

### Main Demo Executable Build and Linking

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/multimodal_model_demo/deploy/CMakeLists.txt

Builds the main demo executable, linking against OpenCV, rknn runtime, and rkllm runtime libraries. It includes platform-specific linking for Android (adding the 'log' library).

```cmake
# demo
add_executable(${PROJECT_NAME} src/image_enc.cc src/main.cpp)
if(CMAKE_SYSTEM_NAME STREQUAL "Android")
  target_link_libraries(${PROJECT_NAME}  PRIVATE ${OpenCV_LIBS} ${LIBRKNNRT} ${LIBRKLLMRT} log)
else()
  target_link_libraries(${PROJECT_NAME} PRIVATE ${OpenCV_LIBS} ${LIBRKNNRT} ${LIBRKLLMRT})
endif()
```

--------------------------------

### Run C++ Demo on Board

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/multimodal_model_demo/README.md

Commands to run the compiled C++ demo on the target board. This includes setting the library path, linking models, and executing the image encoder and the main multimodal demo.

```bash
adb shell
cd /data/demo_Linux_aarch64
# export lib path
export LD_LIBRARY_PATH=./lib
# soft link models dir
ln -s /data/models .
# run imgenc
./imgenc models/qwen2-vl-vision_rk3588.rknn demo.jpg 3
# run demo(Multimodal Example)
./demo demo.jpg models/qwen2-vl-vision_rk3588.rknn models/qwen2-vl-llm_rk3588.rkllm 2048 4096 3 rk3588 "<|vision_start|>" "<|vision_end|>" "<|image_pad|>"
```

--------------------------------

### Run Demo with DeepSeekOCR Model

Source: https://github.com/airockchip/rknn-llm/blob/main/README.md

Execute the demo with the DeepSeekOCR model, specifying image path, RKNN encoder model, RKLLM language model, and specific padding token.

```bash
./demo demo.jpg ./deepseekocr_vision_rk3588.rknn ./deepseekocr_w8a8_rk3588.rkllm 2048 4096 3 rk3588 "" "" "<｜▁pad▁｜>"
```

--------------------------------

### Run Function Calling Demo

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_server_demo/README.md

Launches the built-in interactive demo for function calling (Demo 2). Replace <board-ip> with the actual IP address.

```bash
python chat_api_flask.py --server http://<board-ip>:8080 --demo 2
```

--------------------------------

### Build Flask Server for RKLLM

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_server_demo/README.md

Builds and deploys the Flask server for an OpenAI-compatible API. Ensure the model path and platform are correctly specified.

```bash
# Usage: ./build_rkllm_server_flask.sh --workshop [Working Path] --model_path [Absolute Path of RKLLM Model on Board] --platform [Target Platform: rk3588/rk3576] [--lora_model_path [Lora Model Path]] [--prompt_cache_path [Prompt Cache File Path]] [--adb_device [ADB Device Serial]]
./build_rkllm_server_flask.sh --workshop /userdata --model_path /userdata/model.rkllm --platform rk3588 --adb_device 1234567890abcdef
```

--------------------------------

### RKLLM Runtime Path and Include Directories

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/multimodal_model_demo/deploy/CMakeLists.txt

Defines the path to the rkllm runtime library and its include directories. It also finds the OpenMP package, which might be required by the rkllm runtime.

```cmake
# rkllm runtime
set(RKLLM_PATH ${CMAKE_CURRENT_SOURCE_DIR}/../../../rkllm-runtime)
set(LIBRKLLMRT ${RKLLM_PATH}/${CMAKE_SYSTEM_NAME}/librkllm_api/${TARGET_LIB_ARCH}/librkllmrt.so)
set(LIBRKLLMRT_INCLUDES ${RKLLM_PATH}/${CMAKE_SYSTEM_NAME}/librkllm_api/include)
find_package(OpenMP REQUIRED)

include_directories(${LIBRKLLMRT_INCLUDES})
```

--------------------------------

### Generate Input Embeds for RKLLM Quantization

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/multimodal_model_demo/README.md

Create input embeddings for quantizing an RKLLM model using a specified model path and type. This step is required before exporting the RKLLM model.

```bash
python data/make_input_embeds_for_quantize.py --path /path/to/Qwen2-VL-model --model_type qwen2vl
```

--------------------------------

### RKNN Runtime Path and Include Directories

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/multimodal_model_demo/deploy/CMakeLists.txt

Defines the path to the rknn runtime library and its include directories. This allows the project to link against the rknnrt.so library.

```cmake
# rknn runtime
set(RKNN_PATH ${CMAKE_CURRENT_SOURCE_DIR}/3rdparty/librknnrt)
set(LIBRKNNRT ${RKNN_PATH}/${CMAKE_SYSTEM_NAME}/librknn_api/${TARGET_LIB_ARCH}/librknnrt.so)
set(LIBRKNNRT_INCLUDES ${RKNN_PATH}/${CMAKE_SYSTEM_NAME}/librknn_api/include)
```

--------------------------------

### Convert Qwen2-VL Vision Component to ONNX

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/multimodal_model_demo/README.md

Export the Vision + Projector component of a Qwen2-VL model to ONNX format. The first run generates necessary embeddings with 'step' set to 1, and the second run exports the ONNX model with 'step' set to 0. Ensure 'use_flash_attn' is false in config.json if float32 is required.

```bash
# First time to generate cu_seqlens and rotary_pos_emb, need to set 'step' to 1
python export/export_vision_qwen2.py --step 1 --path /path/to/Qwen2-VL-model --batch 1 --height 392 --width 392
# Second time to export onnx model, need to set 'step' to any number except 1
python export/export_vision_qwen2.py --step 0 --path /path/to/Qwen2-VL-model --savepath /path/to/save/qwen2-vl-vision.onnx --batch 1 --height 392 --width 392
```

--------------------------------

### Convert Qwen2-VL Vision Component to RKNN

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/multimodal_model_demo/README.md

Convert an exported ONNX vision model to an RKNN model using rknn-toolkit2. Specify the path to the ONNX model and the target platform.

```bash
python export/export_vision_rknn.py --path /path/to/save/qwen2-vl-vision.onnx --target-platform rk3588
```

--------------------------------

### Export RKLLM Model

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_api_demo/Readme.md

Commands to generate quantization data and export the RKLLM model. Ensure you are in the 'export' directory and have the fp16 model available.

```bash
cd export
python generate_data_quant.py -m /path/to/DeepSeek-R1-Distill-Qwen-1.5B
python export_rkllm.py
```

--------------------------------

### Define Tool Descriptions (OpenAI Function Calling Format)

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_server_demo/README.md

Define the tools available to the model using the OpenAI Function Calling specification. This includes the function name, description, and parameter schema.

```python
TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "get_current_temperature",
            "description": "Get current temperature at a location.",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "City, State, Country"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "get_temperature_date",
            "description": "Get temperature at a location and date.",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "City, State, Country"},
                    "date": {"type": "string", "description": "YYYY-MM-DD"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location", "date"],
            },
        },
    },
]
```

--------------------------------

### OpenCV Configuration and Find Package

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/multimodal_model_demo/deploy/CMakeLists.txt

Sets the OpenCV directory based on the system name (Linux or Android) and then finds the OpenCV package. This is required for using OpenCV functionalities in the project.

```cmake
# opencv
if (CMAKE_SYSTEM_NAME MATCHES "Linux")
  set(OpenCV_DIR ${CMAKE_CURRENT_SOURCE_DIR}/3rdparty/opencv/opencv-linux-aarch64/share/OpenCV)
elseif(CMAKE_SYSTEM_NAME MATCHES "Android")
  set(OpenCV_DIR ${CMAKE_CURRENT_SOURCE_DIR}/3rdparty/opencv/OpenCV-android-sdk/sdk/native/jni/abi-${CMAKE_ANDROID_ARCH_ABI})
endif()
find_package(OpenCV REQUIRED)
```

--------------------------------

### Regular Chat - Manual HTTP Calls

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_server_demo/README.md

Manually construct and send HTTP requests to the chat completions endpoint for more control over the interaction. This includes setting up the server URL, creating a session, building the request body, sending the request, and parsing the response for both streaming and non-streaming modes.

```python
# 1) Set the server URL
server_url = "http://x.x.x.x:8080/v1/chat/completions"

# 2) Create a session
import requests

session = requests.Session()
session.keep_alive = False
adapter = requests.adapters.HTTPAdapter(max_retries=3)
session.mount("https://", adapter)
session.mount("http://", adapter)

# 3) Build the request body
data = {
    "model": "rkllm",                    # Model identifier (customizable)
    "messages": [
        {"role": "user", "content": "Hello"}
    ],
    "stream": False,                     # True = streaming, False = non-streaming
    "temperature": 0.8,                  # Sampling temperature (default: 0.8)
    "top_p": 0.9,                        # Nucleus sampling threshold (default: 0.9)
    "top_k": 1,                          # Top-k sampling (default: 1)
    "max_tokens": 4096,                  # Max tokens to generate (default: 4096)
    "repeat_penalty": 1.1,               # Repeat penalty (default: 1.1)
    "frequency_penalty": 0.0,            # Frequency penalty (default: 0.0)
    "presence_penalty": 0.0,             # Presence penalty (default: 0.0)
    "enable_thinking": False,            # Enable deep thinking mode
}

# 4) Send the request
headers = {"Content-Type": "application/json", "Authorization": "not_required"}
resp = session.post(server_url, json=data, headers=headers, stream=data["stream"])

# 5) Parse the response
import json

# Non-streaming
if resp.status_code == 200:
    result = resp.json()
    print("A:", result["choices"][0]["message"]["content"])
else:
    print("Error:", resp.text)

# Streaming (SSE)
if resp.status_code == 200:
    for line in resp.iter_lines(decode_unicode=True):
        if line.startswith("data: ") and line[6:].strip() != "[DONE]":
            chunk = json.loads(line[6:])
            delta = chunk["choices"][0].get("delta", {})
            if delta.get("content"):
                print(delta["content"], end="", flush=True)
```

--------------------------------

### Manual HTTP Call: Create Requests Session

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_server_demo/README.md

Initializes a requests session with keep-alive disabled and retries configured for robustness.

```python
import requests

session = requests.Session()
session.keep_alive = False
adapter = requests.adapters.HTTPAdapter(max_retries=3)
session.mount("https://", adapter)
session.mount("http://", adapter)
```

--------------------------------

### Model Conversion Requirements

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/rkllm_api_demo/Readme.md

Specifies the necessary toolkit and runtime versions for RKLLM model conversion. Ensure Python version is 3.8 or higher.

```bash
rkllm-toolkit==1.2.x
rkllm-runtime==1.2.x
python >=3.8
```

--------------------------------

### Export RKLLM Model

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/multimodal_model_demo/README.md

Export the LLM component to an RKLLM model. This command specifies the model path, target platform, number of NPU cores, quantized data type, device, and save path.

```bash
python export/export_rkllm.py --path /path/to/Qwen2-VL-model --target-platform rk3588 --num_npu_core 3 --quantized_dtype w8a8 --device cpu --savepath /path/to/save/qwen2-vl-llm_rk3588.rkllm
```

--------------------------------

### Image Encoder Executable Build

Source: https://github.com/airockchip/rknn-llm/blob/main/examples/multimodal_model_demo/deploy/CMakeLists.txt

Compiles the image encoder source files into an executable named 'imgenc'. It includes necessary header directories and links against OpenCV and the rknn runtime library.

```cmake
# imgenc
include_directories(src/image_enc.h ${LIBRKNNRT_INCLUDES})
add_executable(imgenc src/image_enc.cc src/img_encoder.cpp)
target_link_libraries(imgenc PRIVATE ${OpenCV_LIBS} ${LIBRKNNRT})
```