### Install VLMEvalKit

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Quickstart.md

Clone the repository and install the package in editable mode. This is the initial setup step.

```bash
git clone https://github.com/open-compass/VLMEvalKit.git
cd VLMEvalKit
pip install -e .
```

--------------------------------

### Environment Setup with Docker

Source: https://github.com/kwai-keye/keye/blob/main/asset/readme.md

Run the SGLang Docker image to set up the environment. Ensure each node installs the custom SGLang branch.

```shell
docker run -it --gpus all lmsysorg/sglang:v0.5.2
# make sure each node use the following commands to install the custom SGLang branch
git clone -b keye-dpsk-infer-fp8-release https://github.com/Kwai-Keye/sglang.git sglang
pip install -e sglang/python[all]
```

--------------------------------

### Install keye-vl-utils

Source: https://github.com/kwai-keye/keye/blob/main/keye-vl-utils/README.md

Install the keye-vl-utils library using pip.

```bash
pip install keye-vl-utils
```

--------------------------------

### Install DeepGEMM from Source

Source: https://github.com/kwai-keye/keye/blob/main/README.md

Install DeepGEMM from its Keye support branch. This involves cloning the repository and running an installation script.

```shell
# DeepGEMM (Keye support branch)
git clone -b keye_support https://github.com/Kwai-Keye/DeepGEMM.git
cd DeepGEMM
bash install.sh
cd ..
```

--------------------------------

### Install SGLang from Source

Source: https://github.com/kwai-keye/keye/blob/main/README.md

Install SGLang from its custom branch. This involves cloning the repository and installing it using pip.

```shell
# SGLang (custom branch)
git clone -b keye-vl-v2-30b-release https://github.com/Kwai-Keye/sglang.git
cd sglang
pip install -e python[all]
cd ..
```

--------------------------------

### Run Prebuilt Docker Image

Source: https://github.com/kwai-keye/keye/blob/main/README.md

Use this command to run the prebuilt Docker image for the Keye project. Ensure you have Docker installed and GPUs available.

```shell
docker run -it --gpus all kwaikeye/kwai-keye-vl:keye_vl_v2_30b_a3b
```

--------------------------------

### Install lmdeploy and openai

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Quickstart.md

Install the necessary libraries for deploying a local language model as a judge.

```bash
pip install lmdeploy openai
```

--------------------------------

### KeyeVL Usage Example

Source: https://github.com/kwai-keye/keye/blob/main/keye-vl-utils/README.md

Demonstrates how to load a Keye-VL model, prepare messages with various image and video inputs, process them using the utility functions, and generate model inputs for inference. Ensure environment variables like VIDEO_MAX_PIXELS are set if needed.

```python
from transformers import AutoModel, AutoProcessor
from keye_vl_utils import process_vision_info

# default: Load the model on the available device(s)
model_path = "Kwai-Keye/Keye-VL-8B-Preview"

model = AutoModel.from_pretrained(
    model_path, torch_dtype="auto", device_map="auto", attn_implementation="flash_attention_2", trust_remote_code=True,
)

# You can set the maximum tokens for a video through the environment variable VIDEO_MAX_PIXELS
# based on the maximum tokens that the model can accept. 
# export VIDEO_MAX_PIXELS = 32000 * 28 * 28 * 0.9


# You can directly insert a local file path, a URL, or a base64-encoded image into the position where you want in the text.
messages = [
    # Image
    ## Local file path
    [{"role": "user", "content": [{"type": "image", "image": "file:///path/to/your/image.jpg"}, {"type": "text", "text": "Describe this image."}]}],
    ## Image URL
    [{"role": "user", "content": [{"type": "image", "image": "http://path/to/your/image.jpg"}, {"type": "text", "text": "Describe this image."}]}],
    ## Base64 encoded image
    [{"role": "user", "content": [{"type": "image", "image": "data:image;base64,/9j/..."}, {"type": "text", "text": "Describe this image."}]}],
    ## PIL.Image.Image
    [{"role": "user", "content": [{"type": "image", "image": pil_image}, {"type": "text", "text": "Describe this image."}]}],
    ## Model dynamically adjusts image size, specify dimensions if required.
    [{"role": "user", "content": [{"type": "image", "image": "file:///path/to/your/image.jpg", "resized_height": 280, "resized_width": 420}, {"type": "text", "text": "Describe this image."}]}],
    # Video
    ## Local video path
    [{"role": "user", "content": [{"type": "video", "video": "file:///path/to/video1.mp4"}, {"type": "text", "text": "Describe this video."}]}],
    ## Local video frames
    [{"role": "user", "content": [{"type": "video", "video": ["file:///path/to/extracted_frame1.jpg", "file:///path/to/extracted_frame2.jpg", "file:///path/to/extracted_frame3.jpg"],}, {"type": "text", "text": "Describe this video."},],}],
    ## Model dynamically adjusts video nframes, video height and width. specify args if required.
    [{"role": "user", "content": [{"type": "video", "video": "file:///path/to/video1.mp4", "fps": 2.0, "resized_height": 280, "resized_width": 280}, {"type": "text", "text": "Describe this video."}]}],
]

processor = AutoProcessor.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, torch_dtype="auto", device_map="auto").to('cuda')
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
images, videos, video_kwargs = process_vision_info(messages, return_video_kwargs=True)
inputs = processor(text=text, images=images, videos=videos, padding=True, return_tensors="pt", **video_kwargs).to("cuda")
print(inputs)
generated_ids = model.generate(**inputs)
print(generated_ids)
```

--------------------------------

### Install EffectiveKernels from Source

Source: https://github.com/kwai-keye/keye/blob/main/README.md

Install EffectiveKernels from its repository. This involves cloning the repository and installing it using pip without additional dependencies.

```shell
# EffectiveKernels
git clone https://github.com/Kwai-Keye/EffectiveKernels.git
cd EffectiveKernels
pip install -e . --no-deps --no-build-isolation
cd ..
```

--------------------------------

### Download and Setup for AI2D Preprocessing

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/scripts/AI2D_preproc.ipynb

Downloads required font and test split CSV files if they do not exist locally. It then loads the test IDs from the CSV into a set for quick lookups.

```python
import os, cv2
import string
import os.path as osp
import numpy as np
from collections import defaultdict
from vlmeval.smp import ls, load, dump, download_file, encode_image_file_to_base64, md5, mrlines
import pandas as pd
import matplotlib.pyplot as plt
import multiprocessing as mp
from PIL import Image, ImageFont, ImageDraw

font_URL = 'http://opencompass.openxlab.space/utils/Fonts/timesb.ttf'
font_file = 'timesb.ttf'
if not osp.exists(font_file):
    download_file(font_URL)
    
test_split_URL = 'https://s3-us-east-2.amazonaws.com/prior-datasets/ai2d_test_ids.csv'
test_split_file = 'ai2d_test_ids.csv'
if not osp.exists(test_split_file):
    download_file(test_split_URL)
    
test_ids = set(mrlines(test_split_file))
```

--------------------------------

### Install LMDeploy Package

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/EvalByLMDeploy.md

Install the LMDeploy package using pip. Refer to the official documentation for alternative installation methods.

```bash
pip install lmdeploy
```

--------------------------------

### Deploy judge LLM on a single GPU and evaluate on others

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Quickstart.md

This example shows how to deploy the judge LLM on a single GPU (GPU 0) and then run evaluation on other GPUs (1, 2, 3) using torchrun.

```bash
CUDA_VISIBLE_DEVICES=0 lmdeploy serve api_server internlm/internlm2-chat-1_8b --server-port 23333
```

```bash
CUDA_VISIBLE_DEVICES=1,2,3 torchrun --nproc-per-node=3 run.py --data HallusionBench  --model qwen_chat --verbose
```

--------------------------------

### Install and Run Pre-commit Hooks

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Development.md

Before submitting a Pull Request, install pre-commit and run all checks to maintain code tidiness. This ensures code style consistency across the project.

```bash
pip install pre-commit
pre-commit install
pre-commit run --all-files
```

--------------------------------

### Start LMDeploy Inference Service

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/EvalByLMDeploy.md

Launch the LMDeploy API server for a specific model. It is crucial to specify the --model-name to ensure VLMEvalKit uses the correct prompt construction strategy for datasets with custom prompt behaviors.

```bash
lmdeploy serve api_server OpenGVLab/InternVL2-8B --model-name InternVL2-8B
```

--------------------------------

### Handle Multimodal Messages

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Development.md

Multimodal messages are lists of dictionaries, each with 'type' ('image' or 'text') and 'value'. The 'value' can be a local path, URL for images, or text string. This example shows how to construct and use such messages.

```python
IMAGE_PTH = 'assets/apple.jpg'
IMAGE_URL = 'https://raw.githubusercontent.com/open-compass/VLMEvalKit/main/assets/apple.jpg'
msg1 = [
    dict(type='image', value=IMAGE_PTH),
    dict(type='text', value='What is in this image?')
]
msg2 = [
    dict(type='image', value=IMAGE_URL),
    dict(type='image', value=IMAGE_URL),
    dict(type='text', value='How many apples are there in these images?')
]
response = model.generate(msg1)
```

--------------------------------

### Get model name registered by LMDeploy

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Quickstart.md

Use this Python code to get the model name registered by LMDeploy, which is needed for configuring VLMEvalKit to use the local judge LLM.

```python
from openai import OpenAI
client = OpenAI(
    api_key='sk-123456',
    base_url="http://0.0.0.0:23333/v1"
)
model_name = client.models.list().data[0].id
```

--------------------------------

### Launch one model instance with python

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Quickstart.md

Use this command to launch a single model instance using all available GPUs.

```bash
python run.py --data MMBench_DEV_EN --model InternVL3-78B
```

--------------------------------

### Keye-VL-8B-Preview Evaluation Script

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/README.md

Shell script to set up the environment and run evaluations for the Keye-VL-8B-Preview model. Includes cloning the dataset and setting environment variables.

```bash
cd evaluation/KC-MMBench/scripts/Keye-VL-8B-Preview

# download dataset
git clone https://huggingface.co/datasets/KwaiKeye/KC-MMbench
cp -r ./KC-MMbench/images/* ./KC-MMbench/subsets/
export LMUData=./KC-MMbench/subsets

# run eval
bash run.sh
```

--------------------------------

### Launch Evaluation with Config File

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/ConfigSystem.md

Execute the evaluation process using the `run.py` script by providing the path to your custom configuration JSON file via the `--config` argument. Ensure the config file is saved before running.

```bash
python run.py --config config.json
```

--------------------------------

### Launch three model instances with CUDA_VISIBLE_DEVICES

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Quickstart.md

Launch three model instances, each using 2 GPUs, while explicitly specifying which GPUs to use via CUDA_VISIBLE_DEVICES.

```bash
CUDA_VISIBLE_DEVICES=1,2,3,4,5,6 torchrun --nproc-per-node=3 run.py --data MMBench_DEV_EN --model InternVL3-38B
```

--------------------------------

### Launch SGLang Server

Source: https://github.com/kwai-keye/keye/blob/main/README.md

Launch the SGLang server with specified model path and tensor parallelism size. Ensure to replace MODEL_NAME with your actual model path.

```shell
python3 -m sglang.launch_server \
    --model-path=MODEL_NAME \
    --tp-size=2 \
    --trust-remote-code \
    --mem-fraction-static=0.8
```

--------------------------------

### Sample Model and Dataset Configuration JSON

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/ConfigSystem.md

This JSON defines configurations for various models and datasets, including parameters like class, model name, temperature, and image detail for models, and class, dataset name, and frame count for datasets. Use this structure to customize evaluation settings.

```json
{
    "model": {
        "GPT4o_20240806_T00_HIGH": {
            "class": "GPT4V",
            "model": "gpt-4o-2024-08-06",
            "temperature": 0,
            "img_detail": "high"
        },
        "GPT4o_20240806_T10_Low": {
            "class": "GPT4V",
            "model": "gpt-4o-2024-08-06",
            "temperature": 1.0,
            "img_detail": "low"
        },
        "GPT4o_20241120": {}
    },
    "data": {
        "MME-RealWorld-Lite": {
            "class": "MMERealWorld",
            "dataset": "MME-RealWorld-Lite"
        },
        "MMBench_DEV_EN_V11": {
            "class": "ImageMCQDataset",
            "dataset": "MMBench_DEV_EN_V11"
        },
        "MMBench_Video_8frame_nopack":{},
        "Video-MME_16frame_subs": {
            "class": "VideoMME",
            "dataset": "Video-MME",
            "nframe": 16,
            "use_subtitle": true
        }
    }
}
```

--------------------------------

### Configure API Keys

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Quickstart.md

Set up API keys for proprietary VLMs or LLM judges by placing them in a `.env` file or as environment variables. This enables API-based inference and LLM-assisted evaluation.

```bash
# The .env file, place it under $VLMEvalKit
# API Keys of Proprietary VLMs
# QwenVL APIs
DASHSCOPE_API_KEY=
# Gemini w. Google Cloud Backends
GOOGLE_API_KEY=
# OpenAI API
OPENAI_API_KEY=
OPENAI_API_BASE=
# StepAI API
STEPAI_API_KEY=
# REKA API
REKA_API_KEY=
# GLMV API
GLMV_API_KEY=
# CongRong API
CW_API_BASE=
CW_API_KEY=
# SenseNova API
SENSENOVA_API_KEY=
# Hunyuan-Vision API
HUNYUAN_SECRET_KEY=
HUNYUAN_SECRET_ID=
# LMDeploy API
LMDEPLOY_API_BASE=
# You can also set a proxy for calling api models during the evaluation stage
EVAL_PROXY=
```

--------------------------------

### Run IDEFICS2-8B Evaluation on MMBench-Video

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Quickstart.md

Execute an evaluation for the IDEFICS2-8B model on the MMBench-Video dataset using 8 frames as input and vanilla evaluation. This command is intended for use on a node with 8 GPUs.

```bash
torchrun --nproc-per-node=8 run.py --data MMBench_Video_8frame_nopack --model idefics2_8
```

--------------------------------

### Evaluate Image Benchmarks with `python`

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Quickstart.md

Run evaluations for image benchmarks using the `run.py` script with `python`. This is recommended for very large VLMs that might utilize multiple GPUs.

```bash
# IDEFICS-80B-Instruct on MMBench_DEV_EN, MME, and SEEDBench_IMG, Inference and Evalution
python run.py --data MMBench_DEV_EN MME SEEDBench_IMG --model idefics_80b_instruct --verbose
# IDEFICS-80B-Instruct on MMBench_DEV_EN, MME, and SEEDBench_IMG, Inference only
python run.py --data MMBench_DEV_EN MME SEEDBench_IMG --model idefics_80b_instruct --verbose --mode infer
```

--------------------------------

### Launch two model instances with torchrun

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Quickstart.md

Use this command to launch two model instances in data parallel, with each instance using 4 GPUs on a machine with 8 GPUs.

```bash
torchrun --nproc-per-node=2 run.py --data MMBench_DEV_EN --model InternVL3-78B
```

--------------------------------

### Implement Custom Prompt Building

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Development.md

Optionally implement `use_custom_prompt` and `build_prompt` to customize prompt generation for specific datasets. `use_custom_prompt` returns a boolean, and `build_prompt` constructs the multimodal message.

```python
def use_custom_prompt(dataset):
    # ... return True or False ...

def build_prompt(line, dataset=None):
    # ... construct and return multimodal message ...
    return custom_message
```

--------------------------------

### Run GPT-4o Evaluation on MMBench-Video

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Quickstart.md

Execute an evaluation for the GPT-4o model on the MMBench-Video dataset, processing 1 frame per second with pack evaluation. This command uses the standard Python interpreter.

```bash
python run.py --data MMBench_Video_1fps_pack --model GPT4o
```

--------------------------------

### Load and Prepare MMBench Data

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/scripts/visualize.ipynb

Loads MMBench results from a JSON file and prepares a subset of the data for visualization. It defines models to be visualized and their corresponding colors.

```python
# Draw MMBench Radar Graph
data = json.loads(open('OpenVLM.json').read())['results']
models = list(data)
print(models)

# model2vis = [
#     'GPT-4v (detail: low)', 'GeminiProVision', 'Qwen-VL-Plus', 
#     'InternLM-XComposer2-VL', 'LLaVA-v1.5-13B', 'CogVLM-17B-Chat',
#     'mPLUG-Owl2', 'Qwen-VL-Chat', 'IDEFICS-80B-Instruct'
# ]

model2vis = [
    # 'GPT-4v (detail: low)', 'GeminiProVision', 'InternLM-XComposer2-VL', 
    'GPT-4v (1106, detail-low)', 'Gemini-1.0-Pro', 'Gemini-1.5-Pro', #'Gemini-1.5-Flash', 'Qwen-VL-Plus', 
    'InternLM-XComposer2', 'LLaVA-v1.5-13B', 'CogVLM-17B-Chat',
    'mPLUG-Owl2', 'Qwen-VL-Chat', 'IDEFICS-80B-Instruct'
]

colors = [
    '#1f77b4', '#ff7f0e', '#2ca02c', '#d62728', '#9467bd', '#8c564b', 
    '#e377c2', '#7f7f7f', '#bcbd22'
]
```

--------------------------------

### Node 1 (Master) Deployment Command

Source: https://github.com/kwai-keye/keye/blob/main/asset/readme.md

Launch the SGLang server on the master node for distributed deployment. Configure model path, network settings, and performance parameters.

```shell
MODEL_PATH=/path/to/Keye-VL-671B-A37B
DIST_INIT_ADDR="MASTER_NODE_IP:29500"   # e.g. 192.168.1.100:29500
PORT=30000                              # listening port on each node
python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 0.0.0.0 \
    --port $PORT \
    --tp-size 16 \
    --nnodes 2 \
    --node-rank 0 \
    --dist-init-addr $DIST_INIT_ADDR \
    --trust-remote-code \
    --mm-attention-backend fa3 \
    --attention-backend fa3 \
    --disable-radix-cache \
    --mem-fraction-static 0.8 \
    --cuda-graph-max-bs 64 \
    --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 32}'
```

--------------------------------

### Using Callable Objects

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/_templates/callable.rst

This section explains how to invoke objects that are designed to be called like functions. It highlights the use of the `__call__` special method.

```APIDOC
## Invoking Callable Objects

### Description
Objects with a `__call__` method can be invoked directly as if they were functions. This allows for flexible object-oriented designs where objects can encapsulate behavior that is executed upon invocation.

### Method
`__call__`

### Usage
```python
# Assuming 'my_object' is an instance of a class with a __call__ method
result = my_object(*args, **kwargs)
```

### Parameters
Parameters are passed directly to the `__call__` method of the object. The specific parameters depend on the implementation of the `__call__` method within the class.
```

--------------------------------

### Evaluate Image Benchmarks with `torchrun`

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Quickstart.md

Use `torchrun` to distribute VLM instances across multiple GPUs for faster inference, suitable for VLMs with lower memory consumption. This command runs inference and evaluation.

```bash
# IDEFICS-9B-Instruct, Qwen-VL-Chat, mPLUG-Owl2 on MMBench_DEV_EN, MME, and SEEDBench_IMG. On a node with 8 GPU. Inference and Evaluation.
torchrun --nproc-per-node=8 run.py --data MMBench_DEV_EN MME SEEDBench_IMG --model idefics_80b_instruct qwen_chat mPLUG-Owl2 --verbose
# Qwen-VL-Chat on MME. On a node with 2 GPU. Inference and Evaluation.
torchrun --nproc-per-node=2 run.py --data MME --model qwen_chat --verbose
```

--------------------------------

### Customizing Model Prompt Construction

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Quickstart.md

This Python code defines a `use_custom_prompt` method for a model. It determines whether to use the model's custom prompt logic for a given dataset based on dataset type, modality, and specific dataset names.

```python
def use_custom_prompt(self, dataset: str) -> bool:
    from vlmeval.dataset import DATASET_TYPE, DATASET_MODALITY
    dataset_type = DATASET_TYPE(dataset, default=None)
    if not self._use_custom_prompt:
        return False
    if listinstr(['MMVet'], dataset):
        return True
    if dataset_type == 'MCQ':
        return True
    if DATASET_MODALITY(dataset) == 'VIDEO':
        return False
    return False
```

--------------------------------

### Customizing Benchmark Prompts

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Development.md

Implement the build_prompt function to create custom prompt formats for datasets or interleave inputs. This function takes a TSV line and returns a list of multimodal messages.

```python
def build_prompt(line):
    # ... implementation ...
    return [dict(type='image', value=IMAGE_PTH), dict(type='text', value=prompt)]
```

--------------------------------

### Generate Image and Text Description

Source: https://github.com/kwai-keye/keye/blob/main/asset/readme.md

Use this snippet to send an image and a text prompt to the model for detailed image description. Ensure the image URL is accessible.

```python
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"},
            },
            {"type": "text", "text": "Describe this image in detail."},
        ],
    }
]

result = generate(messages)
print(result["choices"][0]["message"]["content"])
```

--------------------------------

### Deploy a local judge LLM with LMDeploy

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Quickstart.md

Deploy a local judge LLM using LMDeploy, specifying the model, server port, and an API key.

```bash
lmdeploy serve api_server internlm/internlm2-chat-1_8b --server-port 23333
```

--------------------------------

### Node 2 (Worker) Deployment Command

Source: https://github.com/kwai-keye/keye/blob/main/asset/readme.md

Launch the SGLang server on a worker node for distributed deployment. Ensure network settings match the master node.

```shell
MODEL_PATH=/path/to/Keye-VL-671B-A37B
DIST_INIT_ADDR="MASTER_NODE_IP:29500"   # e.g. 192.168.1.100:29500
PORT=30000                              # listening port on each node
python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 0.0.0.0 \
    --port $PORT \
    --tp-size 16 \
    --nnodes 2 \
    --node-rank 1 \
    --dist-init-addr $DIST_INIT_ADDR \
    --trust-remote-code \
    --mm-attention-backend fa3 \
    --attention-backend fa3 \
    --disable-radix-cache \
    --mem-fraction-static 0.8 \
    --cuda-graph-max-bs 64 \
    --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 32}'
```

--------------------------------

### Python Client for Text Generation

Source: https://github.com/kwai-keye/keye/blob/main/asset/readme.md

Use the requests library to send chat completion requests to the deployed SGLang server. Adjust temperature and max_tokens as needed.

```python
import json
import requests

BASE_URL = "http://MASTER_NODE_IP:30000"

def generate(messages):
    payload = {
        "model": "",
        "messages": messages,
        "n": 1,
        "temperature": 0.0,
        "max_tokens": 256,
        "top_k": 1,
        "ignore_eos": False,
        "skip_special_tokens": True,
    }
    resp = requests.post(
        f"{BASE_URL}/v1/chat/completions",
        headers={"Content-Type": "application/json"},
        data=json.dumps(payload),
        timeout=1800,
    )
    resp.raise_for_status()
    return resp.json()

```

--------------------------------

### Download and Unzip AI2D Dataset

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/scripts/AI2D_preproc.ipynb

Downloads the AI2D dataset zip file and extracts its contents. It then lists the image, question, and annotation files, and loads category information.

```python
download_file('https://ai2-public-datasets.s3.amazonaws.com/diagrams/ai2d-all.zip')
os.system('unzip -o ai2d-all.zip')

images = ls('ai2d/images/')
questions = ls('ai2d/questions/')
annotations = ls('ai2d/annotations/')
cates = load('ai2d/categories.json')
```

--------------------------------

### Select Specific MMBench Split

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/scripts/visualize.ipynb

Selects a specific split (e.g., 'MMBench_TEST_EN') from the loaded MMBench data. It uses `defaultdict` to handle cases where a split might be missing, defaulting its values to 0.

```python
from collections import defaultdict

split = 'MMBench_TEST_EN'
# data_sub = {k: v[split] for k, v in data.items()}
data_sub = {k: defaultdict(int, v)[split] for k, v in data.items()}
```

--------------------------------

### Run Evaluation with LMDeploy

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/EvalByLMDeploy.md

Execute the evaluation script using VLMEvalKit, specifying the dataset, model type as 'lmdeploy', and enabling verbose output. The --api-nproc argument controls the number of processes for API calls.

```python
python run.py --data MMStar --model lmdeploy --verbose --api-nproc 64
```

--------------------------------

### Download Files with URL

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/scripts/visualize.ipynb

Downloads a file from a given URL. If no filename is specified, it infers the filename from the URL.

```python
import json
import copy as cp
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.font_manager as fm

def download_file(url, filename=None):
    from urllib.request import urlretrieve
    if filename is None:
        filename = url.split('/')[-1]
    urlretrieve(url, filename)

font_URL = 'http://opencompass.openxlab.space/utils/Fonts/segoepr.ttf'
download_file(font_URL)

font12 = fm.FontProperties(fname='segoepr.ttf', size=12)
font15 = fm.FontProperties(fname='segoepr.ttf', size=15, weight='bold')
font18 = fm.FontProperties(fname='segoepr.ttf', size=18, weight='bold')

DATA_URL = 'http://opencompass.openxlab.space/utils/OpenVLM.json'
download_file(DATA_URL)
```

--------------------------------

### SGLang Client for Video Input

Source: https://github.com/kwai-keye/keye/blob/main/README.md

Python script to interact with the SGLang server for video and text inputs. Customize sampling parameters and video frame-sampling settings as needed.

```python
import json
import requests

BASE_URL = "http://MASTER_NODE_IP:8000"

def generate(messages):
    payload = {
        "model": "",
        "messages": messages,
        "n": 1,
        "temperature": 0.1,
        "max_tokens": 32760,
        "top_p": 0.001,
        "ignore_eos": False,
        "skip_special_tokens": True,
    }
    resp = requests.post(
        f"{BASE_URL}/v1/chat/completions",
        headers={"Content-Type": "application/json"},
        data=json.dumps(payload),
        timeout=1800,
    )
    resp.raise_for_status()
    return resp.json()

# Example: Video + text
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "video_url",
                "video_url": {
                    "url": video_url,
                    "preprocess_kwargs": {
                        "fps": 2.0,
                        "min_pixels": 128*28*28,
                        "max_pixels": 512*28*28,
                        "video_total_pixels":180*1024*28*28,
                    }     
                },          
            },
            {"type": "text", "text": "Describe this video."},
        ],
    },
]

result = generate(messages)
print(result["choices"][0]["message"]["content"])
```

--------------------------------

### Run CC-OCR Benchmark Evaluation Scripts

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/vlmeval/dataset/utils/ccocr_evaluator/README.md

Execute these shell scripts from the root directory of VLMEvalKit to perform batch inference and evaluation for various CC-OCR subsets. Ensure your environment is set up correctly before running.

```shell
MODEL_NAME="QwenVLMax"
OUTPUT_DIR="/your/path/to/output_dir"

SUB_OUTPUT_DIR=${OUTPUT_DIR}/multi_scene_ocr
python run.py --data CCOCR_MultiSceneOcr_Cord CCOCR_MultiSceneOcr_Funsd CCOCR_MultiSceneOcr_Iam CCOCR_MultiSceneOcr_ZhDoc CCOCR_MultiSceneOcr_ZhHandwriting CCOCR_MultiSceneOcr_Hieragent CCOCR_MultiSceneOcr_Ic15 CCOCR_MultiSceneOcr_Inversetext CCOCR_MultiSceneOcr_Totaltext CCOCR_MultiSceneOcr_ZhScene CCOCR_MultiSceneOcr_UgcLaion CCOCR_MultiSceneOcr_ZhDense CCOCR_MultiSceneOcr_ZhVertical --model ${MODEL_NAME} --work-dir ${SUB_OUTPUT_DIR} --verbose
python vlmeval/dataset/utils/ccocr_evaluator/common.py ${SUB_OUTPUT_DIR}
```

```shell
SUB_OUTPUT_DIR=${OUTPUT_DIR}/multi_lan_ocr
python run.py --data CCOCR_MultiLanOcr_Arabic CCOCR_MultiLanOcr_French CCOCR_MultiLanOcr_German CCOCR_MultiLanOcr_Italian CCOCR_MultiLanOcr_Japanese CCOCR_MultiLanOcr_Korean CCOCR_MultiLanOcr_Portuguese CCOCR_MultiLanOcr_Russian CCOCR_MultiLanOcr_Spanish CCOCR_MultiLanOcr_Vietnamese --model ${MODEL_NAME} --work-dir ${SUB_OUTPUT_DIR} --verbose
python vlmeval/dataset/utils/ccocr_evaluator/common.py ${SUB_OUTPUT_DIR}
```

```shell
SUB_OUTPUT_DIR=${OUTPUT_DIR}/doc_parsing
python run.py --data CCOCR_DocParsing_DocPhotoChn CCOCR_DocParsing_DocPhotoEng CCOCR_DocParsing_DocScanChn CCOCR_DocParsing_DocScanEng CCOCR_DocParsing_TablePhotoChn CCOCR_DocParsing_TablePhotoEng CCOCR_DocParsing_TableScanChn CCOCR_DocParsing_TableScanEng CCOCR_DocParsing_MolecularHandwriting CCOCR_DocParsing_FormulaHandwriting --model ${MODEL_NAME} --work-dir ${SUB_OUTPUT_DIR} --verbose
python vlmeval/dataset/utils/ccocr_evaluator/common.py ${SUB_OUTPUT_DIR}
```

```shell
SUB_OUTPUT_DIR=${OUTPUT_DIR}/kie
python run.py --data CCOCR_Kie_Sroie2019Word CCOCR_Kie_Cord CCOCR_Kie_EphoieScut CCOCR_Kie_Poie CCOCR_Kie_ColdSibr CCOCR_Kie_ColdCell --model ${MODEL_NAME} --work-dir ${SUB_OUTPUT_DIR} --verbose
python vlmeval/dataset/utils/ccocr_evaluator/common.py ${SUB_OUTPUT_DIR}
```

--------------------------------

### Set environment variables for local LLM judge

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Quickstart.md

Configure VLMEvalKit to use a local judge LLM by setting these environment variables. Alternatively, these can be set in the $VLMEvalKit/.env file.

```bash
OPENAI_API_KEY=sk-123456
OPENAI_API_BASE=http://0.0.0.0:23333/v1/chat/completions
LOCAL_LLM=<model_name you get>
```

--------------------------------

### VLM Evaluation Configuration

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/README.md

Configuration for evaluating Vision-Language Models on KC-MMBench datasets. Specifies dataset classes and names for different tasks.

```python
{

    "model":'...'    "data": {
        "CPV": {
            "class": "KwaiVQADataset",
            "dataset": "CPV"
        },
        "Hot_Videos_Aggregation": {
            "class": "KwaiVQADataset",
            "dataset": "Hot_Videos_Aggregation"
        },
        "Collection_Order": {
            "class": "KwaiVQADataset",
            "dataset": "Collection_Order"
        },
        "Pornographic_Comment": {
            "class": "KwaiYORNDataset",
            "dataset": "Pornographic_Comment"
        },
        "High_like":{
            "class":"KwaiYORNDataset",
            "dataset":"High_like"
        },
        "SPU": {
            "class": "KwaiYORNDataset",
            "dataset": "SPU"
        }
    }
}
```

--------------------------------

### Cover Image with Replacement Text

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/scripts/AI2D_preproc.ipynb

Processes an annotation file to cover existing text in an image with a white rectangle and then overlays replacement text. It calculates the appropriate font size for the replacement text to fit within the covered area.

```python
def cover_image(ann_path):
    data = load(ann_path)
    texts = list(data['text'].values())
    raw_img = ann_path.replace('annotations', 'images').replace('.json', '')
    tgt_img = raw_img.replace('images', 'images_abc')
    img = Image.open(raw_img)
    draw = ImageDraw.Draw(img)
    for text in texts:
        st, ed = tuple(text['rectangle'][0]), tuple(text['rectangle'][1])
        T = text['replacementText']
        draw.rectangle((st, ed), fill='white')
        font_size = proper_font_size(font_file, (ed[0] - st[0], ed[1] - st[1]), T, ratio=1)
        font = ImageFont.truetype(font_file, font_size)
        text_box = font.getbbox(T)
        text_wh = (text_box[2] - text_box[0], text_box[3] - text_box[1])
        cx, cy = (st[0] + ed[0]) // 2, st[1]
        stx = cx - text_wh[0] // 2
        sty = cy - text_wh[1] // 2
        draw.text((stx, sty), T, font=font, fill='black')
    img.save(tgt_img)
```

--------------------------------

### Simplified Multimodal Message Input

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Development.md

The `generate` method can accept a list of strings as input. It automatically detects image paths or URLs and converts them into the standard multimodal message format.

```python
IMAGE_PTH = 'assets/apple.jpg'
IMAGE_URL = 'https://raw.githubusercontent.com/open-compass/VLMEvalKit/main/assets/apple.jpg'
msg1 = [IMAGE_PTH, 'What is in this image?']
msg2 = [IMAGE_URL, IMAGE_URL,  'How many apples are there in these images?']
response = model.generate(msg1)
```

--------------------------------

### Set Comparison Metrics

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/vlmeval/dataset/utils/megabench/README.md

Metrics for evaluating responses where order does not matter, treating inputs as sets. Handles potential parsing failures by defaulting to empty sets.

```python
set_equality
jaccard_index
set_precision
chess_move_list_jaccard_index
```

--------------------------------

### String Comparison Metrics

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/vlmeval/dataset/utils/megabench/README.md

Metrics for evaluating string responses, ranging from exact matches to fuzzy comparisons and normalized similarity.

```python
exact_str_match
simple_str_match
exact_str_match_case_insensitive
normalized_similarity_demarau_levenshtein
near_str_match
program_judge
```

--------------------------------

### SGLang Client for Image Input

Source: https://github.com/kwai-keye/keye/blob/main/README.md

Python script to interact with the SGLang server for image and text inputs. Customize sampling parameters as needed.

```python
import json
import requests

BASE_URL = "http://MASTER_NODE_IP:8000"

def generate(messages):
    payload = {
        "model": "",
        "messages": messages,
        "n": 1,
        "temperature": 0.0,
        "max_tokens": 256,
        "top_k": 1,
        "ignore_eos": False,
        "skip_special_tokens": True,
    }
    resp = requests.post(
        f"{BASE_URL}/v1/chat/completions",
        headers={"Content-Type": "application/json"},
        data=json.dumps(payload),
        timeout=1800,
    )
    resp.raise_for_status()
    return resp.json()

# Example: image + text
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"},
            },
            {"type": "text", "text": "Describe this image in detail."},
        ],
    }
]

result = generate(messages)
print(result["choices"][0]["message"]["content"])
```

--------------------------------

### Generate Radar Chart for MMBench Evaluation

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/scripts/visualize.ipynb

This code snippet generates a radar chart to visualize model performance on MMBench. It preprocesses data, normalizes scores, and plots the radar chart with fill areas and custom labels. Ensure matplotlib and numpy are imported.

```python
labels = list(data_sub[model2vis[0]])
labels.remove('Overall')
num_vars = len(labels)

raw_data = [data_sub[m] for m in model2vis]
data_list, range_map = pre_normalize(raw_data, labels)

alpha = 0.25
angless = np.linspace(0, 2 * np.pi, num_vars, endpoint=False).tolist()
angless_deg = np.linspace(0, 360, num_vars, endpoint=False).tolist()
fig, ax_base = plt.subplots(nrows=1, ncols=1, figsize=(10, 10), subplot_kw=dict(polar=True))

for i in range(len(data_list)):
    item = data_list[i]
    model_name = model2vis[i]
    color = colors[i]
    tmp_angles = angless[:] + [angless[0]]
    tmp_values = [item[lb] for lb in labels] + [item[labels[0]]]
    ax_base.plot(tmp_angles, tmp_values, color=color, linewidth=1, linestyle='solid', label=model_name)
    ax_base.fill(tmp_angles, tmp_values, color=color, alpha=alpha)
    
angless += [angless[0]]
ax_base.set_ylim(0, 100)
ax_base.set_yticks([40, 60, 80, 100])
ax_base.set_yticklabels([''] * 4)

ax_base.tick_params(pad=25)
ax_base.set_xticks(angless[:-1])
ax_base.set_xticklabels(labels, fontproperties=font18)

leg = ax_base.legend(loc='center right', bbox_to_anchor=(1.6, 0.5), prop=font15, ncol=1, frameon=True, labelspacing=1.2)
for line in leg.get_lines():
    line.set_linewidth(2.5)

cx, cy, sz = 0.44, 0.435, 0.34
axes = [fig.add_axes([cx - sz, cy - sz, cx + sz, cy + sz], projection='polar', label='axes%d' % i) for i in range(num_vars)]
    
for ax, angle, label in zip(axes, angless_deg, labels):
    ax.patch.set_visible(False)
    ax.grid(False)
    ax.xaxis.set_visible(False)
    cur_range = range_map[label]
    label_list = [cur_range[0] + (cur_range[1] - cur_range[0]) / 5 * i for i in range(2, 6)]
    label_list = [f'{x:.1f}' for x in label_list]
    ax.set_rgrids(range(40, 120, 20), angle=angle, labels=label_list, font_properties=font12)
    ax.spines['polar'].set_visible(False)
    ax.set_ylim(0, 100)

title_text = f'{len(model2vis)} Representative VLMs on MMBench Test.'
plt.figtext(.7, .95, title_text, fontproperties=font18, ha='center')
plt.show()
```

--------------------------------

### AI2D Dataset Processing and DataFrame Creation

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/scripts/AI2D_preproc.ipynb

Iterates through questions, extracts relevant data, determines 'abcLabel', and constructs a Pandas DataFrame. It handles image path selection based on the 'abcLabel'.

```python
data_all = defaultdict(list)
for qfile in questions:
    data = load(qfile)
    idx = data['imageName'].split('.')[0]
    if idx not in test_ids:
        continue
    image_pth = qfile.replace('questions', 'images').replace('.json', '')
    cate = cates[image_pth.split('/')[-1]]
    for q, qmeta in data['questions'].items():
        assert '.png-' in qmeta['questionId']
        main, sub = qmeta['questionId'].split('.png-')
        idx = int(main) * 100 + int(sub)
        
        answers = qmeta['answerTexts']
        correct = qmeta['correctAnswer']
        
        data_all['index'].append(idx)
        data_all['question'].append(q)
        assert len(answers) == 4
        for c, a in zip('ABCD', answers):
            data_all[c].append(a)
        data_all['answer'].append('ABCD'[qmeta['correctAnswer']])
        data_all['category'].append(cate)
        data_all['abcLabel'].append(qmeta['abcLabel'])
        abc = is_abc(qmeta['abcLabel'], {x: data_all[x][-1] for x in 'ABCD'}, q)
        # if qmeta['abcLabel'] and not abc:
        #     print(qmeta['abcLabel'], {x: data_all[x][-1] for x in 'ABCD'}, q)
        data_all['image_path'].append(image_pth.replace('images', 'images_abc') if abc else image_pth)
data = pd.DataFrame(data_all)
```

--------------------------------

### Implement Multi-turn Conversation API

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/zh-CN/Development.md

To enable multi-turn conversations, implement the `chat_inner` API. It accepts a list of messages representing the chat history, with alternating 'user' and 'assistant' roles.

```python
# Assume msg1, msg2, msg3, ... are multi-modal messages following the previously described format
# `chat_inner` take the following chat history list as input:
message = [
    dict(role='user', content=msg1),
    dict(role='assistant', content=msg2),
    dict(role='user', content=msg3),
    dict(role='assistant', content=msg4),
	......
    dict(role='user', content=msgn),
]
# `message` should contain an odd number of chat utterances, the role of utterances should be interleaved "user" and "assistant", with the role of the last utterance to be "user".
# The chat function will call `chat_inner`
response = model.chat(message)
```

--------------------------------

### List Comparison Metrics

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/vlmeval/dataset/utils/megabench/README.md

Metrics for evaluating ordered sequences, such as lists or strings representing lists. Handles potential parsing failures by defaulting to empty lists.

```python
longest_common_list_prefix_ratio
```

--------------------------------

### Implement Model Generation API

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Development.md

The `generate_inner` method is mandatory for new models. It takes multimodal messages and returns the model's string prediction. Use `self.message_to_promptimg` if your model does not support interleaved images and text.

```python
def generate_inner(msgs, dataset=None):
    # ... implementation to generate prediction ...
    return prediction_string
```

--------------------------------

### Bounding Box Metrics

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/vlmeval/dataset/utils/megabench/README.md

Specialized list metrics for bounding boxes, using Intersection over Union (IoU) for normalized coordinates.

```python
nbbox_iou_tuple
```

--------------------------------

### Implement generate_inner API for New Models

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/zh-CN/Development.md

Models must implement the `generate_inner` method to process multimodal inputs and return predictions. The `msgs` parameter is a list of dictionaries, each with 'type' (image/text) and 'value' (path/URL/string).

```python
IMAGE_PTH = 'assets/apple.jpg'
IMAGE_URL = 'https://raw.githubusercontent.com/open-compass/VLMEvalKit/main/assets/apple.jpg'
msg1 = [
    dict(type='image', value=IMAGE_PTH),
    dict(type='text', value='What is in this image?')
]
msg2 = [
    dict(type='image', value=IMAGE_URL),
    dict(type='image', value=IMAGE_URL),
    dict(type='text', value='How many apples are there in these images?')
]
response = model.generate(msg1)
```

--------------------------------

### Image Encoding and Decoding APIs

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/docs/en/Development.md

APIs for encoding images to base64 and decoding base64 back to images. Used for handling image data in benchmark datasets.

```python
encode_image_to_base64 (for PIL Image) / encode_image_file_to_base64 (for image file path)
```

```python
decode_base64_to_image(for PIL Image) / decode_base64_to_image_file (for image file path)
```

--------------------------------

### Response Parsing Functions

Source: https://github.com/kwai-keye/keye/blob/main/evaluation/KC-MMBench/vlmeval/dataset/utils/megabench/README.md

Functions used to parse the raw model output into a structured format suitable for metric calculation.

```python
json
odd_one_out
logical_2d_views_3d_shapes
```