### Install lmms-eval via PyPI

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/README.md

Standard installation command for formal usage.

```bash
pip install lmms-eval
```

--------------------------------

### Setup development environment for lmms-eval

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/model_guide.md

Commands to clone the repository, create a feature branch, and install the package in editable mode.

```sh
# After forking...
git clone https://github.com/<YOUR-USERNAME>/lmms-eval.git
cd lmms-eval
git checkout -b <model-type>
pip install -e .
```

--------------------------------

### Run Video-ChatGPT Evaluation

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/run_examples.md

Installs dependencies and launches Video-ChatGPT evaluation.

```bash
cd /path/to/lmms-eval
python3 -m pip install -e .;

python3 -m pip install sentencepiece av;

TASK=$1
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

accelerate launch --num_processes 8 --main_process_port 12345 -m lmms_eval \
    --model video_chatgpt \
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path ./logs/ 
```

--------------------------------

### Run LLaVA-VID Evaluation

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/run_examples.md

Installs dependencies and launches LLaVA-VID evaluation.

```bash
cd /path/to/lmms-eval
python3 -m pip install -e .;

cd /path/to/LLaVA-NeXT;
python3 -m pip install -e ".[train]";

python3 -m pip install flash-attn --no-build-isolation;

python3 -m pip install av;


TASK=$1
CKPT_PATH=$2
CONV_TEMPLATE=$3
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

accelerate launch --num_processes 8 --main_process_port 12345 -m lmms_eval \
    --model llavavid \
    --model_args pretrained=$CKPT_PATH,conv_template=$CONV_TEMPLATE,video_decode_backend=decord,max_frames_num=32 \
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path ./logs/
```

--------------------------------

### Install LLaVA-NeXT dependencies

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/README.md

Clones the LLaVA-NeXT repository and installs it in editable mode.

```bash
# for llava 1.5
# git clone https://github.com/haotian-liu/LLaVA
# cd LLaVA
# pip install -e .

# for llava-next (1.6)
git clone https://github.com/LLaVA-VL/LLaVA-NeXT
cd LLaVA-NeXT
pip install -e .
```

--------------------------------

### Setup Driver and Screenshotter

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/live_bench/example.ipynb

Configure the screenshotter and driver to capture the website content.

```python
from live_bench.data_generator.utils.extract_information import InfomationExtractor
from live_bench.screen_shoter import get_shoter
from live_bench.driver import load_driver

shoter = get_shoter("single_screen")
driver = load_driver()
w = shoter(driver, website[0])
```

--------------------------------

### Run Video-LLaVA Evaluation

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/run_examples.md

Upgrades transformers and installs dependencies before launching Video-LLaVA evaluation.

```bash
cd /path/to/lmms-eval
python3 -m pip install -e .;

python3 -m pip install transformers --upgrade;
python3 -m pip install av sentencepiece;


TASK=$1
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

accelerate launch --num_processes 8 --main_process_port 12345 -m lmms_eval \
    --model video_llava \
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path ./logs/
```

--------------------------------

### Install Environment Dependencies

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/lmms_eval/tasks/ifeval/README.md

Required Python packages for running the IFEval benchmark.

```bash
pip install langdetect
pip install immutabledict
```

--------------------------------

### Install lmms-eval for development

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/README.md

Clones the repository and installs the package in editable mode.

```bash
git clone https://github.com/EvolvingLMMs-Lab/lmms-eval
cd lmms-eval
pip install -e .
```

--------------------------------

### Install Required Libraries

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/make_image_hf_dataset.ipynb

Install the datasets and Pillow libraries required for dataset creation and image processing.

```python
!pip install datasets Pillow
```

--------------------------------

### Verify ffprobe Installation

Source: https://github.com/wenhaochai/moviechat/blob/main/README.md

Ensure ffprobe is installed correctly before running inference.

```bash
ffprobe -version
```

--------------------------------

### Install MovieChat

Source: https://github.com/wenhaochai/moviechat/blob/main/README.md

Install the MovieChat package from PyPI. Version 0.6.3 is recommended.

```bash
pip install MovieChat
```

--------------------------------

### Install SGLang Dependencies

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/commands.md

Commands to clone the SGLang repository and install the required Python environment and FlashInfer dependencies.

```bash
git clone https://github.com/sgl-project/sglang.git
cd sglang;
pip install -e "python[srt]"
python3 -m pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.3/
```

--------------------------------

### Importing Dataset Libraries

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/live_bench/filter.ipynb

Initial setup to import the necessary classes for dataset manipulation.

```python
from datasets import Dataset, load_dataset
```

--------------------------------

### Run LLaVA Evaluation

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/run_examples.md

Setup the lmms-eval and LLaVA-NeXT repositories before launching the evaluation script.

```bash
cd /path/to/lmms-eval
python3 -m pip install -e .;

cd /path/to/LLaVA-NeXT;
python3 -m pip install -e ".[train]";


TASK=$1
CKPT_PATH=$2
CONV_TEMPLATE=$3
MODEL_NAME=$4
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

#mmbench_en_dev,mathvista_testmini,llava_in_the_wild,mmvet
accelerate launch --num_processes 8 --main_process_port 12345 -m lmms_eval \
    --model llava \
    --model_args pretrained=$CKPT_PATH,conv_template=$CONV_TEMPLATE,model_name=$MODEL_NAME \
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path ./logs/
```

--------------------------------

### Run InternVL-1.5 Evaluation

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/run_examples.md

Fork the InternVL repository and install dependencies before running the evaluation.

```bash
cd /path/to/lmms-eval
python3 -m pip install -e .;

cd /path/to/InternVL/internvl_chat
python3 -m pip install -e .;

python3 -m pip install flash-attn==2.3.6 --no-build-isolation;


TASK=$1
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

accelerate launch --num_processes 8 --main_process_port 12345 -m lmms_eval \
    --model internvl \
    --model_args pretrained="OpenGVLab/InternVL-Chat-V1-5"\
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path ./logs/ 
```

--------------------------------

### Install Java for caption datasets

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/README.md

Installs OpenJDK 8 via conda to support the pycocoeval API, followed by a version check command.

```bash
conda install openjdk=8
```

```bash
java -version
```

--------------------------------

### Install VILA dependencies

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/README.md

Installs the s2wrapper dependency required for testing VILA models.

```bash
pip install s2wrapper@git+https://github.com/bfshi/scaling_on_scales
```

--------------------------------

### Dataset JSON Structure Example

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/make_image_hf_dataset.ipynb

Example of the expected JSON structure for the VQAv2_TOY dataset organization.

```json
{
    "info": { /* some infomation */ },
    "task_type": "TASK_TYPE", "data_type": "mscoco",
    "license": { /* some license */ },
    "questions": [
        {
            "image_id": 262144, // integer id of the image
            "question": "Is the ball flying towards the batter?",
            "question_id": 262144000
        },
        /* ... */
    ]
}
```

--------------------------------

### Run InternVL2 Evaluation

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/run_examples.md

Install required dependencies including flash-attn before launching the evaluation.

```bash
cd /path/to/lmms-eval
python3 -m pip install -e .;


python3 -m pip install flash-attn --no-build-isolation;
python3 -m pip install torchvision einops timm sentencepiece;


TASK=$1
CKPT_PATH=$2
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

accelerate launch --num_processes 8 --main_process_port 12380 -m lmms_eval \
    --model internvl2 \
    --model_args pretrained=$CKPT_PATH \
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path ./logs/ 
```

--------------------------------

### Install Additional Dependencies

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/commands.md

Required packages to resolve potential errors during service execution.

```bash
pip install httpx==0.23.3
pip install protobuf==3.20
pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.3/
```

--------------------------------

### PPL-based Task Configuration

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/task_guide.md

Example YAML configuration for a perplexity-based task using SEED-Bench.

```yaml
dataset_path: lmms-lab/SEED-Bench
dataset_kwargs:
  token: True
task: "seedbench_ppl"
test_split: test
output_type: multiple_choice
doc_to_visual: !function utils.seed_doc_to_visual
doc_to_text: !function utils.seed_doc_to_text_mc
doc_to_choice : !function utils.seed_doc_to_choice
doc_to_target: !function utils.seed_doc_to_mc_target
# Note that the metric name can be either a registed metric function (such as the case for GQA) or a key name returned by process_results
metric_list:
  - metric: acc
metadata:
  - version: 0.0
```

--------------------------------

### Run MPlug-Owl Evaluation

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/run_examples.md

Installs specific legacy versions of dependencies required for MPlug-Owl evaluation.

```bash
cd /path/to/lmms-eval
python3 -m pip install -e .;

# It has to use an old transformers version to run
python3 -m pip install av sentencepiece protobuf==3.20 transformers==4.28.1 einops;

TASK=$1
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

accelerate launch --num_processes 8 --main_process_port 12345 -m lmms_eval \
    --model mplug_owl_video \
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path ./logs/ 
```

--------------------------------

### MME Task Configuration Example

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/task_guide.md

A comprehensive YAML configuration for the MME task, demonstrating dataset paths, output types, and custom function integration.

```yaml
dataset_path: lmms-lab/MME # The name of the dataset as listed by HF in the datasets Hub.
dataset_kwargs:
  token: True # Auxiliary arguments that `datasets.load_dataset` accepts. This can be used to specify arguments such as `data_files` or `data_dir` if you want to use local datafiles such as json or csv.
task: "mme" # The name of the task, this should be registered in the task manager. If successful, you can call lmms_eval with this task name by setting `--tasks mme`.
test_split: test # The split of the dataset to use as the test split.
output_type: generate_until # The type of model output for the given task. Options are `generate_until`, `loglikelihood`, and `multiple_choice`.
doc_to_visual: !function utils.mme_doc_to_visual # The function to process a sample into the appropriate input for the model. 
doc_to_text: !function utils.mme_doc_to_text # The function to process a sample into the appropriate target output for the model.
doc_to_target: "answer" # The function to process a sample into a list of possible string choices for `multiple_choice` tasks.
generation_kwargs: # Auxiliary arguments for the `generate` function from HF transformers library. This would be used in different models files.
  max_new_tokens: 16
  temperature: 0
  top_p: 1.0
  num_beams: 1
  do_sample: false
# The return value of process_results will be used by metrics
process_results: !function utils.mme_process_results
# Note that the metric name can be either a registed metric function (such as the case for GQA) or a key name returned by process_results
# e.g. Following metrics `mme_perception_score` is custom defined. 
# So `mme_process_results` function should return the dict `{"mme_perception_score": {sub_k:sub_v, ..., } }`
# And the `mme_aggregate_results` function could get the dict `{sub_k:sub_v, ..., }`, and use the information to gather the final accuracy.
metric_list:
  - metric: mme_percetion_score # The name of the metric to use for evaluation. The process_results function should return the metric name and the metric value, in format of `{metric_name: results}`. And the aggregation function will use the results to get the final score.
    aggregation: !function utils.mme_aggregate_results # The name of the aggregation function to use for evaluation.
    higher_is_better: true # Whether the metric is better when the value is higher.
  - metric: mme_cognition_score
    aggregation: !function utils.mme_aggregate_results
    higher_is_better: true
lmms_eval_specific_kwargs:
  default:
    pre_prompt: ""
    post_prompt: "\nAnswer the question using a single word or phrase."
  qwen_vl:  
    pre_prompt: ""
    post_prompt: " Answer:"
metadata:
  - version: 0.0
```

--------------------------------

### Configure Weights and Biases logging

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/commands.md

Example usage of the --wandb_args flag to specify project and run name for tracking.

```bash
--wandb_args project=test-project,name=test-run
```

--------------------------------

### Generation-based Task Configuration

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/task_guide.md

Example YAML configuration for a generation-based task using MME as a reference.

```yaml
dataset_path: lmms-lab/MME
dataset_kwargs:
  token: True
task: "mme"
test_split: test
output_type: generate_until
doc_to_visual: !function utils.mme_doc_to_visual
doc_to_text: !function utils.mme_doc_to_text
doc_to_target: "answer"
generation_kwargs:
  max_new_tokens: 16
  temperature: 0
  top_p: 1.0
  num_beams: 1
  do_sample: false
# The return value of process_results will be used by metrics
process_results: !function utils.mme_process_results
# Note that the metric name can be either a registed metric function (such as the case for GQA) or a key name returned by process_results
# e.g. Following metrics `mme_perception_score` is custom defined. 
# So `mme_process_results` function should return the dict `{"mme_perception_score": {sub_k:sub_v, ..., } }`
# And the `mme_aggregate_results` function could get the dict `{sub_k:sub_v, ..., }`, and use the information to gather the final accuracy.
metric_list:
  - metric: mme_percetion_score
    aggregation: !function utils.mme_aggregate_results
    higher_is_better: true
  - metric: mme_cognition_score
    aggregation: !function utils.mme_aggregate_results
    higher_is_better: true
lmms_eval_specific_kwargs:
  default:
    pre_prompt: ""
    post_prompt: "\nAnswer the question using a single word or phrase."
  qwen_vl:  
    pre_prompt: ""
    post_prompt: " Answer:"
metadata:
  - version: 0.0
```

--------------------------------

### Multi-round Generation-based Task Configuration

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/task_guide.md

Example YAML configuration for a multi-round generation task using MMSearch.

```yaml
dataset_path: CaraJ/MMSearch
dataset_name: end2end
dataset_kwargs:
  token: False
task: "mmsearch_end2end"
test_split: end2end
output_type: generate_until_multi_round # Note that here we use the new output_type here for multi-round generation. It basicly follows generate_until but incorporate multi-round inference
doc_to_visual: !function lmms_eval_utils.mmsearch_end2end_doc_to_visual
doc_to_text: !function lmms_eval_utils.mmsearch_end2end_doc_to_text
doc_to_target: "answer"
generation_kwargs:
  until:
    - "ASSISTANT:"
  max_new_tokens: 512
  temperature: 0
  top_p: 0
  num_beams: 1
  do_sample: false
process_results: !function lmms_eval_utils.mmsearch_end2end_process_results
metric_list:
  - metric: end2end_f1_score
    aggregation: !function lmms_eval_utils.mmsearch_aggregate_results_f1_score
    higher_is_better: true
  - metric: requery_score
    aggregation: !function lmms_eval_utils.mmsearch_aggregate_results_req_score
    higher_is_better: true
lmms_eval_specific_kwargs: # Note that here we cache the result of every sample whenever the it is inferenced
  middle_resules_dir: /data1/zrr/jdz/mmsearch/mmsearch_middile_results
  result_cache_dir: /data1/zrr/jdz/mmsearch/mmsearch_result_cache_dir
```

--------------------------------

### Metric List Configuration

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/task_guide.md

Example of defining multiple metrics within the metric_list argument, including auxiliary arguments for metric functions.

```yaml
metric_list:
  - metric: acc
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
    ignore_case: true
    ignore_punctuation: false
    regexes_to_ignore:
      - ","
      - "\\$"
```

--------------------------------

### Filter Dataset (Commented)

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/live_bench/script/update_banchmark.ipynb

Example of filtering out specific models from the dataset.

```python
# data = data.filter(lambda x: x["Model Name"] != "LLaVA-OneVision-1.5B")
```

--------------------------------

### Resolve Common Environment Issues

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/run_examples.md

Install specific versions of dependencies to resolve conflicts with httpx, protobuf, or numpy.

```bash
python3 -m pip install httpx==0.23.3;
python3 -m pip install protobuf==3.20;
# If you are using numpy==2.x, sometimes may causing errors
python3 -m pip install numpy==1.26;
# Someties sentencepiece are required for tokenizer to work
python3 -m pip install sentencepiece;
```

--------------------------------

### Chess Board Representation Example

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/live_bench/script/select.ipynb

A Python implementation to represent and print a chess board from a 2D list.

```python
def create_chess_board():
    board = [
        ['r', '0', '0', '0', '0', '0', 'k', 'r'],
        ['p', '0', '0', 'P', '0', '0', '0', 'b'],
        ['0', '0', 'p', '0', '0', '0', 'p', '0'],
        ['0', '0', '0', '0', 'Q', 'P', '0', '0'],
        ['q', '0', '0', '0', '0', '0', '0', '0'],
        ['0', '0', '0', '0', '0', '0', '0', '0'],
        ['P', 'P', 'P', '0', '0', '0', 'P', 'P'],
        ['0', '0', '0', 'R', '0', 'R', 'K', '0']
    ]
    return board


def print_board(board):
    for row in board:
        print(' '.join(piece if piece != '0' else '.' for piece in row))


chess_board = create_chess_board()
print_board(chess_board)
```

```python
def create_chess_board():
    board = [
        ["r", "0", "0", "0", "0", "0", "k", "r"],
        ["p", "0", "0", "P", "0", "0", "0", "b"],
        ["0", "0", "p", "0", "0", "0", "p", "0"],
        ["0", "0", "0", "0", "Q", "P", "0", "0"],
        ["q", "0", "0", "0", "0", "0", "0", "0"],
        ["0", "0", "0", "0", "0", "0", "0", "0"],
        ["P", "P", "P", "0", "0", "0", "P", "P"],
        ["0", "0", "0", "R", "0", "R", "K", "0"],
    ]
    return board


def print_board(board):
    for row in board:
        print(" ".join(piece if piece != "0" else "." for piece in row))


chess_board = create_chess_board()
print_board(chess_board)
```

--------------------------------

### Initialize Website

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/live_bench/example.ipynb

Define the target website for the extraction process.

```python
from random import sample

from live_bench.websites.website import DefaultWebsite
from live_bench.websites import load_websites

# website = load_websites()
# website = sample(website, 1)
# website[0].url
website = [DefaultWebsite(url="https://www.asahi.com/")]  # , DefaultWebsite(url="https://www.bbc.com/sport"), DefaultWebsite(url="https://www.bbc.com/business")
```

--------------------------------

### Initialize LiveBench

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/live_bench/example.ipynb

Import the LiveBench class and initialize the dataset instance.

```python
from live_bench import LiveBench
```

```python
dataset = LiveBench(force_clear=True)
```

--------------------------------

### Run InstructBLIP Evaluation

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/run_examples.md

Configures the environment and launches an evaluation task for InstructBLIP using accelerate.

```bash
cd /path/to/lmms-eval
python3 -m pip install -e .;

python3 -m pip install transformers --upgrade;

CKPT_PATH=$1
TASK=$2
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

accelerate launch --num_processes 8 --main_process_port 12345 -m lmms_eval \
    --model instructblip \
    --model_args pretrained=$CKPT_PATH \
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix instructblip \
    --output_path ./logs/ 
```

--------------------------------

### Run GPT Evaluation

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/run_examples.md

Configures the environment and launches GPT-4V evaluation using accelerate.

```bash
cd /path/to/lmms-eval
python3 -m pip install -e .;

export OPENAI_API_KEY="<YOUR_API_KEY>"

TASK=$1
MODEL_VERSION=$2
MODALITIES=$3
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

accelerate launch --num_processes 8 --main_process_port 30000 -m lmms_eval \
    --model gpt4v \
    --model_args model_version=$MODEL_VERSION,modality=$MODALITIES\
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path ./logs/
```

--------------------------------

### Configure Environment Variables

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/run_examples.md

Export necessary API keys and paths before running experiments.

```bash
export OPENAI_API_KEY="<YOUR_API_KEY>"
export HF_HOME="<Path to HF cache>" 
export HF_TOKEN="<YOUR_API_KEY>"
export HF_HUB_ENABLE_HF_TRANSFER="1"
export REKA_API_KEY="<YOUR_API_KEY>"
# Other possible environment variables include 
# ANTHROPIC_API_KEY,DASHSCOPE_API_KEY etc.
```

--------------------------------

### Download and Extract Dataset

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/make_image_hf_dataset.ipynb

Download the sample dataset zip file and extract it into the local data directory.

```bash
!wget https://huggingface.co/datasets/lmms-lab/VQAv2_TOY/resolve/main/source_data/sample_data.zip -P data
!unzip data/sample_data.zip -d data
```

--------------------------------

### Create Conda Environment

Source: https://github.com/wenhaochai/moviechat/blob/main/README.md

Initializes the project environment from the provided configuration file.

```bash
conda env create -f environment.yml
conda activate moviechat
```

--------------------------------

### Evaluate video models

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/README.md

Run evaluation for video-capable models like LLaVA-NeXT-Video-32B with specific backend and frame settings.

```bash
accelerate launch --num_processes 8 --main_process_port 12345 -m lmms_eval \
    --model llavavid \
    --model_args pretrained=lmms-lab/LLaVA-NeXT-Video-32B-Qwen,conv_template=qwen_1_5,video_decode_backend=decord,max_frames_num=32，mm_spatial_pool_mode=average,mm_newline_position=grid,mm_resampler_location=after \
    --tasks videomme \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_vid_32B \
    --output_path ./logs/
```

--------------------------------

### Evaluate using configuration files

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/README.md

Execute evaluation using a YAML configuration file to manage multiple models and datasets.

```bash
python3 -m accelerate.commands.launch --num_processes=8 -m lmms_eval --config ./miscs/example_eval.yaml
```

--------------------------------

### Create a new model file

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/model_guide.md

Command to initialize a new Python file for the custom model implementation.

```sh
touch lmms_eval/models/<my_model_filename>.py
```

--------------------------------

### Initialize Information Extractor

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/live_bench/example.ipynb

Create an instance of the InformationExtractor class.

```python
extractor = InfomationExtractor()
```

--------------------------------

### Run Claude Evaluation

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/run_examples.md

Configures the environment and launches Claude evaluation using accelerate.

```bash
cd /path/to/lmms-eval
python3 -m pip install -e .;

export ANTHROPIC_API_KEY="<YOUR_API_KEY>"

TASK=$1
MODEL_VERSION=$2
MODALITIES=$3
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

accelerate launch --num_processes 8 --main_process_port 12345 -m lmms_eval \
    --model claude \
    --model_args model_version=$MODEL_VERSION\
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path ./logs/
```

--------------------------------

### Run Gradio Inference Demo

Source: https://github.com/wenhaochai/moviechat/blob/main/Gradio_demo/GRADIO_DEMO.md

Executes the Gradio application with the specified configuration file and GPU ID.

```shell
python app_gradio.py \
    --cfg-path eval_configs/MovieChat.yaml \
    --gpu-id 0 \
```

--------------------------------

### Initialize Dataset Features and Structure

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/makecvrr.ipynb

Define the schema for the dataset and populate a dictionary from an existing DataFrame.

```python
from datasets import Dataset, Features, Value
import pandas as pd

# Define the features for the dataset
features = Features(
    {
        "Q": Value(dtype="string"),
        "A": Value(dtype="string"),
        "DimensionName": Value(dtype="string"),
        "VideoID": Value(dtype="string"),
    }
)

df_items = {
    "Q": [],
    "A": [],
    "DimensionName": [],
    "VideoID": [],
}

for idx, row in df.iterrows():
    df_items["Q"].append(str(row["Q"]))
    df_items["A"].append(str(row["A"]))
    df_items["DimensionName"].append(str(row["DimensionName"]))
    df_items["VideoID"].append(str(row["VideoID"]))

df_items = pd.DataFrame(df_items)
```

--------------------------------

### Initialize Dataset Features and Process Video Data

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/make_video_hf_dataset.ipynb

Define the schema for the dataset and populate a pandas DataFrame by reading video descriptions from local files.

```python
from datasets import Dataset, Features, Value, Image
import pandas as pd
from tqdm import tqdm
import os

# Define the features for the dataset
features = Features(
    {
        "video_name": Value(dtype="string"),
        "question": Value(dtype="string"),
        "answer": Value(dtype="string"),
    }
)

df_items = {
    "video_name": [],
    "question": [],
    "answer": [],
}

description_root = "/mnt/bn/vl-research/workspace/yhzhang/data/llava_video/video_detail_description/Test_Human_Annotated_Captions"
videos = os.listdir(description_root)
for cur_video_name in tqdm(videos):
    sample_set = {}
    video_name = cur_video_name.split(".")[0]
    with open(f"{description_root}/{cur_video_name}", encoding="utf-8-sig") as f:
        description = f.readlines()[0]
    question = "Please provide a detailed description of the video, focusing on the main subjects, their actions, and the background scenes"
    df_items["video_name"].append(video_name)
    df_items["question"].append(question)
    df_items["answer"].append(description)
    # Add other fields as necessary

df_items = pd.DataFrame(df_items)
```

--------------------------------

### Evaluate with naive model sharding

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/README.md

Perform evaluation for large models using device_map=auto for automatic sharding.

```bash
python3 -m lmms_eval \
    --model=llava \
    --model_args=pretrained=lmms-lab/llava-next-72b,conv_template=qwen_1_5,device_map=auto,model_name=llava_qwen \
    --tasks=pope,vizwiz_vqa_val,scienceqa_img \
    --batch_size=1 \
    --log_samples \
    --log_samples_suffix=llava_qwen \
    --output_path="./logs/" \
    --wandb_args=project=lmms-eval,job_type=eval,entity=llava-vl
```

--------------------------------

### Run Xcomposer Evaluation

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/run_examples.md

Commands for evaluating Xcomposer-4KHD and Xcomposer-2d5 models.

```bash
cd /path/to/lmms-eval
python3 -m pip install -e .;


python3 -m pip install flash-attn --no-build-isolation;
python3 -m pip install torchvision einops timm sentencepiece;

TASK=$1
MODALITY=$2
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

# For Xcomposer2d5
accelerate launch --num_processes 8 --main_process_port 10000 -m lmms_eval \
    --model xcomposer2d5 \
    --model_args pretrained="internlm/internlm-xcomposer2d5-7b",device="cuda",modality=$MODALITY\
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path ./logs/ 

# For Xcomposer-4kHD
accelerate launch --num_processes 8 --main_process_port 12345 -m lmms_eval \
    --model xcomposer2_4khd \
    --model_args pretrained="internlm/internlm-xcomposer2-4khd-7b" \
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path ./logs/
```

--------------------------------

### Run Inference for Video QA Datasets

Source: https://github.com/wenhaochai/moviechat/blob/main/eval_code/Acc_score/EVAL_SHORT_VIDEO_QA.md

Commands to execute inference on MSVD-QA, MSRVTT-QA, and ActivityNet-QA datasets. Ensure paths to video files, ground truth files, and output directories are correctly configured.

```shell
python run_inference_qa_msvd.py \
    --cfg-path eval_configs/MovieChat.yaml \
    --gpu-id 0 \
    --num-beams 1 \
    --temperature 1.0 \
    --video-path /path/to/your/video \
    --gt_file /path/to/your/question and answer file \
    --output_dir /path/to/your/output \
    --output_name msvd-qa \
    --fragment-video-path src/video_fragment/output.mp4 \
```

```shell
python run_inference_qa_msrvtt.py \
    --cfg-path eval_configs/MovieChat.yaml \
    --gpu-id 0 \
    --num-beams 1 \
    --temperature 1.0 \
    --video-path /path/to/your/video \
    --gt_file /path/to/your/question and answer file \
    --output_dir /path/to/your/output \
    --output_name msrvtt-qa \
    --fragment-video-path src/video_fragment/output.mp4 \
```

```shell
python run_inference_qa_activitynet.py \
    --cfg-path eval_configs/MovieChat.yaml \
    --gpu-id 0 \
    --num-beams 1 \
    --temperature 1.0 \
    --video-path /path/to/your/video \
    --gt_file /path/to/your/question file \
    --gt_file_answers /path/to/your/answer file \
    --output_dir /path/to/your/output \
    --output_name activitynet-qa \
    --fragment-video-path src/video_fragment/output_act.mp4 \
```

--------------------------------

### Evaluate LLaVA on multiple datasets

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/README.md

Execute evaluation for LLaVA v1.5-7b across multiple benchmarks simultaneously.

```bash
python3 -m accelerate.commands.launch \
    --num_processes=8 \
    -m lmms_eval \
    --model llava \
    --model_args pretrained="liuhaotian/llava-v1.5-7b" \
    --tasks mme,mmbench_en \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_v1.5_mme_mmbenchen \
    --output_path ./logs/
```

--------------------------------

### Evaluate LLaVA on MME

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/README.md

Run evaluation for LLaVA v1.5-7b on the MME benchmark using 8 processes.

```bash
python3 -m accelerate.commands.launch \
    --num_processes=8 \
    -m lmms_eval \
    --model llava \
    --model_args pretrained="liuhaotian/llava-v1.5-7b" \
    --tasks mme \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_v1.5_mme \
    --output_path ./logs/
```

--------------------------------

### Run Local Inference Demo

Source: https://github.com/wenhaochai/moviechat/blob/main/README.md

Executes the MovieChat inference script with specified configuration and video inputs.

```bash
python inference.py \
    --cfg-path eval_configs/MovieChat.yaml \
    --gpu-id 0 \
    --num-beams 1 \
    --temperature 1.0 \
    --text-query "What is he doing?" \
    --video-path src/examples/Cooking_cake.mp4 \
    --fragment-video-path src/video_fragment/output.mp4 \
    --cur-min 1 \
    --cur-sec 1 \
    --middle-video 1
```

--------------------------------

### Run SRT API Model Evaluation

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/run_examples.md

Sets up the SGLang environment and executes evaluation tasks using the SRT API model.

```bash
cd /path/to/lmms-eval
python3 -m pip install -e .;

cd /path/to/sglang;
python3 -m pip install -e "python[all]";


python3 -m pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.3/


CKPT_PATH=$1
TASK=$2
MODALITY=$3
TP_SIZE=$4
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

python3 -m lmms_eval \
    --model srt_api \
    --model_args modality=$MODALITY,model_version=$CKPT_PATH,tp=$TP_SIZE,host=127.0.0.1,port=30000,timeout=600 \
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path ./logs/
```

--------------------------------

### Loading the LiveBench Dataset

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/live_bench/filter.ipynb

Load the specified version of the LiveBench dataset from the Hugging Face Hub.

```python
data = load_dataset("lmms-lab/LiveBench", "2024-07")
```

--------------------------------

### Load and Convert Dataset

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/live_bench/summerize.ipynb

Load a dataset from the Hub and convert it to a pandas DataFrame.

```python
new_data = load_dataset("lmms-lab/LiveBench", "2024-09", split="test")
df = new_data.to_pandas()
```

--------------------------------

### Using evaluate for core evaluation functionality

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/commands.md

Shows how to use the evaluate function with a custom task dictionary and task manager.

```python
import lmms_eval

# suppose you've defined a custom lm_eval.api.Task subclass in your own external codebase
from my_tasks import MyTask1
...

# create your model (could be running finetuning with some custom modeling code)
my_model = initialize_my_model()
...

# instantiate an LM subclass that takes your initialized model and can run
# - `Your_LM.loglikelihood()`
# - `Your_LM.loglikelihood_rolling()`
# - `Your_LM.generate_until()`
lmm_obj = Your_LMM(model=my_model, batch_size=16)

# optional: the task_manager indexes tasks including ones
# specified by the user through `include_path`.
task_manager = lmms_eval.tasks.TaskManager(
    include_path="/path/to/custom/yaml"
    )

# To get a task dict for `evaluate`
task_dict = lmms_eval.tasks.get_task_dict(
    [
        "mmlu", # A stock task
        "my_custom_task", # A custom task
        {
            "task": ..., # A dict that configures a task
            "doc_to_text": ...,
            },
        MyTask1 # A task object from `lm_eval.task.Task`
        ],
    task_manager # A task manager that allows lm_eval to
                 # load the task during evaluation.
                 # If none is provided, `get_task_dict`
                 # will instantiate one itself, but this
                 # only includes the stock tasks so users
                 # will need to set this if including
                 # custom paths is required.
    )

results = evaluate(
    lm=lmm_obj,
    task_dict=task_dict,
    ...
)
```

--------------------------------

### Run LLaMA-VID Evaluation

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/run_examples.md

Executes evaluation while maintaining the LLaMA-VID directory context for configuration access.

```bash
cd /path/to/lmms-eval
python3 -m pip install -e .;

# Notice that you should not leave the folder of LLaMA-VID when calling lmms-eval
# Because they left their processor's config inside the repo
cd /path/to/LLaMA-VID;
python3 -m pip install -e .

python3 -m pip install av sentencepiece;

TASK=$1
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

accelerate launch --num_processes 8 --main_process_port 12345 -m lmms_eval \
    --model llama_vid \
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path ./logs/
```

--------------------------------

### Load LiveBench Dataset

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/live_bench/summerize.ipynb

Imports the load_dataset function and retrieves the test split of the LiveBench dataset.

```python
from datasets import load_dataset
```

```python
data = load_dataset("lmms-lab/LiveBench", "2024-09", split="test")
```

--------------------------------

### Define and format model names

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/live_bench/script/compare.ipynb

Prepare model name strings by replacing hyphens with underscores for consistent variable naming.

```python
model_1 = "gpt-4o"
model_2 = "Qwen-VL-72B-Instruct"

model_1 = model_1.replace("-", "_")
model_2 = model_2.replace("-", "_")
```

--------------------------------

### Print Response

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/live_bench/example.ipynb

Output the extracted information as a string.

```python
print(str(response))
```

--------------------------------

### Run SGLang Backend Service

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/docs/commands.md

Shell script to configure environment variables and execute the lmms_eval service using the SRT API.

```bash

CKPT_PATH=$1
TASK=$2
MODALITY=$3
TP_SIZE=$4
echo $TASK
TASK_SUFFIX="${TASK//,/_}"
echo $TASK_SUFFIX

python3 -m lmms_eval \
    --model srt_api \
    --model_args modality=$MODALITY,model_version=$CKPT_PATH,tp=$TP_SIZE,host=127.0.0.1,port=30000,timeout=600 \
    --tasks $TASK \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $TASK_SUFFIX \
    --output_path ./logs/
```

--------------------------------

### Cite MMLU-Pro in BibTeX

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/lmms_eval/tasks/mmlu_pro/README.md

Use this BibTeX entry to cite the MMLU-Pro benchmark in academic publications.

```bibtex
@misc{wang2024mmlupro,
      title={MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark},
      author={Yubo Wang and Xueguang Ma and Ge Zhang and Yuansheng Ni and Abhranil Chandra and Shiguang Guo and Weiming Ren and Aaran Arulraj and Xuan He and Ziyan Jiang and Tianle Li and Max Ku and Kai Wang and Alex Zhuang and Rongqi Fan and Xiang Yue and Wenhu Chen},
      year={2024},
      eprint={2406.01574},
      archivePrefix={arXiv},
      primaryClass={id='cs.CL' full_name='Computation and Language' is_active=True alt_name='cmp-lg' in_archive='cs' is_general=False description='Covers natural language processing. Roughly includes material in ACM Subject Class I.2.7. Note that work on artificial languages (programming languages, logics, formal systems) that does not explicitly address natural-language issues broadly construed (natural-language processing, computational linguistics, speech, text retrieval, etc.) is not appropriate for this area.'}
}
```

--------------------------------

### Cite SeeClick Paper

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/lmms_eval/tasks/screenspot/README.md

BibTeX citation for the SeeClick paper which serves as the reference for the ScreenSpot benchmark.

```bibtex
@misc{cheng2024seeclick,
      title={SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents}, 
      author={Kanzhi Cheng and Qiushi Sun and Yougang Chu and Fangzhi Xu and Yantao Li and Jianbing Zhang and Zhiyong Wu},
      year={2024},
      eprint={2401.10935},
      archivePrefix={arXiv},
      primaryClass={cs.HC}
}
```

--------------------------------

### Run MovieChat Inference

Source: https://github.com/wenhaochai/moviechat/blob/main/README.md

Initialize the MovieChat model and process a video to generate an answer to a question. Requires a CUDA-enabled device.

```python
from PIL import Image
import cv2

from MovieChat.processors.video_processor import AlproVideoEvalProcessor
from MovieChat.models.chat_model import Chat
from MovieChat.models.moviechat import MovieChat

device = 'cuda:0'
print('Initializing Chat')
moviechat_model = MovieChat.from_config(device=device).to(device)
vis_processor_cfg = {'name': 'alpro_video_eval', 'n_frms': 8, 'image_size': 224}
frame_processor = AlproVideoEvalProcessor.from_config(vis_processor_cfg)
chat = Chat(moviechat_model, frame_processor, device=device)
print('Initialization Finished')

video_path = "Your video path, end with mp4"
fragment_video_path = "The path to store tmp video clips"
middle_video = False # True->Breakpoint mode, False->Global mode
question = "Your Question"
cur_min = 0 # Change it when Breakpoint mode
cur_sec = 0 # Change it when Breakpoint mode

cap = cv2.VideoCapture(video_path)
cur_fps = cap.get(cv2.CAP_PROP_FPS)
cap.set(cv2.CAP_PROP_POS_FRAMES, cur_fps)
ret, frame = cap.read()
rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
pil_image = Image.fromarray(rgb_frame)
image = chat.image_vis_processor(pil_image).unsqueeze(0).unsqueeze(2).half().to(device)
cur_image = chat.model.encode_image(image)

img_list = []
msg = chat.upload_video_without_audio(
    video_path=video_path, 
    fragment_video_path=fragment_video_path,
    cur_min=cur_min, 
    cur_sec=cur_sec, 
    cur_image=cur_image, 
    img_list=img_list, 
    middle_video=middle_video,
    question=question
)
answer = chat.answer(
    img_list=img_list,
    input_text=question,
    msg = msg,
    num_beams=1,
    temperature=1.0,
    max_new_tokens=300,
    max_length=2000)[0]

print(answer)
```

--------------------------------

### Load LiveBench dataset

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/live_bench/script/compare.ipynb

Load the specified version of the LiveBenchDetailedResults dataset using the datasets library.

```python
import datasets

data = datasets.load_dataset("lmms-lab/LiveBenchDetailedResults", "2024-09")
```

--------------------------------

### Group and Upload Dataset to Hub

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/make_image_hf_dataset.ipynb

Organizes dataset splits into a DatasetDict and pushes the collection to the Hugging Face Hub.

```python
data = datasets.DatasetDict({"val": data_val, "test": data_test, "test_dev": data_test_dev})
```

```python
data.push_to_hub("lmms-lab/VQAv2_TOY")  # replace lmms-lab to your username
```

--------------------------------

### Evaluate larger LLaVA models

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/README.md

Run evaluation for larger models like LLaVA-v1.6-34b with specific conversation templates.

```bash
python3 -m accelerate.commands.launch \
    --num_processes=8 \
    -m lmms_eval \
    --model llava \
    --model_args pretrained="liuhaotian/llava-v1.6-34b,conv_template=mistral_direct" \
    --tasks mme,mmbench_en \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_v1.5_mme_mmbenchen \
    --output_path ./logs/
```

--------------------------------

### Citation for Video-ChatGPT

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/lmms_eval/tasks/video_detail_description/README.md

BibTeX citation for the Video-ChatGPT research paper.

```bibtex
@article{Maaz2023VideoChatGPT,
    title={Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models},
    author={Maaz, Muhammad and Rasheed, Hanoona and Khan, Salman and Khan, Fahad Shahbaz},
    journal={arXiv:2306.05424},
    year={2023}
}
```

--------------------------------

### Authenticate with Hugging Face

Source: https://github.com/wenhaochai/moviechat/blob/main/MovieChat_Onevision/tools/make_image_hf_dataset.ipynb

Log in to the Hugging Face CLI using a personal API token.

```bash
!huggingface-cli login --token hf_YOUR_HF_TOKEN # replace hf_YOUR_HF_TOKEN to your own Hugging Face token.
```