### Set Up CLIP Benchmark Development Environment

Source: https://github.com/opengvlab/internvl/blob/main/clip_benchmark/README.md

Clone the repository and install the package locally for development. This involves cloning the Git repository and running the setup script.

```bash
git clone https://github.com/LAION-AI/CLIP_benchmark
cd CLIP_benchmark
python setup.py install
```

--------------------------------

### Setup CLIP Benchmark

Source: https://github.com/opengvlab/internvl/blob/main/clip_benchmark/README.md

Installs the CLIP benchmark package in development mode. Alternatively, you can add the current directory to your PYTHONPATH.

```bash
python setup.py develop
```

```bash
export PYTHONPATH="${PYTHONPATH}:$(pwd)"
```

--------------------------------

### Install Additional Training Packages

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat_llava/README.md

Install 'ninja' and 'flash-attn' for training functionalities. These are optional dependencies.

```shell
pip install ninja
pip install flash-attn --no-build-isolation
```

--------------------------------

### Install Dependencies

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat_llava/README.md

Installs or upgrades pip and then installs the project's requirements. Ensure you are in the project's root directory.

```bash
pip install --upgrade pip  # enable PEP 660 support
pip install -e .
```

--------------------------------

### Install Default Dependencies

Source: https://github.com/opengvlab/internvl/blob/main/INSTALLATION.md

Install the core dependencies for InternVL using the provided requirements.txt file.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install Webdataset Library

Source: https://github.com/opengvlab/internvl/blob/main/clip_benchmark/README.md

Install the webdataset library using pip. This is a prerequisite for working with webdatasets.

```bash
pip install webdataset
```

--------------------------------

### LMDeploy 'Hello, world' Example

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat/README.md

A basic example demonstrating how to use LMDeploy to load a model and perform image description. Loads an image from a URL and uses the Turbomind engine for inference. Requires `lmdeploy` and `lmdeploy.vl`.

```python
from lmdeploy import pipeline, TurbomindEngineConfig
from lmdeploy.vl import load_image

model = 'OpenGVLab/InternVL2_5-8B'
image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
pipe = pipeline(model, backend_config=TurbomindEngineConfig(session_len=8192))
response = pipe(('describe this image', image))
print(response.text)
```

--------------------------------

### Install Clip Benchmark Dependencies

Source: https://github.com/opengvlab/internvl/blob/main/INSTALLATION.md

Manually install dependencies for clip benchmark functionality if needed, as it's not included by default.

```bash
pip install -r requirements/clip_benchmark.txt
```

--------------------------------

### Compile Flash Attention from Source

Source: https://github.com/opengvlab/internvl/blob/main/INSTALLATION.md

Alternative method to install flash-attention by cloning the repository, checking out a specific version, and installing from source.

```bash
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention
git checkout v2.3.6
python setup.py install
```

--------------------------------

### Install Flash Attention

Source: https://github.com/opengvlab/internvl/blob/main/INSTALLATION.md

Install flash-attn version 2.3.6. This command uses --no-build-isolation for potentially faster installation.

```bash
pip install flash-attn==2.3.6 --no-build-isolation
```

--------------------------------

### Training Start Log

Source: https://github.com/opengvlab/internvl/blob/main/classification/work_dirs/intern_vit_6b_1k_224/log_rank0.txt

Confirms the initiation of the training process.

```python
[2023-11-09 22:24:18 intern_vit_6b_1k_224_cls_patch_sgd_lr0.1] (main.py 307): INFO Start training
```

--------------------------------

### Launch LLaVA Controller

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat_llava/docs/LoRA.md

Starts the LLaVA controller service. This is the first step in launching the web UI.

```Shell
python -m llava.serve.controller --host 0.0.0.0 --port 10000
```

--------------------------------

### Install Apex with CUDA Extensions

Source: https://github.com/opengvlab/internvl/blob/main/INSTALLATION.md

Install the Apex library, including CUDA extensions, for segmentation. This involves cloning the repo, checking out a specific commit, and installing with build options. Ensure your environment is set up for CUDA compilation.

```bash
git clone https://github.com/NVIDIA/apex.git
git checkout 2386a912164b0c5cfcd8be7a2b890fbac5607c82  # https://github.com/NVIDIA/apex/issues/1735
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./
```

--------------------------------

### Import Libraries and Setup

Source: https://github.com/opengvlab/internvl/blob/main/clip_benchmark/benchmark/results.ipynb

Imports necessary libraries and sets up global configurations for pandas and matplotlib. Includes a helper function to extract model architecture details.

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
pd.options.display.float_format = '{:.3f}'.format
def extract_arch(model):
    vit, size, patch_size, *rest = model.split("-")
    return vit+"-"+size+"-"+patch_size
plt.rcParams['figure.dpi'] = 200
```

--------------------------------

### Clone the repository

Source: https://github.com/opengvlab/internvl/blob/main/clip_benchmark/CONTRIBUTING.rst

Clone your forked repository locally to start development.

```shell
$ git clone git@github.com:your_name_here/clip_benchmark.git
```

--------------------------------

### Install OpenAI Client

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat/README.md

Install the OpenAI Python package to interact with LMDeploy's OpenAI-compatible API.

```shell
pip install openai
```

--------------------------------

### Training Progress Log Example

Source: https://github.com/opengvlab/internvl/blob/main/classification/work_dirs/intern_vit_6b_1k_224/log_rank0.txt

Shows a snapshot of the training progress, including epoch, batch, ETA, learning rate, time, loss, gradient norm, and memory usage.

```python
[2023-11-09 22:24:27 intern_vit_6b_1k_224_cls_patch_sgd_lr0.1] (main.py 506): INFO Train: [0/10][0/1251]	lr 0.000000	time 9.1551 (9.1551)	model_time 5.9911 (5.9911)	loss 6.9080 (6.9080)	grad_norm 0.0390 (0.0390/0.0000)	mem 48414MB
```

```python
[2023-11-09 22:24:51 intern_vit_6b_1k_224_cls_patch_sgd_lr0.1] (main.py 506): INFO Train: [0/10][10/1251]	lr 0.001599	time 2.3801 (2.9854)	model_time 2.3799 (2.6974)	loss 6.9079 (6.9081)	grad_norm 0.0403 (0.0404/0.0008)	mem 48463MB
```

```python
[2023-11-09 22:25:15 intern_vit_6b_1k_224_cls_patch_sgd_lr0.1] (main.py 506): INFO Train: [0/10][20/1251]	lr 0.003197	time 2.3878 (2.6994)	model_time 2.3875 (2.5484)	loss 6.9083 (6.9080)	grad_norm 0.0404 (0.0402/0.0007)	mem 48463MB
```

```python
[2023-11-09 22:25:39 intern_vit_6b_1k_224_cls_patch_sgd_lr0.1] (main.py 506): INFO Train: [0/10][30/1251]	lr 0.004796	time 2.3913 (2.5997)	model_time 2.3911 (2.4972)	loss 6.9086 (6.9080)	grad_norm 0.0410 (0.0402/0.0007)	mem 48463MB
```

```python
[2023-11-09 22:26:03 intern_vit_6b_1k_224_cls_patch_sgd_lr0.1] (main.py 506): INFO Train: [0/10][40/1251]	lr 0.006395	time 2.3897 (2.5483)	model_time 2.3895 (2.4708)	loss 6.9069 (6.9079)	grad_norm 0.0402 (0.0401/0.0007)	mem 48463MB
```

```python
[2023-11-09 22:26:26 intern_vit_6b_1k_224_cls_patch_sgd_lr0.1] (main.py 506): INFO Train: [0/10][50/1251]	lr 0.007994	time 2.3920 (2.5173)	model_time 2.3918 (2.4549)	loss 6.9076 (6.9079)	grad_norm 0.0389 (0.0401/0.0008)	mem 48463MB
```

```python
[2023-11-09 22:26:51 intern_vit_6b_1k_224_cls_patch_sgd_lr0.1] (main.py 506): INFO Train: [0/10][60/1251]	lr 0.009592	time 2.3922 (2.4997)	model_time 2.3920 (2.4475)	loss 6.9084 (6.9078)	grad_norm 0.0402 (0.0400/0.0008)	mem 48463MB
```

```python
[2023-11-09 22:27:14 intern_vit_6b_1k_224_cls_patch_sgd_lr0.1] (main.py 506): INFO Train: [0/10][70/1251]	lr 0.011191	time 2.3895 (2.4844)	model_time 2.3892 (2.4395)	loss 6.9079 (6.9078)	grad_norm 0.0399 (0.0400/0.0008)	mem 48463MB
```

```python
[2023-11-09 22:27:38 intern_vit_6b_1k_224_cls_patch_sgd_lr0.1] (main.py 506): INFO Train: [0/10][80/1251]	lr 0.012790	time 2.3893 (2.4729)	model_time 2.3891 (2.4334)	loss 6.9083 (6.9077)	grad_norm 0.0389 (0.0400/0.0008)	mem 48463MB
```

```python
[2023-11-09 22:28:02 intern_vit_6b_1k_224_cls_patch_sgd_lr0.1] (main.py 506): INFO Train: [0/10][90/1251]	lr 0.014388	time 2.3926 (2.4639)	model_time 2.3922 (2.4287)	loss 6.9077 (6.9078)	grad_norm 0.0407 (0.0400/0.0008)	mem 48463MB
```

--------------------------------

### Execute MathVista Evaluation (torchrun)

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat/eval/mathvista/README.md

Run the evaluation script using torchrun on an 8-GPU setup. Ensure the OPENAI_API_KEY is exported. The --dynamic flag is recommended for InternVL versions 1.5 and later.

```shell
export OPENAI_API_KEY="your_openai_api_key"
# Test the testmini set
torchrun --nproc_per_node=8 eval/mathvista/evaluate_mathvista.py --checkpoint ${CHECKPOINT} --dynamic --datasets MathVista_testmini
# Test the test set
torchrun --nproc_per_node=8 eval/mathvista/evaluate_mathvista.py --checkpoint ${CHECKPOINT} --dynamic --datasets MathVista_test
```

--------------------------------

### Launch InternVL Model Worker

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat_llava/README.md

Start a model worker for InternVL. Specify the controller address, worker port, and the local path to the downloaded model weights. This example shows workers for both 7B and 13B parameter versions.

```shell
# OpenGVLab/InternVL-Chat-ViT-6B-Vicuna-7B
python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path ./pretrained/InternVL-Chat-ViT-6B-Vicuna-7B
```

```shell
# OpenGVLab/InternVL-Chat-ViT-6B-Vicuna-13B
python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40001 --worker http://localhost:40001 --model-path ./pretrained/InternVL-Chat-ViT-6B-Vicuna-13B
```

--------------------------------

### Install lmdeploy

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat/eval/mmmu_pro/README.md

Install lmdeploy with version 0.5.3 or higher. No additional dependencies are required.

```shell
pip install lmdeploy>=0.5.3 --no-deps
```

--------------------------------

### Prepare GQA Dataset

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat/eval/vqa/README.md

Steps to set up the GQA dataset directory, including downloading the evaluation script, images, and converted .jsonl files. Ensure you are in the data/gqa directory.

```shell
mkdir -p data/gqa && cd data/gqa
```

```shell
wget https://nlp.stanford.edu/data/gqa/eval.zip
unzip eval.zip
```

```shell
wget https://downloads.cs.stanford.edu/nlp/data/gqa/images.zip
unzip images.zip
```

```shell
wget https://ofasys-wlcb.oss-cn-wulanchabu.aliyuncs.com/Qwen-VL/evaluation/gqa/testdev_balanced.jsonl
```

```shell
wget https://ofasys-wlcb.oss-cn-wulanchabu.aliyuncs.com/Qwen-VL/evaluation/gqa/train_balanced.jsonl
```

```shell
wget https://github.com/OpenGVLab/InternVL/releases/download/data/llava_gqa_testdev_balanced_qwen_format.jsonl
```

--------------------------------

### Create models.txt for Evaluation

Source: https://github.com/opengvlab/internvl/blob/main/clip_benchmark/README.md

Prepare a text file listing models for evaluation. Each line should contain the model name and its source, separated by a comma.

```text
ViT-B-32,openai
```

--------------------------------

### Download and Prepare SEED-Image Dataset

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat/eval/seed/README.md

Use these shell commands to create the necessary data directory and download the SEED-Bench-image dataset and its associated JSONL file. Ensure you are in the correct directory before executing.

```shell
mkdir -p data/SEED && cd data/SEED

wget https://huggingface.co/OpenGVLab/InternVL/resolve/main/SEED-Bench-image.zip
unzip SEED-Bench-image.zip
wget https://huggingface.co/OpenGVLab/InternVL/resolve/main/seed.jsonl

cd ../..
```

--------------------------------

### Japanese-CLIP Evaluation Result Example

Source: https://github.com/opengvlab/internvl/blob/main/clip_benchmark/README.md

Example JSON output for a Japanese-CLIP zero-shot classification evaluation on ImageNet1k.

```json
{"dataset": "imagenet1k", "model": "ViT-B-32-quickgelu", "pretrained": "rinna/japanese-cloob-vit-b-16", "task": "zeroshot_classification", "metrics": {"acc1": 0.54636, "acc5": 0.72856, "mean_per_class_recall": 0.54522}, "language": "jp"}
```

--------------------------------

### Install MMCV-Full for Segmentation

Source: https://github.com/opengvlab/internvl/blob/main/INSTALLATION.md

Install mmcv-full version 1.6.2, which is an optional dependency for segmentation tasks. Requires openmim.

```bash
pip install -U openmim
mim install mmcv-full==1.6.2
```

--------------------------------

### Download LLaVA-Bench Dataset

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat/eval/llava_bench/README.md

Use this command to clone the LLaVA-Bench dataset into the specified directory. Ensure you are in the correct project directory before execution.

```shell
cd data/
git clone https://huggingface.co/datasets/liuhaotian/llava-bench-in-the-wild
cd ../
```

--------------------------------

### Download ImageNet-Sketch Dataset

Source: https://github.com/opengvlab/internvl/blob/main/classification/README.md

Use gdown to download the ImageNet-Sketch dataset. Ensure gdown is installed via `pip install gdown`.

```shell
# GDown is needed to download the dataset.
# Please install it via `pip install gdown`
gdown --id 1Mj0i5HBthqH1p_yeXzsg22gZduvgoNeA
```

--------------------------------

### Install LMDeploy

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat/README.md

Installs the LMDeploy toolkit, which is used for compressing, deploying, and serving LLMs & VLMs. Ensure you have pip version 0.6.4 or higher.

```sh
pip install lmdeploy>=0.6.4 --no-deps
```

--------------------------------

### Short-answer format example

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat_llava/docs/Evaluation.md

Example format for short-answer questions, used in benchmarks like VQAv2 and MME. No special postprocessing is typically needed.

```text
<question>
Answer the question using a single word or phrase.
```

--------------------------------

### Download and Prepare MVBench Dataset

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat/eval/mvbench/README.md

Use these shell commands to download the MVBench dataset from Hugging Face and unzip the video files. Ensure you have the huggingface-cli installed. The process involves downloading to the 'data/' directory and then extracting the video archives.

```shell
cd data/
huggingface-cli download --repo-type dataset --resume-download OpenGVLab/MVBench --local-dir MVBench --local-dir-use-symlinks False

cd MVBench/video/
for file in *.zip; do unzip "$file" -d "${file%.*}"; done
cd ../../..
```

--------------------------------

### Prepare ChartQA Dataset

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat/eval/vqa/README.md

Commands to create the data directory, download images, and fetch converted .jsonl files for the ChartQA dataset. Navigate to the data/chartqa directory before executing.

```shell
mkdir -p data/chartqa && cd data/chartqa
```

```shell
wget https://ofasys-wlcb.oss-cn-wulanchabu.aliyuncs.com/Qwen-VL/evaluation/chartqa/train_human.jsonl
```

```shell
wget https://ofasys-wlcb.oss-cn-wulanchabu.aliyuncs.com/Qwen-VL/evaluation/chartqa/train_augmented.jsonl
```

```shell
wget https://ofasys-wlcb.oss-cn-wulanchabu.aliyuncs.com/Qwen-VL/evaluation/chartqa/test_human.jsonl
```

```shell
wget https://ofasys-wlcb.oss-cn-wulanchabu.aliyuncs.com/Qwen-VL/evaluation/chartqa/test_augmented.jsonl
```

--------------------------------

### Install LLaVA Core Packages

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat_llava/README.md

Create a Conda environment with Python 3.10, activate it, upgrade pip, and install the LLaVA package in editable mode.

```shell
conda create -n llava python=3.10 -y
conda activate llava
pip install --upgrade pip  # enable PEP 660 support
pip install -e .
```

--------------------------------

### Option-only multiple-choice format example

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat_llava/docs/Evaluation.md

Example format for multiple-choice questions, used in benchmarks like MMBench and SEED-Bench. The answer should be the option's letter.

```text
<question>
A. <option_1>
B. <option_2>
C. <option_3>
D. <option_4>
Answer with the option's letter from the given choices directly.
```

--------------------------------

### Start LMDeploy API Server

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat/README.md

Launch the LMDeploy API server to expose models as services. This command starts the server for the specified model and port.

```shell
lmdeploy serve api_server OpenGVLab/InternVL2_5-8B --server-port 23333
```

--------------------------------

### Finetune InternViT-6B-224px with Vicuna-7B

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat_llava/README.md

Initiates the fine-tuning process for InternViT-6B-224px combined with Vicuna-7B. Requires 8 GPUs.

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 sh scripts_internvl/finetune_internvit6b_224to336_vicuna7b.sh
```

--------------------------------

### Clone LLaVA Repository

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat_llava/README.md

Clone the LLaVA repository and navigate into the directory. This is the first step for setting up the project.

```bash
git clone https://github.com/haotian-liu/LLaVA.git
cd LLaVA
```

--------------------------------

### Prepare MathVista Data

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat/eval/mathvista/README.md

Use these bash commands to create the necessary directory and download the annotation file for the MathVista testmini dataset.

```bash
mkdir -p data/MathVista && cd data/MathVista
wget https://huggingface.co/datasets/AI4Math/MathVista/raw/main/annot_testmini.json
cd ../..
```

--------------------------------

### Finetune InternViT-6B-448px with Vicuna-7B

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat_llava/README.md

Initiates the fine-tuning process for InternViT-6B-448px combined with Vicuna-7B. Requires 8 GPUs.

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 sh scripts_internvl/finetune_internvit6b_448_vicuna7b.sh
```

--------------------------------

### Finetune InternViT-6B-224px with Vicuna-13B

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat_llava/README.md

Initiates the fine-tuning process for InternViT-6B-224px combined with Vicuna-13B. Requires 8 GPUs.

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 sh scripts_internvl/finetune_internvit6b_224to336_vicuna13b.sh
```

--------------------------------

### Finetune InternViT-6B-448px with Vicuna-13B

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat_llava/README.md

Initiates the fine-tuning process for InternViT-6B-448px combined with Vicuna-13B. Requires 8 GPUs.

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 sh scripts_internvl/finetune_internvit6b_448_vicuna13b.sh
```

--------------------------------

### Get MMMU-Pro Evaluation Score

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat/eval/mmmu_pro/README.md

After completing the evaluation tests, run this command to retrieve the final scores.

```shell
python eval/mmmu_pro/evaluate.py
```

--------------------------------

### Download InternVL-14B-224px Model Weights

Source: https://github.com/opengvlab/internvl/blob/main/internvl_g/README.md

This command downloads the InternVL-14B-224px model weights using huggingface-cli. Ensure you have `huggingface_hub` installed.

```bash
cd pretrained/
# pip install -U huggingface_hub
huggingface-cli download --resume-download --local-dir-use-symlinks False OpenGVLab/InternVL-14B-224px --local-dir InternVL-14B-224px
```

--------------------------------

### Download and Prepare MME Benchmark Data

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat/eval/mme/README.md

Use these shell commands to create the necessary data directory and download the MME benchmark release. Ensure you are in the correct directory before execution.

```shell
mkdir -p data/mme && cd data/mme
wget https://huggingface.co/OpenGVLab/InternVL/resolve/main/MME_Benchmark_release_version.zip
unzip MME_Benchmark_release_version.zip
cd ../..
```

--------------------------------

### Launch Gradio Web Server for Demo

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat_llava/README.md

Launch the Gradio web server for the demo. Ensure the controller is running and accessible. The `--model-list-mode reload` option allows for dynamic model reloading.

```shell
python -m llava.serve.controller --host 0.0.0.0 --port 10000
```

```shell
python -m llava.serve.gradio_web_server --controller http://localhost:10000 --model-list-mode reload --port 10038
```

--------------------------------

### Adapt InternVL2/2.5 in PaddleMIX

Source: https://github.com/opengvlab/internvl/blob/main/README_zh.md

Example configuration or script snippet demonstrating how to adapt InternVL2/2.5 models within the PaddleMIX framework.

```python
from paddlemix.examples.internvl2 import InternVL2Config

config = InternVL2Config()
config.load_pretrained_model("path/to/internvl2_model")

```

--------------------------------

### Projector Weights for Visual Instruction Tuning (v1.1)

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat_llava/docs/MODEL_ZOO.md

Configuration options for using legacy projector weights with Vicuna-7B-v1.1 and CLIP-L. Ensure the base LLM and vision encoder match the pretraining configuration.

```shell
--mm_use_im_start_end True
--mm_use_im_patch_token False
```

--------------------------------

### Download Model Weights

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat_llava/README.md

Downloads specified model weights using huggingface-cli and places them in the 'pretrained/' directory. Ensure 'huggingface_hub' is installed.

```bash
cd pretrained/
# pip install -U huggingface_hub
huggingface-cli download --resume-download --local-dir-use-symlinks False OpenGVLab/InternViT-6B-224px --local-dir InternViT-6B-224px
huggingface-cli download --resume-download --local-dir-use-symlinks False OpenGVLab/InternViT-6B-448px-V1-0 --local-dir InternViT-6B-448px
huggingface-cli download --resume-download --local-dir-use-symlinks False lmsys/vicuna-13b-v1.5 --local-dir vicuna-13b-v1.5
huggingface-cli download --resume-download --local-dir-use-symlinks False lmsys/vicuna-7b-v1.5 --local-dir vicuna-7b-v1.5
```

--------------------------------

### Evaluate VTAB Eurosat Dataset

Source: https://github.com/opengvlab/internvl/blob/main/clip_benchmark/README.md

Perform zero-shot classification on the Eurosat dataset from the VTAB benchmark suite. Ensure the 'task_adaptation' package is installed.

```bash
clip_benchmark eval --dataset=vtab/eurosat --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64
```

--------------------------------

### Benchmark Multiple Models and Datasets with Files

Source: https://github.com/opengvlab/internvl/blob/main/clip_benchmark/README.md

Benchmark multiple models and datasets by specifying file paths containing lists of models and datasets. Each line in the model file should be 'model,pretrained'.

```bash
clip_benchmark eval --pretrained_model  benchmark/models.txt \\
--dataset benchmark/datasets.txt --dataset_root "clip_benchmark_datasets/{dataset}"  \\\n --output "{dataset}_{pretrained}_{model}_{language}_{task}.json"
```

--------------------------------

### Evaluate COCO Captions Zero-Shot Retrieval

Source: https://github.com/opengvlab/internvl/blob/main/clip_benchmark/README.md

Use this command to perform zero-shot retrieval evaluation on the COCO captions dataset. Ensure pycocotools is installed.

```bash
clip_benchmark eval --dataset=mscoco_captions --task=zeroshot_retrieval --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64
```

--------------------------------

### Download Dataset List

Source: https://github.com/opengvlab/internvl/blob/main/clip_benchmark/README.md

Use wget to download a list of datasets for benchmarking.

```bash
wget https://raw.githubusercontent.com/LAION-AI/CLIP_benchmark/main/benchmark/webdatasets.txt
```

--------------------------------

### Import necessary libraries for data analysis and visualization

Source: https://github.com/opengvlab/internvl/blob/main/clip_benchmark/probe_benchmark/scaling_plot.ipynb

Imports common data science and plotting libraries. Ensure these libraries are installed before running the code.

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import os
from collections import defaultdict
%matplotlib inline
from operator import truediv
from tokenize import group
import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.ticker as mticker

from matplotlib.gridspec import GridSpec
import matplotlib.pyplot as plt

from matplotlib.lines import Line2D
import plotly.express as px
from matplotlib import cm
from matplotlib.colors import ListedColormap, LinearSegmentedColormap
#new_porange = ListedColormap(newcolors)
import statsmodels.api as sm
def lstsq(x, y):
    A = np.vstack([x, np.ones(len(x))]).T
    coefs, *rest = np.linalg.lstsq(A, y, rcond=None)
    return coefs
plt.rcParams.update({'font.size': 15})
```

--------------------------------

### Projector Weights for Visual Instruction Tuning (v0, no im token)

Source: https://github.com/opengvlab/internvl/blob/main/internvl_chat_llava/docs/MODEL_ZOO.md

Configuration options for using legacy projector weights with Vicuna-13B-v0 and CLIP-L, specifically when the 'im_token' is not used. Ensure the base LLM and vision encoder match the pretraining configuration.

```shell
--mm_use_im_start_end False
--mm_use_im_patch_token False
```

--------------------------------

### Load InternVL Model with bfloat16

Source: https://context7.com/opengvlab/internvl/llms.txt

Loads the InternVL model with bfloat16 precision. Ensure `torch` and `transformers` are installed. The model is evaluated and moved to CUDA.

```python
import torch
from transformers import AutoModel, AutoTokenizer

# Basic model loading with bfloat16 precision
model = AutoModel.from_pretrained(
    "OpenGVLab/InternVL2_5-8B",
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    use_flash_attn=True,
    trust_remote_code=True
).eval().cuda()

tokenizer = AutoTokenizer.from_pretrained(
    "OpenGVLab/InternVL2_5-8B",
    trust_remote_code=True,
    use_fast=False
)
```