### Install ShareGPT4V Project

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/README.md

Follow these bash commands to clone the repository, set up a conda environment, and install the project dependencies.

```bash
git clone https://github.com/InternLM/InternLM-XComposer --depth=1
cd projects/ShareGPT4V
conda create -n share4v python=3.10 -y
conda activate share4v

pip install --upgrade pip
pip install -e .
pip install -e ".[train]"
pip install flash-attn --no-build-isolation
```

--------------------------------

### Evaluate Model Output in Python

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/README.md

This snippet demonstrates how to use the `eval_model` function to get model output for a given prompt and image. It requires setting up arguments similar to command-line arguments.

```Python
model_path = "Lin-Chen/ShareGPT4V-7B"
prompt = "What is the most common catchphrase of the character on the right?"
image_file = "examples/breaking_bad.png"

args = type('Args', (), {
    "model_path": model_path,
    "model_base": None,
    "model_name": get_model_name_from_path(model_path),
    "query": prompt,
    "conv_mode": None,
    "image_file": image_file,
    "sep": ",",
    "temperature": 0,
    "top_p": None,
    "num_beams": 1,
    "max_new_tokens": 512
})()

eval_model(args)
```

--------------------------------

### Run Local Demo

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/README.md

Execute this command to build and run the local demo for ShareGPT4V.

```bash
# run script
python tools/app.py
```

--------------------------------

### Initializing Gradio Demo

Source: https://context7.com/sharegpt4omni/sharegpt4v/llms.txt

Import necessary modules and utilities to set up a Gradio web interface for ShareGPT4V.

```python
import gradio as gr
import torch
from share4v.model.builder import load_pretrained_model
from share4v.conversation import conv_templates, default_conversation
from share4v.mm_utils import process_images, tokenizer_image_token, KeywordsStoppingCriteria
from share4v.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN
from transformers import TextIteratorStreamer
from threading import Thread
```

--------------------------------

### Run Q-Bench Evaluation

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Execute single-GPU inference for the Q-Bench dataset.

```Shell
# for single node inference
CUDA_VISIBLE_DEVICES=0 bash scripts/sharegpt4v/eval/qbench.sh
```

```Shell
# for slurm inference
srun -p Your partion --gres gpu:1 bash scripts/sharegpt4v/eval/qbench.sh
```

--------------------------------

### Evaluate on standard benchmarks

Source: https://context7.com/sharegpt4omni/sharegpt4v/llms.txt

Execute evaluation scripts for GQA, MME, MMBench, and SEED-Bench using shell commands.

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/sharegpt4v/eval/gqa.sh
```

```bash
CUDA_VISIBLE_DEVICES=0 bash scripts/sharegpt4v/eval/mme.sh
```

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/sharegpt4v/eval/mmbench_en.sh
```

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/sharegpt4v/eval/seed.sh
```

--------------------------------

### Run LLaVA-Bench-in-the-Wild Evaluation

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Execute single-GPU inference for the LLaVA-Bench-in-the-Wild dataset.

```Shell
# for single node inference
CUDA_VISIBLE_DEVICES=0 bash scripts/sharegpt4v/eval/llavabench.sh
```

```Shell
# for slurm inference
srun -p Your partion --gres gpu:1 bash scripts/sharegpt4v/eval/llavabench.sh
```

--------------------------------

### Run MMBench-EN Evaluation

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Execute inference for the English MMBench dataset on single nodes or via Slurm.

```Shell
# for single node inference
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/sharegpt4v/eval/mmbench_en.sh
```

```Shell
# for slurm inference
srun -p Your partion --gres gpu:8 bash scripts/sharegpt4v/eval/mmbench_en.sh
```

--------------------------------

### Load and Run ShareGPT4V Model with Gradio Interface

Source: https://context7.com/sharegpt4omni/sharegpt4v/llms.txt

Loads the ShareGPT4V model and sets up a Gradio interface for interactive use. This includes model loading, response generation with image and text inputs, and launching the web demo.

```python
model_path = "Lin-Chen/ShareGPT4V-7B"
tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path, None, "share4v-7b", False, False
)

def generate_response(image, text, temperature=0.2, max_tokens=512):
    # Prepare conversation
    conv = conv_templates["vicuna_v1"].copy()
    text_with_image = text + '\n<image>' if '<image>' not in text else text
    conv.append_message(conv.roles[0], text_with_image)
    conv.append_message(conv.roles[1], None)
    prompt = conv.get_prompt()

    # Process image
    image_tensor = process_images([image], image_processor, model.config)
    image_tensor = image_tensor.to(model.device, dtype=torch.float16)

    # Tokenize
    input_ids = tokenizer_image_token(
        prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt'
    ).unsqueeze(0).to(model.device)

    # Setup streaming
    streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
    stop_str = conv.sep2
    stopping_criteria = KeywordsStoppingCriteria([stop_str], tokenizer, input_ids)

    # Generate
    thread = Thread(target=model.generate, kwargs=dict(
        inputs=input_ids,
        images=image_tensor,
        do_sample=temperature > 0,
        temperature=temperature,
        max_new_tokens=max_tokens,
        streamer=streamer,
        stopping_criteria=[stopping_criteria],
        use_cache=True
    ))
    thread.start()

    output = ""
    for new_text in streamer:
        output += new_text
        yield output

# Create Gradio interface
demo = gr.Interface(
    fn=generate_response,
    inputs=[
        gr.Image(type="pil", label="Upload Image"),
        gr.Textbox(label="Question"),
        gr.Slider(0, 1, value=0.2, label="Temperature"),
        gr.Slider(64, 1024, value=512, step=64, label="Max Tokens")
    ],
    outputs=gr.Textbox(label="Response"),
    title="ShareGPT4V Demo"
)
demo.launch(server_name="0.0.0.0", port=7860, share=True)
```

--------------------------------

### Run MMBench-CN Evaluation

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Execute inference for the Chinese MMBench dataset on single nodes or via Slurm.

```Shell
# for single node inference
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/sharegpt4v/eval/mmbench_cn.sh
```

```Shell
# for slurm inference
srun -p Your partion --gres gpu:8 bash scripts/sharegpt4v/eval/mmbench_cn.sh
```

--------------------------------

### Finetune ShareGPT4V 7B Model Script

Source: https://context7.com/sharegpt4omni/sharegpt4v/llms.txt

Shell script for finetuning the ShareGPT4V 7B model after pretraining. It utilizes DeepSpeed and specifies paths to the pretrained model, data, and vision tower, along with finetuning-specific hyperparameters.

```bash
deepspeed share4v/train/train.py \
    --deepspeed scripts/zero3.json \
    --model_name_or_path path/to/sharegpt4v-7b-pretrain \
    --version v1 \
    --data_path data/sharegpt4v/sharegpt4v_mix665k_cap23k_coco-ap9k_lcs3k_sam9k_div2k.json \
    --image_folder data \
    --vision_tower path/to/sharegpt4v-7b-pretrain/vision_tower \
    --mm_projector_type mlp2x_gelu \
    --mm_vision_select_layer -2 \
    --mm_use_im_start_end False \
    --mm_use_im_patch_token False \
    --image_aspect_ratio pad \
    --group_by_modality_length True \
    --bf16 True \
    --output_dir ./checkpoints/sharegpt4v-7b \
    --num_train_epochs 1 \
    --per_device_train_batch_size 8 \
    --gradient_accumulation_steps 2 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type cosine \
    --logging_steps 1 \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --dataloader_num_workers 4 \
    --save_strategy steps \
    --save_steps 1000 \
    --save_total_limit 1
```

--------------------------------

### Load Pretrained Model in Python

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/README.md

Use this code to load a pretrained ShareGPT4V model. Ensure necessary imports are present.

```Python
from share4v.model.builder import load_pretrained_model
from share4v.mm_utils import get_model_name_from_path
from share4v.eval.run_share4v import eval_model

model_path = "Lin-Chen/ShareGPT4V-7B"

tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path=model_path,
    model_base=None,
    model_name=get_model_name_from_path(model_path)
)
```

--------------------------------

### Run MM-Vet Evaluation

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Execute single-GPU inference for the MM-Vet benchmark.

```Shell
# for single node inference
CUDA_VISIBLE_DEVICES=0 bash scripts/sharegpt4v/eval/mmvet.sh
```

```Shell
# for slurm inference
srun -p Your partion --gres gpu:1 bash scripts/sharegpt4v/eval/mmvet.sh
```

--------------------------------

### VisWiz Evaluation Script (Single GPU)

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Shell script to run VisWiz evaluation on a single node using a single GPU.

```shell
CUDA_VISIBLE_DEVICES=0 bash scripts/sharegpt4v/eval/vizwiz.sh

```

--------------------------------

### Load Pretrained ShareGPT4V Model

Source: https://context7.com/sharegpt4omni/sharegpt4v/llms.txt

Loads ShareGPT4V models with support for 8-bit/4-bit quantization, LoRA adapters, and automatic device mapping. Use load_4bit=True for reduced memory usage during inference.

```python
from share4v.model.builder import load_pretrained_model
from share4v.mm_utils import get_model_name_from_path

# Load the full ShareGPT4V-7B model
model_path = "Lin-Chen/ShareGPT4V-7B"
model_name = get_model_name_from_path(model_path)

tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path=model_path,
    model_base=None,
    model_name=model_name,
    load_8bit=False,  # Enable for 8-bit quantization
    load_4bit=False,  # Enable for 4-bit quantization
    device_map="auto",
    device="cuda"
)

# For 4-bit quantized inference (reduced memory)
tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path="Lin-Chen/ShareGPT4V-7B",
    model_base=None,
    model_name="sharegpt4v-7b",
    load_4bit=True,
    device_map="auto"
)
```

--------------------------------

### Constructing Conversation Prompts

Source: https://context7.com/sharegpt4omni/sharegpt4v/llms.txt

Use conversation templates to format multi-turn interactions and determine appropriate stop sequences.

```python
# Create a new conversation
conv = conv_templates["share4v_v1"].copy()

# Add messages
conv.append_message(conv.roles[0], "<image>\nWhat do you see in this image?")
conv.append_message(conv.roles[1], None)  # Model response placeholder

# Get formatted prompt
prompt = conv.get_prompt()
# Output: "A chat between a curious human and an artificial intelligence assistant..."

# For multi-turn conversation
conv = conv_templates["share4v_v1"].copy()
conv.append_message(conv.roles[0], "<image>\nDescribe this image.")
conv.append_message(conv.roles[1], "This image shows a beautiful sunset over mountains.")
conv.append_message(conv.roles[0], "What colors are prominent?")
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()

# Determine stop string based on separator style
if conv.sep_style == SeparatorStyle.TWO:
    stop_str = conv.sep2  # "</s>"
else:
    stop_str = conv.sep   # "###"
```

--------------------------------

### Run VQAv2 Benchmark Evaluation

Source: https://context7.com/sharegpt4omni/sharegpt4v/llms.txt

Shell command to execute the VQAv2 benchmark evaluation for ShareGPT4V. This command specifies the visible CUDA devices and points to the evaluation script.

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/sharegpt4v/eval/vqav2.sh
```

--------------------------------

### Run SEED-Bench-Image Evaluation

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Execute inference for the SEED-Bench-Image dataset on single nodes or via Slurm.

```Shell
# for single node inference
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/sharegpt4v/eval/seed.sh
```

```Shell
# for slurm inference
srun -p Your partion --gres gpu:8 bash scripts/sharegpt4v/eval/seed.sh
```

--------------------------------

### VisWiz Evaluation Script (SLURM)

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Shell script to run VisWiz evaluation on a SLURM cluster using a single GPU.

```shell
srun -p Your partion --gres gpu:1 bash scripts/sharegpt4v/eval/vizwiz.sh

```

--------------------------------

### Programmatic evaluation with model_vqa_loader

Source: https://context7.com/sharegpt4omni/sharegpt4v/llms.txt

Run model evaluation programmatically by configuring arguments via argparse.Namespace and calling eval_model.

```python
import argparse
from share4v.eval.model_vqa_loader import eval_model

args = argparse.Namespace(
    model_path="Lin-Chen/ShareGPT4V-7B",
    model_base=None,
    image_folder="./playground/data/eval/vqav2/test2015",
    question_file="./playground/data/eval/vqav2/llava_vqav2_mscoco_test-dev2015.jsonl",
    answers_file="./playground/data/eval/vqav2/answers/sharegpt4v-7b.jsonl",
    conv_mode="share4v_v1",
    num_chunks=1,
    chunk_idx=0,
    temperature=0,  # Greedy decoding
    top_p=None,
    num_beams=1
)

eval_model(args)
```

--------------------------------

### ScienceQA Evaluation Script (SLURM)

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Shell script to run ScienceQA evaluation on a SLURM cluster using multiple GPUs.

```shell
srun -p Your partion --gres gpu:8 bash scripts/sharegpt4v/eval/sqa.sh

```

--------------------------------

### Run Model Evaluation on Images

Source: https://context7.com/sharegpt4omni/sharegpt4v/llms.txt

Evaluates the ShareGPT4V model on image-based queries. Customize 'temperature' and 'top_p' for different response styles, from greedy decoding to more creative outputs.

```python
from share4v.model.builder import load_pretrained_model
from share4v.mm_utils import get_model_name_from_path
from share4v.eval.run_share4v import eval_model

model_path = "Lin-Chen/ShareGPT4V-7B"

# Define inference arguments
args = type('Args', (), {
    "model_path": model_path,
    "model_base": None,
    "model_name": get_model_name_from_path(model_path),
    "query": "What is the most common catchphrase of the character on the right?",
    "conv_mode": None,  # Auto-detected based on model
    "image_file": "examples/breaking_bad.png",
    "sep": ",",
    "temperature": 0,  # Greedy decoding
    "top_p": None,
    "num_beams": 1,
    "max_new_tokens": 512
})()

# Run inference and print output
eval_model(args)

# For more creative responses
args.temperature = 0.7
args.top_p = 0.9
eval_model(args)
```

--------------------------------

### Pretrain ShareGPT4V 7B Model Script

Source: https://context7.com/sharegpt4omni/sharegpt4v/llms.txt

Shell script to initiate the pretraining of the ShareGPT4V 7B model. It configures DeepSpeed, specifies model paths, data sources, vision tower, and various training hyperparameters.

```bash
deepspeed share4v/train/train.py \
    --deepspeed scripts/zero3.json \
    --model_name_or_path lmsys/vicuna-7b-v1.5 \
    --version v1 \
    --data_path data/sharegpt4v/share-captioner_coco_lcs_sam_1246k_1107.json \
    --image_folder data \
    --vision_tower openai/clip-vit-large-patch14-336 \
    --pretrain_mm_mlp_adapter path/to/llava-v1.5-mlp2x-336px-pretrain-vicuna-7b-v1.5/mm_projector.bin \
    --tune_entire_model True \
    --tune_vit_from_layer 12 \
    --mm_projector_type mlp2x_gelu \
    --mm_vision_select_layer -2 \
    --mm_use_im_start_end False \
    --mm_use_im_patch_token False \
    --image_aspect_ratio pad \
    --bf16 True \
    --output_dir ./checkpoints/sharegpt4v-7b-pretrain \
    --num_train_epochs 1 \
    --per_device_train_batch_size 16 \
    --gradient_accumulation_steps 1 \
    --learning_rate 2e-5 \
    --vision_tower_lr 2e-6 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type cosine \
    --logging_steps 1 \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --dataloader_num_workers 4 \
    --save_strategy steps \
    --save_steps 500 \
    --save_total_limit 1
```

--------------------------------

### VQAv2 Evaluation Script (Single Node)

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Shell script to run VQAv2 evaluation on a single node using multiple GPUs.

```shell
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/sharegpt4v/eval/vqav2.sh

```

--------------------------------

### VQAv2 Evaluation Script (SLURM)

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Shell script to run VQAv2 evaluation on a SLURM cluster using multiple GPUs.

```shell
srun -p Your partion --gres gpu:8 bash scripts/sharegpt4v/eval/vqav2.sh

```

--------------------------------

### ScienceQA Evaluation Script (Single GPU)

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Shell script to run ScienceQA evaluation on a single node using a single GPU.

```shell
CUDA_VISIBLE_DEVICES=0 bash scripts/sharegpt4v/eval/sqa.sh

```

--------------------------------

### TextVQA Evaluation Script (Single GPU)

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Shell script to run TextVQA evaluation on a single node using a single GPU.

```shell
CUDA_VISIBLE_DEVICES=0 bash scripts/sharegpt4v/eval/textvqa.sh

```

--------------------------------

### Process Images for Model Input

Source: https://context7.com/sharegpt4omni/sharegpt4v/llms.txt

Prepares images for model input, including aspect ratio handling and padding. Combines with tokenizer_image_token to create multimodal inputs. Ensure images are converted to RGB format.

```python
import torch
from PIL import Image
from share4v.mm_utils import process_images, tokenizer_image_token
from share4v.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN

# Load and preprocess image
image = Image.open("path/to/image.jpg").convert('RGB')
images = [image]

# Process images with the model's image processor
image_tensors = process_images(images, image_processor, model.config)
# Result: torch.Tensor of shape [batch, channels, height, width]

# For single image, move to GPU
image_tensor = image_tensors[0].unsqueeze(0).half().cuda()

# Tokenize prompt with image token
prompt = "<image>\nDescribe this image in detail."
input_ids = tokenizer_image_token(
    prompt,
    tokenizer,
    IMAGE_TOKEN_INDEX,
    return_tensors='pt'
).unsqueeze(0).cuda()

# Generate response
with torch.inference_mode():
    output_ids = model.generate(
        input_ids,
        images=image_tensor,
        do_sample=False,
        temperature=0,
        max_new_tokens=512,
        use_cache=True
    )

# Decode output
output = tokenizer.batch_decode(output_ids[:, input_ids.shape[1]:], skip_special_tokens=True)[0]
print(output.strip())
```

--------------------------------

### Manage Conversation Templates

Source: https://context7.com/sharegpt4omni/sharegpt4v/llms.txt

Manages multi-turn dialogues using predefined conversation templates like Vicuna v0/v1 and LLaMA-2 styles. The 'plain' template is available for pretraining.

```python
from share4v.conversation import conv_templates, SeparatorStyle

# Available conversation templates
templates = [
    "share4v_v0",     # Vicuna v0 style
    "share4v_v1",     # Vicuna v1 style (default)
    "share4v_llama_2", # LLaMA-2 style
    "plain",          # Plain text format for pretraining
]

```

--------------------------------

### GQA Evaluation Script (SLURM)

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Shell script to run GQA evaluation on a SLURM cluster using multiple GPUs.

```shell
srun -p Your partion --gres gpu:8 bash scripts/sharegpt4v/eval/gqa.sh

```

--------------------------------

### GQA Evaluation Script (Single Node)

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Shell script to run GQA evaluation on a single node using multiple GPUs.

```shell
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/sharegpt4v/eval/gqa.sh

```

--------------------------------

### TextVQA Evaluation Script (SLURM)

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Shell script to run TextVQA evaluation on a SLURM cluster using a single GPU.

```shell
srun -p Your partion --gres gpu:1 bash scripts/sharegpt4v/eval/textvqa.sh

```

--------------------------------

### Batch Inference with ShareCaptioner

Source: https://context7.com/sharegpt4omni/sharegpt4v/llms.txt

Process multiple images in batches to generate captions using the ShareCaptioner model.

```python
import json
import torch
from PIL import Image
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load ShareCaptioner model
model_name = "Lin-Chen/ShareCaptioner"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="cuda",
    trust_remote_code=True
).eval().half()
model.tokenizer = tokenizer

# Prepare batch of images
image_paths = ["image1.jpg", "image2.jpg", "image3.jpg"]
batch_size = 4

# Build prompt embeddings
seg1 = '<|User|>:'
seg2 = f'Analyze the image in a comprehensive and detailed manner.{model.eoh}\n<|Bot|>:'
seg_emb1 = model.encode_text(seg1, add_special_tokens=True)
seg_emb2 = model.encode_text(seg2, add_special_tokens=False)

captions = []
for i in range(0, len(image_paths), batch_size):
    batch_paths = image_paths[i:i+batch_size]
    batch_images = []

    for path in batch_paths:
        image = Image.open(path).convert("RGB")
        batch_images.append(model.vis_processor(image).unsqueeze(0))

    batch_tensor = torch.cat(batch_images, dim=0).cuda()
    tmp_seg_emb1 = seg_emb1.repeat(len(batch_images), 1, 1)
    tmp_seg_emb2 = seg_emb2.repeat(len(batch_images), 1, 1)

    with torch.cuda.amp.autocast():
        with torch.no_grad():
            img_embeds = model.encode_img(batch_tensor)
            input_emb = torch.cat([tmp_seg_emb1, img_embeds, tmp_seg_emb2], dim=1)

            outputs = model.internlm_model.generate(
                inputs_embeds=input_emb,
                max_length=500,
                num_beams=3,
                min_length=1,
                do_sample=True,
                repetition_penalty=1.5,
                temperature=1.0,
                eos_token_id=tokenizer.eos_token_id,
            )

    for j, out in enumerate(outputs):
        out[out == -1] = 2
        response = model.decode_text([out])
        captions.append({batch_paths[j]: response})

# Save captions
with open("captions.json", "w") as f:
    json.dump(captions, f, indent=4)
```

--------------------------------

### MME Evaluation Script (Single GPU)

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Shell script to run MME evaluation on a single node using a single GPU.

```shell
CUDA_VISIBLE_DEVICES=0 bash scripts/sharegpt4v/eval/mme.sh

```

--------------------------------

### Evaluate custom VQA datasets

Source: https://context7.com/sharegpt4omni/sharegpt4v/llms.txt

Format custom data into JSONL and evaluate using the model_vqa module.

```python
import json
from share4v.eval.model_vqa import eval_model

# Prepare custom evaluation data (questions.jsonl)
questions = [
    {
        "question_id": 1,
        "image": "image1.jpg",
        "text": "What color is the car?\nAnswer the question using a single word or phrase."
    },
    {
        "question_id": 2,
        "image": "image2.jpg",
        "text": "How many people are in this image?\nAnswer the question using a single word or phrase."
    },
    {
        "question_id": 3,
        "image": "image3.jpg",
        "text": "What is the main subject of this image?\nA. A dog\nB. A cat\nC. A bird\nD. A fish\nAnswer with the option's letter from the given choices directly."
    }
]

# Save to JSONL format
with open("custom_questions.jsonl", "w") as f:
    for q in questions:
        f.write(json.dumps(q) + "\n")

# Run evaluation
# python share4v/eval/model_vqa.py \
#     --model-path Lin-Chen/ShareGPT4V-7B \
#     --image-folder ./custom_images \
#     --question-file custom_questions.jsonl \
#     --answers-file custom_answers.jsonl \
#     --conv-mode share4v_v1 \
#     --temperature 0
```

--------------------------------

### MME Evaluation Script (SLURM)

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Shell script to run MME evaluation on a SLURM cluster using a single GPU.

```shell
srun -p Your partion --gres gpu:1 bash scripts/sharegpt4v/eval/mme.sh

```

--------------------------------

### Short-Answer Question Format

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Format for short-answer questions, expecting a single word or phrase as the answer.

```text
<question>
Answer the question using a single word or phrase.

```

--------------------------------

### Option-Only Multiple Choice Format

Source: https://github.com/sharegpt4omni/sharegpt4v/blob/master/docs/Evaluation.md

Format for multiple-choice questions, requiring the answer to be one of the provided option letters.

```text
<question>
A. <option_1>
B. <option_2>
C. <option_3>
D. <option_4>
Answer with the option's letter from the given choices directly.

```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.