### Install Additional Training Packages

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Install extra packages required for training LLaVA, including flash-attn for optimized attention mechanisms.

```shell
pip install -e ".[train]"
pip install flash-attn --no-build-isolation
```

--------------------------------

### Launch Gradio Web Server

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/LoRA.md

Starts the Gradio web interface, connecting it to the previously launched controller.

```Shell
python -m llava.serve.gradio_web_server --controller http://localhost:10000 --model-list-mode reload
```

--------------------------------

### Start Full Fine-tuning Script

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Execute the main fine-tuning script using DeepSpeed ZeRO-3 for large model training.

```shell
bash scripts/v1_5/finetune.sh
```

--------------------------------

### Start LoRA Fine-tuning Script

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Use this script for fine-tuning with LoRA to reduce GPU memory requirements.

```shell
bash scripts/v1_5/finetune_lora.sh
```

--------------------------------

### Launch LLaVA Controller

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/LoRA.md

Starts the central controller service for the LLaVA web interface.

```Shell
python -m llava.serve.controller --host 0.0.0.0 --port 10000
```

--------------------------------

### LLaVA from LLaMA-2

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Follow this guide to set up and train LLaVA using the LLaMA-2 base model. It details the process for integrating LLaMA-2 with LLaVA's architecture.

```bash
https://github.com/haotian-liu/LLaVA/blob/main/docs/LLaVA_from_LLaMA2.md
```

--------------------------------

### LoRA Training Documentation

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

This documentation provides details on implementing and utilizing LoRA (Low-Rank Adaptation) for efficient finetuning of LLaVA models. It covers setup and usage.

```bash
./docs/LoRA.md
```

--------------------------------

### Clone LLaVA Repository

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/macOS.md

Clone the LLaVA repository to your local machine and navigate into the LLaVA directory. This is the first step in the installation process.

```bash
git clone https://github.com/haotian-liu/LLaVA.git
cd LLaVA
```

--------------------------------

### Install LLaVA Dependencies

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/Windows.md

Commands to create a conda environment and install required Python packages for LLaVA.

```Shell
conda create -n llava python=3.10 -y
conda activate llava
python -mpip install --upgrade pip  # enable PEP 660 support
pip install torch==2.0.1+cu117 torchvision==0.15.2+cu117 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu117
pip install -e .
pip uninstall bitsandbytes
```

--------------------------------

### Intel Platform Support

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Instructions and details for enabling LLaVA support on Intel dGPU and CPU platforms. This guide helps users leverage Intel hardware for LLaVA.

```bash
https://github.com/haotian-liu/LLaVA/tree/intel/docs/intel
```

--------------------------------

### Install LLaVA Python Packages

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/macOS.md

Set up a conda environment for LLaVA with Python 3.10, activate it, upgrade pip, and then install LLaVA and PyTorch. Note that bitsandbytes must be uninstalled.

```shell
conda create -n llava python=3.10 -y
conda activate llava
python -mpip install --upgrade pip  # enable PEP 660 support
pip install -e .
pip install torch==2.1.0 torchvision==0.16.0
pip uninstall bitsandbytes
```

--------------------------------

### Install LLaVA Environment

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Create a conda environment for LLaVA with Python 3.10, activate it, and install the package. Upgrade pip to enable PEP 660 support.

```shell
conda create -n llava python=3.10 -y
conda activate llava
pip install --upgrade pip  # enable PEP 660 support
pip install -e .
```

--------------------------------

### Launch Model Worker

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Starts a worker process for model inference. Multiple workers can be launched by varying the port and worker URL.

```Shell
python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path liuhaotian/llava-v1.5-13b
```

```Shell
python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port <different from 40000, say 40001> --worker http://localhost:<change accordingly, i.e. 40001> --model-path <ckpt2>
```

--------------------------------

### Evaluate LLaVA Model with Image and Prompt

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Use the `eval_model` function to get model output for a given image and prompt. This allows direct use on platforms like Colab after downloading the repository.

```python
model_path = "liuhaotian/llava-v1.5-7b"
prompt = "What are the things I should be cautious about when I visit here?"
image_file = "https://llava-vl.github.io/static/images/view.jpg"

args = type('Args', (), {
    "model_path": model_path,
    "model_base": None,
    "model_name": get_model_name_from_path(model_path),
    "query": prompt,
    "conv_mode": None,
    "image_file": image_file,
    "sep": ",",
})()

eval_model(args)
```

--------------------------------

### Configure Legacy Projector Options

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/MODEL_ZOO.md

Required command-line arguments for instruction tuning with specific legacy projector weights.

```shell
--mm_use_im_start_end True
--mm_use_im_patch_token False
```

```shell
--mm_use_im_start_end False
--mm_use_im_patch_token False
```

--------------------------------

### Prepare Data Directory Structure

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Organize downloaded datasets into the specified directory structure for training.

```shell
├── coco
│   └── train2017
├── gqa
│   └── images
├── ocr_vqa
│   └── images
├── textvqa
│   └── train_images
└── vg
    ├── VG_100K
    └── VG_100K_2
```

--------------------------------

### Prepare Multi-Image Prompt

Source: https://context7.com/opengvlab/all-seeing/llms.txt

Construct a prompt for models that handle multiple images by concatenating image tokens and a descriptive query.

```python
multi_image_prompt = f"{DEFAULT_IMAGE_TOKEN}\n{DEFAULT_IMAGE_TOKEN}\nCompare these two images."
multi_input_ids = tokenizer_image_token(
    prompt=multi_image_prompt,
    tokenizer=tokenizer,
    image_token_index=IMAGE_TOKEN_INDEX,
    return_tensors='pt'
)
```

--------------------------------

### Initialize and Run Interactive Session

Source: https://context7.com/opengvlab/all-seeing/llms.txt

Sets up arguments, loads an image, and initiates an interactive session with the model. Ensure the model path and image file are correctly specified.

```python
args = argparse.Namespace(
    model_path="OpenGVLab/ASMv2",
    model_base=None,
    image_file="example.jpg",
    device="cuda",
    conv_mode="vicuna_v1",
    temperature=0.2,
    max_new_tokens=512,
    load_8bit=False,
    load_4bit=False,
    debug=False
)

# Load image from URL or local path
image = load_image("https://example.com/image.jpg")
# Or: image = load_image("local_image.jpg")

# Run interactive session
main(args)
# USER: What do you see?
# ASSISTANT: I can see a red bus driving on a street...
# USER: What is the bus doing?
# ASSISTANT: The bus appears to be...
```

--------------------------------

### Run Q-Bench Evaluation

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/Evaluation.md

Executes the Q-Bench evaluation script on a single GPU, accepting a subset argument.

```Shell
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/qbench.sh dev
```

--------------------------------

### Run Chinese-Q-Bench Evaluation

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/Evaluation.md

Executes the Chinese-Q-Bench evaluation script on a single GPU, accepting a subset argument.

```Shell
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/qbench_zh.sh dev
```

--------------------------------

### Load Pretrained ASMv2 Models

Source: https://context7.com/opengvlab/all-seeing/llms.txt

Initializes the model, tokenizer, and image processor. Supports full precision, 4-bit/8-bit quantization, and LoRA adapter loading.

```python
from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path

# Load the ASMv2 model from HuggingFace
model_path = "OpenGVLab/ASMv2"
model_name = get_model_name_from_path(model_path)

# Full precision loading
tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path=model_path,
    model_base=None,
    model_name=model_name,
    device_map="auto",
    device="cuda"
)

# Load with 4-bit quantization for reduced memory
tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path=model_path,
    model_base=None,
    model_name=model_name,
    load_4bit=True,
    device_map="auto"
)

# Load with 8-bit quantization
tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path=model_path,
    model_base=None,
    model_name=model_name,
    load_8bit=True,
    device_map="auto"
)

# Load LoRA fine-tuned model
tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path="path/to/lora/weights",
    model_base="OpenGVLab/ASMv2",  # Base model for LoRA
    model_name="asmv2-lora"
)

print(f"Model loaded with context length: {context_len}")
print(f"Tokenizer vocabulary size: {len(tokenizer)}")
```

--------------------------------

### Run MMBench-CN Evaluation

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/Evaluation.md

Executes the MMBench-CN evaluation script on a single GPU.

```Shell
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mmbench_cn.sh
```

--------------------------------

### Execute training with SLURM

Source: https://context7.com/opengvlab/all-seeing/llms.txt

Command to launch the training process on a SLURM-managed cluster using 8 GPUs.

```bash
srun --gres=gpu:8 --ntasks=32 --ntasks-per-node=8 \
python -u llava/train/train_mem.py \
    --deepspeed ./scripts/zero2.json \
    --model_name_or_path checkpoints/asmv2-13b-stage2-pretrain \
    --version v1 \
    --data_path playground/as_mix_4m.json \
    --image_folder playground/data \
    --vision_tower openai/clip-vit-large-patch14-336 \
    --mm_projector_type mlp2x_gelu \
    --mm_vision_select_layer -2 \
    --image_aspect_ratio pad \
    --bf16 True \
    --output_dir ./checkpoints/asmv2-13b-stage2-ft \
    --num_train_epochs 3 \
    --per_device_train_batch_size 16 \
    --learning_rate 2e-5 \
    --warmup_ratio 0.03 \
    --lr_scheduler_type cosine \
    --model_max_length 2048 \
    --gradient_checkpointing True
```

--------------------------------

### Run LLaVA-Bench-in-the-Wild Evaluation

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/Evaluation.md

Executes the LLaVA-Bench-in-the-Wild evaluation script on a single GPU.

```Shell
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/llavabench.sh
```

--------------------------------

### Configure instruction tuning flags

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/MODEL_ZOO.md

Required command-line arguments when using pretrained projector weights for visual instruction tuning.

```Shell
--mm_use_im_start_end False
--mm_use_im_patch_token False
```

--------------------------------

### Sample JSON for LLaVA Finetuning

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/Finetune_Custom_Data.md

This JSON structure represents a single sample for finetuning LLaVA. Ensure each sample includes a unique ID, image path, and conversation data.

```json
[
  {
    "id": "997bb945-628d-4724-b370-b84de974a19f",
    "image": "part-000001/997bb945-628d-4724-b370-b84de974a19f.jpg",
    "conversations": [
      {
        "from": "human",
        "value": "<image>\nWrite a prompt for Stable Diffusion to generate this image."
      },
      {
        "from": "gpt",
        "value": "a beautiful painting of chernobyl by nekro, pascal blanche, john harris, greg rutkowski, sin jong hun, moebius, simon stalenhag. in style of cg art. ray tracing. cel shading. hyper detailed. realistic. ue 5. maya. octane render. "
      },
    ]
  },
  ...
]
```

--------------------------------

### Train LLaVA-1.5 with LoRA

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Use this script to finetune LLaVA-1.5 on your own dataset with LoRA, achieving comparable performance to full-model finetuning with reduced GPU RAM.

```bash
python -m torch.distributed.launch --nproc_per_node=8 --master_port=20001 
    llava/train/train_lora.py 
    --model_name_or_path /path/to/llama-7b-hf 
    --data_path /path/to/your/data.json 
    --bf16 True 
    --output_dir /path/to/save/lora_weights 
    --num_train_epochs 3 
    --per_device_train_batch_size 4 
    --per_device_eval_batch_size 4 
    --gradient_accumulation_steps 8 
    --evaluation_strategy "no" 
    --save_strategy "steps" 
    --save_steps 1200 
    --save_total_limit 1 
    --learning_rate 2e-5 
    --weight_decay 0. 
    --warmup_ratio 0.03 
    --lr_scheduler_type "cosine" 
    --logging_steps 1 
    --tf32 True 
    --model_max_length 2048 
    --gradient_checkpointing True 
    --lazy_preprocess True 
    --report_to wandb 
    --lora_r 8 
    --lora_alpha 16 
    --lora_dropout 0.05 
    --lora_target_modules "q_proj,v_proj,k_proj,o_proj,gate_proj,up_proj,down_proj"
```

--------------------------------

### LLaVA Full-Model Finetuning Script

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/Finetune_Custom_Data.md

Employ this script for full-model finetuning of LLaVA when your task-specific dataset is sufficiently large. Hyperparameters may require adjustment.

```bash
bash scripts/v1_5/finetune_task.sh
```

--------------------------------

### Programmatic CLI-style Interaction

Source: https://context7.com/opengvlab/all-seeing/llms.txt

Import and utilize functions from `llava.serve.cli` to programmatically control LLaVA's command-line interface behavior, such as loading images and initiating the main interaction loop.

```python
from llava.serve.cli import load_image, main
import argparse
```

--------------------------------

### Run stage 2 fine-tuning

Source: https://context7.com/opengvlab/all-seeing/llms.txt

Executes the stage 2 fine-tuning script for ASMv2.

```bash
# Stage 2 fine-tuning script
sh scripts_asmv2/stage2-finetune.sh
```

--------------------------------

### Stage2 Finetune Script

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README.md

Execute this shell script to perform the stage 2 finetuning process. Ensure all datasets are downloaded and organized correctly beforehand.

```shell
sh scripts_asmv2/stage2-finetune.sh
```

--------------------------------

### LLaVA LoRA Finetuning Script

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/Finetune_Custom_Data.md

Use this script for finetuning LLaVA with LoRA when you have a limited amount of task-specific data. Adjust hyperparameters as needed.

```bash
bash scripts/v1_5/finetune_task_lora.sh
```

--------------------------------

### Train LLaVA-1.5

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

This script is used for training LLaVA-1.5. It utilizes distributed training and supports various configurations for model parameters and data loading.

```bash
python -m torch.distributed.launch --nproc_per_node=8 --master_port=20001 
    llava/train/train.py 
    --model_name_or_path /path/to/llama-7b-hf 
    --data_path /path/to/your/data.json 
    --bf16 True 
    --output_dir /path/to/save/model_weights 
    --num_train_epochs 1 
    --per_device_train_batch_size 4 
    --per_device_eval_batch_size 4 
    --gradient_accumulation_steps 16 
    --evaluation_strategy "no" 
    --save_strategy "steps" 
    --save_steps 1200 
    --save_total_limit 1 
    --learning_rate 2e-5 
    --weight_decay 0. 
    --warmup_ratio 0.03 
    --lr_scheduler_type "cosine" 
    --logging_steps 1 
    --tf32 True 
    --model_max_length 2048 
    --gradient_checkpointing True 
    --lazy_preprocess True 
    --report_to wandb
```

--------------------------------

### LLaVA Training Data and Scripts

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Access the training data and scripts for LLaVA-1.5. This provides the necessary resources to replicate or build upon the LLaVA-1.5 model.

```bash
https://github.com/haotian-liu/LLaVA#train
```

--------------------------------

### Run LLaVA CLI Chat with Images

Source: https://context7.com/opengvlab/all-seeing/llms.txt

Interact with LLaVA models via the command-line interface for multi-turn conversations about images, supporting streaming output and various customization options. Use flags like `--load-4bit` for memory optimization or `--temperature` and `--max-new-tokens` to control generation.

```bash
# Basic CLI usage
python -m llava.serve.cli \
    --model-path OpenGVLab/ASMv2 \
    --image-file example.jpg

# With 4-bit quantization for lower memory
python -m llava.serve.cli \
    --model-path OpenGVLab/ASMv2 \
    --image-file example.jpg \
    --load-4bit

# With custom generation parameters
python -m llava.serve.cli \
    --model-path OpenGVLab/ASMv2 \
    --image-file example.jpg \
    --temperature 0.2 \
    --max-new-tokens 1024 \
    --conv-mode vicuna_v1
```

--------------------------------

### Execute ASMv2 Training Scripts

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README.md

Shell commands to initiate the various stages of the ASMv2 training process.

```shell
# stage1 pretrain
sh scripts_asmv2/stage1-pretrain.sh
```

```shell
# stage2 finetune
sh scripts_asmv2/stage1-finetune.sh
```

```shell
# stage1 pretrain
sh scripts_asmv2/stage2-pretrain.sh
```

--------------------------------

### LLaVA Benchmarking

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Utilize LLaVA Bench for benchmarking open-ended visual chat capabilities. It includes results from major models like Bard and Bing-Chat.

```bash
https://github.com/haotian-liu/LLaVA/blob/main/docs/LLaVA_Bench.md
```

--------------------------------

### LLaVA Evaluation Scripts

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Find the evaluation scripts for LLaVA models. These are essential for benchmarking performance and comparing different versions or approaches.

```bash
https://github.com/haotian-liu/LLaVA/blob/main/docs/Evaluation.md
```

--------------------------------

### Configure DeepSpeed ZeRO-2

Source: https://context7.com/opengvlab/all-seeing/llms.txt

JSON configuration for DeepSpeed ZeRO-2 optimization, enabling efficient distributed training.

```json
deepspeed_config = {
    "fp16": {"enabled": "auto"},
    "bf16": {"enabled": "auto"},
    "zero_optimization": {
        "stage": 2,
        "offload_optimizer": {"device": "none"},
        "contiguous_gradients": True
    },
    "gradient_accumulation_steps": "auto",
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto"
}
```

--------------------------------

### LLaVA Model Zoo

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Explore the LLaVA Model Zoo for pre-trained weights and checkpoints. This includes various versions and configurations of LLaVA models.

```bash
https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md
```

--------------------------------

### Launch Model Worker with Quantization

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Reduces GPU memory footprint by using 4-bit or 8-bit quantization.

```Shell
python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path liuhaotian/llava-v1.5-13b --load-4bit
```

--------------------------------

### Run SEED-Bench Evaluation

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/Evaluation.md

Executes the SEED-Bench evaluation script using multiple GPUs.

```Shell
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/v1_5/eval/seed.sh
```

--------------------------------

### MMBench Single-GPU Inference

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/Evaluation.md

Run MMBench evaluation using a single GPU. Download mmbench_dev_20230712.tsv and place it under ./playground/data/eval/mmbench.

```shell
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mmbench.sh
```

--------------------------------

### Run LLaVA CLI Inference

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Executes the LLaVA CLI for image-based chat. Supports quantized inference modes like 4-bit to reduce VRAM usage.

```Shell
python -m llava.serve.cli \
    --model-path liuhaotian/llava-v1.5-7b \
    --image-file "https://llava-vl.github.io/static/images/view.jpg" \
    --load-4bit
```

--------------------------------

### Run MM-Vet Evaluation

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/Evaluation.md

Executes the MM-Vet evaluation script on a single GPU.

```Shell
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mmvet.sh
```

--------------------------------

### Run Single Image Inference Pipeline

Source: https://context7.com/opengvlab/all-seeing/llms.txt

Execute the complete inference pipeline for visual question answering, from loading and processing an image to generating a text response using a pre-trained LLaVA model. Ensure necessary imports and model configuration are set up.

```python
import torch
from PIL import Image
from llava.model.builder import load_pretrained_model
from llava.mm_utils import (
    process_images,
    tokenizer_image_token,
    get_model_name_from_path,
    KeywordsStoppingCriteria
)
from llava.conversation import conv_templates, SeparatorStyle
from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN
from llava.utils import disable_torch_init

# Disable default weight initialization for faster loading
disable_torch_init()

# Load model
model_path = "OpenGVLab/ASMv2"
model_name = get_model_name_from_path(model_path)
tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path, None, model_name
)

# Load and process image
image = Image.open("example.jpg").convert("RGB")
image_tensor = process_images([image], image_processor, model.config)
image_tensor = image_tensor.to(model.device, dtype=torch.float16)

# Prepare conversation
query = "Describe the main objects in this image and their relationships."
conv = conv_templates["vicuna_v1"].copy()
conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\n" + query)
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()

# Tokenize
input_ids = tokenizer_image_token(
    prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt'
).unsqueeze(0).cuda()

# Setup stopping criteria
stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2
stopping_criteria = KeywordsStoppingCriteria([stop_str], tokenizer, input_ids)

# Generate response
with torch.inference_mode():
    output_ids = model.generate(
        input_ids,
        images=image_tensor,
        do_sample=False,
        temperature=0,
        top_p=None,
        num_beams=1,
        max_new_tokens=512,
        use_cache=True,
        stopping_criteria=[stopping_criteria]
    )

# Decode output
output = tokenizer.batch_decode(
    output_ids[:, input_ids.shape[1]:],
    skip_special_tokens=True
)[0].strip()

if output.endswith(stop_str):
    output = output[:-len(stop_str)].strip()

print(f"Response: {output}")
```

--------------------------------

### VisWiz Single-GPU Inference

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/Evaluation.md

Perform VisWiz evaluation using a single GPU. Download test.json and test.zip, extracting them to ./playground/data/eval/vizwiz.

```shell
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/vizwiz.sh
```

--------------------------------

### Evaluate PSG benchmark

Source: https://context7.com/opengvlab/all-seeing/llms.txt

Runs the PSG evaluation script for the ASMv2 model.

```bash
# Run PSG evaluation
sh scripts_asmv2/eval/psg.sh OpenGVLab/ASMv2
```

--------------------------------

### Convert ScienceQA to LLaVA format

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/ScienceQA.md

Use this script to transform the ScienceQA dataset into the LLaVA conversation-style format.

```Shell
python scripts/convert_sqa_to_llava.py \
    convert_to_llava \
    --base-dir /path/to/ScienceQA/data/scienceqa \
    --prompt-format "QCM-LEA" \
    --split {train,val,minival,test,minitest}
```

--------------------------------

### Option-Only Multiple-Choice Format

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/Evaluation.md

This format is for multiple-choice questions, common in benchmarks like MMBench and SEED-Bench. The model should respond with the letter of the correct option.

```text
<question>
A. <option_1>
B. <option_2>
C. <option_3>
D. <option_4>
Answer with the option's letter from the given choices directly.
```

--------------------------------

### Summarize GPT Review Results

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Run the script to summarize the evaluation results generated by the GPT-assisted review process.

```shell
python summarize_gpt_review.py
```

--------------------------------

### Command-Line VQA Evaluation

Source: https://context7.com/opengvlab/all-seeing/llms.txt

Executes batch visual question answering evaluation directly from the command line. Requires specifying model path, question file, image folder, and output file for answers.

```bash
# Command-line evaluation
python -m llava.eval.model_vqa_loader \
    --model-path OpenGVLab/ASMv2 \
    --question-file playground/data/eval/questions.jsonl \
    --image-folder playground/data \
    --answers-file playground/data/eval/answers.jsonl \
    --conv-mode vicuna_v1 \
    --temperature 0 \
    --max_new_tokens 128
```

--------------------------------

### Fetch and Initialize Evaluation Data

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/llava/eval/webpage/index.html

Loads evaluation data from a local JSON file and triggers the population of models and questions in the UI. Requires the existence of populateModels, populateQuestions, and displayQuestion functions.

```javascript
// Fetch the JSON file
fetch('data.json')
.then(response => response.json())
.then(json_data => {
  // Populate the models and questions.
  populateModels(json_data.models);
  populateQuestions(json_data.questions);
  displayQuestion(currentQuestionIndex);
}).catch(error => console.error(error));
```

--------------------------------

### Grounding Evaluation Script

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README.md

Run this script to evaluate the model's performance on referring expression comprehension tasks. Specify the model identifier as an argument.

```shell
sh scripts_asmv2/eval/grounding.sh OpenGVLab/ASMv2
```

--------------------------------

### TextVQA Single-GPU Inference and Evaluation

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/Evaluation.md

Run TextVQA evaluation using a single GPU. Download TextVQA_0.5.1_val.json and images, extracting them to ./playground/data/eval/textvqa.

```shell
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/textvqa.sh
```

--------------------------------

### Load Pretrained LLaVA Model

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Load a pretrained LLaVA model using `load_pretrained_model`. Ensure the model path and name are correctly specified. This function is detailed in `llava/model/builder.py`.

```python
from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path
from llava.eval.run_llava import eval_model

model_path = "liuhaotian/llava-v1.5-7b"

tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path=model_path,
    model_base=None,
    model_name=get_model_name_from_path(model_path)
)
```

--------------------------------

### Evaluate Generated Responses with GPT-4

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Compare generated responses against a reference answer file using GPT-4 for review. Requires setting the OPENAI_API_KEY environment variable.

```shell
OPENAI_API_KEY="sk-***********************************" python llava/eval/eval_gpt_review_visual.py \
    --question playground/data/coco2014_val_qa_eval/qa90_questions.jsonl \
    --context llava/eval/table/caps_boxes_coco2014_val_80.jsonl \
    --answer-list \
    /path/to/answer-file-ref.jsonl \
    /path/to/answer-file-our.jsonl \
    --rule llava/eval/table/rule.json \
    --output /path/to/review.json
```

--------------------------------

### Launch Model Worker with LoRA Weights

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README_llava.md

Loads unmerged LoRA weights by specifying the base model path.

```Shell
python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path liuhaotian/llava-v1-0719-336px-lora-vicuna-13b-v1.3 --model-base lmsys/vicuna-13b-v1.3
```

--------------------------------

### Apply Delta Weights to LLaMA

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/MODEL_ZOO.md

Use this script to merge delta weights with base LLaMA weights to generate LLaVA checkpoints.

```bash
python3 -m llava.model.apply_delta \
    --base /path/to/llama-7b \
    --target /output/path/to/LLaVA-7B-v0 \
    --delta liuhaotian/LLaVA-7b-delta-v0
```

--------------------------------

### GQA Multi-GPU Inference

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/Evaluation.md

Run GQA evaluation with multiple GPUs. Download the GQA data and evaluation scripts, placing them under ./playground/data/eval/gqa/data. Modifications to eval.py might be necessary.

```shell
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 bash scripts/v1_5/eval/gqa.sh
```

--------------------------------

### MME Single-GPU Inference and Evaluation

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/Evaluation.md

Perform MME evaluation on a single GPU. Download the MME benchmark data and the official eval_tool, placing them under ./playground/data/eval/MME.

```shell
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mme.sh
```

--------------------------------

### Parse and evaluate scene graphs

Source: https://context7.com/opengvlab/all-seeing/llms.txt

Parses scene graph captions and runs the full PSG evaluation pipeline.

```python
from llava.eval.eval_psg import eval_psg, parse_scene_graph
import json

# Load synonyms for text normalization
synonyms = {}
with open("llava/eval/synonyms.txt", "r") as f:
    for line in f:
        phrases = line.strip().split(",")
        for p in phrases:
            synonyms[p.strip()] = phrases[0].strip()

# Parse scene graph from model output
scene_graph_caption = """
A <ref>person</ref><box>[[100, 200, 300, 500]]</box> is
<pred>riding</pred><box>[[100, 200, 300, 500]]</box><box>[[350, 300, 600, 550]]</box>
a <ref>bicycle</ref><box>[[350, 300, 600, 550]]</box>.
"""

parsed_sg, correct_format = parse_scene_graph(
    scene_graph_caption=scene_graph_caption,
    width=800,
    height=600,
    square_pad_postprocess=True,
    bbox_scale=999,
    synonyms=synonyms
)

print("Parsed Scene Graph Triplets:")
for triplet in parsed_sg:
    subj, subj_box, obj, obj_box, pred = triplet
    print(f"  ({subj}, {pred}, {obj})")
    print(f"    Subject box: {subj_box.tolist()}")
    print(f"    Object box: {obj_box.tolist()}")

# Run full evaluation
questions = {q['question_id']: q for q in [json.loads(l) for l in open("psg_eval.jsonl")]}
answers = [json.loads(l) for l in open("psg_answers.jsonl")]

eval_psg(
    questions=questions,
    answers=answers,
    synonyms=synonyms,
    square_pad_postprocess=True,
    bbox_scale=999
)
# Output:
# Recall(riding): 45.23
# Recall(beside): 38.91
# Recall: 14.20
# Mean Recall for 56 predicates: 10.30
```

--------------------------------

### ScienceQA Single-GPU Inference and Evaluation

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/Evaluation.md

Execute ScienceQA evaluation on a single GPU. Ensure images, pid_splits.json, and problems.json are placed under ./playground/data/eval/scienceqa.

```shell
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/sqa.sh
```

--------------------------------

### Merge LoRA Weights

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/docs/LoRA.md

Combines LoRA weights with the base model to create a standalone checkpoint.

```Shell
python scripts/merge_lora_weights.py \
    --model-path /path/to/lora_model \
    --model-base /path/to/base_model \
    --save-model-path /path/to/merge_model
```

--------------------------------

### Open-ended Scene Graph Generation Evaluation Script

Source: https://github.com/opengvlab/all-seeing/blob/main/all-seeing-v2/README.md

Execute this script for evaluating open-ended scene graph generation capabilities. The model identifier is required.

```shell
sh scripts_asmv2/eval/psg.sh OpenGVLab/ASMv2
```