### Install MiniGPT-4 and Setup Environment (Bash)

Source: https://github.com/vision-cair/minigpt-4/blob/main/README.md

Clones the MiniGPT-4 repository from GitHub, changes the current directory to the repository folder, creates a Conda environment named 'minigptv' using the provided environment.yml file, and then activates the newly created environment.

```bash
git clone https://github.com/Vision-CAIR/MiniGPT-4.git
cd MiniGPT-4
conda env create -f environment.yml
conda activate minigptv
```

--------------------------------

### Setup Dataset Directories and Move Annotations (Bash)

Source: https://github.com/vision-cair/minigpt-4/blob/main/dataset/README_1_STAGE.md

Defines an environment variable for the dataset storage path, creates dedicated subdirectories for the CC/SBU and LAION datasets, and moves the previously downloaded filtered JSON annotation files into their respective locations.

```bash
export MINIGPT4_DATASET=/YOUR/PATH/FOR/LARGE/DATASET/
mkdir ${MINIGPT4_DATASET}/cc_sbu
mkdir ${MINIGPT4_DATASET}/laion
mv ccs_synthetic_filtered_large.json ${MINIGPT4_DATASET}/cc_sbu
mv laion_synthetic_filtered_large.json ${MINIGPT4_DATASET}/laion
```

--------------------------------

### Launch MiniGPT-v2 Demo (Bash)

Source: https://github.com/vision-cair/minigpt-4/blob/main/README.md

Executes the Python script 'demo_v2.py' to launch the MiniGPT-v2 demo. It specifies the configuration file 'eval_configs/minigptv2_eval.yaml' and assigns the demo to GPU ID 0.

```bash
python demo_v2.py --cfg-path eval_configs/minigptv2_eval.yaml  --gpu-id 0
```

--------------------------------

### Launch MiniGPT-4 Demo (Vicuna) (Bash)

Source: https://github.com/vision-cair/minigpt-4/blob/main/README.md

Executes the Python script 'demo.py' to launch the MiniGPT-4 demo using the Vicuna model. It specifies the configuration file 'eval_configs/minigpt4_eval.yaml' and assigns the demo to GPU ID 0.

```bash
python demo.py --cfg-path eval_configs/minigpt4_eval.yaml  --gpu-id 0
```

--------------------------------

### BibTeX Citation for MiniGPT-4 and MiniGPT-v2

Source: https://github.com/vision-cair/minigpt-4/blob/main/README.md

Provides the BibTeX entries for citing the MiniGPT-v2 and MiniGPT-4 research papers. Use these entries when referencing the work in publications.

```BibTeX
@article{chen2023minigptv2,
      title={MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning}, 
      author={Chen, Jun and Zhu, Deyao and Shen, Xiaoqian and Li, Xiang and Liu, Zechu and Zhang, Pengchuan and Krishnamoorthi, Raghuraman and Chandra, Vikas and Xiong, Yunyang and Elhoseiny, Mohamed},
      year={2023},
      journal={arXiv preprint arXiv:2310.09478},
}

@article{zhu2023minigpt,
  title={MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models},
  author={Zhu, Deyao and Chen, Jun and Shen, Xiaoqian and Li, Xiang and Elhoseiny, Mohamed},
  journal={arXiv preprint arXiv:2304.10592},
  year={2023}
}
```

--------------------------------

### Launch MiniGPT-4 Demo (Llama2) (Bash)

Source: https://github.com/vision-cair/minigpt-4/blob/main/README.md

Executes the Python script 'demo.py' to launch the MiniGPT-4 demo using the Llama2 model. It specifies the configuration file 'eval_configs/minigpt4_llama2_eval.yaml' and assigns the demo to GPU ID 0.

```bash
python demo.py --cfg-path eval_configs/minigpt4_llama2_eval.yaml  --gpu-id 0
```

--------------------------------

### Run MiniGPT-v2 Finetuning Training (Bash)

Source: https://github.com/vision-cair/minigpt-4/blob/main/MiniGPTv2_Train.md

This command initiates the finetuning process for MiniGPT-v2 using torchrun. It specifies the number of GPUs to use and points to the configuration file containing model paths and training settings. Replace NUM_GPU with the desired number of GPUs.

```bash
torchrun --nproc-per-node NUM_GPU train.py --cfg-path train_configs/minigptv2_finetune.yaml
```

--------------------------------

### Launching MiniGPT-4 Stage 2 Finetuning (Bash)

Source: https://github.com/vision-cair/minigpt-4/blob/main/MiniGPT4_Train.md

Command to launch the second finetuning stage of MiniGPT-4. This stage further aligns the model using a small, high-quality image-text pair dataset converted to a conversation format. It requires specifying the number of GPUs (`NUM_GPU`) and uses the configuration defined in `train_configs/minigpt4_stage2_finetune.yaml`, which must include the path to the stage 1 checkpoint.

```Bash
torchrun --nproc-per-node NUM_GPU train.py --cfg-path train_configs/minigpt4_stage2_finetune.yaml
```

--------------------------------

### RefCOCO Dataset Directory Structure

Source: https://github.com/vision-cair/minigpt-4/blob/main/dataset/README_MINIGPTv2_FINETUNE.md

Illustrates the expected directory structure for the RefCOCO, RefCOCO+, and RefCOCOg annotation files after download. Shows the location of instance and reference files within the ${MINIGPTv2_DATASET} path.

```text
${MINIGPTv2_DATASET}
├── refcoco_annotations
│   ├── refcoco
│   │   ├── instances.json
│   │   ├── refs(google).p
│   │   └── refs(unc).p
│   ├── refcoco+
│   │   ├── instances.json
│   │   └── refs(unc).p
│   └── refcocog
│       ├── instances.json
│       ├── refs(google).p
│       └─── refs(und).p
...
```

--------------------------------

### Copy Conversion and Download Scripts (Bash)

Source: https://github.com/vision-cair/minigpt-4/blob/main/dataset/README_1_STAGE.md

Copies the necessary Python scripts for annotation conversion and Bash scripts for dataset downloading into the newly created CC/SBU and LAION dataset directories. These scripts are essential for the subsequent data preparation steps.

```bash
cp convert_cc_sbu.py ${MINIGPT4_DATASET}/cc_sbu
cp download_cc_sbu.sh ${MINIGPT4_DATASET}/cc_sbu
cp convert_laion.py ${MINIGPT4_DATASET}/laion
cp download_laion.sh ${MINIGPT4_DATASET}/laion
```

--------------------------------

### Running VQA Evaluation with torchrun

Source: https://github.com/vision-cair/minigpt-4/blob/main/eval_scripts/EVAL_README.md

Executes the `eval_vqa.py` script using `torchrun` for distributed execution. It specifies the master port, number of processes, configuration file path, and the datasets (okvqa, vizwiz, iconvqa, gqa, vsr, hm) for visual question answering evaluation.

```shell
torchrun --master-port ${port} --nproc_per_node 1 eval_vqa.py \
 --cfg-path ${cfg_path} --dataset okvqa,vizwiz,iconvqa,gqa,vsr,hm
```

--------------------------------

### Running RefCOCO Evaluation with torchrun

Source: https://github.com/vision-cair/minigpt-4/blob/main/eval_scripts/EVAL_README.md

Executes the `eval_ref.py` script using `torchrun` for distributed execution. It specifies the master port, number of processes, configuration file path, and the datasets (refcoco, refcoco+, refcocog) to evaluate, enabling resampling.

```shell
torchrun --master-port ${port} --nproc_per_node 1 eval_ref.py \
 --cfg-path ${cfg_path} --dataset refcoco,refcoco+,refcocog --resample
```

--------------------------------

### GQA Dataset Directory Structure

Source: https://github.com/vision-cair/minigpt-4/blob/main/dataset/README_MINIGPTv2_FINETUNE.md

Illustrates the expected directory structure for the GQA dataset, including images and the main questions JSON file, within the ${MINIGPTv2_DATASET} path.

```text
Location_you_like
├── ${MINIGPTv2_DATASET}
│   ├── gqa
│       ├── images
│       ├── train_balanced_questions.json
```

--------------------------------

### Launching MiniGPT-4 Stage 1 Pretraining (Bash)

Source: https://github.com/vision-cair/minigpt-4/blob/main/MiniGPT4_Train.md

Command to launch the first pretraining stage of MiniGPT-4. This stage aligns the vision and language models using large image-text datasets like Laion and CC. It requires specifying the number of GPUs (`NUM_GPU`) and uses the configuration defined in `train_configs/minigpt4_stage1_pretrain.yaml`.

```Bash
torchrun --nproc-per-node NUM_GPU train.py --cfg-path train_configs/minigpt4_stage1_pretrain.yaml
```

--------------------------------

### Download Datasets with img2dataset (Bash)

Source: https://github.com/vision-cair/minigpt-4/blob/main/dataset/README_1_STAGE.md

Changes directory into the CC/SBU and LAION folders and executes the Bash download scripts. These scripts utilize the img2dataset tool to download the actual images based on the converted annotation files, storing them in a sharded format.

```bash
cd ${MINIGPT4_DATASET}/cc_sbu
sh download_cc_sbu.sh
cd ${MINIGPT4_DATASET}/laion
sh download_laion.sh
```

--------------------------------

### Multi-task Conversation Dataset Directory Structure

Source: https://github.com/vision-cair/minigpt-4/blob/main/dataset/README_MINIGPTv2_FINETUNE.md

Illustrates the expected directory structure for the multi-task conversation dataset annotation file within the ${MINIGPTv2_DATASET} path.

```text
Location_you_like
${MINIGPTv2_DATASET}
├── multitask_conversation
│   └── multitask_conversation.json
...
```

--------------------------------

### OKVQA Dataset Directory Structure

Source: https://github.com/vision-cair/minigpt-4/blob/main/dataset/README_MINIGPTv2_FINETUNE.md

Shows the expected directory structure for the OKVQA dataset annotations within the ${MINIGPTv2_DATASET} path. Indicates the location of the main annotation file.

```text
Location_you_like
├── ${MINIGPTv2_DATASET}
│   ├── okvqa
│       ├── okvqa_train.json
```

--------------------------------

### Convert Annotation Files (Bash/Python)

Source: https://github.com/vision-cair/minigpt-4/blob/main/dataset/README_1_STAGE.md

Navigates into the CC/SBU and LAION directories and executes the respective Python conversion scripts. These scripts process the downloaded JSON annotation files to convert them into a format compatible with the img2dataset tool, typically TSV.

```bash
cd ${MINIGPT4_DATASET}/cc_sbu
python convert_cc_sbu.py

cd ${MINIGPT4_DATASET}/laion
python convert_laion.py
```

--------------------------------

### AOK-VQA Dataset Directory Structure

Source: https://github.com/vision-cair/minigpt-4/blob/main/dataset/README_MINIGPTv2_FINETUNE.md

Illustrates the expected directory structure for the AOK-VQA dataset annotations after download and extraction. Shows the location of the main annotation file within the ${MINIGPTv2_DATASET} path.

```text
Location_you_like
├── ${MINIGPTv2_DATASET}
│   ├── aokvqa
│       ├── aokvqa_v1p0_train.json
```

--------------------------------

### OCR-VQA Dataset Directory Structure

Source: https://github.com/vision-cair/minigpt-4/blob/main/dataset/README_MINIGPTv2_FINETUNE.md

Shows the expected directory structure for the OCR-VQA dataset, including images and the main dataset JSON file, within the ${MINIGPTv2_DATASET} path.

```text
Location_you_like
├── ${MINIGPTv2_DATASET}
│   ├── ocrvqa
│       ├── images
│       ├── dataset.json
```

--------------------------------

### Filtered Flickr-30k Dataset Directory Structure

Source: https://github.com/vision-cair/minigpt-4/blob/main/dataset/README_MINIGPTv2_FINETUNE.md

Shows the expected directory structure for the filtered Flickr-30k dataset, including images and various annotation JSON files, within the ${MINIGPTv2_DATASET} path.

```text
${MINIGPTv2_DATASET}
├── filtered_flickr
│   ├── images
│   ├── captiontobbox.json
│   ├── groundedcaption.json
│   └── phrasetobbox.json
...
```

--------------------------------

### Download and Extract AOK-VQA Dataset

Source: https://github.com/vision-cair/minigpt-4/blob/main/dataset/README_MINIGPTv2_FINETUNE.md

Provides shell commands to download and extract the AOK-VQA dataset archive to a specified directory. Sets an environment variable for the dataset path.

```bash
export AOKVQA_DIR=YOUR_DATASET_PATH
mkdir -p ${AOKVQA_DIR}
curl -fsSL https://prior-datasets.s3.us-east-2.amazonaws.com/aokvqa/aokvqa_v1p0.tar.gz | tar xvz -C ${AOKVQA_DIR}
```

--------------------------------

### TextCaps Directory Structure

Source: https://github.com/vision-cair/minigpt-4/blob/main/dataset/README_MINIGPTv2_FINETUNE.md

Details the expected directory structure for TextCaps images and annotation files relative to the base dataset path (${MINIGPTv2_DATASET}). This structure is used to configure the dataset location for TextCaps tasks, typically by setting 'image_path' to the train_images folder and 'ann_path' to the annotation file.

```YAML
├── ${MINIGPTv2_DATASET}
│   ├── textcaps
│       ├── train_images
│       ├── TextCaps_0.1_train.json
```

--------------------------------

### Visual Genome Directory Structure

Source: https://github.com/vision-cair/minigpt-4/blob/main/dataset/README_MINIGPTv2_FINETUNE.md

Describes the required directory structure for Visual Genome images and annotation files relative to the base dataset path (${MINIGPTv2_DATASET}). This structure is used to configure the dataset location for Visual Genome tasks, typically by setting 'image_path' and 'ann_path' to the visual_genome folder.

```YAML
${MINIGPTv2_DATASET}
├── visual_genome
│   ├── VG_100K
│   ├── VG_100K_2
│   └── region_descriptions.json
│   └── image_data.json
...
```

--------------------------------

### COCO Captions Annotation Directory Structure

Source: https://github.com/vision-cair/minigpt-4/blob/main/dataset/README_MINIGPTv2_FINETUNE.md

Specifies the required directory structure for COCO Karpathy captions annotations relative to the base dataset path (${MINIGPTv2_DATASET}). This path is used to configure the annotation file location for COCO captioning tasks, typically set as 'ann_path' in the dataset configuration file.

```YAML
${MINIGPTv2_DATASET}
├── coco_captions
│   └── annotations
│       ├── coco_karpathy_train.json
...
```

--------------------------------

### COCO VQA Directory Structure

Source: https://github.com/vision-cair/minigpt-4/blob/main/dataset/README_MINIGPTv2_FINETUNE.md

Outlines the expected directory structure for COCO VQA v2 train and validation annotation files relative to the base dataset path (${MINIGPTv2_DATASET}). These paths are used to configure the annotation file locations for COCO VQA tasks, typically set as 'ann_path' in the dataset configuration file.

```YAML
├── ${MINIGPTv2_DATASET}
│   ├── vqav2
│       ├── vqa_train.json
|       ├── vqa_val.json
```

--------------------------------

### COCO Captions Image Directory Structure

Source: https://github.com/vision-cair/minigpt-4/blob/main/dataset/README_MINIGPTv2_FINETUNE.md

Defines the expected directory structure for COCO 2014 images relative to the base dataset path (${MINIGPTv2_DATASET}). This path is used to configure the image location for COCO captioning tasks, typically set as 'image_path' in the dataset configuration file.

```YAML
${MINIGPTv2_DATASET}
├── coco
│   ├── images
...
```

--------------------------------

### Setting PYTHONPATH for MiniGPT-4

Source: https://github.com/vision-cair/minigpt-4/blob/main/eval_scripts/EVAL_README.md

Sets the PYTHONPATH environment variable to include the directory containing the MiniGPT-4 project, allowing Python scripts to import modules from that location. This is a common step for making local packages discoverable.

```shell
export PYTHONPATH=$PYTHONPATH:/path/to/directory/of/MiniGPT-4
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.