### Environment Setup for X-InstructBLIP

Source: https://github.com/salesforce/lavis/blob/main/projects/xinstructblip/demo/demo.ipynb

This section provides commands to set up the environment by cloning the LAVIS-XInstructBLIP repository, installing its dependencies, and installing a specific version of KNN_CUDA.

```shell
!git clone https://github.com/artemisp/LAVIS-XInstructBLIP.git
!cd LAVIS-XInstructBLIP && python -m pip install -e .
!python -m pip install --upgrade https://github.com/unlimblue/KNN_CUDA/releases/download/0.2/KNN_CUDA-0.2-py3-none-any.whl
!wget -P /usr/bin https://github.com/unlimblue/KNN_CUDA/raw/master/ninja
```

--------------------------------

### Install InstructBLIP from Source

Source: https://github.com/salesforce/lavis/blob/main/projects/instructblip/README.md

Installs the InstructBLIP project from its GitHub repository. This involves cloning the repository, navigating into the directory, and installing the package in editable mode.

```bash
git clone https://github.com/salesforce/LAVIS.git
cd LAVIS
pip install -e .
```

--------------------------------

### Setup Device

Source: https://github.com/salesforce/lavis/blob/main/examples/blip_zero_shot_classification.ipynb

Sets up the computation device, prioritizing CUDA if available, otherwise falling back to the CPU. This ensures optimal performance based on the system's hardware.

```python
# setup device to use
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
```

--------------------------------

### Install Dependencies and Setup Environment

Source: https://github.com/salesforce/lavis/blob/main/projects/img2llm-vqa/img2llm_vqa.ipynb

Installs necessary libraries and clones the LAVIS repository. It handles different setup procedures for Google Colab and local environments, including installing specific spaCy models and dependencies.

```Python
# install requirements
import sys
if 'google.colab' in sys.modules:
    print('Running in Colab.')
    !git clone https://github.com/salesforce/LAVIS
    %cd LAVIS
    !pip install .
    !pip3 install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.0.0/en_core_web_sm-3.0.0.tar.gz
else:
    !pip install omegaconf
    %cd ../..
    !pip install .
    !pip3 install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.0.0/en_core_web_sm-3.0.0.tar.gz

%cd projects/img2prompt-vqa
```

--------------------------------

### Install LAVIS Library

Source: https://github.com/salesforce/lavis/blob/main/projects/blip-diffusion/README.md

Installs the LAVIS library from source using pip. This is a prerequisite for running the BLIP-Diffusion examples.

```bash
pip install -e .
```

--------------------------------

### Install LAVIS Repository and Dependencies

Source: https://github.com/salesforce/lavis/blob/main/projects/xinstructblip/README.md

This section details how to clone the LAVIS repository, install it using pip, and set up evaluation dependencies including Java and KNN Cuda. It also includes instructions for installing NLTK's CMUDict.

```shell
git clone https://github.com/salesforce/LAVIS.git
cd LAVIS-XInstructBLIP
pip install -e .

apt-get update
apt-get install default-jre

pip install --upgrade https://github.com/unlimblue/KNN_CUDA/releases/download/0.2/KNN_CUDA-0.2-py3-none-any.whl
wget -P /usr/bin https://github.com/unlimblue/KNN_CUDA/raw/master/ninja

import nltk
nltk.download('cmudict')
```

--------------------------------

### Install salesforce-lavis

Source: https://github.com/salesforce/lavis/blob/main/projects/blip2/README.md

Installs the salesforce-lavis library using pip. This is the first step to use BLIP-2 models.

```Shell
pip install salesforce-lavis
```

--------------------------------

### Load and Display Example Image

Source: https://github.com/salesforce/lavis/blob/main/examples/blip_image_captioning.ipynb

Loads an example image from a specified path, converts it to RGB format, and displays it after resizing. This is a preparatory step for image processing.

```Python
raw_image = Image.open("../docs/_static/merlion.png").convert("RGB")
display(raw_image.resize((596, 437)))
```

--------------------------------

### Load Example Image

Source: https://github.com/salesforce/lavis/blob/main/examples/blip2_instructed_generation.ipynb

Loads an example image from a URL using PIL and requests, then displays it after resizing.

```Python
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/LAVIS/assets/merlion.png' 
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')   
display(raw_image.resize((596, 437)))
```

--------------------------------

### Install LAVIS

Source: https://github.com/salesforce/lavis/blob/main/projects/blip-diffusion/notebooks/editing_tryon_zeroshot.ipynb

Installs the LAVIS library from its GitHub repository using pip. This is the first step to use the library's functionalities.

```Python
!pip install 'git+https://github.com/salesforce/LAVIS.git'
```

--------------------------------

### Install LAVIS from Source

Source: https://github.com/salesforce/lavis/blob/main/docs/intro.rst

Steps to clone the LAVIS repository from GitHub and install it using pip. It also includes instructions for an editable installation for development purposes.

```bash
git clone https://github.com/salesforce/LAVIS.git
cd LAVIS
pip install .
pip install -e .
```

--------------------------------

### Load and Display Example Image

Source: https://github.com/salesforce/lavis/blob/main/examples/clip_zero_shot_classification.ipynb

Loads an example image from a specified path, converts it to RGB format, and displays it with a resized dimension. This step is for visualizing the input image.

```Python
from PIL import Image

raw_image = Image.open("../docs/_static/merlion.png").convert("RGB")
display(raw_image.resize((596, 437)))
```

--------------------------------

### Load Example Image and Question

Source: https://github.com/salesforce/lavis/blob/main/projects/pnp-vqa/pnp_vqa.ipynb

Loads an example image from a URL and defines a sample question for the VQA task. The image is displayed after being resized.

```Python
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/LAVIS/projects/pnp-vqa/demo.png' 
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')   
# raw_image = Image.open("./demo.png").convert("RGB")
display(raw_image.resize((400, 300)))
question = "What is the black objects on the salad called?"
print(question)
```

--------------------------------

### Download Videos with youtube-dl

Source: https://github.com/salesforce/lavis/blob/main/projects/xinstructblip/README.md

Example command to download videos using youtube-dl, which can be used to obtain the necessary video files for the Audio-Video DisCRn dataset generation.

```Bash
youtube-dl <youtube-id>
```

--------------------------------

### Train 3D Q-Former with Vicuna7b

Source: https://github.com/salesforce/lavis/blob/main/projects/xinstructblip/README.md

Example command to train a 3D Q-Former model using Vicuna7b, specifying a particular training configuration file.

```bash
python -m torch.distributed.run --nproc_per_node=8 train.py --cfg-path lavis/projects/xinstruct_blip/train/vicuna7b/pc_training.yaml
```

--------------------------------

### Load and Display Example Image

Source: https://github.com/salesforce/lavis/blob/main/examples/albef_zero_shot_classification.ipynb

Loads an example image from a specified path, converts it to RGB format, and displays it with a resized dimension. This step is for visualizing the input image.

```python
raw_image = Image.open("../docs/_static/merlion.png").convert("RGB")
display(raw_image.resize((596, 437)))
```

--------------------------------

### Install LAVIS from PyPI

Source: https://github.com/salesforce/lavis/blob/main/README.md

Installs the LAVIS library from the Python Package Index (PyPI). This is the standard method for users who want to quickly integrate LAVIS into their projects.

```Bash
pip install salesforce-lavis
```

--------------------------------

### Install Salesforce LAVIS

Source: https://github.com/salesforce/lavis/blob/main/examples/blip2_instructed_generation.ipynb

Installs the salesforce-lavis library, with a check for Google Colab environment.

```Python
import sys
if 'google.colab' in sys.modules:
    print('Running in Colab.')
    !pip3 install salesforce-lavis
```

--------------------------------

### Load Sample Image and Device Setup

Source: https://github.com/salesforce/lavis/blob/main/projects/blip2/README.md

Loads a sample image using PIL and sets up the device (GPU or CPU) for model processing. This is a prerequisite for image-to-text generation.

```Python
import torch
from PIL import Image
# setup device to use
device = torch.device("cuda") if torch.cuda.is_available() else "cpu"
# load sample image
raw_image = Image.open("../../docs/_static/merlion.png").convert("RGB")
display(raw_image.resize((596, 437)))
```

--------------------------------

### Load Example Image and Question

Source: https://github.com/salesforce/lavis/blob/main/projects/img2llm-vqa/img2llm_vqa.ipynb

Loads a sample image from a local path and defines a question for visual question answering. It then displays the resized image.

```Python
# img_url = 'https://storage.googleapis.com/sfr-vision-language-research/LAVIS/projects/pnp-vqa/demo.png'
# raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')

raw_image = Image.open("./demo.png").convert("RGB")
question = "What item s are spinning which can be used to control electric?"
print(question)
display(raw_image.resize((400, 300)))
```

--------------------------------

### Setup Device

Source: https://github.com/salesforce/lavis/blob/main/projects/pnp-vqa/pnp_vqa.ipynb

Determines and sets the device to be used for computation, prioritizing CUDA (GPU) if available, otherwise falling back to CPU.

```Python
# setup device to use
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
device
```

--------------------------------

### LLM Inference Setup Note

Source: https://github.com/salesforce/lavis/blob/main/projects/img2llm-vqa/img2llm_vqa.ipynb

A comment indicating that for this specific notebook demonstration, only CPU will be used for LLM inference.

```Python
# In this notebook, we only use CPU for LLM inference

```

--------------------------------

### Setup random seeds and device

Source: https://github.com/salesforce/lavis/blob/main/projects/xinstructblip/demo/demo.ipynb

Configures random seeds for reproducibility across runs and determines the appropriate device (CUDA or CPU) for model execution.

```python
def setup_seeds(seed=42):
    seed = seed

    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)

device = torch.device("cuda") if torch.cuda.is_available() else "cpu"
```

--------------------------------

### Setup Device

Source: https://github.com/salesforce/lavis/blob/main/examples/albef_zero_shot_classification.ipynb

Sets up the computation device, prioritizing CUDA if available, otherwise falling back to CPU. This is crucial for performance optimization.

```python
# setup device to use
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
```

--------------------------------

### Load and Display Image

Source: https://github.com/salesforce/lavis/blob/main/examples/blip_zero_shot_classification.ipynb

Loads an example image ('merlion.png') using PIL, converts it to RGB format, and displays it with a specified size. This step prepares the image for feature extraction.

```python
raw_image = Image.open("../docs/_static/merlion.png").convert("RGB")
display(raw_image.resize((596, 437)))
```

--------------------------------

### Initialize LAVIS Model and Preprocessors

Source: https://github.com/salesforce/lavis/blob/main/examples/blip2_feature_extraction.ipynb

Sets up the device (GPU or CPU) and loads the BLIP-2 feature extractor model along with its associated visual and text processors. It then preprocesses a sample image and text caption for feature extraction.

```python
import torch
from PIL import Image

from lavis.models import load_model_and_preprocess

# setup device to use
device = torch.device("cuda") if torch.cuda.is_available() else "cpu"

# Load an example image
raw_image = Image.open("../docs/_static/merlion.png").convert("RGB")
caption = "a large fountain spewing water into the "

model, vis_processors, txt_processors = load_model_and_preprocess(name="blip2_feature_extractor", model_type="pretrain", is_eval=True, device=device)
image = vis_processors["eval"](raw_image).unsqueeze(0).to(device)
text_input = txt_processors["eval"](caption)
sample = {"image": image, "text_input": [text_input]}
```

--------------------------------

### Load Sample Image with PyTorch

Source: https://github.com/salesforce/lavis/blob/main/README.md

Loads a sample image using PIL and PyTorch, setting up the device for computation (CUDA or CPU). This is a common setup step for many inference tasks.

```Python
import torch
from PIL import Image
# setup device to use
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# load sample image
raw_image = Image.open("docs/_static/merlion.png").convert("RGB")
```

--------------------------------

### Run Training with Configuration

Source: https://github.com/salesforce/lavis/blob/main/docs/tutorial.tasks.rst

Shows the command-line interface command to start the training process using a specified configuration file (e.g., `dialogue_avsd_ft.yaml`). This command loads the configuration to assign the correct task and parameters.

```sh
python train.py --cfg-path dialogue_avsd_ft.yaml
```

--------------------------------

### Load and Display Image

Source: https://github.com/salesforce/lavis/blob/main/examples/clip_feature_extraction.ipynb

Loads an image from a specified path using Pillow, converts it to RGB format, and displays it after resizing. This is a preparatory step before feature extraction.

```Python
from PIL import Image

raw_image = Image.open("../docs/_static/merlion.png").convert("RGB")
display(raw_image.resize((596, 437)))
```

--------------------------------

### Load and Display Example Image

Source: https://github.com/salesforce/lavis/blob/main/examples/blip_vqa.ipynb

Loads an image file using PIL, converts it to RGB format, and displays it after resizing. This is a prerequisite for image analysis tasks.

```Python
raw_image = Image.open("../docs/_static/merlion.png").convert("RGB")
display(raw_image.resize((596, 437)))
```

--------------------------------

### Load and Display Image

Source: https://github.com/salesforce/lavis/blob/main/examples/blip_image_text_matching.ipynb

Loads an image file using Pillow, converts it to RGB format, and displays it after resizing. This is a preparatory step for image processing in vision-language tasks.

```Python
from PIL import Image

raw_image = Image.open("../docs/_static/merlion.png").convert("RGB")
display(raw_image.resize((596, 437)))
```

--------------------------------

### LAVIS Runner Configuration Example

Source: https://github.com/salesforce/lavis/blob/main/docs/tutorial.training-example.rst

This YAML snippet illustrates runner configurations for training in LAVIS. It includes task-specific arguments, learning rate schedulers, optimizers, distributed training settings, and logging/checkpointing configurations.

```yaml
runner:
    # runner-specific arguments
    task: "caption"
    max_len: 50
    min_len: 1
    # ... other runner arguments
```

--------------------------------

### Load Lavis Model and Preprocessors

Source: https://github.com/salesforce/lavis/blob/main/examples/blip_feature_extraction.ipynb

Loads the 'blip_feature_extractor' model and its associated visual and text processors. It also sets up the device to use CUDA if available, otherwise falling back to CPU.

```Python
import torch
from PIL import Image

from lavis.models import load_model_and_preprocess

raw_image = Image.open("../docs/_static/merlion.png").convert("RGB")
caption = "a large fountain spewing water into the air"

display(raw_image.resize((596, 437)))

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model, vis_processors, txt_processors = load_model_and_preprocess(name="blip_feature_extractor", model_type="base", is_eval=True, device=device)
image = vis_processors["eval"](raw_image).unsqueeze(0).to(device)
text_input = txt_processors["eval"](caption)
sample = {"image": image, "text_input": [text_input]}
```

--------------------------------

### Load and Display Image

Source: https://github.com/salesforce/lavis/blob/main/examples/blip2_image_text_matching.ipynb

Loads an image file using Pillow, converts it to RGB format, and displays it after resizing. This step is crucial for preparing the visual input for the model.

```Python
from PIL import Image

raw_image = Image.open("../docs/_static/merlion.png").convert("RGB")
display(raw_image.resize((596, 437)))
```

--------------------------------

### Setup Device and Load Sample Image for Inference

Source: https://github.com/salesforce/lavis/blob/main/docs/getting_started.rst

Configures the execution device (GPU or CPU) and loads a sample image ('merlion.png') for model inference. This is a prerequisite for performing tasks like image captioning.

```python
from PIL import Image

# setup device to use
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# load sample image
raw_image = Image.open("docs/_static/merlion.png").convert("RGB")
```

--------------------------------

### Load Preprocessors for different modalities

Source: https://github.com/salesforce/lavis/blob/main/projects/xinstructblip/demo/demo.ipynb

Initializes various preprocessors from the LAVIS library, including ULIPPCProcessor for point clouds, ClipImageEvalProcessor for images, BeatsAudioProcessor for audio, and AlproVideoEvalProcessor for videos.

```python
## Load Preprocessors
from lavis.processors.ulip_processors import ULIPPCProcessor
from lavis.processors.clip_processors import ClipImageEvalProcessor
from lavis.processors.audio_processors import BeatsAudioProcessor
from lavis.processors.alpro_processors import AlproVideoEvalProcessor

pc_pocessor = ULIPPCProcessor()
image_pocessor = ClipImageEvalProcessor()
audio_processor = BeatsAudioProcessor(model_name='iter3', sampling_rate=16000, n_frames=2, is_eval=False, frame_length=512)
video_processor = AlproVideoEvalProcessor(n_frms=4, image_size=224)
```

--------------------------------

### Match Embeddings and Calculate Probabilities

Source: https://github.com/salesforce/lavis/blob/main/examples/clip_feature_extraction.ipynb

Calculates the similarity between the image and text embeddings using dot product and applies a softmax function to get probabilities. This step is used to determine how well the text describes the image.

```Python
sims = (image_features @ text_features.t())[0] / 0.01
probs = torch.nn.Softmax(dim=0)(sims).tolist()

for cls_nm, prob in zip(cls_names, probs):
    print(f"{cls_nm}: 	 {prob:.3%}")
```

--------------------------------

### Run X-InstructBLIP Demo

Source: https://github.com/salesforce/lavis/blob/main/projects/xinstructblip/README.md

Executes the Gradio-based demo for X-InstructBLIP. Requires navigating to the demo directory and specifying the model type.

```bash
cd projects/xinstructblip/demo
python run_demo.py --model_type vicuna7b
```

--------------------------------

### Load and Inspect CC3M Dataset

Source: https://github.com/salesforce/lavis/blob/main/lavis/datasets/download_scripts/DownloadConceptualCaptions/create_annotation_3m.ipynb

Loads the conceptual caption 3 million (CC3M) dataset from a gzipped TSV file using pandas. It specifies the delimiter, compression, and column names. It also shows how to access the first record and get the total number of records.

```Python
cc3m = pd.read_csv("downloaded_cc3m_report.tsv.gz", compression="gzip", sep="\t", names=["caption", "path", "dataset", "mimetype", "size", "status", "url"])

cc3m.iloc[0]

len(cc3m)
```

--------------------------------

### Install Requirements and Clone Repository

Source: https://github.com/salesforce/lavis/blob/main/projects/pnp-vqa/pnp_vqa.ipynb

Installs necessary Python packages and clones the LAVIS repository if running in Google Colab. It specifically installs the LAVIS package and transformers version 4.25.

```Python
# install requirements
import sys
if 'google.colab' in sys.modules:
    print('Running in Colab.')
    !git clone https://github.com/salesforce/LAVIS
    %cd LAVIS
    !pip install .
    !pip install transformers==4.25
```

--------------------------------

### Load InstructBLIP Model and Preprocessors

Source: https://github.com/salesforce/lavis/blob/main/projects/instructblip/README.md

Loads an InstructBLIP model and its associated preprocessors for evaluation. It specifies the model name, type, and device (GPU or CPU). The preprocessor is then used to prepare a raw image for the model.

```python
import torch
from PIL import Image
from lavis.models import load_model_and_preprocess

# setup device to use
device = torch.device("cuda") if torch.cuda.is_available() else "cpu"

# load sample image
raw_image = Image.open("../../docs/_static/Confusing-Pictures.jpg").convert("RGB")

# loads InstructBLIP model
model, vis_processors, _ = load_model_and_preprocess(name="blip2_vicuna_instruct", model_type="vicuna7b", is_eval=True, device=device)

# prepare the image
image = vis_processors["eval"](raw_image).unsqueeze(0).to(device)
```

--------------------------------

### Dialogue Dataset Builder Example (AVSD)

Source: https://github.com/salesforce/lavis/blob/main/docs/tutorial.datasets.rst

Example of a Dialogue Dataset Builder subclass for AVSD datasets. It inherits from BaseDatasetBuilder and registers itself with the 'avsd_dialogue' key in the registry.

```Python
from lavis.datasets.builders.base_dataset_builder import BaseDatasetBuilder
from lavis.datasets.datasets.avsd_dialogue_datasets import (
    AVSDDialDataset, 
    AVSDDialEvalDataset 
)

from lavis.common.registry import registry


@registry.register_builder("avsd_dialogue")
class AVSDDialBuilder(BaseDatasetBuilder):
    train_dataset_cls = AVSDDialDataset 
    eval_dataset_cls = AVSDDialEvalDataset 

    DATASET_CONFIG_DICT = {
        "default": "configs/datasets/avsd/defaults_dial.yaml"
    }
```

--------------------------------

### InstructBLIP BibTeX Citation

Source: https://github.com/salesforce/lavis/blob/main/projects/instructblip/README.md

Provides the BibTeX entry for the InstructBLIP paper, 'InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning'. This is useful for citing the model in academic work.

```BibTeX
@misc{instructblip,
      title={InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning},
      author={Wenliang Dai and Junnan Li and Dongxu Li and Anthony Meng Huat Tiong and Junqi Zhao and Weisheng Wang and Boyang Li and Pascale Fung and Steven Hoi},
      year={2023},
      eprint={2305.06500},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
```

--------------------------------

### Generate Audio-Video DisCRn Data

Source: https://github.com/salesforce/lavis/blob/main/projects/xinstructblip/README.md

Python script to create audio-video cross-modal reasoning pairs for the DisCRn dataset. It requires downloaded Audiocaps captions and videos, with modes for property generation, pair identification, instruction-answer generation, and roundtrip consistency.

```Python
python projects/xinstructblip/discrn/data_generation/audiocaps_video_audio.py --shard <shard> --original_data_file <original_data_file> --split <split> --mode <mode> --rnd <rnd>
```

--------------------------------

### Train or Evaluate X-InstructBLIP Model

Source: https://github.com/salesforce/lavis/blob/main/projects/xinstructblip/README.md

Command to initiate training or evaluation of the X-InstructBLIP model using distributed processing. Requires specifying the configuration file path.

```bash
python -m torch.distributed.run --nproc_per_node=8 train.py --cfg-path path/to/train/or/eval/config
```

--------------------------------

### Load X-InstructBLIP Model

Source: https://github.com/salesforce/lavis/blob/main/projects/xinstructblip/README.md

Demonstrates how to load the X-InstructBLIP model with specified weights using the LAVIS library. This is a key step for utilizing the pretrained models.

```python
from lavis.models import load_model
model = load_model("blip2_vicuna_xinstruct", "vicuna7b")
```

--------------------------------

### Image Captioning

Source: https://github.com/salesforce/lavis/blob/main/projects/img2llm-vqa/img2llm_vqa.ipynb

Generates question-guided captions for the input image. The model produces multiple captions, and the first five are printed as examples.

```Python
samples = model.forward_cap(samples=samples, num_captions=50, num_patches=20)
print('Examples of question-guided captions: ')
samples['captions'][0][:5]
```

--------------------------------

### Import necessary libraries

Source: https://github.com/salesforce/lavis/blob/main/projects/xinstructblip/demo/demo.ipynb

Imports essential libraries for the project, including Gradio for UI, PyTorch for deep learning, argument parsing, NumPy for numerical operations, OmegaConf for configuration management, and specific modules from the LAVIS library.

```python
import gradio as gr
import torch
import argparse
import numpy as np
from omegaconf import OmegaConf
from lavis.common.registry import registry
import random

import trimesh
import pyvista as pv
```

--------------------------------

### Install LAVIS with Conda

Source: https://github.com/salesforce/lavis/blob/main/docs/intro.rst

Instructions for creating and activating a conda environment for the LAVIS project. This is an optional step to manage project dependencies.

```bash
conda create -n lavis python=3.8
conda activate lavis
```

--------------------------------

### Model Configuration YAML

Source: https://github.com/salesforce/lavis/blob/main/docs/tutorial.models.rst

Example of a YAML configuration file specifying the model architecture ('gpt_dialogue') and type ('base') for assigning the GPTDialogue model.

```yaml
model:
  arch: gpt_dialogue # name of the model 
  model_type: base

```

--------------------------------

### Download Flickr30k Dataset Script

Source: https://github.com/salesforce/lavis/blob/main/docs/benchmark.rst

Provides a Python script to download the Flickr30k dataset. This dataset is used for retrieval tasks and requires this script for setup.

```python
from lavis.datasets.download_scripts import download_flickr

# Example usage (assuming the script is runnable directly or imported)
# download_flickr()
```

--------------------------------

### Build LAVIS from Source

Source: https://github.com/salesforce/lavis/blob/main/README.md

Clones the LAVIS repository and installs it in editable mode. This method is suitable for developers who want to contribute to LAVIS or make modifications.

```Bash
git clone https://github.com/salesforce/LAVIS.git
cd LAVIS
pip install -e .
```

--------------------------------

### InstructBLIP Implementation (Python)

Source: https://github.com/salesforce/lavis/blob/main/README.md

Implementation of InstructBLIP, a vision-language instruction-tuning framework built upon BLIP-2 models. It achieves state-of-the-art zero-shot generalization performance across various vision-language tasks.

```Python
from lavis.projects.instructblip.models import *
from lavis.projects.instructblip.configs import *

# Example usage (conceptual):
# model = load_model("instructblip_vicuna7b_instruct_v1")
# output = model.generate({"image": "path/to/image.jpg", "instruction": "What is in the image?"})

```

--------------------------------

### Import Libraries

Source: https://github.com/salesforce/lavis/blob/main/examples/blip_zero_shot_classification.ipynb

Imports necessary libraries including torch for tensor operations and PIL for image manipulation. It also loads the load_model_and_preprocess function from Lavis.

```python
import torch
from PIL import Image

from lavis.models import load_model_and_preprocess
```

--------------------------------

### BLIP-2 Stage-1 Pre-training Script

Source: https://github.com/salesforce/lavis/blob/main/projects/blip2/README.md

This bash script is used for the stage-1 pre-training of the BLIP-2 model from scratch. It likely involves setting up the environment and executing the main training pipeline.

```bash
run_scripts/blip2/train/pretrain_stage1.sh
```

--------------------------------

### Extract Unimodal Features (Image and Text)

Source: https://github.com/salesforce/lavis/blob/main/examples/blip2_feature_extraction.ipynb

Extracts unimodal features separately for the image and the text from the sample. It prints the shapes of both the image embeddings and the text embeddings.

```python
# Unimodal features

features_image = model.extract_features(sample, mode="image")
features_text = model.extract_features(sample, mode="text")
print(features_image.image_embeds.shape)
# torch.Size([1, 32, 768])
print(features_text.text_embeds.shape)
# torch.Size([1, 12, 768])
```

--------------------------------

### Load Lavis Model and Preprocessors

Source: https://github.com/salesforce/lavis/blob/main/examples/blip2_image_text_matching.ipynb

Loads the BLIP-2 model and associated visual and text processors for image-text matching. It specifies the model variant ('blip2_image_text_matching'), stage ('pretrain'), the target device (GPU or CPU), and sets the evaluation mode.

```Python
import torch
from PIL import Image

from lavis.models import load_model_and_preprocess
from lavis.processors import load_processor

# setup device to use
device = torch.device("cuda") if torch.cuda.is_available() else "cpu"

model, vis_processors, text_processors = load_model_and_preprocess("blip2_image_text_matching", "pretrain", device=device, is_eval=True)
# model, vis_processors, text_processors = load_model_and_preprocess("blip2_image_text_matching", "coco", device=device, is_eval=True)
```

--------------------------------

### Extract Image and Text Features

Source: https://github.com/salesforce/lavis/blob/main/examples/blip_zero_shot_classification.ipynb

Extracts feature embeddings for both the image and the class names using the loaded model. The 'image' and 'text_input' are passed in a dictionary to the extract_features method.

```python
# Extract image embedding and class name embeddings

sample = {"image": image, "text_input": cls_names}

image_features = model.extract_features(sample, mode="image").image_embeds_proj[:, 0]
text_features = model.extract_features(sample, mode="text").text_embeds_proj[:, 0]
```

--------------------------------

### Generate Text Description for Image (InstructBLIP)

Source: https://github.com/salesforce/lavis/blob/main/projects/instructblip/README.md

Generates a text description for a given image using the InstructBLIP model. It takes the processed image and a text prompt as input and returns the model's generated response.

```python
model.generate({"image": image, "prompt": "What is unusual about this image?"})
```

```python
model.generate({"image": image, "prompt": "Write a short description for the image."})

```

```python
model.generate({"image": image, "prompt": "Write a detailed description."})

```

```python
model.generate({"image": image, "prompt":"Describe the image in details."}, use_nucleus_sampling=True, top_p=0.9, temperature=1)

```

--------------------------------

### Specify Class Names and Processor

Source: https://github.com/salesforce/lavis/blob/main/examples/blip_zero_shot_classification.ipynb

Defines a list of class names for classification and initializes a BlipCaptionProcessor with a specific prompt. This processor is then used to format the class names for the model.

```python
# Specify class names

cls_names = ["merlion", "sky", "giraffe", "fountain", "marina bay"]

# (optional) add prompt when we want to use the model for zero-shot classification
from lavis.processors.blip_processors import BlipCaptionProcessor
text_processor = BlipCaptionProcessor(prompt="A picture of ")

cls_prompt = [text_processor(cls_nm) for cls_nm in cls_names]
```

--------------------------------

### BLIP-2 Explainability for Generated Text

Source: https://github.com/salesforce/lavis/blob/main/projects/blip2/README.md

Asks the BLIP-2 model to explain its previous answer, demonstrating its ability to provide reasoning for its generations.

```Python
model.generate({
    "image": image,
    "prompt": "Question: which city is this? Answer: singapore. Question: why?"})
```

--------------------------------

### Extract Unimodal Features (Image and Text)

Source: https://github.com/salesforce/lavis/blob/main/examples/blip_feature_extraction.ipynb

Extracts unimodal features separately for the image and text from the sample. It prints the shapes of both the image embeddings and text embeddings.

```Python
features_image = model.extract_features(sample, mode="image")
features_text = model.extract_features(sample, mode="text")
print(features_image.image_embeds.shape)
# torch.Size([1, 197, 768])
print(features_text.text_embeds.shape)
# torch.Size([1, 12, 768])
```

--------------------------------

### Extract Multimodal Features

Source: https://github.com/salesforce/lavis/blob/main/examples/blip2_feature_extraction.ipynb

Extracts multimodal features from the preprocessed image and text sample using the loaded LAVIS model. It then prints the shape of the resulting multimodal embeddings.

```python
# Multimodal features

features_multimodal = model.extract_features(sample)
print(features_multimodal.multimodal_embeds.shape)
# torch.Size([1, 32, 768]), 32 is the number of queries
```

--------------------------------

### Generate 3D QA Data

Source: https://github.com/salesforce/lavis/blob/main/projects/xinstructblip/README.md

Python script to generate question-answer pairs for 3D data. It supports modes for color removal, QA generation, and roundtrip consistency checks. Requires downloaded Cap3D captions and specifies data splits.

```Python
python projects/xinstructblip/data_aug/3d_qa_data_generation.py --shard <shard> --mode <mode> --split <split> --original_data_file <original_data_file>
```

--------------------------------

### Import Libraries for Lavis

Source: https://github.com/salesforce/lavis/blob/main/examples/blip_image_captioning.ipynb

Imports necessary libraries including torch, PIL Image, and load_model_and_preprocess from lavis.models for setting up the image captioning environment.

```Python
import torch
from PIL import Image

from lavis.models import load_model_and_preprocess
```

--------------------------------

### Extract Multimodal Features

Source: https://github.com/salesforce/lavis/blob/main/examples/blip_feature_extraction.ipynb

Extracts multimodal features from the preprocessed image and text sample using the loaded Lavis model. It then prints the shape of the resulting multimodal embeddings.

```Python
features_multimodal = model.extract_features(sample)
print(features_multimodal.multimodal_embeds.shape)
# # torch.Size([1, 12, 768]), use features_multimodal[:,0,:] for multimodal classification tasks
```

--------------------------------

### Structure-Controlled Generation / Stylization

Source: https://github.com/salesforce/lavis/blob/main/projects/blip-diffusion/README.md

Enables structure-controlled generation and stylization using BLIP-Diffusion. This notebook likely explores how to guide image generation based on structural or stylistic inputs.

```python
# This is a placeholder for the actual Python code that would be in the notebook.
# The notebook linked demonstrates structure-controlled generation and stylization.
```

--------------------------------

### Generate Audio QA Data

Source: https://github.com/salesforce/lavis/blob/main/projects/xinstructblip/README.md

Python script for augmenting audio data to generate question-answer pairs. Similar to the 3D QA script, it includes modes for QA generation and roundtrip consistency. Requires Audiocaps captions.

```Python
python projects/xinstructblip/data_aug/audio_qa_data_generation.py --shard <shard> --mode <mode> --split <split> --original_data_file <original_data_file>
```

--------------------------------

### Run LAVIS Demo

Source: https://github.com/salesforce/lavis/blob/main/README.md

Provides the command to run the LAVIS demo script. This script initiates a local demo that can be accessed via a web browser, allowing users to interact with the library's functionalities.

```Bash
run_scripts/run_demo.sh
```

--------------------------------

### Get Output Image Size

Source: https://github.com/salesforce/lavis/blob/main/projects/blip-diffusion/notebooks/editing_tryon_zeroshot.ipynb

Retrieves the size (dimensions) of the first output image from the editing process. This is a common operation to verify the dimensions of the generated or modified image.

```Python
output[0].size
```

--------------------------------

### Markdown Dataset Card for AVSD

Source: https://github.com/salesforce/lavis/blob/main/docs/tutorial.datasets.rst

An example of a dataset card in Markdown format for the AVSD dataset. It includes a description, task definition, metrics, and auto-download instructions, providing comprehensive information about the dataset.

```markdown
![Samples from the AVSD dataset (Image credit: "https://arxiv.org/pdf/1901.09107.pdf").](imgs/avsd_dialogue.png)(Samples from the AVSD dataset. Image credit: "https://arxiv.org/pdf/1901.09107.pdf")

# Audio-Visual Scene-Aware Dialogues (AVSD) 

## Description
[Audio-Visual Scene-Aware Dialogues (AVSD)](https://github.com/hudaAlamri/DSTC7-Audio-Visual-Scene-Aware-Dialog-AVSD-Challenge) contains more than 10,000 dialogues, each of which is grounded on a unique video. In the test split, for each test sample, 6 reference dialogue responses are provided. 


## Task

(https://github.com/hudaAlamri/DSTC7-Audio-Visual-Scene-Aware-Dialog-AVSD-Challenge)

In a **video-grounded dialogue task**, the system must generate responses to user input in the context of a given dialog.
This context consists of a dialog history (previous utterances by both user and system) in addition to video and audio information that comprise the scene. The quality of a system’s automatically generated sentences is evaluated using objective measures to determine whether or not the generated responses are natural and informative

## Metrics
Models are typically evaluated according to [BLEU](https://aclanthology.org/P02-1040/), [CIDER](https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Vedantam_CIDEr_Consensus-Based_Image_2015_CVPR_paper.pdf), [METEOR](https://aclanthology.org/W05-0909/), and [ROUGE-L](https://aclanthology.org/W04-1013/) metrics. 

## Leaderboard

....


## Auto-Downloading

Please refer to [benchmark webite](https://github.com/hudaAlamri/DSTC7-Audio-Visual-Scene-Aware-Dialog-AVSD-Challenge) for instructions to download the dataset. 


## References
"Audio Visual Scene-Aware Dialog", Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K. Marks, Chiori Hori, Peter Anderson, Stefan Lee, Devi Parikh
```

--------------------------------

### LLM Model Path Configuration

Source: https://github.com/salesforce/lavis/blob/main/projects/xinstructblip/demo/demo.ipynb

This code snippet defines a placeholder for the path to the Vicuna v1.1 model weights, which need to be downloaded separately.

```python
LLM_MODEL_PATH = "<add/llm/path/here>"
```

--------------------------------

### Load BLIP-2 Model and Preprocessors

Source: https://github.com/salesforce/lavis/blob/main/projects/blip2/README.md

Loads a pre-trained BLIP-2 model and its associated visual preprocessors. This step is necessary before performing inference.

```Python
import torch
from lavis.models import load_model_and_preprocess
# loads BLIP-2 pre-trained model
model, vis_processors, _ = load_model_and_preprocess(name="blip2_t5", model_type="pretrain_flant5xxl", is_eval=True, device=device)
# prepare the image
image = vis_processors["eval"](raw_image).unsqueeze(0).to(device)
```