### Quick Start: Using NVILA-HD-Video

Source: https://github.com/nvlabs/vila/blob/main/vila_hd/nvila_hd_video/README.md

This script demonstrates how to load the NVILA-HD-Video model and processor, set up video processing arguments, and run inference on a given video and prompt. Ensure AutoGaze is installed before running.

```python
import torch
from transformers import AutoModel, AutoProcessor

model_path = "nvidia/NVILA-8B-HD-Video"
video_path = "https://huggingface.co/datasets/bfshi/HLVid/resolve/main/example/clip_av_video_5_001.mp4"
prompt = "Question: What does the white text on the green road sign say?\n \nA. Hampden St\n \nB. Hampden Ave\n \nC. HampdenBlvd\n \nD. Hampden Rd\n \nPlease answer directly with the letter of the correct answer."

# ----- Video processing args ----- 
num_video_frames = 128           # Total sampled frames for tiles
num_video_frames_thumbnail = 64  # Total sampled frames for thumbnails
max_tiles_video = 48             # Max spatial tiles per video (one tile is 392x392)

# ----- AutoGaze args (tiles) -----
gazing_ratio_tile = [0.2] + [0.06] * 15  # Per-frame max gazing ratios (single float or list)
task_loss_requirement_tile = 0.6

# ----- AutoGaze args (thumbnails) -----
gazing_ratio_thumbnail = 1       # Set to None to skip gazing on thumbnails
task_loss_requirement_thumbnail = None

# ----- Batching -----
max_batch_size_autogaze = 16
max_batch_size_siglip = 32

# Load processor and model
processor = AutoProcessor.from_pretrained(
    model_path,
    num_video_frames=num_video_frames,
    num_video_frames_thumbnail=num_video_frames_thumbnail,
    max_tiles_video=max_tiles_video,
    gazing_ratio_tile=gazing_ratio_tile,
    gazing_ratio_thumbnail=gazing_ratio_thumbnail,
    task_loss_requirement_tile=task_loss_requirement_tile,
    task_loss_requirement_thumbnail=task_loss_requirement_thumbnail,
    max_batch_size_autogaze=max_batch_size_autogaze,
    trust_remote_code=True,
)

model = AutoModel.from_pretrained(
    model_path,
    trust_remote_code=True,
    device_map="auto",
    max_batch_size_siglip=max_batch_size_siglip,
)
model.eval()

# Run inference
video_token = processor.tokenizer.video_token
inputs = processor(text=f"{video_token}\n\n{prompt}", videos=video_path, return_tensors="pt")
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}

outputs = model.generate(**inputs)
response = processor.batch_decode(outputs[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0].strip()
print(response)

```

--------------------------------

### Start VILA Server (CLI)

Source: https://context7.com/nvlabs/vila/llms.txt

Starts the VILA server using the command-line interface. Specify the port, model path, and conversation mode.

```bash
python serving/server.py \
    --port 8000 \
    --model-path Efficient-Large-Model/NVILA-15B \
    --conv-mode auto
```

--------------------------------

### Install VILA Dependencies

Source: https://context7.com/nvlabs/vila/llms.txt

Use the provided script to set up the conda environment and install all necessary dependencies. Alternatively, manual installation steps are provided for specific packages.

```bash
# Create conda environment named 'vila' and install all dependencies
./environment_setup.sh vila

# Activate the environment
conda activate vila

# Manual install (alternative)
pip install --upgrade pip setuptools
pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.5.8/flash_attn-2.5.8+cu122torch2.3cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
pip install -e ".[train,eval]"
pip install triton==3.1.0 protobuf==3.20.*

# For HuggingFace-compatible remote code usage only
pip install transformers==4.46 accelerate opencv-python torchvision einops pillow
pip install git+https://github.com/bfshi/scaling_on_scales.git
```

--------------------------------

### Install LongVILA Environment

Source: https://github.com/nvlabs/vila/blob/main/longvila/README.md

Run this script to set up the necessary environment for the VILA project. Ensure you have Python 3.10+ installed.

```bash
./environment_setup.sh vila
```

--------------------------------

### Install One Logger Utility

Source: https://github.com/nvlabs/vila/blob/main/README.md

Optional: Install the one-logger utility for NVIDIA employees with a wandb account. Enable it by setting 'training_args.use_one_logger' to True in 'llava/train/args.py'.

```bash
pip install --index-url=https://sc-hw-artf.nvidia.com/artifactory/api/pypi/hwinf-mlwfo-pypi/simple --upgrade one-logger-utils
```

--------------------------------

### VILA API Client Example

Source: https://github.com/nvlabs/vila/blob/main/README.md

Python code example using the OpenAI SDK to interact with the VILA API server.

```APIDOC
## VILA API Client Example

### Description
Example of how to use the OpenAI Python SDK to send requests to the VILA API server.

### Code
```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000",
    api_key="fake-key",
)
response = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What’s in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://blog.logomyway.com/wp-content/uploads/2022/01/NVIDIA-logo.jpg",
                        # Or you can pass in a base64 encoded image
                        # "url": "data:image/png;base64,<base64_encoded_image>",
                    },
                },
            ],
        }
    ],
    model="NVILA-15B",
)
print(response.choices[0].message.content)
```
```

--------------------------------

### VILA API Server - CLI

Source: https://github.com/nvlabs/vila/blob/main/README.md

Command to start the VILA API server using the CLI.

```APIDOC
## VILA API Server (CLI)

### Description
Starts the VILA API server using the command line interface.

### Command
```bash
python -W ignore server.py \
    --port 8000 \
    --model-path Efficient-Large-Model/NVILA-15B \
    --conv-mode auto
```

### Parameters
- `--port` (integer): The port to run the server on.
- `--model-path` (string): Path to the VILA model.
- `--conv-mode` (string): Conversation mode.
```

--------------------------------

### Launch NVILA Serving Server

Source: https://github.com/nvlabs/vila/blob/main/serving/README.md

Launches the NVILA serving server with a specified port and model path. Use this to start the model inference service.

```bash
python serving/server.py --port 8001 --model-path Efficient-Large-Model/NVILA-15B --conv-mode auto
```

--------------------------------

### Install NVILA Dependencies

Source: https://github.com/nvlabs/vila/blob/main/llava/remote_code/INSTRUCTIONS.md

Install necessary libraries for NVILA, including transformers, accelerate, and specific git repositories. Ensure the transformers version is compatible.

```bash
pip install transformers==4.46 accelerate opencv-python torchvision einops pillow
pip install git+https://github.com/bfshi/scaling_on_scales.git
```

--------------------------------

### Enable FP8 Training Arguments

Source: https://github.com/nvlabs/vila/blob/main/scripts/fp8training/README.md

Add these arguments to your training script to enable FP8 training. Ensure COAT is installed and configured.

```diff
        --gradient_checkpointing True \
        --dataloader_num_workers 16 \
        --vflan_no_system_prompt True \
-        --report_to wandb
+        --report_to wandb \
+        --quantize_model "fp8Linear_qwen2" \
+        --fabit "E4M3" \
+        --fwbit "E4M3" \
+        --bobit "E5M2" \
+        --row_blocksize -1 \
+        --col_blocksize -1 \
+        --pad_to_multiple_of 128
```

--------------------------------

### Run VILA API Server with CLI

Source: https://github.com/nvlabs/vila/blob/main/README.md

Start the VILA API server using Python and FastAPI. Specify the port, model path, and conversation mode.

```bash
python -W ignore server.py \
    --port 8000 \
    --model-path Efficient-Large-Model/NVILA-15B \
    --conv-mode auto
```

--------------------------------

### Query VILA Server with OpenAI SDK (Image URL)

Source: https://context7.com/nvlabs/vila/llms.txt

Queries the VILA server using the OpenAI SDK. This example demonstrates sending a text prompt along with an image URL.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000", api_key="fake-key")

response = client.chat.completions.create(
    model="NVILA-15B",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {"type": "image_url", "image_url": {
                    "url": "https://blog.logomyway.com/wp-content/uploads/2022/01/NVIDIA-logo.jpg"
                }},
            ],
        }
    ]
)
print(response.choices[0].message.content)
```

--------------------------------

### VILA Inference for Image Description

Source: https://github.com/nvlabs/vila/blob/main/README.md

Perform quick inference with user prompts and images using the 'vila-infer' command. This example shows how to get an image description.

```bash
# image description
vila-infer \
    --model-path Efficient-Large-Model/NVILA-15B \
    --conv-mode auto \
    --text "Please describe the image" \
    --media demo_images/demo_img.png
```

--------------------------------

### VILA Training: Stage 1 Alignment

Source: https://context7.com/nvlabs/vila/llms.txt

Starts Stage 1 of the VILA training pipeline for alignment on LLaVA-CC3M-595K. Requires specifying the pretrained LLM backbone and data alias.

```bash
# Stage 1: Alignment — align vision and language on LLaVA-CC3M-595K (single 8xA100 node)
bash scripts/NVILA-Lite/align.sh \
    Efficient-Large-Model/Qwen2-VL-7B-Instruct \
    <alias_to_data>
# Saves to: runs/train/nvila-8b-align
```

--------------------------------

### VILA-HD Evaluation CLI

Source: https://github.com/nvlabs/vila/blob/main/vila_hd/vila_hd/README.md

Use `vila-eval` to evaluate VILA-HD model performance. This example reproduces results from Table 3 of the paper, specifying environment variables for patch selection and a list of datasets.

```bash
NUM_LOOK_CLOSE=6 SELECT_NUM_EACH_SCALE=512+2048 vila-eval -m nvidia/VILA-HD-8B-PS3-1.5K-SigLIP -c auto -t textvqa,lmms-chartqa,lmms-docvqa_val,lmms-ocrbench,lmms-realworldqa,mathvista_testmini,lmms-mmbench,lmms-mme,lmms-mmmu_pro,lmms-mmmu_val,lmms-scienceqa_full,lmms-infovqa_val
```

--------------------------------

### Activate Conda Environment

Source: https://github.com/nvlabs/vila/blob/main/README.md

Activate the 'vila' conda environment after installation.

```bash
conda activate vila
```

--------------------------------

### Batched Inference with AutoProcessor (HuggingFace)

Source: https://context7.com/nvlabs/vila/llms.txt

Performs batched inference using AutoProcessor and a VILA model. This example shows how to apply a chat template and generate output IDs.

```python
# --- Using AutoProcessor for batched inference ---
processor = AutoProcessor.from_pretrained(
    "Efficient-Large-Model/NVILA-Lite-2B-hf-preview",
    trust_remote_code=True
)
model.eval()

# Single-image batch
conv = [{
    "role": "user",
    "content": [
        {"type": "image", "path": "https://nvlabs.github.io/VILA/asset/example.jpg"},
        {"type": "text", "text": "Describe this image."}
    ]
}]
text = processor.apply_chat_template(conv, tokenize=False, add_generation_prompt=True)
inputs = processor([text])
output_ids = model.generate(
    input_ids=inputs.input_ids,
    media=inputs.media,
    media_config=inputs.media_config,
    generation_config=model.generation_config,
    max_new_tokens=256,
)
print(processor.tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0])
```

--------------------------------

### Image + Text Generation with VILA (HuggingFace)

Source: https://context7.com/nvlabs/vila/llms.txt

Generates content using a VILA model loaded via AutoModel, combining an image and a text prompt. Ensure PIL is installed for image handling.

```python
# Image + text generation
response = model.generate_content([
    PIL.Image.open("demo_images/demo_img.png"),
    "Describe this image in detail."
])
print(response)
```

--------------------------------

### Interact with VILA API using OpenAI SDK

Source: https://github.com/nvlabs/vila/blob/main/README.md

Example of how to send a chat completion request to the VILA API server using the OpenAI Python client. The request includes text and an image URL.

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000",
    api_key="fake-key",
)
response = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What’s in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://blog.logomyway.com/wp-content/uploads/2022/01/NVIDIA-logo.jpg",
                        # Or you can pass in a base64 encoded image
                        # "url": "data:image/png;base64,<base64_encoded_image>",
                    },
                },
            ]
        }
    ],
    model="NVILA-15B",
)
print(response.choices[0].message.content)
```

--------------------------------

### Download LLaVA-Instruct-150K Dataset

Source: https://github.com/nvlabs/vila/blob/main/data_prepare/README.md

Command to download the LLaVA-Instruct-150K dataset using huggingface-cli.

```bash
huggingface-cli download liuhaotian/LLaVA-Instruct-150K llava_v1_5_mix665k.json --repo-type dataset
```

--------------------------------

### Enable FP8 Training with COAT

Source: https://context7.com/nvlabs/vila/llms.txt

Arguments to enable FP8 mixed-precision training for faster and memory-efficient runs. Use these with SFT training scripts.

```bash
# Add these arguments to any SFT training script to enable FP8
bash scripts/fp8training/sft_qwen_fp8.sh

# Key FP8 arguments to append to your training command:
#   --quantize_model "fp8Linear_qwen2"
#   --fabit "E4M3"          # forward activation bits
#   --fwbit "E4M3"          # forward weight bits
#   --bobit "E5M2"          # backward output bits
#   --row_blocksize -1
#   --col_blocksize -1
#   --pad_to_multiple_of 128
```

```diff
# Diff showing how to add FP8 to an existing training config:
        --gradient_checkpointing True \
        --dataloader_num_workers 16 \
        --vflan_no_system_prompt True \
-       --report_to wandb
+       --report_to wandb \
+       --quantize_model "fp8Linear_qwen2" \
+       --fabit "E4M3" \
+       --fwbit "E4M3" \
+       --bobit "E5M2" \
+       --row_blocksize -1 \
+       --col_blocksize -1 \
+       --pad_to_multiple_of 128
```

--------------------------------

### Launch Single Dataset Training

Source: https://github.com/nvlabs/vila/blob/main/finetuning/README.md

Run the training script with the desired dataset indicated. Adjust global batch size or gradient accumulation steps if facing OOM issues.

```bash
# reduce the global bs or increase the accum steps if facing OOM issues
DEFAULT_RUN_NAME="NVILA-Lite-8B-finetune-trial" \
DEFAULT_GLOBAL_TRAIN_BATCH_SIZE=64 \
DEFAULT_GRADIENT_ACCUMULATION_STEPS=2 \
    bash scripts/NVILA-Lite/sft.sh \
        Efficient-Large-Model/NVILA-Lite-8B \
        SampleQA
```

--------------------------------

### VILA Training: Stage 3 Supervised Fine-Tuning (SFT)

Source: https://context7.com/nvlabs/vila/llms.txt

Executes Stage 3 of the VILA training pipeline for supervised fine-tuning on multimodal instruction data. Requires the pretrained model from Stage 2 and a data alias.

```bash
# Stage 3: Supervised Fine-Tuning on multimodal instruction data (8xA100)
bash scripts/NVILA-Lite/sft.sh \
    runs/train/nvila-8b-pretraining \
    <alias_to_data>
# Saves to: runs/train/nvila-8b-SFT
```

--------------------------------

### Launch Multi-Dataset Training

Source: https://github.com/nvlabs/vila/blob/main/finetuning/README.md

Concatenate multiple dataset names with '+' to finetune on several datasets simultaneously. Ensure to adjust batch size and accumulation steps as needed.

```bash
DEFAULT_RUN_NAME="NVILA-Lite-8B-finetune-trial" \
DEFAULT_GLOBAL_TRAIN_BATCH_SIZE=64 \
DEFAULT_GRADIENT_ACCUMULATION_STEPS=2 \
    bash scripts/NVILA-Lite/sft.sh \
        Efficient-Large-Model/NVILA-Lite-8B \
        SampleQA+SampleVideo+SampleOCR
```

--------------------------------

### Download WIT Dataset

Source: https://github.com/nvlabs/vila/blob/main/data_prepare/README.md

Use this command to download the subsampled WIT dataset in a LLaVA conversation format. The data is stored locally in a directory named WIT.

```bash
huggingface-cli download mit-han-lab/vila-dataset wit_processed_538k.json --repo-type dataset --local-dir WIT --local-dir-use-symlinks False
```

--------------------------------

### Load and Use NVILA Model

Source: https://github.com/nvlabs/vila/blob/main/llava/remote_code/README.md

Demonstrates loading the NVILA model from Hugging Face with `device_map='auto'` for automatic sharding and then using it for text-only and image-text generation. Ensure the model path is correct and `trust_remote_code=True` is set.

```python
from transformers import AutoConfig, AutoModel
from termcolor import colored

model_path = "Efficient-Large-Model/nvila_lite_2b_dev"
print("main_dev.py, loading from ", model_path)

# config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
# model = AutoModel.from_config(config, trust_remote_code=True)
model = AutoModel.from_pretrained(model_path, trust_remote_code=True, device_map="auto")
res = model.generate_content([
    "how are you today?"
])
print(colored(res, "cyan", attrs=["bold"]))

print("---" * 40)

import PIL.Image
response = model.generate_content([
    PIL.Image.open("inference_test/test_data/caption_meat.jpeg"),
    "describe the image?"
])
print(colored(response, "cyan", attrs=["bold"])).
```

--------------------------------

### VILA-HD Inference with Custom High-Res Patches

Source: https://github.com/nvlabs/vila/blob/main/vila_hd/vila_hd/README.md

Control the number of high-resolution patches processed during inference using environment variables like `NUM_LOOK_CLOSE`. This example increases the high-res encoding runs for potentially better accuracy.

```bash
NUM_LOOK_CLOSE=12 vila-infer --model-path nvidia/VILA-HD-8B-PS3-4K-SigLIP --conv-mode auto --text "Where does the exit lead to?" --media assets/av_example_1.jpg
```

--------------------------------

### LongVILA: Long-Context Video Training

Source: https://context7.com/nvlabs/vila/llms.txt

Scripts for extending VILA to support long-context video understanding up to 2048 frames. Includes LLM context extension and long video SFT stages.

```bash
# Stage 4: LLM context extension to 64k tokens
bash longvila/train/4_extend_llm_64k.sh \
    runs/train/nvila-8b-SFT \
    longvila-8b-64k \
    /data/slimpajama_64k.jsonl
# Saves to: checkpoints/longvila-8b-64k

# Progressive extension to 256k tokens
bash longvila/train/4_extend_llm_256k.sh \
    checkpoints/longvila-8b-64k \
    longvila-8b-256k \
    /data/slimpajama_256k.jsonl

# Stage 5: Long video SFT (256-frame, requires 32x 8xH100)
bash longvila/train/5_long_sft_256frames.sh \
    checkpoints/longvila-8b-64k \
    longvila-8b-256f

# Stage 5: Long video SFT (512-frame)
bash longvila/train/5_long_sft_512frames.sh \
    checkpoints/longvila-8b-256k \
    longvila-8b-512f

# Needle-in-a-haystack evaluation (6000 frames)
bash scripts/eval/needle.sh \
    LongVILA-7B-1M \
    Efficient-Large-Model/qwen2-7b-longvila-1M \
    /path/to/long_video.mp4 \
    6000 \
    300      # needle frame interval

# Benchmark evaluation with 256 frames
vila-eval \
    -m Efficient-Large-Model/qwen2-7b-longvila-256f \
    -c auto \
    --num_video_frames 256 \
    --tasks lmms-videomme-256,lmms-videomme_w_subtitle-256,egoschema_test
```

--------------------------------

### Video Dataset Structure

Source: https://github.com/nvlabs/vila/blob/main/finetuning/README.md

Organize your video dataset similarly to image datasets, with a JSON file and a videos directory.

```bash
/dataset/vila-dataset/sample-qa:
.
├── dataset.json
└── videos
    ├── 1.mp4
    ├── 2.mp4
    └── ......
```

--------------------------------

### VILA API Server - Docker

Source: https://github.com/nvlabs/vila/blob/main/README.md

Commands to build and run the VILA API server using Docker.

```APIDOC
## VILA API Server (Docker)

### Description
Builds and runs the VILA API server using Docker.

### Build Command
```bash
docker build -t vila-server:latest .
```

### Run Command
```bash
docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
    -v ./hub:/root/.cache/huggingface/hub \
    -it --rm -p 8000:8000 \
    -e VILA_MODEL_PATH=Efficient-Large-Model/NVILA-15B \
    -e VILA_CONV_MODE=auto \
    vila-server:latest
```

### Environment Variables
- `VILA_MODEL_PATH` (string): Path to the VILA model.
- `VILA_CONV_MODE` (string): Conversation mode.
```

--------------------------------

### Stage5 Long Supervised Fine-tuning (256 frames)

Source: https://github.com/nvlabs/vila/blob/main/longvila/README.md

Fine-tune the model to follow long video instructions for 256 frames. Requires the path to the Stage 4 extended model (64k) and an output name for the checkpoint.

```bash
bash longvila/train/5_long_sft_256frames.sh [EXTENDED_64k_PATH]
[OUTPUT_NAME]
```

--------------------------------

### Stage5 Long Supervised Fine-tuning (512 frames)

Source: https://github.com/nvlabs/vila/blob/main/longvila/README.md

Fine-tune the model to follow long video instructions for 512 frames. Requires the path to the Stage 4 extended model (256k) and an output name for the checkpoint.

```bash
bash longvila/train/5_long_sft_512frames.sh [EXTENDED_256k_PATH]
[OUTPUT_NAME]
```

--------------------------------

### Stage4 LLM Context Extension (256k)

Source: https://github.com/nvlabs/vila/blob/main/longvila/README.md

Progressively train the model from 64k context to 256k context. Requires the path to the 64k extended model, an output name, and the data file for 256k context. This step is optional if training for contexts longer than 256 frames is not required.

```bash
bash longvila/train/4_extend_llm_256k.sh [EXTENDED_64k_PATH] [OUTPUT_NAME] [DATA_FILE]
```

--------------------------------

### Stage4 LLM Context Extension (64k)

Source: https://github.com/nvlabs/vila/blob/main/longvila/README.md

Use this script to extend the LLM context to 64k using the SlimPajama dataset. Requires the path to the Stage 3 trained VILA model, an output name for the checkpoint, and the data file for 64k context.

```bash
bash longvila/train/4_extend_llm_64k.sh [STAGE3_PATH] [OUTPUT_NAME] [DATA_FILE]
```

--------------------------------

### Run VILA API Server with Docker

Source: https://github.com/nvlabs/vila/blob/main/README.md

Deploy the VILA API server using Docker. This command builds the Docker image and runs it, mapping ports and volumes, and setting environment variables for the model path and conversation mode.

```bash
docker build -t vila-server:latest .
docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
    -v ./hub:/root/.cache/huggingface/hub \
    -it --rm -p 8000:8000 \
    -e VILA_MODEL_PATH=Efficient-Large-Model/NVILA-15B \
    -e VILA_CONV_MODE=auto \
    vila-server:latest
```

--------------------------------

### Download M3IT Dataset

Source: https://github.com/nvlabs/vila/blob/main/data_prepare/README.md

Command to download the M3IT dataset using huggingface-cli.

```bash
huggingface-cli download MMInstruction/M3IT --repo-type dataset --local-dir M3IT --local-dir-use-symlinks False
```

--------------------------------

### VILA Training: Stage 2 Pretraining

Source: https://context7.com/nvlabs/vila/llms.txt

Initiates Stage 2 of the VILA training pipeline for pretraining on MMC4 + Coyo datasets. Requires the model from Stage 1.5 and a data alias.

```bash
# Stage 2: Pretraining on MMC4 + Coyo (interleaved image-text)
bash scripts/NVILA-Lite/pretrain.sh \
    runs/train/nvila-8b-align-1.5 \
    <alias_to_data>
# Saves to: runs/train/nvila-8b-pretraining
```

--------------------------------

### llava.load

Source: https://context7.com/nvlabs/vila/llms.txt

Load a pretrained VILA/NVILA model from a local path or HuggingFace Hub, with optional LoRA adapter and GPU device control.

```APIDOC
## Python API: llava.load

Load a pretrained VILA/NVILA model from a local path or HuggingFace Hub, with optional LoRA adapter and GPU device control.

```python
import llava
from llava import conversation as clib
from llava.media import Image, Video

# Load NVILA-15B from HuggingFace Hub onto all available GPUs
model = llava.load("Efficient-Large-Model/NVILA-15B")

# Load with a LoRA adapter (model_base=base, path=lora)
model = llava.load("path/to/lora-adapter", model_base="Efficient-Large-Model/NVILA-15B")

# Load onto specific GPU devices only
model = llava.load("Efficient-Large-Model/NVILA-8B", devices=[0, 1])

# Set conversation template (auto, vicuna_v1, llama_3, mpt, ...)
clib.default_conversation = clib.conv_templates["auto"].copy()

# --- Image inference ---
response = model.generate_content([
    Image("demo_images/demo_img.png"),
    "Please describe the image in detail."
])
print(response)
# Output: "The image shows a person sitting at a desk with a laptop..."

# --- Video inference ---
response = model.generate_content([
    Video("path/to/video.mp4"),
    "Summarize the key events in this video."
])
print(response)

# --- Multi-image in-context learning ---
response = model.generate_content([
    Image("example1.jpg"),
    "This is a cat.",
    Image("example2.jpg"),
    "This is a dog.",
    Image("query.jpg"),
    "What animal is this?"
])
print(response)

# --- Text-only ---
response = model.generate_content(["What is the capital of France?"])
print(response)  # "Paris"
```
```

--------------------------------

### VILA Training Step-1.5

Source: https://github.com/nvlabs/vila/blob/main/README.md

Execute the script for stage 1.5 of VILA training, using the aligned models from the previous step. Trained models are saved to 'runs/train/nvila-8b-align-1.5'.

```bash
bash scripts/NVILA-Lite/stage15.sh runs/train/nvila-8b-align/model <alias to data>
```

--------------------------------

### Download FLAN Datasets

Source: https://github.com/nvlabs/vila/blob/main/data_prepare/README.md

Command to download the FLAN datasets using huggingface-cli.

```bash
huggingface-cli download Open-Orca/FLAN --repo-type dataset --local-dir FLAN --local-dir-use-symlinks False
```

--------------------------------

### Video Dataset Registration in default.yaml

Source: https://github.com/nvlabs/vila/blob/main/finetuning/README.md

Register video datasets in default.yaml, including the 'is_video: true' flag.

```yaml
SampleVideo:
    _target_: llava.data.LLaVADataset
    data_path: /dataset/vila-dataset/sample-video/dataset.json
    media_dir: /dataset/vila-dataset/sample-video
    is_video: true
```

--------------------------------

### Custom Dataset Folder Structure for Image QA

Source: https://context7.com/nvlabs/vila/llms.txt

Illustrates the required folder structure for preparing custom image question-answering datasets for VILA fine-tuning. Includes a dataset JSON file and an images directory.

```bash
# Dataset folder structure for image QA:
# /dataset/vila-dataset/sample-qa/
# 	├── dataset.json
#     └── images/
#         ├── 1.jpg

```

--------------------------------

### Download ScienceQA Dataset

Source: https://github.com/nvlabs/vila/blob/main/data_prepare/README.md

Download the training split of the ScienceQA dataset, including image data, formatted for LLaVA conversations. The dataset is saved to a local directory named scienceqa.

```bash
huggingface-cli download mit-han-lab/vila-dataset scienceqa_train_12k.json --repo-type dataset --local-dir scienceqa --local-dir-use-symlinks False
```

--------------------------------

### Query NVILA Serving Model

Source: https://github.com/nvlabs/vila/blob/main/serving/README.md

Initiates a query to the already launched NVILA serving model. This script is used to interact with the deployed model.

```bash
python serving/query_nvila.py
```

--------------------------------

### NVILA Model Inference with Text

Source: https://github.com/nvlabs/vila/blob/main/llava/remote_code/INSTRUCTIONS.md

Load the NVILA model and generate content using only text prompts. This demonstrates basic text-based interaction with the model.

```python
from transformers import AutoConfig, AutoModel
from termcolor import colored

model_path = "Efficient-Large-Model/NVILA-Lite-2B-hf-preview"

# you can use config
config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
model = AutoModel.from_config(config, trust_remote_code=True)
# or directly from_pretrained
model = AutoModel.from_pretrained(model_path, trust_remote_code=True, device_map="auto")

# examples generate with raw text
res = model.generate_content([
    "how are you today?"
])
print(colored(res, "cyan", attrs=["bold"]))

print("---" * 40)

# examples generate with text + image
import PIL.Image
response = model.generate_content([
    PIL.Image.open("inference_test/test_data/caption_meat.jpeg"),
    "describe the image?"
])
print(colored(response, "cyan", attrs=["bold"])))
```

--------------------------------

### Dataset Registration in default.yaml

Source: https://github.com/nvlabs/vila/blob/main/finetuning/README.md

Add your dataset to llava/data/registry/datasets/default.yaml using the LLaVADataset target, specifying the data path and media directory.

```yaml
SampleQA:
    _target_: llava.data.LLaVADataset
    data_path: /dataset/vila-dataset/sample-qa/dataset.json
    media_dir: /dataset/vila-dataset/sample-qa
```

--------------------------------

### Evaluation: Needle in the Haystack Experiments

Source: https://github.com/nvlabs/vila/blob/main/longvila/README.md

Run the Needle in the Haystack evaluation experiment. Requires the model name, video path, and two numerical parameters.

```bash
bash scripts/eval/needle.sh LongVILA-7B-1M Efficient-Large-Model/qwen2-7b-longvila-1M $VIDEO_PATH 6000 300
```

--------------------------------

### Run VILA Inference with CLI

Source: https://github.com/nvlabs/vila/blob/main/README.md

Use the `vila-infer` command to perform inference on video or image files. Specify the model path, conversation mode, text prompt, and media source.

```bash
vila-infer \
    --model-path Efficient-Large-Model/NVILA-15B \
    --conv-mode auto \
    --text "Please describe the video" \
    --media https://huggingface.co/datasets/Efficient-Large-Model/VILA-inference-demos/resolve/main/OAI-sora-tokyo-walk.mp4
```

```bash
vila-infer \
    --model-path Efficient-Large-Model/VILA1.5-3b \
    --conv-mode vicuna_v1 \
    --text "Please describe the image" \
    --media demo_images/demo_img.png
```

```bash
vila-infer \
    --model-path Efficient-Large-Model/VILA1.5-3b \
    --conv-mode vicuna_v1 \
    --text "Please describe the video" \
    --media https://huggingface.co/datasets/Efficient-Large-Model/VILA-inference-demos/resolve/main/OAI-sora-tokyo-walk.mp4
```

```bash
vila-infer \
    --model-path Efficient-Large-Model/NVILA-15B \
    --conv-mode auto \
    --text "Please describe the video" \
    --media https://huggingface.co/datasets/Efficient-Large-Model/VILA-inference-demos/resolve/main/OAI-sora-tokyo-walk.mp4
```

--------------------------------

### VILA Training Step-1: Alignment

Source: https://github.com/nvlabs/vila/blob/main/README.md

Run the alignment script for VILA training using the LLaVA-CC3M-Pretrain-595K dataset. This stage aligns textual and visual modalities and can run on a single 8xA100 node. Trained models are saved to 'runs/train/nvila-8b-align'.

```bash
bash scripts/NVILA-Lite/align.sh Efficient-Large-Model/Qwen2-VL-7B-Instruct <alias to data>
```

--------------------------------

### VILA Training Step-3: Supervised Fine-tuning

Source: https://github.com/nvlabs/vila/blob/main/README.md

The final stage of VILA training involves supervised fine-tuning on a subset of M3IT, FLAN, and ShareGPT4V datasets to follow multimodal instructions. This stage runs on an 8xA100 node. Trained models are saved to 'runs/train/nvila-8b-SFT'.

```bash
bash scripts/NVILA-Lite/sft.sh runs/train/nvila-8b-pretraining <alias to data>
```

--------------------------------

### Citation for AutoGaze

Source: https://github.com/nvlabs/vila/blob/main/vila_hd/nvila_hd_video/README.md

This BibTeX entry should be used for citing the AutoGaze work when referencing NVILA-HD-Video.

```bibtex
@article{autogaze,
  title={AutoGaze},
  author={},
  year={2025}
}

```

--------------------------------

### Query VILA Server with OpenAI SDK (Base64 Image)

Source: https://context7.com/nvlabs/vila/llms.txt

Queries the VILA server using the OpenAI SDK with a base64-encoded local image. Ensure the image file exists and is correctly encoded.

```python
import base64
with open("local_image.jpg", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="NVILA-15B",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this diagram."},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}},
            ],
        }
    ]
)
print(response.choices[0].message.content)
```

--------------------------------

### Download COYO-700M Metadata

Source: https://github.com/nvlabs/vila/blob/main/data_prepare/README.md

Command to download the metadata for the COYO-700M dataset using huggingface-cli.

```bash
huggingface-cli download kakaobrain/coyo-700m --repo-type dataset --local-dir coyo-700m --local-dir-use-symlinks False
```

--------------------------------

### Preprocess M3IT Dataset

Source: https://github.com/nvlabs/vila/blob/main/data_prepare/README.md

Script to preprocess the M3IT dataset.

```bash
python preprocess_m3it.py
```

--------------------------------

### Convert VILA Checkpoint to HuggingFace Format

Source: https://context7.com/nvlabs/vila/llms.txt

Converts a native VILA checkpoint to the HuggingFace format using `VILAForCasualLM.convert_vila_dev_ckpt_to_remote`. Set `copy=True` to keep the original checkpoint.

```python
# --- Convert a native VILA checkpoint to HF format ---
from llava.remote_code.modeling_vila import VILAForCasualLM
VILAForCasualLM.convert_vila_dev_ckpt_to_remote(
    "Efficient-Large-Model/NVILA-Lite-2B",
    "NVILA-Lite-2B-hf-preview",
    copy=False
)
```

--------------------------------

### VILA Inference Command

Source: https://github.com/nvlabs/vila/blob/main/README.md

Command to perform inference with VILA models for video description.

```APIDOC
## vila-infer

### Description
Performs inference with VILA models for video description.

### Usage
```bash
vila-infer \
    --model-path Efficient-Large-Model/NVILA-15B \
    --conv-mode auto \
    --text "Please describe the video" \
    --media https://huggingface.co/datasets/Efficient-Large-Model/VILA-inference-demos/resolve/main/OAI-sora-tokyo-walk.mp4
```

### Parameters
- `--model-path` (string): Path to the VILA model.
- `--conv-mode` (string): Conversation mode.
- `--text` (string): The text prompt for description.
- `--media` (string): The URL or path to the media file (video or image).
```

--------------------------------

### VILA Training Step-2: Pretraining

Source: https://github.com/nvlabs/vila/blob/main/README.md

Perform VLM pretraining using MMC4 and Coyo datasets with interleaved image-text pairs. This script uses the models from Step 1.5. Trained models are saved to 'runs/train/nvila-8b-pretraining'.

```bash
bash scripts/NVILA-Lite/pretrain.sh runs/train/nvila-8b-align-1.5 <alias to data>
```

--------------------------------

### Download MMC4-Core Images

Source: https://github.com/nvlabs/vila/blob/main/data_prepare/README.md

Use this script to download images for the MMC4-core dataset. Modify input and output paths before running. Consider sharding for faster processing.

```bash
cd mmc4
python mmc4_downloader.py
```

```bash
# provide the start and end index of the jsonl shard. There are 23098 - 14 shards totally
# python mmc4_downloader.py <start_idx> <end_idx>
python mmc4_downloader.py 0 1000  # worker 1
python mmc4_downloader.py 1000 2000  # worker 2
```

--------------------------------

### Single Image Dataset Structure

Source: https://github.com/nvlabs/vila/blob/main/finetuning/README.md

Organize your single-image dataset with a JSON file and an images directory. The JSON file should contain image paths relative to the dataset root.

```bash
/dataset/vila-dataset/sample-qa:
.
├── dataset.json
└── images
    ├── 1.jpg
    ├── 2.jpg
    └── ......
```

--------------------------------

### Launch VILA Evaluations

Source: https://github.com/nvlabs/vila/blob/main/README.md

Use the 'vila-eval' command to simplify the evaluation process. This command launches all evaluations and returns a summarized result after data preparation.

```bash
MODEL_NAME=NVILA-15B
MODEL_ID=Efficient-Large-Model/$MODEL_NAME
huggingface-cli download $MODEL_ID

vila-eval \
    --model-name $MODEL_NAME \
    --model-path $MODEL_ID \
    --conv-mode auto \
    --tags-include local
```

--------------------------------

### Download COYO-700M Images

Source: https://github.com/nvlabs/vila/blob/main/data_prepare/README.md

Script to download COYO images. It processes each shard and keeps a subset with high CLIP similarity.

```bash
cd coyo
for SHARD in {0..127}; do
    python coyo_downloader.py $SHARD
done
```

--------------------------------

### VILA Training: Stage 1.5 Extended Alignment

Source: https://context7.com/nvlabs/vila/llms.txt

Performs extended alignment in Stage 1.5 of the VILA training pipeline. Requires the path to the model from Stage 1 and a data alias.

```bash
# Stage 1.5: Extended alignment
bash scripts/NVILA-Lite/stage15.sh \
    runs/train/nvila-8b-align/model \
    <alias_to_data>
# Saves to: runs/train/nvila-8b-align-1.5
```

--------------------------------

### Load VILA Model with AutoModel (HuggingFace)

Source: https://context7.com/nvlabs/vila/llms.txt

Loads a VILA model using HuggingFace's AutoModel. Requires `trust_remote_code=True`. Automatically maps the model to available devices.

```python
from transformers import AutoModel, AutoProcessor, AutoConfig
import PIL.Image

# Load NVILA-Lite-2B with automatic device placement
model = AutoModel.from_pretrained(
    "Efficient-Large-Model/NVILA-Lite-2B-hf-preview",
    trust_remote_code=True,
    device_map="auto"
)

# Text-only generation
res = model.generate_content(["How are you today?"])
print(res)
```

--------------------------------

### Uploading Models and Datasets with vila-upload

Source: https://context7.com/nvlabs/vila/llms.txt

Upload local model checkpoints or datasets to HuggingFace Hub. Supports auto-repo creation, specific repo IDs, and fast checking. Exclude files using patterns.

```bash
vila-upload runs/train/nvila-8b-SFT \
    --repo-type model \
    --repo-org Efficient-Large-Model
```

```bash
vila-upload runs/train/nvila-8b-SFT \
    --repo-type model \
    --repo-id MyOrg/my-custom-model
```

```bash
vila-upload /data/my-video-dataset \
    --repo-type dataset \
    --repo-org MyOrg \
    --model-name my-video-dataset
```

```bash
vila-upload runs/train/nvila-8b-SFT \
    --repo-type model \
    --fast-check
```

```bash
vila-upload runs/train/nvila-8b-SFT \
    --repo-type model \
    --exclude "*.log" \
    --exclude "tmp/.*"
```