### Install Training Dependencies

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/README.md

Clones the VideoLLaMA3 repository and installs all required packages for training, including flash-attn. Ensure you are in the cloned repository directory.

```bash
git clone https://github.com/DAMO-NLP-SG/VideoLLaMA3
cd VideoLLaMA3
pip install -r requirements.txt
pip install flash-attn --no-build-isolation
```

--------------------------------

### Start Training Scripts

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/README.md

Execute the provided bash scripts to start training for VideoLLaMA3 Stage 1 and Stage 2.

```bash
# VideoLLaMA3 Stage 1
bash scripts/train/stage1_2b.sh
# VideoLLaMA3 Stage 2
bash scripts/train/stage2_2b.sh
```

--------------------------------

### Data Organization Example

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/README.md

Organize your image and video data under a root directory and use annotation files (JSONL format) to record conversation data and file paths.

```bash
data_root
├── LLaVA-Video-178K
│   ├── video_1.mp4
│   └── ...
├── LLaVA-OneVision-Data
│   ├── image_1.jpg
│   └── ...
├── annotations_video.jsonl
├── annotations_image.jsonl
└── ...
```

--------------------------------

### Launch VideoLLaMA3 Gradio Demo

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/README.md

Start a local Gradio application for the VideoLLaMA3 model. Specify the model path and optional server/interface ports.

```bash
python inference/launch_gradio_demo.py --model-path DAMO-NLP-SG/VideoLLaMA3-7B
```

--------------------------------

### Install VideoLLaMA 3 Dependencies

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Installs necessary packages for VideoLLaMA 3 inference with CUDA 11.8. For full training, clone the repository and install from requirements.txt.

```bash
pip install torch==2.4.0 torchvision==0.19.0 --extra-index-url https://download.pytorch.org/whl/cu118
pip install flash-attn==2.7.3 --no-build-isolation --upgrade
pip install transformers==4.46.3 accelerate==1.0.1
pip install decord ffmpeg-python imageio opencv-python

# Full training install (clone repo first)
git clone https://github.com/DAMO-NLP-SG/VideoLLaMA3
cd VideoLLaMA3
pip install -r requirements.txt
pip install flash-attn --no-build-isolation
```

--------------------------------

### VideoLLaMA 3 Evaluation Script

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/README.md

Use this bash script to start the evaluation process. Ensure you set the MODEL_PATH, BENCHMARKS, NUM_NODES, and NUM_GPUS variables. You can customize DATA_ROOT and SAVE_DIR within the script.

```bash
bash scripts/eval/eval_video.sh ${MODEL_PATH} ${BENCHMARKS} ${NUM_NODES} ${NUM_GPUS}
```

--------------------------------

### VideoLLaMA Dataset Directory Structure

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/README.md

This is an example of the expected directory organization for evaluation datasets. Ensure your downloaded data is structured similarly.

```bash
benchmarks
└── video
│   ├── activitynet_qa
│   │   ├── all_test
│   │   ├── test_a.json
│   │   └── test_q.json
│   ├── charades
│   │   ├── Charades_v1
│   │   └── charades_annotations_test-random_prompt.json
│   ├── egoschema
│   │   ├── good_clips_git
│   │   └── questions.json
│   ├── longvideobench
│   │   ├── lvb_val.json
│   │   ├── subtitles
│   │   └── videos
│   ├── lvbench
│   │   ├── video
│   │   └── video_info.meta.jsonl
│   ├── mlvu
│   │   ├── json
│   │   └── video
│   ├── mvbench
│   │   ├── json
│   │   └── video
│   ├── nextqa
│   │   ├── map_vid_vidorID.json
│   │   ├── NExTVideo
│   │   └── test.csv
│   ├── perception_test
│   │   ├── mc_question_test.json
│   │   └── videos
│   ├── tempcompass
│   │   ├── captioning
│   │   ├── caption_matching
│   │   ├── multi-choice
│   │   ├── videos
│   │   └── yes_no
│   ├── videomme
│   │   ├── subtitles
│   │   ├── test-00000-of-00001.parquet
│   │   └── videos

```

--------------------------------

### Install Inference Dependencies

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/README.md

Installs specific versions of PyTorch, torchvision, flash-attn, transformers, and accelerate for stable inference. Also includes video processing libraries. Ensure CUDA version compatibility.

```bash
pip install torch==2.4.0 torchvision==0.19.0 --extra-index-url https://download.pytorch.org/whl/cu118
pip install flash-attn==2.7.3 --no-build-isolation --upgrade
pip install transformers==4.46.3 accelerate==1.0.1
pip install decord ffmpeg-python imageio opencv-python
```

--------------------------------

### Setup Helper Functions for Box Visualization

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/inference/notebooks/03_visual_referring_and_grounding.ipynb

Imports necessary libraries and defines helper functions for image manipulation, bounding box extraction, and visualization. These functions are used to process model outputs and display bounding boxes on images.

```python
from PIL import Image, ImageDraw
from IPython.display import Markdown, display
import matplotlib.pyplot as plt

import re

def extract_boxes(text):
    matches = re.findall(r'[[(.*?) ]]', text)
    result = []
    
    for match in matches:
        arrays = match.split('],[')
        inner_list = []
        
        for array in arrays:
            # Remove any remaining brackets and split by comma to get individual numbers
            numbers = array.replace('[', '').replace(']', '').split(',')
            inner_list.append([int(num) for num in numbers])
        
        result.extend(inner_list)

    return result

def normalized2raw(box, h, w):
    box = [int(box[0]/1000*w), int(box[1]/1000*h), int(box[2]/1000*w), int(box[3]/1000*h)]
    return box 

def raw2normalized(box, h, w):
    box = [int(box[0]/w*1000), int(box[1]/h*1000), int(box[2]/w*1000), int(box[3]/h*1000)]
    box = [min(b, 1000) for b in box]
    return box 

def show_box(raw_box, image, process=True):
    box_image = image.copy()
    if process:
        bbox = normalized2raw(raw_box, box_image.size[1], box_image.size[0])
    else:
        bbox = raw_box
    draw = ImageDraw.Draw(box_image)
    draw.rectangle([bbox[0], bbox[1], bbox[2], bbox[3]], outline='red', width=2)
    
    plt.imshow(box_image)
    plt.axis('off')
    plt.show()
```

--------------------------------

### Training Script Configuration Example

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/README.md

Modify training script variables to fit your data and model settings. Key parameters include data folder, data path, model path, and vision encoder.

```bash
  --data_folder ./datasets \
  --data_path ./datasets/annotations_video.jsonl ./datasets/annotations_image.jsonl \
  --model_path Qwen/Qwen2.5-1.5B-Instruct \
  --vision_encoder DAMO-NLP-SG/SigLIP-NaViT \

```

--------------------------------

### VideoLLaMA3 Training Annotation Format

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Example format for training annotations in JSONL. Includes samples for both image and video inputs with corresponding conversations.

```json
# Image sample:
# {"image": ["images/cat.jpg"], "conversations": [{"from": "human", "value": "<image>\nWhat is this?"}, {"from": "gpt", "value": "A cat."}]}
# Video sample:

```

--------------------------------

### Temporal Grounding within a Video

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/inference/notebooks/04_video_understanding.ipynb

Shows how to ask specific temporal questions about a video, such as when an event occurred. The model can output start and end timestamps for requested events.

```python
video_path = 'visuals/cola.mp4'
display(Video(video_path, width=480, height=300))
```

```python
conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "video",
                "video": {"video_path": video_path, "fps": 1, "max_frames": 180}
            },
            {
                "type": "text",
                "text": "When did the man pour the cola into the cup? Please output the start and end timestamps."
            },
        ]
    }
]



```

--------------------------------

### VideoLLaMA3 Processor for Inference and Training

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Use this processor for both inference and training. For inference, provide a conversation and images/videos. For training, set `return_labels=True` to get target tensors for supervised fine-tuning.

```python
from transformers import AutoProcessor
from videollama3.mm_utils import load_video, load_images

processor = AutoProcessor.from_pretrained(
    "DAMO-NLP-SG/VideoLLaMA3-7B", trust_remote_code=True
)

# --- Inference mode ---
frames, timestamps = load_video("assets/cat_and_chicken.mp4", fps=1, max_frames=32)
conversation = [
    {"role": "system", "content": "You are a helpful assistant."},
    {
        "role": "user",
        "content": [
            {"type": "video", "timestamps": timestamps, "num_frames": len(frames)},
            {"type": "text", "text": "Summarize the video."},
        ],
    },
]
inputs = processor(
    conversation=conversation,
    images=[("video", frames)],   # pre-loaded; skip automatic loading
    add_system_prompt=True,
    add_generation_prompt=True,
    return_tensors="pt",
)
print(inputs.keys())
# dict_keys(['input_ids', 'attention_mask', 'pixel_values', 'grid_sizes', 'merge_sizes', 'modals'])

# --- Training mode (returns labels) ---
training_conv = [
    {"role": "user",      "content": [{"type": "image", "image": {"image_path": "img.jpg"}},
                                       {"type": "text", "text": "Describe this image."}]},
    {"role": "assistant", "content": "The image shows a sunny beach."},
]
train_inputs = processor(
    conversation=training_conv,
    return_labels=True,
    return_tensors="pt",
)
print(train_inputs["labels"].shape)  # torch.Size([sequence_length])
```

--------------------------------

### Perform Visual Grounding for 'logo of the yellow car'

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/inference/notebooks/03_visual_referring_and_grounding.ipynb

Performs visual grounding to find the 'logo of the yellow car' in the image. This example demonstrates how to query for specific details within an image using a text prompt and the expected bounding box output format.

```python
conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "image", 
                "image": {"image_path": image_path}
            },
            {
                "type": "text", 
                "text": "Where is the logo of the yellow car? Answer in [[x0,y0,x1,y1]] format."
            },
        ]
    }
]



```

--------------------------------

### Launch Gradio Demo for VideoLLaMA3

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Launch an interactive web-based demo using Gradio. Supports image and video input directly from the browser. Customize model path, server port, interface port, and number of processes.

```bash
# Single GPU / default settings
python inference/launch_gradio_demo.py --model-path DAMO-NLP-SG/VideoLLaMA3-7B

# Multi-GPU with custom ports
python inference/launch_gradio_demo.py \
    --model-path DAMO-NLP-SG/VideoLLaMA3-7B \
    --server-port 16666 \
    --interface-port 7860 \
    --nproc 4
```

--------------------------------

### Understand a Common Video

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/inference/notebooks/04_video_understanding.ipynb

Demonstrates how to process a common video file for detailed description using VideoLLaMA3. It involves setting up the conversation with video input and then generating a response.

```python
video_path = 'visuals/basketball.mp4'
display(Video(video_path, width=480, height=300))
```

```python
conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "video",
                "video": {"video_path": video_path, "fps": 1, "max_frames": 180}
            },
            {
                "type": "text",
                "text": "Describe the video in detial."
            },
        ]
    }
]


# Single-turn conversation
inputs = processor(conversation=conversation, return_tensors="pt")
inputs = {k: v.cuda() if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
if "pixel_values" in inputs:
    inputs["pixel_values"] = inputs["pixel_values"].to(torch.bfloat16)

output_ids = model.generate(**inputs, max_new_tokens=256)
response = processor.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
display(Markdown(response))
```

--------------------------------

### model_init(model_path, max_visual_tokens=None, **kwargs)

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Loads a VideoLLaMA 3 checkpoint from a local path or HuggingFace Hub, constructs a Videollama3Processor from the image processor and tokenizer, and optionally caps the number of visual tokens processed per image/frame. Returns (model, processor).

```APIDOC
## model_init(model_path, max_visual_tokens=None, **kwargs)

### Description
Loads a VideoLLaMA 3 checkpoint from a local path or HuggingFace Hub, constructs a `Videollama3Processor` from the image processor and tokenizer, and optionally caps the number of visual tokens processed per image/frame. Returns `(model, processor)`.

### Parameters
- **model_path** (str) - Path to the model checkpoint or HuggingFace Hub identifier.
- **max_visual_tokens** (int, optional) - Maximum number of visual tokens to process per image/frame.
- **kwargs** - Additional keyword arguments to pass to the underlying model loading functions (e.g., `device_map`).

### Usage
```python
from videollama3 import disable_torch_init, model_init

disable_torch_init()

# Load 7B model onto a single GPU, capping visual tokens at 8192
model, processor = model_init(
    "DAMO-NLP-SG/VideoLLaMA3-7B",
    max_visual_tokens=8192,
    device_map={"ளுக்காக": "cuda:0"},
)
print(type(model))     # <class 'videollama3.model.videollama3_qwen2.VideoLLaMA3Qwen2ForCausalLM'>
print(type(processor)) # <class 'videollama3.model.processor.Videollama3Processor'>
```
```

--------------------------------

### Initialize VideoLLaMA 3 Model and Processor

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Loads a VideoLLaMA 3 checkpoint, creates a `Videollama3Processor`, and optionally limits visual tokens. Specify `device_map` for GPU allocation.

```python
from videollama3 import disable_torch_init, model_init

disable_torch_init()

# Load 7B model onto a single GPU, capping visual tokens at 8192
model, processor = model_init(
    "DAMO-NLP-SG/VideoLLaMA3-7B",
    max_visual_tokens=8192,
    device_map={"": "cuda:0"},
)
print(type(model))     # <class 'videollama3.model.videollama3_qwen2.VideoLLaMA3Qwen2ForCausalLM'>
print(type(processor)) # <class 'videollama3.model.processor.Videollama3Processor'>
```

--------------------------------

### Gradio Demo

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Launches a web-based interactive demo with a Gradio frontend backed by a VideoLLaMA3PlainServer. Supports both image and video input directly from the browser.

```APIDOC
## Gradio Demo

### Description
Launches an interactive web-based demo for VideoLLaMA3.

### Usage
```bash
# Single GPU / default settings
python inference/launch_gradio_demo.py --model-path DAMO-NLP-SG/VideoLLaMA3-7B

# Multi-GPU with custom ports
python inference/launch_gradio_demo.py \
    --model-path DAMO-NLP-SG/VideoLLaMA3-7B \
    --server-port 16666 \
    --interface-port 7860 \
    --nproc 4
```

### Parameters
- **--model-path** (str) - Path to the model.
- **--server-port** (int, optional) - Port for the backend server.
- **--interface-port** (int, optional) - Port for the Gradio interface.
- **--nproc** (int, optional) - Number of processes (GPU workers).
```

--------------------------------

### VideoLLaMA3 HuggingFace Transformers API - Video Inference

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Performs video inference using the HuggingFace Transformers API. This allows for model registration via `AutoModelForCausalLM` and `AutoProcessor` without installing the `videollama3` package, making it suitable for deployment.

```APIDOC
## HuggingFace Transformers API — Video Inference

### Description
The repository ships a fully self-contained Transformers-compatible implementation under `inference/transformers_api/`. Models registered via `AutoModelForCausalLM` and `AutoProcessor` work without installing the `videollama3` package, making them suitable for deployment environments.

### Usage Example
```python
import torch
from transformers import AutoModelForCausalLM, AutoProcessor

device = "cuda:0"
model_path = "DAMO-NLP-SG/VideoLLaMA3-7B"

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    trust_remote_code=True,
    device_map={"": device},
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

conversation = [
    {"role": "system", "content": "You are a helpful assistant."},
    {
        "role": "user",
        "content": [
            {
                "type": "video",
                "video": {
                    "video_path": "assets/cat_and_chicken.mp4",
                    "fps": 1,
                    "max_frames": 180,
                },
            },
            {"type": "text", "text": "What is the cat doing?"},
        ],
    },
]

inputs = processor(
    conversation=conversation,
    add_system_prompt=True,
    add_generation_prompt=True,
    return_tensors="pt",
)
inputs = {k: v.to(device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
if "pixel_values" in inputs:
    inputs["pixel_values"] = inputs["pixel_values"].to(torch.bfloat16)

output_ids = model.generate(**inputs, max_new_tokens=1024)
response = processor.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
print(response)
# Expected: "The cat is playfully chasing the chicken..."
```
```

--------------------------------

### Training Pipeline - 4-Stage Fine-tuning

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Details the four-stage curriculum for fine-tuning VideoLLaMA 3, using torchrun with DeepSpeed ZeRO.

```APIDOC
## Training Pipeline — 4-Stage Fine-tuning

### Description
VideoLLaMA 3 uses a four-stage curriculum for fine-tuning: Stage 1 aligns the vision encoder and projector on image-text data; Stage 2 adds video data; Stages 3 and 4 apply full multimodal instruction tuning. Each stage uses `torchrun` with DeepSpeed ZeRO.

### Stages
- **Stage 1 — vision-language alignment (2B model, 8 GPUs)**
  ```bash
  bash scripts/train/stage1_2b.sh
  ```
- **Stage 2 — video alignment**
  ```bash
  bash scripts/train/stage2_2b.sh
  ```
- **Stages 3 & 4 — instruction tuning**
  ```bash
  bash scripts/train/stage3_2b.sh
  bash scripts/train/stage4_2b.sh
  ```

### Training Annotation Format (`annotations.jsonl`)
- **Image sample:**
  ```json
  {"image": ["images/cat.jpg"], "conversations": [{"from": "human", "value": "<image>\nWhat is this?"}, {"from": "gpt", "value": "A cat."}]}
  ```
- **Video sample:**
  (Format details for video samples are not provided in the source text.)
```

--------------------------------

### Run VideoLLaMA3 Inference

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/README.md

Load the model and processor, prepare a conversation with video and text inputs, and generate a response. Ensure the correct device and data types are used for optimal performance.

```python
import torch
from transformers import AutoModelForCausalLM, AutoProcessor

device = "cuda:0"
model_path = "DAMO-NLP-SG/VideoLLaMA3-7B"
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    trust_remote_code=True,
    device_map={"": device},
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

conversation = [
    {"role": "system", "content": "You are a helpful assistant."},
    {
        "role": "user",
        "content": [
            {"type": "video", "video": {"video_path": "./assets/cat_and_chicken.mp4", "fps": 1, "max_frames": 180}},
            {"type": "text", "text": "What is the cat doing?"},
        ]
    },
]

inputs = processor(
    conversation=conversation,
    add_system_prompt=True,
    add_generation_prompt=True,
    return_tensors="pt"
)
inputs = {k: v.to(device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
if "pixel_values" in inputs:
    inputs["pixel_values"] = inputs["pixel_values"].to(torch.bfloat16)
output_ids = model.generate(**inputs, max_new_tokens=1024)
response = processor.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
print(response)
```

--------------------------------

### Load and Display Image

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/inference/notebooks/03_visual_referring_and_grounding.ipynb

Loads an image from a specified path and displays it. Ensure the image path is correct and the image is in RGB format.

```python
image_path = "visuals/cars.jpg"
image = Image.open(image_path).convert("RGB")

plt.imshow(image)
plt.axis('on')
plt.show()
```

--------------------------------

### Load and Display Image

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/inference/notebooks/01_single_image_understanding.ipynb

Loads an image from a specified path and displays it. Ensure the image path is correct.

```python
image_path = "visuals/doc.jpg"
image = Image.open(image_path).convert("RGB")

display(image)
```

--------------------------------

### Convert HuggingFace Checkpoint to Local Format

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Converts a HuggingFace-format VideoLLaMA 3 checkpoint to the local training format, remapping vision encoder weight keys. Ensure the target directory exists.

```bash
python scripts/convert_hf_checkpoint.py \
    --model_path DAMO-NLP-SG/VideoLLaMA3-7B \
    --save_path weights/videollama3_7b_local
```

--------------------------------

### VideoLLaMA3 Training Pipeline Scripts

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Scripts for the four-stage fine-tuning curriculum of VideoLLaMA3. Each stage uses `torchrun` with DeepSpeed ZeRO for distributed training.

```bash
# Stage 1 — vision-language alignment (2B model, 8 GPUs)
bash scripts/train/stage1_2b.sh

# Stage 2 — video alignment
bash scripts/train/stage2_2b.sh

# Stages 3 & 4 — instruction tuning
bash scripts/train/stage3_2b.sh
bash scripts/train/stage4_2b.sh
```

--------------------------------

### Annotation File Format

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/README.md

Annotation files are lists of dictionaries, each containing 'image' or 'video' paths and a list of 'conversations' with 'from' and 'value' fields. Use .jsonl for efficiency.

```json
[
    {
        "image": ["images/xxx.jpg"],
        "conversations": [
            {
                "from": "human",
                "value": "<image>\nWhat are the colors of the bus in the image?"
            },
            {
                "from": "gpt",
                "value": "The bus in the image is white and red."
            },
            ...
        ]
    },
    {
        "video": ["videos/xxx.mp4"],
        "conversations": [
            {
                "from": "human",
                "value": "<video>\nWhat are the main activities that take place in the video?"
            },
            {
                "from": "gpt",
                "value": "The main activities that take place in the video are the preparation of camera equipment by a man, a group of men riding a helicopter, and a man sailing a boat through the water."
            },
            ...
        ]
    },
    ...
]
```

--------------------------------

### DeepSpeed ZeRO Configuration

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/README.md

Use DeepSpeed ZeRO-2/3 for memory optimization by passing the appropriate JSON configuration file to the training script.

```bash
--deepspeed scripts/zero2.json / zero3.json
```

--------------------------------

### Load and Display Algorithm Image

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/inference/notebooks/01_single_image_understanding.ipynb

Loads an image containing an algorithm problem. Ensure the image path is correct.

```python
image_path = "visuals/algorithm.jpg"
image = Image.open(image_path).convert("RGB")

display(image)
```

--------------------------------

### Video Frame Sampling and Display Helpers

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/inference/notebooks/04_video_understanding.ipynb

Provides utility functions to sample frames from a video and display them in a grid. Useful for visualizing video content before or during analysis.

```python
import cv2
import matplotlib.pyplot as plt
import numpy as np

def sample_frames_from_video(video_path, num_frames=64):
    cap = cv2.VideoCapture(video_path)
    total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    step = total_frames // num_frames
    
    frames = []
    for i in range(num_frames):
        cap.set(cv2.CAP_PROP_POS_FRAMES, i * step)

        ret, frame = cap.read()
        if not ret:
            break

        frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        frames.append(frame_rgb)
    
    cap.release()
    return frames

def display_frames_grid(frames, grid_size=(8, 8)):
    fig, axes = plt.subplots(grid_size[0], grid_size[1], figsize=(16, 10))
    
    for i, ax in enumerate(axes.flat):
        if i < len(frames):
            ax.imshow(frames[i])
            ax.axis('off')
        else:
            ax.axis('off')
    
    plt.tight_layout()
    plt.show()
```

--------------------------------

### VideoLLaMA3 Plain TCP Server and Client

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Implement a lightweight TCP socket server for hosting model workers and streaming token-by-token output. The client connects to this server to submit requests and receive streamed responses.

```python
# ---- server side (run in a separate process) ----
from inference.server.plain_server import VideoLLaMA3PlainServer
import torch

server = VideoLLaMA3PlainServer(
    model_path="DAMO-NLP-SG/VideoLLaMA3-7B",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    num_processes=2,   # 2 GPU workers
    port=16666,
)
server.launch()  # blocks; spawns workers and opens TCP listener

# ---- client side ----
from inference.server.plain_server import VideoLLaMA3PlainClient

client = VideoLLaMA3PlainClient(host="localhost", port=16666)

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "video", "video": {"video_path": "assets/cat_and_chicken.mp4", "fps": 1, "max_frames": 60}},
            {"type": "text", "text": "What happens in this video?"},
        ],
    }
]
request_data = {
    "conversation": conversation,
    "generation_config": {"max_new_tokens": 512, "do_sample": False},
}
streamer = client.submit(request_data)

for token in streamer:
    print(token, end="", flush=True)
print()
```

--------------------------------

### VideoLLaMA3PlainServer / VideoLLaMA3PlainClient

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

A lightweight TCP socket server and client for streaming token-by-token output. Useful for building custom serving pipelines without a web framework dependency.

```APIDOC
## VideoLLaMA3PlainServer

### Description
Hosts model worker processes and streams token-by-token output back to clients via TCP sockets.

### Parameters
- **model_path** (str) - Path to the model.
- **torch_dtype** (torch.dtype) - The desired torch data type (e.g., torch.bfloat16).
- **attn_implementation** (str) - Attention implementation to use (e.g., "flash_attention_2").
- **num_processes** (int) - Number of GPU workers.
- **port** (int) - The port to listen on.

### Method
`server.launch()`

## VideoLLaMA3PlainClient

### Description
Connects to a VideoLLaMA3PlainServer and submits requests for processing.

### Parameters
- **host** (str) - The hostname of the server.
- **port** (int) - The port of the server.

### Method
`client.submit(request_data)`

### Request Example
```python
from inference.server.plain_server import VideoLLaMA3PlainClient

client = VideoLLaMA3PlainClient(host="localhost", port=16666)

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "video", "video": {"video_path": "assets/cat_and_chicken.mp4", "fps": 1, "max_frames": 60}},
            {"type": "text", "text": "What happens in this video?"},
        ],
    }
]
request_data = {
    "conversation": conversation,
    "generation_config": {"max_new_tokens": 512, "do_sample": False},
}
streamer = client.submit(request_data)

for token in streamer:
    print(token, end="", flush=True)
print()
```
```

--------------------------------

### Load VideoLLaMA3 Model and Processor

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/inference/notebooks/02_multi_image_understanding.ipynb

Loads the VideoLLaMA3 model and its corresponding processor. Requires significant VRAM and sets CUDA visible devices. Ensure `flash_attention_2` is available for optimal performance.

```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"

from PIL import Image
from IPython.display import Markdown, display
import matplotlib.pyplot as plt

import torch
from transformers import AutoModelForCausalLM, AutoProcessor, AutoModel, AutoImageProcessor

model_path = "DAMO-NLP-SG/VideoLLaMA3-7B-Image"
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    trust_remote_code=True,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
```

--------------------------------

### Load and Display Chart Image

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/inference/notebooks/01_single_image_understanding.ipynb

Loads a chart image using PIL and displays it using Matplotlib. This prepares the image for analysis by the model.

```python
image_path = "visuals/chart.jpg"
image = Image.open(image_path).convert("RGB")

plt.imshow(image)
plt.axis('off')
plt.show()
```

--------------------------------

### Load and Display Images

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/inference/notebooks/02_multi_image_understanding.ipynb

Loads two images from specified paths using PIL and displays them using Matplotlib. Ensure the image files exist at the provided paths.

```python
image_path1 = "visuals/city1.jpg"
image1 = Image.open(image_path1).convert("RGB")

print('image1')
plt.imshow(image1)
plt.axis('off')
plt.show()

image_path2 = "visuals/city2.jpg"
image2 = Image.open(image_path2).convert("RGB")

print('image2')
plt.imshow(image2)
plt.axis('off')
plt.show()
```

--------------------------------

### Distributed Evaluation of VideoLLaMA 3

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Runs distributed inference across benchmarks using torchrun. This script shards the dataset across GPUs, gathers results on rank 0, and prints accuracy tables. Ensure you have the correct number of GPUs available.

```bash
# Evaluate VideoLLaMA3-7B on VideoMME and MVBench with 8 GPUs
bash scripts/eval/eval_video.sh \
    DAMO-NLP-SG/VideoLLaMA3-7B \
    "videomme,mvbench" \
    1 \
    8
```

```bash
# Evaluate image benchmarks
bash scripts/eval/eval_image.sh \
    DAMO-NLP-SG/VideoLLaMA3-7B-Image \
    "mmmu,mmbench,mathvista" \
    1 \
    8
```

```bash
# Programmatic usage (single benchmark)
# torchrun --nproc_per_node 4 evaluation/evaluate.py \
#     --model_path DAMO-NLP-SG/VideoLLaMA3-7B \
#     --benchmark videomme \
#     --data_root /data/benchmarks/video/videomme \
#     --fps 1 \
#     --max_frames 180 \
#     --max_visual_tokens 16384 \
#     --save_path results/videomme.json
```

--------------------------------

### Flash Loss Implementation

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/README.md

Enable experimental flash loss for reduced memory consumption, especially with long contexts or large vocabularies, by setting `--use_flash_loss True`.

```bash
--use_flash_loss True
```

--------------------------------

### Convert Hugging Face Checkpoint

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/README.md

Convert checkpoints from Hugging Face format to local format for finetuning. Specify the model path and save path.

```bash
python scripts/convert_hf_checkpoint.py --model_path DAMO-NLP-SG/VideoLLaMA3-7B --save_path weights/videollama3_7b_local
```

--------------------------------

### Load and Display Image for Visual Grounding

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/inference/notebooks/03_visual_referring_and_grounding.ipynb

Loads an image from a specified path and displays it using matplotlib. This is a prerequisite for visual grounding tasks where the model needs to identify objects within an image.

```python
image_path = "visuals/cars.jpg"
image = Image.open(image_path).convert("RGB")

plt.imshow(image)
plt.axis('off')
plt.show()
```

--------------------------------

### Perform Multimodal Inference with VideoLLaMA 3

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Runs inference using preprocessed inputs. The `mm_infer` function handles tensor movement to CUDA and calls `model.generate`. Ensure `modal` is set to 'video' for video inputs.

```python
from videollama3 import disable_torch_init, model_init, mm_infer
from videollama3.mm_utils import load_video

disable_torch_init()
model, processor = model_init("DAMO-NLP-SG/VideoLLaMA3-7B")

frames, timestamps = load_video("assets/cat_and_chicken.mp4", fps=1, max_frames=180)

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "video", "timestamps": timestamps, "num_frames": len(frames)},
            {"type": "text", "text": "Describe in detail what the cat is doing."},
        ],
    }
]

inputs = processor(
    images=[frames],
    text=conversation,
    merge_size=2,          # merge_size=2 for video, =1 for image
    return_tensors="pt",
)

response = mm_infer(
    inputs,
    model=model,
    tokenizer=processor.tokenizer,
    modal="video",
    do_sample=False,
    max_new_tokens=512,
)
print(response)
# Expected: "The cat is chasing a chicken around the yard, pouncing and batting..."
```

--------------------------------

### Load and Display Table Image

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/inference/notebooks/01_single_image_understanding.ipynb

Loads an image containing a table using PIL and displays it using Matplotlib, preparing it for subsequent analysis.

```python
image_path = "visuals/table.jpg"
image = Image.open(image_path).convert("RGB")

plt.imshow(image)
plt.axis('off')
plt.show()
```

--------------------------------

### Analyze Chart Image with VideoLLaMA3

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/inference/notebooks/01_single_image_understanding.ipynb

Analyzes a chart image by engaging in a single-turn conversation with the VideoLLaMA3 model. The prompt asks for an opinion on a stock and whether it's worth holding, with the model's response displayed.

```python
conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "image", 
                "image": {"image_path": image_path}
            },
            {
                "type": "text", 
                "text": "What do you think of this stock? Is it worth holding? Why?"},
        ]
    }
]


# Single-turn conversation
inputs = processor(conversation=conversation, return_tensors="pt")
inputs = {k: v.cuda() if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
if "pixel_values" in inputs:
    inputs["pixel_values"] = inputs["pixel_values"].to(torch.bfloat16)

output_ids = model.generate(**inputs, max_new_tokens=256)
response = processor.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
display(Markdown(response))
```

--------------------------------

### Load VideoLLaMA3 Model and Processor

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/inference/notebooks/04_video_understanding.ipynb

Loads the VideoLLaMA3 model and its corresponding processor. Requires significant VRAM (approx. 15,967 MB at BFloat16). Ensures only GPU 0 is used by setting CUDA_VISIBLE_DEVICES.

```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"

from PIL import Image
from IPython.display import Markdown, clear_output, display, Video
import matplotlib.pyplot as plt

import torch
from transformers import AutoModelForCausalLM, AutoProcessor, AutoModel, AutoImageProcessor

model_path = "DAMO-NLP-SG/VideoLLaMA3-7B"
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    trust_remote_code=True,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
```

--------------------------------

### load_video

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Decodes a video file using FFmpeg (with fallback to `decord` for `.gif` and frame directories), applies optional temporal trimming and spatial rescaling, uniformly samples up to `max_frames` frames, and returns `(frames, timestamps)`. Supports MP4, MKV, GIF, and frame-directory inputs.

```APIDOC
## `load_video(video_path, start_time, end_time, fps, max_frames, size, ...)`

### Description
Decodes a video file using FFmpeg (with fallback to `decord` for `.gif` and frame directories), applies optional temporal trimming and spatial rescaling, uniformly samples up to `max_frames` frames, and returns `(frames, timestamps)`. Supports MP4, MKV, GIF, and frame-directory inputs.

### Usage Examples
```python
from videollama3.mm_utils import load_video

# Full video at 1 fps, max 180 frames
frames, timestamps = load_video("assets/cat_and_chicken.mp4", fps=1, max_frames=180)
print(len(frames), timestamps[:3])
# 12 [0.0, 1.0, 2.0]

# Clip from 30s to 90s, resize shortest side to 336px
frames, timestamps = load_video(
    "long_video.mp4",
    start_time=30.0,
    end_time=90.0,
    fps=2,
    max_frames=64,
    size=336,
)
print(len(frames))  # up to 64
```
```

--------------------------------

### Load and Decode Video

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Decodes a video file using FFmpeg (with fallback to decord for .gif and frame directories). Applies optional temporal trimming and spatial rescaling, uniformly samples up to max_frames frames, and returns (frames, timestamps). Supports MP4, MKV, GIF, and frame-directory inputs.

```python
from videollama3.mm_utils import load_video

# Full video at 1 fps, max 180 frames
frames, timestamps = load_video("assets/cat_and_chicken.mp4", fps=1, max_frames=180)
print(len(frames), timestamps[:3])
# 12 [0.0, 1.0, 2.0]

# Clip from 30s to 90s, resize shortest side to 336px
frames, timestamps = load_video(
    "long_video.mp4",
    start_time=30.0,
    end_time=90.0,
    fps=2,
    max_frames=64,
    size=336,
)
print(len(frames))  # up to 64
```

--------------------------------

### load_images

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Loads one or more images from various sources like file paths, directories, lists of paths, or existing PIL Image objects. It normalizes them to RGB PIL.Image instances.

```APIDOC
## `load_images(image_path)`

### Description
Loads one or more images from a file path, directory, list of paths, or existing PIL `Image` objects, normalizing them to RGB `PIL.Image` instances. Supports all common input forms in a single interface.

### Usage Examples
```python
from videollama3.mm_utils import load_images

# Single file
images = load_images("assets/desert.jpg")          # [<PIL.Image RGB>]

# Directory of frames
images = load_images("frames/")                    # sorted list of PIL images

# List of paths
images = load_images(["img1.jpg", "img2.jpg"])

# Already a PIL Image
from PIL import Image
pil_img = Image.open("assets/sora.png")
images = load_images(pil_img)                      # [<PIL.Image RGB>]

print(len(images), images[0].size)  # 1 (1280, 720)
```
```

--------------------------------

### mm_infer(data_dict, model, tokenizer, modal, **kwargs)

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Runs inference on preprocessed inputs. Accepts the BatchFeature dict returned by the processor, moves tensors to CUDA, and calls model.generate with configurable sampling parameters. Returns the decoded response string.

```APIDOC
## mm_infer(data_dict, model, tokenizer, modal, **kwargs)

### Description
Runs inference on preprocessed inputs. Accepts the `BatchFeature` dict returned by the processor, moves tensors to CUDA, and calls `model.generate` with configurable sampling parameters. Returns the decoded response string.

### Parameters
- **data_dict** (BatchFeature) - The preprocessed input data dictionary.
- **model** (nn.Module) - The loaded VideoLLaMA 3 model.
- **tokenizer** (PreTrainedTokenizer) - The tokenizer associated with the model.
- **modal** (str) - The modality of the input ('video' or 'image').
- **kwargs** - Additional keyword arguments to pass to `model.generate` (e.g., `do_sample`, `max_new_tokens`).

### Usage
```python
from videollama3 import disable_torch_init, model_init, mm_infer
from videollama3.mm_utils import load_video

disable_torch_init()
model, processor = model_init("DAMO-NLP-SG/VideoLLaMA3-7B")

frames, timestamps = load_video("assets/cat_and_chicken.mp4", fps=1, max_frames=180)

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "video", "timestamps": timestamps, "num_frames": len(frames)},
            {"type": "text", "text": "Describe in detail what the cat is doing."},
        ],
    }
]

inputs = processor(
    images=[frames],
    text=conversation,
    merge_size=2,          # merge_size=2 for video, =1 for image
    return_tensors="pt",
)

response = mm_infer(
    inputs,
    model=model,
    tokenizer=processor.tokenizer,
    modal="video",
    do_sample=False,
    max_new_tokens=512,
)
print(response)
# Expected: "The cat is chasing a chicken around the yard, pouncing and batting..."
```
```

--------------------------------

### Load VideoLLaMA3 Model and Processor

Source: https://github.com/damo-nlp-sg/videollama3/blob/main/inference/notebooks/03_visual_referring_and_grounding.ipynb

Loads the VideoLLaMA3 model and processor for image-based tasks. Requires approximately 15.9 GB of VRAM at BFloat16 precision. Ensures only one GPU is used by setting CUDA_VISIBLE_DEVICES.

```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"

import torch
from transformers import AutoModelForCausalLM, AutoProcessor, AutoModel, AutoImageProcessor

model_path = "DAMO-NLP-SG/VideoLLaMA3-7B-Image"
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    trust_remote_code=True,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
```

--------------------------------

### Load Images from Various Sources

Source: https://context7.com/damo-nlp-sg/videollama3/llms.txt

Loads images from a file path, directory, list of paths, or existing PIL Image objects. Normalizes them to RGB PIL.Image instances. Supports common input forms.

```python
from videollama3.mm_utils import load_images

# Single file
images = load_images("assets/desert.jpg")          # [<PIL.Image RGB>]

# Directory of frames
images = load_images("frames/")                    # sorted list of PIL images

# List of paths
images = load_images(["img1.jpg", "img2.jpg"])

# Already a PIL Image
from PIL import Image
pil_img = Image.open("assets/sora.png")
images = load_images(pil_img)                      # [<PIL.Image RGB>]

print(len(images), images[0].size)  # 1 (1280, 720)
```