### Install VideoMAE V2 Environment

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Sets up the conda environment and installs necessary dependencies including PyTorch, timm, and decord.

```bash
# Create and activate conda environment
conda create --name videomae python=3.8 -y
conda activate videomae

# Install PyTorch (>= 1.12.0 recommended for lower GPU memory usage)
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 -c pytorch

# Install dependencies
pip install -r requirements.txt

# Key dependencies:
# - timm==0.4.12 (required for API compatibility)
# - decord (for video loading)
# - deepspeed (for distributed training)
# - einops, opencv-python, tensorboard
```

--------------------------------

### Inference Setup

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Imports necessary libraries for running inference on a single video using a fine-tuned model. This includes PyTorch, NumPy, timm, torchvision, and decord.

```python
import torch
import numpy as np
from timm.models import create_model
from torchvision import transforms
from decord import VideoReader, cpu
import models  # Register models
```

--------------------------------

### Pre-train Data List Format Examples

Source: https://github.com/opengvlab/videomaev2/blob/master/docs/DATASET.md

Examples of data list file formats for pre-training. For video data, use 'video_path 0 -1'. For rawframes data, use 'frame_folder_path start_index total_frames'. The path prefix can be set using `--data_root`.

```text
# The path prefix 'your_path' can be specified by `--data_root ${PATH_PREFIX}` in scripts when training or inferencing.

your_path/k400/---QUuC4vJs.mp4 0 -1
your_path/k400/--VnA3ztuZg.mp4 0 -1
...
your_path/k700/-0H3T2B9PH4_000025_000035.mp4 0 -1
your_path/k700/-1IlTIWPNs4_000043_000053.mp4 0 -1
...
your_path/webvid2m/016401_016450/1017127174.mp4 0 -1
your_path/webvid2m/026551_026600/1056070034.mp4 0 -1
...
your_path/AVA/frames/clip/zlVkeKC6Ha8 9601 300
your_path/AVA/frames/clip/zlVkeKC6Ha8 9901 300
...
your_path/SomethingV2/frames/182040 1 58
your_path/SomethingV2/frames/197728 1 29
...
```

--------------------------------

### Distributed Finetuning Command

Source: https://github.com/opengvlab/videomaev2/blob/master/docs/FINETUNE.md

Use this command to start the finetuning process. Ensure environment variables like GPUS_PER_NODE, MASTER_PORT, DATA_PATH, MODEL_PATH, and OUTPUT_DIR are set. Refer to `run_class_finetuning.py` for hyperparameter meanings.

```bash
OMP_NUM_THREADS=1 python -m torch.distributed.launch --nproc_per_node=${GPUS_PER_NODE} \
        --master_port ${MASTER_PORT} --nnodes=${N_NODES} --node_rank=$1 --master_addr=$2 \
        run_class_finetuning.py \
        --model vit_giant_patch14_224 \
        --data_set Kinetics-710 \
        --nb_classes 710 \
        --data_path ${DATA_PATH} \
        --finetune ${MODEL_PATH} \
        --log_dir ${OUTPUT_DIR} \
        --output_dir ${OUTPUT_DIR} \
        --batch_size 3 \
        --input_size 224 \
        --short_side_size 224 \
        --save_ckpt_freq 10 \
        --num_frames 16 \
        --sampling_rate 4 \
        --num_sample 2 \
        --num_workers 10 \
        --opt adamw \
        --lr 1e-3 \
        --drop_path 0.3 \
        --clip_grad 5.0 \
        --layer_decay 0.9 \
        --opt_betas 0.9 0.999 \
        --weight_decay 0.1 \
        --warmup_epochs 5 \
        --epochs 35 \
        --test_num_segment 5 \
        --test_num_crop 3 \
        --dist_eval --enable_deepspeed \
        ${PY_ARGS}
```

--------------------------------

### Fine-tune Video Data List Format Example

Source: https://github.com/opengvlab/videomaev2/blob/master/docs/DATASET.md

Example of a data list file format for fine-tuning with video data. The format is 'video_path label'. The path prefix can be set using `--data_root`.

```text
# The path prefix 'your_path' can be specified by `--data_root ${PATH_PREFIX}` in scripts when training or inferencing.

# k400 video data validation list
your_path/k400/jf7RDuUTrsQ.mp4 325
your_path/k400/JTlatknwOrY.mp4 233
your_path/k400/NUG7kwJ-614.mp4 103
your_path/k400/y9r115bgfNk.mp4 320
your_path/k400/ZnIDviwA8CE.mp4 244
...
```

--------------------------------

### Install Project Requirements

Source: https://github.com/opengvlab/videomaev2/blob/master/docs/INSTALL.md

Installs all necessary Python packages listed in the 'requirements.txt' file using pip. This command should be run after activating the conda environment.

```bash
pip install -r requirements.txt
```

--------------------------------

### Initiate Distributed Training Script

Source: https://github.com/opengvlab/videomaev2/blob/master/docs/FINETUNE.md

Execute this script on each node to start the distributed training. Set NODE_RANK to the current node's rank and MASTER_ADDR to the IP address of the master node.

```bash
NODE_RANK=0  # 0 for the first node 0, 1 for the second node, and so on.
# MASTER_ADDR should be set as the ip of current node

bash dist_train_vit_g_k710_ft.sh $NODE_RANK $MASTER_ADDR
```

--------------------------------

### Fine-tune Rawframes Data List Format Example

Source: https://github.com/opengvlab/videomaev2/blob/master/docs/DATASET.md

Example of a data list file format for fine-tuning with rawframes data. The format is 'frame_folder_path total_frames label'. The path prefix can be set using `--data_root`.

```text
# The path prefix 'your_path' can be specified by `--data_root ${PATH_PREFIX}` in scripts when training or inferencing.

# ssv2 rawframes data validation list
your_path/SomethingV2/frames/74225 62 140
your_path/SomethingV2/frames/116154 51 127
your_path/SomethingV2/frames/198186 47 173
your_path/SomethingV2/frames/137878 29 99
your_path/SomethingV2/frames/151151 31 166
...
```

--------------------------------

### Install PyTorch with Conda

Source: https://github.com/opengvlab/videomaev2/blob/master/docs/INSTALL.md

Installs specific versions of PyTorch, torchvision, and torchaudio using the conda package manager. Ensure compatibility with your CUDA version if using GPU.

```bash
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 -c pytorch
```

--------------------------------

### Pre-train VideoMAE V2 with Dual Masking

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Example of ViT-giant pre-training using dual masking (encoder tube masking and decoder running cell masking) on the UnlabeledHybrid dataset.

```bash
#!/usr/bin/env bash
set -x

export MASTER_PORT=$((12000 + $RANDOM % 20000))
export OMP_NUM_THREADS=1

OUTPUT_DIR='YOUR_PATH/work_dir/vit_g_hybrid_pt_1200e'
DATA_PATH='YOUR_PATH/data/hybrid_train.csv'

# Slurm multi-node training (64 GPUs = 8 nodes x 8 GPUs)
srun -p video \
    --job-name=hybrid_pretrain \
    --gres=gpu:8 \
    --ntasks=64 \
    --ntasks-per-node=8 \
    --cpus-per-task=14 \
    --kill-on-bad-exit=1 \
    --async \
    python run_mae_pretraining.py \
    --data_path ${DATA_PATH} \
    --mask_type tube \
    --mask_ratio 0.9 \
    --decoder_mask_type run_cell \
    --decoder_mask_ratio 0.5 \
    --model pretrain_videomae_giant_patch14_224 \
    --decoder_depth 4 \
    --batch_size 32 \
    --with_checkpoint \
    --num_frames 16 \
    --sampling_rate 4 \
    --num_sample 4 \
    --num_workers 10 \
    --opt adamw \
    --lr 6e-4 \
    --clip_grad 0.02 \
    --opt_betas 0.9 0.95 \
    --warmup_epochs 30 \
    --save_ckpt_freq 5 \
    --epochs 300 \
    --log_dir ${OUTPUT_DIR} \
    --output_dir ${OUTPUT_DIR}

# Key pre-training parameters:
# --mask_type tube: Tube masking for encoder (90% masked)
# --decoder_mask_type run_cell: Running cell masking for decoder (50% masked)
# --num_frames 16: Number of input frames
# --sampling_rate 4: Frame sampling interval
# --num_sample 4: Number of augmented samples per video
```

--------------------------------

### Pre-train VideoMAE V2 with Distributed Launch

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Use this script for multi-node pre-training without Slurm. Ensure MASTER_PORT is set and adjust NODE_RANK for each node.

```bash
#!/usr/bin/env bash
set -x

export MASTER_PORT=${MASTER_PORT:-12320}

OUTPUT_DIR='YOUR_PATH/work_dir/vit_g_hybrid_pt_1200e'
DATA_PATH='YOUR_PATH/data/hybrid_train.csv'

N_NODES=8
GPUS_PER_NODE=8

# Run on each node (change NODE_RANK for each node: 0, 1, 2, ...)
NODE_RANK=0
MASTER_ADDR=192.168.1.1  # IP of node 0

OMP_NUM_THREADS=1 python -m torch.distributed.launch \
    --nproc_per_node=${GPUS_PER_NODE} \
    --master_port ${MASTER_PORT} \
    --nnodes=${N_NODES} \
    --node_rank=${NODE_RANK} \
    --master_addr=${MASTER_ADDR} \
    run_mae_pretraining.py \
    --data_path ${DATA_PATH} \
    --mask_type tube \
    --mask_ratio 0.9 \
    --decoder_mask_type run_cell \
    --decoder_mask_ratio 0.5 \
    --model pretrain_videomae_giant_patch14_224 \
    --decoder_depth 4 \
    --batch_size 32 \
    --with_checkpoint \
    --num_frames 16 \
    --sampling_rate 4 \
    --num_sample 4 \
    --num_workers 10 \
    --opt adamw \
    --lr 6e-4 \
    --clip_grad 0.02 \
    --opt_betas 0.9 0.95 \
    --warmup_epochs 30 \
    --save_ckpt_freq 5 \
    --epochs 300 \
    --log_dir ${OUTPUT_DIR} \
    --output_dir ${OUTPUT_DIR}
```

--------------------------------

### Build Fine-tuning Dataset

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Builds a dataset for fine-tuning using the 'build_dataset' function. Requires a configuration object 'args' with dataset-specific parameters like data path, number of classes, and video/frame settings.

```python
from dataset import build_dataset, build_pretraining_dataset
from dataset.datasets import VideoClsDataset, RawFrameClsDataset

# Build fine-tuning dataset
# Requires args with: data_path, data_root, data_set, nb_classes,
#                     num_frames, sampling_rate, input_size, short_side_size,
#                     test_num_segment, test_num_crop, sparse_sample, etc.

class Args:
    data_path = '/path/to/k400'
    data_root = ''
    data_set = 'Kinetics-400'
    nb_classes = 400
    num_frames = 16
    sampling_rate = 4
    input_size = 224
    short_side_size = 224
    test_num_segment = 5
    test_num_crop = 3
    sparse_sample = False
    fname_tmpl = 'img_{:05}.jpg'
    start_idx = 1

args = Args()
train_dataset, nb_classes = build_dataset(is_train=True, test_mode=False, args=args)
val_dataset, _ = build_dataset(is_train=False, test_mode=False, args=args)
test_dataset, _ = build_dataset(is_train=False, test_mode=True, args=args)
```

--------------------------------

### Distributed Pre-training Script for VideoMAEv2

Source: https://github.com/opengvlab/videomaev2/blob/master/scripts/pretrain/README.md

This script is for multi-node distributed training of VideoMAEv2. It requires setting MASTER_PORT, OUTPUT_DIR, and DATA_PATH. The script uses `torch.distributed.launch` and requires node rank and master address as arguments.

```bash
#!/usr/bin/env bash
set -x  # print the commands

export MASTER_PORT=${MASTER_PORT:-12320}  # You should set the same master_port in all the nodes

OUTPUT_DIR='YOUR_PATH/work_dir/vit_g_hybrid_pt_1200e'  # Your output folder for deepspeed config file, logs and checkpoints
DATA_PATH='YOUR_PATH/data/hybrid_train.csv'  # The data list file path.
# pretrain data list file follows the following format
# for the video data line: video_path, 0, -1, 0
# for the rawframe data line: frame_folder_path, start_index, total_frames, 0

N_NODES=${N_NODES:-8}  # Number of nodes
GPUS_PER_NODE=${GPUS_PER_NODE:-8}  # Number of GPUs in each node
SRUN_ARGS=${SRUN_ARGS:-""}  # Other slurm task args
PY_ARGS=${@:3}  # Other training args

# Please refer to `run_mae_pretraining.py` for the meaning of the following hyperreferences
OMP_NUM_THREADS=1 python -m torch.distributed.launch --nproc_per_node=${GPUS_PER_NODE} \
        --master_port ${MASTER_PORT} --nnodes=${N_NODES} --node_rank=$1 --master_addr=$2 \
        run_mae_pretraining.py \
        --data_path ${DATA_PATH} \
        --mask_type tube \
        --mask_ratio 0.9 \
        --decoder_mask_type run_cell \
        --decoder_mask_ratio 0.5 \
        --model pretrain_videomae_giant_patch14_224 \
        --decoder_depth 4 \
        --batch_size 32 \
        --with_checkpoint \
        --num_frames 16 \
        --sampling_rate 4 \
        --num_sample 4 \
        --num_workers 10 \
        --opt adamw \
        --lr 6e-4 \
        --clip_grad 0.02 \
        --opt_betas 0.9 0.95 \
        --warmup_epochs 30 \
        --save_ckpt_freq 5 \
        --epochs 300 \
        --log_dir ${OUTPUT_DIR} \
        --output_dir ${OUTPUT_DIR} \
        ${PY_ARGS}

```

--------------------------------

### Create VideoMAE Giant Model

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Initializes a pre-trained VideoMAE giant model with specified parameters for fine-tuning. Ensure 'create_model' is imported.

```python
pretrain_model = create_model(
    'pretrain_videomae_giant_patch14_224',
    pretrained=False,
    drop_path_rate=0.0,
    all_frames=16,
    tubelet_size=2,
    decoder_depth=4,         # Number of decoder layers
    with_cp=True,
)
```

--------------------------------

### Load and Prepare Video for Inference

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Loads a pre-trained VideoMAE model, prepares video data with specified transforms, and performs inference to predict class and confidence.

```python
model = create_model(
    'vit_giant_patch14_224',
    pretrained=False,
    num_classes=710,
    all_frames=16,
    tubelet_size=2,
    use_mean_pooling=True,
)

checkpoint = torch.load('vit_g_hybrid_pt_1200e_k710_ft.pth', map_location='cpu')
model.load_state_dict(checkpoint['model'], strict=False)
model.eval()
model.cuda()

transform = transforms.Compose([
    transforms.ToPILImage(),
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                        std=[0.229, 0.224, 0.225])
])

video_path = 'sample_video.mp4'
vr = VideoReader(video_path, ctx=cpu(0))
total_frames = len(vr)

frame_indices = np.arange(0, 64, 4)[:16]  # Sample every 4th frame
frame_indices = np.clip(frame_indices, 0, total_frames - 1)
frames = vr.get_batch(frame_indices).asnumpy()

processed_frames = []
for frame in frames:
    processed_frames.append(transform(frame))
video_tensor = torch.stack(processed_frames)  # [T, C, H, W]
video_tensor = video_tensor.permute(1, 0, 2, 3)  # [C, T, H, W]
video_tensor = video_tensor.unsqueeze(0).cuda()  # [1, C, T, H, W]

with torch.no_grad():
    output = model(video_tensor)
    probabilities = torch.softmax(output, dim=1)
    predicted_class = torch.argmax(probabilities, dim=1).item()
    confidence = probabilities[0, predicted_class].item()

print(f"Predicted class: {predicted_class}, Confidence: {confidence:.4f}")
```

--------------------------------

### Create VideoMAE V2 Models with Timm

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Create VideoMAE V2 models, including fine-tuning and pre-training variants, using the timm library's create_model function. Ensure the 'models' module is imported to register custom model architectures.

```python
from timm.models import create_model
import models  # Register custom models

# Fine-tuning models (VisionTransformer)
# Available: vit_small_patch16_224, vit_base_patch16_224,
#            vit_large_patch16_224, vit_huge_patch16_224, vit_giant_patch14_224

model = create_model(
    'vit_giant_patch14_224',
    img_size=224,
    pretrained=False,
    num_classes=710,
    all_frames=16,           # Number of input frames
    tubelet_size=2,          # Temporal patch size
    drop_rate=0.0,           # Dropout rate
    drop_path_rate=0.3,      # Stochastic depth rate
    attn_drop_rate=0.0,      # Attention dropout
    head_drop_rate=0.0,      # Classification head dropout
    use_mean_pooling=True,   # Use mean pooling (True) or CLS token (False)
    init_scale=0.001,        # Scale for head initialization
    with_cp=True,            # Gradient checkpointing for memory efficiency
)

# Pre-training models (PretrainVisionTransformer with encoder-decoder)
# Available: pretrain_videomae_small_patch16_224, pretrain_videomae_base_patch16_224,
#            pretrain_videomae_large_patch16_224, pretrain_videomae_huge_patch16_224,

```

--------------------------------

### Distributed Training Script for VideoMAEv2

Source: https://github.com/opengvlab/videomaev2/blob/master/docs/FINETUNE.md

This script is a modification of the slurm training script for distributed training. It requires setting the same master_port across all nodes. Configure the number of nodes, GPUs per node, and other training arguments as needed.

```bash
#!/usr/bin/env bash
set -x  # print the commands

export MASTER_PORT=${MASTER_PORT:-12320}  # You should set the same master_port in all the nodes

OUTPUT_DIR='YOUR_PATH/work_dir/vit_g_hybrid_pt_1200e_k710_ft'  # Your output folder for deepspeed config file, logs and checkpoints
DATA_PATH='YOUR_PATH/data/k710'  # The data list folder. the folder has three files: train.csv, val.csv, test.csv
# finetune data list file follows the following format
# for the video data line: video_path, label
# for the rawframe data line: frame_folder_path, total_frames, label
MODEL_PATH='YOUR_PATH/model_zoo/vit_g_hybrid_pt_1200e.pth'  # Model for initializing parameters

N_NODES=${N_NODES:-4}  # Number of nodes
GPUS_PER_NODE=${GPUS_PER_NODE:-8}  # Number of GPUs in each node
SRUN_ARGS=${SRUN_ARGS:-""}  # Other slurm task args
PY_ARGS=${@:3}  # Other training args

```

--------------------------------

### Create and Activate Conda Environment

Source: https://github.com/opengvlab/videomaev2/blob/master/docs/INSTALL.md

Use these commands to create a new conda environment named 'videomae' with Python 3.8 and then activate it. This isolates project dependencies.

```bash
conda create --name videomae python=3.8 -y
conda activate videomae
```

--------------------------------

### Slurm Pre-training Script for VideoMAEv2

Source: https://github.com/opengvlab/videomaev2/blob/master/scripts/pretrain/README.md

Use this script for multi-node Slurm training of VideoMAEv2. Ensure to set the OUTPUT_DIR and DATA_PATH variables. The script configures training parameters like mask type, ratio, model, batch size, and learning rate.

```bash
#!/usr/bin/env bash
set -x  # print the commands

export MASTER_PORT=$((12000 + $RANDOM % 20000))  # Randomly set master_port to avoid port conflicts
export OMP_NUM_THREADS=1  # Control the number of threads

OUTPUT_DIR='YOUR_PATH/work_dir/vit_g_hybrid_pt_1200e'  # Your output folder for deepspeed config file, logs and checkpoints
DATA_PATH='YOUR_PATH/data/hybrid_train.csv'  # The data list file path.
# pretrain data list file follows the following format
# for the video data line: video_path, 0, -1
# for the rawframe data line: frame_folder_path, start_index, total_frames

JOB_NAME=$1  # the job name of the slurm task
PARTITION=${PARTITION:-"video"}  # Name of the partition
# 8 for 1 node, 16 for 2 node, etc.
GPUS=${GPUS:-64}  # Number of GPUs
GPUS_PER_NODE=${GPUS_PER_NODE:-8}  # Number of GPUs in each node
CPUS_PER_TASK=${CPUS_PER_TASK:-14}  # Number of CPU cores allocated, number of tasks equal to the number of GPUs used
SRUN_ARGS=${SRUN_ARGS:-""}  # Other slurm task args
PY_ARGS=${@:2}  # Other training args

# Please refer to `run_mae_pretraining.py` for the meaning of the following hyperreferences
srun -p $PARTITION \
        --job-name=${JOB_NAME} \
        --gres=gpu:${GPUS_PER_NODE} \
        --ntasks=${GPUS} \
        --ntasks-per-node=${GPUS_PER_NODE} \
        --cpus-per-task=${CPUS_PER_TASK} \
        --kill-on-bad-exit=1 \
        --async \
        ${SRUN_ARGS} \
        python -u run_mae_pretraining.py \
        --data_path ${DATA_PATH} \
        --mask_type tube \
        --mask_ratio 0.9 \
        --decoder_mask_type run_cell \
        --decoder_mask_ratio 0.5 \
        --model pretrain_videomae_giant_patch14_224 \
        --decoder_depth 4 \
        --batch_size 32 \
        --with_checkpoint \
        --num_frames 16 \
        --sampling_rate 4 \
        --num_sample 4 \
        --num_workers 10 \
        --opt adamw \
        --lr 6e-4 \
        --clip_grad 0.02 \
        --opt_betas 0.9 0.95 \
        --warmup_epochs 30 \
        --save_ckpt_freq 5 \
        --epochs 300 \
        --log_dir ${OUTPUT_DIR} \
        --output_dir ${OUTPUT_DIR} \
        ${PY_ARGS}

```

--------------------------------

### Evaluate VideoMAE V2 on Kinetics-400

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Evaluate a fine-tuned VideoMAE V2 model on the Kinetics-400 test set. Ensure the data path points to the Kinetics-400 dataset.

```bash
#!/usr/bin/env bash
set -x

export MASTER_PORT=$((12000 + $RANDOM % 20000))
export OMP_NUM_THREADS=1

OUTPUT_DIR='YOUR_PATH/eval_results'
DATA_PATH='YOUR_PATH/data/k400'
MODEL_PATH='YOUR_PATH/model_zoo/vit_g_k710_it_k400_ft.pth'

# Evaluation only (add --eval flag)
srun -p video \
    --job-name=k400_eval \
    --gres=gpu:8 \
    --ntasks=8 \
    --ntasks-per-node=8 \
    --cpus-per-task=12 \
    --kill-on-bad-exit=1 \
    python run_class_finetuning.py \
    --model vit_giant_patch14_224 \
    --data_set Kinetics-400 \
    --nb_classes 400 \
    --data_path ${DATA_PATH} \
    --finetune ${MODEL_PATH} \
    --output_dir ${OUTPUT_DIR} \
    --batch_size 8 \
    --input_size 224 \
    --short_side_size 224 \
    --num_frames 16 \
    --sampling_rate 4 \
    --num_workers 10 \
    --test_num_segment 5 \
    --test_num_crop 3 \
    --dist_eval \
    --eval

# Expected output format:
# Accuracy of the network on the 19796 test videos: Top-1: 88.40%, Top-5: 98.00%
```

--------------------------------

### Load and Convert Pre-trained Checkpoint

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Loads a pre-trained PyTorch checkpoint, handles potential key prefixes like '_orig_mod.' from torch.compile, and converts encoder/backbone prefixes for transfer learning. It also includes logic to convert K710 head weights to K400/K600/K700 using label mapping files.

```python
import torch
import json
from collections import OrderedDict

# Load checkpoint
checkpoint = torch.load('vit_g_hybrid_pt_1200e_k710_ft.pth', map_location='cpu')

# Extract model state dict
checkpoint_model = None
for model_key in ['model', 'module']:
    if model_key in checkpoint:
        checkpoint_model = checkpoint[model_key]
        break
if checkpoint_model is None:
    checkpoint_model = checkpoint

# Handle torch.compile prefix (_orig_mod.)
for old_key in list(checkpoint_model.keys()):
    if old_key.startswith('_orig_mod.'):
        new_key = old_key[10:]
        checkpoint_model[new_key] = checkpoint_model.pop(old_key)

# Convert encoder/backbone prefix for pre-train -> fine-tune transfer
new_dict = OrderedDict()
for key in checkpoint_model.keys():
    if key.startswith('backbone.'):
        new_dict[key[9:]] = checkpoint_model[key]
    elif key.startswith('encoder.'):
        new_dict[key[8:]] = checkpoint_model[key]
    else:
        new_dict[key] = checkpoint_model[key]

# Convert K710 head to K400/K600/K700
# Use label mapping files from misc/ folder
if checkpoint_model['head.weight'].shape[0] == 710:
    # For Kinetics-400
    label_map = json.load(open('misc/label_710to400.json'))
    checkpoint_model['head.weight'] = checkpoint_model['head.weight'][label_map]
    checkpoint_model['head.bias'] = checkpoint_model['head.bias'][label_map]

# Load into model
model.load_state_dict(new_dict, strict=False)
```

--------------------------------

### Fine-tune ViT-base on Kinetics-400 with Slurm

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Fine-tune a ViT-base model on Kinetics-400 using Slurm with optimized hyperparameters. Ensure model and data paths are correctly specified.

```bash
#!/usr/bin/env bash
set -x

export MASTER_PORT=$((12000 + $RANDOM % 20000))
export OMP_NUM_THREADS=1

OUTPUT_DIR='YOUR_PATH/work_dir/vit_b_hybrid_pt_800e_k400_ft'
DATA_PATH='YOUR_PATH/data/k400'
MODEL_PATH='YOUR_PATH/model_zoo/vit_b_hybrid_pt_800e.pth'

srun -p video \
    --job-name=k400_finetune \
    --gres=gpu:8 \
    --ntasks=32 \
    --ntasks-per-node=8 \
    --cpus-per-task=12 \
    --kill-on-bad-exit=1 \
    --async \
    python run_class_finetuning.py \
    --model vit_base_patch16_224 \
    --data_path ${DATA_PATH} \
    --finetune ${MODEL_PATH} \
    --log_dir ${OUTPUT_DIR} \
    --output_dir ${OUTPUT_DIR} \
    --batch_size 16 \
    --input_size 224 \
    --short_side_size 224 \
    --save_ckpt_freq 10 \
    --num_frames 16 \
    --sampling_rate 4 \
    --num_workers 10 \
    --opt adamw \
    --lr 7e-4 \
    --opt_betas 0.9 0.999 \
    --weight_decay 0.05 \
    --layer_decay 0.75 \
    --test_num_segment 5 \
    --test_num_crop 3 \
    --epochs 90 \
    --dist_eval --enable_deepspeed
```

--------------------------------

### Download Full Model Weights

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Request full model weights (e.g., ViT-giant, ViT-huge) by filling out a Google Forms request.

```bash
# Full model weights (ViT-giant, ViT-huge, etc.)
# Fill out the download request form:
# https://docs.google.com/forms/d/e/1FAIpQLSd1SjKMtD8piL9uxGEUwicerxd46bs12QojQt92rzalnoI3JA/viewform
```

--------------------------------

### Fine-tune VideoMAE V2 on Something-Something V2

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Fine-tune a pre-trained VideoMAE V2 model on the SSV2 dataset using raw frames with sparse sampling. Ensure the data path points to a directory containing train.csv and val.csv.

```bash
#!/usr/bin/env bash
set -x

export MASTER_PORT=$((12000 + $RANDOM % 20000))
export OMP_NUM_THREADS=1

OUTPUT_DIR='YOUR_PATH/work_dir/vit_g_hybrid_pt_1200e_ssv2_ft'
DATA_PATH='YOUR_PATH/data/sthv2'  # Contains train.csv, val.csv with raw frame paths
MODEL_PATH='YOUR_PATH/model_zoo/vit_g_hybrid_pt_1200e.pth'

srun -p video \
    --job-name=ssv2_finetune \
    --gres=gpu:8 \
    --ntasks=32 \
    --ntasks-per-node=8 \
    --cpus-per-task=14 \
    --kill-on-bad-exit=1 \
    --async \
    python run_class_finetuning.py \
    --model vit_giant_patch14_224 \
    --data_set SSV2 \
    --nb_classes 174 \
    --data_path ${DATA_PATH} \
    --finetune ${MODEL_PATH} \
    --log_dir ${OUTPUT_DIR} \
    --output_dir ${OUTPUT_DIR} \
    --batch_size 3 \
    --input_size 224 \
    --short_side_size 224 \
    --save_ckpt_freq 10 \
    --num_frames 16 \
    --sampling_rate 1 \
    --num_sample 2 \
    --num_workers 10 \
    --opt adamw \
    --lr 1e-3 \
    --drop_path 0.3 \
    --clip_grad 5.0 \
    --layer_decay 0.9 \
    --opt_betas 0.9 0.999 \
    --weight_decay 0.1 \
    --warmup_epochs 5 \
    --epochs 50 \
    --test_num_segment 2 \
    --test_num_crop 3 \
    --dist_eval --enable_deepspeed \
    --sparse_sample

# Note: SSV2 uses RawFrameClsDataset with sparse_sample=True
# Frame filename template can be set with --fname_tmpl 'img_{:05}.jpg'
```

--------------------------------

### Pre-training Data List Format

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Defines the format for pre-training data lists, supporting both video files and raw frame directories.

```python
# Pre-training data list format examples:

# Video data line format: video_path, start_index, total_frames
# For video files, use: video_path 0 -1
your_path/k400/---QUuC4vJs.mp4 0 -1
your_path/k700/-0H3T2B9PH4_000025_000035.mp4 0 -1
your_path/webvid2m/016401_016450/1017127174.mp4 0 -1

# Raw frames line format: frame_folder_path start_index total_frames
your_path/AVA/frames/clip/zlVkeKC6Ha8 9601 300
your_path/SomethingV2/frames/182040 1 58

# Mixed sources in UnlabeledHybrid dataset:
# - Kinetics-400/600/700 videos
# - WebVid2M videos
# - AVA raw frames
# - Something-Something V2 raw frames
```

--------------------------------

### Fine-tune VideoMAE V2 on Kinetics-710 with Slurm

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Fine-tune a pre-trained VideoMAE V2 model on Kinetics-710 using Slurm. Ensure the model path and dataset path are correctly set.

```bash
#!/usr/bin/env bash
set -x

export MASTER_PORT=$((12000 + $RANDOM % 20000))
export OMP_NUM_THREADS=1

OUTPUT_DIR='YOUR_PATH/work_dir/vit_g_hybrid_pt_1200e_k710_ft'
DATA_PATH='YOUR_PATH/data/k710'  # Contains train.csv, val.csv
MODEL_PATH='YOUR_PATH/model_zoo/vit_g_hybrid_pt_1200e.pth'

# Slurm training (32 GPUs = 4 nodes x 8 GPUs)
srun -p video \
    --job-name=k710_finetune \
    --gres=gpu:8 \
    --ntasks=32 \
    --ntasks-per-node=8 \
    --cpus-per-task=14 \
    --kill-on-bad-exit=1 \
    --async \
    python run_class_finetuning.py \
    --model vit_giant_patch14_224 \
    --data_set Kinetics-710 \
    --nb_classes 710 \
    --data_path ${DATA_PATH} \
    --finetune ${MODEL_PATH} \
    --log_dir ${OUTPUT_DIR} \
    --output_dir ${OUTPUT_DIR} \
    --batch_size 3 \
    --input_size 224 \
    --short_side_size 224 \
    --save_ckpt_freq 10 \
    --num_frames 16 \
    --sampling_rate 4 \
    --num_sample 2 \
    --num_workers 10 \
    --opt adamw \
    --lr 1e-3 \
    --drop_path 0.3 \
    --clip_grad 5.0 \
    --layer_decay 0.9 \
    --opt_betas 0.9 0.999 \
    --weight_decay 0.1 \
    --warmup_epochs 5 \
    --epochs 35 \
    --test_num_segment 5 \
    --test_num_crop 3 \
    --dist_eval --enable_deepspeed
```

--------------------------------

### Download Distilled Model Weights

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Download distilled model weights directly from Hugging Face using wget. These models are distilled from larger ViT-giant models.

```bash
# Distilled models (direct download from Hugging Face)
# ViT-small distilled from ViT-giant
wget https://huggingface.co/OpenGVLab/VideoMAE2/resolve/main/distill/vit_s_k710_dl_from_giant.pth

# ViT-base distilled from ViT-giant
wget https://huggingface.co/OpenGVLab/VideoMAE2/resolve/main/distill/vit_b_k710_dl_from_giant.pth
```

--------------------------------

### Download Pre-extracted TAD Features

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Download pre-extracted Temporal Action Detection (TAD) features for THUMOS14 and FineAction datasets using wget.

```bash
# Pre-extracted TAD features
# THUMOS14 features
wget https://huggingface.co/datasets/OpenGVLab/VideoMAEv2-TAL-Features/resolve/main/th14_mae_g_16_4.tar.gz

# FineAction features
wget https://huggingface.co/datasets/OpenGVLab/VideoMAEv2-TAL-Features/resolve/main/fineaction_mae_g.tar.gz
```

--------------------------------

### Slurm Training Script for VideoMAEv2

Source: https://github.com/opengvlab/videomaev2/blob/master/docs/FINETUNE.md

Use this script to fine-tune VideoMAEv2 ViT-giant on Kinetics-710 with multi-node slurm training. Ensure to set your output directory, data path, and model path correctly. This script is designed for multi-node slurm training environments.

```bash
#!/usr/bin/env bash
set -x  # print the commands

export MASTER_PORT=$((12000 + $RANDOM % 20000))  # Randomly set master_port to avoid port conflicts
export OMP_NUM_THREADS=1  # Control the number of threads

OUTPUT_DIR='YOUR_PATH/work_dir/vit_g_hybrid_pt_1200e_k710_ft'  # Your output folder for deepspeed config file, logs and checkpoints
DATA_PATH='YOUR_PATH/data/k710'  # The data list folder. the folder has three files: train.csv, val.csv, test.csv
# finetune data list file follows the following format
# for the video data line: video_path, label
# for the rawframe data line: frame_folder_path, total_frames, label
MODEL_PATH='YOUR_PATH/model_zoo/vit_g_hybrid_pt_1200e.pth'  # Model for initializing parameters

JOB_NAME=$1  # the job name of the slurm task
PARTITION=${PARTITION:-"video"}  # Name of the partition
# 8 for 1 node, 16 for 2 node, etc.
GPUS=${GPUS:-32}  # Number of GPUs
GPUS_PER_NODE=${GPUS_PER_NODE:-8}  # Number of GPUs in each node
CPUS_PER_TASK=${CPUS_PER_TASK:-14}  # Number of CPU cores allocated, number of tasks equal to the number of GPUs used
SRUN_ARGS=${SRUN_ARGS:-""}  # Other slurm task args
PY_ARGS=${@:2}  # Other training args

# Please refer to `run_class_finetuning.py` for the meaning of the following hyperreferences
srun -p $PARTITION \
        --job-name=${JOB_NAME} \
        --gres=gpu:${GPUS_PER_NODE} \
        --ntasks=${GPUS} \
        --ntasks-per-node=${GPUS_PER_NODE} \
        --cpus-per-task=${CPUS_PER_TASK} \
        --kill-on-bad-exit=1 \
        --async \
        ${SRUN_ARGS} \
        python run_class_finetuning.py \
        --model vit_giant_patch14_224 \
        --data_set Kinetics-710 \
        --nb_classes 710 \
        --data_path ${DATA_PATH} \
        --finetune ${MODEL_PATH} \
        --log_dir ${OUTPUT_DIR} \
        --output_dir ${OUTPUT_DIR} \
        --batch_size 3 \
        --input_size 224 \
        --short_side_size 224 \
        --save_ckpt_freq 10 \
        --num_frames 16 \
        --sampling_rate 4 \
        --num_sample 2 \
        --num_workers 10 \
        --opt adamw \
        --lr 1e-3 \
        --drop_path 0.3 \
        --clip_grad 5.0 \
        --layer_decay 0.9 \
        --opt_betas 0.9 0.999 \
        --weight_decay 0.1 \
        --warmup_epochs 5 \
        --epochs 35 \
        --test_num_segment 5 \
        --test_num_crop 3 \
        --dist_eval --enable_deepspeed \
        ${PY_ARGS}
```

```bash
bash script/finetune/vit_g_k710_ft.sh k710_finetune
```

```bash
bash script/finetune/vit_g_k710_ft.sh k710_model_test --eval
```

--------------------------------

### Direct VideoClsDataset Usage

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Directly instantiates VideoClsDataset for video classification tasks. Configure parameters like clip length, frame sampling rate, and cropping settings for training or testing.

```python
video_dataset = VideoClsDataset(
    anno_path='/path/to/train.csv',
    data_root='/path/to/videos',
    mode='train',                    # 'train', 'validation', or 'test'
    clip_len=16,                     # Number of frames per clip
    frame_sample_rate=4,             # Frame sampling rate
    num_segment=1,                   # Number of segments for training
    test_num_segment=5,              # Number of segments for testing
    test_num_crop=3,                 # Number of crops for testing
    num_crop=1,                      # Number of crops for training
    keep_aspect_ratio=True,
    crop_size=224,
    short_side_size=224,
    new_height=256,
    new_width=320,
    sparse_sample=False,
    args=args
)
```

--------------------------------

### Fine-tuning Data List Format

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Specifies the format for fine-tuning data lists, including video paths and class labels for both video and raw frame data.

```python
# Fine-tuning data list format

# VideoClsDataset (for video data): video_path label
# Example: Kinetics-400 validation list
your_path/k400/jf7RDuUTrsQ.mp4 325
your_path/k400/JTlatknwOrY.mp4 233
your_path/k400/NUG7kwJ-614.mp4 103
your_path/k400/y9r115bgfNk.mp4 320

# RawFrameClsDataset (for raw frames): frame_folder_path total_frames label
# Example: Something-Something V2 validation list
your_path/SomethingV2/frames/74225 62 140
your_path/SomethingV2/frames/116154 51 127
your_path/SomethingV2/frames/198186 47 173

# Directory structure for fine-tuning:
# data_path/
#   train.csv    # Training annotations
#   val.csv      # Validation annotations
#   test.csv     # Test annotations (optional, uses val.csv if missing)
```

--------------------------------

### Extract Features for THUMOS14

Source: https://github.com/opengvlab/videomaev2/blob/master/docs/TAD.md

Use this command to extract features for the THUMOS14 dataset using VideoMAE V2-g. Ensure you replace YOUR_PATH with the actual directory paths.

```bash
python extract_tad_feature.py \
    --data_set THUMOS14 \
    --data_path YOUR_PATH/thumos14_videos \
    --save_path YOUR_PATH/th14_vit_g_16_4 \
    --model vit_giant_patch14_224 \
    --ckpt_path YOUR_PATH/vit_g_hyrbid_pt_1200e_k710_ft.pth
```

--------------------------------

### RawFrameClsDataset Usage

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Instantiates RawFrameClsDataset for classification tasks using raw video frames. Specify parameters for frame sampling, number of segments, and cropping, along with file naming conventions.

```python
frame_dataset = RawFrameClsDataset(
    anno_path='/path/to/sthv2/train.csv',
    data_root='/path/to/frames',
    mode='train',
    clip_len=1,
    num_segment=16,                  # Number of frames to sample
    test_num_segment=2,
    test_num_crop=3,
    num_crop=1,
    keep_aspect_ratio=True,
    crop_size=224,
    short_side_size=224,
    new_height=256,
    new_width=320,
    filename_tmpl='img_{:05}.jpg',
    start_idx=1,
    args=args
)
```

--------------------------------

### Extract Features for Temporal Action Detection

Source: https://context7.com/opengvlab/videomaev2/llms.txt

Extract video features for Temporal Action Detection (TAD) tasks using a fine-tuned VideoMAE V2 model. Features are saved as .npy files.

```python
# extract_tad_feature.py usage

# Extract features for THUMOS14 dataset
python extract_tad_feature.py \
    --data_set THUMOS14 \
    --data_path YOUR_PATH/thumos14_videos \
    --save_path YOUR_PATH/th14_vit_g_16_4 \
    --model vit_giant_patch14_224 \
    --ckpt_path YOUR_PATH/vit_g_hybrid_pt_1200e_k710_ft.pth

# Extract features for FineAction dataset
python extract_tad_feature.py \
    --data_set FINEACTION \
    --data_path YOUR_PATH/fineaction_videos \
    --save_path YOUR_PATH/fineaction_vit_g \
    --model vit_giant_patch14_224 \
    --ckpt_path YOUR_PATH/vit_g_hybrid_pt_1200e_k710_ft.pth

# Output: .npy files with shape [N, C] where N=number of clips, C=feature_dim
# Features are extracted using model.forward_features() (before classification head)
# THUMOS14: sliding window with stride 4, FineAction: stride 16
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.