### Clone Repository and Navigate

Source: https://github.com/microsoft/vidtok/blob/main/README.md

Clone the VidTok repository and change the directory to the VidTok folder to begin setup.

```bash
git clone https://github.com/microsoft/VidTok
cd VidTok
```

--------------------------------

### Start Training with Weights & Biases

Source: https://github.com/microsoft/vidtok/blob/main/README.md

Run the training script with integrated Weights & Biases logging.

```bash
python main.py -b CONFIG --logdir LOGDIR --wandb --wandb_entity ENTITY --wandb_project PROJECT
```

--------------------------------

### Start Training Command

Source: https://github.com/microsoft/vidtok/blob/main/README.md

Execute the training script using the specified configuration file and log directory.

```bash
python main.py -b CONFIG --logdir LOGDIR

# You can also use `torchrun` to start the training code.
```

--------------------------------

### Set Up Conda Environment

Source: https://github.com/microsoft/vidtok/blob/main/README.md

Create and activate a Conda environment using the provided environment.yaml file. Ensure Conda is installed.

```bash
# 1. Prepare conda environment
conda env create -f environment.yaml
# 2. Activate the environment
conda activate vidtok
```

--------------------------------

### Install Additional Dependencies

Source: https://github.com/microsoft/vidtok/blob/main/vidtwin/README.md

Install the necessary Python packages required specifically for the VidTwin model.

```bash
pip install tranformers
pip install timm
pip install flash-attn --no-build-isolation
```

--------------------------------

### VidTok Data Preparation Example

Source: https://context7.com/microsoft/vidtok/llms.txt

Illustrates the expected directory structure for VidTok data preparation, showing how video files should be organized within subset directories.

```python
# Example directory structure:
# DATA_DIR/
#   subset1/
#     video1.mp4
#     video2.mp4
#   subset2/
#     video3.mp4

```

--------------------------------

### Setup Conda Environment for VidTok

Source: https://github.com/microsoft/vidtok/blob/main/vidtwin/README.md

Navigate to the VidTok directory and initialize the environment using the provided YAML file.

```bash
cd VidTok
# Prepare conda environment
conda env create -f environment.yaml
# Activate the environment
conda activate vidtok
```

--------------------------------

### VidTok Configuration File Structure

Source: https://context7.com/microsoft/vidtok/llms.txt

An example YAML configuration file for VidTok, detailing model parameters, data loading settings, and training configurations. This file is crucial for defining model architecture and training behavior.

```yaml
# Example: configs/vidtok_kl_causal_488_4chn.yaml
model:
  base_learning_rate: 1e-5
  target: vidtok.models.autoencoder.AutoencodingEngine
  params:
    monitor: val/rec_loss
    mode: min
    ckpt_path: checkpoints/vidtok_kl_causal_488_4chn.ckpt  # Fine-tune from checkpoint

    encoder_config:
      target: vidtok.modules.model_3dcausal.EncoderCausal3DPadding
      params:
        double_z: true
        z_channels: 4           # Latent channels
        in_channels: 3
        ch: 128                 # Base channel width
        ch_mult: [1, 2, 4, 4]   # Channel multipliers per stage
        time_downsample_factor: 4
        num_res_blocks: 2
        fix_encoder: false      # Set true to freeze encoder
        fix_decoder: false      # Set true to freeze decoder

    decoder_config:
      target: vidtok.modules.model_3dcausal.DecoderCausal3DPadding
      params: ${model.params.encoder_config.params}

    regularizer_config:
      target: vidtok.modules.regularizers.DiagonalGaussianRegularizer  # KL
      # For FSQ: target: vidtok.modules.regularizers.FSQRegularizer

data:
  params:
    batch_size: 2
    num_workers: 12
    train:
      target: vidtok.data.vidtok.VidTokDataset
      params:
        data_dir: /path/to/train/videos
        meta_path: /path/to/train.csv
        video_params:
          input_height: 256
          input_width: 256
          sample_num_frames: 17   # 17 for causal, 16 for non-causal
          sample_fps: 30

lightning:
  trainer:
    precision: bf16-mixed
    devices: auto
    max_epochs: 1000
```

--------------------------------

### Launch Training Commands

Source: https://github.com/microsoft/vidtok/blob/main/vidtwin/README.md

Commands to initiate training, including optional Weights & Biases integration.

```bash
python main.py -b CONFIG --logdir LOGDIR

# You can also use `torchrun` to start the training code.
```

```bash
python main.py -b CONFIG --logdir LOGDIR --wandb --wandb_entity ENTITY --wandb_project PROJECT
```

--------------------------------

### Configure First Stage Training

Source: https://github.com/microsoft/vidtok/blob/main/README.md

Set the configuration to enable full model training from scratch with low-resolution data.

```yaml
model:
  params:
    # ckpt_path:  # disable this parameter so as to train from scratch
    encoder_config:
      params:
        fix_encoder: false
        fix_decoder: false
data:
  params:
    train:
      params:
        video_params:
            input_height: 128
            input_width: 128
```

--------------------------------

### Data Directory Structure for Training

Source: https://github.com/microsoft/vidtok/blob/main/README.md

Organize your training videos under a main directory. Subdirectories can be used to group videos.

```bash
└── DATA_DIR
    ├── subset1
    │   ├── videoname11.mp4
    │   └── videoname12.mp4
    ├── subset2
    │   ├── videoname21.mp4
    │   ├── videoname22.mp4
    │   └── subsubset1
    │       ├── videoname211.mp4
    │       └── videoname212.mp4
    └── ...

```

--------------------------------

### Initialize VidTokDataset and DataLoader

Source: https://context7.com/microsoft/vidtok/llms.txt

Configures the dataset with video parameters and wraps it in a PyTorch DataLoader for batch processing.

```python
from vidtok.data.vidtok import VidTokDataset
import torch

dataset = VidTokDataset(
    data_dir="/path/to/videos",
    meta_path="/path/to/train.csv",
    video_params={
        "input_height": 256,
        "input_width": 256,
        "sample_num_frames": 17,
        "sample_fps": 30
    },
    data_frac=1.0,  # Use full dataset
    skip_missing_files=True
)

dataloader = torch.utils.data.DataLoader(dataset, batch_size=4, shuffle=True)

for batch in dataloader:
    video_tensor = batch['jpg']  # Shape: [B, C, T, H, W], range [-1, 1]
    video_paths = batch['path']
    print(f"Batch shape: {video_tensor.shape}")
    break
```

--------------------------------

### Define Data Directory Structure

Source: https://github.com/microsoft/vidtok/blob/main/vidtwin/README.md

Organize training videos within a root data directory.

```text
└── DATA_DIR
    ├── subset1
    │   ├── videoname11.mp4
    │   └── videoname12.mp4
    ├── subset2
    │   ├── videoname21.mp4
    │   ├── videoname22.mp4
    │   └── subsubset1
    │       ├── videoname211.mp4
    │       └── videoname212.mp4
    └── ...
```

--------------------------------

### Multi-GPU Training with Torchrun

Source: https://context7.com/microsoft/vidtok/llms.txt

This command demonstrates multi-GPU training using torchrun. Adjust `nproc_per_node` based on your available GPUs.

```bash
torchrun --nproc_per_node=4 main.py \
    -b configs/vidtok_kl_causal_488_4chn.yaml \
    --logdir logs/my_experiment
```

--------------------------------

### Evaluate Specific Videos with Meta File

Source: https://context7.com/microsoft/vidtok/llms.txt

Use this command to evaluate specific videos using a meta file for detailed analysis. Ensure all paths and parameters are correctly set for your dataset.

```bash
python scripts/inference_evaluate.py \
    --config configs/vidtok_kl_causal_488_4chn.yaml \
    --ckpt checkpoints/vidtok_kl_causal_488_4chn.ckpt \
    --data_dir /path/to/test/videos \
    --meta_path /path/to/test_videos.csv \
    --input_height 256 \
    --input_width 256 \
    --sample_fps 30
```

--------------------------------

### Load and Evaluate VidTwin Model

Source: https://context7.com/microsoft/vidtok/llms.txt

This Python script demonstrates how to load a VidTwin model from configuration and checkpoint files, and perform a reconstruction task. It includes device handling for CUDA or CPU.

```python
import torch
from scripts.inference_evaluate import load_model_from_config

# Load VidTwin model (CVPR 2025)
cfg_path = "configs/vidtwin/vidtwin_structure_7_7_8_dynamics_7_8.yaml"
ckpt_path = "checkpoints/vidtwin_structure_7_7_8_dynamics_7_8.ckpt"

device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
model = load_model_from_config(cfg_path, ckpt_path)
model.to(device).eval()

# VidTwin uses 16 frames at 224x224 resolution
x_input = (torch.rand(1, 3, 16, 224, 224) * 2 - 1).to(device)

with torch.no_grad():
    _, x_recon, *_ = model(x_input)

assert x_input.shape == x_recon.shape
print(f"VidTwin reconstruction shape: {x_recon.shape}")
```

--------------------------------

### Create CSV Meta File

Source: https://github.com/microsoft/vidtok/blob/main/vidtwin/README.md

Format the metadata file to list relative paths of videos for the dataloader.

```text
videos
subset1/videoname11.mp4
subset2/videoname21.mp4
subset2/subsubset1/videoname211.mp4
```

--------------------------------

### Configure Second Stage Training

Source: https://github.com/microsoft/vidtok/blob/main/README.md

Set the configuration to fine-tune the decoder with high-resolution data using a pre-trained checkpoint.

```yaml
model:
  params:
    ckpt_path: CKPT_PATH  # path to the saved checkpoint after the first stage of training
    encoder_config:
      params:
        fix_encoder: true
        fix_decoder: false
data:
  params:
    train:
      params:
        video_params:
            input_height: 256
            input_width: 256
```

--------------------------------

### CSV Metafile for Video Paths

Source: https://github.com/microsoft/vidtok/blob/main/README.md

Create a CSV file listing the relative paths of videos with respect to the DATA_DIR. This file is used to reference training and validation data.

```csv
videos
subset1/videoname11.mp4
subset2/videoname21.mp4
subset2/subsubset1/videoname211.mp4

```

--------------------------------

### Run Docker Container for NVIDIA GPUs

Source: https://github.com/microsoft/vidtok/blob/main/README.md

Execute a Docker container with all necessary dependencies for NVIDIA GPUs. Mounts the current directory and sets the working directory.

```bash
# NVIDIA GPUs
docker run -it --gpus all --shm-size 256G --rm -v `pwd`:/workspace --workdir /workspace \
    deeptimhe/ubuntu22.04-cuda12.1-python3.10-pytorch2.5:orig-vidtok bash
```

--------------------------------

### Configure VidTok Dataset Parameters

Source: https://github.com/microsoft/vidtok/blob/main/README.md

Define training and validation dataset paths, video dimensions, and sampling parameters in the YAML configuration file.

```yaml
    train:
        target: vidtok.data.vidtok.VidTokDataset
        params:
          data_dir: DATA_DIR_1  # DATA_DIR for training data
          meta_path: META_PATH_1  # path to the .csv meta file of training data
          video_params:
            input_height: INPUT_HEIGHT_1  # vary in different training stages
            input_width: INPUT_WIDTH_1  # vary in different training stages
            sample_num_frames: NUM_FRAMES_1  # typically set to 17 for causal models and 16 for non-causal models
            sample_fps: SAMPLE_FPS_1  # sample fps for training data
    validation:
        target: vidtok.data.vidtok.VidTokDataset
        params:
          data_dir: DATA_DIR_2  # DATA_DIR for validation data
          meta_path: META_PATH_2  # path to the .csv meta file of validation data
          video_params: 
            input_height: INPUT_HEIGHT_2
            input_width: INPUT_WIDTH_2
            sample_num_frames: NUM_FRAMES_2  # typically set to 17 for causal models and 16 for non-causal models
            sample_fps: SAMPLE_FPS_2  # sample fps for validation data
          start_index: 0  # fixed value to ensure the same sampled data
```

--------------------------------

### Reconstruct Video Files via CLI

Source: https://context7.com/microsoft/vidtok/llms.txt

Command-line interface for video reconstruction, supporting causal padding and long video processing.

```bash
# Basic reconstruction
python scripts/inference_reconstruct.py \
    --config configs/vidtok_kl_causal_488_4chn.yaml \
    --ckpt checkpoints/vidtok_kl_causal_488_4chn.ckpt \
    --input_video_path assets/example.mp4 \
    --input_height 256 \
    --input_width 256 \
    --sample_fps 30 \
    --output_video_dir ./output

# For causal models, add --pad_gen_frames for smoother output
python scripts/inference_reconstruct.py \
    --config configs/vidtok_kl_causal_488_4chn.yaml \
    --ckpt checkpoints/vidtok_kl_causal_488_4chn.ckpt \
    --input_video_path assets/example.mp4 \
    --input_height 256 \
    --input_width 256 \
    --sample_fps 30 \
    --output_video_dir ./output \
    --pad_gen_frames

# Long video reconstruction (v1.1 models)
python scripts/inference_reconstruct.py \
    --config configs/vidtok_v1_1/vidtok_kl_causal_488_4chn_v1_1.yaml \
    --ckpt checkpoints/vidtok_v1_1/vidtok_kl_causal_488_4chn_v1_1.ckpt \
    --input_video_path assets/example.mp4 \
    --input_height 256 \
    --input_width 256 \
    --sample_fps 30 \
    --chunk_size 16 \
    --output_video_dir ./output \
    --read_long_video
```

--------------------------------

### Evaluate Reconstruction Quality

Source: https://context7.com/microsoft/vidtok/llms.txt

Computes PSNR, SSIM, and LPIPS metrics for a directory of test videos.

```bash
# Evaluate on all videos in a directory
python scripts/inference_evaluate.py \
    --config configs/vidtok_kl_causal_488_4chn.yaml \
    --ckpt checkpoints/vidtok_kl_causal_488_4chn.ckpt \
    --data_dir /path/to/test/videos \
    --input_height 256 \
    --input_width 256 \
    --sample_fps 30
```

--------------------------------

### Train VidTok with Weights & Biases Logging

Source: https://context7.com/microsoft/vidtok/llms.txt

This command enables training with Weights & Biases (WandB) for experiment tracking. Configure your entity and project name for effective logging.

```bash
python main.py -b configs/vidtok_kl_causal_488_4chn.yaml \
    --logdir logs/my_experiment \
    --wandb \
    --wandb_entity your_entity \
    --wandb_project vidtok
```

--------------------------------

### Evaluate Reconstruction Performance with VidTok v1.1

Source: https://github.com/microsoft/vidtok/blob/main/README.md

This script evaluates the reconstruction performance of VidTok v1.1 models. Similar to reconstruction, CHUNK_SIZE should be set based on GPU memory, with 16 being a common recommendation. The `--read_long_video` flag is essential for long video evaluation.

```bash
python scripts/inference_evaluate.py --config CONFIG_v1_1 --ckpt CKPT_v1_1 --data_dir DATA_DIR --input_height 256 --input_width 256 --sample_fps 30 --chunk_size CHUNK_SIZE --read_long_video
# Set `CHUNK_SIZE` according to your GPU memory, recommendly 16.
```

--------------------------------

### Resume VidTok Training from Checkpoint

Source: https://context7.com/microsoft/vidtok/llms.txt

Use this command to resume training from a previously saved checkpoint. Specify the log directory containing the experiment state.

```bash
python main.py -r logs/my_experiment
```

--------------------------------

### Evaluate Reconstruction Performance

Source: https://github.com/microsoft/vidtok/blob/main/README.md

Runs performance evaluation metrics including PSNR, SSIM, and LPIPS on videos within a directory.

```bash
python scripts/inference_evaluate.py --config CONFIG --ckpt CKPT --data_dir DATA_DIR --input_height 256 --input_width 256 --sample_fps 30
```

--------------------------------

### Evaluate Long Videos with v1.1 Models

Source: https://context7.com/microsoft/vidtok/llms.txt

This command is for evaluating long videos using v1.1 models. It includes parameters for chunking to handle longer sequences efficiently.

```bash
python scripts/inference_evaluate.py \
    --config configs/vidtok_v1_1/vidtok_kl_causal_488_16chn_v1_1.yaml \
    --ckpt checkpoints/vidtok_v1_1/vidtok_kl_causal_488_16chn_v1_1.ckpt \
    --data_dir /path/to/test/videos \
    --input_height 256 \
    --input_width 256 \
    --sample_fps 30 \
    --chunk_size 16 \
    --read_long_video
```

--------------------------------

### Optimize Inference with Torch Compile

Source: https://github.com/microsoft/vidtok/blob/main/README.md

Uses torch.compile on model components to accelerate inference. Requires pre-trained model configuration and checkpoint files.

```python
import torch
from scripts.inference_evaluate import load_model_from_config

torch._inductor.config.cpp.weight_prepack=True
torch._inductor.config.freezing=True

cfg_path = "configs/vidtok_kl_causal_488_4chn.yaml"
ckpt_path = "checkpoints/vidtok_kl_causal_488_4chn.ckpt"

# load pre-trained model
model = load_model_from_config(cfg_path, ckpt_path)
model.to('cuda').eval()
model.encoder = torch.compile(model.encoder)
model.decoder = torch.compile(model.decoder)

# random input
num_frames = 17 if model.is_causal else 16
x_input = (torch.rand(1, 3, num_frames, 256, 256) * 2 - 1).to('cuda')  # [B,C,T,H,W], range -1~1

# Warm Up
with torch.no_grad(), torch.autocast(device_type='cuda', dtype=torch.float16):
    _, x_recon, _ = model(x_input)

torch.cuda.synchronize()
import time
start = time.time()
with torch.no_grad(), torch.autocast(device_type='cuda', dtype=torch.float16):
    for i in range(10):
        _, x_recon, _ = model(x_input) 
torch.cuda.synchronize()
print(f"Average inference time: {(time.time() - start)/10 :.4f} seconds")
```

--------------------------------

### Fine-tune VidTok Model

Source: https://context7.com/microsoft/vidtok/llms.txt

Use this command to fine-tune a VidTok model from a checkpoint. Specify the configuration file and the log directory for experiment tracking.

```bash
python main.py -b configs/vidtok_kl_causal_488_4chn.yaml --logdir logs/my_experiment
```

--------------------------------

### Configure Checkpoint Path

Source: https://github.com/microsoft/vidtok/blob/main/vidtwin/README.md

Specify an existing checkpoint path to enable fine-tuning.

```yaml
model:
  params:
    ckpt_path: PATH_TO_CHECKPOINT  # train from existing checkpoint
```

--------------------------------

### Evaluate Video Reconstruction Performance

Source: https://github.com/microsoft/vidtok/blob/main/vidtwin/README.md

Run the inference evaluation script to calculate PSNR, SSIM, and LPIPS metrics for videos in the specified directory.

```bash
python vidtwin/scripts/inference_evaluate.py --config CONFIG --ckpt CKPT --data_dir DATA_DIR --num_frames_per_batch NUM_FRAMES_PER_BATCH --input_height 224 --input_width 224 --sample_fps 25
```

--------------------------------

### Reconstruct Input Video

Source: https://github.com/microsoft/vidtok/blob/main/vidtwin/README.md

Run the reconstruction script on a specific video file.

```bash
python vidtwin/scripts/inference_reconstruct.py --config CONFIG --ckpt CKPT --input_video_path VIDEO_PATH --num_frames_per_batch NUM_FRAMES_PER_BATCH --input_height 224 --input_width 224 --sample_fps 25 --output_video_dir OUTPUT_DIR
```

--------------------------------

### Configure VidTok v1.1 for Long Video Inference

Source: https://github.com/microsoft/vidtok/blob/main/README.md

This Python snippet demonstrates how to configure VidTok v1.1 models for inference, specifically enabling tiling for memory efficiency and setting chunk sizes for encoding and decoding. It also shows how to handle random input for long video processing and adjust the reconstructed output shape.

```python
# Use VidTok v1.1 models
cfg_path = "configs/vidtok_v1_1/vidtok_kl_causal_488_4chn_v1_1.yaml"
ckpt_path = "checkpoints/vidtok_v1_1/vidtok_kl_causal_488_4chn_v1_1.ckpt"

... 

model.to('cuda').eval()
# Using tiling inference to save memory usage
model.use_tiling = True
model.t_chunk_enc = 16
model.t_chunk_dec = model.t_chunk_enc // model.encoder.time_downsample_factor
model.use_overlap = True
# random input: long video
x_input = (torch.rand(1, 3, 129, 256, 256) * 2 - 1).to('cuda') 

... 

if x_recon.shape[2] != x_input.shape[2]:
    x_recon = x_recon[:, :, -x_input.shape[2]:, ...]

```

--------------------------------

### Configure Training and Validation Data

Source: https://github.com/microsoft/vidtok/blob/main/vidtwin/README.md

Define dataset paths and video processing parameters for training and validation splits.

```yaml
train:
    target: vidtok.data.vidtok.VidTokDataset
    params:
      data_dir: DATA_DIR_1  # DATA_DIR for training data
      meta_path: META_PATH_1  # path to the .csv meta file of training data
      video_params:
        input_height: INPUT_HEIGHT_1 # 224 for our VidTwin model
        input_width: INPUT_WIDTH_1 # 224 for our VidTwin model
        sample_num_frames: NUM_FRAMES_1  # set to 16 for our VidTwin model
        sample_fps: SAMPLE_FPS_1  # sample fps for training data, 8 for VidTwin model
validation:
    target: vidtok.data.vidtok.VidTokDataset
    params:
      data_dir: DATA_DIR_2  # DATA_DIR for validation data
      meta_path: META_PATH_2  # path to the .csv meta file of validation data
      video_params: 
        input_height: INPUT_HEIGHT_2 # 224 for our VidTwin model
        input_width: INPUT_WIDTH_2 # 224 for our VidTwin model
        sample_num_frames: NUM_FRAMES_2  # set to 16 for our VidTwin model
        sample_fps: SAMPLE_FPS_2  # sample fps for validation data
      start_index: 0  # fixed value to ensure the same sampled data
```

--------------------------------

### Load VidTok Model from Config

Source: https://context7.com/microsoft/vidtok/llms.txt

Loads a pre-trained VidTok model using its configuration and checkpoint files. Ensure the model is moved to the appropriate device (e.g., 'cuda') and set to evaluation mode.

```python
import torch
from scripts.inference_evaluate import load_model_from_config

# Load a pre-trained KL-regularized causal model (4x8x8 compression, 4 channels)
cfg_path = "configs/vidtok_kl_causal_488_4chn.yaml"
ckpt_path = "checkpoints/vidtok_kl_causal_488_4chn.ckpt"

# Load the model
model = load_model_from_config(cfg_path, ckpt_path)
model.to('cuda').eval()

# Check if model is causal (affects input frame count)
print(f"Is causal model: {model.is_causal}")
# Output: Is causal model: True
```

--------------------------------

### Run Docker Container for AMD GPUs

Source: https://github.com/microsoft/vidtok/blob/main/README.md

Execute a Docker container with all necessary dependencies for AMD GPUs using ROCm. Mounts the current directory and sets the working directory.

```bash
# AMD GPUs
docker run -it --gpus all --shm-size 256G --rm -v `pwd`:/workspace --workdir /workspace \
    deeptimhe/ubuntu22.04-rocm6.2.4-python3.10-pytorch2.5:orig-vidtok bash
```

--------------------------------

### Optimize Inference with Torch Compile

Source: https://context7.com/microsoft/vidtok/llms.txt

Accelerates inference by compiling encoder and decoder modules. Requires a warm-up pass before benchmarking.

```python
import torch
from scripts.inference_evaluate import load_model_from_config

# Enable optimizations
torch._inductor.config.cpp.weight_prepack = True
torch._inductor.config.freezing = True

model = load_model_from_config(
    "configs/vidtok_kl_causal_488_4chn.yaml",
    "checkpoints/vidtok_kl_causal_488_4chn.ckpt"
)
model.to('cuda').eval()

# Compile encoder and decoder
model.encoder = torch.compile(model.encoder)
model.decoder = torch.compile(model.decoder)

x_input = (torch.rand(1, 3, 17, 256, 256) * 2 - 1).to('cuda')

# Warm up (first run triggers compilation)
with torch.no_grad(), torch.autocast(device_type='cuda', dtype=torch.float16):
    _, x_recon, _ = model(x_input)

# Benchmark
torch.cuda.synchronize()
import time
start = time.time()
with torch.no_grad(), torch.autocast(device_type='cuda', dtype=torch.float16):
    for _ in range(10):
        _, x_recon, _ = model(x_input)
torch.cuda.synchronize()
print(f"Average inference time: {(time.time() - start) / 10:.4f} seconds")
```

--------------------------------

### Reconstruct Long Video with VidTok v1.1

Source: https://github.com/microsoft/vidtok/blob/main/README.md

Use this script for reconstructing an input video with VidTok v1.1 models. Adjust CHUNK_SIZE based on GPU memory; 16 is recommended. Ensure to use the `--read_long_video` flag for processing long videos.

```bash
python scripts/inference_reconstruct.py --config CONFIG_v1_1 --ckpt CKPT_v1_1 --input_video_path VIDEO_PATH --input_height 256 --input_width 256 --sample_fps 30 --chunk_size CHUNK_SIZE --output_video_dir OUTPUT_DIR --read_long_video
# Set `CHUNK_SIZE` according to your GPU memory, recommendly 16.
```

--------------------------------

### Reconstruct Long Videos with v1.1 Models

Source: https://context7.com/microsoft/vidtok/llms.txt

Supports arbitrary length videos using temporal chunking. Ensure frame count mismatches are handled if necessary.

```python
import torch
from scripts.inference_evaluate import load_model_from_config

# Load v1.1 model for long video support
model = load_model_from_config(
    "configs/vidtok_v1_1/vidtok_kl_causal_488_4chn_v1_1.yaml",
    "checkpoints/vidtok_v1_1/vidtok_kl_causal_488_4chn_v1_1.ckpt"
)
model.to('cuda').eval()

# Enable tiling with temporal chunking for long videos
model.use_tiling = True
model.t_chunk_enc = 16  # Chunk size for encoding
model.t_chunk_dec = model.t_chunk_enc // model.encoder.time_downsample_factor
model.use_overlap = True  # Enable temporal overlap for smooth transitions

# Process long video (129 frames)
x_input = (torch.rand(1, 3, 129, 256, 256) * 2 - 1).to('cuda')

with torch.no_grad(), torch.autocast(device_type='cuda', dtype=torch.float16):
    z, x_recon, reg_log = model(x_input)

# Handle potential frame count mismatch
if x_recon.shape[2] != x_input.shape[2]:
    x_recon = x_recon[:, :, -x_input.shape[2]:, ...]

print(f"Long video reconstruction shape: {x_recon.shape}")
```

--------------------------------

### Enable Full Model Fine-tuning

Source: https://github.com/microsoft/vidtok/blob/main/README.md

Set encoder and decoder parameters to false to allow weight updates during training.

```yaml
model:
  params:
    encoder_config:
      params:
        fix_encoder: false
        fix_decoder: false
```

--------------------------------

### VidTwin Video Reconstruction

Source: https://context7.com/microsoft/vidtok/llms.txt

This command performs video reconstruction using the VidTwin model. It takes an input video path and outputs the reconstructed video.

```bash
python vidtwin/scripts/inference_reconstruct.py \
    --config configs/vidtwin/vidtwin_structure_7_7_8_dynamics_7_8.yaml \
    --ckpt checkpoints/vidtwin_structure_7_7_8_dynamics_7_8.ckpt \
    --input_video_path assets/example.mp4 \
    --num_frames_per_batch 16 \
    --input_height 224 \
    --input_width 224 \
    --sample_fps 25 \
    --output_video_dir ./output
```

--------------------------------

### Enable Tiling for High-Resolution Videos

Source: https://context7.com/microsoft/vidtok/llms.txt

Reduces GPU memory usage by processing spatial tiles. Call disable_tiling() after processing to revert settings.

```python
import torch
from scripts.inference_evaluate import load_model_from_config

model = load_model_from_config(
    "configs/vidtok_kl_causal_488_4chn.yaml",
    "checkpoints/vidtok_kl_causal_488_4chn.ckpt"
)
model.to('cuda').eval()

# Enable tiling for 768x768 resolution (reduces memory to ~6GB for 17x768x768)
model.enable_tiling(
    tile_sample_min_height=256,
    tile_sample_min_width=256,
    tile_overlap_factor_height=0.125,  # 1/8 overlap for blending
    tile_overlap_factor_width=0.125
)

# Process high-resolution video
x_input = (torch.rand(1, 3, 17, 768, 768) * 2 - 1).to('cuda')

with torch.no_grad(), torch.autocast(device_type='cuda', dtype=torch.float16):
    z, x_recon, reg_log = model(x_input)

print(f"High-res reconstruction shape: {x_recon.shape}")

# Disable tiling when done
model.disable_tiling()
```

--------------------------------

### Reconstruct Input Video

Source: https://github.com/microsoft/vidtok/blob/main/README.md

Executes video reconstruction using a specified configuration and checkpoint. Supports optional frame padding for causal models.

```bash
python scripts/inference_reconstruct.py --config CONFIG --ckpt CKPT --input_video_path VIDEO_PATH --input_height 256 --input_width 256 --sample_fps 30 --output_video_dir OUTPUT_DIR
```

--------------------------------

### Configure Model Parameters

Source: https://github.com/microsoft/vidtok/blob/main/vidtwin/README.md

Define the model structure and latent space dimensions in the YAML configuration file.

```yaml
model:
  params:
    expect_ch: 8  # the dimension of the Structure Latent, d_S
    cont_num_blocks: 1 # downsample blocks of the Structure Latent, 1 -> h_S = 7, 2 -> h_S = 4, 3 -> h_S = 2
    downsample_motion: True
    motion_num_blocks: 1 # downsample blocks of the Dynamics Latent, 1 -> h_D = 7, 2 -> h_D = 4, 3 -> h_D = 2
    d_dim: 8 # the dimension of the Dynamics Latent, d_D
```

--------------------------------

### Perform Model Inference

Source: https://github.com/microsoft/vidtok/blob/main/vidtwin/README.md

Load a pre-trained model and run a forward pass with random input.

```python
import torch
from scripts.inference_evaluate import load_model_from_config

cfg_path = "configs/vidtwin/vidtwin_structure_7_7_8_dynamics_7_8.yaml"
ckpt_path = "checkpoints/vidtwin_structure_7_7_8_dynamics_7_8.ckpt"

device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")    
# load pre-trained model
model = load_model_from_config(cfg_path, ckpt_path)
model.to(device).eval()
# random input
num_frames = 16
x_input = (torch.rand(1, 3, num_frames, 224, 224) * 2 - 1).to(device)  # [B, C, T, H, W], range -1~1
# model forward
_, x_recon, *_ = model(x_input)
assert x_input.shape == x_recon.shape
```

--------------------------------

### VidTwin Cross-Reenactment

Source: https://context7.com/microsoft/vidtok/llms.txt

This command enables cross-reenactment in VidTwin, combining structure from one video with dynamics from another. Specify paths for both structure and dynamics input videos.

```bash
python vidtwin/scripts/inference_vidtwin_cross_reconstruct.py \
    --config configs/vidtwin/vidtwin_structure_7_7_8_dynamics_7_8.yaml \
    --ckpt checkpoints/vidtwin_structure_7_7_8_dynamics_7_8.ckpt \
    --input_video_path_structure assets/video_a.mp4 \
    --input_video_path_dynamics assets/video_b.mp4 \
    --output_video_dir ./output
```

--------------------------------

### Cite VidTwin in BibTeX

Source: https://github.com/microsoft/vidtok/blob/main/vidtwin/README.md

Use this BibTeX entry to cite the VidTwin paper in research publications.

```bibtex
@article{wang2024vidtwin,
  title={VidTwin: Video VAE with Decoupled Structure and Dynamics},
  author={Wang, Yuchi and Guo, Junliang and Xie, Xinyi and He, Tianyu and Sun, Xu and Bian, Jiang},
  year={2024},
  journal={arXiv preprint arXiv:2412.17726},
}
```

--------------------------------

### Cite VidTok

Source: https://github.com/microsoft/vidtok/blob/main/README.md

BibTeX entry for referencing the VidTok research paper.

```bibtex
@article{tang2024vidtok,
  title={VidTok: A Versatile and Open-Source Video Tokenizer},
  author={Tang, Anni and He, Tianyu and Guo, Junliang and Cheng, Xinle and Song, Li and Bian, Jiang},
  year={2024},
  journal={arXiv preprint arXiv:2412.13061},
}
```

--------------------------------

### Encode Video to Latent Space

Source: https://context7.com/microsoft/vidtok/llms.txt

Compresses video frames into a latent representation using the `encode` method. Optionally returns regularization logs, which contain token indices for FSQ models or KL divergence for continuous models.

```python
import torch
from scripts.inference_evaluate import load_model_from_config

model = load_model_from_config(
    "configs/vidtok_kl_causal_488_4chn.yaml",
    "checkpoints/vidtok_kl_causal_488_4chn.ckpt"
)
model.to('cuda').eval()

# Input video tensor
x_input = (torch.rand(1, 3, 17, 256, 256) * 2 - 1).to('cuda')

with torch.no_grad(), torch.autocast(device_type='cuda', dtype=torch.float16):
    # Encode with regularization log (contains KL loss for continuous, indices for discrete)
    z, reg_log = model.encode(x_input, return_reg_log=True)

    # For FSQ (discrete) models, reg_log contains token indices
    if 'indices' in reg_log:
        token_indices = reg_log['indices']
        print(f"Token indices shape: {token_indices.shape}")

    # For KL (continuous) models, reg_log contains KL divergence
    if 'kl_loss' in reg_log:
        print(f"KL loss: {reg_log['kl_loss'].item():.4f}")

print(f"Latent shape: {z.shape}")  # Compressed representation
```

--------------------------------

### VidTok Model Forward Pass (Encode-Decode)

Source: https://context7.com/microsoft/vidtok/llms.txt

Performs a full encode-decode reconstruction of input video frames. The forward pass returns the latent representation, the reconstructed video, and regularization logs. Input tensors should be in the range [-1, 1]. Causal models require more frames than non-causal models.

```python
import torch
from scripts.inference_evaluate import load_model_from_config

cfg_path = "configs/vidtok_kl_causal_488_4chn.yaml"
ckpt_path = "checkpoints/vidtok_kl_causal_488_4chn.ckpt"

model = load_model_from_config(cfg_path, ckpt_path)
model.to('cuda').eval()

# Prepare input: [B, C, T, H, W] with values in range [-1, 1]
# Causal models use 17 frames, non-causal use 16 frames
num_frames = 17 if model.is_causal else 16
x_input = (torch.rand(1, 3, num_frames, 256, 256) * 2 - 1).to('cuda')

# Forward pass with mixed precision
with torch.no_grad(), torch.autocast(device_type='cuda', dtype=torch.float16):
    z, x_recon, reg_log = model(x_input)

print(f"Input shape: {x_input.shape}")       # torch.Size([1, 3, 17, 256, 256])
print(f"Latent shape: {z.shape}")            # torch.Size([1, 4, 5, 32, 32]) for 4x8x8 compression
print(f"Reconstruction shape: {x_recon.shape}")  # torch.Size([1, 3, 17, 256, 256])
assert x_input.shape == x_recon.shape
```

--------------------------------

### Decode from Latent Space

Source: https://context7.com/microsoft/vidtok/llms.txt

Reconstructs video frames from either continuous latent vectors or discrete token indices. FSQ models can decode directly from token indices using `decode_from_indices=True`.

```python
import torch
from scripts.inference_evaluate import load_model_from_config

model = load_model_from_config(
    "configs/vidtok_fsq_causal_488_262144.yaml",  # FSQ discrete model
    "checkpoints/vidtok_fsq_causal_488_262144.ckpt"
)
model.to('cuda').eval()

x_input = (torch.rand(1, 3, 17, 256, 256) * 2 - 1).to('cuda')

with torch.no_grad(), torch.autocast(device_type='cuda', dtype=torch.float16):
    z, reg_log = model.encode(x_input, return_reg_log=True)

    # Decode from continuous latent space
    x_recon_continuous = model.decode(z)

    # Decode from discrete token indices (FSQ models only)
    if 'indices' in reg_log:
        x_recon_discrete = model.decode(reg_log['indices'], decode_from_indices=True)
        print(f"Discrete reconstruction shape: {x_recon_discrete.shape}")

print(f"Continuous reconstruction shape: {x_recon_continuous.shape}")
```

--------------------------------

### Infer from Latent Tokens

Source: https://github.com/microsoft/vidtok/blob/main/README.md

Perform inference directly from continuous or discrete latent spaces using the model's encoder and decoder.

```python
z, reg_log = model.encode(x_input, return_reg_log=True)
# infer from continuous latent space
x_recon = model.decode(z)
# infer from discrete latent tokens
x_recon = model.decode(reg_log['indices'], decode_from_indices=True)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.