### Install Inference Dependencies

Source: https://github.com/peterl1n/robustvideomatting/blob/master/README.md

Install necessary Python packages for inference by running this command. Ensure you have the requirements_inference.txt file.

```sh
pip install -r requirements_inference.txt
```

--------------------------------

### Load Model and Converter via TorchHub

Source: https://github.com/peterl1n/robustvideomatting/blob/master/README.md

Load the Robust Video Matting model and the `convert_video` API directly from TorchHub. This simplifies setup by handling model fetching and loading.

```python
# Load the model.
model = torch.hub.load("PeterL1n/RobustVideoMatting", "mobilenetv3") # or "resnet50"

# Converter API.
convert_video = torch.hub.load("PeterL1n/RobustVideoMatting", "converter")
```

--------------------------------

### Custom PyTorch Inference Loop

Source: https://github.com/peterl1n/robustvideomatting/blob/master/README.md

Implement a custom inference loop for fine-grained control. This example reads frames using `VideoReader`, processes them with the model, composites the foreground onto a background, and writes the output using `VideoWriter`. It demonstrates managing recurrent states and adjusting `downsample_ratio`.

```python
from torch.utils.data import DataLoader
from torchvision.transforms import ToTensor
from inference_utils import VideoReader, VideoWriter

reader = VideoReader('input.mp4', transform=ToTensor())
writer = VideoWriter('output.mp4', frame_rate=30)

bgr = torch.tensor([.47, 1, .6]).view(3, 1, 1).cuda()  # Green background. 
rec = [None] * 4                                       # Initial recurrent states. 
downsample_ratio = 0.25                                # Adjust based on your video. 

with torch.no_grad():
    for src in DataLoader(reader):                     # RGB tensor normalized to 0 ~ 1. 
        fgr, pha, *rec = model(src.cuda(), *rec, downsample_ratio)  # Cycle the recurrent states. 
        com = fgr * pha + bgr * (1 - pha)              # Composite to green background. 
        writer.write(com)                              # Write frame.
```

--------------------------------

### Initialize and Run MattingNetwork

Source: https://context7.com/peterl1n/robustvideomatting/llms.txt

Demonstrates loading the MattingNetwork model, managing recurrent states for temporal consistency, and performing inference on single frames or chunks.

```python
import torch
from model import MattingNetwork

# Initialize the model with MobileNetV3 backbone (lightweight, fast)
model = MattingNetwork(variant='mobilenetv3').eval().cuda()
model.load_state_dict(torch.load('rvm_mobilenetv3.pth'))

# Or use ResNet50 backbone (larger, slightly better quality)
# model = MattingNetwork(variant='resnet50').eval().cuda()
# model.load_state_dict(torch.load('rvm_resnet50.pth'))

# Initialize recurrent states (4 ConvGRU layers in the decoder)
rec = [None] * 4

# Process video frames - src can be [B, C, H, W] or [B, T, C, H, W]
# RGB input normalized to 0~1 range
src = torch.rand(1, 3, 1080, 1920).cuda()  # Single frame
downsample_ratio = 0.25  # Adjust based on resolution

with torch.no_grad():
    # Forward pass returns foreground, alpha, and updated recurrent states
    fgr, pha, *rec = model(src, *rec, downsample_ratio=downsample_ratio)

    # fgr: [B, 3, H, W] - foreground RGB prediction (0~1)
    # pha: [B, 1, H, W] - alpha matte prediction (0~1)
    # rec: list of 4 tensors - recurrent states for next frame

# Process multiple frames at once for better parallelism
src_chunk = torch.rand(1, 12, 3, 1080, 1920).cuda()  # 12 frames
fgr, pha, *rec = model(src_chunk, *rec, downsample_ratio=downsample_ratio)
# fgr: [B, T, 3, H, W], pha: [B, T, 1, H, W]

# Composite foreground onto a new background (green screen example)
bgr = torch.tensor([0.47, 1.0, 0.6]).view(3, 1, 1).cuda()
composite = fgr * pha + bgr * (1 - pha)
```

--------------------------------

### Execute Command-Line Inference

Source: https://context7.com/peterl1n/robustvideomatting/llms.txt

Run video conversion tasks directly from the terminal.

```bash
# Basic video conversion with green screen output
python inference.py \
    --variant mobilenetv3 \
    --checkpoint rvm_mobilenetv3.pth \
    --device cuda \
    --input-source "input.mp4" \
    --output-type video \
    --output-composition "composition.mp4" \
    --output-video-mbps 4 \
    --seq-chunk 12

# Full options with all outputs
python inference.py \
    --variant mobilenetv3 \
    --checkpoint rvm_mobilenetv3.pth \
    --device cuda \
    --input-source "input.mp4" \
    --input-resize 1920 1080 \
    --downsample-ratio 0.25 \
    --output-type video \
    --output-composition "composition.mp4" \
    --output-alpha "alpha.mp4" \
    --output-foreground "foreground.mp4" \
    --output-video-mbps 4 \
    --seq-chunk 12 \
    --num-workers 2

# Output as PNG sequence
python inference.py \
    --variant resnet50 \
    --checkpoint rvm_resnet50.pth \
    --device cuda \
    --input-source "frames_dir/" \
    --output-type png_sequence \
    --output-composition "output_frames/" \
    --seq-chunk 8
```

--------------------------------

### Command Line Interface for Video Conversion

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Run the video conversion process directly from the command line using the provided inference script.

```APIDOC
## Command Line Interface for Video Conversion

Use the `inference.py` script to convert videos from the command line.

```sh
python inference.py \
    --variant mobilenetv3 \
    --checkpoint "CHECKPOINT" \
    --device cuda \
    --input-source "input.mp4" \
    --downsample-ratio 0.25 \
    --output-type video \
    --output-composition "composition.mp4" \
    --output-alpha "alpha.mp4" \
    --output-foreground "foreground.mp4" \
    --output-video-mbps 4 \
    --seq-chunk 12
```
```

--------------------------------

### MNN C++ Demo for Robust Video Matting

Source: https://github.com/peterl1n/robustvideomatting/blob/master/README.md

This C++ code demonstrates the integration of Robust Video Matting with the MNN inference engine. It is intended for use on platforms supporting MNN.

```cpp
#include <iostream>
#include <vector>
#include <memory>
#include <algorithm>

#include "MNN/Interpreter.hpp"
#include "MNN/MNNDefault.h"
#include "MNN/MNNForwardType.h"

#include "core/Macro.h"
#include "core/Tensor.h"
#include "core/Types.h"
#include "utils/Log.h"
#include "utils/String.h"
#include "cv/cv.hpp"
#include "cv/ImageProcess.h"

using namespace MNN;

// Assuming RVM_API is defined elsewhere or not needed for this snippet
// Assuming cv::Mat is a type defined for image representation

int main(int argc, char **argv) {
    // ... (model loading and initialization code) ...

    // Example of processing an image (assuming 'input_image' is a cv::Mat)
    // MNN::cv::Mat mnn_input_image(input_image.rows, input_image.cols, MNN::cv::NCHW, input_image.data, MNN::cv::BGRA);
    // MNN::cv::Mat mnn_output_image(output_mask.rows, output_mask.cols, MNN::cv::NCHW, output_mask.data, MNN::cv::RGBA);

    // ... (inference and post-processing code) ...

    std::cout << "MNN C++ Demo executed." << std::endl;

    return 0;
}

```

--------------------------------

### Inference with Frozen TorchScript Model

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Demonstrates how to use a frozen TorchScript model with the `convert_video` function. Note that `device` and `dtype` must be manually provided for frozen models.

```python
convert_video(frozen_model, ...args..., device='cuda', dtype=torch.float32)
```

--------------------------------

### TNN C++ Demo for Robust Video Matting

Source: https://github.com/peterl1n/robustvideomatting/blob/master/README.md

This C++ code showcases the implementation of Robust Video Matting using the TNN inference framework. It is designed for environments where TNN is supported.

```cpp
#include <iostream>
#include <vector>
#include <memory>
#include <algorithm>

#include "tnn/core/macro.h"
#include "tnn/core/session.h"
#include "tnn/core/blob.h"
#include "tnn/utils/dims_vector_utils.h"
#include "tnn/utils/blob_memory_utils.h"
#include "tnn/network/network_info.h"
#include "tnn/network/network_builder.h"
#include "tnn/core/common.h"

#include "cv/cv.hpp"
#include "cv/ImageProcess.h"

using namespace TNN;

// Assuming RVM_API is defined elsewhere or not needed for this snippet
// Assuming cv::Mat is a type defined for image representation

int main(int argc, char **argv) {
    // ... (model loading and initialization code) ...

    // Example of processing an image (assuming 'input_image' is a cv::Mat)
    // TNN::cv::Mat tnn_input_image(input_image.rows, input_image.cols, TNN::cv::NCHW, input_image.data, TNN::cv::BGRA);
    // TNN::cv::Mat tnn_output_image(output_mask.rows, output_mask.cols, TNN::cv::NCHW, output_mask.data, TNN::cv::RGBA);

    // ... (inference and post-processing code) ...

    std::cout << "TNN C++ Demo executed." << std::endl;

    return 0;
}

```

--------------------------------

### Stage 1 Training Command

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/training.md

Command to initiate the first stage of training for the video matting model. Specifies model variant, dataset, resolution, sequence length, learning rates, and directories for checkpoints and logs.

```sh
python train.py \
    --model-variant mobilenetv3 \
    --dataset videomatte \
    --resolution-lr 512 \
    --seq-length-lr 15 \
    --learning-rate-backbone 0.0001 \
    --learning-rate-aspp 0.0002 \
    --learning-rate-decoder 0.0002 \
    --learning-rate-refiner 0 \
    --checkpoint-dir checkpoint/stage1 \
    --log-dir log/stage1 \
    --epoch-start 0 \
    --epoch-end 20
```

--------------------------------

### Load PyTorch Model

Source: https://github.com/peterl1n/robustvideomatting/blob/master/README.md

Load a pre-trained Robust Video Matting model in PyTorch. Specify the model architecture ('mobilenetv3' or 'resnet50') and move it to the CUDA device if available. Ensure the model weights file is present.

```python
import torch
from model import MattingNetwork

model = MattingNetwork('mobilenetv3').eval().cuda()  # or "resnet50"
model.load_state_dict(torch.load('rvm_mobilenetv3.pth'))
```

--------------------------------

### Process Image Sequences with ImageSequenceReader and ImageSequenceWriter

Source: https://context7.com/peterl1n/robustvideomatting/llms.txt

Handle directories of images as video frames for matting tasks.

```python
from inference_utils import ImageSequenceReader, ImageSequenceWriter
from torchvision.transforms import ToTensor
from torch.utils.data import DataLoader
import torch

# Read from directory of sorted images (PNG/JPG)
# Directory structure: frames/0001.png, frames/0002.png, ...
reader = ImageSequenceReader('input_frames/', transform=ToTensor())

# Write RGBA PNG sequence
writer = ImageSequenceWriter(
    path='output_frames/',
    extension='png'  # or 'jpg'
)

model = torch.hub.load("PeterL1n/RobustVideoMatting", "mobilenetv3").eval().cuda()
rec = [None] * 4

with torch.no_grad():
    for src in DataLoader(reader, batch_size=4, num_workers=2):
        src = src.cuda()
        fgr, pha, *rec = model(src, *rec, downsample_ratio=0.25)

        # Create RGBA output (foreground with alpha channel)
        fgr = fgr * pha.gt(0)  # Zero out background pixels
        rgba = torch.cat([fgr, pha], dim=1)  # [B, 4, H, W]

        writer.write(rgba)

writer.close()
```

--------------------------------

### Command Line Video Conversion

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Invokes the video conversion utility from the command line using `inference.py`. This allows for easy execution of video processing tasks with various parameters.

```sh
python inference.py \
    --variant mobilenetv3 \
    --checkpoint "CHECKPOINT" \
    --device cuda \
    --input-source "input.mp4" \
    --downsample-ratio 0.25 \
    --output-type video \
    --output-composition "composition.mp4" \
    --output-alpha "alpha.mp4" \
    --output-foreground "foreground.mp4" \
    --output-video-mbps 4 \
    --seq-chunk 12
```

--------------------------------

### Load and Run RVM Model via Torch Hub

Source: https://context7.com/peterl1n/robustvideomatting/llms.txt

Load a pre-trained model and use the converter function for video processing.

```python
# Load ResNet50 model (larger, better quality)
model_resnet = torch.hub.load("PeterL1n/RobustVideoMatting", "resnet50")

# Load the converter function
convert_video = torch.hub.load("PeterL1n/RobustVideoMatting", "converter")

# Use the converter with loaded model
convert_video(
    model,
    input_source='my_video.mp4',
    output_type='video',
    output_composition='output.mp4',
    output_video_mbps=4,
    seq_chunk=12
)
```

--------------------------------

### Load ONNX Model

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Initialize an ONNX Runtime inference session for the specified model file.

```python
import onnxruntime as ort

sess = ort.InferenceSession('rvm_mobilenetv3_fp16.onnx')
```

--------------------------------

### Perform Custom Inference with VideoReader and VideoWriter

Source: https://context7.com/peterl1n/robustvideomatting/llms.txt

Use PyTorch Dataset-compatible classes to read and write video files for custom inference loops.

```python
from torch.utils.data import DataLoader
from torchvision.transforms import ToTensor
from inference_utils import VideoReader, VideoWriter
import torch

# Initialize video reader as a PyTorch Dataset
reader = VideoReader('input.mp4', transform=ToTensor())
print(f"Total frames: {len(reader)}")
print(f"Frame rate: {reader.frame_rate}")

# Initialize video writer
writer = VideoWriter(
    path='output.mp4',
    frame_rate=30,
    bit_rate=4000000  # 4 Mbps
)

# Load model
model = torch.hub.load("PeterL1n/RobustVideoMatting", "mobilenetv3").eval().cuda()

# Custom inference loop with green background compositing
bgr = torch.tensor([0.47, 1.0, 0.6]).view(3, 1, 1).cuda()
rec = [None] * 4

with torch.no_grad():
    for src in DataLoader(reader, batch_size=1):
        src = src.cuda()
        fgr, pha, *rec = model(src, *rec, downsample_ratio=0.25)

        # Composite with green background
        composite = fgr * pha + bgr * (1 - pha)

        # Write frame - expects [T, C, H, W] tensor
        writer.write(composite)

writer.close()
```

--------------------------------

### Load Model via TorchHub

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Loads the RobustVideoMatting model directly using `torch.hub.load`. This is a convenient way to access the model without needing to clone the repository.

```python
model = torch.hub.load("PeterL1n/RobustVideoMatting", "mobilenetv3") # or "resnet50"
```

--------------------------------

### Load Video Converter via TorchHub

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Loads the `convert_video` function using `torch.hub.load`. This allows using the conversion utility without direct import from the `inference_utils` module.

```python
convert_video = torch.hub.load("PeterL1n/RobustVideoMatting", "converter")

convert_video(model, ...args...)
```

--------------------------------

### TorchScript Model Loading and Usage

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Load a TorchScript model, optionally freeze it for optimization, and use it with the converter utility.

```APIDOC
## TorchScript Model Loading and Usage

### Model Loading

```python
import torch
model = torch.jit.load('rvm_mobilenetv3.torchscript')
```

### Freezing the Model (Optional)

Freezing optimizes the model graph for faster inference.

```python
model = torch.jit.freeze(model)
```

### Using with Converter API

When using a frozen TorchScript model with the converter, you must manually provide `device` and `dtype`.

```python
convert_video(frozen_model, ...args..., device='cuda', dtype=torch.float32)
```
```

--------------------------------

### Configure Downsample Ratio for Robust Video Matting

Source: https://context7.com/peterl1n/robustvideomatting/llms.txt

Provides a helper function for automatic ratio calculation and demonstrates its usage within the convert_video inference function.

```python
# Recommended downsample_ratio values by resolution and shot type
# Resolution      Portrait    Full-Body
# <= 512x512      1.0         1.0
# 1280x720        0.375       0.6
# 1920x1080       0.25        0.4
# 3840x2160       0.125       0.2

# Auto-calculation function (makes max side 512px)
def auto_downsample_ratio(height, width):
    return min(512 / max(height, width), 1.0)

# Example usage
h, w = 1920, 1080
ratio = auto_downsample_ratio(h, w)  # Returns 0.266...

# The convert_video function auto-calculates if None
from inference import convert_video
convert_video(
    model,
    input_source='input.mp4',
    downsample_ratio=None,  # Auto-calculate based on input size
    output_type='video',
    output_composition='output.mp4'
)
```

--------------------------------

### Perform Optimized ONNX Inference with IOBinding

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Use IOBinding to keep recurrent states on the GPU, avoiding unnecessary CPU-GPU transfers.

```python
import onnxruntime as ort
import numpy as np

# Load model.
sess = ort.InferenceSession('rvm_mobilenetv3_fp16.onnx')

# Create an io binding.
io = sess.io_binding()

# Create tensors on CUDA.
rec = [ ort.OrtValue.ortvalue_from_numpy(np.zeros([1, 1, 1, 1], dtype=np.float16), 'cuda') ] * 4
downsample_ratio = ort.OrtValue.ortvalue_from_numpy(np.asarray([0.25], dtype=np.float32), 'cuda')

# Set output binding.
for name in ['fgr', 'pha', 'r1o', 'r2o', 'r3o', 'r4o']:
    io.bind_output(name, 'cuda')

# Inference loop
for src in YOUR_VIDEO:
    io.bind_cpu_input('src', src)
    io.bind_ortvalue_input('r1i', rec[0])
    io.bind_ortvalue_input('r2i', rec[1])
    io.bind_ortvalue_input('r3i', rec[2])
    io.bind_ortvalue_input('r4i', rec[3])
    io.bind_ortvalue_input('downsample_ratio', downsample_ratio)

    sess.run_with_iobinding(io)

    fgr, pha, *rec = io.get_outputs()

    # Only transfer `fgr` and `pha` to CPU.
    fgr = fgr.numpy()
    pha = pha.numpy()
```

--------------------------------

### Manage Recurrent States for Video Inference

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Demonstrates the correct and incorrect ways to handle recurrent states when processing video frames sequentially.

```python
rec = [None] * 4  # Initial recurrent states are None

for frame in YOUR_VIDEO:
    fgr, pha, *rec = model(frame, *rec, downsample_ratio)
```

```python
for frame in YOUR_VIDEO:
    fgr, pha = model(frame, downsample_ratio)[:2]
```

--------------------------------

### Convert Video using `convert_video` Function

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Utilizes the `convert_video` utility function for streamlined video processing. This function handles model loading, inference, and output generation with various configuration options.

```python
from inference import convert_video

convert_video(
    model,                           # The loaded model, can be on any device (cpu or cuda).
    input_source='input.mp4',        # A video file or an image sequence directory.
    input_resize=(1920, 1080),       # [Optional] Resize the input (also the output).
    downsample_ratio=0.25,           # [Optional] If None, make downsampled max size be 512px.
    output_type='video',             # Choose "video" or "png_sequence"
    output_composition='com.mp4',    # File path if video; directory path if png sequence.
    output_alpha="pha.mp4",          # [Optional] Output the raw alpha prediction.
    output_foreground="fgr.mp4",     # [Optional] Output the raw foreground prediction.
    output_video_mbps=4,             # Output video mbps. Not needed for png sequence.
    seq_chunk=12,                    # Process n frames at once for better parallelism.
    num_workers=1,                   # Only for image sequence input. Reader threads.
    progress=True                    # Print conversion progress.
)
```

--------------------------------

### PyTorch Model Loading and Inference

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Load the PyTorch model and perform inference on video frames. Includes details on input/output shapes and recurrent states.

```APIDOC
## PyTorch Model Loading and Inference

### Model Loading

```python
import torch
from model import MattingNetwork

model = MattingNetwork(variant='mobilenetv3').eval().cuda() # Or variant="resnet50"
model.load_state_dict(torch.load('rvm_mobilenetv3.pth'))
```

### Example Inference Loop

```python
rec = [None] * 4 # Set initial recurrent states to None

for src in YOUR_VIDEO:  # src can be [B, C, H, W] or [B, T, C, H, W]
    fgr, pha, *rec = model(src, *rec, downsample_ratio=0.25)
```

### Input (`src`)

*   **Shape**: Can be `[B, C, H, W]` or `[B, T, C, H, W]`.
*   **Batching**: If `[B, T, C, H, W]`, a chunk of `T` frames can be processed at once for better parallelism.
*   **Normalization**: RGB input should be normalized to the `0~1` range.

### Outputs (`fgr`, `pha`, `rec`)

*   `fgr` (Foreground): RGB prediction with shape `[B, C, H, W]` or `[B, T, C, H, W]`. Normalized to `0~1` range.
*   `pha` (Alpha): Alpha prediction with shape `[B, C, H, W]` or `[B, T, C, H, W]`. Normalized to `0~1` range.
*   `rec` (Recurrent States): A list of 4 tensors representing the recurrent states. Initialized with `[None, None, None, None]`. All tensors are rank 4. If a chunk of `T` frames is processed, only the last frame's recurrent states are returned.

### Complete Video Inference Example

```python
from torch.utils.data import DataLoader
from torchvision.transforms import ToTensor
from inference_utils import VideoReader, VideoWriter

reader = VideoReader('input.mp4', transform=ToTensor())
writer = VideoWriter('output.mp4', frame_rate=30)

bgr = torch.tensor([.47, 1, .6]).view(3, 1, 1).cuda()  # Green background.
rec = [None] * 4                                       # Initial recurrent states.

with torch.no_grad():
    for src in DataLoader(reader):
        fgr, pha, *rec = model(src.cuda(), *rec, downsample_ratio=0.25)  # Cycle the recurrent states.
        writer.write(fgr * pha + bgr * (1 - pha))
```

### Using the `convert_video` Utility

```python
from inference import convert_video

convert_video(
    model,
    input_source='input.mp4',
    input_resize=(1920, 1080),       # [Optional] Resize the input (also the output).
    downsample_ratio=0.25,           # [Optional] If None, make downsampled max size be 512px.
    output_type='video',             # Choose "video" or "png_sequence"
    output_composition='com.mp4',    # File path if video; directory path if png sequence.
    output_alpha="pha.mp4",          # [Optional] Output the raw alpha prediction.
    output_foreground="fgr.mp4",     # [Optional] Output the raw foreground prediction.
    output_video_mbps=4,             # Output video mbps. Not needed for png sequence.
    seq_chunk=12,                    # Process n frames at once for better parallelism.
    num_workers=1,                   # Only for image sequence input. Reader threads.
    progress=True                    # Print conversion progress.
)
```
```

--------------------------------

### TorchHub Model Loading

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Load the model and the conversion utility directly from TorchHub.

```APIDOC
## TorchHub Model Loading

### Model Loading

```python
model = torch.hub.load("PeterL1n/RobustVideoMatting", "mobilenetv3") # or "resnet50"
```

### Using the Conversion Function

```python
convert_video = torch.hub.load("PeterL1n/RobustVideoMatting", "converter")

convert_video(model, ...args...)
```
```

--------------------------------

### Perform Naive ONNX Inference

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Execute inference on a video loop using standard CPU-based input handling.

```python
import numpy as np

rec = [ np.zeros([1, 1, 1, 1], dtype=np.float16) ] * 4  # Must match dtype of the model.
downsample_ratio = np.array([0.25], dtype=np.float32)  # dtype always FP32

for src in YOUR_VIDEO:  # src is of [B, C, H, W] with dtype of the model.
    fgr, pha, *rec = sess.run([], {
        'src': src, 
        'r1i': rec[0], 
        'r2i': rec[1], 
        'r3i': rec[2], 
        'r4i': rec[3], 
        'downsample_ratio': downsample_ratio
    })
```

--------------------------------

### Load TorchScript Model

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Loads a pre-compiled TorchScript model from a file. TorchScript models are optimized for inference and can be deployed in environments without Python.

```python
import torch
model = torch.jit.load('rvm_mobilenetv3.torchscript')
```

--------------------------------

### Convert Video using API

Source: https://github.com/peterl1n/robustvideomatting/blob/master/README.md

Use the provided `convert_video` API for straightforward video processing. This function handles model inference and output generation. Specify input and output paths, and optionally raw alpha/foreground outputs. Adjust `downsample_ratio` for performance and quality trade-offs.

```python
from inference import convert_video

convert_video(
    model,                           # The model, can be on any device (cpu or cuda).
    input_source='input.mp4',        # A video file or an image sequence directory.
    output_type='video',             # Choose "video" or "png_sequence"
    output_composition='com.mp4',    # File path if video; directory path if png sequence.
    output_alpha="pha.mp4",          # [Optional] Output the raw alpha prediction.
    output_foreground="fgr.mp4",     # [Optional] Output the raw foreground prediction.
    output_video_mbps=4,             # Output video mbps. Not needed for png sequence.
    downsample_ratio=None,           # A hyperparameter to adjust or use None for auto.
    seq_chunk=12,                    # Process n frames at once for better parallelism.
)
```

--------------------------------

### CoreML Inference (iOS/macOS)

Source: https://context7.com/peterl1n/robustvideomatting/llms.txt

Loads a CoreML model for deployment on Apple platforms. This model supports fixed resolutions and requires PIL Images as input.

```python
import coremltools as ct
from PIL import Image

# Load CoreML model (fixed resolution, adjust as needed)
model = ct.models.model.MLModel('rvm_mobilenetv3_1920x1080_s0.25_fp16.mlmodel')

r1, r2, r3, r4 = None, None, None, None

for frame in video_frames:  # frame: PIL.Image
    if r1 is None:
        # First frame - no recurrent states
        inputs = {'src': frame}
    else:
        # Subsequent frames - include recurrent states
        inputs = {'src': frame, 'r1i': r1, 'r2i': r2, 'r3i': r3, 'r4i': r4}

    outputs = model.predict(inputs)

    fgr = outputs['fgr']  # PIL.Image
    pha = outputs['pha']  # PIL.Image

    # Update recurrent states for next frame
    r1, r2, r3, r4 = outputs['r1o'], outputs['r2o'], outputs['r3o'], outputs['r4o']
```

--------------------------------

### Process Video Frames with CoreML Model

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Use this snippet to process video frames sequentially using a CoreML model. For the first frame, omit recurrent states. For subsequent frames, provide the recurrent states from the previous frame's output. Ensure the model path is correct and the input 'src' is a PIL Image.

```python
import coremltools as ct

model = ct.models.model.MLModel('rvm_mobilenetv3_1920x1080_s0.25_int8.mlmodel')

r1, r2, r3, r4 = None, None, None, None

for src in YOUR_VIDEO:  # src is PIL.Image.
    
    if r1 is None:
        # Initial frame, do not provide recurrent states.
        inputs = {'src': src}
    else:
        # Subsequent frames, provide recurrent states.
        inputs = {'src': src, 'r1i': r1, 'r2i': r2, 'r3i': r3, 'r4i': r4}

    outputs = model.predict(inputs)

    fgr = outputs['fgr']  # PIL.Image.
    pha = outputs['pha']  # PIL.Image.
    
    r1 = outputs['r1o']  # Numpy array.
    r2 = outputs['r2o']  # Numpy array.
    r3 = outputs['r3o']  # Numpy array.
    r4 = outputs['r4o']  # Numpy array.
```

--------------------------------

### Load PyTorch Matting Network

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Loads the MattingNetwork model with a specified variant and moves it to the CUDA-enabled GPU. Ensure the model weights file ('rvm_mobilenetv3.pth') is in the correct directory.

```python
import torch
from model import MattingNetwork

model = MattingNetwork(variant='mobilenetv3').eval().cuda() # Or variant="resnet50"
model.load_state_dict(torch.load('rvm_mobilenetv3.pth'))
```

--------------------------------

### Load Model via TorchHub

Source: https://context7.com/peterl1n/robustvideomatting/llms.txt

Loads pre-trained RVM models directly from GitHub using TorchHub, which handles automatic downloading and caching.

```python
import torch

# Load MobileNetV3 model (recommended for most cases)
model = torch.hub.load("PeterL1n/RobustVideoMatting", "mobilenetv3")
model = model.eval().cuda()
```

--------------------------------

### Optimized GPU Inference with IO Binding (ONNX)

Source: https://context7.com/peterl1n/robustvideomatting/llms.txt

Utilizes ONNX Runtime with IO binding for efficient GPU inference, minimizing CPU-GPU data transfers. Ensure tensors are created on the desired device (e.g., 'cuda').

```python
import onnxruntime as ort
import numpy as np

sess = ort.InferenceSession('rvm_mobilenetv3_fp16.onnx')
io = sess.io_binding()

# Create tensors on CUDA
rec = [ort.OrtValue.ortvalue_from_numpy(
    np.zeros([1, 1, 1, 1], dtype=np.float16), 'cuda'
)] * 4
downsample_ratio = ort.OrtValue.ortvalue_from_numpy(
    np.array([0.25], dtype=np.float32), 'cuda'
)

# Bind outputs to CUDA
for name in ['fgr', 'pha', 'r1o', 'r2o', 'r3o', 'r4o']:
    io.bind_output(name, 'cuda')

for src in video_frames:
    io.bind_cpu_input('src', src)
    io.bind_ortvalue_input('r1i', rec[0])
    io.bind_ortvalue_input('r2i', rec[1])
    io.bind_ortvalue_input('r3i', rec[2])
    io.bind_ortvalue_input('r4i', rec[3])
    io.bind_ortvalue_input('downsample_ratio', downsample_ratio)

    sess.run_with_iobinding(io)
    fgr, pha, *rec = io.get_outputs()

    # Only transfer results to CPU when needed
    fgr_np = fgr.numpy()
    pha_np = pha.numpy()
```

--------------------------------

### convert_video Utility

Source: https://context7.com/peterl1n/robustvideomatting/llms.txt

High-level API for processing entire video files or image sequences with automatic batching and output handling.

```APIDOC
## convert_video

### Description
Automates the process of reading input sources, performing inference, and writing output files (video or image sequences).

### Parameters
- **model** (MattingNetwork) - Required - The loaded RVM model.
- **input_source** (str) - Required - Path to video file or image sequence directory.
- **input_resize** (tuple) - Optional - Target (width, height) for input resizing.
- **downsample_ratio** (float) - Optional - Downsampling ratio for inference.
- **output_type** (str) - Required - 'video' or 'png_sequence'.
- **output_composition** (str) - Optional - Path for composite output.
- **output_alpha** (str) - Optional - Path for raw alpha matte output.
- **output_foreground** (str) - Optional - Path for raw foreground output.
- **seq_chunk** (int) - Optional - Number of frames per batch.

### Request Example
```python
convert_video(model, input_source='input.mp4', output_type='video', output_composition='out.mp4')
```
```

--------------------------------

### Complete Video Inference with PyTorch DataLoader

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Infers on a video using PyTorch's DataLoader for efficient frame loading and processing. It cycles recurrent states and writes the composited output to a new video file.

```python
from torch.utils.data import DataLoader
from torchvision.transforms import ToTensor
from inference_utils import VideoReader, VideoWriter

reader = VideoReader('input.mp4', transform=ToTensor())
writer = VideoWriter('output.mp4', frame_rate=30)

bgr = torch.tensor([.47, 1, .6]).view(3, 1, 1).cuda()  # Green background.
rec = [None] * 4                                       # Initial recurrent states.

with torch.no_grad():
    for src in DataLoader(reader):
        fgr, pha, *rec = model(src.cuda(), *rec, downsample_ratio=0.25)  # Cycle the recurrent states.
        writer.write(fgr * pha + bgr * (1 - pha))
```

--------------------------------

### Run ONNX Runtime Inference

Source: https://context7.com/peterl1n/robustvideomatting/llms.txt

Perform inference using exported ONNX models with explicit recurrent state management.

```python
import onnxruntime as ort
import numpy as np

# Load ONNX model
sess = ort.InferenceSession('rvm_mobilenetv3_fp16.onnx')

# Initialize recurrent states (must match model dtype)
rec = [np.zeros([1, 1, 1, 1], dtype=np.float16)] * 4
downsample_ratio = np.array([0.25], dtype=np.float32)

# Inference loop
for src in video_frames:  # src: [B, C, H, W] numpy array
    fgr, pha, *rec = sess.run([], {
        'src': src.astype(np.float16),
        'r1i': rec[0],
        'r2i': rec[1],
        'r3i': rec[2],
        'r4i': rec[3],
        'downsample_ratio': downsample_ratio
    })
    # fgr, pha: numpy arrays, rec: updated recurrent states
```

--------------------------------

### Stage 2 Training Command

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/training.md

Command for the second training stage, building upon the checkpoint from Stage 1. Adjusts sequence length and learning rates, and specifies the previous checkpoint.

```sh
python train.py \
    --model-variant mobilenetv3 \
    --dataset videomatte \
    --resolution-lr 512 \
    --seq-length-lr 50 \
    --learning-rate-backbone 0.00005 \
    --learning-rate-aspp 0.0001 \
    --learning-rate-decoder 0.0001 \
    --learning-rate-refiner 0 \
    --checkpoint checkpoint/stage1/epoch-19.pth \
    --checkpoint-dir checkpoint/stage2 \
    --log-dir log/stage2 \
    --epoch-start 20 \
    --epoch-end 22
```

--------------------------------

### Stage 4 Training Command

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/training.md

Command for the fourth training stage, focusing on image matting with high-resolution training. It uses a checkpoint from Stage 3 and modifies learning rates and dataset.

```sh
python train.py \
    --model-variant mobilenetv3 \
    --dataset imagematte \
    --train-hr \
    --resolution-lr 512 \
    --resolution-hr 2048 \
    --seq-length-lr 40 \
    --seq-length-hr 6 \
    --learning-rate-backbone 0.00001 \
    --learning-rate-aspp 0.00001 \
    --learning-rate-decoder 0.00005 \
    --learning-rate-refiner 0.0002 \
    --checkpoint checkpoint/stage3/epoch-22.pth \
    --checkpoint-dir checkpoint/stage4 \
    --log-dir log/stage4 \
    --epoch-start 23 \
    --epoch-end 28
```

--------------------------------

### MattingNetwork Inference

Source: https://context7.com/peterl1n/robustvideomatting/llms.txt

The MattingNetwork class performs the core matting inference, maintaining recurrent states for temporal consistency across video frames.

```APIDOC
## MattingNetwork Inference

### Description
Processes video frames to generate foreground RGB and alpha matte predictions using recurrent neural network states.

### Parameters
- **src** (Tensor) - Required - Input tensor of shape [B, C, H, W] or [B, T, C, H, W].
- **rec** (List) - Required - List of 4 recurrent state tensors.
- **downsample_ratio** (float) - Optional - Ratio to downsample input for performance.

### Response
- **fgr** (Tensor) - Foreground RGB prediction (0~1).
- **pha** (Tensor) - Alpha matte prediction (0~1).
- **rec** (List) - Updated recurrent states for the next frame.

### Request Example
```python
# Single frame inference
fgr, pha, *rec = model(src, *rec, downsample_ratio=0.25)
```
```

--------------------------------

### Freeze TorchScript Model for Optimization

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Freezes a loaded TorchScript model to apply graph optimizations, such as BatchNorm fusion, potentially leading to faster inference. The frozen model can then be used like a standard PyTorch model.

```python
model = torch.jit.freeze(model)
```

--------------------------------

### Stage 3 Training Command

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/training.md

Command for the third training stage, enabling high-resolution training. It uses a checkpoint from Stage 2 and adjusts learning rates, sequence lengths, and resolutions.

```sh
python train.py \
    --model-variant mobilenetv3 \
    --dataset videomatte \
    --train-hr \
    --resolution-lr 512 \
    --resolution-hr 2048 \
    --seq-length-lr 40 \
    --seq-length-hr 6 \
    --learning-rate-backbone 0.00001 \
    --learning-rate-aspp 0.00001 \
    --learning-rate-decoder 0.00001 \
    --learning-rate-refiner 0.0002 \
    --checkpoint checkpoint/stage2/epoch-21.pth \
    --checkpoint-dir checkpoint/stage3 \
    --log-dir log/stage3 \
    --epoch-start 22 \
    --epoch-end 23
```

--------------------------------

### PyTorch Inference Loop

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Performs inference on video frames using the loaded PyTorch model. Handles recurrent states for sequential processing. Input `src` can be a single frame or a chunk of frames.

```python
rec = [None] * 4 # Set initial recurrent states to None

for src in YOUR_VIDEO:  # src can be [B, C, H, W] or [B, T, C, H, W]
    fgr, pha, *rec = model(src, *rec, downsample_ratio=0.25)
```

--------------------------------

### Convert Video with High-Level API

Source: https://context7.com/peterl1n/robustvideomatting/llms.txt

Uses the convert_video function to process video files or image sequences, supporting various output formats and configurations.

```python
from model import MattingNetwork
from inference import convert_video
import torch

# Load the model
model = MattingNetwork('mobilenetv3').eval().cuda()
model.load_state_dict(torch.load('rvm_mobilenetv3.pth'))

# Convert video with green screen composition output
convert_video(
    model,                              # The loaded model (can be on cpu or cuda)
    input_source='input.mp4',           # Video file or image sequence directory
    input_resize=(1920, 1080),          # Optional: resize input (width, height)
    downsample_ratio=0.25,              # None for auto (512px max side)
    output_type='video',                # 'video' or 'png_sequence'
    output_composition='composition.mp4',  # Green screen composite output
    output_alpha='alpha.mp4',           # Optional: raw alpha matte output
    output_foreground='foreground.mp4', # Optional: raw foreground output
    output_video_mbps=4,                # Output video bitrate
    seq_chunk=12,                       # Frames per batch (higher = faster)
    num_workers=1,                      # DataLoader workers (for image sequences)
    progress=True                       # Show progress bar
)

# Output as PNG sequence with RGBA (transparent background)
convert_video(
    model,
    input_source='input_frames/',       # Directory with sorted images
    output_type='png_sequence',
    output_composition='output_frames/', # Directory for RGBA PNG output
    downsample_ratio=0.4,
    seq_chunk=8
)
```

--------------------------------

### Perform TensorFlow Inference

Source: https://github.com/peterl1n/robustvideomatting/blob/master/documentation/inference.md

Run inference using a loaded TensorFlow SavedModel, noting that inputs must be in channel-last format.

```python
import tensorflow as tf

model = tf.keras.models.load_model('rvm_mobilenetv3_tf')
model = tf.function(model)

rec = [ tf.constant(0.) ] * 4         # Initial recurrent states.
downsample_ratio = tf.constant(0.25)  # Adjust based on your video.

for src in YOUR_VIDEO:  # src is of shape [B, H, W, C], not [B, C, H, W]!
    out = model([src, *rec, downsample_ratio])
    fgr, pha, *rec = out['fgr'], out['pha'], out['r1o'], out['r2o'], out['r3o'], out['r4o']
```

--------------------------------

### TensorFlow Inference

Source: https://context7.com/peterl1n/robustvideomatting/llms.txt

Loads a TensorFlow SavedModel for integration within the TensorFlow ecosystem. Note that TensorFlow uses channel-last format [B, H, W, C].

```python
import tensorflow as tf

# Load TensorFlow model
model = tf.keras.models.load_model('rvm_mobilenetv3_tf')
model = tf.function(model)

# Initialize recurrent states
rec = [tf.constant(0.)] * 4
downsample_ratio = tf.constant(0.25)

# Inference loop - note: TensorFlow uses channel-last format [B, H, W, C]
for src in video_frames:  # src: [B, H, W, C] not [B, C, H, W]!
    out = model([src, *rec, downsample_ratio])
    fgr = out['fgr']      # [B, H, W, 3]
    pha = out['pha']      # [B, H, W, 1]
    rec = [out['r1o'], out['r2o'], out['r3o'], out['r4o']]

    # Composite with background
    bgr = tf.constant([0.47, 1.0, 0.6])
    composite = fgr * pha + bgr * (1 - pha)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.