### Quick Start Example with DiffuEraser and Propainter

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/README.md

Initializes the DiffuEraser and Propainter models for a quick start. This example sets up the necessary imports and instantiates the core components for basic usage.

```python
#!/usr/bin/env python3

import torch
import os
from diffueraser.diffueraser import DiffuEraser
from propainter.inference import Propainter

```

--------------------------------

### Setup and Model Loading

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/README.md

Initializes the PyTorch device, creates an output directory, and loads the Propainter and DiffuEraser models.

```python
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
os.makedirs('results', exist_ok=True)

# Load models
print("Loading Propainter...")
propainter = Propainter('weights/propainter', device=device)

print("Loading DiffuEraser...")
diffueraser = DiffuEraser(
    device=device,
    base_model_path='weights/stable-diffusion-v1-5',
    vae_path='weights/sd-vae-ft-mse',
    diffueraser_path='weights/diffuEraser',
    ckpt='4-Step'
)
```

--------------------------------

### BrushNetModel Example Usage

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/api-reference/BrushNetModel.md

Demonstrates how to instantiate BrushNetModel from pretrained weights and perform a forward pass with sample conditioning inputs.

```python
import torch
from libs.brushnet_CA import BrushNetModel

brushnet = BrushNetModel.from_pretrained("weights/diffuEraser", subfolder="brushnet")

# Shape: (batch, 5, 64, 64) where 5 = 4 (latent) + 1 (mask)
conditioning = torch.randn(1, 5, 64, 64)
timestep = torch.tensor([500])
encoder_hidden_states = torch.randn(1, 77, 768)  # CLIP text embeddings

output = brushnet(
    conditioning_image=conditioning,
    timestep=timestep,
    encoder_hidden_states=encoder_hidden_states
)

# Access features
down_features = output.down_block_res_samples
mid_feature = output.mid_block_res_sample
up_features = output.up_block_res_samples
```

--------------------------------

### Install Python Dependencies

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/README.md

Install all required Python packages for the DiffuEraser project using the provided requirements.txt file.

```bash
# install python dependencies
pip install -r requirements.txt 
```

--------------------------------

### Executing the DiffuEraser Forward Method

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/api-reference/DiffuEraser.md

Example of how to call the `forward` method to perform video inpainting. Ensure all required paths and parameters are correctly specified.

```python
output_video = diffueraser.forward(
    validation_image="examples/example3/video.mp4",
    validation_mask="examples/example3/mask.mp4",
    priori="results/priori.mp4",
    output_path="results/diffueraser_result.mp4",
    max_img_size=960,
    video_length=10,
    mask_dilation_iter=8,
    guidance_scale=0.0
)
print(f"Output saved to: {output_video}")
```

--------------------------------

### Video Frame Tensor Format Example

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/types.md

Illustrates the shape and example dimensions for RGB video frames, typically normalized to [0, 1] or uint8 [0, 255].

```text
Shape: (batch_frames, color_channels, height, width)
       (T, 3, H, W)
       
Example: (22, 3, 540, 960) for 22 frames of 960x540 video
```

--------------------------------

### Clone DiffuEraser Repository

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/README.md

Clone the DiffuEraser repository to your local machine to get started.

```bash
git clone https://github.com/lixiaowen-xw/DiffuEraser.git
```

--------------------------------

### Example Error Handling for DiffuEraser

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/README.md

Demonstrates how to catch and handle specific `ValueError` exceptions during DiffuEraser's forward pass, providing user-friendly messages for common issues like invalid resolution, FPS mismatch, or insufficient frames.

```python
from diffueraser.diffueraser import DiffuEraser

try:
    diffueraser.forward(
        validation_image="input.mp4",
        validation_mask="mask.mp4",
        priori="priori.mp4",
        output_path="output.mp4",
        max_img_size=960
    )
except ValueError as e:
    if "max_img_size" in str(e):
        print("Invalid resolution - use 256-1920")
    elif "frame rate" in str(e):
        print("Video FPS mismatch - convert to same FPS")
    elif "too short" in str(e):
        print("Video too short - minimum 22 frames needed")
```

--------------------------------

### Initialize StableDiffusionDiffuEraserPipeline

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/api-reference/StableDiffusionDiffuEraserPipeline.md

Initializes the DiffuEraser diffusion pipeline by registering all component models and setting up the image processor. The pipeline integrates text encoding, VAE latent processing, BrushNet conditioning, and video denoising via the motion-aware UNet. This example shows how to load the pipeline from a pre-trained model.

```python
from diffusers import AutoencoderKL, DDPMScheduler
from transformers import AutoTokenizer, CLIPTextModel
from libs.unet_motion_model import UNetMotionModel
from libs.brushnet_CA import BrushNetModel
from diffueraser.pipeline_diffueraser import StableDiffusionDiffuEraserPipeline

pipeline = StableDiffusionDiffuEraserPipeline.from_pretrained(
    "stable-diffusion-v1-5",
    vae=vae,
    text_encoder=text_encoder,
    tokenizer=tokenizer,
    unet=unet,
    brushnet=brushnet,
    scheduler=scheduler,
    safety_checker=safety_checker,
    feature_extractor=feature_extractor
)
```

--------------------------------

### Mask Tensor Format Example

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/types.md

Shows the shape and value range for binary or grayscale masks, where values from 0.0 to 1.0 indicate regions to be inpainted (1.0 being the target).

```text
Shape: (batch_frames, channels, height, width)
       (T, 1, H, W) or (T, 3, H, W)
       
Values: 0.0-1.0 where 1.0 = region to inpaint
```

--------------------------------

### Initialize and Run DiffuEraser Pipeline

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/README.md

Demonstrates the basic usage of the DiffuEraser class for video inpainting. It involves initializing ProPainter for prior generation and then DiffuEraser for refinement, processing input video and mask files.

```python
import torch
from diffueraser.diffueraser import DiffuEraser
from propainter.inference import Propainter

device = torch.device('cuda:0')

# Initialize models
propainter = Propainter("weights/propainter", device=device)
diffueraser = DiffuEraser(
    device=device,
    base_model_path="weights/stable-diffusion-v1-5",
    vae_path="weights/sd-vae-ft-mse",
    diffueraser_path="weights/diffuEraser",
    ckpt="4-Step"
)

# Stage 1: Generate prior with ProPainter
priori_path = propainter.forward(
    video="input.mp4",
    mask="mask.mp4",
    output_path="results/priori.mp4",
    video_length=10,
    mask_dilation=8
)

# Stage 2: Refine with DiffuEraser
output = diffueraser.forward(
    validation_image="input.mp4",
    validation_mask="mask.mp4",
    priori=priori_path,
    output_path="results/output.mp4",
    max_img_size=960,
    video_length=10
)
```

--------------------------------

### Initialize DiffuEraser Pipeline

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/api-reference/DiffuEraser.md

Instantiate the DiffuEraser class to set up the video inpainting pipeline. Specify the target device, paths to base Stable Diffusion models, VAE, and DiffuEraser checkpoints. The `ckpt` parameter selects the inference step configuration.

```python
import torch
from diffueraser.diffueraser import DiffuEraser

device = torch.device('cuda:0')
diffueraser = DiffuEraser(
    device=device,
    base_model_path="weights/stable-diffusion-v1-5",
    vae_path="weights/sd-vae-ft-mse",
    diffueraser_path="weights/diffuEraser",
    ckpt="4-Step"
)
```

--------------------------------

### Initialize DiffuEraser with Different Checkpoints

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/README.md

Shows how to select different inference speed/quality trade-offs by specifying the 'ckpt' parameter during DiffuEraser initialization. Options range from fastest (2-Step) to moderate (8-Step).

```python
# Fast inference (2-4 steps)
diffueraser = DiffuEraser(..., ckpt="2-Step")      # Fastest
diffueraser = DiffuEraser(..., ckpt="4-Step")      # Very fast

# Balanced (4-8 steps)
diffueraser = DiffuEraser(..., ckpt="Normal CFG 4-Step")  # Fast + guidance
diffueraser = DiffuEraser(..., ckpt="8-Step")       # Moderate
```

--------------------------------

### Loading and Saving BrushNetModel

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/api-reference/BrushNetModel.md

Shows how to load a BrushNetModel from pretrained weights and save it to disk. Also demonstrates loading a model from its configuration.

```python
# Load from pretrained weights
brushnet = BrushNetModel.from_pretrained(
    "weights/diffuEraser",
    subfolder="brushnet"
)

# Save to diskrushnet.save_pretrained("path/to/save", subfolder="brushnet")

# Load from config
from diffusers.utils import ConfigMixin
config = ConfigMixin.load_config("path/to/config.json")
brushnet = BrushNetModel.from_config(config)
```

--------------------------------

### Initialize DiffuEraser with Checkpoint

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/README.md

Instantiate the DiffuEraser model, specifying the checkpoint for high-quality results. This configuration is noted as slower but yields the best quality.

```python
diffueraser = DiffuEraser(..., ckpt="Normal CFG 16-Step") # Slow but best
```

--------------------------------

### Text Embedding Tensor Format Example

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/types.md

Details the shape for text embeddings generated by the CLIP text encoder, with a fixed sequence length of 77 tokens and an embedding dimension of 768.

```text
Shape: (batch, sequence_length, embedding_dim)
       (N, 77, 768) for CLIP text encoder
       
Example: (1, 77, 768) for single prompt
```

--------------------------------

### Propainter.forward

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/api-reference/Propainter.md

Generates prior/reference frames for video inpainting using temporal propagation guided by optical flow. This method processes input video and mask data to produce coherent reference frames for a diffusion-based inpainting model.

```APIDOC
## Method: forward

### Description
Generates prior/reference frames for video inpainting using temporal propagation guided by optical flow. The workflow includes optical flow estimation, flow completion in masked regions, frame propagation for pre-inpainting, and final propagation with updated masks. The output provides coherent reference frames that guide the diffusion-based DiffuEraser model.

### Parameters
#### Path Parameters
- **video** (str) - Required - Path to input video file (MP4, MOV, AVI) or directory containing frame images. Must have consistent frame rate with mask.
- **mask** (str) - Required - Path to mask video (MP4, MOV, AVI), mask image (PNG, JPG), or directory containing mask frames. White regions indicate areas to inpaint.
- **output_path** (str) - Required - Path to save the output prior video file in MP4 format.

#### Query Parameters
- **resize_ratio** (float) - Optional - Default: 0.6 - Resize ratio for input video (0.0-1.0). Smaller values reduce computation but may affect quality. Automatically adjusted if longer edge exceeds 960 pixels.
- **video_length** (float) - Optional - Default: 2 - Maximum video duration in seconds. Limits number of frames loaded from input video.
- **height** (int) - Optional - Default: -1 - Target output height in pixels. -1 means use original or resized height. Must be divisible by 8.
- **width** (int) - Optional - Default: -1 - Target output width in pixels. -1 means use original or resized width. Must be divisible by 8.
- **mask_dilation** (int) - Optional - Default: 4 - Dilation iterations for mask expansion (0-20). Larger values expand the inpainting region more.
- **ref_stride** (int) - Optional - Default: 10 - Stride for selecting reference frames during temporal propagation (1-20). Larger values use fewer reference frames.
- **neighbor_length** (int) - Optional - Default: 10 - Number of neighbor frames to consider (4-20). Larger values improve temporal consistency but increase computation.
- **subvideo_length** (int) - Optional - Default: 80 - Batch size for processing frames in subvideos (20-200). Affects GPU memory usage.
- **raft_iter** (int) - Optional - Default: 20 - Number of RAFT iterations for optical flow estimation (10-50). More iterations improve flow quality but increase time.
- **save_fps** (int) - Optional - Default: 24 - Frame rate (fps) for output video file. If input video has valid fps, input fps is used instead.
- **save_frames** (bool) - Optional - Default: False - Whether to also save individual frame files in output directory.
- **fp16** (bool) - Optional - Default: True - Whether to use FP16 precision for faster inference with less GPU memory. Automatically disabled on CPU.

### Returns
- **str** - Path to the generated prior video file

### Raises
- **ValueError** - If frame rate of mask doesn't match input video
- **RuntimeError** - If video cannot be opened or read

### Example
```python
propainter = Propainter("weights/propainter", device)

priori_path = propainter.forward(
    video="examples/example3/video.mp4",
    mask="examples/example3/mask.mp4",
    output_path="results/priori.mp4",
    video_length=10,
    mask_dilation=8,
    neighbor_length=10,
    ref_stride=10,
    subvideo_length=50,
    save_fps=25
)
```
```

--------------------------------

### Initialize Propainter Model

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/api-reference/Propainter.md

Initializes the Propainter prior generation model. Ensure the specified directory contains the necessary model weights or they will be downloaded from HuggingFace.

```python
import torch
from propainter.inference import Propainter

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
propainter = Propainter(
    propainter_model_dir="weights/propainter",
    device=device
)
```

--------------------------------

### Stage 1: Generate Prior with Propainter

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/README.md

Generates a prior video using the Propainter model. This step requires input video and mask files, along with several parameters controlling the generation process.

```python
# Stage 1: Generate prior
print("Generating prior with ProPainter...")
priori = propainter.forward(
    video='examples/video.mp4',
    mask='examples/mask.mp4',
    output_path='results/priori.mp4',
    video_length=10,
    mask_dilation=8,
    ref_stride=10,
    neighbor_length=10,
    subvideo_length=50
)
```

--------------------------------

### Latent Tensor Format Example

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/types.md

Illustrates the standard format for latent tensors in diffusion models. The shape is (batch_frames, channels, height, width), typically (T, 4, H, W), where H and W are 1/8 of the original image dimensions.

```text
Shape: (batch_frames, channels, height, width)
       (T, 4, H, W)       
Example: (22, 4, 64, 64) for 22 video frames in latent space
```

--------------------------------

### Propainter Class Initialization

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/architecture.md

Details the initialization process for the Propainter class, including loading optical flow and inpainting models.

```python
Propainter
├── __init__()
│   ├── Load RAFT_bi (optical flow)
│   ├── Load RecurrentFlowCompleteNet (flow inpainting)
│   └── Load InpaintGenerator (frame propagation)
```

--------------------------------

### Run DiffuEraser Inference

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/README.md

Navigate to the DiffuEraser directory and run the main inference script. Results are saved in the 'results' folder. Ensure input video and mask have consistent frame rates.

```shell
cd DiffuEraser
python run_diffueraser.py
```

--------------------------------

### Train and Evaluate Stage 1 Model

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/README.md

Scripts for training, saving checkpoints, and evaluating the first stage of the DiffuEraser model.

```bash
# train
sh train_DiffuEraser_stage1.sh
# save checkpoint
python save_checkpoint_stage1.py
# eval
python eval_DiffuEraser_stage1.py
```

--------------------------------

### Inference Configuration Variants: Checkpoint Selection

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/architecture.md

Illustrates the trade-offs between different checkpoints in terms of steps, guidance, and speed/quality for inference.

```text
┌────────────────────┬──────────┬──────────┬─────────────┐
│ Checkpoint         │ Steps    │ Guidance │ Speed/Quality│
├────────────────────┼──────────┼──────────┼─────────────┤
│ 2-Step             │ 2        │ 0.0      │ Fastest     │
│ 4-Step             │ 4        │ 0.0      │ Very fast   │
│ 8-Step             │ 8        │ 0.0      │ Fast        │
│ 16-Step            │ 16       │ 0.0      │ Moderate    │
│ Normal CFG 4-Step  │ 4        │ 7.5      │ Fast        │
│ Normal CFG 8-Step  │ 8        │ 7.5      │ Moderate    │
│ Normal CFG 16-Step │ 16       │ 7.5      │ Slow        │
│ LCM-Like LoRA      │ 4        │ 0.0      │ Very fast   │
└────────────────────┴──────────┴──────────┴─────────────┘
```

--------------------------------

### DiffuEraser Class Initialization

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/architecture.md

Details the initialization process for the DiffuEraser class, including loading models and creating the pipeline.

```python
DiffuEraser
├── __init__()
│   ├── Load VAE (AutoencoderKL)
│   ├── Load Tokenizer (CLIPTokenizer)
│   ├── Load Text Encoder (CLIPTextModel)
│   ├── Load UNet Motion (UNetMotionModel)
│   ├── Load BrushNet (BrushNetModel)
│   ├── Create Pipeline (StableDiffusionDiffuEraserPipeline)
│   └── Load LoRA weights (PCM checkpoints)
```

--------------------------------

### Inference Stage Selection Logic

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/architecture.md

Explains the two main inference stages: pre-inference for long videos and frame-by-frame refinement for all videos.

```text
Pre-inference (when len(frames) > 44):
├─ Purpose: Refine heavily masked regions
├─ Applies to: Sampled frames (every Nth frame)
├─ Updates: Masks to black (region solved)
└─ Benefit: Better temporal coherence for long videos

Frame-by-frame (always):
├─ Purpose: Final refinement of all frames
├─ Applies to: All frames with remaining masks
├─ Uses: Updated masks from pre-inference
└─ Benefit: Ensures smooth final output
```

--------------------------------

### Load UNetMotionModel from Configuration

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/api-reference/UNetMotionModel.md

Instantiate a UNetMotionModel directly from a configuration dictionary. This is useful when not loading from pretrained weights.

```python
unet = UNetMotionModel.from_config(config_dict)
```

--------------------------------

### Train and Evaluate Stage 2 Model

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/README.md

Scripts for training, saving checkpoints, and evaluating the second stage of the DiffuEraser model. Ensure stage1 weights are specified.

```bash
# train
sh train_DiffuEraser_stage2.sh
# save checkpoint
python save_checkpoint_stage2.py
# eval
python eval_DiffuEraser_stage2.py
```

--------------------------------

### Load Pretrained Model

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/configuration.md

Standard method for loading a model from a pretrained path. Specify subfolder, data type, and device mapping.

```python
model = ModelClass.from_pretrained(
    pretrained_model_name_or_path,
    subfolder="subfolder_name",
    torch_dtype=torch.float16,
    device_map="cuda"
)
```

--------------------------------

### Propainter Constructor

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/api-reference/Propainter.md

Initializes the ProPainter prior generation model. It loads RAFT-bi for optical flow estimation, RecurrentFlowCompleteNet for flow inpainting, and InpaintGenerator for frame propagation. Models are automatically downloaded if not found locally.

```APIDOC
## Propainter Constructor

### Description
Initializes the ProPainter prior generation model by loading three components: (1) RAFT-bi for bidirectional optical flow estimation, (2) RecurrentFlowCompleteNet for flow inpainting in masked regions, and (3) InpaintGenerator for frame-wise propagation. Models are automatically downloaded from the official repository if not found locally.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Constructor Signature
```python
def __init__(
    self,
    propainter_model_dir,
    device
)
```

### Parameters
- **propainter_model_dir** (str) - Required - Directory path containing ProPainter model weights. Will download pretrained models from HuggingFace if not found locally. Expected files: raft-things.pth, recurrent_flow_completion.pth, ProPainter.pth
- **device** (torch.device) - Required - Target device for inference (e.g., torch.device('cuda:0'), torch.device('cpu'))

### Returns
`Propainter` instance

### Example
```python
import torch
from propainter.inference import Propainter

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
propainter = Propainter(
    propainter_model_dir="weights/propainter",
    device=device
)
```
```

--------------------------------

### Propainter Forward Method

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/api-reference/Propainter.md

Generates prior/reference frames for video inpainting. Use this method to process input videos and masks, specifying various parameters to control the inpainting process and output quality. Ensure input video and mask have consistent frame rates.

```python
def forward(
    self,
    video,
    mask,
    output_path,
    resize_ratio=0.6,
    video_length=2,
    height=-1,
    width=-1,
    mask_dilation=4,
    ref_stride=10,
    neighbor_length=10,
    subvideo_length=80,
    raft_iter=20,
    save_fps=24,
    save_frames=False,
    fp16=True
)

```

```python
propainter = Propainter("weights/propainter", device)

priori_path = propainter.forward(
    video="examples/example3/video.mp4",
    mask="examples/example3/mask.mp4",
    output_path="results/priori.mp4",
    video_length=10,
    mask_dilation=8,
    neighbor_length=10,
    ref_stride=10,
    subvideo_length=50,
    save_fps=25
)

```

--------------------------------

### Load UNetMotionModel from Pretrained Weights

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/api-reference/UNetMotionModel.md

Load a UNetMotionModel from a specified pretrained directory. Ensure the 'weights/diffuEraser' path and 'unet_main' subfolder are correct.

```python
unet = UNetMotionModel.from_pretrained(
    "weights/diffuEraser",
    subfolder="unet_main"
)
```

--------------------------------

### Initialize UNetMotionModel

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/api-reference/UNetMotionModel.md

Initializes the UNetMotionModel with specified parameters for video inpainting. This constructor sets up the network architecture, including downsampling, middle, and upsampling blocks with integrated spatial and temporal attention mechanisms.

```python
def __init__(
    self,
    sample_size=None,
    in_channels=4,
    conditioning_channels=3,
    out_channels=4,
    down_block_types=("CrossAttnDownBlockMotion", "CrossAttnDownBlockMotion", "CrossAttnDownBlockMotion", "DownBlockMotion"),
    mid_block_type="UNetMidBlockCrossAttnMotion",
    up_block_types=("UpBlockMotion", "CrossAttnUpBlockMotion", "CrossAttnUpBlockMotion", "CrossAttnUpBlockMotion"),
    block_out_channels=(320, 640, 1280, 1280),
    layers_per_block=2,
    downsample_padding=1,
    mid_block_scale_factor=1,
    act_fn="silu",
    norm_num_groups=32,
    norm_eps=1e-5,
    cross_attention_dim=1280,
    use_linear_projection=False,
    num_attention_heads=8,
    motion_max_seq_length=32,
    motion_num_attention_heads=8,
    use_motion_mid_block=True,
    encoder_hid_dim=None,
    encoder_hid_dim_type=None,
    conditioning_embedding_out_channels=(16, 32, 96, 256)
):
    pass
```

--------------------------------

### Quality-Optimized DiffuEraser Configuration

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/configuration.md

This configuration is for achieving the best quality output when ample VRAM is available. It utilizes more steps, explicit guidance, and higher resolution.

```python
diffueraser = DiffuEraser(
    device=device,
    base_model_path="weights/stable-diffusion-v1-5",
    vae_path="weights/sd-vae-ft-mse",
    diffueraser_path="weights/diffuEraser",
    ckpt="Normal CFG 8-Step"  # More steps + guidance
)

output = diffueraser.forward(
    validation_image="video.mp4",
    validation_mask="mask.mp4",
    priori="priori.mp4",
    output_path="result.mp4",
    max_img_size=1280,    # Full resolution
    nframes=22,           # Standard batch
    mask_dilation_iter=8, # Better mask expansion
    guidance_scale=7.5    # Explicit guidance
)
```

--------------------------------

### BrushNetModel Initialization

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/api-reference/BrushNetModel.md

Initializes the BrushNetModel with a comprehensive set of parameters that define its architecture. This includes settings for input/output channels, downsampling and upsampling block types, attention mechanisms, and normalization configurations. Use this to create an instance of the model tailored to specific conditioning inputs and architectural preferences.

```python
def __init__(
    self,
    in_channels=4,
    conditioning_channels=5,
    flip_sin_to_cos=True,
    freq_shift=0,
    down_block_types=("CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "DownBlock2D"),
    mid_block_type="UNetMidBlock2DCrossAttn",
    up_block_types=("UpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D"),
    only_cross_attention=False,
    block_out_channels=(320, 640, 1280, 1280),
    layers_per_block=2,
    downsample_padding=1,
    mid_block_scale_factor=1,
    act_fn="silu",
    norm_num_groups=32,
    norm_eps=1e-5,
    cross_attention_dim=1280,
    transformer_layers_per_block=1,
    encoder_hid_dim=None,
    encoder_hid_dim_type=None,
    attention_head_dim=8,
    use_linear_projection=False,
    class_embed_type=None,
    addition_embed_type=None,
    num_class_embeds=0,
    upcast_attention=False,
    resnet_time_scale_shift="default",
    projection_class_embeddings_input_dim=None,
    brushnet_conditioning_channel_order="rgb",
    conditioning_embedding_out_channels=(16, 32, 96, 256),
    global_pool_conditions=False,
    addition_embed_type_num_heads=64
)

```

--------------------------------

### Convert Frames to Video using FFmpeg

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/README.md

Use FFmpeg to convert a sequence of image frames into an MP4 video. This is useful for preparing input videos for the model if you have split frames.

```shell
ffmpeg -i image%03d.jpg -c:v libx264 -r 25 output.mp4
```

--------------------------------

### Use String or List for Prompt

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/errors.md

The 'prompt' parameter must be either a string or a list of strings. Using other data types like dictionaries or tensors will result in an error.

```python
# Use string or list of strings
pipeline(
    prompt="a person",  # str ✓
    ...
)

# Or list
pipeline(
    prompt=["a person", "a dog"],  # list[str] ✓
    ...
)

# Not dict
pipeline(
    prompt={"text": "a person"},  # ✗ Wrong type
    ...
)
```

--------------------------------

### Verify Video File Existence and Format

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/errors.md

Ensure the video file exists at the specified path and is in a supported format (MP4, MOV, AVI). If not, convert it using tools like ffmpeg.

```python
import os
assert os.path.exists("mask.mp4"), "Mask file not found"

propainter.forward(
    ...,
    mask="mask.mp4",  # Use supported format
    ...
)
```

```bash
# ffmpeg -i mask.avi -c:v libx264 mask.mp4
```

--------------------------------

### DiffuEraser Constructor

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/api-reference/DiffuEraser.md

Initializes the DiffuEraser inference pipeline by loading necessary models and configuring the diffusion process. This constructor sets up the complete pipeline for video inpainting.

```APIDOC
## DiffuEraser Constructor

### Description
Initializes the DiffuEraser inference pipeline by loading the VAE, text encoder, tokenizer, UNet with motion adapter, and BrushNet. Configures the scheduler based on the checkpoint type and loads PCM LoRA weights. The constructor sets up the complete diffusion pipeline with support for multiple inference step configurations.

### Method
__init__

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **device** (torch.device) - Required - Target device for model inference (e.g., 'cuda:0', 'cpu')
- **base_model_path** (str) - Required - Path to Stable Diffusion v1.5 base model directory containing feature_extractor, model_index.json, safety_checker, scheduler, text_encoder, tokenizer, and unet folders
- **vae_path** (str) - Required - Path to VAE model directory (e.g., sd-vae-ft-mse)
- **diffueraser_path** (str) - Required - Path to DiffuEraser checkpoint directory containing brushnet and unet_main subfolders
- **revision** (str) - Optional - Revision identifier for model loading (rarely used)
- **ckpt** (str) - Optional - Checkpoint configuration key. Valid values: "2-Step", "4-Step", "8-Step", "16-Step", "Normal CFG 4-Step", "Normal CFG 8-Step", "Normal CFG 16-Step", "LCM-Like LoRA"
- **mode** (str) - Optional - Model mode specification (default is Stable Diffusion 1.5)
- **loaded** (str) - Optional - Internal state tracker for loaded checkpoint (used to avoid reloading the same checkpoint)

### Request Example
```python
import torch
from diffueraser.diffueraser import DiffuEraser

device = torch.device('cuda:0')
diffueraser = DiffuEraser(
    device=device,
    base_model_path="weights/stable-diffusion-v1-5",
    vae_path="weights/sd-vae-ft-mse",
    diffueraser_path="weights/diffuEraser",
    ckpt="4-Step"
)
```

### Response
#### Success Response
Returns an initialized `DiffuEraser` instance.

#### Response Example
None
```

--------------------------------

### prepare_extra_step_kwargs

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/api-reference/StableDiffusionDiffuEraserPipeline.md

Prepares optional parameters for the scheduler step function. Adds eta and generator parameters only if the scheduler accepts them, ensuring compatibility across different scheduler types.

```APIDOC
## prepare_extra_step_kwargs

### Description
Prepares optional parameters for the scheduler step function. Adds eta and generator parameters only if the scheduler accepts them, ensuring compatibility across different scheduler types.

### Method Signature
```python
def prepare_extra_step_kwargs(self, generator, eta)
```

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **generator** (torch.Generator) - Yes - — - Random number generator for reproducibility
- **eta** (float) - Yes - — - Eta parameter for DDIM scheduler (0.0-1.0). Ignored for other schedulers

### Response
#### Success Response
- **kwargs** (Dict[str, Any]) - Dictionary of kwargs for scheduler.step()

### Example
None
```

--------------------------------

### DiffuEraser Checkpoint Configuration Dictionary

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/types.md

Defines the structure for DiffuEraser checkpoints, mapping descriptive keys to filename patterns, number of inference steps, and default guidance scales.

```python
checkpoints = {
    "2-Step": ["pcm_{}_smallcfg_2step_converted.safetensors", 2, 0.0],
    "4-Step": ["pcm_{}_smallcfg_4step_converted.safetensors", 4, 0.0],
    "8-Step": ["pcm_{}_smallcfg_8step_converted.safetensors", 8, 0.0],
    "16-Step": ["pcm_{}_smallcfg_16step_converted.safetensors", 16, 0.0],
    "Normal CFG 4-Step": ["pcm_{}_normalcfg_4step_converted.safetensors", 4, 7.5],
    "Normal CFG 8-Step": ["pcm_{}_normalcfg_8step_converted.safetensors", 8, 7.5],
    "Normal CFG 16-Step": ["pcm_{}_normalcfg_16step_converted.safetensors", 16, 7.5],
    "LCM-Like LoRA": ["pcm_{}_lcmlike_lora_converted.safetensors", 4, 0.0],
}
```

--------------------------------

### Provide Either Prompt or Prompt Embeds

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/errors.md

When calling the pipeline, provide either the 'prompt' parameter or the 'prompt_embeds' parameter, but not both. This ensures clarity in the input instructions.

```python
# Provide only prompt or prompt_embeds, not both
pipeline(
    prompt="a person",           # Use this
    prompt_embeds=None,          # Not this
    ...
)

# Or pre-compute embeddings but don't provide prompt:
pipeline(
    prompt=None,
    prompt_embeds=computed_embeds,  # Use this
    ...
)
```

--------------------------------

### StableDiffusionDiffuEraserPipeline Constructor

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/api-reference/StableDiffusionDiffuEraserPipeline.md

Initializes the DiffuEraser diffusion pipeline by registering all component models and setting up the image processor. The pipeline integrates text encoding, VAE latent processing, BrushNet conditioning, and video denoising via the motion-aware UNet.

```APIDOC
## StableDiffusionDiffuEraserPipeline Constructor

### Description
Initializes the DiffuEraser diffusion pipeline by registering all component models and setting up the image processor. The pipeline integrates text encoding, VAE latent processing, BrushNet conditioning, and video denoising via the motion-aware UNet.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **vae** (AutoencoderKL) - Required - Variational Auto-Encoder for latent space encoding/decoding
- **text_encoder** (CLIPTextModel) - Required - CLIP text encoder for prompt embeddings
- **tokenizer** (CLIPTokenizer) - Required - Tokenizer for converting text to token ids
- **unet** (UNet2DConditionModel) - Required - Main denoising UNet with motion adapter for video denoising
- **brushnet** (BrushNetModel) - Required - Auxiliary BrushNet for conditioning via masks and inpainting guidance
- **scheduler** (KarrasDiffusionSchedulers) - Required - Noise scheduler (e.g., DDPMScheduler, UniPCMultistepScheduler, TCDScheduler)
- **safety_checker** (StableDiffusionSafetyChecker) - Required - Safety checker for filtering NSFW content. Can be None to disable
- **feature_extractor** (CLIPImageProcessor) - Required - Feature extractor for safety checker. Required if safety_checker is not None
- **image_encoder** (CLIPVisionModelWithProjection) - Optional - Optional CLIP vision encoder for IP-Adapter support
- **requires_safety_checker** (bool) - Optional - Default: True - Whether safety checker is required. If True and safety_checker is None, raises error

### Request Example
```python
from diffusers import AutoencoderKL, DDPMScheduler
from transformers import AutoTokenizer, CLIPTextModel
from libs.unet_motion_model import UNetMotionModel
from libs.brushnet_CA import BrushNetModel
from diffueraser.pipeline_diffueraser import StableDiffusionDiffuEraserPipeline

pipeline = StableDiffusionDiffuEraserPipeline.from_pretrained(
    "stable-diffusion-v1-5",
    vae=vae,
    text_encoder=text_encoder,
    tokenizer=tokenizer,
    unet=unet,
    brushnet=brushnet,
    scheduler=scheduler,
    safety_checker=safety_checker,
    feature_extractor=feature_extractor
)
```

### Response
#### Success Response (200)
Returns an instance of `StableDiffusionDiffuEraserPipeline`.

#### Response Example
None
```

--------------------------------

### ProPainter Pipeline Data Flow

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/architecture.md

Diagram illustrating the data flow for the ProPainter pipeline (Stage 1), from input video to prior video.

```text
Input Video (MP4/frames)
    ↓
[read_frame_from_videos] → Frames, FPS, Size
    ↓
[read_mask] → Flow masks, Regular masks
    ↓
Resizing & validation
    ↓
Optical Flow Estimation
    ├→ RAFT_bi → Forward & Backward flows
    └→ RecurrentFlowCompleteNet → Completed flows
    ↓
Pre-propagation (optional)
    ├→ Sample frames
    ├→ Recompute flows on sampled frames
    ├→ InpaintGenerator.img_propagation → Propagated frames
    └→ Update original frames with propagated results
    ↓
Frame-by-frame propagation
    ├→ InpaintGenerator.img_propagation → Propagated frames
    └→ Iterative inpainting
    ↓
Composition & blending
    ↓
MP4 Video Writing
    ↓
Prior Video (MP4)
```

--------------------------------

### Provide Prompt or Prompt Embeds

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/errors.md

Ensure that at least one of the 'prompt' or 'prompt_embeds' parameters is provided when calling the pipeline. This is necessary for generating meaningful output.

```python
# Provide at least one of prompt or prompt_embeds
pipeline(
    prompt="a person",  # Required
    images=images,
    masks=masks,
    ...
)
```

--------------------------------

### Stage 2: Refine with DiffuEraser

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/README.md

Refines the generated prior video using the DiffuEraser model. This stage takes the prior, validation image/mask, and output path as inputs, with parameters to control image size and other aspects of the refinement.

```python
# Stage 2: Refine with DiffuEraser
print("Refining with DiffuEraser...")
output = diffueraser.forward(
    validation_image='examples/video.mp4',
    validation_mask='examples/mask.mp4',
    priori=priori,
    output_path='results/output.mp4',
    max_img_size=960,
    video_length=10,
    mask_dilation_iter=8,
    guidance_scale=0.0
)

print(f"Done! Output saved to {output}")
```

--------------------------------

### Load LoRA Weights

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/configuration.md

Loads LoRA weights into a pipeline. Specify the path to the weights, the specific weight file name, and the subfolder.

```python
pipeline.load_lora_weights(
    "weights/PCM_Weights",
    weight_name="pcm_sd15_smallcfg_2step_converted.safetensors",
    subfolder="sd15"
)
```

--------------------------------

### DiffuEraser.forward

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/api-reference/DiffuEraser.md

Executes the complete video inpainting pipeline. It reads input video and mask, loads prior results, performs optional pre-inference, and then frame-by-frame inpainting with temporal consistency. Results can optionally be blended with the original image.

```APIDOC
## Method: forward

### Description
Executes the complete video inpainting pipeline. Reads input video and mask, loads prior results from ProPainter, performs optional pre-inference on sampled frames for longer sequences, and then frame-by-frame inpainting with temporal consistency. Optionally blends results with original image using blurred masks.

### Method Signature
```python
def forward(
    self,
    validation_image: str,
    validation_mask: str,
    priori: str,
    output_path: str,
    max_img_size: int = 1280,
    video_length: float = 2,
    mask_dilation_iter: int = 4,
    nframes: int = 22,
    seed: Optional[int] = None,
    revision: Optional[str] = None,
    guidance_scale: Optional[float] = None,
    blended: bool = True
)
```

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

#### Method Parameters
- **validation_image** (str) - Required - Path to input video file (MP4 format). Frame rate and resolution must match the mask video. Resolution must be between 256x256 and 4096x4096.
- **validation_mask** (str) - Required - Path to mask video file (MP4 format) or mask image. White regions (>0) indicate areas to inpaint. Must have same frame rate as input video.
- **priori** (str) - Required - Path to prior/reference video file generated by ProPainter. Deleted after reading. Must have same frame rate as input video.
- **output_path** (str) - Required - Path to save the output inpainted video file (MP4 format).
- **max_img_size** (int) - Optional - Maximum dimension (width or height) in pixels. Valid range: 256-1920. Video will be downscaled if larger. Affects GPU memory requirement. Defaults to 1280.
- **video_length** (float) - Optional - Maximum video duration in seconds. Used to limit the number of frames loaded from input video. Defaults to 2.
- **mask_dilation_iter** (int) - Optional - Number of dilation iterations to expand the mask region. Larger values expand the inpainting area more. Defaults to 4.
- **nframes** (int) - Optional - Number of frames per diffusion batch. Larger values improve temporal consistency but require more GPU memory. Defaults to 22.
- **seed** (int) - Optional - Random seed for reproducible results. If None, uses random seed. Defaults to None.
- **revision** (str) - Optional - Model revision parameter (rarely used). Defaults to None.
- **guidance_scale** (float) - Optional - Classifier-free guidance scale. Overrides the default from checkpoint if provided. 0 means no guidance. Defaults to None.
- **blended** (bool) - Optional - Whether to blend output with original image using blurred masks for smoother transitions. Defaults to True.

### Request Example
```python
output_video = diffueraser.forward(
    validation_image="examples/example3/video.mp4",
    validation_mask="examples/example3/mask.mp4",
    priori="results/priori.mp4",
    output_path="results/diffueraser_result.mp4",
    max_img_size=960,
    video_length=10,
    mask_dilation_iter=8,
    guidance_scale=0.0
)
print(f"Output saved to: {output_video}")
```

### Response
#### Success Response
- **str** - Path to the output video file.

#### Response Example
```json
"results/diffueraser_result.mp4"
```

### Error Handling
- `ValueError` - If max_img_size is not in range [256, 1920]
- `ValueError` - If frame rate of mask video doesn't match input video
- `ValueError` - If effective video duration is less than 22 frames
- `ValueError` - If video resolution is less than 256x256 or greater than 4096x4096
```

--------------------------------

### ProPainter Module Structure

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/architecture.md

Module structure for ProPainter, covering inference, model implementations, and optical flow estimation.

```python
propainter/
├── inference.py            # Propainter inference class
├── model/                  # Model implementations
│   ├── propainter.py
│   ├── modules/
│   │   └── flow_comp_raft.py
│   └── recurrent_flow_completion.py
└── RAFT/                   # Optical flow estimation
    ├── raft.py
    ├── extractor.py
    └── update.py
```

--------------------------------

### Match Prompt and Negative Prompt Types

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/errors.md

Ensure that the 'negative_prompt' parameter has the same data type as the 'prompt' parameter. Mismatched types, such as a string prompt with a list negative prompt, will cause an error.

```python
# Match types
pipeline(
    prompt="a person",
    negative_prompt="blurry",  # Same type (str) ✓
    ...
)

# Or use lists for both
pipeline(
    prompt=["a person", "a dog"],
    negative_prompt=["blurry", "low quality"],  # Both list ✓
    ...
)

# Not mixed
pipeline(
    prompt="a person",
    negative_prompt=["blurry"],  # ✗ Type mismatch
    ...
)
```

--------------------------------

### Memory-Optimized DiffuEraser Configuration

Source: https://github.com/lixiaowen-xw/diffueraser/blob/master/_autodocs/configuration.md

Use this configuration for GPUs with limited memory (e.g., 12-16GB). It reduces memory consumption by using fewer steps and a smaller image size.

```python
diffueraser = DiffuEraser(
    device=device,
    base_model_path="weights/stable-diffusion-v1-5",
    vae_path="weights/sd-vae-ft-mse",
    diffueraser_path="weights/diffuEraser",
    ckpt="2-Step"  # Fewer steps = less memory
)

output = diffueraser.forward(
    validation_image="video.mp4",
    validation_mask="mask.mp4",
    priori="priori.mp4",
    output_path="result.mp4",
    max_img_size=640,  # Smaller resolution
    nframes=16,        # Smaller batch
    mask_dilation_iter=4
)
```