### U-Net Training Configuration Example (YAML)

Source: https://context7.com/bytedance/latentsync/llms.txt

An example YAML configuration file for U-Net training, detailing data paths, checkpoint settings, training parameters, and optimizer configurations.

```yaml
data:
  syncnet_config_path: configs/syncnet/syncnet_16_pixel_attn.yaml
  train_output_dir: debug/unet
  train_fileslist: /path/to/fileslist.txt
  val_video_path: assets/demo1_video.mp4
  val_audio_path: assets/demo1_audio.wav
  batch_size: 1
  num_workers: 12
  num_frames: 16
  resolution: 256
  mask_image_path: latentsync/utils/mask.png

ckpt:
  resume_ckpt_path: checkpoints/latentsync_unet.pt
  save_ckpt_steps: 10000

run:
  pixel_space_supervise: true
  use_syncnet: true
  sync_loss_weight: 0.05
  perceptual_loss_weight: 0.1
  recon_loss_weight: 1
  trepa_loss_weight: 10
  guidance_scale: 1.5
  inference_steps: 20
  trainable_modules:
    - motion_modules.
    - attentions.
  seed: 1247
  mixed_precision_training: true
  enable_gradient_checkpointing: true
  max_train_steps: 10000000

optimizer:
  lr: 1e-5
  max_grad_norm: 1.0
  lr_scheduler: constant
```

--------------------------------

### SyncNet Configuration Example (YAML)

Source: https://context7.com/bytedance/latentsync/llms.txt

An example YAML configuration file for SyncNet training, defining audio and visual encoder architectures, checkpoint settings, data parameters, and optimizer configurations.

```yaml
# SyncNet configuration (configs/syncnet/syncnet_16_pixel_attn.yaml)
model:
  audio_encoder:
    in_channels: 1
    block_out_channels: [32, 64, 128, 256, 512, 1024, 2048]
    downsample_factors: [[2, 1], 2, 2, 1, 2, 2, [2, 3]]
    attn_blocks: [0, 0, 0, 1, 1, 0, 0]
    dropout: 0.0
  visual_encoder:
    in_channels: 48      # 16 frames * 3 channels
    block_out_channels: [64, 128, 256, 256, 512, 1024, 2048, 2048]
    downsample_factors: [[1, 2], 2, 2, 2, 2, 2, 2, 2]
    attn_blocks: [0, 0, 0, 0, 1, 1, 0, 0]
    dropout: 0.0

ckpt:
  resume_ckpt_path: ""
  inference_ckpt_path: checkpoints/stable_syncnet.pt
  save_ckpt_steps: 2500

data:
  train_output_dir: debug/syncnet
  batch_size: 256
  num_workers: 12
  latent_space: false
  num_frames: 16
  resolution: 256
  lower_half: true        # Use only lower half of face for sync

optimizer:
  lr: 1e-5
  max_grad_norm: 1.0

run:
  max_train_steps: 10000000
  validation_steps: 2500
  mixed_precision_training: true
  seed: 42
```

--------------------------------

### Train U-Net Model

Source: https://github.com/bytedance/latentsync/blob/main/README.md

Initiates the training process for the U-Net model using specified configuration files. Users must ensure data is pre-processed before starting.

```bash
./train_unet.sh
```

--------------------------------

### Cog Predictor for LatentSync Deployment

Source: https://context7.com/bytedance/latentsync/llms.txt

Implements a Cog predictor for deploying LatentSync as a cloud service. It handles model setup, including downloading weights and setting up symlinks, and defines the prediction interface with parameters for video, audio, guidance scale, inference steps, and seed.

```python
# predict.py - Cog Predictor implementation
from cog import BasePredictor, Input, Path
import os
import subprocess

class Predictor(BasePredictor):
    def setup(self) -> None:
        """Load the model into memory"""
        # Download weights if not present
        if not os.path.exists("checkpoints"):
            subprocess.check_call([
                "pget", "-xf",
                "https://weights.replicate.delivery/default/chunyu-li/LatentSync/model.tar",
                "checkpoints"
            ])

        # Setup auxiliary model symlinks
        os.system("mkdir -p ~/.cache/torch/hub/checkpoints")
        os.system("ln -s $(pwd)/checkpoints/auxiliary/vgg16-397923af.pth "
                  
```

```python
"~/.cache/torch/hub/checkpoints/vgg16-397923af.pth")

    def predict(
        self,
        video: Path = Input(description="Input video"),
        audio: Path = Input(description="Input audio"),
        guidance_scale: float = Input(
            description="Guidance scale (1.0-3.0)",
            ge=1, le=3, default=2.0
        ),
        inference_steps: int = Input(
            description="Inference steps (20-50)",
            ge=20, le=50, default=20
        ),
        seed: int = Input(
            description="Random seed (0 for random)",
            default=0
        ),
    ) -> Path:
        """Run lip-sync prediction"""
        if seed <= 0:
            seed = int.from_bytes(os.urandom(2), "big")

        output_path = "/tmp/video_out.mp4"

        os.system(
            f"python -m scripts.inference "
            f"--unet_config_path configs/unet/stage2.yaml "
            f"--inference_ckpt_path checkpoints/latentsync_unet.pt "
            f"--guidance_scale {guidance_scale} "
            f"--video_path {video} "
            f"--audio_path {audio} "
            f"--video_out_path {output_path} "
            f"--seed {seed} "
            f"--inference_steps {inference_steps}"
        )
        return Path(output_path)
```

--------------------------------

### Configure SyncNet Visual Encoder Architecture (YAML)

Source: https://github.com/bytedance/latentsync/blob/main/docs/syncnet_arch.md

Defines the architecture for the visual encoder in SyncNet. It processes input frames and outputs features. The `in_channels` parameter is critical and calculated as `num_frames * image_channels`. `image_channels` varies based on whether it's a pixel-space (3) or latent-space (e.g., 4 or 16) SyncNet. This example shows a pixel-space SyncNet with 16 frames.

```yaml
visual_encoder:
  in_channels: 48
  block_out_channels: [64, 128, 256, 256, 512, 1024, 2048, 2048]
  downsample_factors: [[1, 2], 2, 2, 2, 2, 2, 2, 2]
  attn_blocks: [0, 0, 0, 0, 1, 1, 0, 0]
  dropout: 0.0
```

--------------------------------

### Train U-Net Model (Various Resolutions and VRAM)

Source: https://context7.com/bytedance/latentsync/llms.txt

Scripts to initiate U-Net training for different resolutions and VRAM requirements. These commands utilize torchrun for distributed training and specify configuration files for U-Net parameters.

```bash
torchrun --nnodes=1 --nproc_per_node=1 --master_port=25679     -m scripts.train_unet     --unet_config_path "configs/unet/stage2.yaml"
```

```bash
torchrun --nnodes=1 --nproc_per_node=1 --master_port=25679     -m scripts.train_unet     --unet_config_path "configs/unet/stage2_efficient.yaml"
```

```bash
torchrun --nnodes=1 --nproc_per_node=1 --master_port=25679     -m scripts.train_unet     --unet_config_path "configs/unet/stage1_512.yaml"
```

```bash
torchrun --nnodes=1 --nproc_per_node=1 --master_port=25679     -m scripts.train_unet     --unet_config_path "configs/unet/stage2_512.yaml"
```

--------------------------------

### U-Net Training Script (Bash)

Source: https://context7.com/bytedance/latentsync/llms.txt

Command to initiate the training of the lip-sync U-Net model using a configuration file. This command is for Stage 1 training, which does not use motion modules or SyncNet supervision and requires approximately 23GB of VRAM.

```bash
# Stage 1 training (256x256, 23GB VRAM)
torchrun --nnodes=1 --nproc_per_node=1 --master_port=25679 \
    -m scripts.train_unet \
    --unet_config_path "configs/unet/stage1.yaml"
```

--------------------------------

### Download Pretrained SyncNet Checkpoint

Source: https://github.com/bytedance/latentsync/blob/main/README.md

Downloads the required SyncNet checkpoint from Hugging Face to the local checkpoints directory for use in U-Net training.

```bash
huggingface-cli download ByteDance/LatentSync-1.6 stable_syncnet.pt --local-dir checkpoints
```

--------------------------------

### Generate Training File Lists with FileslistWriter

Source: https://context7.com/bytedance/latentsync/llms.txt

Demonstrates how to use the FileslistWriter utility to generate a text file containing paths to video files from specified dataset directories. This file list is used for training data loading.

```python
from tools.write_fileslist import FileslistWriter

# Create a new fileslist
fileslist_path = "/path/to/fileslist.txt"
writer = FileslistWriter(fileslist_path)

# Append videos from multiple dataset directories
writer.append_dataset("/path/to/VoxCeleb2/high_visual_quality/train")
writer.append_dataset("/path/to/HDTF/high_visual_quality/train")

# The resulting file contains one video path per line:
# /path/to/VoxCeleb2/high_visual_quality/train/id00001/video1.mp4
# /path/to/VoxCeleb2/high_visual_quality/train/id00001/video2.mp4
# /path/to/HDTF/high_visual_quality/train/video1.mp4
# ...
```

--------------------------------

### Train SyncNet Model (Single and Multi-GPU)

Source: https://context7.com/bytedance/latentsync/llms.txt

Commands to train a SyncNet model for audio-visual synchronization supervision. Supports both single-GPU and multi-GPU configurations using torchrun.

```bash
# Train SyncNet model
torchrun --nnodes=1 --nproc_per_node=1 --master_port=25678 \
    -m scripts.train_syncnet \
    --config_path "configs/syncnet/syncnet_16_pixel_attn.yaml"
```

```bash
# Multi-GPU SyncNet training
torchrun --nnodes=1 --nproc_per_node=4 --master_port=25678 \
    -m scripts.train_syncnet \
    --config_path "configs/syncnet/syncnet_16_pixel_attn.yaml"
```

--------------------------------

### LipsyncPipeline API for Lip-Sync Generation

Source: https://context7.com/bytedance/latentsync/llms.txt

This Python code demonstrates how to use the LipsyncPipeline for programmatic lip-sync generation. It initializes various components including VAE, audio encoder, U-Net, and scheduler, then runs the inference process. Key parameters include video/audio paths, inference steps, guidance scale, and output dimensions. Requires PyTorch, Diffusers, and project-specific modules.

```python
import torch
from omegaconf import OmegaConf
from diffusers import AutoencoderKL, DDIMScheduler
from latentsync.models.unet import UNet3DConditionModel
from latentsync.pipelines.lipsync_pipeline import LipsyncPipeline
from latentsync.whisper.audio2feature import Audio2Feature
from accelerate.utils import set_seed

# Load configuration
config = OmegaConf.load("configs/unet/stage2_512.yaml")

# Check GPU capabilities for dtype selection
is_fp16_supported = torch.cuda.is_available() and torch.cuda.get_device_capability()[0] > 7
dtype = torch.float16 if is_fp16_supported else torch.float32

# Initialize scheduler
scheduler = DDIMScheduler.from_pretrained("configs")

# Initialize Whisper audio encoder
audio_encoder = Audio2Feature(
    model_path="checkpoints/whisper/tiny.pt",
    device="cuda",
    num_frames=config.data.num_frames,          # Default: 16
    audio_feat_length=config.data.audio_feat_length,  # Default: [2, 2]
)

# Load VAE from Stable Diffusion
vae = AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse", torch_dtype=dtype)
vae.config.scaling_factor = 0.18215
vae.config.shift_factor = 0

# Load U-Net model
unet, _ = UNet3DConditionModel.from_pretrained(
    OmegaConf.to_container(config.model),
    "checkpoints/latentsync_unet.pt",
    device="cpu",
)
unet = unet.to(dtype=dtype)

# Create pipeline
pipeline = LipsyncPipeline(
    vae=vae,
    audio_encoder=audio_encoder,
    unet=unet,
    scheduler=scheduler,
).to("cuda")

# Set seed for reproducibility
set_seed(1247)

# Run inference
pipeline(
    video_path="assets/demo1_video.mp4",
    audio_path="assets/demo1_audio.wav",
    video_out_path="output.mp4",
    num_frames=16,                    # Frames processed per batch
    num_inference_steps=20,           # Denoising steps (20-50)
    guidance_scale=1.5,               # CFG scale (1.0-3.0)
    weight_dtype=dtype,
    width=512,                        # Output resolution
    height=512,
    mask_image_path="latentsync/utils/mask.png",
    temp_dir="temp",
)

print("Lip-sync video generated successfully!")

```

--------------------------------

### Run LatentSync Gradio App

Source: https://context7.com/bytedance/latentsync/llms.txt

This Python script sets up and runs a Gradio interface for the LatentSync application. It defines a function to process video and audio inputs using a pre-trained model and configuration, then launches the web UI. Dependencies include gradio, omegaconf, and custom scripts from the project.

```python
import gradio as gr
from pathlib import Path
from scripts.inference import main
from omegaconf import OmegaConf
import argparse
from datetime import datetime

CONFIG_PATH = Path("configs/unet/stage2_512.yaml")
CHECKPOINT_PATH = Path("checkpoints/latentsync_unet.pt")

def process_video(video_path, audio_path, guidance_scale, inference_steps, seed):
    output_dir = Path("./temp")
    output_dir.mkdir(parents=True, exist_ok=True)

    video_file_path = Path(video_path)
    video_path = video_file_path.absolute().as_posix()
    audio_path = Path(audio_path).absolute().as_posix()

    current_time = datetime.now().strftime("%Y%m%d_%H%M%S")
    output_path = str(output_dir / f"{video_file_path.stem}_{current_time}.mp4")

    config = OmegaConf.load(CONFIG_PATH)
    config["run"].update({
        "guidance_scale": guidance_scale,
        "inference_steps": inference_steps,
    })

    # Create argument namespace
    args = argparse.Namespace(
        inference_ckpt_path=str(CHECKPOINT_PATH.absolute()),
        video_path=video_path,
        audio_path=audio_path,
        video_out_path=output_path,
        inference_steps=inference_steps,
        guidance_scale=guidance_scale,
        seed=seed,
        temp_dir="temp",
        enable_deepcache=True
    )

    result = main(config=config, args=args)
    return output_path

# Create Gradio interface
with gr.Blocks(title="LatentSync demo") as demo:
    gr.Markdown("<h1 align='center'>LatentSync</h1>")

    with gr.Row():
        with gr.Column():
            video_input = gr.Video(label="Input Video")
            audio_input = gr.Audio(label="Input Audio", type="filepath")

            with gr.Row():
                guidance_scale = gr.Slider(minimum=1.0, maximum=3.0, value=1.5, step=0.1, label="Guidance Scale")
                inference_steps = gr.Slider(minimum=10, maximum=50, value=20, step=1, label="Inference Steps")

            seed = gr.Number(value=1247, label="Random Seed", precision=0)
            process_btn = gr.Button("Process Video")

        with gr.Column():
            video_output = gr.Video(label="Output Video")

    process_btn.click(
        fn=process_video,
        inputs=[video_input, audio_input, guidance_scale, inference_steps, seed],
        outputs=video_output
    )

if __name__ == "__main__":
    demo.launch(inbrowser=True, share=True)

```

--------------------------------

### Build and Run Cog Container Locally

Source: https://context7.com/bytedance/latentsync/llms.txt

Provides bash commands to build the LatentSync Cog container locally and run predictions with specified input files. It also includes instructions for pushing the container to Replicate.

```bash
# Build and run Cog container locally
cog build -t latentsync
cog predict -i video=@input_video.mp4 -i audio=@input_audio.wav

# Push to Replicate
cog push r8.im/username/latentsync
```

--------------------------------

### Execute Inference Script

Source: https://github.com/bytedance/latentsync/blob/main/README.md

Runs the inference script to generate video outputs. Users can adjust parameters like inference_steps and guidance_scale to balance quality and speed.

```bash
./inference.sh
```

--------------------------------

### Generate Data Files List

Source: https://github.com/bytedance/latentsync/blob/main/README.md

Runs a utility script to generate a list of data files required for the training process.

```python
python -m tools.write_fileslist
```

--------------------------------

### Multi-GPU U-Net Training

Source: https://context7.com/bytedance/latentsync/llms.txt

Command to perform multi-GPU distributed training for the U-Net model, specifying the number of processes per node and the U-Net configuration path.

```bash
torchrun --nnodes=1 --nproc_per_node=4 --master_port=25679     -m scripts.train_unet     --unet_config_path "configs/unet/stage2.yaml"
```

--------------------------------

### LatentSync Gradio Interface

Source: https://context7.com/bytedance/latentsync/llms.txt

This snippet shows how to launch and use the LatentSync Gradio web interface for quick video processing.

```APIDOC
## LatentSync Gradio Interface

### Description
This section details the Gradio interface for LatentSync, allowing users to upload a video and audio file, adjust parameters, and generate a lip-synced output video.

### Method
N/A (Web Interface)

### Endpoint
N/A (Local execution via `gradio_app.py`)

### Parameters
- **Input Video** (Video File) - Required - The input video file to be processed.
- **Input Audio** (Audio File) - Required - The input audio file to synchronize with the video.
- **Guidance Scale** (Slider, 1.0-3.0) - Optional - Controls the influence of the guidance on the generation process.
- **Inference Steps** (Slider, 10-50) - Optional - The number of denoising steps for the diffusion model.
- **Random Seed** (Number) - Optional - Seed for random number generation to ensure reproducibility.

### Request Example
N/A (Web Interface)

### Response
#### Success Response (200)
- **Output Video** (Video File) - The generated lip-synced video.

#### Response Example
N/A (Web Interface)
```

--------------------------------

### Data Processing Pipeline (Bash and Python)

Source: https://context7.com/bytedance/latentsync/llms.txt

A comprehensive pipeline for preparing training data, including video cleaning, resampling, scene detection, segmentation, face alignment, and audio-visual sync filtering. It can be run via a command-line interface or a Python API.

```bash
# Run the complete data processing pipeline
python -m preprocess.data_processing_pipeline \
    --input_dir /path/to/raw/videos \
    --total_num_workers 96 \
    --per_gpu_num_workers 12 \
    --resolution 256 \
    --sync_conf_threshold 3 \
    --temp_dir temp

# The pipeline creates these directories:
# /path/to/resampled/          - Resampled to 25fps, 16kHz audio
# /path/to/shot/               - Scene-detected clips
# /path/to/segmented/          - 5-10 second segments
# /path/to/affine_transformed/ - Face-aligned videos
# /path/to/av_synced_3/        - Sync-filtered videos
# /path/to/high_visual_quality/- Final high-quality dataset
```

```python
# Python API for data processing
from preprocess.data_processing_pipeline import data_processing_pipeline

data_processing_pipeline(
    total_num_workers=96,          # Total CPU workers for multiprocessing
    per_gpu_num_workers=12,        # Workers per GPU for GPU-accelerated steps
    resolution=256,                # Target face crop resolution (256 or 512)
    sync_conf_threshold=3,         # Minimum SyncNet confidence score
    temp_dir="temp",               # Temporary directory for intermediate files
    input_dir="/path/to/raw/videos"
)

# Individual preprocessing steps can also be run separately:
from preprocess.remove_broken_videos import remove_broken_videos_multiprocessing
from preprocess.resample_fps_hz import resample_fps_hz_multiprocessing
from preprocess.detect_shot import detect_shot_multiprocessing
from preprocess.segment_videos import segment_videos_multiprocessing
from preprocess.affine_transform import affine_transform_multi_gpus
from preprocess.sync_av import sync_av_multi_gpus
from preprocess.filter_visual_quality import filter_visual_quality_multi_gpus

# Example: Run only the resampling step
resample_fps_hz_multiprocessing(
    input_dir="/path/to/input",
    output_dir="/path/to/resampled",
    num_workers=96
)
```

--------------------------------

### Sync Confidence Evaluation (Python API)

Source: https://context7.com/bytedance/latentsync/llms.txt

Python code demonstrating how to use the SyncNet evaluation and detection modules to programmatically assess audio-visual synchronization quality. It includes initializing the SyncNet evaluator and face detector.

```python
# Python API for sync evaluation
import torch
from eval.syncnet import SyncNetEval
from eval.syncnet_detect import SyncNetDetector
from eval.eval_sync_conf import syncnet_eval

device = "cuda" if torch.cuda.is_available() else "cpu"

# Initialize SyncNet evaluator
syncnet = SyncNetEval(device=device)
syncnet.loadParameters("checkpoints/auxiliary/syncnet_v2.model")

# Initialize face detector for video preprocessing
syncnet_detector = SyncNetDetector(
    device=device,
    detect_results_dir="detect_results"
)

```

--------------------------------

### Run Data Processing Pipeline

Source: https://github.com/bytedance/latentsync/blob/main/README.md

Executes the data processing pipeline which includes video resampling, scene detection, face transformation, and quality filtering. The input_dir parameter must be configured in the script.

```bash
./data_processing_pipeline.sh
```

--------------------------------

### Evaluate Sync Confidence Score (CLI)

Source: https://context7.com/bytedance/latentsync/llms.txt

Command-line interface commands to evaluate the audio-visual synchronization quality of generated videos using the SyncNet confidence score. Supports single video or directory evaluation, and custom model paths.

```bash
# Evaluate sync confidence for a single video
python -m eval.eval_sync_conf --video_path "video_out.mp4"
```

```bash
# Evaluate all videos in a directory
python -m eval.eval_sync_conf --videos_dir "/path/to/generated/videos"
```

```bash
# With custom model path
python -m eval.eval_sync_conf \
    --video_path "output.mp4" \
    --initial_model "checkpoints/auxiliary/syncnet_v2.model" \
    --temp_dir "temp"
```

--------------------------------

### Evaluate Model Performance

Source: https://github.com/bytedance/latentsync/blob/main/README.md

Provides scripts to evaluate the synchronization confidence score of generated videos and the overall accuracy of the SyncNet model on a dataset.

```bash
./eval/eval_sync_conf.sh
./eval/eval_syncnet_acc.sh
```

--------------------------------

### Train SyncNet Model

Source: https://github.com/bytedance/latentsync/blob/main/README.md

Runs the training script for the SyncNet model on custom datasets. The data processing requirements are identical to the U-Net pipeline.

```bash
./train_syncnet.sh
```

--------------------------------

### Configure SyncNet Audio Encoder Architecture (YAML)

Source: https://github.com/bytedance/latentsync/blob/main/docs/syncnet_arch.md

Defines the architecture for the audio encoder in SyncNet. It accepts a mel spectrogram and outputs a feature map. Key parameters include `in_channels`, `block_out_channels`, `downsample_factors`, `attn_blocks`, and `dropout`. Adjusting `downsample_factors` is crucial when input resolution changes to ensure the output is a `D x 1 x 1` feature map for cosine similarity calculation. Deeper networks often require larger `block_out_channels`.

```yaml
audio_encoder:
  in_channels: 1
  block_out_channels: [32, 64, 128, 256, 512, 1024, 2048]
  downsample_factors: [[2, 1], 2, 2, 1, 2, 2, [2, 3]]
  attn_blocks: [0, 0, 0, 1, 1, 0, 0]
  dropout: 0.0
```

--------------------------------

### Evaluate Video with SyncNet

Source: https://context7.com/bytedance/latentsync/llms.txt

Evaluates a single video using the SyncNet model to determine audio-visual offset and confidence. It also includes a batch evaluation mode to process multiple videos from a directory and calculate the average sync confidence.

```python
video_path = "video_out.mp4"
av_offset, confidence = syncnet_eval(
    syncnet=syncnet,
    syncnet_detector=syncnet_detector,
    video_path=video_path,
    temp_dir="temp"
)

print(f"SyncNet confidence: {confidence:.2f}")
print(f"Audio-visual offset: {av_offset} frames")

# Batch evaluation
import os
from statistics import fmean

videos_dir = "/path/to/videos"
sync_conf_list = []
for video_name in os.listdir(videos_dir):
    if video_name.endswith(".mp4"):
        try:
            _, conf = syncnet_eval(
                syncnet, syncnet_detector,
                os.path.join(videos_dir, video_name),
                "temp"
            )
            sync_conf_list.append(conf)
        except Exception as e:
            print(f"Error processing {video_name}: {e}")

print(f"Average sync confidence: {fmean(sync_conf_list):.2f}")
```

--------------------------------

### LipsyncPipeline API

Source: https://context7.com/bytedance/latentsync/llms.txt

Programmatic usage of the LipsyncPipeline for advanced control over lip-sync video generation.

```APIDOC
## LipsyncPipeline API

### Description
Core diffusion pipeline class for programmatic lip-sync generation. The `LipsyncPipeline` class extends the Diffusers `DiffusionPipeline` and handles the complete inference workflow: video loading, face detection and affine transformation, audio feature extraction via Whisper, latent diffusion denoising, and video reconstruction. It supports classifier-free guidance, various schedulers (DDIM, PNDM, Euler, etc.), and automatic video looping for long audio.

### Method
`LipsyncPipeline()`

### Endpoint
N/A (Python Class)

### Parameters
#### Initialization Parameters
- **vae** (AutoencoderKL) - The Variational Autoencoder model.
- **audio_encoder** (Audio2Feature) - The audio feature extraction model (e.g., Whisper).
- **unet** (UNet3DConditionModel) - The 3D U-Net model for diffusion.
- **scheduler** (SchedulerMixin) - The diffusion scheduler (e.g., DDIMScheduler).

#### Inference Parameters (`pipeline()` method)
- **video_path** (str) - Path to the input video file.
- **audio_path** (str) - Path to the input audio file.
- **video_out_path** (str) - Path to save the output lip-synced video.
- **num_frames** (int) - Number of frames processed per batch.
- **num_inference_steps** (int) - Number of denoising steps (e.g., 20-50).
- **guidance_scale** (float) - Classifier-free guidance scale (e.g., 1.0-3.0).
- **weight_dtype** (torch.dtype) - Data type for model weights (e.g., `torch.float16` or `torch.float32`).
- **width** (int) - Output video width.
- **height** (int) - Output video height.
- **mask_image_path** (str) - Path to the mask image.
- **temp_dir** (str) - Path to a temporary directory for intermediate files.
- **seed** (int) - Random seed for reproducibility (optional, if not set, uses default).

### Request Example
```python
import torch
from omegaconf import OmegaConf
from diffusers import AutoencoderKL, DDIMScheduler
from latentsync.models.unet import UNet3DConditionModel
from latentsync.pipelines.lipsync_pipeline import LipsyncPipeline
from latentsync.whisper.audio2feature import Audio2Feature
from accelerate.utils import set_seed

# Load configuration
config = OmegaConf.load("configs/unet/stage2_512.yaml")

# Check GPU capabilities for dtype selection
is_fp16_supported = torch.cuda.is_available() and torch.cuda.get_device_capability()[0] > 7
dtype = torch.float16 if is_fp16_supported else torch.float32

# Initialize scheduler
scheduler = DDIMScheduler.from_pretrained("configs")

# Initialize Whisper audio encoder
audio_encoder = Audio2Feature(
    model_path="checkpoints/whisper/tiny.pt",
    device="cuda",
    num_frames=config.data.num_frames,          # Default: 16
    audio_feat_length=config.data.audio_feat_length,  # Default: [2, 2]
)

# Load VAE from Stable Diffusion
vae = AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse", torch_dtype=dtype)
vae.config.scaling_factor = 0.18215
vae.config.shift_factor = 0

# Load U-Net model
unet, _ = UNet3DConditionModel.from_pretrained(
    OmegaConf.to_container(config.model),
    "checkpoints/latentsync_unet.pt",
    device="cpu",
)
unet = unet.to(dtype=dtype)

# Create pipeline
pipeline = LipsyncPipeline(
    vae=vae,
    audio_encoder=audio_encoder,
    unet=unet,
    scheduler=scheduler,
).to("cuda")

# Set seed for reproducibility
set_seed(1247)

# Run inference
pipeline(
    video_path="assets/demo1_video.mp4",
    audio_path="assets/demo1_audio.wav",
    video_out_path="output.mp4",
    num_frames=16,
    num_inference_steps=20,
    guidance_scale=1.5,
    weight_dtype=dtype,
    width=512,
    height=512,
    mask_image_path="latentsync/utils/mask.png",
    temp_dir="temp",
)

print("Lip-sync video generated successfully!")
```

### Response
#### Success Response (200)
- **Output Video** (str) - Path to the generated lip-synced video file.

#### Response Example
```
Lip-sync video generated successfully!
```
```

--------------------------------

### Cog Configuration for LatentSync

Source: https://context7.com/bytedance/latentsync/llms.txt

Defines the Cog configuration for building and running the LatentSync Docker container. It specifies GPU usage, CUDA version, system packages, Python version, requirements, and the entry point for prediction.

```yaml
# cog.yaml - Cog configuration
build:
  gpu: true
  cuda: "12.1"
  system_packages:
    - "ffmpeg"
    - "libgl1"
  python_version: "3.10.13"
  python_requirements: requirements.txt
  run:
    - curl -o /usr/local/bin/pget -L "https://github.com/replicate/pget/releases/download/v0.10.2/pget_linux_x86_64" && chmod +x /usr/local/bin/pget

predict: "predict.py:Predictor"
```

--------------------------------

### Audio Feature Extraction with Audio2Feature API (Python)

Source: https://context7.com/bytedance/latentsync/llms.txt

Extracts synchronized audio embeddings using OpenAI's Whisper model. Supports caching and slicing features for video frame alignment. Dependencies include PyTorch and the Whisper model.

```python
from latentsync.whisper.audio2feature import Audio2Feature
import torch

# Initialize audio encoder with Whisper tiny model (384-dim embeddings)
audio_encoder = Audio2Feature(
    model_path="checkpoints/whisper/tiny.pt",
    device="cuda",
    audio_embeds_cache_dir="audio_cache/embeds",  # Optional: cache embeddings
    num_frames=16,
    audio_feat_length=[2, 2],  # Context window: 2 frames before, 2 after
)

# Extract audio features from file
audio_path = "assets/demo1_audio.wav"
audio_features = audio_encoder.audio2feat(audio_path)
print(f"Audio features shape: {audio_features.shape}")  # (num_chunks, embedding_dim)

# Convert features to chunks aligned with video frames
fps = 25  # Video frame rate
whisper_chunks = audio_encoder.feature2chunks(
    feature_array=audio_features,
    fps=fps
)
print(f"Number of whisper chunks: {len(whisper_chunks)}")
print(f"Each chunk shape: {whisper_chunks[0].shape}")  # (50, 384) for tiny model

# Get sliced feature for a specific video frame index
vid_idx = 10
selected_feature, selected_idx = audio_encoder.get_sliced_feature(
    feature_array=audio_features,
    vid_idx=vid_idx,
    fps=fps
)
print(f"Selected feature shape: {selected_feature.shape}")
print(f"Audio indices used: {selected_idx}")

# Crop overlapping audio window for training
start_index = 0
mel_overlap = audio_encoder.crop_overlap_audio_window(
    audio_feat=audio_features,
    start_index=start_index
)
print(f"Overlap window shape: {mel_overlap.shape}")  # (num_frames, 50, 384)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.