### Setup Environment

Source: https://github.com/opentalker/video-retalking/blob/main/README.md

Commands to clone the repository, create a Conda environment, and install dependencies including PyTorch and FFmpeg.

```bash
git clone https://github.com/vinthony/video-retalking.git
cd video-retalking
conda create -n video_retalking python=3.8
conda activate video_retalking

conda install ffmpeg

# Please follow the instructions from https://pytorch.org/get-started/previous-versions/
# This installation command only works on CUDA 11.1
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html

pip install -r requirements.txt
```

--------------------------------

### Setup Environment and Install Requirements

Source: https://github.com/opentalker/video-retalking/blob/main/quick_demo.ipynb

Ensures CUDA is available, installs ffmpeg, clones the project repository, and installs necessary Python packages. Run this in a Colab notebook or similar environment.

```python
#@title
### make sure that CUDA is available in Edit -> Nootbook settings -> GPU
!nvidia-smi

!python --version  
!apt-get update
!apt install ffmpeg &> /dev/null 

print('Git clone project and install requirements...')
!git clone https://github.com/vinthony/video-retalking.git &> /dev/null
%cd video-retalking
# !pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html
!pip install -r requirements.txt
```

--------------------------------

### Setup VideoReTalking Environment

Source: https://context7.com/opentalker/video-retalking/llms.txt

Commands to clone the repository, create a conda environment, and install dependencies with CUDA support.

```bash
# Clone repository
git clone https://github.com/vinthony/video-retalking.git
cd video-retalking

# Create conda environment
conda create -n video_retalking python=3.8
conda activate video_retalking

# Install ffmpeg
conda install ffmpeg

# Install PyTorch with CUDA 11.1 support
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 \
    -f https://download.pytorch.org/whl/torch_stable.html

# Install dependencies
pip install -r requirements.txt

# Download pretrained models to ./checkpoints/
# Required files:
# - DNet.pt
# - LNet.pth
# - ENet.pth
# - face3d_pretrain_epoch_20.pth
# - GFPGANv1.3.pth
# - GPEN-BFR-512.pth
# - shape_predictor_68_face_landmarks.dat
# - expression.mat
# - BFM/ (folder with 3DMM basis)

# Download from Google Drive:
# https://drive.google.com/drive/folders/18rhjMpxK8LVVxf7PI6XwOidt8Vouv_H0
```

--------------------------------

### Install Requirements

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/GPEN/face_morpher/README.rst

Install the necessary dependencies via pip.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install PyTorch v1.7.1

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/face3d/models/arcface_torch/docs/install.md

Commands for installing PyTorch 1.7.1 with support for various CUDA versions or CPU-only environments.

```shell
# CUDA 11.0
pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

# CUDA 10.2
pip install torch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2

# CUDA 10.1
pip install torch==1.7.1+cu101 torchvision==0.8.2+cu101 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

# CUDA 9.2
pip install torch==1.7.1+cu92 torchvision==0.8.2+cu92 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

# CPU only
pip install torch==1.7.1+cpu torchvision==0.8.2+cpu torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html
```

--------------------------------

### Install PyTorch v1.8.0

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/face3d/models/arcface_torch/docs/install.md

Commands for installing PyTorch 1.8.0 with support for CUDA 11.0, 10.2, or CPU-only environments.

```shell
# CUDA 11.0
pip --default-timeout=100 install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html

# CUDA 10.2
pip --default-timeout=100 install torch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0

# CPU only
pip --default-timeout=100 install torch==1.8.0+cpu torchvision==0.9.0+cpu torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html
```

--------------------------------

### Install PyTorch v1.6.0

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/face3d/models/arcface_torch/docs/install.md

Commands for installing PyTorch 1.6.0 with support for CUDA 10.2, 10.1, 9.2, or CPU-only environments.

```shell
# CUDA 10.2
pip install torch==1.6.0 torchvision==0.7.0

# CUDA 10.1
pip install torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html

# CUDA 9.2
pip install torch==1.6.0+cu92 torchvision==0.7.0+cu92 -f https://download.pytorch.org/whl/torch_stable.html

# CPU only
pip install torch==1.6.0+cpu torchvision==0.7.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
```

--------------------------------

### Multi-Node, 8 GPUs Training (Node 0)

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/face3d/models/arcface_torch/README.md

Command for the first node in a multi-node distributed training setup, utilizing 8 GPUs. Replace 'ip1' with the actual master node IP address.

```shell
python -m torch.distributed.launch --nproc_per_node=8 --nnodes=2 --node_rank=0 --master_addr="ip1" --master_port=1234 train.py train.py configs/ms1mv3_r50
```

--------------------------------

### Single Node, 8 GPUs Training

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/face3d/models/arcface_torch/README.md

Use this command to initiate distributed training on a single machine with 8 GPUs. Ensure PyTorch is installed and requirements are met.

```shell
python -m torch.distributed.launch --nproc_per_node=8 --nnodes=1 --node_rank=0 --master_addr="127.0.0.1" --master_port=1234 train.py configs/ms1mv3_r50
```

--------------------------------

### Multi-Node, 8 GPUs Training (Node 1)

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/face3d/models/arcface_torch/README.md

Command for the second node in a multi-node distributed training setup, utilizing 8 GPUs. Ensure the master address and port match Node 0.

```shell
python -m torch.distributed.launch --nproc_per_node=8 --nnodes=2 --node_rank=1 --master_addr="ip1" --master_port=1234 train.py train.py configs/ms1mv3_r50
```

--------------------------------

### List Image Paths with Face Morpher Library

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/GPEN/face_morpher/README.rst

Use this Python function to get a list of image paths from a specified folder. Ensure the 'facemorpher' library is installed.

```python
import facemorpher

# Get a list of image paths in a folder
imgpaths = facemorpher.list_imgpaths('imagefolder')
```

--------------------------------

### Run Face Morpher in Docker

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/GPEN/face_morpher/README.rst

This command launches a Docker container, mounts a local directory to '/images', and provides a bash session. Install 'facemorpher' inside the container to use its command-line tools.

```bash
docker run -v /Users/alyssa/Desktop/images:/images --name py3 --rm -it jjanzic/docker-python3-opencv bash
```

--------------------------------

### Install Face Morpher in Docker Container

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/GPEN/face_morpher/README.rst

Once inside the Docker container, use pip to install the 'facemorpher' library. This makes its command-line utilities available.

```bash
pip install facemorpher
```

--------------------------------

### Face Morpher Command Line Utility

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/GPEN/face_morpher/README.rst

After installing via pip, this command-line utility can be used for face morphing. Specify source and destination images, and optionally plot the results.

```bash
facemorpher --src=<src_imgpath> --dest=<dest_imgpath> --plot
```

--------------------------------

### Face Averager Command Line Utility

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/GPEN/face_morpher/README.rst

This command-line utility, available after pip installation, averages face images from a specified folder. Use the '--plot' flag to visualize the output.

```bash
faceaverager --images=<images_folder> --plot
```

--------------------------------

### Morpher CLI Help

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/GPEN/face_morpher/README.rst

View the available options for the morpher.py script.

```text
Morph from source to destination face or
Morph through all images in a folder

Usage:
    morpher.py (--src=<src_path> --dest=<dest_path> | --images=<folder>)
            [--width=<width>] [--height=<height>]
            [--num=<num_frames>] [--fps=<frames_per_second>]
            [--out_frames=<folder>] [--out_video=<filename>]
            [--plot] [--background=(black|transparent|average)]

Options:
    -h, --help              Show this screen.
    --src=<src_imgpath>     Filepath to source image (.jpg, .jpeg, .png)
    --dest=<dest_imgpath>   Filepath to destination image (.jpg, .jpeg, .png)
    --images=<folder>       Folderpath to images
    --width=<width>         Custom width of the images/video [default: 500]
    --height=<height>       Custom height of the images/video [default: 600]
    --num=<num_frames>      Number of morph frames [default: 20]
    --fps=<fps>             Number frames per second for the video [default: 10]
    --out_frames=<folder>   Folder path to save all image frames
    --out_video=<filename>  Filename to save a video
    --plot                  Flag to plot images to result.png [default: False]
    --background=<bg>       Background of images to be one of (black|transparent|average) [default: black]
    --version               Show version.
```

--------------------------------

### Initialize and Run LNet

Source: https://context7.com/opentalker/video-retalking/llms.txt

Configures the LNet model for lip-syncing and performs a forward pass with dummy audio and face sequences.

```python
lnet = LNet(
    image_nc=3,           # Input image channels
    descriptor_nc=512,    # Audio feature dimension
    layer=3,              # Number of encoder/decoder layers
    base_nc=64,           # Base channel count
    max_nc=512,           # Maximum channels
    num_res_blocks=9      # Residual blocks in decoder
)
lnet.eval()

device = 'cuda' if torch.cuda.is_available() else 'cpu'
lnet = lnet.to(device)

# Prepare inputs (batch of 4)
# Audio: mel-spectrogram chunks
audio_sequences = torch.randn(4, 1, 80, 16).to(device)  # (B, 1, mel_bins, time_steps)

# Face sequences: masked lower face + reference face concatenated
face_sequences = torch.randn(4, 6, 96, 96).to(device)   # (B, 6, H, W) - 3ch masked + 3ch ref

# Forward pass
with torch.no_grad():
    lip_synced_output = lnet(audio_sequences, face_sequences)

# Output shape: (4, 3, 96, 96) - Generated lip-synced faces
print(f"Output shape: {lip_synced_output.shape}")
```

--------------------------------

### Execute Partial FC training performance tests

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/face3d/models/arcface_torch/docs/speed_benchmark.md

Use these shell commands to initiate training with either model parallel or Partial FC 0.1 configurations.

```shell
# Model Parallel
python -m torch.distributed.launch --nproc_per_node=8 --nnodes=1 --node_rank=0 --master_addr="127.0.0.1" --master_port=1234 train.py configs/3millions
# Partial FC 0.1
python -m torch.distributed.launch --nproc_per_node=8 --nnodes=1 --node_rank=0 --master_addr="127.0.0.1" --master_port=1234 train.py configs/3millions_pfc
```

--------------------------------

### Review training speed logs

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/face3d/models/arcface_torch/docs/speed_benchmark.md

Sample log output showing training speed in samples per second for both configurations.

```python
# (Model Parallel) trainging.log
Training: Speed 2271.33 samples/sec   Loss 1.1624   LearningRate 0.2000   Epoch: 0   Global Step: 100 
Training: Speed 2269.94 samples/sec   Loss 0.0000   LearningRate 0.2000   Epoch: 0   Global Step: 150 
Training: Speed 2272.67 samples/sec   Loss 0.0000   LearningRate 0.2000   Epoch: 0   Global Step: 200 
Training: Speed 2266.55 samples/sec   Loss 0.0000   LearningRate 0.2000   Epoch: 0   Global Step: 250 
Training: Speed 2272.54 samples/sec   Loss 0.0000   LearningRate 0.2000   Epoch: 0   Global Step: 300 

# (Partial FC 0.1) trainging.log
Training: Speed 5299.56 samples/sec   Loss 1.0965   LearningRate 0.2000   Epoch: 0   Global Step: 100  
Training: Speed 5296.37 samples/sec   Loss 0.0000   LearningRate 0.2000   Epoch: 0   Global Step: 150  
Training: Speed 5304.37 samples/sec   Loss 0.0000   LearningRate 0.2000   Epoch: 0   Global Step: 200  
Training: Speed 5274.43 samples/sec   Loss 0.0000   LearningRate 0.2000   Epoch: 0   Global Step: 250  
Training: Speed 5300.10 samples/sec   Loss 0.0000   LearningRate 0.2000   Epoch: 0   Global Step: 300  
```

--------------------------------

### Select Input Video and Audio

Source: https://github.com/opentalker/video-retalking/blob/main/quick_demo.ipynb

Uses ipywidgets to create dropdown menus for selecting the input video and audio files from the 'examples/face' and 'examples/audio' directories, respectively. This code is intended for use in an interactive environment like Google Colab.

```python
#@title
import glob, os, sys
import ipywidgets as widgets
from IPython.display import HTML
from base64 import b64encode

print("Choose the Video name to edit: (saved in folder 'examples/face')")
vid_list = glob.glob1('examples/face/', '*.mp4')
vid_list.sort()
default_vid_name = widgets.Dropdown(options=vid_list, value='1.mp4')
display(default_vid_name)

print("Choose the Audio name to edit: (saved in folder 'examples/audio')")
audio_list = glob.glob1('examples/audio/', '*')
audio_list.sort()
default_audio_name = widgets.Dropdown(options=audio_list, value='1.wav')
display(default_audio_name)
```

--------------------------------

### Initialize and Run ENet

Source: https://context7.com/opentalker/video-retalking/llms.txt

Loads pretrained LNet and ENet models to perform high-resolution face enhancement.

```python
import torch
from models.LNet import LNet
from models.ENet import ENet
from models import load_checkpoint

# Load pretrained LNet first
lnet = LNet()
lnet = load_checkpoint('checkpoints/LNet.pth', lnet)

# Initialize ENet with LNet as base
enet = ENet(
    num_style_feat=512,   # Style feature dimension
    lnet=lnet,            # Base lip-sync network
    concat=False          # Concatenation mode
)
enet = load_checkpoint('checkpoints/ENet.pth', enet)
enet.eval()

device = 'cuda' if torch.cuda.is_available() else 'cpu'
enet = enet.to(device)

# Prepare inputs
audio_sequences = torch.randn(1, 1, 80, 16).to(device)     # Mel spectrogram
face_sequences = torch.randn(1, 6, 384, 384).to(device)    # Masked + reference face
gt_sequences = torch.randn(1, 3, 384, 384).to(device)      # Ground truth for reference

# Forward pass
with torch.no_grad():
    enhanced_output, low_res = enet(audio_sequences, face_sequences, gt_sequences)

# enhanced_output: High-resolution result (384x384)
# low_res: Low-resolution intermediate from LNet (96x96 upscaled)
print(f"Enhanced output shape: {enhanced_output.shape}")
print(f"Low-res shape: {low_res.shape}")
```

--------------------------------

### Launch Gradio Web UI

Source: https://context7.com/opentalker/video-retalking/llms.txt

Launch the interactive Gradio web interface for VideoReTalking. Access the UI at http://localhost:7860 after running the command.

```python
# Launch the Gradio web interface
python webUI.py

# The web UI provides:
# - Video upload for source face
# - Audio upload for target speech
# - Optional segment length for long videos
# - Example videos and audio files

# Access at http://localhost:7860 after launch
```

--------------------------------

### Use Face Morpher Command Line in Docker

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/GPEN/face_morpher/README.rst

After installing 'facemorpher' within the Docker container, you can use its command-line interface for morphing operations. Ensure image paths are accessible within the container's mounted volume.

```bash
facemorpher --src=<img1> --dest=<img2> --plot
```

--------------------------------

### Static Image Lip-Sync with Custom FPS

Source: https://context7.com/opentalker/video-retalking/llms.txt

Generate a lip-synced video from a static image and audio, specifying a custom frames per second (FPS). Use --face for the image, --audio for speech, and --fps for the desired frame rate.

```bash
python3 inference.py \
  --face examples/face/portrait.jpg \
  --audio examples/audio/speech.wav \
  --outfile results/talking_portrait.mp4 \
  --fps 30
```

--------------------------------

### Run Inference for Lip Synchronization

Source: https://github.com/opentalker/video-retalking/blob/main/quick_demo.ipynb

Executes the main inference script using the specified input video and audio files. The output video will be saved as 'results/output.mp4'.

```bash
input_video_path = 'examples/face/{}'.format(default_vid_name.value)
input_audio_path = 'examples/audio/{}'.format(default_audio_name.value)

!python3 inference.py \
  --face {input_video_path} \
  --audio {input_audio_path} \
  --outfile results/output.mp4
```

--------------------------------

### Training ResNet2060 with 8 GPUs

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/face3d/models/arcface_torch/README.md

Initiates distributed training for the ResNet2060 model using 8 GPUs on a single node. This command specifies the configuration file for the R2060 architecture.

```shell
python -m torch.distributed.launch --nproc_per_node=8 --nnodes=1 --node_rank=0 --master_addr="127.0.0.1" --master_port=1234 train.py configs/ms1mv3_r2060.py
```

--------------------------------

### Download Pre-trained Models

Source: https://github.com/opentalker/video-retalking/blob/main/quick_demo.ipynb

Downloads and organizes essential pre-trained models and data required for the VideoReTalking inference. This includes various network weights and facial landmark data.

```python
#@title
print('Download pre-trained models...')
!mkdir ./checkpoints  
!wget https://github.com/vinthony/video-retalking/releases/download/v0.0.1/30_net_gen.pth -O ./checkpoints/30_net_gen.pth
!wget https://github.com/vinthony/video-retalking/releases/download/v0.0.1/BFM.zip -O ./checkpoints/BFM.zip
!wget https://github.com/vinthony/video-retalking/releases/download/v0.0.1/DNet.pt -O ./checkpoints/DNet.pt
!wget https://github.com/vinthony/video-retalking/releases/download/v0.0.1/ENet.pth -O ./checkpoints/ENet.pth
!wget https://github.com/vinthony/video-retalking/releases/download/v0.0.1/expression.mat -O ./checkpoints/expression.mat
!wget https://github.com/vinthony/video-retalking/releases/download/v0.0.1/face3d_pretrain_epoch_20.pth -O ./checkpoints/face3d_pretrain_epoch_20.pth
!wget https://github.com/vinthony/video-retalking/releases/download/v0.0.1/GFPGANv1.3.pth -O ./checkpoints/GFPGANv1.3.pth
!wget https://github.com/vinthony/video-retalking/releases/download/v0.0.1/GPEN-BFR-512.pth -O ./checkpoints/GPEN-BFR-512.pth
!wget https://github.com/vinthony/video-retalking/releases/download/v0.0.1/LNet.pth -O ./checkpoints/LNet.pth
!wget https://github.com/vinthony/video-retalking/releases/download/v0.0.1/ParseNet-latest.pth -O ./checkpoints/ParseNet-latest.pth
!wget https://github.com/vinthony/video-retalking/releases/download/v0.0.1/RetinaFace-R50.pth -O ./checkpoints/RetinaFace-R50.pth
!wget https://github.com/vinthony/video-retalking/releases/download/v0.0.1/shape_predictor_68_face_landmarks.dat -O ./checkpoints/shape_predictor_68_face_landmarks.dat
!unzip -d ./checkpoints/BFM ./checkpoints/BFM.zip
```

--------------------------------

### Averager CLI Help

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/GPEN/face_morpher/README.rst

View the available options for the averager.py script.

```text
Face averager

Usage:
    averager.py --images=<images_folder> [--blur] [--plot]
            [--background=(black|transparent|average)]
            [--width=<width>] [--height=<height>]
            [--out=<filename>] [--destimg=<filename>]

Options:
    -h, --help             Show this screen.
    --images=<folder>      Folder to images (.jpg, .jpeg, .png)
    --blur                 Flag to blur edges of image [default: False]
    --width=<width>        Custom width of the images/video [default: 500]
    --height=<height>      Custom height of the images/video [default: 600]
    --out=<filename>       Filename to save the average face [default: result.png]
    --destimg=<filename>   Destination face image to overlay average face
    --plot                 Flag to display the average face [default: False]
    --background=<bg>      Background of image to be one of (black|transparent|average) [default: black]
    --version              Show version.
```

--------------------------------

### Average Faces via CLI

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/GPEN/face_morpher/README.rst

Generate an average face from a collection of images in a folder.

```bash
python facemorpher/averager.py --images=<images_folder> --out=average.png
```

--------------------------------

### Run Inference

Source: https://github.com/opentalker/video-retalking/blob/main/README.md

Command to execute the inference script with specified face video and audio inputs.

```bash
python3 inference.py \
  --face examples/face/1.mp4 \
  --audio examples/audio/1.wav \
  --outfile results/1_1.mp4
```

--------------------------------

### Basic Lip-Sync Inference

Source: https://context7.com/opentalker/video-retalking/llms.txt

Perform basic lip-sync inference using a video and audio file. Specify input face video, audio, and output file path.

```bash
python3 inference.py \
  --face examples/face/1.mp4 \
  --audio examples/audio/1.wav \
  --outfile results/output.mp4
```

--------------------------------

### Averaging Faces Command-Line Utility

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/GPEN/face_morpher/README.rst

This section details the command-line utility for averaging faces from a folder of images.

```APIDOC
## Averaging Faces

### Description
Average faces from all images in a folder.

### Method
`python`

### Endpoint
`facemorpher/averager.py`

### Parameters
#### Path Parameters
None

#### Query Parameters
- **--images** (string) - Required - Folder to images (.jpg, .jpeg, .png)
- **--blur** (boolean) - Optional - Flag to blur edges of image [default: False]
- **--width** (integer) - Optional - Custom width of the images/video [default: 500]
- **--height** (integer) - Optional - Custom height of the images/video [default: 600]
- **--out** (string) - Optional - Filename to save the average face [default: result.png]
- **--destimg** (string) - Optional - Destination face image to overlay average face
- **--plot** (boolean) - Optional - Flag to display the average face [default: False]
- **--background** (string) - Optional - Background of image to be one of (black|transparent|average) [default: black]

### Request Example
```bash
python facemorpher/averager.py --images=./my_faces/ --out=average_face.png
```

### Response
#### Success Response (200)
- **Image file** - Saved image of the average face.

#### Response Example
(No specific response body example provided, output is typically files.)
```

--------------------------------

### LNet Model Import

Source: https://context7.com/opentalker/video-retalking/llms.txt

Import the LNet (Lip Synchronization Network) model from the specified path. This is the initial step for using the lip-sync synthesis network.

```python
import torch
from models.LNet import LNet
```

--------------------------------

### Visualize Input Video and Audio

Source: https://github.com/opentalker/video-retalking/blob/main/quick_demo.ipynb

Displays the selected input video and audio using HTML5 tags. The media files are base64 encoded to be embedded directly into the output.

```python
#@title
input_video_name = './examples/face/{}'.format(default_vid_name.value)
input_video_mp4 = open('{}'.format(input_video_name),'rb').read()
input_video_data_url = "data:video/x-m4v;base64," + b64encode(input_video_mp4).decode()
print('Display input video: {}'.format(input_video_name), file=sys.stderr)
display(HTML(
  """
  <video width=400 controls>
        <source src="%s" type="video/mp4">
  </video>
  """ % input_video_data_url))

input_audio_name = './examples/audio/{}'.format(default_audio_name.value)
input_audio_mp4 = open('{}'.format(input_audio_name),'rb').read()
input_audio_data_url = "data:audio/wav;base64," + b64encode(input_audio_mp4).decode()
print('Display input audio: {}'.format(input_audio_name), file=sys.stderr)
display(HTML(
  """
  <audio width=400 controls>
        <source src="%s" type="audio/wav">
  </audio>
  """ % input_audio_data_url))
```

--------------------------------

### Configure Face Detection

Source: https://context7.com/opentalker/video-retalking/llms.txt

Sets up parameters for face detection, including padding and temporal smoothing, for processing video frames.

```python
import numpy as np
from utils.inference_utils import face_detect

class Args:
    face_det_batch_size = 4
    pads = [0, 20, 0, 0]  # top, bottom, left, right padding
    nosmooth = False       # Enable temporal smoothing
    box = [-1, -1, -1, -1]  # Manual bounding box (disabled)

args = Args()

# Load video frames as numpy arrays (BGR format)
frames = [...]  # List of numpy arrays
```

--------------------------------

### Clone Repository

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/GPEN/face_morpher/README.rst

Clone the face morpher repository to your local machine.

```bash
$ git clone https://github.com/alyssaq/face_morpher
```

--------------------------------

### Morphing Faces Command-Line Utility

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/GPEN/face_morpher/README.rst

This section details the command-line utility for morphing faces. It supports morphing from a source to a destination image, or through a series of images in a folder.

```APIDOC
## Morphing Faces

### Description
Morph from a source to destination face or morph through all images in a folder.

### Method
`python`

### Endpoint
`facemorpher/morpher.py`

### Parameters
#### Path Parameters
None

#### Query Parameters
- **--src** (string) - Required - Filepath to source image (.jpg, .jpeg, .png)
- **--dest** (string) - Required - Filepath to destination image (.jpg, .jpeg, .png)
- **--images** (string) - Required - Folderpath to images
- **--width** (integer) - Optional - Custom width of the images/video [default: 500]
- **--height** (integer) - Optional - Custom height of the images/video [default: 600]
- **--num** (integer) - Optional - Number of morph frames [default: 20]
- **--fps** (integer) - Optional - Number frames per second for the video [default: 10]
- **--out_frames** (string) - Optional - Folder path to save all image frames
- **--out_video** (string) - Optional - Filename to save a video
- **--plot** (boolean) - Optional - Flag to plot images to result.png [default: False]
- **--background** (string) - Optional - Background of images to be one of (black|transparent|average) [default: black]

### Request Example
```bash
python facemorpher/morpher.py --src=alyssa.jpg --dest=john_malkovich.jpg --out_video=out.avi
```

### Response
#### Success Response (200)
- **Video file** - Saved video of the morphing process.
- **Image frames** - Saved individual image frames if `--out_frames` is specified.

#### Response Example
(No specific response body example provided, output is typically files.)
```

--------------------------------

### Publish Face Morpher Documentation

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/GPEN/face_morpher/README.rst

This script is used to build and publish the documentation for the face morpher project to GitHub Pages.

```bash
./scripts/publish_ghpages.sh
```

--------------------------------

### Cog API Predictor Initialization and Prediction

Source: https://context7.com/opentalker/video-retalking/llms.txt

Initialize and run inference using the Cog API predictor. This involves setting up the predictor and calling the predict method with input file paths.

```python
from cog import BasePredictor, Input, Path
from predict import Predictor

# Initialize predictor (loads models once)
predictor = Predictor()
predictor.setup()

# Run prediction
output_path = predictor.predict(
    face=Path("/path/to/input_video.mp4"),
    input_audio=Path("/path/to/audio.wav")
)

print(f"Output saved to: {output_path}")
# Output: /tmp/output.mp4
```

--------------------------------

### Display Input and Lip-Syncing Video Side-by-Side

Source: https://github.com/opentalker/video-retalking/blob/main/quick_demo.ipynb

Reads the concatenated video (input and output side-by-side), encodes it in base64, and displays it as an HTML5 video element. This allows for direct comparison.

```python
output_concat_video_name = './results/output_concat_input.mp4'
output_concat_video_mp4 = open('{}'.format(output_concat_video_name),'rb').read()
output_concat_video_data_url = "data:video/mp4;base64," + b64encode(output_concat_video_mp4).decode()
print('Display input video and lip-syncing video: {}'.format(output_concat_video_name), file=sys.stderr)
display(HTML(
  """
  <video height=400 controls>
        <source src=\"%s\" type=\"video/mp4\">
  </video>
  """ % output_concat_video_data_url))
```

--------------------------------

### DNet Model Loading and Inference

Source: https://context7.com/opentalker/video-retalking/llms.txt

Load the DNet (Expression Editing Network) model and perform a forward pass with dummy inputs. This demonstrates how to prepare inputs and access outputs like optical flow and warped images.

```python
import torch
from models import load_DNet

class Args:
    DNet_path = 'checkpoints/DNet.pt'

args = Args()
device = 'cuda' if torch.cuda.is_available() else 'cpu'

# Load DNet model
D_Net = load_DNet(args).to(device)

# Prepare inputs
source_img = torch.randn(1, 3, 256, 256).to(device)  # Normalized face image
driving_coeff = torch.randn(1, 73, 27).to(device)    # 3DMM coefficients (exp + angles + trans + crop)

# Forward pass
with torch.no_grad():
    output = D_Net(source_img, driving_coeff)

# Access outputs
flow_field = output['flow_field']      # Optical flow for warping
warp_image = output['warp_image']      # Warped intermediate result
fake_image = output['fake_image']      # Final generated image with canonical expression

# Convert to numpy for visualization
result = (fake_image.squeeze(0).permute(1,2,0).cpu().numpy() + 1) / 2. * 255
result = result.astype('uint8')
```

--------------------------------

### Lip-Sync with Custom Expression Image

Source: https://context7.com/opentalker/video-retalking/llms.txt

Apply a custom facial expression from an image template for lip-sync generation. Provide the path to the expression template image using --exp_img.

```bash
python3 inference.py \
  --face examples/face/1.mp4 \
  --audio examples/audio/1.wav \
  --outfile results/output_custom.mp4 \
  --exp_img path/to/expression_template.jpg
```

--------------------------------

### Process Audio for Inference

Source: https://context7.com/opentalker/video-retalking/llms.txt

Loads audio files, computes mel-spectrograms, and chunks them into the required format for the network.

```python
from utils import audio
import numpy as np

# Load audio file at 16kHz sample rate
wav = audio.load_wav('examples/audio/1.wav', sr=16000)

# Compute mel-spectrogram
mel = audio.melspectrogram(wav)
print(f"Mel spectrogram shape: {mel.shape}")  # (80, time_frames)

# Check for invalid values
if np.isnan(mel.reshape(-1)).sum() > 0:
    raise ValueError('Mel contains nan! Add small noise to audio and retry.')

# Chunk mel-spectrogram for inference (fps=25)
fps = 25
mel_step_size = 16
mel_idx_multiplier = 80. / fps

mel_chunks = []
i = 0
while True:
    start_idx = int(i * mel_idx_multiplier)
    if start_idx + mel_step_size > len(mel[0]):
        mel_chunks.append(mel[:, len(mel[0]) - mel_step_size:])
        break
    mel_chunks.append(mel[:, start_idx:start_idx + mel_step_size])
    i += 1

print(f"Number of mel chunks: {len(mel_chunks)}")
# Each chunk shape: (80, 16)
```

--------------------------------

### Inference With Pytorch

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/face3d/models/arcface_torch/docs/eval.md

Perform inference using a pre-trained model. Specify the weight path and network architecture.

```shell
python inference.py --weight ms1mv3_arcface_r50/backbone.pth --network r50
```

--------------------------------

### Average Face Images

Source: https://github.com/opentalker/video-retalking/blob/main/third_part/GPEN/face_morpher/README.rst

This command averages all face images in a specified folder. Options include blurring and setting a transparent background, width, and height.

```bash
python facemorpher/averager.py --images=images --blur --background=transparent --width=220 --height=250
```