SyncTalk

https://github.com/ziqiaopeng/synctalk
Admin
SyncTalk is a method for synthesizing synchronized talking head videos using tri-plane...
Tokens:12,529
Snippets:101
Trust Score:7.4
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install Dependencies

Source: https://github.com/ziqiaopeng/synctalk/blob/main/data_utils/deepspeech_features/README.md

Installs necessary Python packages from the requirements file. Ensure you have pip3 installed.

```bash
pip3 install -r requirements.txt
```

--------------------------------

### Install PyTorch3D via Script

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Alternative method to install PyTorch3D using a provided Python script. Use this if the direct pip installation fails.

```bash
python ./scripts/install_pytorch3d.py
```

--------------------------------

### Install System Dependencies

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Install the portaudio19-dev library, which is required for audio processing. This command is for Debian-based Linux systems.

```bash
sudo apt-get install portaudio19-dev
```

--------------------------------

### Install PyTorch3D

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Install PyTorch3D using the provided command, specifying the Python version, CUDA version, and PyTorch version. This is crucial for 3D operations.

```bash
pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py38_cu113_pyt1121/download.html
```

--------------------------------

### Install Project Requirements

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Install all Python dependencies listed in the requirements.txt file. This command should be run after activating the Conda environment.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install TensorFlow GPU

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Install TensorFlow with GPU support version 2.8.1. This is required for certain components of the project.

```bash
pip install tensorflow-gpu==2.8.1
```

--------------------------------

### Install Local Python Packages

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Install several local Python packages that are part of the SyncTalk project. These include encoders for frequency, Shen, grid, and ray marching.

```bash
pip install ./freqencoder
pip install ./shencoder
pip install ./gridencoder
pip install ./raymarching
```

--------------------------------

### Install PyTorch with CUDA Support

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Install PyTorch version 1.12.1 with CUDA 11.3 support, along with torchvision and torchaudio. Ensure your system has compatible CUDA drivers.

```bash
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113
```

--------------------------------

### Get Audio Features with Context Windowing

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Utility function to slice pre-computed audio features into context windows. Supports no context, left-only, and bidirectional attention modes. Handles edge cases near the beginning of the sequence by zero-padding.

```python
import torch
from nerf_triplane.utils import get_audio_features

# Simulate a full audio features tensor: [T, 29] (DeepSpeech, T frames)
T = 300
features = torch.randn(T, 29)

# Bidirectional window (att_mode=2): returns an 8-frame window [8, 29]
frame_idx = 50
window = get_audio_features(features, att_mode=2, index=frame_idx)
print(window.shape)   # torch.Size([8, 29])

# Left-only window (att_mode=1): returns 8 past frames [8, 29]
window_left = get_audio_features(features, att_mode=1, index=frame_idx)
print(window_left.shape)  # torch.Size([8, 29])

# Single frame (att_mode=0)
single = get_audio_features(features, att_mode=0, index=frame_idx)
print(single.shape)   # torch.Size([1, 29])

# Edge case: near the beginning of the sequence (zero-padded automatically)
window_start = get_audio_features(features, att_mode=2, index=2)
print(window_start.shape)  # torch.Size([8, 29])
```

--------------------------------

### Prepare Pre-trained Models

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Download and place the 'May.zip' and 'trial_may.zip' files into the 'data' and 'model' folders, respectively. Then, unzip these archives.

```bash
# Download May.zip to the data folder
# Download trial_may.zip to the model folder
# Then unzip them
```

--------------------------------

### Clone Repository and Navigate

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Clone the SyncTalk repository and navigate into the project directory. This is the first step for setting up the project on Linux.

```bash
git clone https://github.com/ZiqiaoPeng/SyncTalk.git
cd SyncTalk
```

--------------------------------

### Initialize and Run NeRF Training with Trainer

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Sets up the NeRF model, dataset loaders, optimizer, and metrics, then initiates the training loop using the Trainer class. Ensure all necessary imports and configurations are in place before execution.

```python
import torch
from nerf_triplane.network import NeRFNetwork
from nerf_triplane.provider import NeRFDataset
from nerf_triplane.utils import Trainer, PSNRMeter, LPIPSMeter, LMDMeter
import argparse

opt = argparse.Namespace(
    path='data/May', workspace='model/trial_may',
    asr_model='ave', iters=60000, lr=1e-2, lr_net=1e-3,
    fp16=True, cuda_ray=True, exp_eye=True, emb=False, att=2,
    fix_eye=-1, smooth_eye=False, smooth_lips=False, finetune_lips=False,
    init_lips=False, bs_area='upper', au45=False, torso=False,
    torso_shrink=0.8, bound=1, density_thresh=10, density_thresh_torso=0.01,
    min_near=0.05, patch_size=1, ind_dim=4, ind_dim_torso=8,
    ind_num=20000, amb_dim=2, amb_aud_loss=1, amb_eye_loss=1,
    unc_loss=1, lambda_amb=1e-4, pyramid_loss=0, num_rays=65536,
    max_steps=16, update_extra_interval=16, max_ray_batch=4096,
    color_space='srgb', preload=0, scale=4, offset=[0,0,0],
    dt_gamma=1/256, warmup_step=10000, train_camera=False,
    smooth_path=False, smooth_path_window=7, test_train=False,
    data_range=[0,-1], portrait=False, aud='', part=False, part2=False,
    seed=0, ckpt='latest',
)
device = torch.device('cuda')
model  = NeRFNetwork(opt).to(device)
criterion = torch.nn.L1Loss(reduction='none')
optimizer = lambda m: torch.optim.AdamW(m.get_params(opt.lr, opt.lr_net), betas=(0,0.99), eps=1e-8)

import numpy as np
train_loader = NeRFDataset(opt, device=device, type='train').dataloader()
valid_loader = NeRFDataset(opt, device=device, type='val', downscale=1).dataloader()

scheduler = lambda optim: torch.optim.lr_scheduler.LambdaLR(
    optim, lambda it: 0.5 ** (it / opt.iters))

metrics = [PSNRMeter(), LPIPSMeter(device=device), LMDMeter(backend='fan')]

trainer = Trainer(
    'ngp', opt, model, device=device,
    workspace=opt.workspace,
    optimizer=optimizer, criterion=criterion,
    ema_decay=0.95, fp16=opt.fp16,
    lr_scheduler=scheduler, scheduler_update_every_step=True,
    metrics=metrics, use_checkpoint=opt.ckpt,
    eval_interval=max(1, int(5000 / len(train_loader))),
)

max_epochs = int(np.ceil(opt.iters / len(train_loader)))
trainer.train(train_loader, valid_loader, max_epochs)
# Checkpoints saved to model/trial_may/checkpoints/
# TensorBoard logs in model/trial_may/run/

# Evaluation (reports PSNR / LPIPS / LMD to console and TensorBoard)
trainer.evaluate(valid_loader)
```

--------------------------------

### Instantiate NeRFNetwork and Forward Pass

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Demonstrates the instantiation of the NeRFNetwork model and a basic forward pass with synthetic data. Requires PyTorch and specific project modules.

```python
import torch
from nerf_triplane.network import NeRFNetwork
from nerf_triplane.utils import seed_everything
import argparse

# Minimal option namespace (mirrors argparse output from main.py)
opt = argparse.Namespace(
    asr_model='ave',       # 'ave' | 'deepspeech' | 'hubert'
    emb=False,
    att=2,                 # bi-directional audio attention
    exp_eye=True,          # explicit eye control
    fix_eye=-1,
    smooth_eye=False,
    smooth_lips=False,
    bs_area='upper',       # blendshape area for eye attention
    au45=False,            # use OpenFace AU45
    torso=False,
    torso_shrink=0.8,
    bound=1,
    fp16=True,
    cuda_ray=True,
    density_thresh=10,
    density_thresh_torso=0.01,
    min_near=0.05,
    ind_dim=4,
    ind_dim_torso=8,
    ind_num=20000,
    amb_dim=2,
    train_camera=False,
    test_train=False,
    finetune_lips=False,
    init_lips=False,
    unc_loss=1,
)

seed_everything(0)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model = NeRFNetwork(opt).to(device)
print(f'Parameters: {sum(p.numel() for p in model.parameters() if p.requires_grad):,}')

# Forward pass (single point, for illustration)
N = 128  # number of sample points
x = torch.rand(N, 3, device=device) * 2 - 1          # 3D positions in [-1, 1]
d = torch.rand(N, 3, device=device)                   # view directions
d = d / d.norm(dim=-1, keepdim=True)
enc_a = torch.rand(1, 32, device=device)              # audio embedding [1, audio_dim] [1, audio_dim]
c     = torch.rand(1, 4, device=device)               # individual code [1, ind_dim]
e     = torch.rand(1, 7, device=device)               # eye blendshape  [1, eye_dim]

with torch.no_grad():
    sigma, color, aud_att, eye_att, uncertainty = model(x, d, enc_a, c, e)
# sigma:       [N]   – volume density
# color:       [N,3] – RGB in [0,1]
# aud_att:     [N,1] – audio channel attention magnitude
# eye_att:     [N,1] – eye channel attention magnitude

```

--------------------------------

### Run Full Data Preprocessing Pipeline

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Execute the complete data preparation script for a source video. This command processes a typical video length and resolution, extracting all necessary features.

```bash
python data_utils/process.py data/May/May.mp4 --asr ave
```

--------------------------------

### Download Face Parsing Model

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Use wget to download the face-parsing model. Ensure the destination directory exists.

```bash
wget https://github.com/YudongGuo/AD-NeRF/blob/master/data_util/face_parsing/79999_iter.pth?raw=true -O data_utils/face_parsing/79999_iter.pth
```

--------------------------------

### ASR Context Manager for Real-time Audio Processing

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Demonstrates using the ASR class as a context manager for both full-file processing and step-by-step consumption of audio chunks in a rendering loop. Useful for interactive talking-head rendering.

```python
with ASR(opt) as asr:
    asr.warm_up()          # prime the model (reduces first-frame latency)
    asr.run()              # process entire file, prints transcription
    # demo/test.npy written with shape [N_frames, 16, 32]

# Per-step usage inside a rendering loop:
with ASR(opt) as asr:
    asr.listen()
    while not asr.terminated:
        asr.run_step()                        # consume one 20ms chunk
        feat = asr.get_next_feat()            # [8, audio_dim, 16] window
        # pass feat to model.encode_audio() for synchronized rendering
```

--------------------------------

### Initialize ASR for Real-Time Speech Recognition

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Sets up the Wav2Vec2-based ASR class for processing audio streams from files or live microphones. Configures sliding window parameters for feature extraction.

```python
import argparse, torch
from nerf_triplane.asr import ASR

# Build a minimal options namespace for file-mode ASR
opt = argparse.Namespace(
    asr_model='facebook/wav2vec2-large-960h-lv60-self',
    asr_wav='demo/test.wav',
    asr_play=False,
    asr_save_feats=True,   # save logits to demo/test.npy
    fps=50,
    l=10, m=50, r=10,      # sliding window: left=10, context=50, right=10 (each unit=20ms)
)

```

--------------------------------

### Inference with Custom Target Audio (Portrait Mode)

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Performs inference in portrait mode using a custom WAV audio file. The output video will be saved in the workspace results directory.

```bash
python main.py data/May --workspace model/trial_may \
    -O --test --test_train --asr_model ave --portrait \
    --aud ./demo/test.wav
```

--------------------------------

### Train SyncTalk Model: Lip Fine-tuning

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Execute the second phase of model training for lip-region fine-tuning. This phase runs for 100,000 iterations using LPIPS loss on specified patch sizes.

```bash
python main.py data/May --workspace model/trial_may \
    -O --iters 100000 --finetune_lips --patch_size 64 --asr_model ave
```

--------------------------------

### Perform Video Inference and Export with Trainer.test

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Renders frames from a test data loader and exports them as an MP4 video. Optionally merges original audio and blends the rendered face onto the original image if `--portrait` is enabled.

```python
import torch
from nerf_triplane.provider import NeRFDataset
from nerf_triplane.utils import Trainer, PSNRMeter, LPIPSMeter, LMDMeter
from nerf_triplane.network import NeRFNetwork
import argparse

# Assume opt and model already set up (see Trainer example above)
# Load the latest checkpoint automatically via use_checkpoint='latest'
trainer = Trainer(
    'ngp', opt, model, device=torch.device('cuda'),
    workspace='model/trial_may',
    criterion=torch.nn.L1Loss(reduction='none'), fp16=True,
    metrics=[PSNRMeter(), LPIPSMeter(device=torch.device('cuda')), LMDMeter(backend='fan')],
    use_checkpoint='latest',
)

test_loader = NeRFDataset(opt, device=torch.device('cuda'), type='test').dataloader()
model.aud_features = test_loader._data.auds
model.eye_areas    = test_loader._data.eye_area

# Render test split → saves ngp_ep<N>.mp4 + ngp_ep<N>_depth.mp4
trainer.test(test_loader)
# If --aud is set with AVE model, also produces ngp_ep<N>_audio.mp4 (with audio merged)

# Evaluate ground-truth metrics
if test_loader.has_gt:
    trainer.evaluate(test_loader)
    # Console output example:
    # PSNR  = 37.644
    # LPIPS (alex) = 0.0117
    # LMD (fan)   = 2.825
```

--------------------------------

### Train with Hubert Model

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Use the Hubert model for training. Specify the number of iterations and optionally enable finetuning lips and patch size.

```bash
python main.py data/May --workspace model/trial_may -O --iters 60000 --asr_model hubert
```

```bash
python main.py data/May --workspace model/trial_may -O --iters 100000 --finetune_lips --patch_size 64 --asr_model hubert
```

--------------------------------

### Train Head Model with OpenFace AU45

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Initiates training for the head model, enabling fp16, CUDA raymarching, and explicit eye control. Uses average ASR model.

```bash
python main.py data/May --workspace model/trial_may \
    -O --iters 60000 --asr_model ave --au45
```

--------------------------------

### Test Model

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Run tests on the trained model. This command includes options for specifying the workspace, iterations, and enabling portrait mode.

```bash
python main.py data/May --workspace model/trial_may -O --test --asr_model ave --portrait
```

--------------------------------

### Train with OpenFace AU45

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Incorporate OpenFace AU45 as the eye parameter by adding the '--au45' flag to the command line during training.

```bash
python main.py data/May --workspace model/trial_may -O --iters 60000 --asr_model ave --au45
```

```bash
python main.py data/May --workspace model/trial_may -O --iters 100000 --finetune_lips --patch_size 64 --asr_model ave --au45
```

--------------------------------

### Download 3DMM Models for Head Pose Estimation

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Download necessary 3DMM model files for head pose estimation using wget. These files are crucial for tracking and processing head movements.

```bash
wget https://github.com/YudongGuo/AD-NeRF/blob/master/data_util/face_tracking/3DMM/exp_info.npy?raw=true -O data_utils/face_tracking/3DMM/exp_info.npy
```

```bash
wget https://github.com/YudongGuo/AD-NeRF/blob/master/data_util/face_tracking/3DMM/keys_info.npy?raw=true -O data_utils/face_tracking/3DMM/keys_info.npy
```

```bash
wget https://github.com/YudongGuo/AD-NeRF/blob/master/data_util/face_tracking/3DMM/sub_mesh.obj?raw=true -O data_utils/face_tracking/3DMM/sub_mesh.obj
```

```bash
wget https://github.com/YudongGuo/AD-NeRF/blob/master/data_util/face_tracking/3DMM/topology_info.npy?raw=true -O data_utils/face_tracking/3DMM/topology_info.npy
```

--------------------------------

### Run Specific Data Preprocessing Task

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Execute only a specific task within the data preparation pipeline, such as audio extraction (task 1) or face tracking (task 7). Use `--task N` to specify the task number.

```bash
python data_utils/process.py data/May/May.mp4 --task 1 --asr ave
```

```bash
python data_utils/process.py data/May/May.mp4 --task 7
```

--------------------------------

### ASR Class Usage

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Demonstrates the usage of the ASR class for both processing entire files and step-by-step processing within a rendering loop.

```APIDOC
## ASR Class Usage

### Description
The ASR class is used for automatic speech recognition and can be utilized in two main ways: processing an entire audio file at once or processing audio in steps within a loop.

### Usage

**Processing entire file:**
```python
with ASR(opt) as asr:
    asr.warm_up()          # prime the model (reduces first-frame latency)
    asr.run()              # process entire file, prints transcription
```

**Per-step usage inside a rendering loop:**
```python
with ASR(opt) as asr:
    asr.listen()
    while not asr.terminated:
        asr.run_step()                        # consume one 20ms chunk
        feat = asr.get_next_feat()            # [8, audio_dim, 16] window
        # pass feat to model.encode_audio() for synchronized rendering
```
```

--------------------------------

### Train Model with AVE

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Train the model using the AVE audio-visual encoder. The `--preload` flag controls data loading strategy: 0 for on-the-fly, 1 for CPU, and 2 for GPU.

```bash
# by default, we load data from disk on the fly.
# we can also preload all data to CPU/GPU for faster training, but this is very memory-hungry for large datasets.
# `--preload 0`: load from disk (default, slower).
# `--preload 1`: load to CPU (slightly slower)
# `--preload 2`: load to GPU (fast)
python main.py data/May --workspace model/trial_may -O --iters 60000 --asr_model ave
python main.py data/May --workspace model/trial_may -O --iters 100000 --finetune_lips --patch_size 64 --asr_model ave
```

--------------------------------

### Inference with Custom Audio (Torso Mode)

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Performs inference using the torso model with custom audio input. Includes testing on the training split.

```bash
python main.py data/May --workspace model/trial_may_torso \
    -O --torso --test --test_train \
    --asr_model ave --aud ./demo/test.wav
```

--------------------------------

### Inference Using HuBERT Features

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Run inference using HuBERT features. Set `--asr_model` to `hubert` and provide the path to the HuBERT `.npy` audio features. The `--portrait` flag is optional.

```bash
python main.py data/May --workspace model/trial_may -O \
    --test --asr_model hubert \
    --aud data/May/aud_hu.npy --portrait
```

--------------------------------

### Train using Training Script

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Execute the provided shell script for training. This script likely encapsulates one or more of the Python training commands.

```bash
sh ./scripts/train_may.sh
```

--------------------------------

### Train SyncTalk Model with HuBERT

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Train the SyncTalk model using the HuBERT audio encoder. This includes both the coarse training and lip fine-tuning stages.

```bash
python main.py data/May --workspace model/trial_may \
    -O --iters 60000 --asr_model hubert
```

```bash
python main.py data/May --workspace model/trial_may \
    -O --iters 100000 --finetune_lips --patch_size 64 --asr_model hubert
```

--------------------------------

### Project Citation

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

BibTeX entry for citing the SyncTalk project in academic work.

```bibtex
@inproceedings{peng2024synctalk,
  title={Synctalk: The devil is in the synchronization for talking head synthesis},
  author={Peng, Ziqiao and Hu, Wentao and Shi, Yue and Zhu, Xiangyu and Zhang, Xiaomei and Zhao, Hao and He, Jun and Liu, Hongyan and Fan, Zhaoxin},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={666--676},
  year={2024}
}
```

--------------------------------

### Load Specific Checkpoint for Inference

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Loads a specific checkpoint file for inference. Ensure the checkpoint path is correct.

```bash
python main.py data/May --workspace model/trial_may \
    -O --test --asr_model ave --ckpt model/trial_may/ngp_ep0019.pth
```

--------------------------------

### Process Video for Talking Head Generation

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Place your video in 'data/<ID>/<ID>.mp4'. The video should be 25FPS, around 4-5 minutes long, and 512x512 resolution. This command processes the video using the AVE ASR model.

```bash
python data_utils/process.py data/<ID>/<ID>.mp4 --asr ave
```

--------------------------------

### Train SyncTalk Model: Coarse Training

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Initiate the first phase of model training, focusing on coarse training for 60,000 iterations. This uses the specified audio encoder (e.g., AVE).

```bash
python main.py data/May --workspace model/trial_may \
    -O --iters 60000 --asr_model ave
```

--------------------------------

### Create Conda Environment

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Create a new Conda environment named 'synctalk' with Python 3.8.8. This isolates project dependencies.

```bash
conda create -n synctalk python==3.8.8
conda activate synctalk
```

--------------------------------

### Generate WAV Files

Source: https://github.com/ziqiaopeng/synctalk/blob/main/data_utils/deepspeech_features/README.md

Extracts audio from video files to generate WAV files. Specify the directory containing your video data using the --in-video flag.

```python
python3 extract_wav.py --in-video=<you_data_dir>
```

--------------------------------

### Inference with Target Audio

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Perform inference using the trained torso model with a target audio file. The '--test_train' flag is used, and '--portrait' mode is not supported.

```bash
python main.py data/May --workspace model/trial_may_torso -O  --torso --test --test_train --asr_model ave --aud ./demo/test.wav # not support --portrait
```

--------------------------------

### Inference Using Pre-extracted DeepSpeech Features

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Run inference using pre-extracted DeepSpeech features. Ensure the `--asr_model` is set to `deepspeech` and provide the path to the `.npy` audio features.

```bash
python main.py data/May --workspace model/trial_may -O \
    --test --test_train \
    --asr_model deepspeech \
    --aud data/MyAudio.npy
```

--------------------------------

### Inference with Target Audio

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Perform inference using a target audio file (e.g., 'demo/test.wav'). Results are saved in the specified workspace. If not using AVE, replace '.wav' with the path to a .npy file.

```bash
python main.py data/May --workspace model/trial_may -O --test --test_train --asr_model ave --portrait --aud ./demo/test.wav
```

--------------------------------

### Convert Basel Face Model

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

After downloading the Basel Face Model 2009, copy the '01_MorphableModel.mat' file to the specified directory and run the conversion script.

```bash
# 1. copy 01_MorphableModel.mat to data_util/face_tracking/3DMM/
# 2.
  cd data_utils/face_tracking
    python convert_BFM.py
```

--------------------------------

### Evaluation Metrics for Talking Head Quality

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Provides three metric trackers: PSNRMeter for peak signal-to-noise ratio, LPIPSMeter for perceptual image quality, and LMDMeter for landmark mouth distance (lip-sync accuracy). These are useful for evaluating synthesized talking head videos.

```python
import torch, numpy as np
from nerf_triplane.utils import PSNRMeter, LPIPSMeter, LMDMeter

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Simulate a batch of predicted and ground-truth frames [B, H, W, 3] in [0, 1]
B, H, W = 2, 512, 512
preds  = torch.rand(B, H, W, 3)
truths = torch.rand(B, H, W, 3)

# PSNR
psnr = PSNRMeter()
psnr.update(preds, truths)
print(psnr.report())   # e.g., PSNR = 18.234500

# LPIPS (AlexNet, expects [B, H, W, 3] in [0, 1])
lpips_m = LPIPSMeter(net='alex', device=device)
lpips_m.update(preds.to(device), truths.to(device))
print(lpips_m.report())  # e.g., LPIPS (alex) = 0.412300

# LMD – Landmark Mouth Distance (requires face_alignment)
lmd = LMDMeter(backend='fan', region='mouth')
# preds / truths must have detectable faces (uint8 after *255)
# lmd.update(preds, truths)
# print(lmd.report())   # e.g., LMD (fan) = 3.142000

# Accumulate over multiple batches then read summary
for _ in range(5):
    psnr.update(preds, truths)
print(f'Mean PSNR over 6 batches: {psnr.measure():.4f}')
psnr.clear()
```

--------------------------------

### PSNRMeter, LPIPSMeter, LMDMeter

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Evaluation metric trackers for measuring talking head quality, including PSNR, LPIPS, and Landmark Mouth Distance.

```APIDOC
## `PSNRMeter`, `LPIPSMeter`, `LMDMeter` — Evaluation Metrics

### Description
Three evaluation metric trackers used to measure talking head quality. `PSNRMeter` computes peak signal-to-noise ratio, `LPIPSMeter` computes perceptual image quality (AlexNet LPIPS), and `LMDMeter` computes landmark mouth distance as a lip-sync accuracy measure.

### Usage

```python
import torch, numpy as np
from nerf_triplane.utils import PSNRMeter, LPIPSMeter, LMDMeter

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Simulate a batch of predicted and ground-truth frames [B, H, W, 3] in [0, 1]
B, H, W = 2, 512, 512
preds  = torch.rand(B, H, W, 3)
truths = torch.rand(B, H, W, 3)

# PSNR
psnr = PSNRMeter()
psnr.update(preds, truths)
print(psnr.report())   # e.g., PSNR = 18.234500

# LPIPS (AlexNet, expects [B, H, W, 3] in [0, 1])
lpips_m = LPIPSMeter(net='alex', device=device)
lpips_m.update(preds.to(device), truths.to(device))
print(lpips_m.report())  # e.g., LPIPS (alex) = 0.412300

# LMD – Landmark Mouth Distance (requires face_alignment)
lmd = LMDMeter(backend='fan', region='mouth')
# preds / truths must have detectable faces (uint8 after *255)
# lmd.update(preds, truths)
# print(lmd.report())   # e.g., LMD (fan) = 3.142000

# Accumulate over multiple batches then read summary
for _ in range(5):
    psnr.update(preds, truths)
print(f'Mean PSNR over 6 batches: {psnr.measure():.4f}')
psnr.clear()
```

### Parameters

- **PSNRMeter**: Initializes a PSNR tracker.
- **LPIPSMeter**: Initializes an LPIPS tracker. Accepts `net` (e.g., 'alex') and `device`.
- **LMDMeter**: Initializes an LMD tracker. Accepts `backend` (e.g., 'fan') and `region` (e.g., 'mouth').

### Methods

- **update(preds, truths)**: Updates the metric trackers with predicted and ground-truth data.
- **report()**: Reports the current metric value.
- **measure()**: Reports the accumulated metric value over multiple updates.
- **clear()**: Clears the accumulated metric data.
```

--------------------------------

### Test Torso Model

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Test the trained torso model. Ensure that the '--portrait' flag is not used, as it is not supported with torso training.

```bash
python main.py data/May --workspace model/trial_may_torso -O  --torso --test --asr_model ave  # not support --portrait
```

--------------------------------

### Train Torso Model

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Performs optional second-stage training for a deformable torso network to fix double-chin artifacts. Requires a completed head training checkpoint.

```bash
python main.py data/May/ --workspace model/trial_may_torso/ \
    -O --torso \
    --head_ckpt model/trial_may/ngp_ep0019.pth \
    --iters 150000 --asr_model ave
```

--------------------------------

### Train SyncTalk Model with DeepSpeech

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Train the SyncTalk model using the DeepSpeech audio encoder. This involves both coarse training and lip fine-tuning phases.

```bash
python main.py data/May --workspace model/trial_may \
    -O --iters 60000 --asr_model deepspeech
```

```bash
python main.py data/May --workspace model/trial_may \
    -O --iters 100000 --finetune_lips --patch_size 64 --asr_model deepspeech
```

--------------------------------

### Run Evaluation Code

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Execute the evaluation script. The first command runs without the 'portrait' option, while the second includes it for higher quality output by pasting the generated face back onto the original image.

```bash
python main.py data/May --workspace model/trial_may -O --test --asr_model ave

python main.py data/May --workspace model/trial_may -O --test --asr_model ave --portrait
```

--------------------------------

### Portrait Mode Inference

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Renders a face and blends it back onto the original background during inference. Requires the `--test` flag.

```bash
python main.py data/May --workspace model/trial_may \
    -O --test --asr_model ave --portrait
```

--------------------------------

### Query Volume Density with NeRFNetwork

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Queries density (sigma) and geometric features for 3D points conditioned on audio and eye embeddings using tri-plane hash encoding and the sigma MLP. The output includes sigma, geo_feat, and attention weights for ambient features.

```python
import torch
from nerf_triplane.network import NeRFNetwork
import argparse

opt = argparse.Namespace(
    asr_model='ave', emb=False, att=2, exp_eye=True, fix_eye=-1,
    smooth_eye=False, smooth_lips=False, bs_area='upper', au45=False,
    torso=False, torso_shrink=0.8, bound=1, fp16=False, cuda_ray=True,
    density_thresh=10, density_thresh_torso=0.01, min_near=0.05,
    ind_dim=4, ind_dim_torso=8, ind_num=20000, amb_dim=2,
    train_camera=False, test_train=False, finetune_lips=False,
    init_lips=False, unc_loss=1,
)
model = NeRFNetwork(opt)

N = 512
pts   = torch.rand(N, 3) * 2 - 1   # 3D positions in [-bound, bound]
enc_a = torch.rand(1, 32)           # audio embedding [1, audio_dim]
e     = torch.rand(1, 7)            # eye blendshape  [1, eye_dim]

with torch.no_grad():
    result = model.density(pts, enc_a, e)

print(result['sigma'].shape)        # torch.Size([512])   – volume density
print(result['geo_feat'].shape)     # torch.Size([512, 64]) – geometry feature
print(result['ambient_aud'].shape)  # torch.Size([512, 1])  – audio attention
print(result['ambient_eye'].shape)  # torch.Size([512, 1])  – eye attention
```

--------------------------------

### Train Model with DeepSpeech

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Train the model using the DeepSpeech audio feature encoder. This is an alternative to AVE, recommended if lip jitter is observed.

```bash
# Use deepspeech model
python main.py data/May --workspace model/trial_may -O --iters 60000 --asr_model deepspeech
python main.py data/May --workspace model/trial_may -O --iters 100000 --finetune_lips --patch_size 64 --asr_model deepspeech
```

--------------------------------

### Evaluate on Test Split

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Evaluates the model on the test split, reporting PSNR, LPIPS, and LMD metrics if ground-truth frames are available. Uses average ASR model.

```bash
python main.py data/May --workspace model/trial_may \
    -O --test --asr_model ave
```

--------------------------------

### Use HuBERT for Audio Features in Preprocessing

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Specify HuBERT as the audio feature extraction model during the data preprocessing pipeline. This replaces the default AVE encoder.

```bash
python data_utils/process.py data/MySubject/MySubject.mp4 --asr hubert
```

--------------------------------

### Generate DeepSpeech Features

Source: https://github.com/ziqiaopeng/synctalk/blob/main/data_utils/deepspeech_features/README.md

Generates DeepSpeech features from input data. Use the --input flag to specify the directory containing your data.

```python
python3 extract_ds_features.py --input=<you_data_dir>
```

--------------------------------

### get_audio_features

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Utility function to slice audio features into context windows for a given frame index with different attention modes.

```APIDOC
## `get_audio_features` — Windowed Audio Feature Slicing

### Description
Utility function in `nerf_triplane/utils.py` that slices a pre-computed audio feature tensor into a context window for a given frame index. Supports three attention modes: no context (`att_mode=0`), left-only (`att_mode=1`), and bidirectional (`att_mode=2`, default).

### Parameters

- **features** (torch.Tensor) - The full audio features tensor.
- **att_mode** (int) - Attention mode: 0 for no context, 1 for left-only, 2 for bidirectional.
- **index** (int) - The frame index for which to extract the window.

### Request Example
```python
import torch
from nerf_triplane.utils import get_audio_features

# Simulate a full audio features tensor: [T, 29] (DeepSpeech, T frames)
T = 300
features = torch.randn(T, 29)

# Bidirectional window (att_mode=2): returns an 8-frame window [8, 29]
frame_idx = 50
window = get_audio_features(features, att_mode=2, index=frame_idx)
print(window.shape)   # torch.Size([8, 29])

# Left-only window (att_mode=1): returns 8 past frames [8, 29]
window_left = get_audio_features(features, att_mode=1, index=frame_idx)
print(window_left.shape)  # torch.Size([8, 29])

# Single frame (att_mode=0)
single = get_audio_features(features, att_mode=0, index=frame_idx)
print(single.shape)   # torch.Size([1, 29])

# Edge case: near the beginning of the sequence (zero-padded automatically)
window_start = get_audio_features(features, att_mode=2, index=2)
print(window_start.shape)  # torch.Size([8, 29])
```
```

--------------------------------

### Train Torso Model

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Train the torso model to resolve double chin issues. This requires specifying the head checkpoint and iteration count. Note that '--portrait' mode is not supported when using the torso model.

```bash
# Train
# <head>.pth should be the latest checkpoint in trial_may
python main.py data/May/ --workspace model/trial_may_torso/ -O --torso --head_ckpt <head>.pth --iters 150000 --asr_model ave
```

```bash
# For example
python main.py data/May/ --workspace model/trial_may_torso/ -O --torso --head_ckpt model/trial_may/ngp_ep0019.pth --iters 150000 --asr_model ave
```

--------------------------------

### Extract HuBERT Features

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Extract HuBERT features from a WAV audio file using a script borrowed from GeneFace. The output is saved as a .npy file.

```bash
# Borrowed from GeneFace. English pre-trained.
    python data_utils/hubert.py --wav data/<name>.wav # save to data/<name>_hu.npy
```

--------------------------------

### Extract HuBERT Features for Inference

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Extract HuBERT features from a WAV audio file. The output `.npy` array is used as the `--aud` input during inference.

```bash
python data_utils/hubert.py --wav data/May/aud.wav
```

--------------------------------

### Extract DeepSpeech Features

Source: https://github.com/ziqiaopeng/synctalk/blob/main/README.md

Extract DeepSpeech features from a WAV audio file. The output is saved as a .npy file in the data directory.

```bash
python data_utils/deepspeech_features/extract_ds_features.py --input data/<name>.wav # save to data/<name>.npy
```

--------------------------------

### Extract DeepSpeech Features for Inference

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Standalone script to extract DeepSpeech features from a WAV audio file. The output `.npy` array can be used as the `--aud` input during inference.

```bash
python data_utils/deepspeech_features/extract_ds_features.py \
    --input data/May/aud.wav
```

--------------------------------

### Encode Audio Features with NeRFNetwork

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Encodes raw audio features (DeepSpeech logits, HuBERT embeddings, or AVE features) into a compact audio embedding using `AudioNet` / `AudioNet_ave` followed by optional bi-directional attention (`AudioAttNet`). Ensure the correct `asr_model` is set for the input feature type.

```python
import torch
from nerf_triplane.network import NeRFNetwork
import argparse

opt = argparse.Namespace(
    asr_model='ave', emb=False, att=2, exp_eye=True, fix_eye=-1,
    smooth_eye=False, smooth_lips=False, bs_area='upper', au45=False,
    torso=False, torso_shrink=0.8, bound=1, fp16=False, cuda_ray=True,
    density_thresh=10, density_thresh_torso=0.01, min_near=0.05,
    ind_dim=4, ind_dim_torso=8, ind_num=20000, amb_dim=2,
    train_camera=False, test_train=False, finetune_lips=False,
    init_lips=False, unc_loss=1,
)
model = NeRFNetwork(opt)

# AVE: input is a pre-encoded 512-dim vector per frame window [1, 1, 512]
ave_feat = torch.rand(1, 1, 512)           # from AudioEncoder (visual encoder)
enc_a_ave = model.encode_audio(ave_feat)   # -> [1, 32]

# DeepSpeech: input is [8, 29, 16] (8-frame window, 29 labels, 16 time steps)
ds_feat = torch.rand(8, 29, 16)
model_ds = NeRFNetwork(argparse.Namespace(**{**vars(opt), 'asr_model': 'deepspeech'}))
enc_a_ds = model_ds.encode_audio(ds_feat)  # -> [1, 32]

# HuBERT: input is [8, 1024, 16]
model_hu = NeRFNetwork(argparse.Namespace(**{**vars(opt), 'asr_model': 'hubert'}))
hu_feat  = torch.rand(8, 1024, 16)
enc_a_hu = model_hu.encode_audio(hu_feat)  # -> [1, 32]

print(enc_a_ave.shape)  # torch.Size([1, 32])
```

--------------------------------

### NeRFNetwork.encode_audio

Source: https://context7.com/ziqiaopeng/synctalk/llms.txt

Encodes raw audio features into a compact audio embedding using AudioNet/AudioNet_ave followed by optional bi-directional attention.

```APIDOC
## `NeRFNetwork.encode_audio` — Audio Feature Encoding

Encodes raw audio features (DeepSpeech logits, HuBERT embeddings, or AVE features) into a compact audio embedding using `AudioNet` / `AudioNet_ave` followed by optional bi-directional attention (`AudioAttNet`).

### Method Signature
`encode_audio(self, audio_features)`

### Parameters
- `audio_features`: The raw audio features to encode. The expected shape and format depend on the `asr_model` configuration (e.g., AVE, DeepSpeech, HuBERT).

### Returns
- `torch.Tensor`: A compact audio embedding tensor.

### Example Usage
```python
import torch
from nerf_triplane.network import NeRFNetwork
import argparse

opt = argparse.Namespace(
    asr_model='ave', emb=False, att=2, exp_eye=True, fix_eye=-1,
    smooth_eye=False, smooth_lips=False, bs_area='upper', au45=False,
    torso=False, torso_shrink=0.8, bound=1, fp16=False, cuda_ray=True,
    density_thresh=10, density_thresh_torso=0.01, min_near=0.05,
    ind_dim=4, ind_dim_torso=8, ind_num=20000, amb_dim=2,
    train_camera=False, test_train=False, finetune_lips=False,
    init_lips=False, unc_loss=1,
)
model = NeRFNetwork(opt)

# AVE: input is a pre-encoded 512-dim vector per frame window [1, 1, 512]
ave_feat = torch.rand(1, 1, 512)           # from AudioEncoder (visual encoder)
enc_a_ave = model.encode_audio(ave_feat)   # -> [1, 32]

# DeepSpeech: input is [8, 29, 16] (8-frame window, 29 labels, 16 time steps)
ds_feat = torch.rand(8, 29, 16)
model_ds = NeRFNetwork(argparse.Namespace(**{**vars(opt), 'asr_model': 'deepspeech'}))
enc_a_ds = model_ds.encode_audio(ds_feat)  # -> [1, 32]

# HuBERT: input is [8, 1024, 16]
model_hu = NeRFNetwork(argparse.Namespace(**{**vars(opt), 'asr_model': 'hubert'}))
hu_feat  = torch.rand(8, 1024, 16)
enc_a_hu = model_hu.encode_audio(hu_feat)  # -> [1, 32]

print(enc_a_ave.shape)  # torch.Size([1, 32])
```
```
Tokens:
Raw