### Start Web Demo

Source: https://github.com/jdh-algo/joyvasa/blob/main/README.md

Start the web-based demo for JoyVASA. The demo will be accessible at http://127.0.0.1:7862.

```bash
python app.py
```

--------------------------------

### Download wav2vec2-base Checkpoints

Source: https://github.com/jdh-algo/joyvasa/blob/main/README.md

Use these commands to download the wav2vec2-base pre-trained weights. Ensure git-lfs is installed.

```bash
git lfs install
git clone https://huggingface.co/facebook/wav2vec2-base-960h
```

--------------------------------

### Install FFmpeg on Ubuntu

Source: https://github.com/jdh-algo/joyvasa/blob/main/README.md

Installs FFmpeg, a crucial multimedia framework, on Ubuntu systems. This is necessary for video processing tasks within the project.

```bash
sudo apt-get update
sudo apt-get install ffmpeg -y
```

--------------------------------

### Install MultiScaleDeformableAttention for Animal Animation

Source: https://github.com/jdh-algo/joyvasa/blob/main/README.md

Compiles and installs the MultiScaleDeformableAttention module, which is optional and required for animating animal images. Navigate to the specified directory before running the setup.

```bash
cd src/utils/dependencies/XPose/models/UniPose/ops
python setup.py build install
cd - # equal to cd ../../../../../../../
```

--------------------------------

### Programmatic Training Setup

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

This Python script demonstrates how to set up and configure the training process programmatically, mirroring the functionality of the `train.py` script. It initializes the model, datasets, data loaders, and optimizer.

```APIDOC
# Programmatic training setup (mirrors train.py __main__)
import argparse, torch
from pathlib import Path
from src.modules.dit_talking_head import DitTalkingHead
from src.dataset.talkinghead_dataset_hungry import TalkingHeadDatasetHungry
from torch.utils import data

device = "cuda"
model = DitTalkingHead(
    device=device, target="sample", audio_model="hubert_zh",
    n_motions=100, n_prev_motions=25, feature_dim=256,
    n_diff_steps=50, diff_schedule="cosine",
    cfg_mode="incremental", guiding_conditions="audio,"
)

train_ds = TalkingHeadDatasetHungry("data/", split="train", n_motions=100)
val_ds   = TalkingHeadDatasetHungry("data/", split="val",   n_motions=100)
train_loader = data.DataLoader(train_ds, batch_size=16, shuffle=True, num_workers=4)
val_loader   = data.DataLoader(val_ds,   batch_size=16, shuffle=False, num_workers=4)

optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-4)
# Checkpoints saved every 1000 iters to experiments/JoyVASA/<exp_name>/checkpoints/
```

--------------------------------

### Programmatic Training Setup

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Sets up and configures the training environment programmatically, mirroring the `train.py` script's main execution. Requires PyTorch and custom modules.

```python
import argparse, torch
from pathlib import Path
from src.modules.dit_talking_head import DitTalkingHead
from src.dataset.talkinghead_dataset_hungry import TalkingHeadDatasetHungry
from torch.utils import data

device = "cuda"
model = DitTalkingHead(
    device=device, target="sample", audio_model="hubert_zh",
    n_motions=100, n_prev_motions=25, feature_dim=256,
    n_diff_steps=50, diff_schedule="cosine",
    cfg_mode="incremental", guiding_conditions="audio,"
)

train_ds = TalkingHeadDatasetHungry("data/", split="train", n_motions=100)
val_ds   = TalkingHeadDatasetHungry("data/", split="val",   n_motions=100)
train_loader = data.DataLoader(train_ds, batch_size=16, shuffle=True, num_workers=4)
val_loader   = data.DataLoader(val_ds,   batch_size=16, shuffle=False, num_workers=4)

optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-4)
# Checkpoints saved every 1000 iters to experiments/JoyVASA/<exp_name>/checkpoints/
```

--------------------------------

### Download hubert-chinese Checkpoints

Source: https://github.com/jdh-algo/joyvasa/blob/main/README.md

Use these commands to download the hubert-chinese pre-trained weights. Ensure git-lfs is installed.

```bash
git lfs install
git clone https://huggingface.co/TencentGameMate/chinese-hubert-base
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/jdh-algo/joyvasa/blob/main/README.md

Installs all required Python packages listed in the requirements.txt file. This command should be run after activating the Conda environment.

```bash
pip install -r requirements.txt
```

--------------------------------

### Create and Activate Conda Environment

Source: https://github.com/jdh-algo/joyvasa/blob/main/README.md

Sets up a dedicated Conda environment for JoyVASA with Python 3.10. Ensure you have Conda installed.

```bash
conda create -n joyvasa python=3.10 -y
conda activate joyvasa
```

--------------------------------

### Download JoyVASA Motion Generator Checkpoints

Source: https://github.com/jdh-algo/joyvasa/blob/main/README.md

Clones the Hugging Face repository containing the pre-trained motion generator checkpoints for JoyVASA. Ensure git-lfs is installed and initialized before cloning.

```bash
git lfs install
git clone https://huggingface.co/jdh-algo/JoyVASA
```

--------------------------------

### Basic Training Run with Defaults

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Initiates a standard training process using default parameters for batch size, learning rate, and iterations. Ensure data and motion files are correctly placed.

```bash
python train.py \
  --exp_name my_experiment \
  --data_root data/ \
  --motion_filename motions.pkl \
  --motion_templete_filename motion_templete.pkl \
  --batch_size 16 \
  --max_iter 50000 \
  --lr 1e-4 \
  --audio_model hubert_zh \
  --n_diff_steps 50 \
  --diff_schedule cosine \
  --scheduler WarmupThenDecay \
  --warm_iter 2000 \
  --cos_max_iter 12000
```

--------------------------------

### Run JoyVASA Gradio Demo on All Interfaces

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Bind the Gradio demo to all network interfaces using `--server_name 0.0.0.0` and specify a port with `--server_port`.

```bash
python app.py --server_name 0.0.0.0 --server_port 8080
```

--------------------------------

### DitTalkingHead Initialization and Usage

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Demonstrates how to initialize the DitTalkingHead model with various configurations and perform a basic sampling inference.

```APIDOC
## `DitTalkingHead` — Diffusion Transformer Motion Generator

PyTorch `nn.Module` implementing the core diffusion model for motion generation. Supports `wav2vec2`, `hubert`, and `hubert_zh` audio encoders, configurable diffusion schedules (`linear`, `cosine`, `quadratic`, `sigmoid`), and classifier-free guidance via `cfg_scale`.

```python
import torch
from src.modules.dit_talking_head import DitTalkingHead

device = "cuda" if torch.cuda.is_available() else "cpu"

model = DitTalkingHead(
    device=device,
    target="sample",          # predict original sample (vs "noise")
    architecture="decoder",
    motion_feat_dim=70,       # 63 exp + 1 scale + 3 t + 3 head angles
    fps=25,
    n_motions=100,            # frames per window
    n_prev_motions=25,        # context frames from previous window
    audio_model="hubert_zh",  # "wav2vec2" | "hubert" | "hubert_zh"
    feature_dim=256,
    n_diff_steps=50,
    diff_schedule="cosine",
    cfg_mode="incremental",   # "incremental" | "independent"
    guiding_conditions="audio,"
)
model.eval()

# Sampling (inference) — 4 seconds of audio at 16kHz, window of 100 frames
n_motions = 100
audio_len = int(16000 * n_motions / 25)  # 64000 samples
audio_in = torch.randn(1, audio_len).to(device)  # (B, L_audio)

with torch.no_grad():
    motion_feat, noise, audio_feat = model.sample(
        audio_or_feat=audio_in,
        cfg_scale=2.0,         # classifier-free guidance strength
        cfg_mode="incremental",
        flexibility=0,
    )
print(motion_feat.shape)  # (1, 100, 70) — 100 frames × 70-dim motion vector
```
```

--------------------------------

### Autoregressive Sampling with DitTalkingHead.sample()

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Demonstrates autoregressive motion generation by chaining samples across windows, using previous motion and audio features as context. The `motion_at_T` parameter can be used to warm-start noise for smoother transitions.

```python
import torch
from src.modules.dit_talking_head import DitTalkingHead

device = "cuda"
model = DitTalkingHead(device=device, n_motions=100, n_prev_motions=25,
                       audio_model="hubert_zh", n_diff_steps=50).eval()

n_motions, fps = 100, 25
audio_len = int(16000 * n_motions / fps)
audio_window_1 = torch.randn(1, audio_len).to(device)
audio_window_2 = torch.randn(1, audio_len).to(device)

with torch.no_grad():
    # First window — no previous context
    motion_w1, noise, prev_audio_feat = model.sample(
        audio_window_1, cfg_scale=2.0
    )
    prev_motion_feat = motion_w1[:, -25:]  # last 25 frames as context

    # Second window — continue from first window context
    motion_w2, noise, prev_audio_feat = model.sample(
        audio_window_2,
        prev_motion_feat=prev_motion_feat,
        prev_audio_feat=prev_audio_feat[:, -25:],
        motion_at_T=noise,          # warm-start noise for smooth transitions
        cfg_scale=2.0,
    )

print(motion_w1.shape)  # (1, 100, 70)
print(motion_w2.shape)  # (1, 100, 70)
```

--------------------------------

### DitTalkingHead.sample() Method

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Details the `sample` method for generating motion sequences from audio, including support for autoregressive continuation with previous context.

```APIDOC
## `DitTalkingHead.sample()` — Autoregressive Diffusion Sampling

Runs the full reverse diffusion process to generate a motion sequence from audio, with optional classifier-free guidance. Supports autoregressive continuation across windows by passing `prev_motion_feat` and `prev_audio_feat` from the previous window.

```python
import torch
from src.modules.dit_talking_head import DitTalkingHead

device = "cuda"
model = DitTalkingHead(device=device, n_motions=100, n_prev_motions=25,
                       audio_model="hubert_zh", n_diff_steps=50).eval()

n_motions, fps = 100, 25
audio_len = int(16000 * n_motions / fps)
audio_window_1 = torch.randn(1, audio_len).to(device)
audio_window_2 = torch.randn(1, audio_len).to(device)

with torch.no_grad():
    # First window — no previous context
    motion_w1, noise, prev_audio_feat = model.sample(
        audio_window_1, cfg_scale=2.0
    )
    prev_motion_feat = motion_w1[:, -25:]  # last 25 frames as context

    # Second window — continue from first window context
    motion_w2, noise, prev_audio_feat = model.sample(
        audio_window_2,
        prev_motion_feat=prev_motion_feat,
        prev_audio_feat=prev_audio_feat[:, -25:],
        motion_at_T=noise,          # warm-start noise for smooth transitions
        cfg_scale=2.0,
    )

print(motion_w1.shape)  # (1, 100, 70)
print(motion_w2.shape)  # (1, 100, 70)
```
```

--------------------------------

### Basic Training Run

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

This command initiates a basic training run using default parameters for batch size, learning rate, and iterations. It specifies experiment name, data paths, and various training configurations.

```APIDOC
## Basic training run with defaults (batch=16, lr=1e-4, 50k iters)
python train.py \
  --exp_name my_experiment \
  --data_root data/ \
  --motion_filename motions.pkl \
  --motion_templete_filename motion_templete.pkl \
  --batch_size 16 \
  --max_iter 50000 \
  --lr 1e-4 \
  --audio_model hubert_zh \
  --n_diff_steps 50 \
  --diff_schedule cosine \
  --scheduler WarmupThenDecay \
  --warm_iter 2000 \
  --cos_max_iter 12000

# Results saved to: experiments/JoyVASA/my_experiment/checkpoints/iter_XXXXXXX.pt
# TensorBoard logs:  experiments/JoyVASA/my_experiment/logs/
```

--------------------------------

### Run JoyVASA CLI for Animal Animation

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Use this command to animate an animal image with a WAV audio file. Adjust `--cfg_scale` for diffusion strength.

```bash
python inference.py \
  -r assets/examples/imgs/joyvasa_001.png \
  -a assets/examples/audios/joyvasa_001.wav \
  --animation_mode animal \
  --cfg_scale 2.0
```

--------------------------------

### Initialize DitTalkingHead Model and Perform Sampling

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Initializes the DitTalkingHead model with specified parameters and performs a sampling inference to generate motion features from audio input. Ensure the model is in evaluation mode.

```python
import torch
from src.modules.dit_talking_head import DitTalkingHead

device = "cuda" if torch.cuda.is_available() else "cpu"

model = DitTalkingHead(
    device=device,
    target="sample",          # predict original sample (vs "noise")
    architecture="decoder",
    motion_feat_dim=70,       # 63 exp + 1 scale + 3 t + 3 head angles
    fps=25,
    n_motions=100,            # frames per window
    n_prev_motions=25,        # context frames from previous window
    audio_model="hubert_zh",  # "wav2vec2" | "hubert" | "hubert_zh"
    feature_dim=256,
    n_diff_steps=50,
    diff_schedule="cosine",
    cfg_mode="incremental",   # "incremental" | "independent"
    guiding_conditions="audio,",
)
model.eval()

# Sampling (inference) — 4 seconds of audio at 16kHz, window of 100 frames
n_motions = 100
audio_len = int(16000 * n_motions / 25)  # 64000 samples
audio_in = torch.randn(1, audio_len).to(device)  # (B, L_audio)

with torch.no_grad():
    motion_feat, noise, audio_feat = model.sample(
        audio_or_feat=audio_in,
        cfg_scale=2.0,         # classifier-free guidance strength
        cfg_mode="incremental",
        flexibility=0,
    )
print(motion_feat.shape)  # (1, 100, 70) — 100 frames × 70-dim motion vector
```

--------------------------------

### Run JoyVASA CLI for Human Animation

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Use this command to animate a human portrait with a WAV audio file. Adjust `--cfg_scale` for diffusion strength.

```bash
python inference.py \
  -r assets/examples/imgs/joyvasa_003.png \
  -a assets/examples/audios/joyvasa_003.wav \
  --animation_mode human \
  --cfg_scale 2.0
```

--------------------------------

### Data Preparation Pipeline Steps

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

A sequence of bash scripts to process raw video data into training-ready formats. Ensure to edit `root_dir` in `01_extract_motions.py` before execution.

```bash
# Step 1: Extract 3D motion coefficients from each video → .pkl per video
# Edit root_dir inside the script first:
#   root_dir = "/path/to/your/mp4/dataset"
cd src/prepare_data
python 01_extract_motions.py

# Step 2: Extract audio → .wav per video (16 kHz mono)
python 05_extract_audio.py

# Step 3: Generate train.json / test.json label files (80/20 split)
python 02_gen_labels.py

# Step 4: Merge all per-video .pkl files into one motions.pkl
python 03_merge_motions.py

# Step 5: Compute normalization statistics → motion_templete.pkl
python 04_gen_template.py

# Move outputs to data/ directory
mv motion_templete.pkl motions.pkl train.json test.json ../../data/
cd ../..

# Expected data/ structure:
# data/
# ├── train.json          (list of {video_name, audio_name, motion_name})
# ├── test.json
# ├── motions.pkl         (dict keyed by audio_name → motion sequence)
# └── motion_templete.pkl (mean/std/min/max stats for normalization)
```

--------------------------------

### Share JoyVASA Gradio Demo Publicly

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Use the `--share` flag to create a public Gradio tunnel for the web demo, allowing external access.

```bash
python app.py --share
```

--------------------------------

### Prepare Training Data for Motion Generator

Source: https://github.com/jdh-algo/joyvasa/blob/main/README.md

Prepare training and validation data for the motion generator by extracting motions and audio, generating labels, merging motions, and creating templates. Ensure to update the root_dir in the script.

```bash
cd src/prepare_data
python 01_extract_motions.py
python 05_extract_audio.py
python 02_gen_labels.py
pyhton 03_merge_motions.py
python 04_gen_template.py

mv motion_templete.pkl motions.pkl train.json test.json ../../data
cd ../..

```

--------------------------------

### Execute LivePortrait Pipeline

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Runs the full human animation pipeline, including cropping, motion generation, warping, rendering, and audio muxing. Ensure all necessary configurations are set before execution.

```python
from src.config.argument_config import ArgumentConfig
from src.config.inference_config import InferenceConfig
from src.config.crop_config import CropConfig
from src.live_portrait_wmg_pipeline import LivePortraitPipeline

def partial_fields(target_class, kwargs):
    return target_class(**{k: v for k, v in kwargs.items() if hasattr(target_class, k)})

args = ArgumentConfig()
args.animation_mode = "human"
args.reference = "assets/examples/imgs/joyvasa_003.png"
args.audio = "assets/examples/audios/joyvasa_003.wav"
args.cfg_scale = 2.0
args.flag_do_crop = True
args.flag_pasteback = True
args.output_dir = "./animations/"

inference_cfg = partial_fields(InferenceConfig, args.__dict__)
crop_cfg = partial_fields(CropConfig, args.__dict__)

pipeline = LivePortraitPipeline(inference_cfg=inference_cfg, crop_cfg=crop_cfg)

# Runs full pipeline: crop → motion gen → warp → render → mux audio
output_video_path = pipeline.execute(args)
print(f"Saved to: {output_video_path}")
# Expected output: ./animations/joyvasa_003_joyvasa_003.mp4
```

--------------------------------

### LivePortraitPipeline.execute()

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Executes the full human animation pipeline, including cropping, motion generation, warping, rendering, and audio muxing. Returns the path to the final MP4 video.

```APIDOC
## `LivePortraitPipeline.execute()` — Human Animation Pipeline

Core inference pipeline for human portraits. Loads the reference image, generates a motion sequence from audio via the diffusion model, then renders each frame by warping 3D appearance features with the generated keypoints. Returns the path to the final MP4 video.

```python
from src.config.argument_config import ArgumentConfig
from src.config.inference_config import InferenceConfig
from src.config.crop_config import CropConfig
from src.live_portrait_wmg_pipeline import LivePortraitPipeline

def partial_fields(target_class, kwargs):
    return target_class(**{k: v for k, v in kwargs.items() if hasattr(target_class, k)})

args = ArgumentConfig()
args.animation_mode = "human"
args.reference = "assets/examples/imgs/joyvasa_003.png"
args.audio = "assets/examples/audios/joyvasa_003.wav"
args.cfg_scale = 2.0
args.flag_do_crop = True
args.flag_pasteback = True
args.output_dir = "./animations/"

inference_cfg = partial_fields(InferenceConfig, args.__dict__)
crop_cfg = partial_fields(CropConfig, args.__dict__)

pipeline = LivePortraitPipeline(inference_cfg=inference_cfg, crop_cfg=crop_cfg)

# Runs full pipeline: crop → motion gen → warp → render → mux audio
output_video_path = pipeline.execute(args)
print(f"Saved to: {output_video_path}")
# Expected output: ./animations/joyvasa_003_joyvasa_003.mp4
```
```

--------------------------------

### Download LivePortraits Checkpoints

Source: https://github.com/jdh-algo/joyvasa/blob/main/README.md

Download LivePortraits checkpoints using the huggingface-cli. This command downloads to a specified local directory and excludes certain files.

```bash
# !pip install -U "huggingface_hub[cli]"
huggingface-cli download KwaiVGI/LivePortrait --local-dir pretrained_weights --exclude "*.git*" "README.md" "docs"
```

--------------------------------

### Data Preparation Pipeline

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

This section outlines a five-step process for preparing raw talking-head video data into training-ready formats. It includes extracting motion coefficients, audio, generating labels, merging motion files, and computing normalization statistics.

```APIDOC
## Data Preparation Pipeline — `src/prepare_data/`

A sequence of five scripts that convert raw talking-head video data into training-ready motion coefficient files and audio/label metadata.

```bash
# Step 1: Extract 3D motion coefficients from each video → .pkl per video
# Edit root_dir inside the script first:
#   root_dir = "/path/to/your/mp4/dataset"
cd src/prepare_data
python 01_extract_motions.py

# Step 2: Extract audio → .wav per video (16 kHz mono)
python 05_extract_audio.py

# Step 3: Generate train.json / test.json label files (80/20 split)
python 02_gen_labels.py

# Step 4: Merge all per-video .pkl files into one motions.pkl
python 03_merge_motions.py

# Step 5: Compute normalization statistics → motion_templete.pkl
python 04_gen_template.py

# Move outputs to data/ directory
mv motion_templete.pkl motions.pkl train.json test.json ../../data/
cd ../..

# Expected data/ structure:
# data/
# ├── train.json          (list of {video_name, audio_name, motion_name})
# ├── test.json
# ├── motions.pkl         (dict keyed by audio_name → motion sequence)
# └── motion_templete.pkl (mean/std/min/max stats for normalization)
```
```

--------------------------------

### Initialize and Use TalkingHeadDatasetHungry for Training

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Sets up the TalkingHeadDatasetHungry for training motion generation models. This dataset loads audio-motion pairs, normalizes motion data, and prepares consecutive clips for autoregressive training. Ensure the data directory contains the specified .pkl and .json files.

```python
from torch.utils import data
from src.dataset.talkinghead_dataset_hungry import TalkingHeadDatasetHungry

# Expects data/ to contain: train.json, test.json, motions.pkl, motion_templete.pkl
train_dataset = TalkingHeadDatasetHungry(
    root_dir="data/",
    motion_filename="motions.pkl",
    motion_templete_filename="motion_templete.pkl",
    split="train",
    coef_fps=25,
    n_motions=100,          # frames per training window
    crop_strategy="random", # "random" | "begin" | "end"
    normalize_type="mix",   # std-norm for exp, min-max for pose
)

train_loader = data.DataLoader(
    train_dataset, batch_size=16, shuffle=True, num_workers=4, pin_memory=True
)

for audio_pair, coef_pair in train_loader:
    # audio_pair: list of 2 tensors, each (B, 64000) — two consecutive windows
    # coef_pair:  list of 2 dicts, each with:
    #   'exp':  (B, 100, 63) normalized expression coefficients
    #   'pose': (B, 100, 7)  normalized scale/translation/rotation
    print(audio_pair[0].shape)         # torch.Size([16, 64000])
    print(coef_pair[0]['exp'].shape)   # torch.Size([16, 100, 63])
    print(coef_pair[0]['pose'].shape)  # torch.Size([16, 100, 7])
    break
```

--------------------------------

### Customize JoyVASA CLI Output Directory

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Specify a custom output directory for the generated animation using the `-o` flag. Adjust `--cfg_scale` for diffusion strength.

```bash
python inference.py \
  -r assets/examples/imgs/joyvasa_005.png \
  -a assets/examples/audios/joyvasa_005.wav \
  --animation_mode human \
  --cfg_scale 2.8 \
  -o ./my_output/
```

--------------------------------

### Inference with Animal Animation Mode

Source: https://github.com/jdh-algo/joyvasa/blob/main/README.md

Run inference for animal animation using a reference image and audio file. Adjust cfg_scale for different expressions and poses.

```python
python inference.py -r assets/examples/imgs/joyvasa_001.png -a assets/examples/audios/joyvasa_001.wav --animation_mode animal --cfg_scale 2.0
```

--------------------------------

### LivePortraitWrapper.gen_motion_sequence()

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Generates an audio-driven motion sequence by loading audio, segmenting it, and using a diffusion model to sample motion parameters. Returns a dictionary of per-frame motion parameters.

```APIDOC
## `LivePortraitWrapper.gen_motion_sequence()` — Audio-Driven Motion Generation

Loads a WAV audio file, segments it into overlapping windows aligned to the model's `n_motions` frame count, and autoregressively samples motion coefficient sequences using the diffusion model. Returns a dictionary of per-frame motion parameters (expression, scale, translation, rotation) ready for rendering.

```python
import librosa
import torch
from src.live_portrait_wmg_wrapper import LivePortraitWrapper
from src.config.inference_config import InferenceConfig
from src.config.argument_config import ArgumentConfig

inference_cfg = InferenceConfig()
wrapper = LivePortraitWrapper(inference_cfg=inference_cfg)

args = ArgumentConfig()
args.audio = "assets/examples/audios/joyvasa_003.wav"
args.cfg_scale = 2.0
args.cfg_mode = "incremental"
args.cfg_cond = None          # uses model default guiding conditions
args.is_smooth_motion = True  # apply EMA smoothing

motion_dict = wrapper.gen_motion_sequence(args)

print(motion_dict['n_frames'])    # e.g., 150 (depends on audio length)
print(motion_dict['output_fps'])  # 25
frame_0 = motion_dict['motion'][0]
print(frame_0.keys())
# dict_keys(['exp', 'scale', 'R', 't', 'pitch', 'yaw', 'roll'])
# frame_0['exp'].shape   -> (1, 21, 3)
# frame_0['R'].shape     -> (1, 3, 3)
# frame_0['scale'].shape -> (1, 1)
```
```

--------------------------------

### Inference with Human Animation Mode

Source: https://github.com/jdh-algo/joyvasa/blob/main/README.md

Run inference for human animation using a reference image and audio file. Adjust cfg_scale for different expressions and poses. Ensure the animation mode matches the reference image type.

```python
python inference.py -r assets/examples/imgs/joyvasa_003.png -a assets/examples/audios/joyvasa_003.wav --animation_mode human --cfg_scale 2.0
```

--------------------------------

### Train Motion Generator

Source: https://github.com/jdh-algo/joyvasa/blob/main/README.md

Execute the training script for the motion generator. The experimental results will be saved in the 'experiments/' directory.

```bash
python train.py
```

--------------------------------

### Configure JoyVASA Arguments with ArgumentConfig

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Instantiate and modify the `ArgumentConfig` dataclass to set parameters for JoyVASA inference. This config centralizes all user-facing arguments.

```python
from src.config.argument_config import ArgumentConfig
from dataclasses import asdict

# Default config (animal mode, example assets)
cfg = ArgumentConfig()
print(cfg.animation_mode)   # "animal"
print(cfg.cfg_scale)        # 2.8
print(cfg.driving_option)   # "expression-friendly"

# Modify for a human inference run
cfg.animation_mode = "human"
cfg.reference = "assets/examples/imgs/joyvasa_003.png"
cfg.audio = "assets/examples/audios/joyvasa_003.wav"
cfg.cfg_scale = 2.0
cfg.flag_do_crop = True
cfg.flag_pasteback = True
cfg.flag_stitching = True
cfg.output_dir = "./animations/"

# Key animation parameters:
# cfg.driving_option: "expression-friendly" | "pose-friendly"
# cfg.driving_multiplier: float (default 1.10) — scales expression amplitude
# cfg.animation_region: "all" | "exp" | "pose" | "lip" | "eyes"
# cfg.is_smooth_motion: bool — apply EMA smoothing to motion sequence
# cfg.cfg_mode: "incremental" | "independent" — CFG accumulation strategy
```

--------------------------------

### Gradio Pipeline Execution Handler

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Programmatically executes the animation pipeline using provided configurations and input files. Updates internal settings and returns the path to the generated video.

```python
from src.config.argument_config import ArgumentConfig
from src.config.inference_config import InferenceConfig
from src.config.crop_config import CropConfig
from src.gradio_pipeline import GradioPipeline

def partial_fields(target_class, kwargs):
    return target_class(**{k: v for k, v in kwargs.items() if hasattr(target_class, k)})

args = ArgumentConfig()
inference_cfg = partial_fields(InferenceConfig, args.__dict__)
crop_cfg = partial_fields(CropConfig, args.__dict__)

pipeline = GradioPipeline(inference_cfg=inference_cfg, crop_cfg=crop_cfg, args=args)

output_path = pipeline.execute_a2v(
    input_image="assets/examples/imgs/joyvasa_005.png",
    input_audio="assets/examples/audios/joyvasa_005.wav",
    animation_mode="human",
    flag_normalize_lip=True,
    flag_relative_motion=True,
    driving_multiplier=1.0,
    driving_option_input="expression-friendly",
    flag_do_crop_input=True,
    scale=2.3,
    vx_ratio=0.0,
    vy_ratio=-0.125,
    flag_stitching_input=True,
    flag_remap_input=True,
    cfg_scale=2.0,
)
print(output_path)  # e.g., ./animations/joyvasa_005_joyvasa_005.mp4
```

--------------------------------

### GradioPipeline.execute_a2v()

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

This function serves as the Gradio backend handler, providing a programmatic interface to the animation logic. It updates configuration, runs the full pipeline, and returns the path to the generated output video. It is callable for custom integrations.

```APIDOC
## `GradioPipeline.execute_a2v()` — Gradio Backend Handler

Programmatic interface to the same animation logic as the web demo. Updates internal config from user-provided kwargs, runs the full pipeline, and returns the output video path. Used internally by `app.py` but callable directly for custom integrations.

```python
from src.config.argument_config import ArgumentConfig
from src.config.inference_config import InferenceConfig
from src.config.crop_config import CropConfig
from src.gradio_pipeline import GradioPipeline

def partial_fields(target_class, kwargs):
    return target_class(**{k: v for k, v in kwargs.items() if hasattr(target_class, k)})

args = ArgumentConfig()
inference_cfg = partial_fields(InferenceConfig, args.__dict__)
crop_cfg = partial_fields(CropConfig, args.__dict__)

pipeline = GradioPipeline(inference_cfg=inference_cfg, crop_cfg=crop_cfg, args=args)

output_path = pipeline.execute_a2v(
    input_image="assets/examples/imgs/joyvasa_005.png",
    input_audio="assets/examples/audios/joyvasa_005.wav",
    animation_mode="human",
    flag_normalize_lip=True,
    flag_relative_motion=True,
    driving_multiplier=1.0,
    driving_option_input="expression-friendly",
    flag_do_crop_input=True,
    scale=2.3,
    vx_ratio=0.0,
    vy_ratio=-0.125,
    flag_stitching_input=True,
    flag_remap_input=True,
    cfg_scale=2.0,
)
print(output_path)  # e.g., ./animations/joyvasa_005_joyvasa_005.mp4
```
```

--------------------------------

### Generate Audio-Driven Motion Sequence

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Generates motion sequences from audio files using a diffusion model. Supports various configuration options for motion smoothing and guidance. The output includes frame counts and motion parameters.

```python
import librosa
import torch
from src.live_portrait_wmg_wrapper import LivePortraitWrapper
from src.config.inference_config import InferenceConfig
from src.config.argument_config import ArgumentConfig

inference_cfg = InferenceConfig()
wrapper = LivePortraitWrapper(inference_cfg=inference_cfg)

args = ArgumentConfig()
args.audio = "assets/examples/audios/joyvasa_003.wav"
args.cfg_scale = 2.0
args.cfg_mode = "incremental"
args.cfg_cond = None          # uses model default guiding conditions
args.is_smooth_motion = True  # apply EMA smoothing

motion_dict = wrapper.gen_motion_sequence(args)

print(motion_dict['n_frames'])    # e.g., 150 (depends on audio length)
print(motion_dict['output_fps'])  # 25
frame_0 = motion_dict['motion'][0]
print(frame_0.keys())
# dict_keys(['exp', 'scale', 'R', 't', 'pitch', 'yaw', 'roll'])
# frame_0['exp'].shape   -> (1, 21, 3)
# frame_0['R'].shape     -> (1, 3, 3)
# frame_0['scale'].shape -> (1, 1)
```

--------------------------------

### Warp and Decode Frame for Rendering

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Renders a single frame by warping 3D appearance features using source and target keypoints, then decoding with a SPADE generator. Requires pre-processed image tensors and keypoint information.

```python
import torch
from src.live_portrait_wmg_wrapper import LivePortraitWrapper
from src.config.inference_config import InferenceConfig

inference_cfg = InferenceConfig()
wrapper = LivePortraitWrapper(inference_cfg=inference_cfg)

# Assume source image already prepared (256x256 normalized tensor)
import cv2
import numpy as np
img = cv2.imread("assets/examples/imgs/joyvasa_003.png")
img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img_resized = cv2.resize(img_rgb, (256, 256))

I_s = wrapper.prepare_source(img_resized)           # (1, 3, 256, 256) on device
x_s_info = wrapper.get_kp_info(I_s)                 # dict with 'kp','exp','pitch','yaw','roll','scale','t'
f_s = wrapper.extract_feature_3d(I_s)               # (1, 32, 16, 64, 64)
x_s = wrapper.transform_keypoint(x_s_info)          # (1, N_kp, 3)

# x_d_i_new: target keypoints for a single frame (same shape as x_s)
x_d_i_new = x_s.clone()  # identity: no motion

out_dct = wrapper.warp_decode(f_s, x_s, x_d_i_new)
frame_uint8 = wrapper.parse_output(out_dct['out'])  # (1, 256, 256, 3) uint8
print(frame_uint8.shape)  # (1, 256, 256, 3)
```

--------------------------------

### TalkingHeadDatasetHungry Dataset

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Details the `TalkingHeadDatasetHungry` PyTorch Dataset for preparing audio and motion data for training the motion generator.

```APIDOC
## `TalkingHeadDatasetHungry` — Training Dataset

PyTorch `Dataset` for training the motion generator. Loads pre-extracted motion coefficients (`.pkl`) and audio (`.wav`) pairs from JSON label files, normalizes them using a motion template, and yields paired consecutive clips for autoregressive training.

```python
from torch.utils import data
from src.dataset.talkinghead_dataset_hungry import TalkingHeadDatasetHungry

# Expects data/ to contain: train.json, test.json, motions.pkl, motion_templete.pkl
train_dataset = TalkingHeadDatasetHungry(
    root_dir="data/",
    motion_filename="motions.pkl",
    motion_templete_filename="motion_templete.pkl",
    split="train",
    coef_fps=25,
    n_motions=100,          # frames per training window
    crop_strategy="random", # "random" | "begin" | "end"
    normalize_type="mix",   # std-norm for exp, min-max for pose
)

train_loader = data.DataLoader(
    train_dataset,
    batch_size=16,
    shuffle=True,
    num_workers=4,
    pin_memory=True,
)

for audio_pair, coef_pair in train_loader:
    # audio_pair: list of 2 tensors, each (B, 64000) — two consecutive windows
    # coef_pair:  list of 2 dicts, each with:
    #   'exp':  (B, 100, 63) normalized expression coefficients
    #   'pose': (B, 100, 7)  normalized scale/translation/rotation
    print(audio_pair[0].shape)         # torch.Size([16, 64000])
    print(coef_pair[0]['exp'].shape)   # torch.Size([16, 100, 63])
    print(coef_pair[0]['pose'].shape)  # torch.Size([16, 100, 7])
    break
```
```

--------------------------------

### LivePortraitWrapper.warp_decode()

Source: https://context7.com/jdh-algo/joyvasa/llms.txt

Renders a frame by warping a 3D appearance feature volume using source and target keypoints, then decoding the result with a SPADE generator. Returns the rendered frame tensor.

```APIDOC
## `LivePortraitWrapper.warp_decode()` — Frame Rendering

Takes the 3D appearance feature volume and source/target keypoints, warps the feature volume via the warping module, then decodes the result with the SPADE generator to produce a rendered frame tensor.

```python
import torch
from src.live_portrait_wmg_wrapper import LivePortraitWrapper
from src.config.inference_config import InferenceConfig

inference_cfg = InferenceConfig()
wrapper = LivePortraitWrapper(inference_cfg=inference_cfg)

# Assume source image already prepared (256x256 normalized tensor)
import cv2
import numpy as np
img = cv2.imread("assets/examples/imgs/joyvasa_003.png")
img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img_resized = cv2.resize(img_rgb, (256, 256))

I_s = wrapper.prepare_source(img_resized)           # (1, 3, 256, 256) on device
x_s_info = wrapper.get_kp_info(I_s)                 # dict with 'kp','exp','pitch','yaw','roll','scale','t'
f_s = wrapper.extract_feature_3d(I_s)               # (1, 32, 16, 64, 64)
x_s = wrapper.transform_keypoint(x_s_info)          # (1, N_kp, 3)

# x_d_i_new: target keypoints for a single frame (same shape as x_s)
x_d_i_new = x_s.clone()  # identity: no motion

out_dct = wrapper.warp_decode(f_s, x_s, x_d_i_new)
frame_uint8 = wrapper.parse_output(out_dct['out'])  # (1, 256, 256, 3) uint8
print(frame_uint8.shape)  # (1, 256, 256, 3)
```
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.