### Install Project Requirements

Source: https://github.com/jdh-algo/joyhallo/blob/main/README.md

Install all necessary Python packages listed in the requirements.txt file after activating the environment.

```bash
pip install -r requirements.txt
```

--------------------------------

### Start JoyHallo Web Demo

Source: https://github.com/jdh-algo/joyhallo/blob/main/README.md

Launch the web-based demo for JoyHallo. The demo will be accessible at http://127.0.0.1:7860.

```bash
sh joyhallo-app.sh
```

--------------------------------

### Install Conda Environment and Dependencies

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Sets up the Python 3.10 conda environment and installs all project dependencies from requirements.txt. Also installs ffmpeg for audio/video processing.

```bash
conda create -n joyhallo python=3.10 -y
conda activate joyhallo

pip install -r requirements.txt

sudo apt-get update && sudo apt-get install ffmpeg -y
```

--------------------------------

### Launch Training Stages with Accelerate

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Use the 'accelerate launch' command to start the training process for stage 1 or stage 2. Ensure you have the necessary configuration files.

```bash
#   accelerate launch scripts/train_stage1_alltrain.py --config configs/train/stage1_alltrain.yaml
#   accelerate launch scripts/train_stage2_alltrain.py --config configs/train/stage2_alltrain.yaml
```

```bash
# Stage 2 only (when Stage 1 checkpoint already exists)
sh joyhallo-train.sh
# Internally runs:
#   accelerate launch scripts/train_stage2.py --config configs/train/stage2.yaml
```

--------------------------------

### Launch Joyhallo Web Demo

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Start the Gradio web UI for interactive portrait and audio processing. Users can upload files and adjust generation parameters directly in the browser.

```bash
sh joyhallo-app.sh
```

```bash
python scripts/app.py
```

--------------------------------

### Install FFmpeg

Source: https://github.com/jdh-algo/joyhallo/blob/main/README.md

Install FFmpeg on Ubuntu 20.04 systems, which is required for video processing. This command updates package lists and then installs FFmpeg.

```bash
sudo apt-get update  
sudo apt-get install ffmpeg -y
```

--------------------------------

### Train JoyHallo from Stage 1

Source: https://github.com/jdh-algo/joyhallo/blob/main/README.md

Initiate the full training process for JoyHallo, starting from Stage 1. Training parameters can be adjusted in `configs/train/stage1_alltrain.yaml` and `configs/train/stage2_alltrain.yaml`.

```bash
sh joyhallo-alltrain.sh
```

--------------------------------

### Download Base Checkpoints

Source: https://github.com/jdh-algo/joyhallo/blob/main/README.md

Clone the 'hallo' repository from Hugging Face to download the base model weights. Ensure Git Large File Storage (LFS) is installed.

```shell
git lfs install
git clone https://huggingface.co/fudan-generative-ai/hallo pretrained_models
```

--------------------------------

### Download Chinese Wav2Vec2 Model

Source: https://github.com/jdh-algo/joyhallo/blob/main/README.md

Download the 'chinese-wav2vec2-base' model from Hugging Face into the 'pretrained_models' directory. Git LFS must be installed.

```shell
cd pretrained_models
git lfs install
git clone https://huggingface.co/TencentGameMate/chinese-wav2vec2-base 
```

--------------------------------

### Train JoyHallo Stage 2 Only

Source: https://github.com/jdh-algo/joyhallo/blob/main/README.md

Start training JoyHallo specifically for Stage 2. Training parameters can be adjusted in `configs/train/stage2.yaml`.

```bash
sh joyhallo-train.sh
```

--------------------------------

### Download JoyHallo Model

Source: https://github.com/jdh-algo/joyhallo/blob/main/README.md

Clone the JoyHallo-v1 model from Hugging Face into the 'pretrained_models/joyhallo' directory. Git LFS must be installed.

```shell
git lfs install
git clone https://huggingface.co/jdh-algo/JoyHallo-v1 pretrained_models/joyhallo
```

--------------------------------

### Gradio Predict Function Signature and Example

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Defines the signature for the Gradio predict function, which handles user inputs for image, audio, and various conditioning weights. An example demonstrates how to call this function directly from Python.

```python
# The Gradio predict() function signature — called automatically by the UI
def predict(
    image,             # filepath to a JPEG portrait (no WebP)
    audio,             # filepath to a WAV audio clip
    pose_weight,       # float, default 1.0 — strength of pose conditioning
    face_weight,       # float, default 1.0 — strength of face conditioning
    lip_weight,        # float, default 1.0 — strength of lip conditioning
    face_expand_ratio, # float, default 1.2 — face bounding box expansion
) -> str:              # returns path to generated "output.mp4"
    ...

# Example: call predict() directly from Python
output_path = predict(
    image="examples/reference_images/3.jpg",
    audio="examples/driving_audios/2.wav",
    pose_weight=1.0,
    face_weight=1.0,
    lip_weight=1.2,   # boost lip sync
    face_expand_ratio=1.2,
)
print(output_path)  # "output.mp4"
```

--------------------------------

### Get Raw Audio Embedding

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Obtain raw audio embeddings from a WAV file using the AudioProcessor without performing vocal separation or padding. This provides a lightweight way to get feature vectors directly from the audio.

```python
# Lightweight version — no vocal separation, no padding
raw_emb = processor.get_embedding("examples/driving_audios/0.wav")
print(raw_emb.shape)     # (T, 12, 768)
```

--------------------------------

### Programmatic Inference with OmegaConf

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Demonstrates how to load the inference configuration programmatically using OmegaConf and initiate the inference process.

```python
# Programmatic usage (mirrors what inference.py does internally)
from omegaconf import OmegaConf
from scripts.inference import inference_process

cfg = OmegaConf.load("configs/inference/inference.yaml")

```

--------------------------------

### Download Pretrained Models for JoyHallo

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Downloads all necessary pretrained weights, including base Hallo models, Chinese wav2vec2, and JoyHallo fine-tuned weights, into the ./pretrained_models/ directory.

```bash
git lfs install
git clone https://huggingface.co/fudan-generative-ai/hallo pretrained_models

cd pretrained_models
git lfs install
git clone https://huggingface.co/TencentGameMate/chinese-wav2vec2-base
cd ..

git lfs install
git clone https://huggingface.co/jdh-algo/JoyHallo-v1 pretrained_models/joyhallo
```

--------------------------------

### Data Preprocessing - Step 1

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Extracts frames, audio, and computes landmark-based masks from raw MP4 videos. This step is CPU-friendly and can be parallelized.

```bash
# Step 1 (CPU-friendly): extract frames, audio, and compute landmark-based masks
python -m scripts.data_preprocess \
    --input_dir jdh-Hallo/videos \
    --step 1 \
    -p 4 -r 0   # partition into 4 parallel jobs, run shard 0

# Repeat for -r 1, -r 2, -r 3 in separate processes/machines
```

--------------------------------

### Initialize and Use AudioProjModel

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Initializes the AudioProjModel to project raw wav2vec2 hidden states into per-frame context token sequences. Requires audio embeddings as input.

```python
from joyhallo.models.audio_proj import AudioProjModel
import torch

audioproj = AudioProjModel(
    seq_len=5,            # temporal window (±2 frames context)
    blocks=12,            # number of wav2vec2 hidden layers
    channels=768,         # hidden dimension
    intermediate_dim=512,
    output_dim=768,
    context_tokens=32,    # output tokens per frame fed to cross-attention
).to("cuda", dtype=torch.float16)

# audio_emb from process_audio_emb(): shape (B, T, 5, 12, 768)
audio_emb = torch.randn(1, 16, 5, 12, 768, device="cuda", dtype=torch.float16)
context_tokens = audioproj(audio_emb)
print(context_tokens.shape)  # torch.Size([1, 16, 32, 768])
```

--------------------------------

### Initialize AudioProcessor

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Instantiate the AudioProcessor with specified sample rate, frames per second, and paths to pre-trained models for wav2vec2 feature extraction and audio separation. Configuration options include whether to use only the last hidden layer and the device for processing.

```python
from joyhallo.datasets.audio_processor import AudioProcessor

processor = AudioProcessor(
    sample_rate=16000,
    fps=25,
    wav2vec_model_path="pretrained_models/chinese-wav2vec2-base",
    only_last_features=False,   # False = stack all 12 hidden layers → shape (T, 12, 768)
    audio_separator_model_path="pretrained_models/audio_separator",
    audio_separator_model_name="Kim_Vocal_2.onnx",
    cache_dir=".cache/audio",
    device="cuda:0",
)
```

--------------------------------

### Preprocess Training Data - Step 1

Source: https://github.com/jdh-algo/joyhallo/blob/main/README.md

Extract features from videos, including audio, and generate masks. This is the first step in preparing the dataset. Use `-p` for parallel processing and `-r` to specify the current process part.

```python
python -m scripts.data_preprocess --input_dir jdh-Hallo/videos --step 1 -p 1 -r 0
```

--------------------------------

### Initialize and Run FaceAnimatePipeline

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Initializes the FaceAnimatePipeline with pre-loaded models and runs inference using image and audio inputs. Ensure models are loaded and on CUDA.

```python
pipeline = FaceAnimatePipeline(
    vae=vae,
    reference_unet=reference_unet,
    denoising_unet=denoising_unet,
    face_locator=face_locator,
    image_proj=imageproj,
    scheduler=DDIMScheduler(...),
).to("cuda")

output = pipeline(
    ref_image=pixel_values_ref_img,          # (1, 1+n_motion, C, H, W)
    audio_tensor=audio_tensor,               # (1, clip_len, 32, 768)
    face_emb=source_image_face_emb,          # (1, 512) InsightFace embedding
    face_mask=source_image_face_region,      # (1, 1, C, H, W)
    pixel_values_full_mask=full_mask_list,   # List[4] background masks
    pixel_values_face_mask=face_mask_list,   # List[4] face masks
    pixel_values_lip_mask=lip_mask_list,     # List[4] lip masks
    width=512,
    height=512,
    video_length=16,                         # frames per clip
    num_inference_steps=40,
    guidance_scale=3.5,
    generator=torch.cuda.manual_seed_all(42),
)

# output.videos — torch.Tensor (1, C, 16, H, W), values in [0, 1]
print(output.videos.shape)  # torch.Size([1, 3, 16, 512, 512])
```

--------------------------------

### Data Preprocessing - Step 2

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Extracts InsightFace and wav2vec2 embeddings from preprocessed video data. This step requires a GPU.

```bash
# Step 2 (requires GPU): extract InsightFace + wav2vec2 embeddings
python -m scripts.data_preprocess \
    --input_dir jdh-Hallo/videos \
    --step 2 \
    -p 4 -r 0
```

--------------------------------

### Full Two-Stage Training

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Executes the full two-stage training process from scratch using the provided shell script.

```bash
# Full two-stage training from scratch
sh joyhallo-alltrain.sh
# Internally runs:
```

--------------------------------

### Preprocess Training Data - Step 2

Source: https://github.com/jdh-algo/joyhallo/blob/main/README.md

Generate face and audio embeddings using InsightFace and Chinese wav2vec2, respectively. This step requires a GPU. Use `-p` for parallel processing and `-r` to specify the current process part.

```python
python -m scripts.data_preprocess --input_dir jdh-Hallo/videos --step 2 -p 1 -r 0
```

--------------------------------

### Configure Stage 2 Training Hyperparameters

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Key training hyperparameters for stage 2 are defined in the YAML configuration file. Adjust settings like learning rate, training steps, and mixed precision for optimal results. Enable 8-bit Adam and xformers for memory efficiency if supported.

```yaml
# Key training hyperparameters (configs/train/stage2.yaml excerpt)
solver:
  learning_rate: 1.0e-5
  lr_scheduler: "constant"
  max_train_steps: 30000
  gradient_accumulation_steps: 1
  mixed_precision: "no"           # change to "fp16" to save VRAM
  use_8bit_adam: True             # requires bitsandbytes
  enable_xformers_memory_efficient_attention: True

trainable_para:
  - audio_modules    # AudioProjModel + audio cross-attention layers
  - motion_modules   # Temporal self-attention in UNet3D

# All other weights (VAE, reference UNet, face locator, imageproj) are frozen
```

--------------------------------

### Create and Activate Python Environment

Source: https://github.com/jdh-algo/joyhallo/blob/main/README.md

Use Conda to create a new environment named 'joyhallo' with Python 3.10 and activate it.

```bash
conda create -n joyhallo python=3.10 -y
conda activate joyhallo
```

--------------------------------

### Initialize ImageProcessor

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Set up the ImageProcessor with a target image size and the path to a face analysis model. This processor is responsible for detecting faces, generating masks, and extracting embeddings from input images.

```python
from joyhallo.datasets.image_processor import ImageProcessor

proc = ImageProcessor(
    img_size=(512, 512),
    face_analysis_model_path="pretrained_models/face_analysis",
)
```

--------------------------------

### Run JoyHallo Inference

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Executes the full inference pipeline using the specified configuration. Generates MP4 videos based on reference images and audio inputs.

```bash
# Quickstart — uses defaults from configs/inference/inference.yaml
sh joyhallo-infer.sh

# Or call directly:
python -m scripts.inference --config configs/inference/inference.yaml
```

--------------------------------

### Initialize FaceAnimatePipeline

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Import necessary components for the FaceAnimatePipeline, a subclass of `DiffusionPipeline`. This pipeline is designed for video generation using DDIM denoising conditioned on audio, face embeddings, and spatial masks.

```python
from diffusers import AutoencoderKL, DDIMScheduler
from joyhallo.animate.face_animate import FaceAnimatePipeline
from joyhallo.models.unet_2d_condition import UNet2DConditionModel
from joyhallo.models.unet_3d import UNet3DConditionModel
from joyhallo.models.face_locator import FaceLocator
from joyhallo.models.image_proj import ImageProjModel
```

--------------------------------

### Web Demo - Gradio predict()

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Launches a Gradio web UI for interactive generation. The `predict` function handles image and audio inputs, along with generation weights, and returns the path to the generated video.

```APIDOC
## predict()

### Description
Handles image and audio inputs for video generation, allowing interactive tuning of generation weights. Returns the filepath to the generated "output.mp4".

### Method
`predict(
    image: str,             # filepath to a JPEG portrait (no WebP)
    audio: str,             # filepath to a WAV audio clip
    pose_weight: float = 1.0,       # strength of pose conditioning
    face_weight: float = 1.0,       # strength of face conditioning
    lip_weight: float = 1.0,        # strength of lip conditioning
    face_expand_ratio: float = 1.2  # face bounding box expansion
) -> str:              # returns path to generated "output.mp4"
    ...

### Request Example
```python
output_path = predict(
    image="examples/reference_images/3.jpg",
    audio="examples/driving_audios/2.wav",
    pose_weight=1.0,
    face_weight=1.0,
    lip_weight=1.2,   # boost lip sync
    face_expand_ratio=1.2,
)
print(output_path)  # "output.mp4"
```
```

--------------------------------

### Build Meta JSON Files

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Builds meta JSON files required by the DataLoader after completing both data preprocessing steps.

```bash
# After both steps, build meta JSON files for the DataLoader:
python scripts/extract_meta_info_stage1.py -r jdh-Hallo -n jdh-Hallo
python scripts/extract_meta_info_stage2.py -r jdh-Hallo -n jdh-Hallo
```

--------------------------------

### Extract Meta Info for Stage 1

Source: https://github.com/jdh-algo/joyhallo/blob/main/README.md

Generate meta information for Stage 1 of the JoyHallo dataset. This script processes the prepared data.

```python
python scripts/extract_meta_info_stage1.py -r jdh-Hallo -n jdh-Hallo
```

--------------------------------

### Extract Meta Info for Stage 2

Source: https://github.com/jdh-algo/joyhallo/blob/main/README.md

Generate meta information for Stage 2 of the JoyHallo dataset. This script processes the prepared data.

```python
python scripts/extract_meta_info_stage2.py -r jdh-Hallo -n jdh-Hallo
```

--------------------------------

### Inference Configuration YAML

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Central configuration file for inference, specifying model paths, input data parameters, audio settings, diffusion parameters, and output directories.

```yaml
# configs/inference/inference.yaml

# --- Data ---
data:
  train_width: 512
  train_height: 512
  fps: 25
  sample_rate: 16000        # Audio sample rate expected by wav2vec2
  n_sample_frames: 16       # Frames generated per clip
  n_motion_frames: 2        # Motion context frames carried between clips
  audio_margin: 2

# --- Audio encoder ---
wav2vec_config:
  features: "all"           # Use all hidden states (not just last)
  model_path: ./pretrained_models/chinese-wav2vec2-base

audio_separator:
  model_path: ./pretrained_models/audio_separator/Kim_Vocal_2.onnx

face_expand_ratio: 1.2       # How much to expand the face bounding box for masking

# --- Model paths ---
base_model_path: "./pretrained_models/stable-diffusion-v1-5"
vae_model_path:  "./pretrained_models/sd-vae-ft-mse"
face_analysis_model_path: "./pretrained_models/face_analysis"
mm_path: "./pretrained_models/motion_module/mm_sd_v15_v2.ckpt"
audio_ckpt_dir: "./pretrained_models/joyhallo/net.pth"

# --- Diffusion / quality ---
inference_steps: 40          # Lower to 15 for faster (slightly lower quality) inference
cfg_scale: 3.5               # Classifier-free guidance scale
weight_dtype: "fp16"

# --- Outputs ---
exp_name: "joyhallo"
output_dir: "./opts"
seed: 42

# --- Input pairs (matched by index) ---
ref_img_path:
  - "examples/reference_images/1.jpg"   # square, face 50-70% of frame
audio_path:
  - "examples/driving_audios/0.wav"     # WAV format, 16 kHz preferred
```

--------------------------------

### Run JoyHallo Inference

Source: https://github.com/jdh-algo/joyhallo/blob/main/README.md

Execute the main inference script for JoyHallo. Modify parameters in `configs/inference/inference.yaml` to specify input files and models. Inference results are saved in `opts/joyhallo`.

```bash
sh joyhallo-infer.sh
```

--------------------------------

### Process Audio Embeddings with Sliding Window

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Applies a sliding temporal window to audio embedding frames to create context stacks for AudioProjModel. Handles boundary frames with edge-padding.

```python
from scripts.inference import process_audio_emb
import torch

# audio_emb from AudioProcessor.preprocess(): shape (T, 12, 768)
raw_emb = torch.randn(64, 12, 768)   # 64 frames, 12 layers, 768-dim

windowed = process_audio_emb(raw_emb)
print(windowed.shape)  # torch.Size([64, 5, 12, 768])
# Frame i contains embeddings from frames [i-2, i-1, i, i+1, i+2]
# Boundary frames are clamped (edge-padding behaviour)
```

--------------------------------

### Preprocess Audio with AudioProcessor

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Use the AudioProcessor to preprocess an audio file, generating frame-aligned embeddings and calculating the number of frames. The `preprocess` method handles optional vocal separation and padding to ensure the output duration is a multiple of the specified clip length.

```python
# Returns:
#   audio_emb   — torch.Tensor of shape (T, 12, 768)  [T = ceil(duration * fps)]
#   audio_length — int, number of frames represented
audio_emb, audio_length = processor.preprocess(
    wav_file="examples/driving_audios/0.wav",
    clip_length=16,   # pad so T is a multiple of 16
)
print(audio_emb.shape)   # e.g. torch.Size([400, 12, 768]) for a ~16 s clip
```

--------------------------------

### Override Runtime Inputs for Inference

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Modify configuration parameters like image paths, audio paths, experiment names, and inference steps before running the main inference process. This allows for quick previews and customized runs.

```python
cfg.ref_img_path = ["my_portrait.jpg"]
cfg.audio_path   = ["my_speech.wav"]
cfg.exp_name     = "my_run"
cfg.inference_steps = 15   # faster preview

inference_process(cfg)
# Output saved to: opts/my_run/0_my_portrait_my_speech.mp4
```

--------------------------------

### save_checkpoint / load_checkpoint

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Utilities for managing model checkpoints during training. `save_checkpoint` persists model weights, optionally keeping a limited number of recent checkpoints. `load_checkpoint` resumes training from the latest available checkpoint in a specified directory.

```APIDOC
## save_checkpoint / load_checkpoint — joyhallo/utils/util.py

Checkpoint management utilities used during training to persist model weights and resume interrupted runs.

```python
from joyhallo.utils.util import save_checkpoint, load_checkpoint

# Save current net state_dict, keeping only the 3 most recent checkpoints
save_checkpoint(
    model=net,
    save_dir="./exp_output/stage2/joyhallo",
    prefix="net",
    ckpt_num=500,       # global step
    total_limit=3,
)
# Creates: ./exp_output/stage2/joyhallo/net-500.pth
# Deletes older checkpoints beyond the limit

# Resume from latest checkpoint in a directory
global_step = load_checkpoint(
    cfg=cfg,            # cfg.resume_from_checkpoint = "latest"
    save_dir="./exp_output/stage2/joyhallo",
    accelerator=accelerator,
)
print(f"Resuming from step {global_step}")
```
```

--------------------------------

### Preprocess Image with ImageProcessor

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Process a source image using the ImageProcessor to obtain normalized pixel values, face and lip masks, and face embeddings. The `preprocess` method also generates multi-scale attention masks for UNet, with an option to expand the detected face region.

```python
(
    pixel_values_ref_img,   # torch.Tensor (3, 512, 512), normalized [-1, 1]
    face_mask,              # torch.Tensor (3, 512, 512), binary face region
    face_emb,               # np.ndarray (512,), InsightFace embedding
    pixel_values_full_mask, # List[4] of (1, H*W) background masks at 64/32/16/8
    pixel_values_face_mask, # List[4] of (1, H*W) face masks at 64/32/16/8
    pixel_values_lip_mask,  # List[4] of (1, H*W) lip masks at 64/32/16/8
) = proc.preprocess(
    source_image_path="examples/reference_images/1.jpg",
    cache_dir=".cache/image",
    face_region_ratio=1.2,  # expand detected face bounding box by 20%
)

print(pixel_values_ref_img.shape)       # torch.Size([3, 512, 512])
print(len(pixel_values_lip_mask))       # 4  (one per attention resolution)
print(pixel_values_lip_mask[0].shape)   # torch.Size([1, 4096])  (64×64 flattened)
```

--------------------------------

### Save and Load Model Checkpoints

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Utilize `save_checkpoint` to persist model weights during training, with an option to limit the total number of checkpoints saved. Use `load_checkpoint` to resume training from the latest saved state.

```python
from joyhallo.utils.util import save_checkpoint, load_checkpoint

# Save current net state_dict, keeping only the 3 most recent checkpoints
save_checkpoint(
    model=net,
    save_dir="./exp_output/stage2/joyhallo",
    prefix="net",
    ckpt_num=500,       # global step
    total_limit=3,
)
# Creates: ./exp_output/stage2/joyhallo/net-500.pth
# Deletes older checkpoints beyond the limit

# Resume from latest checkpoint in a directory
global_step = load_checkpoint(
    cfg=cfg,            # cfg.resume_from_checkpoint = "latest"
    save_dir="./exp_output/stage2/joyhallo",
    accelerator=accelerator,
)
print(f"Resuming from step {global_step}")
```

--------------------------------

### AudioProcessor

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Handles audio preprocessing, including optional vocal separation, resampling, and feature extraction for frame-aligned embeddings.

```APIDOC
## AudioProcessor

### Description
Manages the audio preprocessing pipeline, including vocal separation, 16 kHz resampling, and wav2vec2 feature extraction into frame-aligned embeddings.

### Initialization
`AudioProcessor(
    sample_rate: int = 16000,
    fps: int = 25,
    wav2vec_model_path: str,
    only_last_features: bool = False,
    audio_separator_model_path: str,
    audio_separator_model_name: str,
    cache_dir: str,
    device: str,
)

### Method: preprocess
`preprocess(
    wav_file: str,   # filepath to the WAV audio file
    clip_length: int  # pad so T (number of frames) is a multiple of this value
) -> tuple[torch.Tensor, int]: # Returns audio embeddings and audio length (number of frames)
    ...

### Method: get_embedding
`get_embedding(wav_file: str) -> torch.Tensor:
    ...

### Returns
- `audio_emb`: torch.Tensor of shape (T, 12, 768) [T = ceil(duration * fps)]
- `audio_length`: int, number of frames represented

### Example
```python
processor = AudioProcessor(
    sample_rate=16000,
    fps=25,
    wav2vec_model_path="pretrained_models/chinese-wav2vec2-base",
    only_last_features=False,
    audio_separator_model_path="pretrained_models/audio_separator",
    audio_separator_model_name="Kim_Vocal_2.onnx",
    cache_dir=".cache/audio",
    device="cuda:0",
)

audio_emb, audio_length = processor.preprocess(
    wav_file="examples/driving_audios/0.wav",
    clip_length=16,
)
print(audio_emb.shape)   # e.g. torch.Size([400, 12, 768]) for a ~16 s clip

raw_emb = processor.get_embedding("examples/driving_audios/0.wav")
print(raw_emb.shape)     # (T, 12, 768)
```
```

--------------------------------

### Convert Tensor to Video with Audio

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Converts a float tensor representing video frames into an MP4 file, muxing audio from a WAV source using MoviePy and AAC encoding. Ensure the output directory exists.

```python
from joyhallo.utils.util import tensor_to_video
import torch

# video_tensor: shape (3, F, H, W), values in [0, 1]
video_tensor = torch.rand(3, 48, 512, 512)

tensor_to_video(
    tensor=video_tensor,
    output_video_file="output/result.mp4",
    audio_source="examples/driving_audios/0.wav",
    fps=25,
)
# Writes: output/result.mp4  (48 frames @ 25 fps = 1.92 s, with audio)
```

--------------------------------

### FaceAnimatePipeline

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

A DiffusionPipeline subclass for face animation using DDIM denoising conditioned on audio, face embeddings, and spatial masks.

```APIDOC
## FaceAnimatePipeline

### Description
A `DiffusionPipeline` subclass that implements the DDIM denoising loop for face animation. It is conditioned on audio embeddings, face embeddings, and spatial masks to produce video frames.

### Initialization
This class is a subclass of `diffusers.DiffusionPipeline` and requires several components for initialization, including:
- `AutoencoderKL`
- `DDIMScheduler`
- `UNet2DConditionModel`
- `UNet3DConditionModel`
- `FaceLocator`
- `ImageProjModel`

(Specific initialization parameters are not detailed in the provided source.)

### Usage
(Usage details for `FaceAnimatePipeline` are not provided in the source.)
```

--------------------------------

### ImageProcessor

Source: https://context7.com/jdh-algo/joyhallo/llms.txt

Processes portrait images to detect faces, generate masks, and extract embeddings for use in animation.

```APIDOC
## ImageProcessor

### Description
Detects faces in portrait images, generates per-landmark masks (lip, face, background), and extracts pixel tensors and multi-scale attention masks.

### Initialization
`ImageProcessor(
    img_size: tuple[int, int] = (512, 512),
    face_analysis_model_path: str,
)

### Method: preprocess
`preprocess(
    source_image_path: str,  # Path to the source portrait image
    cache_dir: str,
    face_region_ratio: float = 1.2  # Expand detected face bounding box ratio
) -> tuple[
    torch.Tensor, torch.Tensor, np.ndarray, list[torch.Tensor], list[torch.Tensor], list[torch.Tensor]
]:
    ...

### Returns
- `pixel_values_ref_img`: torch.Tensor (3, 512, 512), normalized [-1, 1]
- `face_mask`: torch.Tensor (3, 512, 512), binary face region
- `face_emb`: np.ndarray (512,), InsightFace embedding
- `pixel_values_full_mask`: List[4] of (1, H*W) background masks at 64/32/16/8 resolutions
- `pixel_values_face_mask`: List[4] of (1, H*W) face masks at 64/32/16/8 resolutions
- `pixel_values_lip_mask`: List[4] of (1, H*W) lip masks at 64/32/16/8 resolutions

### Example
```python
proc = ImageProcessor(
    img_size=(512, 512),
    face_analysis_model_path="pretrained_models/face_analysis",
)

(pixel_values_ref_img, face_mask, face_emb, pixel_values_full_mask, pixel_values_face_mask, pixel_values_lip_mask) = proc.preprocess(
    source_image_path="examples/reference_images/1.jpg",
    cache_dir=".cache/image",
    face_region_ratio=1.2,
)

print(pixel_values_ref_img.shape)       # torch.Size([3, 512, 512])
print(len(pixel_values_lip_mask))       # 4  (one per attention resolution)
print(pixel_values_lip_mask[0].shape)   # torch.Size([1, 4096])  (64×64 flattened)
```
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.