### Install Dependencies

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/README.md

Use this command to install necessary dependencies for the environment. Ensure you have pip installed.

```bash
pip install -r requirements.txt
```

--------------------------------

### Example Dockerfile for PyTorch Project

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/configuration.md

This Dockerfile sets up a PyTorch environment with CUDA support, copies project files, installs dependencies, and configures runtime environment variables.

```dockerfile
FROM pytorch/pytorch:2.0-cuda11.8-cudnn8-runtime

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

ENV CUDA_VISIBLE_DEVICES=0
ENV TF_CPP_MIN_LOG_LEVEL=2
ENV GLOG_minloglevel=2

EXPOSE 8888

CMD ["python", "web_demo/server_realtime.py"]
```

--------------------------------

### Run Web Demo Server

Source: https://github.com/kleinlee/dh_live/blob/main/README_en.md

Start the web demo server. Ensure assets are correctly placed in the web_demo/assets folder.

```bash
python web_demo/server.py
```

--------------------------------

### Run Web Server

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/00-START-HERE.md

Navigate to the web_demo directory and run the server script to start the real-time web interface. Access it via the provided URL.

```bash
cd web_demo
python server_realtime.py
# Visit http://localhost:8888
```

--------------------------------

### YAML Configuration File Example

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/configuration.md

A hypothetical YAML configuration file demonstrating settings for the server, models, frontend, ASR, TTS, and LLM.

```yaml
server:
  host: 0.0.0.0
  port: 8888
  workers: 4
  log_level: warning

models:
  audio_checkpoint: checkpoint/lstm/lstm_model_epoch_325.pkl
  render_checkpoint: checkpoint/DINet_mini/epoch_40_new.pth
  enable_matting: false

frontend:
  video_source: assets/01.mp4
  data_source: assets/combined_data.json.gz
  chroma_key_enabled: false
  show_fps: false

asr:
  sample_rate: 16000
  engine: funasr  # or: silero, whisper

tts:
  engine: edge_tts  # or: azure, aliyun
  default_voice: zh_CN-XiaoxiaoNeural
  default_speed: normal

llm:
  api_endpoint: https://api.openai.com/v1
  model: gpt-3.5-turbo
  temperature: 0.7
```

--------------------------------

### Install FFmpeg on Windows

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/configuration.md

For Windows users, FFmpeg can be downloaded directly from the official website. Instructions for installation are available on the download page.

```bash
# Windows
# Download from https://ffmpeg.org/download.html
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/kleinlee/dh_live/blob/main/README_en.md

Install PyTorch with CUDA support and other project requirements using pip.

```bash
pip install torch --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt
```

--------------------------------

### Install MediaPipe and Dependencies

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/configuration.md

Install the mediapipe library along with its essential dependencies, opencv-python and kaldi-native-fbank, to resolve import errors.

```bash
# Solution: Install dependencies
pip install mediapipe opencv-python kaldi-native-fbank
```

--------------------------------

### Use Mesa for Software OpenGL

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/configuration.md

Install PyOpenGL-accelerate to enable software OpenGL rendering, which can be an alternative solution for GLFW initialization issues on headless systems.

```bash
# Or use Mesa for software OpenGL
pip install PyOpenGL-accelerate
```

--------------------------------

### Orthographic Projection Matrix Setup

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/render.md

Sets up the parameters for an orthographic projection matrix, defining the viewing frustum based on output dimensions.

```python
left = 0
right = out_size[0]
bottom = 0
top = out_size[1]
near = 1000
far = -1000

ortho_matrix = glm.ortho(left, right, bottom, top, near, far)
```

--------------------------------

### RenderModel_Mini Usage Workflow

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/render_model_mini.md

This example demonstrates the complete workflow for using the RenderModel_Mini class. It covers model creation, loading pre-trained weights, setting character appearance, and processing frames in a loop. Ensure you have the necessary dependencies and pre-trained model file.

```python
from render_model_mini import RenderModel_Mini
import cv2
import torch
import numpy as np

# Step 1: Create and load model
model = RenderModel_Mini()
model.loadModel('checkpoint/DINet_mini/epoch_40_new.pth')

# Step 2: Set character appearance
ref_img = cv2.imread('avatar.jpg')
ref_kpts = load_keypoints('avatar_keypoints.npy')
model.reset_charactor(ref_img, ref_kpts, standard_size=256)

# Step 3: Process frames in loop
for source_frame, gl_motion_frame in frame_stream:
    # Normalize to [0, 1]
    source_tensor = torch.from_numpy(
        source_frame / 255.0
    ).permute(2, 0, 1).unsqueeze(0).float()
    
    gl_tensor = torch.from_numpy(
        gl_motion_frame / 255.0
    ).permute(2, 0, 1).unsqueeze(0).float()
    
    # Generate output
    with torch.no_grad():
        output = model.interface(source_tensor, gl_tensor)
    
    # Convert back to numpy
    output_frame = (output[0].permute(1, 2, 0) * 255).byte().cpu().numpy()
```

--------------------------------

### Run Gradio App

Source: https://github.com/kleinlee/dh_live/blob/main/README_en.md

Execute the main application script to start the Gradio interface for easy, first-time use.

```bash
python app.py
```

--------------------------------

### Install FFmpeg on Ubuntu/Debian

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/configuration.md

Install FFmpeg on Ubuntu or Debian-based systems using the apt-get package manager. This is necessary for multimedia processing tasks.

```bash
# Solution: Install FFmpeg
# Ubuntu/Debian
sudo apt-get install ffmpeg
```

--------------------------------

### Process Video File with data_preparation_mini

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/00-START-HERE.md

Use this function to process a video file. Ensure the 'data_preparation_mini' library is installed.

```python
from data_preparation_mini import data_preparation_mini
data_preparation_mini('video.mp4', 'output_dir')
```

--------------------------------

### Initialize RVM Model Session

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/RVM.md

Use this function to get an instance of the RVM matting model. The model is loaded only once and cached for subsequent calls. It requires no specific setup beyond importing the function.

```python
from talkingface.RVM import get_onnx_session

# First call: loads model
model = get_onnx_session()

# Subsequent calls: returns same instance
model2 = get_onnx_session()
assert model is model2  # True
```

--------------------------------

### Set up Xvfb for Headless Systems

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/configuration.md

Use Xvfb (virtual framebuffer) on headless systems to simulate a display environment, resolving GLFW initialization failures. This requires installing Xvfb and running scripts with xvfb-run.

```bash
# Solution: Use Xvfb (virtual display)
sudo apt-get install xvfb
xvfb-run python script.py
```

--------------------------------

### Install FFmpeg on macOS

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/configuration.md

Install FFmpeg on macOS using the Homebrew package manager. This command is used for systems running macOS.

```bash
# macOS
brew install ffmpeg
```

--------------------------------

### Production Frontend Configuration

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/configuration.md

Customize frontend configuration for production, including CDN sources, secure WebSocket URLs, and performance optimizations. This example also disables the logo.

```javascript
// Customize for production
const PROD_CONFIG = {
    videoSrc: 'https://cdn.example.com/avatar.mp4',
    dataSrc: 'https://cdn.example.com/avatar_data.json.gz',
    serverUrl: 'https://api.example.com/dialogue',
    asrUrl: 'wss://api.example.com/asr',  // WSS for HTTPS
    chromaKeyEnabled: true,
    backgroundVideoSrc: 'https://cdn.example.com/bg_video.webm',
    renderScale: 0.8,  // Reduce quality for better performance
    showFPS: false,
    logoVisible: false  // Remove logo after license
};
```

--------------------------------

### Install VP9 Codec Dependency

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/data_preparation_web.md

Install the necessary development library for the VP9 codec if FFmpeg encounters issues during WebM encoding. This is a common solution for WebM encoding failures.

```bash
sudo apt-get install libvpx-dev
```

--------------------------------

### Base64 Audio Encoding/Decoding (JavaScript)

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/endpoints.md

Provides examples for encoding audio data to Base64 on the client-side and decoding it on the server-side.

```javascript
// Encode
const audioBlob = new Blob([pcmBuffer], { type: 'audio/wav' });
const reader = new FileReader();
reader.onload = () => {
    const base64 = reader.result.split(',')[1];  // Remove data:audio/wav;base64, prefix
};

// Decode (server-side)
const audioBuffer = Buffer.from(base64String, 'base64');
```

--------------------------------

### GET /static/<path>

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/endpoints.md

Serves static web assets such as HTML, JavaScript, CSS, and media files. It supports various content types and is used for the main web interface and related resources.

```APIDOC
## GET /static/<path>

### Description
Serves static web assets (HTML, JavaScript, CSS, media files).

### Method
GET

### Endpoint
/static/<path>

### Parameters
#### Path Parameters
- **path** (string) - Required - The path to the static file to serve.

### Response
#### Success Response (200)
- **Content** (various) - The requested static file content.

#### Error Response (404)
- **Not Found** - If the requested file does not exist.
```

--------------------------------

### Handle FFmpegError

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/errors.md

Handles FFmpeg-specific errors. Includes a check to guide users on installing FFmpeg if it's not found.

```python
from data_preparation_mini import data_preparation_mini, FFmpegError

try:
    data_preparation_mini('video.mp4', 'output')
except FFmpegError as e:
    # Check FFmpeg installation
    import subprocess
    result = subprocess.run(['ffmpeg', '-version'], capture_output=True)
    if result.returncode != 0:
        print("FFmpeg not installed. Install via: sudo apt-get install ffmpeg")
    else:
        print(f"FFmpeg error: {e}")
```

--------------------------------

### Initialize and Use RenderModel_Mini Interface

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/render_model_mini.md

Demonstrates how to load a model, set reference appearance, prepare input tensors, and generate an output frame using the `interface` method. Ensure input tensors are normalized to [0, 1].

```python
import torch
import cv2
from render_model_mini import RenderModel_Mini

model = RenderModel_Mini()
model.loadModel('checkpoint/DINet_mini/epoch_40_new.pth')
model.reset_charactor(ref_img, ref_keypoints)

# Prepare tensors
source = cv2.imread('source.jpg')
source = cv2.resize(source, (128, 128))
source_tensor = torch.from_numpy(source / 255.0).permute(2, 0, 1).unsqueeze(0)

gl_rendered = np.random.rand(128, 128, 3)  # OpenGL output
gl_tensor = torch.from_numpy(gl_rendered).permute(2, 0, 1).unsqueeze(0)

# Generate output
with torch.no_grad():
    output = model.interface(source_tensor, gl_tensor)

# Extract to numpy
output_img = (output[0].permute(1, 2, 0) * 255).byte().cpu().numpy()
```

--------------------------------

### Run Mini Demo with Audio

Source: https://github.com/kleinlee/dh_live/blob/main/README_en.md

Execute the mini demo script with video assets, an audio file, and an output video name. Note: Linux and MacOS are not supported for this feature.

```bash
python demo_mini.py video_data/000002/assets video_data/audio0.wav 1.mp4
```

--------------------------------

### Custom Video Generation Pipeline

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/README.md

Use this pattern to prepare assets and generate a video from reference and audio files. Ensure the 'data_preparation_mini' and 'interface_mini' functions are available.

```python
from data_preparation_mini import data_preparation_mini
from demo_mini import interface_mini
from talkingface.model_utils import Audio2bs

# Prepare assets
data_preparation_mini('reference.mp4', 'assets')

# Generate video
interface_mini('assets/assets', 'audio.wav', 'output.mp4')
```

--------------------------------

### Core Documentation: Demo Mini

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/README.md

Documentation for the audio-to-video conversion and video generation orchestration module.

```APIDOC
## Main Entry Point

- `interface_mini()`: The primary function for generating talking head videos.

### Pipeline
1. Audio feature extraction
2. OpenGL 3D rendering
3. Neural network warping

### Use Case
Generate synchronized talking head video from audio input.
```

--------------------------------

### Install CPU Version of PyTorch

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/configuration.md

If CUDA is not available, install the CPU-only version of PyTorch using pip. This is a common solution for environments lacking GPU support.

```python
# Solution: Install CPU version
pip install torch --index-url https://download.pytorch.org/whl/cpu
```

--------------------------------

### Prepare Video Asset for Digital Human Generation

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/README.md

Use this function to prepare video assets. Set `matting=True` for green-screen extraction and `resize_option=True` to resize to 720p for mobile compatibility. The output is saved in the specified directory.

```python
from data_preparation_mini import data_preparation_mini

data_preparation_mini(
    'input_video.mp4',
    'output_directory',
    matting=False,        # Enable if green-screen extraction needed
    resize_option=True    # Resize to 720p for mobile
)
# Output: output_directory/assets/ with combined_data.json.gz
```

--------------------------------

### Configure Lightweight Video Preparation

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/configuration.md

Use this snippet for a lightweight mobile version of video preparation, optimizing for 720p resolution.

```python
data_preparation_mini(
    'input.mp4',
    'output',
    matting=False,
    resize_option=True  # 720p max
)
```

--------------------------------

### Run Real-Time Dialogue Server

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/README.md

Deploy the real-time dialogue server by navigating to the 'web_demo' directory and running the 'server_realtime.py' script. The server will be accessible at http://localhost:8888.

```bash
cd web_demo
python server_realtime.py
# Serves on http://localhost:8888
```

--------------------------------

### Complete Web Preparation Pipeline

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/data_preparation_web.md

This script orchestrates the entire web asset preparation and deployment process, from raw video to CDN.

```python
import os
from data_preparation_mini import data_preparation_mini
from data_preparation_web import data_preparation_web

# Source video
video_path = 'raw_video/avatar_source.mp4'
output_base = 'web_assets/avatar_001'

# Step 1: Raw processing
print("Step 1: Processing raw video...")
data_preparation_mini(
    video_path,
    output_base,
    matting=False,
    resize_option=True
)

# Step 2: Web optimization
print("Step 2: Optimizing for web...")
data_preparation_web(output_base, compress=True)

# Step 3: Deploy to CDN
print("Step 3: Deploying to CDN...")
assets_dir = os.path.join(output_base, 'assets')

# Copy optimized files to CDN
import shutil
cdn_path = '/var/www/cdn/avatars/avatar_001/'
if os.path.exists(cdn_path):
    shutil.rmtree(cdn_path)
shutil.copytree(assets_dir, cdn_path)

print(f"✓ Deployment complete: {cdn_path}")

# Check file sizes
for filename in os.listdir(cdn_path):
    filepath = os.path.join(cdn_path, filename)
    size_mb = os.path.getsize(filepath) / (1024 * 1024)
    print(f"  {filename}: {size_mb:.2f}MB")

```

--------------------------------

### Configure High-Quality Video Preparation with Matting

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/configuration.md

Use this snippet for high-quality video preparation that includes green-screen matting and preserves the original resolution.

```python
data_preparation_mini(
    'input.mp4',
    'output',
    matting=True,
    resize_option=False  # Original resolution
)
```

--------------------------------

### Define FFmpegError Class

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/errors.md

Raised when FFmpeg execution fails. This can occur due to FFmpeg not being installed, corrupted video files, or permission issues.

```python
class FFmpegError(VideoProcessingError):
    """Raised when FFmpeg execution fails"""
```

--------------------------------

### Deploy Web Server with server_realtime.py

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/00-START-HERE.md

Deploy a web server for real-time applications. Run this script and access the server at http://localhost:8888.

```bash
python web_demo/server_realtime.py
# Open http://localhost:8888
```

--------------------------------

### Load Model Checkpoints

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/configuration.md

Load audio, rendering, and optional matting models using their respective checkpoint paths. Ensure all required models are loaded before proceeding.

```python
from talkingface.model_utils import LoadAudioModel
from talkingface.render_model_mini import RenderModel_Mini

# Audio model
audio_model = LoadAudioModel('checkpoint/lstm/lstm_model_epoch_325.pkl')

# Rendering model
render_model = RenderModel_Mini()
render_model.loadModel('checkpoint/DINet_mini/epoch_40_new.pth')

# Optional: matting model
from talkingface.RVM import get_onnx_session
matting_model = get_onnx_session()  # Auto-loads rvm_resnet50.pth
```

--------------------------------

### Combined Data Schema Example

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/demo_mini.md

Illustrates the structure of the combined_data.json.gz file, which includes 3D vertex data, per-frame facial keypoints and bounding boxes, reference feature vectors, and model size.

```json
{
    "face3D_obj": ["v 0.1 0.2 0.3 0.4 0.5", ...],
    "json_data": [
        {
            "points": [p0, p1, ..., p467],
            "rect": [x_min, y_min, x_max, y_max]
        }
    ],
    "ref_data": [f0, f1, ..., f127],
    "size": 184
}
```

--------------------------------

### High Quality Desktop Configuration

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/configuration.md

This configuration is for desktop or laptop use, prioritizing maximum quality. It enables matting and higher confidence.

```python
# Maximum quality for desktop/laptop
config = {
    'window_size': (512, 512),
    'matting': True,
    'resize_option': False,
    'confidence': 0.9,
    'batch_size': 1
}
```

--------------------------------

### JavaScript WebSocket Client for ASR

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/endpoints.md

Example JavaScript code to establish a WebSocket connection for ASR, capture microphone audio, process it into PCM data, and send it to the server. Also handles receiving and logging ASR results.

```javascript
const ws = new WebSocket('ws://localhost:8888/asr?samplerate=16000');

ws.onopen = () => {
    // Start audio capture
    navigator.mediaDevices.getUserMedia({ audio: true })
        .then(stream => {
            const audioContext = new AudioContext({ sampleRate: 16000 });
            const processor = audioContext.createScriptProcessor(4096);
            
            processor.onaudioprocess = (e) => {
                const pcmData = e.inputBuffer.getChannelData(0);
                const int16Data = new Int16Array(pcmData.length);
                
                for (let i = 0; i < pcmData.length; i++) {
                    int16Data[i] = Math.max(-1, Math.min(1, pcmData[i])) * 0x7fff;
                }
                
                ws.send(int16Data.buffer);
            };
            
            stream.getTracks()[0].onended = () => {
                ws.send(JSON.stringify({ text: 'vad' }));
            };
        });
};

ws.onmessage = (e) => {
    if (typeof e.data === 'string') {
        const result = JSON.parse(e.data);
        console.log('ASR Result:', result);
    }
};
```

--------------------------------

### Core Documentation: Render Model Mini

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/README.md

Documentation for the lightweight rendering model management.

```APIDOC
## Class: `RenderModel_Mini`

Represents a neural network rendering pipeline for applying motion to source images.

### Methods

- `loadModel()`: Loads the rendering model.
- `reset_charactor()`: Resets the character's state.
- `interface()`: Executes the rendering pipeline.
- `save()`: Saves the generated output.
```

--------------------------------

### Generate Talking Head Video from Audio with interface_mini

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/00-START-HERE.md

Generate a talking head video from an audio file. Requires 'demo_mini' library and specified asset directory.

```python
from demo_mini import interface_mini
interface_mini('assets/dir', 'audio.wav', 'output.mp4')
```

--------------------------------

### Checkpoint Files for Different Modalities

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/00-START-HERE.md

Lists the required checkpoint files for video generation, audio features, and optional matting. Ensure these files are present in the specified directory structure.

```text
checkpoint/
├── DINet_mini/epoch_40_new.pth        # Video generation
├── lstm/lstm_model_epoch_325.pkl      # Audio features
└── rvm_resnet50.pth                   # Matting (optional)
```

--------------------------------

### Prepare Video Data for Mini Demo

Source: https://github.com/kleinlee/dh_live/blob/main/README_en.md

Process input video files using the mini data preparation script. The `--matting` flag enables green screen removal.

```bash
python data_preparation_mini.py video_data/000002/video.mp4 video_data/000002 --matting
```

--------------------------------

### Run Uvicorn Server

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/endpoints.md

Use this command for local development on localhost. For production, a reverse proxy with SSL termination is recommended.

```python
# Development (localhost)
uvicorn.run(app, host='127.0.0.1', port=8888)

# Production (HTTPS)
# Use nginx/Apache reverse proxy with SSL termination
```

--------------------------------

### Load Audio and Rendering Models

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/00-START-HERE.md

Load the necessary models for audio processing and rendering. Ensure the model paths are correct.

```python
from talkingface.model_utils import LoadAudioModel
from talkingface.render_model_mini import RenderModel_Mini

audio_model = LoadAudioModel('checkpoint/lstm/lstm_model_epoch_325.pkl')
render_model = RenderModel_Mini()
render_model.loadModel('checkpoint/DINet_mini/epoch_40_new.pth')
```

--------------------------------

### RenderModel_gl.__init__

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/render.md

Initializes the OpenGL context and rendering resources, including shaders and buffers. It creates an invisible GLFW window for off-screen rendering.

```APIDOC
## __init__

### Description
Initializes OpenGL context and rendering resources. Creates an invisible GLFW window for off-screen rendering.

### Method
__init__

### Parameters
#### Path Parameters
- **window_size** (tuple) - Required - (width, height) in pixels

### Returns
None. Initializes OpenGL state.

### Throws
- Exception — GLFW initialization fails
- Exception — Window creation fails

### Example
```python
from mini_live.render import RenderModel_gl

renderer = RenderModel_gl(window_size=(512, 512))
```
```

--------------------------------

### Initialize Matting Network with CUDA

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/RVM.md

Initializes the MattingNetwork for CUDA. This implementation hard-codes CUDA and will fail if it is unavailable, as there is no CPU fallback.

```python
# Current implementation
_sess = MattingNetwork('resnet50').eval().cuda()  # Hard-coded CUDA
# Will fail if CUDA unavailable—no CPU fallback
```

--------------------------------

### Create and Activate Conda Environment

Source: https://github.com/kleinlee/dh_live/blob/main/README_en.md

Set up a dedicated Python environment for the project using Conda and activate it.

```bash
conda create -n dh_live python=3.11
conda activate dh_live
```

--------------------------------

### Create Render Model with Window Size

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/configuration.md

Use `create_render_model` to initialize a renderer. The `out_size` parameter accepts a tuple of (width, height) to define the rendering canvas dimensions. Recommended sizes vary based on target device and performance needs.

```python
from mini_live.render import create_render_model

# Mobile optimized
renderer = create_render_model(out_size=(256, 256))

# Desktop quality
renderer = create_render_model(out_size=(512, 512))
```

--------------------------------

### data_preparation_mini

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/data_preparation_mini.md

Main entry point that orchestrates the complete video preparation pipeline. Processes a video file to extract facial landmarks, generate face meshes, and create combined data structures.

```APIDOC
## data_preparation_mini(video_path: str, output_dir: str, matting: bool = False, resize_option: bool = True) -> None

### Description
Main entry point that orchestrates the complete video preparation pipeline. Processes a video file to extract facial landmarks, generate face meshes, and create combined data structures.

### Parameters
#### Path Parameters
- **video_path** (str) - Required - Path to input video file
- **output_dir** (str) - Required - Directory where processed assets will be saved

#### Query Parameters
- **matting** (bool) - Optional - Enable green-screen matting/background removal using RVM model. Defaults to False.
- **resize_option** (bool) - Optional - Resize video to maximum 720p for mobile compatibility. Defaults to True.

### Returns
None. Creates output directory structure with assets containing processed video and metadata.

### Throws
- `FFmpegError` — FFmpeg command execution fails
- `FirstFrameFaceDetectionError` — No face detected in first frame
- `FaceDetectionError` — Face angle invalid, face too small, or multiple faces detected
- `FaceMeshDetectionError` — Facial landmark detection fails

### Example
```python
from data_preparation_mini import data_preparation_mini

# Basic usage
data_preparation_mini(
    'path/to/input_video.mp4',
    'video_data/000002'
)

# With matting enabled
data_preparation_mini(
    'path/to/input_video.mp4',
    'video_data/000002',
    matting=True,
    resize_option=True
)
```
```

--------------------------------

### Prepare Video Data for Web Demo

Source: https://github.com/kleinlee/dh_live/blob/main/README_en.md

Process input video files using the web data preparation script. Processed data is stored in the ./video_data directory.

```bash
python data_preparation_web.py video_data/000002
```

--------------------------------

### Initialize RenderModel_gl

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/render.md

Initializes the OpenGL context and rendering resources, including creating an invisible GLFW window for off-screen rendering. Ensure GLFW is initialized successfully before calling.

```python
from mini_live.render import RenderModel_gl

renderer = RenderModel_gl(window_size=(512, 512))
```

--------------------------------

### Frontend Configuration for Web Assets

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/data_preparation_web.md

Defines frontend configuration for loading web-optimized video and compressed metadata. Ensure the paths match the assets served by the web server.

```javascript
const CONFIG = {
    videoSrc: '/static/assets/01.webm',  // Web-optimized video
    dataSrc: '/static/assets/combined_data.json.gz',  // Compressed metadata
};
```

--------------------------------

### Core Documentation: Data Preparation Mini

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/README.md

Documentation for video preprocessing, face detection, and facial landmark extraction functions.

```APIDOC
## Functions

- `data_preparation_mini()`: Preprocesses video assets for digital human generation.
- `detect_face()`: Detects faces within video frames.
- `detect_face_mesh()`: Detects facial landmarks and mesh.
- `encode_binary_pixels()`: Encodes pixel data in binary format.

### Error Handling
Custom exceptions are raised for face detection failures.
```

--------------------------------

### Generate Talking Head Video with interface_mini

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/demo_mini.md

Use this function as the main entry point for generating a talking head video from audio. It handles model loading, audio processing, and video rendering. Ensure your input WAV file is 16kHz, mono, and single-channel.

```python
from demo_mini import interface_mini

interface_mini(
    path='video_data/000002/assets',
    wav_path='audio.wav',
    output_video_path='output.mp4'
)
```

--------------------------------

### Orchestrate Video Preparation Pipeline

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/data_preparation_mini.md

Use this function as the main entry point to process a video file, extract facial landmarks, generate face meshes, and create combined data structures. It supports optional green-screen matting and resizing for mobile compatibility.

```python
from data_preparation_mini import data_preparation_mini

# Basic usage
data_preparation_mini(
    'path/to/input_video.mp4',
    'video_data/000002'
)

# With matting enabled
data_preparation_mini(
    'path/to/input_video.mp4',
    'video_data/000002',
    matting=True,
    resize_option=True
)
```

--------------------------------

### RenderModel_Mini Class Initialization

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/render_model_mini.md

Initializes an empty RenderModel_Mini instance. The actual model is loaded via loadModel().

```APIDOC
## __init__

### Description
Initializes an empty RenderModel_Mini instance. The actual model is loaded via `loadModel()`.

### Method
__init__

### Example
```python
from render_model_mini import RenderModel_Mini

model = RenderModel_Mini()
```
```

--------------------------------

### Run FastAPI Server (Development)

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/configuration.md

Run the FastAPI application for development purposes. This configuration binds to localhost only and enables info-level logging.

```python
import uvicorn

# Development
uvicorn.run(
    app,
    host='127.0.0.1',  # localhost only
    port=8888,
    log_level='info'
)
```

--------------------------------

### Core Documentation: Render

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/README.md

Documentation for the OpenGL-based 3D face mesh rendering.

```APIDOC
## Class: `RenderModel_gl`

Manages a GPU-accelerated face deformation and rendering pipeline using OpenGL.

### Methods

- `setContent()`: Sets the content for rendering.
- `GenVBO()`: Generates Vertex Buffer Objects.
- `GenEBO()`: Generates Element Buffer Objects.
- `GenTexture()`: Generates textures.
- `render2cv()`: Renders the scene to a OpenCV format.
```

--------------------------------

### Frontend Configuration (JavaScript)

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/endpoints.md

Defines the global configuration object for the frontend, including video sources, data paths, and server endpoint.

```javascript
const CONFIG = {
    videoSrc: 'assets/01.mp4',
    dataSrc: 'assets/combined_data.json.gz',
    chromaKeyEnabled: false,
    backgroundVideoSrc: 'assets/bg.mp4',
    showFPS: true,
    serverUrl: 'http://localhost:8888/eb_stream'  // Configurable server endpoint
};
```

--------------------------------

### interface_mini

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/demo_mini.md

Main entry point for generating a talking head video from audio. This function loads all required models, processes audio to extract speech features, renders video frames with OpenGL, and writes the output MP4 file.

```APIDOC
## interface_mini

### Description
Main entry point for generating a talking head video from audio. This function loads all required models, processes audio to extract speech features, renders video frames with OpenGL, and writes the output MP4 file.

### Parameters
#### Path Parameters
- **path** (str) - Required - Path to assets directory containing combined_data.json.gz and 01.mp4 reference video
- **wav_path** (str) - Required - Path to input WAV file (16kHz, mono, single-channel)
- **output_video_path** (str) - Required - Output path for generated MP4 video file

### Returns
None. Writes output video to `output_video_path`.

### Example
```python
from demo_mini import interface_mini

interface_mini(
    path='video_data/000002/assets',
    wav_path='audio.wav',
    output_video_path='output.mp4'
)
```
```

--------------------------------

### Create Render Model Instance

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/render.md

Use this factory function to create and initialize a RenderModel_gl instance. Specify output image dimensions and an optional floor parameter.

```python
from mini_live.render import create_render_model

renderer = create_render_model(out_size=(512, 512), floor=20)
```

--------------------------------

### Streaming Real-time Synthesis

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/README.md

This pattern is for processing streaming audio to generate real-time synthesis. It involves loading audio and rendering models, then processing audio chunks to produce output. Requires 'LoadAudioModel', 'Audio2bs', and 'RenderModel_Mini'.

```python
from talkingface.model_utils import LoadAudioModel, Audio2bs
from talkingface.render_model_mini import RenderModel_Mini

# Load models
audio_model = LoadAudioModel('checkpoint/lstm/lstm_model_epoch_325.pkl')
render_model = RenderModel_Mini()
render_model.loadModel('checkpoint/DINet_mini/epoch_40_new.pth')

# Process streaming audio
for audio_chunk in audio_stream:
    bs = Audio2bs(audio_chunk, audio_model)
    output = render_model.interface(source_tensor, gl_tensor)
```

--------------------------------

### Initialize RenderModel_Mini

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/render_model_mini.md

Creates an instance of the RenderModel_Mini class. The actual model weights are loaded in a subsequent step using loadModel().

```python
from render_model_mini import RenderModel_Mini

model = RenderModel_Mini()
```

--------------------------------

### CDN Compression Settings

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/data_preparation_web.md

Enable these settings to ensure efficient delivery of compressed assets.

```http
Content-Encoding: gzip
Accept-Encoding: gzip, deflate, br
```

--------------------------------

### Optimize Video Assets for Web Deployment

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/data_preparation_web.md

Use this function to re-encode video to WebM and compress JSON metadata for web delivery. Ensure input assets are prepared by `data_preparation_mini` first.

```python
from data_preparation_mini import data_preparation_mini
from data_preparation_web import data_preparation_web

# Step 1: Prepare raw assets
data_preparation_mini('video.mp4', 'assets_dir')

# Step 2: Optimize for web
data_preparation_web('assets_dir', compress=True)

# Result: assets_dir/assets/ ready for web deployment
```

--------------------------------

### Batch Processing Videos for Web Assets

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/api-reference/data_preparation_web.md

Processes multiple video files in a directory using both `data_preparation_mini` and `data_preparation_web`. It handles output path creation, compression, and error reporting for each video.

```python
import os
from pathlib import Path
from data_preparation_mini import data_preparation_mini
from data_preparation_web import data_preparation_web

# Process all videos in directory
videos_dir = 'raw_videos/'
output_base = 'web_assets/'

for video_file in Path(videos_dir).glob('*.mp4'):
    avatar_name = video_file.stem
    output_path = os.path.join(output_base, avatar_name)
    
    print(f"Processing {avatar_name}...")
    
    try:
        data_preparation_mini(str(video_file), output_path)
        data_preparation_web(output_path, compress=True)
        print(f"✓ {avatar_name} complete")
    except Exception as e:
        print(f"✗ {avatar_name} failed: {e}")
        continue
```

--------------------------------

### Configure Audio Feature Extraction Options

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/configuration.md

Set Mel-spectrogram parameters using `knf.FbankOptions`. Key settings include frame length and shift, number of Mel bins, dithering, and edge frame handling. Ensure audio input meets format, sample rate, bit depth, and channel requirements.

```python
# From model_utils.py
opts = knf.FbankOptions()
opts.frame_opts.dither = 0
opts.frame_opts.frame_length_ms = 50
opts.frame_opts.frame_shift_ms = 20
opts.mel_opts.num_bins = 80
opts.frame_opts.snip_edges = False
opts.mel_opts.debug_mel = False
```

--------------------------------

### Configure GPU Settings and Check Availability

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/configuration.md

Check for available GPUs, set the visible devices, and configure memory usage for PyTorch. Use `torch.cuda.empty_cache()` to free up unused memory and `torch.cuda.set_per_process_memory_fraction()` to limit memory allocation.

```python
import torch

# Check available GPUs
print(torch.cuda.is_available())
print(torch.cuda.device_count())
print(torch.cuda.current_device())

# Use specific GPU
os.environ['CUDA_VISIBLE_DEVICES'] = '0'  # Use GPU 0 only

# Reduce memory usage
torch.cuda.empty_cache()
torch.cuda.set_per_process_memory_fraction(0.8)  # Use 80% of GPU memory
```

--------------------------------

### Prepare Video Data

Source: https://github.com/kleinlee/dh_live/blob/main/_autodocs/00-START-HERE.md

Use this function to prepare video data for processing. It handles matting and resizing options.

```python
from data_preparation_mini import data_preparation_mini
data_preparation_mini('video.mp4', 'assets', matting=False, resize_option=True)
```