LeapTalk

https://github.com/zhangrongxiang/leaptalk
Admin
LeapTalk is a real-time streaming talking head generation framework that synthesizes high-fidelity,...
Tokens:27,896
Snippets:172
Trust Score:7.8
License:Apache-2.0
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Start the web server with python web_server.py

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/README.md

Starts the web server for the real-time digital human demo. After running, open http://localhost:7860 to load a portrait image and interact via text or microphone.

```bash
python web_server.py
```

--------------------------------

### Run demo.py to calculate metrics

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/eval/common_metrics_on_video_quality/README.md

Commands to run the demo.py script that calculates the four metrics (FVD, SSIM, PSNR, LPIPS) on the example videos.

```bash
cd common_metrics_on_video_quality
python demo.py
```

--------------------------------

### Install other LeapTalk dependencies

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/README.md

Install all other Python dependencies listed in requirements.txt. Run after installing PyTorch.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install common_metrics_on_video_quality with conda and pip

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/eval/common_metrics_on_video_quality/README.md

Installation commands for the common_metrics_on_video_quality package. Requires Python 3.9, PyTorch 1.13.1 with CUDA 11.7, numpy<2.0, and opencv-python, tqdm, einops. Clones the repository from GitHub.

```bash
conda create -n test python=3.9
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia
conda install "numpy<2.0"
pip install opencv-python tqdm einops
git clone https://github.com/JunyaoHu/common_metrics_on_video_quality
```

--------------------------------

### Run training with bash train.sh

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/README.md

Runs the training script to start model training with the configured paths.

```bash
bash train.sh
```

--------------------------------

### Install PyTorch dependencies

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/README.md

Install PyTorch with specific versions (torch 2.7.1, torchvision 0.22.1, torchaudio 2.7.1) required by LeapTalk. Run after creating the Conda environment.

```bash
pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1
```

--------------------------------

### Setup TensorFlow 1.0 environment and run calculate_fvd_tensorflow.py

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/eval/common_metrics_on_video_quality/README.md

Commands to set up a TensorFlow 1.0 environment and run the original TensorFlow FVD calculation script. Requires Python 3.7 and specific package versions; the export sets the protocol buffers implementation to Python.

```bash
conda create -n tf1 python=3.7
pip install tensorflow==1.15.0 tensorflow-gan==1.0.0.dev0 tensorflow-hub==0.12.0 scipy==1.7.3 tqdm

export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
calculate_fvd_tensorflow.py
```

--------------------------------

### Metric output with only_final=True

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/eval/common_metrics_on_video_quality/README.md

Example output when only_final=True for all-zero and all-one videos of shape [8x30x3x64x64]. FVD is about 151 (may fluctuate by ~1 due to PyTorch version), SSIM and PSNR are near zero, LPIPS is about 0.814.

```json
{
    "fvd":   {"value": [151.25648496845326]},
    "ssim":  {"value": [9.999000099990664e-05],"value_std": [0.0]},
    "psnr":  {"value": [0.0],"value_std": [0.0]},
    "lpips": {"value": [0.8140090703964233],"value_std": [0.0]}
}
```

--------------------------------

### Input video shape and pixel range

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/eval/common_metrics_on_video_quality/README.md

Shape and pixel range of the input videos for the evaluation example. Both original and generated videos are tensors of shape [8x30x3x64x64] with pixel values in [0,1].

```text
original  video: [8x30x3x64x64] pixel in [0,1]
generated video: [8x30x3x64x64] pixel in [0,1]
```

--------------------------------

### Create Conda environment for LeapTalk

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/README.md

Create and activate a new Conda environment named leaptalk with Python 3.12. Must be done before installing PyTorch and other dependencies.

```bash
conda create -n leaptalk python=3.12
conda activate leaptalk
```

--------------------------------

### Metric output with only_final=False

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/eval/common_metrics_on_video_quality/README.md

Example output when only_final=False. The fvd value array contains per-n FVD values for n from 10 to T (here T=30, so 21 values). SSIM, PSNR, and LPIPS arrays are truncated with ellipses in the source.

```json
{
    "fvd": {
        "value": [
            569.2296293622766,
            486.3584254441098,
            551.9610501807822,
            146.36638178542628,
            172.85453222258292,
            133.70311962583372,
            152.91750309134142,
            357.7402855012116,
            382.4668646785276,
            306.73840379649727,
            338.4151811780684,
            78.17255931098194,
            82.33446642508818,
            64.5885972265882,
            66.06281151704604,
            314.4803706985065,
            316.6870909853734,
            288.97196946254184,
            287.7805184515251,
            152.1524775765185,
            151.2564750365302
        ]
    },
    "ssim":  {"value": [9.999000099990664e-05,...,],"value_std": [0.0,...]},
    "psnr":  {"value": [0.0,...],...},
    "lpips": {"value": [0.8140090703964233,...],...}
}
```

--------------------------------

### Configure keys and model paths for web demo

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/README.md

First copy the environment template, then fill in the Doubao credentials (DOUBAO_APP_ID and DOUBAO_ACCESS_TOKEN) and the LeapTalk model paths. The web demo prefers the APP_ID/ACCESS_TOKEN pair when both are present; DOUBAO_API_KEY remains an optional fallback.

```bash
cp .env.example .env
```

```bash
DOUBAO_APP_ID=YOUR_DOUBAO_APP_ID
DOUBAO_ACCESS_TOKEN=YOUR_DOUBAO_ACCESS_TOKEN
LEAPTALK_CKPT_DIR="./models/SoulX-FlashHead-1_3B"
LEAPTALK_WAV2VEC_DIR="./models/wav2vec2-base-960h"
LEAPTALK_LORA_DIR="./models/leaptalk"
LEAPTALK_AUDIO_PROJ="./models/leaptalk/audio_proj_step_10400.pt"
```

--------------------------------

### Configure inf.sh paths

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/README.md

Edit inf.sh with your local paths. Configure the model directories, audio projector checkpoint, inference steps, and enable Lite mode. Replace YOUR_REFERENCE_IMAGE_PATH and YOUR_AUDIO_PATH with your actual files.

```bash
CKPT_DIR="./models/SoulX-FlashHead-1_3B"
WAV2VEC_DIR="./models/wav2vec2-base-960h"
LORA_DIR="./models/leaptalk"
AUDIO_PROJ="./models/leaptalk/audio_proj_step_10400.pt"
COMPILE="off"
NUM_INFERENCE_STEPS="1"
LITE="1"
COND_IMAGE="YOUR_REFERENCE_IMAGE_PATH"
AUDIO_PATH="YOUR_AUDIO_PATH"
```

--------------------------------

### Download required models with huggingface-cli

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/README.md

Download the required models: SoulX-FlashHead-1_3B as the base model, facebook/wav2vec2-base-960h, and the LeapTalk checkpoint (z-rx/leaptalk) containing LoRA weights, audio_proj_step_*.pt, and the Lite TAE checkpoint. The TAE path is resolved automatically when LITE=1.

```bash
pip install "huggingface_hub[cli]"

huggingface-cli download Soul-AILab/SoulX-FlashHead-1_3B \
  --local-dir ./models/SoulX-FlashHead-1_3B

huggingface-cli download facebook/wav2vec2-base-960h \
  --local-dir ./models/wav2vec2-base-960h

huggingface-cli download z-rx/leaptalk \
  --local-dir ./models/leaptalk
```

--------------------------------

### Download VividHead dataset and expected layout

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/README.md

Downloads the VividHead dataset from Hugging Face and shows the expected directory structure. The training script expects paired videos and audios under data/VividHead.

```bash
huggingface-cli download Soul-AILab/VividHead \
  --repo-type dataset \
  --local-dir ./data/VividHead
```

```text
data/VividHead/
├── videos/
└── audios/
```

--------------------------------

### Fill training paths in train.sh

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/README.md

Shows the default training paths used by train.sh. Adjust these variables to point to your downloaded base model, wav2vec model, and dataset directories before training.

```bash
CKPT_DIR="./models/SoulX-FlashHead-1_3B"
WAV2VEC_DIR="./models/wav2vec2-base-960h"
VIDEO_DIR="./data/VividHead/videos"
AUDIO_DIR="./data/VividHead/audios"
SAVE_DIR="./outputs/train"
```

--------------------------------

### Calculate metrics with only_final=False

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/eval/common_metrics_on_video_quality/README.md

Function calls with only_final=False to compute per-frame metrics. FVD is computed for each n from 10 to T, while PSNR/SSIM/LPIPS are averaged over frames 0 to n-1 for each n from 1 to T.

```python
calculate_fvd(videos1, videos2, device, method='styleganv', only_final=False)
calculate_ssim(videos1, videos2, only_final=False)
calculate_psnr(videos1, videos2, only_final=False)
calculate_lpips(videos1, videos2, device, only_final=False)
```

--------------------------------

### Run LeapTalk inference

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/README.md

Run the inference script after filling in paths in inf.sh.

```bash
bash inf.sh
```

--------------------------------

### Calculate metrics with only_final=True

Source: https://github.com/zhangrongxiang/leaptalk/blob/main/eval/common_metrics_on_video_quality/README.md

Key function calls from demo.py to compute FVD, SSIM, PSNR, and LPIPS with only_final=True, which returns only the final metric value.

```python
calculate_fvd(videos1, videos2, device, method='styleganv', only_final=True)
calculate_ssim(videos1, videos2, only_final=True)
calculate_psnr(videos1, videos2, only_final=True)
calculate_lpips(videos1, videos2, device, only_final=True)
```
Tokens:
Raw