MuseTalk

https://huggingface.co/TMElyralab/MuseTalk
Admin
MuseTalk is a real-time high quality lip-syncing model that generates audio-driven facial animations...
Tokens:43,415
Snippets:809
Trust Score:10
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install MMLab Packages

Source: https://huggingface.co/TMElyralab/MuseTalk/blob/2d8793f1b062656796acaa236c30ae49ddee0344/README.md?code=true

Installs essential MMLab packages including mmengine, mmcv, mmdet, and mmpose. Ensure you have openmim installed first.

```bash
pip install --no-cache-dir -U openmim 
 mim install mmengine 
 mim install "mmcv>=2.0.1" 
 mim install "mmdet>=3.1.0" 
 mim install "mmpose>=1.1.0"
```

--------------------------------

### Install MMLab Packages

Source: https://huggingface.co/TMElyralab/MuseTalk/blob/a79e8467733779ca2721efdb373d9174a7bfa3b9/README.md

Installs essential MMLab packages including mmengine, mmcv, mmdet, and mmpose. Use the --no-cache-dir option for a clean installation.

```bash
pip install --no-cache-dir -U openmim 
mim install mmengine 
mim install "mmcv>=2.0.1" 
mim install "mmdet>=3.1.0" 
mim install "mmpose>=1.1.0"

```

--------------------------------

### Install MMLab Packages

Source: https://huggingface.co/TMElyralab/MuseTalk/resolve/2d8793f1b062656796acaa236c30ae49ddee0344/README.md?download=true

Installs essential MMLab packages including mmengine, mmcv, and mmdet using openmim.

```bash
pip install --no-cache-dir -U openmim 
mim install mmengine 
mim install "mmcv>=2.0.1" 
mim install "mmdet>=3.1.0" 
mim install "mmpose>=1.1.0" 
```

--------------------------------

### Install MMLab Packages

Source: https://huggingface.co/TMElyralab/MuseTalk/blob/4fa36a0b379591226e1b8e8e53203a57c0395ff2/README.md

Installs essential MMLab packages including mmengine, mmcv, and mmdet.

```bash
pip install --no-cache-dir -U openmim
mim install mmengine
mim install "mmcv>=2.0.1"
mim install "mmdet>=3.1.0"
mim install "mmpose>=1.1.0"

```

--------------------------------

### Install MMLab Packages

Source: https://huggingface.co/TMElyralab/MuseTalk/blame/2d8793f1b062656796acaa236c30ae49ddee0344/README.md

Installs essential MMLab packages for computer vision tasks. This includes openmim, mmengine, mmcv, mmdet, and mmpose. Use the --no-cache-dir flag for a cleaner installation.

```bash
pip install --no-cache-dir -U openmim 
mim install mmengine 
mim install "mmcv>=2.0.1" 
mim install "mmdet>=3.1.0" 
mim install "mmpose>=1.1.0"
```

--------------------------------

### Configure FFMPEG Path

Source: https://huggingface.co/TMElyralab/MuseTalk/blame/226e00a385661303b3767b0f1ebb6a9e45ea4988/README.md

Sets the FFMPEG_PATH environment variable to the location of the downloaded ffmpeg-static binary. Example path provided.

```shell
export FFMPEG_PATH=/musetalk/ffmpeg-4.4-amd64-static
```

--------------------------------

### Install MMLab Packages

Source: https://huggingface.co/TMElyralab/MuseTalk/blame/b4af7f2733a1e0b132618ddd140a72a3eb7414ae/README.md

Installs essential MMLab packages including mmengine, mmcv, mmdet, and mmpose. Ensure you use the specified version constraints for mmcv and mmdet.

```shell
pip install --no-cache-dir -U openmim
mim install mmengine
mim install "mmcv>=2.0.1"
mim install "mmdet>=3.1.0"
mim install "mmpose>=1.1.0"
```

--------------------------------

### Install MMLab Packages

Source: https://huggingface.co/TMElyralab/MuseTalk/blob/b4af7f2733a1e0b132618ddd140a72a3eb7414ae/README.md?code=true

Installs essential MMLab packages including mmengine, mmcv, mmdet, and mmpose. Ensure you have pip version >= 20.3 and a compatible CUDA toolkit.

```bash
pip install --no-cache-dir -U openmim

```

```bash
mim install mmengine

```

```bash
mim install "mmcv>=2.0.1"

```

```bash
mim install "mmdet>=3.1.0"

```

```bash
mim install "mmpose>=1.1.0"

```

--------------------------------

### Install MMLab Packages

Source: https://huggingface.co/TMElyralab/MuseTalk/blob/2d8793f1b062656796acaa236c30ae49ddee0344/README.md

Installs necessary packages from the MMLab ecosystem, including mmengine, mmcv, mmdet, and mmpose. These are essential for various computer vision tasks within MuseTalk.

```bash
pip install --no-cache-dir -U openmim 
mim install mmengine 
mim install "mmcv>=2.0.1" 
mim install "mmdet>=3.1.0" 
mim install "mmpose>=1.1.0" 

```

--------------------------------

### Organize Model Weights

Source: https://huggingface.co/TMElyralab/MuseTalk/blob/da1406d74b45cd1ad67144928ffe6b67ab3ac921/README.md

Illustrates the required directory structure for organizing downloaded model weights, including MuseTalk, dwpose, face-parse-bisent, sd-vae-ft-mse, and whisper.

```bash
./models/
├── musetalk
│   └── musetalk.json
│   └── pytorch_model.bin
├── dwpose
│   └── dw-ll_ucoco_384.pth
├── face-parse-bisent
│   ├── 79999_iter.pth
│   └── resnet18-5c106cde.pth
├── sd-vae-ft-mse
│   ├── config.json
│   └── diffusion_pytorch_model.bin
└── whisper
    └── tiny.pt
```

--------------------------------

### Install Whisper Encoder

Source: https://huggingface.co/TMElyralab/MuseTalk/blame/2d8793f1b062656796acaa236c30ae49ddee0344/README.md

Installs the whisper encoder for extracting audio features. This command installs the whisper package in editable mode from the local musetalk/whisper directory.

```shell
pip install --editable ./musetalk/whisper
```

--------------------------------

### Organize Downloaded Weights

Source: https://huggingface.co/TMElyralab/MuseTalk/blob/2d8793f1b062656796acaa236c30ae49ddee0344/README.md

Illustrates the required directory structure for organizing downloaded model weights. Ensure weights for MuseTalk, dwpose, face-parse-bisent, sd-vae-ft-mse, and whisper are placed correctly within the 'models' directory.

```bash
./models/
├── musetalk
│   └── musetalk.json
│   └── pytorch_model.bin
├── dwpose
│   └── dw-ll_ucoco_384.pth
├── face-parse-bisent
│   ├── 79999_iter.pth
│   └── resnet18-5c106cde.pth
├── sd-vae-ft-mse
│   ├── config.json
│   └── diffusion_pytorch_model.bin
└── whisper
    └── tiny.pt

```

--------------------------------

### Install Whisper for Audio Feature Extraction

Source: https://huggingface.co/TMElyralab/MuseTalk/blob/2d8793f1b062656796acaa236c30ae49ddee0344/README.md

Installs the Whisper library, specifically for extracting audio features (encoder only). This is a required step for audio processing in MuseTalk.

```bash
pip install --editable ./musetalk/whisper

```

--------------------------------

### Install MuseTalk Dependencies

Source: https://huggingface.co/TMElyralab/MuseTalk/blob/2d8793f1b062656796acaa236c30ae49ddee0344/README.md

Installs required Python packages for MuseTalk using a requirements file. Ensure your Python version is >=3.10 and CUDA version is 11.7.

```bash
pip install -r requirements.txt

```

--------------------------------

### Organize Model Weights

Source: https://huggingface.co/TMElyralab/MuseTalk/blob/3ef28bc5cff08c90ad8178a25f1b570cd800170f/README.md?code=true

Illustrates the required directory structure for organizing downloaded pre-trained model weights. Ensure all necessary weights are placed in their respective subdirectories within the './models/' directory.

```text
./models/ 
├── musetalk 
│ └── musetalk.json 
│ └── pytorch_model.bin 
├── dwpose 
│ └── dw-ll_ucoco_384.pth 
├── face-parse-bisent 
│ ├── 79999_iter.pth 
│ └── resnet18-5c106cde.pth 
├── sd-vae-ft-mse 
│ ├── config.json 
│ └── diffusion_pytorch_model.bin 
└── whisper 
 └── tiny.pt
```

--------------------------------

### Run Inference with MuseTalk

Source: https://huggingface.co/TMElyralab/MuseTalk/blame/2d8793f1b062656796acaa236c30ae49ddee0344/README.md

Execute the inference script using a specified configuration file. The configuration should include paths to video and audio.

```python
python -m scripts.inference --inference_config configs/inference/test.yaml 
```
Tokens:
Raw