AniPortrait

https://github.com/zejun-yang/aniportrait
Admin
AniPortrait is a novel framework for generating high-quality animation driven by audio and a...
Tokens:1,666
Snippets:21
Trust Score:6.1
License:Apache-2.0
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install Project Dependencies

Source: https://github.com/zejun-yang/aniportrait/blob/main/README.md

Install the required Python packages for the AniPortrait project. Ensure you have Python 3.10+ and CUDA 11.7.

```shell
pip install -r requirements.txt
```

--------------------------------

### Run Gradio Web UI

Source: https://github.com/zejun-yang/aniportrait/blob/main/README.md

Execute this command to launch the Gradio web interface for AniPortrait. An online demo is also available on Huggingface Spaces.

```shell
python -m scripts.app

```

--------------------------------

### Training: Stage 1

Source: https://github.com/zejun-yang/aniportrait/blob/main/README.md

Command to launch the Stage 1 training process using accelerate.

```shell
accelerate launch train_stage_1.py --config ./configs/train/stage1.yaml

```

--------------------------------

### Training: Stage 2

Source: https://github.com/zejun-yang/aniportrait/blob/main/README.md

Command to launch the Stage 2 training process. Ensure the pretrained motion module weights are downloaded and placed correctly, and specify stage 1 checkpoint details in the config file.

```shell
accelerate launch train_stage_2.py --config ./configs/train/stage2.yaml

```

--------------------------------

### Directory Structure for Pretrained Weights

Source: https://github.com/zejun-yang/aniportrait/blob/main/README.md

This is the expected directory structure for organizing downloaded pretrained weights. Ensure all necessary files are placed in their respective subdirectories within './pretrained_weights'.

```text
./pretrained_weights/
|-- image_encoder
|   |-- config.json
|   `-- pytorch_model.bin
|-- sd-vae-ft-mse
|   |-- config.json
|   |-- diffusion_pytorch_model.bin
|   `-- diffusion_pytorch_model.safetensors
|-- stable-diffusion-v1-5
|   |-- feature_extractor
|   |   `-- preprocessor_config.json
|   |-- model_index.json
|   |-- unet
|   |   |-- config.json
|   |   `-- diffusion_pytorch_model.bin
|   `-- v1-inference.yaml
|-- wav2vec2-base-960h
|   |-- config.json
|   |-- feature_extractor_config.json
|   |-- preprocessor_config.json
|   |-- pytorch_model.bin
|   |-- README.md
|   |-- special_tokens_map.json
|   |-- tokenizer_config.json
|   `-- vocab.json
|-- audio2mesh.pt
|-- audio2pose.pt
|-- denoising_unet.pth
|-- film_net_fp16.pt
|-- motion_module.pth
|-- pose_guider.pth
`-- reference_unet.pth

```

--------------------------------

### Inference: Convert Video to Pose Video

Source: https://github.com/zejun-yang/aniportrait/blob/main/README.md

Use this command to convert a raw video into a pose video (keypoint sequence) for use in inference.

```shell
python -m scripts.vid2pose --video_path pose_video_path.mp4

```

--------------------------------

### Training: Preprocess Dataset

Source: https://github.com/zejun-yang/aniportrait/blob/main/README.md

Command to preprocess datasets (like VFHQ or CelebV-HQ) for training. This involves extracting keypoints and creating a training JSON file.

```shell
python -m scripts.preprocess_dataset --input_dir VFHQ_PATH --output_dir SAVE_PATH --training_json JSON_PATH

```

--------------------------------

### Inference: Generate Reference Pose for Head Pose Control

Source: https://github.com/zejun-yang/aniportrait/blob/main/README.md

Command to generate a pose_temp.npy file for head pose control, used with the audio-driven model.

```shell
python -m scripts.generate_ref_pose --ref_video ./configs/inference/head_pose_temp/pose_ref_video.mp4 --save_path ./configs/inference/head_pose_temp/pose.npy

```

--------------------------------

### Inference: Self-driven Video Generation

Source: https://github.com/zejun-yang/aniportrait/blob/main/README.md

Command for self-driven video generation using a configuration file. The '-acc' flag enables acceleration if 'film_net_fp16.pt' is downloaded and placed in the weights directory. Adjust '-W' and '-H' for desired output resolution.

```shell
python -m scripts.pose2vid --config ./configs/prompts/animation.yaml -W 512 -H 512 -acc

```

--------------------------------

### Inference: Audio-driven Video Generation

Source: https://github.com/zejun-yang/aniportrait/blob/main/README.md

Command for audio-driven video generation. Specify audios and reference images in the configuration file. The '-acc' flag enables acceleration. Removing 'pose_temp' from the config enables the audio2pose model.

```shell
python -m scripts.audio2vid --config ./configs/prompts/animation_audio.yaml -W 512 -H 512 -acc

```

--------------------------------

### Inference: Face Reenactment

Source: https://github.com/zejun-yang/aniportrait/blob/main/README.md

Command for face reenactment inference. Ensure source face videos and reference images are specified in the configuration file. The '-acc' flag enables acceleration.

```shell
python -m scripts.vid2vid --config ./configs/prompts/animation_facereenac.yaml -W 512 -H 512 -acc

```

--------------------------------

### AniPortrait Citation

Source: https://github.com/zejun-yang/aniportrait/blob/main/README.md

This is the BibTeX citation for the AniPortrait paper, useful for academic referencing.

```bibtex
@misc{wei2024aniportrait,
      title={AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animations},
      author={Huawei Wei and Zejun Yang and Zhisheng Wang},
      year={2024},
      eprint={2403.17694},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
```
Tokens:
Raw