### Example Content of video_rec_path.txt Source: https://github.com/francis-rings/stableavatar/blob/main/README.md This shows an example of the file content for recording video paths, listing directories for speech, singing, and dancing content. ```text path/StableAvatar/talking_face_data/rec/speech/00001 path/StableAvatar/talking_face_data/rec/speech/00002 ... path/StableAvatar/talking_face_data/rec/singing/00003 path/StableAvatar/talking_face_data/rec/singing/00004 ... path/StableAvatar/talking_face_data/rec/dancing/00005 path/StableAvatar/talking_face_data/rec/dancing/00006 ... ``` -------------------------------- ### Run Multi-GPU Inference Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Execute the multi-GPU inference script. This example assumes a setup utilizing 4 GPUs. ```bash bash multiple_gpu_inference.sh ``` -------------------------------- ### Install PyTorch and Dependencies Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Installs PyTorch with CUDA 12.4 support and other project requirements. Optionally installs flash_attn for accelerated attention computation. ```bash pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.1.1 --index-url https://download.pytorch.org/whl/cu124 pip install -r requirements.txt # Optional to install flash_attn to accelerate attention computation pip install flash_attn ``` -------------------------------- ### Download Model Weights Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Installs the necessary Hugging Face library and downloads model weights for StableAvatar. Ensure you are in the StableAvatar directory and have a 'checkpoints' folder created. ```bash pip install "huggingface_hub[cli]" cd StableAvatar mkdir checkpoints huggingface-cli download FrancisRing/StableAvatar --local-dir ./checkpoints ``` -------------------------------- ### Launch Gradio Interface Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Starts a Gradio web interface for interacting with the StableAvatar model. This provides a user-friendly way to test the avatar generation capabilities. ```python python app.py ``` -------------------------------- ### Example Dataset Structure for StableAvatar Source: https://github.com/francis-rings/stableavatar/blob/main/README.md This illustrates the expected directory and file organization for training data, including different video resolutions and content types like speech, singing, and dancing. ```text talking_face_data/ ├── rec │   │  ├──speech │   │  │  ├──00001 │   │  │  │  ├──sub_clip.mp4 │   │  │  │  ├──audio.wav │   │  │  │  ├──images │   │  │  │  │  ├──frame_0.png │   │  │  │  │  ├──frame_1.png │   │  │  │  │  ├──frame_2.png │   │  │  │  │  ├──... │   │  │  │  ├──face_masks │   │  │  │  │  ├──frame_0.png │   │  │  │  │  ├──frame_1.png │   │  │  │  │  ├──frame_2.png │   │  │  │  │  ├──... │   │  │  │  ├──lip_masks │   │  │  │  │  ├──frame_0.png │   │  │  │  │  ├──frame_1.png │   │  │  │  │  ├──frame_2.png │   │  │  │  │  ├──... │   │  │  ├──00002 │   │  │  │  ├──sub_clip.mp4 │   │  │  │  ├──audio.wav │   │  │  │  ├──images │   │  │  │  ├──face_masks │   │  │  │  ├──lip_masks │   │  │  └──... │   │  ├──singing │   │  │  ├──00001 │   │  │  │  ├──sub_clip.mp4 │   │  │  │  ├──audio.wav │   │  │  │  ├──images │   │  │  │  ├──face_masks │   │  │  │  ├──lip_masks │   │  │  └──... │   │  ├──dancing │   │  │  ├──00001 │   │  │  │  ├──sub_clip.mp4 │   │  │  │  ├──audio.wav │   │  │  │  ├──images │   │  │  │  ├──face_masks │   │  │  │  ├──lip_masks │   │  │  └──... ├── vec │   │  ├──speech │   │  │  ├──00001 │   │  │  │  ├──sub_clip.mp4 │   │  │  │  ├──audio.wav │   │  │  │  ├──images │   │  │  │  ├──face_masks │   │  │  │  ├──lip_masks │   │  │  └──... │   │  ├──singing │   │  │  ├──00001 │   │  │  │  ├──sub_clip.mp4 │   │  │  │  ├──audio.wav │   │  │  │  ├──images │   │  │  │  ├──face_masks │   │  │  │  ├──lip_masks │   │  │  └──... │   │  ├──dancing │   │  │  ├──00001 │   │  │  │  ├──sub_clip.mp4 │   │  │  │  ├──audio.wav │   │  │  │  ├──images │   │  │  │  ├──face_masks │   │  │  │  ├──lip_masks │   │  │  └──... ├── square │   │  ├──speech │   │  │  ├──00001 │   │  │  │  ├──sub_clip.mp4 │   │  │  │  ├──audio.wav │   │  │  │  ├──images │   │  │  │  ├──face_masks │   │  │  │  ├──lip_masks │   │  │  └──... │   │  ├──singing │   │  │  ├──00001 │   │  │  │  ├──sub_clip.mp4 │   │  │  │  ├──audio.wav │   │  │  │  ├──images │   │  │  │  ├──face_masks │   │  │  │  ├──lip_masks │   │  │  └──... │   │  ├──dancing │   │  │  ├──00001 │   │  │  │  ├──sub_clip.mp4 │   │  │  │  ├──audio.wav │   │  │  │  ├──images │   │  │  │  ├──face_masks │   │  │  │  ├──lip_masks │   │  │  └──... ├── video_rec_path.txt ├── video_square_path.txt └── video_vec_path.txt ``` -------------------------------- ### Install PyTorch for Blackwell Chips Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Installs PyTorch with CUDA 12.8 support, suitable for Blackwell series chips, and other project requirements. Optionally installs flash_attn for accelerated attention computation. ```bash pip install torch==2.7.0 torchvision==0.22.0 torchaudio==2.7.0 --index-url https://download.pytorch.org/whl/cu128 pip install -r requirements.txt # Optional to install flash_attn to accelerate attention computation pip install flash_attn ``` -------------------------------- ### Extract Lip Masks using Python Script Source: https://github.com/francis-rings/stableavatar/blob/main/README.md This command installs the mediapipe library and runs a Python script to extract lip masks. Adjust `--folder_root`, `--start`, and `--end` as needed. ```bash pip install mediapipe python lip_mask_extractor.py --folder_root="path/StableAvatar/talking_face_data/rec/singing" --start=1 --end=500 ``` -------------------------------- ### Extract Frames from Raw Videos using ffmpeg Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Use this command to extract frames from raw video files and save them as PNG images in the specified directory. Ensure ffmpeg is installed. ```bash ffmpeg -i raw_video_1.mp4 -q:v 1 -start_number 0 path/StableAvatar/talking_face_data/rec/speech/00001/images/frame_%d.png ``` -------------------------------- ### Download Wan2.1-I2V-14B Models Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Download the necessary Wan2.1-I2V-14B models for training. The 720P model is optional. ```bash huggingface-cli download Wan-AI/Wan2.1-I2V-14B-480P --local-dir ./checkpoints/Wan2.1-I2V-14B-480P huggingface-cli download Wan-AI/Wan2.1-I2V-14B-720P --local-dir ./checkpoints/Wan2.1-I2V-14B-720P ``` -------------------------------- ### Finetune StableAvatar (Mixed Resolution) Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Use these scripts to finetune StableAvatar on mixed resolution settings. Add `--transformer_path` to specify the model checkpoint. ```bash # Finetuning StableAvatar on a mixed resolution setting (480x832 and 832x480) in a single machine bash train_1B_rec_vec.sh ``` ```bash # Finetuning StableAvatar on a mixed resolution setting (480x832 and 832x480) in multiple machines bash train_1B_rec_vec_64.sh ``` -------------------------------- ### Integrate Audio with FFmpeg Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Combine a video without audio with a separate audio file to create a final MP4 with sound. Ensure the output path is correctly specified. ```bash ffmpeg -i video_without_audio.mp4 -i /path/audio.wav -c:v copy -c:a aac -shortest /path/output_with_audio.mp4 ``` -------------------------------- ### Run Base Model Inference Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Executes the inference script using a provided configuration file. This script generates avatar videos based on various settings. ```bash bash inference.sh ``` -------------------------------- ### Train Wan2.1-14B StableAvatar on Mixed Resolution Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Command to train the 14B version of StableAvatar on mixed resolutions across multiple machines. ```bash bash train_14B.sh ``` -------------------------------- ### Train StableAvatar-1.3B LoRA on Mixed Resolution Multiple Machines Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Command for LoRA training of StableAvatar-1.3B across multiple machines with mixed resolutions. ```bash bash train_1B_rec_vec_lora_64.sh ``` -------------------------------- ### Train StableAvatar on Single Resolution (512x512) Single Machine Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Use this command to train StableAvatar on a single machine with a fixed 512x512 resolution. ```bash bash train_1B_square.sh ``` -------------------------------- ### Train StableAvatar-1.3B LoRA on Mixed Resolution Single Machine Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Use this command for LoRA training of StableAvatar-1.3B on a single machine with mixed resolutions. ```bash bash train_1B_rec_vec_lora.sh ``` -------------------------------- ### Train StableAvatar on Mixed Resolution (480x832, 832x480) Single Machine Source: https://github.com/francis-rings/stableavatar/blob/main/README.md This command is for training StableAvatar on a single machine using mixed resolutions. ```bash bash train_1B_rec_vec.sh ``` -------------------------------- ### LoRA Finetune StableAvatar Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Use this script for LoRA finetuning of StableAvatar. Modify `--rank` and `--network_alpha` to control LoRA quality. Add `--transformer_path` to specify the model checkpoint. ```bash # Lora-Finetuning StableAvatar-1.3B on a mixed resolution setting (480x832 and 832x480) in a single machine bash train_1B_rec_vec_lora.sh ``` -------------------------------- ### Train StableAvatar on Mixed Resolution (480x832, 832x480) Multiple Machines Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Use this command to train StableAvatar across multiple machines with mixed resolutions. ```bash bash train_1B_rec_vec_64.sh ``` -------------------------------- ### Train StableAvatar Model Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Use this command to train the Wan2.1-1.3B-based StableAvatar model. Ensure your dataset is organized as specified. ```bash python inference.py --model_path "./models/Wan2.1-1.3B" --dataset_path "./talking_face_data/rec/singing" --start 1 --end 500 ``` -------------------------------- ### LoRA Training StableAvatar-14B on Mixed Resolution Multiple Machines Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Command for LoRA training of the 14B version of StableAvatar on multiple machines with mixed resolutions. ```bash bash train_14B_lora.sh ``` -------------------------------- ### Train StableAvatar on Single Resolution (512x512) Multiple Machines Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Use this command for training StableAvatar across multiple machines with a 512x512 resolution. ```bash bash train_1B_square_64.sh ``` -------------------------------- ### Extract Audio from Video Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Extracts audio from a given video file and saves it as a WAV file. Specify the input video path and the desired output audio path. ```python python audio_extractor.py --video_path="path/test/video.mp4" --saved_audio_path="path/test/audio.wav" ``` -------------------------------- ### Separate Vocal from Audio Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Separates vocal signals from an audio file to improve lip synchronization. Requires the 'audio-separator' library and a pre-trained vocal model. ```python pip install audio-separator[gpu] python vocal_seperator.py --audio_separator_model_file="path/StableAvatar/checkpoints/Kim_Vocal_2.onnx" --audio_file_path="path/test/audio.wav" --saved_vocal_path="path/test/vocal.wav" ``` -------------------------------- ### StableAvatar Citation Source: https://github.com/francis-rings/stableavatar/blob/main/README.md Cite this work using the provided BibTeX entry when referencing StableAvatar in academic publications. ```bibtex @article{tu2025stableavatar, title={Stableavatar: Infinite-length audio-driven avatar video generation}, author={Tu, Shuyuan and Pan, Yueming and Huang, Yinming and Han, Xintong and Xing, Zhen and Dai, Qi and Luo, Chong and Wu, Zuxuan and Jiang, Yu-Gang}, journal={arXiv preprint arXiv:2508.08248}, year={2025} } ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.