### Configure Training Environment

Source: https://github.com/tmelyralab/musepose/blob/main/README.md

Commands to install dependencies and configure the accelerate environment for deepspeed.

```bash
pip install accelerate
```

```bash
accelerate config
```

--------------------------------

### Install MMLab Packages

Source: https://github.com/tmelyralab/musepose/blob/main/README.md

Installs essential MMLab packages including openmim, mmengine, mmcv, mmdet, and mmpose. Use --no-cache-dir for cleaner installs.

```bash
pip install --no-cache-dir -U openmim 
mim install mmengine 
mim install "mmcv>=2.0.1" 
mim install "mmdet>=3.1.0" 
mim install "mmpose>=1.1.0"
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/tmelyralab/musepose/blob/main/README.md

Installs the Python dependencies required for the project from the requirements.txt file. Ensure you have Python >= 3.10 and CUDA 11.7.

```shell
pip install -r requirements.txt
```

--------------------------------

### Initialize and Use PoseGuider

Source: https://context7.com/tmelyralab/musepose/llms.txt

Configures the PoseGuider network for pose conditioning and demonstrates loading pretrained weights or initializing from ControlNet-OpenPose.

```python
from musepose.models.pose_guider import PoseGuider
import torch

# Initialize PoseGuider
pose_guider = PoseGuider(
    conditioning_embedding_channels=320,  # Output channels matching UNet
    conditioning_channels=3,              # RGB input
    block_out_channels=(16, 32, 96, 256)  # Progressive channel sizes
)

# Load pretrained weights
pose_guider.load_state_dict(
    torch.load("./pretrained_weights/MusePose/pose_guider.pth")
)
pose_guider = pose_guider.to("cuda", dtype=torch.float16)

# Process pose condition
# Input shape: (batch, channels, frames, height, width)
pose_tensor = torch.randn(1, 3, 48, 768, 768).to("cuda", dtype=torch.float16)
pose_features = pose_guider(pose_tensor)
# Output shape: (batch, 320, frames, height//8, width//8)
print(pose_features.shape)  # torch.Size([1, 320, 48, 96, 96])

# Initialize from ControlNet-OpenPose for training
controlnet_state = torch.load("./pretrained_weights/control_v11p_sd15_openpose/diffusion_pytorch_model.bin")
state_dict_to_load = {}
for k, v in controlnet_state.items():
    if k.startswith("controlnet_cond_embedding.") and "conv_out" not in k:
        new_k = k.replace("controlnet_cond_embedding.", "")
        state_dict_to_load[new_k] = v
pose_guider.load_state_dict(state_dict_to_load, strict=False)
```

--------------------------------

### Launch Training Stages

Source: https://github.com/tmelyralab/musepose/blob/main/README.md

Commands to initiate training for stage 1 and stage 2 using accelerate.

```bash
accelerate launch train_stage_1_multiGPU.py --config configs/train_stage_1.yaml
```

```bash
accelerate launch train_stage_2_multiGPU.py --config configs/train_stage_2.yaml
```

--------------------------------

### Configure Inference Test Cases

Source: https://github.com/tmelyralab/musepose/blob/main/README.md

Sets up the inference configuration file by specifying the reference image and its corresponding aligned pose video.

```yaml
test_cases:
  "./assets/images/ref.png":
    - "./assets/poses/align/img_ref_video_dance.mp4"
```

--------------------------------

### Run Pose2VideoPipeline Programmatically

Source: https://context7.com/tmelyralab/musepose/llms.txt

Initializes the Pose2VideoPipeline with required components and processes a reference image and pose frames to generate a video.

```python
from musepose.pipelines.pipeline_pose2vid_long import Pose2VideoPipeline
from musepose.utils.util import read_frames, get_fps

pipe = Pose2VideoPipeline(
    vae=vae,
    image_encoder=image_enc,
    reference_unet=reference_unet,
    denoising_unet=denoising_unet,
    pose_guider=pose_guider,
    scheduler=scheduler,
)
pipe = pipe.to("cuda", dtype=torch.float16)

ref_image = Image.open("./assets/images/ref.png").convert("RGB")
pose_frames = read_frames("./assets/poses/align/pose_video.mp4")

video = pipe(
    ref_image,
    pose_frames,
    width=768,
    height=768,
    video_length=len(pose_frames),
    num_inference_steps=20,
    guidance_scale=3.5,
    generator=torch.manual_seed(99),
    context_frames=48,
    context_stride=1,
    context_overlap=4,
).videos  # Shape: (b, c, f, h, w)
```

--------------------------------

### Run Stage 1 Training

Source: https://context7.com/tmelyralab/musepose/llms.txt

Executes multi-GPU training for the pose-to-image generation stage using Accelerate.

```bash
# Stage 1 Training: Pose-to-Image
accelerate launch train_stage_1_multiGPU.py --config configs/train_stage_1.yaml
```

```yaml
# Configuration (configs/train_stage_1.yaml):
data:
  train_bs: 8
  train_width: 768
  train_height: 768
  meta_paths:
    - "./meta/dance_dataset.json"
  sample_margin: 128  # Frame margin between ref and target

solver:
  gradient_accumulation_steps: 1
  mixed_precision: 'fp16'
  enable_xformers_memory_efficient_attention: True
  gradient_checkpointing: False
  max_train_steps: 400000
  learning_rate: 1.0e-5
  lr_scheduler: 'constant'
  use_8bit_adam: False

weight_dtype: 'fp16'
uncond_ratio: 0.1
noise_offset: 0.05
snr_gamma: 5.0
enable_zero_snr: True
pose_guider_pretrain: True  # Initialize from ControlNet-OpenPose

seed: 12580
save_model_epoch_interval: 25
exp_name: 'stage_1'
output_dir: './exp_output'

# Pretrained model paths
base_model_path: './pretrained_weights/sd-image-variations-diffusers'
vae_model_path: './pretrained_weights/sd-vae-ft-mse'
image_encoder_path: './pretrained_weights/sd-image-variations-diffusers/image_encoder'
controlnet_openpose_path: './pretrained_weights/control_v11p_sd15_openpose/diffusion_pytorch_model.bin'
```

--------------------------------

### Stage 2 Training Command

Source: https://context7.com/tmelyralab/musepose/llms.txt

Launches the stage 2 training process using accelerate and a specified configuration file. Ensure accelerate is configured for DeepSpeed Zero Stage 2 with appropriate GPU resources.

```bash
accelerate launch train_stage_2_multiGPU.py --config configs/train_stage_2.yaml
```

--------------------------------

### Render DWPose Videos

Source: https://context7.com/tmelyralab/musepose/llms.txt

Converts extracted keypoint numpy files into rendered pose videos for conditioning.

```bash
# Render DWPose videos from extracted keypoints
python draw_dwpose.py \
    --video_dir ./dance_videos \
    --pose_dir ./dance_videos_dwpose_keypoints \
    --save_dir ./dance_videos_dwpose \
    --draw_face False
```

```python
import numpy as np
from pose.script.dwpose import draw_pose

# Load pre-extracted keypoints
poses = np.load("./keypoints.npy", allow_pickle=True).tolist()

# Render each frame
for pose in poses:
    rendered = draw_pose(
        pose,
        H=1024,  # Render height
        W=768,   # Render width
        draw_face=False
    )
    # rendered is a BGR numpy array
```

--------------------------------

### Dataset Preparation: Render Pose Videos

Source: https://context7.com/tmelyralab/musepose/llms.txt

Step 2 of the dataset preparation workflow. Renders pose videos from extracted keypoints. Set `--draw_face False` to exclude facial keypoints.

```python
python draw_dwpose.py --video_dir ./dance_videos --draw_face False
```

--------------------------------

### Stage 2: Pose-to-Video Generation (CLI)

Source: https://context7.com/tmelyralab/musepose/llms.txt

Generates full dance videos from a reference image and aligned pose sequence using command-line arguments. Supports various parameters for video dimensions, length, and processing.

```bash
python test_stage_2.py \
    --config ./configs/test_stage_2.yaml \
    -W 768 \
    -H 768 \
    -L 300 \
    -S 48 \
    -O 4 \
    --cfg 3.5 \
    --seed 99 \
    --steps 20 \
    --skip 1
```

```bash
python test_stage_2.py --config ./configs/test_stage_2.yaml -W 512 -H 512
```

--------------------------------

### Pose2VideoPipeline API: Model Loading and Initialization

Source: https://context7.com/tmelyralab/musepose/llms.txt

Loads necessary models (VAE, image encoder, UNets, pose guider) and a scheduler for the Pose2VideoPipeline. Ensure all pretrained weights paths are correct. The pipeline is moved to CUDA with float16 precision for optimization.

```python
from musepose.pipelines.pipeline_pose2vid import Pose2VideoPipeline
from diffusers import AutoencoderKL, DDIMScheduler
from transformers import CLIPVisionModelWithProjection
from musepose.models.pose_guider import PoseGuider
from musepose.models.unet_2d_condition import UNet2DConditionModel
from musepose.models.unet_3d import UNet3DConditionModel
import torch

# Load models
vae = AutoencoderKL.from_pretrained("./pretrained_weights/sd-vae-ft-mse")
image_enc = CLIPVisionModelWithProjection.from_pretrained(
    "./pretrained_weights/image_encoder"
)
reference_unet = UNet2DConditionModel.from_pretrained(
    "./pretrained_weights/sd-image-variations-diffusers",
    subfolder="unet"
)
denoisin_unet = UNet3DConditionModel.from_pretrained_2d(
    "./pretrained_weights/sd-image-variations-diffusers",
    "./pretrained_weights/MusePose/motion_module.pth",
    subfolder="unet",
    unet_additional_kwargs={"use_motion_module": True}
)
pose_guider = PoseGuider(320, block_out_channels=(16, 32, 96, 256))

# Load trained weights
denoisin_unet.load_state_dict(torch.load("./pretrained_weights/MusePose/denoising_unet.pth"))
reference_unet.load_state_dict(torch.load("./pretrained_weights/MusePose/reference_unet.pth"))
pose_guider.load_state_dict(torch.load("./pretrained_weights/MusePose/pose_guider.pth"))

scheduler = DDIMScheduler(
    beta_start=0.00085,
    beta_end=0.012,
    beta_schedule="scaled_linear",
    prediction_type="v_prediction",
    rescale_betas_zero_snr=True,
    timestep_spacing="trailing"
)

# Create pipeline
pipe = Pose2VideoPipeline(
    vae=vae,
    image_encoder=image_enc,
    reference_unet=reference_unet,
    denoising_unet=denoising_unet,
    pose_guider=pose_guider,
    scheduler=scheduler
)
pipe = pipe.to("cuda", dtype=torch.float16)

# Enable memory optimization
pipe.enable_vae_slicing()
```

--------------------------------

### Utilize MusePose Helper Functions

Source: https://context7.com/tmelyralab/musepose/llms.txt

Provides utilities for video I/O, frame manipulation, and setting random seeds for reproducibility.

```python
from musepose.utils.util import (
    read_frames,
    get_fps,
    save_videos_grid,
    save_videos_from_pil,
    seed_everything
)
import torch

# Set reproducible random seeds
seed_everything(42)

# Read video frames as PIL images
frames = read_frames("./video.mp4")
print(f"Loaded {len(frames)} frames")

# Get video FPS
fps = get_fps("./video.mp4")
print(f"Video FPS: {fps}")

# Save PIL images as video
save_videos_from_pil(
    pil_images=frames,
    path="./output.mp4",
    fps=24
)

# Save tensor as video grid (for comparing multiple outputs)
# video_tensor shape: (batch, channels, frames, height, width)
video_tensor = torch.randn(3, 3, 48, 256, 256)  # 3 videos
save_videos_grid(
    videos=video_tensor,
    path="./comparison_grid.mp4",
    rescale=True,  # Rescale from [-1,1] to [0,1]
    n_rows=3,      # Number of videos per row
    fps=24
)

# Save as GIF instead of MP4
save_videos_from_pil(frames[:30], "./output.gif", fps=10)
```

--------------------------------

### Run MusePose Inference

Source: https://github.com/tmelyralab/musepose/blob/main/README.md

Executes the main inference script using the provided configuration file. Output results will be saved in the ./output/ directory.

```python
python test_stage_2.py --config ./configs/test_stage_2.yaml
```

--------------------------------

### Stage 1: Pose-to-Image Generation (Programmatic)

Source: https://context7.com/tmelyralab/musepose/llms.txt

Programmatic usage of the Pose2ImagePipeline for generating images. Requires loading models and preparing reference and pose images.

```python
from PIL import Image
from musepose.pipelines.pipeline_pose2img import Pose2ImagePipeline

pipe = Pose2ImagePipeline(
    vae=vae,
    image_encoder=image_enc,
    reference_unet=reference_unet,
    denoising_unet=denoising_unet,
    pose_guider=pose_guider,
    scheduler=scheduler,
)
pipe = pipe.to("cuda", dtype=torch.float16)

ref_image = Image.open("./assets/images/ref.png").convert("RGB")
pose_image = Image.open("./assets/poses/pose.jpg").convert("RGB")

result = pipe(
    ref_image,
    pose_image,
    width=768,
    height=768,
    num_inference_steps=20,
    guidance_scale=7.0,
    generator=torch.manual_seed(42),
)
output_image = result.images  # Shape: (b, c, 1, h, w)
```

--------------------------------

### Dataset Preparation: Extract DWPose Keypoints

Source: https://context7.com/tmelyralab/musepose/llms.txt

Step 1 of the dataset preparation workflow. Extracts DensePose keypoints from dance videos. Ensure the video directory is correctly specified.

```python
python extract_dwpose_keypoints.py --video_dir ./dance_videos
```

--------------------------------

### Stage 1: Pose-to-Image Generation (CLI)

Source: https://context7.com/tmelyralab/musepose/llms.txt

Generates single images from a reference image and pose condition using command-line arguments. Configure test cases in the specified YAML file.

```bash
python test_stage_1.py \
    --config ./configs/test_stage_1.yaml \
    -W 768 \
    -H 768 \
    --seed 42 \
    --cfg 7 \
    --steps 20 \
    --cnt 1
```

--------------------------------

### Accelerate Configuration for DeepSpeed

Source: https://context7.com/tmelyralab/musepose/llms.txt

Configures the accelerate library for DeepSpeed with Zero Stage 2. This is recommended for multi-GPU training environments. Select 'DeepSpeed', 'Zero Stage 2', and 'no offloading' when prompted.

```bash
accelerate config
```

--------------------------------

### Stage 2 Training Configuration Additions

Source: https://context7.com/tmelyralab/musepose/llms.txt

Key configuration parameters for Stage 2 training, enabling the motion module and specifying its properties. This section highlights differences from Stage 1 training.

```yaml
unet_additional_kwargs:
  use_motion_module: true
  motion_module_resolutions: [1, 2, 4, 8]
  motion_module_mid_block: true
  motion_module_type: Vanilla
  motion_module_kwargs:
    num_attention_heads: 8
    num_transformer_block: 1
    temporal_position_encoding: true
    temporal_position_encoding_max_len: 128
```

--------------------------------

### Run MusePose Inference with Reduced VRAM

Source: https://github.com/tmelyralab/musepose/blob/main/README.md

Performs inference while reducing VRAM cost by setting specific width and height. The video is generated at the specified resolution and then resized.

```python
python test_stage_2.py --config ./configs/test_stage_2.yaml -W 512 -H 512
```

--------------------------------

### Dataset Preparation: Generate Metadata JSON

Source: https://context7.com/tmelyralab/musepose/llms.txt

Step 3 of the dataset preparation workflow. Creates a JSON file mapping video paths to their corresponding pose sequences. The output file path is determined by the `--dataset_name` argument.

```python
python extract_meta_info_multiple_dataset.py \
    --video_dirs ./dance_videos \
    --dataset_name my_dance_data
```

--------------------------------

### Pose2VideoPipeline API: Video Generation

Source: https://context7.com/tmelyralab/musepose/llms.txt

Generates a video using the Pose2VideoPipeline with a reference image and pose sequences. Key parameters include video dimensions, length, inference steps, and guidance scale. The output is a tensor representing the video frames.

```python
# Generate video
output = pipe(
    ref_image=ref_pil_image,
    pose_images=list_of_pose_pil_images,
    width=768,
    height=768,
    video_length=48,
    num_inference_steps=20,
    guidance_scale=3.5,
    generator=torch.manual_seed(42),
    output_type="tensor"
)
video_tensor = output.videos  # (batch, channels, frames, height, width)
```

--------------------------------

### Pose Alignment Script

Source: https://github.com/tmelyralab/musepose/blob/main/README.md

Runs the pose alignment script using a reference image and a dance video. The output includes aligned poses and debug visualizations.

```python
python pose_align.py --imgfn_refer ./assets/images/ref.png --vidfn ./assets/videos/dance.mp4
```

--------------------------------

### Extract and Process Pose Data

Source: https://github.com/tmelyralab/musepose/blob/main/README.md

Commands to extract DWPose keypoints and generate meta information for training datasets.

```bash
python extract_dwpose_keypoints.py --video_dir ./xxx
```

```bash
python draw_dwpose.py --video_dir ./xxx
```

```bash
python extract_meta_info_multiple_dataset.py --video_dirs ./xxx --dataset_name xxx
```

--------------------------------

### Align Poses with Reference Image

Source: https://context7.com/tmelyralab/musepose/llms.txt

Aligns pose keypoints from a dance video to match the body proportions of a reference image. Specify input image, video, and output paths. Custom weights and configurations can be provided.

```bash
python pose_align.py \
    --imgfn_refer ./assets/images/ref.png \
    --vidfn ./assets/videos/dance.mp4 \
    --detect_resolution 512 \
    --image_resolution 720 \
    --align_frame 0 \
    --max_frame 300
```

```bash
python pose_align.py \
    --imgfn_refer ./my_image.png \
    --vidfn ./my_dance.mp4 \
    --outfn_align_pose_video ./output/aligned_pose.mp4 \
    --outfn ./output/alignment_demo.mp4 \
    --yolox_config ./pose/config/yolox_l_8xb8-300e_coco.py \
    --dwpose_config ./pose/config/dwpose-l_384x288.py \
    --yolox_ckpt ./pretrained_weights/dwpose/yolox_l_8x8_300e_coco.pth \
    --dwpose_ckpt ./pretrained_weights/dwpose/dw-ll_ucoco_384.pth
```

--------------------------------

### Extract DWPose Keypoints

Source: https://context7.com/tmelyralab/musepose/llms.txt

Extracts skeletal keypoints from videos using DWPose and YOLOX. The output is saved as numpy arrays.

```bash
# Extract DWPose keypoints from all videos in a directory
python extract_dwpose_keypoints.py \
    --video_dir ./dance_videos \
    --save_dir ./dance_videos_dwpose_keypoints \
    --yolox_config ./pose/config/yolox_l_8xb8-300e_coco.py \
    --dwpose_config ./pose/config/dwpose-l_384x288.py \
    --yolox_ckpt ./pretrained_weights/dwpose/yolox_l_8x8_300e_coco.pth \
    --dwpose_ckpt ./pretrained_weights/dwpose/dw-ll_ucoco_384.pth
```

```python
from pose.script.dwpose import DWposeDetector

detector = DWposeDetector(
    det_config="./pose/config/yolox_l_8xb8-300e_coco.py",
    det_ckpt="./pretrained_weights/dwpose/yolox_l_8x8_300e_coco.pth",
    pose_config="./pose/config/dwpose-l_384x288.py",
    pose_ckpt="./pretrained_weights/dwpose/dw-ll_ucoco_384.pth",
    keypoints_only=True
)
detector = detector.to("cuda")

# Extract keypoints from a single frame
import cv2
frame = cv2.imread("frame.jpg")
keypoints = detector(frame)
# keypoints contains: bodies, hands, faces
```

--------------------------------

### MusePose Citation

Source: https://github.com/tmelyralab/musepose/blob/main/README.md

BibTeX citation for the MusePose framework.

```bib
@article{musepose,
  title={MusePose: a Pose-Driven Image-to-Video Framework for Virtual Human Generation},
  author={Tong, Zhengyan and Li, Chao and Chen, Zhaokang and Wu, Bin and Zhou, Wenjiang},
  journal={arxiv},
  year={2024}
}
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.