### Install UniAnimate and Dependencies

Source: https://context7.com/ali-vilab/unianimate/llms.txt

Sets up the conda environment and installs all necessary dependencies, including PyTorch with CUDA support and optional ONNX runtime.

```bash
git clone https://github.com/ali-vilab/UniAnimate.git
cd UniAnimate

conda create -n UniAnimate python=3.9
conda activate UniAnimate

# Install PyTorch 2.0.1 with CUDA 11.8
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 \
    pytorch-cuda=11.8 -c pytorch -c nvidia

pip install -r requirements.txt

# Optional: GPU-accelerated ONNX runtime for pose alignment
pip install onnxruntime-gpu==1.13.1
```

--------------------------------

### Install Python Dependencies for UniAnimate

Source: https://github.com/ali-vilab/unianimate/blob/main/README.md

Clone the repository, create and activate a conda environment, install PyTorch with CUDA support, and then install the remaining project dependencies.

```bash
git clone https://github.com/ali-vilab/UniAnimate.git
cd UniAnimate
conda create -n UniAnimate python=3.9
conda activate UniAnimate
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia
pip install -r requirements.txt
```

--------------------------------

### Run Pose Alignment with Reference Image

Source: https://github.com/ali-vilab/unianimate/blob/main/README.md

Execute this command to align poses using a reference image and source video. Ensure onnxruntime-gpu is installed for GPU acceleration. The first frame of the target pose sequence is crucial for accurate scale coefficient calculation.

```bash
python run_align_pose.py  --ref_name data/images/IMG_20240514_104337.jpg --source_video_paths data/videos/source_video.mp4 --saved_pose_dir data/saved_pose/IMG_20240514_104337
```

--------------------------------

### Download Pretrained Checkpoints using ModelScope

Source: https://github.com/ali-vilab/unianimate/blob/main/README.md

Use the ModelScope library to download the necessary pretrained models for UniAnimate. Ensure the checkpoints are organized in the './checkpoints/' directory after downloading.

```python
!pip install modelscope
from modelscope.hub.snapshot_download import snapshot_download
model_dir = snapshot_download('iic/unianimate', cache_dir='checkpoints/')
```

```bash
mv ./checkpoints/iic/unianimate/* ./checkpoints/
```

--------------------------------

### Programmatic Short Video Inference

Source: https://context7.com/ali-vilab/unianimate/llms.txt

Load configuration and invoke the `inference_unianimate_entrance` function programmatically. Ensure all necessary tools and registry entries are imported.

```python
# Programmatic invocation (mirrors what inference.py does)
from utils.config import Config
from utils.registry_class import INFER_ENGINE
from tools import *  # registers all INFER_ENGINE entries

# Load config and dispatch to inference_unianimate_entrance()
cfg_update = Config(load=True)  # reads --cfg flag
INFER_ENGINE.build(
    dict(type=cfg_update.TASK_TYPE),  # "inference_unianimate_entrance"
    cfg_update=cfg_update.cfg_dict
)
# Output MP4 files are saved to cfg.log_dir (default: ./workdir/)
```

--------------------------------

### Download Pretrained Checkpoints with ModelScope

Source: https://context7.com/ali-vilab/unianimate/llms.txt

Downloads required model weights using ModelScope's snapshot_download function and reorganizes them into a flat layout in the checkpoints directory.

```python
from modelscope.hub.snapshot_download import snapshot_download

# Downloads to ./checkpoints/iic/unianimate/ 
model_dir = snapshot_download('iic/unianimate', cache_dir='checkpoints/')

# Move to expected flat layout
import shutil, os
for f in os.listdir('./checkpoints/iic/unianimate/'):
    shutil.move(f'./checkpoints/iic/unianimate/{f}', f'./checkpoints/{f}')

# Expected final layout:
# checkpoints/
# 	├── dw-ll_ucoco_384.onnx       (DWPose keypoint model)
# 	├── open_clip_pytorch_model.bin (CLIP visual encoder)
# 	├── unianimate_16f_32f_non_ema_223000.pth  (main UNet checkpoint)
# 	├── v2-1_512-ema-pruned.ckpt   (VAE / autoencoder)
# 	└── yolox_l.onnx               (person detector for pose extraction)
```

--------------------------------

### Generate Long Videos with UniAnimate

Source: https://github.com/ali-vilab/unianimate/blob/main/README.md

Execute this command to synthesize long videos. Ensure you have the correct configuration file specified.

```bash
python inference.py --cfg configs/UniAnimate_infer_long.yaml
```

--------------------------------

### Short Video Inference Configuration

Source: https://context7.com/ali-vilab/unianimate/llms.txt

Key parameters for `configs/UniAnimate_infer.yaml`. Adjust `max_frames`, `resolution`, and `ddim_timesteps` for desired output. `CPU_CLIP_VAE` saves VRAM.

```yaml
# configs/UniAnimate_infer.yaml — key parameters
max_frames: 32             # number of output frames (8/16/24/32/48/64 all work)
resolution: [512, 768]     # [width, height]; use [768, 1216] for higher resolution
ddim_timesteps: 30         # DDIM denoising steps (25–50 range)
seed: 11
CPU_CLIP_VAE: True         # offload CLIP + VAE to CPU to save ~14GB VRAM
noise_prior_value: 949     # noise prior blend strength (999=strong, 939=subtle)

# Input data list: [frame_interval, reference_image, pose_sequence_dir]
test_list_path: [
    [2, "data/images/musk.jpg", "data/saved_pose/musk"],
    [2, "data/images/WOMEN-Blouses_Shirts-id_00004955-01_4_full.jpg",
        "data/saved_pose/WOMEN-Blouses_Shirts-id_00004955-01_4_full"],
]

# Conditioning modes run per inference item
partial_keys: [
    ['image', 'local_image', 'dwpose'],  # reference image anchored as first frame
    ['image', 'randomref',   'dwpose'],  # reference image as soft appearance prior
]
```

--------------------------------

### Generate Video Clips with UniAnimate Model

Source: https://github.com/ali-vilab/unianimate/blob/main/README.md

Run the UniAnimate model to generate 32-frame video clips at 768x512 resolution. Adjust `max_frames` in the config file to reduce GPU memory usage if necessary.

```bash
python inference.py --cfg configs/UniAnimate_infer.yaml
```

--------------------------------

### Build and Load UniAnimate UNet Model

Source: https://context7.com/ali-vilab/unianimate/llms.txt

Configures and loads the UNetSD_UniAnimate model. Ensure the configuration dictionary matches your desired model parameters and the checkpoint path is correct.

```python
from tools.modules.unet.unet_unianimate import UNetSD_UniAnimate
import torch

# Build as configured in YAML (via registry)
unet_cfg = {
    'type': 'UNetSD_UniAnimate',
    'in_dim': 4,           # latent channel count (VAE output)
    'dim': 320,            # base feature dimension
    'y_dim': 1024,         # text/image conditioning dim (CLIP)
    'context_dim': 1024,
    'out_dim': 4,
    'dim_mult': [1, 2, 4, 4],
    'num_heads': 8,
    'head_dim': 64,
    'num_res_blocks': 2,
    'dropout': 0.1,
    'temporal_attention': True,
    'num_tokens': 4,
    'temporal_attn_times': 1,
    'use_checkpoint': True,    # gradient checkpointing to save memory
    'use_fps_condition': False,
    'use_sim_mask': False,
    'zero_y': zero_y,          # null CLIP embedding for classifier-free guidance
}

# Model is built and weights loaded like this:
from utils.registry_class import MODEL
model = MODEL.build(unet_cfg)
state_dict = torch.load('checkpoints/unianimate_16f_32f_non_ema_223000.pth', map_location='cpu')
model.load_state_dict(state_dict, strict=True)
model = model.to('cuda').eval().to(torch.float16)
```

--------------------------------

### Configure and Initialize DiffusionDDIM Scheduler

Source: https://context7.com/ali-vilab/unianimate/llms.txt

Sets up the DDIM diffusion scheduler with specified parameters for the reverse diffusion process. Adjust 'num_timesteps' and beta schedule for different sampling needs.

```python
from tools.modules.diffusions.diffusion_ddim import DiffusionDDIM
import torch

# Build diffusion scheduler (as in config)
diffusion = DiffusionDDIM(
    schedule='linear_sd',
    schedule_param={
        'num_timesteps': 1000,
        'init_beta': 0.00085,
        'last_beta': 0.0120,
        'zero_terminal_snr': True,
    },
    mean_type='v',
    loss_type='mse',
    var_type='fixed_small',
    rescale_timesteps=False,
    noise_strength=0.1
)
```

--------------------------------

### Troubleshoot inconsistent appearance at high resolution

Source: https://context7.com/ali-vilab/unianimate/llms.txt

If you encounter appearance inconsistencies at high resolutions, try adjusting the seed, falling back to a standard resolution, or increasing the context overlap for long videos.

```bash
# Troubleshooting inconsistent appearance at high resolution:
# 1. Try a different seed: seed: 42
# 2. Fall back to standard resolution: resolution: [512, 768]
# 3. For long videos, increase overlap: context_overlap: 16
```

--------------------------------

### Align Driving Poses to Reference Image

Source: https://context7.com/ali-vilab/unianimate/llms.txt

Extracts and rescales DWPose keypoints from a driving video to match a reference person's body proportions. Uses OneEuroFilter for temporal smoothing and outputs per-frame pose images.

```bash
# Align driving poses to a fashion image reference
python run_align_pose.py \
    --ref_name data/images/WOMEN-Blouses_Shirts-id_00004955-01_4_full.jpg \
    --source_video_paths data/videos/source_video.mp4 \
    --saved_pose_dir data/saved_pose/WOMEN-Blouses_Shirts-id_00004955-01_4_full

# Align poses to a portrait reference
python run_align_pose.py \
    --ref_name data/images/musk.jpg \
    --source_video_paths data/videos/source_video.mp4 \
    --saved_pose_dir data/saved_pose/musk

# Resulting directory structure:
# data/saved_pose/musk/
# 	├── ref_pose.jpg       (pose of the reference image itself)
# 	├── 0000.jpg           (aligned pose frame 0)
# 	├── 0001.jpg           (aligned pose frame 1)
# 	└── ...
```

--------------------------------

### Load Video Frames Utility

Source: https://context7.com/ali-vilab/unianimate/llms.txt

Utility function `load_video_frames` for preprocessing video and pose data. It applies specified transformations and returns conditioning tensors for the UNet.

```python
import torchvision.transforms as T
import utils.transforms as data
from tools.inferences.inference_unianimate_entrance import load_video_frames

resolution = [512, 768]  # [width, height]
vit_resolution = [224, 224]

train_trans = data.Compose([
    data.Resize(resolution),
    data.ToTensor(),
    data.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])
])
train_trans_pose = data.Compose([
    data.Resize(resolution),
    data.ToTensor(),
])
vit_transforms = T.Compose([
    data.Resize(vit_resolution),
    T.ToTensor(),
    T.Normalize(mean=[0.48145466, 0.4578275, 0.40821073],
                std=[0.26862954, 0.26130258, 0.27577711])
])

vit_frame, video_data, misc_data, dwpose_data, \
    random_ref_frame_data, random_ref_dwpose_data = load_video_frames(
        ref_image_path="data/images/musk.jpg",
        pose_file_path="data/saved_pose/musk",
        train_trans=train_trans,
        vit_transforms=vit_transforms,
        train_trans_pose=train_trans_pose,
        max_frames=32,
        frame_interval=2,
        resolution=resolution,
        get_first_frame=True,
        vit_resolution=vit_resolution
    )

```

--------------------------------

### Long Video Inference Configuration

Source: https://context7.com/ali-vilab/unianimate/llms.txt

Key parameters for `configs/UniAnimate_infer_long.yaml`. Use `max_frames: "None"` for full sequence length. Adjust `context_size` and `context_overlap` for windowed denoising.

```yaml
# configs/UniAnimate_infer_long.yaml — key parameters
resolution: [768, 1216]    # higher resolution for long videos
max_frames: "None"         # "None" = use all frames in pose sequence; or 64, 96
context_size: 32           # frames per sliding window
context_stride: 1
context_overlap: 8         # overlapping frames between adjacent windows (use 16 for consistency)
context_batch_size: 1      # increase to 4 on A100 (80GB) for parallel window denoising
ddim_timesteps: 30
seed: 7
CPU_CLIP_VAE: True
noise_prior_value: 939     # lower value = better motion diversity

# frame_interval=2 means every other frame of the pose sequence is used
test_list_path: [
    [2, "data/images/musk.jpg", "data/saved_pose/musk"],
    [1, "data/images/IMG_20240514_104337.jpg",
        "data/saved_pose/IMG_20240514_104337"],  # frame_interval=1: use every frame
]
```

--------------------------------

### High-Resolution Inference Configuration

Source: https://context7.com/ali-vilab/unianimate/llms.txt

Enables high-resolution video generation by adjusting the 'resolution' parameter in the configuration file. The model supports upscaling to 768x1216 without fine-tuning.

```yaml
# configs/UniAnimate_infer.yaml
resolution: [768, 1216]   # change from default [512, 768]
```

--------------------------------

### Run Long Video Inference

Source: https://context7.com/ali-vilab/unianimate/llms.txt

Execute the long video inference script using a specific configuration file. This command generates full-length videos by processing the entire pose sequence.

```bash
# Generate full-length video (all frames in pose sequence)
python inference.py --cfg configs/UniAnimate_infer_long.yaml
```

--------------------------------

### DDIM Sampling Loop with UniAnimate

Source: https://context7.com/ali-vilab/unianimate/llms.txt

Performs the denoising process using DDIM sampling. This includes initializing noise, blending it with a reference latent, and running the UNet with classifier-free guidance. Ensure 'model_kwargs' contains necessary conditioning information.

```python
# Initial noise with offset noise
noise = torch.randn([1, 4, 32, 96, 64], device='cuda')  # [B, C, F, H/8, W/8]
b, c, f, _, _ = noise.shape
offset_noise = torch.randn(b, c, f, 1, 1, device='cuda')
noise = noise + 0.1 * offset_noise

# Blend noise toward reference image latent (appearance prior)
# noise_prior_value: 949 for clips, 939 for long videos
noise = diffusion.q_sample(random_ref_frame_latent.clone(), 949, noise=noise)

# Run DDIM denoising (classifier-free guidance scale=2.5)
video_latents = diffusion.ddim_sample_loop(
    noise=noise,
    model=model.eval(),
    model_kwargs=model_kwargs,  # contains 'image', 'dwpose', 'randomref', etc.
    guide_scale=2.5,
    ddim_timesteps=30,
    eta=0.0  # deterministic DDIM (eta>0 adds stochasticity)
)
# video_latents shape: [1, 4, 32, 96, 64] → decoded by VAE to [1, 3, 32, 768, 512]
```

--------------------------------

### Align Target Pose with Reference Image

Source: https://github.com/ali-vilab/unianimate/blob/main/README.md

Rescale the target pose sequence to match the pose of the reference image using the provided Python script. This is an important step for ensuring accurate animation.

```python
python run_align_pose.py  --ref_name data/images/WOMEN-Blouses_Shirts-id_00004955-01_4_full.jpg --source_video_paths data/videos/source_video.mp4 --saved_pose_dir data/saved_pose/WOMEN-Blouses_Shirts-id_00004955-01_4_full
```

```python
python run_align_pose.py  --ref_name data/images/musk.jpg --source_video_paths data/videos/source_video.mp4 --saved_pose_dir data/saved_pose/musk
```

```python
python run_align_pose.py  --ref_name data/images/WOMEN-Blouses_Shirts-id_00005125-03_4_full.jpg --source_video_paths data/videos/source_video.mp4 --saved_pose_dir data/saved_pose/WOMEN-Blouses_Shirts-id_00005125-03_4_full
```

--------------------------------

### Reduce GPU Memory Usage with CLIP/VAE Offloading and Float16

Source: https://github.com/ali-vilab/unianimate/blob/main/README.md

To significantly reduce GPU memory requirements, offload CLIP and VAE to CPU and explicitly use torch.float16. This allows generating a 32x768x512 video clip with approximately 12GB of GPU memory. Refer to the issue for detailed instructions.

```yaml
CPU_CLIP_VAE: True
```

--------------------------------

### DWposeDetector for Whole-Body Pose Estimation

Source: https://context7.com/ali-vilab/unianimate/llms.txt

Initializes the DWposeDetector, which uses ONNX models for person detection and pose estimation to extract 133 whole-body keypoints (body, hands, face) from an image. Visualizes the detected pose on a canvas.

```python
from run_align_pose import DWposeDetector, draw_pose
import cv2

# Initialize detector (loads ONNX models from checkpoints/)
dwpose_model = DWposeDetector()

frame = cv2.imread("data/images/musk.jpg", cv2.IMREAD_COLOR)
pose = dwpose_model(frame)  # returns dict with 'bodies', 'hands', 'faces'

H, W, _ = frame.shape
canvas_no_face, canvas_with_face = draw_pose(pose, H, W)

cv2.imwrite("pose_vis.jpg", canvas_with_face)

# pose dict structure:
# {
#   'bodies': {'candidate': array(shape=[20, 2]), 'subset': array(shape=[1, 20])},
#   'hands':  array(shape=[2, 21, 2]),
#   'faces':  array(shape=[1, 68, 2])
# }
print("Body keypoints shape:", pose['bodies']['candidate'].shape)  # (20, 2)
```

--------------------------------

### CPU Offloading for Memory Optimization

Source: https://context7.com/ali-vilab/unianimate/llms.txt

Manages GPU memory by offloading CLIP encoder and VAE to CPU during denoising and reloading them afterward. This significantly reduces peak GPU memory usage, especially for high-resolution outputs.

```python
# This pattern is applied automatically inside inference_unianimate_entrance:
if cfg.CPU_CLIP_VAE:
    clip_encoder.cpu()
    autoencoder.cpu()
    torch.cuda.empty_cache()

video_data = diffusion.ddim_sample_loop(...)  # only UNet on GPU

if cfg.CPU_CLIP_VAE:
    clip_encoder.cuda()
    autoencoder.cuda()

# Decode latents back to pixel space
video_data = 1.0 / cfg.scale_factor * video_data
video_data = rearrange(video_data, 'b c f h w -> (b f) c h w')
decoded_frames = autoencoder.decode(video_data)
```

--------------------------------

### UniAnimate Research Paper Citation

Source: https://github.com/ali-vilab/unianimate/blob/main/README.md

Cite this paper if you find the UniAnimate codebase useful for your research. It details the methodology behind the model.

```bibtex
@article{wang2024unianimate,
      title={UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation},
      author={Wang, Xiang and Zhang, Shiwei and Gao, Changxin and Wang, Jiayu and Zhou, Xiaoqiang and Zhang, Yingya and Yan, Luxin and Sang, Nong},
      journal={arXiv preprint arXiv:2406.01188},
      year={2024}
}
```

--------------------------------

### Add Noise Prior for Improved Appearance Preservation

Source: https://github.com/ali-vilab/unianimate/blob/main/README.md

A noise prior has been added to enhance appearance preservation, particularly for background details in long video generation. This modification is applied during the sampling process.

```python
noise = diffusion.q_sample(random_ref_frame.clone(), getattr(cfg, "noise_prior_value", 939), noise=noise)
```

--------------------------------

### Enable Multi-Segment Parallel Denoising for Long Videos

Source: https://github.com/ali-vilab/unianimate/blob/main/README.md

For GPUs with large memory, such as A100, you can enable multi-segment parallel denoising to accelerate long video inference. Modify the `context_batch_size` in the configuration file to a value greater than 1.

```yaml
context_batch_size: 1
```

--------------------------------

### Print Tensor Shapes

Source: https://context7.com/ali-vilab/unianimate/llms.txt

Prints the shapes of various tensor outputs from the model. Useful for debugging and understanding data dimensions.

```python
print(vit_frame.shape)            # torch.Size([3, 224, 224])
print(video_data.shape)           # torch.Size([32, 3, 768, 512])
print(dwpose_data.shape)          # torch.Size([32, 3, 768, 512])
print(random_ref_frame_data.shape)# torch.Size([32, 3, 768, 512])
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.