### Setup Repository and Environment

Source: https://github.com/guoyww/animatediff/blob/main/README.md

Clone the repository and install the required dependencies to prepare the environment.

```bash
git clone https://github.com/guoyww/AnimateDiff.git
cd AnimateDiff

pip install -r requirements.txt
```

--------------------------------

### AnimateDiff Configuration Example

Source: https://context7.com/guoyww/animatediff/llms.txt

Example of an AnimateDiff configuration file in YAML format. These files define model, prompts, and generation parameters for animation creation.

```yaml
# configs/prompts/custom_animation.yaml

```

--------------------------------

### SparseCtrl Image-to-Video Control Setup

Source: https://context7.com/guoyww/animatediff/llms.txt

This Python script sets up an AnimateDiff pipeline with SparseControlNet for image-to-video generation. It loads controlnet configurations, weights, and preprocesses an input image for conditioning.

```python
import torch
import torchvision.transforms as transforms
from PIL import Image
from einops import rearrange

from animatediff.models.sparse_controlnet import SparseControlNetModel
from animatediff.pipelines.pipeline_animation import AnimationPipeline
from animatediff.utils.util import load_weights, auto_download
from omegaconf import OmegaConf

# Load controlnet configuration
controlnet_config = OmegaConf.load("configs/inference/sparsectrl/latent_condition.yaml")

# Create SparseCtrl encoder from UNet
controlnet = SparseControlNetModel.from_unet(
    unet,
    controlnet_additional_kwargs=controlnet_config.get("controlnet_additional_kwargs", {})
)

# Load controlnet weights
controlnet_path = "models/SparseCtrl/v3_sd15_sparsectrl_rgb.ckpt"
auto_download(controlnet_path, is_dreambooth_lora=False)
controlnet_state_dict = torch.load(controlnet_path, map_location="cpu")
controlnet.load_state_dict(controlnet_state_dict, strict=False)
controlnet.cuda()

# Create pipeline with controlnet
pipeline = AnimationPipeline(
    vae=vae,
    text_encoder=text_encoder,
    tokenizer=tokenizer,
    unet=unet,
    controlnet=controlnet,
    scheduler=DDIMScheduler(**OmegaConf.to_container(inference_config.noise_scheduler_kwargs)),
).to("cuda")

# Load and preprocess control image
image_transforms = transforms.Compose([
    transforms.RandomResizedCrop((512, 512), (1.0, 1.0), ratio=(1.0, 1.0)),
    transforms.ToTensor(),
])

control_image = image_transforms(Image.open("input_image.png").convert("RGB"))
controlnet_images = control_image.unsqueeze(0).unsqueeze(2).cuda()  # [B, C, F, H, W]

# Encode to latent space for latent condition mode
controlnet_images = rearrange(controlnet_images, "b c f h w -> (b f) c h w")
controlnet_images = vae.encode(controlnet_images * 2. - 1.).latent_dist.sample() * 0.18215
controlnet_images = rearrange(controlnet_images, "(b f) c h w -> b c f h w", f=1)
```

--------------------------------

### Launch Gradio App

Source: https://github.com/guoyww/animatediff/blob/main/README.md

Start the Gradio web interface for interactive usage, typically accessible at localhost:7860.

```bash
python -u app.py
```

--------------------------------

### Execute Training Commands

Source: https://github.com/guoyww/animatediff/blob/main/__assets__/docs/animatediff.md

Use torchrun to initiate the training process for either image layer finetuning or motion module training.

```bash
torchrun --nnodes=1 --nproc_per_node=1 train.py --config configs/training/v1/image_finetune.yaml
```

```bash
torchrun --nnodes=1 --nproc_per_node=1 train.py --config configs/training/v1/training.yaml
```

--------------------------------

### Single GPU Training Command

Source: https://context7.com/guoyww/animatediff/llms.txt

Command to initiate training on a single GPU using a specified configuration file.

```bash
torchrun --nnodes=1 --nproc_per_node=1 train.py     --config configs/training/v1/training.yaml
```

--------------------------------

### Train Motion Modules with Distributed Training

Source: https://context7.com/guoyww/animatediff/llms.txt

Launch the training script for motion modules using `torchrun` for distributed training. Includes options for configuration file and Weights & Biases integration.

```bash
# Train motion modules with distributed training
torchrun --nnodes=1 --nproc_per_node=4 train.py \
    --config configs/training/v1/training.yaml \
    --launcher pytorch \
    --wandb
```

--------------------------------

### Generate Animation with MotionLoRA Control

Source: https://github.com/guoyww/animatediff/blob/main/README.md

Run the sampling script with a MotionLoRA configuration.

```bash
python -m scripts.animate --config configs/prompts/2_motionlora/2_motionlora_RealisticVision.yaml
```

--------------------------------

### Load Pipeline with MotionLoRA for Camera Control

Source: https://context7.com/guoyww/animatediff/llms.txt

Use this function to load a pipeline and apply MotionLoRA weights for specific camera effects like zoom. Ensure the correct paths for motion modules and Dreambooth models are provided.

```python
from animatediff.utils.util import load_weights

# Load pipeline with MotionLoRA for camera control
pipeline = load_weights(
    pipeline,
    motion_module_path="models/Motion_Module/mm_sd_v15_v2.ckpt",
    dreambooth_model_path="models/DreamBooth_LoRA/realisticVisionV60B1_v51VAE.safetensors",
    # Apply MotionLoRA for zoom in effect
    motion_module_lora_configs=[
        {"path": "models/Motion_Module/v2_lora_ZoomIn.ckpt", "alpha": 0.8}
    ],
)

# Available MotionLoRA effects:
# - v2_lora_ZoomIn.ckpt     : Camera zooms in
# - v2_lora_ZoomOut.ckpt    : Camera zooms out
# - v2_lora_PanLeft.ckpt    : Camera pans left
# - v2_lora_PanRight.ckpt   : Camera pans right
# - v2_lora_TiltUp.ckpt     : Camera tilts up
# - v2_lora_TiltDown.ckpt   : Camera tilts down
# - v2_lora_RollingClockwise.ckpt     : Camera rolls clockwise
# - v2_lora_RollingAnticlockwise.ckpt : Camera rolls anti-clockwise
```

--------------------------------

### Run AnimateDiff with SparseCtrl Configuration

Source: https://context7.com/guoyww/animatediff/llms.txt

Execute the animation generation script using a specified configuration file. Supports both RGB image and sketch-based control.

```bash
# Generate with RGB image control
python -m scripts.animate --config configs/prompts/3_sparsectrl/3_2_sparsectrl_rgb_RealisticVision.yaml

# Generate with sketch/scribble control
python -m scripts.animate --config configs/prompts/3_sparsectrl/3_3_sparsectrl_sketch_RealisticVision.yaml
```

--------------------------------

### Configure Training Data Paths

Source: https://github.com/guoyww/animatediff/blob/main/__assets__/docs/animatediff.md

Update the YAML configuration files in the configs/training/ folder with the local paths to your dataset annotations and video files.

```yaml
train_data:
  csv_path: [Replace with .csv Annotation File Path]
  video_folder: [Replace with Video Folder Path]
  sample_size: 256
```

--------------------------------

### MotionLoRA Camera Control Configuration

Source: https://context7.com/guoyww/animatediff/llms.txt

This YAML configuration demonstrates how to apply multiple MotionLoRA effects simultaneously by specifying their paths and alpha values. It includes settings for Dreambooth path, inference configuration, and generation parameters.

```yaml
# configs/prompts/2_motionlora/motionlora_example.yaml

- dreambooth_path: "models/DreamBooth_LoRA/realisticVisionV60B1_v51VAE.safetensors"

  inference_config: "configs/inference/inference-v2.yaml"
  motion_module: "models/Motion_Module/mm_sd_v15_v2.ckpt"

  # Apply multiple MotionLoRA effects
  motion_module_lora_configs:
    - path: "models/Motion_Module/v2_lora_ZoomIn.ckpt"
      alpha: 0.8
    - path: "models/Motion_Module/v2_lora_PanLeft.ckpt"
      alpha: 0.5

  seed: 42
  steps: 25
  guidance_scale: 8

  prompt:
    - "A majestic lion in the savanna, golden hour lighting, cinematic"
  n_prompt:
    - "blurry, low quality"
```

--------------------------------

### Generate Animations with Community Models

Source: https://github.com/guoyww/animatediff/blob/main/README.md

Execute the sampling script using specific configuration files for various community models.

```bash
python -m scripts.animate --config configs/prompts/1_animate/1_1_animate_RealisticVision.yaml
python -m scripts.animate --config configs/prompts/1_animate/1_2_animate_FilmVelvia.yaml
python -m scripts.animate --config configs/prompts/1_animate/1_3_animate_ToonYou.yaml
python -m scripts.animate --config configs/prompts/1_animate/1_4_animate_MajicMix.yaml
python -m scripts.animate --config configs/prompts/1_animate/1_5_animate_RcnzCartoon.yaml
python -m scripts.animate --config configs/prompts/1_animate/1_6_animate_Lyriel.yaml
python -m scripts.animate --config configs/prompts/1_animate/1_7_animate_Tusun.yaml
```

--------------------------------

### Generate Animations via Command Line

Source: https://context7.com/guoyww/animatediff/llms.txt

Utilize the command-line script to generate animations using predefined configuration files. This allows for batch generation with different prompts and models, and customization of parameters like resolution and length.

```bash
# Generate animations with community Stable Diffusion models
python -m scripts.animate --config configs/prompts/1_animate/1_1_animate_RealisticVision.yaml

# Generate with different community models
python -m scripts.animate --config configs/prompts/1_animate/1_3_animate_ToonYou.yaml

# Custom parameters for resolution and length
python -m scripts.animate \
    --config configs/prompts/1_animate/1_1_animate_RealisticVision.yaml \
    --W 768 \
    --H 512 \
    --L 24

# Disable xformers for compatibility
python -m scripts.animate \
    --config configs/prompts/1_animate/1_1_animate_RealisticVision.yaml \
    --without-xformers
```

--------------------------------

### Initialize UNet3DConditionModel from 2D Weights

Source: https://context7.com/guoyww/animatediff/llms.txt

Load a 3D UNet model for video generation, initialized from a pretrained 2D Stable Diffusion model. Ensure the `unet_additional_kwargs` are correctly configured for motion module integration.

```python
from animatediff.models.unet import UNet3DConditionModel
from omegaconf import OmegaConf

# Load inference config with motion module settings
inference_config = OmegaConf.load("configs/inference/inference-v3.yaml")

# unet_additional_kwargs from config:
# {
#     "use_motion_module": true,
#     "motion_module_resolutions": [1, 2, 4, 8],
#     "motion_module_mid_block": false,
#     "motion_module_type": "Vanilla",
#     "motion_module_kwargs": {
#         "num_attention_heads": 8,
#         "num_transformer_block": 1,
#         "attention_block_types": ["Temporal_Self", "Temporal_Self"],
#         "temporal_position_encoding": true,
#         "temporal_attention_dim_div": 1,
#         "zero_initialize": true
#     }
# }

# Create 3D UNet from pretrained 2D Stable Diffusion
unet = UNet3DConditionModel.from_pretrained_2d(
    "runwayml/stable-diffusion-v1-5",
    subfolder="unet",
    unet_additional_kwargs=OmegaConf.to_container(inference_config.unet_additional_kwargs)
)

# Enable memory optimizations
unet.enable_xformers_memory_efficient_attention()
unet.enable_gradient_checkpointing()

# Forward pass expects 5D tensor: [batch, channels, frames, height, width]
# Input: sample shape [B, 4, 16, 64, 64] for 512x512 video with 16 frames
# Output: UNet3DConditionOutput with sample tensor
```

--------------------------------

### Generate Animation with SparseCtrl

Source: https://github.com/guoyww/animatediff/blob/main/README.md

Use SparseCtrl RGB or sketch configurations for enhanced animation control.

```bash
python -m scripts.animate --config configs/prompts/3_sparsectrl/3_1_sparsectrl_i2v.yaml
python -m scripts.animate --config configs/prompts/3_sparsectrl/3_2_sparsectrl_rgb_RealisticVision.yaml
python -m scripts.animate --config configs/prompts/3_sparsectrl/3_3_sparsectrl_sketch_RealisticVision.yaml
```

--------------------------------

### Load WebVid10M Dataset and DataLoader

Source: https://context7.com/guoyww/animatediff/llms.txt

Python code to instantiate the WebVid10M dataset and create a DataLoader for training motion modules. Ensure the CSV path and video folder are correctly set.

```python
from animatediff.data.dataset import WebVid10M
import torch

# Create dataset instance
dataset = WebVid10M(
    csv_path="path/to/webvid/results_2M_train.csv",
    video_folder="path/to/webvid/videos",
    sample_size=256,        # Output resolution
    sample_stride=4,        # Frame sampling stride
    sample_n_frames=16,     # Number of frames per clip
    is_image=False,         # Set True for image finetuning
)

# Create DataLoader
dataloader = torch.utils.data.DataLoader(
    dataset,
    batch_size=4,
    shuffle=True,
    num_workers=16,
    pin_memory=True,
    drop_last=True,
)

# Batch format:
# {
#     "pixel_values": tensor [B, F, C, H, W],  # Normalized to [-1, 1]
#     "text": list of caption strings
# }

for batch in dataloader:
    pixel_values = batch["pixel_values"]  # [4, 16, 3, 256, 256]
    captions = batch["text"]
    print(f"Shape: {pixel_values.shape}, Captions: {len(captions)}")
    break
```

--------------------------------

### Run AnimateDiff from Command Line

Source: https://context7.com/guoyww/animatediff/llms.txt

Execute batch generation for AnimateDiff using the command line interface. This script requires configuration files for models and generation parameters.

```bash
python -m scripts.animate --config configs/animate/default.yaml
```

--------------------------------

### Generate Animation from Image

Source: https://context7.com/guoyww/animatediff/llms.txt

Use the pipeline to generate an animation from a text prompt and control images. Specify parameters like inference steps, guidance scale, and video length.

```python
sample = pipeline(
    "A man watching fireworks, night city, cinematic",
    negative_prompt="worst quality, low quality",
    num_inference_steps=25,
    guidance_scale=8.5,
    width=512,
    height=512,
    video_length=16,
    controlnet_images=controlnet_images,
    controlnet_image_index=[0],  # Apply control to first frame
    controlnet_conditioning_scale=1.0,
).videos

save_videos_grid(sample, "image_to_video.gif", fps=8)
```

--------------------------------

### SparseCtrl Configuration for RGB Image Control

Source: https://context7.com/guoyww/animatediff/llms.txt

Configure SparseCtrl using a YAML file for RGB image-based animation control. This includes paths to models, inference settings, and control image configurations.

```yaml
# configs/prompts/3_sparsectrl/sparsectrl_rgb.yaml

- adapter_lora_scale: 1.0
  adapter_lora_path: "models/Motion_Module/v3_sd15_adapter.ckpt"
  dreambooth_path: "models/DreamBooth_LoRA/realisticVisionV60B1_v51VAE.safetensors"

  inference_config: "configs/inference/inference-v3.yaml"
  motion_module: "models/Motion_Module/v3_sd15_mm.ckpt"

  # SparseCtrl configuration
  controlnet_config: "configs/inference/sparsectrl/latent_condition.yaml"
  controlnet_path: "models/SparseCtrl/v3_sd15_sparsectrl_rgb.ckpt"

  seed: -1
  steps: 25
  guidance_scale: 8.5
  W: 512
  H: 512
  L: 16

  # Control image configuration
  controlnet_image_indexs: [0]  # Frame indices for control
  controlnet_images:
    - "path/to/control_image.png"

  prompt:
    - "A beautiful sunset over the ocean, golden hour, cinematic"
  n_prompt:
    - "worst quality, low quality, letterboxed"
```

--------------------------------

### Training Configuration for Motion Modules

Source: https://context7.com/guoyww/animatediff/llms.txt

YAML configuration file for training custom motion modules. Specifies dataset paths, model configurations, training parameters, and optimization settings.

```yaml
# configs/training/v1/training.yaml

image_finetune: false

output_dir: "outputs"
pretrained_model_path: "models/StableDiffusion/stable-diffusion-v1-5"

unet_additional_kwargs:
  use_motion_module: true
  motion_module_resolutions: [1, 2, 4, 8]
  motion_module_type: Vanilla
  motion_module_kwargs:
    num_attention_heads: 8
    num_transformer_block: 1
    attention_block_types: ["Temporal_Self", "Temporal_Self"]
    temporal_position_encoding: true
    temporal_position_encoding_max_len: 24
    temporal_attention_dim_div: 1
    zero_initialize: true

noise_scheduler_kwargs:
  num_train_timesteps: 1000
  beta_start: 0.00085
  beta_end: 0.012
  beta_schedule: "linear"

train_data:
  csv_path: "path/to/webvid/annotations.csv"
  video_folder: "path/to/webvid/videos"
  sample_size: 256
  sample_stride: 4
  sample_n_frames: 16

validation_data:
  prompts:
    - "Snow rocky mountains peaks canyon"
    - "Robot dancing in times square"
  num_inference_steps: 25
  guidance_scale: 8.0

trainable_modules:
  - "motion_modules."

learning_rate: 1.e-4
train_batch_size: 4
max_train_steps: 100000
checkpointing_steps: 5000
validation_steps: 5000

mixed_precision_training: true
enable_xformers_memory_efficient_attention: true
```

--------------------------------

### Initialize UNet3DConditionModel for 2D Pretrained Models

Source: https://context7.com/guoyww/animatediff/llms.txt

Use this method to initialize the UNet3DConditionModel when working with 2D pretrained models. Ensure the model path is correct.

```python
UNet3DConditionModel.from_pretrained_2d(model_path, subfolder="unet")
```

--------------------------------

### Load Motion Modules and Personalized Models

Source: https://context7.com/guoyww/animatediff/llms.txt

This function is used to load and combine motion modules with personalized models. Specify the pipeline and the path to the weights.

```python
pipeline.load_weights(motion_module_path, adapter_ids=adapter_ids)
```

--------------------------------

### Generate Animation with AnimationPipeline

Source: https://context7.com/guoyww/animatediff/llms.txt

Use the AnimationPipeline to generate animations from text prompts. This involves loading models, configuring the pipeline, and calling it with prompts and generation parameters. Ensure CUDA is available for GPU acceleration.

```python
import torch
from diffusers import AutoencoderKL, DDIMScheduler
from transformers import CLIPTextModel, CLIPTokenizer
from omegaconf import OmegaConf

from animatediff.models.unet import UNet3DConditionModel
from animatediff.pipelines.pipeline_animation import AnimationPipeline
from animatediff.utils.util import save_videos_grid, load_weights

# Load configuration for motion module version
inference_config = OmegaConf.load("configs/inference/inference-v3.yaml")

# Initialize model components from pretrained Stable Diffusion
pretrained_model_path = "runwayml/stable-diffusion-v1-5"

tokenizer = CLIPTokenizer.from_pretrained(pretrained_model_path, subfolder="tokenizer")
text_encoder = CLIPTextModel.from_pretrained(pretrained_model_path, subfolder="text_encoder").cuda()
vae = AutoencoderKL.from_pretrained(pretrained_model_path, subfolder="vae").cuda()

# Create 3D UNet from 2D pretrained weights with motion module configuration
unet = UNet3DConditionModel.from_pretrained_2d(
    pretrained_model_path,
    subfolder="unet",
    unet_additional_kwargs=OmegaConf.to_container(inference_config.unet_additional_kwargs)
).cuda()

# Enable memory-efficient attention if available
from diffusers.utils.import_utils import is_xformers_available
if is_xformers_available():
    unet.enable_xformers_memory_efficient_attention()

# Create the animation pipeline
pipeline = AnimationPipeline(
    vae=vae,
    text_encoder=text_encoder,
    tokenizer=tokenizer,
    unet=unet,
    scheduler=DDIMScheduler(**OmegaConf.to_container(inference_config.noise_scheduler_kwargs)),
).to("cuda")

# Load motion module and optional DreamBooth model
pipeline = load_weights(
    pipeline,
    motion_module_path="models/Motion_Module/v3_sd15_mm.ckpt",
    dreambooth_model_path="models/DreamBooth_LoRA/realisticVisionV60B1_v51VAE.safetensors",
)

# Generate animation
torch.manual_seed(8893659352891878017)

prompt = "b&w photo of 42 y.o man in black clothes, bald, face, half body, coastline, overcast weather, wind, waves, 8k uhd, dslr, soft lighting, high quality, film grain"
negative_prompt = "semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime, worst quality, low quality"

sample = pipeline(
    prompt,
    negative_prompt=negative_prompt,
    num_inference_steps=25,
    guidance_scale=8.0,
    width=512,
    height=512,
    video_length=16,
).videos

# Save as GIF
save_videos_grid(sample, "output_animation.gif", fps=8)
# Output: Generates 16-frame 512x512 animation saved as GIF
```

--------------------------------

### Load Weights Utility

Source: https://context7.com/guoyww/animatediff/llms.txt

Python code for the load_weights utility function, which comprehensively loads various model weights including motion modules, DreamBooth models, LoRA, and adapters into a pipeline. Weights are automatically downloaded if not found locally.

```python
from animatediff.utils.util import load_weights

# Load all model weights into pipeline
pipeline = load_weights(
    animation_pipeline,

    # Motion module (required for animation)
    motion_module_path="models/Motion_Module/v3_sd15_mm.ckpt",

    # Optional: MotionLoRA for camera control
    motion_module_lora_configs=[
        {"path": "models/Motion_Module/v2_lora_ZoomIn.ckpt", "alpha": 0.8},
        {"path": "models/Motion_Module/v2_lora_PanLeft.ckpt", "alpha": 0.5},
    ],

    # Optional: Domain adapter for v3 models
    adapter_lora_path="models/Motion_Module/v3_sd15_adapter.ckpt",
    adapter_lora_scale=1.0,

    # Optional: Custom DreamBooth model
    dreambooth_model_path="models/DreamBooth_LoRA/realisticVisionV60B1_v51VAE.safetensors",

    # Optional: Additional LoRA
    lora_model_path="models/DreamBooth_LoRA/custom_lora.safetensors",
    lora_alpha=0.8,
)

# Weights are automatically downloaded from HuggingFace if not present locally
# Repository: guoyww/animatediff (motion modules)
# Repository: guoyww/animatediff_t2i_backups (DreamBooth models)
```

--------------------------------

### AnimateController API for Custom Integrations

Source: https://context7.com/guoyww/animatediff/llms.txt

Python code demonstrating the use of the AnimateController API for updating pipeline models and generating animations. This is useful for custom Gradio integrations.

```python
# AnimateController API for custom Gradio integrations
from app import AnimateController

controller = AnimateController()

# Update pipeline with new models
controller.update_pipeline(
    stable_diffusion_dropdown="runwayml/stable-diffusion-v1-5",
    motion_module_dropdown="v3_sd15_mm.ckpt",
    base_model_dropdown="realisticVisionV60B1_v51VAE.safetensors",
    lora_model_dropdown="none",
    lora_alpha_dropdown="0.6",
    sampler_dropdown="DDIM",
)

# Generate animation
result = controller.animate(
    prompt_textbox="A beautiful landscape with mountains",
    negative_prompt_textbox="blurry, low quality",
    sampler_dropdown="DDIM",
    sample_step_slider=25,
    width_slider=512,
    length_slider=16,
    height_slider=512,
    cfg_scale_slider=8.0,
    seed_textbox="42",
)
# Returns: path to generated MP4 file
```

--------------------------------

### Save Videos Grid Utility

Source: https://context7.com/guoyww/animatediff/llms.txt

Python code for the save_videos_grid utility function, used to save video tensors as GIF or MP4 files, with options for grid layout and frame rate. Supports rescaling.

```python
from animatediff.utils.util import save_videos_grid
import torch

# Single video tensor: [1, C, T, H, W]
# - C: channels (3 for RGB)
# - T: number of frames
# - H, W: height and width

video = torch.randn(1, 3, 16, 512, 512)  # Normalized to [0, 1]

# Save as GIF
save_videos_grid(video, "output.gif", fps=8)

# Save multiple videos in grid layout
batch_videos = torch.randn(4, 3, 16, 512, 512)
save_videos_grid(batch_videos, "batch_output.gif", n_rows=2, fps=8)

# Rescale from [-1, 1] to [0, 1] if needed
save_videos_grid(video, "output.gif", rescale=True, fps=8)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.