### Install Example Requirements

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/training/cogvideox.md

Navigate to the specific example directory (e.g., examples/cogvideo) and install its requirements. This is necessary before running training scripts.

```bash
cd examples/cogvideo
pip install -r requirements.txt
```

--------------------------------

### Install Example-Specific Requirements

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/README.md

After installing the library from source, navigate to the desired example folder and install its specific requirements. This ensures all necessary dependencies for that example are met.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install Example Script Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/training/lora.md

Navigate to the specific example directory (e.g., text_to_image) and install its required dependencies. This ensures all necessary packages are available for the training script.

```bash
cd examples/text_to_image
pip install -r requirements.txt
```

--------------------------------

### Install Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/server/README.md

Install the necessary libraries for the server example. Ensure you have diffusers and the requirements.txt file.

```bash
pip install diffusers
pip install -r requirements.txt
```

--------------------------------

### Install Script Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/training/lcm_distill.md

Navigate to the consistency distillation examples directory and install the required dependencies for the training script.

```bash
cd examples/consistency_distillation
pip install -r requirements.txt
```

--------------------------------

### Install Kandinsky Training Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/training/kandinsky.md

Navigate to the Kandinsky example directory and install its requirements.

```bash
cd examples/kandinsky2_2/text_to_image
pip install -r requirements.txt
```

--------------------------------

### Install SDXL Training Script Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/training/sdxl.md

Navigate to the text-to-image examples directory and install the specific requirements for the SDXL training script.

```bash
cd examples/text_to_image
pip install -r requirements_sdxl.txt
```

--------------------------------

### Install Wuerstchen Training Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/training/wuerstchen.md

Navigate to the Wuerstchen example folder and install the necessary dependencies for the text-to-image training script.

```bash
cd examples/wuerstchen/text_to_image
pip install -r requirements.txt
```

--------------------------------

### Install Training Script Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/training/unconditional_training.md

Navigate to the example folder and install the necessary dependencies for the unconditional image generation training script.

```bash
cd examples/unconditional_image_generation
pip install -r requirements.txt
```

--------------------------------

### Install Training Script Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/training/t2i_adapters.md

Navigate to the T2I-Adapter example directory and install the necessary Python packages for the training script.

```bash
cd examples/t2i_adapter
pip install -r requirements.txt
```

--------------------------------

### Install Training Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/CONTRIBUTING.md

After cloning the repository, navigate to the diffusers directory and install the specific dependencies required for a training example using pip.

```bash
cd diffusers
pip install -r examples/<your-example-folder>/requirements.txt
```

--------------------------------

### Install Tensorboard Profiling Tools

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/controlnet/README.md

Install necessary packages for Tensorboard profiling and start Tensorboard to view profile traces.

```bash
pip install tensorflow tensorboard-plugin-profile
tensorboard --logdir runs/fill-circle-100steps-20230411_165612/
```

--------------------------------

### Install Textual Inversion Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/training/text_inversion.md

Navigate to the textual inversion example directory and install its specific requirements. This ensures all necessary libraries are available for the script.

```bash
cd examples/textual_inversion
pip install -r requirements.txt
```

--------------------------------

### Download Example Dataset

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/training/dreambooth.md

Downloads an example dog dataset from Hugging Face Hub to a local directory. Ensure you have the `huggingface_hub` library installed.

```python
from huggingface_hub import snapshot_download

local_dir = "./dog"
snapshot_download(
    "diffusers/dog-example",
    local_dir=local_dir,
    repo_type="dataset",
    ignore_patterns=".gitattributes",
)
```

--------------------------------

### Install Diffusers and Training Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/dreambooth/README_hidream.md

Clone the diffusers repository, install the library from source, and then install specific requirements for the HiDream training example.

```bash
git clone https://github.com/huggingface/diffusers
cd diffusers
pip install -e .
pip install -r requirements_hidream.txt
```

--------------------------------

### Install Diffusers and Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/custom_diffusion/README.md

Clone the diffusers repository, install the library from source, and then install example-specific requirements. This ensures you have the latest versions for running example scripts.

```bash
git clone https://github.com/huggingface/diffusers
cd diffusers
pip install -e .
pip install -r requirements.txt
pip install clip-retrieval
```

--------------------------------

### Install Optimum Intel with OpenVINO support

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/optimization/open_vino.md

Install the Optimum Intel library with the necessary dependencies for OpenVINO. Ensure you use the eager upgrade strategy to get the latest versions.

```bash
pip install --upgrade-strategy eager optimum["openvino"]
```

--------------------------------

### Install Diffusers and Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/consistency_distillation/README.md

Clone the diffusers repository, install it from source, and then install example-specific requirements. This ensures compatibility with the latest example scripts.

```bash
git clone https://github.com/huggingface/diffusers
cd diffusers
pip install -e .
pip install -r requirements.txt
```

--------------------------------

### Install Flax Training Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/dreambooth/README.md

Installs the necessary dependencies for running the Flax-based training example.

```bash
pip install -U -r requirements_flax.txt
```

--------------------------------

### Multi-GPU Training with InstructPix2Pix

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/instruct_pix2pix/README.md

Example command for distributed training using `accelerate` with multiple GPUs. Follow `accelerate` documentation for setup.

```bash
accelerate launch --mixed_precision="fp16" --multi_gpu train_instruct_pix2pix.py \
 --pretrained_model_name_or_path=stable-diffusion-v1-5/stable-diffusion-v1-5 \
 --dataset_name=sayakpaul/instructpix2pix-1000-samples \
 --use_ema \
 --enable_xformers_memory_efficient_attention \
 --resolution=512 --random_flip \
 --train_batch_size=4 --gradient_accumulation_steps=4 --gradient_checkpointing \
 --max_train_steps=15000 \
 --checkpointing_steps=5000 --checkpoints_total_limit=1 \
 --learning_rate=5e-05 --lr_warmup_steps=0 \
 --conditioning_dropout_prob=0.05 \
 --mixed_precision=fp16 \
 --seed=42 \
 --push_to_hub
```

--------------------------------

### Install Optimum-Quanto and Accelerate

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/quantization/quanto.md

Install the necessary libraries for using the Quanto backend with Diffusers. Ensure you have `optimum-quanto` and `accelerate` installed.

```shell
pip install optimum-quanto accelerate
```

--------------------------------

### Install FLUX Specific Requirements

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/controlnet/README_flux.md

Install the requirements specific to the FLUX ControlNet example. Navigate to the examples/controlnet directory before running this command.

```bash
pip install -r requirements_flux.txt
```

--------------------------------

### Multi-GPU Training with Accelerate

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/kandinsky2_2/text_to_image/README.md

Example command for launching text-to-image decoder training across multiple GPUs using the 'accelerate' library. Ensure 'accelerate' is installed and configured.

```bash
export DATASET_NAME="lambdalabs/naruto-blip-captions"

accelerate launch --mixed_precision="fp16" --multi_gpu  train_text_to_image_decoder.py \
  --dataset_name=$DATASET_NAME \
  --resolution=768 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --gradient_checkpointing \
  --max_train_steps=15000 \
  --learning_rate=1e-05 \
  --max_grad_norm=1 \
  --checkpoints_total_limit=3 \
  --lr_scheduler="constant" --lr_warmup_steps=0 \
  --validation_prompts="A robot naruto, 4k photo" \
  --report_to="wandb" \
  --push_to_hub \
  --output_dir="kandi2-decoder-naruto-model"
```

--------------------------------

### Install Diffusers from Source and Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/research_projects/multi_token_textual_inversion/README.md

Clone the diffusers repository, install it from source, and then install example-specific requirements. Initialize an Accelerate environment.

```bash
git clone https://github.com/huggingface/diffusers
cd diffusers
pip install .
cd examples/research_projects/multi_token_textual_inversion/
pip install -r requirements.txt
accelerate config
```

--------------------------------

### Install Necessary Libraries

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/using-diffusers/sdxl.md

Installs the required libraries for using SDXL. Uncomment the lines to run the installation in a Colab environment.

```python
# uncomment to install the necessary libraries in Colab
#!pip install -q diffusers transformers accelerate invisible-watermark>=0.2.0
```

--------------------------------

### Install Accelerate for Source Installation

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/installation.md

Install the Accelerate library, a prerequisite for installing Diffusers from source.

```bash
uv pip install accelerate
```

--------------------------------

### Install necessary libraries

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/quantization/bitsandbytes.md

Ensure you have the required libraries installed for quantization. This command installs diffusers, transformers, accelerate, and bitsandbytes.

```bash
pip install diffusers transformers accelerate bitsandbytes -U
```

--------------------------------

### Install Diffusers and Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/research_projects/instructpix2pix_lora/README.md

Clone the diffusers repository, install it from source, and then install the example-specific requirements. Ensure PEFT is installed for LoRA training.

```bash
git clone https://github.com/huggingface/diffusers
cd diffusers
pip install .
pip install -r requirements.txt
accelerate config
```

--------------------------------

### Install bitsandbytes

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/api/pipelines/stable_diffusion/stable_diffusion_3.md

Install the bitsandbytes library for quantization.

```shell
pip install bitsandbytes
```

--------------------------------

### Install Dreambooth Requirements

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/dreambooth/README_lumina2.md

Install the specific requirements for the DreamBooth example scripts after installing the main diffusers library.

```bash
pip install -r requirements_sana.txt
```

--------------------------------

### Install Server Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/using-diffusers/create_a_server.md

Navigate to the examples/server folder and install the necessary dependencies using pip.

```bash
pip install .
pip install -f requirements.txt
```

--------------------------------

### Load AnimateDiffControlNetPipeline with ControlNet

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/api/pipelines/animatediff.md

Initializes an AnimateDiff pipeline integrated with ControlNet. This setup requires loading a ControlNet model (e.g., depth estimation) and potentially a motion adapter. Ensure `controlnet_aux` is installed for preprocessing. The example uses a ControlNet model loaded from a local file.

```python
import torch
from diffusers import AnimateDiffControlNetPipeline, AutoencoderKL, ControlNetModel, MotionAdapter, LCMScheduler
from diffusers.utils import export_to_gif, load_video

# Additionally, you will need a preprocess videos before they can be used with the ControlNet
# HF maintains just the right package for it: `pip install controlnet_aux`
from controlnet_aux.processor import ZoeDetector

# Download controlnets from https://huggingface.co/lllyasviel/ControlNet-v1-1 to use .from_single_file
# Download Diffusers-format controlnets, such as https://huggingface.co/lllyasviel/sd-controlnet-depth, to use .from_pretrained()
controlnet = ControlNetModel.from_single_file("control_v11f1p_sd15_depth.pth", torch_dtype=torch.float16)
```

--------------------------------

### Install InstructPix2Pix Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/training/instructpix2pix.md

Navigate to the examples/instruct_pix2pix directory and install the necessary dependencies for the training script.

```bash
cd examples/instruct_pix2pix
pip install -r requirements.txt
```

--------------------------------

### Install Diffusers and Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/dreambooth/README_sd3.md

Clone the diffusers repository, install it from source, and then install the specific requirements for the DreamBooth SD3 examples. This ensures compatibility with the latest scripts.

```bash
git clone https://github.com/huggingface/diffusers
cd diffusers
pip install -e .
cd examples/dreambooth
pip install -r requirements_sd3.txt
```

--------------------------------

### Install ControlNet Training Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/training/controlnet.md

Navigate to the examples/controlnet directory and install the required dependencies for the ControlNet training script.

```bash
cd examples/controlnet
pip install -r requirements.txt
```

--------------------------------

### Loading and Running a Text-to-Video Pipeline

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/using-diffusers/text-img2vid.md

This example demonstrates loading a pre-trained text-to-video pipeline, setting up prompts, and generating video frames. It includes loading specific components like VAE and text encoder, and exporting the output to a video file.

```python
pipeline = WanPipeline.from_pretrained(
    "Wan-AI/Wan2.1-T2V-14B-Diffusers",
    vae=vae,
    transformer=transformer,
    text_encoder=text_encoder,
    torch_dtype=torch.bfloat16
)
pipeline.to("cuda")

prompt = """
The camera rushes from far to near in a low-angle shot, 
revealing a white ferret on a log. It plays, leaps into the water, and emerges, as the camera zooms in 
for a close-up. Water splashes berry bushes nearby, while moss, snow, and leaves blanket the ground. 
Birch trees and a light blue sky frame the scene, with ferns in the foreground. Side lighting casts dynamic 
shadows and warm highlights. Medium composition, front view, low angle, with depth of field.
"""
negative_prompt = """
Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality, 
low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, 
misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards
"""

output = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_frames=81,
    guidance_scale=5.0,
).frames[0]
export_to_video(output, "output.mp4", fps=16)
```

--------------------------------

### Install Custom Diffusion Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/training/custom_diffusion.md

Navigate to the custom_diffusion example directory and install the necessary Python packages, including clip-retrieval.

```bash
cd examples/custom_diffusion
pip install -r requirements.txt
pip install clip-retrieval
```

--------------------------------

### Launching Training with Accelerate

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/tutorials/basic_training.md

Use `notebook_launcher` to start the training process. Pass the training loop function, its arguments, and the desired number of processes.

```python
from accelerate import notebook_launcher

args = (config, model, noise_scheduler, optimizer, train_dataloader, lr_scheduler)

notebook_launcher(train_loop, args, num_processes=1)
```

--------------------------------

### Run RealFill Training (Toy Example)

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/research_projects/realfill/README.md

Launch the RealFill training script with specified model, data directory, and output directory. This example uses default settings for a toy run.

```bash
export MODEL_NAME="stabilityai/stable-diffusion-2-inpainting"
export TRAIN_DIR="data/flowerwoman"
export OUTPUT_DIR="flowerwoman-model"

accelerate launch train_realfill.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --train_data_dir=$TRAIN_DIR \
  --output_dir=$OUTPUT_DIR \
  --resolution=512 \
  --train_batch_size=16 \
  --gradient_accumulation_steps=1 \
  --unet_learning_rate=2e-4 \
  --text_encoder_learning_rate=4e-5 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=100 \
  --max_train_steps=2000 \
  --lora_rank=8 \
  --lora_dropout=0.1 \
  --lora_alpha=16 \

```

--------------------------------

### Install Diffusers and Requirements

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/text_to_image/README_sdxl.md

Installs the diffusers library from source and specific requirements for SDXL text-to-image examples. Ensure you are in a virtual environment.

```bash
git clone https://github.com/huggingface/diffusers
cd diffusers
pip install -e .
cd examples/text_to_image
pip install -r requirements_sdxl.txt
```

--------------------------------

### Start the Async Server

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/server-async/README.md

Run the server using the provided Python script. The server will start on http://localhost:8500 by default.

```bash
python serverasync.py
```

--------------------------------

### Install IP-Adapter

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/using-diffusers/ip_adapter.md

Install the necessary libraries for IP-Adapter. This includes diffusers, transformers, and accelerate.

```bash
pip install diffusers transformers accelerate
```

--------------------------------

### Install controlnet_aux Library

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/cogview4-control/README.md

This command installs the `controlnet_aux` library, which is used for computing pose images for the dataset. It's a prerequisite for the inference part of the example.

```bash
pip install controlnet_aux
```

--------------------------------

### Initialize and Configure Pipeline with IP-Adapter

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/modular_diffusers/quickstart.md

Initializes a pipeline, loads components, configures the IP-Adapter with specific weights and scale, and moves the pipeline to the GPU. This sets up the pipeline for inference with IP-Adapter.

```python
dd_pipeline = dd_blocks.init_pipeline("YiYiXu/modular-demo-auto", collection="diffdiff")
dd_pipeline.load_components(torch_dtype=torch.float16)
dd_pipeline.loader.load_ip_adapter("h94/IP-Adapter", subfolder="sdxl_models", weight_name="ip-adapter_sdxl.bin")
dd_pipeline.loader.set_ip_adapter_scale(0.6)
dd_pipeline = dd_pipeline.to(device)
```

--------------------------------

### Install optimum-quanto

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/api/pipelines/flux.md

Installs the optimum-quanto library, which is recommended for running FP8 inference.

```shell
pip install optimum-quanto
```

--------------------------------

### Install RealFill Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/research_projects/realfill/README.md

Navigate to the realfill directory and install the required packages using the provided requirements file.

```bash
cd realfill
pip install -r requirements.txt
```

--------------------------------

### Environment Setup and Dependencies

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/community/README.md

Clone the diffusers repository and install necessary dependencies for adaptive mask inpainting. This includes PyTorch, detectron2, and diffusers itself.

```Shell
git clone https://github.com/huggingface/diffusers.git
cd diffusers

conda create --name ami python=3.9 -y
conda activate ami

conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge -y
python -m pip install detectron2==0.6 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu113/torch1.10/index.html
pip install easydict
pip install diffusers==0.20.2 accelerate safetensors transformers
pip install setuptools==59.5.0
pip install opencv-python
pip install numpy==1.24.1
```

--------------------------------

### Install Benchmark Requirements

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/benchmarks/README.md

Navigate to the benchmarks directory and install the necessary requirements for running benchmarks.

```sh
cd benchmarks/
pip install -r requirements.txt
```

--------------------------------

### Load Framepack Pipeline with Start and End Image Controls

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/api/pipelines/framepack.md

Loads the HunyuanVideoFramepackPipeline with specific transformer, feature extractor, and image encoder components. This setup is used for generating videos with start and end image controls, utilizing the inverted anti-drifting sampling model.

```python
import torch
from diffusers import HunyuanVideoFramepackPipeline, HunyuanVideoFramepackTransformer3DModel
from diffusers.utils import export_to_video, load_image
from transformers import SiglipImageProcessor, SiglipVisionModel

transformer = HunyuanVideoFramepackTransformer3DModel.from_pretrained(
    "lllyasviel/FramePackI2V_HY", torch_dtype=torch.bfloat16
)
feature_extractor = SiglipImageProcessor.from_pretrained(
    "lllyasviel/flux_redux_bfl", subfolder="feature_extractor"
)
image_encoder = SiglipVisionModel.from_pretrained(
    "lllyasviel/flux_redux_bfl", subfolder="image_encoder", torch_dtype=torch.float16
)
pipe = HunyuanVideoFramepackPipeline.from_pretrained(
    "hunyuanvideo-community/HunyuanVideo",
    transformer=transformer,
    feature_extractor=feature_extractor,
    image_encoder=image_encoder,
    torch_dtype=torch.float16,
)
```

--------------------------------

### Launch InstructPix2Pix SDXL Training

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/instruct_pix2pix/README_sdxl.md

Initiate the training process for InstructPix2Pix with SDXL using the 'accelerate launch' command. This example uses a toy dataset and basic training configurations.

```bash
accelerate launch train_instruct_pix2pix_sdxl.py \
    --pretrained_model_name_or_path=$MODEL_NAME \
    --dataset_name=$DATASET_ID \
    --enable_xformers_memory_efficient_attention \
    --resolution=256 --random_flip \
    --train_batch_size=4 --gradient_accumulation_steps=4 --gradient_checkpointing \
    --max_train_steps=15000 \
    --checkpointing_steps=5000 --checkpoints_total_limit=1 \
    --learning_rate=5e-05 --max_grad_norm=1 --lr_warmup_steps=0 \
    --conditioning_dropout_prob=0.05 \
    --seed=42 \
    --push_to_hub
```

--------------------------------

### InstantID Pipeline Example

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/community/README.md

Shows how to set up and use the StableDiffusionXLInstantIDPipeline for ID-preserving generation. Requires installing several libraries and downloading specific model checkpoints.

```Python
# !pip install diffusers opencv-python transformers accelerate insightface
import diffusers
from diffusers.utils import load_image
from diffusers import ControlNetModel

import cv2
import torch
import numpy as np
from PIL import Image

from insightface.app import FaceAnalysis
from pipeline_stable_diffusion_xl_instantid import StableDiffusionXLInstantIDPipeline, draw_kps

# prepare 'antelopev2' under ./models
# https://github.com/deepinsight/insightface/issues/1896#issuecomment-1023867304
app = FaceAnalysis(name='antelopev2', root='./', providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
app.prepare(ctx_id=0, det_size=(640, 640))

# prepare models under ./checkpoints
# https://huggingface.co/InstantX/InstantID
from huggingface_hub import hf_hub_download

hf_hub_download(repo_id="InstantX/InstantID", filename="ControlNetModel/config.json", local_dir="./checkpoints")
hf_hub_download(repo_id="InstantX/InstantID", filename="ControlNetModel/diffusion_pytorch_model.safetensors", local_dir="./checkpoints")
hf_hub_download(repo_id="InstantX/InstantID", filename="ip-adapter.bin", local_dir="./checkpoints")

face_adapter = './checkpoints/ip-adapter.bin'
controlnet_path = './checkpoints/ControlNetModel'

# load IdentityNet
controlnet = ControlNetModel.from_pretrained(controlnet_path, torch_dtype=torch.float16)

base_model = 'wangqixun/YamerMIX_v8'
pipe = StableDiffusionXLInstantIDPipeline.from_pretrained(
    base_model,
    controlnet=controlnet,
    torch_dtype=torch.float16
)
pipe.to("cuda")

# load adapter
pipe.load_ip_adapter_instantid(face_adapter)

# load an image
face_image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/ai_face2.png")

# prepare face emb
face_info = app.get(cv2.cvtColor(np.array(face_image), cv2.COLOR_RGB2BGR))
face_info = sorted(face_info, key=lambda x:(x['bbox'][2]-x['bbox'][0])*x['bbox'][3]-x['bbox'][1])[-1]  # only use the maximum face
face_emb = face_info['embedding']
face_kps = draw_kps(face_image, face_info['kps'])

# prompt
prompt = "film noir style, ink sketch|vector, male man, highly detailed, sharp focus, ultra sharpness, monochrome, high contrast, dramatic shadows, 1940s style, mysterious, cinematic"
negative_prompt = "ugly, deformed, noisy, blurry, low contrast, realism, photorealistic, vibrant, colorful"

# generate image
pipe.set_ip_adapter_scale(0.8)
image = pipe(
    prompt,
    image_embeds=face_emb,
    image=face_kps,
    controlnet_conditioning_scale=0.8,
).images[0]
```

--------------------------------

### Launch InstructPix2Pix SDXL Training with Validation

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/instruct_pix2pix/README_sdxl.md

Launch training with validation inference enabled via Weights and Biases. Requires 'wandb' to be installed.

```bash
accelerate launch train_instruct_pix2pix_sdxl.py \
    --pretrained_model_name_or_path=stabilityai/stable-diffusion-xl-base-1.0 \
    --dataset_name=$DATASET_ID \
    --use_ema \
    --enable_xformers_memory_efficient_attention \
    --resolution=512 --random_flip \
    --train_batch_size=4 --gradient_accumulation_steps=4 --gradient_checkpointing \
    --max_train_steps=15000 \
    --checkpointing_steps=5000 --checkpoints_total_limit=1 \
    --learning_rate=5e-05 --lr_warmup_steps=0 \
    --conditioning_dropout_prob=0.05 \
    --seed=42 \
    --val_image_url_or_path="https://datasets-server.huggingface.co/assets/fusing/instructpix2pix-1000-samples/--/fusing--instructpix2pix-1000-samples/train/23/input_image/image.jpg" \
    --validation_prompt="make it in japan" \
    --report_to=wandb \
    --push_to_hub
```

--------------------------------

### Generate Image from Text with Stable Diffusion

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/README.md

Quickstart example to generate an image from a text prompt using a pre-trained Stable Diffusion pipeline. Requires PyTorch and CUDA.

```python
from diffusers import DiffusionPipeline
import torch

pipeline = DiffusionPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16)
pipeline.to("cuda")
pipeline("An image of a squirrel in Picasso style").images[0]
```

--------------------------------

### Launch SDXL Training Script

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/training/sdxl.md

Set environment variables for the model, VAE, and dataset, then launch the training script with various optimization and reporting parameters. Use `--report_to=wandb` for monitoring and `--validation_prompt` for tracking results.

```bash
export MODEL_NAME="stabilityai/stable-diffusion-xl-base-1.0"
export VAE_NAME="madebyollin/sdxl-vae-fp16-fix"
export DATASET_NAME="lambdalabs/naruto-blip-captions"

accelerate launch train_text_to_image_sdxl.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --pretrained_vae_model_name_or_path=$VAE_NAME \
  --dataset_name=$DATASET_NAME \
  --enable_xformers_memory_efficient_attention \
  --resolution=512 \
  --center_crop \
  --random_flip \
  --proportion_empty_prompts=0.2 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --gradient_checkpointing \
  --max_train_steps=10000 \
  --use_8bit_adam \
  --learning_rate=1e-06 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --mixed_precision="fp16" \
  --report_to="wandb" \
  --validation_prompt="a cute Sundar Pichai creature" \
  --validation_epochs 5 \
  --checkpointing_steps=5000 \
  --output_dir="sdxl-naruto-model" \
  --push_to_hub
```

--------------------------------

### Multi-GPU Training with Mixed Precision

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/unconditional_image_generation/README.md

Example command for training with multiple GPUs using mixed precision (fp16) and logging with Weights and Biases (wandb). Requires `pip install wandb`.

```bash
accelerate launch --mixed_precision="fp16" --multi_gpu train_unconditional.py \
  --dataset_name="huggan/pokemon" \
  --resolution=64 --center_crop --random_flip \
  --output_dir="ddpm-ema-pokemon-64" \
  --train_batch_size=16 \
  --num_epochs=100 \
  --gradient_accumulation_steps=1 \
  --use_ema \
  --learning_rate=1e-4 \
  --lr_warmup_steps=500 \
  --mixed_precision="fp16" \
  --logger="wandb"
```

--------------------------------

### Chroma Inference

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/api/pipelines/chroma.md

Run inference with a pre-trained Chroma model. Ensure you have the `diffusers` and `torch` libraries installed. This example loads the 'lodestones/Chroma1-HD' model and generates an image based on a detailed prompt.

```python
import torch
from diffusers import ChromaPipeline

pipe = ChromaPipeline.from_pretrained("lodestones/Chroma1-HD", torch_dtype=torch.bfloat16)
pipe.enable_model_cpu_offload()

prompt = [
    "A high-fashion close-up portrait of a blonde woman in clear sunglasses. The image uses a bold teal and red color split for dramatic lighting. The background is a simple teal-green. The photo is sharp and well-composed, and is designed for viewing with anaglyph 3D glasses for optimal effect. It looks professionally done."
]
negative_prompt =  ["low quality, ugly, unfinished, out of focus, deformed, disfigure, blurry, smudged, restricted palette, flat colors"]

image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    generator=torch.Generator("cpu").manual_seed(433),
    num_inference_steps=40,
    guidance_scale=3.0,
    num_images_per_prompt=1,
).images[0]
image.save("chroma.png")
```

--------------------------------

### Image-to-Image Generation Setup

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/using-diffusers/sdxl.md

Imports necessary classes for image-to-image generation with SDXL. This snippet sets up the pipeline for conditioning an image with a text prompt.

```python
from diffusers import AutoPipelineForImage2Image
from diffusers.utils import load_image, make_image_grid
```

--------------------------------

### Launch IP Adapter Training with Accelerate

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/research_projects/ip_adapter/README.md

Launches the IP Adapter training script with specified parameters for mixed precision and model configuration.

```bash
accelerate launch --mixed_precision "fp16" \
 tutorial_train_ip-adapter.py \
 --pretrained_model_name_or_path="stable-diffusion-v1-5/stable-diffusion-v1-5/" \
 --image_encoder_path="{image_encoder_path}" \
 --data_json_file="{data.json}" \
 --data_root_path="{image_path}" \
 --mixed_precision="fp16" \
 --resolution=512 \
 --train_batch_size=8 \
 --dataloader_num_workers=4 \
 --learning_rate=1e-04 \
 --weight_decay=0.01 \
 --output_dir="{output_dir}" \
 --save_steps=10000
```

--------------------------------

### Define Prompt and Bounding Boxes

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/examples/research_projects/gligen/demo.ipynb

Defines a text prompt and a list of bounding boxes with corresponding object descriptions for conditional image generation. This setup is used to guide the GLIGEN pipeline.

```python
import numpy as np


# prompt = 'A realistic image of landscape scene depicting a green car parking on the left of a blue truck, with a red air balloon and a bird in the sky'
# gen_boxes = [('a green car', [21, 281, 211, 159]), ('a blue truck', [269, 283, 209, 160]), ('a red air balloon', [66, 8, 145, 135]), ('a bird', [296, 42, 143, 100])]

# prompt = 'A realistic top-down view of a wooden table with two apples on it'
# gen_boxes = [('a wooden table', [20, 148, 472, 216]), ('an apple', [150, 226, 100, 100]), ('an apple', [280, 226, 100, 100])]

# prompt = 'A realistic scene of three skiers standing in a line on the snow near a palm tree'
```

--------------------------------

### Download Demo Video for InstructPix2Pix

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/api/pipelines/text_to_video_zero.md

Downloads a demo video file from Hugging Face Hub for use with InstructPix2Pix. Ensure you have `huggingface_hub` installed.

```python
from huggingface_hub import hf_hub_download

filename = "__assets__/pix2pix video/camel.mp4"
repo_id = "PAIR/Text2Video-Zero"
video_path = hf_hub_download(repo_type="space", repo_id=repo_id, filename=filename)
```

--------------------------------

### Inpainting with Negative Prompts

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/using-diffusers/inpaint.md

Illustrates the setup for using negative prompts in inpainting. Negative prompts guide the model away from generating unwanted elements, improving overall image quality.

```python
import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid

pipeline = AutoPipelineForInpainting.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-inpainting", torch_dtype=torch.float16, variant="fp16"
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()


```

--------------------------------

### Initializing Model with Context Parallelism

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/training/distributed_inference.md

This example demonstrates initializing a transformer model with Ring Attention enabled via `ContextParallelConfig` during model loading. It's useful for setting up distributed inference from the start.

```python
CKPT_ID = "black-forest-labs/FLUX.1-dev"

cp_config = ContextParallelConfig(ring_degree=2)
transformer = AutoModel.from_pretrained(
    CKPT_ID, 
    subfolder="transformer", 
    torch_dtype=torch.bfloat16, 
    parallel_config=cp_config
)

pipeline = DiffusionPipeline.from_pretrained(
    CKPT_ID, transformer=transformer, torch_dtype=torch.bfloat16,
).to(device)
```

--------------------------------

### Download Example Dataset

Source: https://github.com/yfyang007/realrestorer/blob/master/diffusers/docs/source/en/training/text_inversion.md

Downloads a sample dataset for training, in this case, cat toy images, using snapshot_download from huggingface_hub.

```python
from huggingface_hub import snapshot_download

local_dir = "./cat"
snapshot_download(
    "diffusers/cat_toy_example", local_dir=local_dir, repo_type="dataset", ignore_patterns=".gitattributes"
)
```