### Quick Start IP-Adapter Inference

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/ipadapter/README.md

Run a quick start inference example using the IP-Adapter. This command specifies the inference mode, reference image path, prompt, and other inference parameters.

```bash
# Quick start
python3 sample_ipadapter.py  --infer-mode fa --ref-image-path ipadapter/asset/input/tiger.png --i-scale 1.0 --prompt 一只老虎在海洋中游泳，背景是海洋。构图方式是居中构图，呈现了动漫风格和文化，营造了平静的氛围。 --infer-steps 100 --is-ipa True --load-key distill
```

--------------------------------

### Install IndexKits from Source

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/IndexKits/README.md

Install the IndexKits library from its source code. This is an alternative to installing from a pre-compiled wheel.

```shell
cd IndexKits
pip install -e .
```

--------------------------------

### Quick Start Inference with ControlNet

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/controlnet/README.md

Run a sample inference command using ControlNet with specified parameters. This example demonstrates using canny ControlNet with a detailed prompt and condition image.

```python
python sample_controlnet.py --infer-mode fa --no-enhance --load-key distill --infer-steps 50 --control-type canny --prompt "在夜晚的酒店门前，一座古老的中国风格的狮子雕像矗立着，它的眼睛闪烁着光芒，仿佛在守护着这座建筑。背景是夜晚的酒店前，构图方式是特写，平视，居中构图。这张照片呈现了真实摄影风格，蕴含了中国雕塑文化，同时展现了神秘氛围" --condition-image-path controlnet/asset/input/canny.jpg --control-weight 1.0
```

--------------------------------

### Conda Environment Setup

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Set up a Conda environment using the provided environment.yml file, activate it, and install pip dependencies.

```shell
# 1. Prepare conda environment
conda env create -f environment.yml

# 2. Activate the environment
conda activate HunyuanDiT

# 3. Install pip dependencies
python -m pip install -r requirements.txt
```

--------------------------------

### Launch Gradio Controller

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Starts the Gradio controller. This is the first step in launching the Gradio demo.

```bash
cd mllm
python -m llava.serve.controller --host 0.0.0.0 --port 10000
```

--------------------------------

### Install Hugging Face CLI

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Install the huggingface-cli tool, which is required for downloading pretrained models.

```shell
python -m pip install "huggingface_hub[cli]"
```

--------------------------------

### Launch Kohya_ss GUI

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/kohya_ss-hydit/README.md

Start the kohya_ss GUI by running the launch command. This will open the web interface for training.

```shell
python kohya_gui.py
```

--------------------------------

### Launch Gradio Web Server

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Starts the Gradio web server, connecting to the controller. This provides the web interface for the demo.

```bash
python -m llava.serve.gradio_web_server --controller http://0.0.0.0:10000 --model-list-mode reload --port 443
```

--------------------------------

### Start Gradio Inference GUI

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/kohya_ss-hydit/README.md

Run this command to start the basic Gradio interface for inference. Visit the provided URL in your browser to access the GUI.

```bash
python ./sd-scripts/hunyuan_inference_gui.py
```

--------------------------------

### Install Flash Attention v2

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Install flash attention v2 for performance acceleration. This requires CUDA 11.6 or above.

```shell
python -m pip install git+https://github.com/Dao-AILab/flash-attention.git@v2.1.2.post3
```

--------------------------------

### Setup kohya_ss GUI and sd-scripts

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/kohya_ss-hydit/README.md

Clone the kohya_ss GUI and sd-scripts repository, then copy the custom GUI codes. Ensure the sd-scripts are on the 'dev' branch.

```shell
git clone https://github.com/bmaltais/kohya_ss.git
cd kohya_ss/
git clone -b dev https://github.com/kohya-ss/sd-scripts ./sd-scripts
cp -Rf ${HunyuanDiT}/kohya_ss-hydit/* ./
```

--------------------------------

### Launch Gradio Demo Locally

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/mllm/README.md

Start the Gradio web server and model worker for the Hunyuan-Captioner model. Ensure controller is running first.

```bash
cd mllm
python -m llava.serve.controller --host 0.0.0.0 --port 10000
python -m llava.serve.gradio_web_server --controller http://0.0.0.0:10000 --model-list-mode reload --port 443
python -m llava.serve.model_worker --host 0.0.0.0 --controller http://0.0.0.0:10000 --port 40000 --worker http://0.0.0.0:40000 --model-path "../ckpts/captioner" --model-name LlavaMistral
```

--------------------------------

### Install Required Libraries for Low VRAM

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/lite/README.md

Install the bitsandbytes library, the diffusers library from GitHub, and a specific version of PyTorch for running HunyuanDiT with under 6GB GPU VRAM.

```bash
pip install -U bitsandbytes
pip install git+https://github.com/huggingface/diffusers
pip install torch==2.0.0
```

--------------------------------

### Install Training Dependencies

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/controlnet/README.md

Install essential Python packages for ControlNet training. These include matplotlib, onnxruntime_gpu, and opencv-python.

```bash
pip install matplotlib==3.7.5
pip install onnxruntime_gpu==1.16.3
pip install opencv-python==4.8.1.78
```

--------------------------------

### Install Diffusers

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/diffusers/README.md

Install the latest version of the Hugging Face Diffusers library to use Hunyuan-DiT pipelines.

```bash
pip install git+https://github.com/huggingface/diffusers.git
```

--------------------------------

### Install IndexKits Dependency

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Installs the IndexKits library, which is used for efficient data management during training. Navigate to the project directory before running.

```shell
cd HunyuanDiT
pip install -e ./IndexKits
```

--------------------------------

### Multiple Pictures Inference Setup

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/mllm/README.md

Convert multiple images to a CSV file for batch processing and then perform inference.

```bash
python mllm/make_csv.py --img_dir "mllm/images" --input_file "mllm/images/demo.csv"
```

```bash
python mllm/caption_demo.py --mode "caption_zh" --input_file "mllm/images/demo.csv" --output_file "mllm/images/demo_res.csv" --model_path "./ckpts/captioner"
```

--------------------------------

### Start Multi-turn Text-to-Image Generation UI

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Launches the multi-turn Text-to-Image generation UI. If GPU memory is less than 32GB, use '--load-4bit' to enable 4-bit quantization, which requires at least 22GB of memory.

```shell
python app/multiTurnT2I_app.py --infer-mode fa
```

--------------------------------

### Install DWPose Checkpoints

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/controlnet/README.md

Download and extract DWPose checkpoints required for pose extraction. Ensure the checkpoints are saved to the 'hydit/annotator/ckpts' directory.

```bash
mkdir ./hydit/annotator/ckpts
wget -O ./hydit/annotator/ckpts/dwpose.zip https://dit.hunyuan.tencent.com/download/HunyuanDiT/dwpose.zip
unzip ./hydit/annotator/ckpts/dwpose.zip -d ./hydit/annotator/ckpts/
```

--------------------------------

### Start Multi-turn T2I Generation UI

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/mllm/README.md

Launches the Gradio UI for multi-turn text-to-image generation. Use '--load-4bit' for GPUs with less than 32GB memory to enable 4-bit quantization, requiring at least 22GB.

```bash
# Start a multi-turn T2I generation UI. 
# If your GPU memory is less than 32GB, use '--load-4bit' to enable 4-bit quantization, which requires at least 22GB of memory.
python app/multiTurnT2I_app.py

```

--------------------------------

### Install HunyuanDiT Diffusers Library

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Installs the 🤗 diffusers library, ensuring it is version 0.28.1 or higher. PyTorch version 2.0 or higher is required.

```shell
pip install git+https://github.com/huggingface/diffusers.git
```

```shell
pip install diffusers
```

--------------------------------

### Download Torch Model using Hugging Face CLI

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/mllm/trtllm/README.md

Downloads the HunyuanCaptioner model weights from Hugging Face to a local directory. Ensure you have the huggingface-cli installed and configured.

```shell
huggingface-cli download Tencent-Hunyuan/HunyuanCaptioner --local-dir ../ckpts/captioner
```

--------------------------------

### Install Python Environment and Packages

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/kohya_ss-hydit/README.md

Create a conda environment for hydit-kohya and install essential Python packages, including PyTorch with CUDA support and other dependencies from hunyuan_requirements.txt.

```shell
conda create -n hydit-kohya python=3.10.12
conda activate hydit-kohya
# For cuda 12:
pip install torch==2.1.2 torchvision==0.16.2 xformers==0.0.23.post1
# For cuda 11:
pip install torch==2.1.2+cu118 torchvision==0.16.2+cu118 xformers==0.0.23.post1+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
# For cpu offloading to save GPU memory, we recommend to install Deepspeed as follows:
DS_BUILD_CPU_ADAM=1 pip install deepspeed==0.14.1
pip install -r hunyuan_requirements.txt
```

--------------------------------

### Start HunyuanDiT App with English UI

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Launches the HunyuanDiT application with the user interface set to English.

```shell
python app/hydit_app.py --lang en --infer-mode fa
```

--------------------------------

### Quick Start Inference with Downloaded LoRA Weights

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Perform a quick inference test using downloaded LoRA weights. Specify the inference mode, prompt, and the path to the LoRA checkpoint.

```shell
python sample_t2i.py --infer-mode fa --prompt "青花瓷风格，一只猫在追蝴蝶"  --no-enhance --load-key ema --lora-ckpt ./ckpts/t2i/lora/porcelain
```

--------------------------------

### Start HunyuanDiT Gradio App (Custom Port/Host, Distilled Model)

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Launch the HunyuanDiT Gradio application with custom server settings and using the distilled model. This command specifies the server name, port, and load key.

```shell
# Using special port and host
python app/hydit_app.py --infer-mode fa --server_name 0.0.0.0 --server_port 443 --load-key distill
```

--------------------------------

### Start HunyuanDiT Gradio App (Disable Enhancement)

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Launch the HunyuanDiT Gradio application with the enhancement model disabled. This can be useful if GPU memory is insufficient.

```shell
# You can disable the enhancement model if the GPU memory is insufficient.

```

--------------------------------

### ControlNet Inference Example

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Perform inference using ControlNet with specified parameters, including prompt, image path, and control type. Uses a distilled base model.

```python
python3 sample_controlnet.py --infer-mode fa --no-enhance --load-key distill --infer-steps 50 --control-type canny --prompt "在夜晚的酒店门前，一座古老的中国风格的狮子雕像矗立着，它的眼睛闪烁着光芒，仿佛在守护着这座建筑。背景是夜晚的酒店前，构图方式是特写，平视，居中构图。这张照片呈现了真实摄影风格，蕴含了中国雕塑文化，同时展现了神秘氛围" --condition-image-path controlnet/asset/input/canny.jpg --control-weight 1.0
```

--------------------------------

### Inference with Canny ControlNet (Weighted)

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/controlnet/README.md

Perform inference using Canny ControlNet with a specified weight for all layers. This example demonstrates how to apply custom weights to ControlNet layers during inference.

```bash
python sample_controlnet.py --infer-mode fa --control-weight "[1.0 * (0.825 ** float(19 - i)) for i in range(19)]" --no-enhance --load-key distill --infer-steps 50 --control-type canny --prompt "在夜晚的酒店门前，一座古老的中国风格的狮子雕像矗立着，它的眼睛闪烁着光芒，仿佛在守护着这座建筑。背景是夜晚的酒店前，构图方式是特写，平视，居中构图。这张照片呈现了真实摄影风格，蕴含了中国雕塑文化，同时展现了神秘氛围" --condition-image-path controlnet/asset/input/canny.jpg
```

--------------------------------

### Start HunyuanDiT App without Enhancement

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Launches the HunyuanDiT application without prompt enhancement. The enhancement will be unavailable until the app is restarted without the `--no-enhance` flag.

```shell
python app/hydit_app.py --no-enhance --infer-mode fa
```

--------------------------------

### Start HunyuanDiT Gradio App (Chinese UI, Flash Attention)

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Launch the HunyuanDiT Gradio application with Chinese UI and Flash Attention acceleration. Ensure the conda environment is activated.

```shell
# By default, we start a Chinese UI. Using Flash Attention for acceleration.
python app/hydit_app.py --infer-mode fa
```

--------------------------------

### Training Configuration for Canny ControlNet

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/controlnet/README.md

Configure and start the training process for Canny ControlNet. This script sets up various training parameters including task flags, learning rate, batch size, and deepspeed configurations.

```bash
task_flag="canny_controlnet"                                # the task flag is used to identify folders.
control_type=canny
resume_module_root=./ckpts/t2i/model/pytorch_model_distill.pt  # checkpoint root for resume
index_file=/path/to/your/indexfile                           # index file for dataloader
results_dir=./log_EXP                                        # save root for results
batch_size=1                                                 # training batch size
image_size=1024                                              # training image resolution
grad_accu_steps=2                                            # gradient accumulation
warmup_num_steps=0                                           # warm-up steps
lr=0.0001                                                    # learning rate
ckpt_every=10000                                             # create a ckpt every a few steps.
ckpt_latest_every=5000                                       # create a ckpt named `latest.pt` every a few steps.
epochs=100                                                   # total training epochs


sh $(dirname "$0")/run_g_controlnet.sh \
    --task-flag ${task_flag} \
    --control-type ${control_type} \
    --noise-schedule scaled_linear --beta-start 0.00085 --beta-end 0.03 \
    --predict-type v_prediction \
    --multireso \
    --reso-step 64 \
    --uncond-p 0.44 \
    --uncond-p-t5 0.44 \
    --index-file ${index_file} \
    --random-flip \
    --lr ${lr} \
    --batch-size ${batch_size} \
    --image-size ${image_size} \
    --global-seed 999 \
    --grad-accu-steps ${grad_accu_steps} \
    --warmup-num-steps ${warmup_num_steps} \
    --use-flash-attn \
    --use-fp16 \
    --results-dir ${results_dir} \
    --resume \
    --resume-module-root ${resume_module_root} \
    --epochs ${epochs} \
    --ckpt-every ${ckpt_every} \
    --ckpt-latest-every ${ckpt_latest_every} \
    --log-every 10 \
    --deepspeed \
    --deepspeed-optimizer \
    --use-zero-stage 2 \
    --use-style-cond \
    --size-cond 1024 1024 \
    "$@"
```

--------------------------------

### Example Column Filtering Configuration

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/IndexKits/docs/MakeDataset.md

This snippet demonstrates how to configure filtering for 'height' and 'width' columns, ensuring they are greater than or equal to 512, with a default value of 1024 if invalid. This configuration allows for simultaneous filtering conditions.

```yaml
filter:
    column:
        -   name: height
            type: int
            action: ge
            target: 512
            default: 1024
        -   name: width
            type: int
            action: ge
            target: 512
            default: 1024
```

--------------------------------

### Run HunyuanDiT Inference Demo

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/lite/README.md

Execute the inference script for HunyuanDiT with specified model ID, prompt, inference steps, and guidance scale. This command is used for a quick start with the low VRAM version.

```bash
cd HunyuanDiT

# Quick start
model_id=Tencent-Hunyuan/HunyuanDiT-v1.2-Diffusers-Distilled
prompt=一个宇航员在骑马
infer_steps=50
guidance_scale=6
python3 lite/inference.py ${model_id} ${prompt} ${infer_steps} ${guidance_scale}
```

--------------------------------

### Download and Prepare Dataset

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Downloads a demo dataset, extracts it, and creates necessary subdirectories for processed data. Ensure the target directory exists.

```shell
wget -O ./dataset/data_demo.zip https://dit.hunyuan.tencent.com/download/HunyuanDiT/data_demo.zip
unzip ./dataset/data_demo.zip -d ./dataset
mkdir ./dataset/porcelain/arrows ./dataset/porcelain/jsons
```

--------------------------------

### Download IP-Adapter Model Weights

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/ipadapter/README.md

Download the IP-Adapter model weights using the huggingface-cli tool. It's recommended to use module weights as the base model for IP-Adapter inference.

```bash
cd HunyuanDiT
# Use the huggingface-cli tool to download the model.
# We recommend using module weights as the base model for IP-Adapter inference, as our provided pretrained weights are trained on them.
huggingface-cli download Tencent-Hunyuan/IP-Adapter ipa.pt --local-dir ./ckpts/t2i/model
huggingface-cli download Tencent-Hunyuan/IP-Adapter clip_img_encoder.pt  --local-dir ./ckpts/t2i/model/clip_img_encoder
```

--------------------------------

### 解压数据集

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/kohya_ss-hydit/Kohya_ss GUI训练中文教程.ipynb

用于解压上传的ZIP格式数据集文件。请将 "/root/xxx.zip" 替换为你的数据集文件名。

```python
# 演示用
! unzip /root/xxx.zip -d ./root/xxx
```

--------------------------------

### Configuring Dataset and DataLoader

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/IndexKits/README.md

Set up SimpleDataset and DataLoader with BlockDistributedSampler for multi-resolution data. Ensure batch_size and world_size are correctly passed. Use the dataset's shuffle method, not DataLoader's.

```python
dataset = SimpleDataset('data_multireso.json', batch_size=batch_size, world_size=world_size)
sampler = BlockDistributedSampler(dataset, num_replicas=world_size, rank=rank,
                                  shuffle=shuffle, drop_last=drop_last, batch_size=batch_size)
loader = DataLoader(dataset, batch_size=batch_size, shuffle=shuffle, sampler=sampler,
                    num_workers=num_workers, pin_memory=True, drop_last=drop_last)

for epoch in range(10):
   dataset.shuffle(epoch, fast=True)
   for batch in loader:
       pass
```

--------------------------------

### Launch Gradio Model Worker

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Starts the model worker, which loads the model and communicates with the controller. Ensure the model path is correct.

```bash
python -m llava.serve.model_worker --host 0.0.0.0 --controller http://0.0.0.0:10000 --port 40000 --worker http://0.0.0.0:40000 --model-path "../ckpts/captioner" --model-name LlavaMistral
```

--------------------------------

### IP-Adapter Training Script

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/ipadapter/README.md

This script is used for training IP-Adapter models. It requires several configuration parameters to be set, including task flags, file paths, batch size, image resolution, learning rate, and checkpointing settings. Ensure all necessary paths and parameters are correctly specified before execution.

```bash
task_flag="IP_Adapter"                                # the task flag is used to identify folders.                         # checkpoint root for resume
index_file=path/to/your/index_file
results_dir=./log_EXP                                        # save root for results
batch_size=1                                                 # training batch size
image_size=1024                                              # training image resolution
grad_accu_steps=1                                            # gradient accumulation
warmup_num_steps=0                                           # warm-up steps
lr=0.0001                                                    # learning rate
ckpt_every=10                                         # create a ckpt every a few steps.
ckpt_latest_every=10000                                    # create a ckpt named `latest.pt` every a few steps.
ckpt_every_n_epoch=2                                         # create a ckpt every a few epochs.
epochs=8                                                     # total training epochs

PYTHONPATH=. \
sh $(dirname "$0")/run_g_ipadapter.sh \
    --task-flag ${task_flag} \
    --noise-schedule scaled_linear --beta-start 0.00085 --beta-end 0.018 \
    --predict-type v_prediction \
    --multireso \
    --reso-step 64 \
    --uncond-p 0.22 \
    --uncond-p-t5 0.22\
    --uncond-p-img 0.05\
    --index-file ${index_file} \
    --random-flip \
    --lr ${lr} \
    --batch-size ${batch_size} \
    --image-size ${image_size} \
    --global-seed 999 \
    --grad-accu-steps ${grad_accu_steps} \
    --warmup-num-steps ${warmup_num_steps} \
    --use-flash-attn \
    --use-fp16 \
    --extra-fp16 \
    --results-dir ${results_dir} \
    --resume\
    --resume-module-root ckpts/t2i/model/pytorch_model_module.pt \
    --epochs ${epochs} \
    --ckpt-every ${ckpt_every} \
    --ckpt-latest-every ${ckpt_latest_every} \
    --ckpt-every-n-epoch ${ckpt_every_n_epoch} \
    --log-every 10 \
    --deepspeed \
    --use-zero-stage 2 \
    --gradient-checkpointing \
    --no-strict \
    --training-parts ipadapter \
    --is-ipa True \
    --resume-ipa True \
    --resume-ipa-root ckpts/t2i/model/ipa.pt  \
    "$@"
```

--------------------------------

### Download Pretrained Models

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Create a directory for models and download the HunyuanDiT-v1.2 model using the huggingface-cli.

```shell
# Create a directory named 'ckpts' where the model will be saved, fulfilling the prerequisites for running the demo.
mkdir ckpts
# Use the huggingface-cli tool to download the model.
# The download time may vary from 10 minutes to 1 hour depending on network conditions.
huggingface-cli download Tencent-Hunyuan/HunyuanDiT-v1.2 --local-dir ./ckpts
```

--------------------------------

### Train LoRA without Flash Attention

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Use this script for training LoRA when Flash Attention cannot be installed. Ensure the PYTHONPATH is set correctly.

```shell
PYTHONPATH=./ sh lora/train_lora.sh --index-file dataset/porcelain/jsons/porcelain.json
```

--------------------------------

### Train IP-Adapter

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Execute the training script for IP-Adapter. Ensure the PYTHONPATH is set correctly.

```shell
# Training for IP-Adapter.
PYTHONPATH=./ sh hydit/train_ipadapter.sh
```

--------------------------------

### Inference with Pose ControlNet

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/controlnet/README.md

Perform inference using Pose ControlNet with a default weight of 1.0. This command is suitable for generating images guided by pose information.

```bash
python3 sample_controlnet.py --infer-mode fa --control-weight 1.0 --no-enhance --load-key distill --infer-steps 50 --control-type pose --prompt "在白天的森林中，一位穿着绿色上衣的亚洲女性站在大象旁边。照片采用了中景、平视和居中构图的方式，呈现出写实的效果。这张照片蕴含了人物摄影文化，并展现了宁静的氛围" --condition-image-path controlnet/asset/input/pose.jpg
```

--------------------------------

### Create Base Index V2 Dataset

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/IndexKits/docs/MakeDataset.md

Use the `idk base` command to create a Base Index V2 dataset. Specify the configuration file with `-c` and the output path with `-t`.

```shell
idk base -c base_config.yaml -t base_dataset.json
```

--------------------------------

### Inference with HunyuanDiT Base Model

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/diffusers/README.md

Generate images using the base HunyuanDiT model with either Chinese or English prompts. Ensure your environment has CUDA support and PyTorch installed.

```python
import torch
from diffusers import HunyuanDiTPipeline

pipe = HunyuanDiTPipeline.from_pretrained("Tencent-Hunyuan/HunyuanDiT-v1.2-Diffusers", torch_dtype=torch.float16)
pipe.to("cuda")

# You may also use English prompt as HunyuanDiT supports both English and Chinese
# prompt = "An astronaut riding a horse"
prompt = "一个宇航员在骑马"
image = pipe(prompt).images[0]
```

--------------------------------

### Apply LoRA to HunyuanDiT Pipeline

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/diffusers/README.md

Integrate and use pre-trained LoRA checkpoints with the HunyuanDiT pipeline in 🤗 Diffusers. This example demonstrates loading LoRA weights and applying them to the transformer.

```python
import torch
from diffusers import HunyuanDiTPipeline

### convert checkpoint to diffusers format
num_layers = 40
def load_hunyuan_dit_lora(transformer_state_dict, lora_state_dict, lora_scale):
    for i in range(num_layers):
        Wqkv = torch.matmul(lora_state_dict[f"blocks.{i}.attn1.Wqkv.lora_B.weight"], lora_state_dict[f"blocks.{i}.attn1.Wqkv.lora_A.weight"])
        q, k, v = torch.chunk(Wqkv, 3, dim=0)
        transformer_state_dict[f"blocks.{i}.attn1.to_q.weight"] += lora_scale * q
        transformer_state_dict[f"blocks.{i}.attn1.to_k.weight"] += lora_scale * k
        transformer_state_dict[f"blocks.{i}.attn1.to_v.weight"] += lora_scale * v

        out_proj = torch.matmul(lora_state_dict[f"blocks.{i}.attn1.out_proj.lora_B.weight"], lora_state_dict[f"blocks.{i}.attn1.out_proj.lora_A.weight"])
        transformer_state_dict[f"blocks.{i}.attn1.to_out.0.weight"] += lora_scale * out_proj

        q_proj = torch.matmul(lora_state_dict[f"blocks.{i}.attn2.q_proj.lora_B.weight"], lora_state_dict[f"blocks.{i}.attn2.q_proj.lora_A.weight"])
        transformer_state_dict[f"blocks.{i}.attn2.to_q.weight"] += lora_scale * q_proj

        kv_proj = torch.matmul(lora_state_dict[f"blocks.{i}.attn2.kv_proj.lora_B.weight"], lora_state_dict[f"blocks.{i}.attn2.kv_proj.lora_A.weight"])
        k, v = torch.chunk(kv_proj, 2, dim=0)
        transformer_state_dict[f"blocks.{i}.attn2.to_k.weight"] += lora_scale * k
        transformer_state_dict[f"blocks.{i}.attn2.to_v.weight"] += lora_scale * v

        out_proj = torch.matmul(lora_state_dict[f"blocks.{i}.attn2.out_proj.lora_B.weight"], lora_state_dict[f"blocks.{i}.attn2.out_proj.lora_A.weight"])
        transformer_state_dict[f"blocks.{i}.attn2.to_out.0.weight"] += lora_scale * out_proj

    q_proj = torch.matmul(lora_state_dict["pooler.q_proj.lora_B.weight"], lora_state_dict["pooler.q_proj.lora_A.weight"])
    transformer_state_dict["time_extra_emb.pooler.q_proj.weight"] += lora_scale * q_proj

    return transformer_state_dict

### use the diffusers pipeline with lora
pipe = HunyuanDiTPipeline.from_pretrained("Tencent-Hunyuan/HunyuanDiT-v1.2-Diffusers", torch_dtype=torch.float16)
pipe.to("cuda")

from safetensors import safe_open

lora_state_dict = {}
with safe_open("./ckpts/t2i/lora/jade/adapter_model.safetensors", framework="pt", device=0) as f:
    for k in f.keys():
        lora_state_dict[k[17:]] = f.get_tensor(k) # remove 'basemodel.model'

transformer_state_dict = pipe.transformer.state_dict()
transformer_state_dict = load_hunyuan_dit_lora(transformer_state_dict, lora_state_dict, lora_scale=1.0)
pipe.transformer.load_state_dict(transformer_state_dict)

prompt = "玉石绘画风格，一只猫在追蝴蝶"
image = pipe(
    prompt, 
    num_inference_steps=100,
    guidance_scale=6.0, 
).images[0]
image.save('img.png')
```

--------------------------------

### Download Docker Image (CUDA 12)

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Download and import the Docker image for CUDA 12.

```shell
# 1. Use the following link to download the docker image tar file.
# For CUDA 12
wget https://dit.hunyuan.tencent.com/download/HunyuanDiT/hunyuan_dit_cu12.tar

# 2. Import the docker tar file and show the image meta information
# For CUDA 12
docker load -i hunyuan_dit_cu12.tar

docker image ls
```

--------------------------------

### Download Pretrained Models with HF-Mirror

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Use HF-Mirror to potentially speed up model downloads, especially if experiencing slow speeds in China. This command downloads the model to the './ckpts' directory.

```shell
HF_ENDPOINT=https://hf-mirror.com huggingface-cli download Tencent-Hunyuan/HunyuanDiT-v1.2 --local-dir ./ckpts
```

--------------------------------

### Repeat Samples by MD5 File

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/IndexKits/docs/MakeDataset.md

Set up repeating samples using an MD5 file. This allows specifying repeat counts or additional repeats based on MD5 hashes found in a provided file.

```yaml
repeater:
    md5:
        - name: goodcase1
          path: /HunYuanDiT/dataset/porcelain/md5_repeat_1.json
          type: dict
          plus: 3
        - name: goodcase2
          path: /HunYuanDiT/dataset/porcelain/md5_repeat_2.json
          type: list
          repeat: 6
```

--------------------------------

### 复制训练好的LoRA模型

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/kohya_ss-hydit/Kohya_ss GUI训练中文教程.ipynb

将训练完成的LoRA模型文件（safetensors格式）复制到ComfyUI的loras目录下。请将 "/path/to/kohya_ss/outputs/last-step00001600.safetensors" 和 "/path/to/ComfyUI/models/loras/" 替换为你的实际路径。

```bash
! cp /path/to/kohya_ss/outputs/last-step00001600.safetensors /path/to/ComfyUI/models/loras/ && echo "复制完成"
```

--------------------------------

### Multireso Index V2 Dataset Configuration

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/IndexKits/docs/MakeDataset.md

Use this configuration to create a multi-resolution dataset. Specify source files, base size, and resolution step or target ratios.

```shell
src:
    - /HunYuanDiT/dataset/porcelain/jsons/a.json
    - /HunYuanDiT/dataset/porcelain/jsons/b.json
    - /HunYuanDiT/dataset/porcelain/jsons/c.json
base_size: 512
reso_step: 32
min_size: 512
```

--------------------------------

### Download Docker Image (CUDA 11)

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Download and import the Docker image for CUDA 11.

```shell
# 1. Use the following link to download the docker image tar file.
# For CUDA 11
wget https://dit.hunyuan.tencent.com/download/HunyuanDiT/hunyuan_dit_cu11.tar

# 2. Import the docker tar file and show the image meta information
# For CUDA 11
docker load -i hunyuan_dit_cu11.tar  

docker image ls
```

--------------------------------

### Create Multiresolution Dataset with Python (Target Ratios)

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/IndexKits/docs/MakeDataset.md

This Python function creates multiresolution buckets based on a list of target aspect ratios. It prioritizes target ratios over resolution step if both are provided.

```python
from index_kits import build_multi_resolution_bucket  

md5_hw = None  
# If many arrows in your Index V2 lack height and width, you can pre-calculate them and pass through the md5_hw parameter.  
# md5_hw = {'c67be1d8f30fd0edcff6ac99b703879f': (720, 1280), ...}  
# with open('md5_hw.pkl', 'rb') as f:  
#     md5_hw = pickle.load(f)  

index_v2_file_path = 'data_v2.json'  
index_v2_save_path = 'data_multireso.json'  

# Method 2: Given a series of target aspect ratios, automatically calculate all resolution buckets.  
build_multi_resolution_bucket(base_size=1024,  
                              target_ratios=["1:1", "3:4", "4:3", "16:9", "9:16"],  
                              align=16,  
                              min_size=1024,  
                              src_index_files=index_v2_file_path,   
                              save_file=index_v2_save_path,   
                              md5_hw=md5_hw)
```

--------------------------------

### Inference with LoRA using Gradio

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/lora/README.md

Launch the Gradio application for inference with LoRA weights. The LoRA checkpoint is specified via `--lora-ckpt`.

```bash
python app/hydit_app.py --infer-mode fa --no-enhance --lora-ckpt log_EXP/001-lora_porcelain_ema_rank64/checkpoints/0001000.pt/
```

--------------------------------

### Multi Resolution Training

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Use this command for multi-resolution training, specifying the resolution step. Ensure the PYTHONPATH is set correctly.

```shell
PYTHONPATH=./ sh hydit/train.sh --index-file dataset/porcelain/jsons/porcelain_mt.json --multireso --reso-step 64
```

--------------------------------

### Pull and Run Triton Server Docker Container

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/mllm/trtllm/README.md

Pulls the specified Triton Server Docker image and runs it with necessary configurations for GPU access and shared memory. This is the initial setup step for using the TRTLLM version of Hunyuan-Captioner.

```bash
docker pull nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3
docker run --rm --ipc=host --runtime=nvidia --gpus all --entrypoint /bin/bash -it nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3
```

--------------------------------

### Download ControlNet Models

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/controlnet/README.md

Use the huggingface-cli tool to download the necessary ControlNet weights and the distilled base model. It's recommended to use distilled weights for ControlNet inference.

```bash
cd HunyuanDiT
huggingface-cli download Tencent-Hunyuan/HYDiT-ControlNet-v1.2 --local-dir ./ckpts/t2i/controlnet
huggingface-cli download Tencent-Hunyuan/Distillation-v1.2 ./pytorch_model_distill.pt --local-dir ./ckpts/t2i/model
```

--------------------------------

### Create Multiresolution Dataset with Python (Reso Step)

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/IndexKits/docs/MakeDataset.md

This Python function creates multiresolution buckets using a base size and a resolution step. It can optionally use pre-calculated MD5 to height/width mappings.

```python
from index_kits import build_multi_resolution_bucket  

md5_hw = None  
# If many arrows in your Index V2 lack height and width, you can pre-calculate them and pass through the md5_hw parameter.  
# md5_hw = {'c67be1d8f30fd0edcff6ac99b703879f': (720, 1280), ...}  
# with open('md5_hw.pkl', 'rb') as f:  
#     md5_hw = pickle.load(f)  

index_v2_file_path = 'data_v2.json'  
index_v2_save_path = 'data_multireso.json'  

# Method 1: Given base_size and reso_step, automatically calculate all resolution buckets.  
build_multi_resolution_bucket(base_size=1024,  
                              reso_step=64,  
                              min_size=1024,  
                              src_index_files=index_v2_file_path,   
                              save_file=index_v2_save_path,   
                              md5_hw=md5_hw)
```

--------------------------------

### Download LoRA Weights

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/lora/README.md

Use the huggingface-cli tool to download the trained LoRA weights for text-to-image generation. Ensure you are in the HunyuanDiT directory.

```bash
cd HunyuanDiT
huggingface-cli download Tencent-Hunyuan/HYDiT-LoRA --local-dir ./ckpts/t2i/lora
```

--------------------------------

### 一键打标数据集

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/kohya_ss-hydit/Kohya_ss GUI训练中文教程.ipynb

使用预训练的captioner模型为图像生成中文描述，并保存为txt文件。请确保 "./ckpts/captioner" 存在且包含模型文件，并将 "/root/demo_images_wukong" 替换为你的数据集路径。

```python
# 一键打标数据集
%cd HunyuanDiT
! python mllm/caption_imgs_with_txt.py \
    --model_path "./ckpts/captioner" \
    --mode "caption_zh" \
    --image_folder "/root/demo_images_wukong"
%cd /root
```

--------------------------------

### Download Hunyuan-Captioner Model

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/mllm/README.md

Use the huggingface-cli tool to download the Hunyuan-Captioner model to a specified local directory.

```shell
huggingface-cli download Tencent-Hunyuan/HunyuanCaptioner --local-dir ./ckpts/captioner
```

--------------------------------

### Create Base Index V2 Dataset with Python

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/IndexKits/docs/MakeDataset.md

Use the IndexV2Builder class from the index_kits library to create a base Index V2 dataset and save it to a JSON file.

```python
from index_kits import IndexV2Builder

builder = IndexV2Builder(arrow_files)
builder.save('data_v2.json')
```

--------------------------------

### Download ControlNet and Distilled Model

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/controlnet/README.md

Download the v1.1 base model and controlnet using the huggingface-cli tool. It is recommended to use distilled weights for ControlNet inference.

```bash
cd HunyuanDiT
# Use the huggingface-cli tool to download the model.
# We recommend using distilled weights as the base model for ControlNet inference, as our provided pretrained weights are trained on them.
huggingface-cli download Tencent-Hunyuan/HYDiT-ControlNet-v1.1 --local-dir ./HunyuanDiT-v1.1/t2i/controlnet
huggingface-cli download Tencent-Hunyuan/Distillation-v1.1 ./pytorch_model_distill.pt --local-dir ./HunyuanDiT-v1.1/t2i/model
```

--------------------------------

### Basic Configuration for Base Dataset

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/IndexKits/docs/MakeDataset.md

A basic YAML configuration file for creating a base dataset. It specifies the source Arrow files.

```yaml
source:
    - /HunYuanDiT/dataset/porcelain/arrows/00000.arrow
```

--------------------------------

### Inference with LoRA using Command Line

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/lora/README.md

Use this command to perform text-to-image inference with LoRA weights. Specify the LoRA checkpoint path using `--lora-ckpt`.

```bash
python sample_t2i.py --infer-mode fa --prompt "青花瓷风格，一只小狗"  --no-enhance --lora-ckpt log_EXP/001-lora_porcelain_ema_rank64/checkpoints/0001000.pt/
```

--------------------------------

### Single Resolution Training

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Use this command for single resolution training. Ensure the PYTHONPATH is set correctly.

```shell
PYTHONPATH=./ sh hydit/train.sh --index-file dataset/porcelain/jsons/porcelain.json
```

--------------------------------

### Training with Older HunyuanDiT Version (<= v1.1)

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Use this command to train with older versions of HunyuanDiT. Ensure the PYTHONPATH is set correctly.

```shell
PYTHONPATH=./ sh hydit/train_v1.1.sh --index-file dataset/porcelain/jsons/porcelain.json
```

--------------------------------

### Repeat Samples by Source Path

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/IndexKits/docs/MakeDataset.md

Specify the number of times to repeat Arrow files by adding 'repeat: n' to the source configuration. Ensure the Arrow path ends with a colon.

```yaml
source:
    - /HunYuanDiT/dataset/porcelain/arrows/*.arrow:
        repeat: 10
```

--------------------------------

### Load Multi-Bucket (Multi-Resolution) Index V2

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/IndexKits/README.md

Load a multi-bucket or multi-resolution index file using MultiResolutionBucketIndexV2. This involves defining a custom Dataset class that handles data alignment with batch_size and world_size.

```python
from torch.utils.data import DataLoader, Dataset
import torchvision.transforms as T

from index_kits import MultiResolutionBucketIndexV2
from index_kits.sampler import BlockDistributedSampler

class SimpleDataset(Dataset):
    def __init__(self, index_file, batch_size, world_size):
        # When using multi-bucket (multi-resolution), batch_size and world_size need to be specified for underlying data alignment.
        self.index_manager = MultiResolutionBucketIndexV2(index_file, batch_size, world_size)

        self.flip_norm = T.Compose(
            [
                T.RandomHorizontalFlip(),
                T.ToTensor(),
                T.Normalize([0.5], [0.5]),
            ]
        )

    def shuffle(self, seed, fast=False):
        self.index_manager.shuffle(seed, fast=fast)

    def __len__(self):
        return len(self.index_manager)

    def __getitem__(self, idx):
        image = self.index_manager.get_image(idx)
        original_size = image.size    # (w, h)
        target_size = self.index_manager.get_target_size(idx)     # (w, h)
        image, crops_coords_left_top = self.index_manager.resize_and_crop(image, target_size)
        image = self.flip_norm(image)
        text = self.index_manager.get_attribute(idx, column='text_zh')
        return image, text, (original_size, target_size, crops_coords_left_top)

batch_size = 8      # batch_size per GPU
world_size = 8      # total number of GPUs
rank = 0            # rank of the current process
num_workers = 4     # customizable based on actual conditions
shuffle = False     # must be set to False
drop_last = True    # must be set to True
```

--------------------------------

### Create Multi Resolution Training Data Index

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/README.md

Generates a JSON index file for multi-resolution training data using a specified YAML configuration file. This process is similar to single resolution but uses a different configuration for handling varying image resolutions.

```shell
idk multireso -c dataset/yamls/porcelain_mt.yaml -t dataset/porcelain/jsons/porcelain_mt.json
```

--------------------------------

### Run Text-to-Image Inference with LoRA

Source: https://github.com/tencent-hunyuan/hunyuandit/blob/main/lora/README.md

Execute the sample script for text-to-image inference, specifying a prompt, LoRA checkpoint, and inference mode. The `--no-enhance` flag disables enhancement, and `--load-key ema` loads the EMA weights.

```python
python sample_t2i.py --prompt "青花瓷风格，一只猫在追蝴蝶"  --no-enhance --load-key ema --lora-ckpt ./ckpts/t2i/lora/porcelain --infer-mode fa
```