### Clone Repository and Set Up Environment

Source: https://nvlabs.github.io/Sana/docs

Clone the SANA repository and run the environment setup script. Ensure you have the necessary dependencies installed before proceeding with model usage.

```bash
git clone https://github.com/NVlabs/Sana.git
cd Sana
bash ./environment_setup.sh sana
```

--------------------------------

### Download Dog Example Dataset

Source: https://nvlabs.github.io/Sana/docs/sana_lora_dreambooth

Download the dog example dataset locally using `snapshot_download` from the `huggingface_hub` library. Specify the local directory and repository type.

```python
from huggingface_hub import snapshot_download

local_dir = "data/dreambooth/dog"
snapshot_download(
    "diffusers/dog-example",
    local_dir=local_dir, repo_type="dataset",
    ignore_patterns=".gitattributes",
)
```

--------------------------------

### Install Training Dependencies

Source: https://nvlabs.github.io/Sana/docs/sana_lora_dreambooth

Clone the diffusers repository and install its training dependencies from source. Ensure you are in a new virtual environment before proceeding.

```bash
git clone https://github.com/huggingface/diffusers
cd diffusers
pip install -e .
```

--------------------------------

### Configure Accelerate Environment

Source: https://nvlabs.github.io/Sana/docs/sana_lora_dreambooth

Initialize an Accelerate environment for distributed training. Use 'accelerate config' for interactive setup or 'accelerate config default' for a non-interactive setup.

```bash
accelerate config
```

```bash
accelerate config default
```

```python
from accelerate.utils import write_basic_config
write_basic_config()
```

--------------------------------

### Clone Sana Repository and Set Up Environment

Source: https://nvlabs.github.io/Sana/docs/installation

Clone the Sana repository and set up the environment using the provided script. Alternatively, install components step-by-step.

```bash
git clone https://github.com/NVlabs/Sana.git
cd Sana

bash ./environment_setup.sh sana
# or you can install each components step by step following environment_setup.sh

```

--------------------------------

### Quantization Example

Source: https://nvlabs.github.io/Sana/docs/8bit_sana

Demonstrates how to load a quantized SanaPipeline for inference using bitsandbytes for 8-bit quantization.

```APIDOC
## Quantization

Quantization helps reduce the memory requirements of very large models by storing model weights in a lower precision data type. However, quantization may have varying impact on video quality depending on the video model.

Refer to the Quantization overview to learn more about supported quantization backends and selecting a quantization backend that supports your use case. The example below demonstrates how to load a quantized [`SanaPipeline`] for inference with bitsandbytes.

```python
import torch
from diffusers import BitsAndBytesConfig as DiffusersBitsAndBytesConfig, SanaTransformer2DModel, SanaPipeline
from transformers import BitsAndBytesConfig as BitsAndBytesConfig, AutoModel

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

quant_config = BitsAndBytesConfig(load_in_8bit=True)
text_encoder_8bit = AutoModel.from_pretrained(
    "Efficient-Large-Model/Sana_1600M_1024px_diffusers",
    subfolder="text_encoder",
    quantization_config=quant_config,
    torch_dtype=torch.float16,
)

quant_config = DiffusersBitsAndBytesConfig(load_in_8bit=True)
transformer_8bit = SanaTransformer2DModel.from_pretrained(
    "Efficient-Large-Model/Sana_1600M_1024px_diffusers",
    subfolder="transformer",
    quantization_config=quant_config,
    torch_dtype=torch.float16,
)

pipeline = SanaPipeline.from_pretrained(
    "Efficient-Large-Model/Sana_1600M_1024px_diffusers",
    text_encoder=text_encoder_8bit,
    transformer=transformer_8bit,
    torch_dtype=torch.float16,
    device_map="balanced",
)
pipeline.to(device)

prompt = "a tiny astronaut hatching from an egg on the moon"
image = pipeline(prompt).images[0]
image.save("sana.png")

```
```

--------------------------------

### Setup SANA-WM Environment

Source: https://nvlabs.github.io/Sana/docs/sana_wm

Run this script to set up the SANA-WM environment and activate the conda environment.

```bash
bash ./environment_setup.sh sana
conda activate sana

```

--------------------------------

### SANA-WM Training Configuration Example

Source: https://nvlabs.github.io/Sana/docs/sana_wm

This configuration snippet shows key parameters for SANA-WM Stage-1 training, including dataset paths, model loading, attention types, and training settings like FSDP version and CP size.

```yaml
data:
  hf_dataset_repo: Efficient-Large-Model/SANA-WM-example-training-dataset
  hf_dataset_revision: 4d965e94b9ea11b9c5ba085251ffa7a0345e006f
  hf_dataset_local_dir: .
  data_dir:
    sekai_game: data/sekai_game_train_961frames_16fps_ovl640
  vae_cache_dir: data/vae_cache/LTX2VAE_diffusers_704x1280/sekai_game_train_961frames_16fps_ovl640
model:
  load_from: hf://Efficient-Large-Model/SANA-WM_chunk_causal/dit/sana_wm_chunk_causal_1600m_720p.safetensors
  attn_type: ChunkCausalGDNTriton
  camctrl_type: ChunkCausalGDNUCPESinglePathLiteLABothTriton
train:
  use_fsdp: true
  fsdp_version: 2
  cp_size: 2

```

--------------------------------

### Launch Gradio Demo for Sana

Source: https://nvlabs.github.io/Sana/docs/sana

Start the Gradio web demo for Sana. This command launches a local server and can optionally create a public shareable link.

```bash
DEMO_PORT=15432 \
python app/app_sana.py \
    --share \
    --config=configs/sana_config/1024ms/Sana_1600M_img1024.yaml \
    --model_path=hf://Efficient-Large-Model/Sana_1600M_1024px_BF16/checkpoints/Sana_1600M_1024px_BF16.pth \
    --image_size=1024

```

--------------------------------

### Train SANA-WM Chunk-Causal Stage-1 on Sekai-Game

Source: https://nvlabs.github.io/Sana/docs/sana_wm

Execute this command to start training the chunk-causal Stage-1 model using FSDP2 and CP2 on the Sekai-Game dataset. The configuration path points to the specific training setup.

```bash
torchrun --nproc_per_node=8 --master_port=29500 \
  train_video_scripts/train_sana_wm_stage1.py \
  --config_path configs/sana_wm/stage1/sana_wm_stage1_sekai_chunk_causal_cp2_fsdp2.yaml

```

--------------------------------

### Setup SANA Environment

Source: https://nvlabs.github.io/Sana/docs/sana_streaming

Execute these commands to set up the SANA environment and activate the conda environment.

```bash
bash ./environment_setup.sh sana
conda activate sana
```

--------------------------------

### Install Dependencies for Scaling

Source: https://nvlabs.github.io/Sana/docs/inference_scaling/inference_scaling

Installs necessary libraries for inference scaling, including transformers and a custom scaling library. Ensure you are using the specified transformers version for compatibility.

```bash
# other transformers version may also work, but we have not tested
pip install transformers==4.46
pip install git+https://github.com/bfshi/scaling_on_scales.git

```

--------------------------------

### Install SGLang with Diffusion Support

Source: https://nvlabs.github.io/Sana/docs/sglang

Install the SGLang library with diffusion capabilities using pip. This command enables support for image generation models.

```bash
uv pip install 'sglang[diffusion]' --prerelease=allow

```

--------------------------------

### Train LongSANA with Standard Configuration

Source: https://nvlabs.github.io/Sana/docs/longsana

Execute this command for standard LongSANA training. It utilizes a specific configuration file for the LongSANA setup.

```bash
torchrun --nnodes=8 --nproc_per_node=8 --rdzv_id=5235 \
  --rdzv_backend=c10d \
  --rdzv_endpoint $MASTER_ADDR \
  train_video_scripts/train_longsana.py \
  --config_path configs/sana_video_config/longsana/480ms/longsana.yaml \
  --wandb_name debug_480p_longsana --logdir output/debug_480p_longsana
```

--------------------------------

### Launch SGLang Server

Source: https://nvlabs.github.io/Sana/docs/sglang

Start the SGLang server to provide an OpenAI-compatible API for image generation. Specify the model path and host/port.

```bash
sglang serve --model-path Efficient-Large-Model/SANA1.5_1.6B_1024px_diffusers \
    --host 0.0.0.0 --port 30000

```

--------------------------------

### Load Quantized Sana Pipeline

Source: https://nvlabs.github.io/Sana/docs/8bit_sana

Load a Sana pipeline with 8-bit quantization for reduced memory usage. Ensure you have the 'bitsandbytes' library installed. This example uses 'cuda' if available, otherwise 'cpu'.

```python
import torch
from diffusers import BitsAndBytesConfig as DiffusersBitsAndBytesConfig, SanaTransformer2DModel, SanaPipeline
from transformers import BitsAndBytesConfig as BitsAndBytesConfig, AutoModel

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

quant_config = BitsAndBytesConfig(load_in_8bit=True)
text_encoder_8bit = AutoModel.from_pretrained(
    "Efficient-Large-Model/Sana_1600M_1024px_diffusers",
    subfolder="text_encoder",
    quantization_config=quant_config,
    torch_dtype=torch.float16,
)

quant_config = DiffusersBitsAndBytesConfig(load_in_8bit=True)
transformer_8bit = SanaTransformer2DModel.from_pretrained(
    "Efficient-Large-Model/Sana_1600M_1024px_diffusers",
    subfolder="transformer",
    quantization_config=quant_config,
    torch_dtype=torch.float16,
)

pipeline = SanaPipeline.from_pretrained(
    "Efficient-Large-Model/Sana_1600M_1024px_diffusers",
    text_encoder=text_encoder_8bit,
    transformer=transformer_8bit,
    torch_dtype=torch.float16,
    device_map="balanced",
)
pipeline.to(device)

prompt = "a tiny astronaut hatching from an egg on the moon"
image = pipeline(prompt).images[0]
image.save("sana.png")

```

--------------------------------

### Install Transformer Engine for NVFP4

Source: https://nvlabs.github.io/Sana/docs/sol_rl

Install transformer-engine with PyTorch support for NVFP4 path. Ensure it uses the same Python interpreter as torchrun.

```bash
python -m pip install --no-build-isolation "transformer-engine[pytorch]"
```

--------------------------------

### Install Latest Diffusers Library

Source: https://nvlabs.github.io/Sana/docs/installation

Upgrade the Hugging Face Diffusers library to the latest version to ensure compatibility with Sana.

```bash
pip install git+https://github.com/huggingface/diffusers

```

--------------------------------

### Run ComfyUI

Source: https://nvlabs.github.io/Sana/docs/ComfyUI/comfyui

Execute the main Python script to start the ComfyUI application. This command should be run from the ComfyUI directory.

```bash
python main.py
```

--------------------------------

### Quick Start with Sana and Diffusers for Image Generation

Source: https://nvlabs.github.io/Sana/docs/installation

Load the Sana pipeline using Diffusers and generate an image from a text prompt. Ensure your CUDA device is available and set to the correct dtype.

```python
import torch
from diffusers import SanaPipeline

pipe = SanaPipeline.from_pretrained(
    "Efficient-Large-Model/SANA1.5_1.6B_1024px_diffusers",
    torch_dtype=torch.bfloat16,
)
pipe.to("cuda")

pipe.vae.to(torch.bfloat16)
pipe.text_encoder.to(torch.bfloat16)

prompt = 'a cyberpunk cat with a neon sign that says "Sana"'
image = pipe(
    prompt=prompt,
    height=1024,
    width=1024,
    guidance_scale=4.5,
    num_inference_steps=20,
    generator=torch.Generator(device="cuda").manual_seed(42),
)[0]

image[0].save("sana.png")

```

--------------------------------

### Run LongSANA Video Pipeline in diffusers

Source: https://nvlabs.github.io/Sana/docs/longsana

Inference example using the LongSANA video pipeline from Hugging Face's diffusers library. Requires PyTorch and specifies model parameters, prompts, and video generation settings.

```python
import torch
from diffusers import LongSanaVideoPipeline, FlowMatchEulerDiscreteScheduler
from diffusers.utils import export_to_video

pipe = LongSanaVideoPipeline.from_pretrained(
    "Efficient-Large-Model/Sana-Video_2B_480p_LongLive_diffusers",
    torch_dtype=torch.bfloat16,
    base_chunk_frames=10,
    num_cached_blocks=-1,
)
pipe.scheduler = FlowMatchEulerDiscreteScheduler()
pipe.vae.to(torch.float32)
pipe.text_encoder.to(torch.bfloat16)
pipe.to("cuda")

prompt = "Evening, backlight, side lighting, soft light, high contrast, mid-shot, centered composition, clean solo shot, warm color. A young Caucasian man stands in a forest, golden light glimmers on his hair as sunlight filters through the leaves. He wears a light shirt, wind gently blowing his hair and collar, light dances across his face with his movements. The background is blurred, with dappled light and soft tree shadows in the distance. The camera focuses on his lifted gaze, clear and emotional."
negative_prompt = "Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality, low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards"

video = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    height=480,
    width=832,
    frames=161,
    guidance_scale=1.0,
    timesteps=[1000, 960, 889, 727, 0],
    generator=torch.Generator(device="cuda").manual_seed(42),
).frames[0]
export_to_video(video, "longsana.mp4", fps=16)
```

--------------------------------

### Load and Run Sana 4K Model

Source: https://nvlabs.github.io/Sana/docs/model_zoo

Loads the Sana 4K model for image generation. Ensure `pip install git+https://github.com/huggingface/diffusers` is run first. Adjust tile size for OOM issues.

```python
import torch
from diffusers import SanaPipeline

# 2K model: Efficient-Large-Model/Sana_1600M_2Kpx_BF16_diffusers
# 4K model:Efficient-Large-Model/Sana_1600M_4Kpx_BF16_diffusers
pipe = SanaPipeline.from_pretrained(
    "Efficient-Large-Model/Sana_1600M_4Kpx_BF16_diffusers",
    variant="bf16",
    torch_dtype=torch.bfloat16,
)
pipe.to("cuda")

pipe.vae.to(torch.bfloat16)
pipe.text_encoder.to(torch.bfloat16)

# for 4096x4096 image generation OOM issue, feel free adjust the tile size
if pipe.transformer.config.sample_size == 128:
    pipe.vae.enable_tiling(
        tile_sample_min_height=1024,
        tile_sample_min_width=1024,
        tile_sample_stride_height=896,
        tile_sample_stride_width=896,
    )
prompt = 'a cyberpunk cat with a neon sign that says "Sana"'
image = pipe(
    prompt=prompt,
    height=4096,
    width=4096,
    guidance_scale=5.0,
    num_inference_steps=20,
    generator=torch.Generator(device="cuda").manual_seed(42),
)[0]

image[0].save("sana_4K.png")
```

--------------------------------

### Train LongSANA with Self-Forcing Configuration

Source: https://nvlabs.github.io/Sana/docs/longsana

This command starts LongSANA training using the self-forcing configuration. It requires specifying the path to the self-forcing YAML configuration file.

```bash
torchrun --nnodes=8 --nproc_per_node=8 --rdzv_id=5235 \
  --rdzv_backend=c10d \
  --rdzv_endpoint $MASTER_ADDR \
  train_video_scripts/train_longsana.py \
  --config_path configs/sana_video_config/longsana/480ms/self_forcing.yaml \
  --wandb_name debug_480p_self_forcing --logdir output/debug_480p_self_forcing
```

--------------------------------

### Inference with SanaSprintPipeline using diffusers

Source: https://nvlabs.github.io/Sana/docs/sana_sprint

Load and use the SanaSprintPipeline from Hugging Face Hub for inference. Ensure you have the necessary libraries installed and a CUDA-enabled device for optimal performance. The VAE can be optionally replaced with a faster version like DC-AE-Lite.

```python
# test sana sprint
from diffusers import SanaSprintPipeline
import torch

pipeline = SanaSprintPipeline.from_pretrained(
    "Efficient-Large-Model/Sana_Sprint_1.6B_1024px_diffusers",
    torch_dtype=torch.bfloat16
)
# Use DC-AE-Lite for faster speed.
# from diffusers import AutoencoderDC
# vae = AutoencoderDC.from_pretrained("mit-han-lab/dc-ae-lite-f32c32-sana-1.1-diffusers")
# pipeline.vae = vae
pipeline.to("cuda:0")

prompt = "a tiny astronaut hatching from an egg on the moon"

image = pipeline(prompt=prompt, num_inference_steps=2).images[0]
image.save("test_out.png")

```

--------------------------------

### Install diffusers for LongSANA

Source: https://nvlabs.github.io/Sana/docs/longsana

Install the diffusers library to use LongSANA pipelines. This is a prerequisite for running the inference code.

```bash
pip install git+https://github.com/huggingface/diffusers
```

--------------------------------

### Run SANA Training with Specific Config

Source: https://nvlabs.github.io/Sana/docs/sol_rl

Launch SANA training using a specific configuration file and entry point.

```bash
CONFIG_SPEC=configs/sol_rl/sana.py:sana_diffusionnft_pickscore \
bash train_scripts/sol_rl/run_sana_single_node_8gpu.sh
```

--------------------------------

### Download SANA-WM-Bench Dataset

Source: https://nvlabs.github.io/Sana/docs/sana-wm-bench

Use the Hugging Face CLI to download the benchmark dataset. Ensure the `BENCH` environment variable is set to the download directory.

```bash
huggingface-cli download Efficient-Large-Model/SANA-WM-Bench \
  --repo-type dataset \
  --local-dir data/SANA-WM-Bench

```

```bash
BENCH=data/SANA-WM-Bench

```

--------------------------------

### Install diffusers for SanaVideoPipeline

Source: https://nvlabs.github.io/Sana/docs/sana_video

Upgrade your diffusers library to the latest version to ensure compatibility with SanaVideoPipeline.

```bash
pip install git+https://github.com/huggingface/diffusers

```

--------------------------------

### Run SD3 Training with Specific Config

Source: https://nvlabs.github.io/Sana/docs/sol_rl

Launch SD3 training using a specific configuration file and entry point.

```bash
CONFIG_SPEC=configs/sol_rl/sd3.py:sd3_compile_hpsv2 \
bash train_scripts/sol_rl/run_sd3_single_node_8gpu.sh
```

--------------------------------

### Launch 4bit Sana Online Demo

Source: https://nvlabs.github.io/Sana/docs/4bit_sana

Execute this command to launch the web-based demo for the 4bit Sana model.

```bash
python app/app_sana_4bit.py
```

--------------------------------

### Compare 4bit Sana with BF16 Version Demo

Source: https://nvlabs.github.io/Sana/docs/4bit_sana

Run this command to launch a demo that compares the performance of the 4bit Sana model against its BF16 counterpart.

```bash
python app/app_sana_4bit_compare_bf16.py
```

--------------------------------

### Build and Run Sana Docker Image

Source: https://nvlabs.github.io/Sana/docs/installation

Build a Docker image for Sana and run inference using the Docker container. Ensure the `--gpus all` flag is used for GPU access.

```bash
# Build Docker image
docker build -t sana .

# Run inference with Docker
docker run --gpus all -it sana python scripts/inference.py

```

--------------------------------

### Run FLUX1 Training with Specific Config

Source: https://nvlabs.github.io/Sana/docs/sol_rl

Launch FLUX1 training using a specific configuration file and entry point.

```bash
CONFIG_SPEC=configs/sol_rl/flux1.py:flux1_sol_rl_imagereward \
bash train_scripts/sol_rl/run_flux1_single_node_8gpu.sh
```

--------------------------------

### Run 720p Text-to-Video Inference

Source: https://nvlabs.github.io/Sana/docs/sana_video

Use this script for text-to-video generation with the 720p SANA-Video model. Ensure the configuration and model paths are correctly specified.

```bash
bash inference_video_scripts/inference_sana_video.sh \
      --np 1 \
      --config configs/sana_video_config/Sana_2000M_720px_ltx2vae_AdamW_fsdp.yaml \
      --model_path hf://Efficient-Large-Model/SANA-Video_2B_720p/checkpoints/SANA_Video_2B_720p.pth \
      --txt_file=asset/samples/video_prompts_samples.txt \
      --cfg_scale 6 \
      --motion_score 30 \
      --flow_shift 8 \
      --work_dir output/sana_t2v_720p_results

```

--------------------------------

### Run SANA Single-Node Training

Source: https://nvlabs.github.io/Sana/docs/sol_rl

Execute the default single-node launcher script for SANA training.

```bash
bash train_scripts/sol_rl/run_sana_single_node_8gpu.sh
```

--------------------------------

### Inference with SanaSprintPipeline from local repository

Source: https://nvlabs.github.io/Sana/docs/sana_sprint

Load and run the SanaSprintPipeline using its configuration and pre-trained weights directly from the repository. This method requires specifying the configuration file and the path to the pre-trained checkpoint.

```python
import torch
from app.sana_sprint_pipeline import SanaSprintPipeline
from torchvision.utils import save_image

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
generator = torch.Generator(device=device).manual_seed(42)

sana = SanaSprintPipeline("configs/sana_sprint_config/1024ms/SanaSprint_1600M_1024px_allqknorm_bf16_scm_ladd.yaml")
sana.from_pretrained("hf://Efficient-Large-Model/Sana_Sprint_1.6B_1024px/checkpoints/Sana_Sprint_1.6B_1024px.pth")

prompt = "a tiny astronaut hatching from an egg on the moon",

image = sana(
    prompt=prompt,
    height=1024,
    width=1024,
    guidance_scale=4.5,
    num_inference_steps=2,
    generator=generator,
)
save_image(image, 'sana_sprint.png', nrow=1, normalize=True, value_range=(-1, 1))

```

--------------------------------

### Train LongSANA with ODE Configuration

Source: https://nvlabs.github.io/Sana/docs/longsana

Use this command to initiate LongSANA training with the ODE configuration. Ensure the `data_path` in your config file is correctly set.

```bash
torchrun --nnodes=8 --nproc_per_node=8 --rdzv_id=5235 \
  --rdzv_backend=c10d \
  --rdzv_endpoint $MASTER_ADDR \
  train_video_scripts/train_longsana.py \
  --config_path configs/sana_video_config/longsana/480ms/ode.yaml \
  --wandb_name debug_480p_ode --logdir output/debug_480p_ode
```

--------------------------------

### Run SD3 Single-Node Training

Source: https://nvlabs.github.io/Sana/docs/sol_rl

Execute the default single-node launcher script for SD3 training.

```bash
bash train_scripts/sol_rl/run_sd3_single_node_8gpu.sh
```

--------------------------------

### Download Toy Dataset for FSDP Training

Source: https://nvlabs.github.io/Sana/docs/sana

Download a toy dataset from Hugging Face Hub for testing or demonstrating training with FSDP (Fully Sharded Data Parallel).

```bash
# Download toy dataset
huggingface-cli download Efficient-Large-Model/toy_data --repo-type dataset --local-dir ./data/toy_data

```

--------------------------------

### Run 720p Image-to-Video Inference

Source: https://nvlabs.github.io/Sana/docs/sana_video

This script is for image-to-video generation using the 720p SANA-Video model. It requires a text file specifying input images and task parameters.

```bash
bash inference_video_scripts/inference_sana_video.sh \
      --np 1 \
      --config configs/sana_video_config/Sana_2000M_720px_ltx2vae_AdamW_fsdp.yaml \
      --model_path hf://Efficient-Large-Model/SANA-Video_2B_720p/checkpoints/SANA_Video_2B_720p.pth \
      --txt_file=asset/samples/sample_i2v.txt \
      --task=ltx \
      --cfg_scale 6 \
      --motion_score 30 \
      --flow_shift 8 \
      --work_dir output/sana_ti2v_720p_results

```

--------------------------------

### Send Image Generation Request via OpenAI-Compatible API

Source: https://nvlabs.github.io/Sana/docs/sglang

Send a POST request to the running SGLang server's image generation endpoint using the OpenAI-compatible API format. This example uses the 'requests' library in Python.

```python
import requests

response = requests.post(
    "http://127.0.0.1:30000/v1/images/generations",
    json={
        "prompt": 'a cyberpunk cat with a neon sign that says "Sana"',
        "size": "1024x1024",
        "num_inference_steps": 20,
        "guidance_scale": 4.5,
        "seed": 42,
        "response_format": "b64_json",
        "n": 1,
    },
)

result = response.json()

```

--------------------------------

### Load and Run fp16 Sana Model with Diffusers

Source: https://nvlabs.github.io/Sana/docs/model_zoo

Use this snippet to load and run a Sana model with fp16 precision using the diffusers library. Ensure you have installed the diffusers library. Set `variant` and `torch_dtype` to `fp16` for fp16 models.

```python
# run `pip install git+https://github.com/huggingface/diffusers` before use Sana in diffusers
import torch
from diffusers import SanaPipeline

pipe = SanaPipeline.from_pretrained(
    "Efficient-Large-Model/Sana_1600M_1024px_diffusers",
    variant="fp16",
    torch_dtype=torch.float16,
)
pipe.to("cuda")

pipe.vae.to(torch.bfloat16)
pipe.text_encoder.to(torch.bfloat16)

prompt = 'a cyberpunk cat with a neon sign that says "Sana"'
image = pipe(
    prompt=prompt,
    height=1024,
    width=1024,
    guidance_scale=5.0,
    num_inference_steps=20,
    generator=torch.Generator(device="cuda").manual_seed(42),
)[0]

image[0].save("sana.png")

```

--------------------------------

### Load and Run bf16 Sana Model with Diffusers

Source: https://nvlabs.github.io/Sana/docs/model_zoo

This code snippet demonstrates how to load and run a Sana model with bf16 precision using the diffusers library. Install the diffusers library and set `variant` and `torch_dtype` to `bf16` for bf16 models. Note the use of `SanaPAGPipeline` and `pag_applied_layers`.

```python
# run `pip install git+https://github.com/huggingface/diffusers` before use Sana in diffusers
import torch
from diffusers import SanaPAGPipeline

pipe = SanaPAGPipeline.from_pretrained(
  "Efficient-Large-Model/Sana_1600M_1024px_BF16_diffusers",
  variant="bf16",
  torch_dtype=torch.bfloat16,
  pag_applied_layers="transformer_blocks.8",
)
pipe.to("cuda")

pipe.text_encoder.to(torch.bfloat16)
pipe.vae.to(torch.bfloat16)

prompt = 'a cyberpunk cat with a neon sign that says "Sana"'
image = pipe(
    prompt=prompt,
    guidance_scale=5.0,
    pag_scale=2.0,
    num_inference_steps=20,
    generator=torch.Generator(device="cuda").manual_seed(42),
)[0]
image[0].save('sana.png')

```

--------------------------------

### Run FLUX1 Single-Node Training

Source: https://nvlabs.github.io/Sana/docs/sol_rl

Execute the default single-node launcher script for FLUX1 training.

```bash
bash train_scripts/sol_rl/run_flux1_single_node_8gpu.sh
```

--------------------------------

### Run SANA-Video Inference CLI

Source: https://nvlabs.github.io/Sana/docs/sana_video_inference

Use this CLI command to perform video inference with the SANA-Video model. Specify configuration, model path, save path, and a detailed prompt.

```bash
python app/sana_video_pipeline.py \
        --config configs/sana_video_config/480ms/Sana_1600M_480px_adamW_fsdp.yaml \
        --model_path "hf://Efficient-Large-Model/SanaVideo_willquant/checkpoints/model.pth" \
        --save_path sana_video.mp4 \
        --prompt "In a whimsical forest setting, a small deer with antlers stands amidst oversized mushrooms and scattered carrots. The scene is vibrant with lush green moss and rocks, creating a magical atmosphere. The deer appears curious, moving slowly across the ground, surrounded by the towering fungi and colorful vegetables. The sky above is clear and bright, adding to the enchanting ambiance. A low-angle shot captures the deer's gentle exploration of this fantastical landscape."
```

--------------------------------

### Train Sana 0.6B from Scratch

Source: https://nvlabs.github.io/Sana/docs/sana

Train the Sana 0.6B model from scratch for 512x512 resolution. This command specifies the configuration, data directory, dataset type, and disables multi-scale training.

```bash
# Train Sana 0.6B with 512x512 resolution
bash train_scripts/train.sh \
    configs/sana_config/512ms/Sana_600M_img512.yaml \
    --data.data_dir="[asset/example_data]" \
    --data.type=SanaImgDataset \
    --model.multi_scale=false \
    --train.train_batch_size=32

```

--------------------------------

### Run SANA-Video Chunked Inference CLI

Source: https://nvlabs.github.io/Sana/docs/sana_video_inference

This CLI command is for chunked video inference using SANA-Video. It requires a specific chunk configuration, model path, save path, interval, image path, and a prompt.

```bash
python app/sana_video_pipeline.py \
        --config configs/sana_video_config/480ms/Sana_1600M_480px_adamW_fsdp_chunk.yaml \
        --model_path "hf://Efficient-Large-Model/SanaVideo_chunk/checkpoints/model.pth" \
        --save_path sana_video_chunk_i2v.mp4 \
        --interval_k 0.2 \
        --image_path output/tmp_videos/wan_goodcase_i2v_eval/00000000_video_001.jpg \
        --prompt "In a whimsical forest setting, a small deer with antlers stands amidst oversized mushrooms and scattered carrots. The scene is vibrant with lush green moss and rocks, creating a magical atmosphere. The deer appears curious, moving slowly across the ground, surrounded by the towering fungi and colorful vegetables. The sky above is clear and bright, adding to the enchanting ambiance. A low-angle shot captures the deer's gentle exploration of this fantastical landscape."
```

--------------------------------

### Run Sana Video + LTX2 Refiner Pipeline

Source: https://nvlabs.github.io/Sana/docs/sana_video

Execute the two-stage inference pipeline using SANA-Video for initial generation and LTX2 Refiner for upscaling to 2K quality. Specify model IDs, prompt, dimensions, and output path.

```python
python app/sana_video_refiner_pipeline_diffusers.py \
      --sana_model_id Efficient-Large-Model/SANA-Video_2B_720p_diffusers \
      --ltx2_model_id Lightricks/LTX-2 \
      --prompt "A cat and a dog baking a cake together in a kitchen." \
      --sana_height 704 \
      --sana_width 1280 \
      --sana_frames 81 \
      --output_path sana_ltx2_refined.mp4

```

--------------------------------

### Train 480p SANA-Video Model

Source: https://nvlabs.github.io/Sana/docs/sana_video

Use this script to pre-train a 480p video model. Ensure the data directory and working directory are correctly specified.

```bash
# 5s Video Model Pre-Training
bash train_video_scripts/train_video_ivjoint.sh \
      configs/sana_video_config/Sana_2000M_480px_AdamW_fsdp.yaml \
      --data.data_dir="[data/toy_data]" \
      --train.train_batch_size=1 \
      --work_dir=output/sana_video \
      --train.num_workers=10 \
      --train.visualize=true

```

--------------------------------

### Run One Scene with SANA-WM-Bench

Source: https://nvlabs.github.io/Sana/docs/sana-wm-bench

This script processes a single scene from the benchmark, extracting camera information and prompts, and then runs inference. Ensure all required environment variables are set.

```bash
BENCH=data/SANA-WM-Bench
SPLIT=benchmark_v2_smooth_60s
SCENE=game_style_001
WORK=results/sana_wm_bench_inputs/$SPLIT/$SCENE
mkdir -p "$WORK"
export BENCH SPLIT SCENE WORK

```

```python
import json
import os
from pathlib import Path
import numpy as np

bench = Path(os.environ["BENCH"])
split = os.environ["SPLIT"]
scene = os.environ["SCENE"]
work = Path(os.environ["WORK"])

manifest = bench / split / "sanawm_export_v2/run_manifest.jsonl"
rows = [json.loads(line) for line in manifest.read_text(encoding="utf-8").splitlines() if line.strip()]
row = next(r for r in rows if r["id"] == scene)
traj = np.load(bench / row["camera_path"])

np.save(work / f"{scene}_c2w.npy", traj["c2w"])
np.save(work / f"{scene}_intrinsics.npy", traj["intrinsics"])
(work / f"{scene}.txt").write_text(row["prompt"], encoding="utf-8")

```

```bash
python inference_video_scripts/wm/inference_sana_wm.py \
  --image "$BENCH/images/$SCENE.png" \
  --prompt "$WORK/$SCENE.txt" \
  --camera "$WORK/${SCENE}_c2w.npy" \
  --intrinsics "$WORK/${SCENE}_intrinsics.npy" \
  --num_frames 961 \
  --fps 16 \
  --step 60 \
  --cfg_scale 5.0 \
  --flow_shift 8.0 \
  --sampling_algo flow_euler_ltx \
  --seed 42 \
  --refiner_seed 42 \
  --no_action_overlay \
  --output_dir "results/sana_wm_bidirectional_refined/simple_60s" \
  --name "$SCENE"

```

```text
results/sana_wm_bidirectional_refined/simple_60s/<scene_id>_generated.mp4

```

--------------------------------

### Image-to-Video Inference using TXT file

Source: https://nvlabs.github.io/Sana/docs/sana_video

Execute image-to-video inference via a shell script. Key parameters include the configuration file, model path, text prompt file, task type ('ltx' for image-to-video), and output directory.

```bash
bash inference_video_scripts/inference_sana_video.sh \
      --np 1 \
      --config configs/sana_video_config/Sana_2000M_480px_AdamW_fsdp.yaml \
      --model_path hf://Efficient-Large-Model/SANA-Video_2B_480p/checkpoints/SANA_Video_2B_480p.pth \
      --txt_file=asset/samples/sample_i2v.txt \
      --task=ltx \
      --cfg_scale 6 \
      --motion_score 30 \
      --flow_shift 8 \
      --work_dir output/sana_ti2v_video_results

```

--------------------------------

### Launch DreamBooth LoRA Training for SANA

Source: https://nvlabs.github.io/Sana/docs/sana_lora_dreambooth

Launch the DreamBooth LoRA training script for SANA. This command includes settings for model name, dataset directory, output directory, mixed precision, instance prompt, resolution, batch size, gradient accumulation, optimizer, learning rate, reporting, scheduler, and push to hub.

```bash
export MODEL_NAME="Efficient-Large-Model/Sana_1600M_1024px_BF16_diffusers"
export INSTANCE_DIR="data/dreambooth/dog"
export OUTPUT_DIR="trained-sana-lora"

accelerate launch --num_processes 8 --main_process_port 29500 --gpu_ids 0,1,2,3 \
  train_scripts/train_dreambooth_lora_sana.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --instance_data_dir=$INSTANCE_DIR \
  --output_dir=$OUTPUT_DIR \
  --mixed_precision="bf16" \
  --instance_prompt="a photo of sks dog" \
  --resolution=1024 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --use_8bit_adam \
  --learning_rate=1e-4 \
  --report_to="wandb" \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --max_train_steps=500 \
  --validation_prompt="A photo of sks dog in a pond, yarn art style" \
  --validation_epochs=25 \
  --seed="0" \
  --push_to_hub
```

--------------------------------

### Run Pose Evaluation Script (8-GPU Slurm Job)

Source: https://nvlabs.github.io/Sana/docs/sana-wm-bench

Configure and submit a Slurm job for camera/pose accuracy evaluation using 8 GPUs. This script should be run from the dataset root directory. It utilizes the 'accelerate' library for distributed training and evaluates specified splits.

```bash
#!/usr/bin/env bash
#SBATCH --job-name=swm_pose80
#SBATCH --account=<account>
#SBATCH --partition=<gpu-partition>
#SBATCH --gres=gpu:8
#SBATCH --cpus-per-task=64
#SBATCH --mem=500G
#SBATCH --time=16:00:00
#SBATCH --output=logs/%x_%j.out
set -euo pipefail

PY=/path/to/python
ACCEL=/path/to/accelerate
BENCH=data/SANA-WM-Bench
METHOD_DIR=/path/to/results/sana_wm_bidirectional_refined

cd "$BENCH"
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

run_pose() {
  local split="$1"
  local split_dir="$2"
  "$ACCEL" launch --num_processes=8 --mixed_precision=bf16 \
    /path/to/eval_benchmark_poses.py \
    --result_folder "$METHOD_DIR/$split" \
    --manifest "$BENCH/$split_dir/sanawm_export_v2/run_manifest.jsonl" \
    --interval 4 \
    --batch_size 1 \
    --output "$METHOD_DIR/$split/eval_poses.json" \
    --skip_first_frame auto
}

run_pose simple_60s benchmark_v2_smooth_60s
run_pose hard_60s benchmark_v2_hard_60s

"$PY" /path/to/aggregate_results.py \
  --results_root "$(dirname "$METHOD_DIR")" \
  --json_out "$METHOD_DIR/metrics_80scene/aggregate_results.json" \
  --md_out "$METHOD_DIR/metrics_80scene/aggregate_results.md"
```

--------------------------------

### SFT Training with Image LoRA

Source: https://nvlabs.github.io/Sana/docs/sana_cosmos_rl

Use this command to perform Supervised Fine-Tuning (SFT) with LoRA for image models. Ensure the configuration file path is correct.

```bash
cosmos-rl --config ./configs/sana/sana-image-sft-lora.toml cosmos_rl.tools.dataset.diffusers_dataset
```

--------------------------------

### RL Training with Image NFT

Source: https://nvlabs.github.io/Sana/docs/sana_cosmos_rl

Execute this command for Reinforcement Learning (RL) training using the image NFT configuration. This command is suitable for image-based RL tasks.

```bash
cosmos-rl --config ./configs/sana/sana-image-nft.toml cosmos_rl.tools.dataset.diffusion_nft
```

--------------------------------

### Run SANA-WM Chunk-Causal Stage-1 Inference

Source: https://nvlabs.github.io/Sana/docs/sana_wm

Use this command to run inference with the chunk-causal Stage-1 teacher model. Ensure the refiner is enabled by default for qualitative results, or disable it with `--no_refiner` for debugging.

```bash
python inference_video_scripts/wm/inference_sana_wm.py \
  --config     hf://Efficient-Large-Model/SANA-WM_chunk_causal/config.yaml \
  --model_path hf://Efficient-Large-Model/SANA-WM_chunk_causal/dit/sana_wm_chunk_causal_1600m_720p.safetensors \
  --image      asset/sana_wm/demo_0.png \
  --prompt     asset/sana_wm/demo_0.txt \
  --action     "w-240,dw-120,w-120,aw-180,w-300" \
  --num_frames 961 \
  --offload_refiner \
  --output_dir results/sana_wm_chunk_causal

```

--------------------------------

### FSDP Training Script

Source: https://nvlabs.github.io/Sana/docs/sana

This script is for Fully Sharded Data Parallel (FSDP) training. It requires specifying FSDP-related flags and a compatible configuration file.

```bash
bash train_scripts/train.sh     configs/sana1-5_config/1024ms/Sana_1600M_1024px_AdamW_fsdp.yaml     --data.data_dir="[data/toy_data]"     --data.type=SanaWebDatasetMS     --model.multi_scale=true     --data.load_vae_feat=true     --train.use_fsdp=true     --train.train_batch_size=2
```