SCAIL-2

https://github.com/zai-org/scail-2
Admin
SCAIL-2 is an open-source model and framework that unifies controlled character animation and...
Tokens:3,653
Snippets:45
Trust Score:7.8
License:Apache-2.0
Update:2 months ago
Show doc for...
Tokens:
Raw
### Setup SCAIL-Pose Environment

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Navigate to the submodule directory and install its specific requirements.

```bash
cd SCAIL-Pose
pip install -r requirements.txt
```

--------------------------------

### Install Training Dependencies

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Install required packages for video processing and tensor compression.

```sh
pip install decord zstandard
```

--------------------------------

### File Structure Example

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Expected directory structure for the downloaded SCAIL-2 model files.

```text
SCAIL-2/
├── Wan2.1_VAE.pth
├── model
│   ├── 1
│   │   └── fsdp2_rank_0000_checkpoint.pt
│   └── latest
└── umt5-xxl
    ├── ...
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Install the required Python packages for the project.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install Gemini SDK

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Install the Google GenAI library required for the prompt enhancer.

```bash
pip install google-genai
```

--------------------------------

### Initialize SCAIL-Pose Submodule

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Initialize and update the SCAIL-Pose submodule after cloning the repository.

```bash
git submodule update --init --recursive
```

--------------------------------

### Initiate SAT Training

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Switch to the training branch and execute the training scripts.

```sh
git checkout sat-scail2
export SCAIL2_POSE_LATENT_TRAIN_DIR=/path/to/pose_latent_train
bash scripts/train_mpi_14Bsc_xc_latent_example.sh
# or
bash scripts/train_mpi_14Bsc_xc_latent_fsdp_example.sh
```

--------------------------------

### Download Model Weights

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Use the Hugging Face CLI to download the SCAIL-2 model weights.

```bash
hf download zai-org/SCAIL-2
```

--------------------------------

### Run Animation Preprocessing

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Execute the all-in-one preprocessing script for animation inputs in either end-to-end or pose-driven mode.

```bash
# Recommended end-to-end mode: rendered_v2.mp4 is the driving video copy,
# and the mask video is generated from SAM3 masks.
python NLFPoseExtract/process_animation_aio.py --subdir /path/to/input --e2e_mode

# Pose-driven mode: runs NLF + DWPose and writes a skeleton render.
python NLFPoseExtract/process_animation_aio.py --subdir /path/to/input
```

--------------------------------

### Run Character Replacement Preprocessing

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Execute the preprocessing script for character replacement, with an optional flag for multi-person video handling.

```bash
python NLFPoseExtract/process_replacement.py --subdir /path/to/input

# If the driving video has multiple people and only one should be replaced:
python NLFPoseExtract/process_replacement.py --subdir /path/to/input --matchnearest
```

--------------------------------

### Set Gemini API Key

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Configure the environment variable for the Gemini API key.

```bash
export GEMINI_API_KEY=your_api_key
```

--------------------------------

### Define Training Case Directory Structure

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Required file structure for each training case directory.

```text
case_dir/
├── GT.mp4                  # target video to reconstruct / train on
├── rendered_v2.mp4         # driving video or pose-rendered video
├── rendered_mask_v2.mp4    # driving mask video; replace_mask.mp4 is also supported
├── ref.jpg                 # reference image
├── ref_mask.jpg            # reference mask image
└── prompt.txt              # optional caption; text.txt / caption.txt / recaption.txt also work
```

--------------------------------

### Cache Training Cases into WebDataset Shards

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Commands to cache individual cases or a list of cases into .tar WebDataset shards.

```sh
python cache_scail2_wds.py \
  --input_dir /path/to/case_dir \
  --output /path/to/pose_latent_train/000000.tar \
  --ckpt_dir /path/to/SCAIL-2 \
  --target_h 512 \
  --target_w 896 \
  --max_frames 81

python cache_scail2_wds.py \
  --input_list /path/to/train_cases.txt \
  --output /path/to/pose_latent_train/000000.tar \
  --ckpt_dir /path/to/SCAIL-2 \
  --target_h 512 \
  --target_w 896 \
  --max_frames 81
```

--------------------------------

### SCAIL-2 Input Directory Structure

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Required file structure for running a SCAIL-2 inference job.

```text
examples/001/
├── ref.jpg                 # reference character image
├── ref_mask.jpg            # foreground mask of the reference image
├── rendered_v2.mp4         # driving / pose video consumed by --pose
└── rendered_mask_v2.mp4    # per-frame driving mask consumed by --mask_video
```

--------------------------------

### Run SCAIL-2 Inference

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Execute the main generation script using the enhanced prompt for character replacement.

```bash
python generate.py \
    --model SCAIL-14B \
    --ckpt_dir /path/to/SCAIL-2 \
    --scail_path /path/to/SCAIL-2.safetensors \
    --replace_flag \
    --target_w 896 --target_h 512 \
    --image /path/to/ref.png \
    --mask_image /path/to/ref_mask.png \
    --pose /path/to/driving.mp4 \
    --mask_video /path/to/replace_mask.mp4 \
    --prompt "$(cat enhanced_prompt.txt)" \
    --save_file replacement_output.mp4
```

--------------------------------

### Run Multi-Reference Inference

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Executes the generation script with multiple reference images and masks to provide additional visual evidence.

```bash
python generate.py \
    --model SCAIL-14B \
    --ckpt_dir /path/to/SCAIL-2 \
    --scail_path /path/to/SCAIL-2.safetensors \
    --target_w 896 --target_h 512 \
    --image examples/animation_003_multi_ref/ref.png \
    --mask_image examples/animation_003_multi_ref/ref_mask.jpg \
    --pose examples/animation_003_multi_ref/rendered_v2.mp4 \
    --mask_video examples/animation_003_multi_ref/rendered_mask_v2.mp4 \
    --additional_ref_image \
        examples/animation_003_multi_ref/background.png \
        examples/animation_003_multi_ref/character_1.png \
        examples/animation_003_multi_ref/character_0.png \
    --additional_ref_mask_image \
        examples/animation_003_multi_ref/background_mask.png \
        examples/animation_003_multi_ref/character_1_mask.png \
        examples/animation_003_multi_ref/character_0_mask.png \
    --prompt "An anime style character with yellow hair, wearing a white and green sailor uniform and a green skirt, is dancing in a warm anime-style classroom." \
    --save_file output_multi_ref.mp4
```

--------------------------------

### Run Single-GPU Inference

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Execute inference for animation or replacement tasks using generate.py with specific model paths and input assets.

```bash
python generate.py \
    --model SCAIL-14B \
    --ckpt_dir /path/to/SCAIL-2 \
    --scail_path /path/to/SCAIL-2.safetensors \
    --target_w 896 --target_h 512 \
    --image examples/001/ref.jpg \
    --mask_image examples/001/ref_mask.jpg \
    --pose examples/001/rendered_v2.mp4 \
    --mask_video examples/001/rendered_mask_v2.mp4 \
    --prompt "The girl is dancing" \
    --save_file output.mp4
```

```bash
python generate.py \
    --model SCAIL-14B \
    --ckpt_dir /path/to/SCAIL-2 \
    --scail_path /path/to/SCAIL-2.safetensors \
    --target_w 896 --target_h 512 \
    --image examples/replace_001/ref.png \
    --mask_image examples/replace_001/ref_mask.png \
    --pose examples/replace_001/rendered_v2.mp4 \
    --mask_video examples/replace_001/replace_mask.mp4 \
    --prompt "A blond white male wearing a black suit, trousers, and leather shoes is playing the violin on the street while pedestrians walk past him." \
    --save_file output.mp4 \
    --replace_flag
```

--------------------------------

### Integrate LoRA Checkpoints

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Apply a LoRA checkpoint during inference by specifying the path and alpha strength parameters.

```bash
python generate.py \
    --model SCAIL-14B \
    --ckpt_dir /path/to/SCAIL-2 \
    --scail_path /path/to/SCAIL-2.safetensors \
    --lora_path Lightx2v/lightx2v_I2V_14B_480p_cfg_step_distill_rank128_bf16.safetensors \
    --lora_alpha 1.0 \
    --sample_steps 8 \
    --sample_shift 1 \
    --sample_guide_scale 1.0 \
    --target_w 896 --target_h 512 \
    --image examples/001/ref.jpg \
    --mask_image examples/001/ref_mask.jpg \
    --pose examples/001/rendered_v2.mp4 \
    --mask_video examples/001/rendered_mask_v2.mp4 \
    --prompt "The girl is dancing" \
    --save_file output.mp4
```

--------------------------------

### Run Prompt Enhancer

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Execute the prompt enhancer script to generate a detailed prompt for character replacement.

```bash
python prompt_enhancer.py \
    --video /path/to/driving.mp4 \
    --image /path/to/ref.png \
    --instruction "replace the man in the blue jacket in the video with the person in the image" \
    --examples prompt_examples.txt \
    --num_frames 8 \
    --output enhanced_prompt.txt \
    --caption_out source_caption.txt
```

--------------------------------

### Convert Relighting LoRA

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Convert a SAT format LoRA checkpoint to safetensors format for compatibility with the wan branch.

```bash
python convert_lora.py --input /path/to/relighting-lora.pt --output /path/to/relighting_lora.safetensors
```

--------------------------------

### Convert Model to Safetensors

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Convert the model weights from the 'sat' branch format to 'safetensors' for use in the 'wan' branch.

```python
python convert.py --scail-dir /path/to/SCAIL-2 --save-path /path/to/SCAIL-2.safetensors
```

--------------------------------

### Cite SCAIL-2 in BibTeX

Source: https://github.com/zai-org/scail-2/blob/wan-scail2/README.md

Use this BibTeX entry to cite the SCAIL-2 research paper in academic publications.

```bibtex
@misc{yan2026scail2,
      title={SCAIL-2: Unifying Controlled Character Animation with End-to-end In-Context Conditioning}, 
      author={Wenhao Yan and Fengjia Guo and Zhuoyi Yang and Jie Tang},
      year={2026},
      eprint={2606.10804},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2606.10804}, 
}
```
Tokens:
Raw