### Start CausalVAE Training

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/docs/VAE.md

Execute the training script for CausalVAE. Key arguments control training size, data processing, and model loading.

```shell
bash scripts/causalvae/train.sh
```

--------------------------------

### Commit Message Format Example

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/docs/Contribution_Guidelines.md

Illustrates the required format for commit messages, which includes a type (e.g., docs, feat, fix, refactor) and a summary of the changes.

```bash
[<type>]: <summary>
  │        │
  │        └─⫸ Briefly describe your changes, without ending with a period.
  │
  └─⫸ Commit Type: |docs|feat|fix|refactor|
```

--------------------------------

### Prepare Videos for Evaluation

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/docs/VAE.md

Run the script to prepare videos for the evaluation step. Environment variables like experiment name, video parameters, and checkpoint path are used for control.

```bash
# Experiment name
EXP_NAME=wfvae
# Video parameters
SAMPLE_RATE=1
NUM_FRAMES=33
RESOLUTION=256
# Model weights
CKPT=ckpt
# Select subset size (0 for full set)
SUBSET_SIZE=0
# Dataset directory
DATASET_DIR=test_video

bash scripts/causalvae/prepare_eval.sh
```

--------------------------------

### Training Dataset Structure

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/docs/VAE.md

Organize your training videos recursively in a directory for convenient use with multiple datasets.

```shell
Training Dataset
|——sub_dataset1
    |——sub_sub_dataset1
        |——video1.mp4
        |——video2.mp4
        ......
    |——sub_sub_dataset2
        |——video3.mp4
        |——video4.mp4
        ......
|——sub_dataset2
    |——video5.mp4
    |——video6.mp4
    ......
|——video7.mp4
|——video8.mp4

```

--------------------------------

### Train Prompt Refiner Model

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/docs/Prompt_Refiner.md

Use this command to train the prompt refiner model. Ensure you provide the correct paths for the data, base model, and LoRA output.

```bash
cd opensora/models/prompt_refiner
CUDA_VISIBLE_DEVICES=0 python train.py \
    --data_path path/to/data.json \
    --model_path path/to/llama_model \
    --lora_out_path path/to/save/lora_model
```

--------------------------------

### Evaluate Video Metrics

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/docs/VAE.md

Execute the script to evaluate video metrics. Environment variables specify the experiment name, video parameters, evaluation metric, and paths to ground truth and reconstructed videos.

```bash
# Experiment name
EXP_NAME=wfvae-4dim
# Video parameters
SAMPLE_RATE=1
NUM_FRAMES=33
RESOLUTION=256
# Evaluation metric
METRIC=lpips
# Select subset size (0 for full set)
SUBSET_SIZE=0
# Path to the ground truth videos, which can be saved during video reconstruction by setting `--output_origin`
ORIGIN_DIR=video_gen/${EXP_NAME}_sr${SAMPLE_RATE}_nf${NUM_FRAMES}_res${RESOLUTION}_subset${SUBSET_SIZE}/origin
# Path to the reconstructed videos
RECON_DIR=video_gen/${EXP_NAME}_sr${SAMPLE_RATE}_nf${NUM_FRAMES}_res${RESOLUTION}_subset${SUBSET_SIZE}

bash scripts/causalvae/eval.sh
```

--------------------------------

### Run Video Reconstruction for Inference

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/docs/VAE.md

Initiate video reconstruction using the CausalVAE model for inference. Important arguments control output video size, data paths, and model loading.

```shell
bash scripts/causalvae/rec_video.sh
```

--------------------------------

### Clone Forked Repository

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/docs/Contribution_Guidelines.md

Clone your forked repository to your local machine to begin making changes.

```bash
git clone [your-forked-repository-url]
```

--------------------------------

### Add Upstream Remote

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/docs/Contribution_Guidelines.md

Add the original Open-Sora Plan repository as a remote named 'upstream' to easily sync with the latest updates.

```bash
git remote add upstream https://github.com/PKU-YuanGroup/Open-Sora-Plan
```

--------------------------------

### Run Frame Interpolation Script

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/models/frame_interpolation/readme.md

Execute the frame interpolation script using the AMT model. Ensure you have downloaded the pretrained weights and specify the correct paths for the checkpoint, input video, and output folder.

```python
python opensora/models/frame_interpolation/interpolation.py --ckpt /path/to/ckpt --niters 1 --input /path/to/input/video.mp4 --output_path /path/to/output/floder --frame_rate 30
```

--------------------------------

### Commit Changes with Type

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/docs/Contribution_Guidelines.md

Commit your changes using the specified format, including a commit type (e.g., 'docs') and a brief summary of the modifications.

```bash
git commit -m "[docs]: xxxx"
```

--------------------------------

### Inference with Prompt Refiner

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/docs/Prompt_Refiner.md

Run inference to refine a given prompt using the trained or merged model. Provide the path to the model weights and the prompt text.

```bash
cd opensora/models/prompt_refiner
CUDA_VISIBLE_DEVICES=0 python merge.py \
    --mode_path path/to/data.json \
    --prompt path/to/save/lora_model
```

--------------------------------

### Prompt Refiner Dataset Structure

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/docs/Prompt_Refiner.md

The dataset for the prompt refiner is a JSON file containing instruction, input, and output fields. This structure is used for training the refiner model.

```json
[
  {
    "instruction": "Refine the sentence: \"A newly married couple sharing a piece of there wedding cake.\" to contain subject description, action, scene description. (Optional: camera language, light and shadow, atmosphere) and conceive some additional actions to make the sentence more dynamic. Make sure it is a fluent sentence, not nonsense.",
    "input": "",
    "output": "The newlywed couple, dressed in elegant attire..."
  },
  ...
]
```

--------------------------------

### BibTeX Entry for WF-VAE Paper

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/README.md

This BibTeX entry provides citation details for the WF-VAE paper, focusing on enhancing video VAE with wavelet-driven energy flow. It includes title, authors, journal, and year.

```bibtex
@article{li2024wf,
  title={WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Model},
  author={Li, Zongjian and Lin, Bin and Ye, Yang and Chen, Liuhan and Cheng, Xinhua and Yuan, Shenghai and Yuan, Li},
  journal={arXiv preprint arXiv:2411.17459},
  year={2024}
}
```

--------------------------------

### Sync Code from Upstream

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/docs/Contribution_Guidelines.md

Fetch the latest code from the upstream repository, merge it into your local main branch, and push the synchronized branch to your forked remote repository.

```bash
git fetch upstream
git checkout main
git merge upstream/main
git push origin main
```

--------------------------------

### Push Changes to Fork

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/docs/Contribution_Guidelines.md

Push your committed changes from your local feature branch to your forked repository on GitHub.

```bash
git push origin my-docs-branch
```

--------------------------------

### Create Feature Branch

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/docs/Contribution_Guidelines.md

Create a new branch in your forked repository for your specific changes, ensuring the branch name is descriptive and based on the main branch.

```bash
git checkout -b my-docs-branch main
```

--------------------------------

### BibTeX Entry for Open-Sora Plan Paper

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/README.md

This BibTeX entry can be used to cite the Open-Sora Plan paper in academic works. It includes title, authors, journal, and year.

```bibtex
@article{lin2024open,
  title={Open-Sora Plan: Open-Source Large Video Generation Model},
  author={Lin, Bin and Ge, Yunyang and Cheng, Xinhua and Li, Zongjian and Zhu, Bin and Wang, Shaodong and He, Xianyi and Ye, Yang and Yuan, Shenghai and Chen, Liuhan and others},
  journal={arXiv preprint arXiv:2412.00131},
  year={2024}
}
```

--------------------------------

### Merge LoRA Model with Base Model

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/docs/Prompt_Refiner.md

This command merges a pre-trained LoRA model with the base LLaMA model. Specify the paths for the base model, LoRA input, and the desired output path for the merged model.

```bash
cd opensora/models/prompt_refiner
CUDA_VISIBLE_DEVICES=0 python merge.py \
    --base_path path/to/llama_model \
    --lora_in_path path/to/save/lora_model \
    --lora_out_path path/to/save/merge_model
```

--------------------------------

### BibTeX Entry for Helios Paper

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/README.md

This BibTeX entry is for citing the Helios paper, a real-time long video generation model. It details the title, authors, journal, and publication year.

```bibtex
@article{helios,
  title={Helios: Real Real-Time Long Video Generation Model},
  author={Yuan, Shenghai and Yin, Yuanyang and Li, Zongjian and Huang, Xinwei and Yang, Xiao and Yuan, Li},
  journal={arXiv preprint arXiv:2603.04379},
  year={2026}
}
```

--------------------------------

### Adaptive Grad Clipping with EMA

Source: https://github.com/pku-yuangroup/open-sora-plan/blob/main/docs/Report-v1.5.0.md

This snippet implements an Adaptive Grad Clipping strategy using an Exponential Moving Average (EMA) to dynamically maintain the gradient norm threshold. It adjusts gradient clipping based on the current gradient norm relative to a calculated threshold, improving stability during diffusion model training.

```python
clip_threshold = moving_avg_max_grad_norm + 3.0 * (moving_avg_max_grad_norm_var ** 0.5)
if grad_norm <= clip_threshold:
    # If the gradient norm is below the clipping threshold, the parameters are updated normally at this step, and both the moving_avg_max_grad_norm and moving_avg_max_grad_norm_var are updated accordingly.
    moving_avg_max_grad_norm = ema_decay * moving_avg_max_grad_norm + (1 - ema_decay) * grad_norm
    max_grad_norm_var = (moving_avg_max_grad_norm - grad_norm) ** 2
    moving_avg_max_grad_norm_var = ema_decay * moving_avg_max_grad_norm_var + (1 - ema_decay) * max_grad_norm_var
    # update weights...
else:
    # If the gradient norm exceeds the clipping threshold, the gradients are first clipped to reduce the norm to the threshold value before updating the parameters.
    clip_coef = grad_norm / clip_threshold
    grads = clip(grads, clip_coef) # clipping grads
    # update weights...
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.