CogVideoX

https://github.com/zai-org/cogvideo
Admin
CogVideoX is an open-source video generation model providing various models for text-to-video,...
Tokens:12,112
Snippets:176
Trust Score:7.8
License:Apache-2.0
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install Dependencies

Source: https://github.com/zai-org/cogvideo/blob/main/tools/caption/README.md

Install the required packages for the captioning model by running this command.

```shell
pip install -r requirements.txt
```

--------------------------------

### Start Fine-tuning

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Run these commands to initiate the fine-tuning process on a single or multiple GPUs.

```bash
bash finetune_single_gpu.sh # Single GPU
bash finetune_multi_gpus.sh # Multi GPUs

```

--------------------------------

### CogVideoX Inference CLI Demo

Source: https://github.com/zai-org/cogvideo/blob/main/README.md

Example of how to perform inference using the CogVideoX model via a command-line interface. Ensure dependencies are installed according to requirements.txt.

```python
from diffusers import DiffusionPipeline
import torch

pipeline = DiffusionPipeline.from_pretrained("THUDM/cogvideo-base", torch_dtype=torch.float16, variant="fp16")
pipeline.enable_xformers_memory_efficient_attention()
pipeline.to("cuda")

prompt = "A majestic cat sitting on a throne."
video = pipeline(prompt, num_frames=16).frames

# Save video to disk
from decord import VideoWriter
from decord import cpu, gpu

vr = VideoWriter("output.mp4", width=512, height=512, fps=10)
for frame in video:
    vr.write_frame(frame)
vr.close()
```

--------------------------------

### Run Fine-tuning Script (Full)

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Execute this command to start full parameter fine-tuning with CogVideoX-2B. Ensure the correct config files are specified.

```bash
run_cmd="torchrun --standalone --nproc_per_node=8 train_video.py --base configs/cogvideox_2b.yaml configs/sft.yaml --seed $RANDOM"

```

--------------------------------

### Install VEnhancer Dependencies

Source: https://github.com/zai-org/cogvideo/blob/main/tools/venhancer/README.md

Clone the repository and install necessary dependencies, including PyTorch and other requirements. Ensure PyTorch version compatibility.

```shell
git clone https://github.com/Vchitect/VEnhancer.git
cd VEnhancer
## Torch and other dependencies can use those from CogVideoX. If you need to create a new environment, use the following commands:
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2

## Install required dependencies
pip install -r requirements.txt
```

--------------------------------

### Run Gradio Demo

Source: https://github.com/zai-org/cogvideo/blob/main/inference/gradio_composite_demo/README.md

Execute the main application script to launch the Gradio interface. This command starts the web server for the demo.

```bash
python app.py
```

--------------------------------

### Run Fine-tuning Script (Lora)

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Execute this command to start fine-tuning using Lora with CogVideoX-2B. Ensure the correct config files are specified.

```bash
run_cmd="torchrun --standalone --nproc_per_node=8 train_video.py --base configs/cogvideox_2b_lora.yaml configs/sft.yaml --seed $RANDOM"

```

--------------------------------

### Install Diffusers Dependencies

Source: https://github.com/zai-org/cogvideo/blob/main/finetune/README.md

Clone the diffusers repository and install it in editable mode. This is necessary as relevant code may not be in the official release.

```shell
git clone https://github.com/huggingface/diffusers.git
cd diffusers # Now on the Main branch
pip install -e .
```

--------------------------------

### Install Git LFS

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Ensure Git Large File Storage (LFS) is installed, which is recommended for handling large model weight files.

```bash
git lfs install
```

--------------------------------

### CogVideoX-5B Image-to-Video Inference

Source: https://github.com/zai-org/cogvideo/blob/main/README.md

Example code for generating videos from an initial image using the CogVideoX-5B model.

```python
from cogvideo import CogVideoXPipeline
from PIL import Image
import torch

pipe = CogVideoXPipeline.from_pretrained("cogvideo/cogvideox-5b", torch_dtype=torch.float16, variant="fp16")
pipe = pipe.to("cuda")

image = Image.open("input_image.png")

prompt = "A dog is playing fetch in a park."
video = pipe(prompt, image=image, num_frames=16).videos[0]

# Save the video
video.save_to_disk("dog_playing_fetch_from_image.mp4")
```

--------------------------------

### VEnhancer Runtime Logs Example

Source: https://github.com/zai-org/cogvideo/blob/main/tools/venhancer/README.md

Example of typical runtime logs when executing VEnhancer, including warnings and informational messages about model loading and checkpoint paths.

```shell
/share/home/zyx/.conda/envs/cogvideox/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
  @torch.library.impl_abstract("xformers_flash::flash_fwd")
/share/home/zyx/.conda/envs/cogvideox/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
  @torch.library.impl_abstract("xformers_flash::flash_bwd")
2024-08-20 13:25:17,553 - video_to_video - INFO - checkpoint_path: ./ckpts/venhancer_paper.pt
/share/home/zyx/.conda/envs/cogvideox/lib/python3.10/site-packages/open_clip/factory.py:88: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  checkpoint = torch.load(checkpoint_path, map_location=map_location)
2024-08-20 13:25:37,486 - video_to_video - INFO - Build encoder with FrozenOpenCLIPEmbedder
```

--------------------------------

### CogVideoX-5B Video-to-Video Inference

Source: https://github.com/zai-org/cogvideo/blob/main/README.md

Example code for transforming an input video based on a text prompt using the CogVideoX-5B model.

```python
from cogvideo import CogVideoXPipeline
from diffusers.utils import load_image
import torch

pipe = CogVideoXPipeline.from_pretrained("cogvideo/cogvideox-5b", torch_dtype=torch.float16, variant="fp16")
pipe = pipe.to("cuda")

video_path = "input_video.mp4"
video_frames = load_image(video_path)

prompt = "A cat is sleeping on a couch."
video = pipe(prompt, video=video_frames, num_frames=16).videos[0]

# Save the video
video.save_to_disk("cat_sleeping_on_couch.mp4")
```

--------------------------------

### CogVideoX-5B Text-to-Video Inference

Source: https://github.com/zai-org/cogvideo/blob/main/README.md

Example code for performing text-to-video generation using the CogVideoX-5B model. This is a direct inference script.

```python
from cogvideo import CogVideoXPipeline
import torch

pipe = CogVideoXPipeline.from_pretrained("cogvideo/cogvideox-5b", torch_dtype=torch.float16, variant="fp16")
pipe = pipe.to("cuda")

prompt = "A dog is playing fetch in a park."
video = pipe(prompt, num_frames=16).videos[0]

# Save the video
video.save_to_disk("dog_playing_fetch.mp4")
```

--------------------------------

### Run I2V SFT Fine-tuning

Source: https://github.com/zai-org/cogvideo/blob/main/finetune/README.md

Initiate Image-to-Video fine-tuning using the SFT method. Configure training options in the zero configuration file and bash script.

```bash
bash train_zero_i2v.sh  # Image-to-Video (I2V) fine-tuning
```

--------------------------------

### Run T2V SFT Fine-tuning

Source: https://github.com/zai-org/cogvideo/blob/main/finetune/README.md

Initiate Text-to-Video fine-tuning using the SFT method. Choose an appropriate configuration file and ensure parameters match the bash script.

```bash
bash train_zero_t2v.sh  # Text-to-Video (T2V) fine-tuning
```

--------------------------------

### Dataset Structure for Fine-tuning

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Organize your dataset with separate 'labels' and 'videos' directories. Each video file must have a corresponding '.txt' label file with the same base name.

```tree
. 
├── labels
│   ├── 1.txt
│   ├── 2.txt
│   ├── ...
└── videos
    ├── 1.mp4
    ├── 2.mp4
    ├── ...

```

--------------------------------

### Run Video Captioning Script

Source: https://github.com/zai-org/cogvideo/blob/main/tools/caption/README.md

Execute the main script to perform video captioning.

```shell
python video_caption.py
```

--------------------------------

### Dataset Structure for Fine-tuning

Source: https://github.com/zai-org/cogvideo/blob/main/finetune/README.md

Illustrates the expected directory structure for T2V and I2V datasets. Optional image files can be provided for I2V tasks.

```text
.
├── prompts.txt
├── videos
├── videos.txt
├── images     # (Optional) For I2V, if not provided, first frame will be extracted from video as reference
└── images.txt # (Optional) For I2V, if not provided, first frame will be extracted from video as reference
```

--------------------------------

### Run I2V LoRA Fine-tuning

Source: https://github.com/zai-org/cogvideo/blob/main/finetune/README.md

Execute the I2V fine-tuning script using LoRA. Configure parameters such as image column if using reference images.

```bash
bash train_ddp_i2v.sh  # Image-to-Video (I2V) fine-tuning
```

--------------------------------

### Clone and Prepare T5 Encoder Model

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Clone the T5 model from Hugging Face or Modelscope, then move the necessary files into a 't5-v1_1-xxl' directory. This T5 model is used as an encoder and does not require training.

```bash
git clone https://huggingface.co/THUDM/CogVideoX-2b.git # Download model from Huggingface
# git clone https://www.modelscope.cn/ZhipuAI/CogVideoX-2b.git # Download from Modelscope
mkdir t5-v1_1-xxl
mv CogVideoX-2b/text_encoder/* CogVideoX-2b/tokenizer/* t5-v1_1-xxl
```

--------------------------------

### Configure SAT Fine-tuning (Full)

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Modify this YAML configuration for full parameter fine-tuning. Ensure 'load' points to your Transformer model path.

```yaml
  # checkpoint_activations: True ## using gradient checkpointing (both `checkpoint_activations` in the config file need to be set to True)
  model_parallel_size: 1 # Model parallel size
  experiment_name: lora-disney  # Experiment name (do not change)
  mode: finetune # Mode (do not change)
  load: "{your_CogVideoX-2b-sat_path}/transformer" ## Path to Transformer model
  no_load_rng: True # Whether to load random number seed
  train_iters: 1000 # Training iterations
  eval_iters: 1 # Evaluation iterations
  eval_interval: 100    # Evaluation interval
  eval_batch_size: 1  # Evaluation batch size
  save: ckpts # Model save path
  save_interval: 100 # Save interval
  log_interval: 20 # Log output interval
  train_data: [ "your train data path" ]
  valid_data: [ "your val data path" ] # Training and validation sets can be the same
  split: 1,0,0 # Proportion for training, validation, and test sets
  num_workers: 8 # Number of data loader workers
  force_train: True # Allow missing keys when loading checkpoint (T5 and VAE loaded separately)
  only_log_video_latents: True # Avoid memory usage from VAE decoding
  deepspeed:
    bf16:
      enabled: False # For CogVideoX-2B Turn to False and For CogVideoX-5B Turn to True
    fp16:
      enabled: True  # For CogVideoX-2B Turn to True and For CogVideoX-5B Turn to False

```

--------------------------------

### Run Inference Script

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Execute the inference script using the bash command to perform video generation based on the configured settings.

```bash
inference.sh
```

--------------------------------

### Download CogVideoX-2B Model Weights

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Download and extract the VAE and Transformer model weights for CogVideoX-2B from Tsinghua Cloud. The VAE file is also used for the 5B model.

```bash
mkdir CogVideoX-2b-sat
cd CogVideoX-2b-sat
wget https://cloud.tsinghua.edu.cn/f/fdba7608a49c463ba754/?dl=1
mv 'index.html?dl=1' vae.zip
unzip vae.zip
wget https://cloud.tsinghua.edu.cn/f/556a3e1329e74f1bac45/?dl=1
mv 'index.html?dl=1' transformer.zip
unzip transformer.zip
```

--------------------------------

### Run Inference

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Execute the inference script after configuring it to use the fine-tuned model.

```bash
bash inference.sh

```

--------------------------------

### Set Input Type to Command-Line

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Change the input type to 'cli' to enable entering prompts directly from the command line during inference.

```yaml
input_type: cli
```

--------------------------------

### Export SAT Fine-tuned Lora Adapter

Source: https://github.com/zai-org/cogvideo/blob/main/README.md

Tool to export a fine-tuned SAT Lora adapter in diffusers format.

```python
from cogvideo.utils.export_lora import export_lora_adapter

# Load fine-tuned Lora weights
lora_weights = torch.load("sat_lora_finetuned.pth")

# Export to diffusers format
export_lora_adapter(lora_weights, "output_lora_adapter")
```

--------------------------------

### Clone CogVideoX1.5-5B Model Weights

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Clone the CogVideoX1.5-5B model weights from Hugging Face using git LFS. This downloads the Transformers, VAE, and T5 Encoder models.

```bash
git lfs install
git clone https://huggingface.co/THUDM/CogVideoX1.5-5B-SAT
```

--------------------------------

### Convert SAT Weights to Huggingface

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Run this Python script to convert SAT weight format to Huggingface's format.

```python
python ../tools/convert_weight_sat2hf.py

```

--------------------------------

### Configure Inference Script

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Modify the inference configuration file to use the fine-tuned Lora model. Replace placeholders with your model parameters.

```bash
run_cmd="$environs python sample_video.py --base configs/cogvideox_<model parameters>_lora.yaml configs/inference.yaml --seed 42"

```

--------------------------------

### Run T2V LoRA Fine-tuning

Source: https://github.com/zai-org/cogvideo/blob/main/finetune/README.md

Execute the T2V fine-tuning script using LoRA. Ensure configuration parameters like output directory, dataset root, and caption file are correctly set in the shell script.

```bash
bash train_ddp_t2v.sh  # Text-to-Video (T2V) fine-tuning
```

--------------------------------

### SAT to HF Lora Weight Mapping

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

This dictionary shows the mapping between SAT's Lora weight structure and Huggingface Diffusers' structure.

```python
'attention.query_key_value.matrix_A.0': 'attn1.to_q.lora_A.weight',
'attention.query_key_value.matrix_A.1': 'attn1.to_k.lora_A.weight',
'attention.query_key_value.matrix_A.2': 'attn1.to_v.lora_A.weight',
'attention.query_key_value.matrix_B.0': 'attn1.to_q.lora_B.weight',
'attention.query_key_value.matrix_B.1': 'attn1.to_k.lora_B.weight',
'attention.query_key_value.matrix_B.2': 'attn1.to_v.lora_B.weight',
'attention.dense.matrix_A.0': 'attn1.to_out.0.lora_A.weight',
'attention.dense.matrix_B.0': 'attn1.to_out.0.lora_B.weight'

```

--------------------------------

### Load Diffusers Version of Fine-tuned Lora Adapter

Source: https://github.com/zai-org/cogvideo/blob/main/README.md

Tool code for loading a fine-tuned Lora adapter in the diffusers format.

```python
from cogvideo.utils.load_lora import load_lora_adapter
from cogvideo import CogVideoXPipeline
import torch

# Load base model
pipe = CogVideoXPipeline.from_pretrained("cogvideo/cogvideox-5b", torch_dtype=torch.float16, variant="fp16")
pipe = pipe.to("cuda")

# Load Lora adapter
pipe = load_lora_adapter(pipe, "output_lora_adapter")

# Now use the pipe for inference with the loaded Lora adapter
```

--------------------------------

### Configure SAT Fine-tuning (Lora)

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Modify this YAML configuration for Lora fine-tuning. Uncomment 'lora_config' to enable Lora parameters.

```yaml
model:
  scale_factor: 1.55258426
  disable_first_stage_autocast: true
  not_trainable_prefixes: [ 'all' ] ## Uncomment to unlock
  log_keys:
    - txt

  lora_config: ## Uncomment to unlock
    target: sat.model.finetune.lora2.LoraMixin
    params:
      r: 256

```

--------------------------------

### Convert Input Text for CogVideoX

Source: https://github.com/zai-org/cogvideo/blob/main/README.md

Script to convert user input text into a format suitable for CogVideoX training. It uses an LLM (defaulting to GLM-4) to match the input distribution to the training data.

```python
from cogvideo.utils.convert_input import convert_input_to_long_form

input_text = "A short description of a video."
long_form_text = convert_input_to_long_form(input_text)

print(long_form_text)
```

--------------------------------

### CogVideoX-5B Quantized Text-to-Video Inference

Source: https://github.com/zai-org/cogvideo/blob/main/README.md

Code for quantized text-to-video inference, suitable for devices with lower memory. This script can be modified for FP8 precision.

```python
from cogvideo import CogVideoXPipeline
import torch

pipe = CogVideoXPipeline.from_pretrained("cogvideo/cogvideox-5b-int8", torch_dtype=torch.float16, variant="fp16")
pipe = pipe.to("cuda")

prompt = "A dog is playing fetch in a park."
video = pipe(prompt, num_frames=16).videos[0]

# Save the video
video.save_to_disk("dog_playing_fetch_quantized.mp4")
```

--------------------------------

### Modify Output Directory

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Update the 'output_dir' parameter to specify a custom location for saving generated videos.

```yaml
output_dir: outputs/
```

--------------------------------

### Export Lora Weights from SAT to Huggingface

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Use this command to export Lora weights trained with SAT to Huggingface Diffusers format. Specify the path to your SAT model states and the desired export directory.

```python
python tools/export_sat_lora_weight.py --sat_pt_path {args.save}/{experiment_name}-09-09-21-10/1000/mp_rank_00_model_states.pt --lora_save_directory   {args.save}/export_hf_lora_weights_1/

```

--------------------------------

### Configure Inference Settings in YAML

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Modify this YAML file to set inference parameters such as model path, input type, sampling settings, and output directory.

```yaml
args:
  latent_channels: 16
  mode: inference
  load: "{absolute_path/to/your}/transformer" # Absolute path to CogVideoX-2b-sat/transformer folder
  # load: "{your lora folder} such as zRzRzRzRzRzRzR/lora-disney-08-20-13-28" # This is for Full model without lora adapter

  batch_size: 1
  input_type: txt # You can choose "txt" for plain text input or change to "cli" for command-line input
  input_file: configs/test.txt # Plain text file, can be edited
  sampling_num_frames: 13  # For CogVideoX1.5-5B it must be 42 or 22. For CogVideoX-5B / 2B, it must be 13, 11, or 9.
  sampling_fps: 8
  fp16: True # For CogVideoX-2B
  # bf16: True # For CogVideoX-5B
  output_dir: outputs/
  force_inference: True
```

--------------------------------

### CogVideoX Model Configuration

Source: https://github.com/zai-org/cogvideo/blob/main/sat/README.md

Defines the core model architecture, including diffusion parameters, network layers, and attention mechanisms. Adjust scale_factor and disable_first_stage_autocast for specific training or inference needs.

```yaml
model:
  scale_factor: 1.55258426
  disable_first_stage_autocast: true
  log_keys:
    - txt

  denoiser_config:
    target: sgm.modules.diffusionmodules.denoiser.DiscreteDenoiser
    params:
      num_idx: 1000
      quantize_c_noise: False

      weighting_config:
        target: sgm.modules.diffusionmodules.denoiser_weighting.EpsWeighting
      scaling_config:
        target: sgm.modules.diffusionmodules.denoiser_scaling.VideoScaling
      discretization_config:
        target: sgm.modules.diffusionmodules.discretizer.ZeroSNRDDPMDiscretization
        params:
          shift_scale: 3.0

  network_config:
    target: dit_video_concat.DiffusionTransformer
    params:
      time_embed_dim: 512
      elementwise_affine: True
      num_frames: 49
      time_compressed_rate: 4
      latent_width: 90
      latent_height: 60
      num_layers: 30
      patch_size: 2
      in_channels: 16
      out_channels: 16
      hidden_size: 1920
      adm_in_channels: 256
      num_attention_heads: 30

      transformer_args:
        checkpoint_activations: True ## using gradient checkpointing
        vocab_size: 1
        max_sequence_length: 64
        layernorm_order: pre
        skip_init: false
        model_parallel_size: 1
        is_decoder: false

      modules:
        pos_embed_config:
          target: dit_video_concat.Basic3DPositionEmbeddingMixin
          params:
            text_length: 226
            height_interpolation: 1.875
            width_interpolation: 1.875

        patch_embed_config:
          target: dit_video_concat.ImagePatchEmbeddingMixin
          params:
            text_hidden_size: 4096

        adaln_layer_config:
          target: dit_video_concat.AdaLNMixin
          params:
            qk_ln: True

        final_layer_config:
          target: dit_video_concat.FinalLayerMixin

  conditioner_config:
    target: sgm.modules.GeneralConditioner
    params:
      emb_models:
        - is_trainable: false
          input_key: txt
          ucg_rate: 0.1
          target: sgm.modules.encoders.modules.FrozenT5Embedder
          params:
            model_dir: "t5-v1_1-xxl" # absolute path to CogVideoX-2b/t5-v1_1-xxl weight folder
            max_length: 226

  first_stage_config:
    target: vae_modules.autoencoder.VideoAutoencoderInferenceWrapper
    params:
      cp_size: 1
      ckpt_path: "CogVideoX-2b-sat/vae/3d-vae.pt" # absolute path to CogVideoX-2b-sat/vae/3d-vae.pt file
      ignore_keys: [ 'loss' ]

      loss_config:
        target: torch.nn.Identity

      regularizer_config:
        target: vae_modules.regularizers.DiagonalGaussianRegularizer

      encoder_config:
        target: vae_modules.cp_enc_dec.ContextParallelEncoder3D
        params:
          double_z: true
          z_channels: 16
          resolution: 256
          in_channels: 3
          out_ch: 3
          ch: 128
          ch_mult: [ 1, 2, 2, 4 ]
          attn_resolutions: [ ]
          num_res_blocks: 3
          dropout: 0.0
          gather_norm: True

      decoder_config:
        target: vae_modules.cp_enc_dec.ContextParallelDecoder3D
        params:
          double_z: True
          z_channels: 16
          resolution: 256
          in_channels: 3
          out_ch: 3
          ch: 128
          ch_mult: [ 1, 2, 2, 4 ]
          attn_resolutions: [ ]
          num_res_blocks: 3
          dropout: 0.0
          gather_norm: False

  loss_fn_config:
    target: sgm.modules.diffusionmodules.loss.VideoDiffusionLoss
    params:
      offset_noise_level: 0
      sigma_sampler_config:
        target: sgm.modules.diffusionmodules.sigma_sampling.DiscreteSampling
        params:
          uniform_sampling: True
          num_idx: 1000
          discretization_config:
            target: sgm.modules.diffusionmodules.discretizer.ZeroSNRDDPMDiscretization
            params:
              shift_scale: 3.0

  sampler_config:
    target: sgm.modules.diffusionmodules.sampling.VPSDEDPMPP2MSampler
    params:
      num_steps: 50
      verbose: True

      discretization_config:
        target: sgm.modules.diffusionmodules.discretizer.ZeroSNRDDPMDiscretization
        params:
          shift_scale: 3.0

      guider_config:
        target: sgm.modules.diffusionmodules.guiders.DynamicCFG
        params:
          scale: 6
          exp: 5
          num_steps: 50

```

--------------------------------

### Parallel Inference with xDiT

Source: https://github.com/zai-org/cogvideo/blob/main/README.md

Utilizes xDiT to parallelize video generation across multiple GPUs.

```python
from cogvideo.parallel_inference.parallel_inference_xdit import parallel_inference
import torch

# Assuming you have a model and prompt
model = "cogvideo/cogvideox-5b"
prompt = "A futuristic cityscape at sunset."
output_path = "parallel_video.mp4"

# Run parallel inference
parallel_inference(model, prompt, output_path, num_frames=32, num_gpus=4)
```

--------------------------------

### CogVLM2-Video Citation

Source: https://github.com/zai-org/cogvideo/blob/main/tools/caption/README.md

BibTeX entry for citing the CogVLM2-Video model in research.

```bibtex
@article{hong2024cogvlm2,
  title={CogVLM2: Visual Language Models for Image and Video Understanding},
  author={Hong, Wenyi and Wang, Weihan and Ding, Ming and Yu, Wenmeng and Lv, Qingsong and Wang, Yan and Cheng, Yean and Huang, Shiyu and Ji, Junhui and Xue, Zhao and others},
  journal={arXiv preprint arXiv:2408.16500},
  year={2024}
}
```

--------------------------------

### Enable Diffusers Optimizations

Source: https://github.com/zai-org/cogvideo/blob/main/README.md

These optimizations are enabled by default when testing with the diffusers library. They are generally compatible with NVIDIA Ampere architecture and above. Disabling them increases memory consumption but also speeds up inference.

```python
pipe.enable_sequential_cpu_offload()
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()
```

--------------------------------

### CogVLM2-Caption Citation

Source: https://github.com/zai-org/cogvideo/blob/main/tools/caption/README.md

BibTeX entry for citing the CogVLM2-Caption model in research.

```bibtex
@article{yang2024cogvideox,
  title={CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer},
  author={Yang, Zhuoyi and Teng, Jiayan and Zheng, Wendi and Ding, Ming and Huang, Shiyu and Xu, Jiazheng and Yang, Yuanming and Hong, Wenyi and Zhang, Xiaohan and Feng, Guanyu and others},
  journal={arXiv preprint arXiv:2408.06072},
  year={2024}
}
```
Tokens:
Raw