### Start Training Source: https://context7.com/lucidrains/phenaki-pytorch/llms.txt Initiates the training process for the model. Can be configured to train only the generator or critic phases. ```python trainer.train() ``` ```python trainer.train(only_train_generator=True) ``` ```python trainer.train(only_train_critic=True) ``` -------------------------------- ### Install phenaki-pytorch Source: https://github.com/lucidrains/phenaki-pytorch/blob/main/README.md Install the phenaki-pytorch library using pip. ```bash pip install phenaki-pytorch ``` -------------------------------- ### Phenaki.sample - Generate Video with Scene Continuation Source: https://context7.com/lucidrains/phenaki-pytorch/llms.txt Demonstrates generating a video with scene continuation by sampling the first part of a video. This can be used as a starting point for subsequent generation steps to maintain scene consistency. ```python import torch from phenaki_pytorch import CViViT, MaskGit, Phenaki # Assuming phenaki is initialized as in the previous example # cvivit = CViViT(...) # maskgit = MaskGit(...) # phenaki = Phenaki(cvivit=cvivit, maskgit=maskgit).cuda() # Generate with scene continuation using prime frames video_scene1 = phenaki.sample(texts='a squirrel examines an acorn', num_frames=17) ``` -------------------------------- ### MaskGit Forward Pass with Video Tokens and Text Context Source: https://context7.com/lucidrains/phenaki-pytorch/llms.txt Demonstrates a forward pass through the MaskGit model using video tokens and text embeddings. Includes an example of using classifier-free guidance for improved results. Ensure CUDA is available for .cuda() calls. ```python import torch # Assuming maskgit is already initialized # maskgit = MaskGit(...).cuda() batch_size = 2 seq_len = 64 video_tokens = torch.randint(0, 65536, (batch_size, seq_len)).cuda() text_embeddings = torch.randn(batch_size, 128, 768).cuda() # From T5 encoder text_mask = torch.ones(batch_size, 128).bool().cuda() # Get logits for masked token prediction logits = maskgit( video_tokens, context=text_embeddings, text_mask=text_mask, video_patch_shape=(4, 4, 4), # (frames, height, width) in patches cond_drop_prob=0.1 # Dropout for classifier-free guidance training ) print(f"Logits shape: {logits.shape}") # (2, 64, 65536) # Forward with classifier-free guidance scaling logits_cfg = maskgit.forward_with_cond_scale( video_tokens, context=text_embeddings, text_mask=text_mask, video_patch_shape=(4, 4, 4), cond_scale=3.0 # Guidance scale ) ``` -------------------------------- ### Get T5 Model Embedding Dimension Source: https://context7.com/lucidrains/phenaki-pytorch/llms.txt Retrieves the embedding dimension for a specified T5 model. Useful for understanding model capacity. ```python dim = get_encoded_dim('google/t5-v1_1-base') # Returns 768 ``` ```python dim_large = get_encoded_dim('google/t5-v1_1-large') # Returns 1024 ``` -------------------------------- ### Initialize Phenaki Model for Video Generation Source: https://github.com/lucidrains/phenaki-pytorch/blob/main/README.md Load a pre-trained C-ViViT model and initialize the MaskGit model. Then, combine them into the Phenaki model for text-guided video generation. The model can handle videos of varying lengths and can be co-trained with images. ```python import torch from phenaki_pytorch import CViViT, MaskGit, Phenaki cvivit = CViViT( dim = 512, codebook_size = 65536, image_size = (256, 128), # video with rectangular screen allowed patch_size = 32, temporal_patch_size = 2, spatial_depth = 4, temporal_depth = 4, dim_head = 64, heads = 8 ) cvivit.load('/path/to/trained/cvivit.pt') maskgit = MaskGit( num_tokens = 5000, max_seq_len = 1024, dim = 512, dim_context = 768, depth = 6, ) phenaki = Phenaki( cvivit = cvivit, maskgit = maskgit ).cuda() videos = torch.randn(3, 3, 17, 256, 128).cuda() # (batch, channels, frames, height, width) mask = torch.ones((3, 17)).bool().cuda() # [optional] (batch, frames) - allows for co-training videos of different lengths as well as video and images in the same batch texts = [ 'a whale breaching from afar', 'young girl blowing out candles on her birthday cake', 'fireworks with blue and green sparkles' ] loss = phenaki(videos, texts = texts, video_frame_mask = mask) loss.backward() # do the above for many steps, then ... video = phenaki.sample(texts = 'a squirrel examines an acorn', num_frames = 17, cond_scale = 5.) # (1, 3, 17, 256, 128) # so in the paper, they do not really achieve 2 minutes of coherent video # at each new scene with new text conditioning, they condition on the previous K frames # you can easily achieve this with this framework as so video_prime = video[:, :, -3:] # (1, 3, 3, 256, 128) # say K = 3 video_next = phenaki.sample(texts = 'a cat watches the squirrel from afar', prime_frames = video_prime, num_frames = 14) # (1, 3, 14, 256, 128) # the total video entire_video = torch.cat((video, video_next), dim = 2) # (1, 3, 17 + 14, 256, 128) # and so on... ``` -------------------------------- ### Initialize Phenaki Components for Text-to-Video Source: https://context7.com/lucidrains/phenaki-pytorch/llms.txt Initializes the CViViT, MaskGit, and Phenaki components for a complete text-to-video generation pipeline. Ensure CViViT weights are loaded before initializing Phenaki. The `steps` parameter is optimal with the critic. ```python import torch from phenaki_pytorch import CViViT, MaskGit, Phenaki # Initialize components cvivit = CViViT( dim=512, codebook_size=65536, image_size=(256, 128), # Rectangular video support patch_size=32, temporal_patch_size=2, spatial_depth=4, temporal_depth=4, dim_head=64, heads=8 ) # Load pretrained CViViT weights cvivit.load('/path/to/trained/cvivit.pt') maskgit = MaskGit( num_tokens=65536, max_seq_len=1024, dim=512, dim_context=768, depth=6, unconditional=False ) phenaki = Phenaki( cvivit=cvivit, maskgit=maskgit, steps=18, # Sampling steps (18 optimal with critic) t5_name='google/t5-v1_1-base', sample_temperature=0.0, cond_drop_prob=0.25, # Classifier-free guidance dropout max_text_len=128 ).cuda() ``` -------------------------------- ### Initialize and Train C-ViViT Model Source: https://github.com/lucidrains/phenaki-pytorch/blob/main/README.md Initialize the C-ViViT model for video tokenization and set up the CViViTTrainer. The trainer can be configured to train on images or videos and supports EMA for improved generation. Training progress and checkpoints are saved to './results'. ```python import torch from phenaki_pytorch import CViViT, CViViTTrainer cvivit = CViViT( dim = 512, codebook_size = 65536, image_size = 256, patch_size = 32, temporal_patch_size = 2, spatial_depth = 4, temporal_depth = 4, dim_head = 64, heads = 8 ).cuda() trainer = CViViTTrainer( cvivit, folder = '/path/to/images/or/videos', batch_size = 4, grad_accum_every = 4, train_on_images = False, # you can train on images first, before fine tuning on video, for sample efficiency use_ema = False, # recommended to be turned on (keeps exponential moving averaged cvivit) unless if you don't have enough resources num_train_steps = 10000 ) trainer.train() # reconstructions and checkpoints will be saved periodically to ./results ``` -------------------------------- ### Implement PhenakiTrainer Pipeline Source: https://context7.com/lucidrains/phenaki-pytorch/llms.txt Set up a full training pipeline using PhenakiTrainer, supporting custom datasets and distributed training. ```python import torch from torch.utils.data import Dataset from phenaki_pytorch import CViViT, MaskGit, Phenaki, PhenakiTrainer # Initialize models cvivit = CViViT(dim=512, codebook_size=65536, image_size=256, patch_size=32, temporal_patch_size=2, spatial_depth=4, temporal_depth=4, dim_head=64, heads=8) cvivit.load('/path/to/trained/cvivit.pt') maskgit = MaskGit(num_tokens=65536, max_seq_len=1024, dim=512, dim_context=768, depth=6, unconditional=False) phenaki = Phenaki(cvivit=cvivit, maskgit=maskgit).cuda() # Custom dataset returning (video_tensor, caption) pairs class TextVideoDataset(Dataset): def __init__(self, length=1000, image_size=256, num_frames=17): self.num_frames = num_frames self.image_size = image_size self.len = length def __len__(self): return self.len def __getitem__(self, idx): video = torch.randn(3, self.num_frames, self.image_size, self.image_size) caption = f'sample video caption {idx}' return video, caption dataset = TextVideoDataset() trainer = PhenakiTrainer( phenaki=phenaki, dataset=dataset, batch_size=4, grad_accum_every=4, train_on_images=False, num_frames=17, train_lr=1e-4, train_num_steps=100000, max_grad_norm=1.0, save_and_sample_every=1000, num_samples=4, results_folder='./phenaki_results', sample_texts_file_path='/path/to/captions.txt', # For sampling during training amp=True # Enable automatic mixed precision ) ``` -------------------------------- ### Train CViViT with CViViTTrainer Source: https://context7.com/lucidrains/phenaki-pytorch/llms.txt Set up a training loop for the CViViT model with support for gradient accumulation, EMA, and checkpointing. ```python import torch from phenaki_pytorch import CViViT, CViViTTrainer cvivit = CViViT( dim=512, codebook_size=65536, image_size=256, patch_size=32, temporal_patch_size=2, spatial_depth=4, temporal_depth=4, dim_head=64, heads=8 ).cuda() trainer = CViViTTrainer( vae=cvivit, folder='/path/to/videos', # Folder containing .gif or .mp4 files batch_size=4, grad_accum_every=4, # Accumulate gradients for effective batch of 16 train_on_images=False, # Set True to pretrain on images first num_frames=17, # Number of frames per video clip num_train_steps=100000, lr=3e-4, use_ema=True, # Use exponential moving average ema_beta=0.995, save_results_every=100, # Save reconstructions every N steps save_model_every=1000, # Save model checkpoint every N steps results_folder='./cvivit_results' ) # Start training - reconstructions and checkpoints saved to results_folder trainer.train() # Load a saved checkpoint trainer.load('/path/to/checkpoint.pt') ``` -------------------------------- ### Initialize and Use CViViT Video Tokenizer Source: https://context7.com/lucidrains/phenaki-pytorch/llms.txt Configure the CViViT autoencoder to convert videos into discrete tokens and reconstruct them. Requires a CUDA-enabled environment for processing. ```python import torch from phenaki_pytorch import CViViT # Initialize the video tokenizer cvivit = CViViT( dim=512, # Model dimension codebook_size=65536, # Number of discrete tokens in codebook image_size=256, # Video frame size (can be tuple for rectangular) patch_size=32, # Spatial patch size temporal_patch_size=2, # Temporal patch size for frame grouping spatial_depth=4, # Number of spatial transformer layers temporal_depth=4, # Number of temporal transformer layers dim_head=64, # Attention head dimension heads=8, # Number of attention heads channels=3, # Input video channels (RGB) use_vgg_and_gan=True, # Enable perceptual and GAN losses lookup_free_quantization=True # Use LFQ for better codebook utilization ).cuda() # Encode and decode a video (batch, channels, frames, height, width) video = torch.randn(2, 3, 17, 256, 256).cuda() # Get reconstruction loss during training loss = cvivit(video) loss.backward() # Get only reconstructions for evaluation reconstructed_video = cvivit(video, return_recons_only=True) print(f"Reconstructed shape: {reconstructed_video.shape}") # (2, 3, 17, 256, 256) # Get codebook indices for use with MaskGit codebook_ids = cvivit(video, return_only_codebook_ids=True) print(f"Codebook IDs shape: {codebook_ids.shape}") # (2, frames, h_patches, w_patches) # Decode from codebook indices decoded_video = cvivit.decode_from_codebook_indices(codebook_ids.flatten(1)) ``` -------------------------------- ### Initialize MaskGit Predictor Source: https://context7.com/lucidrains/phenaki-pytorch/llms.txt Configure the MaskGit generative model to predict masked video tokens based on text embeddings. ```python import torch from phenaki_pytorch import MaskGit maskgit = MaskGit( num_tokens=65536, # Must match CViViT codebook_size max_seq_len=1024, # Maximum sequence length for video tokens dim=512, # Model dimension dim_context=768, # Text embedding dimension (T5-base: 768) depth=6, # Number of transformer layers dim_head=64, heads=8, attn_dropout=0.0, ff_dropout=0.0, unconditional=False # Set True for unconditional generation ).cuda() ``` -------------------------------- ### Phenaki.sample - Generate Single Video Source: https://context7.com/lucidrains/phenaki-pytorch/llms.txt Generates a single video from a text description using the `phenaki.sample` method. Parameters like `num_frames`, `cond_scale`, and `starting_temperature` control the generation process. The output shape is (batch_size, channels, frames, height, width). ```python import torch from phenaki_pytorch import CViViT, MaskGit, Phenaki # Setup Phenaki (assuming pretrained weights loaded) cvivit = CViViT(dim=512, codebook_size=65536, image_size=256, patch_size=32, temporal_patch_size=2, spatial_depth=4, temporal_depth=4, dim_head=64, heads=8) maskgit = MaskGit(num_tokens=65536, max_seq_len=1024, dim=512, dim_context=768, depth=6) phenaki = Phenaki(cvivit=cvivit, maskgit=maskgit).cuda() # Generate a single video from text video = phenaki.sample( texts='a squirrel examines an acorn', num_frames=17, cond_scale=5.0, # Classifier-free guidance scale starting_temperature=0.9 # Sampling temperature (annealed during generation) ) print(f"Generated video shape: {video.shape}") # (1, 3, 17, 256, 256) ``` -------------------------------- ### Generate Video using make_video Function Source: https://github.com/lucidrains/phenaki-pytorch/blob/main/README.md Utilize the `make_video` function for a streamlined approach to generating videos from a sequence of text prompts. This function handles the concatenation of video segments generated for each scene, allowing for the creation of longer, coherent videos. It also returns the individual video segments for each scene. ```python # ... above code from phenaki_pytorch import make_video entire_video, scenes = make_video(phenaki, texts = [ 'a squirrel examines an acorn buried in the snow', 'a cat watches the squirrel from a frosted window sill', 'zoom out to show the entire living room, with the cat residing by the window sill' ], num_frames = (17, 14, 14), prime_lengths = (5, 5)) entire_video.shape # (1, 3, 17 + 14 + 14 = 45, 256, 256) # scenes - List[Tensor[3]] - video segment of each scene ``` -------------------------------- ### Train Phenaki with Text-to-Video Dataset Source: https://github.com/lucidrains/phenaki-pytorch/blob/main/README.md Use PhenakiTrainer to train on text-video pairs by providing a custom dataset. ```python import torch from torch.utils.data import Dataset from phenaki_pytorch import CViViT, MaskGit, Phenaki, PhenakiTrainer cvivit = CViViT( dim = 512, codebook_size = 65536, image_size = 256, patch_size = 32, temporal_patch_size = 2, spatial_depth = 4, temporal_depth = 4, dim_head = 64, heads = 8 ) cvivit.load('/path/to/trained/cvivit.pt') maskgit = MaskGit( num_tokens = 5000, max_seq_len = 1024, dim = 512, dim_context = 768, depth = 6, unconditional = False ) phenaki = Phenaki( cvivit = cvivit, maskgit = maskgit ).cuda() # mock text video dataset # you will have to extend your own, and return the (