### Create Conda Environment Source: https://github.com/sallymmx/actionclip/blob/master/INSTALL.md Use the provided environment.yml file to install all necessary project dependencies. ```bash conda env create -f environment.yml ``` -------------------------------- ### Train ActionCLIP on HMDB51 Source: https://github.com/sallymmx/actionclip/blob/master/README.md This command trains ActionCLIP on the HMDB51 dataset, starting from Kinetics400 pretrained models. Verify the configuration path. ```bash # train bash scripts/run_train.sh ./configs/hmdb51/hmdb_train.yaml ``` -------------------------------- ### Configure Optimizer and Scheduler Source: https://context7.com/sallymmx/actionclip/llms.txt Sets up the optimizer (e.g., AdamW) and learning rate scheduler (e.g., Cosine annealing) with warmup. Requires 'utils.solver' module and a 'model' object. ```python from utils.solver import _optimizer, _lr_scheduler from dotmap import DotMap # Configuration config = DotMap({ 'solver': { 'optim': 'adamw', 'type': 'cosine', 'lr': 5e-6, 'lr_warmup_step': 5, 'momentum': 0.9, 'weight_decay': 0.2, 'epochs': 50, 'ratio': 1, 'f_ratio': 10, 'lr_decay_step': 15, 'lr_decay_factor': 0.1 } }) # Create optimizer with different learning rates for components optimizer = _optimizer(config, model, fusion_model) # AdamW with: # - text_params: lr = 5e-6 # - visual_params: lr = 5e-6 * ratio = 5e-6 # - fusion_model: lr = 5e-6 * f_ratio = 5e-5 # Create learning rate scheduler lr_scheduler = _lr_scheduler(config, optimizer) # Cosine annealing with 5 epochs warmup # Training loop with scheduler for epoch in range(config.solver.epochs): for batch_idx, (images, labels) in enumerate(train_loader): # Update LR every 10 steps if (batch_idx + 1) % 10 == 0: lr_scheduler.step(epoch + batch_idx / len(train_loader)) # Training step... optimizer.zero_grad() # ... compute loss ... # loss.backward() optimizer.step() ``` -------------------------------- ### Run Testing Script Source: https://github.com/sallymmx/actionclip/blob/master/README.md Execute the testing script with a specified configuration file for evaluating downloaded pretrained models. ```bash bash scripts/run_test.sh ./configs/k400/k400_test.yaml ``` -------------------------------- ### Initialize and Use Visual Prompt Module Source: https://context7.com/sallymmx/actionclip/llms.txt Initializes the visual prompt module for aggregating frame-level features into video-level representations. Supports various temporal fusion strategies and can be moved to GPU with DataParallel. ```python import torch from modules.Visual_Prompt import visual_prompt # Initialize visual prompt module # clip_state_dict comes from clip.load() fusion_model = visual_prompt( sim_head="Transf", # Options: "meanP", "LSTM", "Transf", "Conv_1D", "Transf_cls" clip_state_dict=clip_state_dict, T=8 # Number of frames ) # Move to GPU with DataParallel fusion_model = torch.nn.DataParallel(fusion_model).cuda() # Forward pass - aggregate frame features to video features # Input: [batch_size, num_frames, embed_dim] batch_size = 4 num_frames = 8 embed_dim = 512 # ViT-B/32 embedding dimension frame_features = torch.randn(batch_size, num_frames, embed_dim).cuda() video_features = fusion_model(frame_features) print(f"Video features shape: {video_features.shape}") # Output: Video features shape: torch.Size([4, 512]) ``` -------------------------------- ### Initialize Models and Losses Source: https://context7.com/sallymmx/actionclip/llms.txt Initializes the CLIP model, fusion model, and loss functions for training. Requires PyTorch, CLIP library, and custom utility modules. ```python import torch import clip from utils.KLLoss import KLLoss from utils.tools import create_logits, gen_label, convert_models_to_fp32 # Initialize models model, clip_state_dict = clip.load("ViT-B/32", device="cuda", jit=False, tsm=False, T=8, dropout=0.0, emb_dropout=0.0, pretrain=True) fusion_model = visual_prompt("Transf", clip_state_dict, T=8) model_text = TextCLIP(model) model_image = ImageCLIP(model) # Losses loss_img = KLLoss() loss_txt = KLLoss() ``` -------------------------------- ### Run Training Script Source: https://context7.com/sallymmx/actionclip/llms.txt Executes the training process using provided configuration files for different datasets. Ensure the 'scripts/run_train.sh' script is executable. ```bash # Train on Kinetics-400 bash scripts/run_train.sh ./configs/k400/k400_train.yaml # Train on HMDB51 (from K400 pretrained) bash scripts/run_train.sh ./configs/hmdb51/hmdb_train.yaml # Train on UCF101 (from K400 pretrained) bash scripts/run_train.sh ./configs/ucf101/ucf_train.yaml ``` -------------------------------- ### Configure Training Schedule and Solver Source: https://github.com/sallymmx/actionclip/blob/master/configs/README.md Set training parameters such as the learning policy, number of epochs, optimizer type, gradient clipping, loss function, learning rate, and weight decay. This configuration is crucial for fine-tuning. ```yaml solver: # learning policy type: cosine #cosine multistep epochs: 50 start_epoch: 0 epoch_offset: 0 # optimizer optim: adamw #adam sgd adamw clip_gradient: 20 loss_type: nll lr: 5.e-6 lr_warmup_step: 5 momentum: 0.9 weight_decay: 0.0005 lr_decay_step: 15 lr_decay_factor: 0.1 ``` -------------------------------- ### Configure Data Augmentations Source: https://context7.com/sallymmx/actionclip/llms.txt Sets up training and validation data augmentation pipelines, including RandAugment. Ensure the 'utils.Augmentation' module is available. ```python import torchvision from utils.Augmentation import get_augmentation, randAugment from dotmap import DotMap # Configuration object config = DotMap({ 'data': { 'input_size': 224, 'dataset': 'kinetics400', 'randaug': {'N': 2, 'M': 9} } }) # Training augmentation pipeline transform_train = get_augmentation(training=True, config=config) # Includes: MultiScaleCrop, RandomHorizontalFlip, ColorJitter, # RandomGrayscale, GaussianBlur, Solarization # Validation augmentation (minimal) transform_val = get_augmentation(training=False, config=config) # Includes: Scale, CenterCrop # Add RandAugment for stronger augmentation if config.data.randaug.N > 0: transform_train = randAugment(transform_train, config) print("RandAugment enabled with N=2, M=9") print(f"Training transforms: {transform_train.transforms}") print(f"Validation transforms: {transform_val.transforms}") ``` -------------------------------- ### Resume Training from Checkpoint Source: https://github.com/sallymmx/actionclip/blob/master/configs/README.md Use the 'resume' parameter to specify the path to a saved model checkpoint. This allows you to continue training an interrupted or previously stopped process. ```yaml resume: /home/ock/video_clip/video-clip/exp/clip_hmdb/ViT-B/32/hmdb/20210916_145953/3.pth.tar ``` -------------------------------- ### Load Pretrained ActionCLIP Model Source: https://context7.com/sallymmx/actionclip/llms.txt Loads model and fusion model states from a specified checkpoint file. Requires a configuration object containing the path to the pretrain checkpoint. ```python if config.pretrain: checkpoint = torch.load(config.pretrain) model.load_state_dict(checkpoint['model_state_dict']) fusion_model.load_state_dict(checkpoint['fusion_model_state_dict']) print(f"Loaded pretrained model from {config.pretrain}") ``` -------------------------------- ### Load CLIP Model with ActionCLIP Extensions Source: https://context7.com/sallymmx/actionclip/llms.txt Loads a pre-trained CLIP model with extensions for video understanding. Configure backbone, temporal shift modules, frame count, and dropout for fine-tuning. ```python import clip import torch # Load CLIP model with ActionCLIP extensions device = "cuda" if torch.cuda.is_available() else "cpu" # Basic loading - ViT-B/32 backbone with 8 frames model, clip_state_dict = clip.load( "ViT-B/32", # Model architecture: "ViT-B/32" or "ViT-B/16" device=device, jit=False, # Must be False for training tsm=False, # Temporal Shift Module (Pre-network Prompt) T=8, # Number of input frames dropout=0.0, # Stochastic depth dropout emb_dropout=0.0, # Embedding dropout pretrain=True, # Load CLIP pre-trained weights joint=False # Joint space-time modeling ) # Available models available = clip.available_models() print(f"Available models: {available}") # Output: Available models: ['ViT-B/32', 'ViT-B/16'] # Model with Temporal Shift Module for better temporal modeling model_tsm, state_dict = clip.load( "ViT-B/16", device=device, jit=False, tsm=True, # Enable temporal shift T=16, # 16-frame input dropout=0.1, # Add dropout for regularization emb_dropout=0.1, pretrain=True ) ``` -------------------------------- ### Configure Visual Prompt Settings Source: https://github.com/sallymmx/actionclip/blob/master/configs/README.md Configure visual prompt settings, including options for pre-network, in-network, and post-network prompts. The 'sim_header' parameter determines the type of post-network prompt. ```yaml #visual prompt settings network: tsm: False #Pre-network Prompt joint: False #In-network Prompt sim_header: "seqTransf" #Post-network Prompt(seqTransf meanP seqLSTM conv_1D seqTransf_cls) ``` -------------------------------- ### Train ActionCLIP on UCF101 Source: https://github.com/sallymmx/actionclip/blob/master/README.md Execute this command to train ActionCLIP on the UCF101 dataset, using Kinetics400 pretrained models. Check the configuration file path. ```bash # train bash scripts/run_train.sh ./configs/ucf101/ucf_train.yaml ``` -------------------------------- ### Train ActionCLIP on Kinetics Source: https://github.com/sallymmx/actionclip/blob/master/README.md Use this command to train ActionCLIP on the Kinetics dataset from CLIP pretrained models. Ensure you have the correct configuration file. ```bash # train bash scripts/run_train.sh ./configs/k400/k400_train.yaml ``` -------------------------------- ### Configure Dataset Settings Source: https://github.com/sallymmx/actionclip/blob/master/configs/README.md Define dataset parameters including the dataset name, number of classes, image naming format, and paths to training, validation, and label lists. ```yaml #dataset settings data: dataset: hmdb #dataset names num_classes: 51 #dataset classes image_tmpl: 'img_{:05d}.jpg' #Picture naming format train_list: 'lists/hmdb51/train_rgb_split1.txt' #dataset traning list val_list: 'lists/hmdb51/val_rgb_split1.txt' #dataset validation list label_list: 'lists/hmdb51_labels.csv' #dataset label list ``` -------------------------------- ### Select Backbone Architecture Source: https://github.com/sallymmx/actionclip/blob/master/configs/README.md Specify the backbone architecture for the model. ActionCLIP supports ViT-B/32 and ViT-B/16. ```yaml #model settings network: arch: ViT-B/32 #ViT-B/32 ViT-B/16 #Backbone ``` -------------------------------- ### Load Video Datasets with Temporal Sampling Source: https://context7.com/sallymmx/actionclip/llms.txt Configures PyTorch Dataset and DataLoader for video frame loading. Supports random temporal sampling for training and uniform sampling for validation. ```python from datasets import Action_DATASETS from torch.utils.data import DataLoader # Initialize dataset for training train_dataset = Action_DATASETS( list_file='lists/k4001/train_frames_new.txt', # Video list file labels_file='lists/kinetics_400_labels.csv', # Labels CSV num_segments=8, # Number of frames to sample new_length=1, # Frames per segment image_tmpl='img_{:05d}.jpg', # Frame filename template transform=transform_train, # Data augmentation random_shift=True, # Random temporal sampling test_mode=False, index_bias=1 # Frame index starts at 1 ) # Initialize dataset for validation val_dataset = Action_DATASETS( list_file='lists/k4001/val_frames_re.txt', labels_file='lists/kinetics_400_labels.csv', num_segments=8, image_tmpl='img_{:05d}.jpg', transform=transform_val, random_shift=False, # Uniform sampling for val test_mode=False ) # Create data loaders train_loader = DataLoader( train_dataset, batch_size=128, num_workers=16, shuffle=True, pin_memory=True, drop_last=True ) val_loader = DataLoader( val_dataset, batch_size=128, num_workers=16, shuffle=False, pin_memory=True, drop_last=True ) # Dataset returns (frames_tensor, label) for images, labels in train_loader: print(f"Batch images shape: {images.shape}") print(f"Batch labels shape: {labels.shape}") # Output: Batch images shape: torch.Size([128, 24, 224, 224]) # (128 videos, 8 frames * 3 channels, 224, 224) break ``` -------------------------------- ### Test on Kinetics-400 Source: https://context7.com/sallymmx/actionclip/llms.txt Execute bash scripts to test trained models on the Kinetics-400 dataset. This includes both standard testing and zero-shot evaluation. ```bash # Test on Kinetics-400 bash scripts/run_test.sh ./configs/k400/k400_test.yaml ``` ```bash # Zero-shot evaluation on Kinetics-400 (from CLIP) bash scripts/run_test.sh ./configs/k400/k400_zero_shot.yaml ``` ```bash # Zero-shot on UCF101/HMDB51 (from K400 pretrained) bash scripts/run_test.sh ./configs/hmdb51/hmdb_zero_shot.yaml ``` -------------------------------- ### Save and Load Model Checkpoints Source: https://context7.com/sallymmx/actionclip/llms.txt Use `epoch_saving` to save the model state after each epoch. Use `best_saving` to save the best model based on validation performance. Load checkpoints using `torch.load` to resume training. ```python import torch from utils.saving import epoch_saving, best_saving # Save checkpoint after each epoch working_dir = './exp/clip_k400/ViT-B/32/kinetics400/20240101_120000' filename = f"{working_dir}/last_model.pt" # Save epoch checkpoint epoch_saving(epoch, model, fusion_model, optimizer, filename) # Save best model when validation improves if is_best: best_saving(working_dir, epoch, model, fusion_model, optimizer) # Load checkpoint for resuming training checkpoint = torch.load(config.resume) model.load_state_dict(checkpoint['model_state_dict']) fusion_model.load_state_dict(checkpoint['fusion_model_state_dict']) start_epoch = checkpoint['epoch'] print(f"Resumed from epoch {start_epoch}") ``` -------------------------------- ### ActionClip Training Loop Source: https://context7.com/sallymmx/actionclip/llms.txt This snippet outlines the core training loop for ActionClip, including data loading, image and text embedding generation, contrastive loss calculation, and optimizer steps. It handles reshaping of image data and text prompt retrieval. ```python for epoch in range(50): model_image.train() model_text.train() fusion_model.train() for images, list_id in train_loader: optimizer.zero_grad() # Reshape: [B, T*C, H, W] -> [B, T, C, H, W] -> [B*T, C, H, W] images = images.view((-1, 8, 3) + images.size()[-2:]) b, t, c, h, w = images.size() images = images.to("cuda").view(-1, c, h, w) # Get text prompts for batch text_id = numpy.random.randint(num_text_aug, size=len(list_id)) texts = torch.stack([text_dict[j][i,:] for i,j in zip(list_id, text_id)]) texts = texts.to("cuda") # Forward pass image_embedding = model_image(images) # [B*T, D] image_embedding = image_embedding.view(b, t, -1) # [B, T, D] image_embedding = fusion_model(image_embedding) # [B, D] text_embedding = model_text(texts) # [B, D] # Compute contrastive loss logit_scale = model.logit_scale.exp() logits_per_image, logits_per_text = create_logits( image_embedding, text_embedding, logit_scale ) ground_truth = torch.tensor(gen_label(list_id), dtype=image_embedding.dtype, device="cuda") loss_imgs = loss_img(logits_per_image, ground_truth) loss_texts = loss_txt(logits_per_text, ground_truth) total_loss = (loss_imgs + loss_texts) / 2 # Backward pass total_loss.backward() convert_models_to_fp32(model) optimizer.step() clip.model.convert_weights(model) ``` -------------------------------- ### Encode Images and Text with CLIP Wrappers Source: https://context7.com/sallymmx/actionclip/llms.txt Wraps CLIP encoders in PyTorch modules to support multi-GPU training via DataParallel. Requires a pre-initialized CLIP model. ```python import torch import torch.nn as nn class TextCLIP(nn.Module): """Wrapper for CLIP text encoder""" def __init__(self, model): super(TextCLIP, self).__init__() self.model = model def forward(self, text): return self.model.encode_text(text) class ImageCLIP(nn.Module): """Wrapper for CLIP image encoder""" def __init__(self, model): super(ImageCLIP, self).__init__() self.model = model def forward(self, image): return self.model.encode_image(image) # Usage example model_text = TextCLIP(model) model_image = ImageCLIP(model) # Wrap with DataParallel for multi-GPU training model_text = torch.nn.DataParallel(model_text).cuda() model_image = torch.nn.DataParallel(model_image).cuda() # Encode images (frames) # Input shape: [batch_size * num_frames, channels, height, width] images = torch.randn(32, 3, 224, 224).cuda() # 4 videos x 8 frames image_features = model_image(images) print(f"Image features: {image_features.shape}") # Output: Image features: torch.Size([32, 512]) # Encode text text_tokens = clip.tokenize(["playing basketball", "running"]).cuda() text_features = model_text(text_tokens) print(f"Text features: {text_features.shape}") # Output: Text features: torch.Size([2, 512]) ``` -------------------------------- ### Tokenize Text Descriptions for Action Labels Source: https://context7.com/sallymmx/actionclip/llms.txt Tokenizes text descriptions for action labels using CLIP's tokenizer. Handles single or multiple descriptions and supports custom context lengths. ```python import clip import torch # Tokenize single text text = "a person playing basketball" tokens = clip.tokenize(text) print(f"Token shape: {tokens.shape}") # Output: Token shape: torch.Size([1, 77]) # Tokenize multiple action descriptions action_descriptions = [ "a photo of action playing basketball", "a picture of action running", "Human action of jumping", "swimming, an action" ] tokens = clip.tokenize(action_descriptions) print(f"Batch token shape: {tokens.shape}") # Output: Batch token shape: torch.Size([4, 77]) # Custom context length (default is 77) tokens_custom = clip.tokenize("short text", context_length=77) ``` -------------------------------- ### Run Zero-Shot Validation on Kinetics Source: https://github.com/sallymmx/actionclip/blob/master/README.md Perform zero-shot validation on the Kinetics dataset using CLIP pretrained models by running the test script with the appropriate configuration. ```bash bash scripts/run_test.sh ./configs/k400/k400_ft_zero_shot.yaml ``` -------------------------------- ### Specify Pre-trained Model Path Source: https://github.com/sallymmx/actionclip/blob/master/configs/README.md Set the 'pretrain' parameter to the path of a pre-trained model checkpoint. This is used for initializing the network with weights from a previously trained model. ```yaml pretrain: /home/ock/video_clip/video-clip/clip_models/k400_vit32_8frame.pt ``` -------------------------------- ### Run Zero-Shot Validation on HMDB51 Source: https://github.com/sallymmx/actionclip/blob/master/README.md Conduct zero-shot validation on the HMDB51 dataset using Kinetics pretrained models. Ensure the Kinetics pretrained model is prepared before execution. ```bash bash scripts/run_test.sh ./configs/hmdb51/hmdb_ft_zero_shot.yaml ``` -------------------------------- ### Generate Augmented Text Prompts Source: https://context7.com/sallymmx/actionclip/llms.txt Generates multiple text augmentations per class to improve action recognition robustness. Requires a dataset object with a .classes attribute. ```python import torch import clip from utils.Text_Prompt import text_prompt # text_prompt function generates multiple text augmentations per class # Requires a dataset object with .classes attribute # Example classes from dataset class MockDataset: @property def classes(self): return [ (0, "playing basketball"), (1, "running"), (2, "swimming"), (3, "jumping") ] dataset = MockDataset() classes, num_text_aug, text_dict = text_prompt(dataset) print(f"Number of text augmentations: {num_text_aug}") # Output: Number of text augmentations: 16 print(f"Total text embeddings shape: {classes.shape}") # Output: Total text embeddings shape: torch.Size([64, 77]) # 4 classes x 16 augmentations = 64 total text prompts # Text augmentation templates used: # - "a photo of action {}" # - "a picture of action {}" # - "Human action of {}" # - "{}, an action" # - "{} this is an action" # - "{}, a video of action" # - "Playing action of {}" # - "{}" # - "Playing a kind of action, {}" # - "Doing a kind of action, {}" # - "Look, the human is {}" # - "Can you recognize the action of {}?" # - "Video classification of {}" # - "A video of {}" # - "The man is {}" # - "The woman is {}" ``` -------------------------------- ### ActionClip Validation Function Source: https://context7.com/sallymmx/actionclip/llms.txt This Python function computes Top-1 and Top-5 accuracy for ActionClip during validation. It encodes text features once and then iterates through the validation loader, calculating similarity scores and accuracy. ```python import torch from tqdm import tqdm def validate(epoch, val_loader, classes, device, model, fusion_model, config, num_text_aug): """Validation function computing Top-1 and Top-5 accuracy""" model.eval() fusion_model.eval() num = 0 corr_1 = 0 corr_5 = 0 with torch.no_grad(): # Encode all class text embeddings once text_inputs = classes.to(device) text_features = model.encode_text(text_inputs) for image, class_id in tqdm(val_loader): # Reshape images: [B, T*C, H, W] -> [B*T, C, H, W] image = image.view((-1, config.data.num_segments, 3) + image.size()[-2:]) b, t, c, h, w = image.size() class_id = class_id.to(device) image_input = image.to(device).view(-1, c, h, w) # Encode and fuse video features image_features = model.encode_image(image_input).view(b, t, -1) image_features = fusion_model(image_features) # Normalize features image_features /= image_features.norm(dim=-1, keepdim=True) text_features_norm = text_features / text_features.norm(dim=-1, keepdim=True) # Compute similarity similarity = (100.0 * image_features @ text_features_norm.T) similarity = similarity.view(b, num_text_aug, -1).softmax(dim=-1) similarity = similarity.mean(dim=1, keepdim=False) # Get predictions values_1, indices_1 = similarity.topk(1, dim=-1) values_5, indices_5 = similarity.topk(5, dim=-1) # Compute accuracy num += b for i in range(b): if indices_1[i] == class_id[i]: corr_1 += 1 if class_id[i] in indices_5[i]: corr_5 += 1 top1 = float(corr_1) / num * 100 top5 = float(corr_5) / num * 100 print(f'Epoch [{epoch}]: Top1: {top1:.2f}%, Top5: {top5:.2f}%') return top1 # Usage prec1 = validate(0, val_loader, classes, "cuda", model, fusion_model, config, num_text_aug) ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.