### Process Portion of Long Video Source: https://context7.com/junhua-liao/light-asd/llms.txt Processes a specific segment of a long video file for active speaker detection. Specify the video name, folder, start time in seconds, and duration in seconds. ```python python Columbia_test.py \ --videoName long_video \ --videoFolder demo \ --start 60 \ --duration 30 ``` -------------------------------- ### Initialize AVA Dataset Preprocessing Arguments Source: https://context7.com/junhua-liao/light-asd/llms.txt Sets up argument namespace for AVA dataset preprocessing, including data paths. ```python from utils.tools import init_args, preprocess_AVA import argparse # Setup argument parser with paths args = argparse.Namespace() args.dataPathAVA = '/path/to/AVAData' args.savePath = 'exps/exp1' args.evalDataType = 'val' # Initialize all required paths args = init_args(args) # Creates: args.modelSavePath, args.trialPathAVA, args.audioPathAVA, etc. ``` -------------------------------- ### Initialize Training Data Loader with Augmentation Source: https://context7.com/junhua-liao/light-asd/llms.txt Sets up the training data loader with specified file paths, batch size, and PyTorch DataLoader parameters. Includes audio and visual augmentation. ```python import torch from dataLoader import train_loader import os # Training data loader with augmentation train_dataset = train_loader( trialFileName='AVAData/csv/train_loader.csv', audioPath='AVAData/clips_audios/train', visualPath='AVAData/clips_videos/train', batchSize=2000 # Dynamic batching by frame count ) train_dataloader = torch.utils.data.DataLoader( train_dataset, batch_size=1, # Each item is a dynamic batch shuffle=True, num_workers=64, pin_memory=True ) # Data format returned by loaders: # audioFeature: FloatTensor (1, batch, frames*4, 13) - MFCC features # visualFeature: FloatTensor (1, batch, frames, 112, 112) - Grayscale faces # labels: LongTensor (1, batch, frames) - 0 or 1 per frame for audio, visual, labels in train_dataloader: audio = audio[0].cuda() # (batch, T*4, 13) visual = visual[0].cuda() # (batch, T, 112, 112) labels = labels[0].cuda() # (batch, T) break ``` -------------------------------- ### Download and Preprocess AVA Dataset Source: https://context7.com/junhua-liao/light-asd/llms.txt Use this command to download and preprocess the AVA-ActiveSpeaker dataset for training. Ensure the specified data path exists. ```python python train.py --dataPathAVA /path/to/AVAData --downloadAVA ``` -------------------------------- ### Run ASD Demo with TalkSet Fine-tuned Weights Source: https://github.com/junhua-liao/light-asd/blob/main/README.md Executes a demo for active speaker detection using a provided video file and weights fine-tuned on the TalkSet dataset. Specify the path to the fine-tuned model. ```python python Columbia_test.py --videoName 0001 --videoFolder demo --pretrainModel weight/finetuning_TalkSet.model ``` -------------------------------- ### Run ASD Demo with AVA Pretrained Weights Source: https://github.com/junhua-liao/light-asd/blob/main/README.md Executes a demo for active speaker detection using a provided video file. By default, it loads weights trained on the AVA-ActiveSpeaker dataset. ```python python Columbia_test.py --videoName 0001 --videoFolder demo ``` -------------------------------- ### ASD Training Wrapper Initialization and Usage Source: https://context7.com/junhua-liao/light-asd/llms.txt Initializes the ASD training wrapper, loads parameters, saves parameters, and demonstrates the training and evaluation loops. Requires the ASD class. ```python import torch from ASD import ASD # Initialize with training parameters asd = ASD(lr=0.001, lrDecay=0.95) # Load pre-trained weights asd.loadParameters('weight/pretrain_AVA_CVPR.model') # Save model parameters asd.saveParameters('my_model.model') # Training loop (single epoch) # loader provides: (audioFeature, visualFeature, labels) for epoch in range(1, 31): loss, lr = asd.train_network(loader=train_loader, epoch=epoch) print(f"Epoch {epoch}: Loss={loss:.5f}, LR={lr:.5f}") # Evaluation asd.eval() mAP = asd.evaluate_network( loader=val_loader, evalCsvSave='results/val_res.csv', evalOrig='data/csv/val_orig.csv' ) print(f"Validation mAP: {mAP:.2f}%") ``` -------------------------------- ### Train Light-ASD Model from Scratch Source: https://context7.com/junhua-liao/light-asd/llms.txt Initiates training of the Light-ASD model from scratch. Configurable parameters include learning rate, learning rate decay, maximum epochs, and batch size. ```python python train.py --dataPathAVA /path/to/AVAData --lr 0.001 --lrDecay 0.95 --maxEpoch 30 --batchSize 2000 ``` -------------------------------- ### Initialize and Use ASD Loss Functions Source: https://context7.com/junhua-liao/light-asd/llms.txt Initializes the audio-visual (lossAV) and visual-only (lossV) loss functions. Demonstrates their forward pass during training with sample features and labels. Requires lossAV and lossV classes. ```python import torch from loss import lossAV, lossV # Initialize loss functions criterion_av = lossAV().cuda() # Audio-Visual loss with FC layer criterion_v = lossV().cuda() # Visual-only loss with FC layer # Forward pass during training # x: model output features (batch*frames, 128) # labels: ground truth (batch*frames,) with values 0 or 1 # r: temperature parameter (decreases from 1.3 during training) features_av = torch.randn(100, 128).cuda() # Combined AV features features_v = torch.randn(100, 128).cuda() # Visual-only features labels = torch.randint(0, 2, (100,)).cuda() # Training mode: returns loss, scores, predictions, correct count r = 1.2 # Temperature parameter loss_av, pred_scores, pred_labels, correct_num = criterion_av(features_av, labels, r) loss_v = criterion_v(features_v, labels, r) # Combined loss (visual loss weighted by 0.5) total_loss = loss_av + 0.5 * loss_v ``` -------------------------------- ### Download and Preprocess AVA Dataset Source: https://github.com/junhua-liao/light-asd/blob/main/README.md Use this command to download and preprocess the AVA dataset. The dataset and labels will be stored in the specified AVADataPath. ```python python train.py --dataPathAVA AVADataPath --download ``` -------------------------------- ### Initialize and Use ASD Model Source: https://context7.com/junhua-liao/light-asd/llms.txt Initializes the ASD model and demonstrates its forward pass with sample audio and visual features. Requires the ASD_Model class and PyTorch. ```python import torch from model.Model import ASD_Model # Initialize the model model = ASD_Model().cuda() # Model components: # - visualEncoder: 3D CNN (1 -> 32 -> 64 -> 128 channels) # - audioEncoder: 2D CNN (1 -> 32 -> 64 -> 128 channels) # - GRU: Bidirectional GRU classifier (128 hidden units) # Forward pass with sample data # Audio: MFCC features (batch, frames*4, 13) # Visual: Grayscale faces (batch, frames, 112, 112) audio_features = torch.randn(1, 100, 13).cuda() # 25 frames * 4 = 100 visual_features = torch.randn(1, 25, 112, 112).cuda() # Process audio frontend audio_embed = model.forward_audio_frontend(audio_features) # (B, T, 128) # Process visual frontend visual_embed = model.forward_visual_frontend(visual_features) # (B, T, 128) # Combine audio-visual features through GRU av_output = model.forward_audio_visual_backend(audio_embed, visual_embed) # (B*T, 128) # Visual-only output (for auxiliary loss) v_output = model.forward_visual_backend(visual_embed) # (B*T, 128) # Full forward pass av_out, v_out = model(audio_features, visual_features) ``` -------------------------------- ### Run Demo Inference with Custom Parameters Source: https://context7.com/junhua-liao/light-asd/llms.txt Performs inference on a custom video with specified parameters including model weights, face detection scale, minimum track length, and crop scale. The video is located in the provided folder. ```python python Columbia_test.py \ --videoName meeting_recording \ --videoFolder /path/to/videos \ --pretrainModel weight/finetuning_TalkSet.model \ --facedetScale 0.25 \ --minTrack 10 \ --cropScale 0.40 ``` -------------------------------- ### Initialize Validation Data Loader Source: https://context7.com/junhua-liao/light-asd/llms.txt Sets up the validation data loader with specified file paths and PyTorch DataLoader parameters. No augmentation is applied. ```python import torch from dataLoader import val_loader import os # Validation data loader (no augmentation) val_dataset = val_loader( trialFileName='AVAData/csv/val_loader.csv', audioPath='AVAData/clips_audios/val', visualPath='AVAData/clips_videos/val' ) val_dataloader = torch.utils.data.DataLoader( val_dataset, batch_size=1, shuffle=False, num_workers=64, pin_memory=True ) ``` -------------------------------- ### Run Full AVA Dataset Preprocessing Source: https://context7.com/junhua-liao/light-asd/llms.txt Executes the complete preprocessing pipeline for the AVA dataset. This process is time-consuming and typically run once. ```python # Full preprocessing pipeline (run once, takes ~2 days) preprocess_AVA(args) ``` -------------------------------- ### Run Demo Inference on Custom Video Source: https://context7.com/junhua-liao/light-asd/llms.txt Processes a custom video file to detect active speakers. This command assumes the video is placed in the 'demo' folder. It handles face detection, tracking, and outputs an annotated video. ```python python Columbia_test.py --videoName my_video --videoFolder demo ``` -------------------------------- ### Train Model on AVA Dataset Source: https://github.com/junhua-liao/light-asd/blob/main/README.md Initiates the training process for the model on the AVA dataset. Output files include scores, trained models, and validation predictions. ```python python train.py --dataPathAVA AVADataPath ``` -------------------------------- ### Initialize S3FD Face Detector Source: https://context7.com/junhua-liao/light-asd/llms.txt Initializes the S3FD face detector on the specified device (e.g., 'cuda'). ```python import cv2 from model.faceDetector.s3fd import S3FD # Initialize face detector detector = S3FD(device='cuda') ``` -------------------------------- ### Initialize Bidirectional GRU Classifier Source: https://context7.com/junhua-liao/light-asd/llms.txt Initializes the BGRU classifier and moves it to CUDA. Processes combined audio-visual features for speaker activity classification. ```python import torch from model.Classifier import BGRU # Initialize bidirectional GRU gru = BGRU(channel=128).cuda() # Input: temporal features (batch, time, channels) # Output: refined temporal features (batch, time, channels) combined_features = torch.randn(2, 25, 128).cuda() output = gru(combined_features) # (2, 25, 128) ``` -------------------------------- ### Initialize Audio Encoder Source: https://context7.com/junhua-liao/light-asd/llms.txt Initializes and moves the audio encoder to CUDA. Processes MFCC spectrograms. ```python # Audio encoder: processes MFCC spectrograms # Input: (B, 1, 13, T*4) - batch, channel, mfcc_coeffs, frames # Output: (B, T, 128) - temporal features a_encoder = audio_encoder().cuda() audio_input = torch.randn(2, 1, 13, 100).cuda() # 25 frames * 4 audio_features = a_encoder(audio_input) # (2, 25, 128) ``` -------------------------------- ### Evaluate on Columbia ASD Dataset with AVA Weights Source: https://context7.com/junhua-liao/light-asd/llms.txt Tests the model on the Columbia ASD benchmark dataset using weights trained on the AVA dataset. The script automatically downloads the dataset and computes per-speaker F1 scores. ```python python Columbia_test.py --evalCol --colSavePath /path/to/colData ``` -------------------------------- ### Initialize Visual Encoder Source: https://context7.com/junhua-liao/light-asd/llms.txt Initializes and moves the visual encoder to CUDA. Processes grayscale face sequences. ```python import torch from model.Encoder import visual_encoder, audio_encoder # Visual encoder: processes grayscale face sequences # Input: (B, 1, T, 112, 112) - batch, channel, frames, height, width # Output: (B, T, 128) - temporal features v_encoder = visual_encoder().cuda() visual_input = torch.randn(2, 1, 25, 112, 112).cuda() visual_features = v_encoder(visual_input) # (2, 25, 128) ``` -------------------------------- ### Train Light-ASD with Custom Parameters Source: https://context7.com/junhua-liao/light-asd/llms.txt Trains the model with custom hyperparameters such as learning rate, decay, epochs, test interval, batch size, data loader threads, and save path for experiments. ```python python train.py --dataPathAVA /path/to/AVAData \ --lr 0.0005 \ --lrDecay 0.90 \ --maxEpoch 50 \ --testInterval 2 \ --batchSize 1500 \ --nDataLoaderThread 32 \ --savePath exps/custom_exp ``` -------------------------------- ### Test Model on Columbia ASD Dataset with TalkSet Fine-tuning Source: https://github.com/junhua-liao/light-asd/blob/main/README.md Tests the model on the Columbia ASD dataset using weights fine-tuned on the TalkSet dataset. This requires specifying the path to the fine-tuned model. ```python python Columbia_test.py --evalCol --pretrainModel weight/finetuning_TalkSet.model --colSavePath colDataPath ``` -------------------------------- ### Test Model on Columbia ASD Dataset Source: https://github.com/junhua-liao/light-asd/blob/main/README.md Tests the model on the Columbia ASD dataset using weights pre-trained on the AVA dataset. The dataset and labels will be downloaded into colDataPath. ```python python Columbia_test.py --evalCol --colSavePath colDataPath ``` -------------------------------- ### Evaluate on Columbia ASD Dataset with TalkSet Weights Source: https://context7.com/junhua-liao/light-asd/llms.txt Fine-tunes the model on the Columbia ASD dataset using TalkSet pre-trained weights for improved performance. Specify the path to the fine-tuning model weights. ```python python Columbia_test.py --evalCol \ --pretrainModel weight/finetuning_TalkSet.model \ --colSavePath /path/to/colData ``` -------------------------------- ### Evaluate Model on AVA Dataset Source: https://github.com/junhua-liao/light-asd/blob/main/README.md Performs evaluation of the trained model on the AVA dataset's validation set. This command checks the model's performance metrics. ```python python train.py --dataPathAVA AVADataPath --evaluation ``` -------------------------------- ### Evaluate Pre-trained Model on AVA Dataset Source: https://context7.com/junhua-liao/light-asd/llms.txt Evaluates the performance of the pre-trained CVPR model weights on the AVA dataset. Set '--evalDataType test' to evaluate on the test set instead of the validation set. ```python python train.py --dataPathAVA /path/to/AVAData --evaluation ``` ```python python train.py --dataPathAVA /path/to/AVAData --evaluation --evalDataType test ``` -------------------------------- ### Detect Faces in Image using S3FD Source: https://context7.com/junhua-liao/light-asd/llms.txt Detects faces in an image using the S3FD detector with specified confidence threshold and scales. Returns bounding boxes with confidence scores. ```python # Detect faces in an image image = cv2.imread('frame.jpg') image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # Returns bounding boxes with confidence scores # bboxes format: [[x1, y1, x2, y2, confidence], ...] bboxes = detector.detect_faces( image_rgb, conf_th=0.9, # Confidence threshold scales=[0.25] # Scale factor for detection ) for bbox in bboxes: x1, y1, x2, y2, conf = bbox cv2.rectangle(image, (int(x1), int(y1)), (int(x2), int(y2)), (0, 255, 0), 2) ``` -------------------------------- ### CVPR 2023 Paper Citation Source: https://github.com/junhua-liao/light-asd/blob/main/README.md Citation details for the paper 'A Light Weight Model for Active Speaker Detection' presented at CVPR 2023. ```bibtex @InProceedings{Liao_2023_CVPR, author = {Liao, Junhua and Duan, Haihan and Feng, Kanghui and Zhao, Wanbing and Yang, Yanbing and Chen, Liangyin}, title = {A Light Weight Model for Active Speaker Detection}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2023}, pages = {22932-22941} } ``` -------------------------------- ### IJCV 2025 Paper Citation Source: https://github.com/junhua-liao/light-asd/blob/main/README.md Citation details for the paper 'LR-ASD: Lightweight and Robust Network for Active Speaker Detection' published in IJCV 2025. ```bibtex @article{Liao_2025_IJCV, title = {LR-ASD: Lightweight and Robust Network for Active Speaker Detection}, author = {Liao, Junhua and Duan, Haihan and Feng, Kanghui and Zhao, Wanbing and Yang, Yanbing and Chen, Liangyin and Chen, Yanru}, journal = {International Journal of Computer Vision}, pages = {1--21}, year = {2025}, publisher = {Springer} } ``` -------------------------------- ### Inference Mode: Prediction Scores Only Source: https://context7.com/junhua-liao/light-asd/llms.txt Sets the model to evaluation mode and disables gradient calculation for inference. Use this when you only need prediction scores. ```python criterion_av.eval() with torch.no_grad(): scores = criterion_av(features_av, labels=None) # numpy array of scores ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.