### Setup Colab Environment for pyannote.audio Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/add_your_own_model.ipynb Installs pyannote.audio, IPython, and downloads necessary data for the tutorial within a Google Colab environment. Remember to restart the runtime after execution. ```python !pip install -qq pyannote.audio==3.1.1 !pip install -qq ipython==7.34.0 !git clone https://github.com/pyannote/AMI-diarization-setup.git %cd ./AMI-diarization-setup/pyannote/ !bash ./download_ami_mini.sh %cd /content ``` -------------------------------- ### Install DiariZen and Dependencies Source: https://github.com/butspeechfit/diarizen/blob/main/README.md Commands to set up a Python environment, install PyTorch, DiariZen, and pyannote-audio. ```bash conda create --name diarizen python=3.10 conda activate diarizen conda install pytorch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 pytorch-cuda=12.1 "mkl<2024.1" -c pytorch -c nvidia -c defaults pip install -r requirements.txt && pip install -e . cd pyannote-audio && pip install -e .[dev,testing] git submodule init git submodule update ``` -------------------------------- ### Install pyannote-audio with CLI dependencies Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/training_with_cli.md Install pyannote-audio with the necessary dependencies for using the command-line training tools. ```bash pip install pyannote-audio[cli] ``` -------------------------------- ### Install pyannote.audio and ipython Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/intro.ipynb Installs the pyannote.audio library and ipython. Ensure you are using a GPU for faster processing. ```python !pip install -qq pyannote.audio==3.1.1 !pip install -qq ipython==7.34.0 ``` -------------------------------- ### Install pyannote.audio with Conda Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/doc/source/index.rst Instructions for creating a new Conda environment and installing pyannote.audio. ```bash $ conda create -n pyannote python=3.10 $ conda activate pyannote $ pip install pyannote.audio ``` -------------------------------- ### Install pyannote.audio and rich Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/adapting_pretrained_pipeline.ipynb Installs the pyannote.audio library version 2.1.1 and the rich library for enhanced output. Ensure you restart the runtime after installation. ```python !pip install -qq pyannote.audio==2.1.1 !pip install -qq rich ``` -------------------------------- ### Multi-GPU Training Setup Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/training_a_model.ipynb Configure a PyTorch Lightning Trainer for multi-GPU training using DDP strategy. This example specifies 4 devices. ```python trainer = Trainer(devices=4, accelerator="gpu", strategy='ddp') trainer.fit(model) ``` -------------------------------- ### Install Development Dependencies Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/README.md Installs the pyannote.audio library in editable mode with development and testing packages. Also installs pre-commit hooks for code quality. ```bash pip install -e .[dev,testing] pre-commit install ``` -------------------------------- ### Install and Download Resources in Colab Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/applying_a_model.ipynb Installs necessary packages (pyannote.audio, ipython) and downloads sample audio and reference files for use in a Google Colab environment. Remember to restart the runtime after execution. ```bash !pip install -qq pyannote-audio==3.1.1 !pip install -qq ipython==7.34.0 !wget -q "https://github.com/pyannote/pyannote-audio/raw/develop/tutorials/assets/sample.wav" !wget -q "https://github.com/pyannote/pyannote-audio/raw/develop/tutorials/assets/sample.rttm" !wget -q -P ./assets/ "https://github.com/pyannote/pyannote-audio/blob/develop/tutorials/assets/download-model.png" ``` -------------------------------- ### Install pyannote.audio and download assets Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/applying_a_pipeline.ipynb Installs the pyannote.audio library and downloads necessary sample audio and RTTM files for use in Google Colab. ```shell !pip install -qq pyannote.audio==3.1.1 !pip install -qq ipython==7.34.0 !wget -q "https://github.com/pyannote/pyannote-audio/raw/develop/tutorials/assets/sample.wav" !wget -q "https://github.com/pyannote/pyannote-audio/raw/develop/tutorials/assets/sample.rttm" !wget -q -P ./assets/ "https://github.com/pyannote/pyannote-audio/blob/develop/tutorials/assets/download-model.png" !wget -q -P ./assets/ "https://github.com/pyannote/pyannote-audio/blob/develop/tutorials/assets/download-pipeline.png" ``` -------------------------------- ### Listen to Audio Example Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/notebook/example.ipynb Uses the 'listen' utility to play a segment of audio from the loaded protocol. This is useful for quick audio verification. ```python from pyannote.audio.utils.preview import listen listen(next(protocol.train())) ``` -------------------------------- ### Install pyannote-audio Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/community/offline_usage_speaker_diarization.ipynb Install the pyannote-audio package using pip. This is a prerequisite for using the library. ```python from pathlib import Path from pyannote.audio import Pipeline import os ``` -------------------------------- ### Download and Setup AMI-SDM Mini Corpus Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/adapting_pretrained_pipeline.ipynb Downloads the AMI-SDM mini corpus and sets up the pyannote database configuration. This involves cloning a GitHub repository and executing a shell script. ```bash # download AMI-SDM mini corpus %cd /content/ !git clone https://github.com/pyannote/AMI-diarization-setup %cd /content/AMI-diarization-setup/pyannote/ !bash download_ami_sdm_mini.sh ``` -------------------------------- ### Install pyannote.audio Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/MRE_template.ipynb Installs a specific version of the pyannote.audio library. Use this to ensure the issue is reproducible with a known version. ```python !pip install -qqq pyannote.audio==3.1.1 ``` -------------------------------- ### Install hydra-submitit-launcher Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/training_with_cli.md Install the hydra-submitit-launcher package to enable launching multiple jobs on a Slurm cluster. This is a prerequisite for distributed training and hyperparameter sweeps. ```bash pip install hydra-submitit-launcher --upgrade ``` -------------------------------- ### Setup for Non-Colab Environments Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/applying_a_model.ipynb Clones the pyannote.audio repository and sets the ROOT_DIR variable to the local path of the repository. This is for users not running the tutorial in Google Colab. ```python # clone pyannote-audio Github repository and update ROOT_DIR accordingly ROOT_DIR = "/pyannote-audio" AUDIO_FILE = f"{ROOT_DIR}/tutorials/assets/sample.wav" REFERENCE = f"{ROOT_DIR}/tutorials/assets/sample.rttm" ``` -------------------------------- ### Train the Model with PyTorch Lightning Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/training_a_model.ipynb Initialize a PyTorch Lightning Trainer and fit the model. This example uses a single device and a maximum of one epoch. ```python trainer = pl.Trainer(devices=1, max_epochs=1) trainer.fit(osd_model) ``` -------------------------------- ### Get First Training File Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/voice_activity_detection.ipynb Retrieves the first training file from the loaded AMI protocol. This file can then be used for listening or visualization. ```python first_training_file = next(ami.train()) reference = first_training_file["annotation"] ``` -------------------------------- ### Set up Callbacks and Trainer for Fine-tuning Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/adapting_pretrained_pipeline.ipynb Configure PyTorch Lightning callbacks for early stopping and model checkpointing, and initialize the Trainer. This setup monitors validation performance and saves the best model during training. ```python from pytorch_lightning.callbacks import ( EarlyStopping, ModelCheckpoint, RichProgressBar, ) # we monitor diarization error rate on the validation set # and use to keep the best checkpoint and stop early monitor, direction = task.val_monitor checkpoint = ModelCheckpoint( monitor=monitor, mode=direction, save_top_k=1, every_n_epochs=1, save_last=False, save_weights_only=False, filename="{epoch}", verbose=False, ) early_stopping = EarlyStopping( monitor=monitor, mode=direction, min_delta=0.0, patience=10, strict=True, verbose=False, ) callbacks = [RichProgressBar(), checkpoint, early_stopping] # we train for at most 20 epochs (might be shorter in case of early stopping) from pytorch_lightning import Trainer trainer = Trainer(accelerator="gpu", callbacks=callbacks, max_epochs=20, gradient_clip_val=0.5) trainer.fit(model) ``` -------------------------------- ### Define Task Specifications Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/add_your_own_task.ipynb In the `setup` method, call `super().setup(stage)` and then define `self.specifications` to declare the problem type, resolution, duration, and classes. ```python def setup(self, stage: Optional[Union[str, None]] = None): # this method assigns prepared data from task.prepare_data() to the task # and declares the task specifications super().setup(stage) # specify the addressed problem self.specifications = Specifications( # it is a multi-label classification problem problem=Problem.MULTI_LABEL_CLASSIFICATION, # we expect the model to output one prediction # for the whole chunk resolution=Resolution.CHUNK, # the model will ingest chunks with that duration (in seconds) duration=self.duration, # human-readable names of classes classes=self.prepared_data["classes"]) ``` -------------------------------- ### Get First Training File Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/overlapped_speech_detection.ipynb Retrieves the first file from the training set of the loaded protocol. This is useful for initial exploration and testing. ```python first_training_file = next(protocol.train()) ``` -------------------------------- ### Load AMI Diarization Protocol Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/overlapped_speech_detection.ipynb Loads the AMI diarization protocol and sets up a file finder for audio preprocessing. Ensure pyannote.audio is installed and the AMI corpus is set up. ```python from pyannote.database import registry, FileFinder registry.load_database("AMI-diarization-setup/pyannote/database.yml") protocol = registry.get_protocol("AMI.SpeakerDiarization.mini", preprocessors={"audio": FileFinder()}) ``` -------------------------------- ### Inference on a Test File Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/voice_activity_detection.ipynb Apply a trained VAD model to a test audio file. This example uses a file provided by the AMI protocol, but any audio file can be used. ```python test_file = next(ami.test()) # here we use a test file provided by the protocol, but it could be any audio file ``` -------------------------------- ### Train the Model with PyTorch Lightning Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/overlapped_speech_detection.ipynb Train the configured pre-trained model using PyTorch Lightning's Trainer. The example trains for a single epoch. ```python import pytorch_lightning as pl trainer = pl.Trainer(max_epochs=1) trainer.fit(pretrained_model) ``` -------------------------------- ### Train the PyanNet Model Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/training_a_model.ipynb Utilize pytorch-lightning's Trainer to fit the PyanNet model. This example trains for a single epoch on one device. ```python import pytorch_lightning as pl trainer = pl.Trainer(devices=1, max_epochs=1) trainer.fit(vad_model) ``` -------------------------------- ### Inference with a Pre-trained Model Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/training_a_model.ipynb Perform inference using a pre-trained pyannote-audio model. This is often a starting point for transfer learning. ```python Inference('pyannote/segmentation-3.0', use_auth_token=True, step=2.5)(test_file) ``` -------------------------------- ### Define audio file and reference path Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/applying_a_pipeline.ipynb Sets the file paths for the audio file and its reference annotation, typically used after setup in a Colab environment. ```python AUDIO_FILE = "sample.wav" REFERENCE = "sample.rttm" ``` -------------------------------- ### Generate Oracle OSD Timeline Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/overlapped_speech_detection.ipynb Creates an 'oracle' overlapped speech detection (OSD) pipeline and applies it to a file to get the ground truth OSD timeline. This serves as a baseline for evaluating OSD models. ```python from pyannote.audio.pipelines.overlapped_speech_detection import OracleOverlappedSpeechDetection oracle_osd = OracleOverlappedSpeechDetection() oracle_osd(first_training_file).get_timeline() ``` -------------------------------- ### Instantiate an Existing Task (Voice Activity Detection) Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/add_your_own_task.ipynb Loads the AMI-diarization-setup database and instantiates the VoiceActivityDetection task using the 'AMI.SpeakerDiarization.mini' protocol. Assumes the AMI corpus is already set up. ```python # this assumes that the AMI corpus has been setup for diarization # according to https://github.com/pyannote/AMI-diarization-setup from pyannote.database import registry, FileFinder registry.load_database("AMI-diarization-setup/pyannote/database.yml") ami = registry.get_protocol('AMI.SpeakerDiarization.mini', preprocessors={'audio': FileFinder()}) # address voice activity detection from pyannote.audio.tasks import VoiceActivityDetection task = VoiceActivityDetection(ami) ``` -------------------------------- ### Get Overlapped Speech Timeline Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/overlapped_speech_detection.ipynb Load an overlapped speech detection model and get the timeline of detected speech regions for a given audio file. ```python oracle_osd(test_file).get_timeline() ``` -------------------------------- ### Download and prepare demo audio file Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/intro.ipynb Downloads an audio file from the AMI corpus and sets up a dictionary for processing. This file will be used for demonstration purposes. ```python !wget -q http://groups.inf.ed.ac.uk/ami/AMICorpusMirror/amicorpus/ES2004a/audio/ES2004a.Mix-Headset.wav DEMO_FILE = {'uri': 'ES2004a.Mix-Headset', 'audio': 'ES2004a.Mix-Headset.wav'} ``` -------------------------------- ### Initialize Debug Protocol and FileFinder Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/notebook/freeze.ipynb Set up the pyannote database protocol and audio file finder for debugging purposes. This is a prerequisite for defining tasks and models. ```python from pyannote.database import get_protocol, FileFinder protocol = get_protocol('Debug.SpeakerDiarization.Debug', preprocessors={"audio": FileFinder()}) ``` -------------------------------- ### Get OSD Probability Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/overlapped_speech_detection.ipynb Obtain the raw overlapped speech detection probabilities from the inference object. This results in a `SlidingWindowFeature` object. ```python osd_probability ``` -------------------------------- ### Instantiate and Apply Optimized Pipeline Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/overlapped_speech_detection.ipynb Create an optimized pipeline using the best parameters obtained from the tuning process. Then, apply this optimized pipeline to a test file and retrieve the resulting timeline. ```python optimized_pipeline = OverlappedSpeechDetectionPipeline(pretrained_model).instantiate(optimizer.best_params) optimized_pipeline(test_file).get_timeline() ``` -------------------------------- ### Configure Notebook for Visualization Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/training_a_model.ipynb Prepares the notebook for visualization purposes by setting a specific time segment for output display. This is useful for focusing on particular parts of the audio during analysis. ```python # preparing notebook for visualization purposes # (only show outputs between t=180s and t=240s) from pyannote.core import notebook, Segment notebook.crop = Segment(210, 240) ``` -------------------------------- ### Perform VAD Inference on a File Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/notebook/inference.ipynb Apply the VAD inference model to an entire development file to get voice activity scores. ```python # inference dev_file = next(protocol.development()) scores = inference(dev_file) scores ``` -------------------------------- ### Download Sample Audio File Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/MRE_template.ipynb Downloads a sample audio file from a public URL using `wget`. This is useful for testing the pipeline with a known audio input. ```bash !wget https://github.com/pyannote/pyannote-audio/raw/develop/tutorials/assets/sample.wav ``` -------------------------------- ### Get Pretrained Pipeline Hyperparameters Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/adapting_pretrained_pipeline.ipynb Retrieve the hyperparameters of a pretrained speaker diarization pipeline. These parameters are adapted to the internal pretrained segmentation model. ```python pretrained_hyperparameters = pretrained_pipeline.parameters(instantiated=True) pretrained_hyperparameters ``` -------------------------------- ### Perform VAD Inference on an Excerpt Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/notebook/inference.ipynb Use the `crop` method of the Inference object to get VAD scores for a specific segment of an audio file. ```python # inference on an excerpt from pyannote.core import Segment scores = inference.crop(dev_file, Segment(10, 15)) scores ``` -------------------------------- ### Configure Training with Data Augmentation Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/training_a_model.ipynb Set up a task with data augmentation using `torch-audiomentations`. This requires specifying a directory containing background noise files. ```python from torch_audiomentations import AddBackgroundNoise augmentation = AddBackgroundNoise("/path/to/background/noise/directory") vad_task = VoiceActivityDetection(ami, augmentation=augmentation) ``` -------------------------------- ### View training configuration with --cfg job Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/training_with_cli.md Use the --cfg job option with pyannote-audio-train to inspect the detailed Hydra configuration used for training. This helps understand all parameters and their values. ```bash pyannote-audio-train --cfg job \ model=PyanNet \ task=VoiceActivityDetection \ registry="AMI-diarization-setup/pyannote/database.yml" \ protocol=AMI.SpeakerDiarization.only_words ``` -------------------------------- ### Prepare Notebook for Visualization Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/applying_a_model.ipynb Configures the notebook to display outputs only within a specific time segment (0-30 seconds). This is useful for focusing on a particular part of the audio. ```python from pyannote.core import notebook, Segment notebook.crop = Segment(0, 30) ``` -------------------------------- ### Configure Jean Zay cluster training parameters Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/training_with_cli.md Set up specific parameters for training on the Jean Zay cluster using hydra-submitit-launcher. This includes defining account, quality of service, GPU and CPU allocation, and job timeout. ```bash +hydra.launcher.additional_parameters.account=eie@gpu # --account option ``` ```bash hydra.launcher.qos=qos_gpu-dev # QOS ``` ```bash hydra.launcher.gpus_per_task=1 # number of GPUs ``` ```bash hydra.launcher.cpus_per_gpu=10 # number of CPUS per GPUs (10 is ) ``` ```bash hydra.launcher.timeout_min=120 # --time option (in minutes) ``` ```bash task.duration=2,5,10 hydra.sweep.subdir=\"${task.duration}\\s_chunks ``` -------------------------------- ### Load AMI Dataset Protocol Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/voice_activity_detection.ipynb Loads the AMI dataset protocol and sets up a file finder for audio files. This is the first step in preparing data for training. ```python from pyannote.database import registry, FileFinder registry.load_database("AMI-diarization-setup/pyannote/database.yml") preprocessors = {"audio": FileFinder()} ami = registry.get_protocol('AMI.SpeakerDiarization.mini', preprocessors=preprocessors) ``` -------------------------------- ### Launch pyannote-audio training on Slurm Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/training_with_cli.md Initiate a grid search for training pyannote-audio models on a Slurm cluster. This command configures a sweep over LSTM layers and bidirectionality, specifying the task and dataset. ```bash pyannote-audio-train \ --multirun hydra/launcher=submitit_slurm \ model=PyanNet +model.lstm.num_layers=2,3,4 +model.lstm.bidirectional=true,false \ task=VoiceActivityDetection \ registry="AMI-diarization-setup/pyannote/database.yml" \ protocol=AMI.SpeakerDiarization.only_words ``` -------------------------------- ### Get Expected Output Timeline Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/training_a_model.ipynb Retrieve the expected voice activity detection timeline from the test file's annotation. This is useful for comparing against the model's predictions. ```python expected_output = test_file["annotation"].get_timeline().support() expected_output ``` -------------------------------- ### Prepare Test File for Inference Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/training_a_model.ipynb Retrieves a test audio file from the specified protocol (ami) to be used for applying the trained model. ```python test_file = next(ami.test()) ``` -------------------------------- ### Train a Voice Activity Detection Model Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/notebook/sharing.ipynb This code trains a simple segmentation model for Voice Activity Detection using PyTorch Lightning. Ensure PyTorch Lightning is installed. ```python from pyannote.audio.tasks import VoiceActivityDetection from pyannote.audio.models.segmentation.debug import SimpleSegmentationModel import pytorch_lightning as pl vad = VoiceActivityDetection(protocol, duration=2., batch_size=32, num_workers=4) model = SimpleSegmentationModel(task=vad) trainer = pl.Trainer(max_epochs=1, default_root_dir='sharing/') _ = trainer.fit(model) ``` -------------------------------- ### Instantiate and Configure VAD Pipeline Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/voice_activity_detection.ipynb Initialize a VoiceActivityDetection pipeline and set its hyper-parameters for onset, offset, and minimum duration of speech and non-speech segments. ```python from pyannote.audio.pipelines import VoiceActivityDetection as VoiceActivityDetectionPipeline pipeline = VoiceActivityDetectionPipeline(segmentation=model) initial_params = {"onset": 0.6, "offset": 0.4, "min_duration_on": 0.0, "min_duration_off": 0.0} pipeline.instantiate(initial_params) ``` -------------------------------- ### Train Custom Model using pyannote-audio-train CLI Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/add_your_own_model.ipynb Train your custom model using the `pyannote-audio-train` CLI by specifying the configuration directory, protocol, task, model, and any model-specific parameters. ```bash $ pyannote-audio-train --config-dir=/your/favorite/directory/custom_config \ protocol=Debug.SpeakerDiarization.Debug \ task=VoiceActivityDetection \ model=MyCustomModel \ model.param2=12 ``` -------------------------------- ### Load Pre-trained Speaker Embedding Model Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/speaker_verification.ipynb Load a pre-trained speaker embedding model. Ensure you have PyTorch and pyannote.audio installed. Specify the model name and device (e.g., 'cuda' for GPU). ```python import torch from pyannote.audio.pipelines.speaker_verification import PretrainedSpeakerEmbedding model = PretrainedSpeakerEmbedding( "speechbrain/spkrec-ecapa-voxceleb", device=torch.device("cuda")) ``` -------------------------------- ### Iterate and Print Annotation Results Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/applying_a_pipeline.ipynb Iterate through the annotation results (speech turns, tracks, and speakers) and print their start time, end time, and speaker label. This is useful for understanding the structure of the output. ```python from pyannote.core import Annotation assert isinstance(dia, Annotation) ``` ```python for speech_turn, track, speaker in dia.itertracks(yield_label=True): print(f"{speech_turn.start:4.1f} {speech_turn.end:4.1f} {speaker}") ``` -------------------------------- ### Train Voice Activity Detection Model Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/notebook/example.ipynb Defines a Voice Activity Detection task and trains a simple segmentation model using PyTorch Lightning. This example requires a configured protocol and trainer. ```python from pyannote.audio.tasks import VoiceActivityDetection vad = VoiceActivityDetection(protocol, duration=2., batch_size=32, num_workers=4) model = SimpleSegmentationModel(task=vad) trainer = pl.Trainer(max_epochs=1) _ = trainer.fit(model) ``` -------------------------------- ### Load Debug Protocol and File Finder Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/notebook/example.ipynb Loads a specific protocol from the registry and configures a FileFinder for audio preprocessing. Ensure your database YAML file is correctly configured. ```python from pyannote.database import registry, FileFinder registry.load_database('../tests/data/database.yml') protocol = registry.get_protocol('Debug.SpeakerDiarization.Debug', preprocessors={"audio": FileFinder()}) ``` -------------------------------- ### Load AMI-SDM Dataset Protocol Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/adapting_pretrained_pipeline.ipynb Loads the pyannote.database registry and retrieves the AMI-SDM speaker diarization mini protocol, configuring it to use FileFinder for audio access. Ensure the database.yml path is correct. ```python from pyannote.database import registry, FileFinder registry.load_database("AMI-diarization-setup/pyannote/database.yml") dataset = registry.get_protocol("AMI-SDM.SpeakerDiarization.mini", {"audio": FileFinder()}) ``` -------------------------------- ### Get Final Overlapped Speech Timeline with Pipeline Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/overlapped_speech_detection.ipynb Apply the configured overlapped speech detection pipeline to the audio file to obtain the final timeline of overlapped speech regions after thresholding and duration filtering. ```python pipeline(test_file).get_timeline() ``` -------------------------------- ### Run Pruning Training Source: https://github.com/butspeechfit/diarizen/blob/main/recipes/diar_ssl_pruning/README.md Initiates the structured pruning training process for WavLM models. Ensure the environment is activated and necessary scripts are in place. ```bash bash -i run_stage.sh ``` -------------------------------- ### Verify AMI-SDM Mini Corpus Info Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/adapting_pretrained_pipeline.ipynb Uses the pyannote-database CLI to display information about the downloaded AMI-SDM mini corpus, including the number of files, annotated duration, speech duration, and speakers for train, development, and test sets. ```bash !PYANNOTE_DATABASE_CONFIG="/content/AMI-diarization-setup/pyannote/database.yml" pyannote-database info AMI-SDM.SpeakerDiarization.mini ``` -------------------------------- ### Custom Model Forward Pass Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/add_your_own_model.ipynb Implement the forward pass for a custom PyTorch model. This example shows how to process waveforms, apply linear layers, perform temporal pooling based on resolution, and apply a final classifier with activation. ```python # Extract sequence of MFCCs and passed them through two linear layers mfcc = self.mfcc(waveforms).squeeze(dim=1).transpose(1, 2) output = self.linear1(mfcc) output = self.linear2(output) # Apply temporal pooling for tasks which need an output at chunk-level. if self.specifications.resolution == Resolution.CHUNK: output = torch.mean(output, dim=-1) # Keep 'mfcc' frame resolution for frame-level tasks. elif self.specifications.resolution == Resolution.FRAME: pass # Apply final classifier and activation function output = self.classifier(output) return self.activation(output) ``` -------------------------------- ### Train the Model Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/training_a_model.ipynb Initialize a PyTorch Lightning `Trainer` and fit the `finetuned` model for one epoch. This step performs the actual fine-tuning process. ```python import pytorch_lightning as pl trainer = pl.Trainer(devices=1, max_epochs=1) trainer.fit(finetuned) ``` -------------------------------- ### Load a Pretrained Model Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/applying_a_model.ipynb Load a pretrained pyannote.audio model from the Hugging Face Hub using `Model.from_pretrained`. Ensure you have authenticated if the model is gated. ```python from pyannote.audio import Model model = Model.from_pretrained("pyannote/segmentation-3.0", use_auth_token=True) ``` -------------------------------- ### Load a Model from a Checkpoint Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/notebook/sharing.ipynb Load a trained pyannote.audio model from a local checkpoint file using Model.from_pretrained. The checkpoint can also be a URL. ```python from pyannote.audio import Model model = Model.from_pretrained('sharing/lightning_logs/version_0/checkpoints/epoch=0-step=3.ckpt') assert isinstance(model, SimpleSegmentationModel) # checkpoint should work with a URL as well (it relies on pl_load) ``` -------------------------------- ### Load Pretrained Segmentation Model Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/adapting_pretrained_pipeline.ipynb Load a pretrained segmentation model from pyannote-audio. Ensure you have authentication token set up if the model is private or requires it. ```python from pyannote.audio import Model model = Model.from_pretrained("pyannote/segmentation", use_auth_token=True) ``` -------------------------------- ### Tune Pipeline Hyperparameters with Optimizer Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/voice_activity_detection.ipynb Optimize the 'onset' and 'offset' hyperparameters of the pyannote-audio pipeline using the Optimizer class. This requires a development dataset and specifies the number of iterations for tuning. ```python from pyannote.pipeline import Optimizer optimizer = Optimizer(pipeline) optimizer.tune(list(ami.development()), warm_start=initial_params, n_iterations=20, show_progress=False) optimized_params = optimizer.best_params ``` -------------------------------- ### Initialize and Train Custom Model using pyannote.audio API Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/add_your_own_model.ipynb Use your custom model directly within the pyannote.audio API by initializing the protocol, task, and model, then training it with PyTorch Lightning. ```python from pyannote.database import registry, FileFinder registry.load_database("./AMI-diarization-setup/pyannote/database.yml") protocol = registry.get_protocol('AMI.SpeakerDiarization.mini', preprocessors={"audio": FileFinder()}) from pyannote.audio.tasks import VoiceActivityDetection task = VoiceActivityDetection(protocol) model = MyCustomModel(task=task) from pytorch_lightning import Trainer trainer = Trainer(max_epochs=1) trainer.fit(model) ``` -------------------------------- ### Load Test File for Inference Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/overlapped_speech_detection.ipynb Load a test audio file from the specified protocol to prepare for model inference. This file can be any audio file. ```python test_file = next(protocol.test()) ``` -------------------------------- ### Implementing `val__getitem__` for Validation Samples Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/add_your_own_task.ipynb This method loads and prepares a single validation sample based on its index. It crops audio and creates a label array for the specified chunk, returning it in a dictionary format. ```python # load list and number of classes classes = self.specifications.classes num_classes = len(classes) # find which part of the validation set corresponds to sample_idx num_samples = np.cumsum([ validation_file["num_samples"] for validation_file in self.prepared_data["validation"]]) file_idx = np.where(num_samples < sample_idx)[0][0] validation_file = self.prepared_data["validation"][file_idx] idx = sample_idx - (num_samples[file_idx] - validation_file["num_samples"]) chunk = SlidingWindow(start=0., duration=self.duration, step=self.duration)[idx] # load audio excerpt corresponding to current chunk X = self.model.audio.crop(validation_file["audio"], chunk, fixed=self.duration) # load labels corresponding to random chunk as {0|1} numpy array # y[k] = 1 means that kth class is active y = np.zeros((num_classes,)) active_classes = validation_file["annotation"].crop(chunk).labels() for active_class in active_classes: y[classes.index(active_class)] = 1 return {'X': X, 'y': y} ``` -------------------------------- ### Run Tests Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/README.md Executes the test suite for the pyannote.audio library using pytest. ```bash pytest ``` -------------------------------- ### Load and Crop Audio Excerpt Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/notebook/augmentation.ipynb Loads an audio file using pyannote.database and extracts a 5-second excerpt. Requires pyannote.database and pyannote.audio.core.io.Audio. Ensure the 'Debug.SpeakerDiarization.Debug' protocol is available. ```python # gett a 5s excerpt of first test file from pyannote.database import get_protocol, FileFinder protocol = get_protocol('Debug.SpeakerDiarization.Debug', preprocessors={"audio": FileFinder()}) from pyannote.audio.core.io import Audio audio = Audio(sample_rate=16000, mono="downmix") file = next(protocol.test()) from pyannote.core import Segment waveform, sample_rate = audio.crop(file, Segment(5, 10)) import torch waveforms = torch.tensor(waveform)[None, :] ``` -------------------------------- ### List available pyannote pipelines on Hugging Face Hub Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/applying_a_pipeline.ipynb Fetches and lists all available pretrained pipelines tagged with 'pyannote-audio-pipeline' from the Hugging Face Model Hub. ```python from huggingface_hub import HfApi available_pipelines = [p.modelId for p in HfApi().list_models(filter="pyannote-audio-pipeline")] list(filter(lambda p: p.startswith("pyannote/"), available_pipelines)) ``` -------------------------------- ### Import Custom Model from pyannote.audio.models Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/add_your_own_model.ipynb Verify that your custom model, when placed within the `pyannote.audio.models` package, can be imported. ```python >>> from pyannote.audio.models.custom_model import MyCustomModel ``` -------------------------------- ### List Available Tasks in pyannote.audio Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/add_your_own_task.ipynb Prints a list of all available task classes within the pyannote.audio library. ```python from pyannote.audio.tasks import __all__ as TASKS; print('\n'.join(TASKS)) ``` -------------------------------- ### Prepare Model for Fine-tuning Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/adapting_pretrained_pipeline.ipynb Configure a segmentation model for fine-tuning by defining the training task and preparing the data. This involves setting up dataset parameters, duration, number of speakers, batch size, and loss functions. ```python from pyannote.audio.tasks import Segmentation task = Segmentation( dataset, duration=model.specifications.duration, max_num_speakers=len(model.specifications.classes), batch_size=32, num_workers=2, loss="bce", vad_loss="bce") model.task = task model.prepare_data() model.setup() ``` -------------------------------- ### Model Training Output and Summary Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/training_a_model.ipynb Displays the model's architecture, parameter counts, and estimated size. It also shows the training progress and completion status. ```text Output: GPU available: False, used: False TPU available: False, using: 0 TPU cores IPU available: False, using: 0 IPUs HPU available: False, using: 0 HPUs | Name | Type | Params | In sizes | Out sizes --------------------------------------------------------------------------------------------------------------------- 0 | sincnet | SincNet | 42.6 K | [1, 1, 32000] | [1, 60, 115] 1 | lstm | LSTM | 589 K | [1, 115, 60] | [[1, 115, 256], [[4, 1, 128], [4, 1, 128]]] 2 | linear | ModuleList | 49.4 K | ? | ? 3 | classifier | Linear | 129 | [1, 115, 128] | [1, 115, 1] 4 | activation | Sigmoid | 0 | [1, 115, 1] | [1, 115, 1] 5 | validation_metric | MetricCollection | 0 | ? | ? --------------------------------------------------------------------------------------------------------------------- 681 K Trainable params 0 Non-trainable params 681 K Total params 2.728 Total estimated model params size (MB) ``` ```text Result: Sanity Checking: | | 0/? [00:00/pyannote-audio" AUDIO_FILE = f"{ROOT_DIR}/tutorials/assets/sample.wav" REFERENCE = f"{ROOT_DIR}/tutorials/assets/sample.rttm" ``` -------------------------------- ### Visualize Manual Annotation Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/voice_activity_detection.ipynb Displays the manual annotation for a specified segment of an audio file using `pyannote.core.notebook`. Ensure `notebook.crop` is set to the desired segment. ```python from pyannote.core import notebook notebook.crop = one_minute reference ``` -------------------------------- ### Load and play audio excerpt Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/intro.ipynb Loads an audio excerpt using pyannote.audio and displays it using IPython.display.Audio. This allows you to listen to the selected segment of the audio file. ```python from pyannote.audio import Audio from IPython.display import Audio as IPythonAudio waveform, sr = Audio(mono="downmix").crop(DEMO_FILE, EXCERPT) IPythonAudio(waveform.flatten(), rate=sr) ``` -------------------------------- ### Load Audio from Memory for Processing Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/applying_a_pipeline.ipynb Loads an audio file using torchaudio and prepares it as a dictionary containing 'waveform' and 'sample_rate' for processing by pyannote.audio pipelines. This is useful when audio is not stored on disk. ```python import torchaudio waveform, sample_rate = torchaudio.load(AUDIO_FILE) print(f"{type(waveform)=}") print(f"{waveform.shape=}") print(f"{waveform.dtype=}") audio_in_memory = {"waveform": waveform, "sample_rate": sample_rate} ``` -------------------------------- ### Train Model Contributing to pyannote-audio using CLI Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/add_your_own_model.ipynb Train a model that has been integrated into the `pyannote.audio` library using the `pyannote-audio-train` CLI. Specify the protocol, task, and model name. ```bash $ pyannote-audio-train protocol=Debug.SpeakerDiarization.Debug \ task=VoiceActivityDetection \ model=MyCustomModel \ model.param2=12 ``` -------------------------------- ### Listen to Audio File Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/overlapped_speech_detection.ipynb Plays the audio associated with a given file and visualizes its reference annotation. Requires the `listen` utility from `pyannote.audio.utils.preview`. ```python from pyannote.audio.utils.preview import listen listen(first_training_file) ``` -------------------------------- ### Load and Configure Pretrained Model Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/overlapped_speech_detection.ipynb Load a pre-trained segmentation model from Hugging Face Hub and assign the initialized OSD task to it. Ensure you have accepted the model's terms on Hugging Face. ```python from pyannote.audio.core.model import Model pretrained_model = Model.from_pretrained("pyannote/segmentation-3.0", use_auth_token=True) # we assign the OSD task to the model pretrained_model.task = osd ``` -------------------------------- ### Inference with DiariZen Pipeline Source: https://github.com/butspeechfit/diarizen/blob/main/README.md Load a pre-trained model and apply the diarization pipeline to an audio file. Results are printed to the console. ```python from diarizen.pipelines.inference import DiariZenPipeline # load pre-trained model diar_pipeline = DiariZenPipeline.from_pretrained("BUT-FIT/diarizen-wavlm-large-s80-md") # apply diarization pipeline diar_results = diar_pipeline('./example/EN2002a_30s.wav') # print results for turn, _, speaker in diar_results.itertracks(yield_label=True): print(f"start={turn.start:.1f}s stop={turn.end:.1f}s speaker_{speaker}") # start=0.0s stop=2.7s speaker_0 # start=0.8s stop=13.6s speaker_3 # start=5.8s stop=6.4s speaker_0 # ... # load pre-trained model and save RTTM result diar_pipeline = DiariZenPipeline.from_pretrained( "BUT-FIT/diarizen-wavlm-large-s80-md", rttm_out_dir='.' ) # apply diarization pipeline diar_results = diar_pipeline('./example/EN2002a_30s.wav', sess_name='EN2002a') ``` -------------------------------- ### Initialize Voice Activity Detection Task Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/training_a_model.ipynb Configure and initialize a Voice Activity Detection task, specifying the dataset (ami), audio chunk duration, and batch size for training. ```python from pyannote.audio.tasks import VoiceActivityDetection vad_task = VoiceActivityDetection(ami, duration=2.0, batch_size=128) ``` -------------------------------- ### Load AMI Protocol Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/training_a_model.ipynb Loads the AMI database protocol using the pyannote.database registry. This step is necessary to access and utilize the AMI corpus for training or evaluation. ```python from pyannote.database import registry, FileFinder registry.load_database("AMI-diarization-setup/pyannote/database.yml") ami = registry.get_protocol('AMI.SpeakerDiarization.mini') ``` -------------------------------- ### Instantiate Overlapped Speech Detection Pipeline Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/overlapped_speech_detection.ipynb Instantiate the `OverlappedSpeechDetectionPipeline` with manually set hyper-parameters for onset, offset, minimum duration on, and minimum duration off. These parameters control the sensitivity and duration filtering of detected speech regions. ```python from pyannote.audio.pipelines import OverlappedSpeechDetection as OverlappedSpeechDetectionPipeline pipeline = OverlappedSpeechDetectionPipeline(pretrained_model).instantiate({"onset": 0.5, "offset": 0.5, "min_duration_on": 0.1, "min_duration_off": 0.1}) ``` -------------------------------- ### Initialize Inference for Sliding Window Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/overlapped_speech_detection.ipynb Wrap a pre-trained model with `Inference` to handle sliding window processing over audio files longer than the model's training chunk size. This is necessary for processing entire files and aggregating model outputs. ```python from pyannote.audio import Inference inference = Inference(pretrained_model) osd_probability = inference(test_file) ``` -------------------------------- ### Import Necessary Classes for VAD and Model Training Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/notebook/freeze.ipynb Import the required classes for Voice Activity Detection (VAD) tasks, a simple segmentation model, and PyTorch Lightning for training. ```python from pyannote.audio.tasks import VoiceActivityDetection from pyannote.audio.models.segmentation.debug import SimpleSegmentationModel import pytorch_lightning as pl ``` -------------------------------- ### Prepare Task-Specific Data Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/add_your_own_task.ipynb Implement `post_prepare_data` to add task-specific data preparation after the base `Task.prepare_data()` is called. This includes loading metadata and identifying classes. ```python def post_prepare_data(self, prepared_data: Dict): # this method is called at the end of Task.prepare_data() # to complete data preparation with task-specific data, here # the list of classes and some training metadata # load metadata for training subset prepared_data["train_metadata"] = list() for training_file in self.protocol.train(): prepared_data["train_metadata"].append({ # path to audio file (str) "audio": training_file["audio"], # duration of audio file (float) "duration": training_file["torchaudio.info"].num_frames / training_file["torchaudio.info"].sample_rate, # reference annotation (pyannote.core.Annotation) "annotation": training_file["annotation"], }) # gather the list of classes classes = set() for training_file in prepared_data["train_metadata"]: classes.update(training_file["annotation"].labels()) prepared_data["classes"] = sorted(classes) # `has_validation` is True if protocol defines a development set if not self.has_validation: return ``` -------------------------------- ### Train the Model Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/voice_activity_detection.ipynb Trains the initialized PyanNet model using pytorch-lightning's Trainer for one epoch. This requires importing `pytorch_lightning`. ```python import pytorch_lightning as pl trainer = pl.Trainer(devices=1, max_epochs=1) trainer.fit(model) ``` -------------------------------- ### Summarize Model Architecture Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/applying_a_model.ipynb Inspect the architecture of a loaded pyannote.audio model using `pytorch_lightning.utilities.model_summary.summarize`. This helps understand the model's layers and parameters. ```python from pytorch_lightning.utilities.model_summary import summarize summarize(model) ``` -------------------------------- ### Training Output - Sanity Checking Progress Source: https://github.com/butspeechfit/diarizen/blob/main/pyannote-audio/tutorials/overlapped_speech_detection.ipynb This output shows the progress of the sanity checking phase before training begins. '0/?' indicates zero out of an unknown total number of iterations. ```text Sanity Checking: | | 0/? [00:00>> from your_package_name.custom_model import MyCustomModel ```