WeSpeaker

https://github.com/wenet-e2e/wespeaker
Admin
WeSpeaker is a speaker embedding learning toolkit that focuses on speaker verification tasks with...
Tokens:56,285
Snippets:334
Trust Score:7.7
License:Apache-2.0
Update:1 week ago
Show doc for...
Tokens:
Raw
### wav.scp file format example

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/vox.md

Example lines from a wav.scp file, showing the format: wav_id and wav_path separated by a blank.

```text
id10001/1zcIwhmdeo4/00001.wav /exported/data/voxceleb1_wav_v2/id10001/1zcIwhmdeo4/00001.wav
id10001/1zcIwhmdeo4/00002.wav /exported/data/voxceleb1_wav_v2/id10001/1zcIwhmdeo4/00002.wav
...
```

--------------------------------

### spk2utt file format example

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/vox.md

Example lines from a spk2utt file, showing the format: spk_id followed by many wav_ids belonging to that speaker.

```text
id10001 id10001/1zcIwhmdeo4/00001.wav id10001/1zcIwhmdeo4/00002.wav id10001/1zcIwhmdeo4/00003.wav ...
id10002 id10002/0_laIeN-Q44/00001.wav id10002/6WO410QOeuo/00001.wav ...
...
```

--------------------------------

### Development install of WeSpeaker

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/python_package.md

Clone the repository and perform an editable install for development purposes.

```sh
git clone https://github.com/wenet-e2e/wespeaker.git
cd wespeaker
pip install -e .
```

--------------------------------

### Start Triton server

Source: https://github.com/wenet-e2e/wespeaker/blob/master/runtime/server/diarization_gpu/README.md

Starts the Triton server container with GPU support and mounts the model repository and SAD directory. Then runs tritonserver inside the container with the model repository path.

```bash
# Start the docker container
docker run --gpus all -v $PWD/model_repo:/workspace/model_repo -v $SAD_DIR:/workspace/triton/ --shm-size=1g --ulimit memlock=-1 -p 8000:8000 -p 8001:8001 -p 8002:8002 --ulimit stack=67108864 -it  wespeaker_server:latest

# Inside the docker container
tritonserver --model-repository=/workspace/model_repo
```

--------------------------------

### utt2spk file format example

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/vox.md

Example lines from an utt2spk file, showing the format: wav_id and spk_id separated by a blank.

```text
id10001/1zcIwhmdeo4/00001.wav id10001
id10001/1zcIwhmdeo4/00002.wav id10001
...
```

--------------------------------

### trials file format example

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/vox.md

Example lines from a trials file, showing the format: enroll_wav_id, test_wav_id, and label (target or nontarget).

```text
id10001/Y8hIVOBuels/00001.wav id10001/1zcIwhmdeo4/00001.wav target
id10001/Y8hIVOBuels/00001.wav id10943/vNCVj7yLWPU/00005.wav nontarget
id10001/Y8hIVOBuels/00001.wav id10001/7w0IBEWc9Qw/00004.wav target
id10001/Y8hIVOBuels/00001.wav id10999/G5R2-Hl7YX8/00008.wav nontarget
...
```

--------------------------------

### WeSpeaker Command-Line Usage Examples

Source: https://github.com/wenet-e2e/wespeaker/blob/master/README.md

Shows how to use the wespeaker command-line tool for various tasks: embedding extraction, embedding extraction from kaldi-format wav.scp, similarity computation, and diarization. Use -h for parameters.

```sh
$ wespeaker --task embedding --audio_file audio.wav --output_file embedding.txt
$ wespeaker --task embedding_kaldi --wav_scp wav.scp --output_file /path/to/embedding
$ wespeaker --task similarity --audio_file audio.wav --audio_file2 audio2.wav
$ wespeaker --task diarization --audio_file audio.wav
```

--------------------------------

### Install WeSpeaker via pip

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/python_package.md

Install the latest WeSpeaker package directly from the GitHub repository using pip.

```sh
pip install git+https://github.com/wenet-e2e/wespeaker.git
```

--------------------------------

### Create Conda Environment and Install Dependencies for WeSpeaker Development

Source: https://github.com/wenet-e2e/wespeaker/blob/master/README.md

Create a conda environment with Python 3.9, install PyTorch 2.7.1 with CUDA 11.X or 12.X, install sox, requirements, torchnet, and pre-commit. PyTorch version >= 2.0.0 is recommended.

```sh
conda create -n wespeaker python=3.9
conda activate wespeaker
pip install torch==2.7.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu118 # cuda 11.X
pip install torch==2.7.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu128 # cuda 12.X
conda install -c conda-forge sox
pip install -r requirements.txt
# visdom (dep of torchnet) needs pkg_resources at build time, bypass pip build isolation to use the existing setuptools
pip install --no-build-isolation torchnet==0.0.4
pre-commit install  # for clean and tidy code
```

--------------------------------

### WeSpeaker command-line usage examples

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/python_package.md

Shows the wespeaker CLI for tasks: embedding, embedding_kaldi, similarity, diarization, and diarization with device selection (CUDA or MPS).

```sh
$ wespeaker --task embedding --audio_file audio.wav --output_file embedding.txt
$ wespeaker --task embedding_kaldi --wav_scp wav.scp --output_file /path/to/embedding
$ wespeaker --task similarity --audio_file audio.wav --audio_file2 audio2.wav
$ wespeaker --task diarization --audio_file audio.wav
$ wespeaker --task diarization --audio_file audio.wav --device cuda:0 # use CUDA on Windows/Linux
$ wespeaker --task diarization --audio_file audio.wav --device mps    # use Metal Performance Shaders on MacOS
```

--------------------------------

### Run Triton server with model repository

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/runtime.md

Runs the WeSpeaker server container with GPU access, mounts the model repository, and starts Triton. Ports 8000 (HTTP) and 8001 (gRPC) are exposed.

```bash
docker run --gpus '"device=0"' -v $PWD/model_repo:/ws/model_repo --shm-size=1g --ulimit memlock=-1 -p 8000:8000 -p 8001:8001 -p 8002:8002 --ulimit stack=67108864 -ti  wespeaker:latest
tritonserver --model-repository=/ws/model_repo
```

--------------------------------

### Stage 3: Train DINO model with torchrun

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/vox_ssl.md

Starts neural network training using torchrun for DDP with the specified number of GPUs. The command runs wespeaker/ssl/bin/train_dino.py with configuration, experiment directory, data type, training data, and optional reverb/noise data. Supports resuming from a checkpoint via the checkpoint variable.

```bash
if [ ${stage} -le 3 ] && [ ${stop_stage} -ge 3 ]; then
  echo "Start training ..."
  num_gpus=$(echo $gpus | awk -F ',' '{print NF}')
  torchrun --standalone --nnodes=1 --nproc_per_node=$num_gpus \
    wespeaker/ssl/bin/train_dino.py --config $config \
      --exp_dir ${exp_dir} \
      --gpus $gpus \
      --num_avg ${num_avg} \
      --data_type "${data_type}" \
      --train_data ${data}/vox2_dev/${data_type}.list \
      --wav_scp ${data}/vox2_dev/wav.scp \
      --reverb_data ${data}/rirs/lmdb \
      --noise_data ${data}/musan/lmdb \
      ${checkpoint:+--checkpoint $checkpoint}
fi
```

--------------------------------

### Train neural network with torchrun (stage 3)

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/vox.md

Starts multi-GPU DDP training using torchrun with the specified config, experiment directory, and data lists. Supports resuming from a checkpoint via the checkpoint variable.

```bash
if [ ${stage} -le 3 ] && [ ${stop_stage} -ge 3 ]; then
  echo "Start training ..."
  num_gpus=$(echo $gpus | awk -F ',' '{print NF}')
  torchrun --standalone --nnodes=1 --nproc_per_node=$num_gpus \
    wespeaker/bin/train.py --config $config \
      --exp_dir ${exp_dir} \
      --gpus $gpus \
      --num_avg ${num_avg} \
      --data_type "${data_type}" \
      --train_data ${data}/vox2_dev/${data_type}.list \
      --train_label ${data}/vox2_dev/utt2spk \
      --reverb_data ${data}/rirs/lmdb \
      --noise_data ${data}/musan/lmdb \
      ${checkpoint:+--checkpoint $checkpoint}
fi
```

--------------------------------

### Example spectral clustering output format

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/voxconverse_diar.md

Example lines from the spectral clustering output file, showing the format: each line contains a subseg_id and a spk_id separated by a blank space.

```text
abjxc-00000400-00007040-00000000-00000150 0
abjxc-00000400-00007040-00000075-00000225 0
abjxc-00000400-00007040-00000150-00000300 0
abjxc-00000400-00007040-00000225-00000375 0
...
```

--------------------------------

### Embedding processing chain example

Source: https://github.com/wenet-e2e/wespeaker/blob/master/examples/sre/v3/README.md

Specifies a processing chain for embedding preprocessing, including mean-subtract, length-norm, and LDA steps. The chain is passed to the prep_embd_proc.py script, which calculates parameters sequentially and stores the chain in pickle format.

```bash
mean-subtract --scp $mean1_scp | length-norm | lda --scp $lda_scp --utt2spk $utt2spk --dim $lda_dim | length-norm
```

--------------------------------

### Example RTTM file content

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/voxconverse_diar.md

Example lines from an RTTM file showing the ten fields per turn. Each line represents a speaker turn with fields like SPEAKER, file ID, channel, onset, duration, and speaker name.

```text
SPEAKER abjxc 1 0.400 6.640 <NA> <NA> 0 <NA> <NA>
SPEAKER abjxc 1 8.680 55.960 <NA> <NA> 0 <NA> <NA>
```

--------------------------------

### Run client container and enter client directory

Source: https://github.com/wenet-e2e/wespeaker/blob/master/runtime/server/diarization_gpu/README.md

Runs the client Docker container with network host mode, mounting the client code and test audio data. Then changes to the client directory inside the container.

```bash
AUDIO_DATA=<path to your test wav files>
docker run -ti --net host -v $PWD/client:/ws/client -v $AUDIO_DATA:/ws/test_data  wespeaker_client:latest
cd /ws/client
```

--------------------------------

### Stage 1: Download prerequisites (SCTK, Silero VAD, ResNet34)

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/voxconverse_diar.md

Shell script to download and unzip the evaluation toolkit SCTK, the Silero VAD model, and the WeSpeaker ResNet34 speaker model. Creates directories external_tools and pretrained_models.

```bash
if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then
    mkdir -p external_tools

    # [1] Download evaluation toolkit
    wget -c https://github.com/usnistgov/SCTK/archive/refs/tags/v2.4.12.zip -O external_tools/SCTK-v2.4.12.zip
    unzip -o external_tools/SCTK-v2.4.12.zip -d external_tools

    # [2] Download voice activity detection model pretrained by Silero Team
    wget -c https://github.com/snakers4/silero-vad/archive/refs/tags/v3.1.zip -O external_tools/silero-vad-v3.1.zip
    unzip -o external_tools/silero-vad-v3.1.zip -d external_tools

    # [3] Download ResNet34 speaker model pretrained by WeSpeaker Team
    mkdir -p pretrained_models

    wget -c https://wespeaker-1256283475.cos.ap-shanghai.myqcloud.com/models/voxceleb/voxceleb_resnet34_LM.onnx -O pretrained_models/voxceleb_resnet34_LM.onnx
fi
```

--------------------------------

### Run all stages of the VoxCeleb v2 recipe

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/vox.md

Commands to execute each stage of the recipe sequentially. Run them one by one to understand the process.

```bash
cd examples/voxceleb/v2/
bash run.sh --stage 1 --stop_stage 1
bash run.sh --stage 2 --stop_stage 2
bash run.sh --stage 3 --stop_stage 3
bash run.sh --stage 4 --stop_stage 4
bash run.sh --stage 5 --stop_stage 5
bash run.sh --stage 6 --stop_stage 6
bash run.sh --stage 7 --stop_stage 7
bash run.sh --stage 8 --stop_stage 8
```

--------------------------------

### Build client Docker image

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/runtime.md

Builds the WeSpeaker client Docker image using the Dockerfile.client. Uses network host mode.

```bash
docker build . -f Dockerfile/dockerfile.client -t wespeaker_client:latest --network host
```

--------------------------------

### Build server Docker image

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/runtime.md

Builds the WeSpeaker server Docker image using the Dockerfile.server. Uses network host mode.

```bash
docker build . -f Dockerfile/dockerfile.server -t wespeaker:latest --network host
```

--------------------------------

### Build MNN backend with CMake

Source: https://github.com/wenet-e2e/wespeaker/blob/master/runtime/mnn/README.md

Commands to build the project. Requires cmake 3.14+ and gcc/g++ 5.4+. The normal build uses `cmake ..`; the commented alternative enables mini libs with `-DMINI_LIBS=ON`.

```sh
mkdir build && cd build
# 1. normal
cmake  ..
# 2. minimum libs
# cmake -DMINI_LIBS=ON ..
cmake --build .
```

--------------------------------

### Download and prepare Voxconverse data (shell script)

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/voxconverse_diar.md

This shell script downloads the Voxconverse annotations (version 0.0.3) and dev/test audio files, unzips them, and creates wav.scp files for both dev and test sets. It is intended to be run when stage is between 2 and stop_stage. The script uses wget and unzip commands, and creates wav.scp files using awk to extract the wav ID from the file path.

```bash
if [ ${stage} -le 2 ] && [ ${stop_stage} -ge 2 ]; then
    mkdir -p data

    # Download annotations for dev and test sets (version 0.0.3)
    wget -c https://github.com/joonson/voxconverse/archive/refs/heads/master.zip -O data/voxconverse_master.zip
    unzip -o data/voxconverse_master.zip -d data

    # Download annotations from VoxSRC-23 validation toolkit (looks like version 0.0.2)
    # cd data && git clone https://github.com/JaesungHuh/VoxSRC2023.git --recursive && cd -

    # Download dev audios
    mkdir -p data/dev

    wget --no-check-certificate -c https://www.robots.ox.ac.uk/~vgg/data/voxconverse/data/voxconverse_dev_wav.zip -O data/voxconverse_dev_wav.zip
    unzip -o data/voxconverse_dev_wav.zip -d data/dev

    # Create wav.scp for dev audios
    ls `pwd`/data/dev/audio/*.wav | awk -F/ '{print substr($NF, 1, length($NF)-4), $0}' > data/dev/wav.scp

    # Test audios
    mkdir -p data/test

    wget  --no-check-certificate -c https://www.robots.ox.ac.uk/~vgg/data/voxconverse/data/voxconverse_test_wav.zip -O data/voxconverse_test_wav.zip
    unzip -o data/voxconverse_test_wav.zip -d data/test

    # Create wav.scp for test audios
    ls `pwd`/data/test/voxconverse_test_wav/*.wav | awk -F/ '{print substr($NF, 1, length($NF)-4), $0}' > data/test/wav.scp
fi
```

--------------------------------

### Run DINO recipe stages sequentially

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/vox_ssl.md

Commands to run each stage of the DINO recipe one by one, from stage 1 to 6, to understand the process.

```bash
cd examples/voxceleb/v3/dino
bash run.sh --stage 1 --stop_stage 1
bash run.sh --stage 2 --stop_stage 2
bash run.sh --stage 3 --stop_stage 3
bash run.sh --stage 4 --stop_stage 4
bash run.sh --stage 5 --stop_stage 5
bash run.sh --stage 6 --stop_stage 6
```

--------------------------------

### Clone WeSpeaker repository and set environment variables

Source: https://github.com/wenet-e2e/wespeaker/blob/master/runtime/server/diarization_gpu/README.md

Clones the WeSpeaker repository and sets environment variables for the project directory.

```bash
# Clond WeSpeaker repo
git clone https://github.com/wenet-e2e/wespeaker.git
export WeSpeaker=$PWD/wespeaker/
cd runtime/server/diarization_gpu
export PROJECT_DIR=$PWD

```

--------------------------------

### Prepare model repository

Source: https://github.com/wenet-e2e/wespeaker/blob/master/runtime/server/diarization_gpu/README.md

Copies pretrained SAD and Speaker models into the model_repo directory. Requires PROJECT_DIR, SPK_MODEL_REPO, and SAD_DIR environment variables to be set.

```bash
cd ${PROJECT_DIR}
mkdir -p model_repo/run/1
cp -r $SPK_MODEL_REPO/* model_repo/
```

--------------------------------

### Stage 8: Large margin finetuning with run.sh

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/vox.md

Performs large margin finetuning by initializing from the pre-trained average model and running stages 3-7 with a different config. This can further improve performance on Voxceleb data.

```shell
if [ ${stage} -le 8 ] && [ ${stop_stage} -ge 8 ]; then
  echo "Large margin fine-tuning ..."
  lm_exp_dir=${exp_dir}-LM
  mkdir -p ${lm_exp_dir}/models
  # Use the pre-trained average model to initialize the LM training
  cp ${exp_dir}/models/avg_model.pt ${lm_exp_dir}/models/model_0.pt
  bash run.sh --stage 3 --stop_stage 7 \
      --data ${data} \
      --data_type ${data_type} \
      --config ${lm_config} \
      --exp_dir ${lm_exp_dir} \
      --gpus $gpus \
      --num_avg 1 \
      --checkpoint ${lm_exp_dir}/models/model_0.pt \
      --trials "$trials" \
      --score_norm_method ${score_norm_method} \
      --top_n ${top_n}
fi
```

--------------------------------

### Build with cmake

Source: https://github.com/wenet-e2e/wespeaker/blob/master/runtime/onnxruntime/README.md

Builds the project with ONNX support. Requires cmake 3.14 or above and gcc/g++ 5.4 or above. Use -DONNX=ON for CPU-only, or add -DGPU=ON for GPU support.

```sh
mkdir build && cd build
# 1. no gpu
cmake -DONNX=ON ..
# 2. gpu
# cmake -DONNX=ON -DGPU=ON ..
cmake --build .
```

--------------------------------

### Run all stages of the VoxConverse v2 diarization recipe

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/voxconverse_diar.md

Commands to execute each stage of the recipe sequentially, from stage 1 to stage 8. Run them in order to understand the full diarization pipeline.

```bash
cd examples/voxconverse/v2/
bash run.sh --stage 1 --stop_stage 1
bash run.sh --stage 2 --stop_stage 2
bash run.sh --stage 3 --stop_stage 3
bash run.sh --stage 4 --stop_stage 4
bash run.sh --stage 5 --stop_stage 5
bash run.sh --stage 6 --stop_stage 6
bash run.sh --stage 7 --stop_stage 7
bash run.sh --stage 8 --stop_stage 8
```

--------------------------------

### Run client and extract embeddings

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/runtime.md

Runs the client container, mounts the current directory and data path, and executes client.py to extract embeddings from a wav.scp file. The output directory will contain files like xvector_000.ark, xvextor_000.scp, etc.

```bash
docker run -it -v $PWD:/ws -v <data path>:<data path> --network=host wespeaker_client

# example command
cd /ws/client/
python3 client.py --url=<ip of the server>:8001 --wavscp=/raid/dgxsa/slyne/wespeaker/examples/voxceleb/v2/data/vox1/wav.scp --output_directory=<to put the generated embeddings>

# The output direcotry will be something like:
# xvector_000.ark xvextor_000.scp xvector_001.scp .....
```

--------------------------------

### Pull pre-built server image

Source: https://github.com/wenet-e2e/wespeaker/blob/master/runtime/server/diarization_gpu/README.md

Pulls a pre-built server image from Docker Hub as an alternative to building it yourself.

```bash
docker pull wd929/wespeaker_server:v1
```

--------------------------------

### Run U³-xi recipe stages 3 to 5

Source: https://github.com/wenet-e2e/wespeaker/blob/master/examples/voxceleb/v2/README.md

Executes the U³-xi recipe from stage 3 (embedding extraction) through stage 5 (scoring) using the provided script. Stage 4 writes xvector_variance.scp, and stage 5 produces both standard and uncertainty-aware scores in exp_dir/scores and exp_dir/scores_uncertainty.

```bash
bash run_u3xi.sh --stage 3 --stop_stage 5
```

--------------------------------

### Build Onnxruntime (cmake)

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/runtime.md

Builds the runtime using cmake. Requires cmake 3.14 or above and gcc/g++ 5.4 or above. GPU build is not supported on macOS.

```sh
mkdir build && cd build
# 1. no gpu
cmake -DONNX=ON ..
# 2. gpu (macOS don't supported)
# cmake -DONNX=ON -DGPU=ON ..
cmake --build .
```

--------------------------------

### Run W2V-BERT 2.0 training and scoring

Source: https://github.com/wenet-e2e/wespeaker/blob/master/examples/voxceleb/v2/README.md

Run the W2V-BERT 2.0 training and scoring pipeline from stage 3 to stage 9. This uses the W2V-BERT 2.0 pre-trained frontend with a w2v-bert2-LoRA-adapter-MFA speaker model and a Frozen (LoRA) => Joint ft => Joint lmft training strategy.

```bash
bash run_w2v.sh --stage 3 --stop_stage 9
```

--------------------------------

### Build server Docker image

Source: https://github.com/wenet-e2e/wespeaker/blob/master/runtime/server/diarization_gpu/README.md

Builds the Triton server Docker image with tag wespeaker_server:latest. Uses --network host for network access during build.

```bash
docker build . -f Dockerfile/dockerfile.server -t wespeaker_server:latest --network host
```

--------------------------------

### Compute similarity with asv_main

Source: https://github.com/wenet-e2e/wespeaker/blob/master/runtime/mnn/README.md

Command to run the speaker verification tool. It compares two speech files and outputs similarity based on a threshold of 0.5.

```sh
export GLOG_logtostderr=1
export GLOG_v=2
./build/bin/asv_main \
    --enroll_wav wav1_path \
    --test_wav wav2_path \
    --threshold 0.5 \
    --speaker_model_path model.mnn \
    --embedding_size 256
```

--------------------------------

### Download and extract SAD model

Source: https://github.com/wenet-e2e/wespeaker/blob/master/runtime/server/diarization_gpu/README.md

Downloads the Silero VAD v3.1 zip, extracts it, and copies the silero_vad.jit file to the SAD directory.

```bash
export SAD_DIR=/workspace/SAD
wget -c https://github.com/snakers4/silero-vad/archive/refs/tags/v3.1.zip -O external_tools/silero-vad-v3.1.zip
unzip -o external_tools/silero-vad-v3.1.zip -d external_tools
cp external_tools/silero-vad-3.1/files/silero_vad.jit $SAD_DIR/

```

--------------------------------

### Calculate similarity between two speech samples

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/runtime.md

Runs the asv_main tool to compute the similarity between two speech files using a threshold of 0.5. Requires the ONNX model and embedding size.

```sh
export GLOG_logtostderr=1
export GLOG_v=2
onnx_dir=your_model_dir
./build/bin/asv_main \
    --enroll_wav wav1_path \
    --test_wav wav2_path \
    --threshold 0.5 \
    --speaker_model_path $onnx_dir/final.onnx \
    --embedding_size 256
```

--------------------------------

### Run WavLM training and scoring

Source: https://github.com/wenet-e2e/wespeaker/blob/master/examples/voxceleb/v2/README.md

Run the WavLM training and scoring pipeline from stage 3 to stage 9. This uses the WavLM Large pre-trained frontend with ECAPA_TDNN_GLOB_c512-ASTP-emb192 speaker model and a Frozen => Joint ft => Joint lmft training strategy.

```bash
bash run_wavlm.sh --stage 3 --stop_stage 9
```

--------------------------------

### Stage 1: Prepare datasets

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/vox.md

Shell script block that prepares the voxceleb1, voxceleb2, MUSAN, and RIRS_NOISES datasets. It runs the local prepare_data.sh script from stage 2 to 4, assuming data archives are already downloaded.

```bash
if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then
  echo "Prepare datasets ..."
  ./local/prepare_data.sh --stage 2 --stop_stage 4 --data ${data}
fi
```

--------------------------------

### wespeaker CLI

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/python_package.md

Command-line interface for WeSpeaker. Supports tasks: embedding, embedding_kaldi, similarity, diarization, diarization_list. Parameters include task, language, pretrain, device, audio_file, audio_file2, wav_scp, resample_rate, vad, output_file, and model selection flags.

```APIDOC
## wespeaker CLI

### Description
Command-line interface for WeSpeaker speaker recognition tasks.

### Usage
```sh
$ wespeaker --task embedding --audio_file audio.wav --output_file embedding.txt
$ wespeaker --task embedding_kaldi --wav_scp wav.scp --output_file /path/to/embedding
$ wespeaker --task similarity --audio_file audio.wav --audio_file2 audio2.wav
$ wespeaker --task diarization --audio_file audio.wav
$ wespeaker --task diarization --audio_file audio.wav --device cuda:0 # use CUDA on Windows/Linux
$ wespeaker --task diarization --audio_file audio.wav --device mps    # use Metal Performance Shaders on MacOS
```

### Parameters
- `-t` or `--task` (string) - Required - Task to perform. Supported: embedding, embedding_kaldi, similarity, diarization, diarization_list.
- `-l` or `--language` (string) - Optional - Use Chinese/English speaker models.
- `-p` or `--pretrain` (string) - Optional - Path of pretrained model directory containing `avg_model.pt` and `config.yaml`.
- `--device` (string) - Optional - PyTorch device: `cpu`, `cuda`, `cuda:0`, or `mps`.
- `--campplus` (flag) - Optional - Use `campplus_cn_common_200k` of damo.
- `--eres2net` (flag) - Optional - Use `res2net_cn_common_200k` of damo.
- `--vblinkp` (flag) - Optional - Use sam_resnet34 model pretrained on VoxBlink2.
- `--vblinkf` (flag) - Optional - Use sam_resnet34 model pretrained on VoxBlink2 and finetuned on VoxCeleb2.
- `--w2vbert2_mfa` (flag) - Optional - Use w2v-bert2 model pretrained.
- `--audio_file` (string) - Optional - Input audio file path.
- `--audio_file2` (string) - Optional - Input audio file2 path, specifically for similarity task.
- `--wav_scp` (string) - Optional - Input wav.scp file in kaldi format (each line: key wav_path).
- `--resample_rate` (integer) - Optional - Resample rate (default: 16000).
- `--vad` (boolean) - Optional - Apply VAD or not for input audios (default: true).
- `--output_file` (string) - Optional - Output file to save speaker embedding. If using kaldi wav_scp, output will be `output_file.ark` and `output_file.scp`.

### Notes
- Use `-h` for details.
- Pretrained models must be renamed to `avg_model.pt` and `config.yaml`.
- Default models: english: `ResNet221_LM` pretrained on VoxCeleb; chinese: `ResNet34_LM` pretrained on CnCeleb.

```

--------------------------------

### Test single audio with client.py

Source: https://github.com/wenet-e2e/wespeaker/blob/master/runtime/server/diarization_gpu/README.md

Runs the client script to perform diarization on a single audio file. The --url option specifies the server address (localhost:8001 for gRPC). Output RTTM files are concatenated into a single rttm file.

```bash
# Test one audio
export output_directory="output"
mkdir -p $output_directory
python client.py --url=localhost:8001 --audio_file=/ws/test_data/abjxc.wav --output_directory=$output_directory
cat $output_directory/rttm* > $output_directory/rttm
```

--------------------------------

### WeSpeaker Python API usage

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/python_package.md

Demonstrates loading a model, setting device, extracting embeddings, computing similarity, diarization, and speaker registration/recognition.

```python
import wespeaker

model = wespeaker.load_model('chinese')
# set the device on which tensors are or will be allocated.
model.set_device('cuda:0')

# embedding/embedding_kaldi/similarity/diarization
embedding = model.extract_embedding('audio.wav')
utt_names, embeddings = model.extract_embedding_list('wav.scp')
similarity = model.compute_similarity('audio1.wav', 'audio2.wav')
diar_result = model.diarize('audio.wav', 'give_this_utt_a_name')

# register and recognize
model.register('spk1', 'spk1_audio1.wav')
model.register('spk2', 'spk2_audio1.wav')
model.register('spk3', 'spk3_audio1.wav')
result = model.recognize('spk1_audio2.wav')
```

--------------------------------

### Stage 7: Export trained model with export_jit.py

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/vox.md

Exports the trained model using Libtorch via wenet/bin/export_jit.py. The exported model files can be used for C++ inference in the runtime.

```shell
if [ ${stage} -le 6 ] && [ ${stop_stage} -ge 6 ]; then
  # Export the best model you want
  python wenet/bin/export_jit.py \
    --config $dir/train.yaml \
    --checkpoint $dir/avg_${average_num}.pt \
    --output_file $dir/final.zip
fi
```

--------------------------------

### Download pre-trained speaker model

Source: https://github.com/wenet-e2e/wespeaker/blob/master/runtime/server/diarization_gpu/README.md

Downloads the voxceleb_resnet34_LM.onnx speaker model to the specified directory.

```bash
export SPK_MODEL_DIR=/workspace/pretrained_models
mkdir -p ${SPK_MODEL_DIR}
wget -c https://wespeaker-1256283475.cos.ap-shanghai.myqcloud.com/models/voxceleb/voxceleb_resnet34_LM.onnx -O ${SPK_MODEL_DIR}/voxceleb_resnet34_LM.onnx

```

--------------------------------

### Test multiple audios with wav.scp

Source: https://github.com/wenet-e2e/wespeaker/blob/master/runtime/server/diarization_gpu/README.md

Runs the client script to process multiple audio files listed in a wav.scp file. The --wavscp option specifies the path to the wav.scp file. Output RTTM files are concatenated into a single rttm file.

```bash
# Test a bunch of audios
export wav_scp_dir=/ws/test_data
python client.py --url=localhost:8001 --wavscp=$wav_scp_dir/wav.scp --output_directory="outp"
cat $output_directory/rttm* > $output_directory/rttm
```

--------------------------------

### Generate test input and run perf_analyzer

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/runtime.md

Generates a test input for the pipeline and runs perf_analyzer to measure performance. The command uses concurrency range 200 to 1000 in steps of 200.

```bash
cd client/
# generate test input
python3 generate_input.py --audio_file=test.wav --seconds=2.02

perf_analyzer -m speaker -b 1 --concurrency-range 200:1000:200 --input-data=input.json -u localhost:8000
```

--------------------------------

### Set speaker model repository path

Source: https://github.com/wenet-e2e/wespeaker/blob/master/runtime/server/diarization_gpu/README.md

Sets the environment variable for the speaker model repository directory.

```bash
export SPK_MODEL_REPO="wespeaker/runtime/server/x86_gpu/model_repo/"

```

--------------------------------

### model.register

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/python_package.md

Register a speaker with an audio sample.

```APIDOC
## model.register

### Description
Register a speaker with an audio sample.

### Method
register

### Parameters
- `spk_id` (string) - Required - Speaker identifier.
- `audio_file` (string) - Required - Path to the audio file for the speaker.

### Example
```python
model.register('spk1', 'spk1_audio1.wav')
```

```

--------------------------------

### Clone WeSpeaker Repository for Development

Source: https://github.com/wenet-e2e/wespeaker/blob/master/README.md

Clone the WeSpeaker repository from GitHub for development and deployment.

```sh
git clone https://github.com/wenet-e2e/wespeaker.git
```

--------------------------------

### Reformat data to shards or raw lists and convert MUSAN/RIR to LMDB

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/vox.md

Converts vox2_dev and vox1 datasets to shard lists (with 1000 utterances per shard) or raw lists based on data_type, then converts MUSAN and RIR datasets to LMDB format. Used in stage 2 of the recipe.

```bash
if [ ${stage} -le 2 ] && [ ${stop_stage} -ge 2 ]; then
  echo "Covert train and test data to ${data_type}..."
  for dset in vox2_dev vox1; do
    if [ $data_type == "shard" ]; then
      python tools/make_shard_list.py --num_utts_per_shard 1000 \
          --num_threads 16 \
          --prefix shards \
          --shuffle \
          ${data}/$dset/wav.scp ${data}/$dset/utt2spk \
          ${data}/$dset/shards ${data}/$dset/shard.list
    else
      python tools/make_raw_list.py ${data}/$dset/wav.scp \
          ${data}/$dset/utt2spk ${data}/$dset/raw.list
    fi
  done
  # Convert all musan data to LMDB
  python tools/make_lmdb.py ${data}/musan/wav.scp ${data}/musan/lmdb
  # Convert all rirs data to LMDB
  python tools/make_lmdb.py ${data}/rirs/wav.scp ${data}/rirs/lmdb
fi
```

--------------------------------

### Stage 4: Extract Speaker Embeddings (average_model.py, convert_repvgg.py, extract_vox.sh)

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/vox.md

Shell script for stage 4 of the recipe. It averages model weights from the last num_avg checkpoints, optionally converts RepVGG models to evaluation format, and extracts embeddings. Embeddings are stored in exp/your_exp/embeddings in kaldi scp,ark format; logs are in exp/your_exp/embeddings/log.

```bash
if [ ${stage} -le 4 ] && [ ${stop_stage} -ge 4 ]; then
  echo "Do model average ..."
  avg_model=$exp_dir/models/avg_model.pt
  python wespeaker/bin/average_model.py \
    --dst_model $avg_model \
    --src_path $exp_dir/models \
    --num ${num_avg}

  model_path=$avg_model
  if [[ $config == *repvgg*.yaml ]]; then
    echo "convert repvgg model ..."
    python wespeaker/models/convert_repvgg.py \
      --config $exp_dir/config.yaml \
      --load $avg_model \
      --save $exp_dir/models/convert_model.pt
    model_path=$exp_dir/models/convert_model.pt
  fi

  echo "Extract embeddings ..."
  local/extract_vox.sh \
    --exp_dir $exp_dir --model_path $model_path \
    --nj 4 --gpus $gpus --data_type $data_type --data ${data}
fi
```

--------------------------------

### Self-supervised dataloader: data reading and shuffling

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/vox_ssl.md

Defines data reading methods based on data storage format (shard, raw, or feature) and applies global and local shuffling. This is part of the dataset preparation for self-supervised training, as defined in wespeaker/ssl/dataset/dataset.py.

```python
dataset = DataList(lists, shuffle=shuffle)
if data_type == 'shard':
    dataset = Processor(dataset, processor.url_opener)
    dataset = Processor(dataset, processor.tar_file_and_group)
elif data_type == 'raw':
    dataset = Processor(dataset, processor.parse_raw)
else:
    dataset = Processor(dataset, processor.parse_feat)
# Local shuffle
if shuffle:
    dataset = Processor(dataset, processor.shuffle,
                        **configs['shuffle_args'])
```

--------------------------------

### WeSpeaker Python Programming Usage

Source: https://github.com/wenet-e2e/wespeaker/blob/master/README.md

Demonstrates how to use the WeSpeaker Python API to load a model, extract embeddings, compute similarity, and perform diarization.

```python
import wespeaker

model = wespeaker.load_model('chinese')
embedding = model.extract_embedding('audio.wav')
utt_names, embeddings = model.extract_embedding_list('wav.scp')
similarity = model.compute_similarity('audio1.wav', 'audio2.wav')
diar_result = model.diarize('audio.wav')
```

--------------------------------

### Export model to ONNX (dynamic and static shape)

Source: https://github.com/wenet-e2e/wespeaker/blob/master/runtime/mnn/README.md

Commands to export a WeSpeaker model to ONNX format using export_onnx.py. The first command uses dynamic shape; the second (commented out) uses static shape with --num_frames 198.

```sh
python wespeaker/bin/export_onnx.py \
  --config config.yaml \
  --checkpoint model.pt \
  --output_model model.onnx
  # When it finishes, you can find `model.onnx`.
# 2. static shape
# python wespeaker/bin/export_onnx.py \
#   --config config.yaml \
#   --checkpoint model.pt \
#   --output_model model.onnx \
#   --num_frames 198
```

--------------------------------

### Extract speaker embeddings from ONNX model

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/pretrained.md

Use this command to extract speaker embeddings from a pretrained ONNX model. The onnx_path should point to the downloaded ONNX model, and wav_path should point to a 16kHz wave file.

```bash
# Download the pretrained model in onnx format and save it as onnx_path
# wav_path is the path to your wave file (16k)
python wespeaker/bin/infer_onnx.py --onnx_path $onnx_path --wav_path $wav_path
```

--------------------------------

### Apply spectral clustering (Stage 6)

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/voxconverse_diar.md

Shell script to apply spectral clustering on the extracted embeddings. It removes any existing label file, then runs wespeaker/diar/spectral_clusterer.py with the embedding SCP and output path. The output file contains lines with subseg_id and spk_id separated by a blank space.

```bash
if [ ${stage} -le 6 ] && [ ${stop_stage} -ge 6 ]; then

    [ -f "exp/spectral_cluster/${partition}_${sad_type}_sad_labels" ] && rm exp/spectral_cluster/${partition}_${sad_type}_sad_labels

    echo "Doing spectral clustering and store the result in exp/spectral_cluster/${partition}_${sad_type}_sad_labels"
    echo "..."
    python3 wespeaker/diar/spectral_clusterer.py \
            --scp exp/${partition}_${sad_type}_sad_embedding/emb.scp \
            --output exp/spectral_cluster/${partition}_${sad_type}_sad_labels
fi
```

--------------------------------

### Export model to ONNX

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/runtime.md

Exports a trained Wespeaker model to ONNX format using the export_onnx.py script. Requires specifying the experiment directory, checkpoint, and output model path.

```sh
exp=exp  # Change it to your experiment dir
onnx_dir=onnx
python wespeaker/bin/export_onnx.py \
  --config $exp/config.yaml \
  --checkpoint $exp/avg_model.pt \
  --output_model $onnx_dir/final.onnx

# When it finishes, you can find `final.onnx`.
```

--------------------------------

### Run voxconverse evaluation script

Source: https://github.com/wenet-e2e/wespeaker/blob/master/runtime/server/diarization_gpu/README.md

Runs the voxconverse evaluation script to access evaluation data.

```bash
bash $WeSpeaker/examples/voxconverse/v1/run.sh --stage 2 --stop_stage 2

```

--------------------------------

### Stage 6: Score normalization with score_norm.sh

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/vox.md

Runs adaptive score normalization (asnorm or snorm) on the evaluation set. The final results are stored in exp/your_exp/scores/vox1_${}${top_n}_result. --score_norm_method selects asnorm or snorm; --top_n sets the negative cohort size.

```shell
if [ ${stage} -le 6 ] && [ ${stop_stage} -ge 6 ]; then
  echo "Score norm ..."
  local/score_norm.sh \
    --stage 1 --stop-stage 3 \
    --score_norm_method $score_norm_method \
    --cohort_set vox2_dev \
    --top_n $top_n \
    --data ${data} \
    --exp_dir $exp_dir \
    --trials "$trials"
fi
```

--------------------------------

### Set CUDA environment variables for GPU

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/runtime.md

Sets the PATH and LD_LIBRARY_PATH to include CUDA 11.1 when using GPU. Required for GPU builds.

```bash
export PATH=/usr/local/cuda-11.1/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-11.1/lib64:${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
```

--------------------------------

### Extract embeddings with extract_emb_main

Source: https://github.com/wenet-e2e/wespeaker/blob/master/runtime/mnn/README.md

Command to run the embedding extraction tool. It outputs RTF to the console and writes embeddings to a txt file. The `samples_per_chunk` parameter is set to 80000 (5s at 16kHz). Note: if -1, processes whole sentence; otherwise averages chunk embeddings.

```sh
export GLOG_logtostderr=1
export GLOG_v=2
./build/bin/extract_emb_main \
  --wav_scp wav.scp \
  --result embedding.txt \
  --speaker_model_path model.mnn \
  --embedding_size 256 \
  --samples_per_chunk  80000  # 5s
```

--------------------------------

### Stage 5: Score Evaluation Set (score.sh)

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/vox.md

Shell script for stage 5 of the recipe. It scores all trials listed in the local variable 'trials' using cosine similarity. Results (EER, minDCF) are stored in exp/your_exp/scores/vox1_cos_result, and detailed scores for each trial are in exp/your_exp/scores/trial_xx.score.

```bash
if [ ${stage} -le 5 ] && [ ${stop_stage} -ge 5 ]; then
  echo "Score ..."
  local/score.sh \
    --stage 1 --stop-stage 2 \
    --data ${data} \
    --exp_dir $exp_dir \
    --trials "$trials"
fi
```

--------------------------------

### Stage 4: Average model and extract embeddings

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/vox_ssl.md

Shell script for stage 4 of the recipe. It averages the last num_avg checkpoints using average_dino_model.py (or average_contrastive_model.py for MoCo/SimCLR) and then extracts embeddings with local/extract_vox.sh. Embeddings are stored in exp/your_exp/embeddings in kaldi scp,ark format.

```shell
if [ ${stage} -le 4 ] && [ ${stop_stage} -ge 4 ]; then
  echo "Do model average ..."
  avg_model=$exp_dir/models/avg_model.pt
  python wespeaker/ssl/bin/average_dino_model.py \
    --dst_model $avg_model \
    --src_path $exp_dir/models \
    --num ${num_avg}

  echo "Extract embeddings ..."
  local/extract_vox.sh \
    --exp_dir $exp_dir --model_path $avg_model \
    --nj 4 --gpus $gpus --data_type $data_type --data ${data}
fi
```

--------------------------------

### Export trained model with export_jit.py

Source: https://github.com/wenet-e2e/wespeaker/blob/master/docs/vox_ssl.md

Shell script to export the best model using the export_jit.py script. It requires the configuration file, the averaged model checkpoint, and specifies the output file. The exported model is intended for C++ inference in the runtime.

```shell
if [ ${stage} -le 6 ] && [ ${stop_stage} -ge 6 ]; then
  echo "Export the best model ..."
  python wespeaker/bin/export_jit.py \
    --config $exp_dir/config.yaml \
    --checkpoint $exp_dir/models/avg_model.pt \
    --output_file $exp_dir/models/final.zip
fi
```
Tokens:
Raw