WhisperSpeech

https://whisperspeech.github.io/WhisperSpeech/
Admin
WhisperSpeech is an open-source, customizable text-to-speech system built by inverting Whisper,...
Tokens:5,418
Snippets:69
Trust Score:9.7
Update:3 months ago
Show doc for...
Tokens:
Raw
### Run Local Preprocessing

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Example of executing a preprocessing step locally on a single audio file. This command creates a directory and runs the `vad` module.

```bash
mkdir -p guttenberg-preproc && cd guttenberg-preproc
python -m whisperspeech.vad ../guttenberg-audiobooks/guttenberg-audiobooks-raw-000010.tar
```

--------------------------------

### Install Tarp Tool

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Installs the tarp tool, which is used for creating and manipulating webdataset tar files. This is a prerequisite for dataset conversion.

```bash
go install -v github.com/collabora/tarp/tarp@latest
```

--------------------------------

### Example Transcript Line

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

An example of a line from the transcripts.txt file, showing an ID and its corresponding transcription text.

```text
10065_10039_000000	ses vêtements devinrent tout brillants de lumière et blancs comme la neige en sorte qu'il n'y a point de foulon sur la terre qui puisse en faire d'aussi blancs
```

--------------------------------

### Run Parallel Preprocessing on Cluster

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Example of running multiple preprocessing jobs in parallel on a cluster. This command finds input shards and uses a batch script to process them with `prepare_s2a_dataset`.

```bash
mkdir -p whisperspeech-s2a-512c-dim64 && cd whisperspeech-s2a-512c-dim64
find ../librilight/ -name 'librilight-small-*.tar'| ~/clapa1/run-batch 24 \
    'python -m whisperspeech.prepare_s2a_dataset $FILE ../librilight-preproc
            --vq_model ~/clapa1/scratch/vqmodel-512c-dim64-4e-hyptuned-32gpu.model
            --batch_size 8'
```

--------------------------------

### Create Training Dataset Configuration Files

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Define dataset configurations for training using 'cat' and 'EOF' for text-to-speech (t2s) and speech-to-audio (s2a) tasks. These files specify data sources, sample counts, and exclusion lists.

```bash
cat > mls-fr-t2s-train.dataset <<EOF
multilingual-librispeech-webdataset/*-medium-txt-*.tar.gz multilingual-librispeech-webdataset/vq-en+pl/ 390203 --txt_kind='medium-txt' --language=fr --exclude_files multilingual-librispeech-webdataset/unseen-speakers-eqvad
EOF
cat > mls-fr-s2a-train.dataset <<EOF
multilingual-librispeech-webdataset/*-atoks-*.tar.gz multilingual-librispeech-webdataset/vq-en+pl/ 338362  --language=fr --exclude_files multilingual-librispeech-webdataset/unseen-speakers-maxvad
EOF
```

--------------------------------

### Create Validation Dataset Configuration Files

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Define dataset configurations for validation using 'cat' and 'EOF'. These files specify the validation data shards and expected sample counts for unseen speakers.

```bash
cat > mls-fr-s2a-val-unseen-speakers.dataset <<EOF
multilingual-librispeech-webdataset/unseen-speakers-maxvad.tar.gz multilingual-librispeech-webdataset/vq-en+pl/ 512 --language fr
EOF
cat > mls-fr-t2s-val-unseen-speakers.dataset <<EOF
multilingual-librispeech-webdataset/unseen-speakers-eqvad.tar.gz multilingual-librispeech-webdataset/vq-en+pl/ 512 --txt_kind 'medium-txt' --language fr
EOF
```

--------------------------------

### Upload Dataset to Hugging Face

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Logs into the Hugging Face CLI and uploads the prepared webdataset shards to a specified repository.

```bash
huggingface-cli login
huggingface-cli upload --repo-type dataset collabora/multilingual-librispeech-webdataset .
```

--------------------------------

### Create and Split Webdataset Shards

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Uses the generated tarp script to create webdataset tar files and then splits them into 2GB shards. This command is repeated for train, test, and dev splits.

```bash
python3 make-script.py  mls_french_opus/train/transcripts.txt \
  | /root/go/bin/tarp create -o - - \
  | /root/go/bin/tarp split -s 2e9 -o 'mls_french_train-audio-%06d.tar' -
```

--------------------------------

### Download and Unpack Dataset

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Downloads the Multilingual Libri Speech (MLS) French dataset using aria2c and then unpacks the downloaded tar.gz file.

```bash
aria2c -x10 https://dl.fbaipublicfiles.com/mls/mls_french_opus.tar.gz
tar -xf mls_french_opus.tar.gz
```

--------------------------------

### Create Unseen Speaker Validation Sets

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Filter sample IDs to create lists of unseen speakers for validation sets. This ensures validation on speakers not encountered during training.

```bash
< all-samples-maxvad grep 'train/1579\|train/2033\|train/3182\|train/12981\|train/2284\|train/2297\|train/6348\|train/7200\|train/7679\|train/1989' >
unseen-speakers-maxvad
< all-samples-eq grep 'train/1579\|train/2033\|train/3182\|train/12981\|train/2284\|train/2297\|train/6348\|train/7200\|train/7679\|train/1989' > unseen-speakers-eqvad__
```

--------------------------------

### Generate Speaker ID Histogram

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Pipe sample IDs to 'awk', 'sort', and 'uniq' to create a histogram of speaker IDs. This aids in selecting diverse speakers for validation.

```bash
< all-samples-maxvad awk -F_ '{ print $1; }'|sort|uniq -c|sort -n|less__
```

--------------------------------

### Perform Encodec Compression

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Compresses audio data using the Encodec model. This step prepares the audio for Speech-to-Audio (S2A) tokenization.

```bash
parallel --eta -j2 python -m whisperspeech.prepare_s2a_atoks --batch_size 4 {} ::: *.tar
```

--------------------------------

### Extract Sample IDs and Count

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Use 'parallel' and 'sed' to extract sample IDs from tar archives and count them. This helps in analyzing the dataset distribution.

```bash
parallel tar tf {} ::: stoks/*-atoks-3kbps-*.tar.gz | sed -e 's/\.atoks\.npy//' > all-samples-maxvad
parallel tar tf {} ::: stoks/*-small.en-txt-*.tar.gz | sed -e 's/\.txt//' > all-samples-eqvad
wc -l all-samples-maxvad__
```

--------------------------------

### Join Multiple WebDatasets

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Loads all audio shards and merges another WebDataset containing VAD information. Ensure the derived dataset function correctly maps original shard names to corresponding VAD shards.

```python
ds = wds.WebDataset([str(x) for x in Path('librilight/').glob('*.tar')]).compose( # load all audio shards
    wds.decode(wds.torch_audio), # decode the audio data
    vq_stoks.merge_in( # merge another WebDataset
        # for each audio (`raw`) shard, find the path and name of a corresponding `vad` shard
        vq_stoks.derived_dataset('librilight-processed/', 'vad')
    ),
)
```

--------------------------------

### Prepare Transcripts for T2S Training

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Prepares text transcripts for Text-to-Speech (T2S) training using a specified transcription model ('medium') and language ('fr'). The --batch_size parameter controls the processing batch size.

```bash
parallel --eta -j1 python -m whisperspeech.prepare_t2s_txts --transcription_model medium --language fr --batch_size 32 {} ::: *.tar
```

--------------------------------

### List Files in a WebDataset Tarball

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

View the contents of a WebDataset tarball to understand the structure of stored samples. This command is useful for inspecting the files within a shard.

```bash
tar tf whisperspeech-s2a-512c-dim64/librilight-small-000.tar.gz |head -6
```

--------------------------------

### Perform Voice Activity Detection (VAD)

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Processes all .tar files in the current directory to perform voice activity detection using the whisperspeech.vad module. The --eta flag shows an estimated time of arrival.

```bash
parallel --eta -j3 python -m whisperspeech.vad {} ::: ./*.tar
```

--------------------------------

### Split Validation Datasets

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Use the 'whisperspeech.split_out_val_datasets' Python module to split validation samples into separate webdataset shards for efficient validation.

```python
python -m whisperspeech.split_out_val_datasets *-atoks-* unseen-speakers-maxvad
python -m whisperspeech.split_out_val_datasets '*-txt-*' unseen-speakers-eqvad
cd stoks && python -m whisperspeech.split_out_val_datasets '*-maxvad-stoks-*' ../unseen-speakers-maxvad
cd stoks && python -m whisperspeech.split_out_val_datasets '*-eqvad-stoks-*' ../unseen-speakers-eqvad__
```

--------------------------------

### Helper Function for Derived Datasets

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Creates a helper function to open a derived WebDataset based on the original shard file name. This function is used within the `merge_in` operation.

```python
def derived_dataset(path, kind):
    def deriver(url):
        url = str(Path(path)/(Path(url).name.replace("raw", kind) + ".gz"))
        return wds.WebDataset(wds.SimpleShardList([url])).decode()
    return deriver
```

--------------------------------

### Extract Semantic Tokens for T2S and S2A

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Extracts semantic tokens for both Text-to-Speech (T2S) using 'eqvad' and Speech-to-Audio (S2A) using the default kind. Requires a pre-trained VQ model.

```bash
parallel --eta -j1 python -m whisperspeech.extract_stoks --batch_size 16 --vq_model ../nbs/vqmodel-medium-en+pl-512c-dim64.model {} ::: *.tar
parallel --eta -j1 python -m whisperspeech.extract_stoks --kind eqvad --batch_size 16 --vq_model ../nbs/vqmodel-medium-en+pl-512c-dim64.model {} ::: *.tar
```

--------------------------------

### Tarp Script Format

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Shows the format of lines generated for a tarp script, mapping audio and text files to their respective webdataset entries.

```text
train/10065_10039_000000.opus file:mls_french_opus/train/audio/10065/10039/10065_10039_000000.opus
train/10065_10039_000000.txt text:ses vêtements devinrent tout brillants de lumière et blancs comme la neige en sorte qu'il n'y a point de foulon sur la terre qui puisse en faire d'aussi blancs
```

--------------------------------

### Extract Speaker Embeddings

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Extracts speaker embeddings for each fragment in the .tar files using the whisperspeech.extract_spk_emb module. The --batch_size parameter controls the number of samples processed in parallel.

```bash
parallel --eta -j2 python -m whisperspeech.extract_spk_emb --batch_size 16 {} ::: ./*.tar
```

--------------------------------

### Python Script to Generate Tarp Input

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

A Python script that reads a transcript file and generates lines in the format required by tarp for creating webdataset entries.

```python
import sys

fname = sys.argv[1]
dir, split, _ = fname.rsplit("/", 2)

for ln in open(fname):
    id, txt = ln.split("\t")
    a,b,c = id.split("_")
    txt = txt.replace("\n", "")
    print(f"{split}/{id}.opus file:{dir}/{split}/audio/{a}/{b}/{id}.opus
{split}/{id}.txt text:{txt}")
```

--------------------------------

### Perform VAD Segment Merging

Source: https://whisperspeech.github.io/WhisperSpeech/dataset%20preparation.html

Merges Voice Activity Detection segments. This step is performed separately to ensure reproducibility for subsequent steps. It is run twice with different parallelization factors.

```bash
parallel --eta -j16 python -m whisperspeech.vad_merge --eqvad {} ::: *.tar
parallel --eta -j16 python -m whisperspeech.vad_merge {} ::: *.tar
```
Tokens:
Raw