SpeechTokenizer

https://github.com/zhangxinfd/speechtokenizer
Admin
SpeechTokenizer is a unified speech tokenizer for speech language models that utilizes an...
Tokens:1,212
Snippets:15
Trust Score:6.6
License:Apache-2.0
Update:2 months ago
Show doc for...
Tokens:
Raw
### Install SpeechTokenizer

Source: https://github.com/zhangxinfd/speechtokenizer/blob/main/README.md

Commands to install the package using pip or by performing a local installation from the cloned repository.

```bash
pip install -U speechtokenizer

# or you can clone the repo and install locally
git clone https://github.com/ZhangXInFD/SpeechTokenizer.git
cd SpeechTokenizer
pip install .
```

--------------------------------

### Quick Start Training

Source: https://github.com/zhangxinfd/speechtokenizer/blob/main/README.md

Execute the provided shell scripts to extract representations and start the training process.

```shell
cd SpeechTokenizer

# Extact semantic representation
bash scripts/hubert_rep_extract.sh

# Train
bash scripts/train_example.sh
```

--------------------------------

### Load SpeechTokenizer Model

Source: https://github.com/zhangxinfd/speechtokenizer/blob/main/README.md

Initialize the SpeechTokenizer model from a configuration file and a pre-trained checkpoint.

```python
from speechtokenizer import SpeechTokenizer

config_path = '/path/config.json'
ckpt_path = '/path/SpeechTokenizer.pt'
model = SpeechTokenizer.load_from_checkpoint(config_path, ckpt_path)
model.eval()
```

--------------------------------

### Train SpeechTokenizer

Source: https://github.com/zhangxinfd/speechtokenizer/blob/main/README.md

Initialize the generator, discriminators, and trainer to begin or continue the training process.

```python
from speechtokenizer import SpeechTokenizer, SpeechTokenizerTrainer
from speechtokenizer.discriminators import MultiPeriodDiscriminator, MultiScaleDiscriminator, MultiScaleSTFTDiscriminator
import json


# Load model and trainer config
with open('<CONFIG_FILE_PATH>') as f:
    cfg = json.load(f)

# Initialize SpeechTokenizer
generator = SpeechTokenizer(cfg)

# Initialize the discriminators. You can add any discriminator that is not yet implemented in this repository, as long as the output format remains consistent with the discriminators in `speechtokenizer.discriminators`.
discriminators = {'mpd':MultiPeriodDiscriminator(), 'msd':MultiScaleDiscriminator(), 'mstftd':MultiScaleSTFTDiscriminator(32)}

# Initialize Trainer
trainer = SpeechTokenizerTrainer(generator=generator,
                                discriminators=discriminators,
                                cfg=cfg)

# Start training
trainer.train()

# Continue training from checkpoints
trainer.continue_train()
```

--------------------------------

### Decode Discrete Representations

Source: https://github.com/zhangxinfd/speechtokenizer/blob/main/README.md

Reconstruct audio waveforms from extracted semantic and timbre tokens.

```python
# Concatenating semantic tokens (RVQ_1) and supplementary timbre tokens and then decoding
wav = model.decode(torch.cat([RVQ_1, RVQ_supplement], axis=0))

# Decoding from RVQ-i:j tokens from the ith quantizers to the jth quantizers
wav = model.decode(codes[i: (j + 1)], st=i)
```

--------------------------------

### Extract Discrete Representations

Source: https://github.com/zhangxinfd/speechtokenizer/blob/main/README.md

Load audio, resample to the model's required rate, and extract semantic and timbre codes using the encoder.

```python
import torchaudio
import torch

# Load and pre-process speech waveform
wav, sr = torchaudio.load('<SPEECH_FILE_PATH>')

# monophonic checking
if wav.shape(0) > 1:
    wav = wav[:1,:]

if sr != model.sample_rate:
    wav = torchaudio.functional.resample(wav, sr, model.sample_rate)

wav = wav.unsqueeze(0)

# Extract discrete codes from SpeechTokenizer
with torch.no_grad():
    codes = model.encode(wav) # codes: (n_q, B, T)

RVQ_1 = codes[:1, :, :] # Contain content info, can be considered as semantic tokens
RVQ_supplement = codes[1:, :, :] # Contain timbre info, complete info lost by the first quantizer
```

--------------------------------

### Cite SpeechTokenizer in BibTeX

Source: https://github.com/zhangxinfd/speechtokenizer/blob/main/README.md

Use this BibTeX entry to cite the SpeechTokenizer paper in academic publications.

```Tex
@misc{zhang2023speechtokenizer,
      title={SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models}, 
      author={Xin Zhang and Dong Zhang and Shimin Li and Yaqian Zhou and Xipeng Qiu},
      year={2023},
      eprint={2308.16692},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```
Tokens:
Raw