### Setup Conda Environment

Source: https://github.com/jasonppy/voicecraft/blob/master/README.md

Commands to create the Conda environment and install necessary dependencies for Voicecraft.

```bash
conda create -n voicecraft python=3.9.16
conda activate voicecraft

pip install -e git+https://github.com/facebookresearch/audiocraft.git@c5157b5bf14bf83449c17ea1eeb66c19fb4bc7f0#egg=audiocraft
pip install xformers==0.0.22
pip install torchaudio==2.0.2 torch==2.0.1 # this assumes your system is compatible with CUDA 11.7, otherwise checkout https://pytorch.org/get-started/previous-versions/#v201
apt-get install ffmpeg # if you don't already have ffmpeg installed
apt-get install espeak-ng # backend for the phonemizer installed below
pip install tensorboard==2.16.2
pip install phonemizer==3.2.1
pip install datasets==2.16.0
pip install torchmetrics==0.11.1
pip install huggingface_hub==0.22.2
# install MFA for getting forced-alignment, this could take a few minutes
conda install -c conda-forge montreal-forced-aligner=2.2.17 openfst=1.8.2 kaldi=5.5.1068
# install MFA english dictionary and model
mfa model download dictionary english_us_arpa
mfa model download acoustic english_us_arpa
# pip install huggingface_hub
# conda install pocl # above gives an warning for installing pocl, not sure if really need this

# to run ipynb
conda install -n voicecraft ipykernel --no-deps --force-reinstall
```

--------------------------------

### Environment Setup for Training

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Set up the Python environment for training VoiceCraft models using Conda and pip. Install the audiocraft library from a specific Git commit.

```bash
conda create -n voicecraft python=3.9.16
conda activate voicecraft
pip install -e git+https://github.com/facebookresearch/audiocraft.git@c5157b5bf14bf83449c17ea1eeb66c19fb4bc7f0#egg=audiocraft
```

--------------------------------

### Start VoiceCraft Training

Source: https://github.com/jasonppy/voicecraft/blob/master/README.md

Execute the training script after preparing the dataset. Ensure you are in the correct directory.

```bash
conda activate voicecraft
cd ./z_scripts
bash e830M.sh
```

--------------------------------

### Install Dependencies

Source: https://github.com/jasonppy/voicecraft/blob/master/voicecraft-gradio-colab.ipynb

Installs required Python packages, system libraries, and external dependencies for VoiceCraft.

```bash
!pip install tensorboard
!pip install phonemizer
!pip install datasets
!pip install torchmetrics

!apt-get install -y espeak espeak-data libespeak1 libespeak-dev
!apt-get install -y festival*
!apt-get install -y build-essential
!apt-get install -y flac libasound2-dev libsndfile1-dev vorbis-tools
!apt-get install -y libxml2-dev libxslt-dev zlib1g-dev

!pip install -e git+https://github.com/facebookresearch/audiocraft.git@c5157b5bf14bf83449c17ea1eeb66c19fb4bc7f0#egg=audiocraft

!pip install -r "/content/VoiceCraft/gradio_requirements.txt"
!pip install typer==0.7.0
```

--------------------------------

### Launch Gradio Application

Source: https://github.com/jasonppy/voicecraft/blob/master/voicecraft-gradio-colab.ipynb

Starts the VoiceCraft Gradio interface with specified paths and sharing enabled.

```bash
!python /content/VoiceCraft/gradio_app.py --demo-path=/content/VoiceCraft/demo --tmp-path=/content/VoiceCraft/demo/temp --models-path=/content/VoiceCraft/pretrained_models --share
```

--------------------------------

### Install MFA Models

Source: https://github.com/jasonppy/voicecraft/blob/master/inference_tts.ipynb

Downloads required MFA models and dictionaries for forced alignment.

```bash
# # install MFA models and dictionaries if you haven't done so already, already done in the dockerfile or envrionment setup
# !source ~/.bashrc && \
#     conda activate voicecraft && \
#     mfa model download dictionary english_us_arpa && \
#     mfa model download acoustic english_us_arpa
```

--------------------------------

### Build and Run Docker Container

Source: https://github.com/jasonppy/voicecraft/blob/master/README.md

Commands to build the Docker image and start the Jupyter environment for Linux or Windows.

```bash
docker build --tag "voicecraft" .
```

```bash
./start-jupyter.sh  # linux
start-jupyter.bat   # windows
```

--------------------------------

### Launch Voicecraft Application

Source: https://github.com/jasonppy/voicecraft/blob/master/gradio_app.ipynb

Import and launch the Gradio app. This step assumes all prior dependencies and configurations are correctly set up. The application will start and be accessible via a web interface.

```python
from gradio_app import app
app.launch()
```

--------------------------------

### Install System Dependencies with Docker

Source: https://github.com/jasonppy/voicecraft/blob/master/gradio_app.ipynb

Run this command within a Docker container to update package lists and install necessary system libraries for audio processing and text-to-speech.

```bash
!source ~/.bashrc && \
    apt-get update && \
    apt-get install -y espeak espeak-data libespeak1 libespeak-dev && \
    apt-get install -y festival* && \
    apt-get install -y build-essential && \
    apt-get install -y flac libasound2-dev libsndfile1-dev vorbis-tools && \
    apt-get install -y libxml2-dev libxslt-dev zlib1g-dev
```

--------------------------------

### Install Python Dependencies with Docker

Source: https://github.com/jasonppy/voicecraft/blob/master/gradio_app.ipynb

Activate the 'voicecraft' conda environment and install Python packages from 'gradio_requirements.txt'. Ensure the conda environment is correctly set up before running.

```bash
!source ~/.bashrc && \
    conda activate voicecraft && \
    pip install -r gradio_requirements.txt
```

--------------------------------

### Install Gradio Dependencies

Source: https://github.com/jasonppy/voicecraft/blob/master/README.md

System and Python package requirements for running the Gradio interface locally.

```bash
apt-get install -y espeak espeak-data libespeak1 libespeak-dev
apt-get install -y festival*
apt-get install -y build-essential
apt-get install -y flac libasound2-dev libsndfile1-dev vorbis-tools
apt-get install -y libxml2-dev libxslt-dev zlib1g-dev
pip install -r gradio_requirements.txt
```

--------------------------------

### Text-to-Speech Inference Setup

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Prepare for TTS inference by loading the VoiceCraft model and tokenizers. This snippet sets up the necessary components and defines input audio and text paths for generating speech.

```python
import torch
import torchaudio
from data.tokenizer import AudioTokenizer, TextTokenizer, tokenize_audio, tokenize_text
from inference_tts_scale import inference_one_sample
from models import voicecraft

device = "cuda" if torch.cuda.is_available() else "cpu"

# Load model
model = voicecraft.VoiceCraft.from_pretrained("pyp1/VoiceCraft_giga830M_TTSEnhanced")
model.to(device)
model.eval()
config = model.args
phn2num = model.args.phn2num

# Initialize tokenizers
audio_tokenizer = AudioTokenizer(signature="./pretrained_models/encodec_4cb2048_giga.th", device=device)
text_tokenizer = TextTokenizer(backend="espeak")

# Input audio and transcript
audio_path = "./demo/5895_34622_000026_000002.wav"
original_transcript = "Gwynplaine had, besides, for his work and for his feats of strength,"
target_transcript = original_transcript + " I cannot believe this model can do text to speech!"

# Calculate prompt end frame (use 3-4 seconds of reference audio)
info = torchaudio.info(audio_path)
cut_off_sec = 3.6
prompt_end_frame = int(cut_off_sec * info.sample_rate)
```

--------------------------------

### Download MFA Models

Source: https://github.com/jasonppy/voicecraft/blob/master/inference_speech_editing.ipynb

Install Montreal Forced Aligner (MFA) models and dictionaries via shell commands within a conda environment.

```bash
# install MFA models and dictionaries if you haven't done so already
!source ~/.bashrc && \
    conda activate voicecraft && \
    mfa model download dictionary english_us_arpa && \
    mfa model download acoustic english_us_arpa
```

--------------------------------

### Load VoiceCraft Model from HuggingFace

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Load a pretrained VoiceCraft model from HuggingFace Hub. Ensure you have the necessary libraries installed and select the appropriate device (CUDA or CPU).

```python
import torch
from models import voicecraft

# Select device
device = "cuda" if torch.cuda.is_available() else "cpu"

# Load pretrained model from HuggingFace Hub
# Available models: giga330M, giga830M, giga330M_TTSEnhanced, giga830M_TTSEnhanced
model = voicecraft.VoiceCraft.from_pretrained("pyp1/VoiceCraft_giga830M_TTSEnhanced")
model.to(device)
model.eval()

# Access model configuration and phoneme mapping
config = model.args
phn2num = model.args.phn2num

print(f"Model loaded with {config.num_decoder_layers} decoder layers")
print(f"Audio codebooks: {config.n_codebooks}")
print(f"Vocabulary size: {len(phn2num)} phonemes")
```

--------------------------------

### Clone VoiceCraft Repository for Docker Usage

Source: https://github.com/jasonppy/voicecraft/blob/master/README.md

Clone the VoiceCraft repository to set up the environment for running inference with Docker. Ensure you have Docker installed with the NVIDIA container toolkit.

```bash
git clone git@github.com:jasonppy/VoiceCraft.git
cd VoiceCraft
```

--------------------------------

### Initialize Environment and Dependencies

Source: https://github.com/jasonppy/voicecraft/blob/master/inference_tts.ipynb

Sets up CUDA environment variables and imports necessary libraries for VoiceCraft.

```python
# import libs
# if this throws an error, something went wrong installing dependencies or changing the kernel above!
import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"   
os.environ["CUDA_VISIBLE_DEVICES"]="0"
os.environ["USER"] = "me" # TODO change this to your username

import torch
import torchaudio
import numpy as np
import random
from argparse import Namespace

from data.tokenizer import (
    AudioTokenizer,
    TextTokenizer,
)
from huggingface_hub import hf_hub_download
```

--------------------------------

### Initialize Audio and Text Tokenizers

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Initialize the AudioTokenizer using an EnCodec model and the TextTokenizer with an espeak backend. The AudioTokenizer requires a path to the EnCodec model, which will be downloaded if not present.

```python
import os
from data.tokenizer import AudioTokenizer, TextTokenizer

# Download EnCodec model if not present
encodec_path = "./pretrained_models/encodec_4cb2048_giga.th"
if not os.path.exists(encodec_path):
    os.makedirs("./pretrained_models", exist_ok=True)
    os.system(f"wget https://huggingface.co/pyp1/VoiceCraft/resolve/main/encodec_4cb2048_giga.th -O {encodec_path}")

# Initialize audio tokenizer (loads EnCodec model)
audio_tokenizer = AudioTokenizer(signature=encodec_path, device=device)
print(f"Audio sample rate: {audio_tokenizer.sample_rate}")
print(f"Audio channels: {audio_tokenizer.channels}")

# Initialize text tokenizer with espeak backend
text_tokenizer = TextTokenizer(backend="espeak")

# Test phonemization
sample_text = "Hello, this is a test of the voice synthesis system."
phonemes = text_tokenizer([sample_text])
print(f"Phonemes: {phonemes[0][:10]}...")  # First 10 phonemes
```

--------------------------------

### Initialize and Run VoiceCraft Model

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Configure the model architecture using Namespace and perform a forward pass with a prepared batch.

```python
from models.voicecraft import VoiceCraft
from argparse import Namespace
import torch

# Model configuration
args = Namespace(
    n_codebooks=4,
    d_model=2048,
    audio_embedding_dim=2048,
    nhead=16,
    num_decoder_layers=16,
    text_vocab_size=100,
    text_pad_token=100,
    audio_vocab_size=2048,
    empty_token=2048,
    eog=2049,
    audio_pad_token=2050,
    max_n_spans=3,
    text_embedding_dropout=0.1,
    audio_embedding_dropout=0.0,
    text_positional_embedding_dropout=0.1,
    audio_positional_embedding_dropout=0.1,
    trm_dropout=0.1,
    special_first=0,
    n_special=3,
    eos=-1,
    encodec_sr=50,
    phn2num={'a': 0, 'b': 1}  # Phoneme to index mapping
)

# Initialize model
model = VoiceCraft(args=args)
model.to("cuda")

# Forward pass for training (returns loss and metrics)
batch = {
    'x': torch.randint(0, 100, (2, 50)),      # Text tokens [B, T_text]
    'x_lens': torch.tensor([45, 50]),          # Text lengths
    'y': torch.randint(0, 2048, (2, 4, 200)), # Audio codes [B, K, T_audio]
    'y_lens': torch.tensor([180, 200])         # Audio lengths
}
batch = {k: v.to("cuda") for k, v in batch.items()}

output = model(batch)
print(f"Loss: {output['loss']:.4f}")
print(f"Top-10 accuracy: {output['top10acc']:.4f}")
```

--------------------------------

### Import Required Libraries

Source: https://github.com/jasonppy/voicecraft/blob/master/inference_speech_editing.ipynb

Load necessary dependencies for audio processing, tokenization, and model architecture.

```python
# import libs
import torch
import torchaudio
import numpy as np
import random
from argparse import Namespace

from data.tokenizer import (
    AudioTokenizer,
    TextTokenizer,
)

from models import voicecraft
```

--------------------------------

### Initialize Inference and Load Model

Source: https://github.com/jasonppy/voicecraft/blob/master/inference_speech_editing.ipynb

Set hyperparameters, seed the environment, and load the VoiceCraft model from HuggingFace.

```python
# hyperparameters for inference
left_margin = 0.08
right_margin = 0.08
codec_audio_sr = 16000
codec_sr = 50
top_k = 40
top_p = 1
temperature = 1
kvcache = 0
# adjust the below three arguments if the generation is not as good
seed = 1 # random seed magic
silence_tokens = [1388,1898,131] # if there are long silence in the generated audio, reduce the stop_repetition to 3, 2 or even 1
stop_repetition = -1 # -1 means do not adjust prob of silence tokens. if there are long silence or unnaturally strecthed words, increase sample_batch_size to 2, 3 or even 4
# what this will do to the model is that the model will run sample_batch_size examples of the same audio, and pick the one that's the shortest
def seed_everything(seed):
    os.environ['PYTHONHASHSEED'] = str(seed)
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True
seed_everything(seed)
device = "cuda" if torch.cuda.is_available() else "cpu"
# load model, tokenizer, and other necessary files
voicecraft_name="giga330M.pth" # or gigaHalfLibri330M_TTSEnhanced_max16s.pth, giga830M.pth

# the new way of loading the model, with huggingface, recommended
from models import voicecraft
model = voicecraft.VoiceCraft.from_pretrained(f"pyp1/VoiceCraft_{voicecraft_name.replace('.pth', '')}")
phn2num = model.args.phn2num
config = vars(model.args)
model.to(device)

# # the old way of loading the model
# from models import voicecraft
# filepath = hf_hub_download(repo_id="pyp1/VoiceCraft", filename=voicecraft_name, repo_type="model")
# ckpt = torch.load(filepath, map_location="cpu")
# model = voicecraft.VoiceCraft(ckpt["config"])
# model.load_state_dict(ckpt["model"])
# config = vars(model.args)
# phn2num = ckpt["phn2num"]
# model.to(device)
# model.eval()

encodec_fn = "./pretrained_models/encodec_4cb2048_giga.th"
if not os.path.exists(encodec_fn):
    os.system(f"wget https://huggingface.co/pyp1/VoiceCraft/resolve/main/encodec_4cb2048_giga.th")
    os.system(f"mv encodec_4cb2048_giga.th ./pretrained_models/encodec_4cb2048_giga.th")
audio_tokenizer = AudioTokenizer(signature=encodec_fn) # will also put the neural codec model on gpu

text_tokenizer = TextTokenizer(backend="espeak")

# point to the original file or record the file
# write down the transcript for the file, or run whisper to get the transcript (and you can modify it if it's not accurate), save it as a .txt file
orig_audio = "./demo/84_121550_000074_000000.wav"
orig_transcript = "But when I had approached so near to them The common object, which the sense deceives, Lost not by distance any of its marks,"
# move the audio and transcript to temp folder
temp_folder = "./demo/temp"
os.makedirs(temp_folder, exist_ok=True)
os.system(f"cp {orig_audio} {temp_folder}")
filename = os.path.splitext(orig_audio.split("/")[-1])[0]
with open(f"{temp_folder}/{filename}.txt", "w") as f:
    f.write(orig_transcript)
# run MFA to get the alignment
align_temp = f"{temp_folder}/mfa_alignments"
os.makedirs(align_temp, exist_ok=True)
os.system(f"mfa align -j 1 --output_format csv {temp_folder} english_us_arpa english_us_arpa {align_temp}")
# if it fail, it could be because the audio is too hard for the alignment model, increasing the beam size usually solves the issue
# os.system(f"mfa align -j 1 --output_format csv {temp_folder} english_us_arpa english_us_arpa {align_temp} --beam 1000 --retry_beam 2000")
audio_fn = f"{temp_folder}/{filename}.wav"
transcript_fn = f"{temp_folder}/{filename}.txt"
align_fn = f"{align_temp}/{filename}.csv"
```

--------------------------------

### Basic Command-Line TTS Demo

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Run a basic Text-to-Speech demo from the command line using default parameters.

```bash
python tts_demo.py
```

--------------------------------

### Configure Custom Training

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Use this command to initiate training on a custom dataset with specific model architecture and optimization parameters.

```bash
export CUDA_VISIBLE_DEVICES=0,1,2,3
export WORLD_SIZE=4

torchrun --nnodes=1 --rdzv-backend=c10d --rdzv-endpoint=localhost:41977 --nproc_per_node=${WORLD_SIZE} \
main.py \
--reduced_eog 1 \
--drop_long 1 \
--eos 2051 \
--n_special 4 \
--codebook_weight "[5,1,0.5,0.1]" \
--encodec_sr 50 \
--num_steps 50000 \
--lr 0.05 \
--warmup_fraction 0.01 \
--optimizer_name "ScaledAdam" \
--d_model 2048 \
--audio_embedding_dim 2048 \
--nhead 16 \
--num_decoder_layers 16 \
--max_num_tokens 100000 \
--gradient_accumulation_steps 26 \
--n_codebooks 4 \
--max_n_spans 3 \
--dataset gigaspeech \
--exp_dir /path/to/experiments/gigaspeech/e830M \
--dataset_dir /path/to/processed_data/xl
```

--------------------------------

### Run VoiceCraft TTS Demo via Command Line

Source: https://github.com/jasonppy/voicecraft/blob/master/README.md

Execute the TTS demo script using command-line arguments to specify inputs and hyperparameters. Use the help command for detailed options.

```bash
python3 tts_demo.py -h
```

--------------------------------

### Run Gradio Server

Source: https://github.com/jasonppy/voicecraft/blob/master/README.md

Command to launch the Gradio application server.

```bash
python gradio_app.py
```

--------------------------------

### Configure Inference Parameters

Source: https://github.com/jasonppy/voicecraft/blob/master/inference_tts.ipynb

Sets up inference parameters and validates audio duration for the synthesis process.

```python
# take a look at demo/temp/mfa_alignment, decide which part of the audio to use as prompt
cut_off_sec = 3.6 # NOTE: according to forced-alignment file demo/temp/mfa_alignments/5895_34622_000026_000002.wav, the word "strength" stop as 3.561 sec, so we use first 3.6 sec as the prompt. this should be different for different audio
target_transcript = "Gwynplaine had, besides, for his work and for his feats of strength, I cannot believe that the same model can also do text to speech synthesis too!"
# NOTE: 3 sec of reference is generally enough for high quality voice cloning, but longer is generally better, try e.g. 3~6 sec.
audio_fn = f"{temp_folder}/{filename}.wav"
info = torchaudio.info(audio_fn)
audio_dur = info.num_frames / info.sample_rate

assert cut_off_sec < audio_dur, f"cut_off_sec {cut_off_sec} is larger than the audio duration {audio_dur}"
prompt_end_frame = int(cut_off_sec * info.sample_rate)

# run the model to get the output
# hyperparameters for inference
codec_audio_sr = 16000
codec_sr = 50
top_k = 40 # can also try 20, 30, 50
top_p = 1 # 1 means do not do top-p sampling
temperature = 1
silence_tokens=[1388,1898,131]
kvcache = 1 # NOTE if OOM, change this to 0, or try the 330M model
```

--------------------------------

### Prepare Audio and Run MFA Alignment

Source: https://github.com/jasonppy/voicecraft/blob/master/inference_tts.ipynb

Prepares audio files and transcripts for alignment using MFA. Includes a retry mechanism for difficult audio.

```python
# Prepare your audio
# point to the original audio whose speech you want to clone
# write down the transcript for the file, or run whisper to get the transcript (and you can modify it if it's not accurate), save it as a .txt file
orig_audio = "./demo/5895_34622_000026_000002.wav"
orig_transcript = "Gwynplaine had, besides, for his work and for his feats of strength, round his neck and over his shoulders, an esclavine of leather."

# move the audio and transcript to temp folder
temp_folder = "./demo/temp"
os.makedirs(temp_folder, exist_ok=True)
os.system(f"cp {orig_audio} {temp_folder}")
filename = os.path.splitext(orig_audio.split("/")[-1])[0]
with open(f"{temp_folder}/{filename}.txt", "w") as f:
    f.write(orig_transcript)
# run MFA to get the alignment
align_temp = f"{temp_folder}/mfa_alignments"
!source ~/.bashrc && \
    conda activate voicecraft && \
    mfa align -v --clean -j 1 --output_format csv {temp_folder} \
        english_us_arpa english_us_arpa {align_temp}

# # if the above fails, it could be because the audio is too hard for the alignment model, increasing the beam size usually solves the issue
# !source ~/.bashrc && \
#     conda activate voicecraft && \
#     mfa align -v --clean -j 1 --output_format csv {temp_folder} \
#         english_us_arpa english_us_arpa {align_temp} --beam 1000 --retry_beam 2000
```

--------------------------------

### Initialize Tokenizers

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Initialize TextTokenizer and AudioTokenizer for processing text and audio data. Specify the backend for text tokenization and the path to the EnCodec model.

```python
from data.tokenizer import TextTokenizer, AudioTokenizer, tokenize_text, tokenize_audio

text_tokenizer = TextTokenizer(backend="espeak")
audio_tokenizer = AudioTokenizer(signature="./pretrained_models/encodec_4cb2048_giga.th")

text = "Hello world, this is a test."
phonemes = tokenize_text(text_tokenizer, text)
print(f"Phonemes: {' '.join(phonemes)}")

encoded_frames = tokenize_audio(audio_tokenizer, "./demo/sample.wav")
codes = encoded_frames[0][0]  # Shape: [1, n_codebooks, T]
print(f"Audio codes shape: {codes.shape}")
```

--------------------------------

### Load VoiceCraft Model and Tokenizers

Source: https://github.com/jasonppy/voicecraft/blob/master/inference_tts.ipynb

Initializes the VoiceCraft model, audio tokenizer, and text tokenizer. The recommended approach uses HuggingFace Hub.

```python
# load model, encodec, and phn2num
# # load model, tokenizer, and other necessary files
device = "cuda" if torch.cuda.is_available() else "cpu"
voicecraft_name="830M_TTSEnhanced.pth" # or giga330M.pth, 330M_TTSEnhanced.pth, giga830M.pth

# the new way of loading the model, with huggingface, recommended
from models import voicecraft
model = voicecraft.VoiceCraft.from_pretrained(f"pyp1/VoiceCraft_{voicecraft_name.replace('.pth', '')}")
phn2num = model.args.phn2num
config = vars(model.args)
model.to(device)


# # the old way of loading the model
# from models import voicecraft
# filepath = hf_hub_download(repo_id="pyp1/VoiceCraft", filename=voicecraft_name, repo_type="model")
# ckpt = torch.load(filepath, map_location="cpu")
# model = voicecraft.VoiceCraft(ckpt["config"])
# model.load_state_dict(ckpt["model"])
# config = vars(model.args)
# phn2num = ckpt["phn2num"]
# model.to(device)
# model.eval()


encodec_fn = "./pretrained_models/encodec_4cb2048_giga.th"
if not os.path.exists(encodec_fn):
    os.system(f"wget https://huggingface.co/pyp1/VoiceCraft/resolve/main/encodec_4cb2048_giga.th")
    os.system(f"mv encodec_4cb2048_giga.th ./pretrained_models/encodec_4cb2048_giga.th")
audio_tokenizer = AudioTokenizer(signature=encodec_fn, device=device) # will also put the neural codec model on gpu

text_tokenizer = TextTokenizer(backend="espeak")
```

--------------------------------

### Execute Fine-tuning Script

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Run the provided fine-tuning script located in the z_scripts directory.

```bash
# Fine-tuning script for TTS
cd z_scripts
bash e830M_ft.sh
```

--------------------------------

### Launch Gradio Server

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Launch the Gradio interactive web interface for voice cloning and speech editing. Specify port and server name, or use --share for a public link.

```bash
python gradio_app.py --port 7860 --server_name 127.0.0.1
```

```bash
python gradio_app.py --share
```

```bash
python gradio_app.py \
    --demo-path ./demo \
    --tmp-path ./demo/temp \
    --models-path ./pretrained_models \
    --port 7860
```

--------------------------------

### Custom Command-Line TTS Demo

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Perform TTS inference with custom parameters, including model name, audio files, transcripts, and generation settings like temperature and top_k.

```bash
python tts_demo.py \
    --model_name giga830M_TTSEnhanced \
    --original_audio ./demo/5895_34622_000026_000002.wav \
    --original_transcript "Gwynplaine had, besides, for his work and for his feats of strength, round his neck and over his shoulders, an esclavine of leather."
    --target_transcript "I cannot believe that the same model can also do text to speech synthesis too!"
    --cut_off_sec 3.6 \
    --output_dir ./generated_tts \
    --seed 1 \
    --top_k 40 \
    --top_p 0.8 \
    --temperature 1.0 \
    --sample_batch_size 3 \
    --stop_repetition 3
```

```bash
python tts_demo.py -h
```

--------------------------------

### Manage Docker Container

Source: https://github.com/jasonppy/voicecraft/blob/master/README.md

Commands for checking logs, accessing the container shell, and verifying GPU visibility.

```bash
docker logs jupyter
```

```bash
docker exec -it jupyter /bin/bash
export USER=(your_linux_username_used_above)
export HOME=/home/$USER
sudo apt-get update
```

```bash
nvidia-smi
```

--------------------------------

### Prepare Data for VoiceCraft Training

Source: https://github.com/jasonppy/voicecraft/blob/master/README.md

Use this script to download Gigaspeech, extract phoneme sequences and encodec codes, and generate a vocabulary file. Adjust batch sizes and max length to prevent Out-of-Memory errors.

```bash
conda activate voicecraft
export CUDA_VISIBLE_DEVICES=0
cd ./data
python phonemize_encodec_encode_hf.py \
--dataset_size xs \
--download_to path/to/store_huggingface_downloads \
--save_dir path/to/store_extracted_codes_and_phonemes \
--encodec_model_path path/to/encodec_model \
--mega_batch_size 120 \
--batch_size 32 \
--max_len 30000
```

--------------------------------

### Configure Custom Fine-tuning

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Fine-tune a pretrained model using the AdamW optimizer for improved stability.

```bash
# Custom fine-tuning configuration
# Use AdamW optimizer for better stability during fine-tuning
torchrun --nnodes=1 --rdzv-backend=c10d --rdzv-endpoint=localhost:41977 --nproc_per_node=4 \
main.py \
--load_model_from /path/to/pretrained/best_bundle.pth \
--reduced_eog 1 \
--eos 2051 \
--n_special 4 \
--optimizer_name "AdamW" \
--lr 0.0001 \
--num_steps 10000 \
--warmup_fraction 0.05 \
--d_model 2048 \
--num_decoder_layers 16 \
--max_num_tokens 50000 \
--gradient_accumulation_steps 4 \
--audio_max_length 16 \
--text_vocab_size 100 \
--text_pad_token 100 \
--dataset custom_dataset \
--exp_dir /path/to/experiments/finetuned \
--dataset_dir /path/to/custom_data
```

--------------------------------

### Download and Process GigaSpeech Dataset

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Download and process the GigaSpeech dataset for VoiceCraft training. This script requires HuggingFace authentication and specifies paths for dataset download, processed data, and EnCodec models.

```bash
cd data

python phonemize_encodec_encode_hf.py \
    --dataset_size xs \
    --download_to /path/to/huggingface_cache \
    --save_dir /path/to/processed_data \
    --encodec_model_path /path/to/encodec_4cb2048_giga.th \
    --mega_batch_size 120 \
    --batch_size 32 \
    --max_len 30000
```

--------------------------------

### Configure Environment Variables

Source: https://github.com/jasonppy/voicecraft/blob/master/inference_speech_editing.ipynb

Set CUDA device visibility and user-specific environment variables before initializing deep learning components.

```python
import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"   
os.environ["CUDA_VISIBLE_DEVICES"]="7"
os.environ["USER"] = "YOUR_USERNAME" # TODO change this to your username
```

--------------------------------

### Configure and Run TTS Inference

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Sets up decoding parameters and executes the inference pipeline for text-to-speech generation.

```python
decode_config = {
    'top_k': 40,              # Top-k sampling (recommended: 40)
    'top_p': 1.0,             # Nucleus sampling (1.0 = disabled)
    'temperature': 1.0,       # Sampling temperature
    'stop_repetition': 3,     # Stop after N consecutive repetitions (-1 to disable)
    'kvcache': 1,             # Enable KV cache for 4-8x speedup
    'codec_audio_sr': 16000,  # Audio sample rate
    'codec_sr': 50,           # Codec frame rate
    'silence_tokens': [1388, 1898, 131],  # Silence token IDs
    'sample_batch_size': 3    # Generate N samples, pick shortest
}

# Run inference
concat_audio, gen_audio = inference_one_sample(
    model, config, phn2num, text_tokenizer, audio_tokenizer,
    audio_path, target_transcript, device, decode_config, prompt_end_frame
)

# Save generated audio
gen_audio = gen_audio[0].cpu()
torchaudio.save("output_tts.wav", gen_audio, 16000)
print(f"Generated audio saved: {gen_audio.shape[1]/16000:.2f} seconds")
```

--------------------------------

### Initialize Random Seeds

Source: https://github.com/jasonppy/voicecraft/blob/master/inference_tts.ipynb

Ensures reproducibility across various libraries including NumPy, PyTorch, and CUDA.

```python
def seed_everything(seed):
    os.environ['PYTHONHASHSEED'] = str(seed)
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True
seed_everything(seed)
```

--------------------------------

### Finetune VoiceCraft Model

Source: https://github.com/jasonppy/voicecraft/blob/master/README.md

Use this script for finetuning a pre-trained VoiceCraft model. It's recommended to use AdamW for optimization. Ensure vocabulary consistency and adjust text embedding parameters if new phonemes are introduced.

```bash
bash ./z_scripts/e830M_ft.sh
```

--------------------------------

### Configure Generation Parameters

Source: https://github.com/jasonppy/voicecraft/blob/master/inference_tts.ipynb

Adjust these variables to control repetition, batch sampling, and randomness in the generated audio.

```python
stop_repetition = 3 # NOTE if the model generate long silence, reduce the stop_repetition to 3, 2 or even 1
sample_batch_size = 3 # NOTE: if the if there are long silence or unnaturally strecthed words, increase sample_batch_size to 4 or higher. What this will do to the model is that the model will run sample_batch_size examples of the same audio, and pick the one that's the shortest. So if the speech rate of the generated is too fast change it to a smaller number.
seed = 1 # change seed if you are still unhappy with the result
```

--------------------------------

### Programmatic Gradio App Usage

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Programmatically initialize and launch the Gradio application. Load models and configure the app for multi-request handling using queues.

```python
from gradio_app import get_app, load_models

app = get_app()

load_models(
    whisper_backend_name="whisperX",
    whisper_model_name="base.en",
    alignment_model_name="whisperX",
    voicecraft_model_name="830M_TTSEnhanced"
)

app.queue().launch(share=False, server_name="127.0.0.1", server_port=7860)
```

--------------------------------

### Custom Training Configuration

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Configuration for custom training of VoiceCraft models.

```APIDOC
## Custom Training Configuration

### Description
Configuration for custom training of VoiceCraft models with specified distributed training settings and hyperparameters.

### Method
Not Applicable (Shell Script)

### Endpoint
Not Applicable

### Parameters
#### Environment Variables
- **CUDA_VISIBLE_DEVICES** (string) - Specifies the GPU devices to use.
- **WORLD_SIZE** (string) - The total number of processes for distributed training.

#### Command Line Arguments
- **--reduced_eog** (integer) - Whether to reduce end-of-generation tokens.
- **--drop_long** (integer) - Whether to drop long sequences.
- **--eos** (integer) - End-of-sentence token ID.
- **--n_special** (integer) - Number of special tokens.
- **--codebook_weight** (string) - Weights for different codebooks.
- **--encodec_sr** (integer) - Sample rate for Encodec.
- **--num_steps** (integer) - Total number of training steps.
- **--lr** (float) - Learning rate.
- **--warmup_fraction** (float) - Fraction of training steps for learning rate warmup.
- **--optimizer_name** (string) - Name of the optimizer (e.g., "ScaledAdam").
- **--d_model** (integer) - Dimension of the model.
- **--audio_embedding_dim** (integer) - Dimension of audio embeddings.
- **--nhead** (integer) - Number of attention heads.
- **--num_decoder_layers** (integer) - Number of decoder layers.
- **--max_num_tokens** (integer) - Maximum number of tokens in a sequence.
- **--gradient_accumulation_steps** (integer) - Number of steps for gradient accumulation.
- **--n_codebooks** (integer) - Number of codebooks.
- **--max_n_spans** (integer) - Maximum number of spans.
- **--dataset** (string) - Name of the dataset.
- **--exp_dir** (string) - Directory to save experiments.
- **--dataset_dir** (string) - Directory containing the dataset.

### Request Example
```bash
export CUDA_VISIBLE_DEVICES=0,1,2,3
export WORLD_SIZE=4
torchrun --nnodes=1 --rdzv-backend=c10d --rdzv-endpoint=localhost:41977 --nproc_per_node=${WORLD_SIZE} \
main.py \
--reduced_eog 1 \
--drop_long 1 \
--eos 2051 \
--n_special 4 \
--codebook_weight "[5,1,0.5,0.1]" \
--encodec_sr 50 \
--num_steps 50000 \
--lr 0.05 \
--warmup_fraction 0.01 \
--optimizer_name "ScaledAdam" \
--d_model 2048 \
--audio_embedding_dim 2048 \
--nhead 16 \
--num_decoder_layers 16 \
--max_num_tokens 100000 \
--gradient_accumulation_steps 26 \
--n_codebooks 4 \
--max_n_spans 3 \
--dataset gigaspeech \
--exp_dir /path/to/experiments/gigaspeech/e830M \
--dataset_dir /path/to/processed_data/xl
```

### Response
Not Applicable (Shell Script Output)
```

--------------------------------

### Prepare Audio Spans and Mask Intervals

Source: https://github.com/jasonppy/voicecraft/blob/master/inference_speech_editing.ipynb

Calculates the original and new spans for editing and converts them into codec frame intervals for the model.

```python
from edit_utils import get_span
orig_span, new_span = get_span(orig_transcript, target_transcript, edit_type)
if orig_span[0] > orig_span[1]:
    RuntimeError(f"example {audio_fn} failed")
if orig_span[0] == orig_span[1]:
    orig_span_save = [orig_span[0]]
else:
    orig_span_save = orig_span
if new_span[0] == new_span[1]:
    new_span_save = [new_span[0]]
else:
    new_span_save = new_span

orig_span_save = ",".join([str(item) for item in orig_span_save])
new_span_save = ",".join([str(item) for item in new_span_save])
from inference_speech_editing_scale import get_mask_interval

start, end = get_mask_interval(align_fn, orig_span_save, edit_type)
info = torchaudio.info(audio_fn)
audio_dur = info.num_frames / info.sample_rate
morphed_span = (max(start - left_margin, 1/codec_sr), min(end + right_margin, audio_dur)) # in seconds

# span in codec frames
mask_interval = [[round(morphed_span[0]*codec_sr), round(morphed_span[1]*codec_sr)]]
mask_interval = torch.LongTensor(mask_interval) # [M,2], M==1 for now
```

--------------------------------

### Clone VoiceCraft Repository

Source: https://github.com/jasonppy/voicecraft/blob/master/voicecraft-gradio-colab.ipynb

Downloads the VoiceCraft source code from the official repository.

```bash
!git clone https://github.com/jasonppy/VoiceCraft
```

--------------------------------

### Run Inference and Display Audio

Source: https://github.com/jasonppy/voicecraft/blob/master/inference_speech_editing.ipynb

Executes the speech editing model and displays the original and edited audio segments in a notebook environment.

```python
from inference_speech_editing_scale import inference_one_sample

decode_config = {'top_k': top_k, 'top_p': top_p, 'temperature': temperature, 'stop_repetition': stop_repetition, 'kvcache': kvcache, "codec_audio_sr": codec_audio_sr, "codec_sr": codec_sr, "silence_tokens": silence_tokens}
orig_audio, new_audio = inference_one_sample(model, Namespace(**config), phn2num, text_tokenizer, audio_tokenizer, audio_fn, target_transcript, mask_interval, device, decode_config)
        
# save segments for comparison
orig_audio, new_audio = orig_audio[0].cpu(), new_audio[0].cpu()
# logging.info(f"length of the resynthesize orig audio: {orig_audio.shape}")

# display the audio
from IPython.display import Audio
print("original:")
display(Audio(orig_audio, rate=codec_audio_sr))

print("edited:")
display(Audio(new_audio, rate=codec_audio_sr))
```

--------------------------------

### Train 830M Model

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Execute the training script for the 830M VoiceCraft model. This command assumes you are in the 'z_scripts' directory.

```bash
cd z_scripts
bash e830M.sh
```

--------------------------------

### Display Generated Audio

Source: https://github.com/jasonppy/voicecraft/blob/master/inference_tts.ipynb

Uses IPython display tools to play the concatenated and generated audio segments in a notebook environment.

```python
concated_audio, gen_audio = concated_audio[0].cpu(), gen_audio[0].cpu()

from IPython.display import Audio
print("concatenate prompt and generated:")
display(Audio(concated_audio, rate=codec_audio_sr))

print("generated:")
display(Audio(gen_audio, rate=codec_audio_sr))
```

--------------------------------

### Use Core Inference Functions Directly

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Provides low-level control for custom TTS and speech editing workflows by manually preparing tokens and prompts.

```python
import torch
from data.tokenizer import tokenize_audio, tokenize_text
from models import voicecraft

device = "cuda" if torch.cuda.is_available() else "cpu"

# Load model
model = voicecraft.VoiceCraft.from_pretrained("pyp1/VoiceCraft_giga830M_TTSEnhanced")
model.to(device)
model.eval()
phn2num = model.args.phn2num

# Prepare text tokens
text = "The quick brown fox jumps over the lazy dog."
text_tokens = [phn2num[phn] for phn in tokenize_text(text_tokenizer, text) if phn in phn2num]
text_tokens = torch.LongTensor(text_tokens).unsqueeze(0).to(device)
text_tokens_lens = torch.LongTensor([text_tokens.shape[-1]]).to(device)

# Prepare audio prompt (first few seconds of reference)
prompt_audio = tokenize_audio(audio_tokenizer, audio_path, offset=0, num_frames=prompt_end_frame)
original_audio = prompt_audio[0][0].transpose(2, 1).to(device)  # [1, T, K]
```

--------------------------------

### Fine-tuning VoiceCraft Models

Source: https://context7.com/jasonppy/voicecraft/llms.txt

Fine-tune pretrained VoiceCraft models on custom datasets.

```APIDOC
## Fine-tuning VoiceCraft Models

### Description
Fine-tune pretrained VoiceCraft models on custom datasets using the TTS-focused training objective. This script utilizes the AdamW optimizer for improved stability during fine-tuning.

### Method
Not Applicable (Shell Script)

### Endpoint
Not Applicable

### Parameters
#### Command Line Arguments
- **--load_model_from** (string) - Path to the pretrained model to load.
- **--reduced_eog** (integer) - Whether to reduce end-of-generation tokens.
- **--eos** (integer) - End-of-sentence token ID.
- **--n_special** (integer) - Number of special tokens.
- **--optimizer_name** (string) - Name of the optimizer (e.g., "AdamW").
- **--lr** (float) - Learning rate.
- **--num_steps** (integer) - Total number of training steps.
- **--warmup_fraction** (float) - Fraction of training steps for learning rate warmup.
- **--d_model** (integer) - Dimension of the model.
- **--num_decoder_layers** (integer) - Number of decoder layers.
- **--max_num_tokens** (integer) - Maximum number of tokens in a sequence.
- **--gradient_accumulation_steps** (integer) - Number of steps for gradient accumulation.
- **--audio_max_length** (integer) - Maximum audio length.
- **--text_vocab_size** (integer) - Vocabulary size for text.
- **--text_pad_token** (integer) - Padding token ID for text.
- **--dataset** (string) - Name of the custom dataset.
- **--exp_dir** (string) - Directory to save fine-tuned experiments.
- **--dataset_dir** (string) - Directory containing the custom dataset.

### Request Example
```bash
# Fine-tuning script for TTS
cd z_scripts
bash e830M_ft.sh
```

```bash
# Custom fine-tuning configuration
# Use AdamW optimizer for better stability during fine-tuning
torchrun --nnodes=1 --rdzv-backend=c10d --rdzv-endpoint=localhost:41977 --nproc_per_node=4 \
main.py \
--load_model_from /path/to/pretrained/best_bundle.pth \
--reduced_eog 1 \
--eos 2051 \
--n_special 4 \
--optimizer_name "AdamW" \
--lr 0.0001 \
--num_steps 10000 \
--warmup_fraction 0.05 \
--d_model 2048 \
--num_decoder_layers 16 \
--max_num_tokens 50000 \
--gradient_accumulation_steps 4 \
--audio_max_length 16 \
--text_vocab_size 100 \
--text_pad_token 100 \
--dataset custom_dataset \
--exp_dir /path/to/experiments/finetuned \
--dataset_dir /path/to/custom_data
```

### Response
Not Applicable (Shell Script Output)
```

--------------------------------

### Define Edit Parameters

Source: https://github.com/jasonppy/voicecraft/blob/master/inference_speech_editing.ipynb

Specify the target transcript and edit type for the audio modification process.

```python
editTypes_set = set(['substitution', 'insertion', 'deletion'])
# propose what do you want the target modified transcript to be
target_transcript = "But when I saw the mirage of the lake in the distance, which the sense deceives, Lost not by distance any of its marks,"
edit_type = "substitution"
assert edit_type in editTypes_set, f"Invalid edit type {edit_type}. Must be one of {editTypes_set}."
```