NeMo Speech

https://github.com/nvidia-nemo/speech
Admin
NeMo Speech is a toolkit for training and deploying speech models, including Automatic Speech...
Tokens:1,129,354
Snippets:7,204
Trust Score:8
License:Apache-2.0
Update:3 weeks ago
Show doc for...
Tokens:
Raw
### Output of setup cell

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/asr/ASR_with_NeMo.ipynb

Example output from running the setup cell. Shows that wget and text-unidecode are already installed, and that the apt-get command fails with a permission error because the runtime is not running as root.

```text
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: wget in /home/nkoluguri/miniconda3/envs/nemo/lib/python3.10/site-packages (3.2)
E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: text-unidecode in /home/nkoluguri/miniconda3/envs/nemo/lib/python3.10/site-packages (1.3)
```

--------------------------------

### Setup Colab environment and install NeMo

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/asr/asr_adapters/ASR_with_Adapters.ipynb

Sets up the environment for the tutorial, installing dependencies and NeMo from the main branch. Includes commented-out commands to download a configuration file for the example.

```python
"""
You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.

Instructions for setting up Colab are as follows:
1. Open a new Python 3 notebook.
2. Import this notebook from GitHub (File -> Upload Notebook -> "GitHub" tab -> copy/paste GitHub URL)
3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select "GPU" for hardware accelerator)
4. Run this cell to set up dependencies.
5. Restart the runtime (Runtime -> Restart Runtime) for any upgraded packages to take effect


NOTE: User is responsible for checking the content of datasets and the applicable licenses and determining if suitable for the intended use.
"""
# If you're using Google Colab and not running locally, run this cell.
import os

# Install dependencies
!pip install wget
!apt-get install sox libsndfile1 ffmpeg
!pip install text-unidecode
!pip install matplotlib>=3.3.2

## Install NeMo
BRANCH = 'main'
!python -m pip install "nemo_toolkit[all] @ git+https://github.com/NVIDIA-NeMo/Speech.git@$BRANCH"

## Grab the config we'll use in this example
# !mkdir configs
# !wget -P configs/ https://raw.githubusercontent.com/NVIDIA/NeMo/$BRANCH/examples/asr/conf/contextnet_rnnt/contextnet_rnnt.yaml
```

--------------------------------

### Setup NeMo and install nemo_toolkit

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/tools/Multispeaker_Simulator.ipynb

Setup cell for running the notebook locally or on Google Colab. Clones the NeMo repository, installs the nemo_toolkit[all] package from the specified branch, and sets the NEMO_DIR_PATH and BRANCH variables used by later cells.

```python
"""
You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.

Instructions for setting up Colab are as follows:
1. Open a new Python 3 notebook.
2. Import this notebook from GitHub (File -> Upload Notebook -> "GITHUB" tab -> copy/paste GitHub URL)
3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select "GPU" for hardware accelerator)
4. Run this cell to set up dependencies.
5. Restart the runtime (Runtime -> Restart Runtime) for any upgraded packages to take effect
"""

NEMO_DIR_PATH = "NeMo"
BRANCH = 'main'

! git clone https://github.com/NVIDIA-NeMo/Speech
%cd NeMo
! python -m pip install "nemo_toolkit[all] @ git+https://github.com/NVIDIA-NeMo/Speech.git@$BRANCH"
%cd ..
```

--------------------------------

### Setup NeMo and dependencies for CTC-WS tutorial

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/asr/ASR_Context_Biasing.ipynb

Setup cell for the CTC-based Word Spotter tutorial. Installs system dependencies (sox, libsndfile1, ffmpeg), clones the NeMo repository from the specified BRANCH, installs nemo_toolkit[all], and adds the NeMo directory to sys.path. Works both locally and on Google Colab.

```python
BRANCH = 'main'

"""
You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.

Instructions for setting up Colab are as follows:
1. Open a new Python 3 notebook.
2. Import this notebook from GitHub (File -> Upload Notebook -> "GITHUB" tab -> copy/paste GitHub URL)
3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select "GPU" for hardware accelerator)
4. Run this cell to set up dependencies.
"""

import os
# either provide a path to local NeMo repository with NeMo already installed or git clone

# option #1: local path to NeMo repo with NeMo already installed
NEMO_DIR_PATH = os.path.dirname(os.path.dirname(os.path.abspath('')))

# check if Google Colab is being used
try:
    import google.colab
    IN_COLAB = True
except (ImportError, ModuleNotFoundError):
    IN_COLAB = False

# option #2: download NeMo repo
if IN_COLAB or not os.path.exists(os.path.join(NEMO_DIR_PATH, "nemo")):
    ## Install dependencies
    !apt-get install sox libsndfile1 ffmpeg

    !git clone -b $BRANCH https://github.com/NVIDIA-NeMo/Speech
    %cd NeMo
    !python -m pip install "nemo_toolkit[all] @ git+https://github.com/NVIDIA-NeMo/Speech.git@$BRANCH"
    NEMO_DIR_PATH = os.path.abspath('')

import sys
sys.path.insert(0, NEMO_DIR_PATH)
```

--------------------------------

### Launch the client

Source: https://github.com/nvidia-nemo/speech/blob/main/examples/voice_agent/README.md

Installs client dependencies and starts the development server. Run in a separate terminal on the server machine.

```bash
cd client
npm install
npm run dev
```

--------------------------------

### Setup NeMo and install nemo_toolkit[asr]

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/speaker_tasks/End_to_End_Diarization_Training.ipynb

Setup cell for running the notebook on Google Colab or locally. Clones the NVIDIA-NeMo/Speech repository, installs the nemo_toolkit[asr] package from the main branch, and returns to the original directory. Requires a GPU and a runtime restart after execution.

```python
"""
You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.

Instructions for setting up Colab are as follows:
1. Open a new Python 3 notebook.
2. Import this notebook from GitHub (File -> Upload Notebook -> "GITHUB" tab -> copy/paste GitHub URL)
3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select "GPU" for hardware accelerator)
4. Run this cell to set up dependencies.
5. Restart the runtime (Runtime -> Restart Runtime) for any upgraded packages to take effect
"""
# If you're using Google Colab and not running locally, run this cell.

NEMO_DIR_PATH = "NeMo"
BRANCH = 'main'

! git clone https://github.com/NVIDIA-NeMo/Speech
%cd NeMo
! python -m pip install "nemo_toolkit[asr] @ git+https://github.com/NVIDIA-NeMo/Speech.git@{BRANCH}"
%cd ..
```

--------------------------------

### Install dependencies and NeMo toolkit

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/asr/ASR_with_Subword_Tokenization.ipynb

Setup commands for running the notebook locally or on Google Colab. Installs dependencies (wget, sox, libsndfile1, ffmpeg, text-unidecode, matplotlib), installs the NeMo toolkit from the Speech repository, and downloads the FastConformer CTC BPE configuration file. After running, restart the runtime for upgraded packages to take effect.

```bash
!pip install wget
!apt-get install sox libsndfile1 ffmpeg
!pip install text-unidecode
!pip install matplotlib>=3.3.2
```

```python
BRANCH = 'main'
!python -m pip install "nemo_toolkit[all] @ git+https://github.com/NVIDIA-NeMo/Speech.git@$BRANCH"
```

```bash
!mkdir configs
!wget -P configs/ https://raw.githubusercontent.com/NVIDIA/NeMo/$BRANCH/examples/asr/conf/fastconformer/fast-conformer_ctc_bpe.yaml
```

--------------------------------

### Quick start: download ARPA and run WFST decoding

Source: https://github.com/nvidia-nemo/speech/blob/main/docs/source/asr/asr_customization/legacy_language_modeling_and_customization.rst

Quick start example: download and lowercase a pruned 3-gram ARPA model, then run WFST decoding with the stt_en_conformer_ctc_small_ls model on Librispeech test_other. Uses beam_width=[8] and lm_weight=[0.5,0.6,0.7,0.8,0.9]. Building the decoding WFST is a long process, so provide a decoding_wfst_file path to buffer it and avoid rebuilding on the next run.

```bash
wget -O - https://www.openslr.org/resources/11/3-gram.pruned.1e-7.arpa.gz | \
gunzip -c | tr '[:upper:]' '[:lower:]' > 3-gram.pruned.1e-7.arpa && \
python eval_wfst_decoding_ctc.py nemo_model_file="stt_en_conformer_ctc_small_ls" \
       input_manifest="<data_dir>/Librispeech/test_other.json" \
       arpa_model_file="3-gram.pruned.1e-7.arpa" \
       decoding_wfst_file="3-gram.pruned.1e-7.fst" \
       beam_width=[8] \
       lm_weight=[0.5,0.6,0.7,0.8,0.9]
```

--------------------------------

### Install NeMo and dependencies for Colab or local run

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/00_NeMo_Primer.ipynb

Setup cell for running the NeMo tutorial notebook locally or on Google Colab. Installs dependencies (wget, sox, libsndfile1, ffmpeg, text-unidecode), installs the NeMo toolkit from the Speech repository on the 'main' branch, and creates a 'configs' directory for the example configuration.

```python
"""
You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.

Instructions for setting up Colab are as follows:
1. Open a new Python 3 notebook.
2. Import this notebook from GitHub (File -> Upload Notebook -> "GITHUB" tab -> copy/paste GitHub URL)
3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select "GPU" for hardware accelerator)
4. Run this cell to set up dependencies.
"""
# If you're using Google Colab and not running locally, run this cell.

## Install dependencies
!pip install wget
!apt-get install sox libsndfile1 ffmpeg
!pip install text-unidecode

# ## Install NeMo
BRANCH = 'main'
!python -m pip install "nemo_toolkit[all] @ git+https://github.com/NVIDIA-NeMo/Speech.git@$BRANCH"

## Grab the config we'll use in this example
!mkdir configs
```

--------------------------------

### Setup dependencies and install NeMo for VAD tutorial

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/asr/Voice_Activity_Detection.ipynb

This cell installs required packages (wget, sox, libsndfile1, ffmpeg, text-unidecode), installs the NeMo ASR toolkit from the main branch, installs TorchAudio, and creates a configs directory. It is intended for Google Colab or local environments with a GPU.

```python
"""
You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.

Instructions for setting up Colab are as follows:
1. Open a new Python 3 notebook.
2. Import this notebook from GitHub (File -> Upload Notebook -> "GITHUB" tab -> copy/paste GitHub URL)
3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select "GPU" for hardware accelerator)
4. Run this cell to set up dependencies.


NOTE: User is responsible for checking the content of datasets and the applicable licenses and determining if suitable for the intended use.
"""
# If you're using Google Colab and not running locally, run this cell.

## Install dependencies
!pip install wget
!apt-get install sox libsndfile1 ffmpeg
!pip install text-unidecode

# ## Install NeMo
BRANCH = 'main'
!python -m pip install "nemo_toolkit[asr] @ git+https://github.com/NVIDIA-NeMo/Speech.git@$BRANCH"

## Install TorchAudio
## NOTE: TorchAudio installation may not work in all environments, please use Google Colab for best experience
!pip install torchaudio>=0.13.0 -f https://download.pytorch.org/whl/torch_stable.html

## Grab the config we'll use in this example
!mkdir configs
```

--------------------------------

### Install dependencies and NeMo for Colab

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/asr/Online_Noise_Augmentation.ipynb

Setup commands for running the notebook on Google Colab or locally. Installs dependencies (wget, sox, libsndfile1, ffmpeg, text-unidecode), installs NeMo ASR toolkit from the main branch, installs torchaudio, and creates a configs directory.

```bash
!pip install wget
!apt-get install sox libsndfile1 ffmpeg
!pip install text-unidecode
```

```python
BRANCH = 'main'
!python -m pip install "nemo_toolkit[asr] @ git+https://github.com/NVIDIA-NeMo/Speech.git@$BRANCH"
```

```bash
!pip install torchaudio>=0.13.0 -f https://download.pytorch.org/whl/torch_stable.html
```

```bash
!mkdir configs
```

--------------------------------

### Setup dependencies and install NeMo for streaming ASR

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/asr/Streaming_ASR.ipynb

Install required packages and NeMo toolkit from the main branch, and download the configuration file. This is intended for Google Colab or local GPU environments. After running, restart the runtime to apply upgraded packages.

```python
"""
You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.

Instructions for setting up Colab are as follows:
1. Open a new Python 3 notebook.
2. Import this notebook from GitHub (File -> Upload Notebook -> "GITHUB" tab -> copy/paste GitHub URL)
3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select "GPU" for hardware accelerator)
4. Run this cell to set up dependencies.
5. Restart the runtime (Runtime -> Restart Runtime) for any upgraded packages to take effect


NOTE: User is responsible for checking the content of datasets and the applicable licenses and determining if suitable for the intended use.
"""
# If you're using Google Colab and not running locally, run this cell.

## Install dependencies
!pip install wget
!apt-get install sox libsndfile1 ffmpeg
!pip install text-unidecode
!pip install matplotlib>=3.3.2

## Install NeMo
BRANCH = 'main'
!python -m pip install "nemo_toolkit[all] @ git+https://github.com/NVIDIA-NeMo/Speech.git@$BRANCH"

## Grab the config we'll use in this example
!mkdir configs
!wget -P configs/ https://raw.githubusercontent.com/NVIDIA/NeMo/$BRANCH/examples/asr/conf/config.yaml

"""
Remember to restart the runtime for the kernel to pick up any upgraded packages (e.g. matplotlib)!
Alternatively, you can uncomment the exit() below to crash and restart the kernel, in the case
that you want to use the "Run All Cells" (or similar) option.
"""
# exit()
```

--------------------------------

### Set up NeMo repository and install toolkit

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/tools/CTC_Segmentation_Tutorial.ipynb

Sets up the NeMo repository: either uses a local path or clones the Speech repo and installs the NeMo toolkit. The code checks for Google Colab or missing directory to trigger the clone and install.

```python
import os
# either provide a path to local NeMo repository with NeMo already installed or git clone

# option #1: local path to NeMo repo with NeMo already installed
NEMO_DIR_PATH = "NeMo"

# option #2: download NeMo repo
if 'google.colab' in str(get_ipython()) or not os.path.exists(NEMO_DIR_PATH):
  ! git clone -b $BRANCH https://github.com/NVIDIA-NeMo/Speech
  ! cd NeMo
  ! python -m pip install "nemo_toolkit[all] @ git+https://github.com/NVIDIA-NeMo/Speech.git@$BRANCH"
```

--------------------------------

### Install dependencies and NeMo ASR toolkit

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/asr/Canary_Multitask_Speech_Model.ipynb

Sets up the environment for running the Canary tutorial, either locally or on Google Colab. Installs system dependencies (sox, libsndfile1, ffmpeg) and Python packages (wget, text-unidecode, omegaconf), then installs the NeMo ASR toolkit from the NVIDIA-NeMo/Speech repository at the specified branch. Includes Colab setup instructions and a note about dataset licensing.

```python
"""
You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.

Instructions for setting up Colab are as follows:
1. Open a new Python 3 notebook.
2. Import this notebook from GitHub (File -> Upload Notebook -> "GitHub" tab -> copy/paste GitHub URL)
3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select "GPU" for hardware accelerator)
4. Run this cell to set up dependencies.
5. Restart the runtime (Runtime -> Restart Runtime) for any upgraded packages to take effect


NOTE: User is responsible for checking the content of datasets and the applicable licenses and determining if suitable for the intended use.
"""
# If you're using Google Colab and not running locally, run this cell to install dependencies


# Install dependencies
!pip install wget
!apt-get update && apt-get install -y sox libsndfile1 ffmpeg
!pip install text-unidecode
!pip install omegaconf

BRANCH='main'

!python -m pip install "nemo_toolkit[asr] @ git+https://github.com/NVIDIA-NeMo/Speech.git@{BRANCH}"
```

--------------------------------

### Setup Dependencies for Google Colab

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/audio/speech_enhancement/BNR_Speech_enhancement_with_NeMo.ipynb

Instructions to set up the notebook on Google Colab, including installing dependencies and NeMo from the main branch. Run this cell first to prepare the environment.

```python
"""
You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.

Instructions for setting up Colab are as follows:
1. Open a new Python 3 notebook.
2. Import this notebook from GitHub (File -> Upload Notebook -> "GITHUB" tab -> copy/paste GitHub URL)
3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select "GPU" for hardware accelerator)
4. Run this cell to set up dependencies.
5. Restart the runtime (Runtime -> Restart Runtime) for any upgraded packages to take effect
"""

GIT_USER = 'NVIDIA-NeMo'
BRANCH = 'main'

if 'google.colab' in str(get_ipython()):

    # Install dependencies
    !pip install wget
    !apt-get install sox libsndfile1 ffmpeg
    !pip install text-unidecode
    !pip install matplotlib>=3.3.2

    ## Install NeMo
    !python -m pip install "nemo_toolkit[all] @ git+https://github.com/{GIT_USER}/Speech.git@{BRANCH}"

    ## Install TorchAudio
    !pip install torchaudio>=0.13.0 -f https://download.pytorch.org/whl/torch_stable.html

```

--------------------------------

### NeMo manifest file example

Source: https://github.com/nvidia-nemo/speech/blob/main/docs/source/starthere/key_concepts.rst

Example manifest file in JSONL format. Each line is one training example with audio_filepath, text (transcript or input text), and duration in seconds.

```json
{"audio_filepath": "/data/audio/001.wav", "text": "hello world", "duration": 2.5}
{"audio_filepath": "/data/audio/002.wav", "text": "how are you", "duration": 1.8}
```

--------------------------------

### Install SageMaker and AWS CLI

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/cloud/aws/ASR_Finetuning_at_Scale_with_AWS_SageMaker.ipynb

Installs the SageMaker and AWS CLI packages required for the tutorial.

```bash
pip install sagemaker awscli
```

--------------------------------

### Set up tutorial directories and download NeMo scripts

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/tts/Audio_Codec_Training.ipynb

Defines directory paths for the tutorial, sets the NeMo download directory, and clones the NeMo Speech repository. Comment out the clone line and set NEMO_ROOT_DIR manually if running locally.

```python
# Directory where tutorialscripts will run and outputs will be saved.
ROOT_DIR = Path().absolute() / "codec_tutorial"

# Nemo code paths
NEMO_DIR = ROOT_DIR / "nemo"
NEMO_SCRIPT_DIR = NEMO_DIR / "scripts" / "dataset_processing" / "tts"
NEMO_EXAMPLES_DIR = NEMO_DIR / "examples" / "tts"
NEMO_CONFIG_DIR = NEMO_EXAMPLES_DIR / "conf"

nemo_download_dir = str(NEMO_DIR)
# Download local version of NeMo scripts. If you are running locally and want to use your own local NeMo code,
# comment out the below line and set NEMO_ROOT_DIR to your local path.
!git clone -b $BRANCH https://github.com/NVIDIA-NeMo/Speech.git $nemo_download_dir
```

--------------------------------

### Transcribe with timestamps=True and print segment and word timestamps

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/asr/Canary_Multitask_Speech_Model.ipynb

Use the transcribe method with timestamps=True to get timestamp predictions for all languages. The output transcript object contains a 'timestamp' dictionary with 'segment' and 'word' lists, each having 'start' and 'end' keys. The example prints segment-level and word-level timestamps; the commented line can be uncommented to listen to word segments.

```python
# To recognize with timestamps
transcript = canary_model.transcribe(
  audio=[audio_path],
  batch_size=1,
  source_lang='en',	# en: English or other supported language
  target_lang='en',	# should be same as "source_lang" for 'asr'
  timestamps=True
)
print("\n\nEnglish speech to text recognition with timestamp prediction:\n")

print(f'Predicted output: \n"{transcript[0].text}"')

print('\nSegment level timestamps:')
for sample in transcript[0].timestamp['segment']:
    segment, start, end = sample['segment'], sample['start'], sample['end']
    print(f'{segment}')
    print(f'Segment start: {start:.2f}s')
    print(f'Segment end: {end:.2f}s\n')

print('\nWord level timestamps:')
for sample in transcript[0].timestamp['word']:
    word, start, end = sample['word'], sample['start'], sample['end']
    print(f'{word:<15}[{start:.2f}s, {end:.2f}s]')
    # listen_to_audio(audio_path, offset=start, duration=(end-start)) # uncomment to listen to word segments
```

--------------------------------

### Result of setup cell

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/asr/ASR_with_NeMo.ipynb

Result of evaluating the setup cell's final string literal. It displays the reminder to restart the runtime after installing packages.

```python
'\nRemember to restart the runtime for the kernel to pick up any upgraded packages (e.g. matplotlib)!\nAlternatively, you can uncomment the exit() below to crash and restart the kernel, in the case\nthat you want to use the "Run All Cells" (or similar) option.\n'
```

--------------------------------

### Set up NeMo environment and install dependencies

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/asr/ASR_Confidence_Estimation.ipynb

Configures the environment for the tutorial. It either uses a local NeMo repository path or, if running on Colab or the local path is missing, installs system dependencies (sox, libsndfile1, ffmpeg), clones the Speech repository, installs the nemo_toolkit[all] package from the specified branch, and adds the repository path to sys.path.

```python
import os
# either provide a path to local NeMo repository with NeMo already installed or git clone

# option #1: local path to NeMo repo with NeMo already installed
NEMO_DIR_PATH = os.path.dirname(os.path.dirname(os.path.abspath('')))
is_colab = False

# option #2: download NeMo repo
if 'google.colab' in str(get_ipython()) or not os.path.exists(os.path.join(NEMO_DIR_PATH, "nemo")):
    ## Install dependencies
    !apt-get install sox libsndfile1 ffmpeg

    !git clone -b $BRANCH https://github.com/NVIDIA-NeMo/Speech
    %cd NeMo
    !python -m pip install "nemo_toolkit[all] @ git+https://github.com/NVIDIA-NeMo/Speech.git@$BRANCH"
    NEMO_DIR_PATH = os.path.abspath('')
    is_colab = True

import sys
sys.path.insert(0, NEMO_DIR_PATH)
```

--------------------------------

### Set Up Working Directory and Download Sample Audio

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/tts/Audio_Codec_Inference.ipynb

Creates a working directory and downloads a sample audio file (LJ023-0089.wav) from a public URL if it does not already exist.

```python
# Working directory
ROOT_DIR = Path().absolute() / 'codec_tutorial'

# Create dataset directory
DATA_DIR = ROOT_DIR / 'data'
DATA_DIR.mkdir(parents=True, exist_ok=True)

audio_path = DATA_DIR / 'LJ023-0089.wav'
audio_url = "https://multilangaudiosamples.s3.us-east-2.amazonaws.com/LJ023-0089.wav"

if not os.path.exists(audio_path):
  wget.download(audio_url, audio_path.as_posix())
```

--------------------------------

### Start the server

Source: https://github.com/nvidia-nemo/speech/blob/main/examples/voice_agent/README.md

Sets environment variables and runs the server. Replace NEMO_PATH with your local NeMo path. Optionally set HF_TOKEN, HF_HUB_CACHE, SERVER_CONFIG_PATH, and SERVER_PUBLIC_HOST.

```bash
NEMO_PATH=???  # Use your local NeMo path with the latest main branch from: https://github.com/NVIDIA-NeMo/Speech
export PYTHONPATH=$NEMO_PATH:$PYTHONPATH
# export HF_TOKEN="hf_..."  # Use your own HuggingFace API token if needed, as some models may require.
# export HF_HUB_CACHE="/path/to/your/huggingface/cache"  # change where HF cache is stored if you don't want to use the default cache
# export SERVER_CONFIG_PATH="/path/to/your/server/config.yaml"  # change to the server config you want to use, otherwise it will use the default config in `server/server_configs/default.yaml`
# export SERVER_PUBLIC_HOST="127.0.0.1"  # set this to the host/IP clients should use for the WebSocket server
python ./server/server.py
```

--------------------------------

### Setup dependencies for Google Colab and local environment

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/asr/Online_ASR_Microphone_Demo_Buffered_Streaming.ipynb

Install dependencies and NeMo ASR toolkit, and create a configs directory. This cell is intended for Google Colab or local setup with GPU.

```python
```python
"""
Please run notebook locally (if you have all the dependencies and a GPU). 
Technically you can run this notebook on Google Colab but you need to set up microphone for Colab.

Instructions for setting up Colab are as follows:
1. Open a new Python 3 notebook.
2. Import this notebook from GitHub (File -> Upload Notebook -> "GITHUB" tab -> copy/paste GitHub URL)
3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select "GPU" for hardware accelerator)
4. Run this cell to set up dependencies.
5. Set up microphone for Colab


NOTE: User is responsible for checking the content of datasets and the applicable licenses and determining if suitable for the intended use.
"""
# If you're using Google Colab and not running locally, run this cell.

## Install dependencies
!pip install wget
!apt-get install sox libsndfile1 ffmpeg portaudio19-dev
!pip install text-unidecode
!pip install pyaudio

# ## Install NeMo
BRANCH = 'main'
!python -m pip install "nemo_toolkit[asr] @ git+https://github.com/NVIDIA-NeMo/Speech.git@$BRANCH"

## Grab the config we'll use in this example
!mkdir configs
```
```

--------------------------------

### Define Hugging Face model name and markdown template

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/Publish_NeMo_Model_On_Hugging_Face_Hub.ipynb

Defines the Hugging Face model name and a markdown template for the model card. The template includes installation instructions, usage examples, and input/output specifications. The code block contains nested code examples for pip install, model instantiation, transcription, and shell commands.

```python
hf_model_name = f'{username}/{MODEL_NAME}'

TEMPLATE = f"""
## Model Overview

<DESCRIBE IN ONE LINE THE MODEL AND ITS USE>

## NVIDIA NeMo: Training

To train, fine-tune or play with the model you will need to install [NVIDIA NeMo](https://github.com/NVIDIA/NeMo). We recommend you install it after you've installed latest Pytorch version.
```
pip install nemo_toolkit['all']
```

## How to Use This Model

The model is available for use in the NeMo toolkit [3], and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset.

### Automatically instantiate the model

```python
import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.ASRModel.from_pretrained("{hf_model_name}")
```

### Transcribing using Python
First, let's get a sample
```
wget https://dldata-public.s3.us-east-2.amazonaws.com/2086-149220-0033.wav
```
Then simply do:
```
asr_model.transcribe(['2086-149220-0033.wav'])
```

### Transcribing many audio files

```shell
python [NEMO_GIT_FOLDER]/examples/asr/transcribe_speech.py \
 pretrained_name="{hf_model_name}" \
 audio_dir="<DIRECTORY CONTAINING AUDIO FILES>"
```

### Input

This model accepts 16000 KHz Mono-channel Audio (wav files) as input.

### Output

This model provides transcribed speech as a string for a given audio sample.
"""
```

--------------------------------

### Build Sphinx docs with uv

Source: https://github.com/nvidia-nemo/speech/blob/main/AGENTS.md

Build the Sphinx documentation. Run the one-time setup first, then either a full or incremental rebuild. Output is written to docs/build/html/; open docs/build/html/index.html to preview locally.

```bash
uv sync --locked --group docs                        # one-time setup (matches CI)
uv run make -C docs clean html                       # full rebuild
uv run make -C docs html                             # incremental rebuild
```

--------------------------------

### Install NeMo and dependencies for Colab

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/asr/Offline_ASR_with_VAD_for_CTC_models.ipynb

Setup cell for running the notebook on Google Colab. Installs wget and the NeMo toolkit from the main branch of the NVIDIA-NeMo/Speech repository. After installation, restart the runtime to apply upgraded packages; optionally uncomment exit() to force a kernel restart.

```python
"""
You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.

Instructions for setting up Colab are as follows:
1. Open a new Python 3 notebook.
2. Import this notebook from GitHub (File -> Upload Notebook -> "GITHUB" tab -> copy/paste GitHub URL)
3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select "GPU" for hardware accelerator)
4. Run this cell to set up dependencies.
5. Restart the runtime (Runtime -> Restart Runtime) for any upgraded packages to take effect


NOTE: User is responsible for checking the content of datasets and the applicable licenses and determining if suitable for the intended use.
"""
# If you're using Google Colab and not running locally, run this cell.

## Install dependencies
!pip install wget

## Install NeMo
BRANCH = 'main'
!python -m pip install "nemo_toolkit[all] @ git+https://github.com/NVIDIA-NeMo/Speech.git@$BRANCH"

"""
Remember to restart the runtime for the kernel to pick up any upgraded packages (e.g. matplotlib)!
Alternatively, you can uncomment the exit() below to crash and restart the kernel, in the case
that you want to use the "Run All Cells" (or similar) option.
"""
# exit()
```

--------------------------------

### Initialize and start Label Studio ML backend

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/tools/label-studio/setup-asr-preannotations.ipynb

Initializes a new ML backend directory named nemo-ml-backend using the asr.py script, then starts the ML backend serving on the default http://localhost:9090.

```python
!label-studio-ml init nemo-ml-backend --from asr.py
```

```python
!label-studio-ml start nemo-ml-backend
```

--------------------------------

### Install dependencies and NeMo for Colab

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/asr/ASR_with_NeMo.ipynb

Setup cell for running the notebook on Google Colab. Installs system dependencies (sox, libsndfile1, ffmpeg) and Python packages (wget, text-unidecode, matplotlib>=3.3.2). The NeMo installation command is commented out; uncomment it to install NeMo from the Speech repository. After running, restart the runtime to apply upgraded packages.

```python
"""
You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.

Instructions for setting up Colab are as follows:
1. Open a new Python 3 notebook.
2. Import this notebook from GitHub (File -> Upload Notebook -> "GITHUB" tab -> copy/paste GitHub URL)
3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select "GPU" for hardware accelerator)
4. Run this cell to set up dependencies.
5. Restart the runtime (Runtime -> Restart Runtime) for any upgraded packages to take effect


NOTE: User is responsible for checking the content of datasets and the applicable licenses and determining if suitable for the intended use.
"""
# If you're using Google Colab and not running locally, run this cell.

## Install dependencies
!pip install wget
!apt-get install sox libsndfile1 ffmpeg
!pip install text-unidecode
!pip install matplotlib>=3.3.2

## Install NeMo
BRANCH = 'main'
# !python -m pip install "nemo_toolkit[all] @ git+https://github.com/NVIDIA-NeMo/Speech.git@$BRANCH"

"""
Remember to restart the runtime for the kernel to pick up any upgraded packages (e.g. matplotlib)!
Alternatively, you can uncomment the exit() below to crash and restart the kernel, in the case
that you want to use the "Run All Cells" (or similar) option.
"""
# exit()
```

--------------------------------

### Install NeMo ASR and dependencies for Colab

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/speaker_tasks/ASR_with_SpeakerDiarization.ipynb

Setup cell for running the notebook on Google Colab or locally. Installs system dependencies (sox, libsndfile1, ffmpeg) and Python packages (wget, text-unidecode), then installs the NeMo ASR toolkit from the NVIDIA-NeMo/Speech GitHub repository at the 'main' branch.

```python
"""
You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.

Instructions for setting up Colab are as follows:
1. Open a new Python 3 notebook.
2. Import this notebook from GitHub (File -> Upload Notebook -> "GITHUB" tab -> copy/paste GitHub URL)
3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select "GPU" for hardware accelerator)
4. Run this cell to set up dependencies.
"""
# If you're using Google Colab and not running locally, run this cell.

## Install dependencies
!pip install wget
!apt-get install sox libsndfile1 ffmpeg
!pip install text-unidecode

# ## Install NeMo
BRANCH = 'main'
!python -m pip install "nemo_toolkit[asr] @ git+https://github.com/NVIDIA-NeMo/Speech.git@{BRANCH}"
```

--------------------------------

### Set up experiment manager for finetuning

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/speaker_tasks/Speaker_Identification_Verification.ipynb

Sets up the experiment manager for the finetuning trainer and prints the log directory.

```python
log_dir_finetune = exp_manager(trainer_finetune, config.get("exp_manager", None))
print(log_dir_finetune)
```

--------------------------------

### Example UEM file lines

Source: https://github.com/nvidia-nemo/speech/blob/main/docs/source/asr/speaker_diarization/datasets.rst

Example lines of a UEM file specifying scoring regions. Format: <uniq-id> <channel ID> <start time> <end time>, with channel ID set to 1.

```bash
TS3012d.Mix-Headset 1 12.31 108.98
TS3012d.Mix-Headset 1 214.00 857.09
```

--------------------------------

### Install NeMo and dependencies for ASR tutorial

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/asr/Intro_to_Transducers.ipynb

Sets up the environment for running the NeMo ASR tutorial, either locally or on Google Colab. Installs required system and Python dependencies, then installs the NeMo toolkit from the NVIDIA-NeMo/Speech repository at the 'main' branch. Includes a note that users are responsible for checking dataset licenses.

```python
"""
You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.

Instructions for setting up Colab are as follows:
1. Open a new Python 3 notebook.
2. Import this notebook from GitHub (File -> Upload Notebook -> "GITHUB" tab -> copy/paste GitHub URL)
3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select "GPU" for hardware accelerator)
4. Run this cell to set up dependencies.
5. Restart the runtime (Runtime -> Restart Runtime) for any upgraded packages to take effect


NOTE: User is responsible for checking the content of datasets and the applicable licenses and determining if suitable for the intended use.
"""
# If you're using Google Colab and not running locally, run this cell.
import os

# Install dependencies
!pip install wget
!apt-get install sox libsndfile1 ffmpeg
!pip install text-unidecode
!pip install matplotlib>=3.3.2

## Install NeMo
BRANCH = 'main'
!python -m pip install "nemo_toolkit[all] @ git+https://github.com/NVIDIA-NeMo/Speech.git@$BRANCH"
```

--------------------------------

### Initialize SALMAutomodel with EP=2 and run inference

Source: https://github.com/nvidia-nemo/speech/blob/main/docs/source/speechlm2/intro.rst

Initialize distributed setup with EP=2 and load a pretrained SALMAutomodel, then run inference with torch.inference_mode(). The strategy packages the mesh and execution policies in distributed_setup.

```python
# Initialize distributed and resolve the Automodel topology with EP=2.
# The strategy packages the mesh and execution policies in distributed_setup.
strategy = setup_distributed(ep_size=2)

# Load a pretrained SALMAutomodel with the resolved distributed setup.
model = slm.models.SALMAutomodel.from_pretrained(
    "path/to/checkpoint",
    distributed_setup=strategy.distributed_setup,
).eval()

# Inference is identical to SALM
with torch.inference_mode():
    output = model.generate(
        prompts=[prompt],
        audios=audio_signal,
        audio_lens=audio_len,
    )
```

--------------------------------

### Install NeMo Speech and transcribe audio with ASRModel

Source: https://github.com/nvidia-nemo/speech/blob/main/docs/source/index.rst

Install the NeMo Speech toolkit with ASR and TTS extras, then load a pretrained ASR model and transcribe an audio file. The Python example uses the parakeet-tdt-0.6b-v2 model from NVIDIA.

```bash
uv pip install 'nemo-toolkit[asr,tts]'
```

```python
import nemo.collections.asr as nemo_asr
model = nemo_asr.models.ASRModel.from_pretrained("nvidia/parakeet-tdt-0.6b-v2")
print(model.transcribe(["audio.wav"])[0].text)
```

--------------------------------

### Install pre-commit and set up Git hooks

Source: https://github.com/nvidia-nemo/speech/blob/main/AGENTS.md

Installs the pre-commit tool and sets up the repository's Git hooks. Run once to enable automatic checks on staged files for every commit.

```bash
uv tool install pre-commit
pre-commit install
```

--------------------------------

### Instantiate SimpleModel with old config and summarize

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/02_NeMo_Adapters.ipynb

Creates a SimpleModel instance using the original config (which does not yet support adapters) and prints its summary. This is the baseline before adding adapters.

```python
old_config = get_model_config(dim)
model = SimpleModel(old_config)
model.summarize()
```

--------------------------------

### Import libraries and set up device

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/tts/NeMo_TTS_Primer.ipynb

Imports required libraries and sets up the device (CUDA if available, otherwise CPU). Seeds PyTorch for reproducibility and prints the selected device.

```python
from pathlib import Path
import time

import IPython.display as ipd
import matplotlib.pyplot as plt
import soundfile as sf
import torch

from nemo.collections.tts.models import MagpieTTSModel

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
torch.manual_seed(0)
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(0)

print(f"Using {device}")
```

--------------------------------

### Start EasyMagpie server with run_server.sh

Source: https://github.com/nvidia-nemo/speech/blob/main/tools/easymagpie_vllm_omni/README.md

Starts the vllm serve process with the EasyMagpie plugin on port 8091. The converted model path is passed as the first argument. After startup, the HTTP and WebSocket serving APIs become available.

```bash
bash ./scripts/run_server.sh ./converted_model 8091
```

--------------------------------

### Run NeMo Docker container and install Label Studio

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/tools/label-studio/setup-asr-preannotations.ipynb

Starts a NeMo Docker container with GPU support and exposes ports 8888, 6006, and 8080 (the default Label Studio port). Then installs the Label Studio package via pip.

```python
!docker run --gpus all -it --rm --shm-size=8g \
-p 8888:8888 -p 6006:6006 -p 8080:8080 --ulimit memlock=-1 --ulimit \
stack=67108864 --device=/dev/snd nvcr.io/nvidia/nemo:1.0.1
```

```python
!pip install label-studio
```

--------------------------------

### Example CTM file lines

Source: https://github.com/nvidia-nemo/speech/blob/main/docs/source/asr/speaker_diarization/datasets.rst

Example lines of a CTM file for word-level diarization evaluation. Format: <session name> <channel ID> <start time> <duration> <word> <confidence> <type of token> <speaker>. Confidence is NA and type of token is lex.

```bash
TS3012d.Mix-Headset 1 12.879 0.32 okay NA lex MTD046ID
TS3012d.Mix-Headset 1 13.203 0.24 yeah NA lex MTD046ID
```

--------------------------------

### Install NeMo and dependencies for Colab

Source: https://github.com/nvidia-nemo/speech/blob/main/tutorials/asr/Transducers_with_HF_Datasets.ipynb

Setup cell for running the notebook on Google Colab or locally. Installs dependencies (wget, sox, libsndfile1, ffmpeg, text-unidecode, matplotlib>=3.3.2), installs NeMo from the 'main' branch of the NVIDIA-NeMo/Speech repository, and downgrades the datasets library to 3.4.0 because version 4.0.0 has issues. After running, restart the runtime for upgraded packages to take effect.

```python
"""
You can run either this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.

Instructions for setting up Colab are as follows:
1. Open a new Python 3 notebook.
2. Import this notebook from GitHub (File -> Upload Notebook -> "GITHUB" tab -> copy/paste GitHub URL)
3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select "GPU" for hardware accelerator)
4. Run this cell to set up dependencies.
5. Restart the runtime (Runtime -> Restart Runtime) for any upgraded packages to take effect


NOTE: User is responsible for checking the content of datasets and the applicable licenses and determining if suitable for the intended use.
"""
# If you're using Google Colab and not running locally, run this cell.
import os

# # Install dependencies
!pip install wget
!apt-get install sox libsndfile1 ffmpeg
!pip install text-unidecode
!pip install matplotlib>=3.3.2

## Install NeMo
BRANCH = 'main'
!python -m pip install "nemo_toolkit[all] @ git+https://github.com/NVIDIA-NeMo/Speech.git@$BRANCH"

!pip install --upgrade datasets==3.4.0 # downgrading to 3.4.0 because latest version (4.0.0) has some issues
```

--------------------------------

### Initialize S3CheckpointIO and NLPDDPStrategy

Source: https://github.com/nvidia-nemo/speech/blob/main/docs/source/common/s3_checkpointing.rst

Shows how to create an S3CheckpointIO instance with configuration values and pass it to NLPDDPStrategy. Requires dirpath to be an S3 dirpath.

```bash
async_checkpointing = self.cfg.s3_checkpointing.get('enable_async_checkpointing', False)
chunk_size_MB = self.cfg.s3_checkpointing.get('chunk_size_MB')
max_read_concurrency = self.cfg.s3_checkpointing.get('max_read_concurrency')
max_write_concurrency = self.cfg.s3_checkpointing.get('max_write_concurrency')
dirpath = self.cfg.exp_manager.checkpoint_callback_params.get('dirpath')

s3_checkpoint_io = S3CheckpointIO(dirpath=dirpath, chunk_size_MB=chunk_size_MB, max_read_concurrency=max_read_concurrency, max_write_concurrency=max_write_concurrency, async_checkpointing=async_checkpointing)

strategy = NLPDDPStrategy(
    no_ddp_communication_hook=True,
    checkpoint_io=s3_checkpoint_io,
    gradient_as_bucket_view=self.cfg.model.gradient_as_bucket_view,
    find_unused_parameters=False,
    nccl_communicator_config_path=self.cfg.model.get('nccl_communicator_config_path', None),
    sharp=self.cfg.model.get('sharp', False),
)
```
Tokens:
Raw