NVIDIA NeMo

https://docs.nvidia.com/nemo/
Admin
NVIDIA NeMo is a modular software suite of APIs and libraries that help developers manage the AI...
Tokens:80,543
Snippets:1,021
Trust Score:10
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install NeMo-Run

Source: https://docs.nvidia.com/nemo/run/latest

Standard installation command for the NeMo-Run project.

```bash
ip install git+https://github.com/NVIDIA-NeMo/Run.git
```

--------------------------------

### Install TransformerEngine with vLLM backend

Source: https://docs.nvidia.com/nemo/export-deploy/latest/index.html

Install required dependencies and the package with vLLM support.

```bash
pip install torch==2.7.0 setuptools pybind11 wheel_stub # Required for TE
pip install --no-build-isolation .[vllm]
```

--------------------------------

### Install TransformerEngine with InFramework backend

Source: https://docs.nvidia.com/nemo/export-deploy/latest/index.html

Install required dependencies and the main package for the InFramework backend.

```bash
pip install torch==2.7.0 setuptools pybind11 wheel_stub # Required for TE
git clone https://github.com/NVIDIA-NeMo/Export-Deploy
cd Export-Deploy/
pip install --no-build-isolation .
```

--------------------------------

### Install NeMo Agent Toolkit

Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html

Install the toolkit and LangChain integration using either uv or pip.

```bash
uv pip install "nvidia-nat[langchain]"
```

```bash
pip install "nvidia-nat[langchain]"
```

--------------------------------

### Install NeMo Microservices SDK

Source: https://docs.nvidia.com/nemo/microservices/latest/generate-synthetic-data/index.html

Install the SDK with the data-designer extra to enable synthetic data generation capabilities.

```bash
pip install nemo-microservices[data-designer]
```

--------------------------------

### Install NeMo Agent Toolkit

Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html

Installation commands for the toolkit using different package managers.

```bash
uv pip install nvidia-nat
```

```bash
pip install nvidia-nat
```

--------------------------------

### Manifest File Format Example

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/datasets.html

The manifest file should contain relative paths to audio files within the bucket.

```json
{"audio_filepath": "path/to/audio.wav", "text": "transcription of the uterance", "duration": 23.147}
```

--------------------------------

### Complete AIStore Training Configuration

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/datasets.html

Example command configuring environment variables and dataset paths for a training run with AIS.

```bash
export AIS_ENDPOINT=http://hostname:port \
&& export NEMO_DATA_STORE_CACHE_DIR=/tmp \
&& export NEMO_DATA_STORE_CACHE_SHARED=0 \
python speech_to_text_bpe.py \
...
model.train_ds.manifest_filepath=ais://train_bucket/tarred_audio_manifest.json \
model.train_ds.tarred_audio_filepaths=ais://train_bucket/audio__OP_0..511_CL_.tar \
++model.train_ds.defer_setup=true \
mode.validation_ds.manifest_filepath=ais://validation_bucket/validation_manifest.json \
++model.validation_ds.defer_setup=true
```

--------------------------------

### Audio File Path List Example

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/speaker_diarization/datasets.html

Example content for the audio file path list text file.

```text
/path/to/abcd01.wav
/path/to/abcd02.wav
```

--------------------------------

### Install Lepton CLI

Source: https://docs.nvidia.com/nemo/run/latest

Installation command for the Lepton CLI required for DGX Cloud Lepton usage.

```bash
pip install leptonai
```

--------------------------------

### Instantiate a Lhotse dataloader

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/datasets.html

Python example for initializing a dataloader from a configuration dictionary.

```python
from nemo.collections.common.data.lhotse.dataloader import get_lhotse_dataloader_from_config

dl = get_lhotse_dataloader_from_config({
        "input_cfg": [
            {"type": "txt", "paths": "/path/to/document_{0..1}.txt", "language": "en"},
        ],
        "use_multimodal_dataloading": True,
        "batch_size": 4,
    },
    global_rank=0,
    world_size=1,
    dataset=MyDatasetClass(),  # converts CutSet -> dict[str, Tensor]
    tokenizer=my_tokenizer,
)
```

--------------------------------

### Manifest File Line Example

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/speaker_diarization/datasets.html

Example of a single line entry in the generated train_manifest.json file.

```json
{"audio_filepath": "/path/to/abcd01.wav", "offset": 0, "duration": 90, "label": "infer", "text": "-", "num_speakers": 2, "rttm_filepath": "/path/to/rttm/abcd01.rttm"}
```

--------------------------------

### UEM File Example

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/speaker_diarization/datasets.html

Example lines for a UEM file specifying scoring regions.

```text
TS3012d.Mix-Headset 1 12.31 108.98
TS3012d.Mix-Headset 1 214.00 857.09
```

--------------------------------

### Install Skypilot with Optional Features

Source: https://docs.nvidia.com/nemo/run/latest

Commands to install NeMo-Run with specific Skypilot support for Kubernetes or all cloud platforms.

```bash
pip install git+https://github.com/NVIDIA-NeMo/Run.git[skypilot]
```

```bash
pip install git+https://github.com/NVIDIA-NeMo/Run.git[skypilot-all]
```

--------------------------------

### Sample filelist.txt content

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/speaker_recognition/datasets.html

Example of the input text file format containing paths to audio files for manifest generation.

```text
/data/datasets/voxceleb/data/dev/aac_wav/id00179/Q3G6nMr1ji0/00086.wav
/data/datasets/voxceleb/data/dev/aac_wav/id00806/VjpQLxHQQe4/00302.wav
/data/datasets/voxceleb/data/dev/aac_wav/id01510/k2tzXQXvNPU/00132.wav
```

--------------------------------

### TitaNet Dataset Configuration Example

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/speaker_recognition/configs.html

Example configuration for TitaNet training and validation datasets, specifying manifest paths, sample rates, and batch sizes.

```yaml
model:
  train_ds:
    manifest_filepath: ???
    sample_rate: 16000
    labels: None   # finds labels based on manifest file
    batch_size: 32
    trim_silence: False
    shuffle: True

  validation_ds:
    manifest_filepath: ???
    sample_rate: 16000
    labels: None   # Keep None, to match with labels extracted during training
    batch_size: 32
    shuffle: False    # No need to shuffle the validation data
```

--------------------------------

### Install TransformerEngine with TensorRT-LLM backend

Source: https://docs.nvidia.com/nemo/export-deploy/latest/index.html

Install system and Python dependencies followed by the package with TensorRT-LLM support.

```bash
sudo apt-get -y install libopenmpi-dev # Required for TensorRT-LLM
pip install torch==2.7.0 setuptools pybind11 wheel_stub # Required for TE
pip install --no-build-isolation .[trtllm]
```

--------------------------------

### Defer Dataloader Setup for Distributed Caching

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/datasets.html

Set defer_setup to true to ensure distributed environment initialization before caching data.

```bash
++model.train_ds.defer_setup=true
++model.validation_ds.defer_setup=true
++model.test_ds.defer_setup=true
```

--------------------------------

### Define Manifest Format for Prompt Conditioning

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/configs.html

Example of a training manifest entry containing prompt information.

```json
{
  "audio_filepath": "/path/to/audio.wav",
  "text": "transcription text",
  "duration": 10.5,
  "target_lang": "en-US"
}
```

--------------------------------

### Login to HuggingFace CLI

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/speaker_diarization/results.html

Commands to install the HuggingFace CLI and authenticate.

```bash
pip install -U "huggingface_hub[cli]"
huggingface-cli login
```

--------------------------------

### Configure text dataloading

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/datasets.html

Example configuration for loading raw text files.

```yaml
input_cfg:
  - type: txt
    paths: /path/to/document_{0..1}.txt
    language: en  # optional
```

--------------------------------

### Install Nemotron Parse dependencies

Source: https://docs.nvidia.com/nemo/retriever/latest

Install the necessary client dependencies to enable the nemotron_parse extraction method for PDF files.

```bash
pip install "nemo-retriever[nemotron-parse]"
```

--------------------------------

### Defer dataloader setup

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/configs.html

Set defer_setup to true to delay dataloader initialization until the model's setup() method is called.

```yaml
model:
  train_ds:
    # Configure training data as usual
    ...
    # Defer train dataloader setup from `__init__` to `setup`
    defer_setup: true
```

--------------------------------

### Workflow Output

Source: https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html

Example console output after running the workflow.

```text
Workflow Result:
['Here are five subspecies of Aardvarks:

1. Orycteropus afer afer (Southern aardvark)
2. O. a. adametzi  Grote, 1921 (Western aardvark)
3. O. a. aethiopicus  Sundevall, 1843
4. O. a. angolensis  Zukowsky & Haltenorth, 1957
5. O. a. erikssoni  Lönnberg, 1906']
```

--------------------------------

### Start NeMo Guardrails API server

Source: https://docs.nvidia.com/nemo/guardrails/latest

Command to launch a local guardrails server using the CLI.

```bash
nemoguardrails server --config ./config --port 8000
```

--------------------------------

### Install Inference Backends inside Container

Source: https://docs.nvidia.com/nemo/export-deploy/latest/index.html

Commands to install specific inference backends within the NeMo FW container environment.

```bash
cd /opt/Export-Deploy
uv sync --link-mode symlink --locked --extra trtllm $(cat /opt/uv_args.txt)
```

```bash
cd /opt/Export-Deploy
uv sync --link-mode symlink --locked --extra vllm $(cat /opt/uv_args.txt)
```

```bash
cd /opt/Export-Deploy
uv sync --link-mode symlink --locked --extra trt-onnx $(cat /opt/uv_args.txt)
```

--------------------------------

### Example JSON Manifest Format

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/examples/kinyarwanda_asr.html

The expected structure of the JSON manifest file containing audio file paths and metadata.

```json
{"audio_filepath": "cv-corpus-9.0-2022-04-27/rw/clips/common_voice_rw_26273273.mp3", "text": "kandi tuguwe neza kugira ngo twakire amagambo y\u2019ukuri,", "up_votes": 2, "down_votes": 0, "age": "twenties", "gender": "male", "accents": NaN, "client_id": "e2a04c0ecacf81302f4270a3dddaa7a131420f6b7319208473af17d4adf3724ad9a3b6cdee107e2f321495db86f114a50c396e0928464a58dfad472130e7514a"}
{"audio_filepath": "cv-corpus-9.0-2022-04-27/rw/clips/common_voice_rw_26273478.mp3", "text": "Simbi na we akajya kwiga nubwo byari bigoye", "up_votes": 2, "down_votes": 0, "age": "twenties", "gender": "male", "accents": NaN, "client_id": "e2a04c0ecacf81302f4270a3dddaa7a131420f6b7319208473af17d4adf3724ad9a3b6cdee107e2f321495db86f114a50c396e0928464a58dfad472130e7514a"}
{"audio_filepath": "cv-corpus-9.0-2022-04-27/rw/clips/common_voice_rw_26273483.mp3", "text": "Inshuti yanjye yaje kunsura ku biro byanjye.", "up_votes": 2, "down_votes": 0, "age": "twenties", "gender": "male", "accents": NaN, "client_id": "e2a04c0ecacf81302f4270a3dddaa7a131420f6b7319208473af17d4adf3724ad9a3b6cdee107e2f321495db86f114a50c396e0928464a58dfad472130e7514a"}
{"audio_filepath": "cv-corpus-9.0-2022-04-27/rw/clips/common_voice_rw_26273488.mp3", "text": "Grand Canyon ni ahantu hazwi cyane ba mukerarugendo.", "up_votes": 2, "down_votes": 0, "age": "twenties", "gender": "male", "accents": NaN, "client_id": "e2a04c0ecacf81302f4270a3dddaa7a131420f6b7319208473af17d4adf3724ad9a3b6cdee107e2f321495db86f114a50c396e0928464a58dfad472130e7514a"}
```

--------------------------------

### Install TransformerEngine with TRT-ONNX backend

Source: https://docs.nvidia.com/nemo/export-deploy/latest/index.html

Install required dependencies and the package with TRT-ONNX support.

```bash
pip install torch==2.7.0 setuptools pybind11 wheel_stub # Required for TE
pip install --no-build-isolation .[trt-onnx]
```

--------------------------------

### Example Path List Files

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/speaker_diarization/datasets.html

Sample content for audio and RTTM path list files used by the manifest generation script.

```text
/path/to/abcd01.wav
/path/to/abcd02.wav
```

```text
/path/to/abcd01.rttm
/path/to/abcd02.rttm
```

--------------------------------

### Create a TextExample object

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/datasets.html

Python instantiation of a TextExample object for multimodal data.

```python
from nemo.collections.common.data.lhotse.text_adapters import TextExample

example = TextExample(
    text="This is a language modeling example.",
    language="en",  # optional
)
```

--------------------------------

### CTM File Example

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/speaker_diarization/datasets.html

Example lines for a CTM file used for word-level diarization evaluation.

```text
TS3012d.Mix-Headset 1 12.879 0.32 okay NA lex MTD046ID
TS3012d.Mix-Headset 1 13.203 0.24 yeah NA lex MTD046ID
```

--------------------------------

### Start NeMo Framework container

Source: https://docs.nvidia.com/nemo/megatron-bridge/latest

Launch the official NeMo Framework container with GPU support and working directory mapping.

```bash
docker run --rm -it -w /workdir -v $(pwd):/workdir \
 --entrypoint bash \
 --gpus all \
 nvcr.io/nvidia/nemo:${TAG}
```

--------------------------------

### Install NeMo Export-Deploy via pip

Source: https://docs.nvidia.com/nemo/export-deploy/latest/index.html

Installs the base package for exploration without additional heavy dependencies.

```bash
pip install nemo-export-deploy
```

--------------------------------

### Install Megatron Bridge via pip

Source: https://docs.nvidia.com/nemo/megatron-bridge/latest

Install required build dependencies followed by the Megatron Bridge package.

```bash
pip install torch setuptools pybind11 wheel_stub # Required for TE
pip install --no-build-isolation megatron-bridge
```

--------------------------------

### Example MSDD Manifest File Format

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/speaker_diarization/datasets.html

Sample lines from a generated manifest file showing the structure of audio filepaths, offsets, durations, and associated RTTM paths.

```json
{"audio_filepath": "/path/to/abcd01.wav", "offset": 0.007, "duration": 14.046, "label": "infer", "text": "-", "num_speakers": 2, "rttm_filepath": "simulated_train/abcd01.1919_1988.rttm"}
{"audio_filepath": "/path/to/abcd01.wav", "offset": 13.553, "duration": 16.429, "label": "infer", "text": "-", "num_speakers": 2, "rttm_filepath": "simulated_train/abcd01.1919_1988.rttm"}
{"audio_filepath": "/path/to/abcd02.wav", "offset": 0.246, "duration": 15.732, "label": "infer", "text": "-", "num_speakers": 2, "rttm_filepath": "path/to/rttm_output_folder/abcd02.777_5694.rttm"}
{"audio_filepath": "/path/to/abcd02.wav", "offset": 15.478, "duration": 14.47, "label": "infer", "text": "-", "num_speakers": 2, "rttm_filepath": "path/to/rttm_output_folder/abcd02.777_5694.rttm"}
```

--------------------------------

### Raw text file format examples

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/datasets.html

Example content for raw text files used in language modeling.

```text
# file: document_0.txt
This is a language modeling example.
Wall Street is expecting major news tomorrow.

# file: document_1.txt
Invisible bats have stormed the city.
What an incredible event!
```

--------------------------------

### Launch Speech Data Explorer

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/examples/kinyarwanda_asr.html

Start the local server for the Speech Data Explorer tool to visualize and analyze model predictions.

```bash
python ${NEMO_ROOT}/tools/speech_data_explorer/data_explorer.py <your manifest file>
```

--------------------------------

### Inference Output Manifest Example

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/examples/kinyarwanda_asr.html

Example of the JSON manifest output containing the original text and the predicted transcript.

```json
{"audio_filepath": "test/common_voice_rw_19835615.wav", "text": "kw'ibumoso", "up_votes": 2, "down_votes": 0, "age": NaN, "gender": NaN, "accents": NaN, "client_id": "66675a7003e6baa3e7d4af01bff8324ac3c5f15e7f8918180799dd2928227c791f19e2811f9ec5779a2b06dac1b7a97fa7740dcfe98646ea1b5e106250c260be", "duration": 3.672, "pred_text": "n'ibumoso"}
{"audio_filepath": "test/common_voice_rw_24795878.wav", "text": "ni ryari uheruka kurya urusenda", "up_votes": 2, "down_votes": 0, "age": NaN, "gender": NaN, "accents": NaN, "client_id": "90e0438947a75b6c0cf59a0444aee3b81a76c5f9459c4b22df2e14b4ce257aeacaed8ac6092bfcd75b8e831633d58a84287fd62190c21d70d75efe8d93ed74ed", "duration": 3.312, "pred_text": "ni ryari uheruka kurya urusenda"}
{"audio_filepath": "test/common_voice_rw_24256935.wav", "text": "umunani", "up_votes": 2, "down_votes": 0, "age": NaN, "gender": NaN, "accents": NaN, "client_id": "974d4876e99e7437183c20f9107053acc9e514379d448bcf00aaaabc0927f5380128af86d39650867fa80a82525110dfc40784a5371c989de1a5bdf531f6d943", "duration": 3.24, "pred_text": "umunani"}
```

--------------------------------

### Execute Distillation with NeMo-Run

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/model-optimization/distillation/distillation.html

Utilize the distillation_recipe to configure and run a distillation job. Requires pre-configured student and teacher model checkpoints.

```python
import nemo_run as run
from nemo.collections import llm
from nemo.collections.llm.modelopt.recipes import distillation_recipe

recipe = distillation_recipe(
    student_model_path="path/to/student/nemo2-checkpoint/",
    teacher_model_path="path/to/teacher/nemo2-checkpoint/",
    distillation_config_path="path/to/distill-config.yaml",
    dir="./distill_logs",  # Path to store logs and checkpoints
    name="distill_testrun",
    num_nodes=1,
    num_gpus_per_node=8,
)

# Override the configuration with desired components:
recipe.data = run.Config(llm.PreTrainingDataModule, ...)
recipe.trainer.strategy.tensor_model_parallel_size = 8
...

run.run(recipe)
```

--------------------------------

### RTTM File Path List Example

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/speaker_diarization/datasets.html

Example content for the RTTM file path list text file.

```text
/path/to/abcd01.rttm
/path/to/abcd02.rttm
```

--------------------------------

### Verify CUDA and cuDNN installation

Source: https://docs.nvidia.com/nemo/megatron-bridge/latest

Check for installed CUDA toolkit and cuDNN packages on a Linux system using dpkg.

```bash
dpkg -l | grep 'cuda-toolkit'
dpkg -l | grep 'cudnn.*cuda'
```

--------------------------------

### Initialize and use NeMo Guardrails with Python SDK

Source: https://docs.nvidia.com/nemo/guardrails/latest

Demonstrates loading a configuration and generating a response using the LLMRails class.

```python
from nemoguardrails import LLMRails, RailsConfig

config = RailsConfig.from_path("./config")
rails = LLMRails(config)

response = rails.generate(
    messages=[{"role": "user", "content": "Hello!"}]
)
```

--------------------------------

### Launch Training Script

Source: https://docs.nvidia.com/nemo/megatron-bridge/latest

Execute the training script across multiple devices using torchrun.

```bash
torchrun --nproc-per-node=<num devices> /path/to/script.py
```

--------------------------------

### Launch Llama 3 8b Pretraining with NeMo-Run Partial

Source: https://docs.nvidia.com/nemo/megatron-bridge/latest

Execute a pretraining script using NeMo-Run's Partial interface.

```bash
uv run python pretrain_llama3_8b_nemo_run_partial.py \
 --nproc-per-node=2
```

--------------------------------

### RTTM File Example

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/speaker_diarization/datasets.html

A single line example of a Rich Transcription Time Marked (RTTM) file used for ground truth evaluation.

```text
SPEAKER TS3012d.Mix-Headset 1 331.573 0.671 <NA> <NA> MTD046ID <NA> <NA>
```

--------------------------------

### RTTM File Line Example

Source: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/speaker_diarization/datasets.html

A single line example of a Rich Transcription Time Marked (RTTM) file used for ground truth labels.

```text
SPEAKER TS3012d.Mix-Headset 1 32.679 0.671 <NA> <NA> MTD046ID <NA> <NA>
```
Tokens:
Raw