NVIDIA Riva

https://docs.nvidia.com/deeplearning/riva/user-guide/docs
Admin
NVIDIA Riva is a GPU-accelerated SDK for building production-grade Speech AI applications with...
Tokens:279,271
Snippets:2,995
Trust Score:10
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install Riva Client Library

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/samples/sample-apps/virtual-assistant/README.html

Install the Riva client library from the downloaded Quick Start package. Navigate to the extracted directory first.

```bash
cd riva_quickstart_v<x.y.z>
pip install riva_api-<x.y.z>-py3-none-any.whl
```

--------------------------------

### Install NeMo and nemo2riva

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/tts-finetune-nemo.html

Installs the NeMo toolkit and the nemo2riva conversion tool. This includes downloading a specific version of the NeMo quickstart resource and installing a wheel file for nemo2riva. It also installs protobuf and a custom script for pynini.

```bash
!pip install nvidia-pyindex
!pip install nemo_toolkit['all']==1.23.0
!ngc registry resource download-version "nvidia/riva/riva_quickstart:2.8.1"
!pip install "riva_quickstart_v2.8.1/nemo2riva-2.8.1-py3-none-any.whl"
!pip install protobuf==3.20.0
# Installing pynini separately
!wget https://raw.githubusercontent.com/NVIDIA/NeMo/main/nemo_text_processing/install_pynini.sh \
bash install_pynini.sh
```

--------------------------------

### KenLM Setup and Build

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-customizing.html

Steps to set up and build the KenLM language modeling toolkit. Ensure Boost and zlib are installed before proceeding.

```bash
wget -O - https://kheafield.com/code/kenlm.tar.gz | tar xz
mkdir kenlm/build
cd kenlm/build
cmake ..
make -j2
```

--------------------------------

### Install NeMo Toolkit and Protobuf

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/tts-evaluate.html

Installs the NeMo toolkit with all dependencies and upgrades protobuf to a specific version required for the ASR examples.

```bash
!pip install nemo_toolkit['all']
!pip install --upgrade protobuf==3.20.0
```

--------------------------------

### Install Apex

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/nmt-python-advanced-finetune-nmt-model-with-nemo.html

Installs the Apex library, which may be required for certain optimizations. This involves cloning the repository, checking out a specific branch, and installing with custom options.

```bash
!git clone https://github.com/ericharper/apex.git
!cd apex
!git checkout nm_v1.15.0
!pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" --global-option="--fast_layer_norm" --global-option="--distributed_adam" --global-option="--deprecated_fused_adam" ./ 
```

--------------------------------

### Riva Server Startup Output

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/model-overview.html

Example output indicating the Riva server is starting and loading models. This output is typically seen when starting the `riva-server` container.

```text
Starting Riva Speech Services
> Waiting for Riva server to load all models...retrying in 10 seconds
> Waiting for Riva server to load all models...retrying in 10 seconds
> Waiting for Riva server to load all models...retrying in 10 seconds
> Riva server is ready …
```

--------------------------------

### Install NeMo and nemo2riva Packages

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/asr-finetune-parakeet-nemo.html

Installs necessary Python packages for NeMo and the `nemo2riva` converter. It also downloads the Riva quickstart resource. Ensure NGC CLI is installed and configured.

```bash
!pip install nvidia-pyindex
!ngc registry resource download-version "nvidia/riva/riva_quickstart:"$__riva_version__
!pip install nemo2riva
!pip install protobuf==3.20.0

```

--------------------------------

### Example Output of Riva Initialization

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/installation/deploy-local.html

This output indicates that the Riva initialization script has successfully logged into the NGC registry, pulled the required Docker images, and completed the setup. It suggests the next step to launch the services.

```text
Logging into NGC Docker registry if necessary...
Pulling required Docker images if necessary...
 > Pulling Riva Speech Server images.
 > Pulling nvcr.io/nvidia/riva/riva-speech:2.24.0. This may take some time...
Riva initialization complete. Run bash riva_start.sh to launch services.
```

--------------------------------

### Example: Build and Deploy Translation Model

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/translation/translation-customization.html

This example demonstrates building a specific translation model and then deploying it to a specified directory.

```bash
riva-build translation \
--name mnmt_en_deesfr_transformer12x2 \
/data/mnmt_en_deesfr_transformer12x2nmt.rmir \
/data/mnmt_en_deesfr_transformer12x2.riva
```

```bash
riva-deploy  -f /data/mnmt_en_deesfr_transformer12x2nmt.rmir  /data/models
```

--------------------------------

### Navigate to Riva Quick Start Directory

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/quick-start-guide/asr.html

Change the directory to the downloaded Riva Quick Start scripts.

```bash
cd riva_quickstart_arm64_v2.24.0
```

--------------------------------

### Initialize and Start Riva Server

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/quick-start-guide/asr.html

Initialize the Riva server, which downloads necessary Docker images and models, and then start the server.

```bash
bash riva_init.sh
bash riva_start.sh
```

--------------------------------

### Install Riva Client API

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/asr-deploy-citrinet.html

Install the necessary Python client library for NVIDIA Riva.

```bash
! pip install nvidia-riva-client
```

--------------------------------

### Download Riva Quick Start Resources

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/nmt-python-advanced-deploy-nemo-nmt-model-on-riva.html

Downloads the Riva Quick Start resources from NGC, which are necessary for setting up and running the Riva server.

```bash
!ngc registry resource download-version "nvidia/riva/riva_quickstart:$__riva_version__"
```

--------------------------------

### Install Required Packages

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/tts-phoneme-distribution.html

Installs Bokeh, CMUDict, PrettyTable, NeMo Toolkit, and a specific version of Protobuf.

```bash
!pip install bokeh cmudict prettytable nemo_toolkit['all']
!pip install protobuf==3.20.0
```

--------------------------------

### Install KenLM and Text Library

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/asr-python-advanced-nemo-ngram-training-and-finetuning.html

Installs KenLM from source and the flashlight text library. This requires build tools and CMake.

```bash
!apt install libeigen3-dev
!git clone https://github.com/kpu/kenlm.git
!cd kenlm && mkdir build && cd build && cmake .. && make -j

!pip3 install git+https://github.com/kpu/kenlm.git
!pip3 install git+https://github.com/flashlight/text.git
```

--------------------------------

### Start Riva Server

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/asr-deploy-conformer-ctc.html

Execute the Riva start script to deploy your configured models. This command initiates the Riva server process.

```shell
# Run Riva Start. This will deploy your model(s).
! cd $RIVA_DIR && ./riva_start.sh config.sh
```

--------------------------------

### Start Chatbot Client Server

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/samples/sample-apps/virtual-assistant-rasa/README.html

Command to start the Python-based chatbot client server.

```bash
	python3 main.py
```

--------------------------------

### Install Dependencies

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/asr-noise-augmentation.html

Installs necessary Python packages and system libraries for audio processing and data handling.

```bash
!pip install wget
!apt-get install sox libsndfile1 ffmpeg libsox-fmt-mp3
!pip install text-unidecode
!pip install matplotlib>=3.3.2
!pip install Cython
```

--------------------------------

### Install Dependencies

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/asr-noise-augmentation-offline.html

Installs necessary packages like wget and the SoX audio processing tool. This is a prerequisite for downloading and manipulating audio files.

```bash
# If you're using Google Colab and not running locally, run this cell.

## Install dependencies
!pip install wget
!apt-get install sox
```

--------------------------------

### Example Output of Riva Server Startup

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/installation/deploy-local.html

This output shows the progress of the Riva server startup, including waiting for the Triton server to become ready. It confirms that the services are being initialized and are preparing to accept requests.

```text
Starting Riva Speech Services
 > Waiting for Triton server to load all models...retrying in 10 seconds
 > Waiting for Triton server to load all models...retrying in 10 seconds
 > Waiting for Triton server to load all models...retrying in 10 seconds
 > Triton server is ready…
```

--------------------------------

### Watch Server Logs

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/troubleshooting.html

Monitor the logs of the Riva server during installation. The server will not start until the init container finishes.

```bash
kubectl logs $POD --follow
```

--------------------------------

### Create Directories for Data and Results

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/asr-python-advanced-nemo-ngram-training-and-finetuning.html

Sets up the necessary directories for downloading datasets and storing training results. Ensure these utilities are available in your environment.

```bash
# Set the path to a folder where you want your data and results to be saved.
DATA_DOWNLOAD_DIR="content/datasets"
RESULTS_DIR="content/results"
MODELS_DIR="content/models"

!mkdir -p $DATA_DOWNLOAD_DIR $RESULTS_DIR $MODELS_DIR
```

--------------------------------

### Configure Riva ASR Pipeline (Offline, Italian)

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-pipeline-configuration.html?highlight=pipeline+configuration

Example configuration for an offline ASR pipeline for Italian language. This snippet shows basic offline setup parameters.

```bash
riva-build speech_recognition \
  <rmir_filename>:<key> \
  <riva_file>:<key> \
  --offline \
  --name=conformer-it-IT-asr-offline \
  --return_separate_utterances=True \
  --featurizer.use_utterance_norm_params=False \
  --featurizer.precalc_norm_time_steps=0 \
  --featurizer.precalc_norm_params=False \
  --ms_per_timestep=40 \
  --endpointing.start_history=200 \

```

--------------------------------

### Streaming ASR Configuration with Greedy Decoder

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-pipeline-configuration.html?highlight=pipeline+configuration

Configure a streaming ASR pipeline using the greedy decoder. This example shows basic streaming setup without specific decoder configurations.

```bash
riva-build speech_recognition \
  <rmir_filename>:<key> \
  <riva_file>:<key> \
  --name=conformer-unified-de-DE-asr-streaming \
  --return_separate_utterances=False \
  --featurizer.use_utterance_norm_params=False \
  --featurizer.precalc_norm_time_steps=0 \
  --featurizer.precalc_norm_params=False \
  --ms_per_timestep=40 \
  --endpointing.start_history=200 \
  --nn.fp16_needs_obey_precision_pass \
  --endpointing.residue_blanks_at_start=-2 \
  --unified_acoustic_model \
  --chunk_size=0.16 \
  --left_padding_size=1.92 \
  --right_padding_size=1.92 \
  --decoder_type=greedy
```

--------------------------------

### Setup Noise Sample Directory

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/asr-noise-augmentation-offline.html

Creates a directory to store downloaded noise samples if it doesn't already exist. This ensures organized storage of augmentation data.

```python
import os
# This is where the noise samples will be placed.
noise_samples = 'noise_samples'
if not os.path.exists(noise_samples):
  os.makedirs(noise_samples)
```

--------------------------------

### Start Riva Speech Services

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/installation/deploy-local.html

This command launches the Riva Speech AI API server and waits for the Triton server to load all models. It is the primary command to get Riva services running after initialization.

```bash
bash riva_start.sh
```

--------------------------------

### Export NMT Model to Riva Format

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/nmt-python-advanced-finetune%20multilingual-nmt-model-with-nemo.html

Converts a fine-tuned NeMo NMT model (`.nemo`) into a format (`.riva`) deployable with NVIDIA Riva. This tool is typically available via the Riva Quick Start Guide.

```bash
!nemo2riva --out $model_dir/megatronnmt_custom_any_en_500m.riva <saved_custom_nemo_model_path>
```

--------------------------------

### Get riva-build Help

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/model-overview.html

Displays help information for the riva-build command, detailing available pipelines and optional parameters for customization.

```bash
riva-build <pipeline> -h
```

--------------------------------

### Install NeMo Toolkit

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/asr-python-advanced-nemo-ngram-training-and-finetuning.html

Installs the NeMo toolkit from a specific branch of its GitHub repository. Ensure you have git and Python installed.

```bash
BRANCH = 'main'
!git clone https://github.com/NVIDIA/NeMo.git
!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[all]
```

--------------------------------

### Install Apex

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/nmt-python-advanced-synthetic-data-generation.html

Clones the Apex repository, checks out a specific commit, and installs it with CUDA and C++ extensions enabled. Apex is used for mixed-precision training and other optimizations.

```shell
!git clone https://github.com/NVIDIA/apex.git
%cd apex
!git checkout a32d7a6dddcf4e39d241b0d139c222a97c91887d
!pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" --global-option="--fast_layer_norm" --global-option="--distributed_adam" --global-option="--deprecated_fused_adam" ./
%cd ..
```

--------------------------------

### Initialize Riva Server and Deploy Models

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/tts-deploy.html

Fetches containers and models, then runs the riva-deploy command. This step can be skipped if Riva deploy has already been performed.

```bash
! cd {quickstart_dir} && ./riva_init.sh config.sh
```

--------------------------------

### Install NeMo and Dependencies

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/asr-finetune-conformer-ctc-nemo.html

Installs necessary Python packages and the NeMo toolkit from a specific branch. Restart the runtime after installation for changes to take effect.

```python
!pip install wget
!apt-get install sox libsndfile1 ffmpeg libsox-fmt-mp3
!pip install text-unidecode
!pip install matplotlib>=3.3.2
!pip install Cython

## Install NeMo
BRANCH = 'v1.23.0'
!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[all]
```

--------------------------------

### Riva Server Startup Log

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/model-overview.html?highlight=nemo2riva

This is an example log output indicating that the Riva server is ready and listening for connections.

```text
Starting Riva Speech Services
> Waiting for Riva server to load all models...retrying in 10 seconds
> Waiting for Riva server to load all models...retrying in 10 seconds
> Waiting for Riva server to load all models...retrying in 10 seconds
> Riva server is ready…

```

--------------------------------

### Get riva-build Help

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/model-overview.html?highlight=nemo2riva

Displays help information for the riva-build command, detailing optional parameters for pipeline customization.

```bash
riva-build <pipeline> -h

```

--------------------------------

### Install Dependencies for NeMo

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/asr-finetune-parakeet-nemo.html

Installs necessary Python packages and system dependencies for NeMo and speech processing. Remember to restart the runtime after installation for changes to take effect.

```python
!pip install wget
!apt-get install sox libsndfile1 ffmpeg libsox-fmt-mp3 jq
!pip install text-unidecode
!pip install matplotlib>=3.3.2
!pip install Cython
!pip3 install --no-cache-dir huggingface-hub==0.23.2

## Install NeMo
BRANCH = 'v1.23.0'
!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[all]
```

--------------------------------

### Install NeMo and nemo2riva Packages

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/asr-finetune-conformer-ctc-nemo.html

Install necessary packages for NeMo ASR model fine-tuning and conversion to Riva format. Ensure NGC CLI is installed for downloading Riva resources.

```python
from version import __riva_version__
print(__riva_version__)
!pip install nvidia-pyindex
!ngc registry resource download-version "nvidia/riva/riva_quickstart:"$__riva_version__
!pip install nemo2riva
!pip install protobuf==3.20.0
```

--------------------------------

### Install NLTK

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/nmt-python-basics.html

Installs the Natural Language Toolkit (NLTK) library for Python.

```bash
!python -m pip install nltk
```

--------------------------------

### Navigate to Virtual Assistant Directory

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/samples/sample-apps/virtual-assistant/README.html

Change your current directory to the virtual assistant sample application folder.

```bash
cd sample-apps/virtual-assistant
```

--------------------------------

### Clone NeMo Repository and Install Dependencies

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/nmt-python-advanced-synthetic-data-generation.html

Clones the NeMo repository from GitHub, installs system dependencies, and then installs NeMo and its Python dependencies. This ensures all necessary libraries and code are available for the tutorial.

```shell
NeMoBranch = "v1.17.0_pt_23.04"
!git clone -b $NeMoBranch https://github.com/NVIDIA/NeMo $base_dir/NeMo
!apt-get update && apt-get install -y libsndfile1 ffmpeg
!pip3 install "cython<3.0.0" wheel && pip3 install pyyaml==5.4.1 --no-build-isolation
%cd $base_dir/NeMo
!./reinstall.sh
!pip install torchmetrics==0.11.4
%cd ..
```

--------------------------------

### Install Node.js Dependencies

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/samples/sample-apps/riva-contact/README.html

Install all the necessary Node.js packages for the Riva Contact application.

```bash
npm install
```

--------------------------------

### Install Kubectl Plugin

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/deploy-gke.html

Installs the kubectl plugin for gcloud, which is necessary for interacting with the GKE cluster.

```bash
gcloud components install kubectl
```

--------------------------------

### Compile and Run Golang Riva Client

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/development-golang.html

This command sequence shows how to prepare and execute the Go client. It first ensures dependencies are managed with `go mod tidy`, then builds the executable, and finally runs it with specified server URI, query, and context.

```bash
go mod tidy
go build qa.go
./qa -server <riva_uri> \
        -query "Which founder of NVIDIA previously worked at Sun?" \
        -context "Nvidia was founded on April 5, 1993, by Jensen Huang (CEO as of
                  2020), a Taiwanese American, previously director of CoreWare at LSI
                  Logic and a microprocessor designer at Advanced Micro Devices (AMD),
                  Chris Malachowsky, an electrical engineer who worked at Sun
                  Microsystems, and Curtis Priem, previously a senior staff engineer
                  and graphics chip designer at Sun Microsystems."

2021/02/04 13:17:51 Question: Which founder of NVIDIA previously worked at Sun?
2021/02/04 13:17:51 Answer:   Chris Malachowsky,
```

--------------------------------

### Download Riva Quick Start Scripts

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/quick-start-guide/asr.html

Use the NGC CLI tool to download the Riva Quick Start scripts for ARM64 platforms.

```bash
ngc registry resource download-version nvidia/riva/riva_quickstart_arm64:2.24.0
```

--------------------------------

### Install Golang gRPC Protoc Plugins

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/development-golang.html

Install the necessary protoc plugins for Golang, including the protobuf compiler and gRPC compiler, to generate language-specific bindings.

```bash
export GO111MODULE=on
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
```

--------------------------------

### Example Manifest Row

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/tts-finetune-nemo.html

Provides a concrete example of a single row in a dataset manifest file.

```json
{"audio_filepath": "actressinhighlife_01_bowen_0001.flac", "text": "the pleasant season did my heart employ", "duration": 2.4}
```

--------------------------------

### Initialize Golang Project Module

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/development-golang.html

Create a new directory for your Golang project and initialize a Go module to manage dependencies.

```bash
mkdir riva-qa-golang-client
cd riva-qa-golang-client
go mod init riva-qa
```

--------------------------------

### Start Riva Server with Model

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tutorials/asr-deploy-citrinet.html

Execute the riva_start.sh script to deploy your configured model. This command initiates the Riva server, making your ASR model available for inference.

```shell
# Run Riva Start. This will deploy your model.
! cd $RIVA_DIR && ./riva_start.sh config.sh
```

--------------------------------

### Install virtualenv

Source: https://docs.nvidia.com/deeplearning/riva/user-guide/docs/samples/sample-apps/virtual-assistant/README.html

Installs the virtualenv tool, which is a prerequisite for creating Python virtual environments for the sample application.

```bash
pip install virtualenv
```
Tokens:
Raw