NVIDIA Speech NIM Microservices

https://docs.nvidia.com/nim/speech/latest/
Admin
GPU-accelerated Docker containers that provide speech AI capabilities for ASR, TTS, and machine...
Tokens:129,024
Snippets:1,496
Trust Score:10
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install NIM Helm Chart

Source: https://docs.nvidia.com/nim/speech/latest/deployment/helm.html

Launch the NIM service by installing the Helm chart using the `helm install` command. Specify the chart name, version, and your custom values file.

```bash
helm install riva-nim riva-nim-<version_number>.tgz -f path/to/your/custom-values.yaml
```

--------------------------------

### Clone Repository and Install Dependencies

Source: https://docs.nvidia.com/nim/speech/latest/reference/api-references/asr/realtime-asr.html

Clone the Python clients repository and install the required packages. Then, run the help command for the real-time ASR client to see available options.

```bash
git clone https://github.com/nvidia-riva/python-clients.git
pip install -r requirements.txt
python scripts/asr/realtime_asr_client.py --help
```

--------------------------------

### Basic Translation Example

Source: https://docs.nvidia.com/nim/speech/latest/nmt/translate-with-python.html

A simple example demonstrating how to perform text translation using the `riva.client.NeuralMachineTranslationClient` and includes basic gRPC error handling.

```APIDOC
## Basic Translation with Error Handling

### Description

This example shows how to initialize the NMT client, perform a translation, and catch potential gRPC errors.

### Code

```python
import grpc
import riva.client

auth = riva.client.Auth(uri="localhost:50051")
nmt_client = riva.client.NeuralMachineTranslationClient(auth)

try:
    response = nmt_client.translate(
        texts=["Hello, world!"],
        model="",
        source_language="en-US",
        target_language="de-DE",
    )
    print(response.translations[0].text)
except grpc.RpcError as e:
    print(f"gRPC error: {e.code()} - {e.details()}")
```

### Error Handling

Common gRPC status codes and their meanings are provided for debugging:

- `UNAVAILABLE`: The NMT NIM is not running or not yet ready. Check the health endpoint.
- `INVALID_ARGUMENT`: Invalid language code or empty text. Refer to supported languages.
- `ALREADY_EXISTS`: Duplicate request.
- `RESOURCE_EXHAUSTED`: Batch too large or GPU memory exhausted. Reduce batch size.
```

--------------------------------

### Realtime API WebSocket Connection URL Example

Source: https://docs.nvidia.com/nim/speech/latest/reference/api-references/asr/realtime-asr.html

An example of a complete URL for connecting to the Realtime API via WebSocket. The 'intent' parameter is required.

```bash
ws://localhost:9000/v1/realtime?intent=transcription
```

--------------------------------

### Clone Repository and Install Dependencies

Source: https://docs.nvidia.com/nim/speech/latest/reference/api-references/tts/realtime-tts.html

Clone the Python clients repository and install the required packages. Then, run the help command for the real-time TTS client script to see available options.

```bash
git clone https://github.com/nvidia-riva/python-clients.git
pip install -r requirements.txt
python scripts/tts/realtime_tts_client.py --help

```

--------------------------------

### Install NVIDIA Riva Python Client Package

Source: https://docs.nvidia.com/nim/speech/latest/get-started/riva-client.html

Install the Python package for the Riva client. This package provides the `riva.client` library for gRPC communication with NIM.

```bash
pip install -U nvidia-riva-client
```

--------------------------------

### Copy Audio File from Container

Source: https://docs.nvidia.com/nim/speech/latest/asr/customization/customization.html

Copy an example audio file from the NIM container to the host machine for use in ASR transcription examples.

```bash
docker cp $CONTAINER_ID:/opt/riva/wav/en-US_wordboosting_sample.wav .

```

--------------------------------

### Example Realtime WebSocket Connection URL

Source: https://docs.nvidia.com/nim/speech/latest/reference/api-references/tts/realtime-tts.html

This is a complete example of a connection URL for the Realtime API's WebSocket server. Ensure the `intent` query parameter is set to `synthesize` for text-to-speech functionality.

```url
ws://localhost:9000/v1/realtime?intent=synthesize
```

--------------------------------

### Get Riva Build Configuration Help

Source: https://docs.nvidia.com/nim/speech/latest/asr/customization/pipeline-configuration.html

Run this command to display help information for the `riva-build` command, specifically for customizing ASR pipeline configurations. It shows available parameters and their usage.

```bash
riva-build --config-path=pkg://servicemaker.configs.asr -h
```

--------------------------------

### List Voices Response Example

Source: https://docs.nvidia.com/nim/speech/latest/reference/api-references/tts/realtime-tts.html

Example response from the /v1/audio/list_voices endpoint, showing available voices categorized by language. The specific voices available depend on your Riva deployment configuration.

```json
{
  "en-US": {
    "voices": [
      "English-US.Male-1",
      "English-US.Female-1",
      "English-US.Male-2"
    ]
  },
  "es-US": {
    "voices": [
      "Magpie-Multilingual.ES-US.Diego",
      "Magpie-Multilingual.ES-US.Isabela"
    ]
  }
}
```

--------------------------------

### Offline ASR Pipeline Configuration

Source: https://docs.nvidia.com/nim/speech/latest/asr/customization/pipeline-configuration.html

Configure an offline ASR pipeline with specified parameters. This example sets up a multilingual pipeline.

```bash
ms_per_timestep=80 \
   nn.fp16_needs_obey_precision_pass=True \
   chunk_size=16 \
   left_padding_size=0.0 \
   right_padding_size=0.0 \
   featurizer.max_batch_size=256 \
   featurizer.max_execution_batch_size=256 \
   featurizer.right_pad_features=true \
   max_batch_size=64 \
   nn.opt_batch_size=64 \
   unified_acoustic_model=true \
   endpointing_type=niva \
   endpointing.stop_history=0 \
   nemo_decoder.use_stateful_decoding=False \
   decoder=nemo \
   language_code=en-US \
   wfst_tokenizer_model=<far_tokenizer_file> \
   wfst_verbalizer_model=<far_verbalizer_file> \
   speech_hints_model=<far_speech_hints_file>
```

--------------------------------

### Synthesize Speech with Realtime API (Websocket)

Source: https://docs.nvidia.com/nim/speech/latest/get-started/tutorials/tts.html

This example demonstrates synthesizing speech using the Realtime Websocket API. It's designed for applications needing the lowest possible latency. Ensure the correct server address and port are specified.

```python
python3 python-clients/scripts/tts/realtime_tts_client.py \
   --server 0.0.0.0:9000 \
   --language-code en-US \
   --text "Experience the future of speech AI with Riva, where every word comes to life with clarity and emotion." \
   --zero_shot_audio_prompt_file python-clients/data/examples/sample_audio_prompt.wav \
   --output output.wav

```

--------------------------------

### Deploy ASR NIM Microservice with Docker

Source: https://docs.nvidia.com/nim/speech/latest/get-started/tutorials/asr.html

Exports environment variables for the container ID and model profile selector, then runs the Docker container. This command downloads the specified model from NGC, optimizes it for your GPU, and starts serving requests on HTTP (port 9000) and gRPC (port 50051). The first run may take time for model download and engine building. Use `mode=str` or `mode=offline` for reduced memory usage.

```bash
export CONTAINER_ID=parakeet-1-1b-ctc-en-us
export NIM_TAGS_SELECTOR="name=parakeet-1-1b-ctc-en-us,mode=all"

docker run -it --rm --name=$CONTAINER_ID \
  --runtime=nvidia \
  --gpus '"device=0"' \
  --shm-size=8GB \
  -e NGC_API_KEY \
  -e NIM_HTTP_API_PORT=9000 \
  -e NIM_GRPC_API_PORT=50051 \
  -p 9000:9000 \
  -p 50051:50051 \
  -e NIM_TAGS_SELECTOR \
  nvcr.io/nim/nvidia/$CONTAINER_ID:latest

```

--------------------------------

### Synthesize speech with different accent using gRPC

Source: https://docs.nvidia.com/nim/speech/latest/get-started/tutorials/tts.html

Demonstrates synthesizing speech with a different accent by intermixing language codes and voice configurations. This example synthesizes English speech with a French accent.

```bash
python3 python-clients/scripts/tts/talk.py --server 0.0.0.0:50051 \
   --language-code en-US \
   --text "Experience the future of speech AI with Riva, where every word comes to life with clarity and emotion." \
   --voice Magpie-Multilingual.FR-FR.Pascal \
   --output output.wav

```

--------------------------------

### Helm Deployment Configuration

Source: https://docs.nvidia.com/nim/speech/latest/nmt/deploy-nmt-model.html

Example `custom-values.yaml` file for deploying the NMT NIM microservice using Helm on Kubernetes. This specifies image details, pull policy, NGC API secret, and environment variables.

```yaml
image:
  repository: nvcr.io/nim/nvidia/riva-translate-1_6b
  pullPolicy: IfNotPresent
  tag: latest
nim:
  ngcAPISecret: ngc-api
imagePullSecrets:
  - name: ngc-secret
envVars:
  NIM_TAGS_SELECTOR: name=riva-translate-1_6b

```

--------------------------------

### Check NIM Pod Status

Source: https://docs.nvidia.com/nim/speech/latest/deployment/helm.html

Verify that the NIM service pod is running and ready after installation. Use `kubectl get pod` to monitor the status.

```bash
kubectl get pod
```

--------------------------------

### List Sample Audio Files in Container

Source: https://docs.nvidia.com/nim/speech/latest/asr/deploy-asr-models/parakeet-ctc-es-us.html

Lists the sample audio files available within the deployed NIM container.

```bash
docker exec $CONTAINER_ID ls /opt/riva/wav/

```

--------------------------------

### List Sample Audio Files in Container

Source: https://docs.nvidia.com/nim/speech/latest/asr/deploy-asr-models/parakeet-ctc-en-us.html

Lists the sample audio files available within the deployed NIM container. This helps in identifying files for testing inference.

```bash
docker exec $CONTAINER_ID ls /opt/riva/wav/
```

--------------------------------

### Launch NIM Container for Riva Build

Source: https://docs.nvidia.com/nim/speech/latest/asr/customization/custom-deployment.html

Launches an interactive session inside the NIM Docker image to prepare for Riva model building. Ensure the artifact directory and export path are correctly mounted.

```bash
# Create the cache directory on the host machine:
 export NIM_EXPORT_PATH=~/nim_export
 mkdir -p $NIM_EXPORT_PATH
 chmod 777 $NIM_EXPORT_PATH
# Set the appropriate value
 export CONTAINER_ID=parakeet-1-1b-ctc-en-us
 docker run --gpus all -it --rm \
     -v <artifact_dir>:/riva_build_deploy \
     -v $NIM_EXPORT_PATH:/model_tar \
     --entrypoint="/bin/bash" \
     --name riva-build-deploy \
     nvcr.io/nim/nvidia/$CONTAINER_ID:latest

```

--------------------------------

### EndpointingConfig

Source: https://docs.nvidia.com/nim/speech/latest/reference/api-references/asr/protos.html

Configuration for utterance start and end detection.

```APIDOC
### EndpointingConfig

EndpointingConfig is used for configuring different fields related to start or end of utterance

#### Fields
- **start_history** (int32, optional) - `start_history` is the size of the window, in milliseconds, used to detect start of utterance. `start_threshold` is the percentage threshold used to detect start of utterance. (0.0 to 1.0) If `start_threshold` of `start_history` ms of the acoustic model output have non-blank tokens, start of utterance is detected.
- **start_threshold** (float, optional) - Percentage threshold for detecting the start of an utterance.
- **stop_history** (int32, optional) - `stop_history` is the size of the window, in milliseconds, used to detect end of utterance. `stop_threshold` is the percentage threshold used to detect end of utterance. (0.0 to 1.0) If `stop_threshold` of `stop_history` ms of the acoustic model output have blank tokens, end of utterance is detected and decoder will be reset.
- **stop_threshold** (float, optional) - Percentage threshold for detecting the end of an utterance.
- **stop_history_eou** (int32, optional) - `stop_history_eou` and `stop_threshold_eou` are used for 2-pass end of utterance. `stop_history_eou` is the size of the window, in milliseconds, used to trigger 1st pass of end of utterance and generate a partial transcript with stability of 1. (stop_history_eou < stop_history) `stop_threshold_eou` is the percentage threshold used to trigger 1st pass of end of utterance. (0.0 to 1.0) If `stop_threshold_eou` of `stop_history_eou` ms of the acoustic model output have blank tokens, 1st pass of end of utterance is triggered.
- **stop_threshold_eou** (float, optional) - Percentage threshold for triggering the 1st pass of end of utterance detection.
```

--------------------------------

### Copy Sample Audio Files

Source: https://docs.nvidia.com/nim/speech/latest/asr/deploy-asr-models/parakeet-rnnt.html

Copy sample audio files from the deployed NIM container to the local filesystem for inference testing.

```bash
docker cp $CONTAINER_ID:/opt/riva/wav/en-US_sample.wav .
docker cp $CONTAINER_ID:/opt/riva/wav/fr-FR_sample.wav .

```

--------------------------------

### Legal Document Translation Dictionary

Source: https://docs.nvidia.com/nim/speech/latest/nmt/custom-dictionaries.html

Example dictionary for legal terms, enforcing consistent terminology across documents.

```text
Force Majeure##Force Majeure
indemnification##Schadloshaltung
liability##Haftung
warranty##Gewährleistung

```

--------------------------------

### Copy Sample Audio File

Source: https://docs.nvidia.com/nim/speech/latest/asr/deploy-asr-models/nemotron-asr-streaming.html

Copies a sample audio file from the running NIM container to the local directory.

```bash
docker cp $CONTAINER_ID:/opt/riva/wav/en-US_sample.wav .

```

--------------------------------

### Medical Translation Dictionary

Source: https://docs.nvidia.com/nim/speech/latest/nmt/custom-dictionaries.html

Example dictionary for medical terms, protecting drug names and standardizing medical term translations.

```text
Aspirin##Aspirin
COVID-19##COVID-19
MRI##MRT
CT scan##CT-Untersuchung
blood pressure##Blutdruck

```

--------------------------------

### Copy Sample Audio File

Source: https://docs.nvidia.com/nim/speech/latest/asr/deploy-asr-models/conformer-ctc.html

Copies a sample audio file from the deployed NIM container to the local directory.

```bash
docker cp $CONTAINER_ID:/opt/riva/wav/es-US_sample.wav .

```

--------------------------------

### Product Catalog Translation Dictionary

Source: https://docs.nvidia.com/nim/speech/latest/nmt/custom-dictionaries.html

Example dictionary for product catalog terms, ensuring product names and SKUs remain unchanged.

```text
NIM-PRO-100
NIM-LITE-50
TensorRT
Triton

```

--------------------------------

### Create Export Directory for RMIR Models

Source: https://docs.nvidia.com/nim/speech/latest/deployment/docker/model-caching.html

Prepare a directory on the host to store RMIR models after they are built and exported by the container. This directory will be mounted into the container.

```bash
# Create the export directory on the host
export NIM_EXPORT_PATH=~/nim_export
mkdir -p $NIM_EXPORT_PATH
chmod 777 $NIM_EXPORT_PATH

```

--------------------------------

### Batch Translation with Python

Source: https://docs.nvidia.com/nim/speech/latest/nmt/translate-with-python.html

Pass multiple texts in a single request for higher throughput. This example demonstrates translating a list of predefined sentences.

```python
import riva.client

auth = riva.client.Auth(uri="localhost:50051")
nmt_client = riva.client.NeuralMachineTranslationClient(auth)

texts = [
    "Deploy containers on Kubernetes.",
    "Monitor GPU utilization with nvidia-smi.",
    "Scale inference with Triton Inference Server.",
    "Optimize models with TensorRT.",
]

response = nmt_client.translate(
    texts=texts,
    model="",
    source_language="en-US",
    target_language="fr",
)

for i, translation in enumerate(response.translations):
    print(f"[{i}] {translation.text}")

```

--------------------------------

### Copy Sample Audio File

Source: https://docs.nvidia.com/nim/speech/latest/asr/deploy-asr-models/parakeet-tdt.html

Copies a sample audio file from the NIM container to the host machine. This is a prerequisite for running inference.

```bash
docker cp $CONTAINER_ID:/opt/riva/wav/en-US_sample.wav .

```

```bash
docker cp $CONTAINER_ID:/opt/riva/wav/fr-FR_sample.wav .

```

--------------------------------

### Streaming ASR Pipeline Configuration (Mandarin)

Source: https://docs.nvidia.com/nim/speech/latest/asr/customization/pipeline-configuration.html

Configure a streaming ASR pipeline for Mandarin (CN) with low-latency requirements. This setup is optimized for real-time transcription.

```bash
riva-build --config-path=pkg://servicemaker.configs.asr --config-name=streaming \
   output_path=<rmir_filename>:<key> \
   'source_path=[<riva_file>:<key>]' \
   profane_words_file=<txt_profane_words_file> \
   name=parakeet-0.6b-unified-zh-CN-asr-streaming \
   featurizer.use_utterance_norm_params=False \
   featurizer.precalc_norm_time_steps=0 \
   featurizer.precalc_norm_params=False \
   ms_per_timestep=80 \
   endpointing.residue_blanks_at_start=-2 \

```

--------------------------------

### Stop a Container (with --rm flag)

Source: https://docs.nvidia.com/nim/speech/latest/get-started/tutorials/stopping-the-container.html

If the container was started with the `--rm` flag, only the `docker stop` command is necessary as Docker automatically removes the container upon stopping.

```bash
docker stop $CONTAINER_ID

```

--------------------------------

### List Available GPUs with `nvidia-smi -L`

Source: https://docs.nvidia.com/nim/speech/latest/deployment/docker/configuration.html

This command lists all available GPUs on the system, along with their device IDs and UUIDs, which are useful for configuring container GPU visibility.

```bash
nvidia-smi -L
```

--------------------------------

### Get WSL2 IP Address

Source: https://docs.nvidia.com/nim/speech/latest/get-started/tutorials/wsl.html

Retrieve the IP address of your WSL2 instance. This IP is necessary for configuring port forwarding when accessing the NIM from a different machine than the WSL2 host.

```bash
hostname -I

```

--------------------------------

### Configure Offline ASR Pipeline

Source: https://docs.nvidia.com/nim/speech/latest/asr/customization/pipeline-configuration.html

Configure an offline ASR pipeline. This setup is suitable for non-streaming audio processing and includes parameters for model configuration, language, and decoder settings.

```bash
riva-build --config-path=pkg://servicemaker.configs.asr --config-name=offline \
   output_path=<rmir_filename>:<key> \
   'source_path=[<riva_file>:<key>]' \
   profane_words_file=<txt_profane_words_file> \
   name=parakeet-0.6b-en-US-asr-offline \
   featurizer.use_utterance_norm_params=False \
   featurizer.precalc_norm_time_steps=0 \
   featurizer.precalc_norm_params=False \
   ms_per_timestep=80 \
   nn.fp16_needs_obey_precision_pass=True \
   chunk_size=4.8 \
   left_padding_size=1.6 \
   right_padding_size=1.6 \
   featurizer.max_batch_size=256 \
   featurizer.max_execution_batch_size=256 \
   decoder=flashlight \
   flashlight_decoder.asr_model_delay=-1 \
   decoding_language_model_binary=<bin_file> \
   decoding_vocab=<txt_decoding_lexicon_file> \
   flashlight_decoder.lm_weight=0.8 \
   flashlight_decoder.word_insertion_score=1.0 \
   flashlight_decoder.beam_size=32 \
   flashlight_decoder.beam_threshold=20. \
   flashlight_decoder.num_tokenization=1 \
   language_code=en-US \
   wfst_tokenizer_model=<far_tokenizer_file> \
   wfst_verbalizer_model=<far_verbalizer_file> \
   speech_hints_model=<far_speech_hints_file>
```

--------------------------------

### Run ASR Client with Boosted Words

Source: https://docs.nvidia.com/nim/speech/latest/asr/customization/customization.html

Execute the ASR client, specifying the boosted words file and a score to control the strength of custom pronunciation mapping.

```bash
riva_streaming_asr_client \
    --audio_file=audio.wav \
    --boosted_words_file=boost.txt \
    --boosted_words_score=20

```

--------------------------------

### NVIDIA Riva Python Clients Repository

Source: https://docs.nvidia.com/nim/speech/latest/reference/api-references/tts/realtime-tts.html

For developing realtime WebSocket clients in Python, consult the NVIDIA Riva Python Clients repository. This resource provides the necessary tools and examples.

```text
For building realtime WebSocket clients in Python, refer to the NVIDIA Riva Python Clients repository.
```

--------------------------------

### Copy Sample Audio File

Source: https://docs.nvidia.com/nim/speech/latest/asr/deploy-asr-models/parakeet-ctc-vi-vn.html

Copies a sample Vietnamese audio file from the deployed NIM container to the local directory.

```bash
docker cp $CONTAINER_ID:/opt/riva/wav/vi-VN_sample.wav .

```

--------------------------------

### Transcribe Audio File Offline via HTTP

Source: https://docs.nvidia.com/nim/speech/latest/asr/deploy-asr-models/parakeet-ctc-en-us.html

Send an input speech file to the service in one shot for transcription using the HTTP protocol. This example uses a curl command.

```bash
curl -s http://0.0.0.0:9000/v1/audio/transcriptions -F language=en-US \
  -F file="@en-US_sample.wav"

```

--------------------------------

### List Available Models and Language Pairs

Source: https://docs.nvidia.com/nim/speech/latest/nmt/deploy-nmt-model.html

Query the running NMT NIM to list all available models and their supported language pairs.

```python
python3 python-clients/scripts/nmt/nmt.py --server 0.0.0.0:50051 --list-models

```

--------------------------------

### Transcribe Audio File Offline via gRPC

Source: https://docs.nvidia.com/nim/speech/latest/asr/deploy-asr-models/parakeet-ctc-en-us.html

Send an input speech file to the service in one shot for transcription using the gRPC protocol. This example includes automatic punctuation.

```python
python3 python-clients/scripts/asr/transcribe_file_offline.py \
  --server 0.0.0.0:50051 \
  --language-code en-US --automatic-punctuation \
  --input-file en-US_sample.wav

```

--------------------------------

### Create NIM Model Store for NMT

Source: https://docs.nvidia.com/nim/speech/latest/reference/utilities.html

Command to set up a model store for Neural Machine Translation (NMT). It utilizes a specific container and requires the NGC_API_KEY to be configured.

```bash
export CONTAINER_ID=riva-translate-1_6b
export LOCAL_NIM_CACHE=$HOME/cache
docker run -it --rm --gpus all -e NGC_API_KEY \
    -v $LOCAL_NIM_CACHE:/opt/nim/.cache --entrypoint nim_create_model_store \
    nvcr.io/nim/nvidia/$CONTAINER_ID:latest \
    -p 9f54ea088ebb1e656ee51b6311b4b8ad25e919faab129aa3ccbdd2b984570cd6 \
    -m /tmp

```

```log
...
INFO 2025-05-13 10:25:13.833 create_model_store.py:57] Fetching contents for profile 9f54ea088ebb1e656ee51b6311b4b8ad25e919faab129aa3ccbdd2b984570cd6
INFO 2025-05-13 10:25:13.833 nim_sdk.py:368] Using the default model_cache_path: /opt/nim/workspace
INFO 2025-05-13 10:25:13.833 nim_sdk.py:378] Creating model store at /tmp
...

```

--------------------------------

### FP8 Quantization Configuration

Source: https://docs.nvidia.com/nim/speech/latest/asr/customization/pipeline-configuration.html

Enable FP8 precision for deployment by adding `nn.use_trt_fp8=True` to the build command. This requires GPUs with compute capability 8.9 or higher.

```bash
nn.use_trt_fp8=True
```

--------------------------------

### Split Large Audio Files (Sox)

Source: https://docs.nvidia.com/nim/speech/latest/troubleshooting/asr.html

Use the `sox` command-line tool to split large audio files into smaller segments, each under the 25 MB limit. This example splits into 60-second chunks.

```bash
sox input.wav output_%03d.wav trim 0 60 : newp : restart
```

--------------------------------

### Display Riva NIM Helm Chart README

Source: https://docs.nvidia.com/nim/speech/latest/deployment/helm.html

This command displays the README for the riva-nim helm chart, rendering it in the terminal using 'glow'. This helps in understanding configuration options.

```bash
helm show readme riva-nim-<version_number>.tgz | glow -p -
```

--------------------------------

### List NMT Model Profiles

Source: https://docs.nvidia.com/nim/speech/latest/reference/utilities.html

Prints system information and model profiles for an NMT NIM container. Ensure CONTAINER_ID is set to a valid NMT model.

```bash
export CONTAINER_ID=riva-translate-1_6b
docker run -it --rm --gpus all --entrypoint nim_list_model_profiles \
    nvcr.io/nim/nvidia/$CONTAINER_ID:latest

```

--------------------------------

### Deploy Parakeet TDT NIM Container

Source: https://docs.nvidia.com/nim/speech/latest/asr/deploy-asr-models/parakeet-tdt.html

Use this command to run the Parakeet TDT NIM container. Ensure you have Docker and NVIDIA container toolkit installed. Set NGC_API_KEY and other environment variables as needed.

```bash
export CONTAINER_ID=parakeet-0.6b-tdt
export NIM_TAGS_SELECTOR="type=multi"

docker run -it --rm --name=$CONTAINER_ID \
  --runtime=nvidia \
  --gpus '"device=0"' \
  --shm-size=8GB \
  -e NGC_API_KEY \
  -e NIM_HTTP_API_PORT=9000 \
  -e NIM_GRPC_API_PORT=50051 \
  -p 9000:9000 \
  -p 50051:50051 \
  -e NIM_TAGS_SELECTOR \
  nvcr.io/nim/nvidia/$CONTAINER_ID:latest

```

--------------------------------

### Copy Sample Audio from NIM Container

Source: https://docs.nvidia.com/nim/speech/latest/asr/deploy-asr-models/parakeet-ctc-zh-cn.html

Copies a sample Mandarin audio file from the running NIM container to the local host for inference testing.

```bash
docker cp $CONTAINER_ID:/opt/riva/wav/zh-CN_sample.wav .

```

--------------------------------

### Configure Utterance Endpointing Parameters

Source: https://docs.nvidia.com/nim/speech/latest/asr/customization/pipeline-configuration.html

Tune the start and stop thresholds and history windows for utterance beginning and end detection. These parameters control when the ASR decoder state is reset and the punctuator model is triggered.

```bash
endpointing.start_history=300 \
  endpointing.start_th=0.2 \
  endpointing.stop_history=800 \
  endpointing.stop_th=0.98
```

--------------------------------

### Configure Nemotron ASR Streaming Pipeline

Source: https://docs.nvidia.com/nim/speech/latest/asr/customization/pipeline-configuration.html

Use this configuration for building a streaming ASR pipeline with Nemotron. It includes parameters for featurization, decoding, and endpointing. Ensure all required Riva files and models are specified in source_path.

```bash
riva-build --config-path=pkg://servicemaker.configs.asr --config-name=streaming \
   output_path=<rmir_filename>:<key> \
   'source_path=[<riva_file>:<key>]' \
   profane_words_file=<txt_profane_words_file> \
   name=cache-aware-parakeet-rnnt-en-US-asr-streaming \
   featurizer.use_utterance_norm_params=False \
   featurizer.precalc_norm_time_steps=0 \
   featurizer.precalc_norm_params=False \
   ms_per_timestep=80 \
   endpointing.residue_blanks_at_start=-2 \
   nemo_decoder.use_stateful_decoding=true \
   endpointing_type=niva \
   endpointing.stop_history=800 \
   endpointing.residue_blanks_at_end=0 \
   unified_acoustic_model=true \
   feature_extractor_type=torch \
   torch_feature_type=nemo \
   featurizer.use_streaming_torch_fe=true \
   nn.fp16_needs_obey_precision_pass=True \
   nn.am_cache_len_input_use_int64=true \
   att_context_size='[70,1]' \
   max_batch_size=32 \
   nn.max_batch_size=32 \
   nn.opt_batch_size=32 \
   decoder=nemo \
   language_code=en-US \
   wfst_tokenizer_model=<far_tokenizer_file> \
   wfst_verbalizer_model=<far_verbalizer_file> \
   speech_hints_model=<far_speech_hints_file>
```

--------------------------------

### Whisper-Large Offline ASR Pipeline (TRTLLM)

Source: https://docs.nvidia.com/nim/speech/latest/asr/customization/pipeline-configuration.html

Configure an offline ASR pipeline for the Whisper-Large model using TensorRT-LLM (TRTLLM) decoder. This setup supports numerous languages and specific feature extraction settings.

```bash
riva-build --config-path=pkg://servicemaker.configs.asr --config-name=offline \
  output_path=<rmir_filename>:<key> \
  'source_path=[<riva_file>:<key>]' \
  profane_words_file=<txt_profane_words_file> \
  name=whisper-large-v3-multi-asr-offline \
  unified_acoustic_model=true \
  chunk_size=30 \
  left_padding_size=0 \
  right_padding_size=0 \
  decoder=trtllm \
  feature_extractor_type=torch \
  torch_feature_type=whisper \
  featurizer.norm_per_feature=false \
  max_batch_size=8 \
  featurizer.precalc_norm_params=false \
  featurizer.max_batch_size=8 \
  featurizer.max_execution_batch_size=8 \
  language_code=\'en,zh,de,es,ru,ko,fr,ja,pt,tr,pl,ca,nl,ar,sv,it,id,hi,fi,vi,he,uk,el,ms,cs,ro,da,hu,ta,no,th,ur,hr,bg,lt,la,mi,ml,cy,sk,te,fa,lv,bn,sr,az,sl,kn,et,mk,br,eu,is,hy,ne,mn,bs,kk,sq,sw,gl,mr,pa,si,km,sn,yo,so,af,oc,ka,be,tg,sd,gu,am,yi,lo,uz,fo,ht,ps,tk,nn,mt,sa,lb,my,bo,tl,mg,as,tt,haw,ln,ha,ba,jw,su,yue,multi\'
```

--------------------------------

### Run Docker Container with Prebuilt Model Cache

Source: https://docs.nvidia.com/nim/speech/latest/deployment/docker/model-caching.html

Mount a local cache directory to persist prebuilt models across container runs. Ensure the cache directory has appropriate permissions.

```bash
# Create the cache directory on the host
export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p $LOCAL_NIM_CACHE
chmod 777 $LOCAL_NIM_CACHE

# Set values for your model (example: ASR Parakeet 1.1b en-US streaming)
export CONTAINER_ID=parakeet-1-1b-ctc-en-us
export NIM_TAGS_SELECTOR="name=parakeet-1-1b-ctc-en-us,mode=str,model_type=prebuilt"

docker run -it --rm --name=$CONTAINER_ID \
  --runtime=nvidia \
  --gpus '"device=0"' \
  --shm-size=8GB \
  -e NGC_API_KEY \
  -e NIM_TAGS_SELECTOR \
  -e NIM_HTTP_API_PORT=9000 \
  -e NIM_GRPC_API_PORT=50051 \
  -p 9000:9000 \
  -p 50051:50051 \
  -v $LOCAL_NIM_CACHE:/opt/nim/.cache \
  nvcr.io/nim/nvidia/$CONTAINER_ID:latest

```

--------------------------------

### Export German ITN Grammars with Pynini

Source: https://docs.nvidia.com/nim/speech/latest/asr/customization/customization.html

Export German ITN grammars using the `pynini_export.py` tool. This generates OpenFst finite state archive (FAR) files for deployment with Riva. Ensure you have a functional NeMo installation.

```bash
python3 pynini_export.py --output_dir . --grammars itn_grammars --input_case cased --language de

```

--------------------------------

### List Voices HTTP Endpoint

Source: https://docs.nvidia.com/nim/speech/latest/reference/api-references/tts/realtime-tts.html

Use this HTTP GET endpoint to query the available pre-trained voices and models provided by the TTS service. The response is a JSON object mapping language codes to lists of available voices.

```http
GET /v1/audio/list_voices
```

--------------------------------

### Verify Port Forwarding Configuration

Source: https://docs.nvidia.com/nim/speech/latest/get-started/tutorials/wsl.html

Execute this command in an elevated Windows command prompt to display the current port proxy configuration, allowing you to verify that the NIM ports are correctly forwarded.

```bash
netsh interface portproxy show v4tov4
```
Tokens:
Raw