NVIDIA TAO Toolkit

https://docs.nvidia.com/tao/tao-toolkit
Admin
NVIDIA TAO Toolkit is a low-code AI framework that provides tools and workflows for fine-tuning,...
Tokens:256,756
Snippets:3,197
Trust Score:10
Update:2 months ago
Show doc for...
Tokens:
Raw
### Install TAO CLI

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/tao_deploy/nsight_dl_designer_integration.html

Commands to set up a Python virtual environment and install the TAO client.

```bash
# Create and activate virtual environment
python3 -m venv tao_env
source tao_env/bin/activate

# Install TAO
pip install nvidia-tao-client
```

--------------------------------

### Legacy TAO CLI Installation and Login

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/migration_guides/migrating_to_tao_7.0.html

The previous method for installing and authenticating with the TAO toolkit.

```bash
pip install nvidia-tao-client
tao login --ngc-key $NGC_KEY --ngc-org-name $NGC_ORG
tao --help
```

--------------------------------

### Launch TensorBoard

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/deprecations/tensorflow_1/object_detection/fasterrcnn.html

Command to start the TensorBoard server for monitoring training logs.

```bash
tensorboard --logdir=/path/to/logs
```

--------------------------------

### Sparse4D Training Configuration Example

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/cv_finetuning/pytorch/sparse4d/sparse4d.html

Example configuration block for training parameters, including optimizer settings, learning rate scheduling, and hardware resource allocation.

```yaml
train:
  num_epochs: 5
  num_nodes: 1
  num_gpus: 1
  validation_interval: 1
  checkpoint_interval: 1
  pretrained_model_path: ???
  precision: bf16
  optim:
    type: "adamw"
    lr: 0.0001
    weight_decay: 0.001
    paramwise_cfg:
      custom_keys:
        img_backbone:
          lr_mult: 0.25
    grad_clip:
      max_norm: 25
      norm_type: 2
    lr_scheduler:
      policy: "cosine"
      warmup: "linear"
      warmup_iters: 500
      warmup_ratio: 0.333333
      min_lr_ratio: 0.001
```

--------------------------------

### TAO 7.0 Plugin Installation and Environment Setup

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/migration_guides/migrating_to_tao_7.0.html

The new workflow using Claude Code plugins and secure environment variable injection for credentials.

```bash
# In a Claude Code session:
/plugin marketplace add https://github.com/NVIDIA-TAO/tao-skills-bank.git#7.0.1
/plugin install tao-skills@tao-skill-bank

# Export credentials in your shell BEFORE launching the agent.
# Use your shell's secret-loading mechanism of choice — for example a
# password manager that prints to stdout, your OS keychain, or a CI-injected
# environment. Do NOT write these values to a file on disk.
export NGC_KEY="$(security find-generic-password -s NGC_KEY -w)"
export NGC_ORG="my-org"
export ACCESS_KEY="$(op read 'op://Private/AWS/access_key')"
export SECRET_KEY="$(op read 'op://Private/AWS/secret_key')"
export S3_BUCKET_NAME="my-bucket"
export S3_ENDPOINT_URL="https://s3.us-west-1.amazonaws.com"

# Now launch the agent so it inherits the exported env.
claude

# Verify your GPU host is ready (only needed for the local-docker platform):
docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
```

--------------------------------

### Install NSight DL Designer

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/tao_deploy/nsight_dl_designer_integration.html

Commands to extract and execute the NSight DL Designer installer script.

```bash
# Extract and install NDLD
chmod +x NVIDIA_DeepLearning_*.run
./NVIDIA_DeepLearning_*.run
```

--------------------------------

### Example Specification File for Distillation

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/cv_finetuning/pytorch/self_supervised_learning/nvdinov2.html

A YAML configuration file demonstrating the setup for distilling a ViT Large model to a ViT Base model, including dataset paths, training hyperparameters, and scheduler settings.

```yaml
encryption_key: tlt_encode
  results_dir: /path/to/experiment_results
  model:
    distill:
      enable: True
      disable_masking: False
      pretrained_non_distill_pl_model_path: /path/to/pretrained/model.pth
    backbone:
      teacher_type: "vit_l"
      student_type: "vit_l"
      drop_path_rate: 0.4
      patch_size: 14
      img_size: 518
    head:
      num_layers: 3
      hidden_dim: 2048
      bottleneck_dim: 384
  dataset:
    train_dataset:
      images_dir: /path/to/img_dir
    test_dataset:
      images_dir: "???"
    batch_size: 16
    workers: 10
    transform:
      n_global_crops: 2
      global_crops_scale: [0.32, 1.0]
      global_crops_size: 224
      n_local_crops: 8
      local_crops_scale: [0.05, 0.32]
      local_crops_size: 98
  train:
    resume_training_checkpoint_path: null
    pretrained_model_path: /path/to/pretrained/model.pth
    num_nodes: 1
    num_gpus: 1
    num_epochs: 10
    checkpoint_interval: 1
    layerwise_decay: 1.0
    clip_grad_norm: 3.0
    optim:
      optim: "adamw"
    schedulers:
      learning_rate:
        val_base: "${eval: '2e-4 * (${dataset.batch_size} * ${train.num_gpus} * ${train.num_nodes} / 1024) ** (1/2)'}"
        val_final: 1e-6
        warm_up_steps: 100000
      last_layer_learning_rate:
        val_base: "${eval: '2e-4 * (${dataset.batch_size} * ${train.num_gpus} * ${train.num_nodes} / 1024) ** (1/2)'}"
        val_final: 1e-6
        warm_up_steps: 100000
        freeze_steps: 1250
      weight_decay:
        val_base: 0.04
        val_final: 0.2
      momentum:
        val_base: 0.994
        val_final: 1.0
      teacher_temperature:
        val_base: 0.07
        val_final: 0.07
        val_start: 0.04
        warm_up_steps: 37500
    num_prototypes: 131072
    results_dir: "${results_dir}/train"
  inference:
    checkpoint: "???"
  export:
    gpu_id: 0
    checkpoint: "???"
    onnx_file: "???"
    input_width: 518
    input_height: 518
```

--------------------------------

### Full DeepStream Configuration Example

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/ds_tao/multitask_classification_ds.html

A complete example of a DeepStream configuration file for a multitask classification model.

```text
[property]
gpu-id=0
net-scale-factor=1.0
offsets=103.939;116.779;123.68
model-color-format=1
labelfile-path=multi_task_labels.txt
onnx-file=../../../models/multi_task/abc.onnx
model-engine-file=../../../models/multi_task/abc.etlt_b1_gpu0_fp16.engine
infer-dims=3;80;60
batch-size=1
```

--------------------------------

### Example deviceQuery Output

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/ds_tao/deformable_detr_ds.html

Sample output showing CUDA capability version.

```text
Detected 2 CUDA Capable device(s)

Device 0: "Tesla T4"
  CUDA Driver Version / Runtime Version          10.2 / 10.2
  CUDA Capability Major/Minor version number:    7.5
```

--------------------------------

### Start Triton Server

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/cv_finetuning/pytorch/re_identification/re_identification.html

Command to initialize the Triton server for ReIdentificationNet inference.

```bash
bash scripts/start_server.sh
```

--------------------------------

### Cosmos-RL Inference Specification

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/vlm_finetuning/cosmos_rl.html

Example configuration file for defining inference parameters such as media path, prompt, and hardware settings.

```yaml
inference:
  media: "path/to/video.mp4"
  prompt: "Describe this video."
  fps: 4
  total_pixels: 6422528
  max_new_tokens: 4096
  num_gpus: 1  # Optional: Number of GPUs for inference (enables tensor parallelism)
```

--------------------------------

### Visual ChangeNet Dataset Configuration Example

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/cv_finetuning/pytorch/visual_changenet/visual_changenet_classify.html

Example YAML configuration for defining dataset paths, batch sizes, and input mapping for Visual ChangeNet classification.

```yaml
dataset:
  classify:
    train_dataset:
      csv_path: /path/to/train.csv
      images_dir: /path/to/img_dir
    validation_dataset:
      csv_path: /path/to/val.csv
      images_dir: /path/to/img_dir
    test_dataset:
      csv_path: /path/to/test.csv
      images_dir: /path/to/img_dir
    infer_dataset:
      csv_path: /path/to/infer.csv
      images_dir: /path/to/img_dir
    image_ext: .jpg
    batch_size: 16
    workers: 2
    fpratio_sampling: 0.2
    num_input: 4
    input_map:
      LowAngleLight: 0
      SolderLight: 1
      UniformLight: 2
      WhiteLight: 3
    concat_type: linear
    grid_map:
      x: 2
      y: 2
    image_width: 128
    image_height: 128
    augmentation_config:
      rgb_input_mean: [0.485, 0.456, 0.406]
      rgb_input_std: [0.229, 0.224, 0.225]
    num_classes: 2
```

--------------------------------

### Run gen_trt_engine Action

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/tao_deploy/mask_grounding_dino.html

Example command description for executing the engine generation process.

```text
Build an FP16 TensorRT engine for Mask Grounding DINO from the exported
ONNX at ``s3://my-bucket/mgdino/model.onnx`` using ``trt-spec.yaml``.
Write the engine to ``s3://my-bucket/mgdino/model.engine``. Run on
the local Docker backend.
```

--------------------------------

### Cosmos-RL Training Specification File

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/vlm_finetuning/cosmos_rl.html

Example configuration file defining training, validation, policy, logging, and custom parameters for Cosmos-RL.

```yaml
train:
  resume: false
  epoch: 10
  compile: false
  train_batch_per_replica: 1
  output_dir: "output"
  optm_lr: 1e-6
  optm_impl: "foreach"
  optm_weight_decay: 0.01
  optm_min_lr_factor: 0.0
  optm_grad_norm_clip: 1.0
  epsilon: 1e-8
  optm_name: "AdamW"
  optm_betas: [0.9, 0.999]
  optm_warmup_epochs: 0
  async_tp_enabled: false
  master_dtype: "float32"
  param_dtype: "bfloat16"
  fsdp_reduce_dtype: "float32"
  fsdp_offload: false
  fsdp_reshard_after_forward: "default"
  sync_weight_interval: 1
  ckpt:
    enable_checkpoint: true
    save_freq_in_epoch: 10
    save_mode: "sync"
    max_keep: 8
    export_safetensors: true
  train_policy:
    type: "sft"
    mini_batch: 4
    enable_dataset_cache: true
    dataloader_num_workers: 8
    dataloader_prefetch_factor: 8
    conversation_column_name: "conversations"
    dataset:
      name: "its"
      test_size: 1
  fp8:
    enable_fp8: false
    fp8_recipe: "dynamic_scaling"
    quant_recipe: "rowwise"

validation:
  enable: true
  freq_in_epoch: 10

policy:
  model_name_or_path: "nvidia/Cosmos-Reason1-7B"
  model_max_length: 4096
  model_gradient_checkpointing: true
  parallelism:
    n_init_replicas: 1
    tp_size: 1
    cp_size: 1
    dp_shard_size: 1
    dp_replicate_size: 1
    pp_size: 1
    cp_rotate_method: "allgather"
  lora:
    r: 8
    lora_alpha: 8
    lora_dropout: 0.0
    target_modules: ["q_proj", "v_proj"]
    use_rslora: false
    modules_to_save: []
    init_lora_weights: true

logging:
  logger: ["console", "tao"]
  project_name: "cosmos-rl"
  experiment_name: "cosmos-rl"

custom:
  dataset:
    annotation_path: "data/sft/annotations.json"
    media_path: "data/sft/train2017"
    system_prompt: ""
  vision:
    fps: 1
    total_pixels: 313600

redis: "12800"
results_dir: "/results"
```

--------------------------------

### Cosmos-RL Evaluation Specification File

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/vlm_finetuning/cosmos_rl.html

Example configuration file defining parameters for model evaluation, including dataset paths, model settings, and generation constraints.

```yaml
# Multi-GPU configuration (optional - automatically configures parallelism)
num_gpus: 1  # Use 1 for single GPU, 2+ for multi-GPU evaluation

evaluate:
  dataset:
    annotation_path: "path/to/eval_annotations.json"
    media_dir: "path/to/eval_media/"
    system_prompt: "You are a helpful assistant that can answer questions about a street-view CCTV footage. The vehicles that need attention are marked with bounding boxes and IDs."
  model:
    model_name: "nvidia/Cosmos-Reason1-7B"
    save_folder: "cr1_1_zero_shot"
    tokenizer_model_name: "qwen2.5-vl-7b"
    dtype: "bfloat16"
    tp_size: 1
    max_length: 128000
    enable_lora: false
    base_model_path: ""
  evaluation:
    answer_type: "freeform"
    num_processes: 40
    skip_saved: false
    seed: 1
    limit: -1
    total_shard: 1
    shard_id: 0
  vision:
    fps: 4
    total_pixels: 3136000
  generation:
    max_retries: 10
    max_tokens: 1024
    temperature: 0
    repetition_penalty: 1
    presence_penalty: 0
    frequency_penalty: 0
  results:
    save_individual_results: true
    save_confusion_matrix: true
    save_metrics_summary: true

results_dir: "/results"
```

--------------------------------

### LLaVA annotation format example

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/vlm_finetuning/cosmos_rl.html

JSON structure for annotations following the LLaVA conversation format.

```json
{
  "id": "d460df3a29cc7d208d4d588c63e83579",
  "images": [
    "images/001354.png",
    "som_images/001354.d460df3a29cc7d208d4d588c63e83579.png"
  ],
  "conversations": [
    {
      "from": "human",
      "value": "<image>\nThe first image is the original, and the second is an overlay. Bright numeric IDs are labeled at the center of certain visual objects in the second image.\nBased on pallet positions in Region [0] Region [1] Region [2] Region [3] Region [4] Region [5] Region [6] Region [7] Region [8] Region [9], which one should the transporter at Region [10] retrieve?\nPlease answer with only the integer number of the correct region the number should be one that is both shown in the image and mentioned in this question. Do not include any explanation or extra text."
    },
    {
      "from": "gpt",
      "value": "3"
    }
  ],
  "category": "mcq",
  "normalized_answer": "3"
}
```

--------------------------------

### Resolved Training Specification

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/vlm_finetuning/cosmos_rl.html

Example of a resolved training configuration generated by the agent, including training parameters and a custom script path.

```yaml
train:
  epoch: 10
  optm_lr: 0.0001
custom_script: s3://my-bucket/custom_sft.py
results_dir: /results
```

--------------------------------

### Initialize DeepStream Containers

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/ds_tao/nvocdr_ds.html

Commands to start the DeepStream development environment on x86 and Jetson platforms, including OpenCV installation.

```bash
docker run --gpus=all -v <work_path>:<work_path> --rm -it --privileged --net=host nvcr.io/nvidia/deepstream:6.2-triton bash

# install opencv
apt update && apt install -y libopencv-dev
```

```bash
docker run --gpus=all -v <work_path>:<work_path> --rm -it --privileged --net=host nvcr.io/nvidia/deepstream-l4t:6.2-triton bash

# install opencv
apt update && apt install -y libopencv-dev
```

--------------------------------

### Setup and Run PeopleSegNet

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/ds_tao/deepstream_tao_integration.html

Commands to clone the repository, download the model, build the application, and run inference.

```bash
git clone https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps.git
```

```bash
ngc registry model download-version "nvidia/tao/peoplesegnet:deployable_v2.0"
```

```bash
wget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplesegnet/versions/deployable_v2.0/zip \
-O peoplesegnet_deployable_v2.0.zip
```

```bash
export CUDA_VER=xy.z              // xy.z is CUDA version, e.g. 10.2
make
```

```bash
SHOW_MASK=1 ./apps/ds-tlt -c configs/peopleSegNet_tlt/pgie_peopleSegNetv2_tlt_config.txt -i \
/opt/nvidia/deepstream/deepstream-5.1/samples/streams/sample_720p.h264 -d
```

--------------------------------

### SSD Specification File Example

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/deprecations/tensorflow_1/object_detection/ssd.html

A sample prototxt configuration file for SSD, defining model architecture, training parameters, evaluation settings, NMS thresholds, data augmentation, and dataset paths.

```protobuf
random_seed: 42
ssd_config {
  aspect_ratios: "[[1.0, 2.0, 0.5],  [1.0, 2.0, 0.5, 3.0, 1.0/3.0],  [1.0, 2.0, 0.5, 3.0, 1.0/3.0],  [1.0, 2.0, 0.5, 3.0, 1.0/3.0],  [1.0, 2.0, 0.5],  [1.0, 2.0, 0.5]]"
  scales: "[0.07, 0.15, 0.33, 0.51, 0.69, 0.87, 1.05]"
  two_boxes_for_ar1: true
  clip_boxes: false
  variances: "[0.1, 0.1, 0.2, 0.2]"
  arch: "resnet"
  nlayers: 18
  freeze_bn: false
  freeze_blocks: 0
}
training_config {
  batch_size_per_gpu: 16
  num_epochs: 80
  enable_qat: false
  learning_rate {
  soft_start_annealing_schedule {
    min_learning_rate: 5e-5
    max_learning_rate: 2e-2
    soft_start: 0.15
    annealing: 0.8
    }
  }
  regularizer {
    type: L1
    weight: 3e-5
  }
}
eval_config {
  validation_period_during_training: 10
  average_precision_mode: SAMPLE
  batch_size: 16
  matching_iou_threshold: 0.5
}
nms_config {
  confidence_threshold: 0.01
  clustering_iou_threshold: 0.6
  top_k: 200
}
augmentation_config {
  output_width: 300
  output_height: 300
  output_channel: 3
  image_mean {
    key: 'b'
    value: 103.9
  }
  image_mean {
    key: 'g'
    value: 116.8
  }
  image_mean {
    key: 'r'
    value: 123.7
  }
}
dataset_config {
  data_sources: {
      # option 1
      tfrecords_path: "/path/to/train/tfrecord"

      # option 2
      # label_directory_path: "/path/to/train/labels"
      # image_directory_path: "/path/to/train/images"
  }
  include_difficult_in_training: true
  target_class_mapping {
      key: "car"
      value: "car"
  }
  target_class_mapping {
      key: "pedestrian"
      value: "pedestrian"
  }
  target_class_mapping {
      key: "cyclist"
      value: "cyclist"
  }
  target_class_mapping {
      key: "van"
      value: "car"
  }
  target_class_mapping {
      key: "person_sitting"
      value: "pedestrian"
  }
  validation_data_sources: {
    label_directory_path: "/path/to/val/labels"
    image_directory_path: "/path/to/val/images"
  }
}
```

--------------------------------

### COCO Annotation File Example

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/data_annotation_format.html

A sample JSON structure representing annotations, images, and categories. Ensure category IDs start from 1.

```json
"annotations": [{"segmentation": [[510.66,423.01,511.72,420.03,510.45,416.0,510.34,413.02,510.77,410.26,510.77,407.5,510.34,405.16,511.51,402.83,511.41,400.49,510.24,398.16,509.39,397.31,504.61,399.22,502.17,399.64,500.89,401.66,500.47,402.08,499.09,401.87,495.79,401.98,490.59,401.77,488.79,401.77,485.39,398.58,483.9,397.31,481.56,396.35,478.48,395.93,476.68,396.03,475.4,396.77,473.92,398.79,473.28,399.96,473.49,401.87,474.56,403.47,473.07,405.59,473.39,407.71,476.68,409.41,479.23,409.73,481.56,410.69,480.4,411.85,481.35,414.93,479.86,418.65,477.32,420.03,476.04,422.58,479.02,422.58,480.29,423.01,483.79,419.93,486.66,416.21,490.06,415.57,492.18,416.85,491.65,420.24,492.82,422.9,493.56,424.39,496.43,424.6,498.02,423.01,498.13,421.31,497.07,420.03,497.07,415.15,496.33,414.51,501.1,411.96,502.06,411.32,503.02,415.04,503.33,418.12,501.1,420.24,498.98,421.63,500.47,424.39,505.03,423.32,506.2,421.31,507.69,419.5,506.31,423.32,510.03,423.01,510.45,423.01]],"area": 702.1057499999998,"iscrowd": 0,"image_id": 289343,"bbox": [473.07,395.93,38.65,28.67],"category_id": 18,"id": 1768}],
"images": [{"license": 1,"file_name": "000000407646.jpg","coco_url": "http://images.cocodataset.org/val2017/000000407646.jpg","height": 400,"width": 500,"date_captured": "2013-11-23 03:58:53","flickr_url": "http://farm4.staticflickr.com/3110/2855627782_17b93a684e_z.jpg","id": 407646}],
"categories": [{"supercategory": "person","id": 1,"name": "person"},{"supercategory": "vehicle","id": 2,"name": "bicycle"},{"supercategory": "vehicle","id": 3,"name": "car"},{"supercategory": "vehicle","id": 4,"name": "motorcycle"}]
```

--------------------------------

### Example COCO annotation file

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/data_annotation_format.html

A sample JSON file demonstrating the COCO annotation format. Ensure category IDs start from 1.

```json
{
    "annotations": [{"area": 702.1057499999998,"iscrowd": 0,"image_id": 289343,"bbox": [473.07,395.93,38.65,28.67],"category_id": 18,"id": 1768}],
    "images": [{"license": 1,"file_name": "000000407646.jpg","coco_url": "http://images.cocodataset.org/val2017/000000407646.jpg","height": 400,"width": 500,"date_captured": "2013-11-23 03:58:53","flickr_url": "http://farm4.staticflickr.com/3110/2855627782_17b93a684e_z.jpg","id": 407646}],
    "categories": [{"supercategory": "person","id": 1,"name": "person"},{"supercategory": "vehicle","id": 2,"name": "bicycle"},{"supercategory": "vehicle","id": 3,"name": "car"},{"supercategory": "vehicle","id": 4,"name": "motorcycle"}]
}
```

--------------------------------

### EfficientDet Specification File Example

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/deprecations/tensorflow_1/object_detection/efficientdet_tf1.html

A sample .prototxt configuration file demonstrating the required structure for training, evaluation, and model parameters.

```protobuf
training_config {
  train_batch_size: 16
  iterations_per_loop: 10
  checkpoint_period: 10
  num_examples_per_epoch: 14700
  num_epochs: 300
  model_name: 'efficientdet-d0'
  profile_skip_steps: 100
  tf_random_seed: 42
  lr_warmup_epoch: 5
  lr_warmup_init: 0.00005
  learning_rate: 0.1
  amp: True
  moving_average_decay: 0.9999
  l2_weight_decay: 0.00004
  l1_weight_decay: 0.0
  checkpoint: "/path/to/your/pretrained_model"
  # pruned_model_path: "/path/to/your/pruned/model"
}
dataset_config {
  num_classes: 91
  image_size: "512,512"
  training_file_pattern: "/path/to/coco/train-*"
  validation_file_pattern: "/path/to/coco/val-*"
  validation_json_file: "/path/to/coco/annotations/instances_val2017.json"
}
eval_config {
  eval_batch_size: 16
  eval_epoch_cycle: 10
  eval_after_training: True
  eval_samples: 5000
  min_score_thresh: 0.4
  max_detections_per_image: 100
}
model_config {
  model_name: 'efficientdet-d0'
  min_level: 3
  max_level: 7
  num_scales: 3
}
augmentation_config {
  rand_hflip: True
  random_crop_min_scale: 0.1
  random_crop_max_scale: 2.0
}
```

--------------------------------

### Launching DetectNet_v2 Training

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/migration_guides/migrating_older_tlt_to_tao.html

Example command structure for initiating a training task using the TAO launcher.

```bash
tao detectnet_v2 train <args>
```

--------------------------------

### Install TensorBoard

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/deprecations/tensorflow_1/object_detection/fasterrcnn.html

Command to install the TensorBoard package via pip.

```bash
pip3 install tensorboard
```

--------------------------------

### Inference Specification Example

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/cv_finetuning/pytorch/metric_learning_recognition/metric_learning_recognition.html

Example configuration file for running MLRecogNet model inference.

```yaml
results_dir: /path/to/root/results/dir
model:
  backbone: resnet_50
  input_width: 224
  input_height: 224
  feat_dim: 256
dataset:
  workers: 8
  val_dataset:
    reference: /path/to/dataset/reference
    query: ""
inference:
  input_path: /path/to/dataset/test
  inference_input_type: classification_folder
  checkpoint: /path/to/model/checkpoint
  results_dir: /path/to/results/dir
  batch_size: 128
```

--------------------------------

### DSSD Configuration File Example

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/deprecations/tensorflow_1/object_detection/dssd.html

A sample prototxt configuration file for DSSD, defining model architecture, training parameters, evaluation settings, and dataset paths.

```protobuf
random_seed: 42
dssd_config {
  aspect_ratios: "[[1.0, 2.0, 0.5],  [1.0, 2.0, 0.5, 3.0, 1.0/3.0],  [1.0, 2.0, 0.5, 3.0, 1.0/3.0],  [1.0, 2.0, 0.5, 3.0, 1.0/3.0],  [1.0, 2.0, 0.5],  [1.0, 2.0, 0.5]]"
  scales: "[0.07, 0.15, 0.33, 0.51, 0.69, 0.87, 1.05]"
  two_boxes_for_ar1: true
  clip_boxes: false
  variances: "[0.1, 0.1, 0.2, 0.2]"
  arch: "resnet"
  nlayers: 18
  freeze_bn: false
  freeze_blocks: 0
}
training_config {
  batch_size_per_gpu: 16
  num_epochs: 80
  enable_qat: false
  learning_rate {
  soft_start_annealing_schedule {
    min_learning_rate: 5e-5
    max_learning_rate: 2e-2
    soft_start: 0.15
    annealing: 0.8
    }
  }
  regularizer {
    type: L1
    weight: 3e-5
  }
}
eval_config {
  validation_period_during_training: 10
  average_precision_mode: SAMPLE
  batch_size: 16
  matching_iou_threshold: 0.5
}
nms_config {
  confidence_threshold: 0.01
  clustering_iou_threshold: 0.6
  top_k: 200
}
augmentation_config {
  output_width: 300
  output_height: 300
  output_channel: 3
  image_mean {
    key: 'b'
    value: 103.9
  }
  image_mean {
    key: 'g'
    value: 116.8
  }
  image_mean {
    key: 'r'
    value: 123.7
  }
}
dataset_config {
  data_sources: {
      # option 1
      tfrecords_path: "/path/to/train/tfrecord"

      # option 2
      # label_directory_path: "/path/to/train/labels"
      # image_directory_path: "/path/to/train/images"
  }
  include_difficult_in_training: true
  target_class_mapping {
      key: "car"
      value: "car"
  }
  target_class_mapping {
      key: "pedestrian"
      value: "pedestrian"
  }
  target_class_mapping {
      key: "cyclist"
      value: "cyclist"
  }
  target_class_mapping {
      key: "van"
      value: "car"
  }
  target_class_mapping {
      key: "person_sitting"
      value: "pedestrian"
  }
  validation_data_sources: {
    label_directory_path: "/path/to/val/labels"
    image_directory_path: "/path/to/val/images"
  }
}
```

--------------------------------

### RetinaNet Configuration File Example

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/deprecations/tensorflow_1/object_detection/retinanet.html

A sample prototxt configuration file for RetinaNet, defining model architecture, training parameters, evaluation settings, and dataset paths.

```text
random_seed: 42
retinanet_config {
  aspect_ratios_global: "[1.0, 2.0, 0.5]"
  scales: "[0.045, 0.09, 0.2, 0.4, 0.55, 0.7]"
  two_boxes_for_ar1: false
  clip_boxes: false
  loss_loc_weight: 0.8
  focal_loss_alpha: 0.25
  focal_loss_gamma: 2.0
  variances: "[0.1, 0.1, 0.2, 0.2]"
  arch: "resnet"
  nlayers: 18
  n_kernels: 1
  n_anchor_levels: 1
  feature_size: 256
  freeze_bn: false
  freeze_blocks: 0
}
training_config {
  enable_qat: False
  batch_size_per_gpu: 24
  num_epochs: 100
  pretrain_model_path: "YOUR_PRETRAINED_MODEL"
  optimizer {
    sgd {
      momentum: 0.9
      nesterov: True
    }
  }
  learning_rate {
  soft_start_annealing_schedule {
    min_learning_rate: 4e-5
    max_learning_rate: 1.5e-2
    soft_start: 0.15
    annealing: 0.5
    }
  }
  regularizer {
    type: L1
    weight: 2e-5
  }
}
eval_config {
  validation_period_during_training: 10
  average_precision_mode: SAMPLE
  batch_size: 24
  matching_iou_threshold: 0.5
}
nms_config {
  confidence_threshold: 0.01
  clustering_iou_threshold: 0.6
  top_k: 200
}
augmentation_config {
    output_width: 384
    output_height: 1248
    output_channel: 3
    image_mean {
      key: 'b'
      value: 103.9
    }
    image_mean {
      key: 'g'
      value: 116.8
    }
    image_mean {
      key: 'r'
      value: 123.7
    }
}
dataset_config {
  data_sources: {
      # option 1
      tfrecords_path: "/workspace/tao-experiments/data/tfrecords/kitti_train*"

      # option 2
      # label_directory_path: "/workspace/tao-experiments/data/training/label_2"
      # image_directory_path: "/workspace/tao-experiments/data/training/image_2"
  }
  target_class_mapping {
      key: "car"
      value: "car"
  }
  target_class_mapping {
      key: "pedestrian"
      value: "pedestrian"
  }
  target_class_mapping {
      key: "cyclist"
      value: "cyclist"
  }
  target_class_mapping {
      key: "van"
      value: "car"
  }
  target_class_mapping {
      key: "person_sitting"
      value: "pedestrian"
  }
  validation_data_sources: {
      label_directory_path: "/workspace/tao-experiments/data/val/label"
      image_directory_path: "/workspace/tao-experiments/data/val/image"
  }
}
```

--------------------------------

### BpNetDataloader Configuration Example

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/deprecations/tensorflow_1/bodypose_estimation/bodyposenet.html

A YAML-based configuration block for setting up the BpNetDataloader, including image, pose, and dataset parameters.

```yaml
dataloader:
  __class_name__: BpNetDataloader
  batch_size: 10
  shuffle_buffer_size: 20000
  image_config:
    image_dims:
      height: 256
      width: 256
      channels: 3
    image_encoding: jpg
  pose_config:
    __class_name__: BpNetPoseConfig
    target_shape: [32, 32]
    pose_config_path: /workspace/examples/bpnet/model_pose_config/bpnet_18joints.json
  dataset_config:
    root_data_path: /workspace/tao-experiments/bpnet/data/
    train_records_folder_path: /workspace/tao-experiments/bpnet/data
    train_records_path: [train-fold-000-of-001]
    val_records_folder_path: /workspace/tao-experiments/bpnet/data
    val_records_path: [val-fold-000-of-001]
    dataset_specs:
      coco: /workspace/examples/bpnet/data_pose_config/coco_spec.json
  normalization_params:
    image_scale: [256.0, 256.0, 256.0]
    image_offset: [0.5, 0.5, 0.5]
    mask_scale: [255.0]
    mask_offset: [0.0]
  augmentation_config:
    ...
  label_processor_config:
    ...
```

--------------------------------

### Install git-lfs

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/ds_tao/deformable_detr_ds.html

Commands to install git-lfs on a Debian-based system, which is a prerequisite for downloading the source code.

```bash
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
sudo apt-get install git-lfs
git lfs install
```

--------------------------------

### Install Triton Client Requirements

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/cv_finetuning/pytorch/panoptic_3d_reconstruction/nvpanoptix3d.html

Install the necessary Python dependencies for the Triton client application.

```bash
pip install -r scripts/nvpanoptix3d_e2e_inference/client-requirements.txt
```

--------------------------------

### FpeNetDataloader Configuration Example

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/deprecations/tensorflow_1/facial_landmarks_estimation/facial_landmarks_estimation.html

A sample YAML configuration block for the FpeNetDataloader, defining augmentation, dataset paths, image dimensions, and keypoint settings.

```yaml
dataloader:
  __class_name__: FpeNetDataloader
  augmentation_info:
    augmentation_resize_probability: 0.5
    augmentation_resize_scale: 1.6
    enable_occlusion_augmentation: true
    enable_online_augmentation: true
    enable_resize_augmentation: true
    gamma_augmentation:
      gamma_max: 1.6
      gamma_min: 0.6
      gamma_probability: 0.1
      gamma_type: uniform
    modulus_spatial_augmentation:
      hflip_probability: 0.25
      rotate_rad_max: 0.35
      translate_max_x: 10
      translate_max_y: 10
      zoom_max: 1.2
      zoom_min: 0.8
    patch_probability: 0.5
    size_to_image_ratio: 0.5
    mask_augmentation_patch: true
  batch_size: 64
  dataset_info:
    image_extension: png
    no_occlusion_masking_sets: s578-usercalibration-incar-0 s578-usercalibration-incar-1
    root_path: /workspace/tao-experiments/
    tfrecord_folder_name: FpeTfRecords
    tfrecords_directory_path: /workspace/tao-experiments/models/tfrecords
    tfrecords_set_id_train: s578-usercalibration-incar-0
    tfrecords_set_id_val: s578-usercalibration-incar-0
    tfrecord_file_name: data.tfrecords
  image_info:
    image:
      channel: 1
      height: 80
      width: 80
  kpiset_info:
    tfrecords_set_id_kpi: s578-usercalibration-incar-1
  num_keypoints: 80
```

--------------------------------

### Evaluation Command Example

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/tao_deploy/mask_grounding_dino.html

Example command to trigger evaluation of a Mask Grounding DINO engine.

```text
Evaluate the Mask Grounding DINO TensorRT engine at
``s3://my-bucket/mgdino/model.engine`` against ``eval-spec.yaml``. Run on
local Docker.
```

--------------------------------

### KITTI Label File Examples

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/data_annotation_format.html

Sample content for label files, showing both full KITTI annotations and simplified versions for custom datasets.

```text
car 0.00 0 -1.58 587.01 173.33 614.12 200.12 1.65 1.67 3.64 -0.65 1.71 46.70 -1.59
cyclist 0.00 0 -2.46 665.45 160.00 717.93 217.99 1.72 0.47 1.65 2.45 1.35 22.10 -2.35
pedestrian 0.00 2 0.21 423.17 173.67 433.17 224.03 1.60 0.38 0.30 -5.87 1.63 23.11 -0.03
```

```text
car 0.00 0 0.00 587.01 173.33 614.12 200.12 0.00 0.00 0.00 0.00 0.00 0.00 0.00
cyclist 0.00 0 0.00 665.45 160.00 717.93 217.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00
pedestrian 0.00 0 0.00 423.17 173.67 433.17 224.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00
```

--------------------------------

### Install OpenCV on Jetson

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/ds_tao/nvocdr_ds.html

Installs OpenCV dependencies on Jetson platforms running Jetpack 5.1 or greater.

```bash
# install opencv
apt update && apt install -y libopencv-dev
```

--------------------------------

### Install TAO Skills Bank from Local Archive in Codex

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/getting_started.html

Extract a local zip file and point the Codex marketplace to the absolute path.

```bash
unzip ~/Downloads/tao-skills-bank-7.0.1.zip -d ~/tao-skills-bank
codex plugin marketplace add ~/tao-skills-bank
codex
/plugins                 # select TAO Skill Bank → Install plugin
```

--------------------------------

### Inference Command Example

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/tao_deploy/mask_grounding_dino.html

Example command to trigger inference using a Mask Grounding DINO engine.

```text
Run Mask Grounding DINO inference with the TensorRT engine at
``s3://my-bucket/mgdino/model.engine`` using ``infer-spec.yaml``. Run on
your chosen backend.
```

--------------------------------

### Example evaluation_config configuration

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/deprecations/tensorflow_1/object_detection/detectnet_v2.html

A sample configuration block demonstrating how to set evaluation modes, validation periods, and class-specific overlap and box dimension thresholds.

```protobuf
# Sample evaluation config to run evaluation in integrate mode for the given 3 class model,
# at every 10th epoch starting from the epoch 1.
evaluation_config {
  average_precision_mode: INTEGRATE
  validation_period_during_training: 10
  first_validation_epoch: 1
  minimum_detection_ground_truth_overlap {
    key: "car"
    value: 0.7
  }
  minimum_detection_ground_truth_overlap {
    key: "person"
    value: 0.5
  }
  minimum_detection_ground_truth_overlap {
    key: "bicycle"
    value: 0.5
  }
  evaluation_box_config {
    key: "car"
    value {
      minimum_height: 4
      maximum_height: 9999
      minimum_width: 4
      maximum_width: 9999
    }
  }
  evaluation_box_config {
    key: "person"
    value {
      minimum_height: 4
      maximum_height: 9999
      minimum_width: 4
      maximum_width: 9999
    }
  }
  evaluation_box_config {
    key: "bicycle"
    value {
      minimum_height: 4
      maximum_height: 9999
      minimum_width: 4
      maximum_width: 9999
    }
  }
}
```

--------------------------------

### SegFormer Evaluation Output Example

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/cv_finetuning/pytorch/segformer.html

Example of the mIoU evaluation metric output format for a SegFormer model.

```text
+------------+-------+-------+
| Class      | IoU   | Acc   |
+------------+-------+-------+
| foreground | 37.81 | 44.56 |
| background | 83.81 | 95.51 |
+------------+-------+-------+
Summary:

+--------+-------+-------+-------+
| Scope  | mIoU  | mAcc  | aAcc  |
+--------+-------+-------+-------+
| global | 60.81 | 70.03 | 85.26 |
+--------+-------+-------+-------+
  ...
```

--------------------------------

### Start Triton Inference Server

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/cv_finetuning/pytorch/panoptic_3d_reconstruction/nvpanoptix3d.html

Execute the provided shell script to initialize the Triton server for NvPanoptix3D.

```bash
bash scripts/nvpanoptix3d_e2e_inference/start_server.sh
```

--------------------------------

### Build Triton Server Docker Image

Source: https://docs.nvidia.com/tao/tao-toolkit/latest/text/ds_tao/nvocdr_ds.html

Execute the setup script to build the Triton server image with specific model parameters.

```bash
cd NVIDIA-Optical-Character-Detection-and-Recognition-Solution/triton
bash setup_triton_server.sh [OCD input height] [OCD input width] [OCD input max batchsize] [DEVICE] [ocd onnx path> [ocr onnx path] [ocr character list path]

# For example
bash setup_triton_server.sh 736 1280 4 0 model/ocd.onnx model/ocr.onnx model/ocr_character_list
```
Tokens:
Raw