NVIDIA Nemotron

https://docs.nvidia.com/nemotron/latest/
Admin
Open and efficient models for agentic AI with reproducible training pipelines featuring transparent...
Tokens:70,584
Snippets:931
Trust Score:10
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install Nemotron Training Recipes

Source: https://docs.nvidia.com/nemotron/latest/index.html

Clone the Nemotron training recipes repository to get started with installation.

```bash
git clone htt
```

--------------------------------

### Create Virtual Environment and Install

Source: https://docs.nvidia.com/nemotron/latest/nemotron/super3/evaluate.html

Sets up a dedicated virtual environment and installs the necessary `nemo-evaluator-launcher` package with all dependencies.

```bash
python -m venv eval-venv
source eval-venv/bin/activate
pip install "nemo-evaluator-launcher[all]"

```

--------------------------------

### Local CLI Execution Example

Source: https://docs.nvidia.com/nemotron/latest/nemotron/cli.html

Example command to run the 'myrecipe' training locally with a 'tiny' configuration.

```bash
uv run nemotron myrecipe train -c tiny
```

--------------------------------

### Training Override Examples

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/pretrain.html

Examples of overriding configuration values during pretraining.

```bash
# More training iterations
uv run nemotron nano3 pretrain train.train_iters=5000

```

```bash
# Larger batch size
uv run nemotron nano3 pretrain train.global_batch_size=64

```

```bash
# Different checkpoint location
uv run nemotron nano3 pretrain checkpoint.save=/path/to/checkpoints

```

--------------------------------

### Nemotron Nano3 SFT Local Execution

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/README.html

Runs supervised fine-tuning locally using the 'tiny' configuration. This is a basic example for starting an SFT job.

```bash
... sft -c tiny
```

--------------------------------

### Install Dependencies with uv pip

Source: https://docs.nvidia.com/nemotron/latest/use-case-examples/Intelligent Document Processing with Nemotron RAG/README.html

Install the required libraries for the IDP pipeline using uv pip. This example uses `nv-ingest` in Library Mode for local execution.

```bash
uv pip install -r pyproject.toml

```

--------------------------------

### NeMo-RL GRPO Training Example

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nvidia-stack.html

Starts a GRPO training run with a specified policy model and environments. Configure the number of rollouts for the training process.

```python
from nemo_rl.algorithms.grpo import GRPOConfig, run_grpo

config = GRPOConfig(
    policy_model="nvidia/Nemotron-3-Nano-8B-SFT",
    environments=["math", "code"],
    num_rollouts=32,
)
run_grpo(config)
```

--------------------------------

### OmegaConf Configuration Example

Source: https://docs.nvidia.com/nemotron/latest/nemo_runspec/omegaconf.html

Example of a configuration file using OmegaConf for Nemotron, including run settings, artifact references, environment configuration, and logger settings.

```yaml
# config.yaml
run:
  # Artifact references - automatically resolved with W&B lineage
  data: PretrainBlendsArtifact-default:latest
  model: ModelArtifact-pretrain:v5

  # Environment configuration (container image)
  env:
    container: nvcr.io/nvidia/nemo:25.11.nemotron_3_nano

# Training configuration sections
recipe:
  _target_: megatron.bridge.recipes.nemotronh.nemotron_next_3b_v2_pretrain_config
  per_split_data_args_path: ${art:data,path}/blend.json  # Resolved from artifact

logger:
  wandb_project: ${run.wandb.project}   # Injected from env.toml
  wandb_entity: ${run.wandb.entity}     # Injected from env.toml

checkpoint:
  save: /nemo_run/pretrain
  save_interval: 20


```

--------------------------------

### Full env.toml Configuration Example

Source: https://docs.nvidia.com/nemotron/latest/nemo_runspec/nemo-run.html

A comprehensive example of an env.toml file, demonstrating system sections (wandb, artifacts, cache) and various execution profiles (local, my-cluster, pretrain, pretrain-large, data-prep, rl).

```TOML
# env.toml

# --- System sections (not profiles) ---

[wandb]
project = "nemotron"
entity = "my-team"

[artifacts]
backend = "file"
root = "/lustre/users/me/artifacts"

[cache]
git_dir = "/lustre/users/me/git-cache"

# --- Execution profiles ---

[local]
executor = "local"
nproc_per_node = 8

[my-cluster]
executor = "slurm"
account = "my-account"
partition = "batch"
run_partition = "interactive"
batch_partition = "batch"
time = "04:00:00"
tunnel = "ssh"
host = "login.cluster.com"
user = "me"
remote_job_dir = "/lustre/users/me/run"

[pretrain]
extends = "my-cluster"
nodes = 2
ntasks_per_node = 8
gpus_per_node = 8
mem = "0"
exclusive = true
mounts = ["/lustre:/lustre"]

[pretrain-large]
extends = "pretrain"
nodes = 8
time = "08:00:00"

[data-prep]
extends = "my-cluster"
run_partition = "cpu"
batch_partition = "cpu"
nodes = 1
ntasks_per_node = 1
cpus_per_task = 96

[rl]
extends = "pretrain"
container_image = "/lustre/users/me/nano3-rl.sqsh"


```

--------------------------------

### NeMo-Run Configuration Example

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/pretrain.html

Example TOML configuration for NeMo-Run, including WandB and cluster settings.

```toml
[wandb]
project = "nemotron"
entity = "YOUR-TEAM"

[YOUR-CLUSTER]
executor = "slurm"
account = "YOUR-ACCOUNT"
partition = "batch"
nodes = 2
ntasks_per_node = 8
gpus_per_node = 8
mounts = ["/lustre:/lustre"]


```

--------------------------------

### Run OpenCode Agent

Source: https://docs.nvidia.com/nemotron/latest/usage-cookbook/Nemotron-3-Super/OpenScaffoldingResources/README.html

Navigates to a project directory and starts the OpenCode agent. The '/init' command analyzes the project on the first run.

```bash
cd /path/to/project
opencode
/init   # first run: analyzes the project and creates AGENTS.md

```

--------------------------------

### Install Apptainer and Download SWE Images

Source: https://docs.nvidia.com/nemotron/latest/nemotron/super3/rl/swe.html

Installs Apptainer on Ubuntu/Debian and downloads SWE container images to a specified directory. Ensure you have the necessary permissions and paths.

```bash
# Install Apptainer (Ubuntu/Debian)
wget https://github.com/apptainer/apptainer/releases/download/v1.3.1/apptainer_1.3.1_amd64.deb
sudo apt install -y ./apptainer_1.3.1_amd64.deb

# Download and convert Docker images to .sif files
./examples/nemo_gym/download_swe_images.py --sif-dir /path/to/sif --concurrency 16

```

--------------------------------

### Install vLLM

Source: https://docs.nvidia.com/nemotron/latest/usage-cookbook/Nemotron-3-Super/AdvancedDeploymentGuide/README.html

Installs the vLLM library with a specific version for Nemotron 3 Super deployment.

```bash
pip install vllm==0.17.1

```

--------------------------------

### Install OpenClaw Daemon

Source: https://docs.nvidia.com/nemotron/latest/usage-cookbook/Nemotron-3-Super/OpenScaffoldingResources/README.html

Installs OpenClaw and registers it as a background service. Requires Node.js >= 22 and an OpenRouter API key.

```bash
npm install -g openclaw@latest
openclaw onboard \
    --install-daemon \
    --auth-choice apiKey \
    --token-provider openrouter \
    --token "$OPENROUTER_API_KEY"

```

--------------------------------

### Runspec TOML Configuration Example

Source: https://docs.nvidia.com/nemotron/latest/runspec/v1/spec.html

This snippet shows the structure of a `[tool.runspec]` TOML table within PEP 723 inline script metadata. It defines schema, documentation link, name, container image, setup instructions, launch method, configuration directory, and resource defaults.

```toml
#!/usr/bin/env python3
# /// script
# [tool.runspec]
# schema = "1"
# docs = "https://raw.githubusercontent.com/NVIDIA-NeMo/Nemotron/main/docs/runspec/v1/spec.md"
# name = "nano3/pretrain"
# image = "nvcr.io/nvidia/nemo:25.11.nemotron_3_nano"
# setup = "NeMo and all training dependencies are pre-installed in the image."
#
# [tool.runspec.run]
# launch = "torchrun"
#
# [tool.runspec.config]
# dir = "./config"
# default = "default"
# format = "omegaconf"
#
# [tool.runspec.resources]
# nodes = 1
# gpus_per_node = 8
# ///

```

--------------------------------

### Remote Cluster Execution Example

Source: https://docs.nvidia.com/nemotron/latest/nemotron/cli.html

Example command to run the 'myrecipe' training on a cluster using NeMo-Run.

```bash
uv run nemotron myrecipe train -c tiny --run MY-CLUSTER
```

--------------------------------

### SFT Training Override Examples

Source: https://docs.nvidia.com/nemotron/latest/nemotron/super3/sft.html

Examples of overriding default configuration values for SFT training. These demonstrate how to modify training iterations, learning rate, and checkpoint loading.

```bash
# More training iterations
uv run nemotron super3 sft train.train_iters=5000

# Different learning rate
uv run nemotron super3 sft optimizer.lr=1e-5

# Load specific pretrained checkpoint
uv run nemotron super3 sft checkpoint.pretrained_checkpoint=/path/to/checkpoint
```

--------------------------------

### Environment Configuration TOML Example

Source: https://docs.nvidia.com/nemotron/latest/nemo_runspec/package-readme.html

Example of an env.toml file demonstrating TOML profiles with inheritance for environment configuration. Special sections like wandb, cli, cache, and artifacts are not executor profiles.

```toml
[base]
executor = "slurm"
account = "my-account"
remote_job_dir = "/lustre/jobs"

[dev]
extends = "base"
partition = "dev-gpu"
nodes = 1

[prod]
extends = "base"
partition = "prod-gpu"
nodes = 8

[wandb]
entity = "my-team"
project = "nemotron"

[artifacts]
wandb = true

[artifacts.manifest]
root = "/lustre/artifacts"

[cache]
git_dir = "/lustre/git-cache"

```

--------------------------------

### Container Path Example

Source: https://docs.nvidia.com/nemotron/latest/nemo_runspec/omegaconf.html

Illustrates the transformation of container paths to absolute filesystem paths for artifact metadata.

```shell
# Container path (inside job)
/nemo_run/pretrain/iter_0010000

# Resolved path (for artifact metadata)
/lustre/scratch/user/jobs/12345/pretrain/iter_0010000

```

--------------------------------

### Clone Nemotron Repository and Install Dependencies

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/README.html

Clone the Nemotron repository from GitHub and install the required Python packages using uv.

```bash
git clone https://github.com/NVIDIA/nemotron
cd nemotron
uv sync

```

--------------------------------

### W&B Authentication Setup

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/README.html

Logs into Weights & Biases for experiment tracking. Ensure W&B integration is set up before running this command.

```bash
wandb login
```

--------------------------------

### Install PyTorch First (Standard CUDA)

Source: https://docs.nvidia.com/nemotron/latest/use-case-examples/nemotron-voice-rag-agent-example/README.html

Manually install PyTorch with CUDA 12.4 support before syncing other dependencies on standard CUDA systems to resolve potential conflicts.

```bash
# For standard CUDA systems
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124
uv sync

```

--------------------------------

### Create Recipe Script with Runspec

Source: https://docs.nvidia.com/nemotron/latest/nemotron/cli.html

Example of creating a Python recipe script that includes a TOML runspec configuration for Nemotron.

```python
# src/nemotron/recipes/myrecipe/train.py

# /// script
# [tool.runspec]
# name = "myrecipe/train"
# image = "nvcr.io/nvidia/nemo:25.11"
#
# [tool.runspec.run]
# launch = "torchrun"
#
# [tool.runspec.config]
# dir = "./config"
# default = "default"

```

--------------------------------

### Nemotron Profile Inheritance Configuration

Source: https://docs.nvidia.com/nemotron/latest/nemo_runspec/nemo-run.html

Example `env.toml` demonstrating profile inheritance, where 'YOUR-CLUSTER' and 'YOUR-CLUSTER-large' extend a 'base-slurm' profile.

```toml
[base-slurm]
executor = "slurm"
account = "my-account"
partition = "gpu"
time = "04:00:00"

[YOUR-CLUSTER]
extends = "base-slurm"
nodes = 4
ntasks_per_node = 8
gpus_per_node = 8

[YOUR-CLUSTER-large]
extends = "YOUR-CLUSTER"
nodes = 16
time = "08:00:00"

```

--------------------------------

### Model Import SFT Command Help

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/import.html

Displays help information for importing SFT models with Nemotron Nano3.

```bash
uv run nemotron nano3 model import sft --help
```

--------------------------------

### Run Nemotron 3 Nano Pretraining

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/pretrain.html

Starts the pretraining job for Nemotron 3 Nano using NeMo-Run. This command assumes data preparation is complete.

```bash
uv run nemotron nano3 pretrain --run YOUR-CLUSTER
```

--------------------------------

### Pretraining/SFT Script with W&B Patches

Source: https://docs.nvidia.com/nemotron/latest/nemo_runspec/omegaconf.html

Example of integrating W&B patches for lineage and checkpoint logging in a Megatron-Bridge pretraining script.

```python
from nemo_runspec.config.resolvers import register_resolvers_from_config
from nemotron.kit.wandb_kit import (
    patch_wandb_checkpoint_logging,
    patch_wandb_init_for_lineage,
)

def main():
    config = OmegaConf.load("config.yaml")

    # Resolve artifacts before wandb.init()
    qualified_names = register_resolvers_from_config(
        config,
        artifacts_key="run",
        mode="pre_init",
    )

    # Patch for lineage and checkpoint logging
    patch_wandb_init_for_lineage(artifact_qualified_names=qualified_names)
    patch_wandb_checkpoint_logging()

    # Now Megatron-Bridge handles the rest
    pretrain(config=cfg, forward_step_func=forward_step)


```

--------------------------------

### Configure OpenHands for Custom NVIDIA Model

Source: https://docs.nvidia.com/nemotron/latest/usage-cookbook/Nemotron-3-Super/OpenScaffoldingResources/README.html

Launches OpenHands and guides through settings to configure a custom model ('nvidia/nemotron-3-super-120b-a12b') and base URL ('https://integrate.api.nvidia.com/v1').

```bash
openhands

```

```bash
/settings

```

--------------------------------

### Nemotron Nano3 SFT Command Help

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/README.html

Shows detailed help for the supervised fine-tuning (SFT) command, including options for configuration, cluster submission, and artifact overrides.

```bash
uv run nemotron nano3 sft --help
```

--------------------------------

### Troubleshooting: Install nemo-evaluator-launcher

Source: https://docs.nvidia.com/nemotron/latest/nemotron/super3/evaluate.html

Provides the command to install `nemo-evaluator-launcher` if it is not found, typically used when troubleshooting installation issues.

```bash
pip install "nemotron[evaluator]"

```

--------------------------------

### Install Dependencies with uv

Source: https://docs.nvidia.com/nemotron/latest/use-case-examples/Simple Nemotron-3-Nano Usage Example/README.html

Install project dependencies using the uv package manager. Ensure you have Python 3.13+ and a uv installation.

```bash
uv sync
```

--------------------------------

### Run Jupyter Notebook Tutorial

Source: https://docs.nvidia.com/nemotron/latest/use-case-examples/nemotron-voice-rag-agent-example/README.html

To run the tutorial, navigate to the project directory and execute the specified Jupyter notebook using the 'jupyter notebook' command.

```bash
jupyter notebook voice_rag_agent_tutorial.ipynb

```

--------------------------------

### Import Pretrain Model with Custom Name and Step

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/import.html

Example of importing a pretrain model checkpoint with a custom artifact name and specifying the training step.

```bash
uv run nemotron nano3 model import pretrain /lustre/checkpoints/model --step 50000 --name my-pretrain-model
```

--------------------------------

### Model Import Pretrain Command Help

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/import.html

Displays help information for importing pretrain models with Nemotron Nano3.

```bash
uv run nemotron nano3 model import pretrain --help
```

--------------------------------

### Data Import SFT Command Help

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/import.html

Displays help information for importing SFT data with Nemotron Nano3.

```bash
uv run nemotron nano3 data import sft --help
```

--------------------------------

### Install Nemotron Evaluator

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/evaluate.html

Install the nemotron[evaluator] package to resolve the 'nemo-evaluator-launcher' not found error.

```bash
pip install "nemotron[evaluator]"
```

--------------------------------

### Data Import Pretrain Command Help

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/import.html

Displays help information for importing pretrain data with Nemotron Nano3.

```bash
uv run nemotron nano3 data import pretrain --help
```

--------------------------------

### Runspec Environment Variables Example

Source: https://docs.nvidia.com/nemotron/latest/runspec/v1/spec.html

Example of setting environment variables within the runspec configuration.

```toml
[tool.runspec.env]
NCCL_NVLS_ENABLE = "0"

```

--------------------------------

### HybridPackager Code Configuration Example

Source: https://docs.nvidia.com/nemotron/latest/nemo_runspec/nemo-run.html

Example of configuring a HybridPackager in Python, combining GitArchivePackager and PatternPackager.

```python
import nemo_run as run

hybrid = run.HybridPackager(
    sub_packagers={
        "code": run.GitArchivePackager(subpath="src"),
        "configs": run.PatternPackager(include_pattern="configs/*.yaml"),
        "data": run.PatternPackager(include_pattern="data/*.json"),
    }
)


```

--------------------------------

### Model Import RL Command Help

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/import.html

Displays help information for importing RL models with Nemotron Nano3.

```bash
uv run nemotron nano3 model import rl --help
```

--------------------------------

### Data Preparation with super_launch.sh

Source: https://docs.nvidia.com/nemotron/latest/nemotron/super3/rl/index.html

Download, fill placeholders, and split RL data blends for training and validation. Ensure the script is executable.

```bash
# Download RL data blends (rlvr1, rlvr2, rlvr3, swe1, swe2, rlhf)
uvx --from huggingface-hub hf download nvidia/Nemotron-3-Super-RL-Training-Blends \
    --repo-type dataset --local-dir=data_with_placeholders

# Fill in placeholders in data blends
chmod +x data_with_placeholders/fill_placeholders.py
./data_with_placeholders/fill_placeholders.py \
    --input-dir data_with_placeholders --output-dir data_filled

# Create train/val splits for each data blend (last 100 rows held out for validation)
for f in data_filled/*.jsonl; do
  name=$(basename "$f" .jsonl)
  mkdir -p "data/$name"
  head -n -100 "$f" > "data/$name/train-split.jsonl"
  tail -n 100 "$f" > "data/$name/val-split.jsonl"
done

```

--------------------------------

### Data Import RL Command Help

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/import.html

Displays help information for importing RL data with Nemotron Nano3.

```bash
uv run nemotron nano3 data import rl --help
```

--------------------------------

### Install OpenHands CLI

Source: https://docs.nvidia.com/nemotron/latest/usage-cookbook/Nemotron-3-Super/OpenScaffoldingResources/README.html

Installs the OpenHands CLI tool using uv. Requires Python 3.12.

```bash
uv tool install openhands --python 3.12

```

--------------------------------

### Model Import Command Help

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/import.html

Displays help information for the general Nemotron Nano3 model import command.

```bash
uv run nemotron nano3 model import --help
```

--------------------------------

### CLI Help for Nemotron Super3

Source: https://docs.nvidia.com/nemotron/latest/nemotron/super3/README.html

Displays available commands and training stages for the Nemotron Super3 recipe.

```bash
nemotron super3 --help
```

--------------------------------

### Override Checkpoint Save Location Example

Source: https://docs.nvidia.com/nemotron/latest/nemotron/super3/pretrain.html

Example of overriding the default checkpoint save directory using the CLI.

```bash
uv run nemotron super3 pretrain checkpoint.save=/path/to/checkpoints
```

--------------------------------

### Override Training Iterations Example

Source: https://docs.nvidia.com/nemotron/latest/nemotron/super3/pretrain.html

Example of overriding the default training iterations to a higher value using the CLI.

```bash
uv run nemotron super3 pretrain train.train_iters=5000
```

--------------------------------

### Troubleshooting: List Available Tasks

Source: https://docs.nvidia.com/nemotron/latest/nemotron/super3/evaluate.html

Demonstrates how to list all available tasks that can be run with `nemo-evaluator-launcher`, useful for identifying correct task names.

```bash
nemo-evaluator-launcher ls tasks

```

--------------------------------

### Running Ray-Enabled Recipes

Source: https://docs.nvidia.com/nemotron/latest/nemo_runspec/nemo-run.html

Examples of how to run Ray-enabled recipes using the 'uv run' command. These commands demonstrate initiating data preparation and RL training with distributed execution managed by Ray.

```Shell
# Data prep uses Ray
uv run nemotron nano3 data prep pretrain --run YOUR-CLUSTER

# RL training uses Ray
uv run nemotron nano3 rl -c tiny --run YOUR-CLUSTER


```

--------------------------------

### Clone Nemotron Repository

Source: https://docs.nvidia.com/nemotron/latest/use-case-examples/nemotron-voice-rag-agent-example/README.html

Clone the Nemotron repository to access the voice-powered RAG agent example. Navigate into the example's directory.

```bash
git clone https://github.com/NVIDIA-NeMo/Nemotron.git
cd Nemotron/use-case-examples/nemotron-voice-rag-agent-example

```

--------------------------------

### Override Global Batch Size Example

Source: https://docs.nvidia.com/nemotron/latest/nemotron/super3/pretrain.html

Example of overriding the default global batch size to a larger value using the CLI.

```bash
uv run nemotron super3 pretrain train.global_batch_size=64
```

--------------------------------

### Data Import Command Help

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/import.html

Displays help information for the general Nemotron Nano3 data import command.

```bash
uv run nemotron nano3 data import --help
```

--------------------------------

### Install Dependencies with PyTorch (DGX Spark)

Source: https://docs.nvidia.com/nemotron/latest/use-case-examples/nemotron-voice-rag-agent-example/README.html

Install project dependencies, including PyTorch with CUDA 13.0 support, for DGX Spark systems.

```bash
uv sync --extra cuda --index-url https://download.pytorch.org/whl/cu130

```

--------------------------------

### Install Dependencies with PyTorch (Standard CUDA)

Source: https://docs.nvidia.com/nemotron/latest/use-case-examples/nemotron-voice-rag-agent-example/README.html

Install project dependencies, including PyTorch with CUDA 12.4 support, for standard NVIDIA GPUs.

```bash
uv sync --extra cuda --index-url https://download.pytorch.org/whl/cu124

```

--------------------------------

### Install OpenCode AI Agent

Source: https://docs.nvidia.com/nemotron/latest/usage-cookbook/Nemotron-3-Super/OpenScaffoldingResources/README.html

Installs the OpenCode AI agent globally using npm. This is the first step to using OpenCode for terminal coding.

```bash
npm install -g opencode-ai

```

--------------------------------

### Run Nemotron 3 Nano Data Preparation and Supervised Fine-Tuning (SFT)

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/README.html

Execute the data preparation and supervised fine-tuning (SFT) stages for the Nemotron 3 Nano model using the 'uv run nemotron nano3' command with cluster execution.

```bash
uv run nemotron nano3 data prep sft --run YOUR-CLUSTER
```

```bash
uv run nemotron nano3 sft --run YOUR-CLUSTER
```

--------------------------------

### Install Kilo Code CLI

Source: https://docs.nvidia.com/nemotron/latest/usage-cookbook/Nemotron-3-Super/OpenScaffoldingResources/README.html

Installs the Kilo Code CLI globally using npm. This tool allows agentic coding tasks from the command line.

```bash
npm install -g @kilocode/cli

```

--------------------------------

### Job YAML Structure Example

Source: https://docs.nvidia.com/nemotron/latest/nemotron/cli.html

Illustrates the structure of a 'job.yaml' file, which includes the recipe, training parameters, run configuration, CLI arguments, and Weights & Biases settings.

```yaml
recipe:
  _target_: megatron.bridge.recipes...
  per_split_data_args_path: /data/blend.json
train:
  train_iters: 1000
run:
  mode: "run"
  profile: "my-cluster"
  env:
    executor: "slurm"
    nodes: 4
    gpus_per_node: 8
  cli:
    argv: ["nemotron", "nano3", "pretrain", "-c", "tiny", "--run", "my-cluster"]
    dotlist: ["train.train_iters=1000"]
  wandb:
    entity: "nvidia"
    project: "nemotron"
```

--------------------------------

### Nemotron Nano3 Help

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/README.html

Displays available commands for Nemotron Nano3, including data curation, model operations, training stages, evaluation, and pipeline composition.

```bash
uv run nemotron nano3 --help
```

--------------------------------

### Example JSONL Log Entry

Source: https://docs.nvidia.com/nemotron/latest/nemotron/xenna-observability.html

An example of a single log entry in the JSONL file, containing timestamp, pipeline kind, run hash, and various metrics.

```json
{
  "timestamp": 1706123456.789,
  "pipeline_kind": "pretrain",
  "run_hash": "abc123",
  "metrics": {
    "pipeline_duration_s": 120.5,
    "progress": 50.0,
    "cluster/total_cpus": 64.0,
    "stages/tasks_completed/download": 100
  },
  "stages": ["PlanStage", "DownloadStage", "BinIdxTokenizationStage"]
}

```

--------------------------------

### Model Commands Help

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/import.html

Displays help information for Nemotron Nano3 model commands.

```bash
uv run nemotron nano3 model --help
```

--------------------------------

### Install PyTorch First (DGX Spark)

Source: https://docs.nvidia.com/nemotron/latest/use-case-examples/nemotron-voice-rag-agent-example/README.html

Manually install PyTorch with CUDA 13.0 support before syncing other dependencies on DGX Spark systems to resolve potential conflicts.

```bash
# For Spark/GB10 systems
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130
uv sync

```

--------------------------------

### OpenAI Chat Input Format Example

Source: https://docs.nvidia.com/nemotron/latest/nemotron/data-prep.html

Example of the input data format for chat SFT/RL training, structured as a JSON object with a 'messages' array containing role-based content.

```json
{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"},
    {"role": "assistant", "content": "Hi there!"}
  ]
}


```

--------------------------------

### Install RAPIDS 25.10 with CUDA 13.0

Source: https://docs.nvidia.com/nemotron/latest/use-case-examples/Data%20Science%20ML%20Agent/README.html

Use this command to create a conda environment with RAPIDS 25.10 and Python 3.11, compatible with CUDA 13.0. Ensure you have the necessary CUDA toolkit installed.

```bash
conda create -n rapids-25.10 -c rapidsai -c conda-forge -c nvidia  \
    rapids=25.10 python=3.11 'cuda-version=13.0'

```

--------------------------------

### Import Supervised Fine-Tuning Model

Source: https://docs.nvidia.com/nemotron/latest/nemotron/artifacts.html

Import a supervised fine-tuning (SFT) model using the `nemotron nano3 model import sft` command. Provide the path to the SFT model and the step number.

```bash
uv run nemotron nano3 model import sft /path/to/sft_model --step 10000
```

--------------------------------

### Run Nemotron Pretraining Locally and on Cluster

Source: https://docs.nvidia.com/nemotron/latest/architecture/README.html

Commands to run pretraining locally using 'tiny' configuration and submit to a cluster. Also shows how to view execution logic.

```bash
# Run pretraining locally
nemotron nano3 pretrain -c tiny

# Submit to cluster
nemotron nano3 pretrain -c tiny --run dgx

# See what's happening (execution logic is visible in the code)
# Open: src/nemotron/cli/commands/nano3/pretrain.py

```

--------------------------------

### Project Structure Overview

Source: https://docs.nvidia.com/nemotron/latest/use-case-examples/nemotron-voice-rag-agent-example/README.html

This outlines the directory structure of the Nemotron Voice RAG Agent example project, including the main tutorial notebook, README, dependencies, and blog content.

```text
nemotron-voice-rag-agent-example/
├── voice_rag_agent_tutorial.ipynb  # Main tutorial notebook
├── README.md                        # This file
├── requirements.txt                 # Python dependencies
└── BlogSkeleton/                    # Blog content and model docs
    ├── BLOG.md
    ├── BLOG_UPDATED.md
    ├── Code Snippets/
    └── Model Information/

```

--------------------------------

### Run RL Training

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/rl.html

Executes RL training using Nemotron Nano3. Use '--run <profile>' to submit and wait for logs, or '--batch <profile>' to submit and exit immediately. '--dry-run' previews the execution plan.

```bash
uv run nemotron nano3 rl [options] [overrides...]
```

--------------------------------

### Slurm Execution Profile Example

Source: https://docs.nvidia.com/nemotron/latest/nemo_runspec/nemo-run.html

An example of a Slurm execution profile configuration in TOML format. This profile specifies executor type, account, partitions, time limits, and resource allocations for a Slurm cluster.

```TOML
[my-cluster]
executor = "slurm"
account = "my-account"
partition = "batch"
run_partition = "interactive"
batch_partition = "batch"
tunnel = "ssh"
host = "cluster.example.com"
user = "username"
remote_job_dir = "/lustre/users/me/run"
time = "04:00:00"
nodes = 2
ntasks_per_node = 8
gpus_per_node = 8
mem = "0"
exclusive = true
container_image = "/path/to/image.sqsh"
mounts = ["/lustre:/lustre"]
startup_commands = ["source /opt/env.sh"]


```

--------------------------------

### Multiplication Resolver Example

Source: https://docs.nvidia.com/nemotron/latest/nemo_runspec/omegaconf.html

Demonstrates a custom multiplication resolver for NeMo-RL configuration.

```yaml
# Multiplication resolver (NeMo-RL)
train_mb_tokens: ${mul:${policy.max_total_sequence_length}, ${policy.train_micro_batch_size}}


```

--------------------------------

### Run Nemotron 3 Nano Data Preparation and Reinforcement Learning (RL)

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/README.html

Execute the data preparation and reinforcement learning (RL) stages for the Nemotron 3 Nano model using the 'uv run nemotron nano3' command with cluster execution. Note that RL requires separate execution.

```bash
uv run nemotron nano3 data prep rl --run YOUR-CLUSTER
```

```bash
uv run nemotron nano3 rl --run YOUR-CLUSTER
```

--------------------------------

### Data Preparation Pipeline Overview

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/rl.html

This table outlines the steps involved in preparing the RL dataset for NeMo-Gym, including loading from HuggingFace, resolving placeholders, formatting to JSONL, splitting, and reward computation.

```markdown
Stage | What Happens  
---|---
**HuggingFace Dataset** | Load Nemotron-3-Nano-RL-Training-Blend from HuggingFace Hub  
**Placeholder Resolution** | Resolve `_hf_placeholder` records by fetching from external datasets (DAPO, Skywork) and applying template restoration  
**JSONL Format** | Convert to JSONL with `question`, `expected_answer`, and `responses_create_params` fields  
**Train/Val/Test Split** | Split into training (98%), validation (1%), and test (1%) sets  
**NeMo-Gym Environment** | Route samples to appropriate reward environments based on task type  
**Reward Computation** | Compute verifiable rewards (math correctness, code execution, schema adherence)  
```

--------------------------------

### Disable Sequence Packing

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/rl.html

Example of disabling sequence packing in the policy configuration during RL training.

```bash
uv run nemotron nano3 rl policy.sequence_packing.enabled=false
```

--------------------------------

### Override Generation Temperature

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/rl.html

Example of overriding the generation temperature for policy configuration during RL training.

```bash
uv run nemotron nano3 rl policy.generation.temperature=0.8
```

--------------------------------

### Import Supervised Fine-Tuning Data

Source: https://docs.nvidia.com/nemotron/latest/nemotron/artifacts.html

Import supervised fine-tuning (SFT) data using the `nemotron nano3 data import sft` command. Provide the path to the SFT data directory.

```bash
uv run nemotron nano3 data import sft /path/to/sft_data/
```

--------------------------------

### Override Training Steps

Source: https://docs.nvidia.com/nemotron/latest/nemotron/nano3/rl.html

Example of overriding the maximum number of training steps for RL training.

```bash
uv run nemotron nano3 rl grpo.max_num_steps=200000
```

--------------------------------

### NeMo-Run Attached Execution

Source: https://docs.nvidia.com/nemotron/latest/nemotron/cli.html

Example of submitting a job to NeMo-Run and waiting for its completion while streaming logs.

```bash
uv run nemotron nano3 pretrain -c tiny --run MY-CLUSTER
```

--------------------------------

### Ray Jobs for Data Prep and RL

Source: https://docs.nvidia.com/nemotron/latest/nemotron/cli.html

Demonstrates submitting data preparation and RL training jobs via Ray, which are configured with `launch = "ray"` in their runspec.

```bash
# Data prep uses Ray for distributed processing
uv run nemotron nano3 data prep pretrain --run MY-CLUSTER

# RL uses Ray for actor orchestration
uv run nemotron nano3 rl -c tiny --run MY-CLUSTER
```
Tokens:
Raw