### Setup NeMo Evaluator Environment

Source: https://github.com/nvidia-nemo/evaluator/blob/main/CONTRIBUTING.md

Clone the repository, create a virtual environment, and install the project with development dependencies.

```bash
git clone <repo-url> && cd nemo-evaluator-next
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
```

--------------------------------

### Start NEL Environment Server

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/tutorials/gym-integration.mdx

Start the NEL environment server to serve benchmarks for Gym training. Use the `-b` flag to specify the benchmark and `-p` for the port.

```bash
nel serve -b gsm8k -p 9090
```

--------------------------------

### Serve for Gym Compatibility

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/deployment/local.mdx

Start a local server compatible with Gym using 'nel serve'. Specify the benchmark and port for the service.

```bash
nel serve -b gsm8k --gym-compat --port 9090
```

--------------------------------

### List Available Benchmarks

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/tutorials/skills-integration.mdx

Execute this command to display a list of all available benchmarks, including built-in ones and any installed Skills benchmarks.

```bash
nel list  # shows built-in + skills if installed
```

--------------------------------

### Install NeMo Skills and Prepare Data

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/tutorials/skills-integration.mdx

Install the necessary NeMo Skills package and prepare datasets for benchmarks like GPQA and AIME24. Ensure you have the correct package installed, either editable or via pip.

```bash
pip install -e "'.[skills]'"  # or: pip install nemo-skills
ns prepare_data gpqa
ns prepare_data aime24
```

--------------------------------

### Evaluation Bundle Example (eval-*.json)

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/result-format.mdx

An example of a JSON file representing a benchmark evaluation, including run ID, configuration, benchmark name, and metric scores with confidence intervals.

```json
{
  "run_id": "eval-20260402T120000Z-nemotron-super-bf16",
  "config": {"benchmark": "mmlu_pro", "model": "nemotron-super-120b"},
  "benchmark": {
    "name": "mmlu_pro",
    "samples": 12000,
    "scores": {
      "mean_reward": {
        "value": 0.782,
        "ci_lower": 0.774,
        "ci_upper": 0.790
      }
    },
    "categories": {
      "algebra": {"mean_reward": 0.85, "n": 500},
      "geometry": {"mean_reward": 0.72, "n": 400}
    }
  }
}
```

--------------------------------

### Install NeMo Evaluator with All Integrations

Source: https://github.com/nvidia-nemo/evaluator/blob/main/README.md

Install NeMo Evaluator with all common runtime integrations.

```bash
pip install -e ".[all]"
```

--------------------------------

### Install NeMo Evaluator Launcher from Source

Source: https://github.com/nvidia-nemo/evaluator/blob/main/packages/nemo-evaluator-launcher/examples/nemotron/nemotron-3-super/reproducibility.md

Install the NeMo Evaluator Launcher from its source repository. This involves cloning the repository and performing a local installation.

```bash
git clone https://github.com/NVIDIA-NeMo/Evaluator.git
cd Evaluator/packages/nemo-evaluator-launcher
pip install -e .
```

--------------------------------

### Quick Testing of Base Model Configuration

Source: https://github.com/nvidia-nemo/evaluator/blob/main/packages/nemo-evaluator-launcher/examples/nemotron/nano-v3-reproducibility.md

Perform a quick test of the base model configuration by limiting the number of samples evaluated. This is useful for verifying setup without running a full benchmark suite. Remember to remove `limit_samples` for actual benchmark results.

```bash
nemo-evaluator-launcher run \
  --config local_nvidia-nemotron-3-nano-30b-a3b-base.yaml \
  -o evaluation.nemo_evaluator_config.config.params.limit_samples=10
```

--------------------------------

### Verify NeMo Evaluator Installation

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/get-started/install.mdx

Check if the installation was successful by running the `nel --version` command to see the installed version and `nel list` to view available benchmarks. Ensure your environment is activated.

```bash
nel --version
nel list
```

--------------------------------

### Nemo CLI Examples for Executors

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/architecture/index.mdx

Illustrates how to invoke the Nemo CLI for different executor types: local with an external API, Docker, and SLURM.

```bash
# Local with external API
nel eval run --bench mmlu --model-url https://api.example.com/v1

```

```bash
# Docker
nel eval run config.yaml    # with cluster.type: docker

```

```bash
# SLURM (generates + submits sbatch)
nel eval run config.yaml    # with cluster.type: slurm

```

--------------------------------

### Flat Directory Layout Example

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/result-format.mdx

Illustrates a flat directory structure where evaluation bundles and per-problem records reside in the root directory.

```text
results/
  eval-mmlu_pro.json
  results.jsonl           # per-problem records for the single benchmark
```

--------------------------------

### Serve Model HTTP Server Command

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/api/index.mdx

Starts an HTTP server for an environment, allowing model serving. Can be configured with port, host, and compatibility options.

```bash
nel serve
    --bench, -b TEXT         Benchmark to serve
    --port INT               Port [9090]
    --host TEXT              Host [0.0.0.0]
    --gym-compat             Enable Gym-compatible endpoints
    --export-data TEXT       Export JSONL to path and exit
```

--------------------------------

### Build GDPVal Apptainer SIF

Source: https://github.com/nvidia-nemo/evaluator/blob/main/examples/nemotron/nemotron-3-ultra/v0.2/README.md

Build the Apptainer SIF image required for the GDPVal agent. This is a one-time setup step.

```bash
apptainer build python-3.12.gdpval.sif \
  https://raw.githubusercontent.com/NVIDIA-NeMo/Gym/2502893977e9e9af84adc1fa8d38c9314208d3ee/responses_api_agents/stirrup_agent/containers/gdpval.def
```

--------------------------------

### Install NeMo Evaluator and Set API Key

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/tutorials/walkthrough.mdx

Install the NeMo Evaluator package with scoring dependencies and set your NVIDIA API key as an environment variable.

```bash
pip install -e ".[scoring]"
export NVIDIA_API_KEY="your-api-key-here"
```

--------------------------------

### Nested Directory Layout Example

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/result-format.mdx

Shows a nested directory structure where each benchmark has its own subdirectory containing its evaluation bundle and per-problem records.

```text
results/
  mmlu_pro/
    eval-mmlu_pro.json
    results.jsonl
  gpqa/
    eval-gpqa.json
    results.jsonl
```

--------------------------------

### Serve Benchmark with Evaluator Protocol

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/deployment-patterns.mdx

Serve a benchmark using the default evaluator protocol, suitable for `nel eval run --adapter`. This command starts an HTTP service for the specified benchmark.

```bash
nel serve -b gsm8k --port 9090
```

--------------------------------

### Example Commit Message

Source: https://github.com/nvidia-nemo/evaluator/blob/main/AGENTS.md

Use clear, imperative commit messages following the format '<type>: <short imperative description>'. Optionally include an issue reference.

```text
feat: add server_cmd support for native Gym servers on SLURM

The sbatch generator now honors ServiceConfig.server_cmd for gym
services instead of hardcoding `nel serve`. This enables running
any server that speaks /seed_session + /verify as a managed service.
```

--------------------------------

### Configure Apptainer Execution Arguments

Source: https://github.com/nvidia-nemo/evaluator/blob/main/examples/nemotron/nemotron-3-ultra/v0.2/README.md

Example Apptainer execution configuration for mounting the SIF image and other necessary volumes. This is typically part of an Apptainer configuration file.

```yaml
execution:
  extra_docker_args: >-
    --privileged
    -v /you/python-3.12.gdpval.sif:/gdpval/sif/python-3.12.gdpval.sif
    # ...other mounts
```

--------------------------------

### Dry Run Full Nemotron-3 Super Evaluation

Source: https://github.com/nvidia-nemo/evaluator/blob/main/packages/nemo-evaluator-launcher/examples/nemotron/nemotron-3-super/reproducibility.md

Preview the configuration and planned evaluation steps without executing the actual benchmarks. This is useful for verifying the setup.

```bash
nemo-evaluator-launcher run \
  --config local_nemotron-3-super-120b-a12b.yaml \
  --dry-run
```

--------------------------------

### Inherit from a Base Configuration

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/tutorials/config-composition.mdx

Instead of duplicating extensive configuration, inherit from a base configuration file using the `defaults:` key. This example shows a significantly reduced configuration file size by inheriting from `24_slurm_swebench_verified_super`.

```yaml
defaults:
  - _base_: 24_slurm_swebench_verified_super

benchmarks:
  - name: harbor://swebench-verified@1.0
    max_concurrent: 30
    solver:
      agent: mini-swe-agent
      agent_kwargs: { max_iterations: 30 }
    sandbox: { concurrency: 30 }

output:
  dir: ./eval_results/swebench-verified-miniswe
```

--------------------------------

### Quantization Release Gate Policy Example

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/tutorials/quality-gate.mdx

A sample gate policy configuration for qualifying NVFP4/INT4 checkpoints against a BF16 baseline. It defines default metrics and thresholds, with specific overrides for critical benchmarks.

```yaml
version: 1
defaults:
  tier: supporting
  metric: mean_reward
  max_drop: 0.015
benchmarks:
  mmlu_pro:
    tier: critical
    max_drop: 0.01
  gpqa:
    tier: critical
    max_drop: 0.01
  aime_2025:
    tier: critical
    max_drop: 0.01
    max_relative_drop: 0.02
    relative_guard_below: 0.15
  aa_omniscience:
    tier: critical
    max_drop: 0.01
  scicode:
    metric: pass@1
  aa_lcr: {}
  ifeval:
    tier: advisory
```

--------------------------------

### Install NeMo Evaluator with Specific Extras

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/get-started/install.mdx

Install the NeMo Evaluator with specific optional features by appending the extra name in square brackets to the pip install command. For example, to install with scoring capabilities, use '.[scoring]'.

```bash
pip install -e ".[scoring]"
```

```bash
pip install -e ".[stats]"
```

```bash
pip install -e ".[skills]"
```

```bash
pip install -e ".[harbor]"
```

```bash
pip install -e ".[lm-eval]"
```

```bash
pip install -e ".[inspect]"
```

```bash
pip install -e ".[ray]"
```

```bash
pip install -e ".[harnesses]"
```

```bash
pip install -e ".[export]"
```

```bash
pip install -e ".[docs]"
```

```bash
pip install -e ".[dev]"
```

--------------------------------

### Initialize GymEnvironment

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/api/index.mdx

Use this to consume `nel serve` endpoints or Gym resource servers. Ensure the client is properly initialized.

```python
from nemo_evaluator.environments.gym import GymEnvironment
from nemo_evaluator.solvers import ChatSolver

env = GymEnvironment("http://localhost:9090")
solver = ChatSolver(client)
bundle = await run_evaluation(env, solver, n_repeats=4)
```

--------------------------------

### Install NeMo Evaluator Launcher

Source: https://github.com/nvidia-nemo/evaluator/blob/main/packages/nemo-evaluator-launcher/examples/nemotron/nano-v3-reproducibility.md

Install the NeMo Evaluator Launcher using pip. Alternatively, install from source by cloning the repository and performing a local installation.

```bash
pip install nemo-evaluator-launcher
```

```bash
git clone https://github.com/NVIDIA-NeMo/Evaluator.git
cd Evaluator/packages/nemo-evaluator-launcher
pip install -e .
```

--------------------------------

### Run Nemotron-3 Ultra Instruct Suite

Source: https://github.com/nvidia-nemo/evaluator/blob/main/examples/nemotron/nemotron-3-ultra/v0.2/README.md

Execute the full instruct suite for Nemotron-3 Ultra using a specified configuration and environment file. Use `--dry-run` to preview the execution without actually running.

```bash
nel run --config local_nemotron-3-ultra-550b-a55b.yaml --env-file .env             # full instruct suite
nel run --config local_nemotron-3-ultra-550b-a55b.yaml --env-file .env --dry-run   # preview
nel run --config local_nemotron-3-ultra-550b-a55b.yaml --env-file .env -t ns_gpqa  # one instruct benchmark
nel run --config local_nemotron-3-ultra-550b-a55b-base.yaml --env-file .env        # base suite (local vLLM, BF16-Base, TP=8)
```

--------------------------------

### Install NeMo Evaluator from Source

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/get-started/install.mdx

Clone the repository and install the package in editable mode with all common dependencies. Ensure you have Python 3.12 or 3.13 installed.

```bash
git clone https://github.com/NVIDIA-NeMo/Evaluator.git
cd Evaluator
pip install -e ".[all]"
```

--------------------------------

### Run Specific Benchmark (LiveCodeBench)

Source: https://github.com/nvidia-nemo/evaluator/blob/main/packages/nemo-evaluator-launcher/examples/nemotron/nano-v3-reproducibility.md

Use the `-t` flag to run a single specified benchmark. Ensure you are in the 'examples/nemotron' directory.

```bash
nemo-evaluator-launcher run --config local_nvidia_nemotron_3_nano_30b_a3b.yaml -t ns_livecodebench
```

--------------------------------

### Install nemo-evaluator-launcher

Source: https://github.com/nvidia-nemo/evaluator/blob/main/examples/nemotron/nemotron-3-ultra/v0.2/README.md

Installs the nemo-evaluator-launcher with all extras for version 0.2.6 using a Python virtual environment.

```bash
python3 -m venv .venv
.venv/bin/pip install "nemo-evaluator-launcher[all]==0.2.6"
```

--------------------------------

### Run Full Tool Usage Evaluation

Source: https://github.com/nvidia-nemo/evaluator/blob/main/packages/nemo-evaluator-launcher/examples/nemotron/nemotron-3-super/reproducibility.md

Execute the complete suite of tool usage evaluations using the specified configuration file. This command initiates all configured tool-enabled benchmarks.

```bash
nemo-evaluator-launcher run --config local_nemotron-3-super-120b-a12b_tools.yaml
```

--------------------------------

### Run MMLU-Pro X Benchmark

Source: https://github.com/nvidia-nemo/evaluator/blob/main/examples/nemotron/nemotron-3-ultra/v0.2/README.md

Execute the MMLU-Pro X benchmark. Recommended setting is num_repeats: 1.

```bash
nel run --config local_nemotron-3-ultra-550b-a55b.yaml --env-file .env -t ns_mmlu_prox
```

--------------------------------

### Install NeMo Evaluator with Stats

Source: https://github.com/nvidia-nemo/evaluator/blob/main/README.md

Install NeMo Evaluator with support for regression analysis using scipy.

```bash
pip install -e ".[stats]"
```

--------------------------------

### Install NeMo Evaluator with Scoring

Source: https://github.com/nvidia-nemo/evaluator/blob/main/README.md

Install NeMo Evaluator with support for symbolic math using sympy.

```bash
pip install -e ".[scoring]"
```

--------------------------------

### View Run Command Options

Source: https://github.com/nvidia-nemo/evaluator/blob/main/packages/nemo-evaluator-launcher/examples/nemotron/nemotron-3-super/reproducibility.md

Display the available options for the 'run' command.

```bash
# View run command options
nemo-evaluator-launcher run --help
```

--------------------------------

### Run MMLU-Pro Benchmark

Source: https://github.com/nvidia-nemo/evaluator/blob/main/examples/nemotron/nemotron-3-ultra/v0.2/README.md

Execute the MMLU-Pro benchmark. Recommended settings include num_repeats: 1 and 10-choice boxed prompt_config.

```bash
nel run --config local_nemotron-3-ultra-550b-a55b.yaml --env-file .env -t ns_mmlu_pro
```

--------------------------------

### Install NeMo Evaluator Launcher via Pip

Source: https://github.com/nvidia-nemo/evaluator/blob/main/packages/nemo-evaluator-launcher/examples/nemotron/nemotron-3-super/reproducibility.md

Install the NeMo Evaluator Launcher package using pip.

```bash
pip install nemo-evaluator-launcher
```

--------------------------------

### Initialize VLMEvalKit Environment and Solver

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/deployment-patterns.mdx

Set up the VLMEvalKitEnvironment with a specific benchmark and configure the ModelClient for interacting with the vision-language model API. The VLMSolver is then initialized with the client.

```python
from nemo_evaluator.environments.vlmevalkit import VLMEvalKitEnvironment
from nemo_evaluator.engine import run_evaluation, ModelClient
from nemo_evaluator.solvers import VLMSolver

env = VLMEvalKitEnvironment("MMBench_DEV_EN")
client = ModelClient(base_url="...", model="...", api_key="...")
solver = VLMSolver(client)
bundle = await run_evaluation(env, solver, n_repeats=1)
```

--------------------------------

### Install NeMo Evaluator with Inspect AI

Source: https://github.com/nvidia-nemo/evaluator/blob/main/README.md

Install NeMo Evaluator with support for Inspect AI log export.

```bash
pip install -e ".[inspect]"
```

--------------------------------

### Run GDPVal Comparison Mode

Source: https://github.com/nvidia-nemo/evaluator/blob/main/examples/nemotron/nemotron-3-ultra/v0.2/README.md

Execute Nemotron-3 Ultra in 'comparison' mode against the generated baseline deliverables.

```bash
nel run --config local_nemotron-3-ultra-550b-a55b_gdpval.yaml --env-file .env
```

--------------------------------

### Install NeMo Evaluator with Ray Support

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/deployment/ray.mdx

Install the NeMo evaluator package with the necessary dependencies for Ray integration.

```bash
pip install -e ".[ray]"
```

--------------------------------

### Install NeMo Evaluator with Harbor Agents

Source: https://github.com/nvidia-nemo/evaluator/blob/main/README.md

Install NeMo Evaluator with support for Harbor agents like OpenHands and Terminus-2.

```bash
pip install -e ".[harbor]"
```

--------------------------------

### Run SWE-bench Verified Benchmark

Source: https://github.com/nvidia-nemo/evaluator/blob/main/examples/nemotron/nemotron-3-ultra/v0.3/README.md

Executes the SWE-bench Verified benchmark using the specified configuration file. Results are typically saved to the 'output.dir' defined in the config.

```bash
nel eval run local_nemotron-3-ultra-550b-a55b_swebench-verified.yaml
```

--------------------------------

### Run Specific Benchmark (MMLU-Pro)

Source: https://github.com/nvidia-nemo/evaluator/blob/main/packages/nemo-evaluator-launcher/examples/nemotron/nano-v3-reproducibility.md

Use the `-t` flag to run a single specified benchmark. Ensure you are in the 'examples/nemotron' directory.

```bash
nemo-evaluator-launcher run --config local_nvidia_nemotron_3_nano_30b_a3b.yaml -t ns_mmlu_pro
```

--------------------------------

### Install NeMo Evaluator Core

Source: https://github.com/nvidia-nemo/evaluator/blob/main/README.md

Install the core NeMo Evaluator package. Use additional flags for specific features.

```bash
pip install -e .
```

--------------------------------

### Run Full Evaluation Suite

Source: https://github.com/nvidia-nemo/evaluator/blob/main/packages/nemo-evaluator-launcher/examples/nemotron/nano-v3-reproducibility.md

Execute the full evaluation suite by specifying the configuration file path.

```bash
nemo-evaluator-launcher run \
  --config packages/nemo-evaluator-launcher/examples/nemotron/local_nvidia_nemotron_3_nano_30b_a3b.yaml
# Or point to your path if you placed the file under a different location
```

--------------------------------

### Install NeMo Evaluator with Scoring and Stats

Source: https://github.com/nvidia-nemo/evaluator/blob/main/README.md

Install NeMo Evaluator with support for both symbolic math (sympy) and regression analysis (scipy).

```bash
pip install -e ".[scoring,stats]"
```

--------------------------------

### Install nemo-evaluator with Harbor extra

Source: https://github.com/nvidia-nemo/evaluator/blob/main/examples/nemotron/nemotron-3-ultra/v0.3/README.md

Installs the nemo-evaluator package with the 'harbor' extra, which includes agentic dependencies. Ensure you are using a virtual environment.

```bash
python3 -m venv .venv
.venv/bin/pip install "nemo-evaluator[harbor]==0.3.*"
```

--------------------------------

### Quick Testing of Base Models with Limited Samples

Source: https://github.com/nvidia-nemo/evaluator/blob/main/packages/nemo-evaluator-launcher/examples/nemotron/nemotron-3-super/reproducibility.md

Perform quick configuration tests by limiting the number of samples evaluated. This is useful for rapid checks but should not be used for actual benchmark results.

```bash
nemo-evaluator-launcher run \
  --config local_nemotron-3-super-120b-a12b-base.yaml \
  -o evaluation.nemo_evaluator_config.config.params.limit_samples=10
```

--------------------------------

### Initialize Skills Environment with Custom Prompt Template

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/tutorials/skills-integration.mdx

Instantiate the SkillsEnvironment in Python with a custom prompt template. This allows for programmatic control over the prompts used for a given benchmark.

```python
from nemo_evaluator.environments.skills import SkillsEnvironment

env = SkillsEnvironment(
    "gpqa",
    prompt_template="Answer the following question.\n\n{problem}\n\nAnswer:",
)
```

--------------------------------

### Initialize Container Environment and Run Evaluation

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/tutorials/legacy-containers.mdx

This Python snippet demonstrates how to set up a `ContainerEnvironment` with a specific Docker image and task, then use a `ModelClient` and `ChatSolver` to run the evaluation via `run_evaluation`.

```python
from nemo_evaluator.environments.container import ContainerEnvironment
from nemo_evaluator.engine.eval_loop import run_evaluation
from nemo_evaluator.solvers import ChatSolver
from nemo_evaluator.engine.model_client import ModelClient

env = ContainerEnvironment(
    image="nvcr.io/.../simple-evals:26.01",
    task="GPQA_diamond",
)

client = ModelClient(
    base_url="https://inference-api.nvidia.com/v1",
    model="azure/openai/gpt-5.2",
)
solver = ChatSolver(client)
bundle = await run_evaluation(env, solver)
```

--------------------------------

### Failure Analysis JSON Output Example

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/evaluation/scoring.mdx

This JSON structure summarizes the evaluation failures, including the total number of failures, the overall failure rate, a breakdown of failures by category, and examples of specific failures.

```json
{
  "total_failures": 12,
  "failure_rate": 0.06,
  "categories": {
    "refusal": {"count": 5, "rate": 0.025},
    "format_error": {"count": 4, "rate": 0.02},
    "timeout": {"count": 3, "rate": 0.015}
  },
  "exemplars": [
    {"category": "refusal", "problem_idx": 42, "response_preview": "I'm sorry, I cannot..."}
  ]
}
```

--------------------------------

### Docker Sandbox Configuration

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/architecture/sandbox.mdx

Configure a Docker sandbox for benchmarks. This setup is suitable for general-purpose containerized environments.

```yaml
benchmarks:
  - name: gym://swebench-server
    sandbox:
      type: docker
      image_template: "swebench/sweb.eval.x86_64.{instance_id}:latest"
      concurrency: 8
      memory: 4g
      network: bridge
```

--------------------------------

### Serve Benchmark with Gym Protocol and Data Export

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/deployment-patterns.mdx

Serve a benchmark using the Gym-compatible protocol and also export data in JSONL format for `ng_collect_rollouts`. This is useful for collecting training rollouts.

```bash
nel serve -b gsm8k --port 9090 --gym-compat --export-data /data
```

--------------------------------

### Merge Semantics Example

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/tutorials/config-composition.mdx

Demonstrates how different data types are merged recursively. `null` explicitly deletes a key.

```yaml
defaults:
  - _base_: base_config

output:
  extra_field: null     # removes extra_field inherited from base
  dir: /new/output      # overrides base's dir
```

--------------------------------

### Docker Compose: Local Eval (No Server)

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/deployment/docker.mdx

Runs a local evaluation using Docker Compose without starting a server.

```bash
docker compose -f deploy/docker-compose.yaml run eval-local
```

--------------------------------

### Run SWE-bench Multilingual Benchmark

Source: https://github.com/nvidia-nemo/evaluator/blob/main/examples/nemotron/nemotron-3-ultra/v0.3/README.md

Executes the SWE-bench Multilingual benchmark. Ensure all sandbox prerequisites and environment variables are correctly configured.

```bash
nel eval run local_nemotron-3-ultra-550b-a55b_swebench-multilingual.yaml
```

--------------------------------

### Docker Compose: Serve + Remote Eval

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/deployment/docker.mdx

Starts the 'serve' and 'eval-remote' services using Docker Compose for remote evaluations.

```bash
docker compose -f deploy/docker-compose.yaml up serve eval-remote
```

--------------------------------

### Run IMO-AnswerBench Benchmark

Source: https://github.com/nvidia-nemo/evaluator/blob/main/examples/nemotron/nemotron-3-ultra/v0.2/README.md

Execute the IMO-AnswerBench benchmark. Requires JUDGE_API_KEY in .env. Recommended setting is num_repeats: 5.

```bash
nel run --config local_nemotron-3-ultra-550b-a55b.yaml --env-file .env -t nemo_skills.ns_imo_answerbench
```

--------------------------------

### Low-Budget Configuration Override

Source: https://github.com/nvidia-nemo/evaluator/blob/main/packages/nemo-evaluator-launcher/examples/nemotron/nemotron-3-super/reproducibility.md

Example of a YAML configuration snippet that overrides adapter parameters to enable low-effort thinking for evaluations.

```yaml
adapter_config:
  params_to_add: {"chat_template_kwargs": {"enable_thinking": true, "low_effort": true}}
```

--------------------------------

### Run Multiple Specific Benchmarks

Source: https://github.com/nvidia-nemo/evaluator/blob/main/packages/nemo-evaluator-launcher/examples/nemotron/nano-v3-reproducibility.md

Use the `-t` flag multiple times to run several specific benchmarks. Ensure you are in the 'examples/nemotron' directory.

```bash
nemo-evaluator-launcher run --config local_nvidia_nemotron_3_nano_30b_a3b.yaml -t ns_gpqa -t ns_aime2025
```

--------------------------------

### List Benchmarks and Environments Command

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/api/index.mdx

Displays available benchmarks and environments. Can be filtered by source.

```bash
nel list
    --source TEXT            Filter by source (e.g., lm-eval)
```

--------------------------------

### Serve Benchmark with Gym-Compatible Protocol

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/deployment-patterns.mdx

Serve a benchmark using a Gym-compatible protocol, which accepts `NeMoGymResponse` in the `/verify` endpoint. This allows Gym agents to consume the benchmark during training.

```bash
nel serve -b gsm8k --port 9090 --gym-compat
```

--------------------------------

### Run IFBench Benchmark

Source: https://github.com/nvidia-nemo/evaluator/blob/main/examples/nemotron/nemotron-3-ultra/v0.2/README.md

Execute the IFBench benchmark. Recommended setting is num_repeats: 8.

```bash
nel run --config local_nemotron-3-ultra-550b-a55b.yaml --env-file .env -t nemo_skills.ns_ifbench
```

--------------------------------

### Run SLURM Evaluation

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/deployment/slurm.mdx

Configure and run a standard evaluation job on a SLURM cluster. This setup deploys a vLLM service and runs benchmarks.

```yaml
# slurm_eval.yaml
services:
  model:
    type: vllm
    model: nvidia/Llama-3.1-70B-Instruct
    protocol: chat_comcompletions
    tensor_parallel_size: 4
    port: 8000
    node_pool: compute

benchmarks:
  - name: gsm8k
    repeats: 8
    solver:
      type: simple
      service: model

cluster:
  type: slurm
  walltime: "02:00:00"
  node_pools:
    compute:
      partition: batch
      nodes: 1
      ntasks_per_node: 1
      gres: "gpu:4"
```

```bash
nel eval run slurm_eval.yaml
```

--------------------------------

### Package Benchmark Module for Sharing

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/tutorials/byob.mdx

Generate a Dockerfile for your benchmark module using `nel package`. Omit `--output` to build the image directly, and add `--push` to push it.

```bash
nel package \
  --module ./benchmarks/my_reasoning.py \
  --tag my-reasoning:latest \
  --output Dockerfile
```

--------------------------------

### Initialize SkillsEnvironment

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/api/index.mdx

Wraps any NeMo Skills benchmark as an `EvalEnvironment`. Specify the benchmark name and optionally dataset split, data directory, prompt template, or evaluation type.

```python
from nemo_evaluator.environments.skills import SkillsEnvironment

env = SkillsEnvironment("gpqa")
solver = ChatSolver(client)
bundle = await run_evaluation(env, solver, n_repeats=4)
```

--------------------------------

### Export Evaluation Results

Source: https://github.com/nvidia-nemo/evaluator/blob/main/README.md

Configure the output to export evaluation results to inspect, wandb, or mlflow. Ensure necessary packages are installed for each export type.

```yaml
output:
  export: [inspect, wandb, mlflow]
```

--------------------------------

### List Available Harnesses

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/tutorials/legacy-containers.mdx

Use the `nel list` command to see available harnesses for a given source. This helps in identifying compatible container images and tasks.

```bash
nel list --source lm-eval
```

--------------------------------

### Sample Minimal Terraform Variables

Source: https://github.com/nvidia-nemo/evaluator/blob/main/terraform/stacks/ecs-sandbox/README.md

Example of a minimal `terraform.tfvars` file required for deploying the ECS sandbox. Customize `project`, `regions`, and `tags` as needed.

```hcl
project = "my-org-eval"
regions = ["us-west-2"]
enable_efs = false

tags = {
  Project   = "my-org-eval"
  ManagedBy = "terraform"
  Owner     = "platform-team"
}
```

--------------------------------

### generate_app()

Source: https://github.com/nvidia-nemo/evaluator/blob/main/docs/api/index.mdx

Serve an `EvalEnvironment` as a Gym-compatible HTTP endpoint.

```APIDOC
## generate_app()

### Description
Serve an `EvalEnvironment` as a Gym-compatible HTTP endpoint.

### Usage
```python
from nemo_evaluator.serving.app import generate_app

app = generate_app(env, gym_compat=True)
# Use with uvicorn: uvicorn.run(app, port=9090)
```
```

--------------------------------

### Add Benchmark using Decorator API

Source: https://github.com/nvidia-nemo/evaluator/blob/main/CONTRIBUTING.md

Example of defining a new benchmark using the `@benchmark` and `@scorer` decorators. Ensure the benchmark is imported in `__init__.py`.

```python
from nemo_evaluator import benchmark, scorer, ScorerInput, answer_line

@benchmark(
    name="my_bench",
    dataset="hf://my-org/my-data?split=test",
    prompt="Question: {question}\nAnswer:",
    target_field="answer",
)
@scorer
def my_scorer(sample: ScorerInput) -> dict:
    return answer_line(sample)
```

```python
import nemo_evaluator.benchmarks.your_bench  # noqa: F401
```

--------------------------------

### Run Qwen3 Base Model Evaluation

Source: https://github.com/nvidia-nemo/evaluator/blob/main/packages/nemo-evaluator-launcher/examples/nemotron/nano-v3-reproducibility.md

Execute the evaluation suite for the Qwen3 base model using its local configuration file. Ensure you are in the correct directory before running.

```bash
cd packages/nemo-evaluator-launcher/examples/nemotron

# Run Qwen3 base model evaluation
nemo-evaluator-launcher run --config local_qwen3-30b-a3b-base.yaml
```

--------------------------------

### Run Tau2 (TauBench V3)

Source: https://github.com/nvidia-nemo/evaluator/blob/main/examples/nemotron/nemotron-3-ultra/v0.2/README.md

Execute the Tau2 (TauBench V3) benchmark for tool-use conversations. Requires TAU2_SIMULATOR_API_KEY in .env and simulator configuration.

```bash
nel run --config local_nemotron-3-ultra-550b-a55b_tau2.yaml --env-file .env
```