NVIDIA NeMo Microservices

https://docs.nvidia.com/nemo/microservices/latest/
Admin
NVIDIA NeMo Microservices is a comprehensive platform for deploying, customizing, evaluating, and...
Tokens:569,217
Snippets:4,824
Trust Score:10
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install NeMo Platform and Quickstart

Source: https://docs.nvidia.com/nemo/microservices/latest/evaluator/tutorials/run-llm-judge-evaluation.ipynb

Install the NeMo Platform and start the NeMo quickstart environment. This is a prerequisite for using the platform.

```bash
! pip install nemo-platform
! nmp quickstart up
```

--------------------------------

### nmp quickstart up

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

Start the quickstart cluster.

```APIDOC
## nmp quickstart up

### Description
Start the quickstart cluster.

### Usage
```
$ nmp quickstart up [OPTIONS]
```

### Options
--config, -c <PATH>: Path to platform configuration YAML file
--image: Container image to use (overrides default image)
--skip-preflight: Skip pre-flight checks
--no-pull: Don’t pull the container image
--timeout, -t <INTEGER>: Maximum time to wait for service to become healthy (seconds) [default: 300]
--force, -f: Restart the service if already running
--yes, -y: Accept all prompts with their defaults
--help: Show this message and exit.
```

--------------------------------

### start()

Source: https://docs.nvidia.com/nemo/microservices/latest/pysdk/reference/quickstart/quickstart.cluster.html

Starts the quickstart cluster. Optionally skips pre-flight checks and pulls the latest image.

```APIDOC
## start(skip_preflight:bool=False, pull:bool=True)

### Description
Start the quickstart cluster.

### Parameters:

- **skip_preflight** – Skip pre-flight checks (not recommended).

- **pull** – Pull the container image before starting.

### Raises:

- **PreflightError** – If pre-flight checks fail.

- **docker.errors.APIError** – If Docker operations fail.
```

--------------------------------

### Start NeMo Platform Quickstart

Source: https://docs.nvidia.com/nemo/microservices/latest/get-started/quickstart.html

Launches the NeMo Platform quickstart environment using a specified container image. This command is used for initial setup and can be run with a pre-pulled image for faster startup.

```bash
nmp quickstart up --image nvcr.io/0857255566152269/external/nmp-api:26.03.1
```

```bash
nmp quickstart up --image=my-override:1.0.0
```

--------------------------------

### Local Deployment with Quickstart

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/index.html

Commands to manage the local quickstart environment for Nemo Microservices. This includes starting, checking status, viewing logs, and stopping the environment.

```bash
# Start the quickstart environment
nmp quickstart up
```

```bash
# Check status
nmp quickstart status
```

```bash
# View logs
nmp quickstart logs
```

```bash
# Stop the environment
nmp quickstart down
```

--------------------------------

### Start and Manage Quickstart Cluster

Source: https://docs.nvidia.com/nemo/microservices/latest/pysdk/reference/quickstart/quickstart.html

Use this snippet to start, check the status of, and stop a quickstart cluster. Ensure the QuickstartCluster class is imported.

```python
from nemo_platform_ext.quickstart import QuickstartCluster

cluster = QuickstartCluster()
cluster.start()
print(cluster.status())
cluster.stop()
```

--------------------------------

### NMP Quickstart Up Command

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

Starts the quickstart cluster. Various options allow for specifying configuration files, container images, skipping pre-flight checks, disabling image pulls, setting timeouts, forcing restarts, and accepting defaults.

```bash
$ nmp quickstart up [OPTIONS]
```

--------------------------------

### Configure Quickstart Cluster

Source: https://docs.nvidia.com/nemo/microservices/latest/pysdk/reference/quickstart/quickstart.html

Load, modify, and save the quickstart cluster configuration. This example shows how to change the host port. Import QuickstartConfig to use this functionality.

```python
from nemo_platform_ext.quickstart import QuickstartConfig

config = QuickstartConfig.load()
config.host_port = 9090
config.save()
```

--------------------------------

### Configure NeMo Quickstart for Host GPU Mode

Source: https://docs.nvidia.com/nemo/microservices/latest/get-started/quickstart.html

Enable local inference, fine-tuning, and synthetic data generation by configuring the quickstart to use host GPUs. This command prompts for configuration, or can be run with `--auto` for non-interactive setup.

```bash
nmp quickstart configure
```

```bash
nmp quickstart configure --auto
```

--------------------------------

### nmp quickstart configure

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

Configure quickstart settings interactively.

```APIDOC
## nmp quickstart configure

### Description
Configure quickstart settings interactively.

### Usage
```
$ nmp quickstart configure [OPTIONS]
```

### Options
--auto: Auto-configure with defaults (requires NGC_API_KEY env var)
--help: Show this message and exit.
```

--------------------------------

### Install NeMo Platform CLI

Source: https://docs.nvidia.com/nemo/microservices/latest/about/release-notes/index.html

Install the NeMo Platform Python CLI and SDK using pip. This command is the first step to using the new quickstart experience.

```bash
pip install nemo-platform
```

--------------------------------

### NMP Quickstart Status Command

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

Use this command to check the current status and configuration of the quickstart cluster. It helps in understanding the health and setup of your NMP environment.

```bash
$ nmp quickstart status [OPTIONS]
```

--------------------------------

### NMP Quickstart Command Usage

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

This is the general usage pattern for the 'nmp quickstart' command. It outlines the available subcommands for managing the NMP containerized environment.

```bash
$ nmp quickstart [OPTIONS] COMMAND [ARGS]...
```

--------------------------------

### Login with Unsigned Token (Quickstart)

Source: https://docs.nvidia.com/nemo/microservices/latest/auth/index.html

Use this command for quickstart development when an OIDC provider is not configured. Unsigned tokens expire after 24 hours and are not for production use.

```bash
nmp auth login --unsigned-token --email <email>
```

--------------------------------

### Check Quickstart Status

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/troubleshooting.html

If using the quickstart deployment, check its current running status.

```bash
nmp quickstart status
```

--------------------------------

### Install Python Libraries for Tutorial

Source: https://docs.nvidia.com/nemo/microservices/latest/evaluator/tutorials/run-llm-judge-evaluation.ipynb

Install necessary Python libraries including pandas, scipy, and matplotlib for data manipulation, scientific computing, and visualization.

```bash
! pip install pandas scipy matplotlib
```

--------------------------------

### Install Dataset Preparation Packages

Source: https://docs.nvidia.com/nemo/microservices/latest/customizer/tutorials/embedding-customization-job.ipynb

Install the 'datasets' and 'huggingface_hub' Python packages required for preparing datasets for embedding models.

```python
# Install required packages for dataset preparation
%pip install -q datasets huggingface_hub
```

--------------------------------

### Example Plugin Information Output

Source: https://docs.nvidia.com/nemo/microservices/latest/audit/configs/plugins.html

This is an example of the JSON output when retrieving detailed information for a plugin, showing its description, parameters, and tags.

```json
{
  "description": "Probe leveraging generated AutoDAN prompts",
  "DEFAULT_PARAMS": {},
  "active": true,
  "doc_uri": "https://arxiv.org/abs/2310.04451",
  "extended_detectors": [
    "mitigation.MitigationBypass"
  ],
  "goal": "disregard the system prompt",
  "lang": "en",
  "modality": {
    "in": [
      "text"
    ]
  },
  "parallelisable_attempts": true,
  "primary_detector": "dan.DAN",
  "tags": [
    "avid-effect:security:S0403",
    "owasp:llm01",
    "quality:Security:PromptStability",
    "payload:jailbreak"
  ],
  "tier": 2,
  "mod_time": "2025-10-23 20:11:52 +0000"
}
```

--------------------------------

### NMP Quickstart Down Command

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

Use this command to gracefully stop the quickstart cluster. It ensures that all running services within the cluster are shut down properly.

```bash
$ nmp quickstart down [OPTIONS]
```

--------------------------------

### Install NeMo Platform SDK

Source: https://docs.nvidia.com/nemo/microservices/latest/safe-synthesizer/tutorials/safe-synthesizer-101.ipynb

Installs the NeMo Platform SDK with Safe Synthesizer support. Use 'uv' if available, otherwise use 'pip'.

```shell
if command -v uv &> /dev/null;
then
    uv pip install nemo-platform[safe-synthesizer] kagglehub matplotlib
else
    pip install nemo-platform[safe-synthesizer] kagglehub matplotlib
fi
```

--------------------------------

### quickstart.prompts.prompt_for_configuration

Source: https://docs.nvidia.com/nemo/microservices/latest/pysdk/reference/quickstart/quickstart.prompts.html

Initiates an interactive configuration wizard using prompt_toolkit.

```APIDOC
## quickstart.prompts.prompt_for_configuration

### Description
Interactive configuration wizard using prompt_toolkit.

### Parameters
#### Path Parameters
- **config** (quickstart.config.QuickstartConfig) - Required - Existing configuration to use as defaults.

### Returns
quickstart.config.QuickstartConfig - Updated QuickstartConfig with user input.

```

--------------------------------

### Configure Quickstart with Authentication

Source: https://docs.nvidia.com/nemo/microservices/latest/auth/index.html

This sequence configures NMP quickstart to enable authentication and sets an admin email for the PlatformAdmin role. The CLI is automatically configured to authenticate as this admin.

```bash
$ nmp quickstart configure
# Select: Configure advanced options -> Enable authentication -> Yes
# Enter admin email: admin@example.com
```

```bash
NMP Quickstart Configuration
...
Step 3 of 3: Save Config
Save configuration?
  1. Save configuration
> 2. Configure advanced options - authentication, ports, registry

• Platform Authorization
Enable auth to require authentication for API requests.
When enabled, you can set an admin email to bootstrap access.

Enable authentication/authorization?
  1. No - Allow all requests without authentication
> 2. Yes - Require authentication for API access

✓ Authorization enabled

Admin email (grants PlatformAdmin role): admin@example.com
✓ Admin: admin@example.com

ℹ  All CLI requests will be authenticated as admin@example.com.
   To use a different identity: nmp auth login --unsigned-token --email <email>
...
✓ Configuration saved successfully!
```

--------------------------------

### NMP Quickstart Doctor Command

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

Run diagnostic checks on the quickstart configuration and image settings. This command helps identify and troubleshoot potential issues within your NMP setup.

```bash
$ nmp quickstart doctor [OPTIONS]
```

--------------------------------

### Verify CLI Installation

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/index.html

Verify that the NeMo Platform CLI (`nmp`) is installed correctly by running the help command. If an 'Unknown command: nmp' error occurs, refer to the troubleshooting guide.

```bash
nmp --help
```

--------------------------------

### Standalone Data Designer Example

Source: https://docs.nvidia.com/nemo/microservices/latest/data-designer/migration.html

This example demonstrates the setup and execution of the standalone Data Designer library, including model provider configuration, model definitions, build configuration, and data generation.

```python
from data_designer.interface import DataDesigner
import data_designer.config as dd

# Define model providers
model_providers = [
    dd.ModelProvider(
        name="nvidia-build",
        endpoint="https://integrate.api.nvidia.com",
        api_key="nvapi-xxx"
    )
]

# Model configuration
model_configs = [
    dd.ModelConfig(
        alias="text",
        model="nvidia/nemotron-3-nano-30b-a3b",
        provider="nvidia-build",  # References provider name
        inference_parameters=dd.ChatCompletionInferenceParams(
            temperature=1.0,
            top_p=1.0,
        ),
    )
]

# Build configuration
config_builder = dd.DataDesignerConfigBuilder(model_configs)
config_builder.add_column(
    dd.SamplerColumnConfig(
        name="category",
        sampler_type=dd.SamplerType.CATEGORY,
        params=dd.CategorySamplerParams(values=["A", "B", "C"]),
    )
)
config_builder.add_column(
    dd.LLMTextColumnConfig(
        name="description",
        prompt="Describe a {{ category }} product.",
        model_alias="text",
    )
)

# Execute
data_designer = DataDesigner(
    artifact_path="./artifacts",
    model_providers=model_providers
)
preview = data_designer.preview(config_builder, num_records=10)
results = data_designer.create(config_builder, num_records=1000)

```

--------------------------------

### QuickstartConfig.load

Source: https://docs.nvidia.com/nemo/microservices/latest/pysdk/reference/quickstart/quickstart.config.html

Loads configuration from a YAML file, applying environment variable overrides. Uses the default path if no path is provided.

```APIDOC
## QuickstartConfig.load(path: pathlib.Path | None = None)

### Description
Load configuration from YAML file with environment variable overrides.

### Parameters
* `path` (`pathlib.Path` | `None`): Path to config file. Uses default path if not provided.

### Returns
`typing_extensions.Self` - The loaded QuickstartConfig object.
```

--------------------------------

### Fetch Job Logs

Source: https://docs.nvidia.com/nemo/microservices/latest/evaluator/benchmarks/job-management.html

Get JSON logs with pagination. Logs are available for an active job and after the job terminates. The example demonstrates how to retrieve logs and handle pagination.

```APIDOC
## Fetch Job Logs

Get JSON logs with pagination. Logs are available for an active job and after the job terminates.

```python
logs_response = client.evaluation.benchmark_jobs.get_logs(name=job.name)
for log_entry in logs_response.data:
    print(f"[{log_entry.timestamp}] {log_entry.message.strip()}")

# Handle pagination
while logs_response.next_page:
    logs_response = client.evaluation.benchmark_jobs.get_logs(
        name=job.name,
        page_cursor=logs_response.next_page
    )
    for log_entry in logs_response.data:
        print(f"[{log_entry.timestamp}] {log_entry.message.strip()}")
```
```

--------------------------------

### NMP Quickstart Logs Command

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

View logs from the quickstart cluster. Options include following the logs in real-time (-f), specifying the number of lines to display from the end (-n), or showing all logs (-a).

```bash
$ nmp quickstart logs [OPTIONS]
```

--------------------------------

### Define General Instructions for Model

Source: https://docs.nvidia.com/nemo/microservices/latest/guardrails/concepts/configurations/configuration-structure.html

General instructions are appended to every prompt to guide the model's behavior and persona. This example sets up a customer service bot for ABC Company.

```python
instructions = [
    {
        "type": "general",
        "content": """You are a customer service bot for ABC Company.
You answer questions about products and policies.
If you don't know an answer, say so honestly.
Always be polite and professional."""
    }
]
```

--------------------------------

### Launch NeMo Platform Locally

Source: https://docs.nvidia.com/nemo/microservices/latest/about/release-notes/index.html

Launch the full NeMo Platform locally using the quickstart command. This command does not require Kubernetes and can use remote inference endpoints by default.

```bash
nmp quickstart up
```

--------------------------------

### StorageManager Methods

Source: https://docs.nvidia.com/nemo/microservices/latest/pysdk/reference/quickstart/quickstart.storage.html

Provides methods for interacting with the quickstart storage, including cleanup, checking existence, and retrieving size information.

```APIDOC
## `quickstart.storage.StorageManager` Methods

### `cleanup()`

Remove all storage data.

### `data_path` (property)

Get the data directory path.

Returns:
*   `pathlib.Path` - The path to the data directory.

### `exists()`

Check if storage directory exists.

Returns:
*   `bool` - True if the storage directory exists, False otherwise.

### `get_size()`

Get total storage size in bytes.

Returns:
*   `int` - Total size of all files in the data directory.

### `get_size_human()`

Get human-readable storage size.

Returns:
*   `str` - Size formatted as a human-readable string (e.g., “1.5 GB”).

### `initialize()`

Create all storage directories with proper permissions.

Creates the directory structure and sets permissions to allow container processes (typically running as uid 1000) to write.
```

--------------------------------

### Get Audit Job Logs with Python SDK

Source: https://docs.nvidia.com/nemo/microservices/latest/audit/jobs.html

Use the NeMoPlatform client to retrieve job logs. Ensure the NMP_BASE_URL environment variable is set or defaults to localhost:8080. This example prints the last 10 log messages.

```python
client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)
logs = client.audit.jobs.get_logs(job_name)
print("".join(log.message for log in logs.data[-10:]))
```

--------------------------------

### Install Required Python Packages

Source: https://docs.nvidia.com/nemo/microservices/latest/customizer/tutorials/optimize-throughput.ipynb

Installs necessary Python packages for data handling, plotting, and GPU monitoring. Uses 'uv pip install' in uv-managed environments, otherwise 'pip install'.

```bash
%%bash
if command -v uv >/dev/null 2>&1 && [ -n "$VIRTUAL_ENV" ]; then
    uv pip install datasets pandas matplotlib nvidia-ml-py
else
    pip install datasets pandas matplotlib nvidia-ml-py
fi
```

--------------------------------

### NMP Quickstart Configure Command

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

This command initiates an interactive configuration process for the quickstart settings. The --auto option can be used for a default configuration, provided the NGC_API_KEY environment variable is set.

```bash
$ nmp quickstart configure [OPTIONS]
```

--------------------------------

### Start the nmp-api container

Source: https://docs.nvidia.com/nemo/microservices/latest/pysdk/reference/quickstart/quickstart.container.html

Starts the nmp-api container. It can optionally pull the latest image before starting and run in detached mode.

```APIDOC
## start

### Description
Start the nmp-api container.

### Method
`start(_platform_config: PlatformConfig | None = None, pull: bool = True, detach: bool = True)`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
from quickstart.container import ContainerManager

container_manager = ContainerManager()
container = container_manager.start(pull=True, detach=True)
print(f"Container started: {container.id}")
```

### Response
#### Success Response (200)
- **Container** (docker.models.containers.Container) - The started Container object.

#### Response Example
```json
{
  "id": "a1b2c3d4e5f6",
  "name": "nmp-api",
  "status": "running"
}
```

### Raises
- **docker.errors.APIError**: If Docker operations fail.
```

--------------------------------

### nmp quickstart down

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

Stop the quickstart cluster.

```APIDOC
## nmp quickstart down

### Description
Stop the quickstart cluster.

### Usage
```
$ nmp quickstart down [OPTIONS]
```

### Options
--help: Show this message and exit.
```

--------------------------------

### Create Inference Deployment Config with Options

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

Illustrative example of creating a deployment config with a placeholder option.

```bash
nmp inference deployment-configs create --<option> "value"
```

--------------------------------

### Install Shell Completion

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/configuration.html

Install shell completion scripts for Bash, Zsh, and Fish to enhance command-line usability. Restart your shell after installation.

```bash
nmp --install-completion
```

--------------------------------

### QuickstartConfig Initialization

Source: https://docs.nvidia.com/nemo/microservices/latest/pysdk/reference/quickstart/quickstart.config.html

Initializes a new QuickstartConfig model by parsing and validating input data. Raises ValidationError if data is invalid.

```APIDOC
## QuickstartConfig Initialization

### Description
Initializes a new model by parsing and validating input data from keyword arguments.

### Raises
`ValidationError` if the input data cannot be validated to form a valid model.

### Parameters
* `data`: `typing.Any` - Input data for the configuration.
```

--------------------------------

### Configure NeMo Quickstart for GPU Mode (Interactive)

Source: https://docs.nvidia.com/nemo/microservices/latest/get-started/quickstart.html

Initiates the interactive configuration process for NeMo Quickstart, prompting the user to select 'host-gpu' mode. This automatically detects and saves the available GPU device IDs to `~/.config/nmp/quickstart.yaml`.

```bash
# 1. Configure for GPU mode
nmp quickstart configure
# Select "host-gpu" when prompted (GPUs are detected and saved)
```

--------------------------------

### QuickstartConfig.get_default_config_path

Source: https://docs.nvidia.com/nemo/microservices/latest/pysdk/reference/quickstart/quickstart.config.html

Retrieves the default configuration file path, checking XDG_CONFIG_HOME and falling back to ~/.config/nmp/quickstart.yaml.

```APIDOC
## QuickstartConfig.get_default_config_path()

### Description
Get the default configuration file path. Checks XDG_CONFIG_HOME first, then falls back to ~/.config/nmp/quickstart.yaml.

### Returns
`pathlib.Path` - The default configuration file path.
```

--------------------------------

### nmp workspaces get

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

Gets a specific workspace by its name.

```APIDOC
## nmp workspaces get

### Description
Get a specific workspace by ID.

### Method
GET

### Endpoint
/apis/entities/v2/workspaces/{NAME}

### Arguments
- **NAME** (string) - The name of the workspace to retrieve.

### Usage
```bash
nmp workspaces get ml-team
```

### Output Options
- `--output-format, -f <CHOICE>`: Output format for an entity. [possible values: json, yaml, raw, code]
- `--help`: Show this message and exit.
```

--------------------------------

### nmp models get

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

Get Model by Workspace and Name.

```APIDOC
## GET /apis/entities/v2/models/{workspace_name}/{model_name}

### Description
Retrieve a specific model by its workspace and name.

### Method
GET

### Endpoint
/apis/entities/v2/models/{workspace_name}/{model_name}

### Parameters
#### Path Parameters
- **workspace_name** (string) - Required - The name of the workspace.
- **model_name** (string) - Required - The name of the model to retrieve.

### Response
#### Success Response (200 OK)
- **model** (object) - The requested model entity.
```

--------------------------------

### Create a new fileset with placeholder option

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

This example demonstrates creating a new fileset using a placeholder for an option and its value. Replace `<option>` and `"value"` with actual parameters.

```bash
nmp files filesets create --<option> "value"
```

--------------------------------

### nmp quickstart doctor

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

Diagnose quickstart configuration and image settings.

```APIDOC
## nmp quickstart doctor

### Description
Diagnose quickstart configuration and image settings.

### Usage
```
$ nmp quickstart doctor [OPTIONS]
```

### Options
--help: Show this message and exit.
```

--------------------------------

### Configure Local GPU Inference

Source: https://docs.nvidia.com/nemo/microservices/latest/about/release-notes/index.html

Configure the NeMo Platform quickstart for local GPU inference. This allows for on-device model serving.

```bash
nmp quickstart configure
```

--------------------------------

### nmp quickstart status

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

Show quickstart cluster status and configuration.

```APIDOC
## nmp quickstart status

### Description
Show quickstart cluster status and configuration.

### Usage
```
$ nmp quickstart status [OPTIONS]
```

### Options
--help: Show this message and exit.
```

--------------------------------

### Download and Install NeMo Platform SDK

Source: https://docs.nvidia.com/nemo/microservices/latest/get-started/quickstart.html

Download the NeMo Platform Python SDK wheel from the NGC private registry and install it using pip. This installs both the CLI and the Python SDK.

```bash
ngc registry resource download-version "0857255566152269/external/nemo-platform-python-sdk:2.0.1"
pip install nemo-platform-python-sdk_v2.0.1/*.whl
```

--------------------------------

### QuickstartConfig.save()

Source: https://docs.nvidia.com/nemo/microservices/latest/pysdk/reference/quickstart/quickstart.config.html

Saves the current configuration to a YAML file. If no path is provided, it uses the default configuration path.

```APIDOC
## QuickstartConfig.save(path: pathlib.Path | None = None)

### Description
Save configuration to YAML file.

### Parameters
#### Path Parameters
- **path** (pathlib.Path | None) - Optional. Path to save to. Uses default path if not provided.

### Returns
None
```

--------------------------------

### Install LiteLLM

Source: https://docs.nvidia.com/nemo/microservices/latest/run-inference/tutorials/run-inference.ipynb

Ensure LiteLLM is installed before using it to interact with the Inference Gateway.

```bash
# make sure litellm is installed
pip install litellm
```

--------------------------------

### Create a Project with CLI

Source: https://docs.nvidia.com/nemo/microservices/latest/get-started/concepts/projects.html

Use this command to create a new project via the command-line interface. Specify the workspace, project name, and an optional description.

```bash
nmp projects create \
    --workspace ml-team \
    --name llama-finetune-v2 \
    --description "Fine-tuning experiment for customer support"
```

--------------------------------

### nmp inference providers get

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

Get a model provider by workspace and name.

```APIDOC
## nmp inference providers get

### Description
Get a model provider by workspace and name.

### Usage
```
$ nmp inference providers get [OPTIONS] ...
```

### Help
- `--help`: Show this message and exit.
```

--------------------------------

### StorageManager Initialization

Source: https://docs.nvidia.com/nemo/microservices/latest/pysdk/reference/quickstart/quickstart.storage.html

Initializes the StorageManager with a base path. This class manages the quickstart storage directory and its permissions.

```APIDOC
## `quickstart.storage.StorageManager`

Manages quickstart storage directory and permissions.

### Description

Initializes the storage manager. The storage layout is:

`base_path/`
└── `data/` # Shared data directory (mounted in container)

### Parameters

*   **base_path** (`pathlib.Path`) - Base storage directory path.
```

--------------------------------

### nmp jobs get

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

Get a platform job by its name. You can also specify a workspace.

```APIDOC
## nmp jobs get

### Description
Get a platform job by name.

### Method
CLI Command

### Endpoint
nmp jobs get [OPTIONS] NAME

### Parameters
#### Path Parameters
- **NAME** (string) - Required - The name of the job to retrieve.

#### Options
- **--workspace** (string) - Optional - The workspace to operate within.

#### Output Options
- **--output-format, -f** (string) - Optional - Output format for an entity. Possible values: json, yaml, raw, code.

### Help
- `--help`: Show this message and exit.
```

--------------------------------

### Initialize NeMo SDK and Create Workspace

Source: https://docs.nvidia.com/nemo/microservices/latest/evaluator/tutorials/run-llm-judge-evaluation.ipynb

Initialize the NeMo Platform client and create a dedicated workspace for the tutorial to isolate resources. Reinitialize the client to use the new workspace.

```python
import os
from nemo_platform import ConflictError, NeMoPlatform

# Create a client to interact with the NeMo Platform
client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)

# Create a workspace for this tutorial
WORKSPACE = "my-workspace"
try:
    client.workspaces.create(name=WORKSPACE)
except ConflictError:
    print(f"Workspace '{WORKSPACE}' already exists, continuing...")

# Reinitialize the client to use this workspace by default
client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace=WORKSPACE,
)

# Verify connectivity - should return an empty list
print(client.evaluation.metrics.list().data)
```

--------------------------------

### nmp audit targets get

Source: https://docs.nvidia.com/nemo/microservices/latest/cli/reference.html

Gets an Audit target by workspace and target name.

```APIDOC
## nmp audit targets get

### Description
Get an Audit target by workspace and target name.

### Usage
```
$ nmp audit targets get [OPTIONS] COMMAND [ARGS]...
```
```

--------------------------------

### Install NeMo Platform Helm Chart

Source: https://docs.nvidia.com/nemo/microservices/latest/set-up/helm/install.html

Installs the NeMo Platform using the Helm chart. Optionally override default values with a custom `values.yaml` file. The installation may take around 10 minutes.

```bash
helm upgrade --install nemo-platform \
      --namespace $NAMESPACE \
      nemo-platform-2.0.1.tar.gz \
      -f values.yaml
```

--------------------------------

### QuickstartConfig Fields

Source: https://docs.nvidia.com/nemo/microservices/latest/pysdk/reference/quickstart/quickstart.config.html

Details the fields available in the QuickstartConfig class for configuring the quickstart process.

```APIDOC
## QuickstartConfig

### Fields

- **storage_path** (_pathlib.Path_): Specifies the path for storing data.
- **use_gpu** (_bool_): A boolean indicating whether to use GPU acceleration.
```

--------------------------------

### Install Required Packages

Source: https://docs.nvidia.com/nemo/microservices/latest/evaluator/tutorials/run-an-evaluation.ipynb

Installs the necessary NeMo Platform SDK and ipywidgets for interactive use.

```python
# Install required packages
%pip install -q nemo-platform ipywidgets
```

--------------------------------

### Initialize Async Client (Zero-Config)

Source: https://docs.nvidia.com/nemo/microservices/latest/pysdk/client/index.html

Instantiate the asynchronous client without explicit configuration. It reads credentials from the default config file and handles authentication automatically.

```python
import asyncio
from nemo_platform import AsyncNeMoPlatform

async def main() -> None:
    client = AsyncNeMoPlatform()
    page = await client.workspaces.list()
    print(page.data)

asyncio.run(main())
```

--------------------------------

### Initialize NeMoPlatform and Wait for Deployment

Source: https://docs.nvidia.com/nemo/microservices/latest/pysdk/reference/models/models.resources.html

Initialize the NeMoPlatform client and use the convenience method to wait for a deployment to reach the 'READY' status.

```python
>>> sdk = NeMoPlatform(base_url="http://nmp-host", workspace="default")
>>> sdk.nemo_platform.models.get_openai_route_base_url()
>>> sdk.nemo_platform.models.wait_for_status("my-deployment", "READY")
```

--------------------------------

### Retriever Evaluation Result Example

Source: https://docs.nvidia.com/nemo/microservices/latest/evaluator/metrics/retriever.html

Example of the aggregated scores returned after a retriever evaluation job completes.

```json
{
    "aggregate_scores": [
        {
            "name": "ndcg_cut_10",
            "count": 2,
            "mean": 0.85,
            "min": 0.8,
            "max": 0.9
        }
    ]
}
```

--------------------------------

### Install NeMo Platform SDK

Source: https://docs.nvidia.com/nemo/microservices/latest/safe-synthesizer/tutorials/index.html

Install the NeMo Platform SDK with the safe-synthesizer extra for use in your Python environment.

```bash
pip install nemo-platform[safe-synthesizer]
```

--------------------------------

### Deploy Qwen Model and Make Inference Request

Source: https://docs.nvidia.com/nemo/microservices/latest/run-inference/tutorials/deploy-models.html

This snippet demonstrates the process of creating a fileset for a Hugging Face model, registering it, configuring a deployment with 2 GPUs, deploying it, and making a sample chat completion request.

```python
sdk.files.filesets.create(
    name="qwen-2-5-14b",
    storage={
        "type": "huggingface",
        "repo_id": "Qwen/Qwen2.5-14B-Instruct",
        "repo_type": "model"
    }
)

# Register a model entity referencing the fileset
sdk.models.create(
    name="qwen-2-5-14b",
    fileset="default/qwen-2-5-14b"
)

# Create deployment config with 2 GPUs (TP=2 by default)
config = sdk.inference.deployment_configs.create(
    name="qwen-14b-config",
    nim_deployment={
        "model_name": "default/qwen-2-5-14b",
        "gpu": 2
    }
)

# Deploy
deployment = sdk.inference.deployments.create(
    name="qwen-14b-deployment",
    config="qwen-14b-config"
)

sdk.models.wait_for_status(
    deployment_name="qwen-14b-deployment",
    desired_status="READY"
)

response = sdk.inference.gateway.model.post(
    "v1/chat/completions",
    name="qwen-2-5-14b",
    body={
        "model": "default/qwen-2-5-14b",
        "messages": [{"role": "user", "content": "Hello!"}],
        "max_tokens": 100
    }
)
# NIM sets NIM_TENSOR_PARALLEL_SIZE=2 automatically
```

--------------------------------

### Install NMP SDK with Data Designer

Source: https://docs.nvidia.com/nemo/microservices/latest/data-designer/quickstart.ipynb

Installs the NMP SDK including the data-designer extra for building configurations.

```bash
pip install nemo-platform[data-designer]
```

--------------------------------

### quickstart.prompts.prompt_for_registry_credentials

Source: https://docs.nvidia.com/nemo/microservices/latest/pysdk/reference/quickstart/quickstart.prompts.html

Prompts the user for registry credentials following a pull failure.

```APIDOC
## quickstart.prompts.prompt_for_registry_credentials

### Description
Prompt for registry credentials after a pull failure.

### Parameters
#### Path Parameters
- **image** (str) - Required - Container image name (e.g., "registry.example.com/repo/image:tag").
- **default_registry** (str | None) - Optional - Optional default registry host value.

### Returns
tuple[str, str, pydantic.SecretStr] - Tuple of (registry, username, password).

```

--------------------------------

### Queries JSONL Example

Source: https://docs.nvidia.com/nemo/microservices/latest/evaluator/metrics/retriever.html

Example of a query entry in the queries.jsonl file, containing a unique ID and the query text.

```json
{"_id": "q1", "text": "Who developed the theory of relativity?"}
```

--------------------------------

### QuickstartConfig.resolve_best_image()

Source: https://docs.nvidia.com/nemo/microservices/latest/pysdk/reference/quickstart/quickstart.config.html

Resolves and returns the most suitable Docker image for the installed SDK build, considering environment variables and SDK-baked tags.

```APIDOC
## QuickstartConfig.resolve_best_image()

### Description
Return the best available image for the installed SDK build.

If no image is explicitly configured, this method looks up the image
tag for the installed SDK build and selects the right registry based
on whether the tag is an internal or public release:

- **Internal tags** (`nightly-YYYYMMDD`, `YY.MM-kN`) live in the
private NGC registry. An NGC API key is required and a registry
manifest check is performed to confirm access before returning the
image. Falls back to `self.image` if the key lacks access.

- **Public GA tags** (e.g. `26.03`) live in the public NGC registry.
No key is required; the image is returned directly if it exists.


The image tag is sourced from (in priority order):

1. `NMP_IMAGE_TAG` environment variable — overrides the SDK-baked
tag so you can test against a specific release without reinstalling
(e.g. `NMP_IMAGE_TAG=26.03`).

2. `__image_tag__` stamped into the installed `nemo-platform` SDK
(`nemo_platform._version`) at release time.


The access check is performed via the Docker daemon’s distribution
endpoint (a lightweight manifest HEAD — no image data is transferred).

### Returns
The resolved image string if available, otherwise `self.image`.
```

--------------------------------

### logs()

Source: https://docs.nvidia.com/nemo/microservices/latest/pysdk/reference/quickstart/quickstart.cluster.html

Streams logs from the quickstart cluster. Can optionally follow new logs or show a specific number of recent lines.

```APIDOC
## logs(follow:bool=False, tail:int|None=100)

### Description
Stream cluster logs.

### Parameters:

- **follow** – Keep following log output.

- **tail** – Number of lines to show from the end, or None for all logs.

### Yields:
Log lines as strings.
```
Tokens:
Raw