NVIDIA Documentation Hub

https://docs.nvidia.com/
Admin
NVIDIA Documentation Hub is a central resource providing technical information and product...
Tokens:877,059
Snippets:8,287
Trust Score:10
Update:2 months ago
Show doc for...
Tokens:
Raw
### Complete CUPTI Setup Example

Source: https://docs.nvidia.com/cupti/tutorial/tutorial.html

A condensed example demonstrating header inclusion, error handling macros, and buffer callback implementation.

```C
#include <cuda.h>
#include <cuda_runtime.h>
#include <cupti.h>
#include <stdio.h>
#include <stdlib.h>

#define CUPTI_API_CALL(call)                                              \
do {                                                                      \
    CUptiResult status = call;                                            \
    if (status != CUPTI_SUCCESS) {                                        \
        const char *errstr;                                               \
        cuptiGetResultString(status, &errstr);                            \
        fprintf(stderr, "CUPTI Error: %s at %s:%d\n",                     \
                errstr, __FILE__, __LINE__);                              \
        exit(EXIT_FAILURE);                                               \
    }                                                                     \
} while (0)

#define SUBSCRIBER_COUNT 2
#define BUF_SIZE (8 * 1024 * 1024)

CUpti_SubscriberHandle g_subscriberHandle[SUBSCRIBER_COUNT];

__global__ void VectorAdd(const int *pA, const int *pB, int *pC, int N)
{
    int i = blockDim.x * blockIdx.x + threadIdx.x;
    if (i < N)
        pC[i] = pA[i] + pB[i];
}

void CUPTIAPI BufferRequested_subscriber0(
    uint8_t **ppBuffer, size_t *pSize,
    size_t *pMaxNumRecords, CUpti_BufferCallbackRequestInfo *pRequestInfo)
{
    *ppBuffer = (uint8_t *)malloc(BUF_SIZE);
    if (*ppBuffer == NULL)
    {
        fprintf(stderr, "Failed to allocate CUPTI activity buffer.\n");
        exit(EXIT_FAILURE);
    }
    *pSize = BUF_SIZE;
    *pMaxNumRecords = 0;
}
```

--------------------------------

### Quick start with s3cmd

Source: https://docs.nvidia.com/aistore/s3compat

Example of uploading a file using s3cmd with explicit host configuration flags.

```bash
s3cmd put README.md s3://demo \
  --no-ssl \
  --host=localhost:8080/s3 \
  --host-bucket="localhost:8080/s3/%(bucket)"
  
# Expected output:  
# upload: 'README.md' -> 's3://demo/README.md' [1 of 1]  
# 10493 of 10493   100% in    0s     4.20 MB/s  done
```

--------------------------------

### Start Maintenance Example

Source: https://docs.nvidia.com/aistore/cli/cluster

Example of checking cluster status and putting a target node into maintenance mode.

```bash
$ ais show cluster
PROXY            MEM USED %      MEM AVAIL       UPTIME
202446p8082      0.09%           31.28GiB        70s
279128p8080[P]   0.11%           31.28GiB        80s

TARGET           MEM USED %      MEM AVAIL       CAP USED %      CAP AVAIL       CPU USED %      REBALANCE       UPTIME
147665t8084      0.10%           31.28GiB        16%             2.458TiB        0.12%           -               70s
165274t8087      0.10%           31.28GiB        16%             2.458TiB        0.12%           -               70s

$ ais cluster add-remove-nodes start-maintenance 147665t8084
$ ais show cluster
PROXY            MEM USED %      MEM AVAIL       UPTIME
202446p8082      0.09%           31.28GiB        70s
279128p8080[P]   0.11%           31.28GiB        80s

TARGET           MEM USED %      MEM AVAIL       CAP USED %      CAP AVAIL       CPU USED %      REBALANCE       UPTIME  STATUS
147665t8084      0.10%           31.28GiB        16%             2.458TiB        0.12%           -               71s     maintenance
165274t8087      0.10%           31.28GiB        16%             2.458TiB        0.12%           -               71s     online
```

--------------------------------

### Install and Import Dependencies

Source: https://docs.nvidia.com/base-command-platform/user-guide/latest/index.html

Initial setup for AWS SDK interactions within the platform environment.

```python
!pip install boto3
import boto3
from botocore import UNSIGNED
from bot ocore.config import Config
```

--------------------------------

### BSR ILU02 Preconditioner Setup

Source: https://docs.nvidia.com/cuda/cusparse/index.html

This example demonstrates the initialization of matrix descriptors, info structures, and buffer size querying required to perform an ILU02 factorization and solve using BSR format.

```cpp
// Suppose that A is m x m sparse matrix represented by BSR format,
// The number of block rows/columns is mb, and
// the number of nonzero blocks is nnzb.
// Assumption:
// - handle is already created by cusparseCreate(),
// - (d_bsrRowPtr, d_bsrColInd, d_bsrVal) is BSR of A on device memory,
// - d_x is right hand side vector on device memory.
// - d_y is solution vector on device memory.
// - d_z is intermediate result on device memory.
// - d_x, d_y and d_z are of size m.
cusparseMatDescr_t descr_M = 0;
cusparseMatDescr_t descr_L = 0;
cusparseMatDescr_t descr_U = 0;
bsrilu02Info_t info_M = 0;
bsrsv2Info_t   info_L = 0;
bsrsv2Info_t   info_U = 0;
int pBufferSize_M;
int pBufferSize_L;
int pBufferSize_U;
int pBufferSize;
void *pBuffer = 0;
int structural_zero;
int numerical_zero;
const double alpha = 1.;
const cusparseSolvePolicy_t policy_M = CUSPARSE_SOLVE_POLICY_NO_LEVEL;
const cusparseSolvePolicy_t policy_L = CUSPARSE_SOLVE_POLICY_NO_LEVEL;
const cusparseSolvePolicy_t policy_U = CUSPARSE_SOLVE_POLICY_USE_LEVEL;
const cusparseOperation_t trans_L  = CUSPARSE_OPERATION_NON_TRANSPOSE;
const cusparseOperation_t trans_U  = CUSPARSE_OPERATION_NON_TRANSPOSE;
const cusparseDirection_t dir = CUSPARSE_DIRECTION_COLUMN;

// step 1: create a descriptor which contains
// - matrix M is base-1
// - matrix L is base-1
// - matrix L is lower triangular
// - matrix L has unit diagonal
// - matrix U is base-1
// - matrix U is upper triangular
// - matrix U has non-unit diagonal
cusparseCreateMatDescr(&descr_M);
cusparseSetMatIndexBase(descr_M, CUSPARSE_INDEX_BASE_ONE);
cusparseSetMatType(descr_M, CUSPARSE_MATRIX_TYPE_GENERAL);

cusparseCreateMatDescr(&descr_L);
cusparseSetMatIndexBase(descr_L, CUSPARSE_INDEX_BASE_ONE);
cusparseSetMatType(descr_L, CUSPARSE_MATRIX_TYPE_GENERAL);
cusparseSetMatFillMode(descr_L, CUSPARSE_FILL_MODE_LOWER);
cusparseSetMatDiagType(descr_L, CUSPARSE_DIAG_TYPE_UNIT);

cusparseCreateMatDescr(&descr_U);
cusparseSetMatIndexBase(descr_U, CUSPARSE_INDEX_BASE_ONE);
cusparseSetMatType(descr_U, CUSPARSE_MATRIX_TYPE_GENERAL);
cusparseSetMatFillMode(descr_U, CUSPARSE_FILL_MODE_UPPER);
cusparseSetMatDiagType(descr_U, CUSPARSE_DIAG_TYPE_NON_UNIT);

// step 2: create a empty info structure
// we need one info for bsrilu02 and two info's for bsrsv2
cusparseCreateBsrilu02Info(&info_M);
cusparseCreateBsrsv2Info(&info_L);
cusparseCreateBsrsv2Info(&info_U);

// step 3: query how much memory used in bsrilu02 and bsrsv2, and allocate the buffer
cusparseDbsrilu02_bufferSize(handle, dir, mb, nnzb,
    descr_M, d_bsrVal, d_bsrRowPtr, d_bsrColInd, blockDim, info_M, &pBufferSize_M);
cusparseDbsrsv2_bufferSize(handle, dir, trans_L, mb, nnzb,
    descr_L, d_bsrVal, d_bsrRowPtr, d_bsrColInd, blockDim, info_L, &pBufferSize_L);
cusparseDbsrsv2_bufferSize(handle, dir, trans_U, mb, nnzb,
    descr_U, d_bsrVal, d_bsrRowPtr, d_bsrColInd, blockDim, info_U, &pBufferSize_U);

pBufferSize = max(pBufferSize_M, max(pBufferSize_L, pBufferSize_U));
```

--------------------------------

### Install AISLoader via Script

Source: https://docs.nvidia.com/aistore/aisloader

Use the provided installation script to set up the tool.

```bash
1| ./scripts/install_from_binaries.sh --help  
---|---  
```

--------------------------------

### Deploying AIS Node Roles

Source: https://docs.nvidia.com/aistore/command_line

Examples for starting an AIS proxy or an AIS storage target using the aisnode binary.

```bash
# Example deploying `aisnode` proxy  
$ aisnode -config=/etc/ais/config.json -local_config=/etc/ais/local_config.json -role=proxy -ntargets=16  
  
# Example deploying `aisnode` target (ie., storage server)  
$ aisnode -config=/etc/ais/config.json -local_config=/etc/ais/local_config.json -role=target
```

--------------------------------

### Install AIStore CLI from GitHub

Source: https://docs.nvidia.com/aistore/cli/help

Installs the CLI and aisloader to a specified directory using the installation script.

```bash
$ scripts/install_from_binaries.sh --dstdir /tmp/www
```

```bash
$ ls /tmp/www
ais  aisloader
```

--------------------------------

### Install CLI from release binaries

Source: https://docs.nvidia.com/aistore/cli

Use the installation script to extract the binary and configure autocompletions.

```bash
$ ./scripts/install_from_binaries.sh --dstdir /tmp/www --completions
```

--------------------------------

### View Quick Start Command Help

Source: https://docs.nvidia.com/base-command-platform/user-guide/latest/index.html

Display help information for the Quick Start project command.

```bash
$ ngc base-command quickstart project -h
```

--------------------------------

### Production Logging Configuration Example

Source: https://docs.nvidia.com/aistore/monitoring-logs

Example output showing production-tuned logging settings.

```text
$ ais config cluster log  
---  
PROPERTY         VALUE  
log.level        3  
log.max_size     4MiB  
log.max_total    512MiB  
log.flush_time   1m  
log.stats_time   3m  
log.to_stderr    false
```

--------------------------------

### Install Grafana via Docker

Source: https://docs.nvidia.com/aistore/monitoring-grafana

Quick setup for a standalone Grafana instance using Docker.

```bash
docker run -d -p 3000:3000 --name grafana grafana/grafana-oss
```

--------------------------------

### CUPTI Multiple Subscriber Implementation

Source: https://docs.nvidia.com/cupti/tutorial/tutorial.html

Complete example showing buffer callbacks, subscriber setup, and teardown logic for multiple CUPTI subscribers.

```c
void CUPTIAPI BufferCompleted_subscriber0(
    uint8_t *pBuffer, size_t size, size_t validSize,
    CUpti_BufferCallbackCompleteInfo *pCompleteInfo)
{
    if (validSize > 0)
    {
        CUpti_Activity *pRecord = NULL;
        while (cuptiActivityGetNextRecord_v2(g_subscriberHandle[0],
                pBuffer, validSize, &pRecord) == CUPTI_SUCCESS)
        {
            printf("[Subscriber 0] Activity kind: %d\n", pRecord->kind);
        }
    }
    free(pBuffer);
}

void CUPTIAPI BufferRequested_subscriber1(
    uint8_t **ppBuffer, size_t *pSize,
    size_t *pMaxNumRecords, CUpti_BufferCallbackRequestInfo *pRequestInfo)
{
    *ppBuffer = (uint8_t *)malloc(BUF_SIZE);
    if (*ppBuffer == NULL)
    {
        fprintf(stderr, "Failed to allocate CUPTI activity buffer.\n");
        exit(EXIT_FAILURE);
    }
    *pSize = BUF_SIZE;
    *pMaxNumRecords = 0;
}

void CUPTIAPI BufferCompleted_subscriber1(
    uint8_t *pBuffer, size_t size, size_t validSize,
    CUpti_BufferCallbackCompleteInfo *pCompleteInfo)
{
    if (validSize > 0)
    {
        CUpti_Activity *pRecord = NULL;
        while (cuptiActivityGetNextRecord_v2(g_subscriberHandle[1],
                pBuffer, validSize, &pRecord) == CUPTI_SUCCESS)
        {
            printf("[Subscriber 1] Activity kind: %d\n", pRecord->kind);
        }
    }
    free(pBuffer);
}

void SetupCupti()
{
    // Check for existing tracing session
    uint8_t isRunning = 0;
    CUPTI_API_CALL(cuptiIsTracingSessionRunning(&isRunning));

    // Set required pre-subscription attributes here before subscribing.

    // Query multi-subscriber state
    uint8_t multiSubState = 0;
    size_t multiSubStateSize = sizeof(uint8_t);
    CUPTI_API_CALL(cuptiActivityGetAttribute_v2(NULL,
                    CUPTI_ACTIVITY_ATTR_MULTIPLE_SUBSCRIBER_STATE,
                    &multiSubStateSize, &multiSubState));
    if (multiSubState == 2) // Not yet set by any subscriber
    {
        // No prior subscriber has set this option yet - we can proceed.
    }
    else if (multiSubState == 1) // Enabled by a prior subscriber
    {
        // Multiple subscribers already enabled by a prior subscriber - we can join.
    }
    else // Disallowed by a prior subscriber (0)
    {
        printf("Multiple subscribers are disallowed by a prior subscriber.\n");
        exit(EXIT_FAILURE);
    }

    // Subscribe both subscribers
    CUpti_SubscriberParams params0 = {};
    params0.structSize = CUpti_SubscriberParams_STRUCT_SIZE;
    params0.subscriberName = "Subscriber 0";
    params0.allowMultipleSubscribers = 1;
    CUPTI_API_CALL(cuptiSubscribe_v2(&g_subscriberHandle[0], NULL, NULL, &params0));

    CUpti_SubscriberParams params1 = {};
    params1.structSize = CUpti_SubscriberParams_STRUCT_SIZE;
    params1.subscriberName = "Subscriber 1";
    params1.allowMultipleSubscribers = 1;
    CUPTI_API_CALL(cuptiSubscribe_v2(&g_subscriberHandle[1], NULL, NULL, &params1));

    // Register per-subscriber buffer callbacks
    CUPTI_API_CALL(cuptiActivityRegisterCallbacks_v2(g_subscriberHandle[0],
                    BufferRequested_subscriber0, BufferCompleted_subscriber0));
    CUPTI_API_CALL(cuptiActivityRegisterCallbacks_v2(g_subscriberHandle[1],
                    BufferRequested_subscriber1, BufferCompleted_subscriber1));

    // Enable different activity kinds for each subscriber
    CUPTI_API_CALL(cuptiActivityEnable_v2(g_subscriberHandle[0],
                    CUPTI_ACTIVITY_KIND_CONCURRENT_KERNEL, NULL));
    CUPTI_API_CALL(cuptiActivityEnable_v2(g_subscriberHandle[0],
                    CUPTI_ACTIVITY_KIND_MEMCPY, NULL));

    CUPTI_API_CALL(cuptiActivityEnable_v2(g_subscriberHandle[1],
                    CUPTI_ACTIVITY_KIND_MEMSET, NULL));
    CUPTI_API_CALL(cuptiActivityEnable_v2(g_subscriberHandle[1],
                    CUPTI_ACTIVITY_KIND_DRIVER, NULL));
}

void TeardownCupti()
{
    CUPTI_API_CALL(cuptiActivityFlushAll(CUPTI_ACTIVITY_FLAG_FLUSH_FORCED));

    CUPTI_API_CALL(cuptiActivityDisable_v2(g_subscriberHandle[0],
                    CUPTI_ACTIVITY_KIND_CONCURRENT_KERNEL, NULL));
    CUPTI_API_CALL(cuptiActivityDisable_v2(g_subscriberHandle[0],
                    CUPTI_ACTIVITY_KIND_MEMCPY, NULL));
    CUPTI_API_CALL(cuptiActivityDisable_v2(g_subscriberHandle[1],
                    CUPTI_ACTIVITY_KIND_MEMSET, NULL));
    CUPTI_API_CALL(cuptiActivityDisable_v2(g_subscriberHandle[1],
                    CUPTI_ACTIVITY_KIND_DRIVER, NULL));

    CUPTI_API_CALL(cuptiUnsubscribe(g_subscriberHandle[0]));
    CUPTI_API_CALL(cuptiUnsubscribe(g_subscriberHandle[1]));
}

int main()
{
    uint32_t version;
    CUPTI_API_CALL(cuptiGetVersion(&version));
    if (version < 130300)
    {
        printf("CUPTI multiple subscribers require version 130300 or higher.\n");
        exit(EXIT_SUCCESS);
    }

    SetupCupti();
```

--------------------------------

### Build and install CLI from source

Source: https://docs.nvidia.com/aistore/cli

Commands to compile the CLI binary and install shell autocompletions.

```bash
$ make cli			# 1. build CLI binary and install it into your `$GOPATH/bin` directory
$ make cli-autocompletions	# 2. install CLI autocompletions (Bash or Zsh)
```

--------------------------------

### Build and Install CLI from Source

Source: https://docs.nvidia.com/attestation/nv-attestation-sdk-cpp/latest/overview.html

Commands to clone the repository, compile the CLI using CMake, and install it to the system.

```bash
git clone https://github.com/NVIDIA/attestation-sdk.git
cd attestation-sdk/nv-attestation-cli
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build
cmake --install build
sudo ldconfig
```

--------------------------------

### Complete PM Sampling Integration Example

Source: https://docs.nvidia.com/cupti/tutorial/tutorial.html

Full implementation example including headers, global variables, kernel definition, and main execution flow.

```cpp
#include <cuda_runtime.h>
#include <cuda.h>
#include <stdio.h>
#include <vector>
#include <cupti_target.h>
#include <cupti_pmsampling.h>
#include <cupti_profiler_target.h>
#include <cupti_profiler_host.h>

// Global variables for PM Sampling
CUpti_PmSampling_Object* g_pPmSamplingObject = NULL;

std::string g_chipName;
std::vector<uint8_t> g_configImage;
std::vector<uint8_t> g_counterDataImage;

std::vector<const char*> g_metrics =
{
    "gr__cycles_active.avg",     // GPU Active Cycles
    "gr__cycles_elapsed.max",     // GPU Elapsed Cycles
    "sm__cycles_active.avg"     // SM Active Cycles
};

// CUDA kernel for vector addition
__global__ void VectorAdd(const int *A, const int *B, int *C, int N)
{
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx < N)
        C[idx] = A[idx] + B[idx];
}

// Helper function to initialize PM Sampling
void InitializeAndEnablePmSampling(int deviceIndex);
void ConfigurePmSampling(uint64_t hardwareBufferSize, uint64_t samplingInterval);
void StartPmSampling();
void StopPmSampling();
void DecodeAndPrintSamplingData();
void CleanupPmSampling();

int main()
{
    const int vectorLen = 4096 * 4096 * 2;
    size_t size = vectorLen * sizeof(int);

    // Initialize CUDA
    cuInit(0);

    // Setup CUDA workload
    int *h_A = (int*)malloc(size);
    int *h_B = (int*)malloc(size);
    int *h_C = (int*)malloc(size);

    for (int i = 0; i < vectorLen; ++i) {
        h_A[i] = i;
        h_B[i] = i * 2;
    }

    int *d_A, *d_B, *d_C;
    cudaMalloc((void **)&d_A, size);
    cudaMalloc((void **)&d_B, size);
    cudaMalloc((void **)&d_C, size);

    cudaMemcpy(d_A, h_A, size, cudaMemcpyHostToDevice);
    cudaMemcpy(d_B, h_B, size, cudaMemcpyHostToDevice);
```

--------------------------------

### Install Toolkit with LangChain Integration

Source: https://docs.nvidia.com/aiqtoolkit/latest/index.html

Install the toolkit with optional LangChain dependencies.

```bash
uv pip install "nvidia-nat[langchain]"
```

```bash
pip install "nvidia-nat[langchain]"
```

--------------------------------

### Example: Launch a JupyterLab Job

Source: https://docs.nvidia.com/base-command-platform/user-guide/latest/index.html

A concrete example of launching a JupyterLab job with defined parameters.

```bash
$ ngc base-command quickstart cluster create --name "Quick Start jupyterlab tensorflow ffb4a" \
--ace ceph-sjc-4-ngc-wfs0 --cluster-lifetime 3600s --cluster-type jupyterlab \
--container-image "nvidia/tensorflow:23.08-tf2-py3" --data-output-mount-point /results \
--scheduler-instance-type dgx1v.32g.4.norm --job-order 50 --job-priority NORMAL \
--min-time-slice 0s --nworkers 1 --org nvidia --label quick_start_jupyterlab \
--workspace-mount ZNqskFA0SC2uMGUa4q-5Vg:/bcp/workspaces/49529_quick-start-jupyterlab-workspace_ceph-sjc-4-ngc-wfs0:RW
```

--------------------------------

### Show bucket properties examples

Source: https://docs.nvidia.com/aistore/cli/bucket

Examples of displaying bucket properties with compact formatting and optional prefix filtering.

```bash
$ ais bucket props show s3://bucket-name --compact
$ ais bucket props show s3://bucket_name lru --compact
$ ais bucket props show s3://ais-abhishek lru
```

--------------------------------

### Example .s3cfg Configuration

Source: https://docs.nvidia.com/aistore/s3compat

Example configuration file content for connecting s3cmd to an AIStore gateway.

```ini
host_base = 10.10.0.1:51080/s3  
host_bucket = 10.10.0.1:51080/s3/%(bucket)  
access_key = FAKEKEY  
secret_key = FAKESECRET  
signature_v2 = False  
use_https = False  
```

--------------------------------

### Development Logging Configuration Example

Source: https://docs.nvidia.com/aistore/monitoring-logs

Example JSON structure for development-oriented logging settings.

```json
"log": {  
    "level": "3",  
    "modules": "none",  
    "max_size": "4MiB",  
    "max_total": "128MiB",  
    "flush_time": "1m",  
    "stats_time": "1m",  
    "to_stderr": false  
}
```

--------------------------------

### Install NVIDIA NeMo Agent Toolkit

Source: https://docs.nvidia.com/aiqtoolkit/latest/index.html

Installation commands for the toolkit using uv or pip.

```bash
uv pip install nvidia-nat
```

```bash
pip install nvidia-nat
```

--------------------------------

### Install cuEquivariance packages

Source: https://docs.nvidia.com/cuda/cuequivariance/index.html

Use pip to install the desired frontend and corresponding CUDA kernels for PyTorch or JAX.

```bash
# Choose the frontend you want to use
pip install cuequivariance-jax
pip install cuequivariance-torch
pip install cuequivariance  # Installs only the core non-ML components

# CUDA kernels
pip install cuequivariance-ops-torch-cu12  # or -cu13
pip install cuequivariance-ops-jax-cu12    # or -cu13
```

--------------------------------

### View Quick Start Help

Source: https://docs.nvidia.com/base-command-platform/user-guide/latest/index.html

Displays the help menu for the cluster creation command.

```bash
$ ngc base-command quickstart cluster create -h
```

--------------------------------

### Digitize input array examples

Source: https://docs.nvidia.com/cupynumeric/latest/api/generated/cupynumeric.digitize.html

Examples showing basic binning behavior and the effect of the right parameter on interval boundaries.

```python
>>> x = np.array([0.2, 6.4, 3.0, 1.6])
>>> bins = np.array([0.0, 1.0, 2.5, 4.0, 10.0])
>>> inds = np.digitize(x, bins)
>>> inds
array([1, 4, 3, 2])
>>> for n in range(x.size):
...   print(bins[inds[n]-1], "<=", x[n], "<", bins[inds[n]])
...
0.0 <= 0.2 < 1.0
4.0 <= 6.4 < 10.0
2.5 <= 3.0 < 4.0
1.0 <= 1.6 < 2.5
>>> x = np.array([1.2, 10.0, 12.4, 15.5, 20.])
>>> bins = np.array([0, 5, 10, 15, 20])
>>> np.digitize(x,bins,right=True)
array([1, 2, 3, 4, 4])
>>> np.digitize(x,bins,right=False)
array([1, 3, 3, 4, 5])
```

--------------------------------

### Get objects with limit

Source: https://docs.nvidia.com/aistore/cli/storage

Example of retrieving objects with a specified limit.

```bash
ais get gs://abc /dev/null --prefix dir --limit 1234
```

--------------------------------

### Benchmark Comparison Example

Source: https://docs.nvidia.com/aistore/development

Example command to compare specific benchmarks between the current commit and a target commit.

```bash
1| $ bash ./scripts/bootstrap.sh bench cmp f9a1536f4c9af0d1ac84c200e68f2ba73676c487 --dir bench/tools/aisloader --bench BenchmarkRandom  
---|---
```

--------------------------------

### bcprun Optional Argument Examples

Source: https://docs.nvidia.com/base-command-platform/user-guide/latest/index.html

Examples demonstrating the usage of various optional flags for configuring node counts, environment variables, and working directories.

```bash
--nnodes 2
```

```bash
--npernode 8
```

```bash
--env 'var1=value1' --env 'var2=value2'
```

```bash
--workdir '$WORK_HOME/scripts' --env
                'WORK_HOME=/mnt/workspace'
```

```bash
--launcher 'mpirun --allow-run-as-root'
```

```bash
bcprun --npernode 8 -d –logdir "/workspace" -c "python3 train.py"
```

--------------------------------

### Install and Run VS Code at Runtime

Source: https://docs.nvidia.com/base-command-platform/user-guide/latest/index.html

NGC CLI command to install code-server dynamically within a standard image at job launch.

```bash
ngc base-command run --image nvidia/pytorch:22.05-py3 --port 8899 \
--name "run_vscode" \
--ace <ace>\
--org <org> \
--team <team> \
--instance dgxa100.40g.1.norm \
--result /results \
--total-runtime 1h \
--commandline "wget -nc https://github.com/coder/code-server/releases/download/v4.4.0/code-server_4.4.0_amd64.deb -o code-server_4.4.0_amd64.deb && dpkg -i ./code-server_4.4.0_amd64.deb && PASSWORD=mypass code-server --auth password --bind-addr 0.0.0.0:8899"
```

--------------------------------

### List Quick Start Clusters

Source: https://docs.nvidia.com/base-command-platform/user-guide/latest/index.html

Displays available Quick Start clusters using the NGC CLI.

```bash
$ ngc base-command quickstart cluster
```

--------------------------------

### Recommended starting configuration

Source: https://docs.nvidia.com/aistore/idle_connections

Suggested baseline settings for intra-cluster and backend traffic.

```bash
$| ais config cluster net.http.idle_conn_time=20s
$| ais config cluster net.http.idle_conns=2000
$| ais config cluster net.http.idle_conns_per_host=100
```

```bash
$| ais config cluster net.http.backend_idle_conn_time=0
```

--------------------------------

### Manage Quick Start Templates via NGC CLI

Source: https://docs.nvidia.com/base-command-platform/user-guide/latest/index.html

Base command for interacting with Quick Start templates in the NGC CLI.

```bash
$ ngc base-command quickstart project
```

--------------------------------

### Display AIStore Job Help and Examples

Source: https://docs.nvidia.com/aistore/batch

View the help documentation for the 'ais show job' command, which lists available job types and provides usage examples for monitoring running or finished jobs.

```bash
$ ais show job --help  
  
   ais show job - Show running and/or finished jobs:
     archive              blob-download  cleanup     copy-bucket   copy-objects   create-inventory  
     delete-objects       download       dsort       ec-bucket     ec-get         ec-put  
     ec-resp              elect-primary  etl-bucket  etl-inline    etl-objects    evict-objects  
     evict-remote-bucket  get-batch      list        lru-eviction  mirror         prefetch-objects  
     promote-files        put-copies     rebalance   rechunk       rename-bucket  resilver  
     summary              warm-up-metadata  
   (use any of these names with 'ais show job' command, or try shortcuts: "evict", "prefetch", "copy", "delete", "ec")
  
   e.g.:
     - show job prefetch-listrange           - show all running prefetch jobs;
     - show job prefetch                     - same as above;
     - show job prefetch --top 5             - show 5 most recent prefetch jobs;
     - show job tco-cysbohAGL                - show a given (multi-object copy/transform) job identified by its unique ID;
     - show job copy-listrange               - show all running multi-object copies;
     - show job copy-objects                 - same as above (using display name);
     - show job copy                         - show all copying jobs including both bucket-to-bucket and multi-object;
     - show job copy-objects --all           - show both running and already finished (or stopped) multi-object copies;
     - show job copy-objects --all --top 10  - show 10 most recent multi-object copy jobs;
     - show job rechunk                      - show all running rechunk jobs;
     - show job ec                           - show all erasure-coding;
     - show job list                         - show all running list-objects jobs;
     - show job ls                           - same as above;
     - show job ls --refresh 10              - same as above with periodic _refreshing_ every 10 seconds;
     - show job ls --refresh 10 --count 4    - same as above but only for the first four 10-seconds intervals;
     - show job prefetch --refresh 1m        - show all running prefetch jobs at 1 minute intervals (until Ctrl-C);
     - show job evict                        - all running bucket and/or data evicting jobs;
     - show job --all                        - show absolutely all jobs, running and finished.
```

--------------------------------

### Example usage of cusparseSbsr2csr

Source: https://docs.nvidia.com/cuda/cusparse/index.html

Demonstrates the setup and execution flow for converting a BSR matrix to CSR format.

```C
// Given BSR format (bsrRowPtrA, bsrcolIndA, bsrValA) and
// blocks of BSR format are stored in column-major order.
cusparseDirection_t dir = CUSPARSE_DIRECTION_COLUMN;
int m = mb*blockDim;
int nnzb = bsrRowPtrA[mb] - bsrRowPtrA[0]; // number of blocks
int nnz  = nnzb * blockDim * blockDim; // number of elements
cudaMalloc((void**)&csrRowPtrC, sizeof(int)*(m+1));
cudaMalloc((void**)&csrColIndC, sizeof(int)*nnz);
cudaMalloc((void**)&csrValC, sizeof(float)*nnz);
cusparseSbsr2csr(handle, dir, mb, nb,
        descrA,
        bsrValA, bsrRowPtrA, bsrColIndA,
        blockDim,
        descrC,
        csrValC, csrRowPtrC, csrColIndC);
```

--------------------------------

### Create a Bucket and Add an Object

Source: https://docs.nvidia.com/aistore/authn

Initialize a new bucket and upload a file to it using the AIStore CLI.

```bash
ais bucket create ais://nnn
ais put README.md ais://nnn
```

--------------------------------

### Display CLI help documentation

Source: https://docs.nvidia.com/aistore/getting_started

Use the --help flag to view usage examples and supported options for a specific command.

```bash
1| $ ais cp --help  
---|---
```

--------------------------------

### CSC Sorting Procedure Example

Source: https://docs.nvidia.com/cuda/cusparse/index.html

Demonstrates the full workflow for sorting a CSC matrix, including buffer allocation, permutation setup, and value gathering.

```c
// A is a 3x3 sparse matrix, base-0
//     | 1 2  |
// A = | 4 0  |
//     | 0 8  |
const int m = 3;
const int n = 2;
const int nnz = 4;
cscColPtr[n+1] = { 0, 2, 4}; // on device
cscRowInd[nnz] = { 1, 0, 2, 0}; // on device
cscVal[nnz]    = { 4.0, 1.0, 8.0, 2.0 }; // on device
size_t pBufferSizeInBytes = 0;
void *pBuffer = NULL;
int *P = NULL;

// step 1: allocate buffer
cusparseXcscsort_bufferSizeExt(handle, m, n, nnz, cscColPtr, cscRowInd, &pBufferSizeInBytes);
cudaMalloc( &pBuffer, sizeof(char)* pBufferSizeInBytes);

// step 2: setup permutation vector P to identity
cudaMalloc( (void**)&P, sizeof(int)*nnz);
cusparseCreateIdentityPermutation(handle, nnz, P);

// step 3: sort CSC format
cusparseXcscsort(handle, m, n, nnz, descrA, cscColPtr, cscRowInd, P, pBuffer);

// step 4: gather sorted cscVal
cusparseDgthr(handle, nnz, cscVal, cscVal_sorted, P, CUSPARSE_INDEX_BASE_ZERO);
```

--------------------------------

### Range Read Request within an Archive

Source: https://docs.nvidia.com/aistore/get_batch

Example of a request entry using start and length parameters to retrieve a specific byte range from an archived file.

```json
{
  "objname": "shard-000042.tar",
  "archpath": "samples/000123.bin",
  "start": 1048576,
  "length": 262144
}
```

--------------------------------

### Download TensorBoard Tutorial

Source: https://docs.nvidia.com/base-command-platform/user-guide/latest/index.html

Downloads the TensorBoard tutorial notebook into the container environment.

```bash
wget https://storage.googleapis.com/tensorflow_docs/tensorboard/docs/get_started.ipynb
```

--------------------------------

### Start AuthN Server

Source: https://docs.nvidia.com/aistore/authn

Launch the AuthN server process pointing to the directory containing the configuration file.

```bash
authn -config=<authn_config_dir>
```

--------------------------------

### Install CUPTI Python

Source: https://docs.nvidia.com/cupti-python/user-guide/topics/getting-started.html

Install the package using pip.

```bash
$ pip install cupti-python
```

--------------------------------

### Install Docker CE

Source: https://docs.nvidia.com/aistore/docker_main

Install the Docker Community Edition package.

```bash
$ sudo apt-get install docker-ce
```

--------------------------------

### Create and List with NBI

Source: https://docs.nvidia.com/aistore/nbi

Demonstrates creating an inventory snapshot for an S3 bucket and subsequently listing objects using that inventory.

```bash
$ ais nbi create s3://training-data-v5
...

$ time ais ls s3://training-data-v5 --inventory | tail
zzy/shard-0489132.tar         128.00MiB       yes
zzy/shard-0489133.tar         128.00MiB       yes
zzz/shard-0491718.tar         128.00MiB       yes
Listed 48,017,395 names

real    0m14.238s
user    0m9.671s
sys     0m3.102s
```

--------------------------------

### 100% GET Workload

Source: https://docs.nvidia.com/aistore/howto_benchmark

Executes a 100% GET operation from an AIStore bucket.

```bash
$ aisloader -bucket=nvais -duration 5s -numworkers=3 -pctput=0 -provider=ais
```

--------------------------------

### Stop Maintenance Example

Source: https://docs.nvidia.com/aistore/cli/cluster

Example of taking a node out of maintenance mode.

```bash
$ ais cluster add-remove-nodes stop-maintenance t[147665t8084]
$ ais show cluster
PROXY            MEM USED %      MEM AVAIL       UPTIME
202446p8082      0.09%           31.28GiB        80s
279128p8080[P]   0.11%           31.28GiB        90s

TARGET           MEM USED %      MEM AVAIL       CAP USED %      CAP AVAIL       CPU USED %      REBALANCE       UPTIME
147665t8084      0.10%           31.28GiB        16%             2.458TiB        0.12%           -               80s
165274t8087      0.10%           31.28GiB        16%             2.458TiB        0.12%           -               80s
```

--------------------------------

### Shutdown Node Example

Source: https://docs.nvidia.com/aistore/cli/cluster

Example of shutting down a specific target node.

```bash
$ ais cluster add-remove-nodes shutdown t[23kfa10f]
```

--------------------------------

### Start an AIStore node with custom overrides

Source: https://docs.nvidia.com/aistore/configuration

Initializes an AIStore node with additional configuration overrides applied at startup.

```bash
$ aisnode -config=/etc/ais/ais.json \
    -local_config=/etc/ais/ais_local.json \
    -role=target \
    -config_custom="client.client_timeout=13s,log.level=4"
```

--------------------------------

### Verify AWS CLI Installation

Source: https://docs.nvidia.com/base-command-platform/user-guide/latest/index.html

Checks the installed version of the AWS CLI.

```bash
aws --version
```

--------------------------------

### Creating Buckets with Namespaces

Source: https://docs.nvidia.com/aistore/bucket

Examples of creating buckets using different AWS profiles, SwiftStack configurations, and custom S3-compatible endpoints.

```bash
ais create s3://#prod/data --props="extra.aws.profile=prod-account"
ais create s3://#dev/data --props="extra.aws.profile=dev-account"

ais create s3://#swift-tenant/bucket \
  --props="extra.aws.profile=swift extra.aws.endpoint=https://swift.example.com"

ais create s3://#minio/images \
  --props="extra.aws.endpoint=http://minio.local:9000"
```

--------------------------------

### Verify Docker Compose installation

Source: https://docs.nvidia.com/aistore/docker_main

Check the installed version of Docker Compose.

```bash
$ docker-compose --version
docker-compose version 1.23.1, build 1719ceb
```

--------------------------------

### View Makefile options

Source: https://docs.nvidia.com/aistore/development

Display available make commands and usage examples.

```bash
$ make help
```

--------------------------------

### Create Buckets with Custom Properties

Source: https://docs.nvidia.com/aistore/cli/bucket

Examples of setting specific bucket properties during the creation process.

```bash
ais create ais://mmm --props="versioning.validate_warm_get=false versioning.synchronize=true"
```

```bash
ais create ais://nnn --props='mirror.enabled=true mirror.copies=4 checksum.type=md5'
```

```bash
ais create s3://bbb --props='extra.cloud.profile=prod extra.cloud.endpoint=https://s3.example.com'
```

--------------------------------

### GET error percentage

Source: https://docs.nvidia.com/aistore/monitoring-prometheus

Calculates the percentage of GET operations that resulted in errors.

```PromQL
sum(rate(ais_target_err_get_count[5m]))  
---  
/ (  
    sum(rate(ais_target_get_count[5m]))  
    + sum(rate(ais_target_err_get_count[5m]))  
  ) * 100
```

--------------------------------

### Start an AIStore node

Source: https://docs.nvidia.com/aistore/configuration

Initializes an AIStore node using specific configuration and local configuration files.

```bash
$ aisnode -config=/etc/ais/ais.json \
    -local_config=/etc/ais/ais_local.json \
    -role=target
```

--------------------------------

### Authentication Response Example

Source: https://docs.nvidia.com/base-command-platform/user-guide/latest/index.html

Example JSON structure returned by the authentication endpoint.

```json
{
  "token": "eyJraW...",
  "expires_in": 600
}
```

--------------------------------

### Initialize Environment and W&B

Source: https://docs.nvidia.com/base-command-platform/user-guide/latest/index.html

Install dependencies, configure the netrc file for W&B authentication, and initialize the tracking run.

```python
!pip install tensorflow
import numpy as np
from tensorflow import keras
from tensorflow.keras import layers
import wandb
import os

# 1. Import the W&B API key from private config workspace by defining NETRC fileos.environ["NETRC"]= "/homews-demouser/bcpwandb/wandbconf/config.netrc"

# 2. Initialize the W&B run
wandb.init(project = "nvtest-repro", id = "MNIST_run_epoch-128_bs-15", name = "NGC-JOB-ID_" + os.environ["NGC_JOB_ID"])
```

--------------------------------

### Log Format Example

Source: https://docs.nvidia.com/aistore/monitoring-logs

Example of a standard AIS informational log entry.

```text
I 2025‑05‑19 13:42:17.791884 cpu:60 Reducing GOMAXPROCS (prev=256) to 32  
---  
│ │            │      │              └─ message  
│ │            │      └─ Go file:line inside AIS source  
│ │            └─ timestamp (µs precision)  
│ └─ severity prefix  
└─ 'I' for INFO  
```

--------------------------------

### Install TensorRT via pip

Source: https://docs.nvidia.com/deeplearning/tensorrt/index.html

Command to install the TensorRT Python bindings and libraries.

```bash
pip install tensorrt
```

--------------------------------

### Verify TensorRT Installation

Source: https://docs.nvidia.com/deeplearning/tensorrt/index.html

Commands to verify the installation of TensorRT via CLI or Python.

```bash
trtexec --help
```

```python
import tensorrt; print(tensorrt.__version__)
```

--------------------------------

### Install numba-cuda

Source: https://docs.nvidia.com/cupti-python/samples.html

Command to install the numba-cuda package and required dependencies for CUDA 13.x.

```bash
$ pip install numba-cuda[cu13]
```

--------------------------------

### List Objects with Basic CLI Usage

Source: https://docs.nvidia.com/aistore/bucket

Examples demonstrating how to list objects in a bucket, including handling remote buckets that are not yet in AIS.

```bash
$ ais ls s3://bucket
Error: ErrRemoteBckNotFound: aws bucket "s3://ais-vm" does not exist
Tip: use '--all' to list all objects including remote

# and note: `--all` can be used just once and only when the remote is not in AIS
$ ais ls s3://bucket --all --limit 4
```

--------------------------------

### Install cuOpt Managed Service

Source: https://docs.nvidia.com/cuopt/service/latest/quickstart.html

Install the package into the active Python virtual environment.

```bash
pip install .
```

--------------------------------

### Create and configure a Dockerfile

Source: https://docs.nvidia.com/base-command-platform/user-guide/latest/index.html

Defines a base image and installation steps for a custom container build.

```bash
$ mkdir mydocker
$ cd mydocker
$ vi Dockerfile
$ more Dockerfile
# This is the base container for the new container.
FROM nvcr.io/<org>/<team>/<container-name>:<tag>
# Update the apt-get database
RUN apt-get update
# Install the package octave with apt-get
RUN apt-get install -y octave
$
```

--------------------------------

### Startup Log Output Example

Source: https://docs.nvidia.com/aistore/monitoring-logs

Sample log lines generated by AIS during the startup process.

```text
I 19:28:24.774518 config:2143 log.dir: "/var/log/ais"; l4.proto: tcp; pub port: 51080; verbosity: 3  
---  
I 19:28:24.774523 config:2145 config: "/etc/ais/.ais.conf"; stats_time: 10s; authentication: false; backends: [aws]
```

--------------------------------

### Kernel Declaration Example

Source: https://docs.nvidia.com/compute-sanitizer/ComputeSanitizer/index.html

Example of global kernel declarations used for filter testing.

```cpp
__global__ void gamma(int *bufer);
__global__ void delta(int *bufer);
__global__ void epsilon(int *bufer);
```

--------------------------------

### Terminate All Jobs Example

Source: https://docs.nvidia.com/cosmos-curator-lha/current/api_reference.html

Example cURL command to terminate all in-progress jobs for the organization.

```bash
curl -X DELETE "https://api.ngc.nvidia.com/v1/cosmos/jobs" \
  -H "Authorization: Bearer ${COSMOS_KEY}"
```

--------------------------------

### Erasure Coding Help and Examples

Source: https://docs.nvidia.com/aistore/cli/bucket

Displays usage information and examples for the ec-encode command.

```bash
1| $ ais start ec-encode --help  
---|---  
2|   
3| NAME:  
4|    ais start ec-encode - Erasure code entire bucket, e.g.:  
5|      - 'ais start ec-encode ais://nnn -d 8 -p 2'                          - erasure-code ais://nnn for 8 data and 2 parity slices;  
6|      - 'ais start ec-encode ais://nnn --data-slices 8 --parity-slices 2'  - same as above;  
7|      - 'ais start ec-encode ais://nnn --recover'                          - check and make sure that every ais://nnn object is properly erasure-coded.  
8|    see also: 'ais start mirror'  
9|   
10| USAGE:  
11|    ais start ec-encode BUCKET [command options]  
12|   
13| OPTIONS:  
14|    --data-slices value, -d value    Number of data slices (default: 2)  
15|    --non-verbose, --nv              Non-verbose (quiet) output, minimized reporting, fewer warnings  
16|    --parity-slices value, -p value  Number of parity slices (default: 2)  
17|    --recover                        Check and make sure that each and every object is properly erasure coded  
18|    --help, -h                       Show help
```

--------------------------------

### Start Dynamo Frontend

Source: https://docs.nvidia.com/dynamo/index.html

Initializes the OpenAI-compatible frontend service using file-based discovery to bypass etcd requirements.

```bash
python3 -m dynamo.frontend --discovery-backend file
```

--------------------------------

### Create and Configure Buckets via CLI

Source: https://docs.nvidia.com/aistore/bucket

Examples of creating buckets with specific properties like versioning, mirroring, and provider-specific configurations using the AIS CLI.

```bash
ais create gs://abc --props="versioning.validate_warm_get=false versioning.synchronize=true"
ais create ais://abc --props="mirror.enabled=true mirror.copies=3"
ais create ais://abc --props='{"mirror": {"enabled": true, "copies": 3}}'
ais create ais://abc \
  --props='{
    "mirror":   {"enabled": true, "copies": 3},
    "checksum": {"type": "xxhash", "validate_warm_get": true}
  }'
ais create s3://logs \
  --props='{
    "extra":      {"aws": {"profile": "prod", "endpoint": "https://s3.example.com"}},
    "rate_limit": {"backend": {"enabled": true, "max_bps": "800MB"}}
  }'
```

--------------------------------

### List objects using inventory via CLI

Source: https://docs.nvidia.com/aistore/nbi

Examples of using the --inventory flag with various cloud providers and options in the AIStore CLI.

```bash
ais ls gs://abc --inventory
```

```bash
ais ls oci://abc --inventory --paged --prefix subdir
```

```bash
ais ls s3://abc --inventory --inv-name my-first-inventory
```

```bash
ais ls s3://abc --inventory --nr --prefix images/
```

--------------------------------

### Verify Docker installation

Source: https://docs.nvidia.com/aistore/docker_main

Run the hello-world image to confirm Docker is functioning correctly.

```bash
$ sudo docker run hello-world
```

--------------------------------

### Install and Import Boto3

Source: https://docs.nvidia.com/base-command-platform/user-guide/latest/index.html

Installs the Boto3 library and imports necessary modules for S3 interaction.

```python
!pip installboto3

import boto3
import io
import os
```

--------------------------------

### Install MsgPack generator

Source: https://docs.nvidia.com/aistore/msgp

Use the make command to install or update the MessagePack generator tool.

```bash
$ make msgp-update
```
Tokens:
Raw