### Example Output of Operator Installation

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-kata.html

This is an example output showing the Kubernetes resources created during the installation of the Confidential Containers Operator. It includes namespaces, custom resource definitions, service accounts, roles, and deployments.

```yaml
namespace/confidential-containers-system created
customresourcedefinition.apiextensions.k8s.io/ccruntimes.confidentialcontainers.org created
serviceaccount/cc-operator-controller-manager created
role.rbac.authorization.k8s.io/cc-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/cc-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/cc-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/cc-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/cc-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/cc-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/cc-operator-proxy-rolebinding created
configmap/cc-operator-manager-config created
service/cc-operator-controller-manager-metrics-service created
deployment.apps/cc-operator-controller-manager create
```

--------------------------------

### Example Output of oc get pods

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/nvaie-with-ocp.html

This is an example output showing the status of various NVIDIA GPU Operator pods, indicating a successful deployment.

```text
NAME                                                                  READY   STATUS      RESTARTS   AGE
pod/gpu-feature-discovery-hlpgs                                       1/1     Running     0          91m
pod/gpu-operator-8dc8d6648-jzhnr                                      1/1     Running     0          94m
pod/nvidia-dcgm-exporter-ds9xd                                        1/1     Running     0          91m
pod/nvidia-dcgm-k7tz6                                                 1/1     Running     0          91m
pod/nvidia-device-plugin-daemonset-nqxmc                              1/1     Running     0          91m
pod/nvidia-driver-daemonset-49.84.202202081504-0-9df9j                2/2     Running     0          91m
pod/nvidia-node-status-exporter-7bhdk                                 1/1     Running     0          91m
```

--------------------------------

### Get InstallPlan Name

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/install-gpu-ocp.html

Retrieve the name of the install plan for the GPU Operator in the specified namespace.

```bash
$ INSTALL_PLAN=$(oc get installplan -n nvidia-gpu-operator -oname)

```

--------------------------------

### Get Starting CSV

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/install-gpu-ocp.html

Determine the current Cluster Service Version (CSV) for the specified channel and store it in a variable. This is used to define the initial version to install.

```bash
$ STARTING_CSV=$(oc get packagemanifests/gpu-operator-certified -n openshift-marketplace -ojson | jq -r '.status.channels[] | select(.name == "'$CHANNEL'") | .currentCSV')

```

--------------------------------

### Install Helm CLI

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html

Use this command to download and install the Helm CLI. Ensure you have `curl` and `chmod` available.

```bash
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 \
    && chmod 700 get_helm.sh \
    && ./get_helm.sh 
```

--------------------------------

### Install Helm 3

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator.html

Download and execute the script to install Helm version 3.

```bash
$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 \
   && chmod 700 get_helm.sh \
   && ./get_helm.sh

```

--------------------------------

### Verify Driver Toolkit Installation

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/steps-overview.html

Use this command to verify if the Driver Toolkit is installed on your OpenShift cluster. The example output shows the image repository and tags for the driver-toolkit.

```bash
$ oc get -n openshift is/driver-toolkit

```

--------------------------------

### Install Packages for Private Registry

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/mirror-gpu-ocp-disconnected.html

Installs necessary packages including podman for containerization and httpd-tools for user authentication.

```bash
$ yum -y install podman httpd httpd-tools

```

--------------------------------

### Install GPU Operator with Default Configuration

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html

Install the NVIDIA GPU Operator using Helm with default settings. The `--wait` flag ensures the installation completes before proceeding.

```bash
helm install --wait --generate-name \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --version=v25.3.3 
```

--------------------------------

### Launch Concurrent Jobs on MIG Devices with Argo Workflows

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html

This example uses Argo Workflows to launch multiple CUDA containers onto MIG devices. Ensure Argo Workflows is installed and the Argo CLI is configured. The A100 GPU must be configured into MIG devices beforehand.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
 generateName: argo-mig-example-
spec:
 entrypoint: argo-mig-result-example
templates:
- name: argo-mig-result-example
 steps:
 - - name: generate
 template: gen-mig-device-list
 # Iterate over the list of numbers generated by the generate step above
 - - name: argo-mig
 template: argo-mig
 arguments:
 parameters:
 - name: argo-mig
 value: "{{item}}"
 withParam: "{{steps.generate.outputs.result}}"

# Generate a list of numbers in JSON format
- name: gen-mig-device-list
 script:
 image: python:alpine3.6
 command: [python]
 source: |
 import json
 import sys
 json.dump([i for i in range(0, 2)], sys.stdout)

- name: argo-mig
 retryStrategy:
 limit: 10
 retryPolicy: "Always"
 inputs:
 parameters:
 - name: argo-mig
 container:
 image: nvidia/samples:vectoradd-cuda11.2.1
 resources:
 limits:
 nvidia.com/gpu: 1
 nodeSelector:
 nvidia.com/gpu.product: A100-SXM4-40GB-MIG-3g.20gb
```

--------------------------------

### Verify Install Plan Status

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/install-gpu-ocp.html

Check the status of the install plan to confirm it has been created and is awaiting approval.

```bash
$ oc get installplan -n nvidia-gpu-operator

```

--------------------------------

### Install GPU Operator Script

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator-nvaie.html

Execute the downloaded GPU Operator installer script with the 'install' command. This script automates the deployment of the GPU Operator on your cluster.

```bash
bash gpu-operator-nvaie.sh install
```

--------------------------------

### Get ClusterPolicy Example

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/install-gpu-ocp.html

Retrieve the default ClusterPolicy example from the Subscription CR annotations. This JSON output can be saved to a file for modification.

```bash
$ oc get csv -n nvidia-gpu-operator $STARTING_CSV -o jsonpath='{.metadata.annotations.alm-examples}' | jq -r 'map(select(.kind == "ClusterPolicy")) | .[0]'> clusterpolicy.json

```

--------------------------------

### Custom MIG Configuration with Helm

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/release-notes.html

Example of creating a config map with custom MIG profiles during installation or upgrade using Helm. This allows for fine-grained control over MIG configurations.

```yaml
helm install gpu-operator community/gpu-operator --version v24.6.0 -n gpu-operator --create-namespace \
  --set mig.config.enabled=true \
  --set mig.config.configMapName=gpu-operator-mig-config \
  --set mig.config.configMapNamespace=gpu-operator
```

--------------------------------

### Example Output of Resource Types

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/deploy-kata-containers.html

This is an example of the output you might see when querying a node for its advertised NVIDIA resource types after configuring model-specific aliases.

```json
"nvidia.com/GH100_H100L_94GB": "1"
```

--------------------------------

### Example Output of Pods and Services

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-kata.html

This example output displays the running pods and services within the 'confidential-containers-system' namespace. It shows the status, readiness, and age of the 'cc-operator-controller-manager' pod and its associated metrics service.

```yaml
NAME READY STATUS RESTARTS AGE
pod/cc-operator-controller-manager-c98c4ff74-ksb4q 2/2 Running 0 2m59s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cc-operator-controller-manager-metrics-service ClusterIP 10.98.221.141 <none> 8443/TCP 2m59s
```

--------------------------------

### Install Apache httpd

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/mirror-gpu-ocp-disconnected.html

Installs the Apache web server package using yum. This is a prerequisite for setting up an HTTP server.

```bash
$ yum install httpd -y

```

--------------------------------

### Install Helm CLI

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator-gov-ready.html

Installs the Helm CLI, a package manager for Kubernetes, which is required for installing the GPU Operator.

```bash
$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 \
    && chmod 700 get_helm.sh \
    && ./get_helm.sh

```

--------------------------------

### Start and Enable Apache httpd Service

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/mirror-gpu-ocp-disconnected.html

Restarts the Apache web server to apply changes and enables it to start automatically on system boot.

```bash
$ systemctl restart httpd

```

```bash
$ systemctl enable httpd

```

--------------------------------

### Verify Node Feature Rules Installation

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-confidential-containers.html

Confirm that the NVIDIA NFD node feature rules have been installed on the cluster.

```bash
kubectl get nodefeaturerules nvidia-nfd-node-featurerules
```

--------------------------------

### Install Argo Workflows

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html

Installs Argo Workflows components into a Kubernetes cluster using kubectl. This command creates a namespace and applies the quick-start PostgreSQL manifest.

```bash
kubectl create ns argo && kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo-workflows/stable/manifests/quick-start-postgres.yaml
```

--------------------------------

### Install GPU Operator with Network Operator Drivers

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

Install the GPU Operator using Helm, relying on the Network Operator for DMA-BUF and network device drivers.

```bash
$ helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --version=v25.3.3 \
    
```

--------------------------------

### Example nvidia-smi Output

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/mig-ocp.html

This is an example output from the nvidia-smi command, showing details about the NVIDIA GPUs, their status, and MIG device configurations. It is used to verify the presence and status of MIG devices.

```text
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.12             Driver Version: 535.104.12   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------|
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA H100 80GB HBM3          On  | 00000000:1B:00.0 Off |                   On |
| N/A   25C    P0              71W / 700W |                  N/A |     N/A      Default |
|                                         |                      |              Enabled |
+-----------------------------------------+----------------------+----------------------|
|   1  NVIDIA H100 80GB HBM3          On  | 00000000:43:00.0 Off |                   On |
| N/A   26C    P0              70W / 700W |                  N/A |     N/A      Default |
|                                         |                      |              Enabled |
+-----------------------------------------+----------------------+----------------------|
|   2  NVIDIA H100 80GB HBM3          On  | 00000000:52:00.0 Off |                   On |
| N/A   31C    P0              72W / 700W |                  N/A |     N/A      Default |
|                                         |                      |              Enabled |
+-----------------------------------------+----------------------+----------------------|
|   3  NVIDIA H100 80GB HBM3          On  | 00000000:61:00.0 Off |                   On |
| N/A   29C    P0              71W / 700W |                  N/A |     N/A      Default |
|                                         |                      |              Enabled |
+-----------------------------------------+----------------------+----------------------|
|   4  NVIDIA H100 80GB HBM3          On  | 00000000:9D:00.0 Off |                   On |
| N/A   26C    P0              71W / 700W |                  N/A |     N/A      Default |
|                                         |                      |              Enabled |
+-----------------------------------------+----------------------+----------------------|
|   5  NVIDIA H100 80GB HBM3          On  | 00000000:C3:00.0 Off |                   On |
| N/A   25C    P0              70W / 700W |                  N/A |     N/A      Default |
|                                         |                      |              Enabled |
+-----------------------------------------+----------------------+----------------------|
|   6  NVIDIA H100 80GB HBM3          On  | 00000000:D1:00.0 Off |                   On |
| N/A   29C    P0              73W / 700W |                  N/A |     N/A      Default |
|                                         |                      |              Enabled |
+-----------------------------------------+----------------------+----------------------|
|   7  NVIDIA H100 80GB HBM3          On  | 00000000:DF:00.0 Off |                   On |
| N/A   31C    P0              72W / 700W |                  N/A |     N/A      Default |
|                                         |                      |              Enabled |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| MIG devices:                                                                          |
+------------------+--------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |                   Memory-Usage |        Vol|      Shared           |
|      ID  ID  Dev |                     BAR1-Usage | SM     Unc| CE ENC DEC OFA JPG    |
|                  |                                |        ECC|                       |
|==================+================================+===========+=======================|
|  0    2   0   0  |              16MiB / 40448MiB  | 60      0 |  3   0    3    0    3 |
|                  |               0MiB / 65535MiB  |           |                       |
+------------------+--------------------------------+-----------+-----------------------+
|  0    3   0   1  |              11MiB / 20096MiB  | 32      0 |  2   0    2    0    2 |
|                  |               0MiB / 32767MiB  |           |                       |
+------------------+--------------------------------+-----------+-----------------------+
|  0    9   0   2  |               5MiB /  9984MiB  | 16      0 |  1   0    1    0    1 |

```

--------------------------------

### Download oc-mirror Tools

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/mirror-gpu-ocp-disconnected.html

Download the `oc-mirror` command-line tools. After downloading, copy the binary to a directory in your PATH, such as `/usr/bin`, and verify the installation with `oc mirror help`.

```bash
$ wget https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable/oc-mirror.tar.gz

```

--------------------------------

### Create Subscription

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/install-gpu-ocp.html

Apply the Subscription custom resource to initiate the installation process for the GPU Operator.

```bash
$ oc create -f nvidia-gpu-sub.yaml

```

--------------------------------

### List Available Runtime Classes

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-confidential-containers.html

After installation, verify that the specific NVIDIA-related runtime classes, such as `kata-qemu-nvidia-gpu` and `kata-qemu-nvidia-gpu-snp`, are registered and available in the cluster.

```bash
$ kubectl get runtimeclass 
```

--------------------------------

### Create Directories for Registry

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/mirror-gpu-ocp-disconnected.html

Sets up the required directory structure for the registry's data, authentication, and certificates.

```bash
$ mkdir -p /opt/registry/{auth,certs,data}

```

--------------------------------

### Create nvidia.conf for GPU Firmware Logging

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/custom-driver-params.html

Create a configuration file for the 'nvidia' module to enable GPU firmware logging. This file should contain parameters as key-value pairs, one per line.

```bash
cat nvidia.conf
NVreg_EnableGpuFirmwareLogs=2
```

--------------------------------

### Get GKE Cluster Credentials

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/google-gke.html

Retrieves authentication credentials for your GKE cluster. Ensure you have the `gcloud` CLI installed and configured.

```bash
gcloud container clusters get-credentials demo-cluster --location us-west1
```

--------------------------------

### Get Nodes and GPU Counts with oc

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/nvaie-with-ocp.html

Use this command to display node names and their associated GPU capacities after the ClusterPolicy installation.

```bash
$ oc get nodes -o=custom-columns='Node:metadata.name,GPUs:status.capacity.nvidia\.com/gpu'
```

--------------------------------

### Create Directory for Serving BaseOS RPMs

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/mirror-gpu-ocp-disconnected.html

Create a directory structure under `/var/www/html` to host and serve the BaseOS RPM packages. This is part of setting up a local repository.

```bash
$ mkdir -p /var/www/html/content/dist/rhel8/8/x86_64/baseos/

```

--------------------------------

### Check default vGPU device allocation

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/openshift-virtualization.html

After installing the GPU Operator without specific vGPU configuration, check the node's allocatable resources to see the default vGPU devices created. This example shows 4 NVIDIA_A10-12Q devices.

```bash
$ oc get node cnt-server-2 -o json | jq '.status.allocatable | with_entries(select(.key | startswith("nvidia.com/"))) | with_entries(select(.value != "0"))'
{
   "nvidia.com/NVIDIA_A10-12Q": "4"
}


```

--------------------------------

### List Kata Runtime Class Files

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-kata.html

Verify that the NVIDIA Kata Manager has correctly installed the necessary configuration files for the 'kata-qemu-nvidia-gpu' runtime class. This ensures the runtime class is properly configured.

```bash
$ ls -1 /opt/nvidia-gpu-operator/artifacts/runtimeclasses/kata-qemu-nvidia-gpu/
```

--------------------------------

### Install GPU Operator in a Specific Namespace

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator.html

Installs the GPU Operator into the 'gpu-operator' namespace. Ensure the namespace is created before installation if it does not exist.

```bash
$ helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator

```

--------------------------------

### List Enabled Repositories

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/mirror-gpu-ocp-disconnected.html

List all repositories currently enabled for your system. This helps identify the `repoid` needed for synchronization.

```bash
$ subscription-manager repos --list-enabled

```

--------------------------------

### Approve Install Plan

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/install-gpu-ocp.html

Approve the pending install plan for the GPU Operator using `oc patch`. This allows the operator to be installed.

```bash
$ oc patch $INSTALL_PLAN -n nvidia-gpu-operator --type merge --patch '{"spec":{"approved":true }}

```

--------------------------------

### Install NVIDIA GPU Operator with Helm

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/google-gke.html

Install the NVIDIA GPU Operator using Helm, specifying custom paths for driver and toolkit installations, and enabling CDI for container image management. Driver installation is disabled as it's managed separately.

```bash
$ helm install --wait --generate-name \
    -n gpu-operator \
    nvidia/gpu-operator \
    --version=v25.3.3 \
    --set hostPaths.driverInstallDir=/home/kubernetes/bin/nvidia \
    --set toolkit.installDir=/home/kubernetes/bin/nvidia \
    --set cdi.enabled=true \
    --set cdi.default=true \
    --set driver.enabled=false
```

--------------------------------

### Create Directory for Serving AppStream RPMs

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/mirror-gpu-ocp-disconnected.html

Create a directory structure under `/var/www/html` to host and serve the AppStream RPM packages. This is part of setting up a local repository.

```bash
mkdir -p /var/www/html/content/dist/rhel8/8/x86_64/appstream/

```

--------------------------------

### Create Namespace

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/install-gpu-ocp.html

Apply the Namespace custom resource to create the specified namespace.

```bash
$ oc create -f nvidia-gpu-operator.yaml

```

--------------------------------

### Verify Kata Runtime Class Availability

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/deploy-kata-containers.html

List available runtime classes to confirm that Kata Container runtime classes, such as `kata-qemu-nvidia-gpu`, have been successfully installed.

```bash
kubectl get runtimeclass | grep kata-qemu-nvidia-gpu

```

--------------------------------

### Install GPU Operator with Pre-installed Drivers

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator.html

Install the GPU Operator when NVIDIA drivers are already pre-installed on the system image. This is done by setting `driver.enabled=false` in the Helm installation.

```bash
$ helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --set driver.enabled=false

```

--------------------------------

### Install GPU Operator for ComputeDomain

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-crds.html

Installs the NVIDIA GPU Operator with the device plugin disabled. This is a basic installation for ComputeDomain support without specific device plugin configurations.

```bash
helm upgrade --install gpu-operator nvidia/gpu-operator \
  --version=v26.3.2 \
  --create-namespace \
  --namespace gpu-operator

```

--------------------------------

### Apply Second Pod Manifest

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

Apply the `demo-pod-2.yaml` manifest to create the second pod, which will participate in the RDMA performance test.

```bash
kubectl apply -f demo-pod-2.yaml
```

--------------------------------

### Install NVIDIA Driver Installer for COS

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/google-gke.html

Apply the DaemonSet manifest to install NVIDIA drivers on Container-Optimized OS (COS) nodes. Ensure you use the correct manifest for your node image.

```bash
$ kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml
```

--------------------------------

### Install Confidential Containers Operator

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-kata.html

Apply the Kubernetes configuration to install the Confidential Containers Operator using its release manifest. The installation uses the version specified in the VERSION environment variable.

```bash
kubectl apply -k "github.com/confidential-containers/operator/config/release?ref=${VERSION}"
```

--------------------------------

### Describe Driver DaemonSet

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

Use this command to inspect the pod template for the driver daemon set and confirm the presence of the mofed-validation init container and nvidia-driver-ctr containers. The nvidia-peermem-ctr container is only present if RDMA was enabled during installation.

```bash
$ kubectl describe ds -n gpu-operator nvidia-driver-daemonset
```

--------------------------------

### Apply First Pod Manifest

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

Apply the `demo-pod-1.yaml` manifest to create the first pod in your Kubernetes cluster.

```bash
kubectl apply -f demo-pod-1.yaml
```

--------------------------------

### Install GPU Operator on RHEL with Pre-installed Drivers

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html

Install the GPU Operator on RHEL when NVIDIA GPU drivers are already present on worker nodes. This command disables the driver installation by the operator.

```bash
helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --version=v25.3.3 \
     --set driver.enabled=false
```

--------------------------------

### Run RDMA Performance Test Client

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

From the second pod, run `ib_write_bw` as a client, connecting to the server pod's IP address. This initiates the RDMA data transfer test and reports bandwidth in Gbps.

```bash
kubectl exec -it demo-pod-2 -- ib_write_bw -n 5000 --use_cuda=0 --use_cuda_dmabuf \
    -d mlx5_0 -a -F --report_gbits -q 1 192.168.38.90
```

--------------------------------

### Create Directory for Mirrored Repos

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/mirror-gpu-ocp-disconnected.html

Create a directory to store the downloaded RPM repositories. Ensure sufficient disk space is available.

```bash
$ mkdir -p /opt/mirror-repos/

```

--------------------------------

### Install GPU Operator with Custom Driver Image

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator.html

Installs the GPU Operator using Helm, overriding the default driver repository and version to use a custom-built driver image. The --wait flag ensures the installation completes before proceeding.

```bash
$ helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --set driver.repository=docker.io/nvidia \
     --set driver.version="465.27"

```

--------------------------------

### Inspect Kata Manager Runtime Class Files

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-confidential-containers.html

Verify that the NVIDIA Kata Manager has correctly installed the necessary runtime class files for confidential containers, specifically for the `kata-qemu-nvidia-gpu-snp` configuration.

```bash
$ ls -1 /opt/nvidia-gpu-operator/artifacts/runtimeclasses/kata-qemu-nvidia-gpu-snp/ 
```

--------------------------------

### Install GPU Operator with Helm

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-driver-configuration.html

Install the NVIDIA GPU Operator using Helm, ensuring the NVIDIA driver custom resource definition (CRD) is enabled. This command installs the operator and creates a default driver CR unless disabled.

```bash
helm install --wait --generate-name \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --version=v25.3.3 \
 --set driver.nvidiaDriverCRD.enabled=true
```

--------------------------------

### Start RDMA Performance Test Server

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

Execute `ib_write_bw` within the first pod to start the RDMA performance test server. This command configures the test to use CUDA DMA buffers and specifies the InfiniBand device.

```bash
kubectl exec -it demo-pod-1 -- ib_write_bw --use_cuda=0 --use_cuda_dmabuf \
    -d mlx5_0 -a -F --report_gbits -q 1
```

--------------------------------

### Describe Node for GPU Resources

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html

Use this command to verify that the node advertises additional GPU resources after configuration changes. The output varies based on your GPU and configuration.

```bash
kubectl describe node <node-name>
```

--------------------------------

### View Runtime Classes

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-kata.html

Lists the available runtime classes after the Operator has created the base runtime classes. This command helps verify the installation.

```bash
$ kubectl get runtimeclass
```

--------------------------------

### Example of Updated Pull Secret

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/mirror-gpu-ocp-disconnected.html

This is an example of how the pull secret file should look after adding your mirror registry configuration.

```json
{
  "auths": {
    "jump_hostname:5000": {
      "auth": "BGVtbYk3ZHAtqXs=",
      "email": "you@example.com"
    },
    "cloud.openshift.com": {
      "auth": "b3BlbnNo...",
      "email": "you@example.com"
    },
    "quay.io": {
      "auth": "b3BlbnNo...",
      "email": "you@example.com"
    },
    "registry.connect.redhat.com": {
      "auth": "NTE3Njg5Nj...",
      "email": "you@example.com"
    },
    "registry.redhat.io": {
      "auth": "NTE3Njg5Nj...",
      "email": "you@example.com"
    }
  }
}

```

--------------------------------

### Create NLS Client License Token ConfigMap

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator-nvaie.html

Create a Kubernetes ConfigMap named 'licensing-config-new' in the 'gpu-operator' namespace. This ConfigMap will contain the vGPU license configuration file and the NLS client license token.

```bash
sudo touch gridd.conf
kubectl create configmap licensing-config-new \
    -n gpu-operator --from-file=gridd.conf --from-file=<path>/client_configuration_token.tok
```

--------------------------------

### Install GPU Operator

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator-proxy.html

Use this command to install the GPU Operator using Helm. Ensure you have a values.yaml file configured.

```bash
helm upgrade --install gpu-operator gpu-operator-v25.3.3.tgz -n gpu-operator --create-namespace -f values.yaml
```

--------------------------------

### Initialize ImageSet Configuration for GPU Operator

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/mirror-gpu-ocp-disconnected.html

Initialize the imageset-config-gpu.yaml file for mirroring GPU operator images. Replace ${JUMP_HOST} with your registry host.

```bash
oc-mirror init --registry ${JUMP_HOST}:5000/oc-mirror-gpu-metadata > imageset-config-gpu.yaml

```

--------------------------------

### Install Kubeflow MPI Operator

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-cds.html

Use this command to install the Kubeflow MPI Operator. Ensure you are using a compatible version.

```bash
kubectl create -f https://github.com/kubeflow/mpi-operator/releases/download/v0.6.0/mpi-operator.yaml
```

--------------------------------

### View Deployed Resources

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/enable-gpu-op-dashboard.html

Lists all resources associated with the console plugin for the NVIDIA GPU Operator in the specified namespace. Useful for verifying the installation.

```bash
$ oc -n nvidia-gpu-operator get all -l app.kubernetes.io/name=console-plugin-nvidia-gpu
```

--------------------------------

### Install GPU Operator (Drivers Disabled)

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator.html

Install the GPU Operator using Helm, disabling the driver provisioning. This is for scenarios where drivers are pre-installed.

```bash
$ helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
      nvidia/gpu-operator \
      --set driver.enabled=false \
      --set toolkit.enabled=false
```

--------------------------------

### Install GPU Operator for GPU Allocation

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-crds.html

Installs the NVIDIA GPU Operator with the NVIDIA Kubernetes Device Plugin disabled and configures the driver manager to use a specific node label for GPU pod eviction. Ensure the node label matches the one used for DRA driver installation.

```bash
helm upgrade --install gpu-operator nvidia/gpu-operator \
  --version=v26.3.2 \
  --create-namespace \
  --namespace gpu-operator \
  --set devicePlugin.enabled=false \
  --set driver.manager.env[0].name=NODE_LABEL_FOR_GPU_POD_EVICTION \
  --set driver.manager.env[0].value="nvidia.com/dra-kubelet-plugin"

```

--------------------------------

### Copy vGPU Driver Files

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator-vgpu.html

Copy the NVIDIA vGPU guest driver and catalog file into the drivers directory. Ensure the driver file matches the expected pattern `*-grid.run`.

```bash
cp <local-driver-download-directory>/*-grid.run drivers/
cp vgpuDriverCatalog.yaml drivers/
```

--------------------------------

### Install GPU Operator with MIG Disabled

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html

Use this Helm command to install the GPU Operator, ensuring driver management is disabled for MIG configurations.

```bash
helm install gpu-operator \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --version=v25.3.3 \
    --set driver.enabled=false
```

--------------------------------

### Define First Pod for RDMA Test

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

Create a Pod manifest (`demo-pod-1.yaml`) that uses the `mellanox/cuda-perftest` container and is configured to use the previously created `demo-macvlannetwork`. Ensure GPU and RDMA resources are requested.

```yaml
apiVersion: v1
kind: Pod
metadata:
 name: demo-pod-1
annotations:
 k8s.v1.cni.cncf.io/networks: demo-macvlannetwork
 # If a network with static IPAM is used replace network annotation with the below.
 # k8s.v1.cni.cncf.io/networks: '[
 # {"name": "rdma-net",
 # "ips": ["192.168.111.101/24"],
 # "gateway": ["192.168.111.1"]
 # }
 # ]'
spec:
 nodeSelector:
 # Note: Replace hostname or remove selector altogether
 kubernetes.io/hostname: nvnode1
restartPolicy: OnFailure
containers:
 - image: mellanox/cuda-perftest
 name: rdma-gpu-test-ctr
 securityContext:
 capabilities:
 add: [ "IPC_LOCK" ]
 resources:
 limits:
 nvidia.com/gpu: 1
 rdma/rdma_shared_device_a: 1
 requests:
 nvidia.com/gpu: 1
 rdma/rdma_shared_device_a: 1
```

--------------------------------

### Install GPU Operator (Toolkit Disabled)

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator.html

Install the GPU Operator using Helm, disabling the toolkit provisioning. This is for scenarios where the NVIDIA Container Toolkit is pre-installed.

```bash
$ helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --set toolkit.enabled=false
```

--------------------------------

### Example Output of GPU Attestation Verification

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-confidential-containers.html

Illustrates the expected output when the `SmallGPUTest.py` script successfully verifies GPU attestation, including driver and VBIOS versions, certificate chain validation, signature verification, and measurement comparisons.

```text
[SmallGPUTest] node name : thisNode1
[['LOCAL_GPU_CLAIMS', <Devices.GPU: 2>, <Environment.LOCAL: 2>, '', '', '']]
[SmallGPUTest] call attest() - expecting True
Number of GPUs available : 1
-----------------------------------
Fetching GPU 0 information from GPU driver.
VERIFYING GPU : 0
 Driver version fetched : 535.86.05
 VBIOS version fetched : 96.00.5e.00.01
 Validating GPU certificate chains.
 GPU attestation report certificate chain validation successful.
 The certificate chain revocation status verification successful.
 Authenticating attestation report
 The nonce in the SPDM GET MEASUREMENT request message is matching with the generated nonce.
 Driver version fetched from the attestation report : 535.86.05
 VBIOS version fetched from the attestation report : 96.00.5e.00.01
 Attestation report signature verification successful.
 Attestation report verification successful.
 Authenticating the RIMs.
 Authenticating Driver RIM
 Schema validation passed.
 driver RIM certificate chain verification successful.
 The certificate chain revocation status verification successful.
 driver RIM signature verification successful.
 Driver RIM verification successful
 Authenticating VBIOS RIM.
RIM Schema validation passed.
vbios RIM certificate chain verification successful.
 The certificate chain revocation status verification successful.
vbios RIM signature verification successful.
 VBIOS RIM verification successful
 Comparing measurements (runtime vs golden)
 The runtime measurements are matching with the golden measurements.
 GPU is in the expected state.
 GPU 0 verified successfully.
 attestation result: True
 claims list:: {'x-nv-gpu-availability': True, 'x-nv-gpu-attestation-report-available': ...
 True
[SmallGPUTest] token : [["JWT", "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.e..."],
 {"LOCAL_GPU_CLAIMS": "eyJhbGciOiJIUzI1NiIsInR5cCI..."}]
[SmallGPUTest] call validate_token() - expecting True
 True
```

--------------------------------

### Enable IOMMU in GRUB Configuration

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/deploy-kata-containers.html

Add the `intel_iommu=on` or `amd_iommu=on` kernel command-line argument to the GRUB configuration file to enable IOMMU support. This is typically done in `/etc/default/grub`.

```bash
... 
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on modprobe.blacklist=nouveau"
...
```

--------------------------------

### Initialize oc-mirror for NFD

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/mirror-gpu-ocp-disconnected.html

Initializes the oc-mirror configuration for mirroring the NFD CatalogSource. Ensure JUMP_HOST is set to your private registry's hostname.

```bash
$ oc mirror init --registry ${JUMP_HOST}:5000/oc-mirror-nfd-metadata > imageset-config-nfd.yaml
```

--------------------------------

### Install GPU Operator with MicroK8s Configuration

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html

Install the GPU Operator using Helm, specifying the MicroK8s-specific environment variables via the --set flag.

```bash
helm install gpu-operator -n gpu-operator --create-namespace \
 nvidia/gpu-operator $HELM_OPTIONS \
 --version=v25.3.3 \
 --set toolkit.env[0].name=CONTAINERD_CONFIG \
 --set toolkit.env[0].value=/var/snap/microk8s/current/args/containerd-template.toml \
 --set toolkit.env[1].name=CONTAINERD_SOCKET \
 --set toolkit.env[1].value=/var/snap/microk8s/common/run/containerd.sock \
 --set toolkit.env[2].name=CONTAINERD_RUNTIME_CLASS \
 --set toolkit.env[2].value=nvidia \
 --set toolkit.env[3].name=CONTAINERD_SET_AS_DEFAULT \
 --set-string toolkit.env[3].value=true
```

--------------------------------

### Install GPU Operator with Custom Driver Image

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html

Install the GPU Operator, overriding the default driver image repository and version with your custom image details.

```bash
helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --version=v25.3.3 \
     --set driver.repository=docker.io/nvidia \
     --set driver.version="465.27"
```

--------------------------------

### Install GPU Operator with Legacy nvidia-peermem

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

Install the GPU Operator using Helm, enabling the legacy nvidia-peermem kernel module for RDMA instead of DMA-BUF.

```bash
$ helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --version=v25.3.3 \
     --set driver.rdma.enabled=true
```

--------------------------------

### Install Helm on Kubernetes Cluster

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/deploy-kata-containers.html

Install Helm version 3 on your Kubernetes cluster using the provided script. Ensure the script is downloaded, made executable, and then run.

```bash
$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 \
      && chmod 700 get_helm.sh \
      && ./get_helm.sh

```

--------------------------------

### Install GPU Operator with RDMA Enabled

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

Install the NVIDIA GPU Operator using Helm, enabling GPUDirect Storage and the legacy nvidia-peermem kernel module for RDMA.

```bash
helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --version=v25.3.3 \
     --set gds.enabled=true \
     --set driver.rdma.enabled=true
```