NVIDIA GPU Operator

https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/
Admin
NVIDIA GPU Operator automates the management of NVIDIA software components needed to provision GPUs...
Tokens:97,258
Snippets:1,631
Trust Score:10
Update:1 month ago
Show doc for...
Tokens:
Raw
### Example Helm Install Output

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/deploy-kata-containers.html

Sample output indicating a successful Helm deployment.

```text
LAST DEPLOYED: Wed Apr  1 17:03:00 2026
NAMESPACE: kata-system
STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete
TEST SUITE: None
```

--------------------------------

### Example Output of Operator Installation

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-kata.html

This is an example output showing the Kubernetes resources created during the installation of the Confidential Containers Operator. It includes namespaces, custom resource definitions, service accounts, roles, and deployments.

```yaml
namespace/confidential-containers-system created
customresourcedefinition.apiextensions.k8s.io/ccruntimes.confidentialcontainers.org created
serviceaccount/cc-operator-controller-manager created
role.rbac.authorization.k8s.io/cc-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/cc-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/cc-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/cc-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/cc-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/cc-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/cc-operator-proxy-rolebinding created
configmap/cc-operator-manager-config created
service/cc-operator-controller-manager-metrics-service created
deployment.apps/cc-operator-controller-manager create
```

--------------------------------

### Install Helm

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/deploy-kata-containers.html

Download and execute the official Helm installation script.

```bash
$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 \
      && chmod 700 get_helm.sh \
      && ./get_helm.sh
```

--------------------------------

### Install Helm 3

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator.html

Download and execute the script to install Helm version 3.

```bash
$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 \
   && chmod 700 get_helm.sh \
   && ./get_helm.sh

```

--------------------------------

### Install Helm CLI

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html

Use this command to download and install the Helm CLI. Ensure you have `curl` and `chmod` available.

```bash
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 \
    && chmod 700 get_helm.sh \
    && ./get_helm.sh 
```

--------------------------------

### Launch Concurrent Jobs on MIG Devices with Argo Workflows

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html

This example uses Argo Workflows to launch multiple CUDA containers onto MIG devices. Ensure Argo Workflows is installed and the Argo CLI is configured. The A100 GPU must be configured into MIG devices beforehand.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
 generateName: argo-mig-example-
spec:
 entrypoint: argo-mig-result-example
templates:
- name: argo-mig-result-example
 steps:
 - - name: generate
 template: gen-mig-device-list
 # Iterate over the list of numbers generated by the generate step above
 - - name: argo-mig
 template: argo-mig
 arguments:
 parameters:
 - name: argo-mig
 value: "{{item}}"
 withParam: "{{steps.generate.outputs.result}}"

# Generate a list of numbers in JSON format
- name: gen-mig-device-list
 script:
 image: python:alpine3.6
 command: [python]
 source: |
 import json
 import sys
 json.dump([i for i in range(0, 2)], sys.stdout)

- name: argo-mig
 retryStrategy:
 limit: 10
 retryPolicy: "Always"
 inputs:
 parameters:
 - name: argo-mig
 container:
 image: nvidia/samples:vectoradd-cuda11.2.1
 resources:
 limits:
 nvidia.com/gpu: 1
 nodeSelector:
 nvidia.com/gpu.product: A100-SXM4-40GB-MIG-3g.20gb
```

--------------------------------

### Install GPU Operator Script

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator-nvaie.html

Execute the downloaded GPU Operator installer script with the 'install' command. This script automates the deployment of the GPU Operator on your cluster.

```bash
bash gpu-operator-nvaie.sh install
```

--------------------------------

### Install GPU Operator with Default Configuration

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html

Install the NVIDIA GPU Operator using Helm with default settings. The `--wait` flag ensures the installation completes before proceeding.

```bash
helm install --wait --generate-name \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --version=v25.3.3 
```

--------------------------------

### Custom MIG Configuration with Helm

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/release-notes.html

Example of creating a config map with custom MIG profiles during installation or upgrade using Helm. This allows for fine-grained control over MIG configurations.

```yaml
helm install gpu-operator community/gpu-operator --version v24.6.0 -n gpu-operator --create-namespace \
  --set mig.config.enabled=true \
  --set mig.config.configMapName=gpu-operator-mig-config \
  --set mig.config.configMapNamespace=gpu-operator
```

--------------------------------

### Install GPU Operator with Pre-Installed Driver

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-crds.html

Installs the GPU Operator when the driver is already present on the host.

```bash
$ helm upgrade --install gpu-operator nvidia/gpu-operator \
    --version=v26.7.0 \
    --namespace gpu-operator \
    --create-namespace \
    --set clusterPolicy.deployCR=false \
    --set gpuCluster.deployCR=true \
    --set driver.enabled=false
```

--------------------------------

### Install NVIDIA Guest Driver on Fedora

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-kubevirt-dra.html

Commands to install the necessary kernel headers and NVIDIA drivers within a Fedora guest VM.

```bash
$ sudo dnf -y install kernel-devel-matched kernel-headers dnf-plugins-core
$ sudo dnf config-manager addrepo \
    --from-repofile=https://developer.download.nvidia.com/compute/cuda/repos/fedora44/x86_64/cuda-fedora44.repo
$ sudo dnf clean expire-cache
$ sudo dnf -y install nvidia-driver-cuda kmod-nvidia-open-dkms
$ sudo reboot
```

--------------------------------

### Example Output of Pods and Services

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-kata.html

This example output displays the running pods and services within the 'confidential-containers-system' namespace. It shows the status, readiness, and age of the 'cc-operator-controller-manager' pod and its associated metrics service.

```yaml
NAME READY STATUS RESTARTS AGE
pod/cc-operator-controller-manager-c98c4ff74-ksb4q 2/2 Running 0 2m59s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cc-operator-controller-manager-metrics-service ClusterIP 10.98.221.141 <none> 8443/TCP 2m59s
```

--------------------------------

### Verify Node Feature Rules Installation

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-confidential-containers.html

Confirm that the NVIDIA NFD node feature rules have been installed on the cluster.

```bash
kubectl get nodefeaturerules nvidia-nfd-node-featurerules
```

--------------------------------

### Install GPU Operator with Operator-Managed Driver

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-crds.html

Installs the GPU Operator with the driver managed by the operator.

```bash
$ helm upgrade --install gpu-operator nvidia/gpu-operator \
    --version=v26.7.0 \
    --namespace gpu-operator \
    --create-namespace \
    --set clusterPolicy.deployCR=false \
    --set gpuCluster.deployCR=true \
    --set driver.nvidiaDriverCRD.enabled=true
```

--------------------------------

### Install Argo Workflows

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html

Installs Argo Workflows components into a Kubernetes cluster using kubectl. This command creates a namespace and applies the quick-start PostgreSQL manifest.

```bash
kubectl create ns argo && kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo-workflows/stable/manifests/quick-start-postgres.yaml
```

--------------------------------

### Install GPU Operator Components

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator-gov-ready.html

Labels the namespace, adds the NVIDIA Helm repository, and installs the GPU Operator with specific FIPS-compliant driver settings.

```bash
$ kubectl label --overwrite ns gpu-operator pod-security.kubernetes.io/enforce=privileged
```

```bash
$ helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \
    && helm repo update
```

```bash
$  helm install gpu-operator nvidia/gpu-operator \
     --namespace gpu-operator \
     --set driver.secretEnv=ubuntu-fips-secret \
     --set driver.repository=nvcr.io/nvidia \
     --set driver.version=580.95.05-stig-fips \
     --set driver.image=gpu-driver-stig-fips \
     --set driver.imagePullSecrets={ngc-secret} \
     --set nfd.enabled=false
```

--------------------------------

### Install GPU Operator with Network Operator Drivers

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

Install the GPU Operator using Helm, relying on the Network Operator for DMA-BUF and network device drivers.

```bash
$ helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --version=v25.3.3 \
    
```

--------------------------------

### Start RDMA Performance Test Server

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

Execute `ib_write_bw` within the first pod to start the RDMA performance test server. This command configures the test to use CUDA DMA buffers and specifies the InfiniBand device.

```bash
kubectl exec -it demo-pod-1 -- ib_write_bw --use_cuda=0 --use_cuda_dmabuf \
    -d mlx5_0 -a -F --report_gbits -q 1
```

--------------------------------

### Install GPU Operator with Host Network Drivers

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

Install the GPU Operator using Helm, enabling GPUDirect RDMA with host-installed network device drivers (MOFED).

```bash
$ helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --version=v25.3.3 \
     --set driver.rdma.useHostMofed=true
```

--------------------------------

### Example Rollout Status Output

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/deploy-kata-containers.html

Sample output showing the progress of the DaemonSet rollout.

```text
Waiting for daemon set "kata-deploy" rollout to finish: 0 of 1 updated pods are available...
daemon set "kata-deploy" successfully rolled out
```

--------------------------------

### Get GKE Cluster Credentials

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/google-gke.html

Retrieves authentication credentials for your GKE cluster. Ensure you have the `gcloud` CLI installed and configured.

```bash
gcloud container clusters get-credentials demo-cluster --location us-west1
```

--------------------------------

### List Available Runtime Classes

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-confidential-containers.html

After installation, verify that the specific NVIDIA-related runtime classes, such as `kata-qemu-nvidia-gpu` and `kata-qemu-nvidia-gpu-snp`, are registered and available in the cluster.

```bash
$ kubectl get runtimeclass 
```

--------------------------------

### Example Runtime Class Output

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/deploy-kata-containers.html

Sample output listing available runtime classes.

```text
NAME                       HANDLER                    AGE
kata-qemu-nvidia-gpu       kata-qemu-nvidia-gpu       40s
kata-qemu-nvidia-gpu-snp   kata-qemu-nvidia-gpu-snp   40s
kata-qemu-nvidia-gpu-tdx   kata-qemu-nvidia-gpu-tdx   40s
```

--------------------------------

### Install GPU Operator in a Specific Namespace

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator.html

Installs the GPU Operator into the 'gpu-operator' namespace. Ensure the namespace is created before installation if it does not exist.

```bash
$ helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator

```

--------------------------------

### Install NVIDIA GPU Operator with Helm

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/google-gke.html

Install the NVIDIA GPU Operator using Helm, specifying custom paths for driver and toolkit installations, and enabling CDI for container image management. Driver installation is disabled as it's managed separately.

```bash
$ helm install --wait --generate-name \
    -n gpu-operator \
    nvidia/gpu-operator \
    --version=v25.3.3 \
    --set hostPaths.driverInstallDir=/home/kubernetes/bin/nvidia \
    --set toolkit.installDir=/home/kubernetes/bin/nvidia \
    --set cdi.enabled=true \
    --set cdi.default=true \
    --set driver.enabled=false
```

--------------------------------

### Example Pod Status Output

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/deploy-kata-containers.html

Sample output showing running pods in the kata-system namespace.

```text
NAME                READY   STATUS    RESTARTS   AGE
kata-deploy-b2lzs   1/1     Running   0          6m37s
```

--------------------------------

### Example DeviceClass output

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-crds.html

Expected output showing available DeviceClasses.

```text
NAME                                        AGE
compute-domain-daemon.nvidia.com            2m
compute-domain-default-channel.nvidia.com   2m
gpu.nvidia.com                              2m
mig.nvidia.com                              2m
vfio.gpu.nvidia.com                         2m
```

--------------------------------

### Install GPU Operator with Pre-installed Drivers

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator.html

Install the GPU Operator when NVIDIA drivers are already pre-installed on the system image. This is done by setting `driver.enabled=false` in the Helm installation.

```bash
$ helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --set driver.enabled=false

```

--------------------------------

### Example Node List Output

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/deploy-kata-containers.html

Sample output showing node status and versions.

```text
NAME          STATUS   ROLES           AGE   VERSION
node-01       Ready    <none>          10d   v1.34.0
node-02       Ready    <none>          10d   v1.34.0
```

--------------------------------

### Example resource output

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/deploy-kata-containers.html

Shows the format of model-specific resource types advertised on a node.

```json
"nvidia.com/GH100_H100L_94GB": "1"
```

--------------------------------

### Create nvidia.conf for GPU Firmware Logging

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/custom-driver-params.html

Create a configuration file for the 'nvidia' module to enable GPU firmware logging. This file should contain parameters as key-value pairs, one per line.

```bash
cat nvidia.conf
NVreg_EnableGpuFirmwareLogs=2
```

--------------------------------

### Install Confidential Containers Operator

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-kata.html

Apply the Kubernetes configuration to install the Confidential Containers Operator using its release manifest. The installation uses the version specified in the VERSION environment variable.

```bash
kubectl apply -k "github.com/confidential-containers/operator/config/release?ref=${VERSION}"
```

--------------------------------

### Example Node Label Output

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/deploy-kata-containers.html

Sample output confirming the presence of the workload configuration label.

```text
nvidia.com/gpu.workload.config: vm-passthrough
```

--------------------------------

### Install NVIDIA Driver Installer for COS

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/google-gke.html

Apply the DaemonSet manifest to install NVIDIA drivers on Container-Optimized OS (COS) nodes. Ensure you use the correct manifest for your node image.

```bash
$ kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml
```

--------------------------------

### Build and Execute Partition Verification Tools

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-kubevirt-dra.html

Steps to build the list-partitions utility and execute it within the driver container to verify partition status.

```bash
$ git clone --depth 1 https://github.com/NVIDIA/go-nvfm.git
$ cd go-nvfm
$ make example-list-partitions
```

```bash
$ kubectl get pods --namespace gpu-operator \
    --selector app.kubernetes.io/component=nvidia-driver \
    --field-selector spec.nodeName=<node-name>
```

```bash
$ kubectl cp --namespace gpu-operator \
    --container nvidia-driver-ctr \
    ./list-partitions <driver-pod>:/run/nvidia/list-partitions
```

```bash
$ kubectl exec --namespace gpu-operator \
    --container nvidia-driver-ctr \
    <driver-pod> -- bash -c \
    "NVFM_UNIX_SOCKET_PATH=/run/nvidia-fabricmanager/socket /run/nvidia/list-partitions"
```

--------------------------------

### Install GPU Operator on RHEL with Pre-installed Drivers

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html

Install the GPU Operator on RHEL when NVIDIA GPU drivers are already present on worker nodes. This command disables the driver installation by the operator.

```bash
helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --version=v25.3.3 \
     --set driver.enabled=false
```

--------------------------------

### List Kata Runtime Class Files

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-kata.html

Verify that the NVIDIA Kata Manager has correctly installed the necessary configuration files for the 'kata-qemu-nvidia-gpu' runtime class. This ensures the runtime class is properly configured.

```bash
$ ls -1 /opt/nvidia-gpu-operator/artifacts/runtimeclasses/kata-qemu-nvidia-gpu/
```

--------------------------------

### Install GPU Operator with Custom Driver Image

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator.html

Installs the GPU Operator using Helm, overriding the default driver repository and version to use a custom-built driver image. The --wait flag ensures the installation completes before proceeding.

```bash
$ helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --set driver.repository=docker.io/nvidia \
     --set driver.version="465.27"

```

--------------------------------

### Apply and verify the VirtualMachine

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-kubevirt-dra.html

Commands to apply the VM manifest and wait for the instance to reach a ready state.

```bash
$ kubectl apply -f dra-gpu-vm.yaml
$ kubectl wait vmi/dra-gpu-vm --for=condition=Ready --timeout=10m
```

--------------------------------

### Install GPU Operator with Helm

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-driver-configuration.html

Install the NVIDIA GPU Operator using Helm, ensuring the NVIDIA driver custom resource definition (CRD) is enabled. This command installs the operator and creates a default driver CR unless disabled.

```bash
helm install --wait --generate-name \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --version=v25.3.3 \
 --set driver.nvidiaDriverCRD.enabled=true
```

--------------------------------

### Install Kata Deploy Helm Chart

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/deploy-kata-containers.html

Deploy the kata-deploy chart with specific configurations for NVIDIA GPU support.

```bash
$ helm install kata-deploy "${CHART}" \
   --namespace kata-system --create-namespace \
   --set nfd.enabled=false \
   -f kata-nvidia-gpu-values.yaml \
   --version "${VERSION}"
```

--------------------------------

### Apply Second Pod Manifest

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

Apply the `demo-pod-2.yaml` manifest to create the second pod, which will participate in the RDMA performance test.

```bash
kubectl apply -f demo-pod-2.yaml
```

--------------------------------

### Verify DeviceClasses availability

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-crds.html

List all available DeviceClasses to ensure proper configuration.

```bash
$ kubectl get deviceclass
```

--------------------------------

### Install GPU Operator

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator-proxy.html

Use this command to install the GPU Operator using Helm. Ensure you have a values.yaml file configured.

```bash
helm upgrade --install gpu-operator gpu-operator-v25.3.3.tgz -n gpu-operator --create-namespace -f values.yaml
```

--------------------------------

### Inspect GPUCluster conditions

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-crds.html

Use this command to troubleshoot if the GPUCluster resource fails to reach the ready state.

```bash
$ kubectl describe gpucluster gpu-cluster
```

--------------------------------

### Install Kubeflow MPI Operator

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-cds.html

Use this command to install the Kubeflow MPI Operator. Ensure you are using a compatible version.

```bash
kubectl create -f https://github.com/kubeflow/mpi-operator/releases/download/v0.6.0/mpi-operator.yaml
```

--------------------------------

### Describe Driver DaemonSet

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

Use this command to inspect the pod template for the driver daemon set and confirm the presence of the mofed-validation init container and nvidia-driver-ctr containers. The nvidia-peermem-ctr container is only present if RDMA was enabled during installation.

```bash
$ kubectl describe ds -n gpu-operator nvidia-driver-daemonset
```

--------------------------------

### Apply First Pod Manifest

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

Apply the `demo-pod-1.yaml` manifest to create the first pod in your Kubernetes cluster.

```bash
kubectl apply -f demo-pod-1.yaml
```

--------------------------------

### Install GPU Operator (Drivers Disabled)

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator.html

Install the GPU Operator using Helm, disabling the driver provisioning. This is for scenarios where drivers are pre-installed.

```bash
$ helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
      nvidia/gpu-operator \
      --set driver.enabled=false \
      --set toolkit.enabled=false
```

--------------------------------

### Run RDMA Performance Test Client

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

From the second pod, run `ib_write_bw` as a client, connecting to the server pod's IP address. This initiates the RDMA data transfer test and reports bandwidth in Gbps.

```bash
kubectl exec -it demo-pod-2 -- ib_write_bw -n 5000 --use_cuda=0 --use_cuda_dmabuf \
    -d mlx5_0 -a -F --report_gbits -q 1 192.168.38.90
```

--------------------------------

### Example GPUCluster output

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-crds.html

Expected output format for a successful GPUCluster deployment.

```text
NAME          STATUS   AGE
gpu-cluster   ready    3m
```

--------------------------------

### Install GPU Operator with MIG Disabled

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html

Use this Helm command to install the GPU Operator, ensuring driver management is disabled for MIG configurations.

```bash
helm install gpu-operator \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --version=v25.3.3 \
    --set driver.enabled=false
```

--------------------------------

### Inspect Kata Manager Runtime Class Files

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-confidential-containers.html

Verify that the NVIDIA Kata Manager has correctly installed the necessary runtime class files for confidential containers, specifically for the `kata-qemu-nvidia-gpu-snp` configuration.

```bash
$ ls -1 /opt/nvidia-gpu-operator/artifacts/runtimeclasses/kata-qemu-nvidia-gpu-snp/ 
```

--------------------------------

### Install GPU Operator (Toolkit Disabled)

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator.html

Install the GPU Operator using Helm, disabling the toolkit provisioning. This is for scenarios where the NVIDIA Container Toolkit is pre-installed.

```bash
$ helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --set toolkit.enabled=false
```

--------------------------------

### Install GPU Operator with MicroK8s Configuration

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html

Install the GPU Operator using Helm, specifying the MicroK8s-specific environment variables via the --set flag.

```bash
helm install gpu-operator -n gpu-operator --create-namespace \
 nvidia/gpu-operator $HELM_OPTIONS \
 --version=v25.3.3 \
 --set toolkit.env[0].name=CONTAINERD_CONFIG \
 --set toolkit.env[0].value=/var/snap/microk8s/current/args/containerd-template.toml \
 --set toolkit.env[1].name=CONTAINERD_SOCKET \
 --set toolkit.env[1].value=/var/snap/microk8s/common/run/containerd.sock \
 --set toolkit.env[2].name=CONTAINERD_RUNTIME_CLASS \
 --set toolkit.env[2].value=nvidia \
 --set toolkit.env[3].name=CONTAINERD_SET_AS_DEFAULT \
 --set-string toolkit.env[3].value=true
```

--------------------------------

### Describe Node for GPU Resources

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html

Use this command to verify that the node advertises additional GPU resources after configuration changes. The output varies based on your GPU and configuration.

```bash
kubectl describe node <node-name>
```

--------------------------------

### Install GPU Operator with Custom Driver Image

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html

Install the GPU Operator, overriding the default driver image repository and version with your custom image details.

```bash
helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --version=v25.3.3 \
     --set driver.repository=docker.io/nvidia \
     --set driver.version="465.27"
```

--------------------------------

### Install GPU Operator with Legacy nvidia-peermem

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

Install the GPU Operator using Helm, enabling the legacy nvidia-peermem kernel module for RDMA instead of DMA-BUF.

```bash
$ helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --version=v25.3.3 \
     --set driver.rdma.enabled=true
```

--------------------------------

### View Runtime Classes

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-kata.html

Lists the available runtime classes after the Operator has created the base runtime classes. This command helps verify the installation.

```bash
$ kubectl get runtimeclass
```

--------------------------------

### Install Console Plugin Helm Chart

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/enable-gpu-op-dashboard.html

Installs the console plugin for the NVIDIA GPU Operator in the specified namespace. This enables GPU usage information in the OpenShift console.

```bash
$ helm install -n nvidia-gpu-operator console-plugin-nvidia-gpu rh-ecosystem-edge/console-plugin-nvidia-gpu
```

--------------------------------

### Create NLS Client License Token ConfigMap

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator-nvaie.html

Create a Kubernetes ConfigMap named 'licensing-config-new' in the 'gpu-operator' namespace. This ConfigMap will contain the vGPU license configuration file and the NLS client license token.

```bash
sudo touch gridd.conf
kubectl create configmap licensing-config-new \
    -n gpu-operator --from-file=gridd.conf --from-file=<path>/client_configuration_token.tok
```

--------------------------------

### View Deployed Resources

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/openshift/enable-gpu-op-dashboard.html

Lists all resources associated with the console plugin for the NVIDIA GPU Operator in the specified namespace. Useful for verifying the installation.

```bash
$ oc -n nvidia-gpu-operator get all -l app.kubernetes.io/name=console-plugin-nvidia-gpu
```

--------------------------------

### Install GPU Operator with RDMA Enabled

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

Install the NVIDIA GPU Operator using Helm, enabling GPUDirect Storage and the legacy nvidia-peermem kernel module for RDMA.

```bash
helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --version=v25.3.3 \
     --set gds.enabled=true \
     --set driver.rdma.enabled=true
```

--------------------------------

### Install GPU Operator with Time-Slicing Configuration

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html

Install the NVIDIA GPU Operator using Helm, enabling time-slicing by specifying the device plugin configuration map name.

```bash
helm install gpu-operator nvidia/gpu-operator \
    -n gpu-operator \
    --version=v25.3.3 \
    --set devicePlugin.config.name=time-slicing-config
```

--------------------------------

### Define First Pod for RDMA Test

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

Create a Pod manifest (`demo-pod-1.yaml`) that uses the `mellanox/cuda-perftest` container and is configured to use the previously created `demo-macvlannetwork`. Ensure GPU and RDMA resources are requested.

```yaml
apiVersion: v1
kind: Pod
metadata:
 name: demo-pod-1
annotations:
 k8s.v1.cni.cncf.io/networks: demo-macvlannetwork
 # If a network with static IPAM is used replace network annotation with the below.
 # k8s.v1.cni.cncf.io/networks: '[
 # {"name": "rdma-net",
 # "ips": ["192.168.111.101/24"],
 # "gateway": ["192.168.111.1"]
 # }
 # ]'
spec:
 nodeSelector:
 # Note: Replace hostname or remove selector altogether
 kubernetes.io/hostname: nvnode1
restartPolicy: OnFailure
containers:
 - image: mellanox/cuda-perftest
 name: rdma-gpu-test-ctr
 securityContext:
 capabilities:
 add: [ "IPC_LOCK" ]
 resources:
 limits:
 nvidia.com/gpu: 1
 rdma/rdma_shared_device_a: 1
 requests:
 nvidia.com/gpu: 1
 rdma/rdma_shared_device_a: 1
```

--------------------------------

### Install GPU Operator with Pre-installed Container Toolkit

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html

Install the GPU Operator when only the NVIDIA Container Toolkit is pre-installed on worker nodes. This command provisions the GPU driver.

```bash
helm install --wait --generate-name \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --version=v25.3.3 \
    --set toolkit.enabled=false
```

--------------------------------

### Copy vGPU Driver Files

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator-vgpu.html

Copy the NVIDIA vGPU guest driver and catalog file into the drivers directory. Ensure the driver file matches the expected pattern `*-grid.run`.

```bash
cp <local-driver-download-directory>/*-grid.run drivers/
cp vgpuDriverCatalog.yaml drivers/
```

--------------------------------

### Configure GRUB for IOMMU

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/deploy-kata-containers.html

Add the IOMMU kernel parameter to the GRUB configuration file.

```text
...
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on modprobe.blacklist=nouveau"
...
```

--------------------------------

### Install DRA Driver with Operator-Provided GPU Driver

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-intro-install.html

Installs the nvidia-dra-driver-gpu Helm chart with configuration for an operator-provided GPU driver. Sets resources.gpus.enabled to false and specifies the nvidiaDriverRoot.

```bash
helm install nvidia-dra-driver-gpu nvidia/nvidia-dra-driver-gpu \
    --version="25.3.0-rc.5" \
    --create-namespace \
    --namespace nvidia-dra-driver-gpu \
    --set resources.gpus.enabled=false \
    --set nvidiaDriverRoot=/run/nvidia/driver 
```

--------------------------------

### Install GPU Operator with Helm

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator-vgpu.html

Install the NVIDIA GPU Operator using Helm. This command deploys the operator with specified configurations for drivers, image pull secrets, and licensing.

```bash
helm install --wait --generate-name \
     -n gpu-operator --create-namespace \
     nvidia/gpu-operator \
     --set driver.repository=${PRIVATE_REGISTRY} \
     --set driver.version=${VERSION} \
     --set driver.imagePullSecrets={$REGISTRY_SECRET_NAME} \
     --set driver.licensingConfig.configMapName=licensing-config
```

--------------------------------

### Example Output of GPU Attestation Verification

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-confidential-containers.html

Illustrates the expected output when the `SmallGPUTest.py` script successfully verifies GPU attestation, including driver and VBIOS versions, certificate chain validation, signature verification, and measurement comparisons.

```text
[SmallGPUTest] node name : thisNode1
[['LOCAL_GPU_CLAIMS', <Devices.GPU: 2>, <Environment.LOCAL: 2>, '', '', '']]
[SmallGPUTest] call attest() - expecting True
Number of GPUs available : 1
-----------------------------------
Fetching GPU 0 information from GPU driver.
VERIFYING GPU : 0
 Driver version fetched : 535.86.05
 VBIOS version fetched : 96.00.5e.00.01
 Validating GPU certificate chains.
 GPU attestation report certificate chain validation successful.
 The certificate chain revocation status verification successful.
 Authenticating attestation report
 The nonce in the SPDM GET MEASUREMENT request message is matching with the generated nonce.
 Driver version fetched from the attestation report : 535.86.05
 VBIOS version fetched from the attestation report : 96.00.5e.00.01
 Attestation report signature verification successful.
 Attestation report verification successful.
 Authenticating the RIMs.
 Authenticating Driver RIM
 Schema validation passed.
 driver RIM certificate chain verification successful.
 The certificate chain revocation status verification successful.
 driver RIM signature verification successful.
 Driver RIM verification successful
 Authenticating VBIOS RIM.
RIM Schema validation passed.
vbios RIM certificate chain verification successful.
 The certificate chain revocation status verification successful.
vbios RIM signature verification successful.
 VBIOS RIM verification successful
 Comparing measurements (runtime vs golden)
 The runtime measurements are matching with the golden measurements.
 GPU is in the expected state.
 GPU 0 verified successfully.
 attestation result: True
 claims list:: {'x-nv-gpu-availability': True, 'x-nv-gpu-attestation-report-available': ...
 True
[SmallGPUTest] token : [["JWT", "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.e..."],
 {"LOCAL_GPU_CLAIMS": "eyJhbGciOiJIUzI1NiIsInR5cCI..."}]
[SmallGPUTest] call validate_token() - expecting True
 True
```

--------------------------------

### Install GPU Operator with Custom GPU Clients

Source: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html

Install the GPU Operator using Helm, specifying a custom config map for GPU clients and disabling driver management.

```bash
helm install gpu-operator \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --version=v25.3.3 \
    --set migManager.gpuClientsConfig.name=gpu-clients
 --set driver.enabled=false
```
Tokens:
Raw