NVIDIA Container Toolkit

https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/
Admin
NVIDIA Container Toolkit is a collection of libraries and utilities that enable users to build and...
Tokens:26,730
Snippets:433
Trust Score:10
Update:1 month ago
Show doc for...
Tokens:
Raw
### Run Sample CUDA Container with Podman

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/sample-workload.html

Use this command to run a sample CUDA container with Podman. This verifies the installation and configuration, similar to the Docker example.

```bash
podman run --rm --security-opt=label=disable \
 --device=nvidia.com/gpu=all \
 ubuntu nvidia-smi
```

--------------------------------

### Install Prerequisites for APT

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install/repo-apt.html

Install necessary packages for repository management on Debian-based systems.

```bash
$ sudo apt-get update && sudo apt-get install -y --no-install-recommends \
   ca-certificates \
   curl \
   gnupg2
```

--------------------------------

### Install NVIDIA Container Toolkit via Zypper

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install/repo-apt.html

Install the toolkit packages using zypper with automatic GPG key import.

```bash
$  export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.0-1
   sudo zypper --gpg-auto-import-keys install -y \
      nvidia-container-toolkit-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      nvidia-container-toolkit-base-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      libnvidia-container-tools-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      libnvidia-container1-${NVIDIA_CONTAINER_TOOLKIT_VERSION}
```

--------------------------------

### Install Prerequisites for DNF

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install/repo-apt.html

Install curl as a prerequisite for repository configuration on RPM-based systems.

```bash
$ sudo dnf install -y \
   curl
```

--------------------------------

### Start a GPU-enabled CUDA container using --gpus

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/runtime/docker.html

Basic command to run a container with all available GPUs.

```bash
$ docker run --rm --gpus all nvidia/cuda nvidia-smi
```

--------------------------------

### Run Container with CUDA and NVML Capabilities

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/runtime/docker.html

Examples of using the legacy runtime and the modern --gpus flag to enable specific driver capabilities.

```bash
$ docker run --rm --runtime=nvidia \
    -e NVIDIA_VISIBLE_DEVICES=2,3 \
    -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \
    nvidia/cuda nvidia-smi
```

```bash
$ docker run --rm --gpus 'all,"capabilities=compute,utility"' \
    nvidia/cuda:12.5.0-base-ubuntu22.04 nvidia-smi
```

--------------------------------

### Install NVIDIA Container Toolkit Packages with zypper

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

Install the NVIDIA Container Toolkit and its dependencies using zypper, specifying a version.

```bash
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.17.8-1
 sudo zypper --gpg-auto-import-keys install -y \
 nvidia-container-toolkit-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
 nvidia-container-toolkit-base-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
 libnvidia-container-tools-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
 libnvidia-container1-${NVIDIA_CONTAINER_TOOLKIT_VERSION}
```

--------------------------------

### Start a container on two GPUs

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/runtime/docker.html

Request two GPUs using the --gpus flag.

```bash
$ docker run --rm --gpus 2 nvidia/cuda nvidia-smi
```

--------------------------------

### Install NVIDIA Container Toolkit with apt

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

Installs the NVIDIA Container Toolkit packages on Ubuntu/Debian systems after configuring the repository. Ensure the NVIDIA GPU driver is installed prior to this step.

```bash
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.17.8-1
 sudo apt-get update 
 sudo apt-get install -y \
 nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
 nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
 libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
 libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}
```

--------------------------------

### Run Sample CUDA Container with Docker

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/sample-workload.html

Use this command to run a sample CUDA container with Docker. Ensure the NVIDIA Container Toolkit and drivers are installed.

```bash
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
```

--------------------------------

### Start a container on specific GPUs using device IDs

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/runtime/docker.html

Specify specific GPU indices using the device parameter format.

```bash
$ docker run --gpus '"device=1,2"' \
    nvidia/cuda nvidia-smi --query-gpu=uuid --format=csv
```

```text
uuid
GPU-ad2367dd-a40e-6b86-6fc3-c44a2cc92c7e
GPU-16a23983-e73e-0945-2095-cdeb50696982
```

--------------------------------

### Install NVIDIA Container Toolkit with dnf

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

Installs the NVIDIA Container Toolkit packages on RHEL/CentOS/Fedora/Amazon Linux systems after configuring the repository. Ensure the NVIDIA GPU driver is installed prior to this step.

```bash
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.17.8-1
 sudo dnf install -y \
 nvidia-container-toolkit-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
 nvidia-container-toolkit-base-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
 lib-nvidia-container-tools-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
 lib-nvidia-container1-${NVIDIA_CONTAINER_TOOLKIT_VERSION}
```

--------------------------------

### Pin All NVIDIA Container Stack Components

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/release-notes.html

To ensure compatibility when using a specific version of `nvidia-container-runtime` (e.g., 3.5.0), it is recommended to pin all related NVIDIA container stack components to their corresponding versions. This command installs the runtime, toolkit, tools, and library at the specified versions.

```bash
sudo apt-get install \
 nvidia-container-runtime=3.5.0-1 \
 nvidia-container-toolkit=1.5.1-1 \
 lib-nvidia-container-tools=1.5.1-1 \
 lib-nvidia-container1==1.5.1-1
```

--------------------------------

### Install Specific nvidia-container-runtime Version

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/release-notes.html

When installing older versions of `nvidia-container-runtime` on Debian-based systems, explicitly specify the `nvidia-container-toolkit` version to avoid unmet dependency errors. This command pins both packages to compatible versions.

```bash
sudo apt-get install \
 nvidia-container-runtime=3.5.0-1 \
 nvidia-container-toolkit=1.5.1-1
```

--------------------------------

### Run CUDA Container with All GPUs

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/docker-specialized.html

Use the `--gpus all` option to make all available GPUs accessible within the container. This is a common way to start a GPU-enabled CUDA container.

```bash
docker run --rm --gpus all nvidia/cuda nvidia-smi
```

--------------------------------

### Run CUDA Container on Two GPUs

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/docker-specialized.html

To run a container on a specific number of GPUs, use the `--gpus` option followed by the count of GPUs. This example starts a container with access to two GPUs.

```bash
docker run --rm --gpus 2 nvidia/cuda nvidia-smi
```

--------------------------------

### Configure Production Repository for APT

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install/repo-apt.html

Add the NVIDIA Container Toolkit GPG key and repository to the system sources.

```bash
$ curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
```

--------------------------------

### Enable Experimental Packages for APT

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install/repo-apt.html

Uncomment the experimental repository configuration in the apt source list.

```bash
$ sudo sed -i -e '/experimental/ s/^#//g' /etc/apt/sources.list.d/nvidia-container-toolkit.list
```

--------------------------------

### Preview nvidia-ctk configuration changes

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/user-guide.html

Preview configuration updates by printing to standard output without modifying the source file.

```bash
$ sudo nvidia-ctk config \
    --set nvidia-container-runtime.log-level=debug

```

--------------------------------

### Run Container with Graphics and Utility Capabilities

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/runtime/docker.html

Enables GPU 0 with graphics and utility components for rendering applications.

```bash
$ docker run --rm --runtime=nvidia \
    -e NVIDIA_VISIBLE_DEVICES=0 \
    -e NVIDIA_DRIVER_CAPABILITIES=graphics,utility \
    <image> <command>
```

--------------------------------

### Run Container with Specific GPU

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/release-notes.html

Example of running a CUDA container and listing GPUs. This may fail if the selected device does not have /dev/dri or /dev/nvidia-caps nodes.

```bash
$ docker run -it --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=nvidia.com/gpu=0 nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi -L
 docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #1: error running hook: exit status 1, stdout: , stderr: chmod: cannot access '/var/lib/docker/overlay2/9069fafcb6e39ccf704fa47b52ca92a1d48ca5ccfedd381f407456fb6cd3f9f0/merged/dev/dri': No such file or directory: unknown.
 ERRO[0000] error waiting for container: context canceled
```

--------------------------------

### Configure containerd for Kubernetes

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install/repo-apt.html

Configure containerd to use the NVIDIA Container Runtime by creating a drop-in configuration file.

```bash
$ sudo nvidia-ctk runtime configure --runtime=containerd
```

```bash
$ sudo systemctl restart containerd
```

--------------------------------

### Configure Production Repository for Zypper

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install/repo-apt.html

Add the repository using the zypper addrepo command.

```bash
$ sudo zypper ar https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
```

--------------------------------

### Apply configuration changes in-place

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/distro/suse15.html

Update the configuration file directly using the --in-place flag.

```bash
$ sudo nvidia-ctk config --in-place \
    --set nvidia-container-runtime.log-level=debug
```

```bash
$ sudo nvidia-ctk config --in-place \
    --set nvidia-container-runtime.log-level=debug \
    --set nvidia-container-runtime.debug=/var/log/nvidia-container-runtime.log
```

```bash
$ sudo nvidia-ctk config --in-place \
    --set nvidia-container-runtime.runtimes=crun:runc
```

--------------------------------

### Run Container with All GPUs using Podman

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html

Launches an Ubuntu container with access to all NVIDIA GPUs using Podman. Requires Podman v4.1.0 or later for `--device` argument support. Ensure no NVIDIA Container Runtime hook is active.

```bash
podman run --rm --device nvidia.com/gpu=all --security-opt=label=disable ubuntu nvidia-smi -L
```

--------------------------------

### Configure containerd for Kubernetes

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/user-guide.html

Configure containerd to use the NVIDIA Container Runtime and restart the service.

```bash
$ sudo nvidia-ctk runtime configure --runtime=containerd

```

```bash
$ sudo systemctl restart containerd

```

--------------------------------

### Preview and Update nvidia-container-runtime Configuration

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install/repo-apt.html

Use nvidia-ctk to preview or apply changes to the config.toml file. The --in-place flag is required to persist changes to the source file.

```bash
$ sudo nvidia-ctk config \
    --set nvidia-container-runtime.log-level=debug
```

```bash
$ sudo nvidia-ctk config --in-place \
    --set nvidia-container-runtime.log-level=debug
```

```bash
$ sudo nvidia-ctk config --in-place \
    --set nvidia-container-runtime.log-level=debug \
    --set nvidia-container-runtime.debug=/var/log/nvidia-container-runtime.log
```

```bash
$ sudo nvidia-ctk config --in-place \
    --set nvidia-container-runtime.runtimes=crun:runc
```

--------------------------------

### Configure Rootless Docker for NVIDIA Container Runtime

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/user-guide.html

Set up the NVIDIA Container Runtime for Docker running in rootless mode.

```bash
$ nvidia-ctk runtime configure --runtime=docker --config=$HOME/.config/docker/daemon.json

```

```bash
$ systemctl --user restart docker

```

```bash
$ sudo nvidia-ctk config --set nvidia-container-cli.no-cgroups --in-place

```

--------------------------------

### Preview configuration changes

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/distro/suse15.html

Preview updates to the configuration file without modifying the source.

```bash
$ sudo nvidia-ctk config \
    --set nvidia-container-runtime.log-level=debug
```

--------------------------------

### Enable Experimental Packages for Zypper

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install/repo-apt.html

Enable the experimental repository using zypper modifyrepo.

```bash
$ sudo zypper modifyrepo --enable nvidia-container-toolkit-experimental
```

--------------------------------

### Configure apt Repository for NVIDIA Container Toolkit

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

Configures the production repository for NVIDIA Container Toolkit on Debian-based systems. Optionally enables experimental packages.

```bash
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
```

```bash
sed -i -e '/experimental/ s/^#//g' /etc/apt/sources.list.d/nvidia-container-toolkit.list
```

--------------------------------

### Enable Experimental Repository with zypper

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

Optionally enable the experimental repository for NVIDIA Container Toolkit packages using zypper.

```bash
sudo zypper modifyrepo --enable nvidia-container-toolkit-experimental
```

--------------------------------

### Update nvidia-ctk configuration in-place

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/user-guide.html

Apply configuration changes directly to the source file using the --in-place flag.

```bash
$ sudo nvidia-ctk config --in-place \
    --set nvidia-container-runtime.log-level=debug

```

```bash
$ sudo nvidia-ctk config --in-place \
    --set nvidia-container-runtime.log-level=debug \
    --set nvidia-container-runtime.debug=/var/log/nvidia-container-runtime.log

```

```bash
$ sudo nvidia-ctk config --in-place \
    --set nvidia-container-runtime.runtimes=crun:runc

```

--------------------------------

### Use NVIDIA_VISIBLE_DEVICES with the NVIDIA runtime

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/runtime/docker.html

Alternative method to enable GPU access by setting the runtime and environment variable.

```bash
$ docker run --rm --runtime=nvidia \
    -e NVIDIA_VISIBLE_DEVICES=all nvidia/cuda nvidia-smi
```

--------------------------------

### Enable Experimental Packages for DNF

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install/repo-apt.html

Use dnf config-manager to enable the experimental repository.

```bash
$ sudo dnf config-manager --enable nvidia-container-toolkit-experimental
```

--------------------------------

### Run Container with Specific GPUs using Podman

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html

Launches an Ubuntu container requesting specific GPUs (e.g., GPU 0 and the first MIG device on GPU 1) using Podman. The output will show only the UUIDs of the requested devices.

```bash
podman run --rm \
    --device nvidia.com/gpu=0 \
    --device nvidia.com/gpu=1:0 \
    --security-opt=label=disable \
    ubuntu nvidia-smi -L
```

--------------------------------

### Identify Conflicting apt Sources

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/troubleshooting.html

Use this command to find repository configuration files that might be causing conflicts with the NVIDIA Container Toolkit's signed-by directive.

```bash
grep "nvidia.github.io" /etc/apt/sources.list.d/*
```

--------------------------------

### Configure Containerd Runtime with nvidia-ctk

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

Configure the containerd container runtime to use the NVIDIA Container Runtime.

```bash
sudo nvidia-ctk runtime configure --runtime=containerd
```

--------------------------------

### Run Container with Compute and Utility Capabilities

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/docker-specialized.html

Use this command to run a container with specific NVIDIA driver capabilities enabled, such as compute and utility, allowing CUDA and NVML usage. It also specifies visible devices.

```bash
docker run --rm --runtime=nvidia \
    -e NVIDIA_VISIBLE_DEVICES=2,3 \
    -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \
    nvidia/cuda nvidia-smi
```

```bash
docker run --rm --gpus 'all,"capabilities=compute,utility"' \
    nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi
```

--------------------------------

### Configure CRI-O Runtime

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install/repo-apt.html

Configure CRI-O to use the NVIDIA Container Runtime and restart the daemon.

```bash
$ sudo nvidia-ctk runtime configure --runtime=crio
```

```bash
$ sudo systemctl restart crio
```

--------------------------------

### Configure Rootless Docker Runtime with nvidia-ctk

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

Configure the container runtime for Docker running in rootless mode, specifying a custom daemon.json path.

```bash
nvidia-ctk runtime configure --runtime=docker --config=$HOME/.config/docker/daemon.json
```

--------------------------------

### Configure CRI-O Runtime with nvidia-ctk

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

Configure the CRI-O container runtime to use the NVIDIA Container Runtime.

```bash
sudo nvidia-ctk runtime configure --runtime=crio
```

--------------------------------

### List Potentially Conflicting apt Files

Source: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/troubleshooting.html

This command helps identify specific files in the sources.list.d directory that reference the NVIDIA repository and do not use the expected toolkit list file, aiding in resolving apt update errors.

```bash
grep -l "nvidia.github.io" /etc/apt/sources.list.d/* | grep -vE "/nvidia-container-toolkit.list$"
```
Tokens:
Raw