NVIDIA DOCA

https://networking-docs.nvidia.com/doca/archive/3-4-0
Admin
NVIDIA DOCA is a framework that enables developers to rapidly create and manage high-performance...
Tokens:2,161,762
Snippets:13,283
Trust Score:10
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Full Configuration Example - Client Side

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/macsec-full-offload.md

Complete client-side setup using example values: physical device enp8s0f0, MACsec device macsec0, SCI 1, and specific keys. Includes offload and rule addition.

```bash
ip address flush enp8s0f0
ip address add 1.1.1.1/24 dev enp8s0f0
ip link set dev enp8s0f0 up
ip link add link enp8s0f0 macsec0 type macsec sci 1 encrypt on
ip macsec offload macsec0 mac
ip macsec add macsec0 tx sa 0 pn 1 on key 00 dffafc8d7b9a43d5b9a3dfbbf6a30c16
ip macsec add macsec0 rx sci 2 on
ip macsec add macsec0 rx sci 2 sa 0 pn 1 on key 00 ead3664f508eb06c40ac7104cdae4ce5
ip address flush macsec0
ip address add 2.2.2.1/24 dev macsec0
ip link set dev macsec0 up
```

--------------------------------

### Full Configuration Example - Server Side

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/macsec-full-offload.md

Complete server-side setup using example values: physical device enp8s0f0, MACsec device macsec0, SCI 2, and specific keys. Includes offload and rule addition.

```bash
ip link del macsec0
ip address flush enp8s0f0
ip address add 1.1.1.2/24 dev enp8s0f0
ip link set dev enp8s0f0 up
ip link add link enp8s0f0 macsec0 type macsec sci 2 encrypt on
ip macsec offload macsec0 mac
ip macsec add macsec0 tx sa 0 pn 1 on key 00 ead3664f508eb06c40ac7104cdae4ce5
ip macsec add macsec0 rx sci 1 on
ip macsec add macsec0 rx sci 1 sa 0 pn 1 on key 00 dffafc8d7b9a43d5b9a3dfbbf6a30c16
ip address flush macsec0
ip address add 2.2.2.2/24 dev macsec0
ip link set dev macsec0 up
```

--------------------------------

### Example commands for installing doca-host repository (DEB-based)

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/mlnx_ofed-to-doca-ofed-transition-guide.md

Example commands showing the actual repository file name and the apt update command.

```bash
$ sudo dpkg -i doca-host_2.10.0-093442-25.01-ubuntu2204_amd64.deb

$ sudo apt update
```

--------------------------------

### Install kernel repo on RPM-based distributions (example)

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/doca-host-installation-and-upgrade.md

Example commands for RPM-based distributions using a specific RPM file and kernel version.

```bash
host# rpm -Uvh /tmp/DOCA.EuUfkWfV7Z/doca-kernel-repo-2.9.0-1.kver.5.14.0.356.el9.x86.64.x86_64.rpm
host# dnf makecache
host# dnf install doca-ofed-userspace
host# dnf install --disablerepo=doca doca-kernel-5.14.0.356.el9.x86.64.x86_64.noarch
```

--------------------------------

### Example bfb-install output

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/bf-bundle-installation-and-upgrade.md

Example output of Ubuntu-22.04 BFB bundle installation showing progress and INFO logs from PSC, BL2, BL31, UEFI, MISC, and NIC firmware update. The output indicates successful installation and password change.

```bash
host# sudo bfb-install --rshim rshim0 --bfb bf-bundle-2.7.0_24.04_ubuntu-22.04_prod.bfb --config bf.cfg
Pushing bfb 1.41GiB 0:02:02 [11.7MiB/s] [           <=>                                                                                                                                ]
Collecting BlueField booting status. Press Ctrl+C to stop
 INFO[PSC]: PSC BL1 START
 INFO[BL2]: start
 INFO[BL2]: boot mode (rshim)
 INFO[BL2]: VDDQ: 1120 mV
 INFO[BL2]: DDR POST passed
 INFO[BL2]: UEFI loaded
 INFO[BL31]: start
 INFO[BL31]: lifecycle GA Secured
 INFO[BL31]: VDD: 850 mV
 INFO[BL31]: runtime
 INFO[BL31]: MB ping success
 INFO[UEFI]: eMMC init
 INFO[UEFI]: eMMC probed
 INFO[UEFI]: UPVS valid
 INFO[UEFI]: PMI: updates started
 INFO[UEFI]: PMI: total updates: 1
 INFO[UEFI]: PMI: updates completed, status 0
 INFO[UEFI]: PCIe enum start
 INFO[UEFI]: PCIe enum end
 INFO[UEFI]: UEFI Secure Boot 
 INFO[UEFI]: PK configured
 INFO[UEFI]: Redfish enabled
 INFO[UEFI]: exit Boot Service
 INFO[MISC]: Found bf.cfg
 INFO[MISC]: Ubuntu installation started
 INFO[MISC]: Installing OS image
 INFO[MISC]: Changing the default password for user ubuntu
 INFO[MISC]: Ubuntu installation completed
 INFO[MISC]: Updating NIC firmware...
 INFO[MISC]: NIC firmware update done
 INFO[MISC]: Installation finished 
```

--------------------------------

### virtnet update -s example

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/doca-virtio-net-service-guide.md

Example of starting a live update with virtnet update -s, showing the output and a message to check status.

```bash
# virtnet update -s
{'start': '0x1'}
"Update started, use 'virtnet update -t' or check logs for status"
```

--------------------------------

### Kickstart script configuration for HBN

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/hbn-service-deployment.md

Example lines to append to /etc/bf.cfg within a kickstart script for HBN setup.

```Bash
cat >> /etc/bf.cfg << EOF 

ENABLE_BR_HBN=yes
BR_HBN_UPLINKS='p0,p1'
BR_HBN_REPS='pf0hpf,pf1hpf,pf0vf0-pf0vf7,pf1vf0-pf1vf7'
BR_HBN_SFS='pf0dpu1,pf0dpu3' 
EOF
```

--------------------------------

### Complete DPL example (header and control start)

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/equal-cost-multi-path-routing-example.md

Complete DPL example header and control block setup, including includes, ECMP_BUCKETS definition, usermeta_t struct, and the start of the control block with the NvHash object.

```C
/*
 * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 * SPDX-License-Identifier: LicenseRef-NvidiaProprietary
 *
 * NVIDIA CORPORATION, its affiliates and licensors retain all intellectual
 * property and proprietary rights in and to this material, related
 * documentation and any modifications thereto. Any use, reproduction,
 * disclosure or distribution of this material and related documentation
 * without an express license agreement from NVIDIA CORPORATION or
 * its affiliates is strictly prohibited.
 */

#include <doca_model.p4>
#include <doca_headers.p4>
#include <doca_externs.p4>
#include <doca_parser.p4>

#define ECMP_BUCKETS 16

struct usermeta_t {
    bit<4> ecmp_select;
};

control c(
    inout nv_headers_t headers,
    in nv_standard_metadata_t std_meta,
    inout usermeta_t user_meta,
    inout nv_empty_metadata_t pkt_out_meta
) {
    // Create a hash object with the selected algorithm
    NvHash(NvHashAlgorithm.CRC32) hash;
```

--------------------------------

### Host-side pseudo-code for DPA thread setup and run

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/doca-dpa.md

Example showing how to create, configure, start, and run a DPA thread with EU affinity, TLS, and a completion context. The completion context must be started before running the thread.

```c
extern doca_dpa_func_t hello_kernel;

// create DPA thread
doca_dpa_thread_create(&dpa_thread);

// set thread kernel
doca_dpa_thread_set_func_arg(dpa_thread, &hello_kernel, func_arg);

// set thread affinity
doca_dpa_eu_affinity_create(&eu_affinity);
doca_dpa_eu_affinity_set(eu_affinity, 10 /* EU ID */);
doca_dpa_thread_set_affinity(dpa_thread, eu_affinity);

// set thread local storage
doca_dpa_mem_alloc(&tls_dev_ptr);
doca_dpa_thread_set_local_storage(dpa_thread, tls_dev_ptr);

// start thread
doca_dpa_thread_start(dpa_thread);

// create and initialize DPA Completion Context
doca_dpa_completion_create(&dpa_comp);
doca_dpa_completion_set_thread(dpa_comp, dpa_thread);
doca_dpa_completion_start(dpa_comp);

// run thread only after both thread is started and the attached completion context is started
doca_dpa_thread_run(dpa_thread);
```

--------------------------------

### CLI Example of the Server Application

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/doca-dpu-gpu-remote-offload-application-guide.md

Example command to run the server with mandatory device ID, representor ID, listen port, and CPU options. The PCIe addresses must match the desired devices.

```bash
./doca_dpu_gpu_remote_offload_server --device-id 03:00.0 --representor-id d8:00.0 \
    -p 12345 --max-concurrent-messages 128 --cpu 1 --cpu 2
```

--------------------------------

### Port group definition example

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/nvidia-sm.md

Defines a port group named 'grp1' with three port GUIDs. Each group starts with 'port-group' and ends with 'end-port-group'.

```plaintext
port-group
name: grp1
port-guid: 0x281, 0x282, 0x283
end-port-group
```

--------------------------------

### Prepare build environment and clone SPDK

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/snap-4-service-appendixes.md

Sets up the build environment by creating a working directory, cloning the SPDK source tree at the version matching the SNAP release, and verifying the expected directory layout.

```bash
# Create working directory (example: /opt/build)
mkdir -p /opt/build
cd /opt/build

# Clone SPDK sources
git clone https://github.com/Mellanox/spdk --branch v25.05.2.nvda

# Verify directory structure
ls -F
# Output should show: Dockerfile  spdk/
```

--------------------------------

### DPU End-to-End Installation Steps

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/dpl-container-deployment.md

End-to-end installation steps for a DPU (BlueField) environment. Downloads the NGC CLI and DPL Runtime Service resources, runs the system setup script, restarts kubelet and containerd services, creates a device configuration file, and starts the service. Replace device IDs and filenames as appropriate for your setup.

```Bash
# Download NGC CLI tool:
wget --content-disposition https://api.ngc.nvidia.com/v2/resources/nvidia/ngc-apps/ngc_cli/versions/4.5.2/files/ngccli_arm64.zip -O ngccli_arm64.zip
unzip ngccli_arm64.zip

# Download the DPL Runtime Service Resources bundle:
./ngc-cli/ngc registry resource download-version "nvidia/doca/dpl_rt_service"

# Prepare DPU and restart services:
cd dpl_rt_service_va.b.c-docax.y.z
chmod +x ./scripts/dpl_system_setup.sh
sudo ./scripts/dpl_system_setup.sh
sudo systemctl restart kubelet.service
sudo systemctl restart containerd.service

# Create a device configuration file with relevant interfaces info:
sudo cp /etc/dpl_rt_service/devices.d/NAME.conf.template /etc/dpl_rt_service/devices.d/1000.conf
sudo vim /etc/dpl_rt_service/devices.d/1000.conf

# Launch the Pod and container:
sudo ./scripts/dpl_rt_service_ctl.sh --start
```

--------------------------------

### Set up SNAP ML Optimizer

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/snap-4-service-advanced-features.md

Steps to set up the SNAP ML optimizer: copy the snap_ml folder from the container, create and activate a virtual environment, install dependencies, and view help options.

```bash
crictl exec -it $(crictl ps -s running -q --name snap) cp -r /opt/nvidia/nvda_snap/bin/snap_ml /etc/nvda_snap/
```

```bash
cd tools/snap_ml
```

```bash
python3 -m venv snap_ml
```

```bash
source snap_ml/bin/activate
```

```bash
pip3 install --no-cache-dir -r requirements.txt
```

```bash
python3 snap_ml.py --help
```

--------------------------------

### Host End-to-End Installation Steps

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/dpl-container-deployment.md

End-to-end installation steps for a host (x86) environment. Downloads the NGC CLI and DPL Runtime Service resources, runs the system setup script with the --dev flag specifying the device ID, creates a device configuration file, and starts the Docker container. Replace device IDs and filenames as appropriate for your setup.

```Bash
# Download NGC CLI tool:
wget --content-disposition https://api.ngc.nvidia.com/v2/resources/nvidia/ngc-apps/ngc_cli/versions/4.5.2/files/ngccli_linux.zip -O ngccli_linux.zip
unzip ngccli_linux.zip

# Download the DPL Runtime Service Resources bundle:
./ngc-cli/ngc registry resource download-version "nvidia/doca/dpl_rt_service"

# Prepare DPU and restart services:
cd dpl_rt_service_va.b.c-docax.y.z
chmod +x ./scripts/dpl_system_setup.sh
sudo ./scripts/dpl_system_setup.sh --dev 0000:08:00.0

# Create a device configuration file with relevant interfaces info:
sudo cp /etc/dpl_rt_service/devices.d/NAME.conf.template /etc/dpl_rt_service/devices.d/1000.conf
sudo vim /etc/dpl_rt_service/devices.d/1000.conf

# Launch the Docker container:
sudo ./scripts/dpl_rt_service_ctl.sh --start
```

--------------------------------

### Intel server boot parameters example

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/doca-snap-virtio-fs-service-guide.md

Example output of cat /proc/cmdline on an Intel server, showing the required pci=realloc and modprobe.blacklist=virtio_pci,virtiofs parameters.

```bash
cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.15.0_mlnx root=UUID=91528e6a-b7d3-4e78-9d2e-9d5ad60e8273 ro crashkernel=auto resume=UUID=06ff0f35-0282-4812-894e-111ae8d76768 rhgb quiet pci=realloc modprobe.blacklist=virtio_pci,virtiofs
```

--------------------------------

### Enhanced QoS Policy Example with Defaults

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/nvidia-sm.md

A more detailed policy example using multiple bandwidth names (bw1 to bw4) and the 'default' keyword to apply a rule to all unlisted GUIDs. Shows rules for a physical GUID and a virtual-port GUID.

```plaintext
BW_NAMES
bw1 = 50
bw2 = 25
bw3 = 15
bw4 = 10

BW_RULES
default = 1:bw1
0x2c90000000025 = 1:bw2

VPORT_BW_RULES
default = all:bw3
0x2c90000000026 = 3:bw4
```

--------------------------------

### Install Storage Packages on SLES

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/doca-host-installation-and-upgrade.md

Example installation commands for SLES systems: searches for the mlnx-nvme package and installs the mlnx-nvme-kmp-default package.

```bash
# zypper search mlnx-nvme
# zypper install mlnx-nvme-kmp-default
```

--------------------------------

### Run the basic sample application

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/dpl-runtime-controller-sdk.md

Run the basic sample application with a privileged user. Replace <path-to-p4info.txt> and <path-to-dplconfig> with the actual paths. The first argument is 1000.

```bash
sudo /tmp/build/dpl_sample_basic 1000 <path-to-p4info.txt> <path-to-dplconfig>
```

--------------------------------

### Install Storage Packages on Debian-based Systems

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/doca-host-installation-and-upgrade.md

Example installation commands for Debian-based systems: searches for the mlnx-nvme and mlnx-nfsrdma packages, then installs the mlnx-nvme-dkms and mlnx-nfsrdma-dkms packages.

```bash
# apt search mlnx-nvme
# apt search mlnx-nfsrdma
# apt install mlnx-nvme-dkms
# apt install mlnx-nfsrdma-dkms
```

--------------------------------

### Example output of devlink dev show

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/bluefield-scalable-functions.md

Example output after creating and activating the SF, showing auxiliary devices.

```text
pci/0000:03:00.0
pci/0000:03:00.1
auxiliary/mlx5_core.sf.2
auxiliary/mlx5_core.sf.3
auxiliary/mlx5_core.sf.4
```

--------------------------------

### Recommended Boot Parameters for virtio-blk

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/snap-4-service-appendixes.md

Example of the recommended boot parameter configuration for virtio-blk support.

```bash
[dpu] cat /proc/cmdline BOOT_IMAGE ... pci=realloc modprobe.blacklist=virtio_blk,virtio_pci
```

--------------------------------

### Install kernel repo on SLES (example)

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/doca-host-installation-and-upgrade.md

Example commands for SLES using the doca-ofed-userspace metapackage and a specific kernel version.

```bash
host# rpm -Uvh <doca-kernel-repo>
host# zypper refresh
host# zypper install doca-ofed-userspace
host# zypper modifyrepo --disable doca 
host# zypper install doca-kernel-5.14.21.150500.53.default     # For all the meta-package, not just this single package
host# zypper modifyrepo --enable doca
```

--------------------------------

### Start IKE with swanctl

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/doca-ipsec-security-gateway-application-guide.md

Start the IKE protocol on either the initiator or target side. In the example, the child name is 'bf'.

```bash
swanctl -i --child <child_name>
```

--------------------------------

### Configure and activate SF (example)

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/bluefield-scalable-functions.md

Example command that sets the MAC address, enables trust, and activates SF 4 on controller 0.

```bash
/opt/mellanox/iproute2/sbin/mlxdevm port function set pci/0000:03:00.0/229409 \
    hw_addr 02:25:f2:8d:a2:4c \
    trust on \
    state active
```

--------------------------------

### Configure static virtio-blk PF in snap_rpc_init.conf

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/doca-snap-3-user-guide.md

Add this line to /etc/mlnx_snap/snap_rpc_init.conf to create a static virtio-blk PF with no backing device. Required before using static virtio-blk PFs to avoid host boot failure.

```bash
controller_virtio_blk_create mlx5_0 --pf_id 0 --bdev_type none
```

--------------------------------

### hbn-dpu-setup.sh help menu

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/hbn-service-deployment.md

Displays the usage and options for the hbn-dpu-setup.sh script, including MTU, username, password, and REST API access flags.

```Bash
./hbn-dpu-setup.sh -h
usage: hbn-dpu-setup.sh
hbn-dpu-setup.sh -m|--mtu <MTU> Use <MTU> bytes for all HBN interfaces (default 9216)
hbn-dpu-setup.sh -u|--username <username> User creation
hbn-dpu-setup.sh -p|--password <password> Password for --username <username>
hbn-dpu-setup.sh -e|--enable-rest-api-access Enable REST API from external IPs
hbn-dpu-setup.sh -h|--help
```

--------------------------------

### GET /interfaces/interface[name=p0]/config/mtu

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/doca-management-service-guide.md

Example of a Get request using the DMS client to retrieve the MTU configuration for a specific interface.

```APIDOC
## [GET] /interfaces/interface[name=p0]/config/mtu

### Description
Retrieves real-time device data for a specific YANG XPath path. This command queries the MTU configuration for a specific interface.

### Method
GET

### Endpoint
/interfaces/interface[name=p0]/config/mtu

### Parameters
#### Path Parameters
- **name** (string) - Required - The name of the interface to query.

### Request Example
```bash
/opt/mellanox/doca/services/dms/dmsc <flags> get --path /interfaces/interface[name=p0]/config/mtu
```

### Response
#### Success Response (200)
- **source** (string) - The source address of the response.
- **timestamp** (integer) - The timestamp of the response.
- **time** (string) - The time of the response in RFC3339 format.
- **updates** (array) - List of updates.
  - **Path** (string) - The path of the updated value.
  - **values** (object) - The updated values.

#### Response Example
```json
[
  {
    "source": "localhost:9339",
    "timestamp": 1712485149723248511,
    "time": "2024-04-07T10:19:09.723248511Z",
    "updates": [
      {
        "Path": "interfaces/interface[name=p0]/config/mtu",
        "values": {
          "interfaces/interface/config/mtu": "1500"
        }
      }
    ]
  }
]
```
```

--------------------------------

### Build SNAP with custom SPDK - meson setup

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/snap-4-service-deployment.md

Configure the build with meson, enabling SPDK compatibility and providing the path to the custom SPDK. Replace </path/to/custom/spdk> with the actual path.

```bash
meson setup /tmp/build -Denable-spdk-compat=true -Dsnap_spdk_prefix=</path/to/custom/spdk>
```

--------------------------------

### AMD server boot parameters example

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/doca-snap-virtio-fs-service-guide.md

Example output of cat /proc/cmdline on an AMD server, showing the required pci=realloc and modprobe.blacklist=virtio_pci,virtiofs parameters.

```bash
cat /proc/cmdline
cat /proc/cmdline BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.15.0_mlnx root=UUID=91528e6a-b7d3-4e78-9d2e-9d5ad60e8273 ro crashkernel=auto resume=UUID=06ff0f35-0282-4812-894e-111ae8d76768 rhgb quiet pci=realloc modprobe.blacklist=virtio_pci,virtiofs
```

--------------------------------

### Install kernel repo on DEB-based distributions (example)

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/doca-host-installation-and-upgrade.md

Example commands for DEB-based distributions using a specific DEB file and the doca-ofed profile.

```bash
host# dpkg -i /tmp/DOCA.J8JYxEmffD/doca-kernel-repo-2.9.0-6.4.0.mlnx_2.9.0_amd64.deb
host# apt update
host# apt install doca-ofed
```

--------------------------------

### Enable DIM configuration example

Source: https://networking-docs.nvidia.com/doca/archive/3-4-0/doca-virtio-net-service-guide.md

Configuration example for enabling DIM: unload guest-OS drivers, enable DIM on the DPU, reload drivers, and verify the setting. Using `disable` disables DIM.

```bash
[host]# modprobe -rv virtio_net && modprobe -rv virtio_pci
```

```bash
[dpu]# virtnet modify -p 0 device -dim enable
{'pf': '0x0', 'all': '0x0', 'subcmd': '0x0', 'dim_config': 'enable'}
{
  "errno": 0,
  "errstr": "Success"
}
```

```bash
[host]# modprobe -v virtio_pci && modprobe -v virtio_net
```

```bash
[dpu]# virtnet query -p 0 -b | grep -i dim
      "dim": "enabled"
```
Tokens:
Raw