GPUDirect Storage

https://docs.nvidia.com/gpudirect-storage/
Admin
GPUDirect Storage enables direct DMA transfers between GPU memory and storage, reducing CPU overhead...
Tokens:137,470
Snippets:1,459
Trust Score:10
Update:2 months ago
Show doc for...
Tokens:
Raw
### Example Output: IOMMU Enabled

Source: https://docs.nvidia.com/gpudirect-storage/best-practices-guide/index.html

This is an example output from 'cat /proc/cmdline' indicating that IOMMU is enabled on the system. IOMMU should be disabled for GDS.

```bash
BOOT_IMAGE=/boot/vmlinuz-5.19.0-38-generic root=UUID=fb2a25a8-9d2e-4e1c-9d8a-efabdf165adc ro rootflags=data=ordered amd_iommu=on
```

--------------------------------

### cuFile API Call Example

Source: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

This example demonstrates the usage of cuFile APIs for driver, handle, and buffer registration and deregistration. It highlights the explicit and implicit behaviors of these operations.

```c
cuFileDriverOpen();
cuFileHandleRegister(fd, &handle);
cuFileBufRegister(buffer, size, &buf_handle);

// ... operations ...

cuFileBufDeregister(buf_handle);
cuFileHandleDeregister(handle);
cuFileDriverClose();
```

--------------------------------

### Example Output: IOMMU Disabled

Source: https://docs.nvidia.com/gpudirect-storage/best-practices-guide/index.html

This is an example output from '/usr/local/cuda/gds/tools/gdscheck -p' showing that IOMMU is disabled. This is the recommended state for optimal GDS performance.

```bash
 IOMMU: disabled
 Platform verification succeeded 
```

--------------------------------

### Install WekaIO

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Run this command to install WekaIO. Ensure you replace <IP of one of the WekaIO hosts' IB interface> with the correct IP address.

```bash
$ curl http://<IP of one of the WekaIO hosts' IB interface>:14000/dist/v1/install | sh
```

--------------------------------

### Install NFS Server and Configure Storage

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Steps to install the Linux NFS server, format and mount NVMe storage, and configure exports. Ensure the server has Mellanox connect-X4/5 NIC with MLNX_OFED 5.3 or later.

```bash
sudo apt-get install nfs-kernel-server

```

```bash
mkfs.ext4 /dev/nvme0n1

```

```bash
mount -o data=ordered /dev/nvme0n1 /mnt/nvme

```

```bash
service nfs-kernel-server restart

```

```bash
modprobe rpcrdma

```

```bash
echo rdma 20049 > /proc/fs/nfsd/portlist

```

--------------------------------

### Install NFS Client with GPUDirect Storage Support

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Installs the NFS client and configures it for RDMA and GDS. Ensure MLNX_OFED 5.3 or later is installed on the client with a compatible Mellanox NIC.

```bash
$ ofed_info -s MLNX_OFED_LINUX-5.3-1.0.5.0:
```

```bash
$ sudo apt-get install nfs-common
```

```bash
$ modprobe rpcrdma
```

```bash
$ mkdir -p /mnt/nfs_rdma_gds
```

```bash
$ sudo mount -v -o proto=rdma,port=20049,vers=3 172.16.0.101:/ /mnt/nfs_rdma_gds
```

```bash
$ sudo mount -v -o proto=rdma,port=20049,vers=3,nconnect=20,localports=172.16.0.17-172.16.0.24,remoteports=172.16.0.101-172.16.0.120 172.16.0.101:/ /mnt/nfs_rdma_gds
```

--------------------------------

### Check GPUDirect Storage Installation

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Run this command to verify the GPUDirect Storage installation and check available BAR memory per GPU. Review the output for GPU index and BAR size.

```bash
/usr/local/cuda-x.y/gds/tools/gdscheck
```

--------------------------------

### Check for libibverbs.so

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Confirm the installation of the 'libibverbs-dev' package, which provides the 'libibverbs.so' library.

```bash
$ dpkg -s libibverbs-dev Package: libibverbs-dev Status: install ok installed Priority: optional Section: libdevel Installed-Size: 1151 Maintainer: Linux RDMA Mailing List <linux-rdma@vger.kernel.org> Architecture: amd64 Multi-Arch: same Source: rdma-core Version: 47mlnx1-1.47329
```

--------------------------------

### Example Custom cufile.json for Application Logging

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Example of a cufile.json file configured for a specific application, demonstrating how to set the log directory. This file should be placed at the path specified by CUFILE_ENV_PATH_JSON.

```json
"logging": {
    // log directory, if not enabled

```

--------------------------------

### Install VAST NFSoRDMA+Multipath Package

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Install the VAST NFSoRDMA+Multipath package using apt-get. Ensure you have the correct .deb file that matches your kernel and MLNX_OFED version.

```bash
$ sudo apt-get install mlnx-nfsrdma-*.deb
```

--------------------------------

### startRDMASession()

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjServer-api/index.html

Starts the RDMA session for the initialized connection.

```APIDOC
## int startRDMASession()

### Description
Starts the RDMA session. 

### Returns
- **int** - 0 on success, -1 if the address is invalid or the session cannot be opened.
```

--------------------------------

### Verify cuObject Client Installation

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuobject-client-release-notes/index.html

Commands to verify the presence of cuObject libraries, check package installation, and validate GDS support.

```bash
# Check library presence
$ ldconfig -p | grep cuobj
$ ldconfig -p | grep cufile

# Verify cuObjClient installation
$ rpm -qa | grep cuobjclient    # RHEL/CentOS
$ dpkg -l | grep cuobjclient    # Ubuntu/Debian

# Verify GDS support
$ /usr/local/cuda/gds/tools/gdscheck -p
```

--------------------------------

### Initialize cuObjServer with IPv6 and IPv4

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjServer-api/index.html

Examples of initializing the server with different IP address formats. Ensure the port is passed as a separate integer argument.

```cpp
cuObjServer server("fd00::10", 18515, CUOBJ_PROTO_RDMA_DC_V1);
```

```cpp
cuObjServer server("192.168.1.100", 18515, CUOBJ_PROTO_RDMA_DC_V1);
```

--------------------------------

### Verify cuObject Server Installation

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuobject-server-release-notes/index.html

Commands to check for the presence of the cuObject library and verify the installation of the cuObjServer package on Linux systems.

```bash
# Check library presence
ldconfig -p | grep cuobj

# Verify cuObjServer installation
rpm -qa | grep cuobjserver    # RHEL/CentOS
dpkg -l | grep cuobjserver    # Ubuntu/Debian
```

--------------------------------

### Compile GDSIO Example Program

Source: https://docs.nvidia.com/gpudirect-storage/getting-started/index.html

Command to compile the C++ GDSIO example program, linking necessary CUDA and GDS libraries.

```bash
g++ -o gds_example gds_example.cc -I/usr/local/cuda/include -L/usr/local/cuda/lib64 -lcuda -lcufile -lcudart
```

--------------------------------

### Start RDMA Session

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjServer-api/index.html

Initiates the RDMA session; returns 0 on success or -1 on failure.

```cpp
int startRDMASession();
```

--------------------------------

### Implement Custom Callbacks

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjClient-api/index.html

Shows how to implement GET and PUT callbacks to communicate with remote endpoints via a control path.

```cpp
ssize_t my_get_callback(const void* handle, char* ptr, size_t size,
                        loff_t offset, const cufileRDMAInfo_t* rdma_info) {
    // Extract user context
    MyContext* ctx = (MyContext*)cuObjClient::getCtx(handle);
    if (ctx == nullptr || rdma_info == nullptr || rdma_info->desc_str == nullptr) {
        return -EINVAL;
    }

    // Get RDMA descriptor string
    const char* rdma_desc = rdma_info->desc_str;

    // Communicate with the remote endpoint via control path
    // Send: operation=GET, size, offset, rdma_desc
    ssize_t bytes_read = send_request_on_control_path(ctx->connection,
                                                      "GET", size, offset, rdma_desc);

    // Return the amount of data transferred
    return bytes_read;
}

ssize_t my_put_callback(const void* handle, const char* ptr, size_t size,
                        loff_t offset, const cufileRDMAInfo_t* rdma_info) {
    // Extract user context
    MyContext* ctx = (MyContext*)cuObjClient::getCtx(handle);
    if (ctx == nullptr || rdma_info == nullptr || rdma_info->desc_str == nullptr) {
        return -EINVAL;
    }

    // Get RDMA descriptor string
    const char* rdma_desc = rdma_info->desc_str;

    // Communicate with the remote endpoint via control path
    // Send: operation=PUT, size, offset, rdma_desc
    ssize_t bytes_written = send_request_on_control_path(ctx->connection,
                                                         "PUT", size, offset, rdma_desc);

    // Return the amount of data transferred
    return bytes_written;
}
```

--------------------------------

### Check EXAScaler File System Client Module Version

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Run this command to verify the installed EXAScaler file system client module version. The output will display the Lustre version.

```bash
$ sudo lctl get_param version
```

--------------------------------

### Configure Lustre Module Parameters

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Example Lustre module configuration. These options can be tuned for LNet performance based on the network type and host channel adapter (HCA).

```bash
cat /etc/modprobe.d/lustre.conf
```

--------------------------------

### Configure RDMA Multipathing in JSON

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjClient-api/index.html

Example JSON configuration for enabling and tuning RDMA multipathing properties.

```json
{
  "properties": {
    "rdma_dev_addr_list": [
      "192.168.1.10",
      "192.168.1.11",
      "fd00::10",
      "fd00::11"
    ],
    "rdma_peer_type": "dmabuf",
    "rdma_multipath_enabled": true,
    "rdma_max_backup_devices": 2,
    "rdma_io_retry_count": 3,
    "rdma_io_retry_delay_ms": 100,
    "rdma_failback_enabled": true,
    "rdma_failback_delay_ms": 100,
    "rdma_health_check_interval_ms": 100,
    "rdma_async_event_monitoring": true,
    "rdma_unhealthy_threshold": 3
  }
}
```

--------------------------------

### Get LLite Tuning Parameters

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Retrieves EXAScaler file system client tuning parameters for LLite (Lustre Lite).

```bash
lctl get_param llite.*.*
```

--------------------------------

### Check LNet Network Setup on Client

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Execute this command to inspect the LNet network configuration on a client system. Review the output for network details.

```bash
$ sudo lnetctl net show
```

--------------------------------

### List GDS Tools and Samples

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

List the contents of the GPUDirect Storage tools and samples directory. This helps in locating available utilities and example code.

```bash
ls -lh /usr/local/cuda-X.Y/gds/
```

--------------------------------

### Configure RDMA Device Names

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjClient-api/index.html

Example of specifying RDMA device names in the address list.

```json
"rdma_dev_addr_list": ["mlx5_0", "mlx5_1"]
```

--------------------------------

### Check libibverbs-dev on Ubuntu

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Verify the installation status and details of the `libibverbs-dev` package on Ubuntu systems, which provides development headers for RDMA verbs.

```bash
$ dpkg -s libibverbs-dev
root@fscc-sr650-59:~# dpkg -s libibverbs-dev
Package: libibverbs-dev
Status: install ok installed
Priority: optional
Section: libdevel
Installed-Size: 1428
Maintainer: Linux RDMA Mailing List <linux-rdma@vger.kernel.org>
Architecture: amd64
Multi-Arch: same
Source: rdma-core
Version: 54mlnx1-1.54103 
```

--------------------------------

### Accessing Infiniband Network Counters

Source: https://docs.nvidia.com/gpudirect-storage/configuration-guide/index.html

Example path to access performance counters for an Infiniband network interface. These counters provide detailed statistics on network traffic.

```bash
/sys/class/infiniband/[INTERFACE]/ports/[PORT NUMBER]/counters
```

--------------------------------

### Check WekaIO Version

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Verify the installed WekaIO version using the 'weka status' command.

```bash
$ weka status WekaIO v3.6.2.5-rdma-beta (CLI build 3.6.2.5-rdma-beta)
```

--------------------------------

### Configure Mixed RDMA Addresses

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjClient-api/index.html

Example of mixing IP addresses and device names in the address list.

```json
"rdma_dev_addr_list": ["192.168.1.10", "mlx5_1", "fd00::11"]
```

--------------------------------

### Show Server LNet Network Configuration (Sub-optimal)

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Display the LNet network configuration on the server, highlighting a sub-optimal setup where a required LNet subnet might be missing.

```bash
[root@ai200-090a-vm01 ~]# lnetctl net show
```

--------------------------------

### Install Matching Fabric Manager Version

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Ensure the installed version of Fabric Manager matches the CUDA driver version to resolve CUDA_ERROR_SYSTEM_NOT_READY on systems with NVSwitch. This example shows installing nvidia-fabricmanager-460 for nvidia-driver-460-server.

```bash
$ sudo apt install nvidia-driver-460-server -y

then use:

$ apt-get install nvidia-fabricmanager-460
```

--------------------------------

### Perform Basic Client Operations

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjClient-api/index.html

Demonstrates the standard workflow for initializing a client, registering memory, and executing GET/PUT operations.

```cpp
// 1. Define callback operations
CUObjOps_t ops = {
    .get = my_get_callback,
    .put = my_put_callback
};

// 2. Create client instance
cuObjClient client(ops, CUOBJ_PROTO_RDMA_DC_V1);
if (!client.isConnected()) {
    // Handle connection error
    return;
}

// 3. Allocate and register memory
void* buffer = malloc(1024 * 1024);  // 1MB buffer
if (buffer == nullptr) {
    // Handle allocation error
    return;
}

if (client.cuMemObjGetDescriptor(buffer, 1024 * 1024) != CU_OBJ_SUCCESS) {
    // Handle registration error
    free(buffer);
    return;
}

// 4. Perform GET operation
MyContext ctx = { /* user data */ };
ssize_t result = client.cuObjGet(&ctx, buffer, 1024, 0, 0);
if (result < 0) {
    // Handle error
    client.cuMemObjPutDescriptor(buffer);
    free(buffer);
    return;
}

// 5. Perform PUT operation
result = client.cuObjPut(&ctx, buffer, 1024, 0, 0);
if (result < 0) {
    // Handle error
    client.cuMemObjPutDescriptor(buffer);
    free(buffer);
    return;
}

// 6. Cleanup
client.cuMemObjPutDescriptor(buffer);
free(buffer);
```

--------------------------------

### Manage RDMA Tokens Manually

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjClient-api/index.html

Provides an example of manually retrieving and releasing RDMA tokens for specific memory regions.

```cpp
// Register memory
void* buffer = malloc(1024 * 1024);
if (buffer == nullptr) {
    // Handle allocation error
    return;
}

if (client.cuMemObjGetDescriptor(buffer, 1024 * 1024) != CU_OBJ_SUCCESS) {
    // Handle registration error
    free(buffer);
    return;
}

// Get RDMA token for a specific region
char* rdma_token = nullptr;
size_t region_size = 4096;
size_t region_offset = 1024;
cuObjErr_t err = client.cuMemObjGetRDMAToken(buffer, region_size,
                                             region_offset, CUOBJ_GET,
                                             &rdma_token);
if (err == CU_OBJ_SUCCESS) {
    // Pass rdma_token on the application control path
    // ...

    // Free the token
    client.cuMemObjPutRDMAToken(rdma_token);
}

// Cleanup
client.cuMemObjPutDescriptor(buffer);
free(buffer);
```

--------------------------------

### Initialize and use cuObjServer

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjServer-api/index.html

Demonstrates basic server connection, channel allocation, host buffer registration, and object retrieval.

```cpp
cuObjServer server("192.168.1.100", 18515, CUOBJ_PROTO_RDMA_DC_V1);
if (!server.isConnected()) {
    return;
}

uint16_t channel = server.allocateChannelId();
if (channel == INVALID_CHANNEL_ID) {
    return;
}

void* buffer = server.allocHostBuffer(1024 * 1024);
if (buffer == nullptr) {
    server.freeChannelId(channel);
    return;
}

struct rdma_buffer* rdma_buf = server.registerBuffer(buffer, 1024 * 1024);
if (rdma_buf == nullptr) {
    free(buffer);
    server.freeChannelId(channel);
    return;
}

ssize_t result = server.handleGetObject("request-1",
                                        rdma_buf,
                                        remote_addr,
                                        request_size,
                                        rdma_descriptor,
                                        channel);
if (result < 0) {
    // Handle error.
}

server.deRegisterBuffer(rdma_buf);
free(buffer);
server.freeChannelId(channel);
```

--------------------------------

### Verify VAST Multipath Package Installation

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Check if the VAST NFSoRDMA+Multipath package is installed and verify its version using dpkg. This confirms the successful installation of the package.

```bash
$ dpkg -l | grep mlnx-nfsrdma
```

--------------------------------

### Verify GDS Installation with gdsio (Default Mode)

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Use the gdsio tool to test storage to GPU memory transfer. Ensure the path to gdsio and the file are correct for your system.

```bash
/usr/local/cuda-x.y/tool/gdsio -f /mnt/ai200/fio-seq-read-1 -d 0 -w 4 -s 10G -i 1M -I 0 -x 5
```

--------------------------------

### Get OSC Tuning Parameters

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Retrieves EXAScaler file system client tuning parameters for OSC (Object Storage Client), such as max_pages_per_rpc, max_dirty_mb, and max_rpcs_in_flight. Useful for performance tuning.

```bash
lctl get_param osc.*.max* osc.*.checksums
```

--------------------------------

### Profile Commands with Nsight Systems

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Use Nsight Systems to profile commands for RDMA connectivity and performance issues. This command starts the profiling session.

```bash
/usr/local/cuda/bin/nsys profile <command>
```

--------------------------------

### Perform asynchronous GET operation

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjServer-api/index.html

Submits an asynchronous GET request and polls for completion events.

```cpp
void* async_handle = reinterpret_cast<void*>(0x1);

ssize_t submit = server.handleGetObject("async-get",
                                        rdma_buf,
                                        remote_addr,
                                        request_size,
                                        rdma_descriptor,
                                        channel,
                                        0,
                                        nullptr,
                                        async_handle);
if (submit == 0) {
    cuObjAsyncEvent_t events[16];
    int count = 0;
    do {
        count = server.poll(events, 16, channel);
    } while (count == 0);

    if (count > 0) {
        for (int i = 0; i < count; ++i) {
            if (events[i].status == IBV_WC_SUCCESS) {
                // events[i].async_handle identifies the completed request.
            }
        }
    }
}
```

--------------------------------

### cuFileStatsStart

Source: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

Starts the collection of cuFile statistics. This function initiates performance data gathering, which can be controlled by the previously set statistics level.

```APIDOC
## cuFileStatsStart

### Description
Start collecting cuFile statistics. The statistics level must be set with a level greater than 0 before calling this function. Statistics counters begin accumulating from this point and can be stopped and restarted without resetting counters unless `cuFileStatsReset()` is called.

### Signature
```c
CUfileError_t cuFileStatsStart(void)
```

### Parameters
None.

### Returns
* CU_FILE_SUCCESS - On success.
* CU_FILE_DRIVER_NOT_INITIALIZED - If the driver is not initialized.
```

--------------------------------

### handleGetObject()

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjServer-api/index.html

Services a GET request using data already present in the registered server buffer by performing an RDMA WRITE to the client buffer.

```APIDOC
## handleGetObject()

### Description
Services a GET request using data already present in the registered server buffer. This method performs an RDMA WRITE to the client buffer.

### Signature
```cpp
ssize_t handleGetObject(const std::string& key, struct rdma_buffer* local_rdma_buff, uint64_t remote_buf_start, size_t size, const std::string& rdma_descr, uint16_t channel = 0, uint64_t local_offset = 0, ibv_wc_status* status = nullptr, void* async_handle = nullptr);

ssize_t handleGetObject(const std::string& key, struct rdma_buffer* local_rdma_buff, uint64_t remote_buf_start, size_t size, const std::string& rdma_descr, uint32_t poll_delay, uint16_t channel = 0, uint64_t local_offset = 0, ibv_wc_status* status = nullptr, void* async_handle = nullptr);
```

### Parameters
- **key** (std::string) - Request identifier used for telemetry.
- **local_rdma_buff** (struct rdma_buffer*) - Registered server buffer containing the data to send.
- **remote_buf_start** (uint64_t) - Remote buffer start address for this request.
- **size** (size_t) - Transfer size in bytes.
- **rdma_descr** (std::string) - Opaque RDMA descriptor string received from the client.
- **poll_delay** (uint32_t) - Per-request polling delay in nanoseconds for synchronous calls.
- **channel** (uint16_t) - Channel returned by allocateChannelId().
- **local_offset** (uint64_t) - Offset into local_rdma_buff.
- **status** (ibv_wc_status*) - Optional completion status output for synchronous calls.
- **async_handle** (void*) - User handle for asynchronous completion. Use nullptr for synchronous operation.

### Returns
- **ssize_t** - Number of bytes transferred on synchronous success, 0 when asynchronous submission succeeds, or a negative errno-style value on failure.
```

--------------------------------

### GET Callback

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjClient-api/index.html

The get callback is invoked during a cuObjGet() operation to handle data retrieval from the object store.

```APIDOC
## GET Callback

### Description
Called during a cuObjGet() operation to handle data retrieval. The callback must communicate with cuObjServer using the provided RDMA descriptor information.

### Signature
ssize_t (*get)(const void* handle, char* ptr, size_t size, loff_t offset, const cufileRDMAInfo_t* rdma_info);

### Parameters
- **handle** (const void*) - Cookie to user context. Use cuObjClient::getCtx(handle) to extract the context.
- **ptr** (char*) - Pointer to the memory chunk to be filled with data.
- **size** (size_t) - Size of the memory chunk being read.
- **offset** (loff_t) - Starting object offset for this chunk.
- **rdma_info** (const cufileRDMAInfo_t*) - RDMA memory descriptor string for out-of-band communication.

### Returns
- **ssize_t** - Size of data read on success, or a negative failure value on failure. May return retryable negative errno values.
```

--------------------------------

### Check libibverbs on RHEL

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Query the RPM package manager to check the installation status and details of the `libibverbs` library on RHEL-based systems.

```bash
> []# rpm -qi libibverbs
>     Name        : libibverbs
>     Version     : 54mlnx1
>     Release     : 1.54103
>     Architecture: x86_64
>     Install Date: Tue 13 Jul 2021 10:21:18 AM CEST
>     Group       : System Environment/Libraries
>     Size        : 535489
>     License     : GPLv2 or BSD
>     Signature   : DSA/SHA1, Fri 02 Jul 2021 08:14:44 PM CEST, Key ID c5ed83e26224c050
>     Source RPM  : rdma-core-54mlnx1-1.54103.src.rpm
>     Build Date  : Fri 02 Jul 2021 06:59:01 PM CEST
>     Build Host  : c-141-24-1-005.mtl.labs.mlnx
>     Relocations : (not relocatable)
>     URL         : https://github.com/linux-rdma/rdma-core
>     Summary     : A library and drivers for direct userspace use of RDMA (InfiniBand/iWARP/RoCE) hardware
>     Description :
>     libibverbs is a library that allows userspace processes to use RDMA
>     "verbs" as described in the InfiniBand Architecture Specification and
>     the RDMA Protocol Verbs Specification.  This includes direct hardware
>     access from userspace to InfiniBand/iWARP adapters (kernel bypass) for
>     fast path operations.
> 
>     Device-specific plug-in ibverbs userspace drivers are included:
> 
> - libmlx5: Mellanox ConnectX-4+ InfiniBand HCA 
```

--------------------------------

### Display Docker Information

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Show general information about the Docker installation, including version, storage driver, and number of containers. Useful for diagnosing container-related issues.

```bash
# docker info
```

--------------------------------

### Configure Dynamic Routing with GPU Memory

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Example configuration for enabling dynamic routing and prioritizing GPU memory for RDMA devices. Ensure 'rdma_dev_addr_list' is correctly set for your network.

```json
"rdma_dev_addr_list": [ "192.168.0.12", "192.168.1.12" ],
"rdma_dynamic_routing": true,
"rdma_dynamic_routing_order": [ "GPU_MEM_NVLINKS", "GPU_MEM", "SYS_MEM" ]
```

--------------------------------

### Check GDS Installation Version

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Run the `gdscheck.py -v` command to determine the installed version of GPUDirect Storage and its related components.

```bash
$ gdscheck.py -v

Example output:
GDS release version: 1.0.0.78
 nvidia_fs version:  2.7 libcufile version: 2.4
```

--------------------------------

### Configure Telemetry

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjClient-api/index.html

Demonstrates how to set up telemetry logging to a file and define log levels.

```cpp
// Setup telemetry with custom output stream
std::ofstream log_file("cuobj_client.log");
cuObjClient::setupTelemetry(false, &log_file);
cuObjClient::setTelemFlags(CUOBJ_LOG_PATH_ERROR |
                           CUOBJ_LOG_PATH_INFO |
                           CUOBJ_LOG_PATH_DEBUG);

{
    // Create and use client
    cuObjClient client(ops);
    // ... perform operations ...
}

// Shutdown telemetry before closing file
cuObjClient::shutdownTelemetry();
log_file.close();
```

--------------------------------

### Batch IO Operations with cuFile in C++

Source: https://docs.nvidia.com/gpudirect-storage/best-practices-guide/index.html

Illustrates using cuFile batch APIs for performing multiple read operations concurrently. This example shows setting up batch parameters, submitting IO, and retrieving status, highlighting how 'min_nr' can be adjusted for enhanced throughput.

```cpp
int main(int argc, char *argv[]) {
 int fd[MAX_BATCH_IOS];
 void *devPtr[MAX_BATCH_IOS];
 CUfileDescr_t cf_descr[MAX_BATCH_IOS];
 CUfileHandle_t cf_handle[MAX_BATCH_IOS];
 CUfileIOParams_t io_batch_params[MAX_BATCH_IOS];
 CUfileIOEvents_t io_batch_events[MAX_BATCH_IOS];

 <Get program inputs>

 status = cuFileDriverOpen();
 if (status.err != CU_FILE_SUCCESS) {
 std::cerr << "cufile driver open error: "
 << cuFileGetErrorString(status) << std::endl;
 return -1;
 }

 <Open files and call cuFileHandleRegister for each of the batch entry file handles>

 <Allocate cuda memory and register buffers using cuFileBufRegister for each of the
 batch entries>

 for(i = 0; i < batch_size; i++) {
 io_batch_params[i].mode = CUFILE_BATCH;
 io_batch_params[i].fh = cf_handle[i];
 io_batch_params[i].u.batch.devPtr_base = devPtr[i];
 io_batch_params[i].u.batch.file_offset = i * size;
 io_batch_params[i].u.batch.devPtr_offset = 0;
 io_batch_params[i].u.batch.size = size;
 io_batch_params[i].opcode = CUFILE_READ;
 }
 std::cout << "Setting Up Batch" << std::endl;
 errorBatch = cuFileBatchIOSetUp(&batch_id, batch_size);
 if(errorBatch.err != 0) {
 std::cerr << "Error in setting Up Batch" << std::endl;
 goto error;
 }

 errorBatch = cuFileBatchIOSubmit(batch_id, batch_size, io_batch_params, flags);
 if(errorBatch.err != 0) {
 std::cerr << "Error in IO Batch Submit" << std::endl;
 goto error;
 }

 // Setting min_nr to batch_size for this example.
 min_nr = batch_size;
 while(num_completed != min_nr) {
 memset(io_batch_events, 0, sizeof(*io_batch_events));
 nr = batch_size;
 errorBatch = cuFileBatchIOGetStatus(batch_id, batch_size, &nr, io_batch_events, NULL);
 if(errorBatch.err != 0) {
 std::cerr << "Error in IO Batch Get Status" << std::endl;
 goto error;
 }
 std::cout << "Got events " << nr << std::endl;
 num_completed += nr;
 <Copy to the user buffer>
 }

 cuFileBatchIODestroy(batch_id);
 < Deregister the device memory using cuFileBufDeregister>

 status = cuFileDriverClose();
 std::cout << "cuFileDriverClose Done" << std::endl;
 if (status.err != CU_FILE_SUCCESS) {
 ...
 }
 ret = 0;
 return ret;
 ...
}
```

--------------------------------

### Perform GET operation with cuObjGet

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjClient-api/index.html

Executes a synchronous GET operation. Requires non-null context and pointer, with size greater than zero.

```cpp
ssize_t cuObjGet(void* ctx, void* ptr, size_t size, loff_t offset = 0,
                 loff_t buf_offset = 0);
```

--------------------------------

### Add NVIDIA GDS Preview Repository for Ubuntu

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Add the NVIDIA GDS preview repository for Ubuntu 20.04 to install GDS 1.0.0, NVSM, and MLNX_OFED packages via network.

```bash
$ sudo apt-key adv --fetch-keys https://repo.download.nvidia.com/baseos/GPG-KEY-dgx-cosmos-support

$ sudo add-apt-repository "deb https://repo.download.nvidia.com/baseos/ubuntu/focal/x86_64/ focal-updates preview"

$ sudo apt update
```

--------------------------------

### cuFileGetBARSizeInKB

Source: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

Get the BAR size for a specific GPU in KB. This API is used to get the BAR size of the GPU passed in as a parameter.

```APIDOC
## cuFileGetBARSizeInKB

### Description
Retrieves the Base Address Register (BAR) size for a specified GPU in kilobytes.

### Signature
CUfileError_t cuFileGetBARSizeInKB(int gpuIndex, size_t *barSize)

### Parameters
- **gpuIndex** (int) - The index of the GPU to query.
- **barSize** (size_t *) - Pointer to store the BAR size in KB.

### Returns
- CU_FILE_SUCCESS on success.
- CU_FILE_DRIVER_NOT_INITIALIZED if the driver is not initialized.
- CU_FILE_INVALID_VALUE on invalid input.
```

--------------------------------

### static void setupTelemetry(bool use_OTEL, std::ostream* os)

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjClient-api/index.html

Configures the telemetry output stream for logging and monitoring. This should be called before creating the first cuObjClient object.

```APIDOC
## static void setupTelemetry(bool use_OTEL, std::ostream* os)

### Description
Configures the telemetry output stream for logging and monitoring. Call this before creating the first cuObjClient object.

### Parameters
- **use_OTEL** (bool) - Required - If true, requests OpenTelemetry integration (if supported). If false, uses stream telemetry through the provided output stream.
- **os** (std::ostream*) - Required - Output stream for telemetry data. Must be non-null when stream logging is used and must remain valid until all cuObjClient objects are destroyed.
```

--------------------------------

### Custom RDMA Dynamic Routing Order Example 1

Source: https://docs.nvidia.com/gpudirect-storage/overview-guide/index.html

A sample routing order that prioritizes NVLink-based GPU memory transfers and falls back to system memory if NVLinks are not available or applicable.

```json
"rdma_dynamic_routing_order": [ "GPU_MEM_NVLINKS", "SYS_MEM"]
```

--------------------------------

### Check nvidia-fs.ko Support for Memory Peer Direct

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Run this command to verify if nvidia-fs.ko is loaded and supports memory peer direct. Review the output for confirmation.

```bash
lsmod | grep nvidia_fs | grep ib_core && echo "Ready for Memory Peer Direct"
```

--------------------------------

### Verify GDS Installation Log File

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Check for the presence of the cufile.log file to verify successful GDS installation. This command lists the log file if it exists in the specified directory.

```bash
$ ls -l /opt/myapp/cufile.log
```

--------------------------------

### Install DOCA NVMe Packages on RHEL

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Install DOCA packages for NVMe and NVMe-oF support on RHEL systems. This includes necessary firmware updaters and kernel modules. A reboot is required.

```bash
sudo dnf install doca-ofed mlnx-fw-updater kmod-mlnx-nvme
$ sudo dracut -f
$ reboot
```

--------------------------------

### Install DOCA NVMe Packages on Ubuntu

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Install DOCA packages for NVMe and NVMe-oF support on Ubuntu systems. This includes necessary firmware updaters and kernel modules. A reboot is required.

```bash
sudo apt install doca-ofed mlnx-fw-updater mlnx-nvme-dkms
$ sudo update-initramfs -u -k `uname -r`
$ reboot
```

--------------------------------

### Start gdsio job and collect level 3 stats with gds_stats

Source: https://docs.nvidia.com/gpudirect-storage/configuration-guide/index.html

Starts a gdsio job in the background and then uses gds_stats to extract detailed statistics. Ensure 'profile:cufile_stats' is enabled in /etc/cufile.json.

```bash
dgx2> gdsio -D /nvme23/gds_dir -d 2 -w 8 -s 1G -i 1M -x 0 -I 0 -T 300 &
[1] 850272
dgx2> gds_stats -p 850272 -l 3
```

--------------------------------

### cuFileBatchIOSetUp

Source: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

Initializes a batch I/O operation, specifying the maximum number of entries the batch will hold. It returns a handle for subsequent batch I/O calls.

```APIDOC
## cuFileBatchIOSetUp

### Description
This interface should be the first call in the sequence of batch I/O operation. This takes the maximum number of batch entries the caller intends to use and returns a `CUFileBatchHandle_t` which should be used by the caller for subsequent batch I/O calls.

### Parameters
#### Path Parameters
- None

#### Query Parameters
- None

#### Request Body
- None

### Parameters
- **max_nr** (Input) - The maximum number of events this batch will hold. The number should be between 1 - `properties.io_batch_size`.
- **batch_idp** (Output) - Will be used in subsequent batch IO calls.

### Returns
- `CU_FILE_SUCCESS` on success.
- `CU_FILE_INTERNAL_ERROR` on any failures.

### Request Example
```c
CUfileError_t cuFileBatchIOSetUp(CUfileBatchHandle_t *batch_idp, int max_nr);
```

### Response
#### Success Response (CU_FILE_SUCCESS)
- None (Indicates successful setup of batch I/O)

#### Response Example
```c
// Success is indicated by the return value CU_FILE_SUCCESS
```
```

--------------------------------

### Install MLNX_OFED with GDS and NVMe Support

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Install MLNX_OFED with support for NVMe, NVMe-oF, and GPU Direct Storage. This command enables necessary features for GDS functionality. Reboot is required for changes to take effect.

```bash
sudo ./mlnxofedinstall --with-nvmf --with-nfsrdma --enable-gds --add-kernel-support --dkms
```

--------------------------------

### Run gdsio with a configuration file

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Use the gdsio command with a configuration file to test storage performance. Refer to the 'Tabulated Fields' section for details on available options.

```bash
./gdsio <config file name>
```

--------------------------------

### Install and Dump Health with NVSM on DGX OS

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

On DGX OS systems, install the NVIDIA System Management (NVSM) tool and use the 'nvsm dump health' command to collect system health information for troubleshooting.

```bash
$ sudo apt-get install nvsm
$ sudo nvsm dump health
```

--------------------------------

### Verify GDS Installation with gdscheck

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Run `gdscheck` to verify a successful GDS installation. This command checks for supported file systems, devices, and PCIe ACS configuration. Ensure python3 is available or specify the path to python.

```bash
$ /usr/local/cuda-<x>.<y>/gds/tools/gdscheck.py -p
```

```bash
$ /usr/bin/python /usr/local/cuda-<x>.<y>/gds/tools/gdscheck.py -p
```

--------------------------------

### Initialize RDMA Configuration

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjServer-api/index.html

Copies tunable values into the connection object.

```cpp
void initRDMAConfigParams(cuObjRDMATunable config_params);
```

--------------------------------

### cuMemObjGetMaxRequestCallbackSize

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjClient-api/index.html

Gets the maximum callback size for a memory pointer.

```APIDOC
## cuMemObjGetMaxRequestCallbackSize

### Description
Retrieves the maximum callback size in bytes for a given memory pointer, which is useful for determining chunk sizes for I/O operations.

### Signature
`ssize_t cuMemObjGetMaxRequestCallbackSize(void* ptr);`

### Parameters
- **ptr** (void*) - Required - Start address of user memory.

### Returns
- **ssize_t** - Maximum callback size in bytes, or -1 if the size cannot be determined or the pointer is null.
```

--------------------------------

### Get Channel Port

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjServer-api/index.html

Returns the source port of the channel.

```cpp
int getChannelPort(struct rdma_cm_id* id);
```

--------------------------------

### isConnected()

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjServer-api/index.html

Checks if the server RDMA session was started successfully.

```APIDOC
## isConnected()

### Description
Returns true if the server RDMA session was started successfully, false otherwise.

### Signature
```cpp
bool isConnected();
```
```

--------------------------------

### Enable Peer Affinity Stats for nvidia-fs

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Execute this command to enable peer affinity statistics for nvidia-fs. This requires sudo privileges.

```bash
sudo bash
echo 1 > /sys/module/nvidia_fs/parameters/peer_stats_enabled
```

--------------------------------

### GET Callback Signature

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjClient-api/index.html

Signature for the callback function invoked during cuObjGet operations.

```c
ssize_t (*get)(const void* handle, char* ptr, size_t size, loff_t offset,
               const cufileRDMAInfo_t* rdma_info);
```

--------------------------------

### Sample Program using cuFile APIs

Source: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

A sample C++ program demonstrating the usage of cuFile APIs for file operations directly from CUDA memory. It covers file opening, cuFile driver initialization, handle registration, buffer allocation and registration, writing data to a file, and resource cleanup. Ensure the TESTFILE environment variable is set to a GDS-enabled filesystem path.

```c++
// To compile this sample code:
//
// nvcc gds_helloworld.cxx -o gds_helloworld -lcufile
//
// Set the environment variable TESTFILE
// to specify the name of the file on a GDS enabled filesystem
//
// Ex: TESTFILE=/mnt/gds/gds_test ./gds_helloworld
//
//
#include <fcntl.h>
#include <errno.h>
#include <unistd.h>

#include <cstdlib>
#include <cstring>
#include <iostream>
#include <cuda_runtime.h>
#include "cufile.h"

//#include "cufile_sample_utils.h"
using namespace std;

int main(void) {
 int fd;
 ssize_t ret;
 void *devPtr_base;
 off_t file_offset = 0x2000;
 off_t devPtr_offset = 0x1000;
 ssize_t IO_size = 1UL << 24;
 size_t buff_size = IO_size + 0x1000;
 CUfileError_t status;
 // CUResult cuda_result;
 int cuda_result;
 CUfileDescr_t cf_descr;
 CUfileHandle_t cf_handle;
 char *testfn;

 testfn=getenv("TESTFILE");
 if (testfn==NULL) {
 std::cerr << "No testfile defined via TESTFILE. Exiting." << std::endl;
 return -1;
 }

 cout << std::endl;
 cout << "Opening File " << testfn << std::endl;

 fd = open(testfn, O_CREAT|O_WRONLY|O_DIRECT, 0644);
 if(fd < 0) {
 std::cerr << "file open " << testfn << "errno " << errno << std::endl;
 return -1;
 }

 // the above fd could also have been opened without O_DIRECT starting CUDA toolkit 12.2
 // (gds 1.7.x version) as follows
 // fd = open(testfn, O_CREAT|O_WRONLY, 0644);

 cout << "Opening cuFileDriver." << std::endl;
 status = cuFileDriverOpen();
 if (status.err != CU_FILE_SUCCESS) {
 std::cerr << " cuFile driver failed to open " << std::endl;
 close(fd);
 return -1;
 }

 cout << "Registering cuFile handle to " << testfn << "." << std::endl;

 memset((void *)&cf_descr, 0, sizeof(CUfileDescr_t));
 cf_descr.handle.fd = fd;
 cf_descr.type = CU_FILE_HANDLE_TYPE_OPAQUE_FD;
 status = cuFileHandleRegister(&cf_handle, &cf_descr);
 if (status.err != CU_FILE_SUCCESS) {
 std::cerr << "cuFileHandleRegister fd " << fd << " status " << status.err << std::endl;
 close(fd);
 return -1;
 }

 cout << "Allocating CUDA buffer of " << buff_size << " bytes." << std::endl;

 cuda_result = cudaMalloc(&devPtr_base, buff_size);
 if (cuda_result != CUDA_SUCCESS) {
 std::cerr << "buffer allocation failed " << cuda_result << std::endl;
 cuFileHandleDeregister(cf_handle);
 close(fd);
 return -1;
 }

 cout << "Registering Buffer of " << buff_size << " bytes." << std::endl;
 status = cuFileBufRegister(devPtr_base, buff_size, 0);
 if (status.err != CU_FILE_SUCCESS) {
 std::cerr << "buffer registration failed " << status.err << std::endl;
 cuFileHandleDeregister(cf_handle);
 close(fd);
 cudaFree(devPtr_base);
 return -1;
 }

 // fill a pattern
 cout << "Filling memory." << std::endl;

 cudaMemset((void *) devPtr_base, 0xab, buff_size);
 cuStreamSynchronize(0);

 // perform write operation directly from GPU mem to file
 cout << "Writing buffer to file." << std::endl;
 ret = cuFileWrite(cf_handle, devPtr_base, IO_size, file_offset, devPtr_offset);

 if (ret < 0 || ret != IO_size) {
 std::cerr << "cuFileWrite failed " << ret << std::endl;
 }

 // release the GPU memory pinning
 cout << "Releasing cuFile buffer." << std::endl;
 status = cuFileBufDeregister(devPtr_base);
 if (status.err != CU_FILE_SUCCESS) {
 std::cerr << "buffer deregister failed" << std::endl;
 cudaFree(devPtr_base);
 cuFileHandleDeregister(cf_handle);
 close(fd);
 return -1;
 }

 cout << "Freeing CUDA buffer." << std::endl;
 cudaFree(devPtr_base);
 // deregister the handle from cuFile
 cout << "Releasing file handle. " << std::endl;

```

--------------------------------

### cuObjGet

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/index.html

Initiates a GET operation by inserting the x-amz-rdma-token tag in the HTTP header.

```APIDOC
## cuObjGet

### Signature
`cuObjGet(void *ctx, void *ptr, size_t size, loff_t offset)`

### Description
Initiates a GET operation by inserting the `x-amz-rdma-token` tag in the HTTP header.
```

--------------------------------

### Check Connection Status

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjServer-api/index.html

Verifies if the server RDMA session was started successfully.

```cpp
bool isConnected();
```

--------------------------------

### Run gdsio for GPUDirect Storage Reads

Source: https://docs.nvidia.com/gpudirect-storage/configuration-guide/index.html

Use the `gdsio` command to initiate GPUDirect Storage reads from storage to the GPU. This example specifies the data directory, GPU ID, number of threads, dataset size, IO size, and transfer timeout.

```bash
dgx2> gdsio -D /nvme23/gds_dir -d 12 -w 8 -s 500M -i 1M -x 0 -I 0 -T 120
```

--------------------------------

### Verify EXT4 Mount Options

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Check the current mount options for a file system to ensure it is using the correct journaling mode for GDS. This example shows a file system mounted with 'data=writeback', which is not supported.

```bash
mount | grep /mnt1
```

--------------------------------

### Get Configuration Tunable Parameter

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjServer-api/index.html

Copies current tunable values into the provided parameter structure.

```cpp
void getConfigTunableParam(struct cuObjRDMATunableParam* param);
```

--------------------------------

### Check EXT4 Mount Options

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Verify the mount options for EXT4 file systems. Ensure journaling mode is set to `data=ordered`.

```bash
$ mount | grep ext4
   /dev/sda2 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
   /dev/nvme1n1 on /mnt type ext4 (rw,relatime,data=ordered)
   /dev/nvme0n1p2 on /mnt1 type ext4 (rw,relatime,data=writeback)
```

--------------------------------

### Configure custom RDMA tuning parameters

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjServer-api/index.html

Sets specific RDMA parameters before initializing the server instance.

```cpp
cuObjRDMATunable params;
params.setNumDcis(256);
params.setCqDepth(1024);
params.setTimeout(20);
params.setRetryCount(7);
params.setDelayInterval(10000);
params.setQPResetOnFailure(true);
params.setMaxRdAtomic(16);

cuObjServer server("fd00::10", 18515, CUOBJ_PROTO_RDMA_DC_V1, params);
```

--------------------------------

### Get Channel IP

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjServer-api/index.html

Returns a pointer to a static string containing the channel IP address.

```cpp
char* getChannelIP(struct rdma_cm_id* id);
```

--------------------------------

### Check XFS Mount Options

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Use this command to check mount options for XFS file systems.

```bash
mount | grep xfs
```
Tokens:
Raw