### Example Output: IOMMU Enabled

Source: https://docs.nvidia.com/gpudirect-storage/best-practices-guide/index.html

This is an example output from 'cat /proc/cmdline' indicating that IOMMU is enabled on the system. IOMMU should be disabled for GDS.

```bash
BOOT_IMAGE=/boot/vmlinuz-5.19.0-38-generic root=UUID=fb2a25a8-9d2e-4e1c-9d8a-efabdf165adc ro rootflags=data=ordered amd_iommu=on
```

--------------------------------

### Callback Implementation for GET and PUT Operations

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjClient-api/index.html

Provides example implementations for `my_get_callback` and `my_put_callback` functions. These callbacks handle communication with the server for GET and PUT operations, leveraging RDMA for direct data transfer.

```cpp
ssize_t my_get_callback(const void* handle, char* ptr, size_t size,
                        loff_t offset, const cufileRDMAInfo_t* rdma_info) {
    // Extract user context
    MyContext* ctx = (MyContext*)cuObjClient::getCtx(handle);

    // Get RDMA descriptor information
    const char* rdma_desc = (const char*)rdma_info;

    // Communicate with server via control path
    // Send: operation=GET, size, offset, rdma_desc
    ssize_t bytes_read = send_request_to_server(ctx->server_conn,
                                                "GET", size, offset, rdma_desc);

    // Server performs RDMA_WRITE directly to ptr
    return bytes_read;
}

ssize_t my_put_callback(const void* handle, const char* ptr, size_t size,
                        loff_t offset, const cufileRDMAInfo_t* rdma_info) {
    // Extract user context
    MyContext* ctx = (MyContext*)cuObjClient::getCtx(handle);

    // Get RDMA descriptor information
    const char* rdma_desc = (const char*)rdma_info;

    // Communicate with server via control path
    // Send: operation=PUT, size, offset, rdma_desc
    ssize_t bytes_written = send_request_to_server(ctx->server_conn,
                                                   "PUT", size, offset, rdma_desc);

    // Server performs RDMA_READ directly from ptr
    return bytes_written;
}

```

--------------------------------

### cuFile API Call Example

Source: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

This example demonstrates the usage of cuFile APIs for driver, handle, and buffer registration and deregistration. It highlights the explicit and implicit behaviors of these operations.

```c
cuFileDriverOpen();
cuFileHandleRegister(fd, &handle);
cuFileBufRegister(buffer, size, &buf_handle);

// ... operations ...

cuFileBufDeregister(buf_handle);
cuFileHandleDeregister(handle);
cuFileDriverClose();
```

--------------------------------

### Example Output: IOMMU Disabled

Source: https://docs.nvidia.com/gpudirect-storage/best-practices-guide/index.html

This is an example output from '/usr/local/cuda/gds/tools/gdscheck -p' showing that IOMMU is disabled. This is the recommended state for optimal GDS performance.

```bash
 IOMMU: disabled
 Platform verification succeeded 
```

--------------------------------

### Install WekaIO

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Run this command to install WekaIO. Ensure you replace <IP of one of the WekaIO hosts' IB interface> with the correct IP address.

```bash
$ curl http://<IP of one of the WekaIO hosts' IB interface>:14000/dist/v1/install | sh
```

--------------------------------

### Install NFS Server and Configure Storage

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Steps to install the Linux NFS server, format and mount NVMe storage, and configure exports. Ensure the server has Mellanox connect-X4/5 NIC with MLNX_OFED 5.3 or later.

```bash
sudo apt-get install nfs-kernel-server

```

```bash
mkfs.ext4 /dev/nvme0n1

```

```bash
mount -o data=ordered /dev/nvme0n1 /mnt/nvme

```

```bash
service nfs-kernel-server restart

```

```bash
modprobe rpcrdma

```

```bash
echo rdma 20049 > /proc/fs/nfsd/portlist

```

--------------------------------

### Install NFS Client with GPUDirect Storage Support

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Installs the NFS client and configures it for RDMA and GDS. Ensure MLNX_OFED 5.3 or later is installed on the client with a compatible Mellanox NIC.

```bash
$ ofed_info -s MLNX_OFED_LINUX-5.3-1.0.5.0:
```

```bash
$ sudo apt-get install nfs-common
```

```bash
$ modprobe rpcrdma
```

```bash
$ mkdir -p /mnt/nfs_rdma_gds
```

```bash
$ sudo mount -v -o proto=rdma,port=20049,vers=3 172.16.0.101:/ /mnt/nfs_rdma_gds
```

```bash
$ sudo mount -v -o proto=rdma,port=20049,vers=3,nconnect=20,localports=172.16.0.17-172.16.0.24,remoteports=172.16.0.101-172.16.0.120 172.16.0.101:/ /mnt/nfs_rdma_gds
```

--------------------------------

### Check GPUDirect Storage Installation

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Run this command to verify the GPUDirect Storage installation and check available BAR memory per GPU. Review the output for GPU index and BAR size.

```bash
/usr/local/cuda-x.y/gds/tools/gdscheck
```

--------------------------------

### Install VAST NFSoRDMA+Multipath Package

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Install the VAST NFSoRDMA+Multipath package using apt-get. Ensure you have the correct .deb file that matches your kernel and MLNX_OFED version.

```bash
$ sudo apt-get install mlnx-nfsrdma-*.deb
```

--------------------------------

### Check for libibverbs.so

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Confirm the installation of the 'libibverbs-dev' package, which provides the 'libibverbs.so' library.

```bash
$ dpkg -s libibverbs-dev Package: libibverbs-dev Status: install ok installed Priority: optional Section: libdevel Installed-Size: 1151 Maintainer: Linux RDMA Mailing List <linux-rdma@vger.kernel.org> Architecture: amd64 Multi-Arch: same Source: rdma-core Version: 47mlnx1-1.47329
```

--------------------------------

### Example Custom cufile.json for Application Logging

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Example of a cufile.json file configured for a specific application, demonstrating how to set the log directory. This file should be placed at the path specified by CUFILE_ENV_PATH_JSON.

```json
"logging": {
    // log directory, if not enabled

```

--------------------------------

### Verify cuObject Client Installation

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuobject-client-release-notes/index.html

Commands to check if cuObject client libraries and GDS support are installed correctly on the system.

```bash
# Check library presence
$ ldconfig -p | grep cuobj
$ ldconfig -p | grep cufile

# Verify cuObjClient installation
$ rpm -qa | grep cuobjclient    # RHEL/CentOS
$ dpkg -l | grep cuobjclient    # Ubuntu/Debian

# Verify GDS support
$ /usr/local/cuda/gds/tools/gdscheck -p

```

--------------------------------

### Compile GDSIO Example Program

Source: https://docs.nvidia.com/gpudirect-storage/getting-started/index.html

Command to compile the C++ GDSIO example program, linking necessary CUDA and GDS libraries.

```bash
g++ -o gds_example gds_example.cc -I/usr/local/cuda/include -L/usr/local/cuda/lib64 -lcuda -lcufile -lcudart
```

--------------------------------

### Get LLite Tuning Parameters

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Retrieves EXAScaler file system client tuning parameters for LLite (Lustre Lite).

```bash
lctl get_param llite.*.*
```

--------------------------------

### Check libibverbs-dev on Ubuntu

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Verify the installation status and details of the `libibverbs-dev` package on Ubuntu systems, which provides development headers for RDMA verbs.

```bash
$ dpkg -s libibverbs-dev
root@fscc-sr650-59:~# dpkg -s libibverbs-dev
Package: libibverbs-dev
Status: install ok installed
Priority: optional
Section: libdevel
Installed-Size: 1428
Maintainer: Linux RDMA Mailing List <linux-rdma@vger.kernel.org>
Architecture: amd64
Multi-Arch: same
Source: rdma-core
Version: 54mlnx1-1.54103 
```

--------------------------------

### Basic Client Usage with cuObjClient

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjClient-api/index.html

Demonstrates the fundamental steps for using the cuObjClient, including defining operations, creating a client instance, allocating and registering memory, performing GET and PUT operations, and cleanup.

```cpp
// 1. Define callback operations
CUObjOps_t ops = {
    .get = my_get_callback,
    .put = my_put_callback
};

// 2. Create client instance
cuObjClient client(ops, CUOBJ_PROTO_RDMA_DC_V1);

// 3. Allocate and register memory
void* buffer = malloc(1024 * 1024);  // 1MB buffer
if (client.cuMemObjGetDescriptor(buffer, 1024 * 1024) != CU_OBJ_SUCCESS) {
    // Handle error
}

// 4. Perform GET operation
MyContext ctx = { /* user data */ };
ssize_t result = client.cuObjGet(&ctx, buffer, 1024, 0, 0);
if (result < 0) {
    // Handle error
}

// 5. Perform PUT operation
result = client.cuObjPut(&ctx, buffer, 1024, 0, 0);
if (result < 0) {
    // Handle error
}

// 6. Cleanup
client.cuMemObjPutDescriptor(buffer);
free(buffer);

```

--------------------------------

### Install Matching Fabric Manager Version

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Ensure the installed version of Fabric Manager matches the CUDA driver version to resolve CUDA_ERROR_SYSTEM_NOT_READY on systems with NVSwitch. This example shows installing nvidia-fabricmanager-460 for nvidia-driver-460-server.

```bash
$ sudo apt install nvidia-driver-460-server -y

then use:

$ apt-get install nvidia-fabricmanager-460
```

--------------------------------

### Check MLNX_OFED Version

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Verify the installed MLNX_OFED version on the server. This command is used to confirm the correct version for NFS server setup with RDMA support.

```bash
ofed_info -s MLNX_OFED_LINUX-5.3-1.0.5.1:
```

--------------------------------

### Verify VAST Multipath Package Installation

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Check if the VAST NFSoRDMA+Multipath package is installed and verify its version using dpkg. This confirms the successful installation of the package.

```bash
$ dpkg -l | grep mlnx-nfsrdma
```

--------------------------------

### Check EXAScaler File System Client Module Version

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Run this command to verify the installed EXAScaler file system client module version. The output will display the Lustre version.

```bash
$ sudo lctl get_param version
```

--------------------------------

### Basic Server Usage (Single-Threaded)

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjServer-api/index.html

Demonstrates the fundamental steps for setting up and using the cuObjServer in a single-threaded environment, including buffer allocation, registration, and object handling.

```c
cuObjServer server("192.168.1.100", 8080, CUOBJ_PROTO_RDMA_DC_V1);
if (!server.isConnected()) {
    /* Handle error */
}

void* buffer = server.allocHostBuffer(1024 * 1024);
struct rdma_buffer* rdma_buf = server.registerBuffer(buffer, 1024 * 1024);
if (!rdma_buf) {
    /* Handle error */
}

memcpy(buffer, data_source, data_size);

ssize_t result = server.handleGetObject("request_123",
                                        rdma_buf,
                                        remote_addr,
                                        data_size,
                                        rdma_descriptor,
                                        0);
if (result < 0) {
    /* Handle error */
}

result = server.handlePutObject("request_456",
                                rdma_buf,
                                remote_addr,
                                data_size,
                                rdma_descriptor,
                                0);
if (result > 0) {
    process_data(buffer, (size_t)result);
}

server.deRegisterBuffer(rdma_buf);
free(buffer);


```

--------------------------------

### Batch IO Operations with cuFile in C++

Source: https://docs.nvidia.com/gpudirect-storage/best-practices-guide/index.html

Illustrates using cuFile batch APIs for performing multiple read operations concurrently. This example shows setting up batch parameters, submitting IO, and retrieving status, highlighting how 'min_nr' can be adjusted for enhanced throughput.

```cpp
int main(int argc, char *argv[]) {
 int fd[MAX_BATCH_IOS];
 void *devPtr[MAX_BATCH_IOS];
 CUfileDescr_t cf_descr[MAX_BATCH_IOS];
 CUfileHandle_t cf_handle[MAX_BATCH_IOS];
 CUfileIOParams_t io_batch_params[MAX_BATCH_IOS];
 CUfileIOEvents_t io_batch_events[MAX_BATCH_IOS];

 <Get program inputs>

 status = cuFileDriverOpen();
 if (status.err != CU_FILE_SUCCESS) {
 std::cerr << "cufile driver open error: "
 << cuFileGetErrorString(status) << std::endl;
 return -1;
 }

 <Open files and call cuFileHandleRegister for each of the batch entry file handles>

 <Allocate cuda memory and register buffers using cuFileBufRegister for each of the
 batch entries>

 for(i = 0; i < batch_size; i++) {
 io_batch_params[i].mode = CUFILE_BATCH;
 io_batch_params[i].fh = cf_handle[i];
 io_batch_params[i].u.batch.devPtr_base = devPtr[i];
 io_batch_params[i].u.batch.file_offset = i * size;
 io_batch_params[i].u.batch.devPtr_offset = 0;
 io_batch_params[i].u.batch.size = size;
 io_batch_params[i].opcode = CUFILE_READ;
 }
 std::cout << "Setting Up Batch" << std::endl;
 errorBatch = cuFileBatchIOSetUp(&batch_id, batch_size);
 if(errorBatch.err != 0) {
 std::cerr << "Error in setting Up Batch" << std::endl;
 goto error;
 }

 errorBatch = cuFileBatchIOSubmit(batch_id, batch_size, io_batch_params, flags);
 if(errorBatch.err != 0) {
 std::cerr << "Error in IO Batch Submit" << std::endl;
 goto error;
 }

 // Setting min_nr to batch_size for this example.
 min_nr = batch_size;
 while(num_completed != min_nr) {
 memset(io_batch_events, 0, sizeof(*io_batch_events));
 nr = batch_size;
 errorBatch = cuFileBatchIOGetStatus(batch_id, batch_size, &nr, io_batch_events, NULL);
 if(errorBatch.err != 0) {
 std::cerr << "Error in IO Batch Get Status" << std::endl;
 goto error;
 }
 std::cout << "Got events " << nr << std::endl;
 num_completed += nr;
 <Copy to the user buffer>
 }

 cuFileBatchIODestroy(batch_id);
 < Deregister the device memory using cuFileBufDeregister>

 status = cuFileDriverClose();
 std::cout << "cuFileDriverClose Done" << std::endl;
 if (status.err != CU_FILE_SUCCESS) {
 ...
 }
 ret = 0;
 return ret;
 ...
}
```

--------------------------------

### Check GDS Installation Version

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Run the `gdscheck.py -v` command to determine the installed version of GPUDirect Storage and its related components.

```bash
$ gdscheck.py -v

Example output:
GDS release version: 1.0.0.78
 nvidia_fs version:  2.7 libcufile version: 2.4
```

--------------------------------

### Configure Lustre Module Parameters

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Example Lustre module configuration. These options can be tuned for LNet performance based on the network type and host channel adapter (HCA).

```bash
cat /etc/modprobe.d/lustre.conf
```

--------------------------------

### cuFileGetBARSizeInKB

Source: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

Get the BAR size for a specific GPU in KB. This API is used to get the BAR size of the GPU passed in as a parameter.

```APIDOC
## cuFileGetBARSizeInKB

### Description
Retrieves the Base Address Register (BAR) size for a specified GPU in kilobytes.

### Signature
CUfileError_t cuFileGetBARSizeInKB(int gpuIndex, size_t *barSize)

### Parameters
- **gpuIndex** (int) - The index of the GPU to query.
- **barSize** (size_t *) - Pointer to store the BAR size in KB.

### Returns
- CU_FILE_SUCCESS on success.
- CU_FILE_DRIVER_NOT_INITIALIZED if the driver is not initialized.
- CU_FILE_INVALID_VALUE on invalid input.
```

--------------------------------

### Verify cuObject Server Installation

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuobject-server-release-notes/index.html

Commands to check for the presence of cuObject libraries and verify the installation of the cuObjServer package on RHEL/CentOS and Ubuntu/Debian systems.

```bash
# Check library presence
ldconfig -p | grep cuobj

# Verify cuObjServer installation
rpm -qa | grep cuobjserver    # RHEL/CentOS
dpkg -l | grep cuobjserver    # Ubuntu/Debian
```

--------------------------------

### Custom RDMA Dynamic Routing Order Example 1

Source: https://docs.nvidia.com/gpudirect-storage/overview-guide/index.html

A sample routing order that prioritizes NVLink-based GPU memory transfers and falls back to system memory if NVLinks are not available or applicable.

```json
"rdma_dynamic_routing_order": [ "GPU_MEM_NVLINKS", "SYS_MEM"]
```

--------------------------------

### Check nvidia-fs.ko Support for Memory Peer Direct

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Run this command to verify if nvidia-fs.ko is loaded and supports memory peer direct. Review the output for confirmation.

```bash
lsmod | grep nvidia_fs | grep ib_core && echo "Ready for Memory Peer Direct"
```

--------------------------------

### Check LNet Network Setup on Client

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Execute this command to inspect the LNet network configuration on a client system. Review the output for network details.

```bash
$ sudo lnetctl net show
```

--------------------------------

### Verify GDS Installation Log File

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Check for the presence of the cufile.log file to verify successful GDS installation. This command lists the log file if it exists in the specified directory.

```bash
$ ls -l /opt/myapp/cufile.log
```

--------------------------------

### Handle GET Object Operation with Poll Delay

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjServer-api/index.html

Handles a GET operation with an optional per-request poll delay in nanoseconds. This overrides the server's default polling configuration.

```c
ssize_t handleGetObject(const std::string& key,
                        struct rdma_buffer* local_rdma_buff,
                        uint64_t remote_buf_start, size_t size,
                        const std::string& rdma_descr,
                        uint32_t poll_delay,
                        uint16_t channel = 0, uint64_t local_offset = 0,
                        ibv_wc_status* status = nullptr,
                        void* async_handle = nullptr);
```

--------------------------------

### List GDS Tools and Samples

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

List the contents of the GPUDirect Storage tools and samples directory. This helps in locating available utilities and example code.

```bash
ls -lh /usr/local/cuda-X.Y/gds/
```

--------------------------------

### cuFileBatchIOSetUp

Source: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

Initializes a batch I/O operation, specifying the maximum number of entries the batch will hold. It returns a handle for subsequent batch I/O calls.

```APIDOC
## cuFileBatchIOSetUp

### Description
This interface should be the first call in the sequence of batch I/O operation. This takes the maximum number of batch entries the caller intends to use and returns a `CUFileBatchHandle_t` which should be used by the caller for subsequent batch I/O calls.

### Parameters
#### Path Parameters
- None

#### Query Parameters
- None

#### Request Body
- None

### Parameters
- **max_nr** (Input) - The maximum number of events this batch will hold. The number should be between 1 - `properties.io_batch_size`.
- **batch_idp** (Output) - Will be used in subsequent batch IO calls.

### Returns
- `CU_FILE_SUCCESS` on success.
- `CU_FILE_INTERNAL_ERROR` on any failures.

### Request Example
```c
CUfileError_t cuFileBatchIOSetUp(CUfileBatchHandle_t *batch_idp, int max_nr);
```

### Response
#### Success Response (CU_FILE_SUCCESS)
- None (Indicates successful setup of batch I/O)

#### Response Example
```c
// Success is indicated by the return value CU_FILE_SUCCESS
```
```

--------------------------------

### Run gdsio with a configuration file

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Use the gdsio command with a configuration file to test storage performance. Refer to the 'Tabulated Fields' section for details on available options.

```bash
./gdsio <config file name>
```

--------------------------------

### Install DOCA NVMe Packages on RHEL

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Install DOCA packages for NVMe and NVMe-oF support on RHEL systems. This includes necessary firmware updaters and kernel modules. A reboot is required.

```bash
sudo dnf install doca-ofed mlnx-fw-updater kmod-mlnx-nvme
$ sudo dracut -f
$ reboot
```

--------------------------------

### Install DOCA NVMe Packages on Ubuntu

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Install DOCA packages for NVMe and NVMe-oF support on Ubuntu systems. This includes necessary firmware updaters and kernel modules. A reboot is required.

```bash
sudo apt install doca-ofed mlnx-fw-updater mlnx-nvme-dkms
$ sudo update-initramfs -u -k `uname -r`
$ reboot
```

--------------------------------

### Telemetry Configuration with Custom Output

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjClient-api/index.html

Demonstrates how to configure telemetry logging, including setting a custom output stream (e.g., a file) and enabling specific logging flags. Remember to shut down telemetry before closing the output stream.

```cpp
// Setup telemetry with custom output stream
std::ofstream log_file("cuobj_client.log");
cuObjClient::setupTelemetry(false, &log_file);
cuObjClient::setTelemFlags(0xFFFF);  // Enable all logging

// Create and use client
cuObjClient client(ops);
// ... perform operations ...

// Shutdown telemetry before closing file
cuObjClient::shutdownTelemetry();
log_file.close();


```

--------------------------------

### GET Callback Function Signature

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjClient-api/index.html

Signature for the GET callback function, used by cuObjGet() to retrieve data. This callback handles data retrieval into a provided memory buffer, supporting RDMA communication.

```c
ssize_t (*get)(const void* handle, char* ptr, size_t size, loff_t offset,
               const cufileRDMAInfo_t* rdma_info);
```

--------------------------------

### Start gdsio job and collect level 3 stats with gds_stats

Source: https://docs.nvidia.com/gpudirect-storage/configuration-guide/index.html

Starts a gdsio job in the background and then uses gds_stats to extract detailed statistics. Ensure 'profile:cufile_stats' is enabled in /etc/cufile.json.

```bash
dgx2> gdsio -D /nvme23/gds_dir -d 2 -w 8 -s 1G -i 1M -x 0 -I 0 -T 300 &
[1] 850272
dgx2> gds_stats -p 850272 -l 3
```

--------------------------------

### Install MLNX_OFED with GDS and NVMe Support

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Install MLNX_OFED with support for NVMe, NVMe-oF, and GPU Direct Storage. This command enables necessary features for GDS functionality. Reboot is required for changes to take effect.

```bash
sudo ./mlnxofedinstall --with-nvmf --with-nfsrdma --enable-gds --add-kernel-support --dkms
```

--------------------------------

### Enable Peer Affinity Stats for nvidia-fs

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Execute this command to enable peer affinity statistics for nvidia-fs. This requires sudo privileges.

```bash
sudo bash
echo 1 > /sys/module/nvidia_fs/parameters/peer_stats_enabled
```

--------------------------------

### Handle GET Object Operation (Synchronous/Async)

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjServer-api/index.html

Performs a GET operation using RDMA_WRITE to client memory. Supports both synchronous and asynchronous modes. For async operations, `poll()` must be called to retrieve the completion status.

```c
ssize_t handleGetObject(const std::string& key,
                        struct rdma_buffer* local_rdma_buff,
                        uint64_t remote_buf_start, size_t size,
                        const std::string& rdma_descr,
                        uint16_t channel = 0, uint64_t local_offset = 0,
                        ibv_wc_status* status = nullptr,
                        void* async_handle = nullptr);
```

--------------------------------

### Accessing Infiniband Network Counters

Source: https://docs.nvidia.com/gpudirect-storage/configuration-guide/index.html

Example path to access performance counters for an Infiniband network interface. These counters provide detailed statistics on network traffic.

```bash
/sys/class/infiniband/[INTERFACE]/ports/[PORT NUMBER]/counters
```

--------------------------------

### Sample Program using cuFile APIs

Source: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

A sample C++ program demonstrating the usage of cuFile APIs for file operations directly from CUDA memory. It covers file opening, cuFile driver initialization, handle registration, buffer allocation and registration, writing data to a file, and resource cleanup. Ensure the TESTFILE environment variable is set to a GDS-enabled filesystem path.

```c++
// To compile this sample code:
//
// nvcc gds_helloworld.cxx -o gds_helloworld -lcufile
//
// Set the environment variable TESTFILE
// to specify the name of the file on a GDS enabled filesystem
//
// Ex: TESTFILE=/mnt/gds/gds_test ./gds_helloworld
//
//
#include <fcntl.h>
#include <errno.h>
#include <unistd.h>

#include <cstdlib>
#include <cstring>
#include <iostream>
#include <cuda_runtime.h>
#include "cufile.h"

//#include "cufile_sample_utils.h"
using namespace std;

int main(void) {
 int fd;
 ssize_t ret;
 void *devPtr_base;
 off_t file_offset = 0x2000;
 off_t devPtr_offset = 0x1000;
 ssize_t IO_size = 1UL << 24;
 size_t buff_size = IO_size + 0x1000;
 CUfileError_t status;
 // CUResult cuda_result;
 int cuda_result;
 CUfileDescr_t cf_descr;
 CUfileHandle_t cf_handle;
 char *testfn;

 testfn=getenv("TESTFILE");
 if (testfn==NULL) {
 std::cerr << "No testfile defined via TESTFILE. Exiting." << std::endl;
 return -1;
 }

 cout << std::endl;
 cout << "Opening File " << testfn << std::endl;

 fd = open(testfn, O_CREAT|O_WRONLY|O_DIRECT, 0644);
 if(fd < 0) {
 std::cerr << "file open " << testfn << "errno " << errno << std::endl;
 return -1;
 }

 // the above fd could also have been opened without O_DIRECT starting CUDA toolkit 12.2
 // (gds 1.7.x version) as follows
 // fd = open(testfn, O_CREAT|O_WRONLY, 0644);

 cout << "Opening cuFileDriver." << std::endl;
 status = cuFileDriverOpen();
 if (status.err != CU_FILE_SUCCESS) {
 std::cerr << " cuFile driver failed to open " << std::endl;
 close(fd);
 return -1;
 }

 cout << "Registering cuFile handle to " << testfn << "." << std::endl;

 memset((void *)&cf_descr, 0, sizeof(CUfileDescr_t));
 cf_descr.handle.fd = fd;
 cf_descr.type = CU_FILE_HANDLE_TYPE_OPAQUE_FD;
 status = cuFileHandleRegister(&cf_handle, &cf_descr);
 if (status.err != CU_FILE_SUCCESS) {
 std::cerr << "cuFileHandleRegister fd " << fd << " status " << status.err << std::endl;
 close(fd);
 return -1;
 }

 cout << "Allocating CUDA buffer of " << buff_size << " bytes." << std::endl;

 cuda_result = cudaMalloc(&devPtr_base, buff_size);
 if (cuda_result != CUDA_SUCCESS) {
 std::cerr << "buffer allocation failed " << cuda_result << std::endl;
 cuFileHandleDeregister(cf_handle);
 close(fd);
 return -1;
 }

 cout << "Registering Buffer of " << buff_size << " bytes." << std::endl;
 status = cuFileBufRegister(devPtr_base, buff_size, 0);
 if (status.err != CU_FILE_SUCCESS) {
 std::cerr << "buffer registration failed " << status.err << std::endl;
 cuFileHandleDeregister(cf_handle);
 close(fd);
 cudaFree(devPtr_base);
 return -1;
 }

 // fill a pattern
 cout << "Filling memory." << std::endl;

 cudaMemset((void *) devPtr_base, 0xab, buff_size);
 cuStreamSynchronize(0);

 // perform write operation directly from GPU mem to file
 cout << "Writing buffer to file." << std::endl;
 ret = cuFileWrite(cf_handle, devPtr_base, IO_size, file_offset, devPtr_offset);

 if (ret < 0 || ret != IO_size) {
 std::cerr << "cuFileWrite failed " << ret << std::endl;
 }

 // release the GPU memory pinning
 cout << "Releasing cuFile buffer." << std::endl;
 status = cuFileBufDeregister(devPtr_base);
 if (status.err != CU_FILE_SUCCESS) {
 std::cerr << "buffer deregister failed" << std::endl;
 cudaFree(devPtr_base);
 cuFileHandleDeregister(cf_handle);
 close(fd);
 return -1;
 }

 cout << "Freeing CUDA buffer." << std::endl;
 cudaFree(devPtr_base);
 // deregister the handle from cuFile
 cout << "Releasing file handle. " << std::endl;

```

--------------------------------

### Install and Dump Health with NVSM on DGX OS

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

On DGX OS systems, install the NVIDIA System Management (NVSM) tool and use the 'nvsm dump health' command to collect system health information for troubleshooting.

```bash
$ sudo apt-get install nvsm
$ sudo nvsm dump health
```

--------------------------------

### Verify GDS Installation with gdscheck

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Run `gdscheck` to verify a successful GDS installation. This command checks for supported file systems, devices, and PCIe ACS configuration. Ensure python3 is available or specify the path to python.

```bash
$ /usr/local/cuda-<x>.<y>/gds/tools/gdscheck.py -p
```

```bash
$ /usr/bin/python /usr/local/cuda-<x>.<y>/gds/tools/gdscheck.py -p
```

--------------------------------

### Run gdsio for GPUDirect Storage Reads

Source: https://docs.nvidia.com/gpudirect-storage/configuration-guide/index.html

Use the `gdsio` command to initiate GPUDirect Storage reads from storage to the GPU. This example specifies the data directory, GPU ID, number of threads, dataset size, IO size, and transfer timeout.

```bash
dgx2> gdsio -D /nvme23/gds_dir -d 12 -w 8 -s 500M -i 1M -x 0 -I 0 -T 120
```

--------------------------------

### Perform GET Operation with cuObjGet

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjClient-api/index.html

Executes a synchronous GET operation to read data from a remote server into a registered local memory buffer. The memory must be registered via `cuMemObjGetDescriptor` prior to calling. If the requested size exceeds `MaxRequestCallbackSize`, multiple callbacks will be invoked.

```c
ssize_t cuObjGet(void* ctx, void* ptr, size_t size, loff_t offset = 0,
                 loff_t buf_offset = 0);
```

--------------------------------

### Verify EXT4 Mount Options

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Check the current mount options for a file system to ensure it is using the correct journaling mode for GDS. This example shows a file system mounted with 'data=writeback', which is not supported.

```bash
mount | grep /mnt1
```

--------------------------------

### Check EXT4 Mount Options

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Verify the mount options for EXT4 file systems. Ensure journaling mode is set to `data=ordered`.

```bash
$ mount | grep ext4
   /dev/sda2 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
   /dev/nvme1n1 on /mnt type ext4 (rw,relatime,data=ordered)
   /dev/nvme0n1p2 on /mnt1 type ext4 (rw,relatime,data=writeback)
```

--------------------------------

### I/O Operations

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjClient-api/contents.html

APIs for performing object get and put operations.

```APIDOC
## I/O Operations

### `cuObjGet`

#### Description
Retrieves an object from the storage.

### `cuObjPut`

#### Description
Stores an object to the storage.
```

--------------------------------

### setupTelemetry

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjServer-api/index.html

Configures telemetry output stream for logging and monitoring.

```APIDOC
## setupTelemetry

### Description
Configures telemetry output stream for logging and monitoring.

### Signature
```cpp
static void setupTelemetry(bool use_OTEL, std::ostream* os);
```

### Parameters
#### Path Parameters
- **use_OTEL** (bool) - Enables OpenTelemetry integration.
- **os** (std::ostream*) - Output stream for telemetry data.

### Notes
- The output stream must remain valid until `shutdownTelemetry()` is called.
- Affects all `cuObjServer` instances.
```

--------------------------------

### Run GDS Sample and Check Stats

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Executes a GPU Direct Storage sample and then displays statistics for the NVFS driver. This is useful for checking driver version, Mellanox PeerDirect support, and IO statistics.

```bash
$ /usr/local/cuda-x.y/gds/samples /mnt/lustre/test 0
```

```bash
$ cat /proc/driver/nvidia-fs/stats
```

```text
GDS Version: 1.0.0.71
NVFS statistics(ver: 4.0)
NVFS Driver(version: 2:7:47)
Mellanox PeerDirect Supported: True
IO stats: Enabled, peer IO stats: Enabled
Logging level: info

Active Shadow-Buffer (MB): 0
Active Process: 0
```

--------------------------------

### I/O Operations (Synchronous)

Source: https://docs.nvidia.com/gpudirect-storage/cuobject/cuObjServer-api/index.html

Synchronous methods for handling object GET and PUT operations.

```APIDOC
## handleGetObject(const std::string& key, struct rdma_buffer* local_rdma_buff, uint64_t remote_buf_start, size_t size, const std::string& rdma_descr, uint16_t channel = 0, uint64_t local_offset = 0, ibv_wc_status* status = nullptr, void* async_handle = nullptr)

### Description
Synchronously retrieves an object from the remote server.

### Parameters
- **key** (const std::string&) - The key of the object to retrieve.
- **local_rdma_buff** (struct rdma_buffer*) - The local RDMA buffer to store the object data.
- **remote_buf_start** (uint64_t) - The starting address of the remote buffer.
- **size** (size_t) - The size of the object to retrieve.
- **rdma_descr** (const std::string&) - RDMA descriptor string.
- **channel** (uint16_t, optional) - The channel ID to use (default: 0).
- **local_offset** (uint64_t, optional) - The offset in the local buffer (default: 0).
- **status** (ibv_wc_status*, optional) - Pointer to store the completion status.
- **async_handle** (void*, optional) - Handle for asynchronous operations.

### Returns
- The number of bytes read on success.
- A negative value on failure.
```

```APIDOC
## handlePutObject(const std::string& key, struct rdma_buffer* local_rdma_buff, uint64_t remote_buf_start, size_t size, const std::string& rdma_descr, uint16_t channel = 0, uint64_t local_offset = 0, ibv_wc_status* status = nullptr, void* async_handle = nullptr)

### Description
Synchronously sends an object to the remote server.

### Parameters
- **key** (const std::string&) - The key of the object to put.
- **local_rdma_buff** (struct rdma_buffer*) - The local RDMA buffer containing the object data.
- **remote_buf_start** (uint64_t) - The starting address of the remote buffer.
- **size** (size_t) - The size of the object to put.
- **rdma_descr** (const std::string&) - RDMA descriptor string.
- **channel** (uint16_t, optional) - The channel ID to use (default: 0).
- **local_offset** (uint64_t, optional) - The offset in the local buffer (default: 0).
- **status** (ibv_wc_status*, optional) - Pointer to store the completion status.
- **async_handle** (void*, optional) - Handle for asynchronous operations.

### Returns
- The number of bytes written on success.
- A negative value on failure.
```

--------------------------------

### Check XFS Mount Options

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Use this command to check mount options for XFS file systems.

```bash
mount | grep xfs
```

--------------------------------

### Check WekaIO Version

Source: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html

Verify the installed WekaIO version using the 'weka status' command.

```bash
$ weka status WekaIO v3.6.2.5-rdma-beta (CLI build 3.6.2.5-rdma-beta)
```