NVIDIA Fabric Manager

https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/
Admin
NVIDIA Fabric Manager configures and monitors NVSwitch memory fabrics to enable multi-GPU...
Tokens:30,609
Snippets:273
Trust Score:10
Update:9 months ago
Show doc for...
Tokens:
Raw
### Fabric Manager Startup Script Location (Linux)

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

Provides the default installation path for the NVIDIA Fabric Manager startup script (`nv-fabricmanager-start.sh`) on Linux. This script is invoked by the systemd service unit file to selectively start FM and NVLSM processes based on the platform.

```bash
/usr/bin/nv-fabricmanager-start.sh
```

--------------------------------

### Install NVLink5 and NVIDIA Open Driver on Debian/Ubuntu

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

Installs the necessary NVIDIA open driver and NVLink5 components for systems using fourth-generation NVSwitches on Debian and Ubuntu-based operating systems. Requires specifying the driver branch.

```bash
sudo apt-get install -V nvidia-open-<driver-branch>
sudo apt-get install -V nvlink5-<driver-branch>
```

```bash
sudo apt-get install -V nvidia-open-570
sudo apt-get install -V nvlink5-570
```

--------------------------------

### Getting a List of Supported Partitions

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

Queries and retrieves a list of supported static GPU fabric partitions within an NVSwitch-based system.

```APIDOC
## fmGetSupportedFabricPartitions

### Description
Queries the list of supported (static) GPU fabric partitions in an NVSwitch-based system.

### Method
`fmGetSupportedFabricPartitions(fmHandle_t pFmHandle, fmFabricPartitionList_t *pFmFabricPartition)`

### Parameters
#### Request Body
- **pFmHandle** (fmHandle_t) - Handle returned by `fmConnect()`.
- **pFmFabricPartition** (fmFabricPartitionList_t *) - A pointer to a structure that will be populated with the list of supported partition information upon success.

### Return Values
- **FM_ST_SUCCESS**: Successfully queried the list of supported partitions.
- **FM_ST_UNINITIALIZED**: The FM interface library has not been initialized.
- **FM_ST_BADPARAM**: Invalid input parameters.
- **FM_ST_GENERIC_ERROR**: An unspecified internal error occurred.
- **FM_ST_NOT_SUPPORTED**: The requested feature is not supported or enabled.
- **FM_ST_NOT_CONFIGURED**: Fabric Manager is initializing and no data is available.
- **FM_ST_VERSION_MISMATCH**: Provided versions of parameters do not match.
```

--------------------------------

### Get Supported GPU Fabric Partitions

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

Queries the list of supported static GPU fabric partitions in an NVSwitch-based system. It requires a handle from fmConnect() and a pointer to fmFabricPartitionList_t to populate the partition information. Success returns FM_ST_SUCCESS, while errors include uninitialized library, bad parameters, generic errors, unsupported features, or configuration issues.

```c
fmReturn_t fmGetSupportedFabricPartitions(fmHandle_t pFmHandle, fmFabricPartitionList_t *pFmFabricPartition)
```

--------------------------------

### Install NVIDIA Driver and NVLink on RHEL

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

Installs the NVIDIA driver and NVLink components for systems using fourth-generation NVSwitches on Red Hat Enterprise Linux 8 and 9. Utilizes dnf module installs and direct installs for specific driver branches.

```bash
sudo dnf module install nvidia-driver-<driver-branch>-open
sudo dnf install nvlink-<driver-branch>
```

```bash
sudo dnf module install nvidia-driver-570-open
sudo dnf install nvlink-570
```

--------------------------------

### Activate GPU Partition with Virtual Functions (C)

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

Activates an available GPU fabric partition with vGPU Virtual Functions (VFs). This function requires a valid handle, partition ID, a list of VFs, and the number of VFs. It must be called before starting a vGPU VM. Activation fails if MIG mode is enabled on the GPUs.

```c
fmReturn_t fmActivateFabricPartitionWithVFs(
    fmHandle_t pFmHandle,
    fmFabricPartitionId_t partitionId,
    fmPciDevice_t *vfList,
    unsigned int numVfs
);
```

--------------------------------

### Get Unsupported Fabric Partitions

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

Queries for all unsupported fabric partitions when Fabric Manager is running in Shared NVSwitch or vGPU multi-tenancy modes.

```APIDOC
## GET /fmGetUnsupportedFabricPartitions

### Description
Retrieves a list of fabric partitions that are unsupported by the Fabric Manager, particularly in multi-tenancy environments.

### Method
GET

### Endpoint
/fmGetUnsupportedFabricPartitions

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```json
{
  "example": "fmGetUnsupportedFabricPartitions(pHandle, pUnsuppList)"
}
```

### Response
#### Success Response (200)
- **pFmHandle** (fmHandle_t) - Handle returned by fmConnect()
- **pFmUnupportedFabricPartition** (fmUnsupportedFabricPartitionList_t) - List of unsupported fabric partitions on the system.

#### Response Example
```json
{
  "example": "FM_ST_SUCCESS: Successfully queried list of devices with failed NVLinks"
}
```

### Return Values
- **FM_ST_SUCCESS**: Successfully queried list of devices with failed NVLinks
- **FM_ST_UNINITIALIZED**: FM interface library has not been initialized.
- **FM_ST_BADPARAM**: Invalid input parameters
- **FM_ST_GENERIC_ERROR**: An unspecified internal error occurred
- **FM_ST_NOT_SUPPORTED**: Requested feature is not supported or enabled
- **FM_ST_NOT_CONFIGURED**: Fabric Manager is initializing and no data
- **FM_ST_VERSION_MISMATCH**: Provided versions of params do not match
```

--------------------------------

### Get Devices with Failed NVLinks (C)

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

Queries all GPUs and NVSwitches with failed NVLinks during Fabric Manager initialization. This API is not supported in specific restart modes. It returns a success status and a list of affected devices, or an empty list if NVLinks are trained at the hardware level.

```c
fmReturn_t fmGetNvlinkFailedDevices(
    fmHandle_t pFmHandle,
    fmNvlinkFailedDevices_t *pFmNvlinkFailedDevices
);
```

--------------------------------

### Get Unsupported Fabric Partitions (C)

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

Retrieves a list of unsupported fabric partitions when Fabric Manager is running in Shared NVSwitch or vGPU multi-tenancy modes. It requires a valid handle returned by fmConnect() and outputs a list of unsupported partitions. On newer systems like DGX H100, this API is expected to always return success with an empty list.

```c
fmReturn_tfmGetUnsupportedFabricPartitions(fmHandle_t pFmHandle,
 fmUnsupportedFabricPartitionList_t *pFmUnupportedFabricPartition)
```

--------------------------------

### Fabric Manager Startup Options and Usage

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

Displays the usage and available command-line options for the NVIDIA Fabric Manager executable (`nv-fabricmanager`). This helps in understanding how to configure FM at startup, including specifying configuration files and managing its operational mode.

```bash
$ nv-fabricmanager -h
    NVIDIA Fabric Manager
    Runs as a background process to configure the NVSwitches to form
    a single memory fabric among all participating GPUs.
    Usage: nv-fabricmanager [options]
    Options include:
    [-h | --help]:              Displays help information
    [-v | --version]:           Displays the Fabric Manager version and exit.
    [-c | --config]:            Provides Fabric Manager config file path/name which controls all the config options.
    [-r | --restart]:           Restart Fabric Manager after exit. Applicable to Shared NVSwitch and vGPU multitenancy modes.
    [-g | --fm-sm-mgmt-port-guid]: Fabric Manager and NVLink Subnet Manager management port GUID for Control Traffic.
    [-d | --database]: 		Provides Fabric Manager database engine

```

--------------------------------

### Connect to Fabric Manager Instance

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

Establishes a connection to a running Fabric Manager (FM) instance. It accepts connection parameters including IP address and port, or a Unix domain socket path. It returns a handle for subsequent API calls, with potential errors including connection failures, uninitialized library, invalid parameters, or version mismatches.

```c
fmReturn_t fmConnect(fmConnectParams_t *connectParams, fmHandle_t *pFmHandle)
```

--------------------------------

### Set Activated Fabric Partitions (API)

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

This API call is used by the hypervisor after restarting Fabric Manager to inform FM about the currently activated guest VM partitions. This is necessary because FM loses track of VM states when it is not running.

```c++
fmSetActivatedFabricPartitions(numPartitions, partitionList);
```

--------------------------------

### Fabric Manager Default Configuration File Location (Linux)

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

Indicates the default path for the NVIDIA Fabric Manager configuration file on Linux installations. This file contains most of the FM's configurable parameters and options. Changes to this file should be managed carefully, especially during package updates.

```bash
/usr/share/nvidia/nvswitch/fabricmanager.cfg
```

--------------------------------

### Set Activated Fabric Partitions List After Restart (C)

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

Sends a list of currently activated fabric partitions to Fabric Manager after it has been restarted. If no partitions are active upon restart, this function must be called with the number of partitions as zero. It requires a handle and a pointer to the list of activated partitions.

```c
fmReturn_t fmSetActivatedFabricPartitions(
    fmHandle_t pFmHandle,
    fmActivatedFabricPartitionList_t *pFmActivatedPartitionList
);
```

--------------------------------

### Connecting to the Running Fabric Manager Instance

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

Establishes a connection to a running instance of Fabric Manager (FM) to enable API communication.

```APIDOC
## fmConnect

### Description
Connects to a running instance of Fabric Manager (FM). This connection is used by the APIs to exchange information with the running FM instance.

### Method
`fmConnect(fmConnectParams_t *connectParams, fmHandle_t *pFmHandle)`

### Parameters
#### Request Body
- **connectParams** (fmConnectParams_t *) - Specifies connection parameters, including IP address and port. If `x.x.x.x` is used, the default port is attempted. If `x.x.x.x:yyyy` is used, port `yyyy` is attempted. For Unix domain sockets, fill the `addressInfo` member and set the `addressIsUnixSocket` flag.
- **pFmHandle** (fmHandle_t *) - A pointer to a handle that will be used for subsequent API calls.

### Return Values
- **FM_ST_SUCCESS**: Successfully connected to the FM instance.
- **FM_ST_CONNECTION_NOT_VALID**: The FM instance could not be reached.
- **FM_ST_UNINITIALIZED**: The FM interface library has not been initialized.
- **FM_ST_BADPARAM**: `pFmHandle` is NULL or the IP Address/format is invalid.
- **FM_ST_VERSION_MISMATCH**: Provided versions of parameters do not match.
```

--------------------------------

### C++ FM API: Manage Shared Fabric Partitions

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

This C++ code snippet demonstrates how to interact with the Fabric Manager (FM) API to perform operations on shared fabric partitions. It allows listing supported partitions, activating a partition, and deactivating a partition. The code requires the `nv_fm_agent.h` header and links against the FM library.

```c++
#include <iostream>
#include <string.h>


#include "nv_fm_agent.h"


int main(int argc, char **argv)
{
    fmReturn_t fmReturn;
    fmHandle_t fmHandle = NULL;
    char hostIpAddress[16] = {0};
    unsigned int operation = 0;
    fmFabricPartitionId_t partitionId = 0;
    fmFabricPartitionList_t partitionList = {0};


    std::cout << "Select Shared Fabric Partition Operation:\n";
    std::cout << "0 - List Supported Partition\n";
    std::cout << "1 – Activate a Partition\n";
    std::cout << "2 – Deactivate a Partition\n";
    std::cin >> operation;
    if ( operation > 2 ) {
        std::cout << "Invalid input.\n" << std::endl;
        return FM_ST_BADPARAM;
    }
    std::cout << std::endl;


    if ( operation > 0 ) {
        std::cout << "Input Shared Fabric Partition ID: \n";
        std::cin >> partitionId;


        if ( partitionId >= FM_MAX_FABRIC_PARTITIONS ) {
            std::cout << "Invalid partition ID." << std::endl;
            return FM_ST_BADPARAM;
        }
    }
    std::cout << std::endl;


    std::cout << "Please input an IP address to connect to. (Localhost = 127.0.0.1) \n";
    std::string buffer;
    std::cin >> buffer;
    if (buffer.length() > sizeof(hostIpAddress) – 1){
        std::cout << "Invalid IP address.\n" << std::endl;
        return FM_ST_BADPARAM;
    } else {
        buffer += '\0';
        strncpy(hostIpAddress, buffer.c_str(), 15);
    }


    /* Initialize Fabric Manager API interface library */
    fmReturn = fmLibInit();
    if (FM_ST_SUCCESS != fmReturn) {
        std::cout << "Failed to initialize Fabric Manager API interface library." << std::endl;
        return fmReturn;
    }


    /* Connect to Fabric Manager instance */
    fmConnectParams_t connectParams;
    strncpy(connectParams.addressInfo, hostIpAddress, sizeof(hostIpAddress));
    connectParams.timeoutMs = 1000; // in milliseconds
    connectParams.version = fmConnectParams_version;
    connectParams.addressIsUnixSocket = 0;
    fmReturn = fmConnect(&connectParams, &fmHandle);
    if (fmReturn != FM_ST_SUCCESS){
        std::cout << "Failed to connect to Fabric Manager instance." << std::endl;
        return fmReturn;
    }


    if ( operation == 0 ) {
        /* List supported partitions */
        partitionList.version = fmFabricPartitionList_version;
        fmReturn = fmGetSupportedFabricPartitions(fmHandle, &partitionList);
        if (fmReturn != FM_ST_SUCCESS) {
            std::cout << "Failed to get partition list. fmReturn: " << fmReturn << std::endl;
        } else {
            /* Only printing number of partitions for brevity */
            std::cout << "Total number of partitions supported: " << partitionList.numPartitions  << std::endl;
        }


    } else if ( operation == 1 ) {
        /* Activate a partition */
        fmReturn = fmActivateFabricPartition(fmHandle, partitionId);
        if (fmReturn != FM_ST_SUCCESS) {
            std::cout << "Failed to activate partition. fmReturn: " << fmReturn << std::endl;
        }


    } else if ( operation == 2 ) {
        /* Deactivate a partition */
        fmReturn = fmDeactivateFabricPartition(fmHandle, partitionId);
        if (fmReturn != FM_ST_SUCCESS) {
            std::cout << "Failed to deactivate partition. fmReturn: " << fmReturn << std::endl;
        }


    } else {
        std::cout << "Unknown operation." << std::endl;
    }


    /* Clean up */
    fmDisconnect(fmHandle);
    fmLibShutdown();
    return fmReturn;
}

```

--------------------------------

### List NVIDIA Devices on HGX H100 Baseboard (Shell)

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

This snippet shows how to display all NVIDIA devices on an HGX H100 GPU baseboard using the 'lspci' command in a shell. This output is useful for verifying the presence and type of GPUs and NVswitch components connected via PCIe.

```shell
$ lspci | grep -i nvidia
07:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
08:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
09:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
0a:00.0 Bridge: NVIDIA Corporation Device 22a3 (rev a1)
1b:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
43:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
52:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
61:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
9d:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
c3:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
d1:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
df:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)

```

--------------------------------

### Querying PCIe Devices on Linux

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

Demonstrates how to list PCIe devices, specifically filtering for NVIDIA and Mellanox devices using `lspci` on a Linux system. This command is useful for identifying network controllers and GPUs in the system.

```bash
$ lspci | grep -i -E 'nvidia|mella'
05:00.0 Infiniband controller: Mellanox Technologies MT2910 Family [ConnectX-7]
05:00.1 Infiniband controller: Mellanox Technologies MT2910 Family [ConnectX-7]
05:00.2 Infiniband controller: Mellanox Technologies MT2910 Family [ConnectX-7]
05:00.3 Infiniband controller: Mellanox Technologies MT2910 Family [ConnectX-7]
1b:00.0 3D controller: NVIDIA Corporation Device 29bc (rev a1)
43:00.0 3D controller: NVIDIA Corporation Device 29bc (rev a1)
52:00.0 3D controller: NVIDIA Corporation Device 29bc (rev a1)
61:00.0 3D controller: NVIDIA Corporation Device 29bc (rev a1)
9d:00.0 3D controller: NVIDIA Corporation Device 29bc (rev a1)
c3:00.0 3D controller: NVIDIA Corporation Device 29bc (rev a1)
d1:00.0 3D controller: NVIDIA Corporation Device 29bc (rev a1)
df:00.0 3D controller: NVIDIA Corporation Device 29bc (rev a1)

```

--------------------------------

### C++ Make File: Compile FM API Sample Code

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

This Makefile is designed to compile the C++ sample code that uses the Fabric Manager (FM) API. It specifies include paths (`IDIR`) and library paths (`LDIR`), and links the object file with the FM library (`-lnvfm`). The `clean` target removes compiled files.

```makefile
# Make file for the above sample assuming the source is saved into sampleCode.cpp
# Note: Change the default include paths (/usr/include & /usr/lib) based on FM API header files location.


IDIR := /usr/include
CXXFLAGS = -I $(IDIR)


LDIR := /usr/lib
LDFLAGS= -L$(LDIR) -lnvfm


sampleCode: sampleCode.o
	$(CXX) -o $@ $< $(CXXFLAGS) $(LDFLAGS)


clean:
	-@rm -f sameplCode.o
	-@rm -f sampleCode

```

--------------------------------

### List NVIDIA Devices on HGX A100 Baseboard (Shell)

Source: https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index

This snippet demonstrates how to list all NVIDIA devices present on an HGX A100 GPU baseboard using the 'lspci' command in a shell environment. It helps in identifying GPUs and NVswitch bridges connected via PCIe.

```shell
$ lspci | grep -i nvidia
36:00.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
3b:00.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
41:00.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
45:00.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
59:00.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
5d:00.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
63:00.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
67:00.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
6d:00.0 Bridge: NVIDIA Corporation Device 1af1 (rev a1)
6e:00.0 Bridge: NVIDIA Corporation Device 1af1 (rev a1)
6f:00.0 Bridge: NVIDIA Corporation Device 1af1 (rev a1)
70:00.0 Bridge: NVIDIA Corporation Device 1af1 (rev a1)
71:00.0 Bridge: NVIDIA Corporation Device 1af1 (rev a1)
72:00.0 Bridge: NVIDIA Corporation Device 1af1 (rev a1)

```
Tokens:
Raw