Libfabric

https://ofiwg.github.io/libfabric/
Admin
Libfabric (Open Fabrics Interfaces) is a communication library that defines APIs for...
Tokens:192,123
Snippets:2,122
Trust Score:9.7
Update:6 months ago
Show doc for...
Tokens:
Raw
### Example Peer CQ Setup

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_peer.3.html

Provides a step-by-step example of how to set up a peer CQ relationship between two providers.

```APIDOC
## Example Peer CQ Setup

The above description defines the generic mechanism for sharing CQs between providers. This section outlines one possible implementation to demonstrate the use of the APIs.

1.  Provider A is configured to use provider B as a peer. This may be coded into provider A or set through an environment variable.
2.  The application calls: `fi_cq_open(domain_a, attr, &cq_a, app_context)`
3.  Provider A allocates `cq_a` and automatically configures it to be used as a peer CQ.
4.  Provider A takes these steps:
    *   Allocate `peer_cq` and reference `cq_a`.
    *   Set `peer_cq_context->cq = peer_cq`.
    *   Set `attr_b.flags |= FI_PEER`.
    *   Call `fi_cq_open(domain_b, attr_b, &cq_b, peer_cq_context)`.
5.  Provider B allocates a `cq`, but configures it such that all completions are written to the `peer_cq`. The CQ ops to read from the CQ are set to `enosys` calls.
6.  Provider B inserts its own callbacks into the `peer_cq` object. It creates a reference between the `peer_cq` object and its own CQ.
```

--------------------------------

### Example Peer CQ Setup Steps

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_peer.3.html

Outlines a sequence of steps for setting up a peer CQ relationship between two providers (A and B), where provider A uses provider B as a peer.

```text
1. Provider A is configured to use provider B as a peer.  This may be coded
   into provider A or set through an environment variable.
2. The application calls:
   fi_cq_open(domain_a, attr, &cq_a, app_context)
3. Provider A allocates cq_a and automatically configures it to be used
   as a peer cq.
4. Provider A takes these steps:
   allocate peer_cq and reference cq_a
   set peer_cq_context->cq = peer_cq
   set attr_b.flags |= FI_PEER
   fi_cq_open(domain_b, attr_b, &cq_b, peer_cq_context)
5. Provider B allocates a cq, but configures it such that all completions
   are written to the peer_cq.  The cq ops to read from the cq are
   set to enosys calls.
6. Provider B inserts its own callbacks into the peer_cq object.  It
   creates a reference between the peer_cq object and its own cq.


```

--------------------------------

### Example Peer SRX Setup

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_peer.3.html

Illustrates a potential implementation for sharing SRXs between providers, demonstrating how provider A can use provider B as a peer for data transfers.

```APIDOC
## EXAMPLE PEER SRX SETUP

### Description
This section outlines one possible implementation to demonstrate the use of the APIs. In the example, provider A uses provider B as a peer for data transfers targeting endpoints on the local node.

### Steps
1.  **Provider Configuration**: Provider A is configured to use provider B as a peer. This may be coded into provider A or set through an environment variable.
2.  **Application Call**: The application calls `fi_srx_context(domain_a, attr, &srx_a, app_context)`.
3.  **SRX Allocation (Provider A)**: Provider A allocates `srx_a` and automatically configures it to be used as a peer SRX.
4.  **Peer SRX Setup (Provider A)**:
    *   Provider A allocates `peer_srx` and references `srx_a`.
    *   Sets `peer_srx_context->srx = peer_srx`.
    *   Sets `attr_b.flags |= FI_PEER`.
    *   Calls `fi_srx_context(domain_b, attr_b, &srx_b, peer_srx_context)`.
5.  **SRX Allocation (Provider B)**: Provider B allocates an SRX, configuring it such that all receive buffers are obtained from `peer_srx`. The SRX ops to post receives are set to `enosys` calls.
6.  **Callback Insertion (Provider B)**: Provider B inserts its own callbacks into the `peer_srx` object. It creates a reference between the `peer_srx` object and its own SRX.

### Method
N/A (Illustrative example)

### Endpoint
N/A

### Parameters
N/A

### Request Example
None

### Response
None

### Response Example
None
```

--------------------------------

### String Formatted Address Examples

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_getinfo.3.html

Provides examples of string-formatted addresses using different protocols and including query parameters.

```plaintext
fi_sockaddr://10.31.6.12:7471
```

```plaintext
fi_sockaddr_in6://[fe80::6:12]:7471
```

```plaintext
fi_sockaddr://10.31.6.12:7471?qos=3
```

--------------------------------

### Example CSV output

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_mon_sampler.1.html

Sample format of the generated CSV data.

```csv
mon_recv_0_64_c,mon_recv_0_64_s,mon_recv_64_512_c,mon_recv_64_512_s
0,0,0,0
22529,0,0,0
113664,0,0,0
```

--------------------------------

### Configure OPX with AV Table Mode

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_opx.7.html

Example command sequence to build the OPX provider with the address vector mode set to table.

```bash
OPX_AV=table ./configure  
make install
```

--------------------------------

### Example Usage of Version 2 GetInfo

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_usnic.7.html

An example demonstrating how to use the `getinfo` function to retrieve fabric information using version 2 of the usnic extension.

```APIDOC
## Example: Using Version 2 of the usnic GetInfo Extension

### Description
This example demonstrates how to utilize version 2 of the usnic "fabric getinfo" extension to query IP and SR-IOV characteristics of a usNIC device.

### Code Example
```c
#include <stdio.h>
#include <rdma/fabric.h>

/* The usNIC extensions are all in the
   rdma/fi_ext_usnic.h header */
#include <rdma/fi_ext_usnic.h>

int main(int argc, char *argv[]) {
    struct fi_info *info;
    struct fi_info *info_list;
    struct fi_info hints = {0};
    struct fi_ep_attr ep_attr = {0};
    struct fi_fabric_attr fabric_attr = {0};

    fabric_attr.prov_name = "usnic";
    ep_attr.type = FI_EP_DGRAM;

    hints.caps = FI_MSG;
    hints.mode = FI_LOCAL_MR | FI_MSG_PREFIX;
    hints.addr_format = FI_SOCKADDR;
    hints.ep_attr = &ep_attr;
    hints.fabric_attr = &fabric_attr;

    /* Find all usnic providers */
    fi_getinfo(FI_VERSION(1, 0), NULL, 0, 0, &hints, &info_list);

    for (info = info_list; NULL != info; info = info->next) {
        /* Open the fabric on the interface */
        struct fid_fabric *fabric;
        fi_fabric(info->fabric_attr, &fabric, NULL);

        /* Pass FI_USNIC_FABRIC_OPS_1 to get usnic ops
           on the fabric */
        struct fi_usnic_ops_fabric *usnic_fabric_ops;
        fi_open_ops(&fabric->fid, FI_USNIC_FABRIC_OPS_1, 0,
                (void **) &usnic_fabric_ops, NULL);

        /* Now use the returned usnic ops structure to call
           usnic extensions.  The following extension queries
           some IP and SR-IOV characteristics about the
           usNIC device. */
        struct fi_usnic_info usnic_info;

        /* Explicitly request version 2. */
        usnic_fabric_ops->getinfo(2, fabric, &usnic_info);

        printf("Fabric interface %s is %s:\n"
               "\tNetmask:  0x%08x\n\tLink speed: %d\n"
               "\tSR-IOV VFs: %d\n\tQPs per SR-IOV VF: %d\n"
               "\tCQs per SR-IOV VF: %d\n",
               info->fabric_attr->name,
               usnic_info.ui.v2.ui_ifname,
               usnic_info.ui.v2.ui_netmask_be,
               usnic_info.ui.v2.ui_link_speed,
               usnic_info.ui.v2.ui_num_vf,
               usnic_info.ui.v2.ui_qp_per_vf,
               usnic_info.ui.v2.ui_cq_per_vf);

        fi_close(&fabric->fid);
    }

    return 0;
}
```
```

--------------------------------

### Listing Environment Variables via Command Line

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_setup.7.html

Example output of the fi_info command used to list available environment variables.

```bash
$ fi_info -e
# FI_LOG_LEVEL: String
# Specify logging level: warn, trace, info, debug (default: warn)

# FI_LOG_PROV: String
# Specify specific provider to log (default: all)
```

--------------------------------

### Run fi_rdm_atomic test with options

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fabtests.7.html

Example of running the fi_rdm_atomic test with specific options for provider, iterations, message size, and server address.

```bash
run server: fi_rdm_atomic -p psm3 -s 192.168.0.123 -I 1000 -S 1024
run client: fi_rdm_atomic 192.168.0.123 -p psm3 -I 1000 -S 1024
```

--------------------------------

### Start RDM Pingpong Benchmark Server

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_cxi.7.html

Initiate the `fi_rdm_tagged_pingpong` benchmark server using the CXI provider, specifying the source PID and interface.

```bash
./fabtests/benchmarks/fi_rdm_tagged_pingpong -B 10 -s cxi0

```

--------------------------------

### FI_PROVIDER_PATH Configuration Variants

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fabric.7.html

Examples of setting FI_PROVIDER_PATH with various combinations of default paths, forced activation, and specific library files.

```bash
FI_PROVIDER_PATH=
FI_PROVIDER_PATH=@
FI_PROVIDER_PATH=+/opt/libfabric/libtcp-fi.so
FI_PROVIDER_PATH=@+/opt/libfabric/libtcp-fi.so
```

--------------------------------

### Run libfabric test as client

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fabtests.7.html

Example of running a libfabric test as a client, specifying the server address and provider.

```bash
run client: <test_name> <server_addr> -p <provider_name>
	e.g.	fi_msg_rma 192.168.0.123 -p sockets
```

--------------------------------

### Querying usNIC Version 2 Extension

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_usnic.7.html

Example demonstrating how to initialize a fabric and query device characteristics using the version 2 usNIC extension.

```c
#include <stdio.h>
#include <rdma/fabric.h>

/* The usNIC extensions are all in the
   rdma/fi_ext_usnic.h header */
#include <rdma/fi_ext_usnic.h>

int main(int argc, char *argv[]) {
    struct fi_info *info;
    struct fi_info *info_list;
    struct fi_info hints = {0};
    struct fi_ep_attr ep_attr = {0};
    struct fi_fabric_attr fabric_attr = {0};

    fabric_attr.prov_name = "usnic";
    ep_attr.type = FI_EP_DGRAM;

    hints.caps = FI_MSG;
    hints.mode = FI_LOCAL_MR | FI_MSG_PREFIX;
    hints.addr_format = FI_SOCKADDR;
    hints.ep_attr = &ep_attr;
    hints.fabric_attr = &fabric_attr;

    /* Find all usnic providers */
    fi_getinfo(FI_VERSION(1, 0), NULL, 0, 0, &hints, &info_list);

    for (info = info_list; NULL != info; info = info->next) {
        /* Open the fabric on the interface */
        struct fid_fabric *fabric;
        fi_fabric(info->fabric_attr, &fabric, NULL);

        /* Pass FI_USNIC_FABRIC_OPS_1 to get usnic ops
           on the fabric */
        struct fi_usnic_ops_fabric *usnic_fabric_ops;
        fi_open_ops(&fabric->fid, FI_USNIC_FABRIC_OPS_1, 0,
                (void **) &usnic_fabric_ops, NULL);

        /* Now use the returned usnic ops structure to call
           usnic extensions.  The following extension queries
           some IP and SR-IOV characteristics about the
           usNIC device. */
        struct fi_usnic_info usnic_info;

        /* Explicitly request version 2. */
        usnic_fabric_ops->getinfo(2, fabric, &usnic_info);

        printf("Fabric interface %s is %s:\n"
               "\tNetmask:  0x%08x\n\tLink speed: %d\n"
               "\tSR-IOV VFs: %d\n\tQPs per SR-IOV VF: %d\n"
               "\tCQs per SR-IOV VF: %d\n",
               info->fabric_attr->name,
               usnic_info.ui.v2.ui_ifname,
               usnic_info.ui.v2.ui_netmask_be,
               usnic_info.ui.v2.ui_link_speed,
               usnic_info.ui.v2.ui_num_vf,
               usnic_info.ui.v2.ui_qp_per_vf,
               usnic_info.ui.v2.ui_cq_per_vf);

        fi_close(&fabric->fid);
    }
```

--------------------------------

### Run libfabric test as server

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fabtests.7.html

Example of running a libfabric test as a server, specifying the provider and source address.

```bash
run server: <test_name> -p <provider_name> -s <source_addr>
	e.g.	fi_msg_rma -p sockets -s 192.168.0.123
```

--------------------------------

### Create and Bind Passive Endpoint

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_setup.7.html

Example of creating a passive endpoint and binding it to an event queue before listening for connections.

```c
/* Example passive endpoint listen */
fi_passive_ep(fabric, info, &pep, NULL);

fi_eq_open(fabric, &eq_attr, &eq, NULL);
fi_pep_bind(pep, &eq->fid, 0);

fi_listen(pep);

```

--------------------------------

### Start RDM Pingpong Benchmark Client

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_cxi.7.html

Launch the `fi_rdm_tagged_pingpong` benchmark client, providing the server PID and NIC address. The client can optionally be bound to a specific interface.

```bash
# Start client by specifying server PID and NIC address
./fabtests/benchmarks/fi_rdm_tagged_pingpong -P 10 $CXI0_ADDR

# The client may be bound to a specific interface, like:
./fabtests/benchmarks/fi_rdm_tagged_pingpong -B 10 -s cxi1 -P 10 $CXI0_ADDR

```

--------------------------------

### Enable Unconnected Endpoint (fi_enable)

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_setup.7.html

Example demonstrating how to enable an unconnected endpoint. This involves allocating and binding an address vector and a completion queue before calling `fi_enable()`.

```c
/* Example to enable an unconnected endpoint */

/* Allocate an address vector and associated it with the endpoint */
fi_av_open(domain, &av_attr, &av, NULL);
fi_ep_bind(ep, &av->fid, 0);

/* Allocate and associate completion queues with the endpoint */
fi_cq_open(domain, &cq_attr, &cq, NULL);
fi_ep_bind(ep, &cq->fid, FI_TRANSMIT | FI_RECV);

fi_enable(ep);

```

--------------------------------

### Example: fi_mr_desc() with fi_send()

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_setup.7.html

Demonstrates how to use the memory descriptor obtained from fi_mr_desc() as an argument to the fi_send() function for data transfer.

```c
/* fi_mr_desc() example using fi_send() */
fi_send(ep, buf, len, fi_mr_desc(mr), 0, NULL);
```

--------------------------------

### Run multinode tests

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fabtests.7.html

Invokes the multinode test utility. A server process must be started before client processes.

```bash
fi_multinode -n <number of processes> -s <server_addr> -C <mode>
```

--------------------------------

### FI_PEER_TRANSFER Setup using fi_endpoint

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_peer.3.html

This section details how to set up peer transfers using the fi_endpoint API, including the necessary data structures and parameters.

```APIDOC
## POST /fi_endpoint (with FI_PEER_TRANSFER flag)

### Description
Sets up peer transfers by opening a peer transfer endpoint and accepting transfers with the FI_PEER_TRANSFER flag set. This involves using the `fi_endpoint` API with a `fi_peer_transfer_context`.

### Method
POST (Implicitly via `fi_endpoint` call)

### Endpoint
`fi_endpoint()`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
**context** (`struct fi_peer_transfer_context*`) - Required - A pointer to a `fi_peer_transfer_context` structure used to establish a virtual link between endpoints on peer providers.

**info** (`struct fi_info*`) - Optional - Defines the attributes of the main provider's objects, reporting capabilities and restrictions on peer transfers. The importing provider must fail the call if it cannot meet these restrictions.

### Request Example
```c
struct fi_peer_transfer_context context;
struct fi_info *info = NULL; // Or populated with desired attributes

// Populate context and info as needed

// Call fi_endpoint with FI_PEER_TRANSFER flag set in info->mode
ret = fi_endpoint(domain, &info, &ep, &context, FI_PEER_TRANSFER);
```

### Response
#### Success Response (0)
Returns `FI_SUCCESS` on successful setup of the peer transfer endpoint.

#### Response Example
```
FI_SUCCESS (0)
```

### Error Handling
On error, a negative value corresponding to fabric errno is returned. Fabric errno values are defined in `rdma/fi_errno.h`.
```

--------------------------------

### Select HFI for OPX Contexts (Fixed Unit)

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_opx.7.html

Force OPX provider to use a specific HFI unit when opening a context. Example: `FI_OPX_HFI_SELECT=0`.

```bash
FI_OPX_HFI_SELECT=0
```

```bash
FI_OPX_HFI_SELECT=1
```

--------------------------------

### Accept New Connection

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_setup.7.html

Example of accepting a new connection request on a passive endpoint, including allocating a new active endpoint and setting up necessary resources.

```c
/* Example accepting a new connection */

/* Wait for a CONNREQ event */
fi_eq_sread(eq, &event, &cm_entry, sizeof cm_entry, -1, 0);
assert(event == FI_CONNREQ);

/* Allocate a new endpoint for the connection */
if (!cm_entry.info->domain_attr->domain)
    fi_domain(fabric, cm_entry.info, &domain, NULL);
fi_endpoint(domain, cm_entry.info, &ep, NULL);

fi_ep_bind(ep, &eq->fid, 0);
fi_cq_open(domain, &cq_attr, &cq, NULL);
fi_ep_bind(ep, &cq->fid, FI_TRANSMIT | FI_RECV);

fi_enable(ep);
fi_recv(ep, rx_buf, len, NULL, 0, NULL);

fi_accept(ep, NULL, 0);
fi_eq_sread(eq, &event, &cm_entry, sizeof cm_entry, -1, 0);
assert(event == FI_CONNECTED);

```

--------------------------------

### Atomic Write Operation Example

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_atomic.3.html

This pseudo-code demonstrates a 64-bit unsigned atomic write operation using fi_atomic. Ensure the data type and operation are valid for the provider and the count is within the supported range.

```c
fi_atomic(ep, buf, count, NULL, dest_addr, addr, key,
	  FI_UINT64, FI_ATOMIC_WRITE, context)
{
	for (i = 1; i < count; i ++)
		ATOMIC_WRITE_U64(((uint64_t *) addr)[i],
				 ((uint64_t *) buf)[i]);
}
```

--------------------------------

### Launch the sampler

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_mon_sampler.1.html

Execute the sampler to watch and export data to a specified directory.

```bash
fi_mon_sampler -o $HOME -w 1000 -f csv /dev/shm/ofi/$UID/$HOSTNAME
```

--------------------------------

### Initialize Provider and Manage Parameters

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_provider.3.html

Defines the entry point for external providers and provides functions to register and retrieve configuration parameters.

```c
#include <rdma/fabric.h>
#include <rdma/prov/fi_prov.h>

struct fi_provider* fi_prov_ini(void);

int fi_param_define(const struct fi_provider *provider, const char *param_name,
	enum fi_param_type type, const char *help_string_fmt, ...);

int fi_param_get_str(struct fi_provider *provider, const char *param_name,
	char **value);

int fi_param_get_int(struct fi_provider *provider, const char *param_name,
	int *value);

int fi_param_get_bool(struct fi_provider *provider, const char *param_name,
	int *value);

int fi_param_get_size_t(struct fi_provider *provider, const char *param_name,
	size_t *value);
```

--------------------------------

### CXI Domain Operations Structure

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_cxi.7.html

Defines the structure for CXI domain operations, including methods for reading counters, topology, enabling hybrid MR descriptors, getting unexpected messages, and getting DWQ depth.

```c
struct fi_cxi_dom_ops {
	int (*cntr_read)(struct fid *fid, unsigned int cntr, uint64_t *value,
		      struct timespec *ts);
	int (*topology)(struct fid *fid, unsigned int *group_id,
				unsigned int *switch_id, unsigned int *port_id);
	int (*enable_hybrid_mr_desc)(struct fid *fid, bool enable);
	size_t (*ep_get_unexp_msgs)(struct fid_ep *fid_ep,
					struct fi_cq_tagged_entry *entry,
					size_t count, fi_addr_t *src_addr,
					size_t *ux_count);
	int (*get_dwq_depth)(struct fid *fid, size_t *depth);
};

```

--------------------------------

### Endpoint Options

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_endpoint.3.html

Functions to get or set options for an endpoint.

```APIDOC
## Endpoint Options

### Description
Functions to get or set options for an endpoint.

### Endpoints
- `fi_getopt`
- `fi_setopt`

### Arguments
- `ep` (struct fid *): The fabric endpoint.
- `level` (int): Protocol level at which the desired option resides.
- `optname` (int): The protocol option to read or set.
- `optval` (void *): The option value buffer.
- `optlen` (size_t *): The size of the optval buffer.
```

--------------------------------

### Initiate Server Listening

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_intro.7.html

Sets up a non-blocking listening socket on port 7471 using getaddrinfo for address resolution.

```C
/* Example server code flow to initiate listen */
struct addrinfo *ai, hints;
int listen_fd;

memset(&hints, 0, sizeof hints);
hints.ai_socktype = SOCK_STREAM;
hints.ai_flags = AI_PASSIVE;
getaddrinfo(NULL, "7471", &hints, &ai);

listen_fd = socket(ai->ai_family, SOCK_STREAM, 0);
bind(listen_fd, ai->ai_addr, ai->ai_addrlen);
freeaddrinfo(ai);

fcntl(listen_fd, F_SETFL, O_NONBLOCK);
listen(listen_fd, 128);
```

--------------------------------

### GET /fi_getinfo/fabric_attributes

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_getinfo.3.html

Retrieves or specifies fabric attributes using the fi_getinfo interface.

```APIDOC
## GET /fi_getinfo/fabric_attributes

### Description
Retrieves or specifies fabric attributes. When provided as hints, requested values of struct fi_fabric_attr should be set. On output, the actual fabric attributes that can be provided are returned.

### Method
GET

### Endpoint
/fi_getinfo/fabric_attributes

### Parameters
#### Request Body
- **fi_fabric_attr** (struct) - Optional - Fabric attributes to be used as hints for the provider.
```

--------------------------------

### GET fi_eq_read

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_eq.3.html

Performs a non-blocking read of event data from the Event Queue (EQ).

```APIDOC
## GET fi_eq_read

### Description
Performs a non-blocking read of event data from the EQ. At most one event is returned per read operation.

### Method
GET

### Endpoint
fi_eq_read

### Parameters
#### Query Parameters
- **FI_PEEK** (flag) - Optional - If set, reads event data without consuming it from the EQ.

### Response
#### Success Response (200)
- **struct fi_eq_entry** (struct) - Header for all events.
- **struct fi_eq_cm_entry** (struct) - Structure for connection management events.

#### Response Example
struct fi_eq_entry {
	fid_t            fid;
	void            *context;
	uint64_t         data;
};
```

--------------------------------

### GET /fi_tx_context

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_endpoint.3.html

Retrieves a specific transmit context associated with a scalable endpoint, identified by an index.

```APIDOC
## GET /fi_tx_context

### Description
The fi_tx_context call is used to retrieve a specific transmit context, identified by an index. Transmit contexts are independent transmit queues that allow multi-threaded applications to access independent queues.

### Method
GET

### Endpoint
/fi_tx_context

### Parameters
#### Query Parameters
- **index** (integer) - Required - The index identifying the specific transmit context.
- **attr** (object) - Optional - Context specific attributes requested by the application.

### Response
#### Success Response (200)
- **attr** (object) - The actual attributes assigned to the context by the provider.
```

--------------------------------

### Synopsis for fi_mon_sampler

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_mon_sampler.1.html

Basic command-line syntax for invoking the sampler.

```bash
 fi_mon_sampler [OPTIONS] <target>		sample from file(s) at <target>
```

--------------------------------

### GET /fi_getinfo

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_getinfo.3.html

Retrieves fabric information based on provided node, service, and control flags.

```APIDOC
## GET /fi_getinfo

### Description
Retrieves information about available fabric providers and their attributes. The operation can be controlled using specific input flags.

### Method
GET

### Endpoint
/fi_getinfo

### Parameters
#### Query Parameters
- **node** (string) - Optional - The node address to resolve.
- **service** (string) - Optional - The service or port to resolve.
- **flags** (enum) - Optional - Control flags including FI_NUMERICHOST, FI_PROV_ATTR_ONLY, FI_SOURCE, and FI_RESCAN.
```

--------------------------------

### GET /fi_getinfo/nic_attributes

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_getinfo.3.html

Retrieves optional attributes related to the hardware NIC associated with the fabric, domain, and endpoint.

```APIDOC
## GET /fi_getinfo/nic_attributes

### Description
Retrieves optional attributes related to the hardware NIC associated with the specified fabric, domain, and endpoint data. This field is only valid for providers where the corresponding attributes are closely associated with a hardware NIC.

### Method
GET

### Endpoint
/fi_getinfo/nic_attributes

### Parameters
#### Query Parameters
- **_nic** (object) - Optional - Network interface details associated with the fabric.
```

--------------------------------

### Provider Initialization and Parameters

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_provider.3.html

Functions for external provider entry points and managing configuration parameters via environment variables.

```APIDOC
## fi_prov_ini

### Description
Entry point for external providers to return their fi_provider structure to the libfabric core.

### Method
Function Call

### Parameters
- None

## fi_param_define

### Description
Registers a configuration parameter for a provider.

### Parameters
- **provider** (struct fi_provider*) - Required - Reference to the provider.
- **param_name** (const char*) - Required - Name of the parameter.
- **type** (enum fi_param_type) - Required - Type of the parameter.
- **help_string_fmt** (const char*) - Required - Help string format.

## fi_param_get

### Description
Retrieves the value of a previously defined configuration parameter from the environment.

### Parameters
- **provider** (struct fi_provider*) - Required - Reference to the provider.
- **param_name** (const char*) - Required - Name of the parameter.
- **value** (void*) - Required - Pointer to store the retrieved value.
```

--------------------------------

### Enable SELinux for OPX

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_opx.7.html

Enable SELinux support for OPX, which updates the Jkey based on system settings. Defaults to 'No'.

```bash
_FI_OPX_SELINUX_=1
```

--------------------------------

### Configure FI_PROVIDER_PATH Search Directories

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fabric.7.html

Define a colon-separated list of directories to search for dynamically loadable providers.

```bash
FI_PROVIDER_PATH=/opt/libfabric:/opt/libfabric2
```

--------------------------------

### FI_MR_RAW

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_mr.3.html

Supports providers with keys larger than 64-bits or requiring peer setup. Requires `fi_mr_raw_attr()` and `fi_mr_map_raw()` for memory region targeting.

```APIDOC
## FI_MR_RAW

### Description
This mode is used to support providers that handle keys larger than 64-bits or require setup at the peer. When `FI_MR_RAW` is set, applications must use `fi_mr_raw_attr()` locally and `fi_mr_map_raw()` at the peer before targeting a memory region for data transfer.

### Requirements
- Use `fi_mr_raw_attr()` locally.
- Use `fi_mr_map_raw()` at the peer.
```

--------------------------------

### cq_open_ext

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_efa.7.html

Creates a completion queue with external memory provided via dmabuf.

```APIDOC
## cq_open_ext

### Description
This op creates a completion queue with external memory provided via dmabuf. The memory can be passed by supplying the following struct.

```c
struct fi_efa_cq_init_attr {
	uint64_t flags;
	struct {
		uint8_t *buffer;
		uint64_t length;
		uint64_t offset;
		uint32_t fd;
	} ext_mem_dmabuf;
};
```

### Parameters
#### Request Body
- **flags** (uint64_t) - Required - A bitwise OR of the various values described below. FI_EFA_CQ_INIT_FLAGS_EXT_MEM_DMABUF: create CQ with external memory provided via dmabuf.
- **ext_mem_dmabuf** (struct) - Required - Structure containing information about external memory when using FI_EFA_CQ_INIT_FLAGS_EXT_MEM_DMABUF flag.
  - **buffer** (uint8_t *) - Optional - Pointer to the memory mapped in the process’s virtual address space. The field is optional, but if not provided, the use of CQ poll interfaces should be avoided.
  - **length** (uint64_t) - Required - Length of the memory region to use.
  - **offset** (uint64_t) - Required - Offset within the dmabuf.
  - **fd** (uint32_t) - Required - File descriptor of the dmabuf.

### Return Value
- **cq_open_ext()** returns 0 on success, or the value of errno on failure (which indicates the failure reason).
```

--------------------------------

### Initiate Client Connection

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_intro.7.html

Configures a non-blocking client socket and initiates a connection to a specified server address.

```C
/* Example client code flow to start connection */
struct addrinfo *ai, hints;
int client_fd;

memset(&hints, 0, sizeof hints);
hints.ai_socktype = SOCK_STREAM;
getaddrinfo("10.31.20.04", "7471", &hints, &ai);

client_fd = socket(ai->ai_family, SOCK_STREAM, 0);
fcntl(client_fd, F_SETFL, O_NONBLOCK);

connect(client_fd, ai->ai_addr, ai->ai_addrlen);
freeaddrinfo(ai);
```

--------------------------------

### Retrieve and parse Libfabric version

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_version.3.html

Use fi_version to get the current interface version and the FI_MAJOR/FI_MINOR macros to extract specific version components.

```c
#include <rdma/fabric.h>

uint32_t fi_version(void);

FI_MAJOR(version)

FI_MINOR(version)
```

--------------------------------

### Establish Client Connection

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_intro.7.html

Uses poll() to wait for connection completion and getsockopt() to verify success.

```c
/* Example client code flow to establish a connection */
struct pollfd fds;
int err;
socklen_t len;

fds.fd = client_fd;
fds.events = POLLOUT;

poll(&fds, -1);

len = sizeof err;
getsockopt(client_fd, SOL_SOCKET, SO_ERROR, &err, &len);
```

--------------------------------

### CXI Counter Operations Structure

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_cxi.7.html

Defines the structure for CXI counter operations, including methods to set the writeback buffer and get the MMIO address.

```c
struct fi_cxi_cntr_ops {
	/* Set the counter writeback address to a client provided address. */
	int (*set_wb_buffer)(struct fid *fid, const void *buf, size_t len);

	/* Get the counter MMIO region. */
	int (*get_mmio_addr)(struct fid *fid, void **addr, size_t *len);
};

```

--------------------------------

### Launch Libfabric application with monitor

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_mon_sampler.1.html

Enable the ofi_hook_monitor provider for a libfabric application.

```bash
FI_HOOK=monitor fi_pingpong [OPTIONS]
```

--------------------------------

### Force First Provider Activation

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fabric.7.html

Use the @ prefix to force the first discovered provider to be active, ignoring version numbers.

```bash
FI_PROVIDER_PATH=@/opt/libfabric:/opt/libfabric2
```

--------------------------------

### Base FID Definition

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_setup.7.html

Defines the base FID structure and related enums and operation structures. All fabric interface descriptors must start with this structure.

```c
/* Base FID definition */
enum {
    FI_CLASS_UNSPEC,
    FI_CLASS_FABRIC,
    FI_CLASS_DOMAIN,
    ...
};

struct fi_ops {
    size_t size;
    int (*close)(struct fid *fid);
    ...
};

/* All fabric interface descriptors must start with this structure */
struct fid {
    size_t fclass;
    void *context;
    struct fi_ops *ops;
};
```

--------------------------------

### Using fi_trywait for Native Wait Objects

Source: https://ofiwg.github.io/libfabric/v2.5.0/man/fi_setup.7.html

Demonstrates how to safely block on a native file descriptor by checking for race conditions with fi_trywait.

```c
/* Get the native wait object -- an fd in this case */
fi_control(&cq->fid, FI_GETWAIT, (void *) &fd);
FD_ZERO(&fds);
FD_SET(fd, &fds);

while (1) {
    ret = fi_trywait(fabric, &cq->fid, 1);
    if (ret == FI_SUCCESS) {
        /* It’s safe to block on the fd */
        select(fd + 1, &fds, NULL, &fds, &timeout);
    } else if (ret == -FI_EAGAIN) {
        /* Read and process all completions from the CQ */
        do {
            ret = fi_cq_read(cq, &comp, 1);
        } while (ret > 0);
    } else {
        /* something really bad happened */
    }
}
```
Tokens:
Raw