AMD TCPDirect

https://github.com/xilinx-cns/tcpdirect
Admin
Unable to determine project description due to file access restrictions. The agent was blocked from...
Tokens:14,090
Snippets:178
Trust Score:7.5
Update:4 months ago
Show doc for...
Tokens:
Raw
### Building TCPDirect Example Applications

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/030_apps.dox

Navigate to the scripts directory, set the PATH, then go to the build directory to clean and make the example applications.

```shell
cd openonload/scripts/
 export PATH="$PWD:$PATH"
 cd ../build/gnu_x86_64/tests/zf_apps/
 make clean
 make
```

--------------------------------

### Get Free Space for Alternatives

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Demonstrates how to determine the maximum packet size that can be queued on an alternative using zf_alternatives_free_space().

```c
fs = zf_alternatives_free_space(ts, queue_ab);
```

--------------------------------

### Install libstdc++-static

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/DEVELOPING.md

Installs the static version of the libstdc++ library, which is a build dependency.

```bash
sudo yum install libstdc++-static
```

--------------------------------

### Example C Code Snippet

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/010_overview.dox

This is a placeholder for a C code example. The actual code content is missing from the provided text.

```c

```

--------------------------------

### C Style Comment and Code Block Example

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/CONTRIBUTING.md

Illustrates the C-style comment and code block formatting conventions used in the project, including indentation and brace placement.

```c
/* This is a comment */
if( ! conditional_expr ) {
  statement1;
  statement2;
}
```

--------------------------------

### Allocating and Queuing Alternative Packets

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

This example demonstrates allocating alternative queues and queuing initial packet data for potential updates A, B, or both. It shows how to prepare multiple update scenarios in advance.

```c
zf_alternatives_alloc(ts, attr, &queue_a);
zft_alternatives_queue(ts, queue_a, <UpdateA_data>, flags);
zf_alternatives_alloc(ts, attr, &queue_b);
zft_alternatives_queue(ts, queue_b, <UpdateB_data>, flags);
zf_alternatives_alloc(ts, attr, &queue_ab);
zft_alternatives_queue(ts, queue_ab, <UpdateA_data>, flags);
zft_alternatives_queue(ts, queue_ab, <UpdateB_data>, flags);
```

--------------------------------

### Install perl-Test-Harness

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/DEVELOPING.md

Installs the perl-Test-Harness package, which is a dependency for running unit tests.

```bash
$ sudo yum install perl-Test-Harness
```

--------------------------------

### Debian Package Build Instructions

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/DEVELOPING.md

Steps to build a Debian package, including extracting the source tarball, unpacking the source archive, entering the directory, building the package with debuild, and installing the resulting deb file.

```bash
# 1. Extract debian-source tarball:
tar -xf tcpdirect_<version>-debiansource.tgz
# 2. Unpack debian source archive
dpkg-source -x tcpdirect_<version>-1.dsc
# 3. Enter newly created tcpdirect directory
cd tcpdirect-<version>
# 4. Build debian package
debuild -i -uc -us
# 5. Install debian package
dpkg -i ../tcpdirect_<version>-1_amd64.deb
```

--------------------------------

### Sending and Canceling Alternative Queues

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

This example shows how to send packets from a selected alternative queue and cancel others to ensure correct TCP headers. It highlights the process of finalizing a send operation.

```c
zf_alternatives_send(ts, queue_a);
zf_alternatives_cancel(ts, queue_b);
zf_alternatives_cancel(ts, queue_ab);
```

--------------------------------

### Overlapped UDP Receive Example

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Example demonstrating how to set up and use overlapped UDP receives. This involves adding a zocket to a muxer with the ZF_EPOLLIN_OVERLAPPED event and then processing received data using zfur_zc_recv with ZF_OVERLAPPED_WAIT.

```c
 const struct epoll_event in_event_ovl  = {
   .events = ZF_EPOLLIN_OVERLAPPED | EPOLLIN,
   .data = { .ptr = udp_socket },
 };
rc = zf_muxer_add(muxer, zft_to_waitable(udp_socket), &in_event_ovl);
if( rc < 0 )
  return rc;

while( 1 ) { // event loop
  rc = zf_muxer_wait(muxer, events, maxevents, 1000000);
  if( rc == 1 && events[0].events == ZF_EPOLLIN_OVERLAPPED ) {

    // ensure arrival of n bytes of data (optional step)
    struct zfur* us = events[0].ptr;
    msg.iovcnt = 1;
    msg.iov[0].iov_len = n;
    zfur_zc_recv(us, msg, ZF_OVERLAPPED_WAIT);
    if( msg.iovcnt == 0 )
      continue; // overlapped wait has been abandoned

    result = compute_result(msg.iov[0]);

    // wait for packet completion and verification
```

--------------------------------

### Requeuing and Releasing Alternative Queues

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

This example demonstrates updating packet data by requeuing on a new alternative and releasing the old one. It's used when packet data needs modification after initial queuing, avoiding in-place edits.

```c
zf_alternatives_alloc(ts, attr, &queue_new_ab);
zft_alternatives_queue(ts, queue_new_ab, <UpdateA_edited_data>, flags);
zft_alternatives_queue(ts, queue_new_ab, <UpdateB_data>, flags);
zf_alternatives_release(ts, queue_ab);
```

--------------------------------

### Include TCPDirect Header File

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Applications using TCPDirect should include the 'zf.h' header file. This is typically installed in the system include directory.

```c
#include <zf/zf.h>

```

--------------------------------

### UDP Ping Pong Example

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/050_examples.dox

Demonstrates a low-latency UDP ping pong application. It requires setting the 'interface' attribute via ZF_ATTR environment variable. Boilerplate code is omitted for clarity; a full version is available in `src/tests/zf_apps/zfudppingpong.c`.

```c
#define SIZE 12
#define ITERATIONS 1000000
int ping;


static void ping_pongs(struct zf_stack* stack, struct zfur* ur, struct zfut* ut)
{
  char send_buf[SIZE];
  int sends_left = ITERATIONS;
  int recvs_left = ITERATIONS;

  struct {
    /* The iovec used by zfur_msg must be immediately afterwards. */
    struct zfur_msg msg;
    struct iovec iov[2];
  } msg;
  const int max_iov = sizeof(msg.iov) / sizeof(msg.iov[0]);

  if( ping ) {
    ZF_TEST(zfut_send_single(ut, send_buf, SIZE) == SIZE);
    --sends_left;
  }

  do {
    /* Poll the stack until something happens. */
    while( zf_reactor_perform(stack) == 0 )
      ;
    msg.msg.iovcnt = max_iov;
    zfur_zc_recv(ur, &msg.msg, 0);
    if( msg.msg.iovcnt ) {
      if( sends_left ) {
        ZF_TEST(zfut_send_single(ut, send_buf, SIZE) == SIZE);
        --sends_left;
      }
      zfur_zc_recv_done(ur, &msg.msg);
      --recvs_left;
    }
  } while( recvs_left );
}


int main(int argc, char* argv[])
{
  if( argc != 3 )
    usage_err();

  if( ! strcmp(argv[0], "ping") )
    ping = true;
  else if( ! strcmp(argv[0], "pong") )
    ping = false;
  else
    usage_err();

  struct addrinfo *ai_local, *ai_remote;
  if( getaddrinfo_hostport(argv[1], NULL, &ai_local) != 0 ) {
    fprintf(stderr, "ERROR: failed to lookup address '%s'\n", argv[1]);
    exit(2);
  }
  if( getaddrinfo_hostport(argv[2], NULL, &ai_remote) != 0 ) {
    fprintf(stderr, "ERROR: failed to lookup address '%s'\n", argv[2]);
    exit(2);
  }

  /* Initialise the TCPDirect library and allocate a stack. */
  ZF_TRY(zf_init());

  struct zf_attr* attr;
  ZF_TRY(zf_attr_alloc(&attr));

  struct zf_stack* stack;
  ZF_TRY(zf_stack_alloc(attr, &stack));

  /* Allocate zockets and bind them to the given addresses.  TCPDirect has
   * separate objects for sending and receiving UDP datagrams.
   */
  struct zfur* ur;
  ZF_TRY(zfur_alloc(&ur, stack, attr));
  ZF_TRY(zfur_addr_bind(ur, ai_local->ai_addr, ai_local->ai_addrlen,
                        ai_remote->ai_addr, ai_remote->ai_addrlen, 0));

  struct zfut* ut;
  ZF_TRY(zfut_alloc(&ut, stack, ai_local->ai_addr, ai_local->ai_addrlen,
                    ai_remote->ai_addr, ai_remote->ai_addrlen, 0, attr));

  ping_pongs(stack, ur, ut);

  return 0;
}

```

--------------------------------

### TCP Ping Pong Example Snippet

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/050_examples.dox

A snippet for a low-latency TCP ping pong application. Boilerplate code is omitted for clarity; a full version is available in `src/tests/zf_apps/zftcppingpong.c`.

```c
#define SIZE 12

```

--------------------------------

### Build RPM Package from SRPM

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/DEVELOPING.md

Builds an RPM package from the generated SRPM. This command requires the onload and onload-devel packages to be installed or uses a specified onload tarball via --define.

```bash
rpmbuild --rebuild ~/rpmbuild/SRPMS/tcpdirect-${version}-1.src.rpm --define "onload_tarball ${onload}"
```

--------------------------------

### TCPDirect Ping Pong Core Logic

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/050_examples.dox

Implements the main data sending and receiving loop for the ping-pong example. It polls the reactor for events and handles message reception and transmission.

```c
#define ITERATIONS 1000000
int ping;


struct rx_msg {
  /* The iovec used by zft_msg must be immediately afterwards */
  struct zft_msg msg;
  struct iovec iov[1];
};


static void ping_pongs(struct zf_stack* stack, struct zft* zock)
{
  char send_buf[SIZE];
  struct rx_msg msg;
  const int max_iov = sizeof(msg.iov) / sizeof(msg.iov[0]);
  int sends_left = ITERATIONS;
  int recvs_left = ITERATIONS;
  bool zock_has_rx_data = false;

  if( ping ) {
    ZF_TEST(zft_send_single(zock, send_buf, SIZE, 0) == SIZE);
    --sends_left;
  }

  do {
    size_t bytes_left = SIZE;
    do {
      if( ! zock_has_rx_data )
        /* Poll the stack until something happens. */
        while( zf_reactor_perform(stack) == 0 )
          ;
      msg.msg.iovcnt = max_iov;
      zft_zc_recv(zock, &msg.msg, 0);
      if( msg.msg.iovcnt ) {
        /* NB. msg.iov[0].iov_len==0 indicates we're not going to get any
         * more data (ie. the other end has shutdown or connection has
         * died).  We don't check for that here...instead it will be
         * detected if zft_zc_recv_done()!=1.
         */
        ZF_TEST(msg.iov[0].iov_len <= bytes_left);
        bytes_left -= msg.iov[0].iov_len;
        if( bytes_left == 0 )
          /* Break out to do send before zft_zc_recv_done() to save a few
           * nanoseconds.
           */
          break;
        ZF_TEST(zft_zc_recv_done(zock, &msg.msg) == 1);
      }
      zock_has_rx_data = msg.msg.pkts_left != 0;
    } while( bytes_left );

    if( sends_left ) {
      ZF_TEST(zft_send_single(zock, send_buf, SIZE, 0) == SIZE);
      --sends_left;
    }
    ZF_TEST(zft_zc_recv_done(zock, &msg.msg) == 1);
    --recvs_left;
  } while( recvs_left );
}

```

--------------------------------

### Get TX Timestamps

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Use these functions to retrieve transmit timestamps when TX timestamping is enabled via the tx_timestamping attribute. These functions are declared in zf_tcp.h and zf_udp.h.

```c
int zft_get_tx_timestamps(struct zft* ts, struct zf_pkt_report* reports,
                          int* count_in_out);

int zfut_get_tx_timestamps(struct zfut* us, struct zf_pkt_report* reports_out,
                           int* count_in_out);
```

--------------------------------

### Get Local and Remote IP Addresses and Ports for TCP Sockets

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Use zft_getname to discover the local and/or remote IP addresses and ports in use for TCP sockets. This function is declared in zf_tcp.h.

```c
void zft_getname(struct zft* ts, struct sockaddr* laddr_out,
                 struct sockaddr* raddr_out);
```

--------------------------------

### OK: Queue and Send Alternatives

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Demonstrates the correct sequence for queuing data on an alternative and then sending it. The alternative must be cancelled before reuse.

```c
zft_send(ts,<data>, 0);
zft_alternatives_queue(ts, q, <data>, flags);
zf_alternatives_send(stack, q);
```

--------------------------------

### OK: Send Alternatives After Queuing

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Shows an alternative valid sequence where alternatives are queued and sent before a direct send operation.

```c
zft_alternatives_queue(ts, q, <data>, flags);
zf_alternatives_send(stack, q);
zft_send(ts,<data>, 0);
```

--------------------------------

### Set Environment and Run Tests

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/DEVELOPING.md

Sets the ONLOAD_TREE and ZF_DEVEL environment variables, then cleans and runs the unit tests for TCPDirect. Requires sudo access.

```bash
$ export ONLOAD_TREE={path to the checked out onload repository}
$ export ZF_DEVEL=1
$ make clean test
```

--------------------------------

### TCPDirect Ping Pong Main Function

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/050_examples.dox

Handles command-line arguments, initializes TCPDirect, sets up the zocket for either ping or pong mode, establishes the connection, and calls the core ping-pong logic.

```c
int main(int argc, char* argv[])
{
  if( argc != 2 )
    usage_err();

  if( ! strcmp(argv[0], "ping") )
    ping = true;
  else if( ! strcmp(argv[0], "pong") )
    ping = false;
  else
    usage_err();

  struct addrinfo* ai;
  if( getaddrinfo_hostport(argv[1], NULL, &ai) != 0 ) {
    fprintf(stderr, "ERROR: failed to lookup address '%s'\n", argv[1]);
    exit(2);
  }

  /* Initialise the TCPDirect library and allocate a stack. */
  ZF_TRY(zf_init());

  struct zf_attr* attr;
  ZF_TRY(zf_attr_alloc(&attr));

  struct zf_stack* stack;
  ZF_TRY(zf_stack_alloc(attr, &stack));

  struct zft* zock;

  if( ping ) {
    /* In 'ping' mode, connect to the specified remote address. */
    struct zft_handle* tcp_handle;
    ZF_TRY(zft_alloc(stack, attr, &tcp_handle));
    printf("Connecting to ponger\n");
    ZF_TRY(zft_connect(tcp_handle, ai->ai_addr, ai->ai_addrlen, &zock));
    /* The zft_connect() call is non-blocking, so the zocket is not yet
     * connected.  Wait until the connect completes or fails...
     */
    while( zft_state(zock) == TCP_SYN_SENT )
      zf_reactor_perform(stack);
    ZF_TEST( zft_state(zock) == TCP_ESTABLISHED );
  }
  else {
    /* In 'pong' mode, create a listening zocket and wait until we've
     * accepted a connection.
     */
    struct zftl* listener;
    int rc;
    ZF_TRY(zftl_listen(stack, ai->ai_addr, ai->ai_addrlen, attr, &listener));
    printf("Waiting for incoming connection\n");
    do {
      while( zf_reactor_perform(stack) == 0 );
    } while( (rc = zftl_accept(listener, &zock)) == -EAGAIN );
    ZF_TRY(rc);
    ZF_TRY(zftl_free(listener));
  }
  printf("Connection established\n");

  ping_pongs(stack, zock);

  /* Do a clean shutdown and free all resources. */
  printf("Completed\n");
  while( zft_shutdown_tx(zock) == -EAGAIN )
    zf_reactor_perform(stack);

  while( ! zf_stack_is_quiescent(stack) )
    zf_reactor_perform(stack);

  ZF_TRY(zft_free(zock));
  ZF_TRY(zf_stack_free(stack));
  ZF_TRY(zf_deinit());
  return 0;
}

```

--------------------------------

### Trader Application Usage (Normal Sends)

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/030_apps.dox

Run the exchange server with onload and the trader client with its arguments. The server listens on a multicast interface and the client connects to a specified server IP.

```shell
Server: <code>onload -p latency-best ./exchange _mcast-intf_</code>

Client: <code>./trader_tcpdirect_ds_efvi _mcast-intf_ _server_</code>
```

--------------------------------

### Enable Huge Pages

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/DEVELOPING.md

Configures the system to allocate huge pages by setting the vm.nr_hugepages sysctl parameter. This is a solution for the 'Failed to allocate huge page for emu' problem.

```bash
$ sudo sysctl vm.nr_hugepages=4096
```

--------------------------------

### zftcppingpong Client Usage

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/030_apps.dox

Command to run the zftcppingpong application as a client. Ensure the ZF_ATTR environment variable is set to specify the network interface.

```sh
export ZF_ATTR=interface=ethX
zftcppingpong ping serverhost:serverport
```

--------------------------------

### zftcppingpong Server Usage

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/030_apps.dox

Command to run the zftcppingpong application as a server. Ensure the ZF_ATTR environment variable is set to specify the network interface.

```sh
export ZF_ATTR=interface=ethX
zftcppingpong pong serverhost:serverport
```

--------------------------------

### zftcpmtpong Application Usage

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/030_apps.dox

Set the ZF_ATTR environment variable to specify the network interface and then run the zftcpmtpong application with the local address and port.

```shell
export ZF_ATTR=interface=ethX
 zftcpmtpong localaddr:port
```

--------------------------------

### Run Application with Debug Library

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

To use the shared debug library, invoke the application via the 'zf_debug' wrapper. This ensures that debug versions of TCPDirect libraries are used for logging and parameter validation.

```bash
zf_debug app

```

--------------------------------

### Create TCP Listening Zocket

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Function to create a TCP listening zocket. Requires a zf_stack, local address, and optional attributes.

```c
int zftl_listen(struct zf_stack* st,
                const struct sockaddr* laddr,
                socklen_t laddrlen,
                const struct zf_attr* attr,
                struct zftl** tl_out);
```

--------------------------------

### zfudppingpong Client Usage

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/030_apps.dox

Command to run the zfudppingpong application as a client. Ensure the ZF_ATTR environment variable is set to specify the network interface.

```sh
export ZF_ATTR=interface=ethX
zfudppingpong ping clienthost:clientport serverhost:serverport
```

--------------------------------

### Trader Application Usage (Delegated Sends)

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/030_apps.dox

Run the exchange server with onload and the trader client with the -d option for delegated sends. Specify the multicast interface and server IP address.

```shell
Server: <code>onload -p latency-best ./exchange _mcast-intf_</code>

Client: <code>./trader_tcpdirect_ds_efvi -d _mcast-intf_ _server_</code>
```

--------------------------------

### zfudppingpong Server Usage

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/030_apps.dox

Command to run the zfudppingpong application as a server. Ensure the ZF_ATTR environment variable is set to specify the network interface.

```sh
export ZF_ATTR=interface=ethX
zfudppingpong pong serverhost:serverport clienthost:clientport
```

--------------------------------

### Convert Zockets to Waitables

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Shows how to obtain a zf_waitable from different types of zockets (UDP receive, UDP transmit, TCP listening, TCP) for use with the multiplexer.

```c
struct zf_waitable* zfur_to_waitable(struct zfur* us);
struct zf_waitable* zfut_to_waitable(struct zfut* us);
struct zf_waitable* zftl_to_waitable(struct zftl* tl);
struct zf_waitable* zft_to_waitable(struct zft* ts);
```

--------------------------------

### Set Onload Path and Build TCPDirect

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/DEVELOPING.md

Sets the ONLOAD_TREE environment variable to the path of the checked-out Onload repository and then executes the make command to build TCPDirect.

```bash
export ONLOAD_TREE={path to the checked out onload repository}
make
```

--------------------------------

### Create SRPM Package

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/DEVELOPING.md

Uses the zf_make_official_srpm script to create a Source RPM (SRPM) package for TCPDirect. The version can be a git hexsha or a semver string.

```bash
scripts/zf_make_official_srpm --version ${version}
```

--------------------------------

### Send via Alternative TX Queues

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Sends packets through the TX path on the NIC using alternative queues for minimized latency.

```c
int zf_alternatives_send(struct zf_stack* stack,
                           zf_althandle alt);
```

--------------------------------

### Allocate and Free TCPDirect Stack

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

These functions are used to create and destroy the main TCPDirect stack. The `zf_stack_alloc` function takes attributes to configure the stack, such as the network interface and buffer count.

```c
int zf_stack_alloc(struct zf_attr* attr, struct zf_stack** stack_out);
int zf_stack_free(struct zf_stack* stack);
```

--------------------------------

### SPDX License and Copyright Information

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/README.md

This snippet contains the SPDX license identifier and copyright information for the project.

```yaml
SPDX-License-Identifier: MIT
SPDX-FileCopyrightText: Copyright (C) 2020-2024 Advanced Micro Devices, Inc.
```

--------------------------------

### NOT OK: Invalid Sequence for Alternatives

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Illustrates an incorrect sequence where a direct send occurs between queuing and sending alternatives, which will result in an error.

```c
zft_alternatives_queue(ts, q, <data>, flags);
zft_send(ts,<data>, 0);
zf_alternatives_send(stack, q);  // Will return -EINVAL
```

--------------------------------

### Allocate Alternative TX Queues

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Allocates alternative transmit queues for minimizing latency on send operations. Requires a zf_stack and optional attributes.

```c
int zf_alternatives_alloc(struct zf_stack* stack,
                          const struct zf_attr* attr,
                          zf_althandle* alt_out);
```

--------------------------------

### Add Waitable to Multiplexer Set

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Illustrates how to add a zf_waitable to a multiplexer set. Each waitable can only belong to one set, and each set to one stack.

```c
int zf_muxer_add(struct zf_muxer_set*,
                 struct zf_waitable* w,
                 const struct epoll_event* event);
```

--------------------------------

### Wait for Events on Multiplexer Set

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Demonstrates polling a multiplexer set for ready zockets. The timeout parameter controls blocking behavior.

```c
int zf_muxer_wait(struct zf_muxer_set*,
                  struct epoll_event* events,
                  int maxevents,
                  int64_t timeout);
```

--------------------------------

### Clone Onload Repository

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/DEVELOPING.md

Clones the Onload source code repository, which is a required dependency for building TCPDirect.

```bash
git clone git@github.com:Xilinx/onload.git
```

--------------------------------

### Release Alternative TX Queues

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Releases allocated alternative transmit queues.

```c
int zf_alternatives_release(struct zf_stack* stack,
                            zf_althandle alt);
```

--------------------------------

### Bind UDP Receive Zocket to Address and Port

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Associates a UDP receive zocket with a local address and port, and optionally adds filters. This prepares the zocket to receive packets destined for the specified endpoint.

```c
int zfur_addr_bind(struct zfur* us,
                   struct sockaddr* laddr,
				   socklen_t laddrlen,
                   const struct sockaddr* raddr,
				   socklen_t raddrlen,
				   int flags);
```

--------------------------------

### Modify Waitable in Multiplexer Set

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Shows how to change the event configuration for a waitable that is already part of a multiplexer set.

```c
int zf_muxer_mod(struct zf_waitable* w,
                 const struct epoll_event* event);
```

--------------------------------

### Inspect CTPIO Diagnostics

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Use the ethtool command to inspect CTPIO (Cut-through PIO) diagnostics on a network interface. This helps in understanding CTPIO usage and failure reasons.

```sh
ethtool -S ethX | grep ctpio
```

--------------------------------

### Allocate TCP Zocket

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Allocates a TCP zocket handle for both sending and receiving. Requires a zf_stack and optional attributes.

```c
int zft_alloc(struct zf_stack* st,
              const struct zf_attr* attr,
              struct zft_handle** handle_out);
```

--------------------------------

### Delete Waitable from Multiplexer Set

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Illustrates how to remove a waitable from a multiplexer set.

```c
int zf_muxer_del(struct zf_waitable* w);
```

--------------------------------

### zfsink Application Usage

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/030_apps.dox

Command to run the zfsink application to receive UDP datagrams. Ensure the ZF_ATTR environment variable is set to specify the network interface. The localaddr should be an IP address on the interface or a multicast address.

```sh
export ZF_ATTR=interface=ethX
zfsink localaddr:port
```

--------------------------------

### Bind TCP Zocket to Address

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Binds a TCP zocket handle to a local address and port. Flags can be provided.

```c
int zft_addr_bind(struct zft_handle* handle,
                  const struct sockaddr* laddr,
                  socklen_t laddrlen,
                  int flags);
```

--------------------------------

### Copy-Based TCP Receive

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Performs a copy-based receive on a connected TCP zocket, copying data into provided iovec buffers.

```c
int zft_recv(struct zft* ts,
             struct iovec* iov_out,
             int iovcnt,
             int flags);
```

--------------------------------

### Accept Passive TCP Connection

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Function to accept a passively opened TCP connection on a listening zocket.

```c
int zftl_accept(struct zftl* tl,
                struct zft** ts_out);
```

--------------------------------

### UDP Receive Zocket Operations

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Functions for allocating, binding, and receiving UDP datagrams using zero-copy.

```APIDOC
## UDP Receive Zocket Operations

### Description
Functions for allocating, binding, and receiving UDP datagrams using zero-copy.

### Functions

- `int zfur_alloc(struct zfur** us_out, struct zf_stack* st, const struct zf_attr* attr);`
- `int zfur_addr_bind(struct zfur* us, struct sockaddr* laddr, socklen_t laddrlen, const struct sockaddr* raddr, socklen_t raddrlen, int flags);`
- `int zfur_zc_recv(struct zfur* us, struct zfur_msg* msg, int flags);`
- `int zfur_zc_recv_done(struct zfur* us, struct zfur_msg* msg);`

### Parameters

#### `zfur_alloc`
- `us_out` (struct zfur**) - Output parameter to store the pointer to the newly allocated UDP receive zocket.
- `st` (struct zf_stack*) - Pointer to the TCPDirect stack.
- `attr` (const struct zf_attr*) - Pointer to attributes for configuring the zocket.

#### `zfur_addr_bind`
- `us` (struct zfur*) - Pointer to the UDP receive zocket.
- `laddr` (struct sockaddr*) - Local address to bind to.
- `laddrlen` (socklen_t) - Length of the local address structure.
- `raddr` (const struct sockaddr*) - Remote address to associate with (for filtering).
- `raddrlen` (socklen_t) - Length of the remote address structure.
- `flags` (int) - Flags for binding.

#### `zfur_zc_recv`
- `us` (struct zfur*) - Pointer to the UDP receive zocket.
- `msg` (struct zfur_msg*) - Output parameter to store message details, including pointers to zero-copy buffers.
- `flags` (int) - Flags for receiving.

#### `zfur_zc_recv_done`
- `us` (struct zfur*) - Pointer to the UDP receive zocket.
- `msg` (struct zfur_msg*) - Pointer to the message whose buffers are to be released.

### Notes
- These functions are found in `zf_udp.h`.
- `zfur_zc_recv` performs a zero-copy read. The buffers are locked until `zfur_zc_recv_done` is called.
```

--------------------------------

### Stack Allocation and Freeing

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Functions to allocate and free a TCPDirect stack. The stack configuration is controlled by attributes.

```APIDOC
## Stack Allocation and Freeing

### Description
Functions to allocate and free a TCPDirect stack. The stack configuration is controlled by attributes.

### Functions

- `int zf_stack_alloc(struct zf_attr* attr, struct zf_stack** stack_out);`
- `int zf_stack_free(struct zf_stack* stack);`

### Parameters

#### `zf_stack_alloc`
- `attr` (struct zf_attr*) - Pointer to attributes for configuring the stack (e.g., network interface, buffer count).
- `stack_out` (struct zf_stack**) - Output parameter to store the pointer to the newly allocated stack.

#### `zf_stack_free`
- `stack` (struct zf_stack*) - Pointer to the stack to be freed.

### Notes
- The `attr` parameter in `zf_stack_alloc` is crucial for configuring stack behavior, including the network interface and the number of packet buffers (`n_bufs`). Insufficient `n_bufs` can lead to dropped packets and `ENOMEM` errors.
```

--------------------------------

### Check for Pending Work

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Use zf_stack_has_pending_work() to determine if the stack has any pending operations. A non-zero return value indicates that zf_reactor_perform() or zf_muxer_wait() should be called.

```c
int zf_stack_has_pending_work(const struct zf_stack* st);
```

--------------------------------

### UDP Send Function Signature

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Signature for the zfut_send function used for UDP communication.

```c
int zfut_send(struct zfut* us,
              const struct iovec* iov,
              int iov_cnt,
              int flags);
```

--------------------------------

### Copy-Based TCP Send

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Performs a copy-based send on a TCP zocket. Supplied buffers can be reused immediately after the call returns.

```c
int zft_send(struct zft* ts,
             const struct iovec* iov,
             int iov_cnt,
             int flags);
```

--------------------------------

### Connect TCP Zocket

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Connects a TCP zocket to a remote address and port. The zocket handle type may change. This operation is non-blocking.

```c
int zft_connect(struct zft_handle* handle,
                const struct sockaddr* raddr,
                socklen_t raddrlen,
                struct zft** ts_out);
```

--------------------------------

### Receive UDP Datagram with Zero-Copy

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Performs a zero-copy read of a single UDP datagram. The message structure is updated to point to the buffers containing the datagram. These buffers are locked until `zfur_zc_recv_done` is called.

```c
int zfur_zc_recv(struct zfur* us,
                 struct zfur_msg* msg,
                 int flags);
```

--------------------------------

### Free Listening Zocket

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Function to close and free a listening zocket.

```c
int zftl_free(struct zftl* ts);
```

--------------------------------

### Allocate UDP Receive Zocket

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Allocates a UDP receive zocket. This zocket is used to receive UDP datagrams from the network.

```c
int zfur_alloc(struct zfur** us_out,
               struct zf_stack* st,
               const struct zf_attr* attr);
```

--------------------------------

### Complete Zero-Copy Receive

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Indicates that the caller has finished with the received message buffers from a zero-copy receive operation.

```c
void zft_zc_recv_done(struct zft* ts,
                      struct zft_msg* msg);
```

--------------------------------

### Perform Reactor Operations

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Call zf_reactor_perform() to process events and perform actions within the reactor. This function is internally called by blocking functions like zf_muxer_wait().

```c
int zf_reactor_perform(struct zf_stack* st);
```

--------------------------------

### Zero-Copy TCP Receive

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Performs a zero-copy receive on a connected TCP zocket. The received message buffers are locked until zft_zc_recv_done is called.

```c
int zft_zc_recv(struct zft* ts,
                struct zft_msg* msg,
                int flags);
```

--------------------------------

### UDP Send Zocket Allocation

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Function for allocating a UDP transmit zocket with specified local and remote addresses.

```APIDOC
## UDP Send Zocket Allocation

### Description
Function for allocating a UDP transmit zocket with specified local and remote addresses.

### Function

- `int zfut_alloc(struct zfut** us_out, struct zf_stack* st, const struct sockaddr* laddr, socklen_t laddrlen, const struct sockaddr* raddr, socklen_t raddrlen, int flags, const struct zf_attr* attr);`

### Parameters

- `us_out` (struct zfut**) - Output parameter to store the pointer to the newly allocated UDP transmit zocket.
- `st` (struct zf_stack*) - Pointer to the TCPDirect stack.
- `laddr` (const struct sockaddr*) - Local address and port to bind to.
- `laddrlen` (socklen_t) - Length of the local address structure.
- `raddr` (const struct sockaddr*) - Remote address and port to send to.
- `raddrlen` (socklen_t) - Length of the remote address structure.
- `flags` (int) - Flags for allocation.
- `attr` (const struct zf_attr*) - Pointer to attributes for configuring the zocket.
```

--------------------------------

### Complete Zero-Copy UDP Receive

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Releases the buffers used by a zero-copy received UDP datagram, allowing them to be reused by the stack. This must be called after processing a message received via `zfur_zc_recv`.

```c
int zfur_zc_recv_done(struct zfur* us,
                      struct zfur_msg* msg);
```

--------------------------------

### Perform Copy-Based UDP Datagram Send

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Performs a copy-based send of a single UDP datagram. This operation may potentially use Programmed I/O (PIO) for data transfer.

```c
// Function signature for copy-based send is not provided in the source text.
```

--------------------------------

### Allocate UDP Transmit Zocket

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Allocates a UDP transmit zocket. This zocket is used to send UDP datagrams. It requires specifying local and remote addresses and ports, along with attributes for configuration.

```c
int zfut_alloc(struct zfut** us_out,
               struct zf_stack* st,
               const struct sockaddr* laddr,
               socklen_t laddrlen,
               const struct sockaddr* raddr,
               socklen_t raddrlen,
               int flags,
               const struct zf_attr* attr);
```

--------------------------------

### Cancel Alternative TX Send

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Cancels a send operation that was initiated using alternative transmit queues.

```c
int zf_alternatives_cancel(struct zf_stack* stack,
                             zf_althandle alt);
```

--------------------------------

### Workaround for C++ Compiler Error with Flexible Array Members

Source: https://github.com/xilinx-cns/tcpdirect/blob/master/src/lib/zf/doxygen/040_using.dox

Modify application code to resolve the 'flexible array member not at end of struct' error from newer C++ compilers like g++ version 6 and above. This involves changing the struct definition.

```c
struct my_msg {
  zft_msg msg;
  iovec iov[1];
};
```

```c
typedef struct {
  zft_msg msg;
  iovec iov[1];
} my_msg;
```
Tokens:
Raw