### Compile and Run UCX Client-Server Example Source: https://github.com/openucx/ucx/blob/master/docs/source/index.rst Compiles a UCX client-server C example using installed libraries and then runs the server and client. Demonstrates basic UCX communication. ```console $ gcc examples/ucp_client_server.c -lucp -lucs -o ucp_client_server \ -Iinstall/include -Linstall/lib $ export LD_LIBRARY_PATH=$PWD/install/lib $ ./ucp_client_server & $ ./ucp_client_server -a # : IP address of a local RoCE or IPoIB interface ``` -------------------------------- ### Build and Install UCX Source: https://github.com/openucx/ucx/wiki/OpenMPI-and-OpenSHMEM-installation-with-UCX Build and install the UCX library after configuration. ```bash make make install ``` -------------------------------- ### Example of Comment Order and Explanation Source: https://github.com/openucx/ucx/blob/master/REVIEW.md Illustrates the recommended order for code review comments, starting with the requested change and followed by a brief explanation if necessary. ```shell pls add $(UCX_LT_RELEASE) here as well; otherwise libucs_signal stays outside the suffix scheme. ``` -------------------------------- ### Build and Install MPICH Source: https://github.com/openucx/ucx/wiki/MPICH-Installation-with-UCX Build the MPICH library and install it after configuring with UCX support. ```bash $ make $ make install ``` -------------------------------- ### Download, Build, and Install UCX Source: https://github.com/openucx/ucx/blob/master/docs/source/index.rst Steps to download the latest UCX release, build it from source, and install it locally. Includes configuration and compilation commands. ```console $ wget https://github.com/openucx/ucx/releases/download/v{RELEASE}/ucx-{RELEASE}.tar.gz $ tar xzf ucx-{RELEASE}.tar.gz $ cd ucx-{RELEASE} $ ./contrib/configure-release --prefix=$PWD/install $ make -j8 install ``` -------------------------------- ### Dockerfile for UCP Client-Server Image Source: https://github.com/openucx/ucx/wiki/UCP-Client‐Server-Example-on-Kubernetes-Cluster This Dockerfile installs necessary build tools, clones the UCX repository, builds and installs UCX with UCP client-server example, and sets environment variables for its use. ```dockerfile FROM centos:7 RUN yum install -y \ autoconf \ automake \ gcc-c++ \ git \ libtool \ make RUN git clone --depth 1 --branch v1.16.0 https://github.com/openucx/ucx.git WORKDIR /ucx/ RUN ./autogen.sh WORKDIR /ucx/build RUN ../contrib/configure-release --prefix=$PWD/../install RUN make -j install ENV PATH="${PATH}:/ucx/install/bin" ENV LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/ucx/install/lib" ENV C_INCLUDE_PATH="${C_INCLUDE_PATH}:/ucx/install/include" ENV LIBRARY_PATH="${LIBRARY_PATH}:/ucx/install/lib" WORKDIR /ucx RUN gcc examples/ucp_client_server.c -lucp -lucs -o /ucx/install/bin/ucp_client_server ``` -------------------------------- ### Start UCX VFS Daemon Source: https://github.com/openucx/ucx/wiki/Instrumentation-and-monitoring-tool Launches the UCX VFS daemon to enable runtime analysis of UCX-based applications. This command should be run from the UCX installation directory. ```bash $ /bin/ucx_vfs ``` -------------------------------- ### Build and Install UCX Source: https://github.com/openucx/ucx/blob/master/docs/source/running.md Compile and install UCX using make with parallel jobs. ```bash make -j4 make install ``` -------------------------------- ### Download and Install UCX v1.17.0 Source: https://github.com/openucx/ucx/wiki/Home Steps to download, extract, configure, and install the UCX library from a tarball. Ensure you have the necessary build tools. ```console $ wget https://github.com/openucx/ucx/releases/download/v1.17.0/ucx-1.17.0.tar.gz $ tar xzf ucx-1.17.0.tar.gz $ cd ucx-1.17.0 $ ./contrib/configure-release --prefix=$PWD/install $ make -j8 install ``` -------------------------------- ### Install Open MPI Prerequisites Source: https://github.com/openucx/ucx/wiki/Build-and-run-ROCm-UCX-OpenMPI Installs necessary development tools for building Open MPI on Ubuntu systems. This includes version control and build utilities. ```bash sudo apt install git m4 autoconf automake libtool flex ``` -------------------------------- ### Build UCX from Release Tarball Source: https://github.com/openucx/ucx/blob/master/README.md Use these commands to install UCX from a release tarball. Ensure you have the necessary permissions to install to the specified prefix. ```sh $ ./contrib/configure-release --prefix=/where/to/install $ make -j8 $ make install ``` -------------------------------- ### Set Installation Environment Variables Source: https://github.com/openucx/ucx/wiki/Build-and-run-ROCm-UCX-OpenMPI Defines environment variables for specifying installation directories for UCX and Open MPI. Customize these paths as needed. ```bash export INSTALL_DIR=/path/to/install export UCX_DIR=$INSTALL_DIR/ucx export OMPI_DIR=$INSTALL_DIR/ompi ``` -------------------------------- ### Start UCX Server for Performance Testing Source: https://github.com/openucx/ucx/wiki/Performance-measurement Initiates the UCX performance testing server. This command starts the server in client mode (0) and waits for client connections. ```shell $ ucx_perftest -c 0 Waiting for connection... +------------------------------------------------------------------------------------------+ | API: protocol layer | | Test: UCP tag match latency | | Data layout: (automatic) | | Message size: 8 | +------------------------------------------------------------------------------------------+ ``` -------------------------------- ### Configure OpenMPI with UCX Source: https://github.com/openucx/ucx/blob/master/docs/source/running.md Configure OpenMPI to use UCX, specifying installation paths for both. ```bash mkdir build-ucx cd build-ucx ../configure --prefix= --with-ucx= ``` -------------------------------- ### Testing UCX with IMB Benchmark Source: https://github.com/openucx/ucx/wiki/UCX-Testing-Procedure Example command for testing UCX on a single host using the IMB benchmark. Ensure UCX is configured for release mode. ```bash mpirun –H host1 -np 16 -mca pml ucx -x UCX_TLS=mm -x UCX_SHM_DEVICES=sysv --bind-to core --map-by core --display-map IMB-MPI1 ``` -------------------------------- ### Example: Enabling Multi-Rail Support in UCX Source: https://github.com/openucx/ucx/wiki/FAQ Set environment variables to configure the number of rails for eager and rendezvous protocols, and specify network devices for multi-rail communication. ```bash mpirun -x UCX_MAX_EAGER_RAILS=2 -x UCX_MAX_RNDV_RAILS=3 -x UCX_NET_DEVICES=mlx5_0:1,mlx5_1:1 ... ``` -------------------------------- ### Configure and Build librte (Optional) Source: https://github.com/openucx/ucx/wiki/Matt's-Titan-notes Configure and install the librte component, which is optional but required for certain UCX utilities like perf_test that involve remote node communication. ```bash ./configure --prefix=$MEMBERWORK/$PROJNAME/ucx-work/librte --with-pmi=/opt/cray/pmi/default/ --with-pmi-lib=/opt/cray/pmi/default/lib64/ --with-pmi-include=/opt/cray/pmi/default/include/ make make install ``` -------------------------------- ### Configuring UCX Transports and Devices with mpirun Source: https://github.com/openucx/ucx/wiki/UCX-environment-parameters This example demonstrates setting UCX transports (rc_x, sysv) and network devices (mlx5_0:1) directly within an mpirun command. ```bash mpirun -mca pml ucx -x UCX_TLS=rc_x,sysv -x UCX_NET_DEVICES=mlx5_0:1 ... ``` -------------------------------- ### Build UCX from Git Repository Source: https://github.com/openucx/ucx/blob/master/README.md Clone the UCX repository and follow these steps to build and install. This method requires running autogen.sh first. ```sh $ ./autogen.sh $ ./contrib/configure-release --prefix=/where/to/install $ make -j8 $ make install ``` -------------------------------- ### Read Profile Output Source: https://github.com/openucx/ucx/wiki/Profiling Example of reading a collected profile file using ucx_read_profile and the resulting formatted output. ```bash $ ucx_read_profile ucx.prof command : ./app host : my_host pid : 9999 units : usec NAME AVG TOTAL COUNT FILE FUNCTION printf 15.316 15 1 profiling.c:13 my_func_inner() my_func 15.883 16 1 profiling.c:12 my_func() 0.000 my_func 15.883 { 0.332 printf 15.316 { 15.316 } 0.236 } ``` -------------------------------- ### Run MPICH Application with UCX Source: https://github.com/openucx/ucx/wiki/MPICH-Installation-with-UCX Execute an MPICH application using mpirun, specifying UCX network devices if necessary. This example shows how to select a specific InfiniBand device and port. ```bash $ mpirun -np 2 -env UCX_NET_DEVICES=mlx5_0:1 ./app ``` -------------------------------- ### Manage individual iodemo processes Source: https://github.com/openucx/ucx/blob/master/test/apps/iodemo/README.md Control the status, start, stop, or kill of a specific iodemo process (e.g., server_0) using the generated launch script. ```bash %./iodemo_commands_node1.sh -status server_0 %./iodemo_commands_node1.sh -start server_0 %./iodemo_commands_node1.sh -stop server_0 %./iodemo_commands_node1.sh -kill server_0 ``` -------------------------------- ### UCT Operation Handle Allocation Example Source: https://github.com/openucx/ucx/wiki/UCT-Design Illustrates how to initiate an asynchronous operation using UCT and handle its completion status. The callback structure should be initialized with a counter and a callback function. ```c ucs_status_t uct_ep_put_zcopy(uct_ep_h ep, const void *buffer, size_t length, uct_mem_h memh, uint64_t remote_addr, uct_rkey_t rkey, uct_completion_t *comp) ``` ```c status = api_call(..., &my_handle->comp); if (likely(status == UCS_OK)) { /* done */ } else if (status == UCS_INPROGRESS) { /* started */ } else if (status == UCS_ERR_NO_RESOURCE) { /* cannot be started now */ } else { /* error */ } ``` -------------------------------- ### Configure UCX Build Source: https://github.com/openucx/ucx/wiki/OpenMPI-and-OpenSHMEM-installation-with-UCX Configure the UCX build with a specified installation path. Use contrib/configure-release for optimized builds or contrib/configure-devel for debug builds. ```bash mkdir build cd build ../configure --prefix=/your_install_path ``` -------------------------------- ### Configure Open MPI with UCX Support Source: https://github.com/openucx/ucx/wiki/OpenMPI-and-OpenSHMEM-installation-with-UCX Configure Open MPI to use UCX, specifying installation paths. Optionally disable the 'btl-uct' component if compilation errors occur. ```bash mkdir build cd build ../configure --prefix=/your_install_path/ --with-ucx=/path_to_ucx_installation ``` ```bash ./configure ... --enable-mca-no-build=btl-uct ... ``` -------------------------------- ### Configure and Build with Profiling Source: https://github.com/openucx/ucx/wiki/Profiling Instructions for configuring and compiling the project with profiling enabled using contrib/configure-prof. ```bash $ ./contrib/configure-prof --prefix=$PWD/install # etc $ make -j 32 && make install ``` -------------------------------- ### Compile Large BAR Check Example Source: https://github.com/openucx/ucx/wiki/Build-and-run-ROCm-UCX-OpenMPI Compiles the C sanity check code using the ROCm hipcc compiler. Ensure ROCm is installed and hipcc is in your PATH. ```bash $ /opt/rocm/bin/hipcc $(/opt/rocm/bin/hipconfig --cpp_config) -L/opt/rocm/lib/ -lamdhip64 check_large_bar.c -o check_large_bar ``` -------------------------------- ### Generate iodemo launch script Source: https://github.com/openucx/ucx/blob/master/test/apps/iodemo/README.md Use run_io_demo.sh with various options to generate host-specific launch scripts. This command simulates the script generation without actual execution. ```bash %./run_io_demo.sh --dry-run -i eth0 -H node1,node2 --num-clients 5 \ --num-servers 5 --tasks-per-node 10 --duration $((10*60)) --log-dir $PWD/logs $PWD/io_demo Launch configuration: host_list : 'node1,node2' tasks_per_node : '10' map_by : 'node' num_clients : '5' num_servers : '5' iodemo_exe : 'io_demo' iodemo_client_args : '' net_if : 'eth0' base_port_num : '20000' duration : '600' client_wait_time : '2' launcher : 'pdsh -b -w' dry_run : '1' node1 : iodemo_commands_node1.sh node2 : iodemo_commands_node2.sh ``` -------------------------------- ### Run all iodemo processes Source: https://github.com/openucx/ucx/blob/master/test/apps/iodemo/README.md Execute all managed iodemo processes (servers and clients) simultaneously using the generated launch script without specifying a tag. ```bash %./iodemo_commands_node1.sh ``` -------------------------------- ### Example of Filtered Statistics Output Source: https://github.com/openucx/ucx/wiki/Statistics This example shows a summary output when specific counters are filtered using glob patterns. The output aggregates counts for matching counters across different components. ```text [elrond1:13966] ucp_worker{rx_eager_msg:10000 rx_eager_chunk_exp:1670000 rx_eager_chunk_unexp:0} ucp_ep{tx_eager:10000 tx_eager_sync:0} uct_ep{bytes_bcopy:10253440130 uct_ep.bytes_zcopy:0} ``` -------------------------------- ### Display UCX Environment Variable Documentation Source: https://github.com/openucx/ucx/blob/master/docs/source/faq.md To view detailed documentation for all UCX environment variables, run 'ucx_info -cf'. This command provides explanations for each configurable parameter. ```bash ucx_info -cf ``` -------------------------------- ### Clone UCX Repository Source: https://github.com/openucx/ucx/wiki/OpenMPI-and-OpenSHMEM-installation-with-UCX Clone the UCX repository from GitHub to get the latest source code. ```bash git clone https://github.com/openucx/ucx.git ucx cd ucx ``` -------------------------------- ### Configure MPICH with UCX Source: https://github.com/openucx/ucx/blob/master/docs/source/running.md Configure MPICH to use UCX as its communication device, specifying the installation path. ```bash mkdir build cd build ../configure --prefix= --with-device=ch4:ucx ``` -------------------------------- ### Configure Librte on Cray Source: https://github.com/openucx/ucx/wiki/Pasha's-notes Configures the librte library with specified installation prefix and PMI settings for Cray systems. ```bash /ccs/home/pavel/work/librte/configure --prefix=/lustre/atlas/scratch/pavel/stf010/install --with-pmi=/opt/cray/pmi/default/ --with-pmi-lib=/opt/cray/pmi/default/lib64/ --with-pmi-include=/opt/cray/pmi/default/include/ ``` -------------------------------- ### Configure UCX Build Source: https://github.com/openucx/ucx/blob/master/docs/source/running.md Configure the UCX build process, specifying the installation path. For optimized builds, use contrib/configure-release. ```bash mkdir build cd build ../configure --prefix= ``` -------------------------------- ### Build UCX with ROCm Support Source: https://github.com/openucx/ucx/wiki/Build-and-run-ROCm-UCX-OpenMPI Compiles and installs UCX with ROCm support, disabling other features like CUDA and Knem. ```bash # git checkout v1.10.x # optional to use v1.10.x branch ./autogen.sh mkdir build cd build ../contrib/configure-opt --prefix=$UCX_DIR --with-rocm=/opt/rocm --without-knem --without-cuda --enable-gtest --enable-examples make make install ``` -------------------------------- ### UCX Performance Test - Server Source: https://github.com/openucx/ucx/blob/master/README.md Start the UCX performance test server. The -c flag sets CPU affinity. ```sh $ ./src/tools/perf/ucx_perftest -c 0 ``` -------------------------------- ### Help Message Option Source: https://github.com/openucx/ucx/wiki/Performance-measurement Displays the help message for the performance measurement tool. ```bash -h show this help message ``` -------------------------------- ### Output with Large BAR Enabled Source: https://github.com/openucx/ucx/wiki/Build-and-run-ROCm-UCX-OpenMPI Example output from the Large BAR sanity check when the setting is correctly enabled in the BIOS. ```bash $ ./check_large_bar address buf 0x14642b400000 Buf[0] = -1094795586 Buf[0] = 1 ``` -------------------------------- ### Displaying UCX Info Source: https://github.com/openucx/ucx/wiki/UCX-environment-parameters Use the ucx_info tool to display environment parameters. The -f flag shows all parameters, and grep can filter for specific ones like devices. ```bash $ ./bin/ucx_info -f ``` ```bash $ ./bin/ucx_info -f | grep DEVICES ``` -------------------------------- ### Get UCP Server Service Details Source: https://github.com/openucx/ucx/wiki/UCP-Client‐Server-Example-on-Kubernetes-Cluster Command to retrieve information about the UCP server service, including its ClusterIP and port. ```bash $ kubectl get svc ucp-server-svc ``` -------------------------------- ### ucx_perftest Command-Line Help Source: https://github.com/openucx/ucx/wiki/Performance-measurement Displays the usage and available options for the ucx_perftest command-line utility. This includes test types, message sizes, iteration counts, and various configuration flags. ```bash $ ucx_perftest -h Note: test can be also launched as an MPI application Usage: lt-ucx_perftest [ server-hostname ] [ options ] Common options: -t test to run: am_lat - active message latency put_lat - put latency add_lat - atomic add latency get - get latency / bandwidth / message rate fadd - atomic fetch-and-add latency / message rate swap - atomic swap latency / message rate cswap - atomic compare-and-swap latency / message rate am_bw - active message bandwidth / message rate put_bw - put bandwidth / message rate add_mr - atomic add message rate tag_lat - UCP tag match latency tag_bw - UCP tag match bandwidth tag_sync_lat - UCP tag sync match latency ucp_put_lat - UCP put latency ucp_put_bw - UCP put bandwidth ucp_get - UCP get latency / bandwidth / message rate ucp_add - UCP atomic add bandwidth / message rate ucp_fadd - UCP atomic fetch-and-add latency / bandwidth / message rate ucp_swap - UCP atomic swap latency / bandwidth / message rate ucp_cswap - UCP atomic compare-and-swap latency / bandwidth / message rate stream_bw - UCP stream bandwidth stream_lat - UCP stream latency -s list of scatter-gather sizes for single message (8) for example: "-s 16,48,8192,8192,14" -n number of iterations to run (1000000) -w number of warm-up iterations (10000) -c set affinity to this CPU (off) -O maximal number of uncompleted outstanding sends (1) -i distance between consecutive scatter-gather entries (0) -l use loopback connection, in this case, the process will communicate with itself, so passing server hostname is not allowed -T number of threads in the test (1), if >1 implies "-M multi" for UCP -B register memory with NONBLOCK flag -b read and execute tests from a batch file: every line in the file is a test to run, first word is test name, the rest of the line is command-line arguments for the test. -p TCP port to use for data exchange (13337) -P <0|1> disable/enable MPI mode (0) ``` -------------------------------- ### Display UCX Device Information Source: https://github.com/openucx/ucx/blob/master/contrib/ibmock/README.md Displays information about the configured UCX devices, useful for verifying the IB mock setup. ```shell ucx_info -d ``` -------------------------------- ### Configure MPICH with UCX Source: https://github.com/openucx/ucx/wiki/MPICH-Installation-with-UCX Configure MPICH to use UCX as the communication device. Ensure you have a build directory and specify the installation path. ```bash $ mkdir build $ cd build $ ../configure --prefix=/your_install_path/ --with-device=ch4:ucx ``` -------------------------------- ### Output with Large BAR Disabled Source: https://github.com/openucx/ucx/wiki/Build-and-run-ROCm-UCX-OpenMPI Example output from the Large BAR sanity check when the setting is not enabled, resulting in a segmentation fault. ```bash $ ./check_large_bar address buf 0x7efa41c00000 Segmentation fault ``` -------------------------------- ### Download Updated config.guess for OpenPOWER Source: https://github.com/openucx/ucx/wiki/OpenMPI-and-OpenSHMEM-installation-with-UCX Download an updated config.guess file for OpenPOWER platforms if the default one is outdated. ```bash wget https://github.com/shamisp/ucx/raw/topic/power8-config/config.guess ```