ROMIO

https://ftp.mcs.anl.gov/pub/romio/users-guide
Admin
ROMIO is a high-performance, portable MPI-IO implementation that provides parallel I/O capabilities...
Tokens:8,987
Snippets:87
Trust Score:9.5
Update:8 months ago
Show doc for...
Tokens:
Raw
### Configure ROMIO with PVFS Support

Source: https://ftp.mcs.anl.gov/pub/romio/users-guide/node29

This example shows how to configure the ROMIO library to support the PVFS parallel file system. It involves specifying the file system type during configuration and providing paths to PVFS include files and libraries.

```shell
configure -cflags=-I/usr/pvfs/include -file_system=pvfs+ufs+nfs
```

--------------------------------

### Configure NFS Mount with 'noac' Option

Source: https://ftp.mcs.anl.gov/pub/romio/users-guide/node12

This section provides instructions on how to edit the `/etc/fstab` file to include the `noac` (no attribute caching) option for NFS mounts. It shows an example of a correct fstab entry and the commands to remount the directory to apply the changes. This is crucial for correct ROMIO behavior on NFS.

```shell
>2. then edit /etc/fstab for each nfs directory read/written by MPIO
>   on each  machine used for multihost MPIO.
>
>    Here is an example of a correct fstab entry for /epm1:
>
>   ie grep epm1 /etc/fstab
>
>      ROOOOT 11>grep epm1 /etc/fstab
>      gershwin:/epm1 /rmt/gershwin/epm1 nfs bg,intr,noac 0 0
>
>   if the noac option is not present, add it 
>   and then remount this directory
>   on each of the machines that will be used to share MPIO files
>
>ie
>
>ROOOOT >umount /rmt/gershwin/epm1
>ROOOOT >mount  /rmt/gershwin/epm1
>
```

--------------------------------

### Verify 'noac' Mount Option

Source: https://ftp.mcs.anl.gov/pub/romio/users-guide/node12

This command verifies that the NFS directory has been successfully mounted with the `noac` option. It uses `grep` on `/etc/mnttab` to display the mount options for a specific NFS share. This confirms the configuration required for ROMIO on NFS.

```shell
>3. Confirm that the directory is mounted noac:
>
>ROOOOT >grep gershwin /etc/mnttab 
>gershwin:/epm1 /rmt/gershwin/epm1 nfs
>noac,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0 0 0 899911504
>
```

--------------------------------

### Compile MPI-IO Programs with Custom MPI Implementation

Source: https://ftp.mcs.anl.gov/pub/romio/users-guide/node20

Compile C or Fortran MPI-IO programs when using an MPI implementation other than the standard ones. This requires explicitly providing the path to the mpio.h/mpiof.h include files and the libmpio.a library.

```shell
cc foo.c -I$(ROMIO_HOME)/include -L$(ROMIO_HOME)/lib/$(ARCH) -lmpio
```

```shell
f77 foo.f -I$(ROMIO_HOME)/include -L$(ROMIO_HOME)/lib/$(ARCH) -lmpio
```

```shell
f90 foo.f -I$(ROMIO_HOME)/include -L$(ROMIO_HOME)/lib/$(ARCH) -lmpio
```

--------------------------------

### Verify NFS Version 3 and Lockd Daemon

Source: https://ftp.mcs.anl.gov/pub/romio/users-guide/node12

This snippet demonstrates how to verify if the system is running NFS version 3 and if the lockd daemon is active. It uses `rpcinfo` to query RPC services and `grep` to filter for NFS-related entries. This is a prerequisite for using ROMIO on NFS.

```shell
>1. first confirm you are running nfs version 3
>
>rpcnfo -p `hostname` | grep nfs
>
>ie 
>
>    goedel >rpcinfo -p goedel | grep nfs
>    100003    2   udp   2049  nfs
>    100003    3   udp   2049  nfs
>
```

--------------------------------

### Run MPI-IO Programs

Source: https://ftp.mcs.anl.gov/pub/romio/users-guide/node20

Execute compiled MPI-IO programs using the appropriate mpirun command for your specific MPI implementation. It is crucial to use the mpirun associated with the MPI library used for compilation to ensure correct execution.

```shell
mpirun -np <num_procs> ./your_program
```

--------------------------------

### Build ROMIO in a Separate Directory using Virtual Paths

Source: https://ftp.mcs.anl.gov/pub/romio/users-guide/node29

This demonstrates how to use virtual paths with the `make` utility to compile ROMIO in a directory separate from the source distribution. This is useful for managing build artifacts, especially when the source is on a slower file system. GNU `make` is an example utility that supports this feature.

```shell
cd /tmp/thakur
/home/thakur/romio/configure
make
```

--------------------------------

### Compile MPI-IO Programs with LAM MPI

Source: https://ftp.mcs.anl.gov/pub/romio/users-guide/node20

Compile C or Fortran MPI-IO programs using LAM MPI compilers (hcc, hf77) and explicitly link the MPI library using the -lmpi flag. This method is for the LAM MPI implementation.

```shell
hcc foo.c -lmpi
```

```shell
hf77 foo.f -lmpi
```

--------------------------------

### Compile MPI-IO Programs with SGI MPI

Source: https://ftp.mcs.anl.gov/pub/romio/users-guide/node20

Compile C or Fortran MPI-IO programs using SGI MPI compilers (cc, f77, f90) and explicitly link the MPI library using the -lmpi flag. This is specific to SGI's MPI implementation.

```shell
cc foo.c -lmpi
```

```shell
f77 foo.f -lmpi
```

```shell
f90 foo.f -lmpi
```

--------------------------------

### Compile MPI-IO Programs with MPICH, HP MPI, or NEC MPI

Source: https://ftp.mcs.anl.gov/pub/romio/users-guide/node20

Compile C or Fortran MPI-IO programs using the standard mpicc or mpifort compilers provided by MPICH, HP MPI, or NEC MPI. These compilers handle the necessary include paths and library linking for MPI-IO.

```shell
mpicc foo.c
```

```shell
mpifort foo.f
```

--------------------------------

### ROMIO cb_config_list Hint Syntax

Source: https://ftp.mcs.anl.gov/pub/romio/users-guide/node6

This hint provides explicit control over aggregators for collective I/O. It specifies a comma-separated list of host specifications, where each host specification can include an optional limit on the number of processes to be used on that node.

```bnf
cb_config_list => hostspec [ ',' cb_config_list ]
hostspec => hostname [ ':' maxprocesses ]
hostname => <alphanumeric string> | '*'
maxprocesses => <digits> | '*'
```

--------------------------------

### ROMIO Hints File Configuration

Source: https://ftp.mcs.anl.gov/pub/romio/users-guide/node11

This snippet demonstrates the format of a ROMIO hints file. It shows how to add comments using '#' and set hint values like 'cb_nodes' and 'cb_config_list'. This file allows for ROMIO-specific configuration that can be overridden by programs.

```plaintext
# this is a comment describing the following setting
cb_nodes 32
# these nodes happen to have the best connection to storage
cb_config_list n01,n11,n21,n31,n41
```

--------------------------------

### ROMIO Collective Buffering Hints

Source: https://ftp.mcs.anl.gov/pub/romio/users-guide/node6

These hints manage the two-phase collective buffering optimization. They allow control over buffer sizes, the number of aggregators, and when collective buffering is applied to read and write operations. The 'romio_no_indep_rw' hint influences deferred open behavior.

```text
cb_buffer_size: Controls the size (in bytes) of the intermediate buffer for two-phase collective I/O. Default is 4194304 (4 Mbytes).
cb_nodes: Controls the maximum number of aggregators to be used. Default is the number of unique hosts in the communicator.
romio_cb_read: Controls collective buffering for collective read operations. Valid values: 'enable', 'disable', 'automatic'. Default: 'automatic'.
romio_cb_write: Controls collective buffering for collective write operations. Valid values: 'enable', 'disable', 'automatic'. Default: 'automatic'.
romio_no_indep_rw: Controls deferred open usage. When 'true', ROMIO avoids file operations on non-aggregator nodes, expecting only collective operations.
```

--------------------------------

### Enable Direct I/O using Environment Variables

Source: https://ftp.mcs.anl.gov/pub/romio/users-guide/node29

This snippet demonstrates how to enable direct I/O for reading and writing operations in ROMIO using environment variables. Direct I/O bypasses the file-system cache for potentially improved performance on high-bandwidth systems. These variables take precedence over MPI hints.

```shell
setenv MPIO_DIRECT_READ TRUE
setenv MPIO_DIRECT_WRITE TRUE
```

--------------------------------

### ROMIO Data Sieving Hints

Source: https://ftp.mcs.anl.gov/pub/romio/users-guide/node6

These hints control the data sieving optimization for read and write operations. They manage buffer sizes and enable/disable the sieving mechanism based on specified values or automatic heuristics. The default values are provided for each hint.

```text
ind_rd_buffer_size: Controls the size (in bytes) of the intermediate buffer used by ROMIO during data sieving read operations. Default is 4194304 (4 Mbytes).
ind_wr_buffer_size: Controls the size (in bytes) of the intermediate buffer used by ROMIO during data sieving write operations. Default is 524288 (512 Kbytes).
romio_ds_read: Determines when ROMIO performs data sieving for reads. Valid values: 'enable', 'disable', 'automatic'. Default: 'automatic'.
romio_ds_write: Determines when ROMIO performs data sieving for writes. Valid values: 'enable', 'disable', 'automatic'. Default: 'automatic'.
```

--------------------------------

### Specify GNU Make for ROMIO Configuration

Source: https://ftp.mcs.anl.gov/pub/romio/users-guide/node29

This command shows how to explicitly tell the ROMIO `configure` script to use a specific `make` utility, such as GNU `make`, if the default `make` does not support virtual paths.

```shell
/home/thakur/romio/configure -make=/usr/gnu/bin/gmake
```

--------------------------------

### Configure ROMIO with Large File Support Flags for Linux

Source: https://ftp.mcs.anl.gov/pub/romio/users-guide/node18

This snippet shows the command to configure ROMIO on Linux systems to enable support for files larger than 2 GBytes. It uses the flags `-D_FILE_OFFSET_BITS=64` and `-D_LARGEFILE64_SOURCE` to inform the GNU libc about the requirement for 64-bit file offsets and large file functions. This is typically used when automatic detection fails.

```bash
./configure -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE
```
Tokens:
Raw