### Build OP2 and Example Application

Source: https://op2-dsl.readthedocs.io/en/latest/getting_started

Builds the OP2 libraries and an example application ('airfoil_plain/dp') using parallel execution. The '-j$(nproc)' flag utilizes all available processor cores for faster compilation.

```bash
make -C op2 -j$(nproc)
make -C apps/c/airfoil/airfoil_plain/dp -j$(nproc)
```

--------------------------------

### Clone OP2 Repository

Source: https://op2-dsl.readthedocs.io/en/latest/getting_started

Clones the OP2-DSL repository from GitHub and navigates into the project directory. This is the initial step for any manual build.

```bash
git clone https://github.com/OP-DSL/OP2-Common.git
cd OP2-Common
```

--------------------------------

### Specify Library Installation Paths

Source: https://op2-dsl.readthedocs.io/en/latest/getting_started

Sets environment variables for the installation paths of optional library dependencies such as PT-Scotch, ParMETIS, KaHIP, HDF5, and CUDA. These paths are used by the build system to locate necessary libraries and headers.

```bash
export PTSCOTCH_INSTALL_PATH=<path/to/ptscotch>
export PARMETIS_INSTALL_PATH=<path/to/parmetis>
export KAHIP_INSTALL_PATH=<path/to/kahip>
export HDF5_{SEQ, PAR}_INSTALL_PATH=<path/to/hdf5>

export CUDA_INSTALL_PATH=<path/to/cuda/toolkit>
```

--------------------------------

### Configure OP2 Detailed Compiler Options

Source: https://op2-dsl.readthedocs.io/en/latest/getting_started

Provides more granular control over compiler selection for C, C++/CUDA, and Fortran. This allows specifying individual compiler types for different language frontends.

```bash
export OP2_C_COMPILER={gnu, clang, cray, intel, xl, nvhpc}
export OP2_C_CUDA_COMPILER={nvhpc}
export OP2_F_COMPILER={gnu, cray, intel, xl, nvhpc}
```

--------------------------------

### Generate OP2 Build Configuration

Source: https://op2-dsl.readthedocs.io/en/latest/getting_started

Generates the build configuration for OP2 by running the 'config' target in the 'op2' subdirectory. This step verifies compiler and library settings before proceeding with the actual build.

```bash
make -C op2 config
```

--------------------------------

### Configure OP2 Compiler

Source: https://op2-dsl.readthedocs.io/en/latest/getting_started

Sets the environment variable OP2_COMPILER to specify the desired C/C++ compiler toolchain for building OP2. Supported options include gnu, cray, intel, xl, and nvhpc.

```bash
export OP2_COMPILER={gnu, cray, intel, xl, nvhpc}
```

--------------------------------

### Specify Library Installation Paths

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/getting_started

Sets environment variables to point to the installation directories of optional library dependencies such as PT-Scotch, ParMETIS, KaHIP, and HDF5. These paths are crucial for the build system to locate the necessary libraries.

```shell
export PTSCOTCH_INSTALL_PATH=<path/to/ptscotch>
export PARMETIS_INSTALL_PATH=<path/to/parmetis>
export KAHIP_INSTALL_PATH=<path/to/kahip>
export HDF5_{SEQ, PAR}_INSTALL_PATH=<path/to/hdf5>
```

--------------------------------

### Configure CUDA Target Architectures

Source: https://op2-dsl.readthedocs.io/en/latest/getting_started

Specifies a comma-separated list of target GPU architectures for which to generate CUDA code when using the NVIDIA HPC SDK (NVHPC) compiler. This is used to optimize for specific hardware.

```bash
export NV_ARCH={Fermi, Kepler, ..., Ampere}[,{Fermi, ...}]
```

--------------------------------

### Generate OpenACC Code with OP2

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

These commands demonstrate running the OP2 executable with OpenACC code generation. The first example runs on a single GPU, while the second example uses MPI to distribute the workload across 4 processes, each with its own GPU. The OP_PART_SIZE and OP_BLOCK_SIZE parameters control the partition and block sizes, respectively.

```bash
#On a single GPU
./airfoil_openacc OP_PART_SIZE=128 OP_BLOCK_SIZE=192
#On 4 mpi procs, each proc having a GPU
$MPI_INSTALL_PATH/bin/mpirun -np 4 ./airfoil_mpi_openacc OP_PART_SIZE=128 OP_BLOCK_SIZE=192
```

--------------------------------

### Execute Developer Sequential and MPI Versions

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

These bash commands demonstrate how to execute the developer versions of the Airfoil application. The first command runs the sequential version, and the second runs the distributed memory MPI version using mpirun on 4 processes. Ensure the MPI installation path is correctly set.

```bash
#developer sequential
./aifroil_seq

#developer distributed memory with mpi, on 4 mpi procs
$MPI_INSTAL_PATH/bin/mpirun -np 4 ./airfoil_mpi_seq

```

--------------------------------

### op_par_loop Example for Direct Loop

Source: https://op2-dsl.readthedocs.io/en/latest/devapp

This example shows how to convert a direct loop that saves the solution from one time iteration to the previous one using the op_par_loop API. It involves outlining the loop body into an elemental kernel and defining the data arguments.

```APIDOC
## POST /op_par_loop

### Description
This endpoint demonstrates the usage of `op_par_loop` to parallelize a direct loop. A direct loop is one where all data accessed is defined on the set the loop iterates over. This example converts a C-style loop that copies `q` to `q_old` into an OP2 parallel loop.

### Method
`op_par_loop` (conceptual representation, not a standard HTTP method)

### Endpoint
`op_par_loop(kernel, name, set, arg1, arg2, ...)`

### Parameters
#### Routine Arguments
- **kernel**: (subroutine) - The outlined elemental kernel function (e.g., `save_soln`).
- **name**: (string) - A descriptive name for the loop (e.g., "save_soln").
- **set**: (op_set) - The iteration set (e.g., `cells`).

#### Data Arguments (`op_arg_dat`)
- **`op_arg_dat(dat, map_index, rule, dim, type, access)`**: Describes data dependencies.
  - **dat**: (op_dat) - The name of the OP2 data set (e.g., `p_q`, `p_qold`).
  - **map_index**: (integer) - Typically -1 for direct access.
  - **rule**: (op_map_rule) - Rule for mapping (e.g., `OP_ID` for direct access).
  - **dim**: (integer) - The dimension of the data per mesh point (e.g., 4 for 4 doubles).
  - **type**: (string) - Data type (e.g., "double").
  - **access**: (op_access) - Data access mode (`OP_READ`, `OP_WRITE`, `OP_RW`).

### Request Example
```c
// Elemental kernel
inline void save_soln(double *q, double *qold) {
  for (int n = 0; n < 4; n++)
    qold[n] = q[n];
}

// OP2 parallel loop call
op_par_loop(save_soln, "save_soln", cells,
            op_arg_dat(p_q,    -1, OP_ID, 4, "double", OP_READ ),
            op_arg_dat(p_qold, -1, OP_ID, 4, "double", OP_WRITE));
```

### Response
#### Success Response
Execution of the parallel loop completes without errors. The results are stored in the specified data sets according to the access modes.

#### Response Example
(No direct response body, execution is side-effecting on data sets)
```

--------------------------------

### Set MPI Installation Path

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/getting_started

Manually sets the path to the MPI executables using the MPI_INSTALL_PATH environment variable. This is useful when the MPI wrapper (e.g., mpicc) is not automatically found by the build system, ensuring correct MPI compilation for OP2.

```shell
export MPI_INSTALL_PATH=<path/to/mpi/executables>
```

--------------------------------

### Run Hybrid CPU/GPU Execution with OP2 MPI CUDA

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

This command executes the OP2 hybrid MPI CUDA version using 12 MPI processes. It demonstrates how to allocate processes to both GPUs and CPU cores for parallel execution. Ensure MPI and the OP2 executable are correctly installed and configured.

```bash
#On 12 mpi procs
$MPI_INSTALL_PATH/bin/mpirun -np 12 ./airfoil_mpi_cuda_hyb OP_PART_SIZE=128 OP_BLOCK_SIZE=192
```

--------------------------------

### Specify CUDA Installation Path and Target Architectures

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/getting_started

Sets the environment variable CUDA_INSTALL_PATH to the CUDA toolkit directory and optionally NV_ARCH for specifying target GPU architectures when building with CUDA support. This is used for CUDA-enabled builds.

```shell
export CUDA_INSTALL_PATH=<path/to/cuda/toolkit>
export NV_ARCH={Fermi, Kepler, ..., Ampere}[,{Fermi, ...}]
```

--------------------------------

### Sequential Loop to Save Solution (C)

Source: https://op2-dsl.readthedocs.io/en/latest/devapp

This C code snippet demonstrates a basic sequential loop that iterates over cells to save the previous time-iteration's solution (`q`) to `q_old`. It serves as the starting point for optimization using the OP2 API.

```c
//save_soln : iterates over cells
for (int iteration = 0; iteration < (ncell * 4); ++iteration) {
  qold[iteration] = q[iteration];
}

```

--------------------------------

### Execute Code-Generated CUDA Versions

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

These bash commands show the execution of code-generated CUDA versions of the Airfoil application, including MPI hybrid versions. They demonstrate running on a single GPU with specified partition and block sizes, and on 4 MPI processes, each with a GPU. Ensure CUDA and MPI are correctly installed and configured.

```bash
#On a single GPU
./airfoil_cuda OP_PART_SIZE=128 OP_BLOCK_SIZE=192
#On 4 mpi procs, each proc having a GPU
$MPI_INSTALL_PATH/bin/mpirun -np 4 ./airfoil_mpi_cuda OP_PART_SIZE=128 OP_BLOCK_SIZE=192

```

--------------------------------

### Execute MPI + Code-Generated Hybrid CUDA Versions

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

This bash command executes the MPI combined with code-generated hybrid CUDA versions of the Airfoil application. It specifies partition and block sizes for the CUDA execution across multiple MPI processes, assuming each process has access to a GPU. The command targets a distributed memory setup with CUDA acceleration.

```bash
# MPI + Code-gen hybrid CUDA with mini-partition size of 128 and CUDA thread-block size of 192

```

--------------------------------

### Initialize and Finalize OP2 Library (C)

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

Initializes the OP2 library with runtime arguments and sets the diagnostics level. It also includes the necessary header file and finalizes the library at the end of the application. Ensure OP2 is set up correctly, including Makefile configurations for include and library paths, and linking against the 'op2_seq' backend library.

```C
#include "op_seq.h"
...
...
int main(int argc, char **argv) {
  //Initialise the OP2 library, passing runtime args, and setting diagnostics level to low (1)
  op_init(argc, argv, 1);
  ...
  ...
  ...
  free(adt);
  free(res);

  //Finalising the OP2 library
  op_exit();
}
```

--------------------------------

### Initialization and Termination

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/api

Routines for initializing and terminating the OP2 runtime environment.

```APIDOC
## OP2 C/C++ API - Initialization and Termination

### Description
Functions to initialize and terminate the OP2 runtime.

### Functions

.. c:function:: void op_init(int argc, char **argv, int diags_level)

   This routine must be called before all other OP routines. Under MPI back-ends, this routine also calls :c:func:`MPI_Init()` unless its already called previously.

   :param argc: The number of command line arguments.
   :param argv: The command line arguments, as passed to :c:func:`main()`.
   :param diags_level: Determines the level of debugging diagnostics and reporting to be performed.

   The values for **diags_level** are as follows:

   - :c:expr:`0`: None.
   - :c:expr:`1`: Error-checking.
   - :c:expr:`2`: Info on plan construction.
   - :c:expr:`3`: Report execution of parallel loops.
   - :c:expr:`4`: Report use of old plans.
   - :c:expr:`7`: Report positive checks in :c:func:`op_plan_check()`

.. c:function:: void op_exit()

   This routine must be called last to cleanly terminate the OP2 runtime. Under MPI back-ends, this routine also calls :c:func:`MPI_Finalize()` unless its has been called previously. A runtime error will occur if :c:func:`MPI_Finalize()` is called after :c:func:`op_exit()`.
```

--------------------------------

### Initialize OP2 and Output Diagnostics (C)

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

Initializes the OP2 library with runtime arguments and a specified diagnostics level. It then outputs detailed mesh information using the op_diagnostic_output() API. This is useful for debugging and understanding the declared mesh structure.

```C
/* main application */
int main(int argc, char **argv) {
  //Initialise the OP2 library, passing runtime args, and setting diagnostics level to low (1)
  op_init(argc, argv, 2);
  ...
  ... op_decl_set ...
  ... op_decl_map ...
  ... op_decl_dat ...
  ... op_decl_const  ...
  ...
  //output mesh information
  op_diagnostic_output();

```

--------------------------------

### Initialization and Termination Routines

Source: https://op2-dsl.readthedocs.io/en/latest/api

Routines for initializing and terminating the OP2 runtime environment, including MPI integration.

```APIDOC
## `op_init`

### Description
Initializes the OP2 runtime. This routine must be called before all other OP routines. It also handles MPI initialization if the MPI back-end is used and MPI has not been initialized previously.

### Method
`void op_init(int argc, char **argv, int diags_level)`

### Parameters
#### Path Parameters
- **argc** (int) - Required - The number of command line arguments.
- **argv** (char **) - Required - The command line arguments, as passed to `main()`.
- **diags_level** (int) - Required - Determines the level of debugging diagnostics and reporting. 
  - 0: None.
  - 1: Error-checking.
  - 2: Info on plan construction.
  - 3: Report execution of parallel loops.
  - 4: Report use of old plans.
  - 7: Report positive checks in `op_plan_check()`

### Response
#### Success Response (void)
This function does not return a value.

## `op_exit`

### Description
Terminates the OP2 runtime cleanly. This routine must be called last. It also handles MPI finalization if the MPI back-end is used and MPI has not been finalized previously. Calling `MPI_Finalize()` after `op_exit()` will result in a runtime error.

### Method
`void op_exit()`

### Parameters
This function does not take any parameters.

### Response
#### Success Response (void)
This function does not return a value.
```

--------------------------------

### OP2 Get Global Size for Sets

Source: https://op2-dsl.readthedocs.io/en/latest/devapp

Retrieves the global size of a given set, which is necessary for certain calculations like validating application behavior over iterations. This API call is used after the mesh has been read via HDF5.

```c
//get global number of cells
ncell = op_get_size(cells);

```

--------------------------------

### Declare Maps using op_decl_map

Source: https://op2-dsl.readthedocs.io/en/latest/devapp

Declares mappings between different mesh sets, representing connectivity information. For example, mapping edges to nodes or cells. Requires the names of the two sets, arity, mapping data type, and a string name.

```c
//declare maps
op_map pedge   = op_decl_map(edges,  nodes, 2, edge,   "pedge"  );
op_map pecell  = op_decl_map(edges,  cells, 2, ecell,  "pecell" );
op_map pbedge  = op_decl_map(bedges, nodes, 2, bedge,  "pbedge" );
op_map pbecell = op_decl_map(bedges, cells, 1, becell, "pbecell");
op_map pcell   = op_decl_map(cells,  nodes, 4, cell,   "pcell"  );

```

--------------------------------

### Initialize and Finalize OP2 Library

Source: https://op2-dsl.readthedocs.io/en/latest/devapp

This C code snippet demonstrates how to include the OP2 header file, initialize the OP2 library with command-line arguments and a diagnostics level, and finalize the library upon application exit. It assumes necessary memory allocations and deallocations are handled elsewhere.

```c
#include "op_seq.h"

int main(int argc, char **argv) {
  //Initialise the OP2 library, passing runtime args, and setting diagnostics level to low (1)
  op_init(argc, argv, 1);
  
  // ... other code ...
  
  free(adt);
  free(res);

  //Finalising the OP2 library
  op_exit();
}
```

--------------------------------

### Initialize OP2 and Output Diagnostics

Source: https://op2-dsl.readthedocs.io/en/latest/devapp

Initializes the OP2 library with runtime arguments and sets a diagnostics level. It also demonstrates calling a function to output detailed information about the declared mesh structure.

```c
/* main application */
int main(int argc, char **argv) {
//Initialise the OP2 library, passing runtime args, and setting diagnostics level to low (1)
op_init(argc, argv, 2);
...
...
//output mesh information
op_diagnostic_output();

}

```

--------------------------------

### Direct Loop Conversion to OP2 API (C)

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

Demonstrates converting a sequential 'save_soln' loop, which iterates over cells and copies data from 'q' to 'q_old', into an OP2 parallel loop. It shows outlining the loop body into an elemental kernel and then using the op_par_loop API with appropriate arguments for data access and iteration sets.

```C
//save_soln : iterates over cells
for (int iteration = 0; iteration < (ncell * 4); ++iteration) {
  qold[iteration] = q[iteration];
}

//outlined elemental kernel
inline void save_soln(double *q, double *qold) {
  for (int n = 0; n < 4; n++)
    qold[n] = q[n];
}

//save_soln : iterates over cells
for (int iteration = 0; iteration < (ncell * 4); ++iteration) {
  save_soln(&q[iteration], &qold[iteration]);
}

op_par_loop(save_soln, "save_soln", cells,
              op_arg_dat(p_q,    -1, OP_ID, 4, "double", OP_READ ),
              op_arg_dat(p_qold, -1, OP_ID, 4, "double", OP_WRITE));

```

--------------------------------

### Indirect Loop Example: adt_calc (C)

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

Presents the 'adt_calc' loop, which iterates over cells to calculate area/timstep. This code snippet illustrates an indirect loop where data on connected nodes ('x') is accessed via a cells-to-nodes mapping, while data on cells ('adt') is accessed directly.

```C
//adt_calc - calculate area/timstep : iterates over cells
for (int iteration = 0; iteration < ncell; ++iteration) {
  int map1idx = cell[iteration * 4 + 0];
  int map2idx = cell[iteration * 4 + 1];
  int map3idx = cell[iteration * 4 + 2];
  int map4idx = cell[iteration * 4 + 3];

  double dx, dy, ri, u, v, c;

  ri = 1.0f / q[4 * iteration + 0];
  u = ri * q[4 * iteration + 1];
  v = ri * q[4 * iteration + 2];
  c = sqrt(gam * gm1 * (ri * q[4 * iteration + 3] - 0.5f * (u * u + v * v)));

  dx = x[2 * map2idx + 0] - x[2 * map1idx + 0];
  dy = x[2 * map2idx + 1] - x[2 * map1idx + 1];
  adt[iteration] = fabs(u * dy - v * dx) + c * sqrt(dx * dx + dy * dy);

  dx = x[2 * map3idx + 0] - x[2 * map2idx + 0];
  dy = x[2 * map3idx + 1] - x[2 * map2idx + 1];
  adt[iteration] += fabs(u * dy - v * dx) + c * sqrt(dx * dx + dy * dy);

  dx = x[2 * map4idx + 0] - x[2 * map3idx + 0];
  dy = x[2 * map4idx + 1] - x[2 * map3idx + 1];
  adt[iteration] += fabs(u * dy - v * dx) + c * sqrt(dx * dx + dy * dy);
}

```

--------------------------------

### Makefile for OP2 Application Compilation

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

This Makefile is used to compile the Airfoil application with OP2. It specifies the application name, entry point, and enables the use of HDF5 libraries. The 'common.mk' and 'c_app.mk' files are included for common build rules and C application specifics, respectively.

```make
APP_NAME := airfoil_step7

APP_ENTRY := $(APP_NAME).cpp
APP_ENTRY_MPI := $(APP_ENTRY)

OP2_LIBS_WITH_HDF5 := true

include ../../../../../makefiles/common.mk
include ../../../../../makefiles/c_app.mk

```

--------------------------------

### C: Initialize Mesh Data from File

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

This C code snippet demonstrates reading mesh data from a file. It opens a file, reads mesh dimensions (nodes, cells, edges, boundary edges), and then allocates memory and reads coordinates, cell connectivity, edge connectivity, and boundary edge information. Error handling is included for file operations and data reading.

```C
FILE *fp;
  if ((fp = fopen(FILE_NAME_PATH, "r")) == NULL) {
    printf("can't open file FILE_NAME_PATH\n");
    exit(-1);
  }
  if (fscanf(fp, "%d %d %d %d \n", &nnode, &ncell, &nedge, &nbedge) != 4) {
	printf("error reading from FILE_NAME_PATH\n");
	exit(-1);
  }

  cell   = (int *)malloc(4 * ncell  * sizeof(int));
  edge   = (int *)malloc(2 * nedge  * sizeof(int));
  ecell  = (int *)malloc(2 * nedge  * sizeof(int));
  bedge  = (int *)malloc(2 * nbedge * sizeof(int));
  becell = (int *)malloc(1 * nbedge * sizeof(int));
  bound  = (int *)malloc(1 * nbedge * sizeof(int));
  x      = (double *)malloc(2 * nnode * sizeof(double));
  q      = (double *)malloc(4 * ncell * sizeof(double));
  qold   = (double *)malloc(4 * ncell * sizeof(double));
  res    = (double *)malloc(4 * ncell * sizeof(double));
  adt    = (double *)malloc(1 * ncell * sizeof(double));

  for (int n = 0; n < nnode; n++) {
	if (fscanf(fp, "%lf %lf \n", &x[2 * n], &x[2 * n + 1]) != 2) {
  	  printf("error reading from FILE_NAME_PATH\n");
  	  exit(-1);
	}
  }

  for (int n = 0; n < ncell; n++) {
    if (fscanf(fp, "%d %d %d %d \n", &cell[4 * n], &cell[4 * n + 1],
        &cell[4 * n + 2], &cell[4 * n + 3]) != 4) {
      printf("error reading from FILE_NAME_PATH\n");
      exit(-1);
    }
  }
  for (int n = 0; n < nedge; n++) {
	if (fscanf(fp, "%d %d %d %d \n", &edge[2 * n], &edge[2 * n + 1],
        &ecell[2 * n], &ecell[2 * n + 1]) != 4) {
      printf("error reading from FILE_NAME_PATH\n");
  	  exit(-1);
    }
  }
  for (int n = 0; n < nbedge; n++) {
    if (fscanf(fp, "%d %d %d %d \n", &bedge[2 * n], &bedge[2 * n + 1],
        &becell[n], &bound[n]) != 4) {
    printf("error reading from FILE_NAME_PATH\n");
    exit(-1);
    }
  }
  fclose(fp);

```

--------------------------------

### Get Global Size of a Set using OP2 API in C

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

This C code snippet shows how to obtain the global number of cells in a set using OP2's `op_get_size()` API. This is essential for tasks like validating application results, such as computing RMS values, after the mesh has been processed and potentially partitioned.

```C
//get global number of cells
ncell = op_get_size(cells);
```

--------------------------------

### Partition Mesh and Create MPI Halos with OP2 in C

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

This C code snippet demonstrates the use of OP2's `op_partition()` function. This call signals to the MPI back-end that all mesh data has been defined, allowing for mesh partitioning and the creation of MPI halos. It's a critical step for enabling distributed memory parallel execution.

```C
...
...
op_decl_const(1, "double", &alpha);
op_decl_const(4, "double", qinf  );

//output mesh information
op_diagnostic_output();

//partition mesh and create mpi halos
op_partition("BLOCK", "ANY", edges, pecell, p_x);

...
...
```

--------------------------------

### Execute Code-Generated SIMD Vectorized Versions

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

These bash commands demonstrate how to execute the code-generated SIMD vectorized versions of the Airfoil application, both sequentially and with MPI. The first command runs the SIMD vectorized version, and the second runs it distributed across 4 MPI processes.

```bash
#SIMD vec
./aifroil_vec
#On 4 mpi procs with each proc running SIMD vec
$MPI_INSTAL_PATH/bin/mpirun -np 4 ./aifroil_mpi_vec

```

--------------------------------

### Execute Code-Generated Sequential and MPI Versions

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

These bash commands show the execution of code-generated sequential and MPI versions of the Airfoil application. The first command runs the code-generated sequential version, and the second runs the MPI version on 4 processes. These versions are produced after running the OP2 code generator.

```bash
# code-gen sequential
./aifroil_genseq
#On 4 mpi procs
$MPI_INSTAL_PATH/bin/mpirun -np 4 ./aifroil_mpi_genseq

```

--------------------------------

### OP2 Initialization and Termination Functions

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/api

These functions are crucial for managing the OP2 runtime environment. op_init must be called before any other OP2 routines, handling initialization and potentially MPI_Init. op_exit must be called last for clean termination, potentially calling MPI_Finalize.

```c
void op_init(int argc, char **argv, int diags_level)

void op_exit()
```

--------------------------------

### Compile and Execute with OP_NO_REALLOC Flag (Bash)

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

Compiles the step2 application and provides instructions for executing it using the OP_NO_REALLOC runtime flag. This flag instructs the OP2 back-end to reuse allocated memory, aiding in gradual conversion to the OP2 API.

```bash
./airfoil_step2 OP_NO_REALLOC

```

--------------------------------

### Outlined Elemental Kernel and Loop for OP2 (C)

Source: https://op2-dsl.readthedocs.io/en/latest/devapp

This C code demonstrates outlining the loop body into an elemental kernel subroutine (`save_soln`) and then modifying the main loop to call this subroutine. This is a preparatory step before using the `op_par_loop` API.

```c
//outlined elemental kernel
inline void save_soln(double *q, double *qold) {
for (int n = 0; n < 4; n++)
  qold[n] = q[n];
}

//save_soln : iterates over cells
for (int iteration = 0; iteration < (ncell * 4); ++iteration) {
  save_soln(&q[iteration], &qold[iteration]);
}

```

--------------------------------

### C: Loop for ADT Calculation Using Outlined Kernel

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

Iterates through each cell in the simulation, using the 'cell' array to map global indices to local cell vertex indices. For each cell, it calls the outlined 'adt_calc' kernel function to compute the area/timestep. Input data (x, q) and output data (adt) are passed as pointers.

```C
//adt_calc - calculate area/timstep : iterates over cells
  for (int iteration = 0; iteration < ncell; ++iteration) {
    int map1idx = cell[iteration * 4 + 0];
    int map2idx = cell[iteration * 4 + 1];
    int map3idx = cell[iteration * 4 + 2];
    int map4idx = cell[iteration * 4 + 3];

    adt_calc(&x[2 * map1idx], &x[2 * map2idx], &x[2 * map3idx],
             &x[2 * map4idx], &q[4 * iteration], &adt[iteration]);
  }
```

--------------------------------

### Initialize Mesh Data from File in C

Source: https://op2-dsl.readthedocs.io/en/latest/devapp

This C code snippet demonstrates reading mesh data from a file ('new_grid.dat'). It allocates memory for mesh components (cells, edges, boundary edges) and their connectivity, then populates these arrays by parsing the file. Error handling for file operations and data reading is included. It assumes FILE_NAME_PATH is defined elsewhere.

```c
FILE *fp;
if ((fp = fopen(FILE_NAME_NAME_PATH, "r")) == NULL) {
  printf("can't open file FILE_NAME_PATH\n");
  exit(-1);
}
if (fscanf(fp, "%d %d %d %d \n", &nnode, &ncell, &nedge, &nbedge) != 4) {
      printf("error reading from FILE_NAME_PATH\n");
      exit(-1);
}

cell   = (int *)malloc(4 * ncell  * sizeof(int));
edge   = (int *)malloc(2 * nedge  * sizeof(int));
ecell  = (int *)malloc(2 * nedge  * sizeof(int));
bedge  = (int *)malloc(2 * nbedge * sizeof(int));
becell = (int *)malloc(1 * nbedge * sizeof(int));
bound  = (int *)malloc(1 * nbedge * sizeof(int));
x      = (double *)malloc(2 * nnode * sizeof(double));
q      = (double *)malloc(4 * ncell * sizeof(double));
qold   = (double *)malloc(4 * ncell * sizeof(double));
res    = (double *)malloc(4 * ncell * sizeof(double));
adt    = (double *)malloc(1 * ncell * sizeof(double));

for (int n = 0; n < nnode; n++) {
      if (fscanf(fp, "%lf %lf \n", &x[2 * n], &x[2 * n + 1]) != 2) {
        printf("error reading from FILE_NAME_PATH\n");
        exit(-1);
      }
}

for (int n = 0; n < ncell; n++) {
  if (fscanf(fp, "%d %d %d %d \n", &cell[4 * n], &cell[4 * n + 1],
      &cell[4 * n + 2], &cell[4 * n + 3]) != 4) {
    printf("error reading from FILE_NAME_PATH\n");
    exit(-1);
  }
}
for (int n = 0; n < nedge; n++) {
      if (fscanf(fp, "%d %d %d %d \n", &edge[2 * n], &edge[2 * n + 1],
      &ecell[2 * n], &ecell[2 * n + 1]) != 4) {
    printf("error reading from FILE_NAME_PATH\n");
        exit(-1);
  }
}
for (int n = 0; n < nbedge; n++) {
  if (fscanf(fp, "%d %d %d %d \n", &bedge[2 * n], &bedge[2 * n + 1],
      &becell[n], &bound[n]) != 4) {
  printf("error reading from FILE_NAME_PATH\n");
  exit(-1);
  }
}
fclose(fp);

```

--------------------------------

### Declare Sets, Maps, Data, and Constants using HDF5 in C

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

This C code snippet demonstrates how to declare sets, maps, data, and global constants using OP2's HDF5 API. It assumes the mesh data is available in an HDF5 file. This is crucial for enabling distributed memory execution with MPI by reading mesh data directly from a file.

```C
// declare sets
op_set nodes  = op_decl_set_hdf5(file,  "nodes" );
op_set edges  = op_decl_set_hdf5(file,  "edges" );
op_set bedges = op_decl_set_hdf5(file, "bedges");
op_set cells  = op_decl_set_hdf5(file,  "cells" );

//declare maps
op_map pedge   = op_decl_map_hdf5(edges,  nodes, 2, file, "pedge"  );
op_map pecell  = op_decl_map_hdf5(edges,  cells, 2, file, "pecell" );
op_map pbedge  = op_decl_map_hdf5(bedges, nodes, 2, file, "pbedge" );
op_map pbecell = op_decl_map_hdf5(bedges, cells, 1, file, "pbecell");
op_map pcell   = op_decl_map_hdf5(cells,  nodes, 4, file, "pcell"  );

//declare data on sets
op_dat p_bound = op_decl_dat_hdf5(bedges, 1, "int",    file, "p_bound");
op_dat p_x     = op_decl_dat_hdf5(nodes,  2, "double", file, "p_x"    );
op_dat p_q     = op_decl_dat_hdf5(cells,  4, "double", file, "p_q"    );
op_dat p_qold  = op_decl_dat_hdf5(cells,  4, "double", file, "p_qold" );
op_dat p_adt   = op_decl_dat_hdf5(cells,  1, "double", file, "p_adt"  );
op_dat p_res   = op_decl_dat_hdf5(cells,  4, "double", file, "p_res"  );

//read and declare global constants
op_get_const_hdf5("gam",   1, "double", (char *)&gam,  file);
op_get_const_hdf5("gm1",   1, "double", (char *)&gm1,  file);
op_get_const_hdf5("cfl",   1, "double", (char *)&cfl,  file);
op_get_const_hdf5("eps",   1, "double", (char *)&eps,  file);
op_get_const_hdf5("alpha", 1, "double", (char *)&alpha,file);
op_get_const_hdf5("qinf",  4, "double", (char *)&qinf, file);

op_decl_const(1, "double", &gam  );
op_decl_const(1, "double", &gm1  );
op_decl_const(1, "double", &cfl  );
op_decl_const(1, "double", &eps  );
op_decl_const(1, "double", &alpha);
op_decl_const(4, "double", qinf  );
```

--------------------------------

### Include Elemental Kernels in C++ Application

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

This C++ code snippet shows how elemental kernels are included as header files in the main application file. This is a preparatory step before code generation to ensure the main application does not contain duplicate kernel definitions. It assumes header files like 'adt_calc.h', 'bres_calc.h', etc., are available.

```C++
  /* Global Constants */
  double gam, gm1, cfl, eps, mach, alpha, qinf[4];

  //
  // kernel routines for parallel loops
  //
  #include "adt_calc.h"
  #include "bres_calc.h"
  #include "res_calc.h"
  #include "save_soln.h"
  #include "update.h"

```

--------------------------------

### C: Loop for Update Using Outlined Kernel

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

Iterates over each cell to update the flow field and compute a part of the global RMS. It calls the 'update' kernel for each cell, passing pointers to the old flow state (qold), current flow state (q), residuals (res), cell-specific adt, and the global RMS accumulator.

```C
//update = update flow field - iterates over cells
  rms = 0.0f;
  for (int iteration = 0; iteration < ncell; ++iteration) {
    update(&qold[iteration * 4], &q[iteration * 4], &res[iteration * 4],
           &adt[iteration], &rms);
  }
```

--------------------------------

### Partitioning

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/api

Routine for controlling the partitioning of sets for distributed memory parallel execution.

```APIDOC
## OP2 C/C++ API - Partitioning

### Description
Controls the partitioning of sets for distributed memory parallel execution.

### Function

.. c:function:: void op_partition(char *lib_name, char *lib_routine, op_set prime_set, op_map prime_map, op_dat coords)

   :param lib_name: The partitioning library to use, see below.
   :param lib_routine: The partitioning algorithm to use. Required if using :c:expr:`"PTSCOTCH"`, :c:expr:`"PARMETIS"` or https://kahip.github.io/ as the **lib_name**.
   :param prime_set: Specifies the set to be partitioned.
   :param prime_map: Specifies the map to be used to create adjacency lists for the **prime_set**. Required if using :c:expr:`"KWAY"` or :c:expr:`"GEOMKWAY"`.
   :param coords: Specifies the geometric coordinates of the **prime_set**. Required if using :c:expr:`"GEOM"` or :c:expr:`"GEOMKWAY"`.

   The current options for **lib_name** are:

   - :c:expr:`"PTSCOTCH"`: The `PT-Scotch <https://www.labri.fr/perso/pelegrin/scotch/>`_ library.
   - :c:expr:`"PARMETIS"`: The `ParMETIS <http://glaros.dtc.umn.edu/gkhome/metis/parmetis/overview>`_ library.
   - :c:expr:`"KAHIP"`: The `KaHIP <https://kahip.github.io/>`_ library.
   - :c:expr:`"INERTIAL"`: Internal 3D recursive inertial bisection partitioning.
   - :c:expr:`"EXTERNAL"`: External partitioning optionally read in when using HDF5 I/O.
   - :c:expr:`"RANDOM"`: Random partitioning, intended for debugging purposes.

   The options for **lib_routine** when using :c:expr:`"PTSCOTCH"` or :c:expr:`"KAHIP"` are:

   - :c:expr:`"KWAY"`: K-way graph partitioning.

   The options for **lib_routine** when using :c:expr:`"PARMETIS"` are:

   - :c:expr:`"KWAY"`: K-way graph partitioning.
   - :c:expr:`"GEOM"`: Geometric graph partitioning.
   - :c:expr:`"GEOMKWAY"`: Geometric followed by k-way graph partitioning.
```

--------------------------------

### Execute Code-Generated OpenMP Versions

Source: https://op2-dsl.readthedocs.io/en/latest/_sources/devapp

These bash commands illustrate the execution of code-generated OpenMP versions of the Airfoil application, both standalone and with MPI. The first command runs on 4 OpenMP threads, while the second runs on 4 MPI processes, each utilizing 8 OpenMP threads. The OP_PART_SIZE environment variable can be set for partitioning.

```bash
# on 4 OMP threads
export OMP_NUM_THREADS=4; ./aifroil_openmp OP_PART_SIZE=256
#On 4 mpi procs with each proc running 8 OpenMP threads
export OMP_NUM_THREADS=8; $MPI_INSTAL_PATH/bin/mpirun -np 4 ./aifroil_mpi_openmp with mini-partition size of 256

```

--------------------------------

### Constant and Dataset Declaration

Source: https://op2-dsl.readthedocs.io/en/latest/api

Routines for declaring constant data and datasets.

```APIDOC
## `op_decl_const`

### Description
Declares constant data with global scope that can be used in kernel functions.

### Method
`void op_decl_const(int dim, char *type, T *dat)`

### Parameters
#### Path Parameters
- **dim** (int) - Required - Number of data elements. For maximum efficiency this should be an integer literal.
- **type** (char *) - Required - The type of the data as a string (e.g., "float", "double", "int").
- **dat** (T *) - Required - A pointer to the data.

### Response
#### Success Response (void)
This function does not return a value.

## `op_decl_dat`

### Description
Declares a dataset associated with a set.

### Method
`op_dat op_decl_dat(op_set set, int dim, char *type, T *data, char *name)`

### Parameters
#### Path Parameters
- **set** (op_set) - Required - The set the data is associated with.
- **dim** (int) - Required - Number of data elements per set element. Must be an integer literal at present.
- **type** (char *) - Required - The datatype as a string (e.g., "float", "double"). A qualifier may be added for data layout.
- **data** (T *) - Required - Input data of type `T`, provided in AoS form.
- **name** (char *) - Required - A name for the dataset, used for output diagnostics.

### Response
#### Success Response (op_dat)
- **op_dat** (op_dat) - A handle representing the declared dataset.

## `op_decl_dat_temp`

### Description
Declares a temporary dataset, which can be released early using `op_free_dat_temp()`.

### Method
`op_dat op_decl_dat_temp(op_set set, int dim, char *type, T *data, char *name)`

### Parameters
#### Path Parameters
- **set** (op_set) - Required - The set the data is associated with.
- **dim** (int) - Required - Number of data elements per set element.
- **type** (char *) - Required - The datatype as a string.
- **data** (T *) - Required - Input data of type `T`.
- **name** (char *) - Required - A name for the dataset.

### Response
#### Success Response (op_dat)
- **op_dat** (op_dat) - A handle representing the declared temporary dataset.

## `op_free_dat_temp`

### Description
Releases a temporary dataset previously defined with `op_decl_dat_temp()`.

### Method
`void op_free_dat_temp(op_dat dat)`

### Parameters
#### Path Parameters
- **dat** (op_dat) - Required - The dataset to free.

### Response
#### Success Response (void)
This function does not return a value.
```

--------------------------------

### HDF5 I/O Routines

Source: https://op2-dsl.readthedocs.io/en/latest/api

Routines for declaring OP2 data structures by reading from HDF5 files.

```APIDOC
## op_decl_set_hdf5

### Description
Declares an OP2 set by reading its size from an HDF5 file.

### Method
OP2 Routine

### Endpoint
N/A (Function call)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```c
op_set my_set = op_decl_set_hdf5("input.h5", "set_name");
```

### Response
#### Success Response (200)
Returns an `op_set` identifier.

#### Response Example
```c
// op_set structure is internal to OP2
```
```

```APIDOC
## op_decl_map_hdf5

### Description
Declares an OP2 map by reading the mapping table from an HDF5 file.

### Method
OP2 Routine

### Endpoint
N/A (Function call)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```c
op_set from_set, to_set;
op_map my_map = op_decl_map_hdf5(from_set, to_set, 2, "input.h5", "map_name");
```

### Response
#### Success Response (200)
Returns an `op_map` identifier.

#### Response Example
```c
// op_map structure is internal to OP2
```
```

```APIDOC
## op_decl_dat_hdf5

### Description
Declares an OP2 dataset by reading its data from an HDF5 file.

### Method
OP2 Routine

### Endpoint
N/A (Function call)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```c
op_set target_set;
op_dat my_dat = op_decl_dat_hdf5(target_set, 3, "double", "input.h5", "data_name");
```

### Response
#### Success Response (200)
Returns an `op_dat` identifier.

#### Response Example
```c
// op_dat structure is internal to OP2
```
```

```APIDOC
## op_get_const_hdf5

### Description
Reads constant data from an HDF5 file into a user-supplied array.

### Method
OP2 Routine

### Endpoint
N/A (Function call)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```c
double constants[10];
op_get_const_hdf5("my_constant", 10, "double", constants, "input.h5");
```

### Response
#### Success Response (200)
Data is read into the `data` parameter. No return value.

#### Response Example
```c
// Data is populated in the 'constants' array from the example.
```
```

--------------------------------

### Execute Code-Generated Hybrid CUDA Version

Source: https://op2-dsl.readthedocs.io/en/latest/devapp

This command demonstrates the execution of a hybrid CUDA version using MPI. It shows how to launch the application across multiple MPI processes, which can then utilize both GPUs and CPU cores, distributing work accordingly.

```bash
#On 12 mpi procs
$MPI_INSTALL_PATH/bin/mpirun -np 12 ./airfoil_mpi_cuda_hyb OP_PART_SIZE=128 OP_BLOCK_SIZE=192

```