### MPI Setup and Execution Example

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-ARM.-Tested-on-GW4-Isambard

Example of loading MPI modules and running an MPI program. This sequence is for setting up and running an mpi4py program.

```bash
module load gcc
module load python/3.8.8
module load openmpi/4.0.4/gcc-11.0
```

--------------------------------

### Setup Global Devito Installation and User Examples Script

Source: https://github.com/devitocodes/devito/wiki/How-to-setup-a-JupyterHub-server-on-Azure-for-using-Devito-in-a-classroom-environment

A bash script to set up a system-wide Devito installation and copy example directories to each user's home directory. It activates the TLJH environment, installs Devito and its optional/MPI requirements, and then deactivates the environment.

```bash
#!/bin/bash
#

# Install the python depedencies (pip)
source /opt/tljh/user/bin/activate
pip3 install -U pip

# Print python3 --version
python3 -c 'import sys; print(".".join(map(str, sys.version_info[:3])))"

# Save current dir
dir=$PWD

# Install Devito in home directory
python3 -m pip install --ignore-installed --no-cache-dir devito # bit redundant setup (useful if other devito may have been installed)
# pip3 install devito # (should be fine in a clean tljh install)

# Clone Devito, not to install (as installed in last step), but only to copy examples
cd $HOME
git clone https://github.com/devitocodes/devito
cd devito
python3 -m pip install --ignore-installed --no-cache-dir -r requirements-optional.txt
python3 -m pip install --ignore-installed --no-cache-dir -r requirements-mpi.txt

https://github.com/devitocodes/Energy-HPC-2022.git
cd Energy-HPC-2022/
# Make examples available or other tutos you may like
# cp -r Energy-HPC-2022/ /etc/skel/Energy-HPC-2022

# Deactivate tljh env
source /opt/tljh/user/bin/deactivate
# Return to $dir
cd $dir
```

--------------------------------

### MPI Interactive Run Setup

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-Isambard-[CPU-GPU-ARM].

Setup for an interactive MPI run with Devito. Includes installing Devito, loading GCC and MPI modules, and installing mpi4py.

```bash
python -m pip install -U -e .
module load gcc
gcc --version
# For MPI
# module load openmpi/4.0.4/gcc-9.3
module load openmpi/mlnx/gcc/64/4.0.3rc4
pip3 install mpi4py

python3 examples/seismic/acoustic/acoustic_example.py -d 256 256 256 --tn 128
```

--------------------------------

### Install Intel oneAPI Base and HPC Toolkits

Source: https://github.com/devitocodes/devito/wiki/Setting-up-a-Devito-compatible-Intel-oneAPI-suite-for-profiling-using-APT

Install the Intel oneAPI basekit and hpckit extension using APT.

```bash
sudo apt-get install intel-basekit
```

```bash
sudo apt-get install intel-hpckit
```

--------------------------------

### Setup Virtual Display for 3D Renders

Source: https://github.com/devitocodes/devito/blob/main/examples/seismic/tutorials/14_creating_synthetics.ipynb

Uncomment and run these lines on platforms that do not support floating windows to prevent 3D renders from crashing the notebook. This installs and configures `xvfb` and `pyvirtualdisplay`.

```python
# NBVAL_IGNORE_OUTPUT
"""
!apt-get update
!apt-get -qq install xvfb
!pip install pyvirtualdisplay

from pyvirtualdisplay import Display
display = Display(visible=0, size=(600, 400))
display.start()
"""
```

--------------------------------

### Load Modules and Install Devito

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-DINE@COSMA

Load necessary modules for Python and compilers, then clone and install Devito using pip. This is the initial setup for Devito on COSMA.

```sh
module unload python/2.7.15 
module load python/3.6.5 

module load gnu_comp/10.2.0 
module load openmpi/4.0.5

git clone 
pip3 install --user -e.[extras]
```

--------------------------------

### Run Devito Acoustic Example on GPU

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-Isambard-[CPU-GPU-ARM].

Execute a Devito acoustic wave equation example on the GPU after setting up the environment.

```bash
python3 examples/seismic/acoustic/acoustic_example.py  -d 256 256 256 --tn 128
```

--------------------------------

### Run Intel Advisor Analysis for Devito Examples

Source: https://github.com/devitocodes/devito/blob/main/benchmarks/user/advisor/README.md

Execute Devito examples with Intel Advisor profiling. Specify the example name, path to the Devito script, and optional execution arguments.

```bash
python3 run_advisor.py --name isotropic --path <path-to-devito>/examples/seismic/acoustic/acoustic_example.py
```

```bash
python3 run_advisor.py --name iso_elastic --path <path-to-devito>/examples/seismic/elastic/elastic_example.py --exec-args "-so 4"
```

```bash
python3 run_advisor.py --name tti_so8 --path <path-to-devito>/examples/seismic/tti/tti_example.py --exec-args "-so 8"
```

--------------------------------

### Install Docker Engine

Source: https://github.com/devitocodes/devito/wiki/Azure-Hyperscale:-Kubernetes-Setup-for-Dask

Installs the Docker engine on Ubuntu. It's recommended to follow the official Docker installation guide for the most up-to-date instructions.

```bash
sudo apt-get update
sudo apt-get remove docker docker_engine docker.io
sudo apt install docker.io
```

--------------------------------

### Running Devito Acoustic Example Interactively on ARCHER2

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-ARCHER2

Execute the Devito acoustic wave equation example script interactively on ARCHER2 using srun. This example demonstrates launching a job with specific MPI and OpenMP configurations for different node counts.

```bash
# In interactive job

# Allocated nodes
OMP_NUM_THREADS=16 DEVITO_MPI=1 DEVITO_ARCH=cray DEVITO_LANGUAGE=openmp DEVITO_LOGGING=DEBUG srun --distribution=block:block --hint=nomultithread python examples/seismic/acoustic/acoustic_example.py -d 1024 1024 1024 --tn 512 -so 12 -a aggressive

# Nodes 1
OMP_NUM_THREADS=16 DEVITO_MPI=1 DEVITO_ARCH=cray DEVITO_LANGUAGE=openmp DEVITO_LOGGING=DEBUG srun -n 8 --distribution=block:block --hint=nomultithread python examples/seismic/acoustic/acoustic_example.py -d 512 512 512 --tn 100

# Nodes 2
OMP_NUM_THREADS=16 DEVITO_MPI=1 DEVITO_ARCH=cray DEVITO_LANGUAGE=openmp DEVITO_LOGGING=DEBUG srun -n 16 --distribution=block:block --hint=nomultithread python examples/seismic/acoustic/acoustic_example.py -d 512 512 512 --tn 512 -so 8
```

--------------------------------

### Clone Devito and Install User Packages

Source: https://github.com/devitocodes/devito/wiki/How-to-setup-a-JupyterHub-server-on-Azure-for-using-Devito-in-a-classroom-environment

Clones the Devito repository and installs it locally for the user, along with Matplotlib for tutorial rendering. This is a deprecated method for user setup.

```bash
# e.g. for Transform 2020
git clone https://github.com/devitocodes/devito.git # Clone latest Devito master
git clone https://github.com/devitocodes/transform2020.git # Clone Devito tutorial
cd devito && pip install --user -e . # Install Devito
pip install --user matplotlib # Matplotlib is needed for tutorials
```

--------------------------------

### Example PBS Job Script

Source: https://github.com/devitocodes/devito/wiki/PBS-file-generation-for-benchmarking

This is an example of a PBS job script generated by `make-pbs.py`. It includes resource requests, module loading, environment variable setup, and the benchmark execution command.

```bash
#!/bin/bash

#PBS -lselect=2:ncpus=24:mem=120gb:mpiprocs=2:ompthreads=12
#PBS -lwalltime=02:00:00

module load anaconda3/personal
module load intel-suite
module load mpi

cd /home/devitocodes/devito

source activate devito

export DEVITO_HOME=/home/devitocodes/devito
export DEVITO_ARCH=intel
export DEVITO_LANGUAGE=openmp
export DEVITO_MPI=basic
export DEVITO_LOGGING=DEBUG

cd benchmarks/user

mpiexec python benchmark.py bench -P acoustic -bm O2 -d 600 600 600 -so 12 --tn 100 -x 1 --arch haswell -r /home/opesci/results/

```

--------------------------------

### Install Devito with Docker

Source: https://github.com/devitocodes/devito/blob/main/README.md

Use these commands to clone the Devito repository and start a Jupyter notebook server using Docker Compose. Access the notebook via the provided URL.

```bash
git clone https://github.com/devitocodes/devito.git
cd devito

docker-compose up devito
```

--------------------------------

### Install and Install Pre-commit Hooks

Source: https://github.com/devitocodes/devito/blob/main/CONTRIBUTING.md

Installs the pre-commit package and sets up the pre-commit hooks for the Devito repository. These hooks run automated checks before each commit.

```bash
pip install pre-commit
pre-commit install
```

--------------------------------

### User Installation Command for Devito

Source: https://github.com/devitocodes/devito/wiki/How-to-setup-a-JupyterHub-server-on-Azure-for-using-Devito-in-a-classroom-environment

Example of how a user can install Devito using pip within their user environment. This command is typically run after a terminal is opened in the JupyterHub session.

```bash
pip install --user ..pkg-name..
```

--------------------------------

### Install Devito with Virtual Environment

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-HX1-(A100-GPUs)

Load production tools and a specific Python version, create a virtual environment, and install Devito in editable mode.

```bash
# https://icl-rcs-user-guide.readthedocs.io/en/latest/hpc/applications/easybuild/
# First, load the production tools

module load tools/prod
# Load Python, create virtual env and activate it
module load Python/3.11.3-GCCcore-12.3.0
# ...create activate and then...
python3 -m pip install -e .
```

--------------------------------

### Install kubectl CLI

Source: https://github.com/devitocodes/devito/wiki/Azure-Hyperscale:-Kubernetes-Setup-for-Dask

Installs the kubectl command-line interface on the system.

```bash
sudo az aks install-cli
```

--------------------------------

### Activate Devito Environment and Run Example

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-CX3--@Imperial-College-HPC

Activate the Devito Python environment and set logging to DEBUG. This example runs a typical acoustic operator.

```bash
# If everything went fine, you should be able to run a typical operator. i.e.:
source ../../environments/python3.11.5-env/bin/activate
export DEVITO_LOGGING=DEBUG
export DEVITO_LANGUAGE=openmp
python examples/seismic/acoustic/acoustic_example.py
```

--------------------------------

### Set up nvptx-tools for CUDA Compilation

Source: https://github.com/devitocodes/devito/wiki/Installing-gcc-10.0.1-with-offloading-support-on-Azure

Clones the nvptx-tools repository, configures it with CUDA include and library paths, and installs it to a specified prefix. Requires a working CUDA installation.

```bash
mkdir -p $HOME/offload/wrk
cd $HOME/offload/wrk
git clone https://github.com/MentorEmbedded/nvptx-tools.git
cd nvptx-tools
./configure \
    --with-cuda-driver-include=/usr/local/cuda-10.1/include \
    --with-cuda-driver-lib=/usr/local/cuda-10.1/lib64 \
    --prefix=$HOME/offload/install
make
make install
cd ..
```

--------------------------------

### Install Build Essentials and Libraries

Source: https://github.com/devitocodes/devito/wiki/Installing-gcc-10.0.1-with-offloading-support-on-Azure

Installs necessary build tools like gcc and make, cmake, and development libraries for libelf and pkg-config on Ubuntu.

```bash
sudo apt update
sudo apt install build-essential
sudo snap install cmake --classic
sudo apt install -y libelf-dev libffi-dev
sudo apt install -y pkg-config
```

--------------------------------

### Clone Devito and Load Modules for Fujitsu Compiler

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-ARM.-Tested-on-GW4-Isambard

Clone Devito, load GCC and Python modules, and install Devito. This section also includes setup for the Fujitsu compiler.

```bash
git clone https://github.com/devitocodes/devito.git
cd devito/

module load gcc
module load python/3.8.8
pip3 install --user -e .

# module load openmpi/4.0.4/gcc-11.0

source ~brx-pridley/arm-sve-tools/isambard-fujitsu.bashrc #add fujitsu
export CC=fcc
export CFLAGS="-Kfast,openmp -fPIC -Nfjomplib"
export CXX=FCC
export CXXFLAGS="-Kfast,openmp -fPIC -Nfjomplib"
export LDSHARED=fcc
export LDFLAGS="-Kfast,openmp -shared -Nfjomplib -lfjomphk -lfjomp -lfj90i -lfj90f -lfjsrcinfo -lfjcrt -lfjompcrt -lelf"
export MPICC=mpifcc
export MPICXX=mpiFCC

# install mpi4py with Fujitsu
module load python/3.8.8
pip3 install --force-reinstall --upgrade --user --no-cache-dir mpi4py
```

--------------------------------

### MPI Run Commands for Devito Examples

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-HX1-(A100-GPUs)

Demonstrates `mpirun` commands for executing Devito seismic examples, showing different mapping strategies and hostfile usage.

```bash
mpirun -n 4 --map-by ppr:4:node --report-bindings
mpirun -n 4 --map-by ppr:2:node -hostfile $PBS_NODEFILE --report-bindings python examples/seismic/elastic/elastic_example.py -d 1024 1024 1024 --tn 1024 -so 8
```

--------------------------------

### Run Devito Example with MPI Trace Collection

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Using-Intel-MPI-Trace-Analyzer-and-Collector-with-Devito

Execute a Devito example script with advanced profiling, ICC architecture, OpenMP, and basic MPI, enabling trace collection for MPI and OpenMP events.

```bash
DEVITO_PROFILING=advanced2 DEVITO_ARCH=icc OMP_PLACES=threads DEVITO_LOGGING=DEBUG DEVITO_LANGUAGE=openmp DEVITO_MPI=basic mpiexec.hydra -trace "libVT.so libmpi.so" -n 2 python3 examples/seismic/acoustic/acoustic_example.py -d 100 100 100 --tn 500
```

--------------------------------

### Devito Seismic Inversion Setup

Source: https://github.com/devitocodes/devito/blob/main/examples/seismic/tutorials/04_dask_pickling.ipynb

Imports necessary libraries and defines parameters for a seismic inversion problem using Devito and Dask. This includes model parameters, acquisition geometry, and solver setup.

```python
# NBVAL_IGNORE_OUTPUT
import numpy as np

from scipy import optimize


from distributed import Client, LocalCluster, wait

import cloudpickle as pickle

# Import acoustic solver, source and receiver modules.
from examples.seismic import demo_model, AcquisitionGeometry, Receiver
from examples.seismic.acoustic import AcousticWaveSolver

# Import convenience function for plotting results
from examples.seismic import plot_image


# Set up inversion parameters.
param = {
    't0': 0.,
    'tn': 1000.,              # Simulation last 1 second (1000 ms)
    'f0': 0.010,              # Source peak frequency is 10Hz (0.010 kHz)
    'nshots': 5,              # Number of shots to create gradient from
    'shape': (101, 101),      # Number of grid points (nx, nz).
    'spacing': (10., 10.),    # Grid spacing in m. The domain size is now 1km by 1km.
    'origin': (0, 0),         # Need origin to define relative source and receiver locations.
    'nbl': 40                 # nbl thickness.
}

```

--------------------------------

### Run Devito Acoustic Example (OpenMP)

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-DINE@COSMA

Execute a Devito acoustic example using OpenMP for parallelism. This command sets environment variables for logging and architecture, then runs the example.

```sh
DEVITO_LOGGING=DEBUG DEVITO_ARCH=intel python examples/seismic/acoustic/acoustic_example.py
```

--------------------------------

### Install and Import GemPy and Dependencies

Source: https://github.com/devitocodes/devito/blob/main/examples/seismic/tutorials/14_creating_synthetics.ipynb

Installs necessary libraries like `aiohttp`, `pyvista`, `pyqt5`, `gempy`, and `jinja2` if they are not already present. It also checks for and installs `ipyvtklink` for VTK notebook backend support.

```python
# NBVAL_IGNORE_OUTPUT
try:
    # Import gempy
    import gempy as gp
except ModuleNotFoundError:
    # Need to install these
    ! pip install aiohttp==3.7.1
    ! pip install pyvista==0.29
    ! pip install pyqt5
    # Install gempy
    ! pip install gempy==2.2.9
    # Import gempy
    import gempy as gp

try:
    # Import jinja2 (used for colour coding geology)
    import jinja2  # noqa: F401
except ModuleNotFoundError:
    # Install jinja2
    ! pip install jinja2
    # Import jinja2

try:
    # Check vtk notebook backend is installed
    import ipyvtklink  # noqa: F401
except ModuleNotFoundError:
    ! pip install ipyvtklink
```

--------------------------------

### Install Devito Prerequisites and TLJH on Azure VM

Source: https://github.com/devitocodes/devito/wiki/How-to-setup-a-JupyterHub-server-on-Azure-for-using-Devito-in-a-classroom-environment

Installs necessary packages like Python 3, Git, and cURL, then bootstraps The Littlest JupyterHub (TLJH) with administrative user configuration.

```bash
sudo apt-get update
sudo apt install python3 python3-dev git curl
curl -L https://tljh.jupyter.org/bootstrap.py | sudo -E python3 - --admin <admin-name>
```

--------------------------------

### Install Devito and Dependencies

Source: https://github.com/devitocodes/devito/wiki/[Manual-WIP]-Devito-on-Jade2

Clone the Devito repository if it's not already present. Install Devito using pip and then install mpi4py, specifying the C compiler and flags from the NVIDIA HPC SDK.

```bash
# If devito is not cloned:
git clone https://github.com/devitocodes/devito
pip3 install --upgrade -e .
# Install mpi4py with HPC SDK MPI, takes a few minutes possibly
CC=nvc CFLAGS="-noswitcherror -tp=px" pip install --ignore-installed --no-cache-dir mpi4py
```

--------------------------------

### Clone and Install Devito on CPU

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-Isambard-[CPU-GPU-ARM].

Clone the Devito repository and install it in a virtual environment using pip. Load GCC compiler and MPI modules if needed.

```bash
git clone https://github.com/devitocodes/devito.git
cd devito
python -m pip install -U -e .
module load gcc
gcc --version
# For MPI
# module load openmpi/4.0.4/gcc-9.3
module load openmpi/mlnx/gcc/64/4.0.3rc4
pip3 install mpi4py
```

--------------------------------

### Install ncurses prerequisite for nvtop

Source: https://github.com/devitocodes/devito/wiki/Installing-gcc-10.0.1-with-offloading-support-on-Azure

Install the ncurses development library, a prerequisite for compiling and running nvtop. This is typically done using a package manager like apt.

```bash
sudo apt-get install libncurses5-dev
```

--------------------------------

### Start CPU Development Environment

Source: https://github.com/devitocodes/devito/blob/main/docker/README.md

Use `docker-compose` to launch a development terminal for CPU-based work. This command starts a container with the necessary tools for CPU development.

```bash
# Start a terminal to develop/run for CPUs using docker compose
docker-compose run devito /bin/bash
```

--------------------------------

### Run Devito Acoustic Example (MPI)

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-DINE@COSMA

Run a Devito acoustic example using MPI for distributed parallelism. This command configures OpenMP thread binding and uses mpirun to launch the process across multiple cores.

```sh
OMP_PLACES=cores OMP_PROC_BIND=close DEVITO_LANGUAGE=openmp OMP_NUM_THREADS=16 mpirun -np 4 --mca btl_tcp_if_include p1p2 --bind-to socket python3 examples/seismic/acoustic/acoustic_example.py -d 1024 1024 1024 --tn 512 -so 12
```

--------------------------------

### Run Devito Acoustic Example (Intel)

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-DINE@COSMA

Execute a Devito acoustic example using Intel compilers and MPI. This involves loading Intel modules, activating a Conda environment, and setting Devito environment variables.

```sh
#load the modules used to build your program.
module purge
module load pythonconda3
source activate devito
module load intel_comp/2020-update2
module load intel_mpi/2020-update2

which mpicc
which mpirun

# Run the program
export DEVITO_LOGGING=DEBUG
export DEVITO_ARCH=intel
export DEVITO_MPI=full
export DEVITO_LANGUAGE=openmp

lscpu
hwloc-ls

export OMP_NUM_THREADS=16
mpirun -genv UCX_NET_DEVICES mlx5_1:1 -n 4 python3 examples/seismic/acoustic/acoustic_example.py -d 1024 1024 1024 --tn 512 -so 12
```

--------------------------------

### NumPy: Initial Condition and Boundary Setup

Source: https://github.com/devitocodes/devito/blob/main/examples/cfd/05_laplace.ipynb

Sets up the initial zero field and applies boundary conditions for the 2D Laplace equation solver. Visualizes the initial field.

```python
# NBVAL_IGNORE_OUTPUT

# Our initial condition is 0 everywhere, except at the boundary
p = np.zeros((ny, nx))

# Boundary conditions
bc_right = np.linspace(0, 1, ny)
p[0, :] = 0  # p = 0 @ x = 0
p[-1, :] = bc_right  # p = y @ x = 2
p[:, 0] = p[:, 1]  # dp/dy = 0 @ y = 0
p[:, -1] = p[:, -2]  # dp/dy = 0 @ y = 1

plot_field(p, ymax=1.0, view=(30, 225))
```

--------------------------------

### Initialize Solver Parameters

Source: https://github.com/devitocodes/devito/blob/main/examples/seismic/self_adjoint/sa_03_iso_correctness.ipynb

Sets up parameters for the acoustic self-adjoint solver, including grid shape and time parameters.

```python
npad = 10
fpeak = 0.010
qmin = 0.1
qmax = 500.0
tmax = 1000.0
shape = (101, 81)
```

--------------------------------

### Install MPI Requirements and Create ipyparallel Profile

Source: https://github.com/devitocodes/devito/blob/main/examples/mpi/data_initialization.ipynb

Install necessary Python packages for MPI and generate an ipyparallel profile for MPI execution. This is a prerequisite for running Devito examples in parallel.

```bash
pip install -r requirements-mpi.txt
./scripts/create_ipyparallel_mpi_profile.sh
```

--------------------------------

### Run Acoustic Example with MPI and OpenACC

Source: https://github.com/devitocodes/devito/wiki/[Manual-WIP]-Devito-on-Jade2

Execute the acoustic example script using `mpirun`. This command configures Devito to use MPI, the NVIDIA C++ compiler, debug logging, the nvidiaX platform, and OpenACC for parallelization across 2 processes.

```bash
DEVITO_MPI=1 DEVITO_ARCH=nvc++ DEVITO_LOGGING=DEBUG DEVITO_PLATFORM=nvidiaX DEVITO_LANGUAGE=openacc mpirun -n 2 python examples/seismic/acoustic/acoustic_example.py -d 124 124 124 --tn 100
```

--------------------------------

### Setup and Run Acoustic Wave Solver

Source: https://github.com/devitocodes/devito/blob/main/examples/seismic/self_adjoint/sa_03_iso_correctness.ipynb

Sets up the model, geometry, and solver for the acoustic wave equation. It then performs a forward simulation to obtain the numerical solution and computes the analytic solution for comparison. Ensure all necessary parameters like time axis, model dimensions, and source properties are correctly defined.

```python
# NBVAL_IGNORE_OUTPUT

# Setup time / frequency
nt = 1001
dt = 0.1
tmin = 0.0
tmax = dt * (nt - 1)
fpeak = 0.090
t0w = 1.0 / fpeak
omega = 2.0 * np.pi * fpeak
time_axis = TimeAxis(start=tmin, stop=tmax, step=dt)
time = np.linspace(tmin, tmax, nt)

# Model
space_order = 8
npad = 50
dx, dz = 0.5, 0.5
nx, nz = 801, 801
shape = (nx, nz)
spacing = (dx, dz)
origin = (0., 0.)

dtype = np.float64
qmin = 0.1
qmax = 100000
v0 = 1.5*np.ones(shape)
b0 = 1.0*np.ones(shape)

# Model
init_damp = lambda func, nbl: setup_w_over_q(func, omega, qmin, qmax, npad, sigma=0)
model = Model(origin=origin, shape=shape, vp=v0, b=b0, spacing=spacing, nbl=npad,
              space_order=space_order, bcs=init_damp, dtype=dtype, dt=dt)

# Source and receiver coordinates
src_coords = np.empty((1, 2), dtype=dtype)
rec_coords = np.empty((1, 2), dtype=dtype)
src_coords[:, :] = np.array(model.domain_size) * .5
rec_coords[:, :] = np.array(model.domain_size) * .5 + 60

geometry = AcquisitionGeometry(model, rec_coords, src_coords,
                               t0=0.0, tn=tmax, src_type='Ricker',
                               f0=fpeak)
# Solver setup
solver = SaIsoAcousticWaveSolver(model, geometry, space_order=space_order)

# Numerical solution
recNum, uNum, _ = solver.forward(dt=dt)

# Analytic solution
recAnaPad = analytic_response(fpeak, time_axis, src_coords, rec_coords, model.vp)
recAna = recAnaPad[0:nt]

# Compute RMS and difference
diff = (recNum.data - recAna)
nrms = np.max(np.abs(recNum.data))
arms = np.max(np.abs(recAna))
drms = np.max(np.abs(diff))

print(f"\nMaximum absolute numerical,analytic,diff; {nrms:+12.6e} {arms:+12.6e} {drms:+12.6e}")

# This isnt a very strict tolerance ...
tol = 0.1
assert np.allclose(diff, 0.0, atol=tol)

nmin, nmax = np.min(recNum.data), np.max(recNum.data)
amin, amax = np.min(recAna), np.max(recAna)

print("")
print(f"Numerical min/max; {nmin:+12.6e} {nmax:+12.6e}")
print(f"Analytic  min/max; {amin:+12.6e} {amax:+12.6e}")
```

--------------------------------

### Example Devito Operator Performance Output

Source: https://github.com/devitocodes/devito/blob/main/FAQ.md

This is an example of the performance profile Devito emits, showing total runtime, global performance (with and without setup overhead), and a breakdown of local performance by code section. Use this to understand where computation time is spent.

```text
Operator `MyOperator` ran in 0.68 s
Global performance: [OI=0.01, 50.59 GFlops/s, 1.16 GPts/s]
Global performance <w/o setup>: [0.24 s, 3.27 GPts/s]
Local performance:
  * section0 ran in 0.07 s [OI=0.06, 2063.30 GFlops/s, 11.28 GPts/s]
  * section1 ran in 0.17 s [OI=0.01, 4984.02 GFlops/s, 4.60 GPts/s]
  * section2 ran in 0.01 s
  * section3 ran in 0.01 s
```

--------------------------------

### Devito Setup with Grid and Functions

Source: https://github.com/devitocodes/devito/blob/main/examples/cfd/06_poisson.ipynb

Initializes Devito's Grid, Function objects for pressure (p) and previous pressure (pd), and the source term (b). Sets boundary conditions.

```python
from devito import Grid, Function, TimeFunction, Operator, configuration, Eq, solve

# Silence the runtime performance logging
configuration['log-level'] = 'ERROR'

grid = Grid(shape=(nx, ny), extent=(xmax, ymax))
p = Function(name='p', grid=grid, space_order=2)
pd = Function(name='pd', grid=grid, space_order=2)
p.data[:] = 0.
pd.data[:] = 0.

# Initialise the source term `b`
b = Function(name='b', grid=grid)
b.data[:] = 0.
b.data[int(nx / 4), int(ny / 4)] = 100
b.data[int(3 * nx / 4), int(3 * ny / 4)] = -100

# Create Laplace equation base on `pd`
eq = Eq(pd.laplace, b, subdomain=grid.interior)
# Let SymPy solve for the central stencil point
stencil = solve(eq, pd)
# # Now we let our stencil populate our second buffer `p`
eq_stencil = Eq(p, stencil)

# Create boundary condition expressions
x, y = grid.dimensions
t = grid.stepping_dim
bc = [Eq(p[x, 0], 0.)]
bc += [Eq(p[x, ny-1], 0.)]
bc += [Eq(p[0, y], 0.)]
bc += [Eq(p[nx-1, y], 0.)]
```

--------------------------------

### Set up Intel Advisor and Compiler Environment Variables

Source: https://github.com/devitocodes/devito/blob/main/benchmarks/user/advisor/README.md

Source these environment variables to make Intel Advisor and compilers accessible in your shell. Adjust the path based on your installation directory.

```sh
source /opt/intel/oneapi/advisor/latest/env/vars.sh
source /opt/intel/oneapi/compiler/latest/env/vars.sh
```

--------------------------------

### Setup Time Axis

Source: https://github.com/devitocodes/devito/blob/main/examples/seismic/tutorials/14_creating_synthetics.ipynb

Defines the time axis for the simulation using TimeAxis. Requires start time, stop time, and a time step (dt).

```python
from examples.seismic import TimeAxis

t0 = 0.  # Simulation starts a t=0
tn = 1000.  # Simulation last 1 second (1000 ms)
dt = seis_model.critical_dt  # Time step from model grid spacing

time_range = TimeAxis(start=t0, stop=tn, step=dt)
```

--------------------------------

### Run Intel Advisor Tripcounts and Cache Simulation

Source: https://github.com/devitocodes/devito/wiki/WIP:-Using-oneapi-Intel-Advisor-with-Devito

Analyze the acoustic example with Intel Advisor, collecting tripcounts, enabling cache simulation, and generating FLOP and map reports. This provides detailed performance insights.

```bash
numactl --cpunodebind=0 advixe-cl -data-limit=500 -project-dir mydir -collect tripcounts --enable-cache-simulation --flop --stacks --collect=map -- python3 examples/seismic/acoustic/acoustic_example.py -d 100 100 100
```

--------------------------------

### SSH and Environment Setup on ARCHER2

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-ARCHER2

Connect to the ARCHER2 login node, navigate to the work filesystem, load the necessary Python module, and create/activate a virtual environment. This is a prerequisite for installing Devito.

```shell
# After completing the registration
# Do `ssh` to your login node (private key needed)
ssh your_username@login.archer2.ac.uk -vv

# ch-dir to work filesystem
cd /work/"project-number"/"project-number"/"username"/
module load cray-python
# Create a python3 virtual env
# Activate it

# If devito is not cloned:
git clone https://github.com/devitocodes/devito
# TOFIX: BUILD WITH PRG-ENV-GNU
pip3 install -e .
```

--------------------------------

### OpenACC Offloading C Code Example

Source: https://github.com/devitocodes/devito/wiki/Installing-gcc-10.0.1-with-offloading-support-on-Azure

A C program demonstrating OpenACC offloading for a vector addition operation. It defines a kernel function for GPU execution and a main function for setup, execution, and result comparison.

```c
#include <stdio.h>
#include <stdlib.h>
void vecaddgpu( float *restrict r, float *a, float *b, int n ){
    #pragma acc kernels loop copyin(a[0:n],b[0:n]) copyout(r[0:n])
    for( int i = 0; i < n; ++i ) r[i] = a[i] + b[i];
}

int main( int argc, char* argv[] ){
    int n; /* vector length */
    float * a; /* input vector 1 */
    float * b; /* input vector 2 */
    float * r; /* output vector */
    float * e; /* expected output values */
    int i, errs;
    if( argc > 1 ) n = atoi( argv[1] );
    else n = 100000; /* default vector length */
    if( n <= 0 ) n = 100000;
    a = (float*)malloc( n*sizeof(float) );
    b = (float*)malloc( n*sizeof(float) );
    r = (float*)malloc( n*sizeof(float) );
    e = (float*)malloc( n*sizeof(float) );
    for( i = 0; i < n; ++i ){
         a[i] = (float)(i+1);
         b[i] = (float)(1000*i);
    }
    /* compute on the GPU */
    vecaddgpu( r, a, b, n );
    /* compute on the host to compare */
    for( i = 0; i < n; ++i ) e[i] = a[i] + b[i];
    /* compare results */
    errs = 0;
    for( i = 0; i < n; ++i ){
       	if( r[i] != e[i] ){
       		++errs;
        }
	}
	printf( "%d errors found\n", errs );
	return 0;
}

```

--------------------------------

### Install Python Development Headers on Linux

Source: https://github.com/devitocodes/devito/wiki/Installation-Issues

If you encounter a 'Python.h: No such file or directory' error during installation on Linux, install the python3-dev package.

```bash
(sudo) apt-get install python3-dev
```

--------------------------------

### Install MPI Requirements

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-CX3--@Imperial-College-HPC

Install or reinstall the necessary requirements for running Devito with MPI. Use `--force-reinstall` to ensure a clean installation.

```bash
# Instal MPI requirements
python3 -m pip install --force-reinstall --upgrade --no-cache-dir -r requirements-mpi.txt
```

--------------------------------

### MPI and mpi4py Installation

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-CSD3@Cambridge

Install mpi4py by first loading the necessary MPI module and then installing from the requirements file. This step is crucial for parallel processing with Devito.

```sh
# First check what mpiicc is available
module load intel-oneapi-compilers/
module load intel-oneapi-mpi
# Then install mpi4py
pip3 install --ignore-installed --no-cache-dir -r requirements-mpi.txt
# and try a small example
```

--------------------------------

### Setup Python Virtual Environment on CPU

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-Isambard-[CPU-GPU-ARM].

Create and activate a Python virtual environment for Devito on a CPU node. Includes updating pip.

```bash
python3 -m venv clx-env
cd clx-env
source bin/activate
python3 -m pip install -U pip
```

--------------------------------

### Download and Install CUDA Toolkit

Source: https://github.com/devitocodes/devito/wiki/Installing-gcc-10.0.1-with-offloading-support-on-Azure

Downloads and installs the CUDA 10.1 toolkit for Ubuntu 18.04. Includes adding NVIDIA's repository and installing the cuda package.

```bash
wget https://developer.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda-repo-ubuntu1804-10-1-local-10.1.105-418.39_1.0-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1804-10-1-local-10.1.105-418.39_1.0-1_amd64.deb
sudo apt-key add /var/cuda-repo-10-1-local-10.1.105-418.39/7fa2af80.pub
sudo apt-get update
sudo apt-get install cuda
```

--------------------------------

### Check Available Software Versions

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-HX1-(A100-GPUs)

Use `module avail` to list available software versions, filtering by a keyword.

```bash
# To quickly see the available versions of any software do not forget that you can do:
module avail -t 2>&1 | grep -i <keyword>
# e.g.
module avail -t 2>&1 | grep -i nvidia
```

--------------------------------

### Setup Snapshotting and Time Subsampling

Source: https://github.com/devitocodes/devito/blob/main/examples/timestepping/superstep.ipynb

Configures time-stepping parameters and sets up a `TimeFunction` for saving snapshots at regular intervals. This is useful for visualizing simulation results.

```python
nt2 = int(np.ceil((t2 - t1)/(dt*superstep_size)))

# Want to snapshot the solution for visualisation
nsnaps = 16
factor = int(np.ceil(nt2/nsnaps))
t_sub = ConditionalDimension('t_sub', parent=grid.time_dim, factor=factor)
u_save = TimeFunction(name='usave', grid=grid, time_order=0, space_order=2, save=nsnaps, time_dim=t_sub)
save = Eq(u_save, new_u)

print(f'The number of supertimesteps required is {nt2}')
```

--------------------------------

### Initialize CFD Parameters and Grid

Source: https://github.com/devitocodes/devito/blob/main/examples/cfd/04_burgers.ipynb

Sets up grid dimensions, time-stepping parameters, and physical constants for the CFD simulation.

```python
from examples.cfd import plot_field, init_hat
import numpy as np
%matplotlib inline

# Some variable declarations
nx = 41  # Grid size on x axis
ny = 41  # Grid size on y axis

batches = 5  # Batches of timesteps, increase number of batches to extend evolution in time
# A figure of the wave state will be produced for each batch.
batch_size = 640  # Number of timesteps for every batch
nt = batches*batch_size  # Number of total timesteps

c = 1
dx = 2. / (nx - 1)
dy = 2. / (ny - 1)
sigma = .0009
nu = 0.01
dt = sigma * dx * dy / nu
```

--------------------------------

### Verify GCC Version

Source: https://github.com/devitocodes/devito/wiki/Installing-gcc-10.0.1-with-offloading-support-on-Azure

Check the installed GCC version to confirm successful installation and configuration.

```bash
gcc --version
```

--------------------------------

### Clone and Install Devito

Source: https://github.com/devitocodes/devito/wiki/[Manual]-Devito-on-ARM.-Tested-on-GW4-Isambard

Clone the Devito repository and install it using pip. Ensure you are in the correct directory after cloning.

```bash
git clone https://github.com/devitocodes/devito.git
cd devito/
pip3 install --user -e .
```

--------------------------------

### Initialize Grid and Dimensions

Source: https://github.com/devitocodes/devito/blob/main/examples/cfd/09_Darcy_flow_equation.ipynb

Sets up the computational grid and dimensions for the simulation. The 'log-level' is set to 'ERROR' to suppress performance logging.

```python
# Silence the runtime performance logging
configuration['log-level'] = 'ERROR'

# Number of grid points on [0,1]^2
s = 256

# Create s x s grid with spacing 1
grid = Grid(shape=(s, s), extent=(1.0, 1.0))

x, y = grid.dimensions
t = grid.stepping_dim
```

--------------------------------

### Install numactl on Linux

Source: https://github.com/devitocodes/devito/blob/main/benchmarks/user/advisor/README.md

Install the numactl utility if it is not already available on your system. This is often a prerequisite for performance analysis tools.

```sh
sudo apt-get install numactl
```

--------------------------------

### Install Azure CLI

Source: https://github.com/devitocodes/devito/wiki/Azure-Hyperscale:-Kubernetes-Setup-for-Dask

Installs the Azure command-line interface on Debian-based systems. This tool is essential for managing Azure resources.

```bash
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
```

--------------------------------

### Initialize Devito Utilities

Source: https://github.com/devitocodes/devito/blob/main/examples/seismic/tutorials/10_nmo_correction.ipynb

Imports necessary libraries for Devito seismic modelling.

```python
import numpy as np
import sympy as sp

from devito import *

```

--------------------------------

### Setting up Time Axis, Source, and Wavefield

Source: https://github.com/devitocodes/devito/blob/main/examples/seismic/tutorials/07_DRP_schemes.ipynb

Configures the simulation time range, defines a Ricker source with specific parameters, and initializes a TimeFunction for the wavefield. Ensure `order` is defined.

```python
t0 = 0.  # Simulation starts a t=0
tn = 500.  # Simulation last 1 second (500 ms)
dt = 1.0  # Time step of 1.0ms

time_range = TimeAxis(start=t0, stop=tn, step=dt)

f0 = 0.025  # Source peak frequency is 25Hz (0.025 kHz)
src = RickerSource(name='src', grid=model.grid, f0=f0,
                   npoint=1, time_range=time_range)

src.coordinates.data[0, :] = np.array(model.domain_size) * .5
src.coordinates.data[0, -1] = 800.  # Depth is 800m

# New wave-field
u_DRP = TimeFunction(name="u_DRP", grid=model.grid, time_order=2, space_order=order)
```

--------------------------------

### Install MPI Dependencies

Source: https://github.com/devitocodes/devito/blob/main/examples/mpi/overview.ipynb

Install necessary dependencies for MPI support, including mpi4py and ipyparallel. Run this from the root Devito directory.

```bash
pip install -r requirements-mpi.txt
```

--------------------------------

### Install Python Packages for Plotting

Source: https://github.com/devitocodes/devito/blob/main/benchmarks/user/advisor/README.md

Install pandas and matplotlib using pip. These libraries are necessary for processing and visualizing the profiling results.

```sh
pip install pandas matplotlib
```

--------------------------------

### Initialize Grid and TimeFunction

Source: https://github.com/devitocodes/devito/blob/main/examples/compiler/01_data_regions.ipynb

Sets up a 3x3 grid and a time-varying function `u` initialized with ones. This establishes the base data structure for the example.

```python
from devito import Eq, Grid, TimeFunction, Operator

grid = Grid(shape=(3, 3))
u = TimeFunction(name='u', grid=grid)
u.data[:] = 1
```

--------------------------------

### Check Intel Advisor Installation

Source: https://github.com/devitocodes/devito/wiki/Using-Intel-Advisor-with-Devito

Verify if Intel Advisor is installed on the system by checking its version. This command is generic for Intel Advisor.

```bash
advixe-cl --version
```

--------------------------------

### Initialize a fine grid model

Source: https://github.com/devitocodes/devito/blob/main/examples/seismic/acoustic/accuracy.ipynb

Creates a `ModelBench` instance with a fine grid resolution, high spatial order, and damping boundary conditions for accurate wave propagation simulation.

```python
# Fine grid model
c0 = 1.5
model = ModelBench(vp=c0, origin=(0., 0.), spacing=(.5, .5), bcs="damp",
                   shape=(801, 801), space_order=20, nbl=40, dtype=np.float64)
```

--------------------------------

### Defining NMO Velocity Guide

Source: https://github.com/devitocodes/devito/blob/main/examples/seismic/tutorials/10_nmo_correction.ipynb

Creates a SparseFunction to represent a constant NMO velocity guide. Requires 'grid' and 'ns' to be defined.

```python
vnmo = 1500
vguide = SparseFunction(name='v', grid=grid, npoint=ns)
vguide.data[:] = vnmo
```