### Test Flux Framework Installation

Source: https://github.com/pyiron/executorlib/blob/main/docs/installation.md

Starts an interactive Flux session to validate the installation and check GPU recognition. The 'flux resource list' command displays available hardware resources recognized by Flux.

```shell
flux start
flux resource list
```

--------------------------------

### Minimal Installation

Source: https://github.com/pyiron/executorlib/blob/main/docs/installation.md

Installs the core executorlib package with its essential dependencies (zero message queue and cloudpickle) for basic functionality.

```shell
pip install executorlib
```

```shell
conda install -c conda-forge executorlib
```

--------------------------------

### Start Flux Jupyter Session

Source: https://github.com/pyiron/executorlib/blob/main/docs/installation.md

Initiates a Jupyter notebook session directly within an active Flux session. This is the simplest method for using Flux and Jupyter together.

```shell
flux start jupyter notebook
```

--------------------------------

### Run Script with Flux via SLURM

Source: https://github.com/pyiron/executorlib/blob/main/docs/installation.md

Executes a Python script using the Flux framework within a SLURM allocation. This command uses 'srun' to start a Flux session and then runs the specified Python script.

```shell
srun flux start python <script.py>
```

--------------------------------

### Install Executorlib Development Branch

Source: https://github.com/pyiron/executorlib/blob/main/docs/installation.md

Installs a specific development branch of the executorlib package directly from its GitHub repository using pip. Replace 'main' with the desired branch name.

```shell
pip install git+https://github.com/pyiron/executorlib.git@main
```

--------------------------------

### Install Executorlib with Graph Dependencies (Pip)

Source: https://github.com/pyiron/executorlib/blob/main/docs/installation.md

Installs the executorlib package along with optional dependencies for graph visualization using pip. This command installs pygraphviz and other necessary libraries.

```shell
pip install executorlib[graph]
```

--------------------------------

### HPC Cluster Executor Installation

Source: https://github.com/pyiron/executorlib/blob/main/docs/installation.md

Installs executorlib with support for HPC Cluster Executors, which requires the Python simple queuing system adapter (pysqa) for job scheduler interfacing and h5py for caching.

```shell
pip install executorlib[cluster]
```

```shell
conda install -c conda-forge executorlib h5py pysqa
```

--------------------------------

### Caching Installation

Source: https://github.com/pyiron/executorlib/blob/main/docs/installation.md

Installs executorlib with caching capabilities based on HDF5, which is required for HPC Cluster Executors. This requires the h5py package.

```shell
pip install executorlib[cache]
```

```shell
conda install -c conda-forge executorlib h5py
```

--------------------------------

### Install Executorlib and Visualization Dependencies (Conda)

Source: https://github.com/pyiron/executorlib/blob/main/docs/installation.md

Installs executorlib and its visualization dependencies, including pygraphviz, matplotlib, networkx, and ipython, using the conda package manager from the conda-forge channel.

```shell
conda install -c conda-forge executorlib pygraphviz matplotlib networkx ipython
```

--------------------------------

### Create Flux Jupyter Kernel Configuration

Source: https://github.com/pyiron/executorlib/blob/main/docs/installation.md

Defines the configuration for a custom Jupyter kernel that launches within a Flux environment. This JSON file specifies the command to execute, display name, and language for the kernel.

```json
{
  "argv": [
    "flux",
    "start",
    "/srv/conda/envs/notebook/bin/python",
    "-m",
    "ipykernel_launcher",
    "-f",
    "{connection_file}"
  ],
  "display_name": "Flux",
  "language": "python",
  "metadata": {
    "debugger": true
  }
}
```

--------------------------------

### Install Flux with OpenMPI 5+ and PMIX (conda)

Source: https://github.com/pyiron/executorlib/blob/main/docs/installation.md

Installs the Flux framework with OpenMPI version 5 or later, utilizing the PMIX backend. This requires the flux-pmix plugin and setting the pmi_mode parameter for executors.

```bash
conda install -c conda-forge flux-core flux-sched flux-pmix openmpi>=5 executorlib
```

--------------------------------

### MPI Support Installation

Source: https://github.com/pyiron/executorlib/blob/main/docs/installation.md

Installs executorlib with support for MPI parallel Python functions, requiring the mpi4py package. This enables distributed computing capabilities.

```shell
pip install executorlib[mpi]
```

```shell
conda install -c conda-forge executorlib mpi4py
```

--------------------------------

### Install Flux with OpenMPI (conda)

Source: https://github.com/pyiron/executorlib/blob/main/docs/installation.md

Installs the Flux framework with OpenMPI support using conda. This command is for general compatibility with OpenMPI versions prior to 5.0.

```bash
conda install -c conda-forge flux-core flux-sched openmpi=4.1.6 executorlib
```

--------------------------------

### Initialize External Process with Subprocess

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/4-developer.ipynb

Sets up the connection to an external executable using Python's subprocess module. It starts the executable and creates pipes for standard input (stdin) and standard output (stdout) for communication.

```python
def init_process():
    import subprocess

    return {
        "process": subprocess.Popen(
            ["python", "count.py"],
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            universal_newlines=True,
            shell=False,
        )
    }
```

--------------------------------

### Install Flux for Intel GPUs (conda)

Source: https://github.com/pyiron/executorlib/blob/main/docs/installation.md

Installs the Flux framework and dependencies for Intel GPUs using conda. This command is for systems like the Aurora HPC, compatible with mpich version 4.X, and does not require specific libhwloc GPU compilation flags.

```bash
conda install -c conda-forge flux-core flux-sched mpich=>4 executorlib
```

--------------------------------

### Run Script with Flux via SLURM using PMI2

Source: https://github.com/pyiron/executorlib/blob/main/docs/installation.md

Executes a Python script with Flux via SLURM, enforcing the use of the PMI2 protocol for MPI communication. This is used when the system's PMI version is 1.

```shell
srun –mpi=pmi2 flux start python <script.py>
```

--------------------------------

### Install Flux for Nvidia GPUs (conda)

Source: https://github.com/pyiron/executorlib/blob/main/docs/installation.md

Installs the Flux framework and dependencies for Nvidia GPUs using conda. It includes libhwloc compiled with CUDA support and mpich version 4.X, tailored for systems like the Perlmutter HPC. An environment variable CONDA_OVERRIDE_CUDA can be set if installing on a node without a GPU.

```bash
conda install -c conda-forge flux-core flux-sched libhwloc=*=cuda* mpich>=4 executorlib
```

--------------------------------

### Submit Shell Command to Executor

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/4-developer.ipynb

Demonstrates submitting the `execute_shell_command` function to a `SingleNodeExecutor`. The example shows how to instantiate the executor, submit the function with its arguments (the shell command and its options), and retrieve the result. The output of the executed command is then printed.

```Python
with SingleNodeExecutor() as exe:
    future = exe.submit(
        execute_shell_command,
        ["echo", "test"],
        universal_newlines=True,
        shell=False,
    )
    print(future.result())
```

--------------------------------

### Example Usage with SingleNodeExecutor

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/4-developer.ipynb

Demonstrates how to integrate the external process interaction functions with executorlib's SingleNodeExecutor. It shows submitting tasks for interaction and shutdown, and retrieving results.

```python
with SingleNodeExecutor(
    max_workers=1,
    init_function=init_process,
    block_allocation=True,
) as exe:
    future = exe.submit(
        interact, shell_input="4\n", lines_to_read=5, stop_read_pattern=None
    )
    print(future.result())
    future_shutdown = exe.submit(shutdown)
    print(future_shutdown.result())
```

--------------------------------

### Install Flux for AMD GPUs (conda)

Source: https://github.com/pyiron/executorlib/blob/main/docs/installation.md

Installs the Flux framework and necessary dependencies for AMD GPUs on HPC clusters using conda. This command specifies libhwloc compiled with ROCm support and mpich version 4.X, suitable for systems like the Frontier HPC.

```bash
conda install -c conda-forge flux-core flux-sched libhwloc=*=rocm* mpich>=4 executorlib
```

--------------------------------

### Start Flux in SLURM Allocation

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/3-hpc-job.ipynb

Illustrates how to start the Flux distributed computing framework within an existing SLURM allocation using the 'srun' command. This is a prerequisite for using the 'flux_allocation' backend.

```shell
srun flux start python <script.py>
```

--------------------------------

### Example Function with Error

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

Defines a sample Python function that simulates a potential error condition. This function is used to demonstrate error logging capabilities.

```python
def my_funct(i, j):
    if i == 2 and j == 2:
        raise ValueError()
    else:
        return i * j + i + j
```

--------------------------------

### Manage Cache Directory

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

Provides an example of how to inspect the contents of the cache directory and remove it using standard Python libraries. This is recommended after development to free up disk space.

```Python
import os
import shutil

cache_dir = "./file"
if os.path.exists(cache_dir):
    print(os.listdir(cache_dir))
    try:
        shutil.rmtree(cache_dir)
    except OSError:
        pass
```

--------------------------------

### Define Addition Function

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

Defines a simple Python function that takes two arguments and returns their sum. This function serves as an example for submission to the executor.

```python
def calc_add(a, b):
    return a + b
```

--------------------------------

### Flux Resource Monitoring Commands

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/3-hpc-job.ipynb

Provides examples of common Flux commands for monitoring computational resources. These commands help track available resources and executed jobs within a Flux session.

```shell
! flux resource list
```

```shell
! flux jobs -a
```

--------------------------------

### Define Simple Addition Function

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/2-hpc-cluster.ipynb

A basic Python function that takes two arguments and returns their sum. This serves as a simple example for submission to the executor.

```python
def add_funct(a, b):
    return a + b
```

--------------------------------

### SLURM Submission Template Example

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/2-hpc-cluster.ipynb

A Jinja2 template for a SLURM job submission script. It defines common SLURM directives like output file, job name, working directory, partition, runtime, dependencies, and memory, using placeholders for dynamic values.

```bash
#!/bin/bash
#SBATCH --output=time.out
#SBATCH --job-name={{job_name}}
#SBATCH --chdir={{working_directory}}
#SBATCH --get-user-env=L
#SBATCH --partition={{partition}}
{%- if run_time_max %}
#SBATCH --time={{ [1, run_time_max // 60]|max }}
{%- endif %}
{%- if dependency %}
#SBATCH --dependency=afterok:{{ dependency | join(',') }}
{%- endif %}
{%- if memory_max %}
#SBATCH --mem={{memory_max}}G
{%- endif %}
#SBATCH --cpus-per-task={{cores}}

{{command}}
```

--------------------------------

### executorlib Block Allocation with max_workers

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

Demonstrates reusing Python processes with `block_allocation=True` and limiting parallel processes using `max_workers` in `executorlib`. This ensures functions start with a fresh process but allows for process reuse to reduce initialization overhead.

```python
from executorlib import SingleNodeExecutor

# Example demonstrating block allocation and max_workers
with SingleNodeExecutor(max_workers=2, block_allocation=True) as exe:
    future = exe.submit(sum, [1, 1])
    print(future.result())
```

--------------------------------

### Python Imports for Executorlib Workflow

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-2-quantum-espresso.ipynb

Imports necessary libraries for building and executing DFT workflows, including ASE for calculator integration, atomistics for helper functions, and subprocess for external command execution.

```python
import os
import subprocess
from ase.build import bulk
from atomistics.workflows.evcurve.helper import (
    analyse_structures_helper as evcurve_analyse_structures,
    generate_structures_helper as evcurve_generate_structures,
)
import matplotlib.pyplot as plt
import pprint
from tqdm import tqdm
from time import sleep
```

--------------------------------

### FluxClusterExecutor Task Submission and Monitoring

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-2-quantum-espresso.ipynb

Demonstrates submitting tasks to FluxClusterExecutor, monitoring job status via 'flux jobs -a', and collecting results. It handles creating directories, setting resources, and processing results from submitted tasks.

```python
from executorlib import FluxClusterExecutor
import os
import subprocess
import pprint
from time import sleep
from tqdm import tqdm

# Assuming task_loop_dict, pseudopotentials, evaluate_with_quantum_espresso, structure_dict, evcurve_analyse_structures, plt are defined elsewhere

future_dict = {}
with FluxClusterExecutor() as exe:
    for k, v in task_loop_dict.items():
        os.makedirs(os.path.abspath(("strain_%0.2f" % k).replace(".", "_")), exist_ok=True)
        future_dict[k] = exe.submit(
            evaluate_with_quantum_espresso, 
            task_dict=v, 
            pseudopotentials=pseudopotentials, 
            kpts=(3, 3, 3), 
            resource_dict={
                "cores": 1,  # As an external executable is used the number of cores is set to one
                "threads_per_core": 2,  # For external executables the number of threads_per_core specifies the parallel execution
                "cwd": os.path.abspath(("strain_%0.2f" % k).replace(".", "_")),
            },
        )
    sleep(1)
    pprint.pp(subprocess.check_output(["flux", "jobs", "-a"], universal_newlines=True).split("\n")))
    result_dict = {
        k: f.result() 
        for k, f in tqdm(future_dict.items())
    }
    sleep(1)
    pprint.pp(subprocess.check_output(["flux", "jobs", "-a"], universal_newlines=True).split("\n")))
```

--------------------------------

### List Flux Cluster Resources

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-2-quantum-espresso.ipynb

Executes the `flux resource list` command to display available resources within the Flux cluster. This is used to validate that Flux has been initialized correctly and can access system resources.

```shell
pprint.pp(subprocess.check_output(["flux", "resource", "list"], universal_newlines=True).split("\n"))
```

--------------------------------

### Import necessary libraries

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-1-gpaw.ipynb

Imports essential Python libraries for running GPAW simulations, building structures, handling workflows, plotting, and managing subprocesses.

```python
import subprocess
from ase.build import bulk
from atomistics.workflows.evcurve.helper import (
    analyse_structures_helper as evcurve_analyse_structures,
    generate_structures_helper as evcurve_generate_structures,
)
import matplotlib.pyplot as plt
import pprint
from tqdm import tqdm
from time import sleep
```

--------------------------------

### Executor Configuration Mappings

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/4-developer.ipynb

This section details the configuration options for various executor classes and how they map to specific task schedulers and spawners. It covers different combinations of dependency handling, block allocation, and dependency graph plotting.

```APIDOC
SingleNodeExecutor:
  __init__(disable_dependencies: bool = False, block_allocation: bool = False)
    - disable_dependencies=False: Maps to DependencyTaskScheduler with MpiExecSpawner.
    - disable_dependencies=True, block_allocation=False: Maps to OneProcessTaskScheduler with MpiExecSpawner.
    - disable_dependencies=True, block_allocation=True: Maps to BlockAllocationTaskScheduler with MpiExecSpawner.

SlurmClusterExecutor:
  __init__(plot_dependency_graph: bool = False)
    - plot_dependency_graph=False: Maps to FileTaskScheduler with pysqa.
    - plot_dependency_graph=True: Maps to DependencyTaskScheduler with SrunSpawner.

SlurmJobExecutor:
  __init__(disable_dependencies: bool = False, block_allocation: bool = False)
    - disable_dependencies=False: Maps to DependencyTaskScheduler with SrunSpawner.
    - disable_dependencies=True, block_allocation=False: Maps to OneProcessTaskScheduler with SrunSpawner.
    - disable_dependencies=True, block_allocation=True: Maps to BlockAllocationTaskScheduler with SrunSpawner.

FluxClusterExecutor:
  __init__(plot_dependency_graph: bool = False)
    - plot_dependency_graph=False: Maps to FileTaskScheduler with pysqa.
    - plot_dependency_graph=True: Maps to DependencyTaskScheduler with FluxPythonSpawner.

FluxJobExecutor:
  __init__(disable_dependencies: bool = False, block_allocation: bool = False)
    - disable_dependencies=False: Maps to DependencyTaskScheduler with FluxPythonSpawner.
    - disable_dependencies=True, block_allocation=False: Maps to OneProcessTaskScheduler with FluxPythonSpawner.
    - disable_dependencies=True, block_allocation=True: Maps to BlockAllocationTaskScheduler with FluxPythonSpawner.
```

--------------------------------

### Initialize and Use SlurmClusterExecutor

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/2-hpc-cluster.ipynb

Demonstrates how to initialize the SlurmClusterExecutor with a cache directory and submit simple Python functions. It shows the basic workflow of submitting tasks and retrieving results.

```python
from executorlib import SlurmClusterExecutor

with SlurmClusterExecutor(cache_directory="./cache") as exe:
    future_lst = [exe.submit(sum, [i, i]) for i in range(1, 4)]
    print([f.result() for f in future_lst])
```

--------------------------------

### Executorlib Unit Test Execution

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/4-developer.ipynb

Provides the command to discover and run unit tests for the executorlib library. Tests are organized by module, with specific naming conventions for different components like SingleNodeExecutor.

```bash
python -m unittest discover .
```

--------------------------------

### Evaluate with Quantum Espresso Function

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-2-quantum-espresso.ipynb

This Python function orchestrates a Quantum Espresso calculation using ASE. It configures the Espresso calculator with pseudopotentials, stress/force flags, k-points, and a specific profile for running via `flux run pw.x`. It returns the calculated energy.

```python
def evaluate_with_quantum_espresso(task_dict, pseudopotentials, kpts):
    from ase.calculators.espresso import Espresso, EspressoProfile
    from atomistics.calculators import evaluate_with_ase
    
    return evaluate_with_ase(
        task_dict=task_dict,
        ase_calculator=Espresso(
            pseudopotentials=pseudopotentials,
            tstress=True,
            tprnfor=True,
            kpts=kpts,
            profile=EspressoProfile(
                 command="flux run pw.x",
                 pseudo_dir="/home/runner/work/executorlib/executorlib/tests/integration",
            ),
        ),
    )["energy"]
```

--------------------------------

### executorlib Parameter Preloading with init_function

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

Explains and demonstrates the `init_function` parameter in `SingleNodeExecutor` for preloading parameters. This allows data to be reused across function calls within the same process, improving performance when dealing with large datasets.

```python
from executorlib import SingleNodeExecutor

def init_function():
    # Returns a dictionary of parameters to be reused
    return {"j": 4, "k": 3, "l": 2}

def calc_with_preload(i, j, k):
    # Function that utilizes preloaded parameters
    return i + j + k

# Example demonstrating init_function for parameter preloading
with SingleNodeExecutor(max_workers=2, init_function=init_function, block_allocation=True) as exe:
    # i=2 and j=5 are provided during submission, k=3 is taken from init_function
    fs = exe.submit(calc_with_preload, 2, j=5)
    print(fs.result())
```

--------------------------------

### Define MPI Calculation Function

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/2-hpc-cluster.ipynb

A Python function designed to run within an MPI environment, utilizing mpi4py to get the total number of processes (size) and the current process rank. This is useful for parallel computations.

```python
def calc(i):
    from mpi4py import MPI

    size = MPI.COMM_WORLD.Get_size()
    rank = MPI.COMM_WORLD.Get_rank()
    return i, size, rank
```

--------------------------------

### Submit Calculation with Resources

Source: https://github.com/pyiron/executorlib/blob/main/README.md

This snippet demonstrates how to submit a calculation task using the executorlib's `submit` method. It shows how to pass a function (`calc`), its arguments (`3`), and a dictionary specifying resource requirements like the number of CPU cores (`resource_dict`). The result of the submitted task can then be retrieved using `fs.result()`.

```python
fs = exe.submit(calc, 3, resource_dict={"cores": 2})
print(fs.result())
```

--------------------------------

### Get Available GPU Information

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/2-hpc-cluster.ipynb

A Python function that queries the system for available GPUs using TensorFlow's device library and returns their names, physical device descriptions, and the hostname. This helps in identifying and utilizing GPU resources.

```python
def get_available_gpus():
    import socket
    from tensorflow.python.client import device_lib
    local_device_protos = device_lib.list_local_devices()
    return [
        (x.name, x.physical_device_desc, socket.gethostname()) 
        for x in local_device_protos if x.device_type == 'GPU'
    ]
```

--------------------------------

### FluxJobExecutor Task Submission and Monitoring

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-2-quantum-espresso.ipynb

Demonstrates submitting tasks to FluxJobExecutor with nesting enabled, monitoring job status via 'flux jobs -a', and collecting results. This is analogous to FluxClusterExecutor but tailored for job allocations within a queuing system.

```python
from executorlib import FluxJobExecutor
import os
import subprocess
import pprint
from time import sleep
from tqdm import tqdm

# Assuming task_loop_dict, pseudopotentials, evaluate_with_quantum_espresso, structure_dict, evcurve_analyse_structures, plt are defined elsewhere

future_dict = {}
with FluxJobExecutor(flux_executor_nesting=True) as exe:
    for k, v in task_loop_dict.items():
        os.makedirs(os.path.abspath(("strain_%0.2f" % k).replace(".", "_")), exist_ok=True)
        future_dict[k] = exe.submit(
            evaluate_with_quantum_espresso, 
            task_dict=v, 
            pseudopotentials=pseudopotentials, 
            kpts=(3, 3, 3), 
            resource_dict={
                "cores": 1,  # As an external executable is used the number of cores is set to one
                "threads_per_core": 2,  # For external executables the number of threads_per_core specifies the parallel execution
                "cwd": os.path.abspath(("strain_%0.2f" % k).replace(".", "_")),
            },
        )
    sleep(1)
    pprint.pp(subprocess.check_output(["flux", "jobs", "-a"], universal_newlines=True).split("\n")))
    result_dict = {
        k: f.result() 
        for k, f in tqdm(future_dict.items())
    }
    sleep(1)
    pprint.pp(subprocess.check_output(["flux", "jobs", "-a"], universal_newlines=True).split("\n")))
```

--------------------------------

### Python: Submit MPI Function with Resource Dict in submit()

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

This example demonstrates submitting the `calc_mpi` function to a `SingleNodeExecutor`. The `resource_dict` is passed directly to the `submit()` method to specify execution resources, such as the number of cores. The result of the parallel function execution is then retrieved using `fs.result()`.

```Python
with SingleNodeExecutor() as exe:
    fs = exe.submit(calc_mpi, 3, resource_dict={"cores": 2})
    print(fs.result())
```

--------------------------------

### FluxClusterExecutor Data Analysis and Plotting

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-2-quantum-espresso.ipynb

This snippet shows how to analyze the results obtained from computational tasks executed via FluxClusterExecutor. It fits the energy-volume data using a polynomial and prepares it for plotting to visualize the bulk modulus.

```python
# Assuming result_dict, structure_dict, evcurve_analyse_structures, plt are defined elsewhere

fit_dict = evcurve_analyse_structures(
    output_dict={"energy": result_dict},
    structure_dict=structure_dict,
    fit_type="polynomial",
    fit_order=3,
)

plt.plot(fit_dict["volume"], fit_dict["energy"], label="$B_0$= %0.2f GPa" % fit_dict["bulkmodul_eq"])
plt.xlabel("Volume [$\\AA^3$]")
plt.ylabel("Energy [eV]")
plt.legend()
```

--------------------------------

### List All Flux Jobs

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-2-quantum-espresso.ipynb

Executes the `flux jobs -a` command to list all active and completed jobs managed by Flux. This serves as a verification step to confirm Flux job management is operational.

```shell
pprint.pp(subprocess.check_output(["flux", "jobs", "-a"], universal_newlines=True).split("\n"))
```

--------------------------------

### Initialize Executor with Thread Resources

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

Shows initializing `SingleNodeExecutor` with thread parallelism settings in `resource_dict` during instantiation. This applies the specified resources to all tasks submitted to this executor instance.

```python
from executorlib import SingleNodeExecutor

# Assuming calc_with_threads is defined as above
with SingleNodeExecutor(resource_dict={"threads_per_core": 2}) as exe:
    fs = exe.submit(calc_with_threads, 3)
    print(fs.result())
```

--------------------------------

### Executorlib Benchmark Execution

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/4-developer.ipynb

Demonstrates how to run the likelihood benchmark for executorlib. It compares performance across different execution backends like static, ProcessPoolExecutor, ThreadPoolExecutor, and executorlib's SingleNodeExecutor, including options with block allocation and mpi4py.

```bash
python llh.py static
mpiexec -n 4 python -m mpi4py.futures llh.py mpi4py
```

--------------------------------

### Python: Submit MPI Function with Resource Dict in Executor Init

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

This snippet shows an alternative way to configure execution resources by providing the `resource_dict` during the initialization of the `SingleNodeExecutor`. This configuration is then applied to all subsequent `submit()` calls within the context manager, simplifying the submission process when resources are consistent.

```Python
with SingleNodeExecutor(resource_dict={"cores": 2}) as exe:
    fs = exe.submit(calc_mpi, 3)
    print(fs.result())
```

--------------------------------

### FluxJobExecutor Data Analysis and Plotting

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-2-quantum-espresso.ipynb

This snippet illustrates the data analysis and plotting steps following task execution with FluxJobExecutor. It involves fitting the energy-volume data to a polynomial to determine the bulk modulus and then plotting the curve.

```python
# Assuming result_dict, structure_dict, evcurve_analyse_structures, plt are defined elsewhere

fit_dict = evcurve_analyse_structures(
    output_dict={"energy": result_dict},
    structure_dict=structure_dict,
    fit_type="polynomial",
    fit_order=3,
)

plt.plot(fit_dict["volume"], fit_dict["energy"], label="$B_0$= %0.2f GPa" % fit_dict["bulkmodul_eq"])
plt.xlabel("Volume [$\\AA^3$]")
plt.ylabel("Energy [eV]")
plt.legend()
```

--------------------------------

### Define Pseudopotentials for Espresso

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-2-quantum-espresso.ipynb

Specifies the pseudopotential file to be used for Aluminium (Al) calculations within the Quantum Espresso workflow. This dictionary maps element symbols to their corresponding UPF files.

```python
pseudopotentials = {"Al": "Al.pbe-n-kjpaw_psl.1.0.0.UPF"}
```

--------------------------------

### Check Flux Cluster Resources

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-1-gpaw.ipynb

Executes `flux resource list` command to display available resources in the Flux cluster. This is a diagnostic step to ensure the cluster is properly configured and accessible.

```bash
pprint.pp(subprocess.check_output(["flux", "resource", "list"], universal_newlines=True).split("\n"))
```

--------------------------------

### Import FluxJobExecutor

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-1-gpaw.ipynb

Imports the necessary FluxJobExecutor class from the executorlib library for managing job execution.

```Python
from executorlib import FluxJobExecutor
```

--------------------------------

### Analyze and Fit Energy-Volume Data

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-1-gpaw.ipynb

Analyzes the collected energy results against structural volumes using a polynomial fit to determine material properties like the bulk modulus. It utilizes the `atomistics` library for analysis.

```python
fit_dict = evcurve_analyse_structures(
    output_dict={"energy": result_dict},
    structure_dict=structure_dict,
    fit_type="polynomial",
    fit_order=3,
)
```

--------------------------------

### SingleNodeExecutor with MPI and Resources

Source: https://github.com/pyiron/executorlib/blob/main/README.md

Shows how to use SingleNodeExecutor with functions that utilize MPI (via mpi4py) and how to specify computing resources like CPU cores using the resource_dict parameter.

```python
from executorlib import SingleNodeExecutor


def calc(i):
    from mpi4py import MPI

    size = MPI.COMM_WORLD.Get_size()
    rank = MPI.COMM_WORLD.Get_rank()
    return i, size, rank


with SingleNodeExecutor() as exe:
    fs = exe.submit(calc, 3, resource_dict={"cores": 2})
    print(fs.result())
```

--------------------------------

### Import SingleNodeExecutor (Python)

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

Demonstrates the basic import statement required to use the `SingleNodeExecutor` class from the `executorlib` library. This is the first step in leveraging its parallel execution capabilities.

```python
from executorlib import SingleNodeExecutor
```

--------------------------------

### Transform Structure Dictionary for Parallel Execution

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-2-quantum-espresso.ipynb

Converts the generated structure dictionary into a task loop dictionary format suitable for parallel execution. Each structure is mapped to a calculation task, specifically calling `calc_energy`.

```python
task_loop_dict = {k: {"calc_energy": v} for k, v in structure_dict.items()}
```

--------------------------------

### Plot Energy-Volume Curve

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-1-gpaw.ipynb

Generates a plot of the energy-volume curve, displaying the calculated bulk modulus as a label. This visualization summarizes the fitting results and equilibrium properties.

```Python
plt.plot(fit_dict["volume"], fit_dict["energy"], label="$B_0$= %0.2f GPa" % fit_dict["bulkmodul_eq"])
plt.xlabel("Volume [$\AA^3$]")
plt.ylabel("Energy [eV]")
plt.legend()
```

--------------------------------

### Executorlib API Reference

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/4-developer.ipynb

Provides access to stable functionalities of executorlib for external library integration. This includes functions for managing queues, handling pickling, retrieving backend commands, and managing communication interfaces.

```APIDOC
executorlib.api:
  cancel_items_in_queue(): Cancels items waiting in the Python standard library queue.
  cloudpickle_register(): Registers functions for pickling by value, not by reference, for communication.
  get_command_path(): Retrieves the file path for the executorlib backend executable script.
  interface_bootup(): Initializes the ZeroMQ communication interface.
  interface_connect(hostname: str, port: int): Connects to an existing SocketInterface instance using hostname and port.
  interface_receive(): Receives instructions or data from a connected SocketInterface instance.
  interface_send(data: any): Sends data or results to a connected SocketInterface instance.
  interface_shutdown(): Closes the connection to a SocketInterface instance.
  MpiExecSpawner: A subprocess interface for starting parallel processes using mpi4py.
  SocketInterface: An abstraction layer over Zero Message Queue (ZeroMQ) for communication.
  SubprocessSpawner: A subprocess interface for starting serial Python processes.
```

--------------------------------

### TestClusterExecutor Usage

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

Demonstrates the usage of TestClusterExecutor, which simulates file-based communication of cluster executors (like SlurmClusterExecutor) on a local machine. It executes a simple sum function and prints the result.

```python
from executorlib.api import TestClusterExecutor

with TestClusterExecutor(cache_directory="test") as exe:
    future = exe.submit(sum, [1, 1])
    print(future.result())
```

--------------------------------

### Submit Multiple Functions with SingleNodeExecutor (Python)

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

Shows how to efficiently submit multiple function calls to `SingleNodeExecutor` using a loop and the `submit` method. This approach is more performant than creating an executor for each call, and results are collected from a list of `Future` objects.

```python
%%time
with SingleNodeExecutor() as exe:
    future_lst = [exe.submit(sum, [i, i]) for i in range(2, 5)]
    print([f.result() for f in future_lst])
```

--------------------------------

### Submit Task with Thread Resources via submit()

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

Demonstrates submitting a function to `SingleNodeExecutor` and specifying thread parallelism via the `resource_dict` parameter directly in the `submit` call. This allows dynamic configuration of execution resources per task.

```python
from executorlib import SingleNodeExecutor

# Assuming calc_with_threads is defined as above
with SingleNodeExecutor() as exe:
    fs = exe.submit(calc_with_threads, 3, resource_dict={"threads_per_core": 2})
    print(fs.result())
```

--------------------------------

### Sequential Execution with SingleNodeExecutor

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

Demonstrates the use of SingleNodeExecutor to manage sequential task dependencies. It shows how a Future object from one submission can be used as input for the next, with the executor handling the internal waiting.

```python
with SingleNodeExecutor() as exe:
    future = 0
    for i in range(1, 4):
        future = exe.submit(calc_add, i, future)
    print(future.result())
```

--------------------------------

### Transform Structure Dictionary

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-1-gpaw.ipynb

Transforms the generated structure dictionary into a format suitable for parallel execution, mapping keys to dictionaries containing the structure for energy calculation.

```python
task_loop_dict = {k: {"calc_energy": v} for k, v in structure_dict.items()}
```

--------------------------------

### Submit Tasks with File Cache

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

Demonstrates submitting tasks to a SingleNodeExecutor with a specified cache directory. Executorlib serializes results using cloudpickle and stores them in the cache for potential reuse.

```Python
%%time
with SingleNodeExecutor(cache_directory="./file") as exe:
    future_lst = [exe.submit(sum, [i, i]) for i in range(1, 4)]
    print([f.result() for f in future_lst])
```

--------------------------------

### Distribute Tasks with FluxJobExecutor

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-1-gpaw.ipynb

Submits simulation tasks to the FluxJobExecutor, monitors job status using 'flux jobs -a', and collects results from completed futures. This demonstrates a typical workflow for parallel execution.

```Python
future_dict = {}
with FluxJobExecutor() as exe:
    for k, v in task_loop_dict.items():
        future_dict[k] = exe.submit(
            evaluate_with_gpaw, 
            task_dict=v, 
            kpts=(3, 3, 3), 
            encut=300,
            resource_dict={"cores": 2},
        )
    sleep(1)
    pprint.pp(subprocess.check_output(["flux", "jobs", "-a"], universal_newlines=True).split("\n"))
    result_dict = {
        k: f.result()[-1] 
        for k, f in tqdm(future_dict.items())
    }
    sleep(1)
    pprint.pp(subprocess.check_output(["flux", "jobs", "-a"], universal_newlines=True).split("\n"))
```

--------------------------------

### Visualize Dependency Graph

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

Enables the visualization of task dependency graphs by setting `plot_dependency_graph=True` during SingleNodeExecutor initialization. This feature is for debugging and requires the optional `pygraphviz` dependency.

```python
with SingleNodeExecutor(plot_dependency_graph=True) as exe:
    future = 0
    for i in range(1, 4):
        future = exe.submit(calc_add, i, future)
    print(future.result())
```

--------------------------------

### Analyze and Fit Energy-Volume Curve

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-1-gpaw.ipynb

Analyzes simulation results by fitting the energy-volume data using a specified polynomial order to derive the bulk modulus. This step is crucial for material property calculations.

```Python
fit_dict = evcurve_analyse_structures(
    output_dict={"energy": result_dict},
    structure_dict=structure_dict,
    fit_type="polynomial",
    fit_order=3,
)
```

--------------------------------

### Executorlib Communication Interface Utilities

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/4-developer.ipynb

Lists utility functions for managing the SocketInterface, which is an abstraction layer for Zero Message Queue (ZeroMQ) communication. These functions are crucial for initializing, connecting, sending, receiving, and shutting down communication channels.

```python
import executorlib.standalone.interactive.communication

# Initialize the interface
executorlib.standalone.interactive.communication.interface_bootup()

# Connect to another instance
# executorlib.standalone.interactive.communication.interface_connect(hostname='...', port=...)

# Send messages
# executorlib.standalone.interactive.communication.interface_send(message)

# Receive messages
# received_message = executorlib.standalone.interactive.communication.interface_receive()

# Shutdown the interface
# executorlib.standalone.interactive.communication.interface_shutdown()
```

--------------------------------

### Submit GPAW Tasks with FluxClusterExecutor

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-1-gpaw.ipynb

Demonstrates submitting the `evaluate_with_gpaw` function for each strained structure to a `FluxClusterExecutor`. It manages task submission, monitors job status, and collects results.

```python
from executorlib import FluxClusterExecutor

future_dict = {}
with FluxClusterExecutor() as exe:
    for k, v in task_loop_dict.items():
        future_dict[k] = exe.submit(
            evaluate_with_gpaw, 
            task_dict=v, 
            kpts=(3, 3, 3), 
            encut=300,
            resource_dict={"cores": 2},
        )
    sleep(1)
    pprint.pp(subprocess.check_output(["flux", "jobs", "-a"], universal_newlines=True).split("\n"))
    result_dict = {
        k: f.result()[-1] 
        for k, f in tqdm(future_dict.items())
    }
    sleep(1)
    pprint.pp(subprocess.check_output(["flux", "jobs", "-a"], universal_newlines=True).split("\n"))
```

--------------------------------

### Import SingleNodeExecutor

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

Imports the SingleNodeExecutor class from the executorlib library, which is used for executing tasks on a single node.

```python
from executorlib import SingleNodeExecutor
```

--------------------------------

### Generate Strained Structures for EV Curve

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-2-quantum-espresso.ipynb

Generates a dictionary of structures for an Aluminium bulk material, strained by 5% along specified axes. This is a preparatory step for calculating the energy-volume curve.

```python
structure_dict = evcurve_generate_structures(
    structure=bulk("Al", a=4.15, cubic=True),
    num_points=7,
    vol_range=0.05,
    axes=("x", "y", "z"),
)
```

--------------------------------

### Submit Tasks with Dependencies using FluxClusterExecutor

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/2-hpc-cluster.ipynb

Demonstrates submitting a series of tasks to a FluxClusterExecutor where each task's input depends on the output of the previous one. This showcases dependency resolution and asynchronous execution using Future objects.

```python
from executorlib import FluxClusterExecutor

with FluxClusterExecutor(cache_directory="./file") as exe:
    future = 0
    for i in range(4, 8):
        future = exe.submit(add_funct, i, future)
    print(future.result())
```

--------------------------------

### Executor Interface and Resource Management

Source: https://github.com/pyiron/executorlib/blob/main/docs/paper/paper.md

Executorlib extends the standard Python Executor interface to support distributed execution on HPC systems. The `submit()` method is enhanced to accept a `resource_dict` for specifying dedicated computing resources, such as cores, threads, GPUs, and scheduler-specific parameters like working directory, runtime, and memory.

```APIDOC
Executor:
  submit(func, *args, resource_dict: dict = None, **kwargs)
    - Submits a Python function for execution.
    - Parameters:
      - func: The Python function to execute.
      - *args: Positional arguments for the function.
      - resource_dict: A dictionary specifying requested computing resources and scheduler-specific parameters.
        - Example resource_dict:
          {
            "cores": 4,
            "threads": 1,
            "gpu": 1,
            "walltime": "01:00:00",
            "memory": "16GB",
            "path": "/path/to/working/dir"
          }
      - **kwargs: Keyword arguments for the function.
    - Returns: A future-like object representing the execution result.
```

--------------------------------

### Submit Single Function with SingleNodeExecutor (Python)

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

Illustrates using the `submit` method of `SingleNodeExecutor` to execute a single Python function (`sum`) with its arguments within a `with`-statement context. The `future.result()` method retrieves the computed output.

```python
%%time
with SingleNodeExecutor() as exe:
    future = exe.submit(sum, [1, 1])
    print(future.result())
```

--------------------------------

### Executor Implementations for Job Schedulers

Source: https://github.com/pyiron/executorlib/blob/main/docs/paper/paper.md

Executorlib offers specific Executor classes for different job schedulers, enabling seamless integration with various HPC environments. These classes handle the underlying submission logic for each scheduler.

```APIDOC
Executor Classes:

- SingleNodeExecutor:
  - For rapid prototyping on local workstations.
  - Functionally similar to Python's `ProcessPoolExecutor`.

- SlurmClusterExecutor:
  - Submits Python functions as individual jobs to a SLURM scheduler using `sbatch`.
  - Suitable for long-running tasks or legacy code execution.

- SlurmJobExecutor:
  - Distributes Python functions within an existing SLURM job using `srun`.

- FluxClusterExecutor:
  - Submits Python functions as individual jobs to a Flux job scheduler.

- FluxJobExecutor:
  - Distributes Python functions within a Flux job.
  - Supports nesting for hierarchical workflows, leveraging Flux's hierarchical approach.
```

--------------------------------

### Generate Strained Structures

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/5-1-gpaw.ipynb

Generates a dictionary of structures by applying volumetric strain to a base structure, typically for calculating energy-volume curves. It uses the `atomistics` library for structure generation.

```python
structure_dict = evcurve_generate_structures(
    structure=bulk("Al", a=4.05, cubic=True),
    num_points=7,
    vol_range=0.05,
    axes=("x", "y", "z"),
)
```

--------------------------------

### pyiron Executorlib Documentation Template

Source: https://github.com/pyiron/executorlib/blob/main/docs/_templates/custom-class-template.rst

This is a reStructuredText template used by Sphinx to generate API documentation for the pyiron executorlib. It leverages directives like 'autoclass', 'automethod', and 'autosummary' to extract and format information from Python docstrings and class structures.

```APIDOC
.. fullname | escape | underline}}

.. currentmodule:: {{ module }}

.. autoclass:: {{ objname }}
   :members:
   :show-inheritance:
   :inherited-members:

   {% block methods %}
   .. automethod:: __init__

   {% if methods %}
   .. rubric:: {{ _('Methods') }}

   .. autosummary::
   {% for item in methods %}
      ~{{ name }}.{{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}

   {% block attributes %}
   {% if attributes %}
   .. rubric:: {{ _('Attributes') }}

   .. autosummary::
   {% for item in attributes %}
      ~{{ name }}.{{ item }}
   {%- endfor %}
   {% endif %}
   {% endblock %}
```

--------------------------------

### Assign GPUs with FluxClusterExecutor

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/2-hpc-cluster.ipynb

Demonstrates submitting tasks to FluxClusterExecutor and requesting GPU resources by specifying `"gpus_per_core": 1` in the `resource_dict`. This allows for GPU-accelerated computations.

```python
with FluxClusterExecutor(
    cache_directory="./cache",
    resource_dict={"gpus_per_core": 1}
) as exe:
    fs_1 = exe.submit(get_available_gpus)
    fs_2 = exe.submit(get_available_gpus)
    print(fs_1.result(), fs_2.result())
```

--------------------------------

### Display Log File Content

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

Prints the content read from the error log file to the console for inspection.

```python
content
```

--------------------------------

### executorlib MPI Function Execution with Block Allocation

Source: https://github.com/pyiron/executorlib/blob/main/notebooks/1-single-node.ipynb

Shows how block allocation applies to MPI parallel Python functions. It illustrates submitting an MPI function (`calc_mpi`) within a `SingleNodeExecutor` configured with `max_workers` and `block_allocation=True`, and how to specify resources via `resource_dict`.

```python
from executorlib import SingleNodeExecutor

def calc_mpi(i):
    from mpi4py import MPI

    size = MPI.COMM_WORLD.Get_size()
    rank = MPI.COMM_WORLD.Get_rank()
    return i, size, rank

# Example with MPI function and resource dictionary
with SingleNodeExecutor(max_workers=2, resource_dict={"cores": 2}, block_allocation=True) as exe:
    fs = exe.submit(calc_mpi, 3)
    print(fs.result())
```