libEnsemble

https://libensemble.readthedocs.io/en/main/
Admin
libEnsemble is a Python toolkit for coordinating dynamic ensembles of calculations to solve design,...
Tokens:276,002
Snippets:2,158
Trust Score:9.6
Update:1 month ago
Show doc for...
Tokens:
Raw
### Getting started with APOSMM: suggest() and ingest() approaches

Source: https://libensemble.readthedocs.io/en/main/examples/gest_api/aposmm.html

Two approaches to provide the initial sample: either retrieve sample points via suggest() and then ingest them, or ingest pre-computed sample points. Both require the initial_sample_size to be met before local optimization points are provided.

```python
# Approach 1: Retrieve sample points via suggest()
gen = APOSMM(vocs, max_active_runs=2, initial_sample_size=10)

# ask APOSMM for some sample points
initial_sample = gen.suggest(10)
for point in initial_sample:
    point["f"] = func(point["x"])
gen.ingest(initial_sample)

# APOSMM will now provide local-optimization points.
points = gen.suggest(10)

# ----------------

# Approach 2: Ingest pre-computed sample points via ingest()
gen = APOSMM(vocs, max_active_runs=2, initial_sample_size=10)

initial_sample = create_initial_sample()
for point in initial_sample:
    point["f"] = func(point["x"])

# provide APOSMM with sample points
gen.ingest(initial_sample)

# APOSMM will now provide local-optimization points.
points = gen.suggest(10)

...
```

--------------------------------

### Install libEnsemble via uv

Source: https://libensemble.readthedocs.io/en/main/_sources/advanced_installation.rst.txt

Install the latest PyPI release using `uv pip install`, which is a recommended virtual environment tool.

```bash
uv pip install libensemble
```

--------------------------------

### Run libEnsemble with persistent generator

Source: https://libensemble.readthedocs.io/en/main/_sources/running_libE.rst.txt

Example command to start a manager, one persistent generator worker, and one simulation worker using mpirun.

```bash
mpirun -np 3 python my_script.py
```

--------------------------------

### Known Platform Example

Source: https://libensemble.readthedocs.io/en/main/data_structures/platform_specs.html

Example of using a known platform, such as PerlmutterGPU.

```APIDOC
## Known Platform Example

### Description
Uses a pre-defined platform object from libEnsemble.

### Usage
```python
from libensemble.resources.platforms import PerlmutterGPU

libE_specs["platform_specs"] = PerlmutterGPU()
```
```

--------------------------------

### Example: ingest() with evaluated point

Source: https://libensemble.readthedocs.io/en/main/examples/gest_api/sampling.html

Shows how to use suggest() to get a point, evaluate it, add the objective value, and pass it to ingest(). The example uses a point with keys 'x' and 'y', then adds 'f' before calling ingest.

```python
>>> point = generator.suggest(1)
>>> point
[{"x": 1, "y": 1}]
>>> point[0]["f"] = objective(point[0])
>>> point
[{"x": 1, "y": 1, "f": 2}]
>>> generator.ingest(point)

```

--------------------------------

### Example usage script

Source: https://libensemble.readthedocs.io/en/main/_sources/examples/sim_funcs/forces_simf_gpu.rst.txt

Example script to run the libEnsemble Forces GPU simulation.

```python
{* ../../../libensemble/tests/scaling_tests/forces/forces_gpu/run_libe_forces.py *}
```

--------------------------------

### Example Run Command

Source: https://libensemble.readthedocs.io/en/main/_sources/tutorials/executor_forces_tutorial.rst.txt

Example command to run the libEnsemble script with 3 workers.

```bash
python run_libe_forces.py --nworkers 3
```

--------------------------------

### Run forces example with gen_on_manager=True

Source: https://libensemble.readthedocs.io/en/main/platforms/perlmutter.html

Configure the libEnsemble generator to run on the manager node, then run the forces example with 4 workers. This setup is used on Perlmutter when the generator does not need dedicated resources.

```python
ensemble.libE_specs = LibeSpecs(
    gen_on_manager=True,

```
```

```bash
python run_libe_forces.py -n 4

```
```

--------------------------------

### Create, upload, and verify libEnsemble PyPI release

Source: https://libensemble.readthedocs.io/en/main/dev_guide/release_management/release_platforms/rel_pypi.html

Create a source distribution with setup.py sdist, upload it to PyPI with twine, then verify the new version is installable with pip install libensemble. Requires PyPI credentials and twine installed.

```bash
python setup.py sdist

```

```bash
twine upload dist/*

```

```bash
pip install libensemble

```

--------------------------------

### Install libEnsemble and dependencies

Source: https://libensemble.readthedocs.io/en/main/_sources/tutorials/local_sine_tutorial.rst.txt

Install the libEnsemble library and optional Matplotlib for visualization.

```bash
pip install libensemble
pip install matplotlib # Optional
```

--------------------------------

### Install ax-platform

Source: https://libensemble.readthedocs.io/en/main/examples/ax_multitask.html

Install the required ax-platform package (version 0.5.0 or later) using pip.

```bash
pip install ax-platform

```

--------------------------------

### Install libEnsemble via Spack with variants

Source: https://libensemble.readthedocs.io/en/main/_sources/advanced_installation.rst.txt

Install a specific version with common variants like MPI, scipy, mpmath, petsc4py, and nlopt.

```bash
spack install py-libensemble @0.7.2 +mpi +scipy +mpmath +petsc4py +nlopt
```

--------------------------------

### Install libensemble with pip

Source: https://libensemble.readthedocs.io/en/main/_sources/dev_guide/release_management/release_platforms/rel_pypi.rst.txt

After uploading, running this command should find and install the new version from PyPI. The package should also be visible at https://pypi.org/project/libensemble/.

```bash
pip install libensemble
```

--------------------------------

### run_libe_forces.py example usage

Source: https://libensemble.readthedocs.io/en/main/_sources/examples/sim_funcs/forces_simf_gpu_multi_app.rst.txt

Example script demonstrating how to configure the generator to assign CPU or GPU applications and specify the number of processors and GPUs for each simulation.

```python
{* ../../../libensemble/tests/scaling_tests/forces/forces_multi_app/run_libe_forces.py *}
```

--------------------------------

### run_libe_forces.py Example usage script

Source: https://libensemble.readthedocs.io/en/main/_sources/examples/sim_funcs/forces_simf.rst.txt

This example shows how to run the forces simulation with libEnsemble, using the `forces_simple.forces_simf` module. It is the usage runner that calls the simulation function.

```python
.. literalinclude:: ../../../libensemble/tests/scaling_tests/forces/forces_simple/run_libe_forces.py
   :language: python
   :linenos:


```

--------------------------------

### APOSMM with libEnsemble

Source: https://libensemble.readthedocs.io/en/main/_sources/examples/gest_api/aposmm.rst.txt

Example of using APOSMM with libEnsemble, showing the setup of the generator on the manager.

```APIDOC
## APOSMM with libEnsemble

### Description
Example of using APOSMM with libEnsemble, showing the setup of the generator on the manager.

### Code
```python
workflow.libE_specs.gen_on_manager = True
```
```

--------------------------------

### Navigate to forces_simple directory

Source: https://libensemble.readthedocs.io/en/main/_sources/platforms/improv.rst.txt

Change to the forces_simple directory, which contains the run script for the tutorial.

```bash
cd ../forces_simple/
```

--------------------------------

### Example command lines for libEnsemble options

Source: https://libensemble.readthedocs.io/en/main/utilities.html

Example command lines for running a calling script with various libEnsemble options. The first three are equivalent for local comms with 4 workers. The second runs with 5 workers (one gen worker, four sim workers). The third uses 8 resource sets for larger simulations. The fourth is the previous example with 'mpi' comms.

```shell
$ python calling_script --comms local --nworkers 4
$ python calling_script --nworkers 4
$ python calling_script -n 4

```

```shell
$ python calling_script --comms local --nsim_workers 4

```

```shell
$ python calling_script --comms local --nresource_sets 8

```

```shell
$ mpirun -np 5 python calling_script --nresource_sets 8

```

--------------------------------

### Use existing platform: PerlmutterGPU

Source: https://libensemble.readthedocs.io/en/main/data_structures/libE_specs.html

Use an existing platform by importing the platform class and assigning an instance to the 'platform_specs' key. The example uses PerlmutterGPU, but other known platforms can be substituted.

```python
from libensemble.resources.platforms import PerlmutterGPU

libE_specs["platform_specs"] = PerlmutterGPU()

```

--------------------------------

### Create source distribution with python setup.py sdist

Source: https://libensemble.readthedocs.io/en/main/_sources/dev_guide/release_management/release_platforms/rel_pypi.rst.txt

Run this command in the package directory on the main branch (containing setup.py) to create the source distribution tarball. The tarball is created using the MANIFEST.in file in the git root directory; do not use the GitHub tarball as it lacks the required PKG-INFO file.

```bash
python setup.py sdist
```

--------------------------------

### Install libEnsemble via Spack default

Source: https://libensemble.readthedocs.io/en/main/_sources/advanced_installation.rst.txt

Install the latest release with required dependencies using `spack install py-libensemble`.

```bash
spack install py-libensemble
```

--------------------------------

### Configure GPU Command Line Option

Source: https://libensemble.readthedocs.io/en/main/data_structures/platform_specs.html

Example settings for assigning GPUs using a command line flag.

```text
"gpu_setting_type" = "option_gpus_per_node"
"gpu_setting_name" = "--gpus-per-node"
```

--------------------------------

### Start interactive PBS session

Source: https://libensemble.readthedocs.io/en/main/_sources/platforms/polaris.rst.txt

Start an interactive session on Polaris. You may need to reload the conda module and reactivate the venv environment after starting the session.

```bash
qsub -A <myproject> -l select=1 -l walltime=15:00 -lfilesystems=home:grand -qdebug -I
```

--------------------------------

### Install libEnsemble via pip from develop branch

Source: https://libensemble.readthedocs.io/en/main/_sources/advanced_installation.rst.txt

Install the latest develop branch directly from GitHub using pip. Use with caution as it installs unreleased code.

```bash
python -m pip install --upgrade git+https://github.com/Libensemble/libensemble.git@develop
```

--------------------------------

### Run forces_simple with five workers

Source: https://libensemble.readthedocs.io/en/main/_sources/platforms/improv.rst.txt

Run the forces_simple tutorial with five workers: one for the generator and four for simulations.

```bash
python run_libe_forces.py --nworkers 5
```

--------------------------------

### Submit with all options in extra_args

Source: https://libensemble.readthedocs.io/en/main/platforms/summit.html

Equivalent to the previous submit example, but expresses everything in extra_args, including the -n 3 option for the number of resource sets. This is an alternative to using the portable num_procs option.

```python
exctr = Executor.executor
task = exctr.submit(app_name="mycode",
                    extra_args="-n 3 -a 1 -g 1 -c 1 --bind=packed:1 --smpiargs="-gpu""
                    app_args="-i input")

```

--------------------------------

### Install mpi4py

Source: https://libensemble.readthedocs.io/en/main/_sources/platforms/bebop.rst.txt

Install mpi4py using the preinstalled Intel MPI compiler.

```console
CC=mpiicc MPICC=mpiicc pip install mpi4py --no-binary mpi4py
```

--------------------------------

### AllocSupport.__init__

Source: https://libensemble.readthedocs.io/en/main/function_guides/allocator.html

Instantiates a new AllocSupport instance with the given parameters.

```APIDOC
## AllocSupport.__init__

### Description
Instantiate a new AllocSupport instance. W is passed in for convenience on init; it is referenced by the various methods, but never modified. By default, an AllocSupport instance uses any initiated libEnsemble resource module and the built-in libEnsemble scheduler.

### Method
Constructor

### Signature
`__init__(W, manage_resources=False, persis_info={}, libE_info={}, user_resources=None, user_scheduler=None)`

### Parameters
- **W** (Worker array) - Required - A Worker array.
- **manage_resources** (bool) - Optional - Boolean for if to assign resource sets when creating work units.
- **persis_info** (dict) - Optional - A dictionary of persistent information.
- **libE_info** (dict) - Optional - A dictionary of libE information.
- **user_resources** (object) - Optional - A user supplied `resources` object.
- **user_scheduler** (object) - Optional - A user supplied `user_scheduler` object.

### Notes
- The `scheduler_opts` parameter is mentioned in the source but not listed in the signature; it is likely a typo and should be ignored.
- Since the alloc_f is called periodically by the Manager, this class instance (if used) will be recreated/destroyed on each loop.
```

--------------------------------

### Install SciPy dependency

Source: https://libensemble.readthedocs.io/en/main/_sources/tutorials/aposmm_tutorial.rst.txt

Install the SciPy library required for APOSMM optimization methods.

```bash
pip install scipy
```

--------------------------------

### Configure GPU Environment Variable

Source: https://libensemble.readthedocs.io/en/main/data_structures/platform_specs.html

Example settings for assigning GPUs using an environment variable.

```text
"gpu_setting_type" = "env"
"gpu_setting_name" = "ROCR_VISIBLE_DEVICES"
```

--------------------------------

### Example usage: configure and run libEnsemble with forces sim

Source: https://libensemble.readthedocs.io/en/main/examples/sim_funcs/forces_simf.html

Example usage that sets up and runs a libEnsemble ensemble with a persistent generator and the `run_forces` simulation function. It registers the `forces.x` executable, configures `SimSpecs`, `GenSpecs`, and `AllocSpecs`, sets `sim_max=8` as the exit criterion, and prints the final energy checksum from the manager. Requires the forces executable to be built at `../forces_app/forces.x`.

```python
#!/usr/bin/env python
import os
import sys

import numpy as np
from forces_simf import run_forces  # Sim func from current dir

from libensemble import Ensemble
from libensemble.alloc_funcs.start_only_persistent import only_persistent_gens as alloc_f
from libensemble.executors import MPIExecutor
from libensemble.gen_funcs.persistent_sampling import persistent_uniform as gen_f
from libensemble.specs import AllocSpecs, ExitCriteria, GenSpecs, LibeSpecs, SimSpecs

if __name__ == "__main__":
    # Initialize MPI Executor
    exctr = MPIExecutor()

    # Register simulation executable with executor
    sim_app = os.path.join(os.getcwd(), "../forces_app/forces.x")

    if not os.path.isfile(sim_app):
        sys.exit("forces.x not found - please build first in ../forces_app dir")

    exctr.register_app(full_path=sim_app, app_name="forces")

    # Parse number of workers, comms type, etc. from arguments
    ensemble = Ensemble(parse_args=True, executor=exctr)
    nsim_workers = ensemble.nworkers - 1  # One worker is for persistent generator

    # Persistent gen does not need resources
    ensemble.libE_specs = LibeSpecs(
        num_resource_sets=nsim_workers,
        sim_dirs_make=True,
    )

    ensemble.sim_specs = SimSpecs(
        sim_f=run_forces,
        inputs=["x"],
        outputs=[("energy", float)],
    )

    ensemble.gen_specs = GenSpecs(
        gen_f=gen_f,
        inputs=[],  # No input when start persistent generator
        persis_in=["sim_id"],  # Return sim_ids of evaluated points to generator
        outputs=[("x", float, (1,))],
        user={
            "initial_batch_size": nsim_workers,
            "lb": np.array([1000]),  # min particles
            "ub": np.array([3000]),  # max particles
        },
    )

    # Starts one persistent generator. Simulated values are returned in batch.
    ensemble.alloc_specs = AllocSpecs(
        alloc_f=alloc_f,
        user={
            "async_return": False,  # False causes batch returns
        },
    )

    # Instruct libEnsemble to exit after this many simulations
    ensemble.exit_criteria = ExitCriteria(sim_max=8)

    # Seed random streams for each worker, particularly for gen_f
    ensemble.add_random_streams()

    # Run ensemble
    ensemble.run()

    if ensemble.is_manager:
        # Note, this will change if changing sim_max, nworkers, lb, ub, etc.
        print(f'Final energy checksum: {np.sum(ensemble.H["energy"])}')
```

--------------------------------

### Check Python version and install libEnsemble

Source: https://libensemble.readthedocs.io/en/main/tutorials/local_sine_tutorial.html

Check the Python version (must be >= 3.10) and install libEnsemble with optional Matplotlib for visualization. Add --user to each command if system permissions prevent installation.

```python
python --version  # This should be >= 3.10

```

```python
pip install libensemble
pip install matplotlib # Optional

```

--------------------------------

### Install libEnsemble via conda on Bebop

Source: https://libensemble.readthedocs.io/en/main/platforms/bebop.html

Installs libEnsemble via conda from the conda-forge channel.

```bash
conda config --add channels conda-forge
conda install -c conda-forge libensemble

```

--------------------------------

### Initializing persis_info structure

Source: https://libensemble.readthedocs.io/en/main/data_structures/persis_info.html

Example of defining the initial structure of persis_info, including manager-level tracking and worker-specific random streams.

```python
persis_info_1 = {
    "total_gen_calls": 0,  # Counts gen calls in alloc_f
    "last_worker": 0,  # Remembers last gen worker in alloc_f
    "next_to_give": 0,  # Remembers next H row to give in alloc_f
}

persis_info_1[0] = {
    "run_order": {},  # Used by manager to remember run order
    "total_runs": 0,  # Used by manager to count total runs
    "rand_stream": np.random.default_rng(1),
}
```

--------------------------------

### Install libEnsemble via pip

Source: https://libensemble.readthedocs.io/en/main/_sources/platforms/summit.rst.txt

Install libEnsemble using pip from the activated conda environment.

```console
(my_env) user@login5:~$ pip install libensemble
```

--------------------------------

### Run libEnsemble forces tutorial

Source: https://libensemble.readthedocs.io/en/main/_sources/platforms/frontier.rst.txt

Execute the libEnsemble forces tutorial script within the interactive session.

```console
python run_libe_forces.py --nworkers 9
```

--------------------------------

### Install Jinja2 dependency

Source: https://libensemble.readthedocs.io/en/main/_sources/examples/sim_funcs/forces_simf_input_file.rst.txt

Command to install the Jinja2 library required for templating input files.

```bash
pip install jinja2
```

--------------------------------

### Install libEnsemble with pip

Source: https://libensemble.readthedocs.io/en/main/platforms/summit.html

Installs libEnsemble via pip. The prompt shows the expected environment and user. If pip errors occur, upgrade pip first with 'python -m pip install --upgrade pip'.

```bash
(my_env) user@login5:~$ pip install libensemble

```

--------------------------------

### Example libE_stats.txt output

Source: https://libensemble.readthedocs.io/en/main/tutorials/executor_forces_tutorial.html

Sample content of the libE_stats.txt file generated after running the simulation.

```text
Manager     : Starting ensemble at: 2023-09-12 18:12:08.517
Worker     2: sim_id     0: sim Time: 0.205 Start: ... End: ... Status: Completed
Worker     3: sim_id     1: sim Time: 0.284 Start: ... End: ... Status: Completed
Worker     2: sim_id     2: sim Time: 0.117 Start: ... End: ... Status: Completed
Worker     3: sim_id     3: sim Time: 0.294 Start: ... End: ... Status: Completed
Worker     2: sim_id     4: sim Time: 0.124 Start: ... End: ... Status: Completed
Worker     3: sim_id     5: sim Time: 0.174 Start: ... End: ... Status: Completed
Worker     3: sim_id     7: sim Time: 0.135 Start: ... End: ... Status: Completed
Worker     2: sim_id     6: sim Time: 0.275 Start: ... End: ... Status: Completed
Worker     1: Gen no     1: gen Time: 1.038 Start: ... End: ... Status: Persis gen finished
Manager     : Exiting ensemble at: 2023-09-12 18:12:09.565 Time Taken: 1.048
```
Tokens:
Raw