EasyBuild

https://docs.easybuild.io/
Admin
EasyBuild is a software build and installation framework that automates and manages scientific...
Tokens:959,867
Snippets:10,186
Trust Score:8.8
Update:1 week ago
Show doc for...
Tokens:
Raw
### guess_start_dir()

Source: https://docs.easybuild.io/api/easybuild/framework/easyblock

Returns the directory where to start the configure/make/make install cycle from.

```APIDOC
## guess_start_dir()

### Description
Return the directory where to start the whole configure/make/make install cycle from - typically self.src[0]['finalpath'] - start_dir option -- if abspath: use that -- else, treat it as subdir for regular procedure.
```

--------------------------------

### Set software subdirectory and modules subdirectory with EASYBUILD_SUBDIR_SOFTWARE and --subdir-modules

Source: https://docs.easybuild.io/configuration

Shows how to configure the software install path subdirectory via the EASYBUILD_SUBDIR_SOFTWARE environment variable and the modules install path subdirectory via the --subdir-modules option. The example uses --installpath to set the base install path.

```bash
export EASYBUILD_SUBDIR_SOFTWARE=installs
eb --installpath=$HOME/easybuild --subdir-modules=module_files ...
```

--------------------------------

### download_instructions parameter example

Source: https://docs.easybuild.io/writing-easyconfig-files

Use when a required installation file cannot be downloaded automatically (e.g., manual download or login required). The string value is printed whenever the build fails because a needed file was not found; if omitted, EasyBuild prints its default message listing the tested paths.

```python
download_instructions = """
  Step 1: Go to example.com and download example.jar.
  Step 2: Install example.jar (ensure Java is installed).
  Step 3: Go to the installation directory and create Tarball of its contents (tar -czvf example.tar.gz *).
  Step 4: Move created Tarball into the same directory, where your easyconfig is located and run build again.
"""
```

--------------------------------

### Set configure/build/install options with preconfigopts, buildopts, installopts

Source: https://docs.easybuild.io/writing-easyconfig-files

Shows how to set preconfigopts, buildopts, and installopts for a ConfigureMake easyblock. The preconfigopts value is prefixed to the configure command, and installopts uses the %(installdir)s template for the installation prefix.

```python
easyblock = 'ConfigureMake'
...
# configure with: ./autogen.sh && ./configure CC="$CC" CFLAGS="$CFLAGS"
preconfigopts = "./autogen.sh && "
buildopts = 'CC="$CC" CFLAGS="$CFLAGS"'
# install with: make install PREFIX=<installation prefix>
installopts = 'PREFIX=%(installdir)s'

```

--------------------------------

### Mixed list and single-string options for multiple cycles

Source: https://docs.easybuild.io/writing-easyconfig-files

Combines list and single-string values for configure/build/install options. When some parameters are lists, all list parameters must have the same number of items; single-string values are reused for each cycle. The example results in three cycles, e.g., './configure --prefix=... --one; make lib; TYPE=one make install'.

```python
easyblock = 'ConfigureMake'
configopts = ['--one', '--two', '--three']
buildopts = 'lib'
preinstallopts = ['TYPE=one', 'TYPE=two', 'TYPE=three']

```

--------------------------------

### Use Binary easyblock with install_cmd

Source: https://docs.easybuild.io/writing-easyconfig-files

Uses the generic Binary easyblock and sets the install_cmd parameter to run a custom installation command. Generic easyblocks often support custom easyconfig parameters to steer their behavior.

```python
easyblock = 'Binary'
[...]
install_cmd = "./install.bin"
[...]

```

--------------------------------

### HPL easyconfig example with sources and patches

Source: https://docs.easybuild.io/writing-easyconfig-files

Example easyconfig file for HPL 2.2 showing how to specify source_urls, sources, and patches. The patches entry fixes Make dependencies so a parallel build also works. Source URLs point to the Netlib benchmark site.

```python
name = 'HPL'
version = '2.2'

[...]

source_urls = ['http://www.netlib.org/benchmark/hpl']
sources = ['hpl-%(version)s.tar.gz']

# fix Make dependencies, so parallel build also works
patches = ['HPL_parallel-make.patch']
```

--------------------------------

### Mandatory easyconfig parameters example

Source: https://docs.easybuild.io/writing-easyconfig-files

Minimal easyconfig file showing the mandatory parameters: name, version, homepage, description, and toolchain. The toolchain is a dictionary with name and version keys, here using the goolf toolchain version 1.4.10.

```python
name = 'HPL'
version = '2.0'

homepage = 'http://www.netlib.org/benchmark/hpl/'
description = "High Performance Computing Linpack Benchmark"

toolchain = {'name': 'goolf', 'version': '1.4.10'}
[...]

```

--------------------------------

### Start RStudio Server with rserver

Source: https://docs.easybuild.io/version-specific/supported-software/r/RStudio-Server

Start the RStudio Server daemon on port 8787 with the --server-daemonize=0 flag to run in the foreground, useful for containers or debugging.

```bash
rserver --server-daemonize=0 --www-port=8787
```

--------------------------------

### List of configopts for multiple configure-build-install cycles

Source: https://docs.easybuild.io/writing-easyconfig-files

Defines a list of configuration option sets. EasyBuild will run the configure-build-install cycle three times, once for each set, cleaning the build directory and resetting the environment between cycles while leaving the installation directory untouched.

```python
configopts = [
    "--common-opt --one --one-more",
    "--common-opt --two",
    "--common-opt --three",
]

```

--------------------------------

### exts_list with llvmlite extension entry

Source: https://docs.easybuild.io/writing-easyconfig-files

Example of an extensions list entry for the llvmlite extension version 0.26.0. The tuple's third component is a dictionary of configuration options for the extension. This example appears in the section on extensions and follows the general `exts_list` syntax shown above it.

```python
exts_list = [
    ('llvmlite', '0.26.0', {
        'buildopts': '...',
        'installopts': '...',
    }),
]

```

--------------------------------

### checksums easyconfig parameter example

Source: https://docs.easybuild.io/writing-easyconfig-files

Shows the format of the checksums easyconfig parameter as a list of strings, where the checksum type is inferred from the string length (32-character strings are MD5, 64-character strings are SHA256).

```text
checksums = [
    '...',
]
```

--------------------------------

### Configure and use SQLite database for rserver

Source: https://docs.easybuild.io/version-specific/supported-software/r/RStudio-Server

Create a SQLite database configuration file for RStudio Server and then start the server with that configuration. The command sets up a temporary directory and writes a db.conf file with the sqlite provider and directory path, which is then passed to rserver via the --database-config-file flag.

```bash
MYTMP=`mktemp -d` && echo -e "provider=sqlite\ndirectory=${MYTMP}/sqlite" > "${MYTMP}/db.conf"
```

```bash
rserver ... --database-config-file="${MYTMP}/db.conf"
```

--------------------------------

### Update GCC version with --try-software-version

Source: https://docs.easybuild.io/writing-easyconfig-files

Use the --try-software-version command line option to update the version of an existing easyconfig file on the fly. The example updates GCC from 4.9.0 to 4.9.1 without manually editing the easyconfig.

```bash
eb GCC-4.9.0.eb --try-software-version=4.9.1

```

--------------------------------

### TestCase.setUp()

Source: https://docs.easybuild.io/api/easybuild/base/testing

Prepare test case.

```APIDOC
## TestCase.setUp()

### Description
Prepare test case.

### Method
TestCase method

### Parameters
None

### Returns
None.
```

--------------------------------

### Example installation of sagecipher keyring backend for headless setup

Source: https://docs.easybuild.io/integration-with-github

Shows a shell script to set up a virtual environment with sagecipher and GitPython, configure the keyring backend, and set environment variables for EasyBuild. Requires a working ssh-agent and a setuptools version providing pkg_resources. The fingerprint can be obtained from ssh-add -l -E md5.

```bash
module load EasyBuild  # ensure the python interpreter matches
python -m venv ~/eb_venv  # your preferred location for this virtual environment
source ~/eb_venv/bin/activate
pip install sagecipher GitPython setuptools==69.5.1  # sagecipher needs a setuptools that provides pkg_resources module
export PYTHON_KEYRING_BACKEND="sagecipher.keyring.Keyring"  # tell keyring to use sagecipher backend
export KEYRING_PROPERTY_SSH_KEY_FINGERPRINT="ff:00:11:22:33:44:55:66:77:88:99:aa:bb:cc:dd:ee"  # your SSH key's MD5 fingerprint
# instead of hard coding, you can get the fingerprint from a running ssh-agent, e.g.
# ssh-add -l -E md5 | awk '/your@email.com/{gsub("MD5:",""); print $2}'
export EB_PYTHON=$(which python)  # tell EB to use the eb_venv's python
# eb --install-github-token  # see docs below
# eb --check-github  # see docs below
```

--------------------------------

### set_up_configuration(args=None, logfile=None, testing=False, silent=False, reconfigure=False)

Source: https://docs.easybuild.io/api/easybuild/tools/options

Set up EasyBuild configuration, by parsing configuration settings & initialising build options.

```APIDOC
## set_up_configuration(args=None, logfile=None, testing=False, silent=False, reconfigure=False)

### Description
Set up EasyBuild configuration, by parsing configuration settings & initialising build options.

### Method
Function

### Parameters
- `args` - command line arguments to take into account when parsing the EasyBuild configuration settings (default: None)
- `logfile` - log file to use (default: None)
- `testing` - enable testing mode (default: False)
- `silent` - stay silent (no printing) (default: False)
- `reconfigure` - reconfigure singletons that hold configuration dictionaries. Use with care: normally, configuration shouldn't be changed during a run. Exceptions are when looping over items in EasyStack files (default: False)

```

--------------------------------

### Install EasyBuild using setup.py directly

Source: https://docs.easybuild.io/installation-alternative

Fallback method if pip or easy_install is not available. Installs the package using the setup.py script with the --prefix option.

```bash
python setup.py --prefix $HOME/EasyBuild install
```

--------------------------------

### extensions_progress_bar()

Source: https://docs.easybuild.io/api/easybuild/tools/output

Get progress bar to show progress for installing extensions.

```APIDOC
## `extensions_progress_bar()`

### Description
Get progress bar to show progress for installing extensions.

### Returns
- (ProgressBar) - The progress bar instance.
```

--------------------------------

### install_step()

Source: https://docs.easybuild.io/api/easybuild/framework/easyblock

Installs built software. This is an abstract method.

```APIDOC
## install_step()

### Description
Install built software (abstract method).
```

--------------------------------

### Generated Singularity container recipe

Source: https://docs.easybuild.io/containers

The generated container recipe includes yum install commands for OS packages, pip install for EasyBuild, creation of the easybuild user, configuration of Lmod, and environment setup to load the installed module.

```singularity
Bootstrap: yum
OSVersion: 7
MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/x86_64/
Include: yum

%post
yum install --quiet --assumeyes epel-release
yum install --quiet --assumeyes python setuptools Lmod
yum install --quiet --assumeyes python-pip
yum install --quiet --assumeyes bzip2 gzip tar zip unzip xz
yum install --quiet --assumeyes curl wget
yum install --quiet --assumeyes patch make
yum install --quiet --assumeyes file git which
yum install --quiet --assumeyes gcc-c++
yum install --quiet --assumeyes perl-Data-Dumper
yum install --quiet --assumeyes perl-Thread-Queue
yum --skip-broken --quiet --assumeyes install libibverbs-dev libibverbs-devel rdma-core-devel
yum --skip-broken --quiet --assumeyes install openssl-devel libssl-dev libopenssl-devel

# install EasyBuild using pip
pip install -U setuptools
pip install 'vsc-install<0.11.4' 'vsc-base<2.9.0'
pip install easybuild

# create 'easybuild' user (if missing)
id easybuild || useradd easybuild

# create /app software installation prefix + /scratch sandbox directory
if [ ! -d /app ]; then mkdir -p /app; chown easybuild:easybuild -R /app; fi
if [ ! -d /scratch ]; then mkdir -p /scratch; chown easybuild:easybuild -R /scratch; fi

# install Lmod RC file
cat > /etc/lmodrc.lua << EOF
scDescriptT = {
  {
    ["dir"]       = "/app/lmodcache",
    ["timestamp"] = "/app/lmodcache/timestamp",
  },
}
EOF

# change to 'easybuild' user
su - easybuild

# verbose commands, exit on first error
set -ve

# configure EasyBuild

# use /scratch as general prefix, used for sources, build directories, etc.
export EASYBUILD_PREFIX=/scratch

# also use /scratch for temporary directories
export EASYBUILD_TMPDIR=/scratch/tmp

# download sources to /scratch/sources, but also consider files located in /tmp/easybuild/sources;
# that way, source files that can not be downloaded can be seeded in
export EASYBUILD_SOURCEPATH=/scratch/sources:/tmp/easybuild/sources

# install software & modules into /app
export EASYBUILD_INSTALLPATH=/app

# use EasyBuild to install specified software
eb Python-3.6.4-foss-2018a.eb --robot

# update Lmod cache
mkdir -p /app/lmodcache
$LMOD_DIR/update_lmod_system_cache_files -d /app/lmodcache -t /app/lmodcache/timestamp /app/modules/all

# exit from 'easybuild' user
exit

# cleanup, everything in /scratch is assumed to be temporary
rm -rf /scratch/*

%runscript
eval "$@"

%environment
# make sure that 'module' and 'ml' commands are defined
source /etc/profile
# increase threshold time for Lmod to write cache in $HOME (which we don't want to do)
export LMOD_SHORT_TIME=86400
# purge any modules that may be loaded outside container
module --force purge
# avoid picking up modules from outside of container
module unuse $MODULEPATH
# pick up modules installed in /app
module use /app/modules/all
# load module(s) corresponding to installed software
module load Python/3.6.4-foss-2018a

%labels
```
Tokens:
Raw