IBM Spectrum LSF

https://www.ibm.com/docs/en/spectrum-lsf
Admin
IBM Spectrum LSF is a workload management platform for distributed computing environments that...
Tokens:802,049
Snippets:8,505
Trust Score:10
Update:1 month ago
Show doc for...
Tokens:
Raw
### Run interactive installation

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=services-installing-lsf-web

Execute the installer interactively to follow the setup prompts.

```bash
./lwsinstall.sh
```

--------------------------------

### Run LSF Web Services Installer

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=services-installing-lsf-web-high-availability-environment

Execute the installation script to begin the setup process.

```bash
# ./lwsinstall.sh
```

--------------------------------

### HOSTTYPE parameter syntax and example

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=windows-installation-parameter-quick-reference

Specifies the type of host to install.

```text
HOSTTYPE= Master | Candidate | Server | Client
```

```text
HOSTTYPE=Server
```

--------------------------------

### Start LSF Simulator with volume mapping

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=using-cluster-configurations

Example command to start the LSF Simulator service with the production cluster configuration directory mapped into the container.

```bash
./bcogn start -v "/production_cluster_sharedir/lsf:/production_cluster_sharedir/lsf"
```

--------------------------------

### TIME_WINDOW configuration example

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=files-lsbresources

A basic configuration example for TIME_WINDOW.

```text
TIME_WINDOW=8:00-14:00
```

--------------------------------

### rhostsetup configuration parameters

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=linux-adding-hosts-cluster

Example configuration settings for the rhostsetup script, defining the remote shell command, target hosts, installation directory, and boot/quiet options.

```bash
LSF_RSHCMD="ssh -n"
LSF_HOSTS="hostA hostB hostC"
LSF_TOPDIR=/usr/local/ls
LSF_BOOT=y
LSF_QUIET=n
```

--------------------------------

### Configure server.config parameters

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=connector-preparing-cloud-gen-2-lsf-resource

Example configuration parameters for the server.config file required during LSF installation.

```text
LSF_TOP="/usr/share/lsf"
LSF_ADMINS="lsfadmin root"
LSF_TARDIR="/opt/lsfinstaller/"
LSF_ENTITLEMENT_FILE="/opt/lsfinstaller/lsf_std_entitlement.dat"
LSF_SERVER_HOSTS="hostm" # LSF management host
LSF_LIM_PORT="7869" # management host LIM port 
LSF_LOCAL_RESOURCES="[resource icgen2host]"
```

--------------------------------

### Run hostsetup with automatic boot configuration

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=linux-adding-hosts-cluster

Sets up a host to use the cluster installed in the specified directory and configures LSF daemons to start automatically.

```bash
# hostsetup --top="/usr/share/lsf" --boot="y"
```

--------------------------------

### Configure mixed local and remote agent host setup

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=scheduler-lsflicensescheduler

Example with one local license server (hostA), one remote license server (hostB), and one remote agent host (hostC) serving two remote license servers (hostD and hostE).

```text
LIC_SERVERS=((1700@hostA)(1700@hostB)(1700@hostC)(1700@hostD)(1700@hostE))
REMOTE_LMSTAT_SERVERS=hostB hostC(hostD hostE)
```

--------------------------------

### Example hostregsetup file content

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=ac-register-lsf-host-names-ip-addresses-lsf-servers

A text file listing the LSF servers to which the local host must register. Empty lines and lines starting with # are ignored.

```text
# Rack 1 hosts
host1A
host1B
host1C

# Rack 2 hosts
host2A
host2B
host2C
```

--------------------------------

### JOBS configuration example

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=files-lsbresources

Example setting the maximum number of jobs to 20.

```text
JOBS=20
```

--------------------------------

### Start LSF Web Services

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=ilwshae-configuring-apache-http-server-as-https-proxy-lsf-web-services

Start the service after generating the required certificates.

```bash
# systemctl start lwsd
```

--------------------------------

### RESOURCE parameter examples

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=files-lsbresources

Examples of RESOURCE configuration in both horizontal and vertical tabular formats.

```text
RESOURCE=[stat_shared,4]
Begin Limit
RESOURCE                       PER_HOST
([stat_shared,4])              (all ~hostA)
([dyn_rsrc,1] [stat_rsrc,2])  (hostA)
End Limit
```

--------------------------------

### Specify domain user on Windows

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=windows-lsf-default-user-mapping

Example command to view jobs for a specific Windows domain user on a Windows host.

```bash
bjobs -u MYDOMAIN\user1
```

--------------------------------

### Specify domain user on UNIX

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=windows-lsf-default-user-mapping

Example command to view jobs for a specific Windows domain user on a UNIX host, requiring a double backslash to escape the special character.

```bash
bjobs -u MYDOMAIN\\user1
```

--------------------------------

### Example lsf.sudoers configuration

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=files-lsfsudoers

A sample configuration showing the definition of LSB_PRE_POST_EXEC_USER, LSF_STARTUP_PATH, and LSF_STARTUP_USERS.

```text
LSB_PRE_POST_EXEC_USER=user100
LSF_STARTUP_PATH=/usr/share/lsf/etc
LSF_STARTUP_USERS="user1 user10 user55"
```

--------------------------------

### ALLOW_OVERWRITE_LSF_TOP configuration

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=linux-installconfig

Example setting for the ALLOW_OVERWRITE_LSF_TOP parameter to permit installation into a non-empty directory during silent installs.

```text
ALLOW_OVERWRITE_LSF_TOP="Y"
```

--------------------------------

### PROJECTS configuration example

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=files-lsbresources

Example of configuring PROJECTS to enforce limits on projA and projB.

```text
PROJECTS=projA projB
```

--------------------------------

### Example lsb.threshold configuration file

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=files-lsbthreshold

This example shows the structure of the lsb.threshold file, including the Min_Energy and Min_Time sections. Uncomment the lines to apply custom values; otherwise, default values are used.

```text
# Copyright International Business Machines Corp,1993-2006, 2013
#....
# Minimize energy policy
#.....
Begin Min_Energy
THRESHOLD_RUNTIME_VAR = 10
End  Min_Energy
#
# Minimize run time policy
#...
Begin Min_Time
CPU_FREQ       RUNTIME_VAR
2100000KHz     -2      
2200000KHz     -4
2300000KHz     -9
2400000KHz     -12
2500000KHz     -15
2700000KHz     -20
End Min_Time
```

--------------------------------

### LSF_QUIET_INST configuration

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=linux-installconfig

Example of enabling quiet installation to hide messages.

```text
LSF_QUIET_INST="Y"
```

--------------------------------

### Example of submitting a job with a task range

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=jobs-specify-minimum-maximum-number-tasks

An example of submitting a job requesting between 4 and 16 tasks. The job will start if at least 4 processors are available and attempt to use up to 16.

```bash
bsub -n 4,16 myjob
```

--------------------------------

### HOSTS configuration example

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=files-lsbhosts

Example showing how to include all hosts in a group except for a specific host.

```text
HOSTS=groupA ~hostL
```

--------------------------------

### lsb.params configuration file example

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=tips-additional-configuration-settings

A sample configuration file for lsb.params showing various parameters. Settings prefixed with # are commented out and require adjustment before use.

```text
# Verify the existing setting
JOB_SCHEDULING_INTERVAL=0
#JOB_SCHEDULING_INTERVAL=250ms

ACCT_ARCHIVE_AGE=30        # 30 days
ACCT_ARCHIVE_SIZE=10000000   # 10 GB
#ACCT_ARCHIVE_TIME=23:55  # Enables archiving, performed @ 11:55PM
DEFAULT_PROJECT=default # Default value set explicitly
DEFAULT_RESREQ_ORDER=r15s:pg # Default value set explicitly
#DEFAULT_USER_GROUP=staff
ENABLE_DIAGNOSE=query
ENABLE_HOST_INTERSECTION=Y
ENFORCE_UG_TREE=Y
#EVALUATE_JOB_DEPENDENCY=100
EXTEND_JOB_EXCEPTION_NOTIFY=Y
JOB_DISTRIBUTE_ON_HOST=any # Default value set explicitly
JOB_INFO_EVENT_DUMP_INTERVAL=15 # Default value set explicitly
JOB_INFO_MEMORY_CACHE_SIZE=1024 # Default value set explicitly
LOCAL_MAX_PREEXEC_RETRY=5
LOCAL_MAX_PREEXEC_RETRY_ACTION=EXIT
MAX_ACCT_ARCHIVE_FILE=10
MAX_JOB_REQUEUE=5
ORPHAN_JOB_TERM_GRACE_PERIOD=60
#PREEMPT_FOR=LEAST_RUN_TIME
RELAX_JOB_DISPATCH_ORDER=ALLOC_REUSE_DURATION[0 30]
SCHED_METRIC_ENABLE=Y
SIMPLIFIED_GURANTEE=Y
```

--------------------------------

### PER_PROJECT configuration example

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=files-lsbresources

Example of configuring PER_PROJECT to enforce limits on specific projects.

```text
PER_PROJECT=proj1 proj2
```

--------------------------------

### Create an advance reservation with pre-script and post-script

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=reservation-adding-reservations

Example of using brsvadd to create a reservation named AR1 with a pre-script starting 5 minutes before the reservation and a post-script starting 10 minutes before the reservation ends.

```bash
brsvadd -N AR1 -n 1 -unit host -m hostA -u user1 -E /home/user1/pre.sh -Et 5 -Ep /home/user1/post.sh -Ept 10 –q normal –nosusp -b 9:00 -e 17:00
Reservation AR1 is created
```

--------------------------------

### Configure NON_SHARED for Hierarchical Groups

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=scheduler-lsflicensescheduler

Example configuration showing the correct setup for NON_SHARED values within hierarchical groups.

```text
GROUP           SHARES    OWNERSHIP   LIMITS      NON_SHARED 
(final (G2 G1)) (1 1)     ()          ()          (2 0) 
(G1 (AP2 AP1))  (1 1)     ()          ()          (1 1)
```

--------------------------------

### ResourceMap configuration example

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=lsbglobalpolicies-resourcemap

This example shows the structure of the ResourceMap section, mapping global_res_static and global_res_dynamic resources to all clusters.

```text
Begin ResourceMap 
RESOURCENAME        LOCATION 
global_res_static   (5@[all]) 
global_res_dynamic  ([all]) 
End ResourceMap
```

--------------------------------

### Register LSF host on Windows

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=101-host-related-features

Command to register a local host on Windows systems. This installs lsreghost as a service that starts automatically.

```cmd
lsreghost -i file_path\hostregsetup
```

--------------------------------

### LSF NotificationParams configuration

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=jsdl-using-files-lsf

Example of the jsdl-lsf:NotificationParams element used to configure job start and finish notifications via email.

```xml
<jsdl-lsf:NotificationParams>
<jsdl-lsf:NotifyAtStart>
true</jsdl-lsf:NotifyAtStart>
<jsdl-lsf:NotifyAtFinish>
true</jsdl-lsf:NotifyAtFinish>
<jsdl-lsf:NotificationEmail>
-u user10</jsdl-lsf:NotificationEmail>
</jsdl-lsf:NotificationParams>
```

--------------------------------

### Example lsf.conf configuration settings

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=tips-additional-configuration-settings

A summary of example settings for lsf.conf. Parameters commented out with a hash (#) require further adjustment before being enabled.

```text
EGO_DEFINE_NCPUS=cores # Default value set explicitly
EGO_PIM_SWAP_REPORT=Y
LSB_DISPLAY_YEAR=Y
LSB_ENABLE_PERF_METRICS_LOG=Y
LSB_SHARE_LOCATION_ENH=Y
LSB_SUB_COMMANDNAME=Y
LSF_ACCEPT_NUMCLIENTS=6 # Default value set explicitly
LSF_DISCARD_LOG=Y # Default value set explicitly
#LSF_GPU_RESOURCE_IGNORE=Y
LSF_INTELLIGENT_CPU_BIND=Y
#LSF_LINUX_CGROUP_ACCT=Y
LSF_LOG_QUEUE_SIZE=100000
LSF_LSLOGIN_SSH=Y
LSF_PROCESS_TRACKING=Y
LSF_RSH="ssh -o 'PasswordAuthentication no' -o 'StrictHostKeyChecking no'"
LSF_STRICT_CHECKING=Y     # Only change when the cluster is completely down
#LSF_STRIP_DOMAIN=.example.com:.domain.example.com
```

--------------------------------

### Example LSF cluster configuration file path

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=overview-cluster-characteristics

The cluster name specified during installation is included in the path of the lsf.cluster.cluster_name file located in the LSF_CONFDIR directory.

```text
/usr/share/lsf/lsf_10/conf/lsf.cluster.lsf_10
```

--------------------------------

### Example Resource section configuration

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=resources-configuring-lsfshared-resource-section

Example of defining various resource types within the Begin Resource and End Resource block.

```text
Begin Resource
RESOURCENAME  TYPE    INTERVAL INCREASING CONSUMABLE DESCRIPTION  # Keywords
   patchrev   Numeric  ()        Y         ()         (Patch revision)
   specman    Numeric  ()        N         ()         (Specman)
   switch     Numeric  ()        Y         N          (Network Switch)
   rack       String   ()        ()        ()         (Server room rack)
   owner      String   ()        ()        ()         (Owner of the host)
   elimres    Numeric  10        Y         ()         (elim generated index)
End Resource
```

--------------------------------

### Configure hostsetup for automatic startup

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=resources-adding-host-your-cluster

Run the hostsetup script to configure LSF daemons to start automatically at boot time, ensuring the host is already defined in lsf.cluster.cluster_name.

```bash
# cd /usr/share/lsf/10.1.0/install
# ./hostsetup --top="/usr/share/lsf" --boot="y"
```

--------------------------------

### Install VPC and DNS plugins

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=connector-preparing-cloud-gen-2-lsf-resource

Add the VPC infrastructure and DNS plug-ins to the IBM Cloud CLI.

```bash
ibmcloud plugin install vpc-infrastructure
ibmcloud plugin install DNS
```

--------------------------------

### Configure install.config for LSF Management Host

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=providers-configuring-microsoft-azure-cyclecloud-lsf-resource-connector

Example parameters to set in the install.config file when installing a new LSF management host to enable dynamic host support.

```text
LSF_TOP="/opt/lsf"
LSF_ADMINS="lsfadmin"
LSF_CLUSTER_NAME="cluster1"
LSF_MASTER_LIST="lsfmanagement1 lsfmanagement2"                  
LSF_ENTITLEMENT_FILE="/root/platform_lsf_std_entitlement.dat"
LSF_TARDIR="/root/"
ENABLE_DYNAMIC_HOSTS="Y"
LSF_DYNAMIC_HOST_WAIT_TIME="2"
```

--------------------------------

### Configure CONTAINER for sequential jobs

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=docker-configuring-lsf-run-jobs

Example configuration for a sequential job using the ubuntu image.

```text
CONTAINER=docker[image(ubuntu) options(--rm)]
```

--------------------------------

### LSF SchedulerParams configuration

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=jsdl-using-files-lsf

Example of the jsdl-lsf:SchedulerParams element used to define complex scheduling parameters like resource requirements, start times, and job groups.

```xml
<jsdl-lsf:SchedulerParams> 
<jsdl-lsf:ResourceRequirements>
"select[swp>15 && hpux] order[ut]"
</jsdl-lsf:ResourceRequirements>
<jsdl-lsf:Start>12:06:09:55
</jsdl-lsf:Start>
<jsdl-lsf:Deadline>8:22:15:50
</jsdl-lsf:Deadline>
<jsdl-lsf:ReservationID>"user1#0"
</jsdl-lsf:ReservationID>
<jsdl-lsf:Dependencies>’done myjob1’
</jsdl-lsf:Dependencies>
<jsdl-lsf:Rerunnable>true
</jsdl-lsf:Rerunnable>
<jsdl-lsf:UserPriority>3
</jsdl-lsf:UserPriority>
<jsdl-lsf:ServiceClass>platinum
</jsdl-lsf:ServiceClass>
<jsdl-lsf:Group>sysadmin</jsdl-lsf:Group>
<jsdl-lsf:ExternalScheduler>pset
</jsdl-lsf:ExternalScheduler>
<jsdl-lsf:Hold>true </jsdl-lsf:Hold> 
<jsdl-lsf:JobGroup>/risk_group/portfolio1
/current
</jsdl-lsf:JobGroup>
</jsdl-lsf:SchedulerParams>
```

--------------------------------

### Configure UNIX pre- and post-execution commands

Source: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=processing-pre-post-execution-behavior

Example configuration lines for UNIX hosts, showing how to specify commands and redirect output.

```text
PRE_EXEC=/usr/share/lsf/misc/testq_pre >> /tmp/pre.out POST_EXEC= /usr/share/lsf/misc/testq_post | grep -v "Testing..."
```
Tokens:
Raw