### Example .rpmmacros File for Slurm Source: https://slurm.schedmd.com/quickstart_admin.html This example `.rpmmacros` file demonstrates how to override default RPM macros for custom Slurm installations, such as specifying installation paths and enabling debug logic. ```bash # .rpmmacros # Override some RPM macros from /usr/lib/rpm/macros # Set Slurm-specific macros for unconventional file locations # %_enable_debug "--with-debug" %_prefix /opt/slurm %_slurm_sysconfdir %{_prefix}/etc/slurm %_defaultdocdir %{_prefix}/doc %with_munge "--with-munge=/opt/munge" ``` -------------------------------- ### Example Slurm ResumeProgram Script Source: https://slurm.schedmd.com/elastic_computing.html This bash script demonstrates a basic ResumeProgram. It takes a hostlist expression as input, converts it to individual node names using 'scontrol show hostnames', and then attempts to start each node using 'sudo node_startup'. Logging is included for tracking invocations. ```bash #!/bin/bash # Example ResumeProgram hosts=$(scontrol show hostnames "$1") logfile=/var/log/power_save.log echo "$(date) Resume invoked $0 $*" >>$logfile for host in $hosts d sudo node_startup "$host" done exit 0 ``` -------------------------------- ### Slurm Configuration Example (slurm.conf) Source: https://slurm.schedmd.com/slurm A sample slurm.conf file demonstrating primary and backup controller setup, authentication, plugin directories, logging, scheduler type, and node and partition configurations. This file is crucial for defining the Slurm cluster's behavior and resources. ```shell # # Sample /etc/slurm.conf for dev[0-25].llnl.gov # Author: John Doe # Date: 11/06/2001 # SlurmctldHost=dev0(12.34.56.78) # Primary server SlurmctldHost=dev1(12.34.56.79) # Backup server # AuthType=auth/munge Epilog=/usr/local/slurm/epilog Prolog=/usr/local/slurm/prolog FirstJobId=65536 InactiveLimit=120 JobCompType=jobcomp/filetxt JobCompLoc=/var/log/slurm/jobcomp KillWait=30 MaxJobCount=10000 MinJobAge=300 PluginDir=/usr/local/lib:/usr/local/slurm/lib ReturnToService=0 SchedulerType=sched/backfill SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdLogFile=/var/log/slurm/slurmd.log SlurmctldPort=7002 SlurmdPort=7003 SlurmdSpoolDir=/var/spool/slurmd.spool StateSaveLocation=/var/spool/slurm.state TmpFS=/tmp WaitTime=30 # # Node Configurations # NodeName=DEFAULT CPUs=2 RealMemory=2000 TmpDisk=64000 NodeName=DEFAULT State=UNKNOWN NodeName=dev[0-25] NodeAddr=edev[0-25] Weight=16 # Update records for specific DOWN nodes DownNodes=dev20 State=DOWN Reason="power,ETA=Dec25" # # Partition Configurations # PartitionName=DEFAULT MaxTime=30 MaxNodes=10 State=UP PartitionName=debug Nodes=dev[0-8,18-25] Default=YES PartitionName=batch Nodes=dev[9-17] MinNodes=4 PartitionName=long Nodes=dev[9-17] MaxTime=120 AllowGroups=admin ``` -------------------------------- ### Install Slurm Source: https://slurm.schedmd.com/quickstart_admin.html Compiles and installs Slurm programs, documentation, and libraries after configuration. ```bash make install ``` -------------------------------- ### Enroot Start Wrapper Script Source: https://slurm.schedmd.com/containers.html A bash script used by Enroot to manage container startup, including environment variable setup and image handling. It imports an image if it's not already present. ```bash #!/bin/bash BUNDLE="$1" SPOOLDIR="$2" ENVFILE="$3" shift 4 IMAGE= export USER=$(whoami) export HOME="$BUNDLE/" export TERM export ENROOT_SQUASH_OPTIONS='-comp gzip -noD' export ENROOT_ALLOW_SUPERUSER=n export ENROOT_MOUNT_HOME=y export ENROOT_REMAP_ROOT=y export ENROOT_ROOTFS_WRITABLE=y export ENROOT_LOGIN_SHELL=n export ENROOT_TRANSFER_RETRIES=2 export ENROOT_CACHE_PATH="$SPOOLDIR/" export ENROOT_DATA_PATH="$SPOOLDIR/" export ENROOT_TEMP_PATH="$SPOOLDIR/" export ENROOT_ENVIRON="$ENVFILE" if [ ! -f "$BUNDLE" ] then IMAGE="$SPOOLDIR/container.sqsh" enroot import -o "$IMAGE" -- "$BUNDLE" && \ enroot create "$IMAGE" CONTAINER="container" else CONTAINER="$BUNDLE" fi enroot start -- "$CONTAINER" "$@" rc=$? [ $IMAGE ] && unlink $IMAGE exit $rc ``` -------------------------------- ### Install Slurm and Verify PMIx Libraries Source: https://slurm.schedmd.com/mpi_guide.html After successful configuration, install Slurm using 'make install'. The output shows the created PMIx MPI plugin libraries, including different versions. ```bash user@testbox:~/slurm/22.05/build$ make -j install user@testbox:~/slurm/22.05/build$ cd /home/user/slurm/22.05/inst/lib/slurm/ user@testbox:~/slurm/22.05/inst/lib/slurm$ ls -l *pmix* lrwxrwxrwx 1 user user 16 jul 6 17:17 mpi_pmix.so -> ./mpi_pmix_v4.so -rw-r--r-- 1 user user 9387254 jul 6 17:17 mpi_pmix_v3.a -rwxr-xr-x 1 user user 1065 jul 6 17:17 mpi_pmix_v3.la -rwxr-xr-x 1 user user 1265840 jul 6 17:17 mpi_pmix_v3.so -rw-r--r-- 1 user user 9935358 jul 6 17:17 mpi_pmix_v4.a -rwxr-xr-x 1 user user 1059 jul 6 17:17 mpi_pmix_v4.la -rwxr-xr-x 1 user user 1286936 jul 6 17:17 mpi_pmix_v4.so ``` -------------------------------- ### Enable and Start MUNGE Service Source: https://slurm.schedmd.com/authentication.html Enables the MUNGE service to start on boot and starts the service immediately. This should be done on all machines using MUNGE for authentication. ```bash systemctl enable munge systemctl start munge ``` -------------------------------- ### Install and configure slurm-spank-x11 plugin Source: https://slurm.schedmd.com/faq.html Steps to install the compiled SPANK plugin and configure Slurm to use it. Ensure the paths match your Slurm installation. ```bash # Install $ mkdir -p /opt/slurm/17.02/libexec $ install -m 755 slurm-spank-x11 /opt/slurm/17.02/libexec $ install -m 755 x11.so /opt/slurm/17.02/lib/slurm # Configure $ echo -e "optional x11.so" >> /opt/slurm/17.02/etc/plugstack.conf $ cd ~/tests ``` -------------------------------- ### Verify PMI-1 and PMI-2 Library Installation Source: https://slurm.schedmd.com/mpi_guide.html After installing PMI-1 and PMI-2 from contribs, verify the presence of their respective libraries in the Slurm installation's lib directory. ```bash user@testbox:~/$ ls -l /home/user/slurm/22.05/inst/lib/*pmi* -rw-r--r-- 1 user user 493024 jul 6 17:27 libpmi2.a -rwxr-xr-x 1 user user 987 jul 6 17:27 libpmi2.la lrwxrwxrwx 1 user user 16 jul 6 17:27 libpmi2.so -> libpmi2.so.0.0.0 lrwxrwxrwx 1 user user 16 jul 6 17:27 libpmi2.so.0 -> libpmi2.so.0.0.0 -rwxr-xr-x 1 user user 219712 jul 6 17:27 libpmi2.so.0.0.0 -rw-r--r-- 1 user user 427768 jul 6 17:27 libpmi.a -rwxr-xr-x 1 user user 1039 jul 6 17:27 libpmi.la lrwxrwxrwx 1 user user 15 jul 6 17:27 libpmi.so -> libpmi.so.0.0.0 lrwxrwxrwx 1 user user 15 jul 6 17:27 libpmi.so.0 -> libpmi.so.0.0.0 -rwxr-xr-x 1 user user 241640 jul 6 17:27 libpmi.so.0.0.0 ``` -------------------------------- ### Slurm Configuration File Example Source: https://slurm.schedmd.com/overview.html A sample configuration file for Slurm daemons, demonstrating settings for hostnames, authentication, plugin directories, ports, timeouts, and spool directories. This file is essential for setting up the Slurm controller and daemon. ```bash # # Sample /etc/slurm.conf # SlurmctldHost=linux0001 # Primary server SlurmctldHost=linux0002 # Backup server # AuthType=auth/munge Epilog=/usr/local/slurm/sbin/epilog PluginDir=/usr/local/slurm/lib Prolog=/usr/local/slurm/sbin/prolog SlurmctldPort=7002 SlurmctldTimeout=120 SlurmdPort=7003 SlurmdSpoolDir=/var/tmp/slurmd.spool SlurmdTimeout=120 StateSaveLocation=/usr/local/slurm/slurm.state TmpFS=/tmp ``` -------------------------------- ### Reservation Data Example Source: https://slurm.schedmd.com/rest_api.html An example of the JSON structure for reservation data, including details like start time, end time, flags, and associated resources. ```APIDOC ## Example Data ### Content-Type application/json ### Request Body ```json { "reservations" : [ { "end_time" : { "number" : 2, "set" : true, "infinite" : true }, "flags" : [ "MAINT", "MAINT" ], "groups" : [ "groups", "groups" ], "users" : [ "users", "users" ], "max_start_delay" : { "number" : 5, "set" : true, "infinite" : true }, "duration" : { "number" : 5, "set" : true, "infinite" : true }, "features" : "features", "start_time" : { "number" : 2, "set" : true, "infinite" : true }, "burst_buffer" : "burst_buffer", "licenses" : [ "licenses", "licenses" ], "partition" : "partition", "name" : "name", "comment" : "comment", "tres" : [ { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, { "name" : "name", "count" : 0, "id" : 7, "type" : "type" } ], "accounts" : [ "accounts", "accounts" ], "node_count" : { "number" : 5, "set" : true, "infinite" : true }, "node_list" : [ "node_list", "node_list" ], "purge_completed" : { "time" : { "number" : 5, "set" : true, "infinite" : true } }, "core_count" : { "number" : 5, "set" : true, "infinite" : true } }, { "end_time" : { "number" : 2, "set" : true, "infinite" : true }, "flags" : [ "MAINT", "MAINT" ], "groups" : [ "groups", "groups" ], "users" : [ "users", "users" ], "max_start_delay" : { "number" : 5, "set" : true, "infinite" : true }, "duration" : { "number" : 5, "set" : true, "infinite" : true }, "features" : "features", "start_time" : { "number" : 2, "set" : true, "infinite" : true }, "burst_buffer" : "burst_buffer", "licenses" : [ "licenses", "licenses" ], "partition" : "partition", "name" : "name", "comment" : "comment", "tres" : [ { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, { "name" : "name", "count" : 0, "id" : 7, "type" : "type" } ], "accounts" : [ "accounts", "accounts" ], "node_count" : { "number" : 5, "set" : true, "infinite" : true }, "node_list" : [ "node_list", "node_list" ], "purge_completed" : { "time" : { "number" : 5, "set" : true, "infinite" : true } }, "core_count" : { "number" : 5, "set" : true, "infinite" : true } } ], "meta" : { "slurm" : { "cluster" : "cluster", "release" : "release", "version" : { "major" : "major", "minor" : "minor", "micro" : "micro" } }, "plugin" : { "accounting_storage" : "accounting_storage", "name" : "name", "type" : "type", "data_parser" : "data_parser" }, "client" : { "source" : "source", "user" : "user", "group" : "group" }, "command" : [ "command", "command" ] }, "warnings" : [ { "description" : "description", "source" : "source" }, { "description" : "description", "source" : "source" } ], "errors" : [ { "description" : "description", "source" : "source", "error" : "error", "error_number" : 7 }, { "description" : "description", "source" : "source", "error" : "error", "error_number" : 7 } ] } ``` ``` -------------------------------- ### Create Resource Allocation and Execute Job Steps Source: https://slurm.schedmd.com/quickstart.html Creates a resource allocation for 1024 nodes, starts a bash shell within it, transfers an executable 'a.out' to '/tmp/joe.a.out' on allocated nodes using 'sbcast', executes the program using 'srun', removes the executable, and then relinquishes the allocation. ```bash salloc -N1024 bash ``` ```bash sbcast a.out /tmp/joe.a.out ``` ```bash srun /tmp/joe.a.out ``` ```bash srun rm /tmp/joe.a.out ``` ```bash exit ``` -------------------------------- ### Example JSON Response for Get WCKey Info Source: https://slurm.schedmd.com/archive/slurm-25.05.2/rest_api Example JSON data structure for the response of the 'Get WCKey Info' API call. It includes WCKey details, metadata about the Slurm cluster, plugins, client, and any warnings or errors encountered. ```json { "wckeys" : [ { "cluster" : "cluster", "name" : "name", "flags" : [ "DELETED", "DELETED" ], "accounting" : [ { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 }, { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 } ], "id" : 7, "user" : "user" }, { "cluster" : "cluster", "name" : "name", "flags" : [ "DELETED", "DELETED" ], "accounting" : [ { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 }, { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 } ], "id" : 7, "user" : "user" } ], "meta" : { "slurm" : { "cluster" : "cluster", "release" : "release", "version" : { "major" : "major", "minor" : "minor", "micro" : "micro" } }, "plugin" : { "accounting_storage" : "accounting_storage", "name" : "name", "type" : "type", "data_parser" : "data_parser" }, "client" : { "source" : "source", "user" : "user", "group" : "group" }, "command" : [ "command", "command" ] }, "warnings" : [ { "description" : "description", "source" : "source" }, { "description" : "description", "source" : "source" } ], "errors" : [ { "description" : "description", "source" : "source", "error" : "error", "error_number" : 7 }, { "description" : "description", "source" : "source", "error" : "error", "error_number" : 7 } ] } ``` -------------------------------- ### Example SPANK Plugin Configuration Source: https://slurm.schedmd.com/spank.html Illustrates how to load an optional SPANK plugin named 'test.so' without any arguments. ```plaintext optional /usr/lib/slurm/test.so ``` -------------------------------- ### Example JSON Response for Get WCKey Info Source: https://slurm.schedmd.com/archive/slurm-25.05.0/rest_api Example JSON data structure for the response of the 'Get WCKey Info' API call. It includes WCKey details, metadata about the Slurm cluster, plugins, client, and any warnings or errors encountered. ```json { "wckeys" : [ { "cluster" : "cluster", "name" : "name", "flags" : [ "DELETED", "DELETED" ], "accounting" : [ { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 }, { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 } ], "id" : 7, "user" : "user" }, { "cluster" : "cluster", "name" : "name", "flags" : [ "DELETED", "DELETED" ], "accounting" : [ { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 }, { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 } ], "id" : 7, "user" : "user" } ], "meta" : { "slurm" : { "cluster" : "cluster", "release" : "release", "version" : { "major" : "major", "minor" : "minor", "micro" : "micro" } }, "plugin" : { "accounting_storage" : "accounting_storage", "name" : "name", "type" : "type", "data_parser" : "data_parser" }, "client" : { "source" : "source", "user" : "user", "group" : "group" }, "command" : [ "command", "command" ] }, "warnings" : [ { "description" : "description", "source" : "source" }, { "description" : "description", "source" : "source" } ], "errors" : [ { "description" : "description", "source" : "source", "error" : "error", "error_number" : 7 }, { "description" : "description", "source" : "source", "error" : "error", "error_number" : 7 } ] } ``` -------------------------------- ### Requesting Heterogeneous Job Steps with srun Source: https://slurm.schedmd.com/heterogeneous_jobs.html This example demonstrates how to request heterogeneous job steps within a non-homogeneous job allocation using salloc and srun. It shows how to allocate nodes and then launch two different job steps with distinct resource requirements on those nodes. ```bash $ salloc -N2 --exclusive --gpus=10 salloc: Granted job allocation 61034 $ srun -N1 -n4 --gpus=4 printenv SLURMD_NODENAME : -N1 -n1 --gpus=6 printenv SLURMD_NODENAME node02 node01 node01 node01 node01 ``` -------------------------------- ### Example JSON Response for Get WCKey Info Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/rest_api Example JSON data structure for the response of the 'Get WCKey Info' API call. It includes WCKey details, metadata about the Slurm cluster, plugins, client, and any warnings or errors encountered. ```json { "wckeys" : [ { "cluster" : "cluster", "name" : "name", "flags" : [ "DELETED", "DELETED" ], "accounting" : [ { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 }, { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 } ], "id" : 7, "user" : "user" }, { "cluster" : "cluster", "name" : "name", "flags" : [ "DELETED", "DELETED" ], "accounting" : [ { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 }, { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 } ], "id" : 7, "user" : "user" } ], "meta" : { "slurm" : { "cluster" : "cluster", "release" : "release", "version" : { "major" : "major", "minor" : "minor", "micro" : "micro" } }, "plugin" : { "accounting_storage" : "accounting_storage", "name" : "name", "type" : "type", "data_parser" : "data_parser" }, "client" : { "source" : "source", "user" : "user", "group" : "group" }, "command" : [ "command", "command" ] }, "warnings" : [ { "description" : "description", "source" : "source" }, { "description" : "description", "source" : "source" } ], "errors" : [ { "description" : "description", "source" : "source", "error" : "error", "error_number" : 7 }, { "description" : "description", "source" : "source", "error" : "error", "error_number" : 7 } ] } ``` -------------------------------- ### Install PMI-1 and PMI-2 Support from Contribs Source: https://slurm.schedmd.com/mpi_guide.html To enable support for PMI-1 or PMI-2, navigate to the respective directories in the Slurm contribs and run 'make install'. ```bash user@testbox:~/slurm/22.05/build/$ cd contribs/pmi1 user@testbox:~/slurm/22.05/build/contribs/pmi1$ make -j install user@testbox:~/slurm/22.05/build/$ cd contribs/pmi2 user@testbox:~/slurm/22.05/build/contribs/pmi2$ make -j install ``` -------------------------------- ### Example JSON Response for Get WCKey Info Source: https://slurm.schedmd.com/archive/slurm-master/rest_api Example JSON data representing the response from the 'Get WCKey Info' API call. It includes WCKey details, metadata about the Slurm cluster, plugin information, client source, and any warnings or errors encountered. ```json { "wckeys" : [ { "cluster" : "cluster", "name" : "name", "flags" : [ "DELETED", "DELETED" ], "accounting" : [ { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 }, { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 } ], "id" : 7, "user" : "user" }, { "cluster" : "cluster", "name" : "name", "flags" : [ "DELETED", "DELETED" ], "accounting" : [ { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 }, { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 } ], "id" : 7, "user" : "user" } ], "meta" : { "slurm" : { "cluster" : "cluster", "release" : "release", "version" : { "major" : "major", "minor" : "minor", "micro" : "micro" } }, "plugin" : { "accounting_storage" : "accounting_storage", "name" : "name", "type" : "type", "data_parser" : "data_parser" }, "client" : { "source" : "source", "user" : "user", "group" : "group" }, "command" : [ "command", "command" ] }, "warnings" : [ { "description" : "description", "source" : "source" }, { "description" : "description", "source" : "source" } ], "errors" : [ { "description" : "description", "source" : "source", "error" : "error", "error_number" : 7 }, { "description" : "description", "source" : "source", "error" : "error", "error_number" : 7 } ] } ``` -------------------------------- ### Slurm Job Placement and Preemption Example Source: https://slurm.schedmd.com/preempt.html This example demonstrates a typical workflow involving checking node status with 'sinfo', submitting multiple jobs with 'sbatch', and monitoring the job queue with 'squeue'. It illustrates how job placement can affect resource availability and preemption. ```bash [user@n8 ~]$ **sinfo** PARTITION AVAIL TIMELIMIT NODES STATE NODELIST active* up infinite 5 idle n[1-5] hipri up infinite 5 idle n[1-5] ``` ```bash [user@n8 ~]$ **sbatch -N1 -n2 ./sleepme 60** sbatch: Submitted batch job 17 ``` ```bash [user@n8 ~]$ **sbatch -N1 -n2 ./sleepme 60** sbatch: Submitted batch job 18 ``` ```bash [user@n8 ~]$ **sbatch -N1 -n2 ./sleepme 60** sbatch: Submitted batch job 19 ``` ```bash [user@n8 ~]$ **squeue** JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 17 active sleepme cholmes R 0:03 1 n1 18 active sleepme cholmes R 0:03 1 n2 19 active sleepme cholmes R 0:02 1 n3 ``` ```bash [user@n8 ~]$ **sbatch -N3 -n6 -p hipri ./sleepme 20** sbatch: Submitted batch job 20 ``` ```bash [user@n8 ~]$ **squeue -Si** JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 17 active sleepme cholmes S 0:16 1 n1 18 active sleepme cholmes S 0:16 1 n2 19 active sleepme cholmes S 0:15 1 n3 20 hipri sleepme cholmes R 0:03 3 n[1-3] ``` ```bash [user@n8 ~]$ **sinfo** PARTITION AVAIL TIMELIMIT NODES STATE NODELIST active* up infinite 3 alloc n[1-3] active* up infinite 2 idle n[4-5] hipri up infinite 3 alloc n[1-3] hipri up infinite 2 idle n[4-5] ``` -------------------------------- ### Sample slurm.conf Configuration Source: https://slurm.schedmd.com/slurm.conf.html A sample slurm.conf file demonstrating controller host configuration, authentication, plugin directories, logging, and node/partition definitions. Ensure SlurmctldHost parameters are correctly set to avoid system instability. ```bash # # Sample /etc/slurm.conf for dev[0-25].llnl.gov # Author: John Doe # Date: 11/06/2001 # SlurmctldHost=dev0(12.34.56.78) # Primary server SlurmctldHost=dev1(12.34.56.79) # Backup server # AuthType=auth/munge Epilog=/usr/local/slurm/epilog Prolog=/usr/local/slurm/prolog FirstJobId=65536 InactiveLimit=120 JobCompType=jobcomp/filetxt JobCompLoc=/var/log/slurm/jobcomp KillWait=30 MaxJobCount=10000 MinJobAge=300 PluginDir=/usr/local/lib:/usr/local/slurm/lib ReturnToService=0 SchedulerType=sched/backfill SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdLogFile=/var/log/slurm/slurmd.log SlurmctldPort=7002 SlurmdPort=7003 SlurmdSpoolDir=/var/spool/slurmd.spool StateSaveLocation=/var/spool/slurm.state TmpFS=/tmp WaitTime=30 # # Node Configurations # NodeName=DEFAULT CPUs=2 RealMemory=2000 TmpDisk=64000 NodeName=DEFAULT State=UNKNOWN NodeName=dev[0-25] NodeAddr=edev[0-25] Weight=16 # Update records for specific DOWN nodes DownNodes=dev20 State=DOWN Reason="power,ETA=Dec25" # # Partition Configurations # PartitionName=DEFAULT MaxTime=30 MaxNodes=10 State=UP PartitionName=debug Nodes=dev[0-8,18-25] Default=YES PartitionName=batch Nodes=dev[9-17] MinNodes=4 PartitionName=long Nodes=dev[9-17] MaxTime=120 AllowGroups=admin ``` -------------------------------- ### Example JSON Response for Get WCKey Info Source: https://slurm.schedmd.com/archive/slurm-25.05.1/rest_api Example JSON data representing the response from the 'Get WCKey Info' API call. It includes WCKey details, metadata about the Slurm cluster, plugin information, client source, and any warnings or errors encountered. ```json { "wckeys" : [ { "cluster" : "cluster", "name" : "name", "flags" : [ "DELETED", "DELETED" ], "accounting" : [ { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 }, { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 } ], "id" : 7, "user" : "user" }, { "cluster" : "cluster", "name" : "name", "flags" : [ "DELETED", "DELETED" ], "accounting" : [ { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 }, { "start" : 2, "id" : 5, "TRES" : { "name" : "name", "count" : 0, "id" : 7, "type" : "type" }, "allocated" : { "seconds" : 1 }, "id_alt" : 5 } ], "id" : 7, "user" : "user" } ], "meta" : { "slurm" : { "cluster" : "cluster", "release" : "release", "version" : { "major" : "major", "minor" : "minor", "micro" : "micro" } }, "plugin" : { "accounting_storage" : "accounting_storage", "name" : "name", "type" : "type", "data_parser" : "data_parser" }, "client" : { "source" : "source", "user" : "user", "group" : "group" }, "command" : [ "command", "command" ] }, "warnings" : [ { "description" : "description", "source" : "source" }, { "description" : "description", "source" : "source" } ], "errors" : [ { "description" : "description", "source" : "source", "error" : "error", "error_number" : 7 }, { "description" : "description", "source" : "source", "error" : "error", "error_number" : 7 } ] } ``` -------------------------------- ### Get TRES Information Example Data Source: https://slurm.schedmd.com/archive/slurm-master/rest_api Example JSON data structure for TRES information, including metadata about the Slurm cluster, plugins, and client. ```json { "meta": { "slurm": { "cluster": "cluster", "release": "release", "version": { "major": "major", "minor": "minor", "micro": "micro" } }, "plugin": { "accounting_storage": "accounting_storage", "name": "name", "type": "type", "data_parser": "data_parser" }, "client": { "source": "source", "user": "user", "group": "group" }, "command": [ "command", "command" ] }, "warnings": [ { "description": "description", "source": "source" } ], "TRES": [ { "name": "name", "count": 0, "id": 7, "type": "type" } ], "errors": [ { "description": "description", "source": "source", "error": "error", "error_number": 7 } ] } ``` -------------------------------- ### Get TRES Information Example Data Source: https://slurm.schedmd.com/archive/slurm-25.05.1/rest_api Example JSON data structure for TRES information, including metadata about the Slurm cluster, plugins, and client. ```json { "meta": { "slurm": { "cluster": "cluster", "release": "release", "version": { "major": "major", "minor": "minor", "micro": "micro" } }, "plugin": { "accounting_storage": "accounting_storage", "name": "name", "type": "type", "data_parser": "data_parser" }, "client": { "source": "source", "user": "user", "group": "group" }, "command": [ "command", "command" ] }, "warnings": [ { "description": "description", "source": "source" } ], "TRES": [ { "name": "name", "count": 0, "id": 7, "type": "type" } ], "errors": [ { "description": "description", "source": "source", "error": "error", "error_number": 7 } ] } ``` -------------------------------- ### Configure Cloud Info on Slurmd Startup Source: https://slurm.schedmd.com/elastic_computing.html Set instance ID, type, and arbitrary extra data when starting slurmd. Ensure all necessary arguments are provided. ```bash $ slurmd --instance-id=12345 --instance-type=m7g.medium --extra="arbitrary string" . . . ``` -------------------------------- ### Get TRES Information Example Data Source: https://slurm.schedmd.com/archive/slurm-25.05.0/rest_api Example JSON data structure for TRES information, including metadata about the Slurm cluster, plugins, and client. ```json { "meta": { "slurm": { "cluster": "cluster", "release": "release", "version": { "major": "major", "minor": "minor", "micro": "micro" } }, "plugin": { "accounting_storage": "accounting_storage", "name": "name", "type": "type", "data_parser": "data_parser" }, "client": { "source": "source", "user": "user", "group": "group" }, "command": [ "command", "command" ] }, "warnings": [ { "description": "description", "source": "source" } ], "TRES": [ { "name": "name", "count": 0, "id": 7, "type": "type" } ], "errors": [ { "description": "description", "source": "source", "error": "error", "error_number": 7 } ] } ```