Slurm Workload Manager

https://slurm.schedmd.com/
Admin
Slurm Workload Manager is an open-source job scheduler and resource management system for HPC...
Tokens:383,314
Snippets:4,445
Trust Score:10
Update:3 months ago
Show doc for...
Tokens:
Raw
### Specify Job Start Time with --begin

Source: https://slurm.schedmd.com/salloc.html

Examples of valid time formats for the --begin option to defer job allocation.

```text
   --begin=16:00
   --begin=now+1hour
   --begin=now+60           (seconds by default)
   --begin=2010-01-20T12:34:00
```

--------------------------------

### Enable and Start MUNGE Service

Source: https://slurm.schedmd.com/authentication.html

Enables the MUNGE service to start on boot and starts the service immediately. This should be done on all machines using MUNGE for authentication.

```bash
systemctl enable munge
systemctl start munge
```

--------------------------------

### Install Slurm and Verify PMIx Libraries

Source: https://slurm.schedmd.com/mpi_guide.html

After successful configuration, install Slurm using 'make install'. The output shows the created PMIx MPI plugin libraries, including different versions.

```bash
user@testbox:~/slurm/22.05/build$ make -j install
user@testbox:~/slurm/22.05/build$ cd /home/user/slurm/22.05/inst/lib/slurm/
user@testbox:~/slurm/22.05/inst/lib/slurm$ ls -l *pmix*
lrwxrwxrwx 1 user user      16 jul  6 17:17 mpi_pmix.so -> ./mpi_pmix_v4.so
-rw-r--r-- 1 user user 9387254 jul  6 17:17 mpi_pmix_v3.a
-rwxr-xr-x 1 user user    1065 jul  6 17:17 mpi_pmix_v3.la
-rwxr-xr-x 1 user user 1265840 jul  6 17:17 mpi_pmix_v3.so
-rw-r--r-- 1 user user 9935358 jul  6 17:17 mpi_pmix_v4.a
-rwxr-xr-x 1 user user    1059 jul  6 17:17 mpi_pmix_v4.la
-rwxr-xr-x 1 user user 1286936 jul  6 17:17 mpi_pmix_v4.so
```

--------------------------------

### Task Invocation Behavior Examples

Source: https://slurm.schedmd.com/mc_support.html

Command line examples demonstrating task distribution across nodes, sockets, and cores.

```bash
% srun -n 4 hostname
hydra12
hydra12
hydra12
hydra12
% srun -n 4 --ntasks-per-node=1 hostname
hydra12
hydra13
hydra14
hydra15
% srun -n 4 --ntasks-per-node=2 hostname
hydra12
hydra12
hydra13
hydra13
% srun -n 4 --ntasks-per-socket=1 hostname
hydra12
hydra12
hydra13
hydra13
% srun -n 4 --ntasks-per-core=1 hostname
hydra12
hydra12
hydra12
hydra12
```

--------------------------------

### Build and install nss_slurm

Source: https://slurm.schedmd.com/nss_slurm.html

Compile and install the plugin from the Slurm source directory.

```bash
make && make install
```

--------------------------------

### Task Distribution Command Examples

Source: https://slurm.schedmd.com/mc_support.html

Examples of combining distribution flags with CPU binding for srun jobs.

```bash
srun -n 16 -N 4 -B '2:*:1' -m block:cyclic --cpu-bind=socket a.out
```

```bash
srun -n 16 -N 4 -B '2:*:1' -m plane=2 --cpu-bind=core a.out
```

```bash
srun -n 16 -N 4 -B '2:*:1' -m plane=2 a.out
```

--------------------------------

### Install PMI-1 and PMI-2 Support from Contribs

Source: https://slurm.schedmd.com/mpi_guide.html

To enable support for PMI-1 or PMI-2, navigate to the respective directories in the Slurm contribs and run 'make install'.

```bash
user@testbox:~/slurm/22.05/build/$ cd contribs/pmi1
user@testbox:~/slurm/22.05/build/contribs/pmi1$ make -j install

user@testbox:~/slurm/22.05/build/$ cd contribs/pmi2
user@testbox:~/slurm/22.05/build/contribs/pmi2$ make -j install
```

--------------------------------

### Memory Binding Examples

Source: https://slurm.schedmd.com/sbatch.html

Examples of memory binding configurations using map_mem and mask_mem syntax.

```text
map_mem:0x0f*4,0xf0*4
```

```text
mask_mem:0*4,1*4
```

--------------------------------

### SrunPortRange Usage Examples

Source: https://slurm.schedmd.com/slurm.conf.html

Examples showing the number of listening ports used by srun based on node count and options.

```text
srun -N 1        will use 4 listening ports.
srun --pty -N 1  will use 5 listening ports.
srun -N 48       will use 4 listening ports.
srun -N 50       will use 6 listening ports.
srun -N 200      will use 12 listening ports.
```

--------------------------------

### Build and install Slurm from source on FreeBSD

Source: https://slurm.schedmd.com/quickstart_admin.html

Navigate to the ports directory and execute the make install command to build from source, allowing for custom configuration options.

```shell
cd /usr/ports/sysutils/slurm-wlm && make install
```

--------------------------------

### Configure nvidia-container-runtime in oci.conf

Source: https://slurm.schedmd.com/containers.html

Examples for setting up nvidia-container-runtime. Using the 'run' command is recommended over 'create/start'.

```text
EnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeEnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeQuery="nvidia-container-runtime --rootless=true --root=/run/user/%U/ state %n.%u.%j.%s.%t"
RunTimeCreate="nvidia-container-runtime --rootless=true --root=/run/user/%U/ create %n.%u.%j.%s.%t -b %b"
RunTimeStart="nvidia-container-runtime --rootless=true --root=/run/user/%U/ start %n.%u.%j.%s.%t"
RunTimeKill="nvidia-container-runtime --rootless=true --root=/run/user/%U/ kill -a %n.%u.%j.%s.%t"
RunTimeDelete="nvidia-container-runtime --rootless=true --root=/run/user/%U/ delete --force %n.%u.%j.%s.%t"
```

```text
EnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeEnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeQuery="nvidia-container-runtime --rootless=true --root=/run/user/%U/ state %n.%u.%j.%s.%t"
RunTimeKill="nvidia-container-runtime --rootless=true --root=/run/user/%U/ kill -a %n.%u.%j.%s.%t"
RunTimeDelete="nvidia-container-runtime --rootless=true --root=/run/user/%U/ delete --force %n.%u.%j.%s.%t"
RunTimeRun="nvidia-container-runtime --rootless=true --root=/run/user/%U/ run %n.%u.%j.%s.%t -b %b"
```

--------------------------------

### Example acct_gather.conf configuration

Source: https://slurm.schedmd.com/acct_gather.conf.html

A sample configuration file demonstrating settings for energy, profile, and interconnect plugins.

```text
###
# Slurm acct_gather configuration file
###
# Parameters for acct_gather_energy/impi plugin
EnergyIPMIFrequency=10
EnergyIPMICalcAdjustment=yes
#
# Parameters for acct_gather_profile/hdf5 plugin
ProfileHDF5Dir=/app/slurm/profile_data
# Parameters for acct_gather_interconnect/ofed plugin
InfinibandOFEDPort=1
```

--------------------------------

### Consumable Resource: CR_CORE_MEMORY

Source: https://slurm.schedmd.com/gang_scheduling.html

This example shows the behavior with CR_CORE_MEMORY configured. Jobs 46 and 47 do not get suspended because they are not sharing cores. Jobs 48 and 49 share cores with jobs 44 and 45, leading to timeslicing only between those pairs.

```bash
[user@n16 load]$ **sinfo**
PARTITION AVAIL  TIMELIMIT NODES  STATE NODELIST
active*      up   infinite     5   idle n[12-16]

[user@n16 load]$ **scontrol show config | grep Select**
SelectType              = select/cons_tres
SelectTypeParameters    = CR_CORE_MEMORY

[user@n16 load]$ **sinfo -o "%20N %5D %5c %5z"**
NODELIST             NODES CPUS  S:C:T
n[12-16]             5     8     2:4:1

[user@n16 load]$ **sbatch -n10 -N5 ./myload 300**
sbatch: Submitted batch job 44

[user@n16 load]$ **sbatch -n10 -N5 ./myload 300**
sbatch: Submitted batch job 45

[user@n16 load]$ **sbatch -n10 -N5 ./myload 300**
sbatch: Submitted batch job 46

[user@n16 load]$ **sbatch -n10 -N5 ./myload 300**
sbatch: Submitted batch job 47

[user@n16 load]$ **sbatch -n10 -N5 ./myload 300**
sbatch: Submitted batch job 48

[user@n16 load]$ **sbatch -n10 -N5 ./myload 300**
sbatch: Submitted batch job 49

[user@n16 load]$ **squeue**
JOBID PARTITION    NAME  USER ST  TIME NODES NODELIST
   44    active  myload  user  R  0:09     5 n[12-16]
   45    active  myload  user  R  0:08     5 n[12-16]
   46    active  myload  user  R  0:08     5 n[12-16]
   47    active  myload  user  R  0:07     5 n[12-16]
   48    active  myload  user  S  0:00     5 n[12-16]
   49    active  myload  user  S  0:00     5 n[12-16]

[user@n16 load]$ **squeue**
JOBID PARTITION    NAME  USER ST  TIME NODES NODELIST
   46    active  myload  user  R  0:49     5 n[12-16]
   47    active  myload  user  R  0:48     5 n[12-16]
   48    active  myload  user  R  0:06     5 n[12-16]
   49    active  myload  user  R  0:06     5 n[12-16]
   44    active  myload  user  S  0:44     5 n[12-16]
   45    active  myload  user  S  0:43     5 n[12-16]

[user@n16 load]$ **squeue**
JOBID PARTITION    NAME  USER ST  TIME NODES NODELIST
   44    active  myload  user  R  1:23     5 n[12-16]
   45    active  myload  user  R  1:22     5 n[12-16]
   46    active  myload  user  R  2:22     5 n[12-16]
   47    active  myload  user  R  2:21     5 n[12-16]
   48    active  myload  user  S  1:00     5 n[12-16]
   49    active  myload  user  S  1:00     5 n[12-16]

```

--------------------------------

### View multiple program configuration file

Source: https://slurm.schedmd.com/srun.html

Example of a configuration file for multiple program execution.

```bash
$ cat silly.conf
###################################################################
```

--------------------------------

### Verify PMI-1 and PMI-2 Library Installation

Source: https://slurm.schedmd.com/mpi_guide.html

After installing PMI-1 and PMI-2 from contribs, verify the presence of their respective libraries in the Slurm installation's lib directory.

```bash
user@testbox:~/$ ls -l /home/user/slurm/22.05/inst/lib/*pmi*
-rw-r--r-- 1 user user 493024 jul  6 17:27 libpmi2.a
-rwxr-xr-x 1 user user    987 jul  6 17:27 libpmi2.la
lrwxrwxrwx 1 user user     16 jul  6 17:27 libpmi2.so -> libpmi2.so.0.0.0
lrwxrwxrwx 1 user user     16 jul  6 17:27 libpmi2.so.0 -> libpmi2.so.0.0.0
-rwxr-xr-x 1 user user 219712 jul  6 17:27 libpmi2.so.0.0.0
-rw-r--r-- 1 user user 427768 jul  6 17:27 libpmi.a
-rwxr-xr-x 1 user user   1039 jul  6 17:27 libpmi.la
lrwxrwxrwx 1 user user     15 jul  6 17:27 libpmi.so -> libpmi.so.0.0.0
lrwxrwxrwx 1 user user     15 jul  6 17:27 libpmi.so.0 -> libpmi.so.0.0.0
-rwxr-xr-x 1 user user 241640 jul  6 17:27 libpmi.so.0.0.0
```

--------------------------------

### TRES Specification Examples

Source: https://slurm.schedmd.com/scontrol.html

Examples of valid TRES configurations for reservations.

```text
TRES=cpu=5,bb/cray=4,license/iop1=1,license/iop2=3
```

```text
TRES=node=5k,license/iop1=2
```

```text
TRES=gres/gpu:a100=2
```

--------------------------------

### Slurm Configuration File Example

Source: https://slurm.schedmd.com/overview.html

A sample configuration file for Slurm daemons, demonstrating settings for hostnames, authentication, plugin directories, ports, timeouts, and spool directories. This file is essential for setting up the Slurm controller and daemon.

```bash
#
# Sample /etc/slurm.conf
#
SlurmctldHost=linux0001  # Primary server
SlurmctldHost=linux0002  # Backup server
#
AuthType=auth/munge
Epilog=/usr/local/slurm/sbin/epilog
PluginDir=/usr/local/slurm/lib
Prolog=/usr/local/slurm/sbin/prolog
SlurmctldPort=7002
SlurmctldTimeout=120
SlurmdPort=7003
SlurmdSpoolDir=/var/tmp/slurmd.spool
SlurmdTimeout=120
StateSaveLocation=/usr/local/slurm/slurm.state
TmpFS=/tmp

```

--------------------------------

### Create Resource Allocation and Execute Job Steps

Source: https://slurm.schedmd.com/quickstart.html

Creates a resource allocation for 1024 nodes, starts a bash shell within it, transfers an executable 'a.out' to '/tmp/joe.a.out' on allocated nodes using 'sbcast', executes the program using 'srun', removes the executable, and then relinquishes the allocation.

```bash
salloc -N1024 bash
```

```bash
sbcast a.out /tmp/joe.a.out
```

```bash
srun /tmp/joe.a.out
```

```bash
srun rm /tmp/joe.a.out
```

```bash
exit
```

--------------------------------

### Reload and Start Docker

Source: https://slurm.schedmd.com/containers.html

Commands to reload systemd and start the docker service.

```bash
systemctl --user daemon-reload
```

```bash
systemctl --user start docker
```

--------------------------------

### Ping Response Example

Source: https://slurm.schedmd.com/rest_api.html

Example JSON response structure for the ping endpoint.

```json
{
  "meta" : {
    "slurm" : {
      "cluster" : "cluster",
      "release" : "release",
      "version" : {
        "major" : "major",
        "minor" : "minor",
        "micro" : "micro"
      }
    },
    "plugin" : {
      "accounting_storage" : "accounting_storage",
      "name" : "name",
      "type" : "type",
      "data_parser" : "data_parser"
    },
    "client" : {
      "source" : "source",
      "user" : "user",
      "group" : "group"
    },
    "command" : [ "command", "command" ]
  },
  "warnings" : [ {
    "description" : "description",
    "source" : "source"
  }, {
    "description" : "description",
    "source" : "source"
  } ],
  "pings" : [ {
    "responding" : true,
    "hostname" : "hostname",
    "latency" : 0,
    "primary" : true,
    "status" : "status"
  }, {
    "responding" : true,
    "hostname" : "hostname",
    "latency" : 0,
    "primary" : true,
    "status" : "status"
  } ],
  "errors" : [ {
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  }, {
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  } ]
}
```

--------------------------------

### Hostlist Function Usage Examples

Source: https://slurm.schedmd.com/topology.html

Demonstrates the application of topology-based hostlist functions in scontrol, sbatch, and configuration files.

```bash
scontrol update node=block{b1} state=resume
sbatch --nodelist=switchwith{node0} -N 10 program
PartitionName=Ring10 Nodes=ring{ring10} ...
```

--------------------------------

### Diagnostic Response Example

Source: https://slurm.schedmd.com/rest_api.html

Example JSON response body for the diagnostic endpoint.

```json
{
  "meta" : {
    "slurm" : {
      "cluster" : "cluster",
      "release" : "release",
      "version" : {
        "major" : "major",
        "minor" : "minor",
        "micro" : "micro"
      }
    },
    "plugin" : {
      "accounting_storage" : "accounting_storage",
      "name" : "name",
      "type" : "type",
      "data_parser" : "data_parser"
    },
    "client" : {
      "source" : "source",
      "user" : "user",
      "group" : "group"
    },
    "command" : [ "command", "command" ]
  },
  "warnings" : [ {
    "description" : "description",
    "source" : "source"
  }, {
    "description" : "description",
    "source" : "source"
  } ],
  "errors" : [ {
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  }, {
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  } ],
  "statistics" : {
    "bf_cycle_max" : 0,
    "rpcs_by_message_type" : [ {
      "cycle_last" : 9,
      "average_time" : {
        "number" : 2,
        "set" : true,
        "infinite" : true
      },
      "type_id" : 4,
      "queued" : 9,
      "count" : 7,
      "dropped" : 0,
      "message_type" : "message_type",
      "total_time" : 7,
      "cycle_max" : 0
    }, {
      "cycle_last" : 9,
      "average_time" : {
        "number" : 2,
        "set" : true,
        "infinite" : true
      },
      "type_id" : 4,
      "queued" : 9,
      "count" : 7,
      "dropped" : 0,
      "message_type" : "message_type",
      "total_time" : 7,
      "cycle_max" : 0
    } ],
    "bf_backfilled_het_jobs" : 6,
    "bf_table_size" : 6,
    "schedule_cycle_depth" : 1,
    "bf_depth_sum" : 7,
    "job_states_ts" : {
      "number" : 2,
      "set" : true,
      "infinite" : true
    },
    "bf_queue_len" : 3,
    "jobs_started" : 6,
    "schedule_cycle_max" : 7,
    "server_thread_count" : 0,
    "bf_queue_len_sum" : 0,
    "bf_cycle_last" : 7,
    "bf_exit" : {
      "state_changed" : 7,
      "bf_max_time" : 4,
      "bf_max_job_start" : 6,
      "bf_node_space_size" : 8,
      "end_job_queue" : 7,
      "bf_max_job_test" : 0
    },
    "agent_thread_count" : 5,
    "jobs_completed" : 8,
    "bf_depth_mean" : 6,
    "bf_depth_try_sum" : 5,
    "schedule_cycle_mean" : 4,
    "bf_table_size_sum" : 4,
    "agent_queue_size" : 6,
    "jobs_failed" : 6,
    "bf_last_depth_try" : 3,
    "req_time" : {
      "number" : 2,
      "set" : true,
      "infinite" : true
    },
    "bf_cycle_counter" : 6,
    "schedule_queue_length" : 9,
    "bf_queue_len_mean" : 4,
    "schedule_exit" : {
      "max_sched_time" : 4,
      "licenses" : 5,
      "default_queue_depth" : 6,
      "max_job_start" : 7,
      "max_rpc_cnt" : 1,
      "end_job_queue" : 1
    },
    "jobs_canceled" : 9,
    "schedule_cycle_sum" : 3,
    "jobs_submitted" : 9,
    "schedule_cycle_mean_depth" : 7,
    "schedule_cycle_per_minute" : 1,
    "req_time_start" : {
      "number" : 2,
      "set" : true,
      "infinite" : true
    },
    "jobs_running" : 6,
    "bf_last_backfilled_jobs" : 2,
    "bf_last_depth" : 3,
    "bf_backfilled_jobs" : 1,
    "rpcs_by_user" : [ {
      "average_time" : {
        "number" : 2,
        "set" : true,
        "infinite" : true
      },
      "user_id" : 5,
      "count" : 4,
      "total_time" : 6,
      "user" : "user"
    }, {
      "average_time" : {
        "number" : 2,
        "set" : true,
        "infinite" : true
      },
      "user_id" : 5,
      "count" : 4,
      "total_time" : 6,
      "user" : "user"
    } ],
    "bf_cycle_mean" : 5,
    "pending_rpcs_by_hostlist" : [ {
      "type_id" : 3,
      "count" : [ "count", "count" ],
      "message_type" : "message_type"
    }, {
      "type_id" : 3,
      "count" : [ "count", "count" ],
      "message_type" : "message_type"
    } ],
    "dbd_agent_queue_size" : 5,
    "bf_table_size_mean" : 1,
    "jobs_pending" : 3,
    "agent_count" : 1,
    "bf_cycle_sum" : 3,
    "bf_active" : true,
    "bf_depth_mean_try" : 3,
    "gettimeofday_latency" : 2,
    "pending_rpcs" : [ {
      "type_id" : 8,
      "count" : 4,
      "message_type" : "message_type"
    }, {
      "type_id" : 8,
      "count" : 4,
      "message_type" : "message_type"
    } ],
    "schedule_cycle_total" : 2,
    "bf_when_last_cycle" : {
      "number" : 2,
      "set" : true,
      "infinite" : true
    },
    "schedule_cycle_last" : 9
  }
}
```

--------------------------------

### Show Available QOS

Source: https://slurm.schedmd.com/sacctmgr.html

List all QOS names configured in the cluster.

```bash
$ sacctmgr show qos format=name
       Name
  ---------
     normal
   expedite
```

--------------------------------

### Cluster Information Response Example

Source: https://slurm.schedmd.com/rest_api.html

Example JSON response for the cluster information endpoint.

```json
{
  "meta" : {
    "slurm" : {
      "cluster" : "cluster",
      "release" : "release",
      "version" : {
        "major" : "major",
        "minor" : "minor",
        "micro" : "micro"
      }
    },
    "plugin" : {
      "accounting_storage" : "accounting_storage",
      "name" : "name",
      "type" : "type",
      "data_parser" : "data_parser"
    },
    "client" : {
      "source" : "source",
      "user" : "user",
      "group" : "group"
    },
    "command" : [ "command", "command" ]
  },
  "warnings" : [ {
    "description" : "description",
    "source" : "source"
  }, {
    "description" : "description",
    "source" : "source"
  } ],
  "clusters" : [ {
    "associations" : {
      "root" : {
        "cluster" : "cluster",
        "partition" : "partition",
        "id" : 5,
        "user" : "user",
        "account" : "account"
      }
    },
    "controller" : {
      "port" : 0,
      "host" : "host"
    },
    "nodes" : "nodes",
    "flags" : [ "DELETED", "DELETED" ],
    "name" : "name",
    "rpc_version" : 6,
    "tres" : [ {
      "name" : "name",
      "count" : 0,
      "id" : 7,
      "type" : "type"
    }, {
      "name" : "name",
      "count" : 0,
      "id" : 7,
      "type" : "type"
    } ]
  }, {
    "associations" : {
      "root" : {
        "cluster" : "cluster",
        "partition" : "partition",
        "id" : 5,
        "user" : "user",
        "account" : "account"
      }
    },
    "controller" : {
      "port" : 0,
      "host" : "host"
    },
    "nodes" : "nodes",
    "flags" : [ "DELETED", "DELETED" ],
    "name" : "name",
    "rpc_version" : 6,
    "tres" : [ {
      "name" : "name",
      "count" : 0,
      "id" : 7,
      "type" : "type"
    }, {
      "name" : "name",
      "count" : 0,
      "id" : 7,
      "type" : "type"
    } ]
  } ],
  "errors" : [ {
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  }, {
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  } ]
}
```

--------------------------------

### Implement ResumeProgram script

Source: https://slurm.schedmd.com/elastic_computing.html

A bash script example for resuming nodes using scontrol to expand hostlists and executing a node_startup command.

```bash
#!/bin/bash
# Example ResumeProgram
hosts=$(scontrol show hostnames "$1")
logfile=/var/log/power_save.log
echo "$(date) Resume invoked $0 $*" >>$logfile
for host in $hosts
do
        sudo node_startup "$host"
done
exit 0
```

--------------------------------

### Automate Keycloak Startup

Source: https://slurm.schedmd.com/jwt.html

A shell script to bootstrap a Keycloak admin service and start the development server.

```bash
#!/bin/bash
export SECRET=secret
/opt/keycloak/bin/kc.sh bootstrap-admin service --client-id test --client-secret:env=SECRET
exec /opt/keycloak/bin/kc.sh start-dev --log=console --log-console-level=debug --http-enabled true
```

--------------------------------

### Delete WCKey Response Example

Source: https://slurm.schedmd.com/rest_api.html

Example JSON response for a successful WCKey deletion.

```json
{
  "deleted_wckeys" : [ "deleted_wckeys", "deleted_wckeys" ],
  "meta" : {
    "slurm" : {
      "cluster" : "cluster",
      "release" : "release",
      "version" : {
        "major" : "major",
        "minor" : "minor",
        "micro" : "micro"
      }
    },
    "plugin" : {
      "accounting_storage" : "accounting_storage",
      "name" : "name",
      "type" : "type",
      "data_parser" : "data_parser"
    },
    "client" : {
      "source" : "source",
      "user" : "user",
      "group" : "group"
    },
    "command" : [ "command", "command" ]
  },
  "warnings" : [ {
    "description" : "description",
    "source" : "source"
  }, {
    "description" : "description",
    "source" : "source"
  } ],
  "errors" : [ {
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  }, {
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  } ]
}
```

--------------------------------

### Delete QOS Response Example

Source: https://slurm.schedmd.com/rest_api.html

Example JSON response for a successful QOS deletion.

```json
{
  "meta" : {
    "slurm" : {
      "cluster" : "cluster",
      "release" : "release",
      "version" : {
        "major" : "major",
        "minor" : "minor",
        "micro" : "micro"
      }
    },
    "plugin" : {
      "accounting_storage" : "accounting_storage",
      "name" : "name",
      "type" : "type",
      "data_parser" : "data_parser"
    },
    "client" : {
      "source" : "source",
      "user" : "user",
      "group" : "group"
    },
    "command" : [ "command", "command" ]
  },
  "warnings" : [ {
    "description" : "description",
    "source" : "source"
  }, {
    "description" : "description",
    "source" : "source"
  } ],
  "removed_qos" : [ "removed_qos", "removed_qos" ],
  "errors" : [ {
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  }, {
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  } ]
}
```

--------------------------------

### Reservation Response Example

Source: https://slurm.schedmd.com/rest_api.html

Example JSON response structure for a reservation information request.

```json
{
  "reservations" : [ {
    "end_time" : {
      "number" : 2,
      "set" : true,
      "infinite" : true
    },
    "flags" : [ "MAINT", "MAINT" ],
    "groups" : "groups",
    "users" : "users",
    "max_start_delay" : 6,
    "features" : "features",
    "start_time" : {
      "number" : 2,
      "set" : true,
      "infinite" : true
    },
    "burst_buffer" : "burst_buffer",
    "licenses" : "licenses",
    "partition" : "partition",
    "core_specializations" : [ {
      "node" : "node",
      "core" : "core"
    }, {
      "node" : "node",
      "core" : "core"
    } ],
    "name" : "name",
    "tres" : "tres",
    "accounts" : "accounts",
    "node_count" : 1,
    "node_list" : "node_list",
    "purge_completed" : {
      "time" : {
        "number" : 5,
        "set" : true,
        "infinite" : true
      }
    },
    "core_count" : 0
  }, {
    "end_time" : {
      "number" : 2,
      "set" : true,
      "infinite" : true
    },
    "flags" : [ "MAINT", "MAINT" ],
    "groups" : "groups",
    "users" : "users",
    "max_start_delay" : 6,
    "features" : "features",
    "start_time" : {
      "number" : 2,
      "set" : true,
      "infinite" : true
    },
    "burst_buffer" : "burst_buffer",
    "licenses" : "licenses",
    "partition" : "partition",
    "core_specializations" : [ {
      "node" : "node",
      "core" : "core"
    }, {
      "node" : "node",
      "core" : "core"
    } ],
    "name" : "name",
    "tres" : "tres",
    "accounts" : "accounts",
    "node_count" : 1,
    "node_list" : "node_list",
    "purge_completed" : {
      "time" : {
        "number" : 5,
        "set" : true,
        "infinite" : true
      }
    },
    "core_count" : 0
  } ],
  "meta" : {
    "slurm" : {
      "cluster" : "cluster",
      "release" : "release",
      "version" : {
        "major" : "major",
        "minor" : "minor",
        "micro" : "micro"
      }
    },
    "plugin" : {
      "accounting_storage" : "accounting_storage",
      "name" : "name",
      "type" : "type",
      "data_parser" : "data_parser"
    },
    "client" : {
      "source" : "source",
      "user" : "user",
      "group" : "group"
    },
    "command" : [ "command", "command" ]
  },
  "last_update" : {
    "number" : 2,
    "set" : true,
    "infinite" : true
  },
  "warnings" : [ {
    "description" : "description",
    "source" : "source"
  }, {
    "description" : "description",
    "source" : "source"
  } ],
  "errors" : [ {
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  }, {
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  } ]
}
```

--------------------------------

### Configure and Run OCI Bundle

Source: https://slurm.schedmd.com/containers.html

Generate a runtime configuration and execute the container using Slurm.

```bash
cd ~/oci_images/alpine
runc --rootless=true spec --rootless
```

```bash
srun --container ~/oci_images/alpine/ uptime
```

--------------------------------

### Partition Response Example

Source: https://slurm.schedmd.com/rest_api.html

Example JSON response body for the partition information endpoint.

```json
{
  "partitions" : [ {
    "cluster" : "cluster",
    "cpus" : {
      "task_binding" : [ "CPU_BIND_TO_THREADS", "CPU_BIND_TO_THREADS" ],
      "total" : 6
    },
    "topology" : "topology",
    "flags" : [ "DEFAULT", "DEFAULT" ],
    "timeouts" : {
      "resume" : {
        "number" : 2,
        "set" : true,
        "infinite" : true
      },
      "suspend" : {
        "number" : 2,
        "set" : true,
        "infinite" : true
      }
    },
    "groups" : {
      "allowed" : "allowed"
    },
    "alternate" : "alternate",
    "select_type" : [ "NONE", "NONE" ],
    "suspend_time" : {
      "number" : 5,
      "set" : true,
      "infinite" : true
    },
    "priority" : {
      "tier" : 2,
      "job_factor" : 3
    },
    "node_sets" : "node_sets",
    "maximums" : {
      "shares" : 2,
      "nodes" : {
        "number" : 5,
        "set" : true,
        "infinite" : true
      },
      "over_time_limit" : {
        "number" : 2,
        "set" : true,
        "infinite" : true
      },
      "cpus_per_node" : {
        "number" : 5,
        "set" : true,
        "infinite" : true
      },
      "cpus_per_socket" : {
        "number" : 5,
        "set" : true,
        "infinite" : true
      },
      "partition_memory_per_node" : {
        "number" : 2,
        "set" : true,
        "infinite" : true
      },
      "oversubscribe" : {
        "jobs" : 7,
        "flags" : [ "force", "force" ]
      },
      "memory_per_cpu" : 5,
      "time" : {
        "number" : 5,
        "set" : true,
        "infinite" : true
      },
      "partition_memory_per_cpu" : {
        "number" : 2,
        "set" : true,
        "infinite" : true
      }
    },
    "preempt_mode" : [ "CLUSTER_GLOBAL", "CLUSTER_GLOBAL" ],
    "nodes" : {
      "configured" : "configured",
      "total" : 0,
      "allowed_allocation" : "allowed_allocation"
    },
    "partition" : {
      "oversubscribe" : "oversubscribe",
      "exclusive" : "exclusive",
      "state" : [ "INACTIVE", "INACTIVE" ]
    },
    "qos" : {
      "deny" : "deny",
      "allowed" : "allowed",
      "assigned" : "assigned"
    },
    "defaults" : {
      "partition_memory_per_node" : {
        "number" : 2,
        "set" : true,
        "infinite" : true
      },
      "memory_per_cpu" : 1,
      "time" : {
        "number" : 5,
        "set" : true,
        "infinite" : true
      },
      "job" : "job",
      "partition_memory_per_cpu" : {
        "number" : 2,
        "set" : true,
        "infinite" : true
      }
    },
    "name" : "name",
    "tres" : {
      "configured" : "configured",
      "billing_weights" : "billing_weights"
    },
    "accounts" : {
      "deny" : "deny",
      "allowed" : "allowed"
    },
    "minimums" : {
      "nodes" : 9
    },
    "grace_time" : 5
  }, {
    "cluster" : "cluster",
    "cpus" : {
      "task_binding" : [ "CPU_BIND_TO_THREADS", "CPU_BIND_TO_THREADS" ],
      "total" : 6
    },
    "topology" : "topology",
    "flags" : [ "DEFAULT", "DEFAULT" ],
    "timeouts" : {
      "resume" : {
        "number" : 2,
        "set" : true,
        "infinite" : true
      },
      "suspend" : {
        "number" : 2,
        "set" : true,
        "infinite" : true
      }
    },
    "groups" : {
      "allowed" : "allowed"
    },
    "alternate" : "alternate",
    "select_type" : [ "NONE", "NONE" ],
    "suspend_time" : {
      "number" : 5,
      "set" : true,
      "infinite" : true
    },
    "priority" : {
      "tier" : 2,
      "job_factor" : 3
    },
    "node_sets" : "node_sets",
    "maximums" : {
      "shares" : 2,
      "nodes" : {
        "number" : 5,
        "set" : true,
        "infinite" : true
      },
      "over_time_limit" : {
        "number" : 2,
        "set" : true,
        "infinite" : true
      },
      "cpus_per_node" : {
        "number" : 5,
        "set" : true,
        "infinite" : true
      },
      "cpus_per_socket" : {
        "number" : 5,
        "set" : true,
        "infinite" : true
      }
```

--------------------------------

### Configure job_container.conf base paths

Source: https://slurm.schedmd.com/job_container.conf.html

Sample configurations for defining base paths and node-specific settings in job_container.conf.

```text
AutoBasePath=true
BasePath=/var/nvme/storage
```

```text
NodeName=largemem[1-2] AutoBasePath=true BasePath=/var/nvme/storage_a
NodeName=gpu[1-10] BasePath=/var/nvme/storage_b InitScript=/etc/slurm/init.sh
```

```text
AutoBasePath=true
BasePath=/var/nvme/storage Dirs=/tmp,/var/tmp
```

--------------------------------

### Configure Node Features in helpers.conf

Source: https://slurm.schedmd.com/helpers.conf.html

Example configuration mapping nodes and features to specific helper scripts.

```text
NodeName=n1_[1-10] Feature=a1,a2 Helper=/path/helper.sh
NodeName=n2_[1-10] Feature=b1,b2 Helper=/path/helper.sh
Feature=c1,c2 Helper=/path/helper.sh
```

--------------------------------

### TRESPerNode Specification Example

Source: https://slurm.schedmd.com/scontrol.html

Example of allocating specific GRES resources per node.

```text
TRESPerNode=gres/gpu:a100=2
```

--------------------------------

### Job Submission Examples for Node Allocation

Source: https://slurm.schedmd.com/cons_tres.html

These examples show job submissions with varying CPU and node counts, illustrating how they are allocated under different resource management strategies.

```bash
# srun -n 4 -N 4 sleep 120 &
# srun -n 3 -N 3 sleep 120 &
# srun -n 1 sleep 120 &
# srun -n 3 sleep 120 &
```

--------------------------------

### scrun Start Operation

Source: https://slurm.schedmd.com/scrun.html

Requests to start and run the specified container within a Slurm job.

```bash
scrun [_GLOBAL OPTIONS_...] _start_ <_container-id_ >
```

--------------------------------

### Configure Charliecloud in oci.conf

Source: https://slurm.schedmd.com/containers.html

Configuration example for Charliecloud v0.30.

```text
IgnoreFileConfigJson=true
CreateEnvFile=newline
EnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeEnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeRun="env -i PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin/:/sbin/ USER=$(whoami) HOME=/home/$(whoami)/ ch-run -w --bind /etc/group:/etc/group --bind /etc/passwd:/etc/passwd --bind /etc/slurm:/etc/slurm --bind %m:/var/run/slurm/ --bind /var/run/munge/:/var/run/munge/ --set-env=%e --no-passwd %r -- %@"
RunTimeKill="kill -s SIGTERM %p"
RunTimeDelete="kill -s SIGKILL %p"
```
Tokens:
Raw