Slurm Workload Manager

https://slurm.schedmd.com/archive/slurm-25.05-latest
Admin
Slurm Workload Manager is an open-source job scheduler for Linux clusters, widely used in...
Tokens:492,933
Snippets:5,821
Trust Score:10
Update:3 months ago
Show doc for...
Tokens:
Raw
### runc oci.conf Example (Create/Start)

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/containers.html

Example configuration for the 'runc' OCI runtime using the 'create' and 'start' operations. This method requires Slurm to poll the runtime for job completion.

```bash
EnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeEnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeQuery="runc --rootless=true --root=/run/user/%U/ state %n.%u.%j.%s.%t"
RunTimeCreate="runc --rootless=true --root=/run/user/%U/ create %n.%u.%j.%s.%t -b %b"
RunTimeStart="runc --rootless=true --root=/run/user/%U/ start %n.%u.%j.%s.%t"
RunTimeKill="runc --rootless=true --root=/run/user/%U/ kill -a %n.%u.%j.%s.%t"
RunTimeDelete="runc --rootless=true --root=/run/user/%U/ delete --force %n.%u.%j.%s.%t"

```

--------------------------------

### crun oci.conf Example (Create/Start)

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/containers.html

Example configuration for the 'crun' OCI runtime using the 'create' and 'start' operations. This method requires Slurm to poll the runtime for job completion.

```bash
EnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeEnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeQuery="crun --rootless=true --root=/run/user/%U/ state %n.%u.%j.%s.%t"
RunTimeKill="crun --rootless=true --root=/run/user/%U/ kill -a %n.%u.%j.%s.%t"
RunTimeDelete="crun --rootless=true --root=/run/user/%U/ delete --force %n.%u.%j.%s.%t"
RunTimeCreate="crun --rootless=true --root=/run/user/%U/ create --bundle %b %n.%u.%j.%s.%t"
RunTimeStart="crun --rootless=true --root=/run/user/%U/ start %n.%u.%j.%s.%t"

```

--------------------------------

### nvidia-container-runtime oci.conf (create/start)

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/containers.html

Example oci.conf for nvidia-container-runtime using create and start commands. This method is not recommended over the 'run' command.

```bash
EnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeEnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeQuery="nvidia-container-runtime --rootless=true --root=/run/user/%U/ state %n.%u.%j.%s.%t"
RunTimeCreate="nvidia-container-runtime --rootless=true --root=/run/user/%U/ create %n.%u.%j.%s.%t -b %b"
RunTimeStart="nvidia-container-runtime --rootless=true --root=/run/user/%U/ start %n.%u.%j.%s.%t"
RunTimeKill="nvidia-container-runtime --rootless=true --root=/run/user/%U/ kill -a %n.%u.%j.%s.%t"
RunTimeDelete="nvidia-container-runtime --rootless=true --root=/run/user/%U/ delete --force %n.%u.%j.%s.%t"
```

--------------------------------

### Local Backfilling Example

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/gang_scheduling.html

Displays 'local backfilling' where a smaller job can start before a larger job finishes if it fits within the remaining resources.

```bash
[user@n16 load]$ **sbatch -N3 ./myload 300**
sbatch: Submitted batch job 12

[user@n16 load]$ **sbatch -N5 ./myload 300**
sbatch: Submitted batch job 13

[user@n16 load]$ **sbatch -N2 ./myload 300**
sbatch: Submitted batch job 14

[user@n16 load]$ **squeue**
JOBID PARTITION    NAME  USER ST  TIME NODES NODELIST
   12    active  myload  user  R  0:14     3 n[12-14]
   14    active  myload  user  R  0:06     2 n[15-16]
   13    active  myload  user  S  0:00     5 n[12-16]

```

--------------------------------

### gres.conf Example 1: Basic Sharding Setup

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/gres.html

Configures four GPUs and sets a shard count of 8 for each, using AutoDetect=nvml.

```gres.conf
# Example 1 of gres.conf
# Configure four GPUs (with Sharding)
AutoDetect=nvml
Name=gpu Type=gp100 File=/dev/nvidia0 Cores=0,1
Name=gpu Type=gp100 File=/dev/nvidia1 Cores=0,1
Name=gpu Type=p6000 File=/dev/nvidia2 Cores=2,3
Name=gpu Type=p6000 File=/dev/nvidia3 Cores=2,3
# Set gres/shard Count value to 8 on each of the 4 available GPUs
Name=shard Count=32


```

--------------------------------

### Example TRESPerNode Specification

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/scontrol.html

This example illustrates how to specify TRES resources that should be allocated per node within a reservation, such as GPUs.

```bash
TRESPerNode=gres/gpu:a100=2
```

--------------------------------

### Transient Systemd Unit File Example

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/cgroup_v2.html

Example of a transient systemd unit file created programmatically. Do not edit this file directly.

```systemd
# /run/systemd/transient/gamba1_slurmstepd.scope
# This is a transient unit file, created programmatically via the systemd API. Do not edit.
[Scope]
Delegate=yes
TasksMax=infinity

```

--------------------------------

### Example TRES Specification with Nodes

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/scontrol.html

This example demonstrates a TRES specification that includes node counts and licenses.

```bash
TRES=node=5k,license/iop1=2
```

--------------------------------

### Example TRES Specification

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/scontrol.html

This example shows a valid specification for the TRES parameter, including CPU, BurstBuffer, License, and GPU resources.

```bash
TRES=cpu=5,bb/cray=4,license/iop1=1,license/iop2=3
```

--------------------------------

### Example .rpmmacros File for Slurm

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/quickstart_admin.html

Configure RPM build options for Slurm by defining macros in a `.rpmmacros` file. This example sets debugging, installation prefix, system configuration directory, documentation directory, and MUNGE installation path.

```bash
# .rpmmacros
# Override some RPM macros from /usr/lib/rpm/macros
# Set Slurm-specific macros for unconventional file locations
#
%_enable_debug     "--with-debug"
%_prefix           /opt/slurm
%_slurm_sysconfdir %{_prefix}/etc/slurm
%_defaultdocdir    %{_prefix}/doc
%with_munge        "--with-munge=/opt/munge"

```

--------------------------------

### Example JSON Response for Get Cluster Info

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/rest_api.html

An example of the JSON response structure when retrieving cluster information. This includes metadata, warnings, cluster details, and errors.

```json
{
  "meta" : {
    "slurm" : {
      "cluster" : "cluster",
      "release" : "release",
      "version" : {
        "major" : "major",
        "minor" : "minor",
        "micro" : "micro"
      }
    },
    "plugin" : {
      "accounting_storage" : "accounting_storage",
      "name" : "name",
      "type" : "type",
      "data_parser" : "data_parser"
    },
    "client" : {
      "source" : "source",
      "user" : "user",
      "group" : "group"
    },
    "command" : [ "command", "command" ]
  },
  "warnings" : [ {
    "description" : "description",
    "source" : "source"
  }, {
    "description" : "description",
    "source" : "source"
  } ],
  "clusters" : [ {
    "associations" : {
      "root" : {
        "cluster" : "cluster",
        "partition" : "partition",
        "id" : 5,
        "user" : "user",
        "account" : "account"
      }
    },
    "controller" : {
      "port" : 0,
      "host" : "host"
    },
    "nodes" : "nodes",
    "flags" : [ "DELETED", "DELETED" ],
    "name" : "name",
    "rpc_version" : 6,
    "tres" : [ {
      "name" : "name",
      "count" : 0,
      "id" : 7,
      "type" : "type"
    }, {
      "name" : "name",
      "count" : 0,
      "id" : 7,
      "type" : "type"
    } ],
    "select_plugin" : "select_plugin"
  }, {
    "associations" : {
      "root" : {
        "cluster" : "cluster",
        "partition" : "partition",
        "id" : 5,
        "user" : "user",
        "account" : "account"
      }
    },
    "controller" : {
      "port" : 0,
      "host" : "host"
    },
    "nodes" : "nodes",
    "flags" : [ "DELETED", "DELETED" ],
    "name" : "name",
    "rpc_version" : 6,
    "tres" : [ {
      "name" : "name",
      "count" : 0,
      "id" : 7,
      "type" : "type"
    }, {
      "name" : "name",
      "count" : 0,
      "id" : 7,
      "type" : "type"
    } ],
    "select_plugin" : "select_plugin"
  } ],
  "errors" : [ {
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  }, {
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  } ]
}
```

--------------------------------

### Equivalent Command-Line Submission for Script Example

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/heterogeneous_jobs.html

This demonstrates the command-line equivalent of the '#SBATCH hetjob' script, showing how to specify different resource configurations for job components.

```bash
$ cat my.bash
#!/bin/bash
srun run.app

$ sbatch --cpus-per-task=4 --mem-per-cpu=16g --ntasks=1 : \
         --cpus-per-task=2 --mem-per-cpu=1g  --ntasks=8 my.bash

```

--------------------------------

### runc oci.conf Example (Recommended 'run')

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/containers.html

Recommended configuration for the 'runc' OCI runtime using the 'run' operation. This is more efficient as it avoids Slurm polling for job completion.

```bash
EnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeEnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeQuery="runc --rootless=true --root=/run/user/%U/ state %n.%u.%j.%s.%t"
RunTimeKill="runc --rootless=true --root=/run/user/%U/ kill -a %n.%u.%j.%s.%t"
RunTimeDelete="runc --rootless=true --root=/run/user/%U/ delete --force %n.%u.%j.%s.%t"
RunTimeRun="runc --rootless=true --root=/run/user/%U/ run %n.%u.%j.%s.%t -b %b"

```

--------------------------------

### API Endpoint for Slurm Daemon Configuration

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/rest_api.html

This is an example of a GET request to the /slurmdb/v0.0.43/config endpoint.

```http
get /slurmdb/v0.0.43/config
```

--------------------------------

### Basic Hourly Job with Scrontab

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/scrontab.html

This example demonstrates how to schedule a job to run at the beginning of each hour. It specifies the partition, account, and walltime for the job.

```bash
DIR=/home/user1
#SCRON -p high
#SCRON -A sub1
#SCRON -t 1:00
@hourly $DIR/date.printer.job

```

--------------------------------

### SLURM Dependency with Time Delay

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/sbatch.html

This example demonstrates specifying a dependency that allows the job to start only after a certain time has elapsed since the preceding job started or was cancelled. The time is specified in minutes.

```bash
after:job_id[+time][:jobid[+time]...]
```

--------------------------------

### Build OpenMPI v5 + PMIx v4 Container with Dockerfile

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/containers.html

This Dockerfile sets up a minimal environment with OpenMPI and PMIx. It installs necessary build tools, compiles and installs PMIx, then compiles and installs OpenMPI with PMIx support, and finally copies example MPI executables to the bin directory.

```docker
FROM almalinux:latest
RUN dnf -y update && dnf -y upgrade && dnf install -y epel-release && dnf -y update
RUN dnf -y install make automake gcc gcc-c++ kernel-devel bzip2 python3 wget libevent-devel hwloc-devel

WORKDIR /usr/local/src/
RUN wget --quiet 'https://github.com/openpmix/openpmix/releases/download/v5.0.7/pmix-5.0.7.tar.bz2' -O - | tar --no-same-owner -xvjf -
WORKDIR /usr/local/src/pmix-5.0.7/
RUN ./configure && make -j && make install

WORKDIR /usr/local/src/
RUN wget --quiet --inet4-only 'https://download.open-mpi.org/release/open-mpi/v5.0/openmpi-5.0.7.tar.bz2' -O - | tar --no-same-owner -xvjf -
WORKDIR /usr/local/src/openmpi-5.0.7/
RUN ./configure --disable-pty-support --enable-ipv6 --without-slurm --with-pmix --enable-debug && make -j && make install

WORKDIR /usr/local/src/openmpi-5.0.7/examples
RUN make && cp -v hello_c ring_c connectivity_c spc_example /usr/local/bin

```

--------------------------------

### crun oci.conf Example (Recommended 'run')

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/containers.html

Recommended configuration for the 'crun' OCI runtime using the 'run' operation. This is more efficient as it avoids Slurm polling for job completion.

```bash
EnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeEnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeQuery="crun --rootless=true --root=/run/user/%U/ state %n.%u.%j.%s.%t"
RunTimeKill="crun --rootless=true --root=/run/user/%U/ kill -a %n.%u.%j.%s.%t"
RunTimeDelete="crun --rootless=true --root=/run/user/%U/ delete --force %n.%u.%j.%s.%t"
RunTimeRun="crun --rootless=true --root=/run/user/%U/ run --bundle %b %n.%u.%j.%s.%t"

```

--------------------------------

### Example Slurm ResumeProgram Script

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/power_save.html

This script demonstrates how to implement a ResumeProgram for Slurm. It takes a hostlist as input, converts it to individual hostnames, and then executes a node startup command for each host. Logging is included for tracking script execution.

```bash
#!/bin/bash
# Example ResumeProgram
hosts=$(scontrol show hostnames "$1")
logfile=/var/log/power_save.log
echo "$(date) Resume invoked $0 $*" >>$logfile
for host in $hosts
do
        sudo node_startup "$host"
done
exit 0

```

--------------------------------

### SRUN Multi-Program Configuration File Example

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/srun.html

This is an example configuration file for srun's multi-program mode. It defines which program to run for specific task ranks and how to pass arguments.

```bash
###################################################################
# srun multiple program configuration file
#
# srun -n8 -l --multi-prog silly.conf
###################################################################
4-6       hostname
1,7       echo  task:%t
0,2-3     echo  offset:%o

```

--------------------------------

### Show All Configured Resources

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/licenses.html

Displays a summary of all configured resources, including their total counts and server types. Useful for a quick overview of license inventory.

```bash
$ sacctmgr show resource
      Name     Server     Type  Count LastConsumed Allocated ServerType                Flags
---------- ---------- -------- ------ ------------ --------- ---------- --------------------
   nastran  flex_host  License    100            0       100     flexlm
    matlab   rlm_host  License     50            0       100        rlm

```

--------------------------------

### Requesting Heterogeneous Job Steps

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/heterogeneous_jobs.html

Example of allocating nodes and then running heterogeneous job steps with different GPU and CPU requirements on each step. The steps must run on unique nodes.

```bash
$ salloc -N2 --exclusive --gpus=10
salloc: Granted job allocation 61034
$ srun -N1 -n4 --gpus=4 printenv SLURMD_NODENAME : -N1 -n1 --gpus=6 printenv SLURMD_NODENAME
node02
node01
node01
node01
node01

```

--------------------------------

### Example JSON for Reservations and Metadata

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/rest_api.html

This JSON object represents example data for Slurm reservations, including details like start and end times, flags, and associated resources. It also includes metadata about the Slurm cluster, plugins, client, and command used.

```json
{
  "reservations" : [ { 
    "end_time" : {
      "number" : 2,
      "set" : true,
      "infinite" : true
    },
    "flags" : [ "MAINT", "MAINT" ],
    "groups" : "groups",
    "users" : "users",
    "max_start_delay" : 6,
    "features" : "features",
    "start_time" : {
      "number" : 2,
      "set" : true,
      "infinite" : true
    },
    "burst_buffer" : "burst_buffer",
    "licenses" : "licenses",
    "partition" : "partition",
    "watts" : {
      "number" : 5,
      "set" : true,
      "infinite" : true
    },
    "core_specializations" : [ { 
      "node" : "node",
      "core" : "core"
    }, { 
      "node" : "node",
      "core" : "core"
    } ],
    "name" : "name",
    "tres" : "tres",
    "accounts" : "accounts",
    "node_count" : 1,
    "node_list" : "node_list",
    "purge_completed" : {
      "time" : {
        "number" : 5,
        "set" : true,
        "infinite" : true
      }
    },
    "core_count" : 0
  }, { 
    "end_time" : {
      "number" : 2,
      "set" : true,
      "infinite" : true
    },
    "flags" : [ "MAINT", "MAINT" ],
    "groups" : "groups",
    "users" : "users",
    "max_start_delay" : 6,
    "features" : "features",
    "start_time" : {
      "number" : 2,
      "set" : true,
      "infinite" : true
    },
    "burst_buffer" : "burst_buffer",
    "licenses" : "licenses",
    "partition" : "partition",
    "watts" : {
      "number" : 5,
      "set" : true,
      "infinite" : true
    },
    "core_specializations" : [ { 
      "node" : "node",
      "core" : "core"
    }, { 
      "node" : "node",
      "core" : "core"
    } ],
    "name" : "name",
    "tres" : "tres",
    "accounts" : "accounts",
    "node_count" : 1,
    "node_list" : "node_list",
    "purge_completed" : {
      "time" : {
        "number" : 5,
        "set" : true,
        "infinite" : true
      }
    },
    "core_count" : 0
  } ],
  "meta" : {
    "slurm" : {
      "cluster" : "cluster",
      "release" : "release",
      "version" : {
        "major" : "major",
        "minor" : "minor",
        "micro" : "micro"
      }
    },
    "plugin" : {
      "accounting_storage" : "accounting_storage",
      "name" : "name",
      "type" : "type",
      "data_parser" : "data_parser"
    },
    "client" : {
      "source" : "source",
      "user" : "user",
      "group" : "group"
    },
    "command" : [ "command", "command" ]
  },
  "last_update" : {
    "number" : 2,
    "set" : true,
    "infinite" : true
  },
  "warnings" : [ { 
    "description" : "description",
    "source" : "source"
  }, { 
    "description" : "description",
    "source" : "source"
  } ],
  "errors" : [ { 
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  }, { 
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  } ]
}
```

--------------------------------

### nvidia-container-runtime oci.conf (run - Recommended)

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/containers.html

Recommended oci.conf for nvidia-container-runtime using the 'run' command. This simplifies container execution compared to create/start.

```bash
EnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeEnvExclude="^(SLURM_CONF|SLURM_CONF_SERVER)="
RunTimeQuery="nvidia-container-runtime --rootless=true --root=/run/user/%U/ state %n.%u.%j.%s.%t"
RunTimeKill="nvidia-container-runtime --rootless=true --root=/run/user/%U/ kill -a %n.%u.%j.%s.%t"
RunTimeDelete="nvidia-container-runtime --rootless=true --root=/run/user/%U/ delete --force %n.%u.%j.%s.%t"
RunTimeRun="nvidia-container-runtime --rootless=true --root=/run/user/%U/ run %n.%u.%j.%s.%t -b %b"
```

--------------------------------

### Allocate memory per CPU with threads-per-core

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/srun.html

This example shows how --threads-per-core=1 affects memory allocation per CPU. Each task gets a full core, but memory is calculated based on usable threads. The sacct output reflects the difference in allocated CPUs and memory compared to the previous example.

```bash
$ salloc -n3 --mem-per-cpu=100 --threads-per-core=1
salloc: Granted job allocation 17200
$ sacct -j $SLURM_JOB_ID -X -o jobid%7,reqtres%35,alloctres%35
  JobID                             ReqTRES                           AllocTRES
-------
  17200     billing=3,cpu=3,mem=300M,node=1     billing=6,cpu=6,mem=300M,node=1
```

--------------------------------

### Sample knl_generic.conf Configuration

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/intel_knl.html

This snippet shows a sample configuration for the knl_generic plugin, specifying paths for syscfg and default/allowed NUMA and MCDRAM modes.

```bash
# Sample knl_generic.conf
SyscfgPath=/usr/bin/syscfg
DefaultNUMA=a2a         # NUMA=all2all
AllowNUMA=a2a,snc2,hemi
DefaultMCDRAM=cache     # MCDRAM=cache

```

--------------------------------

### Query Node Status

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/slurmrestd.html

Queries the status of a specific node (e.g., 'host1') when slurmrestd is running in INETD mode. This example demonstrates a GET request to the node endpoint.

```APIDOC
## Query Node Status

Query the status of a node with **slurmrestd** running in INETD mode:  

```
$ echo -e "GET http://ignored/slurm/v0.0.42/node/host1 HTTP/1.1\r\n" | slurmrestd
HTTP/1.1 200 OK
Content-Length: 3174
Content-Type: application/json

{
  "nodes": [
    {
      "architecture": "x86_64",
      "burstbuffer_network_address": "",
      "boards": 1,
      "boot_time": {
        "set": true,
        "infinite": false,
        "number": 1720820315
      },
      "cluster_name": "",
      "cores": 16,
      "specialized_cores": 0,
      "cpu_binding": 0,
      "cpu_load": 446,
      "free_mem": {
        "set": true,
        "infinite": false,
        "number": 39871
      },
      "cpus": 32,
      "effective_cpus": 32,
      "specialized_cpus": "",
      "energy": {
        "average_watts": 0,
        "base_consumed_energy": 0,
        "consumed_energy": 0,
        "current_watts": {
          "set": false,
          "infinite": false,
          "number": 0
        },
        "previous_consumed_energy": 0,
        "last_collected": 0
      },
      "external_sensors": {},
      "extra": "",
      "power": {},
      "features": [],
      "active_features": [],
      "gpu_spec": "",
      "gres": "gpu:fake1:1(S:0),gpu:fake2:1(S:0)",
      "gres_drained": "N/A",
      "gres_used": "gpu:fake1:0(IDX:N/A),gpu:fake2:0(IDX:N/A)",
      "instance_id": "",
      "instance_type": "",
      "last_busy": {
        "set": true,
        "infinite": false,
        "number": 1722009794
      },
      "mcs_label": "",
      "specialized_memory": 0,
      "name": "host1",
      "next_state_after_reboot": [
        "INVALID"
      ],
      "address": "localhost",
      "hostname": "omicronpersei8",
      "state": [
        "IDLE"
      ],
      "operating_system": "Linux 6.5.0-44-generic #44-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun  7 15:10:09 UTC 2024",
      "owner": "",
      "partitions": [
        "debug"
      ],
      "port": 5015,
      "real_memory": 127927,
      "res_cores_per_gpu": 0,
      "comment": "",
      "reason": "",
      "reason_changed_at": {
        "set": true,
        "infinite": false,
        "number": 0
      },
      "reason_set_by_user": "",
      "resume_after": {
        "set": true,
        "infinite": false,
        "number": 0
      },
      "reservation": "",
      "alloc_memory": 0,
      "alloc_cpus": 0,
      "alloc_idle_cpus": 32,
      "tres_used": "",
      "tres_weighted": 0.0,
      "slurmd_start_time": {
        "set": true,
        "infinite": false,
        "number": 1722009794
      },
      "sockets": 1,
      "threads": 2,
      "temporary_disk": 0,
      "weight": 1,
      "tres": "cpu=32,mem=127927M,billing=32,gres/gpu=2",
      "version": "24.11.0-0rc1"
    }
  ],
  "last_update": {
    "set": true,
    "infinite": false,
    "number": 1722010273
  },
  "meta": {},
  "errors": [],
  "warnings": []
}
```
```

--------------------------------

### Get Slurm Daemon Configuration

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/rest_api.html

Retrieves the Slurm daemon configuration, including association details and job limits. This endpoint is useful for understanding the current Slurm setup.

```json
{
  "associations" : [ {
    "lineage" : "lineage",
    "cluster" : "cluster",
    "shares_raw" : 7,
    "max" : {
      "jobs" : {
        "total" : {
          "number" : 5,
          "set" : true,
          "infinite" : true
        },
        "active" : {
          "number" : 5,
          "set" : true,
          "infinite" : true
        },
        "accruing" : {
          "number" : 5,
          "set" : true,
          "infinite" : true
        },
        "per" : {
          "submitted" : {
            "number" : 5,
            "set" : true,
            "infinite" : true
          },
          "count" : {
            "number" : 5,
            "set" : true,
            "infinite" : true
          },
          "accruing" : {
            "number" : 5,
            "set" : true,
            "infinite" : true
          },
          "wall_clock" : {
            "number" : 5,
            "set" : true,
            "infinite" : true
          }
        }
      },
      "tres" : {
        "total" : [ {
          "name" : "name",
          "count" : 0,
          "id" : 7,
          "type" : "type"
        }, {
          "name" : "name",
          "count" : 0,
          "id" : 7,
          "type" : "type"
        } ],
        "minutes" : {
          "total" : [ {
            "name" : "name",
            "count" : 0,
            "id" : 7,
            "type" : "type"
          }, {
            "name" : "name",
            "count" : 0,
            "id" : 7,
            "type" : "type"
          } ],
          "per" : {
            "job" : [ {
              "name" : "name",
              "count" : 0,
              "id" : 7,
              "type" : "type"
            }, {
              "name" : "name",
              "count" : 0,
              "id" : 7,
              "type" : "type"
            } ]
          }
        },
        "per" : {
          "node" : [ {
            "name" : "name",
            "count" : 0,
            "id" : 7,
            "type" : "type"
          }, {
            "name" : "name",
            "count" : 0,
            "id" : 7,
            "type" : "type"
          } ],
          "job" : [ {
            "name" : "name",
            "count" : 0,
            "id" : 7,
            "type" : "type"
          }, {
            "name" : "name",
            "count" : 0,
            "id" : 7,
            "type" : "type"
          } ]
        }
      }
    }
  } ]
}
```

--------------------------------

### Example JSON Response for Get Instance Info

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/rest_api.html

This JSON structure represents the data returned when querying for instance information. It includes details about instances, metadata, warnings, and errors.

```json
{
  "instances" : [ {
    "cluster" : "cluster",
    "instance_id" : "instance_id",
    "extra" : "extra",
    "node_name" : "node_name",
    "time" : {
      "time_start" : 1,
      "time_end" : 1
    },
    "instance_type" : "instance_type"
  }, {
    "cluster" : "cluster",
    "instance_id" : "instance_id",
    "extra" : "extra",
    "node_name" : "node_name",
    "time" : {
      "time_start" : 1,
      "time_end" : 1
    },
    "instance_type" : "instance_type"
  } ],
  "meta" : {
    "slurm" : {
      "cluster" : "cluster",
      "release" : "release",
      "version" : {
        "major" : "major",
        "minor" : "minor",
        "micro" : "micro"
      }
    },
    "plugin" : {
      "accounting_storage" : "accounting_storage",
      "name" : "name",
      "type" : "type",
      "data_parser" : "data_parser"
    },
    "client" : {
      "source" : "source",
      "user" : "user",
      "group" : "group"
    },
    "command" : [ "command", "command" ]
  },
  "warnings" : [ {
    "description" : "description",
    "source" : "source"
  }, {
    "description" : "description",
    "source" : "source"
  } ],
  "errors" : [ {
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  }, {
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  } ]
}
```

--------------------------------

### Launch applications with specific resource requirements

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/srun.html

Shows how to launch two applications, 'server' and 'client', with distinct task counts, CPU, and memory requirements per task.

```bash
$ srun -n1 -c8 --mem-per-cpu=2gb server : -n16 --mem-per-cpu=1gb client

```

--------------------------------

### MySQL/MariaDB Database Configuration Example

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/accounting.html

This example shows how to display the current 'innodb_buffer_pool_size' variable and provides a sample configuration file snippet for setting key MySQL/MariaDB parameters relevant to Slurm's accounting.

```sql
mysql> SHOW VARIABLES LIKE 'innodb_buffer_pool_size';
+-------------------------+------------+
| Variable_name           | Value      |
+-------------------------+------------+
| innodb_buffer_pool_size | 4294967296 |
+-------------------------+------------+
1 row in set (0.001 sec)

```

```ini
$cat my.cnf
...
[mysqld]
_innodb_buffer_pool_size=4096M_
_innodb_log_file_size=1024M_
_innodb_lock_wait_timeout=900_
_max_allowed_packet=16M_
...

```

--------------------------------

### Example JSON Response for Get Account

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/rest_api.html

Illustrates the structure of a successful JSON response when retrieving account information, including meta-data, warnings, account details, and potential errors.

```json
{
  "meta" : {
    "slurm" : {
      "cluster" : "cluster",
      "release" : "release",
      "version" : {
        "major" : "major",
        "minor" : "minor",
        "micro" : "micro"
      }
    },
    "plugin" : {
      "accounting_storage" : "accounting_storage",
      "name" : "name",
      "type" : "type",
      "data_parser" : "data_parser"
    },
    "client" : {
      "source" : "source",
      "user" : "user",
      "group" : "group"
    },
    "command" : [ "command", "command" ]
  },
  "warnings" : [ {
    "description" : "description",
    "source" : "source"
  }, {
    "description" : "description",
    "source" : "source"
  } ],
  "accounts" : [ {
    "associations" : [ {
      "cluster" : "cluster",
      "partition" : "partition",
      "id" : 5,
      "user" : "user",
      "account" : "account"
    }, {
      "cluster" : "cluster",
      "partition" : "partition",
      "id" : 5,
      "user" : "user",
      "account" : "account"
    } ],
    "coordinators" : [ {
      "name" : "name",
      "direct" : true
    }, {
      "name" : "name",
      "direct" : true
    } ],
    "organization" : "organization",
    "name" : "name",
    "flags" : [ "DELETED", "DELETED" ],
    "description" : "description"
  }, {
    "associations" : [ {
      "cluster" : "cluster",
      "partition" : "partition",
      "id" : 5,
      "user" : "user",
      "account" : "account"
    }, {
      "cluster" : "cluster",
      "partition" : "partition",
      "id" : 5,
      "user" : "user",
      "account" : "account"
    } ],
    "coordinators" : [ {
      "name" : "name",
      "direct" : true
    }, {
      "name" : "name",
      "direct" : true
    } ],
    "organization" : "organization",
    "name" : "name",
    "flags" : [ "DELETED", "DELETED" ],
    "description" : "description"
  } ],
  "errors" : [ {
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  }, {
    "description" : "description",
    "source" : "source",
    "error" : "error",
    "error_number" : 7
  } ]
}
```

--------------------------------

### Manually Starting Multiple slurmd Daemons

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/programmer_guide.html

Demonstrates how to manually start individual slurmd daemons on a host, specifying the unique NodeName for each. This is useful for testing and direct control.

```bash
host1> slurmd -N foo1
host1> slurmd -N foo11
host1> slurmd -N foo21


```

--------------------------------

### Get SlurmDBD Diagnostics

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/rest_api.html

Fetches diagnostic information for the SlurmDBD, including meta-information about the Slurm installation, plugins, client, and command. It also provides details on warnings, errors, and performance statistics.

```APIDOC
## GET /slurmdb/v0.0.43/diag/

### Description
Get slurmdb diagnostics (slurmdbV0043GetDiag)

### Method
GET

### Endpoint
/slurmdb/v0.0.43/diag/

### Return type
v0.0.43_openapi_slurmdbd_stats_resp

### Produces
This API call produces the following media types according to the 
  * `application/json`

### Responses
#### 200
v0.0.43_openapi_slurmdbd_stats_resp
#### default
v0.0.43_openapi_slurmdbd_stats_resp
```

--------------------------------

### Systemd Reconfiguration Check (cgroup v1)

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/cgroup_v2.html

Check if systemd needs reconfiguration for cgroup v2. This example shows the output indicating multiple lines or a non-zero starting value in /proc/1/cgroup.

```text
12:cpuset:/
11:hugetlb:/
10:perf_event:/
. . .

```

--------------------------------

### Get Node Token Script Example

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/certmgr.html

Prints the node's security token to standard output. This script must be executable on slurmd nodes and return a zero exit code on success.

```bash
#!/bin/bash

# Slurm node name is passed in as arg $1
TOKEN_PATH=/etc/slurm/certmgr/$1_token.txt

# Check if token file exists
if [ ! -f $TOKEN_PATH ]
then
    echo "$BASH_SOURCE: Failed to resolve token path '$TOKEN_PATH'"
    exit 1
fi

# Print token to stdout
cat $TOKEN_PATH

# Exit with exit code 0 to indicate success
exit 0

```

--------------------------------

### Example SPANK Plugin Configuration

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/spank.html

Illustrates how to load a SPANK plugin named 'test.so' without any arguments. If a fully-qualified path is not provided, the system searches the configured PluginDir.

```plaintext
  optional /usr/lib/slurm/test.so

```

--------------------------------

### Example librdkafka Configuration File

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/jobcomp_kafka.html

An example file for configuring librdkafka properties, including bootstrap servers, debug settings, and linger time.

```properties
bootstrap.servers=kafkahost1:9092
debug=broker,topic,msg
linger.ms=400
log_level=7
```

--------------------------------

### Create Daily Node Count Reservation for Specific Users

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/reservations.html

Creates a daily reservation for a specified number of nodes, available only to listed users. This example reserves 10 nodes daily starting at noon.

```bash
$ scontrol create reservation user=alan,brenda \
   starttime=noon duration=60 flags=daily nodecnt=10
Reservation created: alan_6
```

```bash
$ scontrol show res
ReservationName=alan_6 StartTime=2009-02-05T12:00:00
   EndTime=2009-02-05T13:00:00 Duration=60
   Nodes=sun[000-003,007,010-013,017] NodeCnt=10
   Features=(null) PartitionName=pdebug
   Flags=DAILY Licenses=(null) BurstBuffers=(null)
   Users=alan,brenda Accounts=(null)
```

--------------------------------

### Configure Two-Level Network with Four Connections per Switch

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/topology.html

Sets up a two-level network where each switch has four connections, including LinkSpeed for performance indication. This example is for a sixteen-node cluster and demonstrates a more complex hierarchical setup.

```bash
# topology.conf
# Switch Configuration
SwitchName=s0 Nodes=tux[0-3]   LinkSpeed=900
SwitchName=s1 Nodes=tux[4-7]   LinkSpeed=900
SwitchName=s2 Nodes=tux[8-11]  LinkSpeed=900
SwitchName=s3 Nodes=tux[12-15] LinkSpeed=1800
SwitchName=s4 Switches=s[0-3]  LinkSpeed=1800
SwitchName=s5 Switches=s[0-3]  LinkSpeed=1800
SwitchName=s6 Switches=s[0-3]  LinkSpeed=1800
SwitchName=s7 Switches=s[0-3]  LinkSpeed=1800

```

--------------------------------

### Multiple Key Setup Example (JWKS-esque)

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/authentication.html

This JSON structure defines multiple cryptographic keys for Slurm authentication. It includes fields like algorithm, key type, identifier, the key itself (Base64 encoded), and optional use or expiration.

```json
{
  "keys": [
    {
      "alg": "HS256",
      "kty": "oct",
      "kid": "key-identifier",
      "k": "VGhlIGtleSBiZWxvdyBtZSBuZXZlciBsaWVz",
      "exp": 1718200800
    },
    {
      "alg": "HS256",
      "kty": "oct",
      "kid": "key-identifier-2",
      "k": "VGhlIGtleSBhYm92ZSBtZSBhbHdheXMgbGllcw==",
      "use": "default"
    }
  ]
}

```

--------------------------------

### Allocate Heterogeneous Resources and Run Single MPI Communicator

Source: https://slurm.schedmd.com/archive/slurm-25.05-latest/heterogeneous_jobs.html

This example allocates heterogeneous resources and runs applications within a single MPI_COMM_WORLD. It's useful when applications need to communicate globally.

```bash
salloc -N1 --mem=256GB -C haswell : \
       -n2176 -N32 --ntasks-per-core=1 -C knl bash
srun server : client

```
Tokens:
Raw