### Start Command Help

Source: https://github.com/azure/cyclecloud-slurm/blob/master/integration/README.md

Show help for the `start` command, including optional arguments like `--skip-tests`.

```bash
$ python3 src/integration.py start --help
```

```bash
$ python3 src/integration.py start
```

--------------------------------

### Start All Integration Tests

Source: https://github.com/azure/cyclecloud-slurm/blob/master/integration/README.md

Start all integration tests using the `start` command in the integration script. This command initiates all defined tests.

```bash
$ python3 src/integration.py start
```

--------------------------------

### Example File Naming for Execution Order

Source: https://github.com/azure/cyclecloud-slurm/blob/master/specs/default/cluster-init/scripts/README.txt

Use leading digits in filenames to control execution order on the host. Ensure files are named to start with digits for predictable execution.

```shell
- 000_run_me_first.sh
- 001_run_me_second.sh
```

--------------------------------

### Azslurm CLI Usage Example

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

The `azslurm` CLI tool, available as root, provides various commands for managing Slurm clusters. This example shows the basic usage and available commands.

```bash
[root@scheduler ~]# azslurm
usage:
accounting_info      -
buckets              - Prints out autoscale bucket information, like limits etc
config               - Writes the effective autoscale config, after any preprocessing, to stdout
connect              - Tests connection to CycleCloud
cost                 - Cost analysis and reporting tool that maps Azure costs to SLURM Job Accounting data. This is an experimental feature.
default_output_columns - Output what are the default output columns for an optional command.
initconfig           - Creates an initial autoscale config. Writes to stdout
keep_alive           - Add, remeove or set which nodes should be prevented from being shutdown.
limits               -
nodes                - Query nodes
partitions           - Generates partition configuration
refresh_autocomplete - Refreshes local autocomplete information for cluster specific resources and nodes.
remove_nodes         - Removes the node from the scheduler without terminating the actual instance.
resume               - Equivalent to ResumeProgram, starts and waits for a set of nodes.
resume_fail          - Equivalent to SuspendFailProgram, shutsdown nodes
retry_failed_nodes   - Retries all nodes in a failed state.
scale                -
shell                - Interactive python shell with relevant objects in local scope. Use --script to run python scripts
suspend              - Equivalent to SuspendProgram, shutsdown nodes
topology             - Generates topology plugin configuration
wait_for_resume      - Wait for a set of nodes to converge.

```

--------------------------------

### Manually Start Nodes

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Use the provided script to start new nodes when autoscale is disabled (SuspendTime=-1). Nodes are denoted by the FUTURE state.

```bash
/opt/azurehpc/slurm/resume_program.sh node_list

```

--------------------------------

### Example Slurm Topology Output (Fabric Manager)

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

An example of the Slurm topology configuration output when using a Fabric Manager. This output details switches and the nodes associated with each.

```text
# Number of Nodes in sw00: 6

SwitchName=sw00 Nodes=ccw-gpu-7,ccw-gpu-99,ccw-gpu-151,ccw-gpu-140,ccw-gpu-167,ccw-gpu-194

# Number of Nodes in sw01: 12

SwitchName=sw01 Nodes=ccw-gpu-30,ccw-gpu-29,ccw-gpu-32,ccw-gpu-87,ccw-gpu-85,ccw-gpu-149,ccw-gpu-150,ccw-gpu-166,ccw-gpu-141,ccw-gpu-162,ccw-gpu-112,ccw-gpu-183

# Number of Nodes in sw02: 1

SwitchName=sw02 Nodes=ccw-gpu-192
```

--------------------------------

### Setup NFS Cluster using Python Script

Source: https://github.com/azure/cyclecloud-slurm/blob/master/integration/README.md

Automatically set up an NFS cluster using the provided Python integration script. Ensure `CS_HOME` is defined and `cyclecloud` is in your PATH.

```bash
python3 src/integration.py setup_nfs -p params.json
```

--------------------------------

### Start Cluster and Apply Scale Changes

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Start a Slurm cluster and apply scaling and configuration changes on the scheduler node. This command ensures the partition is configured correctly and Slurm services are restarted.

```bash
cyclecloud start_cluster MY_CLUSTERNAME
ssh $SCHEDULER_IP
# azslurm scale will configure the partition and restart slurmctld
# - this generally has no impact on the running workload
sudo azslurm scale
```

--------------------------------

### Example Slurm Tree Topology Output (VMSS)

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

An example of the Slurm topology configuration output generated for tree topology using VMSS. This format is used for the topology.conf file.

```text
SwitchName=htc Nodes=cluster-htc-1,cluster-htc-2,cluster-htc-3,cluster-htc-4,cluster-htc-5,cluster-htc-6,cluster-htc-7,cluster-htc-8,cluster-htc-9,cluster-htc-10,cluster-htc-11,cluster-htc-12,cluster-htc-13,cluster-htc-14,cluster-htc-15,cluster-htc-16,cluster-htc-17,cluster-htc-18,cluster-htc-19,cluster-htc-20,cluster-htc-21,cluster-htc-22,cluster-htc-23,cluster-htc-24,cluster-htc-25,cluster-htc-26,cluster-htc-27,cluster-htc-28,cluster-htc-29,cluster-htc-30,cluster-htc-31,cluster-htc-32,cluster-htc-33,cluster-htc-34,cluster-htc-35,cluster-htc-36,cluster-htc-37,cluster-htc-38,cluster-htc-39,cluster-htc-40,cluster-htc-41,cluster-htc-42,cluster-htc-43,cluster-htc-44,cluster-htc-45,cluster-htc-46,cluster-htc-47,cluster-htc-48,cluster-htc-49,cluster-htc-50
SwitchName=Standard_F2s_v2_pg0 Nodes=cluster-hpc-1,cluster-hpc-10,cluster-hpc-11,cluster-hpc-12,cluster-hpc-13,cluster-hpc-14,cluster-hpc-15,cluster-hpc-16,cluster-hpc-2,cluster-hpc-3,cluster-hpc-4,cluster-hpc-5,cluster-hpc-6,cluster-hpc-7,cluster-hpc-8,cluster-hpc-9
```

--------------------------------

### Start a Single Cluster

Source: https://github.com/azure/cyclecloud-slurm/blob/master/integration/README.md

Manually start a specific cluster using the `cyclecloud start_cluster` command. The `--test` flag is used for integration testing purposes.

```bash
cyclecloud start_cluster {cluster_name} --test
```

--------------------------------

### Import Clusters using Python Script

Source: https://github.com/azure/cyclecloud-slurm/blob/master/integration/README.md

Import clusters using the integration script. Pass the parameters file with `-p`. Use `-n` to specify the NFS instance IP address if using your own NFS setup.

```bash
$ python3 src/integration.py import -p param.json
```

--------------------------------

### Generate Monthly Cost Report

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Example command to generate a cost report for March 2023. This creates a directory named 'march-2023' containing CSV files for job and partition costs.

```bash
azslurm cost -s 2023-03-01 -e 2023-03-31 -o march-2023
```

--------------------------------

### Customizing Cost Report Fields

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Example command to customize the fields included in the jobs cost report CSV. This allows appending specific Slurm fields from 'sacct' data.

```bash
azslurm cost -s 2023-03-01 -e 2023-03-31 -o march-2023 -f account,cluster,jobid,jobname,reqtres,start,end,state,qos,priority,container,constraints,user
```

--------------------------------

### Update Slurm Download URL

Source: https://github.com/azure/cyclecloud-slurm/blob/master/DEVEL.md

Update the `CURRENT_DOWNLOAD_URL` variable in `slurm/install/slurm_supported_version.py` to point to the latest Slurm binary release. This ensures that new installations download the correct package.

```python
CURRENT_DOWNLOAD_URL
```

--------------------------------

### Disable Healthagent Entirely

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Set the 'cyclecloud.healthagent.disable' template parameter to true to disable healthagent setup entirely. This prevents healthagent from running.

```text
cyclecloud.healthagent.disable=true
```

--------------------------------

### Import Clusters Help

Source: https://github.com/azure/cyclecloud-slurm/blob/master/integration/README.md

Display help information for the `import` command, showing available arguments like `--properties` and `--nfs-address`.

```bash
$ python3 src/integration.py import --help
```

--------------------------------

### azslurm topology command usage

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Displays the available arguments and options for the `azslurm topology` command. Use this to understand how to configure topology generation.

```bash
usage: azslurm topology [-h] [--config CONFIG] [-p, PARTITION] [-o OUTPUT]
                        [-v | -f | -n] [-b | -t] [-s BLOCK_SIZE]

optional arguments:
  -h, --help            show this help message and exit
  --config CONFIG, -c CONFIG
  -p, PARTITION, --partition PARTITION
                        Specify the parititon
  -o OUTPUT, --output OUTPUT
                        Specify slurm topology file output
  -v, --use_vmss        Use VMSS to map Tree or Block topology along VMSS
                        boundaries without special network consideration
                        (default: True)
  -f, --use_fabric_manager
                        Use Fabric Manager to map Tree topology (Block
                        topology not allowed) according to SHARP network
                        topology tool(default: False)
  -n, --use_nvlink_domain
                        Use NVlink domain to map Block topology (Tree topology
                        not allowed) according to NVLink Domain and Partition
                        for multi-node NVLink (default: False)
  -b, --block           Generate Block Topology output to use Block topology
                        plugin (default: False)
  -t, --tree            Generate Tree Topology output to use Tree topology
                        plugin(default: False)
  -s BLOCK_SIZE, --block_size BLOCK_SIZE
                        Minimum block size required for each block (use with
                        --block or --use_nvlink_domain, default: 1)
```

--------------------------------

### Create Integration Parameters JSON

Source: https://github.com/azure/cyclecloud-slurm/blob/master/integration/README.md

Define common parameters for your clusters. Save this as `integration/params.json`. Replace placeholders like `{myregion}` with your specific values.

```json
{
  "UsePublicNetwork" : false,
  "Region" : "{myregion}",
  "NumberLoginNodes" : 1,
  "Credentials" : "{mycred, probably cloud}",
  "ExecuteNodesPublic" : false,
  "SubnetId" : "{my-persisten-rg}/{myvnet}/default",
  "ReturnProxy" : false
}
```

--------------------------------

### Enable Multiselect for Dynamic VM Types

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Configure the 'Config.Multiselect' parameter for a dynamic node type to true, allowing the dynamic partition to support multiple VM sizes by default.

```ini
        [[[parameter DynamicMachineType]]]
        Label = Dyn VM Type
        Description = The VM type for Dynamic nodes
        ParameterType = Cloud.MachineType
        DefaultValue = Standard_F2s_v2
        Config.Multiselect = true

```

--------------------------------

### Support Multiple VM Sizes in Dynamic Partitions

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Add VM size names as features to dynamic nodes to allow autoscaling to distinguish and use different VM sizes within the same partition. The VM size is added implicitly.

```bash
scontrol create nodename=f4-[1-10] Feature=mydyn,Standard_F4 State=CLOUD
scontrol create nodename=f8-[1-10] Feature=mydyn,Standard_F8 State=CLOUD

```

--------------------------------

### Create Dynamic GPU Nodes (with gres.conf)

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

When adding dynamic nodes with GRES like GPUs, ensure the /etc/slurm/gres.conf file is modified first. This prevents errors when creating nodes with GPU features.

```bash
root@s3072-scheduler:~# scontrol create NodeName=e1 CPUs=24 Gres=gpu:4 Feature=dyn,nv24 State=cloud
scontrol: error: Invalid argument (e1)
Error creating node(s): Invalid argument

```

--------------------------------

### Create Dynamic Nodes for Autoscaling

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Manually create node records with specific features for a dynamic partition to enable Slurm autoscaling. Nodes are created with State=CLOUD.

```bash
scontrol create nodename=f4-[1-10] Feature=mydyn,Standard_F2s_V2 cpus=2 State=CLOUD

```

--------------------------------

### Build Slurm RPMs and DEBs

Source: https://github.com/azure/cyclecloud-slurm/blob/master/DEVEL.md

Build the RPM and DEB packages for Slurm using the provided utility script. This process is essential for creating distributable Slurm packages.

```bash
# this should be all you need, but new versions may require
# updates. See the below for more information, as they are what is run inside the
# container.
# ./specs/default/cluster-init/files/01-build-debs.sh
# ./specs/default/cluster-init/files/00-build-slurm.sh
./util/docker-rpmbuild.sh
```

--------------------------------

### Capture Logs and Configuration Data

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Run this script on any Slurm node (scheduler, login, or execute) to collect logs and configuration data for troubleshooting. Ensure it's executed on all nodes requiring examination. The output is a tarball file.

```bash
/opt/cycle/capture_logs.sh
```

--------------------------------

### Clone CycleCloud Scalelib and Slurm Repositories

Source: https://github.com/azure/cyclecloud-slurm/blob/master/DEVEL.md

Clone the necessary CycleCloud Slurm and Scalelib repositories to your development environment. Ensure you are in the desired development directory before cloning.

```bash
CODE_DIR=~/code  # or where ever you wish to develop this
cd $CODE_DIR
git clone https://github.com/Azure/cyclecloud-scalelib.git
# cd cyclecloud-scalelib
# git checkout specific-branch

cd $CODE_DIR
git clone https://github.com/Azure/cyclecloud-slurm.git
cd cyclecloud-slurm

docker-package.sh ../cyclecloud-scalelib
```

--------------------------------

### Generate Slurm Tree Topology using Fabric Manager

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Generates a Slurm topology configuration using a Fabric Manager with SHARP enabled. A partition must be specified, and tree topology is the default.

```bash
azslurm topology -f -p gpu
azslurm topology -f -p gpu -t
azslurm topology -f -p gpu -o topology.conf
```

--------------------------------

### Enable Dynamic Slurm Partitions

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Configure a nodearray to map to a dynamic Slurm partition by setting slurm.autoscale to true and defining dynamic configuration parameters. The 'slurm.dynamic_config' parameter allows specifying features for the dynamic partition.

```ini
      [[[configuration]]]
      slurm.autoscale = true
      # Set to true if nodes are used for tightly-coupled multi-node jobs
      slurm.hpc = false
      # This is the minimum, but see slurmd --help and [slurm.conf](https://slurm.schedmd.com/slurm.conf.html) for more information.
      slurm.dynamic_config := "-Z --conf \"Feature=mydyn\""

```

--------------------------------

### Allowable File Extensions on Windows

Source: https://github.com/azure/cyclecloud-slurm/blob/master/specs/default/cluster-init/scripts/README.txt

On Windows, scripts can use .bat, .cmd, or .exe file extensions.

```batch
.bat, .cmd, .exe
```

--------------------------------

### Login Node Configuration

Source: https://github.com/azure/cyclecloud-slurm/blob/master/templates/slurm.txt

Sets up login nodes for the Slurm cluster, specifying initial and maximum counts, machine type, image, and cluster-init specs. Use this for interactive access and job submission.

```ini
[[nodearray login]]
        InitialCount = $NumberLoginNodes
        MaxCount = $MaxLoginNodeCount
        MachineType = $loginMachineType
        ImageName = $LoginImageName
        AdditionalClusterInitSpecs = $LoginClusterInitSpecs

        [[[cluster-init cyclecloud/healthagent:default]]]
        [[[cluster-init cyclecloud/monitoring:default]]]
        [[[cluster-init cyclecloud/slurm:login]]]
        [[[configuration]]]
        slurm.role = login
        autoscale.enabled = false
        slurm.node_prefix = ${ifThenElse(NodeNamePrefix=="Cluster Prefix", StrJoin("-", ClusterName, ""), NodeNamePrefix)}
        slurm.use_nodename_as_hostname = $NodeNameIsHostname
```

--------------------------------

### Cost Report Output Files

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Lists the CSV files generated by the 'azslurm cost' command for a given period. These files detail job costs, partition costs, and hourly partition costs.

```bash
[root@slurm301-2-scheduler ~]# ls march-2023/
jobs.csv  partition.csv  partition_hourly.csv
```

--------------------------------

### Check AzSlurm Exporter Metrics

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Run this curl command on the Slurm scheduler VM to verify if the AzSlurm Exporter is exposing metrics. It is available via localhost on port 9101.

```bash
curl -s http://localhost:9101/metrics
```

--------------------------------

### Base Execute Node Configuration

Source: https://github.com/azure/cyclecloud-slurm/blob/master/templates/slurm.txt

Defines the abstract base configuration for execute nodes, setting the Slurm role to 'execute' and enabling autoscaling. This serves as a template for specific execute node arrays.

```ini
[[node nodearraybase]]
    Abstract = true
        [[[configuration]]]
        slurm.role = execute
        slurm.autoscale = true

        slurm.node_prefix = ${ifThenElse(NodeNamePrefix=="Cluster Prefix", StrJoin("-", ClusterName, ""), NodeNamePrefix)}
        slurm.use_nodename_as_hostname = $NodeNameIsHostname

        [[[cluster-init cyclecloud/healthagent:default]]]
        [[[cluster-init cyclecloud/monitoring:default]]]
        [[[cluster-init cyclecloud/slurm:execute]]]

        [[[network-interface eth0]]]
        AssociatePublicIpAddress = $ExecuteNodesPublic
```

--------------------------------

### Customized Jobs CSV Columns

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Illustrates the column headers in the jobs CSV file after applying custom formatting options. Includes default cost-related fields and user-specified Slurm fields.

```text
account,cluster,jobid,jobname,reqtres,start,end,state,qos,priority,container,constraints,user,sku_name,region,spot,meter,meterid,metercat,rate,currency,cost
```

--------------------------------

### Generate Slurm Tree Topology using VMSS

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Generates a Slurm topology configuration using VMSS, defaulting to tree topology. The output can be printed to stdout or saved to a file.

```bash
azslurm topology
azslurm topology -v -t
azslurm topology -o topology.conf
```

--------------------------------

### Generate Slurm Topology

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Generates Slurm topology in slurm format or an output file. Specify a partition and optionally a minimum block size and the block option.

```bash
azslurm topology -n -p gpu
```

```bash
azslurm topology -n -p gpu -b -s 5
```

```bash
azslurm topology -n -p gpu -b -s 5 -o topology.conf
```

--------------------------------

### Define Custom Slurm Nodearray

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Extend an existing nodearray to create a new Slurm partition with custom machine types and core counts. Ensure new parameters are added to the [[parameters]] section.

```ini
   [[nodearray specialgpu]]
   Extends = gpu

   MachineType = $SpecialGPUMachineType

   MaxCoreCount = $MaxSpecialGPUCoreCount

... 

# Any new parameters [SpecialGPUMachineType, MaxSpecialGPUCoreCount] should be added n the [[parameters]] section

```

--------------------------------

### Define Supported Slurm Versions

Source: https://github.com/azure/cyclecloud-slurm/blob/master/DEVEL.md

Add new Slurm version records to the SUPPORTED_VERSIONS dictionary in `slurm/install/slurm_supported_version.py`. This dictionary maps Slurm versions to their supported platform and architecture combinations.

```python
SUPPORTED_VERSIONS = {
    "22.05.8": {
        "rhel": [{"platform_version": "el8", "arch": "x86_64"}],
        "debian": [{"arch": "amd64"}],
    },
    "23.02.0": {
        "rhel": [{"platform_version": "el8", "arch": "x86_64"}],
        "debian": [{"arch": "amd64"}],
    }
}
```

--------------------------------

### Configure IMEX Service

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Parameter to enable or disable the IMEX service on a per-job basis for GB200/GB300 nodes.

```ini
slurm.imex.enabled=True
```

```ini
slurm.imex.enabled=False
```

--------------------------------

### Manual NFS Export Configuration

Source: https://github.com/azure/cyclecloud-slurm/blob/master/integration/README.md

Manually configure NFS exports for `/sched` and `/shared` if not using the automated script. Requires `nfs-mountd` to be restarted.

```bash
echo '/mnt/exports/sched *(rw,sync,no_root_squash)' >> /etc/exports
echo '/mnt/exports/shared *(rw,sync,no_root_squash)' >> /etc/exports
systemctl restart nfs-mountd
```

--------------------------------

### Configure AzSlurm Exporter Port

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Set the AZSLURM_EXPORTER_PORT environment variable to configure the exporter's port. Restart the systemd service to apply changes.

```bash
export AZSLURM_EXPORTER_PORT=<PORT>
/opt/azurehpc/slurm/venv/bin/azslurm-exporter-install
systemctl restart azslurm-exporter
```

--------------------------------

### Rerun Docker Package Script

Source: https://github.com/azure/cyclecloud-slurm/blob/master/DEVEL.md

After updating supported versions and download URLs, rerun the `docker-package.sh` script. This will incorporate the new Slurm versions into the Docker image.

```bash
docker-package.sh
```

--------------------------------

### Add AzSlurm Exporter Dashboards to Azure Managed Grafana

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Clone the repository and run the add_dashboards.sh script to import dashboards into an Azure Managed Grafana instance. Replace placeholders with your resource group and Grafana instance names.

```bash
git clone https://github.com/Azure/cyclecloud-slurm.git
cd cyclecloud-slurm/azure-slurm-exporter
./add_dashboards.sh <resource group name> <grafana name>
```

--------------------------------

### Override Specific VM Size GPU Count

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

This JSON configures a specific override for GPU counts for a particular VM size ('Standard_XYZ') to 8 GPUs. It must be placed before the general default resource definition to take precedence.

```json
{
    "default_resources": [
      {
        "select": {"node.vm_size": "Standard_XYZ"},
        "name": "slurm_gpus",
        "value": 8
      },
      {
        "select": {},
        "name": "slurm_gpus",
        "value": "node.gpu_count"
      }
    ],
  }
```

--------------------------------

### Check DCGM Exporter Metrics

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Execute this curl command on nodes with NVIDIA GPUs to check if the DCGM Exporter is exposing metrics. Access it via localhost on port 9400.

```bash
curl -s http://localhost:9400/metrics
```

--------------------------------

### azslurm cost Command Usage

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

This snippet shows the command-line arguments for the 'azslurm cost' command, used for reporting Slurm job costs. It requires Cyclecloud 8.4+ and enabled Slurm accounting.

```bash
usage: azslurm cost [-h] [--config CONFIG] [-s START] [-e END] -o OUT [-f FMT]

optional arguments:
  -h, --help            show this help message and exit
  --config CONFIG, -c CONFIG
  -s START, --start START
                        Start time period (yyyy-mm-dd), defaults to current
                        day.
  -e END, --end END     End time period (yyyy-mm-dd), defaults to current day.
  -o OUT, --out OUT     Directory name for output CSV
  -f FMT, --fmt FMT     Comma separated list of SLURM formatting options.
                        Otherwise defaults are applied
```

--------------------------------

### Configure Automatic Zombie Node Termination in CycleCloud

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Add this JSON configuration to `/opt/azurehpc/slurm/autoscale.json` to automatically terminate 'zombie' nodes. Zombie nodes are those present in CycleCloud but in a powered-down state in Slurm.

```json
{
  "return-to-idle": {
    "terminate-zombie-nodes": true
  }
}
```

--------------------------------

### Allowable File Extensions on Linux

Source: https://github.com/azure/cyclecloud-slurm/blob/master/specs/default/cluster-init/scripts/README.txt

The only allowable file extension for scripts on Linux is .sh.

```shell
.sh
```

--------------------------------

### Scheduler Node Configuration

Source: https://github.com/azure/cyclecloud-slurm/blob/master/templates/slurm.txt

Defines the primary scheduler node, including HA settings, NFS mount configurations, and network interface details. Use this to set up the central scheduler for the Slurm cluster.

```ini
Zone = ${ifThenElse(configuration_slurm_ha_enabled, SchedulerZone, undefined)}
    
        [[[configuration]]]
        slurm.role = scheduler
        # Disable NFS mount of built-in /sched since it is a local volume mount: cyclecloud.mounts.builtinsched
        cyclecloud.mounts.nfs_sched.disabled = ${UseBuiltinSched && !configuration_slurm_ha_enabled}
        cyclecloud.mounts.nfs_shared.disabled = ${UseBuiltinShared && !configuration_slurm_ha_enabled}
        slurm.secondary_scheduler_name = ${ifThenElse(configuration_slurm_ha_enabled, "scheduler-ha-1", undefined)}


        [[[cluster-init cyclecloud/healthagent:default]]]
        [[[cluster-init cyclecloud/monitoring:default]]]
        [[[cluster-init cyclecloud/slurm:scheduler]]]

        [[[network-interface eth0]]]
        AssociatePublicIpAddress = $UsePublicNetwork

        [[[volume sched]]]
        Size = $SchedFilesystemSize
        SSD = True
        Mount = builtinsched
        Persistent = True
        Disabled = ${!UseBuiltinSched || configuration_slurm_ha_enabled}

        [[[volume shared]]]
        Size = $FilesystemSize
        SSD = True
        Mount = builtinshared
        Persistent = True
        Disabled = ${!UseBuiltinShared || configuration_slurm_ha_enabled}

        [[[configuration cyclecloud.mounts.builtinsched]]]
        disabled = ${!UseBuiltinSched || configuration_slurm_ha_enabled}
        mountpoint = /sched
        fs_type = xfs

        [[[configuration cyclecloud.mounts.builtinshared]]]
        disabled = ${!UseBuiltinShared || configuration_slurm_ha_enabled}
        mountpoint = /shared
        fs_type = xfs

        [[[configuration cyclecloud.exports.builtinsched]]]
	    disabled = ${!UseBuiltinSched || configuration_slurm_ha_enabled}
        export_path = /sched
        options = no_root_squash
        samba.enabled = false
        type = nfs

        [[[configuration cyclecloud.exports.builtinshared]]]
        disabled = ${!UseBuiltinShared || configuration_slurm_ha_enabled}
        export_path = /shared
        samba.enabled = false
        type = nfs
```

--------------------------------

### Override Slurm User UIDs and GIDs

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Configure these attributes in the 'Configuration' section of each nodearray to override default UIDs and GIDs for Slurm, Munge, and Slurmrestd users if conflicts arise. Ensure consistency across all nodearrays.

```text
munge.user.uid
munge.user.gid
slurm.slurmrestd.user.uid
slurm.slurmrestd.user.gid
slurm.user.uid
slurm.user.gid
```

--------------------------------

### Scale Slurm Cluster Configuration

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Apply cluster configuration changes and restart Slurm services on the scheduler node. This command rebuilds `azure.conf`, updates nodes, and restarts `slurmctld`.

```bash
sudo -i
# azslurm scale
```

--------------------------------

### Check Node Exporter Metrics

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Use this curl command to verify if the Prometheus Node Exporter is exposing metrics on all nodes. It is accessible via localhost on port 9100.

```bash
curl -s http://localhost:9100/metrics
```

--------------------------------

### Shutdown Clusters

Source: https://github.com/azure/cyclecloud-slurm/blob/master/integration/README.md

Shutdown clusters using the `shutdown` command. Use the `--include-nfs` flag to also shut down the `integration-nfs` cluster.

```bash
$ python3 src/integration.py shutdown [--include-nfs]
```

--------------------------------

### Import Modified Slurm Cluster Template

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Update a Slurm cluster configuration by exporting parameters, modifying the template, and then importing the changes. This is used for applying template changes to existing clusters.

```bash
# First update a copy of the slurm template (shown as ./MODIFIED_SLURM.txt below)

cyclecloud export_parameters MY_CLUSTERNAME > ./MY_CLUSTERNAME.json
cyclecloud import_cluster MY_CLUSTERNAME -c slurm -f ./MODIFIED_slurm.txt -p ./MY_CLUSTERNAME.json --force
```

--------------------------------

### High Availability Scheduler Node Configuration

Source: https://github.com/azure/cyclecloud-slurm/blob/master/templates/slurm.txt

Configures a secondary scheduler node for high availability. This node extends the base scheduler configuration and is enabled only when HA is configured.

```ini
[[nodearray scheduler-ha]]
    Extends = scheduler
    IsReturnProxy = false
    InitialCount = $configuration_slurm_ha_enabled
    Zone = $SchedulerHAZone
    # Do not inherit property from node that is not used in nodearray
    # The equivalent is ComputerNamePrefix for nodearray, however Cluster-init will handle renaming of all hosts in a VMSS
    ComputerName := undefined
        [[[configuration]]]
        autoscale.enabled = false
        slurm.node_prefix = ${ifThenElse(NodeNamePrefix=="Cluster Prefix", StrJoin("-", ClusterName, ""), NodeNamePrefix)}
        slurm.use_nodename_as_hostname = $NodeNameIsHostname
        slurm.is_primary_scheduler = false
```

--------------------------------

### Delete Clusters

Source: https://github.com/azure/cyclecloud-slurm/blob/master/integration/README.md

Delete clusters using the `delete` command. The `--include-nfs` flag is necessary to delete the `integration-nfs` cluster.

```bash
$ python3 src/integration.py delete [--include-nfs]
```

--------------------------------

### Default GPU Resource Definition

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

This JSON snippet defines the default resource for GPUs in CycleCloud's autoscaler configuration. It sets a resource named 'slurm_gpus' to the value of 'node.gpu_count' for all VM sizes.

```json
{
    "default_resources": [
      {
        "select": {},
        "name": "slurm_gpus",
        "value": "node.gpu_count"
      }
    ],
  }
```

--------------------------------

### Default Memory Resource Definition

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

This JSON snippet defines the default resource for memory in CycleCloud's autoscaler configuration. It sets a resource named 'slurm_memory' to the value of 'node.memory' for all VM sizes.

```json
    "default_resources": [
    {
      "select": {},
      "name": "slurm_memory",
      "value": "node.memory"
    }
  ],
  
```

--------------------------------

### HTC Execute Node Configuration

Source: https://github.com/azure/cyclecloud-slurm/blob/master/templates/slurm.txt

Sets up High-Throughput Computing (HTC) execute nodes, inheriting from the base execute node configuration. Defines machine type, image, max count, and availability zone.

```ini
[[nodearray htc]
    Extends = nodearraybase
    MachineType = $HTCMachineType
    ImageName = $HTCImageName
    MaxCount = $MaxHTCExecuteNodeCount
    Zone = ${ifThenElse(DefineNodesAvailabilityZone, HTCAvailabilityZone, undefined)}
```

--------------------------------

### HPC Execute Node Configuration

Source: https://github.com/azure/cyclecloud-slurm/blob/master/templates/slurm.txt

Configures High-Performance Computing (HPC) execute nodes, extending the base execute node configuration. Specifies machine type, image, max count, availability zone, and health checks.

```ini
[[nodearray hpc]
    Extends = nodearraybase
    MachineType = $HPCMachineType
    ImageName = $HPCImageName
    MaxCount = $MaxHPCExecuteNodeCount
    Zone = ${ifThenElse(DefineNodesAvailabilityZone, HPCAvailabilityZone, undefined)}
    Azure.MaxScalesetSize = $HPCMaxScalesetSize
    AdditionalClusterInitSpecs = $HPCClusterInitSpecs
    EnableNodeHealthChecks = $EnableNodeHealthChecks
    

        [[[configuration]]]
        slurm.default_partition = true
        slurm.hpc = true
        slurm.partition = hpc
```

--------------------------------

### Manually Shutdown Nodes

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Use the provided script to shut down nodes when autoscale is disabled (SuspendTime=-1). Nodes are denoted by the FUTURE state.

```bash
/opt/azurehpc/slurm/suspend_program.sh node_list

```

--------------------------------

### Disable Ubuntu 22+ WAagent DNS Hostname Resolution Fix

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

To disable the mitigation for DNS registration issues on Ubuntu 22+ VMs in a VMSS, set `slurm.ubuntu22_waagent_fix` to `false` in the `Configuration` section of your settings. This mitigation restarts `systemd-networkd` when the VMSS hostname changes.

```ini
      [[[configuration]]]
      slurm.ubuntu22_waagent_fix = false

```

--------------------------------

### Disable Slurm Healthchecks Configuration

Source: https://github.com/azure/cyclecloud-slurm/blob/master/README.md

Set 'slurm.enable_healthchecks' to false to disable Slurm-specific configuration for healthchecks, while still allowing healthagent to run for UI visibility.

```text
slurm.enable_healthchecks=false
```