### Install KEDA Helm chart

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/autoscaling.md

Adds the KEDA Helm repository and installs the KEDA chart in the keda namespace.

```sh
helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda \
  --namespace keda --create-namespace
```

--------------------------------

### Dockerfile: Install PyTorch and GCN example in base-extra layer

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/tutorial-pytorch.md

Modify the base-extra layer of the Slurm container image to install Python and PyTorch packages, clone the PyTorch examples repository, and install the GCN example's dependencies. This is required to run the GCN example on the Slurm cluster.

```dockerfile
################################################################################
FROM base AS base-extra

SHELL ["bash", "-c"]

RUN --mount=type=cache,target=/var/cache/dnf,sharing=locked <<EOR
# Install Extra Packages
set -xeuo pipefail
dnf -q -y install --setopt='install_weak_deps=False' \
  openmpi python3-pip python-setuptools python3-scipy
EOR

# pytorch
RUN git clone --depth=2 https://github.com/pytorch/examples.git
WORKDIR /tmp/examples/gcn
RUN pip install -r requirements.txt

################################################################################

```

--------------------------------

### Install Slurm with accounting enabled

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/installation.md

Installs the Slurm Helm chart with accounting enabled, pointing to the MariaDB database created earlier.

```sh
helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
  --set 'accounting.enabled=true' \
  --namespace=slurm --create-namespace
```

--------------------------------

### Slurm topology.yaml example

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/topology.md

Example topology.yaml file defining three topologies: topo-switch, topo-block, and topo-flat. The topo-switch topology is set as the cluster default.

```yaml
---
- topology: topo-switch
  cluster_default: true
  tree:
    switches:
      - switch: sw_root
        children: s[1-2]
      - switch: s1
        nodes: node[1-2]
      - switch: s2
        nodes: node[3-4]
- topology: topo-block
  cluster_default: false
  block:
    block_sizes:
      - 2
      - 4
    blocks:
      - block: b1
        nodes: node[1-2]
      - block: b2
        nodes: node[3-4]
- topology: topo-flat
  cluster_default: false
  flat: true
```

--------------------------------

### Install cert-manager only

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/development.md

Installs only the external prerequisite cert-manager. Requires Helm and kubectl, but does not require Skaffold or a local container build toolchain.

```sh
make prereqs
```

--------------------------------

### Install Slurm Helm chart with metrics enabled

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/autoscaling.md

Installs the Slurm chart from the OCI registry, enabling controller metrics and service monitor for Prometheus.

```sh
helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
  --set 'controller.metrics.enabled=true' \
  --set 'controller.metrics.serviceMonitor.enabled=true' \
  --namespace slurm --create-namespace
```

--------------------------------

### Create and annotate BYO Secret, then install chart with externalCertInjection

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/installation.md

Run these commands to create a namespace, create a TLS secret from your certificate and key, add the ca.crt data, annotate the secret for cainjector, and install the chart with externalCertInjection enabled. The chart will fail at template time if both certManager.enabled and externalCertInjection.enabled are true. Use helm install --dry-run=server to test the lookup.

```sh
kubectl create namespace slinky
kubectl create secret tls slurm-operator-webhook-byo \
  --namespace=slinky \
  --cert=tls.crt --key=tls.key
kubectl patch secret slurm-operator-webhook-byo \
  --namespace=slinky \
  --type=merge \
  -p "{\"data\":{\"ca.crt\":\"$(base64 < ca.crt | tr -d '\n')\"}}"
kubectl annotate secret slurm-operator-webhook-byo \
  --namespace=slinky \
  cert-manager.io/allow-direct-injection=true

helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
  --set 'certManager.enabled=false' \
  --set 'externalCertInjection.enabled=true' \
  --set 'externalCertInjection.secretName=slurm-operator-webhook-byo' \
  --set 'webhook.validatingAnnotations.cert-manager\.io/inject-ca-from-secret=slinky/slurm-operator-webhook-byo' \
  --set 'webhook.mutatingAnnotations.cert-manager\.io/inject-ca-from-secret=slinky/slurm-operator-webhook-byo' \
  --namespace=slinky --create-namespace
```

--------------------------------

### Install Slurm with metrics and service monitor enabled

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/installation.md

Installs the Slurm chart with controller metrics and Prometheus service monitor enabled, creating the slurm namespace.

```sh
helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
  --set 'controller.metrics.enabled=true' \
  --set 'controller.metrics.serviceMonitor.enabled=true' \
  --namespace=slurm --create-namespace
```

--------------------------------

### Install Slurm with login and SSSD configuration

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/installation.md

Installs the Slurm chart with login sets enabled and provides an SSSD configuration file from the local home directory.

```sh
helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
  --set 'loginsets.slinky.enabled=true' \
  --set-file "loginsets.slinky.sssdConf=${HOME}/sssd.conf" \
  --namespace=slurm --create-namespace
```

--------------------------------

### Create sbatch script for PyTorch GCN example

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/tutorial-pytorch.md

Creates the file `pytorch-sbatsh.sh` (note the typo in the filename) that runs the GCN PyTorch example on three nodes. It requests 3 tasks across 3 nodes with a 10-minute time limit, 100 MB memory, and writes output to `pytorch-output_%j.out` and errors to `pytorch-errors_%j.err`. The `srun` command runs `python3 main.py` with specified hyperparameters.

```bash
#!/bin/bash
# Slurm Parameters
#SBATCH -n 3                      # Run n tasks
#SBATCH -N 3                      # Run across N nodes
#SBATCH -t 0-00:10                # Time limit in D-HH:MM
#SBATCH -p slinky                 # Partition to submit to
#SBATCH --mem=100                 # Memory pool for all cores
#SBATCH -o pytorch-output_%j.out  # File to which STDOUT will be written, %j inserts jobid
#SBATCH -e pytorch-errors_%j.err  # File to which STDERR will be written, %j inserts jobid

echo "This was run on $SLURM_JOB_NODELIST."
srun -D /tmp/examples/gcn python3 main.py --epochs 200 --lr 0.01 --l2 5e-4 --dropout-p 0.5 --hidden-dim 16 --val-every 20 --include-bias

```

--------------------------------

### Start external slurmd

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/hybrid.md

When slurmd is external to Kubernetes, start it with the --conf-server option pointing to the external Slurmctld host and port.

```sh
slurmd --conf-server "${SLURMCTLD_HOST}:${SLURMCTLD_PORT}"
```

--------------------------------

### Install a Slurm cluster

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/index.rst

Installs a Slurm cluster using the slurm chart with a nodeset named 'slinky' and all partitions enabled.

```sh
helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
  --set-json 'nodesets={"slinky":{}}' \
  --set "partitions.all.enabled=true" \
  --namespace=slurm --create-namespace
```

--------------------------------

### Install slurm-operator and its CRDs

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/index.rst

Installs the slurm-operator CRDs chart and the slurm-operator chart using Helm. The CRDs chart must be installed first.

```sh
helm install slurm-operator-crds oci://ghcr.io/slinkyproject/charts/slurm-operator-crds
helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
  --namespace=slinky --create-namespace
```

--------------------------------

### jwtKey configuration example

Source: https://github.com/slinkyproject/slurm-operator/blob/main/helm/slurm/UPGRADE.md

Example YAML showing the replacement of jwtHs256KeyRef with jwtKey. Set create to false and modify secretRef to reference an existing secret.

```yaml
jwtKey:
  create: false
  secretRef:
    name: slurm-auth-jwths256
    key: jwt_hs256.key
```

--------------------------------

### Install without cert-manager

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/installation.md

Installs the chart without cert-manager, generating a self-signed CA and serving certificate via Helm's genCA/genSignedCert functions.

```sh
helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
  --set 'certManager.enabled=false' \
  --namespace=slinky --create-namespace
```

--------------------------------

### Install Prometheus Helm chart

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/autoscaling.md

Adds the Prometheus community Helm repository and installs the kube-prometheus-stack chart with CRDs enabled in the prometheus namespace.

```sh
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack \
  --set 'installCRDs=true' \
  --namespace prometheus --create-namespace
```

--------------------------------

### Install mariadb-operator and CRDs

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/installation.md

Adds the MariaDB Helm repository, updates it, and installs the mariadb-operator CRDs and operator in the 'mariadb' namespace.

```sh
helm repo add mariadb-operator https://helm.mariadb.com/mariadb-operator
helm repo update
helm install mariadb-operator-crds mariadb-operator/mariadb-operator-crds
helm install mariadb-operator mariadb-operator/mariadb-operator \
  --namespace mariadb --create-namespace
```

--------------------------------

### Example Kubernetes node conditions for CPU and GPU problems

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/nodeset-operations.md

Shows the node conditions that would be propagated to Slurm when the node is cordoned.

```yaml
status:
  conditions:
  - type: CPUProblem
    reason: BadCPU
    message: "CPU 17: Machine Check Exception"
  - type: GPUProblem
    reason: GpuCountMismatch
    message: "GPU count mismatch detected: Node has 3, expected 4"
```

--------------------------------

### Install Slurm with root SSH authorized keys

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/installation.md

Installs the Slurm chart with login sets enabled and provides a public SSH key for root access, even if SSSD is misconfigured.

```sh
helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
  --set 'loginsets.slinky.enabled=true' \
  --set-file "loginsets.slinky.rootSshAuthorizedKeys=${HOME}/.ssh/id_ed25519.pub" \
  --namespace=slurm --create-namespace
```

--------------------------------

### Install cert-manager with CRDs

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/index.rst

Installs cert-manager with its CRDs enabled using Helm. Required before installing the slurm-operator.

```sh
helm install \
  cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --namespace cert-manager --create-namespace \
  --set crds.enabled=true
```

--------------------------------

### Start external sackd

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/hybrid.md

When login hosts are external to Kubernetes, start sackd with the --conf-server option pointing to the external Slurmctld host and port.

```sh
sackd --conf-server "${SLURMCTLD_HOST}:${SLURMCTLD_PORT}"
```

--------------------------------

### Install with CRDs as subchart

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/installation.md

Installs the slurm-operator and its CRDs in the same Helm release by enabling the crds subchart.

```sh
helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
  --set 'crds.enabled=true' \
  --namespace=slinky --create-namespace
```

--------------------------------

### Bootstrap current Kubernetes context

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/development.md

Selects the Kubernetes context and installs cert-manager, CRDs, the locally built operator, and Slurm onto an existing cluster. `--registry` sets `SKAFFOLD_DEFAULT_REPO`; setting that environment variable directly is equivalent.

```sh
kubectl config use-context my-remote-cluster
./hack/kind.sh \
  --existing-cluster \
  --registry=ghcr.io/my-user \
  --core
```

--------------------------------

### Install Slurm with custom storage class for controller persistence

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/installation.md

Lists available storage classes and installs Slurm with a specific storage class for the controller's persistent volume claim. Replace $STORAGE_CLASS with an existing storage class name.

```sh
kubectl get storageclasses.storage.k8s.io
```

```sh
helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
  --set "controller.persistence.storageClassName=$STORAGE_CLASS" \
  --namespace=slurm --create-namespace
```

--------------------------------

### Install with externally-managed webhook certificate

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/installation.md

Installs the chart using a pre-existing TLS secret for the webhook certificate, disabling cert-manager and enabling external certificate injection.

```sh
helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
  --set 'certManager.enabled=false' \
  --set 'externalCertInjection.enabled=true' \
  --set 'externalCertInjection.secretName=slurm-operator-webhook-ca' \
  --namespace=slinky --create-namespace
```

--------------------------------

### Install with namespace-scoped watching

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/installation.md

Restricts the operator and webhook to watch resources only in the specified namespaces. Cluster-scoped resources are always watched regardless.

```sh
helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
  --set 'operator.namespaces=slurm-system,production' \
  --set 'webhook.namespaces=slurm-system,production' \
  --namespace=slinky --create-namespace
```

--------------------------------

### Modify base-extra Dockerfile to Install JupyterLab

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/jupyter.md

Edit the base-extra layer in the Dockerfile to install Python and JupyterLab for rockylinux9 images.

```dockerfile
################################################################################
FROM base AS base-extra

SHELL ["bash", "-c"]

RUN --mount=type=cache,target=/var/cache/dnf,sharing=locked <<EOR
# Install Extra Packages
set -xeuo pipefail
dnf -q -y install \
  python3
dnf -q -y install \
  python3-pip
pip install jupyterlab
EOR

################################################################################
```

--------------------------------

### Configure plugstack.conf with pyxis include

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/pyxis.md

Adds an include directive to plugstack.conf using glob syntax to avoid slurmctld failure. Only login and slurmd pods should have pyxis libraries installed.

```yaml
configFiles:
  plugstack.conf: |
    include /usr/share/pyxis/*
```

--------------------------------

### Install Slurm-operator with CRDs

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/tpu-slinky-jax.md

Installs the latest stable Slurm-operator from the OCI chart with CRDs enabled in the slinky namespace. Creates the namespace if it does not exist.

```bash
helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
  --set crds.enabled=true \
  -n slinky \
  --create-namespace
```

--------------------------------

### Create Slurm submission script (submit_jax.sh) for TPU job

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/tpu-slinky-jax.md

Bash script that submits a Slurm job requesting all 8 TPUs (--gres=tpu:8). It installs pip and venv, creates a virtual environment, installs JAX with TPU support from the specified URL, and runs jax_test.py. Requires internet access in the container.

```bash
#!/usr/bin/env bash
set -euo pipefail

#SBATCH --job-name=jax-tpu-test
#SBATCH --partition=tpu
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --gres=tpu:8  # Request all 8 TPUs on the node
#SBATCH --output=jax_tpu_%j.out
#SBATCH --error=jax_tpu_%j.err
#SBATCH --time=00:15:00

echo "=== Job started at $(date) ==="
echo "Running on node: $(hostname)"

# Install pip & venv dynamically (requires internet access in container)
echo "Installing system packages (pip and venv)..."
export DEBIAN_FRONTEND=noninteractive
apt-get update -y && apt-get install -y python3-pip python3-venv python3-full

# Create virtualenv and install JAX with TPU support
VENV_DIR="/tmp/jax_tpu_venv"
echo "Creating virtualenv..."
python3 -m venv --system-site-packages "$VENV_DIR"
source "$VENV_DIR"/bin/activate

echo "Installing JAX..."
python3 -m pip install --upgrade pip
python3 -m pip install "jax[tpu]" -f https://storage.googleapis.com/jax-releases/libtpu_releases.html

# Execute the JAX test
echo "Running JAX test script..."
python3 jax_test.py

echo "=== Job finished at $(date) ==="

```

--------------------------------

### Deploy Slurm cluster with Helm

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/tpu-slinky-jax.md

Helm command to install the Slurm cluster using the custom tpu-values.yaml. Creates the slurm namespace if it does not exist.

```bash
helm install slurm-tpu oci://ghcr.io/slinkyproject/charts/slurm -f tpu-values.yaml -n slurm --create-namespace

```

--------------------------------

### Kubernetes nodes annotated to match topology.yaml

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/topology.md

Four Kubernetes Node manifests annotated with topology.slinky.slurm.net/spec to match the topology.yaml example. Each node specifies its switch and block topology.

```yaml
---
apiVersion: v1
kind: Node
metadata:
  name: node1
  annotations:
    topology.slinky.slurm.net/spec: topo-switch:s1,topo-block:b1
---
apiVersion: v1
kind: Node
metadata:
  name: node2
  annotations:
    topology.slinky.slurm.net/spec: topo-switch:s1,topo-block:b1
---
apiVersion: v1
kind: Node
metadata:
  name: node3
  annotations:
    topology.slinky.slurm.net/spec: topo-switch:s2,topo-block:b2
---
apiVersion: v1
kind: Node
metadata:
  name: node4
  annotations:
    topology.slinky.slurm.net/spec: topo-switch:s2,topo-block:b2
```

--------------------------------

### Get ResourceClaim status

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/installation.md

Command to list ResourceClaims in the slurm namespace, showing allocation state.

```bash
$ kubectl get resourceclaim -n slurm
NAME                                                     STATE                AGE
slurm-worker-slinky-jvb8l-compute-domain-channel-4rpsc   allocated,reserved   7m8s

```

--------------------------------

### Configure extraSshdConfig on a LoginSet

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/ssh-access.md

Appends custom sshd_config settings to the generated file. This example sets ClientAliveInterval and ClientAliveCountMax. Note that keywords already emitted by the operator (Include, UsePAM, Subsystem, X11Forwarding on login pods, AuthenticationMethods on worker pods) cannot be overridden here because sshd_config uses the first value for each keyword.

```yaml
loginsets:
  slinky:
    extraSshdConfig: |
      ClientAliveInterval 60
      ClientAliveCountMax 3

```

--------------------------------

### Verify KEDA Metrics API Server is running

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/autoscaling.md

Checks the availability of the KEDA metrics API server using kubectl get apiservice.

```console
$ kubectl get apiservice -l app.kubernetes.io/instance=keda
NAME                              SERVICE                                AVAILABLE   AGE
v1beta1.external.metrics.k8s.io   keda/keda-operator-metrics-apiserver   True        22h
```

--------------------------------

### Check Slurm pod status

Source: https://github.com/slinkyproject/slurm-operator/blob/main/helm/slurm/templates/NOTES.txt

Run this command to check the status of the Slurm pods after installation. It uses the namespace and chart name from the Helm release.

```bash
kubectl --namespace={{ include "slurm.namespace" $ }} get pods -l helm.sh/chart={{ include "slurm.chart" $ }} --watch
```

--------------------------------

### Show Slurm Node Features with scontrol

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/node-features.md

Displays the available and active features of a Slurm node. In the example, the baseline features `GPU` and `slinky` are preserved, and the prefixed feature `k8s/nn-75bfcf47ca3e4f7dc` is added by the operator.

```console
$ scontrol show node slinky-0 | grep -Eo "NodeName=[^ ]+|[ ]*AvailableFeatures=[^ ]+|[ ]*ActiveFeatures=[^ ]+"
NodeName=slinky-0
   AvailableFeatures=GPU,k8s/nn-75bfcf47ca3e4f7dc,slinky
   ActiveFeatures=GPU,k8s/nn-75bfcf47ca3e4f7dc,slinky

```

--------------------------------

### Kubernetes Node annotation example

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/topology.md

Shows a Kubernetes Node manifest with the topology.slinky.slurm.net/spec annotation. The annotation value is used by the operator to update Slurm node topology when the pod runs as a NodeSet pod.

```yaml
apiVersion: v1
kind: Node
metadata:
  name: node0
  annotations:
    topology.slinky.slurm.net/spec: topo-switch:s0,topo-block:b0
```

--------------------------------

### Get the drain reason

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/nodeset-operations.md

Retrieves the message of the 'SlurmNodeStateDrain' condition to get the drain reason.

```sh
kubectl get pod <pod> -o jsonpath='{.status.conditions[?(@.type=="SlurmNodeStateDrain")].message}'
```

--------------------------------

### Create GKE cluster and TPU node pool

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/tpu-slinky-jax.md

Sets up environment variables and creates a GKE Standard cluster with a TPU v5e Spot node pool (8-chip ct5lp-hightpu-8t). Requires a GCP project ID and zone.

```bash
PROJECT="<YOUR_GCP_PROJECT_ID>"
CLUSTER="tpu-slurm-cluster"
ZONE="us-central1-a"
VERSION="1.35"

# Create the base GKE Standard cluster
gcloud container clusters create "${CLUSTER}" \
    --cluster-version="${VERSION}" \
    --zone="${ZONE}" \
    --project="${PROJECT}" \
    --num-nodes="3"

# Create the TPU v5e Spot node pool (8-chip ct5lp-hightpu-8t)
gcloud container node-pools create tpu-v5-single-host-spot \
    --cluster="${CLUSTER}" \
    --zone="${ZONE}" \
    --project="${PROJECT}" \
    --machine-type="ct5lp-hightpu-8t" \
    --spot \
    --num-nodes="1" \
    --node-locations="${ZONE}"
```

--------------------------------

### List /etc/enroot directory contents

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/override-config-file.md

Executes ls on /etc/enroot inside the pod to show that other files remain unchanged.

```console
$ kubectl exec -it -n slurm slurm-worker-slinky-0 -- ls /etc/enroot
enroot.conf  enroot.conf.d  environ.d  hooks.d	mounts.d

```

--------------------------------

### Copy sbatch script to slurm-login pod

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/tutorial-pytorch.md

Copies the local `pytorch-sbatch.sh` file to the `slurm-login-slinky` pod's `/root/` directory. Replace `<hash>` with the hash of the current slurm-login-slinky pod.

```bash
kubectl -n slurm cp ~/location/of/script/pytorch-sbatch.sh slurm-login-slinky-<hash>:/root/

```

--------------------------------

### Create and delete local Kind cluster

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/development.md

Use `make kind-start` to create a development cluster and deploy the operator, CRDs, and a Slurm cluster. Use `make kind-stop` to delete the cluster. `make deploy` rebuilds once, while `make debug` builds Delve-enabled images.

```sh
make kind-start
```

```sh
make kind-stop
```

--------------------------------

### Create KWOK cluster with fake nodes

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/development.md

Creates a cluster with the operator and KWOK, then adds 100 fake nodes. Each fake node fits 110 pods; for more nodes, run `./hack/kwok-nodes.sh COUNT` with the total needed.

```sh
./hack/kind.sh --crds --operator --kwok
./hack/kwok-nodes.sh
```

--------------------------------

### Verify Virtual Functions with lspci

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/sriov.md

Run this command to confirm that SR-IOV Virtual Functions are created and visible on the system. The output shows Mellanox ConnectX-4 Lx Virtual Function entries.

```console
$ lspci | grep "Virtual Function"
01:00.2 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
01:00.3 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
01:00.4 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
01:00.5 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
01:00.6 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
01:00.7 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
01:01.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
01:01.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
```

--------------------------------

### Test IMEX channels with srun

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/installation.md

Runs a job to verify IMEX channel devices exist on the NodeSet pod, printing job ID, node name, and channel devices.

```bash
$ kubectl exec -n slurm slurm-controller-0 -- srun bash -c \ "printf '\n\n'; echo === IMEX Channel Test ===; \
 echo Job \$SLURM_JOB_ID Node \$SLURMD_NODENAME; \
echo Channel devices:; \
ls /dev/nvidia-caps-imex-channels/ | grep channel || echo No channels found; \
ls -la /dev/nvidia-caps-imex-channels/channel* 2>/dev/null || echo No channel devices;"


=== IMEX Channel Test ===
Job 74 Node validation-k8s-gpu-02
Channel devices:
channel1
crw-rw-rw- 1 root root 509, 1 Apr 20 17:08 /dev/nvidia-caps-imex-channels/channel1

```

--------------------------------

### Apply ScaledObject and verify KEDA and HPA resources

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/autoscaling.md

Commands to apply the ScaledObject YAML and check the created KEDA ScaledObject and HPA resources.

```console
$ kubectl apply -f scaledobject.yaml -n slurm
scaledobject.keda.sh/scale-radar created

$ kubectl get -n slurm scaledobjects
NAME           SCALETARGETKIND                     SCALETARGETNAME        MIN   MAX   TRIGGERS     AUTHENTICATION   READY   ACTIVE   FALLBACK   PAUSED    AGE
scale-radar    slinky.slurm.net/v1beta1.NodeSet    slurm-worker-radar    1     5     prometheus                    True    False    Unknown    Unknown   28s

$ kubectl get -n slurm hpa
NAME                    REFERENCE                      TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-scale-radar    NodeSet/slurm-worker-radar    <unknown>/5   1         5         0          32s
```

--------------------------------

### Submit GPU job and verify mapping files

Source: https://github.com/slinkyproject/slurm-operator/blob/main/helm/slurm/_vendor/nvidia/dcgm/README.md

Use these commands to submit a GPU job and check the mapping files inside the slurmd pod. The mapping files should contain the job ID.

```bash
# Submit job
sbatch --gres=gpu:2 --wrap="nvidia-smi; sleep 60"

# Check mapping files (inside slurmd pod)
ls /var/lib/dcgm-exporter/job-mapping/
# Should show files like: 0, 1 (containing job ID)
```

--------------------------------

### Describe NodeSet to verify volume mounts

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/override-config-file.md

Shows the output of kubectl describe for the NodeSet slurm-worker-slinky, confirming the volumeMounts and volumes are correctly configured.

```console
$ kubectl -n slurm describe nodesets.slinky.slurm.net slurm-worker-slinky
Name:         slurm-worker-slinky
Namespace:    slurm
Spec:
  Slurmd:
    Volume Mounts:
      Mount Path:  /etc/enroot/enroot.conf
      Name:        enroot-config
      Sub Path:    enroot.conf
  Volumes:
    Config Map:
      Items:
        Key:   enroot
        Path:  enroot.conf
      Name:    enroot-config
    Name:      enroot-config

```

--------------------------------

### Disable Slurm controller persistence

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/installation.md

Installs Slurm with controller persistence disabled. This is not recommended for production because cluster state is lost on pod restarts.

```sh
helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
  --set 'controller.persistence.enabled=false' \
  --namespace=slurm --create-namespace
```

--------------------------------

### Submit sbatch script

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/tutorial-pytorch.md

Submits the `pytorch-sbatch.sh` script to Slurm from the slurm-login-slinky node.

```bash
sbatch pytorch-sbatch.sh

```

--------------------------------

### Create webhook certificate Secret

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/installation.md

Creates a Kubernetes TLS secret for the webhook serving certificate, including the CA certificate, before installing the chart.

```sh
kubectl create namespace slinky
kubectl create secret tls slurm-operator-webhook-ca \
  --namespace=slinky \
  --cert=tls.crt --key=tls.key
kubectl patch secret slurm-operator-webhook-ca \
  --namespace=slinky \
  --type=merge \
  -p "{\"data\":{\"ca.crt\":\"$(base64 < ca.crt | tr -d '\n')\"}}"
```

--------------------------------

### Test enroot import

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/pyxis.md

Imports a Docker image using enroot to verify enroot functionality in the login container.

```bash
enroot import docker://alpine:latest
```

--------------------------------

### Uninstall all Slinky charts for 0.Y releases

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/index.rst

Uninstalls the Slurm chart, slurm-operator, and slurm-operator-crds. Required before installing a new release when upgrading between v0.Y versions.

```bash
helm --namespace=slurm uninstall slurm
helm --namespace=slinky uninstall slurm-operator
helm uninstall slurm-operator-crds
```

--------------------------------

### Verify enroot.conf content in pod

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/override-config-file.md

Executes cat on /etc/enroot/enroot.conf inside the slurm-worker-slinky-0 pod to confirm the file is replaced with the ConfigMap content.

```console
$ kubectl exec -it -n slurm slurm-worker-slinky-0 -- cat /etc/enroot/enroot.conf
ENROOT_RUNTIME_PATH         /run/enroot/${UID}/run
ENROOT_CONFIG_PATH          /run/enroot/${UID}/config
ENROOT_CACHE_PATH           /run/enroot/${UID}/cache
ENROOT_DATA_PATH            /run/enroot/${UID}/data
ENROOT_TEMP_PATH            /run/${UID}/tmp

```

--------------------------------

### Configure NodeSets and login pods with pyxis OCI image

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/pyxis.md

Defines a loginset and nodeset named 'pyxis' using the respective OCI images from ghcr.io/slinkyproject.

```yaml
loginsets:
  - name: pyxis
    image:
      repository: ghcr.io/slinkyproject/login-pyxis
nodesets:
  - name: pyxis
    image:
      repository: ghcr.io/slinkyproject/slurmd-pyxis
```

--------------------------------

### Delete Slinky CRDs manually

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/index.rst

Deletes Slinky CRDs using kubectl when they were not installed via the slurm-operator-crds helm chart. Includes a comment marking the clusters CRD as defunct.

```bash
kubectl delete customresourcedefinitions.apiextensions.k8s.io accountings.slinky.slurm.net
kubectl delete customresourcedefinitions.apiextensions.k8s.io clusters.slinky.slurm.net # defunct
kubectl delete customresourcedefinitions.apiextensions.k8s.io loginsets.slinky.slurm.net
kubectl delete customresourcedefinitions.apiextensions.k8s.io nodesets.slinky.slurm.net
kubectl delete customresourcedefinitions.apiextensions.k8s.io restapis.slinky.slurm.net
kubectl delete customresourcedefinitions.apiextensions.k8s.io tokens.slinky.slurm.net
```

--------------------------------

### Get Slurm controller service IP (ExternalName)

Source: https://github.com/slinkyproject/slurm-operator/blob/main/helm/slurm/templates/NOTES.txt

For a controller service of type ExternalName, this retrieves the external name and sets the SLURM_CONF_SERVER variable with the controller port (default 6817).

```bash
SLURM_CONTROLLER_IP="$(kubectl get services -n {{ include "slurm.namespace" $ }} {{ include "slurm.controller.name" . }} -o jsonpath='{.spec.externalName}')"
SLURM_CONF_SERVER=$SLURM_CONTROLLER_IP:{{ .Values.controller | dig "service" "port" "6817" }}
```

--------------------------------

### Build slurmd Image with JupyterLab

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/jupyter.md

Set environment variables and build the slurmd image with the modified base-extra layer.

```bash
export VERSION=26.05
export FLAVOR="rockylinux9"
export BAKE_IMPORTS="--file ./docker-bake.hcl --file ./$VERSION/$FLAVOR/slurm.hcl"
docker bake $BAKE_IMPORTS slurmd
```

--------------------------------

### Test Slurm commands from login pod

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/installation.md

Runs basic Slurm commands to verify the cluster is functioning.

```sh
sinfo
srun hostname
sbatch --wrap="sleep 60"
squeue
sacct
```

--------------------------------

### Get Slurm controller service IP (LoadBalancer)

Source: https://github.com/slinkyproject/slurm-operator/blob/main/helm/slurm/templates/NOTES.txt

For a controller service of type LoadBalancer, this retrieves the external IP and sets the SLURM_CONF_SERVER variable with the controller port (default 6817).

```bash
SLURM_CONTROLLER_IP="$(kubectl get services -n {{ include "slurm.namespace" $ }} {{ include "slurm.controller.name" . }} -o jsonpath='{.status.loadBalancer.ingress[0].ip}')"
SLURM_CONF_SERVER=$SLURM_CONTROLLER_IP:{{ .Values.controller | dig "service" "port" "6817" }}
```

--------------------------------

### Enable node pinning directly in NodeSet CR

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/nodeset-operations.md

Set pinToNode=true in a NodeSet custom resource to enable node pinning. This example creates a NodeSet named gpu-workers with 4 replicas.

```yaml
apiVersion: slinky.slurm.net/v1beta1
kind: NodeSet
metadata:
  name: gpu-workers
spec:
  pinToNode: true
  replicas: 4

```

--------------------------------

### Expected output of JAX TPU training job

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/tpu-slinky-jax.md

This is the expected output from the job log, showing the JAX environment bootstrap, TPU device detection, and training progress. It confirms a successful end-to-end TPU Slurm deployment on GKE.

```default
=== Job started at Thu May 21 18:16:24 UTC 2026 ===
Running on node: tpu-0
Allocated GRES:
...
JAX successfully imported inside job!
Running JAX test script...
--- JAX TPU Model Training Verification ---
JAX version: 0.10.1
Available devices: [TpuDevice(id=0, process_index=0, coords=(0,0,0), core_on_chip=0), TpuDevice(id=1, process_index=0, coords=(1,0,0), core_on_chip=0), TpuDevice(id=2, process_index=0, coords=(0,1,0), core_on_chip=0), TpuDevice(id=3, process_index=0, coords=(1,1,0), core_on_chip=0), TpuDevice(id=4, process_index=0, coords=(0,2,0), core_on_chip=0), TpuDevice(id=5, process_index=0, coords=(1,2,0), core_on_chip=0), TpuDevice(id=6, process_index=0, coords=(0,3,0), core_on_chip=0), TpuDevice(id=7, process_index=0, coords=(1,3,0), core_on_chip=0)]
SUCCESS: JAX is using TPU!
Starting training loop (100 epochs)...
Epoch 0: Loss = 0.693804
Epoch 10: Loss = 0.535265
Epoch 20: Loss = 0.427468
Epoch 30: Loss = 0.329034
Epoch 40: Loss = 0.245817
Epoch 50: Loss = 0.187892
Epoch 60: Loss = 0.152073
Epoch 70: Loss = 0.129508
Epoch 80: Loss = 0.114930
Epoch 90: Loss = 0.104988
Training completed in 0.1254 seconds!
Final Loss: 0.098378
-------------------------------------------
=== Job finished at Thu May 21 18:16:38 UTC 2026 ===
```

--------------------------------

### Get Slurm controller service IP (ClusterIP)

Source: https://github.com/slinkyproject/slurm-operator/blob/main/helm/slurm/templates/NOTES.txt

For a controller service of type ClusterIP (default), this retrieves the cluster IP and sets the SLURM_CONF_SERVER variable with the controller port (default 6817).

```bash
SLURM_CONTROLLER_IP="$(kubectl get services -n {{ include "slurm.namespace" $ }} {{ include "slurm.controller.name" . }} -o jsonpath='{.spec.clusterIP}')"
SLURM_CONF_SERVER=$SLURM_CONTROLLER_IP:{{ .Values.controller | dig "service" "port" "6817" }}
```

--------------------------------

### Submit a test job to trigger scaling

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/autoscaling.md

Command to submit a single-node job to the 'radar' partition to test the scaling behavior.

```sh
sbatch --wrap "sleep 30" --partition radar --exclusive
```

--------------------------------

### sbatch script to launch JupyterLab

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/jupyter.md

Submit this batch script using sbatch to launch JupyterLab on a Slinky cluster. The script sets job parameters and runs 'jupyter lab --port=9999 --no-browser'.

```bash
#!/bin/bash
#SBATCH --job-name=jupyterlab-singleuser
#SBATCH --account=vivian
#SBATCH --nodes=1
#SBATCH --time=01:00:00
#
## Launch the JupyterLab server:
jupyter lab --port=9999 --no-browser
```

--------------------------------

### Check Slurm partition with sinfo

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/autoscaling.md

Verify the Slurm partition 'radar' after scaling the NodeSet. The output shows one idle node.

```console
$ kubectl exec -n slurm statefulset/slurm-controller -- sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
radar        up   infinite      1   idle kind-worker
```

--------------------------------

### Copy scripts to worker pod and submit Slurm job

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/tpu-slinky-jax.md

Commands to copy jax_test.py and submit_jax.sh to the worker pod 'slurm-tpu-worker-tpu-0' in the 'slurm' namespace, then submit the job via sbatch. Uses kubectl cp and kubectl exec.

```bash
# Copy scripts to the worker pod "slurm-tpu-worker-tpu-0"
kubectl cp jax_test.py slurm/slurm-tpu-worker-tpu-0:/tmp/jax_test.py -c slurmd
kubectl cp submit_jax.sh slurm/slurm-tpu-worker-tpu-0:/tmp/submit_jax.sh -c slurmd

# Submit the job
kubectl exec -n slurm slurm-tpu-worker-tpu-0 -c slurmd -- bash -c "cd /tmp && sbatch submit_jax.sh"

```

--------------------------------

### Deploy operator to remote cluster

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/development.md

Sets the Kubernetes context and registry, then deploys the operator and CRDs using Skaffold and Helm. Existing release values are not retained, so add required cluster-specific values to `values-dev.yaml` before deploying.

```sh
export SKAFFOLD_KUBE_CONTEXT=my-remote-cluster
export SKAFFOLD_DEFAULT_REPO=ghcr.io/my-user
make deploy
```

--------------------------------

### ConfigMap for enroot.conf override

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/override-config-file.md

Creates a ConfigMap named enroot-config in the slurm namespace containing the custom enroot.conf content. The data key is 'enroot' and the content is a multi-line string with ENROOT_* environment variables.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: enroot-config
  namespace: slurm
data:
  enroot: |
    ENROOT_RUNTIME_PATH         /run/enroot/${UID}/run
    ENROOT_CONFIG_PATH          /run/enroot/${UID}/config
    ENROOT_CACHE_PATH           /run/enroot/${UID}/cache
    ENROOT_DATA_PATH            /run/enroot/${UID}/data
    ENROOT_TEMP_PATH            /run/${UID}/tmp

```

--------------------------------

### Check Slurm partition node count with sinfo

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/autoscaling.md

Command to verify that the partition 'radar' has no nodes, confirming the NodeSet is scaled to zero.

```console
$ slurm@slurm-controller-0:/tmp$ sinfo -p radar
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
radar        up   infinite      0    n/a
```

--------------------------------

### Verify GPU visibility with nvidia-smi

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/installation.md

Runs nvidia-smi inside a Slurm job to verify GPU visibility. Shows all 4 GB200 GPUs on the underlying host are visible within the NodeSet pod.

```bash
$ kubectl exec -n slurm slurm-controller-0 -- srun bash -c "echo GPU Devices Available on $(hostname):; printf '\n';nvidia-smi"
GPU Devices Available on validation-k8s-headnode:

Mon Apr 20 17:03:28 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 590.48.01              Driver Version: 590.48.01      CUDA Version: 13.1     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB200                   On  |   00000008:01:00.0 Off |                    0 |
| N/A   37C    P0            166W / 1200W |       0MiB / 189471MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GB200                   On  |   00000009:01:00.0 Off |                    0 |
| N/A   37C    P0            157W / 1200W |       0MiB / 189471MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GB200                   On  |   00000018:01:00.0 Off |                    0 |
| N/A   36C    P0            150W / 1200W |       0MiB / 189471MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA GB200                   On  |   00000019:01:00.0 Off |                    0 |
| N/A   36C    P0            172W / 1200W |       0MiB / 189471MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
```

--------------------------------

### Build images with Docker Bake

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/tutorial-pytorch.md

Build the Slurm container images using Docker Bake from the containers repository root. Replace <version> and <distribution> with the desired Slurm version and distribution. The SUFFIX environment variable can be set to append a custom tag to the built images.

```bash
docker bake --file containers/schedmd/slurm/docker-bake.hcl --file containers/schedmd/slurm/<version>/<distribution>/slurm.hcl

```

--------------------------------

### Enable DCGM integration in values.yaml

Source: https://github.com/slinkyproject/slurm-operator/blob/main/helm/slurm/_vendor/nvidia/dcgm/README.md

Add this configuration to your values.yaml to enable the DCGM integration, set the job mapping directory, and specify the script priority.

```yaml
vendor:
  nvidia:
    dcgm:
      enabled: true
      jobMappingDir: "/var/lib/dcgm-exporter/job-mapping"
      scriptPriority: "90"
```

--------------------------------

### Describe NodeSet to observe affinity

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/workload-isolation.md

Run this kubectl command to view the affinity section in the NodeSet after applying the Helm chart with affinity set in values.yaml.

```bash
kubectl describe NodeSet --namespace slurm
```

--------------------------------

### Apply ConfigMap with kubectl

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/override-config-file.md

Applies the ConfigMap defined in enroot-config.yaml to the cluster.

```bash
kubectl apply -f enroot-config.yaml

```

--------------------------------

### Verify Slurm-operator pods

Source: https://github.com/slinkyproject/slurm-operator/blob/main/docs/usage/tpu-slinky-jax.md

Lists pods in the slinky namespace to confirm both operator pods are running successfully.

```bash
kubectl get pods -n slinky
```