JobSet

https://jobset.sigs.k8s.io/
Admin
JobSet is a Kubernetes-native API for managing a group of Kubernetes Jobs as a unit, enabling...
Tokens:17,837
Snippets:189
Trust Score:9.4
Update:5 months ago
Show doc for...
Tokens:
Raw
### Apply JobSet Network Configuration and Verify Pods/Services

Source: https://jobset.sigs.k8s.io/docs/troubleshooting

Deploy a JobSet network example using kubectl and check the status of created pods and services. This helps verify basic cluster communication setup.

```bash
root@VM-0-4-ubuntu:/home/ubuntu# vi jobset-network.yaml
root@VM-0-4-ubuntu:/home/ubuntu# kubectl apply -f jobset-network.yaml
jobset.jobset.x-k8s.io/network-jobset created
root@VM-0-4-ubuntu:/home/ubuntu# kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE   IP          NODE               NOMINATED NODE   READINESS GATES
network-jobset-leader-0-0-5xnzz    1/1     Running   0          17m   10.6.2.27   cluster1-worker    <none>           <none>
network-jobset-workers-0-0-78k9j   1/1     Running   0          17m   10.6.1.16   cluster1-worker2   <none>           <none>
network-jobset-workers-0-1-rmw42   1/1     Running   0          17m   10.6.2.28   cluster1-worker    <none>           <none>
root@VM-0-4-ubuntu:/home/ubuntu# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
example      ClusterIP   None         <none>        <none>    19s
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   2d1h


```

--------------------------------

### Set Up Kustomization Environment

Source: https://jobset.sigs.k8s.io/docs/installation

Creates a directory for Kustomize configuration and navigates into it. This is the first step for customizing your Jobset installation.

```bash
mkdir kustomize-jobset
cd kustomize-jobset
```

--------------------------------

### Install Latest Development Version

Source: https://jobset.sigs.k8s.io/docs/installation

Installs the latest development version of Jobset from the main branch using kubectl and Kustomize. The controller will be deployed in the 'jobset-system' namespace.

```bash
kubectl apply --server-side -k github.com/kubernetes-sigs/jobset/config/default?ref=main
```

--------------------------------

### Install Released Version with Helm

Source: https://jobset.sigs.k8s.io/docs/installation

Installs a specific released version of Jobset using Helm. The chart is installed into the 'jobset-system' namespace.

```bash
helm install jobset oci://registry.k8s.io/jobset/charts/jobset --version ${VERSION#v} --create-namespace --namespace=jobset-system
```

--------------------------------

### Install Released Version with Kubectl

Source: https://jobset.sigs.k8s.io/docs/installation

Installs a specific released version of Jobset using kubectl. Ensure the VERSION variable is set to your desired release.

```bash
VERSION=v0.11.1
kubectl apply --server-side -f https://github.com/kubernetes-sigs/jobset/releases/download/$VERSION/manifests.yaml
```

--------------------------------

### RestartJobSet Action Example

Source: https://jobset.sigs.k8s.io/docs/tasks/failure_policy

Configure the JobSet to restart up to 3 times if the 'leader' job fails. This example demonstrates the default behavior when no other rules match, using RestartJobSet as the action.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: restartjobset-action-example
spec:
  failurePolicy:
    maxRestarts: 3
    rules:
      # The JobSet will restart when the leader job fails.
      - action: RestartJobSet
        targetReplicatedJobs:
        - leader
  replicatedJobs:
  - name: leader
    replicas: 1
    template:
      spec:
        # Set backoff limit to 0 so job will immediately fail if any pod fails.
        backoffLimit: 0
        completions: 2
        parallelism: 2
        template:
          spec:
            containers:
            - name: leader
              image: bash:latest
              command:
              - bash
              - -xc
              - |
                echo "JOB_COMPLETION_INDEX=$JOB_COMPLETION_INDEX"
                if [[ "$JOB_COMPLETION_INDEX" == "0" ]]; then
                  for i in $(seq 10 -1 1)
                  do
                    echo "Sleeping in $i"
                    sleep 1
                  done
                  exit 1
                fi
                for i in $(seq 1 1000)
                do
                  echo "$i"
                  sleep 1
                done
  - name: workers
    replicas: 1
    template:
      spec:
        backoffLimit: 0
        completions: 2
        parallelism: 2
        template:
          spec:
            containers:
            - name: worker
              image: bash:latest
              command:
              - bash
              - -xc
              - |
                sleep 1000

```

--------------------------------

### Install JobSet Controller

Source: https://jobset.sigs.k8s.io/docs/overview

Installs the latest release of JobSet using kubectl. Requires a Kubernetes cluster running one of the last 3 minor versions. The controller will be deployed in the 'jobset-system' namespace.

```bash
kubectl apply --server-side -f https://github.com/kubernetes-sigs/jobset/releases/download/v0.11.1/manifests.yaml
```

--------------------------------

### Kustomization File - Basic Resources

Source: https://jobset.sigs.k8s.io/docs/installation

A basic kustomization.yaml file that references the downloaded manifest. This is the starting point for defining Kustomize configurations.

```yaml
resources:
  - manifests.yaml
```

--------------------------------

### FailJobSet Action Example

Source: https://jobset.sigs.k8s.io/docs/tasks/failure_policy

Configure the JobSet to fail immediately if the 'leader' job fails. This example sets a maxRestarts limit and defines a rule to trigger FailJobSet for the 'leader' job.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: failjobset-action-example
spec:
  failurePolicy:
    maxRestarts: 3
    rules:
      # The JobSet will fail immediately when the leader job fails.
      - action: FailJobSet
        targetReplicatedJobs:
        - leader
  replicatedJobs:
  - name: leader
    replicas: 1
    template:
      spec:
        # Set backoff limit to 0 so job will immediately fail if any pod fails.
        backoffLimit: 0
        completions: 2
        parallelism: 2
        template:
          spec:
            containers:
            - name: leader
              image: bash:latest
              command:
              - bash
              - -xc
              - |
                echo "JOB_COMPLETION_INDEX=$JOB_COMPLETION_INDEX"
                if [[ "$JOB_COMPLETION_INDEX" == "0" ]]; then
                  for i in $(seq 10 -1 1)
                  do
                    echo "Sleeping in $i"
                    sleep 1
                  done
                  exit 1
                fi
                for i in $(seq 1 1000)
                do
                  echo "$i"
                  sleep 1
                done
  - name: workers
    replicas: 1
    template:
      spec:
        backoffLimit: 0
        completions: 2
        parallelism: 2
        template:
          spec:
            containers:
            - name: worker
              image: bash:latest
              command:
              - bash
              - -xc
              - |
                sleep 1000

```

--------------------------------

### RestartJobAndIgnoreMaxRestarts Action Example

Source: https://jobset.sigs.k8s.io/docs/tasks/failure_policy

This example demonstrates how to configure `RestartJobAndIgnoreMaxRestarts` to recreate only the failed worker Job without counting towards the `maxRestarts` limit. It also shows how `RestartJobSetAndIgnoreMaxRestarts` can be used to recreate all jobs when a leader job fails.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: js
spec:
  failurePolicy:
    maxRestarts: 2
    rules:
    # If any Job from replicatedJob `rj-leader` fails, restart the JobSet by recreating ALL Jobs
    # Do NOT count the restart towards `jobSet.spec.failurePolicy.maxRestarts`
    - name: r_leader
      action: RestartJobSetAndIgnoreMaxRestarts
      targetReplicatedJobs:
      - rj-leader
    # If any Job from replicatedJob `rj-worker` fails, restart the JobSet by recreating only the failed Job
    # Do NOT count the restart towards `jobSet.spec.failurePolicy.maxRestarts`
    - name: r_worker
      action: RestartJobAndIgnoreMaxRestarts
      targetReplicatedJobs:
      - rj-worker
  replicatedJobs:
  - name: rj-leader
    replicas: 1
    template:
      spec:
        parallelism: 2
        completions: 2
        backoffLimit: 0 # Job fails if any Pod fails
        template:
          spec:
            restartPolicy: Never
            terminationGracePeriodSeconds: 0 # Useful for testing
            containers:
            - name: leader
              image: busybox:1.37
              command: ["/bin/sh", "-c"]
              # Fail a leader Pod with `kubectl exec ${POD_NAME:?} -c leader -- touch /tmp/fail`
              args:
              - |
                echo "Start"
                while true; do
                  if [ -f /tmp/fail ]; then
                    echo "Exiting 1"
                    exit 1
                  elif [ -f /tmp/succeed ]; then
                    echo "Exiting 0"
                    exit 0
                  fi
                  sleep 1
                done
  - name: rj-worker
    replicas: 1
    template:
      spec:
        parallelism: 2
        completions: 2
        backoffLimit: 0 # Job fails if any Pod fails
        template:
          spec:
            restartPolicy: Never
            terminationGracePeriodSeconds: 0 # Useful for testing
            containers:
            - name: worker
              image: busybox:1.37
              command: ["/bin/sh", "-c"]
              # Fail a worker Pod with `kubectl exec ${POD_NAME:?} -c worker -- touch /tmp/fail`
              args:
              - |
                echo "Start"
                while true; do
                  if [ -f /tmp/fail ]; then
                    echo "Exiting 1"
                    exit 1
                  elif [ -f /tmp/succeed ]; then
                    echo "Exiting 0"
                    exit 0
                  fi
                  sleep 1
                done

```

--------------------------------

### PyTorch Distributed Training Example

Source: https://jobset.sigs.k8s.io/docs/tasks/workload_examples

Demonstrates distributed training of a CNN on the MNIST dataset using PyTorch and JobSet. Ensure machine learning container images are accessible.

```text
Train Epoch: 1 [0/60000 (0%)]   loss=2.3130, accuracy=12.5000
Train Epoch: 1 [6400/60000 (11%)]       loss=0.4624, accuracy=86.4171
Train Epoch: 1 [12800/60000 (21%)]      loss=0.3381, accuracy=90.0109
Train Epoch: 1 [19200/60000 (32%)]      loss=0.2724, accuracy=91.8916
Train Epoch: 1 [25600/60000 (43%)]      loss=0.2367, accuracy=92.9941
Train Epoch: 1 [32000/60000 (53%)]      loss=0.2111, accuracy=93.7063
Train Epoch: 1 [38400/60000 (64%)]      loss=0.1925, accuracy=94.2882
Train Epoch: 1 [44800/60000 (75%)]      loss=0.1796, accuracy=94.6416
Train Epoch: 1 [51200/60000 (85%)]      loss=0.1677, accuracy=94.9945
Train Epoch: 1 [57600/60000 (96%)]      loss=0.1565, accuracy=95.3229
Test Loss: 0.0635, Test Accuracy: 97.8400
```

--------------------------------

### Handle Host Maintenance Events with JobSet

Source: https://jobset.sigs.k8s.io/docs/tasks/failure_policy

This example models handling a host maintenance event. A pod eviction due to maintenance triggers a `DisruptionTarget` pod condition, which in turn triggers the job's `podFailurePolicy`. The JobSet's failure policy rule matches on the `PodFailurePolicy` reason and restarts the JobSet without counting it against `maxRestarts`. Any other type of failure in the `leader` job will trigger a normal restart that counts against `maxRestarts` (which is 0 in this case, meaning any other failure will cause the JobSet to fail immediately).

```yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: host-maintenance-event-model
spec:
  failurePolicy:
    maxRestarts: 0
    rules:
      # The JobSet will restart an unlimited number of times when failure matches the pod failure policy.
      - action: RestartJobSetAndIgnoreMaxRestarts
        onJobFailureReasons:
        - PodFailurePolicy
      # The JobSet is restarted as normal when the leader job fails and the above rule is not matched.
      - action: RestartJobSet
        targetReplicatedJobs:
        - leader
  replicatedJobs:
  - name: leader
    replicas: 1
    template:
      spec:
        # Set backoff limit to 0 so job will immediately fail if any pod fails.
        backoffLimit: 0
        completions: 2
        parallelism: 2
        template:
          spec:
            restartPolicy: Never
            containers:
            - name: leader
              image: bash:latest
              command:
              - bash
              - -xc
              - |
                echo "JOB_COMPLETION_INDEX=$JOB_COMPLETION_INDEX"
                if [[ "$JOB_COMPLETION_INDEX" == "0" ]]; then
                  for i in $(seq 120 -1 1)
                  do
                    echo "Sleeping in $i"
                    sleep 1
                  done
                  exit 1
                fi
                for i in $(seq 1 1000)
                do
                  echo "$i"
                  sleep 1
                done
        # This failure policy is triggered when a node undergoes host maintenace.
        # In such a case, the pods are evicted and the job will fail with a condition
        # of type DisruptionTarget.
        podFailurePolicy:
          rules:
            - action: FailJob
              onPodConditions: 
              - type: DisruptionTarget
  - name: workers
    replicas: 1
    template:
      spec:
        backoffLimit: 0
        completions: 2
        parallelism: 2
        template:
          spec:
            containers:
            - name: worker
              image: bash:latest
              command:
              - bash
              - -xc
              - |
                sleep 1000

```

--------------------------------

### Install Cert-Manager for Jobset Webhooks

Source: https://jobset.sigs.k8s.io/docs/installation

Install cert-manager on your cluster using a specific version. This is an optional step if you prefer to use cert-manager for webhook certificate management instead of Jobset's internal certificate.

```bash
VERSION=v1.11.0
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/$VERSION/cert-manager.yaml

```

--------------------------------

### JobSet with Shared PVCs and Different Retention Policies

Source: https://jobset.sigs.k8s.io/docs/tasks/volume_claim_policies

This example creates two shared PVCs for two ReplicatedJobs. The 'initializer' PVC is deleted when the JobSet is deleted, while the 'checkpoints' PVC is retained. The second ReplicatedJob depends on the first and writes checkpoints to the retained PVC.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: volume-claim-trainjob
spec:
  volumeClaimPolicies:
    - templates:
        - metadata:
            name: initializer
          spec:
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 1Gi
    - templates:
        - metadata:
            name: checkpoints
          spec:
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 1Gi
      retentionPolicy:
        whenDeleted: Retain
  replicatedJobs:
    - name: initializer
      template:
        spec:
          template:
            spec:
              containers:
                - name: initializer
                  image: busybox
                  command:
                    - /bin/sh
                    - -c
                    - |
                      echo "Download pre-trained model into /workspace/model/qwen3.txt"
                      echo 'Qwen3-30b' > /workspace/model/qwen3.txt
                  volumeMounts:
                    - mountPath: /workspace/model
                      name: initializer
    - name: node
      dependsOn:
        - name: initializer
          status: Complete
      template:
        spec:
          parallelism: 3
          completions: 3
          template:
            spec:
              containers:
                - name: node
                  image: busybox
                  command:
                    - /bin/sh
                    - -c
                    - |
                      echo "Read pre-trained model" && cat /workspace/model/qwen3.txt
                      echo "Write model checkpoint to /workspace/checkpoints/model_node_${JOB_COMPLETION_INDEX}"
                      echo "Checkpoint: model_${JOB_COMPLETION_INDEX}" > /workspace/checkpoints/model_${JOB_COMPLETION_INDEX}.txt
                  volumeMounts:
                    - mountPath: /workspace/model
                      name: initializer
                    - mountPath: /workspace/checkpoints
                      name: checkpoints

```

--------------------------------

### Assign Coordinator Pod

Source: https://jobset.sigs.k8s.io/docs/concepts

Configure a specific pod as a coordinator using `spec.coordinator`. This adds a `jobset.sigs.k8s.io/coordinator` annotation and label to all Jobs and Pods, providing a stable network endpoint for communication. Ensure Kueue version v0.8.3+ or v0.9.0+ is installed if using Kueue.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: pytorch
spec:
  coordinator:
    replicatedJob: leader
    jobIndex: 0
    podIndex: 0
  replicatedJobs:
    - name: leader
      replicas: 1
      template:
        spec:
          parallelism: 1
          completions: 1
          ...
    - name: workers
      replicas: 1
      template:
        spec:
          parallelism: 8
          completions: 8
          template:
            spec:
              containers:
              - name: worker
                env:
                - name: LEADER_ADDRESS
                  valueFrom:
                    fieldRef:
                      fieldPath: "metadata.labels['jobset.sigs.k8s.io/coordinator']"
            ...

```

--------------------------------

### List Headless Services for a JobSet

Source: https://jobset.sigs.k8s.io/docs/concepts

This command lists the headless services associated with a JobSet by filtering services that have the jobset-name label. It then uses xargs to get detailed service information.

```bash
kubectl get svc -o jsonpath='{.items[?(@.spec.selector.jobset\.sigs\.k8s\.io/jobset-name == "pytorch")].metadata.name}' | xargs kubectl get services

```

--------------------------------

### JobSet Controller Error Example

Source: https://jobset.sigs.k8s.io/docs/troubleshooting

This log message indicates a potential issue with resuming preempted workloads when using JobSet with Kueue. It suggests a permissions problem where the JobSet controller cannot update job statuses.

```text
ERROR   resuming jobset {"controller": "jobset", "controllerGroup": "jobset.x-k8s.io", "controllerKind": "JobSet", "JobSet": {"name":"js","namespace":"default"}, "namespace": "default", "name": "js", "reconcileID": "e1ab5e21-586c-496e-96b7-8629cd702f3b", "jobset": {"name":"js","namespace":"default"}, "error": "jobs.batch \"js-slice-job-1\" is forbidden: User \"system:serviceaccount:jobset-system:jobset-controller-manager\" cannot update resource \"jobs/status\" in API group \"batch\" in the namespace \"default\""}
```

--------------------------------

### Build and Deploy Jobset from Source

Source: https://jobset.sigs.k8s.io/docs/installation

Clone the Jobset repository, navigate to the directory, and build and push the Docker image, then deploy Jobset to your cluster. Replace `<registry>/<project>` with your container registry details.

```bash
git clone https://github.com/kubernetes-sigs/jobset.git
cd jobset
IMAGE_REGISTRY=<registry>/<project> make image-push deploy

```

--------------------------------

### Serve Documentation Site Locally

Source: https://jobset.sigs.k8s.io/docs/contribution_guidelines/development

Build and serve the JobSet documentation site locally using Hugo and Docsy. Access the provided URL in your browser to preview changes.

```bash
make site-serve
```

--------------------------------

### Run JobSet Tests

Source: https://jobset.sigs.k8s.io/docs/contribution_guidelines/development

Execute unit tests with 'make test' and integration tests with 'make test-integration'. These commands help verify the functionality of your changes.

```bash
make test
```

```bash
make test-integration
```

--------------------------------

### Build and Deploy JobSet

Source: https://jobset.sigs.k8s.io/docs/contribution_guidelines/development

Use make targets to undeploy existing JobSet instances and then deploy a new instance using the previously pushed image. Ensure environment variables match those used during image pushing.

```bash
make undeploy deploy
```

--------------------------------

### List Pod Hostnames and Subdomains for a JobSet

Source: https://jobset.sigs.k8s.io/docs/concepts

Retrieve the hostnames and subdomains for pods belonging to a JobSet using a custom column output and the jobset-name selector.

```bash
kubectl get pods -o custom-columns=Name:.metadata.name,HOSTNAME:.spec.hostname,Subdomain:.spec.subdomain --selector=jobset.sigs.k8s.io/jobset-name=pytorch

```

--------------------------------

### Uninstall Released Version with Helm

Source: https://jobset.sigs.k8s.io/docs/installation

Uninstalls Jobset from the cluster when it was installed using Helm. The command targets the 'jobset' release in the 'jobset-system' namespace.

```bash
helm uninstall jobset --namespace jobset-system
```

--------------------------------

### Enable Prometheus Metrics Scraping

Source: https://jobset.sigs.k8s.io/docs/installation

Run this command to allow prometheus-operator to scrape metrics from Jobset components. This step is optional and may not be needed if using kube-prometheus.

```bash
make prometheus

```

--------------------------------

### Exec into Pod and Test Network Connectivity

Source: https://jobset.sigs.k8s.io/docs/troubleshooting

Access a running pod using kubectl exec and check its /etc/hosts file for domain name resolution. Use ping to test network communication between pods using their domain names.

```bash
root@VM-0-4-ubuntu:/home/ubuntu# kubectl exec -it network-jobset-leader-0-0-5xnzz -- sh
/ # cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1	localhost
...
10.6.2.27	network-jobset-leader-0-0.example.default.svc.cluster.local	network-jobset-leader-0-0
/ # ping network-jobset-workers-0-0.example
PING network-jobset-workers-0-0.example (10.6.1.16): 56 data bytes
64 bytes from 10.6.1.16: seq=0 ttl=62 time=0.121 ms
64 bytes from 10.6.1.16: seq=1 ttl=62 time=0.093 ms
64 bytes from 10.6.1.16: seq=2 ttl=62 time=0.094 ms
64 bytes from 10.6.1.16: seq=3 ttl=62 time=0.103 ms
--- network-jobset-workers-0-0.example ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.093/0.102/0.121 ms
/ # ping network-jobset-workers-0-1.example
PING network-jobset-workers-0-1.example (10.6.2.28): 56 data bytes
64 bytes from 10.6.2.28: seq=0 ttl=63 time=0.068 ms
64 bytes from 10.6.2.28: seq=1 ttl=63 time=0.072 ms
64 bytes from 10.6.2.28: seq=2 ttl=63 time=0.079 ms
--- network-jobset-workers-0-1.example ping statistics ---


```

--------------------------------

### Apply Kustomization Configuration

Source: https://jobset.sigs.k8s.io/docs/installation

Applies the Kustomize configuration to the Kubernetes cluster using kubectl. This command should be run from the directory containing your kustomization.yaml.

```bash
kubectl apply -k .
```

--------------------------------

### Download Remote Manifest for Kustomize

Source: https://jobset.sigs.k8s.io/docs/installation

Downloads the remote manifest file for a specific version of Jobset. This file will be used by Kustomize.

```bash
VERSION=v0.11.1
curl -LO https://github.com/kubernetes-sigs/jobset/releases/download/$VERSION/manifests.yaml
```

--------------------------------

### Source .env file

Source: https://jobset.sigs.k8s.io/docs/contribution_guidelines/development

Load the environment variables defined in the .env file into your current shell session. This is necessary before running make image-push to use your custom registry and tag.

```bash
source .env
```

--------------------------------

### Kustomization File - Including Patch

Source: https://jobset.sigs.k8s.io/docs/installation

Includes both the base manifest and a resource patch file in the kustomization.yaml. This applies the defined resource adjustments.

```yaml
resources:
  - manifests.yaml
patches:
  - path: resource_patch.yaml
```

--------------------------------

### Add Metrics Scraping for Prometheus (Development Version)

Source: https://jobset.sigs.k8s.io/docs/installation

Applies the Prometheus configuration for the latest development version of Jobset. This allows prometheus-operator to scrape metrics.

```bash
kubectl apply --server-side -k github.com/kubernetes-sigs/jobset/config/prometheus?ref=main
```

--------------------------------

### Inspect JobSet Controller Logs

Source: https://jobset.sigs.k8s.io/docs/troubleshooting

Check the JobSet controller logs for errors related to job creation. Look for specific error messages like 'Index with name field:.metadata.controller does not exist' or validation errors.

```bash
kubectl get pods -n jobset-system
```

```bash
kubectl logs <pod> -n jobset-system
```

--------------------------------

### Set Environment Variables for Image Build

Source: https://jobset.sigs.k8s.io/docs/contribution_guidelines/development

Define GIT_TAG and IMAGE_REGISTRY environment variables in a .env file for custom image building and pushing. Ensure your cluster can access the specified image registry.

```bash
export GIT_TAG="dev-$(date +%Y-%m-%d-%H%M%S)"
export IMAGE_REGISTRY=<YOUR_IMAGE_REGISTRY_URL>

```

--------------------------------

### Coordinator Configuration

Source: https://jobset.sigs.k8s.io/docs/reference/jobset.v1alpha2

Defines how to identify the coordinator pod within a JobSet, specifying the replicated job, job index, and pod index.

```APIDOC
## Coordinator

**Appears in:**
  * JobSetSpec


Coordinator defines which pod can be marked as the coordinator for the JobSet workload.
Field| Description  
---|---  
`replicatedJob` **[Required]**  
`string`|  replicatedJob is the name of the ReplicatedJob which contains the coordinator pod.  
`jobIndex` **[Required]**  
`int`|  jobIndex is the index of Job which contains the coordinator pod (i.e., for a ReplicatedJob with N replicas, there are Job indexes 0 to N-1).  
`podIndex` **[Required]**  
`int`|  podIndex is the Job completion index of the coordinator pod.  
```

--------------------------------

### Check JobSet Controller Pods

Source: https://jobset.sigs.k8s.io/docs/troubleshooting

Use this command to check if the JobSet controller manager pods are running. If they are in a Pending state, describe the pod to identify the cause.

```bash
kubectl get pods -n jobset-system
```

```bash
kubectl describe pod <pod> -n jobset-system
```

--------------------------------

### Add Metrics Scraping for Prometheus Operator (Released Version)

Source: https://jobset.sigs.k8s.io/docs/installation

Applies the Prometheus configuration for released versions of Jobset to enable metrics scraping. This is optional if you are using kube-prometheus.

```bash
VERSION=v0.11.1
kubectl apply -f https://github.com/kubernetes-sigs/jobset/releases/download/$VERSION/prometheus.yaml
```

--------------------------------

### Create a PyTorch Distributed Training JobSet

Source: https://jobset.sigs.k8s.io/docs/concepts

This YAML defines a JobSet for a distributed PyTorch training workload. It specifies replicated jobs with templates for containers, environment variables, and commands.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: pytorch
spec:
  replicatedJobs:
    - name: workers
      template:
        spec:
          parallelism: 4
          completions: 4
          backoffLimit: 0
          template:
           spec:
            containers:
            - name: pytorch
              image: gcr.io/k8s-staging-jobset/pytorch-resnet:latest
              ports:
              - containerPort: 3389
              env:
              - name: MASTER_ADDR
                value: "pytorch-workers-0-0.pytorch"
              - name: MASTER_PORT
                value: "3389"
              command:
              - bash
              - -xc
              - |
                torchrun --nproc_per_node=1 --master_addr=$MASTER_ADDR --master_port=$MASTER_PORT resnet.py --backend=gloo


```

--------------------------------

### Configure RestartJob Action

Source: https://jobset.sigs.k8s.io/docs/tasks/failure_policy

This YAML configuration defines a JobSet with two replicated jobs, `rj-leader` and `rj-worker`. It specifies failure policy rules: `r_leader` uses `RestartJobSet` to recreate all jobs upon failure, while `r_worker` uses `RestartJob` to recreate only the failed job. Both actions count towards the `maxRestarts` limit of 2.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: js
spec:
  failurePolicy:
    maxRestarts: 2
    rules:
    # If any Job from replicatedJob `rj-leader` fails, restart the JobSet by recreating ALL Jobs
    # Count the restart towards `jobSet.spec.failurePolicy.maxRestarts`
    - name: r_leader
      action: RestartJobSet
      targetReplicatedJobs:
      - rj-leader
    # If any Job from replicatedJob `rj-worker` fails, restart the JobSet by recreating only the failed Job
    # Count the restart towards `jobSet.spec.failurePolicy.maxRestarts`
    - name: r_worker
      action: RestartJob
      targetReplicatedJobs:
      - rj-worker
  replicatedJobs:
  - name: rj-leader
    replicas: 1
    template:
      spec:
        parallelism: 2
        completions: 2
        backoffLimit: 0 # Job fails if any Pod fails
        template:
          spec:
            restartPolicy: Never
            terminationGracePeriodSeconds: 0 # Useful for testing
            containers:
            - name: leader
              image: busybox:1.37
              command: ["/bin/sh", "-c"]
              # Fail a leader Pod with `kubectl exec ${POD_NAME:?} -c leader -- touch /tmp/fail`
              args:
              - |
                echo "Start"
                while true; do
                  if [ -f /tmp/fail ]; then
                    echo "Exiting 1"
                    exit 1
                  elif [ -f /tmp/succeed ]; then
                    echo "Exiting 0"
                    exit 0
                  fi
                  sleep 1
                done
  - name: rj-worker
    replicas: 1
    template:
      spec:
        parallelism: 2
        completions: 2
        backoffLimit: 0 # Job fails if any Pod fails
        template:
          spec:
            restartPolicy: Never
            terminationGracePeriodSeconds: 0 # Useful for testing
            containers:
            - name: worker
              image: busybox:1.37
              command: ["/bin/sh", "-c"]
              # Fail a worker Pod with `kubectl exec ${POD_NAME:?} -c worker -- touch /tmp/fail`
              args:
              - |
                echo "Start"
                while true; do
                  if [ -f /tmp/fail ]; then
                    echo "Exiting 1"
                    exit 1
                  elif [ -f /tmp/succeed ]; then
                    echo "Exiting 0"
                    exit 0
                  fi
                  sleep 1
                done

```

--------------------------------

### Uninstall Development Version

Source: https://jobset.sigs.k8s.io/docs/installation

Uninstalls the Jobset development version from the cluster using kubectl and the default Kustomize configuration.

```bash
kubectl delete -k github.com/kubernetes-sigs/jobset/config/default
```

--------------------------------

### Handle BackoffLimitExceeded with Unlimited Restarts

Source: https://jobset.sigs.k8s.io/docs/tasks/failure_policy

Configures a JobSet to restart indefinitely when the 'leader' job fails due to exceeding its backoff limit. This is useful for critical jobs where temporary failures should not halt the entire JobSet.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: onjobfailurereasons-present-example
spec:
  failurePolicy:
    maxRestarts: 3
    rules:
      # The JobSet will restart an unlimited number of times when the
      # leader job fails with the failure reason BackoffLimitExceeded.
      - action: RestartJobSetAndIgnoreMaxRestarts 
        targetReplicatedJobs:
        - leader
        onJobFailureReasons:
        - BackoffLimitExceeded
  replicatedJobs:
  - name: leader
    replicas: 1
    template:
      spec:
        # Set backoff limit to 0 so job will immediately fail if any pod fails.
        backoffLimit: 0
        completions: 2
        parallelism: 2
        template:
          spec:
            containers:
            - name: leader
              image: bash:latest
              command:
              - bash
              - -xc
              - |
                echo "JOB_COMPLETION_INDEX=$JOB_COMPLETION_INDEX"
                if [[ "$JOB_COMPLETION_INDEX" == "0" ]]; then
                  for i in $(seq 10 -1 1)
                  do
                    echo "Sleeping in $i"
                    sleep 1
                  done
                  exit 1
                fi
                for i in $(seq 1 1000)
                do
                  echo "$i"
                  sleep 1
                done
  - name: workers
    replicas: 1
    template:
      spec:
        backoffLimit: 0
        completions: 2
        parallelism: 2
        template:
          spec:
            containers:
            - name: worker
              image: bash:latest
              command:
              - bash
              - -xc
              - |
                sleep 1000

```

--------------------------------

### List Jobs Belonging to a JobSet

Source: https://jobset.sigs.k8s.io/docs/concepts

Use this kubectl command to list all jobs associated with a specific JobSet, identified by its name label.

```bash
kubectl get jobs --selector=jobset.sigs.k8s.io/jobset-name=pytorch

```

--------------------------------

### Exclusive Job to Topology Placement

Source: https://jobset.sigs.k8s.io/docs/concepts

Use the `alpha.jobset.sigs.k8s.io/exclusive-topology` annotation to ensure a 1:1 mapping between jobs and a specific topology, such as a rack, optimizing network performance.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: pytorch
  annotations:
    alpha.jobset.sigs.k8s.io/exclusive-topology: rack
spec:
  replicatedJobs:
    - name: workers
      template:
        spec:
          ...

```

--------------------------------

### JobSetRestartStrategy

Source: https://jobset.sigs.k8s.io/docs/reference/jobset.v1alpha2

Represents the restart strategy for a JobSet, aliased as a string.

```APIDOC
## JobSetRestartStrategy

(Alias of `string`)

**Appears in:**
  * FailurePolicy
```

--------------------------------

### Uninstall Jobset

Source: https://jobset.sigs.k8s.io/docs/installation

Run this command to remove Jobset from your cluster.

```bash
make undeploy

```

--------------------------------

### RestartJobSetAndIgnoreMaxRestarts Action

Source: https://jobset.sigs.k8s.io/docs/tasks/failure_policy

Use this action to restart a specific job within a JobSet an unlimited number of times when it fails, without counting towards the overall maxRestarts limit. This is useful for critical jobs like a 'leader' that must eventually succeed.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  # rjimr stands for "restartjobsetandignoremaxrestarts"
  name: rjimr-action-example
spec:
  failurePolicy:
    maxRestarts: 3
    rules:
      # The JobSet will restart an unlimited number of times
      # when the leader job fails.
      - action: RestartJobSetAndIgnoreMaxRestarts
        targetReplicatedJobs:
        - leader
  replicatedJobs:
  - name: leader
    replicas: 1
    template:
      spec:
        # Set backoff limit to 0 so job will immediately fail if any pod fails.
        backoffLimit: 0
        completions: 2
        parallelism: 2
        template:
          spec:
            containers:
            - name: leader
              image: bash:latest
              command:
              - bash
              - -xc
              - |
                echo "JOB_COMPLETION_INDEX=$JOB_COMPLETION_INDEX"
                if [[ "$JOB_COMPLETION_INDEX" == "0" ]]; then
                  for i in $(seq 10 -1 1)
                  do
                    echo "Sleeping in $i"
                    sleep 1
                  done
                  exit 1
                fi
                for i in $(seq 1 1000)
                do
                  echo "$i"
                  sleep 1
                done
  - name: workers
    replicas: 1
    template:
      spec:
        backoffLimit: 0
        completions: 2
        parallelism: 2
        template:
          spec:
            containers:
            - name: worker
              image: bash:latest
              command:
              - bash
              - -xc
              - |
                sleep 1000

```

--------------------------------

### Kustomization File - Resource Patch

Source: https://jobset.sigs.k8s.io/docs/installation

Defines resource adjustments for the jobset-controller-manager deployment, such as modifying CPU and memory requests and limits.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jobset-controller-manager
  namespace: jobset-system
spec:
  template:
    spec:
      containers:
      - name: manager
        resources:
          requests:
            cpu: "1"      
            memory: "256Mi"  
          limits:
            cpu: "4"      
            memory: "1Gi"  
```

--------------------------------

### List Pods Belonging to a JobSet

Source: https://jobset.sigs.k8s.io/docs/concepts

This kubectl command retrieves all pods associated with a given JobSet, using the jobset-name selector.

```bash
kubectl get pods --selector=jobset.sigs.k8s.io/jobset-name=pytorch

```

--------------------------------

### JobSet Controller Transient Admission Error Log

Source: https://jobset.sigs.k8s.io/docs/troubleshooting

This log message indicates a temporary issue where the JobSet controller cannot find the leader pod, which is expected during initial startup or leader pod rescheduling. It does not necessarily indicate a persistent problem.

```log
2026-02-27T19:51:30Z	ERROR	admission	finding leader pod for follower	{"webhookGroup": "", "webhookKind": "Pod", "Pod": {"name":"","namespace":"default"}, "namespace": "default", "name": "", "resource": {"group":"","version":"v1","resource":"pods"}, "user": "system:serviceaccount:kube-system:job-controller", "requestID": "<request-id>", "error": "expected 1 leader pod (jobname-0-0)), but got 0. this is an expected, transient error"}


```

--------------------------------

### JobSetSpec

Source: https://jobset.sigs.k8s.io/docs/reference/jobset.v1alpha2

Defines the desired state of a JobSet, including replicated jobs, network configuration, success and failure policies, and other operational parameters.

```APIDOC
## JobSetSpec

JobSetSpec defines the desired state of JobSet

### Fields

*   `replicatedJobs` **[Required]** (`[]ReplicatedJob`): The group of jobs that will form the set.
*   `network` (`Network`): Defines the networking options for the jobset.
*   `successPolicy` **[Required]** (`SuccessPolicy`): Configures when to declare the JobSet as succeeded. The JobSet is always declared succeeded if all jobs in the set finished with status complete.
*   `failurePolicy` **[Required]** (`FailurePolicy`): Configures when to declare the JobSet as failed. The JobSet is always declared failed if any job in the set finished with status failed.
*   `startupPolicy` **[Required]** (`StartupPolicy`): Configures in what order jobs must be started. Deprecated: StartupPolicy is deprecated, please use the DependsOn API.
*   `suspend` **[Required]** (`bool`): Suspends all running child Jobs when set to true.
*   `coordinator` (`Coordinator`): Can be used to assign a specific pod as the coordinator for the JobSet. If defined, an annotation will be added to all Jobs and pods with coordinator pod, which contains the stable network endpoint where the coordinator pod can be reached. jobset.sigs.k8s.io/coordinator=.
*   `managedBy` (`string`): Used to indicate the controller or entity that manages a JobSet. The built-in JobSet controller reconciles JobSets which don't have this field at all or the field value is the reserved string `jobset.sigs.k8s.io/jobset-controller`, but skips reconciling JobSets with a custom value for this field. The value must be a valid domain-prefixed path (e.g. acme.io/foo) - all characters before the first "/" must be a valid subdomain as defined by RFC 1123. All characters trailing the first "/" must be valid HTTP Path characters as defined by RFC 3986. The value cannot exceed 63 characters. The field is immutable.
*   `ttlSecondsAfterFinished` (`int32`): Limits the lifetime of a JobSet that has finished execution (either Complete or Failed). If this field is set, TTLSecondsAfterFinished after the JobSet finishes, it is eligible to be automatically deleted. When the JobSet is being deleted, its lifecycle guarantees (e.g. finalizers) will be honored. If this field is unset, the JobSet won't be automatically deleted. If this field is set to zero, the JobSet becomes eligible to be deleted immediately after it finishes.
*   `volumeClaimPolicies` (`[]VolumeClaimPolicy`): A list of policies for persistent volume claims that pods are allowed to reference. JobSet controller automatically adds the required volume claims to the pod template. Every claim in this list must have at least one matching (by name) volumeMount in one container in the template.
```

--------------------------------

### JobSet Resource Type

Source: https://jobset.sigs.k8s.io/docs/reference/jobset.v1alpha2

Defines the JobSet resource, which is the primary object for managing batch workloads. It includes spec and status fields.

```APIDOC
## JobSet

**Appears in:**
JobSet is the Schema for the jobsets API

Field| Description  
---|---  
`apiVersion`  
string| `jobset.x-k8s.io/v1alpha2`  
`kind`  
string| `JobSet`  
`spec` **[Required]**  
`JobSetSpec`|  spec is the specification for jobset  
`status` **[Required]**  
`JobSetStatus`|  status is the status of the jobset  
```
Tokens:
Raw