### Install JobSet Python SDK via Setuptools

Source: https://github.com/kubernetes-sigs/jobset/blob/main/sdk/python/README.md

Install the package using Setuptools. You can install it for the current user or for all users.

```sh
python setup.py install --user
```

--------------------------------

### JobSet with InOrder Startup Policy

Source: https://github.com/kubernetes-sigs/jobset/blob/main/keps/104-StartupPolicy/README.md

Example JobSet configuration specifying 'InOrder' startup policy to ensure replicated jobs start sequentially.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: messagequeue-driver-worker
spec:
  startupPolicy:
     startupPolicyOrder: InOrder
  replicatedJobs:
  - name: messagequeue
    replicas: 1
    template:
      spec:
        # Set backoff limit to 0 so job will immediately fail if any pod fails.
        backoffLimit: 0 
        completions: 1
        parallelism: 1
        template:
          spec:
            containers:
            - name: messagequeue
              image: bash:latest
              command:
              - bash
              - -xc
              - |
                sleep 100
  - name: driver
    replicas: 2
    template:
      spec:
        backoffLimit: 0 
        completions: 2
        parallelism: 2
        template:
          spec:
            containers:
            - name: driver
              image: bash:latest
              command:
              - bash
              - -xc
              - |
                sleep 10
  - name: worker
    replicas: 2
    template:
      spec:
        backoffLimit: 0 
        completions: 2
        parallelism: 2
        template:
          spec:
            containers:
            - name: worker
              image: bash:latest
              command:
              - bash
              - -xc
              - |
                sleep 10

```

--------------------------------

### Run JobSet Go Client Example

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/static/examples/client-go/README.md

Command to execute the Go client example. Ensure the KUBE_CONFIG_FILEPATH environment variable is set to your kubeconfig file.

```bash
go run main.go --kubeconfig $KUBE_CONFIG_FILEPATH
```

--------------------------------

### Install JobSet Python SDK

Source: https://github.com/kubernetes-sigs/jobset/blob/main/sdk/python/examples/README.md

Install the JobSet Python SDK into a virtual environment using pip. Ensure you have Python 3 and pip installed.

```bash
python3 -m venv jobset-test
source jobset-test/bin/active
python3 -m pip install sdk/python/.
```

--------------------------------

### Install JobSet Chart from Registry

Source: https://github.com/kubernetes-sigs/jobset/blob/main/charts/jobset/README.md

Installs the JobSet Helm chart version 0.11.1 from the registry.k8s.io repository. A specific version is required as there is no 'latest' tag.

```shell
helm install [RELEASE_NAME] oci://registry.k8s.io/jobset/charts/jobset --version 0.11.1
```

--------------------------------

### Build and Install JobSet from Source

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/installation/_index.md

Build Jobset from source and install it in your cluster by cloning the repository, navigating into the directory, and running the make commands for image push and deployment.

```shell
git clone https://github.com/kubernetes-sigs/jobset.git
cd jobset
IMAGE_REGISTRY=<registry>/<project> make image-push deploy
```

--------------------------------

### Install JobSet Chart Locally

Source: https://github.com/kubernetes-sigs/jobset/blob/main/charts/jobset/README.md

Installs the JobSet Helm chart from a local path. Use this command to deploy the chart to your Kubernetes cluster.

```shell
helm install [RELEASE_NAME] charts/jobset
```

--------------------------------

### Install JobSet Chart with Namespace Creation

Source: https://github.com/kubernetes-sigs/jobset/blob/main/charts/jobset/README.md

Installs the JobSet Helm chart, creating the specified namespace if it does not already exist. This is useful for isolating JobSet resources.

```shell
helm install jobset charts/jobset \
    --namespace jobset-system \
    --create-namespace
```

--------------------------------

### JobSet Creation Output

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/static/examples/client-go/README.md

Expected output when the JobSet Go client example successfully creates a JobSet.

```text
successfully created JobSet: test-js
```

--------------------------------

### Install cert-manager

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/installation/_index.md

Install cert-manager on your cluster. This is a prerequisite for using cert-manager with JobSet webhooks.

```shell
VERSION=v1.11.0
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/$VERSION/cert-manager.yaml
```

--------------------------------

### Install Latest Development Version

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/installation/_index.md

Install the latest development version of Jobset in your cluster using the provided Kustomize command. The controller will be deployed in the `jobset-system` namespace.

```shell
kubectl apply --server-side -k github.com/kubernetes-sigs/jobset/config/default?ref=main
```

--------------------------------

### Install JobSet Python SDK via pip

Source: https://github.com/kubernetes-sigs/jobset/blob/main/sdk/python/README.md

Install the package directly from a Git repository using pip. You may need to run pip with root permissions.

```sh
pip install git+https://github.com/GIT_USER_ID/GIT_REPO_ID.git
```

--------------------------------

### Import JobSet Python SDK

Source: https://github.com/kubernetes-sigs/jobset/blob/main/sdk/python/README.md

Import the JobSet package after installation. This is a common first step for using the SDK.

```python
import jobset
```

--------------------------------

### Patch command for JobSet scaling

Source: https://github.com/kubernetes-sigs/jobset/blob/main/keps/463-ElasticJobsets/README.md

Example command to patch a JobSet to scale its parallelism and completions.

```bash
# kubectl patch jobset pytorch-elastic -p '{"spec":{"replicatedJobs":[{"replicas":1,"template":{"spec":{"parallelism":8,"completions":8}}}]}}'
```

--------------------------------

### StartupPolicy

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/reference/jobset.v1alpha2.md

Defines the startup order for ReplicatedJobs within a JobSet. It can be set to start jobs in any order or in the order they are listed.

```APIDOC
## `StartupPolicy`     {#jobset-x-k8s-io-v1alpha2-StartupPolicy}
    
**Appears in:**

- [JobSetSpec](#jobset-x-k8s-io-v1alpha2-JobSetSpec)

<table class="table">
<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
<tbody>
    
  
<tr><td><code>startupPolicyOrder</code> <B>[Required]</B><br/>
<a href="#jobset-x-k8s-io-v1alpha2-StartupPolicyOptions"><code>StartupPolicyOptions</code></a>
</td>
<td>
   <p>startupPolicyOrder determines the startup order of the ReplicatedJobs. AnyOrder means to start replicated jobs in any order. InOrder means to start them as they are listed in the JobSet. A ReplicatedJob is started only when all the jobs of the previous one are ready.</p>
</td>
</tr>
</tbody>
</table>
```

--------------------------------

### RestartJobSet Action Example

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/tasks/failure_policy.md

Configure the JobSet to restart up to 3 times if the 'leader' job fails. This is the default action if no other rules match.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha1
kind: JobSet
metadata:
  name: restartjobset-example
spec:
  failurePolicy:
    maxRestarts: 3
    rules:
      - onFailure: RestartJobSet
        targetReplicatedJobs: ["leader"]
  replicatedJobs:
    - name: leader
      template:
        spec:
          template:
            spec:
              containers:
                - name: container
                  image: busybox
                  command: ["sh", "-c", "sleep 10 && exit 1"]
      replicas: 1
    - name: worker
      template:
        spec:
          template:
            spec:
              containers:
                - name: container
                  image: busybox
                  command: ["sh", "-c", "sleep 10"]
      replicas: 3

```

--------------------------------

### StartupPolicy Condition Example

Source: https://github.com/kubernetes-sigs/jobset/blob/main/keps/104-StartupPolicy/README.md

This condition indicates that the startup policy is currently running. A 'false' status means the policy is active, and a 'true' status means all jobs have at least reached the ready state. The 'Reason' field specifies the startup policy type, and the 'Message' field identifies the specific job being processed.

```go
metav1.Condition{
   Type:    "JobSetStartupPolicyCompleted",
   Status:  metav1.ConditionStatus(corev1.ConditionFalse),
   Reason:  "StartupPolicyInOrder",
   Message: "replicated job %s is starting",
  })
```

--------------------------------

### Targeting Job Failure Reasons Example

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/tasks/failure_policy.md

Configure unlimited restarts for the 'leader' job only when its backoffLimit is exceeded. This allows specific handling of different failure types.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha1
kind: JobSet
metadata:
  name: onjobfailurereasons-example
spec:
  failurePolicy:
    maxRestarts: 3
    rules:
      - onFailure: RestartJobSetAndIgnoreMaxRestarts
        targetReplicatedJobs: ["leader"]
        onJobFailureReasons: ["BackoffLimitExceeded"]
  replicatedJobs:
    - name: leader
      template:
        spec:
          template:
            spec:
              containers:
                - name: container
                  image: busybox
                  command: ["sh", "-c", "sleep 10 && exit 1"]
      replicas: 1
    - name: worker
      template:
        spec:
          template:
            spec:
              containers:
                - name: container
                  image: busybox
                  command: ["sh", "-c", "sleep 10"]
      replicas: 3

```

--------------------------------

### RestartJob Action Example

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/tasks/failure_policy.md

Configure individual job restarts for failures in 'leader' and 'worker' jobs, up to 3 times. Failures in 'leader' jobs recreate all jobs, while 'worker' job failures only recreate the failed worker.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha1
kind: JobSet
metadata:
  name: restartjob-example
spec:
  failurePolicy:
    maxRestarts: 3
    rules:
      - onFailure: RestartJob
        targetReplicatedJobs: ["leader"]
      - onFailure: RestartJob
        targetReplicatedJobs: ["worker"]
  replicatedJobs:
    - name: leader
      template:
        spec:
          template:
            spec:
              containers:
                - name: container
                  image: busybox
                  command: ["sh", "-c", "sleep 10 && exit 1"]
      replicas: 1
    - name: worker
      template:
        spec:
          template:
            spec:
              containers:
                - name: container
                  image: busybox
                  command: ["sh", "-c", "sleep 10 && exit 1"]
      replicas: 3

```

--------------------------------

### FailJobSet Action Example

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/tasks/failure_policy.md

Configure the JobSet to fail immediately if the 'leader' job fails. This action stops the entire JobSet.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha1
kind: JobSet
metadata:
  name: failjobset-example
spec:
  failurePolicy:
    rules:
      - onFailure: FailJobSet
        targetReplicatedJobs: ["leader"]
  replicatedJobs:
    - name: leader
      template:
        spec:
          template:
            spec:
              containers:
                - name: container
                  image: busybox
                  command: ["sh", "-c", "sleep 10 && exit 1"]
      replicas: 1
    - name: worker
      template:
        spec:
          template:
            spec:
              containers:
                - name: container
                  image: busybox
                  command: ["sh", "-c", "sleep 10"]
      replicas: 3

```

--------------------------------

### Basic Volume Claim Policy Example

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/tasks/volume_claim_policies.md

Demonstrates creating shared PVCs with different retention policies. The PVCs are automatically created with the naming convention: `<claim-name>-<jobset-name>`. The 'initializer-volume' is deleted when the JobSet is deleted, while 'checkpoints-volume' is retained.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha1
kind: JobSet
metadata:
  name: trainjob
spec:
  replicatedJobs:
    - name: initializer
      template:
        spec:
          containers:
            - name: main
              image: alpine
              command: ["sh", "-c", "echo 'Downloading model...' && sleep 5 && echo 'Model downloaded.'"]
              volumeMounts:
                - name: initializer-volume
                  mountPath: /data
    - name: node
      template:
        spec:
          containers:
            - name: main
              image: alpine
              command: ["sh", "-c", "echo 'Reading model and writing checkpoints...' && sleep 10 && echo 'Checkpoints written.'"]
              volumeMounts:
                - name: checkpoints-volume
                  mountPath: /checkpoints
  volumeClaimPolicies:
    - name: initializer-volume
      templates:
        - metadata:
            name: initializer-volume
          spec:
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 10Gi
      retentionPolicy:
        whenDeleted: Delete
    - name: checkpoints-volume
      templates:
        - metadata:
            name: checkpoints-volume
          spec:
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 50Gi
      retentionPolicy:
        whenDeleted: Retain
```

--------------------------------

### Install Prometheus CR for JobSet Metrics

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/static/examples/prometheus-operator/README.md

Applies the Prometheus Custom Resource definition from `prometheus.yaml`. This configures Prometheus to scrape metrics from JobSet.

```bash
kubectl apply -f prometheus.yaml
serviceaccount/prometheus-jobset created
clusterrole.rbac.authorization.k8s.io/prometheus-jobset created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-jobset created
prometheus.monitoring.coreos.com/jobset-metrics created
service/jobset-metrics created
```

--------------------------------

### JobSet with Startup Order Policy

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/overview/_index.md

Configures a startup order for ReplicatedJobs within a JobSet, enabling patterns like leader-worker where specific jobs must start before others.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha1
kind: JobSet
metadata:
  name: jobset-startup-order
spec:
  startupPolicy:
    orderedReplicatedJobs:
      - name: leader
      - name: worker
  replicatedJobs:
    - name: leader
      template:
        spec:
          template:
            spec:
              containers:
                - name: leader
                  image: "busybox"
                  command: ["sleep", "3600"]
      replicas: 1
    - name: worker
      template:
        spec:
          template:
            spec:
              containers:
                - name: worker
                  image: "busybox"
                  command: ["sleep", "3600"]
      replicas: 3
```

--------------------------------

### JobSet with Driver-Ready Worker Start Policy

Source: https://github.com/kubernetes-sigs/jobset/blob/main/keps/104-StartupPolicy/README.md

This YAML configuration demonstrates a JobSet with a StartupPolicy set to 'InOrder'. It defines a 'driver' ReplicatedJob that must be ready before the 'workers' ReplicatedJob is initiated. This is useful for scenarios where a driver component needs to be operational before worker processes begin.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: driver-ready-worker-start
spec:
  startupPolicy:
     startupPolicyOrder: InOrder
  replicatedJobs:
  - name: driver
    replicas: 1
    template:
      spec:
        # Set backoff limit to 0 so job will immediately fail if any pod fails.
        backoffLimit: 0 
        completions: 1
        parallelism: 1
        template:
          spec:
            containers:
            - name: driver
              image: bash:latest
              command:
              - bash
              - -xc
              - |
                sleep 10000
  - name: workers
    replicas: 1
    template:
      spec:
        backoffLimit: 0 
        completions: 2
        parallelism: 2
        template:
          spec:
            containers:
            - name: worker
              image: bash:latest
              command:
              - bash
              - -xc
              - |
                sleep 10
```

--------------------------------

### Set Up Kustomization Environment

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/installation/_index.md

Creates a directory for Kustomize configuration and navigates into it.

```shell
mkdir kustomize-jobset
cd kustomize-jobset
```

--------------------------------

### Go Structs for Startup Policy Options

Source: https://github.com/kubernetes-sigs/jobset/blob/main/keps/104-StartupPolicy/README.md

Defines the possible options for the startup policy, including 'AnyOrder' and 'InOrder'. 'AnyOrder' is the default.

```go
type StartupPolicyOptions string

const (
 // This is the default setting
 // AnyOrder means that we will start jobs without any specific order.
 AnyOrder StartupPolicyOptions = "AnyOrder"
 // InOrder starts the ReplicatedJobs in order that they are listed. Jobs within a ReplicatedJob will 
 // still start in any order.
 InOrder StartupPolicyOptions = "InOrder"
)

type StartupPolicy struct {
 // StartupPolicyOrder is an enum of different options for StartupPolicy
 // AnyOrder means to start replicated jobs in any order.
 // InOrder means to start them as they are listed in the JobSet.
 // +kubebuilder:validation:Enum=AnyOrder;InOrder
 StartupPolicyOrder StartupPolicyOptions `json:"startupPolicyOrder"`
}

```

--------------------------------

### Install CRDs

Source: https://github.com/kubernetes-sigs/jobset/blob/main/CLAUDE.md

Installs only the Custom Resource Definitions (CRDs) for JobSet into the Kubernetes cluster. This is a prerequisite for deploying the controller.

```bash
make install
```

--------------------------------

### Install Released Jobset Version using Helm

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/installation/_index.md

Installs a released version of Jobset into your Kubernetes cluster using Helm.

```shell
helm install jobset oci://registry.k8s.io/jobset/charts/jobset --version ${VERSION#v} --create-namespace --namespace=jobset-system
```

--------------------------------

### Apply JobSet and Check Pods/Services

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/troubleshooting/_index.md

Deploy a JobSet with network configuration and verify the status of created pods and services. This helps in initial network troubleshooting.

```bash
root@VM-0-4-ubuntu:/home/ubuntu# vi jobset-network.yaml
root@VM-0-4-ubuntu:/home/ubuntu# kubectl apply -f jobset-network.yaml
jobset.jobset.x-k8s.io/network-jobset created
root@VM-0-4-ubuntu:/home/ubuntu# kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE   IP          NODE               NOMINATED NODE   READINESS GATES
network-jobset-leader-0-0-5xnzz    1/1     Running   0          17m   10.6.2.27   cluster1-worker    <none>           <none>
network-jobset-workers-0-0-78k9j   1/1     Running   0          17m   10.6.1.16   cluster1-worker2   <none>           <none>
network-jobset-workers-0-1-rmw42   1/1     Running   0          17m   10.6.2.28   cluster1-worker    <none>           <none>
root@VM-0-4-ubuntu:/home/ubuntu# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
example      ClusterIP   None         <none>        <none>    19s
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   2d1h
```

--------------------------------

### Install Released Jobset Version using kubectl

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/installation/_index.md

Installs a released version of Jobset into your Kubernetes cluster using kubectl apply.

```shell
VERSION={{< param "version" >}}
kubectl apply --server-side -f https://github.com/kubernetes-sigs/jobset/releases/download/$VERSION/manifests.yaml
```

--------------------------------

### Programmatic JobSet Creation with Go Client

Source: https://context7.com/kubernetes-sigs/jobset/llms.txt

This Go program demonstrates how to create and manage JobSets programmatically using the JobSet Go client. It includes creating a JobSet and listing existing JobSets. Ensure your Kubernetes configuration is accessible via the KUBECONFIG environment variable or default locations.

```go
package main

import (
    "context"
    "flag"
    "fmt"

    _ "k8s.io/client-go/plugin/pkg/client/auth"
    "k8s.io/client-go/tools/clientcmd"
    "k8s.io/utils/ptr"

    batchv1 "k8s.io/api/batch/v1"
    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

    jobsetv1alpha2 "sigs.k8s.io/jobset/api/jobset/v1alpha2"
    jobsetclient "sigs.k8s.io/jobset/client-go/clientset/versioned"
)

var kubeconfig string

func init() {
    flag.StringVar(&kubeconfig, "kubeconfig", "", "path to Kubernetes config file")
    flag.Parse()
}

func main() {
    config, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
    if err != nil {
        panic(err)
    }

    ctx := context.Background()
    client := jobsetclient.NewForConfigOrDie(config)

    js, err := client.JobsetV1alpha2().JobSets("default").Create(ctx, &jobsetv1alpha2.JobSet{
        ObjectMeta: metav1.ObjectMeta{
            Name: "test-js",
        },
        Spec: jobsetv1alpha2.JobSetSpec{
            ReplicatedJobs: []jobsetv1alpha2.ReplicatedJob{
                {
                    Name: "rjob",
                    Template: batchv1.JobTemplateSpec{
                        ObjectMeta: metav1.ObjectMeta{Name: "job"},
                        Spec: batchv1.JobSpec{
                            Parallelism:  ptr.To(int32(2)),
                            Completions:  ptr.To(int32(2)),
                            BackoffLimit: ptr.To(int32(0)),
                            Template: corev1.PodTemplateSpec{
                                Spec: corev1.PodSpec{
                                    Containers: []corev1.Container{
                                        {
                                            Name:    "bash-container",
                                            Image:   "bash:latest",
                                            Command: []string{"sleep", "60"},
                                        },
                                    },
                                },
                            },
                        },
                    },
                },
            },
        },
    }, metav1.CreateOptions{})
    if err != nil {
        panic(err)
    }

    fmt.Printf("successfully created JobSet: %s\n", js.Name)

    // List all JobSets
    list, err := client.JobsetV1alpha2().JobSets("default").List(ctx, metav1.ListOptions{})
    if err != nil {
        panic(err)
    }
    for _, item := range list.Items {
        fmt.Printf("JobSet: %s, TerminalState: %s, Restarts: %d\n",
            item.Name, item.Status.TerminalState, item.Status.Restarts)
    }
}
```

```bash
# Build and run
go mod tidy
go run main.go --kubeconfig=$HOME/.kube/config
# Output: successfully created JobSet: test-js
```

--------------------------------

### Install JobSet CRD and Controller

Source: https://context7.com/kubernetes-sigs/jobset/llms.txt

Installs the JobSet Custom Resource Definition and controller on a Kubernetes cluster. Ensure your cluster version is 1.27 or higher.

```bash
# Install the latest release
kubectl apply --server-side -f https://github.com/kubernetes-sigs/jobset/releases/download/v0.11.1/manifests.yaml

# Verify the controller is running
kubectl get pods -n jobset-system

# Check installed CRDs
kubectl get crd jobsets.jobset.x-k8s.io
```

--------------------------------

### Serve Documentation Site Locally

Source: https://github.com/kubernetes-sigs/jobset/blob/main/CLAUDE.md

Serves the JobSet documentation site locally using Hugo and Docsy. This allows for previewing changes before building the static site.

```bash
make site-serve
```

--------------------------------

### Kustomization Configuration

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/installation/_index.md

A basic kustomization.yaml file that references a downloaded manifest.

```yaml
resources:
  - manifests.yaml
```

--------------------------------

### Build Documentation Site

Source: https://github.com/kubernetes-sigs/jobset/blob/main/CLAUDE.md

Builds the static HTML files for the JobSet documentation site. The output is typically placed in a 'public' directory.

```bash
make site-build
```

--------------------------------

### Build and Push Image

Source: https://github.com/kubernetes-sigs/jobset/blob/main/CLAUDE.md

Builds the JobSet controller's Docker image and pushes it to a container registry. Requires registry access and authentication.

```bash
make image-push
```

--------------------------------

### Install Latest JobSet Release

Source: https://github.com/kubernetes-sigs/jobset/blob/main/README.md

Installs the latest release of JobSet into your Kubernetes cluster. Ensure your cluster is running one of the last 3 Kubernetes minor versions. The controller will be deployed in the 'jobset-system' namespace.

```shell
kubectl apply --server-side -f https://github.com/kubernetes-sigs/jobset/releases/download/v0.11.1/manifests.yaml
```

--------------------------------

### TensorFlow Distributed JobSet Configuration

Source: https://github.com/kubernetes-sigs/jobset/blob/main/keps/672-serial-job-execution/README.md

Defines a JobSet for a TensorFlow distributed LLM fine-tuning workflow. It includes an initializer job, two parameter server jobs (ps-a, ps-b), and a trainer job, with dependencies ensuring parameter servers start after initialization and the trainer starts after parameter servers are ready.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: tensorflow-distributed
spec:
  replicatedJobs:
    - name: initializer
    - name: ps-a
      dependsOn:
        - name: initializer
          status: Complete
    - name: ps-b
      dependsOn:
        - name: initializer
          status: Complete
    - name: trainer-node
      dependsOn:
        - name: ps-a
          status: Ready
        - name: ps-b
          status: Ready

```

--------------------------------

### Build Kind Image

Source: https://github.com/kubernetes-sigs/jobset/blob/main/CLAUDE.md

Builds a Docker image for the JobSet controller specifically for use with Kind (local Kubernetes cluster).

```bash
make kind-image-build
```

--------------------------------

### Include Patch in Kustomization

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/installation/_index.md

Add the resource_patch.yaml file to your kustomization.yaml to apply the patch. This is part of the installation process.

```yaml
resources:
  - manifests.yaml
patches:
  - path: resource_patch.yaml
```

--------------------------------

### Deploy Controller

Source: https://github.com/kubernetes-sigs/jobset/blob/main/CLAUDE.md

Deploys the JobSet controller to the Kubernetes cluster. This command assumes CRDs are already installed.

```bash
make deploy
```

--------------------------------

### JobSet with Custom Network Configuration

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/overview/_index.md

Demonstrates how to configure a custom subdomain for a JobSet's network configuration to establish stable pod hostnames and enable pod-to-pod communication.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha1
kind: JobSet
metadata:
  name: jobset-with-network
spec:
  network:
    subdomain: "my-custom-subdomain"
  replicatedJobs:
    - name: worker
      template:
        spec:
          template:
            spec:
              containers:
                - name: worker
                  image: "busybox"
                  command: ["sleep", "3600"]
      replicas: 3
```

--------------------------------

### Uninstall JobSet (Make)

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/installation/_index.md

To uninstall JobSet after building and installing from source using make, run the undeploy command.

```shell
make undeploy
```

--------------------------------

### Kueue scaling scenarios

Source: https://github.com/kubernetes-sigs/jobset/blob/main/keps/463-ElasticJobsets/README.md

Illustrates how Kueue scales parallelism based on cluster GPU availability.

```text
Cluster 100 GPUs @ 80% → Kueue scales: parallelism 20→25
Cluster 100 GPUs @ 95% → Kueue scales: parallelism 25→20
```

--------------------------------

### Build Multi-Platform Image

Source: https://github.com/kubernetes-sigs/jobset/blob/main/CLAUDE.md

Builds a multi-platform Docker image for the JobSet controller, suitable for different architectures.

```bash
make image-build
```

--------------------------------

### Verify Prometheus Operator Pod Status

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/static/examples/prometheus-operator/README.md

Check if the Prometheus Operator pod is running after installation. This confirms the operator is active.

```bash
kubectl get pods
NAME                                   READY   STATUS    RESTARTS   AGE
prometheus-operator-76469b7f8c-5wb8x   1/1     Running   0          12h
```

--------------------------------

### Format Code

Source: https://github.com/kubernetes-sigs/jobset/blob/main/CLAUDE.md

Formats the Go code according to project standards using 'go fmt'. This command should be run before committing changes.

```bash
make fmt
```

--------------------------------

### Uninstall JobSet (Helm)

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/installation/_index.md

To uninstall JobSet using Helm, run the following command. This assumes JobSet was installed using Helm.

```shell
helm uninstall jobset --namespace jobset-system
```

--------------------------------

### JobSet Restart Strategies

Source: https://github.com/kubernetes-sigs/jobset/blob/main/keps/262-ConfigurableFailurePolicy/README.md

Defines the restart strategies for a JobSet. Use 'Recreate' to restart by recreating all Jobs as soon as the previous iteration is deleted, or 'BlockingRecreate' to ensure all previous Jobs are deleted before creating new ones.

```go
type JobSetRestartStrategy string

const (
	// Restart the JobSet by recreating all Jobs. Each Job is recreated as soon
	// as its previous iteration (and its Pods) is deleted.
	Recreate JobSetRestartStrategy = "Recreate"

	// Restart the JobSet by recreating all Jobs. Ensures that all Jobs (and
	// their Pods) from the previous iteration are deleted before creating new
	// Jobs.
	BlockingRecreate JobSetRestartStrategy = "BlockingRecreate"
)
```

--------------------------------

### Uninstall JobSet (Kustomize)

Source: https://github.com/kubernetes-sigs/jobset/blob/main/site/content/en/docs/installation/_index.md

To uninstall JobSet installed via Kustomize, use the following command. This command removes the JobSet resources from your cluster.

```shell
kubectl delete -k github.com/kubernetes-sigs/jobset/config/default
```

--------------------------------

### Basic JobSet: Parallel Jobs

Source: https://context7.com/kubernetes-sigs/jobset/llms.txt

Defines a JobSet with two ReplicatedJobs, 'driver' and 'workers', that start in parallel. The JobSet completes when all defined Jobs finish.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: paralleljobs
spec:
  replicatedJobs:
  - name: workers
    template:
      spec:
        parallelism: 4
        completions: 4
        backoffLimit: 0
        template:
          spec:
            containers:
            - name: sleep
              image: busybox
              command: [sleep]
              args: ["100s"]
  - name: driver
    template:
      spec:
        parallelism: 1
        completions: 1
        backoffLimit: 0
        template:
          spec:
            containers:
            - name: sleep
              image: busybox
              command: [sleep]
              args: ["100s"]
```

```bash
kubectl apply -f paralleljobs.yaml
kubectl get jobsets
# NAME           TERMINALSTATE   RESTARTS   COMPLETED   SUSPENDED   AGE
# paralleljobs                   0                      false       10s

kubectl get jobs
# NAME                    COMPLETIONS   DURATION   AGE
# paralleljobs-workers-0  0/4                      10s
# paralleljobs-driver-0   0/1                      10s
```

--------------------------------

### Build Manager Binary

Source: https://github.com/kubernetes-sigs/jobset/blob/main/CLAUDE.md

Builds the manager binary for the JobSet controller. This is a fundamental step for local development and deployment.

```bash
make build
```

--------------------------------

### Fan-In Dependencies with Multiple DependsOn

Source: https://context7.com/kubernetes-sigs/jobset/llms.txt

A ReplicatedJob can be configured to wait for multiple predecessor ReplicatedJobs to complete before it starts. This is achieved by listing multiple dependencies in the `dependsOn` field.

```yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: multiple-depends-on
spec:
  replicatedJobs:
  - name: dataset-initializer
    template:
      spec:
        template:
          spec:
            containers:
            - name: initializer
              image: busybox
              command: [/bin/sh, -c, "echo 'downloading dataset' && sleep 10"]

  - name: model-initializer
    template:
      spec:
        template:
          spec:
            containers:
            - name: initializer
              image: busybox
              command: [/bin/sh, -c, "echo 'downloading model' && sleep 10"]

  - name: worker                      # Waits for BOTH initializers to complete
    dependsOn:
    - name: dataset-initializer
      status: Complete
    - name: model-initializer
      status: Complete
    template:
      spec:
        parallelism: 4
        completions: 4
        template:
          spec:
            containers:
            - name: worker
              image: busybox
              command: [/bin/sh, -c, "echo 'training' && sleep 20"]

```