Kueue

https://kueue.sigs.k8s.io/docs/
Admin
Kueue is a Kubernetes-native system that manages quotas and how jobs consume them, handling job...
Tokens:124,287
Snippets:1,460
Trust Score:9.4
Update:2 months ago
Show doc for...
Tokens:
Raw
### Expected installation output

Source: https://kueue.sigs.k8s.io/docs/tasks/run/python_jobs

Example console output after running the installation script.

```text
⭐️ Installing Kueue...
⭐️ Applying queues from single-clusterqueue-setup.yaml...
```

--------------------------------

### List Workload Example

Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_list/kueuectl_list_workload

A basic example of how to list Kueue workloads.

```bash
  # List Workload
  kueuectl list kueueworkload
```

--------------------------------

### List ClusterQueue Example

Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_list/kueuectl_list_clusterqueue

A simple example showing how to list all ClusterQueues.

```bash
# List ClusterQueue
kueuectl list clusterqueue
```

--------------------------------

### Install via Krew

Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/installation

Use the Krew plugin manager to install kubectl-kueue.

```bash
kubectl krew install kueue
```

--------------------------------

### kueuectl create clusterqueue examples

Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_create/kueuectl_create_clusterqueue

Various examples demonstrating how to create ClusterQueues with different configurations, including simple creation, cohort assignment, and complex resource quota definitions.

```bash
# Create a ClusterQueue
kueuectl create clusterqueue my-cluster-queue

# Create a ClusterQueue with cohort, namespace selector and other details
kueuectl create clusterqueue my-cluster-queue \
--cohort cohortname \
--queuing-strategy StrictFIFO \
--namespace-selector fooX=barX,fooY=barY \
--reclaim-within-cohort Any \
--preemption-within-cluster-queue LowerPriority

# Create a ClusterQueue with nominal quota and one resource flavor named alpha
kueuectl create clusterqueue my-cluster-queue --nominal-quota "alpha:cpu=9;memory=36Gi"

# Create a ClusterQueue with multiple resource flavors named alpha and beta
kueuectl create clusterqueue my-cluster-queue \
--cohort cohortname \
--nominal-quota "alpha:cpu=9;memory=36Gi;nvidia.com/gpu=10,beta:cpu=18;memory=72Gi;nvidia.com/gpu=20" \
--borrowing-limit "alpha:cpu=1;memory=1Gi;nvidia.com/gpu=1,beta:cpu=2;memory=2Gi;nvidia.com/gpu=2" \
--lending-limit "alpha:cpu=1;memory=1Gi;nvidia.com/gpu=1,beta:cpu=2;memory=2Gi;nvidia.com/gpu=2"
```

--------------------------------

### Pod Status Output Examples

Source: https://kueue.sigs.k8s.io/docs/tasks/manage/setup_wait_for_pods_ready

Example outputs showing pod states during job startup and subsequent resource assignment.

```text
NAME            READY   STATUS              RESTARTS   AGE
job1-0-gc284    0/1     ContainerCreating   0          1s
job1-1-xz555    0/1     ContainerCreating   0          1s
job1-10-2ltws   0/1     Pending             0          1s
job1-11-r4778   0/1     ContainerCreating   0          1s
job1-12-xx8mn   0/1     Pending             0          1s
job1-13-glb8j   0/1     Pending             0          1s
job1-14-gnjpg   0/1     Pending             0          1s
job1-15-dzlqh   0/1     Pending             0          1s
job1-16-ljnj9   0/1     Pending             0          1s
job1-17-78tzv   0/1     Pending             0          1s
job1-18-4lhw2   0/1     Pending             0          1s
job1-19-hx6zv   0/1     Pending             0          1s
job1-2-hqlc6    0/1     ContainerCreating   0          1s
job1-3-zx55w    0/1     ContainerCreating   0          1s
job1-4-k2tb4    0/1     Pending             0          1s
job1-5-2zcw2    0/1     ContainerCreating   0          1s
job1-6-m2qzw    0/1     ContainerCreating   0          1s
job1-7-hgp9n    0/1     ContainerCreating   0          1s
job1-8-ss248    0/1     ContainerCreating   0          1s
job1-9-nwqmj    0/1     ContainerCreating   0          1s
```

```text
NAME            READY   STATUS      RESTARTS   AGE
job1-0-gc284    1/1     Running     0          9s
job1-1-xz555    1/1     Running     0          9s
job1-10-2ltws   1/1     Running     0          9s
job1-11-r4778   1/1     Running     0          9s
job1-12-xx8mn   1/1     Running     0          9s
job1-13-glb8j   1/1     Running     0          9s
job1-14-gnjpg   1/1     Running     0          9s
job1-15-dzlqh   1/1     Running     0          9s
job1-16-ljnj9   1/1     Running     0          9s
job1-17-78tzv   1/1     Running     0          9s
job1-18-4lhw2   1/1     Running     0          9s
job1-19-hx6zv   1/1     Running     0          9s
job1-2-hqlc6    1/1     Running     0          9s
job1-3-zx55w    1/1     Running     0          9s
job1-4-k2tb4    1/1     Running     0          9s
job1-5-2zcw2    1/1     Running     0          9s
job1-6-m2qzw    1/1     Running     0          9s
job1-7-hgp9n    1/1     Running     0          9s
job1-8-ss248    1/1     Running     0          9s
job1-9-nwqmj    1/1     Running     0          9s
job2-0-djnjd    1/1     Running     0          3s
job2-1-trw7b    0/1     Pending     0          2s
job2-10-228cc   0/1     Pending     0          2s
job2-11-2ct8m   0/1     Pending     0          2s
job2-12-sxkqm   0/1     Pending     0          2s
job2-13-md92n   0/1     Pending     0          2s
job2-14-4v2ww   0/1     Pending     0          2s
job2-15-sph8h   0/1     Pending     0          2s
job2-16-2nvk2   0/1     Pending     0          2s
job2-17-f7g6z   0/1     Pending     0          2s
job2-18-9t9xd   0/1     Pending     0          2s
job2-19-tgf5c   0/1     Pending     0          2s
job2-2-9hcsd    0/1     Pending     0          2s
job2-3-557lt    0/1     Pending     0          2s
job2-4-k2d6b    0/1     Pending     0          2s
job2-5-nkkhx    0/1     Pending     0          2s
job2-6-5r76n    0/1     Pending     0          2s
job2-7-pmzb5    0/1     Pending     0          2s
job2-8-xdqtp    0/1     Pending     0          2s
job2-9-c4rcl    0/1     Pending     0          2s
```

--------------------------------

### Setup ClusterQueue and LocalQueue Resources

Source: https://kueue.sigs.k8s.io/docs/tasks/manage/monitor_pending_workloads/pending_workloads_in_grafana

Example manifest to define a ResourceFlavor, ClusterQueue, and LocalQueue.

```yaml
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: "default-flavor"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: "cluster-queue"
spec:
  namespaceSelector: {} # match all.
  resourceGroups:
  - coveredResources: ["cpu", "memory"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 9
      - name: "memory"
        nominalQuota: 36Gi
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  namespace: "default"
  name: "user-queue"
spec:
  clusterQueue: "cluster-queue"
```

--------------------------------

### Install From Source

Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/installation

Build the binary from the source repository and move it to the system path.

```bash
make kueuectl
sudo mv ./bin/kubectl-kueue /usr/local/bin/kubectl-kueue
```

--------------------------------

### Run Kueue installation script

Source: https://kueue.sigs.k8s.io/docs/tasks/run/python_jobs

Commands to execute the installation script using uv, including an optional version flag.

```bash
uv run install-kueue-queues.py
```

```bash
uv run install-kueue-queues.py --version v0.19.0
```

--------------------------------

### Apply ClusterQueue Example

Source: https://kueue.sigs.k8s.io/docs/tasks/manage/monitor_pending_workloads/pending_workloads_in_grafana

Command to apply the example ClusterQueue configuration from the official documentation URL.

```bash
kubectl apply -f https://kueue.sigs.k8s.io/examples/admin/single-clusterqueue-setup.yaml
```

--------------------------------

### Examples of creating local queues

Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_create/kueuectl_create_localqueue

Common usage examples for creating a standard local queue and one that ignores unknown cluster queues.

```bash
# Create a local queue
kueuectl create localqueue my-local-queue -c my-cluster-queue

# Create a local queue with unknown cluster queue
kueuectl create localqueue my-local-queue -c my-cluster-queue -i
```

--------------------------------

### Automated MultiKueue TAS Setup

Source: https://kueue.sigs.k8s.io/docs/tasks/dev/setup_multikueue_development_environment

Clones the repository and executes the setup script to configure a MultiKueue environment with Topology-Aware Scheduling.

```bash
git clone https://github.com/kubernetes-sigs/kueue.git
cd kueue/examples/multikueue/dev
./setup-kind-multikueue-tas.sh
```

--------------------------------

### Workload status example

Source: https://kueue.sigs.k8s.io/docs/tasks/troubleshooting/troubleshooting_jobs

Example output showing resource usage and flavor assignments for a Workload.

```yaml
apiVersion: kueue.x-k8s.io/v1beta2
kind: Workload
...
status:
  admission:
    clusterQueue: cluster-queue
    podSetAssignments:
    - count: 3
      flavors:
        cpu: default-flavor
        memory: default-flavor
      name: main
      resourceUsage:
        cpu: "3"
        memory: 600Mi
  ...
```

--------------------------------

### Example: Resume a ClusterQueue

Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_resume/kueuectl_resume_clusterqueue

Example command to resume a ClusterQueue named 'my-clusterqueue'.

```bash
  # Resume the clusterqueue
  kueuectl resume clusterqueue my-clusterqueue
```

--------------------------------

### Example Provisioning Request Name

Source: https://kueue.sigs.k8s.io/docs/tasks/troubleshooting/troubleshooting_provreq

An example of a generated ProvisioningRequest name.

```text
sample-job-2zcsb-57864-sample-admissioncheck-1
```

--------------------------------

### Install Kueue via Helm

Source: https://kueue.sigs.k8s.io/docs/getting-started/installation

Installs Kueue using Helm charts from the OCI registry or a downloaded tarball.

```bash
helm install kueue oci://registry.k8s.io/kueue/charts/kueue \
  --version=0.19.0 \
  --namespace  kueue-system \
  --create-namespace \
  --wait --timeout 300s
```

```bash
helm install kueue https://github.com/kubernetes-sigs/kueue/releases/download/v0.19.0/kueue-0.19.0.tgz \
  --namespace kueue-system \
  --create-namespace \
  --wait --timeout 300s
```

--------------------------------

### Build and install from source

Source: https://kueue.sigs.k8s.io/docs/getting-started/installation

Clones the repository and deploys Kueue using a custom image registry.

```bash
git clone https://github.com/kubernetes-sigs/kueue.git
cd kueue
IMAGE_REGISTRY=registry.example.com/my-user make image-local-push deploy
```

--------------------------------

### Sample Sandbox Configuration

Source: https://kueue.sigs.k8s.io/docs/tasks/run/external_workloads/sandbox

A complete example of a Sandbox resource configured for Kueue management.

```yaml
apiVersion: agents.x-k8s.io/v1alpha1
kind: Sandbox
metadata:
  name: hello-world-kueue
spec:
  podTemplate:
    metadata:
      labels:
        kueue.x-k8s.io/queue-name: user-queue
    spec:
      containers:
      - name: hello
        image: busybox
        command: ["sleep"]
        args: ["6000"]
        resources:
          requests:
            cpu: "100m"
            memory: "200Mi"
      restartPolicy: Never
```

--------------------------------

### Apply Manager Configuration

Source: https://kueue.sigs.k8s.io/docs/tasks/dev/setup_multikueue_development_environment

Apply the manager setup configuration file.

```bash
kubectl --context kind-manager apply -f examples/multikueue/tas/manager-setup.yaml
```

--------------------------------

### Apply RBAC Manifests

Source: https://kueue.sigs.k8s.io/docs/tasks/manage/monitor_pending_workloads/pending_workloads_on_demand

Apply the RBAC configuration using the provided example URL.

```bash
kubectl apply -f https://kueue.sigs.k8s.io/examples/visibility/cluster-role-and-binding.yaml
```

--------------------------------

### List ResourceFlavor example

Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_list/kueuectl_list_resourceflavor

A simple command to list all available ResourceFlavor resources.

```bash
  # List ResourceFlavor
  kueuectl list resourceflavor
```

--------------------------------

### Example Job Event Output

Source: https://kueue.sigs.k8s.io/docs/tasks/troubleshooting/troubleshooting_jobs

Sample output showing the CreatedWorkload event.

```text
  Normal  CreatedWorkload   24s   batch/job-kueue-controller  Created Workload: my-namespace/job-my-job-19797
```

--------------------------------

### Install development version

Source: https://kueue.sigs.k8s.io/docs/getting-started/installation

Deploys the latest development version of Kueue into the cluster.

```bash
kubectl apply --server-side -k "github.com/kubernetes-sigs/kueue/config/default?ref=main"
```

--------------------------------

### Example Output

Source: https://kueue.sigs.k8s.io/docs/tasks/run/python_jobs

Expected output format when running the queue control script.

```text
⛑️  Local Queues
Found queue user-queue
  Admitted workloads: 3
  Pending workloads: 0
  Flavor default-flavor has resources [{'name': 'cpu', 'total': '3'}, {'name': 'memory', 'total': '600Mi'}]

💼️ Jobs
Found job sample-job-8n5sb
  Succeeded: 3
  Ready: 0
Found job sample-job-gnxtl
  Succeeded: 1
  Ready: 0
Found job tacos46bqw
  Succeeded: 1
  Ready: 1
```

--------------------------------

### Sample SparkApplication Manifest

Source: https://kueue.sigs.k8s.io/docs/tasks/run/kubeflow/sparkapplications

A complete example of a SparkApplication configured for use with Kueue.

```yaml
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  type: Scala
  mode: cluster                 # spark-operator supports "cluster" mode only
  sparkVersion: 4.0.0
  image: spark:4.0.0
  imagePullPolicy: IfNotPresent
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples.jar
  arguments:
  - "50000"
  memoryOverheadFactor: "0"     # spark adds extra memory on memory limits
                                # for non-JVM tasks. 0 can avoid it.
  driver:
    coreRequest: "1"
    memory: 1g                  # In Java format (e.g. 512m, 2g)
    serviceAccount: spark       # You need to create this service account beforehand,
                                # and the service account should have proper role
                                # ref: https://github.com/kubeflow/spark-operator/blob/master/config/rbac/spark-application-rbac.yaml
  executor:
    instances: 2
    coreRequest: "1"
    memory: 1g                  # In Java format (e.g. 512m, 2g)
    deleteOnTermination: false  # to keep terminated executor pods for demo purpose
```

--------------------------------

### Apply Sample DRA Capacity Queues

Source: https://kueue.sigs.k8s.io/docs/tasks/manage/setup_dra

Apply the example configuration for capacity-based queues.

```bash
kubectl apply -f https://kueue.sigs.k8s.io/examples/dra/sample-dra-capacity-queues.yaml
```

--------------------------------

### Sample JAXJob configuration

Source: https://kueue.sigs.k8s.io/docs/tasks/run/kubeflow/jaxjobs

A complete example of a JAXJob definition configured for Kueue scheduling.

```yaml
apiVersion: kubeflow.org/v1
kind: JAXJob
metadata:
  name: jax-simple
  namespace: default
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  jaxReplicaSpecs:
    Worker:
      replicas: 2
      restartPolicy: OnFailure
      template:
        spec:
          containers:
            - name: jax
              image: docker.io/kubeflow/jaxjob-simple:latest
              command:
                - "python3"
                - "train.py"
              imagePullPolicy: Always
              resources:
                requests:
                  cpu: 1
                  memory: "200Mi"
```

--------------------------------

### Example ResourceSlice Output

Source: https://kueue.sigs.k8s.io/docs/tasks/manage/setup_dra

Sample output showing the structure of counter data in a ResourceSlice.

```text
gpu.example.com  gpu-0: [{"counterSet":"shared","counters":{"memory":{"value":"10Gi"}}}]
```

--------------------------------

### Install kube-prometheus

Source: https://kueue.sigs.k8s.io/docs/tasks/dev/setup_dev_monitoring

Deploys the full monitoring stack including Prometheus, Grafana, and alerting.

```bash
git clone https://github.com/prometheus-operator/kube-prometheus.git
cd kube-prometheus
kubectl apply --server-side -f manifests/setup
kubectl wait --for condition=Established --all CustomResourceDefinition --namespace=monitoring
kubectl apply -f manifests/
kubectl wait --for=condition=Ready pods --all -n monitoring --timeout=300s
```

--------------------------------

### Start Port Forwarding

Source: https://kueue.sigs.k8s.io/docs/tasks/manage/enable_kueueviz

Exposes the frontend and backend services locally for development access.

```bash
kubectl port-forward svc/kueue-kueueviz-frontend -n kueue-system 8080 &
kubectl port-forward svc/kueue-kueueviz-backend  -n kueue-system 8081:8080
```

--------------------------------

### Install Kueue via kubectl

Source: https://kueue.sigs.k8s.io/docs/getting-started/installation

Deploys the released version of Kueue using server-side apply.

```bash
kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.19.0/manifests.yaml
```

--------------------------------

### Uninstall source installation

Source: https://kueue.sigs.k8s.io/docs/getting-started/installation

Removes the Kueue installation built from source.

```bash
make undeploy
```

--------------------------------

### Sample Deployment Configuration

Source: https://kueue.sigs.k8s.io/docs/tasks/run/deployment

A complete example of a Deployment configured for Kueue, including queue assignment and resource requests.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
    kueue.x-k8s.io/queue-name: user-queue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: registry.k8s.io/nginx-slim:0.27
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: "100m"
```

--------------------------------

### Install ClusterProfile CRDs

Source: https://kueue.sigs.k8s.io/docs/tasks/manage/setup_multikueue

Command to manually install the ClusterProfile CRDs into the cluster.

```bash
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-inventory-api/refs/heads/main/config/crd/bases/multicluster.x-k8s.io_clusterprofiles.yaml
```

--------------------------------

### Example: Stop a LocalQueue

Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_stop/kueuectl_stop_localqueue

Example usage of the stop command for a specific LocalQueue.

```bash
  # Stop the localqueue
  kueuectl stop localqueue my-localqueue
```

--------------------------------

### Listing Pods by Resource

Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_list/kueuectl_list_pods

Examples showing how to list pods for a specific Job or Pod group.

```bash
# List Pods for the Job
kueuectl list pods --for job/job-name

# List Pods for the Pod group
kueuectl list pods --for pod/pod-name
```

--------------------------------

### Workload status output examples

Source: https://kueue.sigs.k8s.io/docs/tasks/troubleshooting/troubleshooting_provreq

Examples of expected and error-related status outputs for a workload.

```text
[...]
Status:
  Conditions:
    Last Transition Time:  2024-05-22T10:26:40Z
    Message:               Quota reserved in ClusterQueue cluster-queue
    Observed Generation:   1
    Reason:                QuotaReserved
    Status:                True
    Type:                  QuotaReserved
```

```text
  Conditions:
    Last Transition Time:  2024-05-22T08:48:47Z
    Message:               couldn't assign flavors to pod set main: insufficient unused quota for memory in flavor default-flavor, 4396Mi more needed
    Observed Generation:   1
    Reason:                Pending
    Status:                False
    Type:                  QuotaReserved
```

```text
Status:
  Conditions:
    Last Transition Time:  2024-05-22T08:57:09Z
    Message:               ClusterQueue cluster-queue doesn't exist
    Observed Generation:   1
    Reason:                Inadmissible
    Status:                False
    Type:                  QuotaReserved
```

--------------------------------

### Run Sample Jobs

Source: https://kueue.sigs.k8s.io/docs/tasks/run/python_jobs

Commands to execute sample jobs using uv.

```bash
uv run sample-job.py
uv run sample-job.py
uv run sample-job.py --job-name tacos
```

--------------------------------

### Example: Stop a ClusterQueue

Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_stop/kueuectl_stop_clusterqueue

Example command to stop a specific ClusterQueue named my-clusterqueue.

```bash
  # Stop the clusterqueue
  kueuectl stop clusterqueue my-clusterqueue
```

--------------------------------

### Create vGPU Deployment

Source: https://kueue.sigs.k8s.io/docs/tasks/run/using_hami

Command to apply the HAMI sample configuration.

```bash
kubectl create -f https://kueue.sigs.k8s.io/examples/serving-workloads/sample-hami.yaml
```

--------------------------------

### Quick start e2e infrastructure

Source: https://kueue.sigs.k8s.io/docs/tasks/dev/setup_dev_monitoring

Provisions a Kind cluster with Prometheus and Kueue pre-configured for development.

```bash
E2E_MODE=dev GINKGO_ARGS="--label-filter=feature:prometheus" make kind-image-build test-e2e
```

--------------------------------

### Example: Resume a specific LocalQueue

Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_resume/kueuectl_resume_localqueue

Example usage to resume a LocalQueue named 'my-localqueue'.

```bash
  # Resume the localqueue
  kueuectl resume localqueue my-localqueue
```

--------------------------------

### LocalQueue status output example

Source: https://kueue.sigs.k8s.io/docs/tasks/troubleshooting/troubleshooting_queues

Example of a LocalQueue status showing an inactive ClusterQueue condition.

```yaml
status:
  admittedWorkloads: 0
  conditions:
  - lastTransitionTime: "2024-05-03T18:57:32Z"
    message: Can't submit new workloads to clusterQueue
    reason: ClusterQueueIsInactive
    status: "False"
    type: Active
```

--------------------------------

### Configure Binary Permissions and Path

Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/installation

Make the downloaded binary executable and move it to a system PATH location.

```bash
chmod +x ./kubectl-kueue
```

```bash
sudo mv ./kubectl-kueue /usr/local/bin/kubectl-kueue
```

--------------------------------

### ProvisioningRequest output examples

Source: https://kueue.sigs.k8s.io/docs/tasks/troubleshooting/troubleshooting_provreq

Sample output showing the status fields of a ProvisioningRequest and a detailed failure condition.

```text
NAME                                                 ACCEPTED   PROVISIONED   FAILED   AGE
sample-job-2zcsb-57864-sample-admissioncheck-1       True       False         False    20s
```

```text
[...]
Status:
  Conditions:
    Last Transition Time:  2024-05-22T13:04:54Z
    Message:               Provisioning Request wasn't accepted.
    Observed Generation:   1
    Reason:                NotAccepted
    Status:                False
    Type:                  Accepted
    Last Transition Time:  2024-05-22T13:04:54Z
    Message:               Provisioning Request wasn't provisioned.
    Observed Generation:   1
    Reason:                NotProvisioned
    Status:                False
    Type:                  Provisioned
    Last Transition Time:  2024-05-22T13:06:49Z
    Message:               max cluster limit reached, nodepools out of resources: default-nodepool (cpu, memory)
    Observed Generation:   1
    Reason:                OutOfResources
    Status:                True
    Type:                  Failed
```

--------------------------------

### Install KubeRay CRDs

Source: https://kueue.sigs.k8s.io/docs/tasks/run/multikueue/kuberay

Install KubeRay CRDs on the manager cluster for versions prior to Kueue v0.11.0.

```bash
kubectl create -k "github.com/ray-project/kuberay/ray-operator/config/crd?ref=v1.3.0"
```

--------------------------------

### Apply Prometheus ServiceMonitor via manifests

Source: https://kueue.sigs.k8s.io/docs/tasks/manage/observability/setup_prometheus

Install the ServiceMonitor using the release manifests for kubectl installations.

```bash
VERSION=v0.19.0
kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/${VERSION}/prometheus.yaml

```

--------------------------------

### Examples of creating resource flavors

Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_create/kueuectl_create_resourceflavor

Various ways to create resource flavors, including options for labels, taints, and tolerations.

```bash
  # Create a resource flavor
  kueuectl create resourceflavor my-resource-flavor
  
  # Create a resource flavor with labels
  kueuectl create resourceflavor my-resource-flavor \
  --node-labels kubernetes.io/arch=arm64,kubernetes.io/os=linux
  
  # Create a resource flavor with node taints
  kueuectl create resourceflavor my-resource-flavor \
  --node-taints key1=value:NoSchedule,key2:NoExecute
  
  # Create a resource flavor with tolerations
  kueuectl create resourceflavor my-resource-flavor \
  --tolerations key1=value:NoSchedule,key2:NoExecute,key3=value,key4,:PreferNoSchedule
```

--------------------------------

### Install uv package manager

Source: https://kueue.sigs.k8s.io/docs/tasks/run/python_jobs

Use this command to install the uv tool, which is required to run the provided Python scripts.

```bash
# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh
```

--------------------------------

### Create Job via URL

Source: https://kueue.sigs.k8s.io/docs/tasks/manage/setup_topology_aware_scheduling

Command to create the sample job directly from the provided URL.

```bash
kubectl create -f https://kueue.sigs.k8s.io/examples/tas/sample-job-required.yaml
```

--------------------------------

### Install Kubeflow Trainer CRDs

Source: https://kueue.sigs.k8s.io/docs/tasks/run/multikueue/kubeflow

Use this command to install the necessary Custom Resource Definitions for Kubeflow Trainer v1.9.0 on the management cluster.

```bash
kubectl apply -k "github.com/kubeflow/trainer.git/manifests/base/crds?ref=v1.9.0"
```

--------------------------------

### Create Deployment via kubectl

Source: https://kueue.sigs.k8s.io/docs/tasks/run/deployment

Command to deploy the sample configuration to the cluster.

```bash
kubectl create -f https://kueue.sigs.k8s.io/examples/serving-workloads/sample-deployment.yaml
```

--------------------------------

### Provision a Kind Cluster

Source: https://kueue.sigs.k8s.io/docs/tasks/manage/setup_failure_recovery

Download the configuration file and create a Kind cluster for testing Kueue features.

```bash
wget https://raw.githubusercontent.com/kubernetes-sigs/kueue/81cae0608cac3d457b8ecf8a0480e7994c2151c1/hack/kind-cluster.yaml
kind create cluster --config hack/kind-cluster.yaml
```

--------------------------------

### ClusterQueue status output examples

Source: https://kueue.sigs.k8s.io/docs/tasks/troubleshooting/troubleshooting_queues

Examples of ClusterQueue status outputs, including error states like FlavorNotFound and a healthy Ready state.

```yaml
status:
  admittedWorkloads: 0
  conditions:
  - lastTransitionTime: "2024-05-03T18:22:30Z"
    message: 'Can''t admit new workloads: FlavorNotFound'
    reason: FlavorNotFound
    status: "False"
    type: Active
```

```yaml
status:
  admittedWorkloads: 1
  conditions:
  - lastTransitionTime: "2024-05-03T18:35:28Z"
    message: Can admit new workloads
    reason: Ready
    status: "True"
    type: Active
```

--------------------------------

### Apply the Job manifest

Source: https://kueue.sigs.k8s.io/docs/getting-started/quick-start

Command to deploy the sample Job using kubectl.

```bash
kubectl apply -f https://kueue.sigs.k8s.io/examples/getting-started/quick-start-job.yaml
```

--------------------------------

### Install Kueue via Python script

Source: https://kueue.sigs.k8s.io/docs/tasks/run/python_jobs

A Python script that installs Kueue on a cluster by fetching manifests from GitHub and applying them via the Kubernetes CustomObjectsApi.

```python
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.9"
# dependencies = [
#     "kubernetes",
#     "requests",
# ]
# ///

from kubernetes import utils, config, client
import tempfile
import requests
import argparse

# install-kueue-queues.py will:
# 1. install queue from the latest or a specific version on GitHub
# This example will demonstrate installing Kueue and applying a YAML file (local) to install Kueue

# Make sure your cluster is running!


def get_parser():
    parser = argparse.ArgumentParser(
        description="Submit Kueue Job Example",
        formatter_class=argparse.RawTextHelpFormatter,
    )
    parser.add_argument(
        "--version",
        help="Version of Kueue to install (if undefined, will install from master branch)",
        default=None,
    )
    return parser


def main():
    """
    Install Kueue and the Queue components.

    This will error if they are already installed.
    """
    config.load_kube_config()
    parser = get_parser()
    args, _ = parser.parse_known_args()
    install_kueue(args.version)


def get_install_url(version):
    """
    Get the install version.

    If a version is specified, use it. Otherwise install
    from the main branch.
    """
    if version is not None:
        return f"https://github.com/kubernetes-sigs/kueue/releases/download/v{version}/manifests.yaml"
    return "https://github.com/kubernetes-sigs/kueue/config/default?ref=main"


def install_kueue(version):
    """
    Install Kueue of a particular version.
    """
    crd_api = client.CustomObjectsApi()
    api_client = crd_api.api_client

    print("⭐️ Installing Kueue...")
    url = get_install_url(version)
    with tempfile.NamedTemporaryFile(delete=True) as install_yaml:
        res = requests.get(url)
        assert res.status_code == 200
        install_yaml.write(res.content)
        utils.create_from_yaml(api_client, install_yaml.name)


if __name__ == "__main__":
    main()
```

--------------------------------

### Example Workload Resource Usage Output

Source: https://kueue.sigs.k8s.io/docs/tasks/manage/setup_dra

Sample output showing the resource usage charged to the workload.

```text
job-sample-dra-counter-job-xxxxx: {"gpu.memory":"85899345920"}
```

--------------------------------

### Stop a Workload

Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_stop/kueuectl_stop_workload

Example of stopping a specific workload.

```bash
  # Stop the workload
  kueuectl stop workload my-workload
```

--------------------------------

### Uninstall development version

Source: https://kueue.sigs.k8s.io/docs/getting-started/installation

Removes the Kueue installation from the cluster.

```bash
kubectl delete -k "github.com/kubernetes-sigs/kueue/config/default?ref=main"
```

--------------------------------

### TrainJob Submission Output

Source: https://kueue.sigs.k8s.io/docs/tasks/run/python_jobs

Example console output after submitting a TrainJob.

```text
⭐️ Submitting TrainJob to runtime torch-distributed...
⭐️ Created TrainJob md3b82e943a5
Use:
"kubectl get queue" to see queue assignment
"kubectl get trainjobs" to see TrainJobs
```

--------------------------------

### AppWrapper with Deployment

Source: https://kueue.sigs.k8s.io/docs/tasks/run/appwrappers

Example of an AppWrapper definition wrapping a Kubernetes Deployment.

```yaml
apiVersion: workload.codeflare.dev/v1beta2
kind: AppWrapper
metadata:
  name: sample-appwrapper-deployment
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  components:
  - podSets:
    - path: "template.spec.template"
      replicas: 3
    template:
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: nginx-deployment
        labels:
          app: nginx
      spec:
        replicas: 3
        selector:
          matchLabels:
            app: nginx
        template:
          metadata:
            labels:
              app: nginx
          spec:
            containers:
              - name: nginx
                image: registry.k8s.io/nginx-slim:0.27
                ports:
                  - containerPort: 80
                resources:
                  requests:
                    cpu: "100m"
```

--------------------------------

### Run Sample TrainJob

Source: https://kueue.sigs.k8s.io/docs/tasks/run/python_jobs

Execute the sample training job script using uv.

```bash
uv run sample-trainjob.py
```

--------------------------------

### AppWrapper with PyTorchJob

Source: https://kueue.sigs.k8s.io/docs/tasks/run/appwrappers

Example of an AppWrapper definition wrapping a PyTorchJob workload.

```yaml
apiVersion: workload.codeflare.dev/v1beta2
kind: AppWrapper
metadata:
  name: sample-appwrapper-pytorch-job
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  components:
  - template:
      apiVersion: "kubeflow.org/v1"
      kind: PyTorchJob
      metadata:
        name: pytorch-simple
      spec:
        pytorchReplicaSpecs:
          Master:
            replicas: 1
            restartPolicy: OnFailure
            template:
              spec:
                containers:
                - name: pytorch
                  image: docker.io/kubeflowkatib/pytorch-mnist-cpu:v1beta1-fc858d1
                  command:
                  - "python3"
                  - "/opt/pytorch-mnist/mnist.py"
                  - "--epochs=1"
                  resources:
                    requests:
                      cpu: 1
          Worker:
            replicas: 1
            restartPolicy: OnFailure
            template:
              spec:
                containers:
                - name: pytorch
                  image: docker.io/kubeflowkatib/pytorch-mnist-cpu:v1beta1-fc858d1
                  command:
                  - "python3"
                  - "/opt/pytorch-mnist/mnist.py"
                  - "--epochs=1"
                  resources:
                    requests:
                      cpu: 1
```

--------------------------------

### Delete a Workload

Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_delete/kueuectl_delete_workload

Example command to delete a specific Workload resource.

```bash
  # Delete the Workload
  kueuectl delete kueueworkload my-workload
```

--------------------------------

### Example Pod Status Output

Source: https://kueue.sigs.k8s.io/docs/tasks/manage/setup_failure_recovery

Sample output showing a pod in the Failed state alongside a running pod.

```text
NAME                     READY   STATUS    RESTARTS   AGE    IP            NODE           NOMINATED NODE   READINESS GATES
sample-job-l8pts-kxpng   1/1     Running   0          17s    10.244.1.9    kind-worker    <none>           <none>
sample-job-l8pts-qcwbc   1/1     Failed    0          8m4s   10.244.2.29   kind-worker2   <none>           <none>
```

--------------------------------

### LocalQueue Output Example

Source: https://kueue.sigs.k8s.io/docs/getting-started/quick-start

Expected output format when querying the LocalQueue.

```text
NAME         CLUSTERQUEUE    PENDING WORKLOADS   ADMITTED WORKLOADS
user-queue   cluster-queue   0                   0
```

--------------------------------

### ClusterQueue Output Example

Source: https://kueue.sigs.k8s.io/docs/getting-started/quick-start

Expected output format when querying the ClusterQueue.

```text
NAME            COHORT   STRATEGY         PENDING WORKLOADS   ADMITTED WORKLOADS
cluster-queue            BestEffortFIFO   0                   0
```

--------------------------------

### Define an AdmissionCheck object

Source: https://kueue.sigs.k8s.io/docs/concepts/admission_check

Example of an AdmissionCheck API object configuration.

```yaml
apiVersion: kueue.x-k8s.io/v1beta2
kind: AdmissionCheck
metadata:
  name: prov-test
spec:
  controllerName: kueue.x-k8s.io/provisioning-request
  parameters:
    apiGroup: kueue.x-k8s.io
    kind: ProvisioningRequestConfig
    name: prov-test-config
```

--------------------------------

### Install Kueue with KueueViz enabled via Helm

Source: https://kueue.sigs.k8s.io/docs/tasks/manage/enable_kueueviz

Use this command to deploy Kueue and enable the KueueViz visualization tool simultaneously.

```bash
helm install kueue oci://registry.k8s.io/kueue/charts/kueue \
  --version=0.19.0 \
  --namespace kueue-system \
  --create-namespace \
  --wait --timeout 300s \
  --set enableKueueViz=true # enable KueueViz
```

--------------------------------

### Install Flux Operator

Source: https://kueue.sigs.k8s.io/docs/tasks/run/python_jobs

Command to apply the Flux Operator manifest to the cluster.

```bash
kubectl apply -f https://raw.githubusercontent.com/flux-framework/flux-operator/main/examples/dist/flux-operator.yaml
```

--------------------------------

### Complete MPIJob Configuration Example

Source: https://kueue.sigs.k8s.io/docs/tasks/run/kubeflow/mpijobs

A full YAML definition for an MPIJob configured to work with Kueue.

```yaml
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: pi
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  slotsPerWorker: 1
  runPolicy:
    cleanPodPolicy: Running
    ttlSecondsAfterFinished: 60
  sshAuthMountPath: /home/mpiuser/.ssh
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          containers:
          - image: mpioperator/mpi-pi:openmpi
            name: mpi-launcher
            securityContext:
              runAsUser: 1000
            command:
            - mpirun
            args:
            - -n
            - "2"
            - /home/mpiuser/pi
            resources:
              limits:
                cpu: 1
                memory: 1Gi
    Worker:
      replicas: 2
      template:
        spec:
          containers:
          - image: mpioperator/mpi-pi:openmpi
            name: mpi-worker
            securityContext:
              runAsUser: 1000
            command:
            - /usr/sbin/sshd
            args:
            - -De
            - -f
            - /home/mpiuser/.sshd_config
            resources:
              limits:
                cpu: 1
                memory: 1Gi
```
Tokens:
Raw