### List ResourceFlavor Example Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_list/kueuectl_list_resourceflavor This example demonstrates how to list all available ResourceFlavors. ```bash kueuectl list resourceflavor ``` -------------------------------- ### View ProvisioningRequest output example Source: https://kueue.sigs.k8s.io/docs/tasks/troubleshooting/troubleshooting_provreq Example output showing the status columns for a ProvisioningRequest. ```text NAME ACCEPTED PROVISIONED FAILED AGE sample-job-2zcsb-57864-sample-admissioncheck-1 True False False 20s ``` -------------------------------- ### Install Single ClusterQueue Setup Source: https://kueue.sigs.k8s.io/docs/tasks/manage/monitor_pending_workloads/pending_workloads_on_demand Apply this YAML configuration to set up a basic ClusterQueue, ResourceFlavor, and LocalQueue. Ensure the Kueue controller manager is running. ```yaml apiVersion: kueue.x-k8s.io/v1beta2 kind: ResourceFlavor metadata: name: "default-flavor" --- apiVersion: kueue.x-k8s.io/v1beta2 kind: ClusterQueue metadata: name: "cluster-queue" spec: namespaceSelector: {} # match all. resourceGroups: - coveredResources: ["cpu", "memory"] flavors: - name: "default-flavor" resources: - name: "cpu" nominalQuota: 9 - name: "memory" nominalQuota: 36Gi --- apiVersion: kueue.x-k8s.io/v1beta2 kind: LocalQueue metadata: namespace: "default" name: "user-queue" spec: clusterQueue: "cluster-queue" ``` -------------------------------- ### List Workload Example Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_list/kueuectl_list_workload A simple example demonstrating how to list workloads using the kueuectl command. ```bash # List Workload kueuectl list kueueworkload ``` -------------------------------- ### List LocalQueue example Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_list/kueuectl_list_localqueue A basic example of listing all LocalQueues. ```bash # List LocalQueue kueuectl list localqueue ``` -------------------------------- ### Resume workload example Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_resume/kueuectl_resume_workload Example command to resume a specific workload named my-workload. ```bash # Resume the workload kueuectl resume kueueworkload my-workload ``` -------------------------------- ### ProvisioningRequest failure status example Source: https://kueue.sigs.k8s.io/docs/tasks/troubleshooting/troubleshooting_provreq Example output showing a failed ProvisioningRequest with specific condition details. ```text [...] Status: Conditions: Last Transition Time: 2024-05-22T13:04:54Z Message: Provisioning Request wasn't accepted. Observed Generation: 1 Reason: NotAccepted Status: False Type: Accepted Last Transition Time: 2024-05-22T13:04:54Z Message: Provisioning Request wasn't provisioned. Observed Generation: 1 Reason: NotProvisioned Status: False Type: Provisioned Last Transition Time: 2024-05-22T13:06:49Z Message: max cluster limit reached, nodepools out of resources: default-nodepool (cpu, memory) Observed Generation: 1 Reason: OutOfResources Status: True Type: Failed ``` -------------------------------- ### Example Pod Status Output Source: https://kueue.sigs.k8s.io/docs/tasks/manage/setup_wait_for_pods_ready This is an example of the output from `kubectl get pods`, showing various job pods and their statuses (Running or Pending). ```text NAME READY STATUS RESTARTS AGE job1-0-9pvs8 1/1 Running 0 28m job1-1-w9zht 1/1 Running 0 28m job1-10-fg99v 1/1 Running 0 28m job1-11-4gspm 1/1 Running 0 28m job1-12-w5jft 1/1 Running 0 28m job1-13-8d5jk 1/1 Running 0 28m job1-14-h5q8x 1/1 Running 0 28m job1-15-kkv4j 0/1 Pending 0 28m job1-16-frs8k 0/1 Pending 0 28m job1-17-g78g8 0/1 Pending 0 28m job1-18-2ghmt 0/1 Pending 0 28m job1-19-4w2j5 0/1 Pending 0 28m job1-2-9s486 1/1 Running 0 28m job1-3-s9kh4 1/1 Running 0 28m job1-4-52mj9 1/1 Running 0 28m job1-5-bpjv5 1/1 Running 0 28m job1-6-7f7tj 1/1 Running 0 28m job1-7-pnq7w 1/1 Running 0 28m job1-8-7s894 1/1 Running 0 28m job1-9-kz4gt 1/1 Running 0 28m job2-0-x6xvg 1/1 Running 0 28m job2-1-flkpj 1/1 Running 0 28m job2-10-vf4j9 1/1 Running 0 28m job2-11-ktbld 0/1 Pending 0 28m job2-12-sf4xb 1/1 Running 0 28m job2-13-9j7lp 0/1 Pending 0 28m job2-14-czc6l 1/1 Running 0 28m job2-15-m77zt 0/1 Pending 0 28m job2-16-7p7fs 0/1 Pending 0 28m job2-17-sfdmj 0/1 Pending 0 28m job2-18-cs4lg 0/1 Pending 0 28m job2-19-x66dt 0/1 Pending 0 28m job2-2-hnqjv 1/1 Running 0 28m job2-3-pkwhw 1/1 Running 0 28m job2-4-gdtsh 1/1 Running 0 28m job2-5-6swdc 1/1 Running 0 28m job2-6-qb6sp 1/1 Running 0 28m job2-7-grcg4 0/1 Pending 0 28m job2-8-kg568 1/1 Running 0 28m job2-9-hvwj8 0/1 Pending 0 28m ``` -------------------------------- ### Configure waitForPodsReady in Kueue Source: https://kueue.sigs.k8s.io/docs/tasks/manage/setup_wait_for_pods_ready Enable the all-or-nothing scheduling by configuring `waitForPodsReady` in Kueue. This example shows the basic setup with timeout and recovery timeout. ```yaml waitForPodsReady: timeout: 10m recoveryTimeout: 3m blockAdmission: false requeuingStrategy: timestamp: Eviction | Creation backoffLimitCount: 5 backoffBaseSeconds: 60 backoffMaxSeconds: 3600 ``` -------------------------------- ### Apply single ClusterQueue setup Source: https://kueue.sigs.k8s.io/docs/tasks/manage/administer_cluster_quotas Apply a pre-configured setup for a single ClusterQueue and ResourceFlavor. ```bash kubectl apply -f https://kueue.sigs.k8s.io/examples/admin/single-clusterqueue-setup.yaml ``` -------------------------------- ### Sample StatefulSet configuration Source: https://kueue.sigs.k8s.io/docs/tasks/run/statefulset A complete example of a StatefulSet configured for Kueue management. ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: nginx-statefulset labels: app: nginx kueue.x-k8s.io/queue-name: user-queue spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: registry.k8s.io/nginx-slim:0.27 ports: - containerPort: 80 resources: requests: cpu: "100m" serviceName: "nginx" ``` -------------------------------- ### PodSetAssignment Topology Example 4 Source: https://kueue.sigs.k8s.io/docs/reference/kueue.v1beta2 Presents an alternative representation of Example 3 using multiple slices to further optimize storage by extracting longer common prefixes. This example demonstrates how to partition assignments into separate slices for efficiency. ```yaml topologyAssignment: levels: - kubernetes.io/hostname slices: - domainCount: 4 valuesPerLevel: - individual: prefix: block-1-rack-1-node- roots: [1, 2, 3, 4] podCounts: universal: 1 - domainCount: 2 valuesPerLevel: - individual: prefix: block-1-rack-2-node- roots: [1, 2] podCounts: universal: 1 ``` -------------------------------- ### Resume ClusterQueue Example Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_resume/kueuectl_resume_clusterqueue Example command to resume a specific ClusterQueue named my-clusterqueue. ```bash # Resume the clusterqueue kueuectl resume clusterqueue my-clusterqueue ``` -------------------------------- ### Install Kueue Released Version with Helm (OCI) Source: https://kueue.sigs.k8s.io/docs/installation Install a released version of Kueue using Helm by referencing the OCI registry. This command creates the 'kueue-system' namespace if it doesn't exist and waits for the installation to complete within 300 seconds. ```bash helm install kueue oci://registry.k8s.io/kueue/charts/kueue \ --version=0.17.0 \ --namespace kueue-system \ --create-namespace \ --wait --timeout 300s ``` -------------------------------- ### Install kubectl-kueue via Krew Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/installation Use this command to install the kubectl-kueue plugin if you have Krew installed. Krew is the recommended package manager for Kubernetes CLI plugins. ```bash kubectl krew install kueue ``` -------------------------------- ### Install Visibility API Priority and Fairness Source: https://kueue.sigs.k8s.io/docs/tasks/manage/monitor_pending_workloads/pending_workloads_on_demand Apply the necessary manifests to install the API Priority and Fairness configuration for the visibility API. This is a prerequisite for accessing the API. ```bash kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.17.0/visibility-apf.yaml ``` -------------------------------- ### Sample SparkApplication Configuration Source: https://kueue.sigs.k8s.io/docs/tasks/run/kubeflow/sparkapplications A complete example of a SparkApplication configured for Kueue scheduling. ```yaml apiVersion: sparkoperator.k8s.io/v1beta2 kind: SparkApplication metadata: name: spark-pi labels: kueue.x-k8s.io/queue-name: user-queue spec: type: Scala mode: cluster # spark-operator supports "cluster" mode only sparkVersion: 4.0.0 image: spark:4.0.0 imagePullPolicy: IfNotPresent mainClass: org.apache.spark.examples.SparkPi mainApplicationFile: local:///opt/spark/examples/jars/spark-examples.jar arguments: - "50000" memoryOverheadFactor: "0" # spark adds extra memory on memory limits # for non-JVM tasks. 0 can avoid it. driver: coreRequest: "1" memory: 1g # In Java format (e.g. 512m, 2g) serviceAccount: spark # You need to create this service account beforehand, # and the service account should have proper role # ref: https://github.com/kubeflow/spark-operator/blob/master/config/rbac/spark-application-rbac.yaml executor: instances: 2 coreRequest: "1" memory: 1g # In Java format (e.g. 512m, 2g) deleteOnTermination: false # to keep terminated executor pods for demo purpose ``` -------------------------------- ### Provisioning Request Example Name Source: https://kueue.sigs.k8s.io/docs/tasks/troubleshooting/troubleshooting_provreq An example of a generated ProvisioningRequest name based on the naming pattern. ```text sample-job-2zcsb-57864-sample-admissioncheck-1 ``` -------------------------------- ### Install Kueue with KueueViz Enabled Source: https://kueue.sigs.k8s.io/docs/tasks/manage/enable_kueueviz Use this Helm command to install Kueue along with KueueViz. Ensure you specify the desired version and namespace. ```bash helm install kueue oci://registry.k8s.io/kueue/charts/kueue \ --version=0.17.0 \ --namespace kueue-system \ --create-namespace \ --wait --timeout 300s \ --set enableKueueViz=true # enable KueueViz ``` -------------------------------- ### Print kueuectl Version Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_version Use this command to display the client version and the Kueue controller manager image if it is installed. This is useful for checking installation status and compatibility. ```bash kueuectl version ``` ```bash # Prints the client version and the kueue controller manager image, if installed kueuectl version ``` -------------------------------- ### Install Kueue Released Version with Helm (tgz) Source: https://kueue.sigs.k8s.io/docs/installation Install a released version of Kueue using Helm by providing a direct URL to the tgz chart archive. This command creates the 'kueue-system' namespace if it doesn't exist and waits for the installation to complete within 300 seconds. ```bash helm install kueue https://github.com/kubernetes-sigs/kueue/releases/download/v0.17.0/kueue-0.17.0.tgz \ --namespace kueue-system \ --create-namespace \ --wait --timeout 300s ``` -------------------------------- ### Example Workload output Source: https://kueue.sigs.k8s.io/docs/tasks/manage/share_quotas_across_flavors Sample output showing admitted and pending workloads. ```text NAME QUEUE RESERVED IN ADMITTED AGE job-sample-job-abc12-xyz team-queue team-cluster-queue True 4s job-sample-job-def34-uvw team-queue team-cluster-queue True 3s job-sample-job-ghi56-rst team-queue team-cluster-queue True 3s job-sample-job-jkl78-opq team-queue team-cluster-queue True 2s job-sample-job-mno90-lmn team-queue 2s ``` -------------------------------- ### Example JobSet Definition Source: https://kueue.sigs.k8s.io/docs/tasks/run/jobsets This is a complete example of a JobSet definition, including network configuration, replicated jobs with their respective templates, and resource requests. It demonstrates how to combine queue selection, resource configuration, and job definitions. ```yaml # sample-jobset.yaml apiVersion: jobset.x-k8s.io/v1alpha2 kind: JobSet metadata: generateName: sleep-job- labels: kueue.x-k8s.io/queue-name: user-queue spec: network: enableDNSHostnames: false subdomain: some-subdomain replicatedJobs: - name: workers replicas: 1 template: spec: parallelism: 1 completions: 1 backoffLimit: 0 template: spec: containers: - name: sleep image: busybox resources: requests: cpu: 1 memory: "200Mi" command: - sleep args: - 100s - name: driver template: spec: parallelism: 1 completions: 1 backoffLimit: 0 template: spec: containers: - name: sleep image: busybox resources: requests: cpu: 2 memory: "200Mi" command: - sleep args: - 100s ``` -------------------------------- ### Install Flux Operator SDK and Controller Source: https://kueue.sigs.k8s.io/docs/tasks/run/python_jobs Commands to install the Python SDK and apply the Flux operator manifest. ```bash pip install fluxoperator ``` ```bash kubectl apply -f https://raw.githubusercontent.com/flux-framework/flux-operator/main/examples/dist/flux-operator.yaml ``` -------------------------------- ### Workload Status Output Example Source: https://kueue.sigs.k8s.io/docs/tasks/troubleshooting/troubleshooting_provreq Example output from kubectl describe showing the status of an Admission Check and associated annotations. ```yaml [...] Status: Admission Checks: Last Transition Time: 2024-05-22T10:47:46Z Message: Provisioning Request was successfully provisioned. Name: sample-admissioncheck Pod Set Updates: Annotations: cluster-autoscaler.kubernetes.io/consume-provisioning-request: sample-job-2zcsb-57864-sample-admissioncheck-1 cluster-autoscaler.kubernetes.io/provisioning-class-name: queued-provisioning.gke.io Name: main State: Ready ``` -------------------------------- ### Port-forward to Prometheus (Manual Setup) Source: https://kueue.sigs.k8s.io/docs/tasks/dev/setup_dev_monitoring Establish a port-forward connection to the Prometheus service after manual setup. Use port 9090 for local access. ```bash kubectl -n monitoring port-forward svc/prometheus-k8s 9090:9090 ``` -------------------------------- ### Setup ClusterQueue and LocalQueue Source: https://kueue.sigs.k8s.io/docs/tasks/manage/monitor_pending_workloads/pending_workloads_in_grafana Configures a basic Kueue ClusterQueue with a default ResourceFlavor and a LocalQueue associated with it. This setup is for demonstrating ClusterQueue visibility. ```yaml apiVersion: kueue.x-k8s.io/v1beta2 kind: ResourceFlavor metadata: name: "default-flavor" --- apiVersion: kueue.x-k8s.io/v1beta2 kind: ClusterQueue metadata: name: "cluster-queue" spec: namespaceSelector: {} # match all. resourceGroups: - coveredResources: ["cpu", "memory"] flavors: - name: "default-flavor" resources: - name: "cpu" nominalQuota: 9 - name: "memory" nominalQuota: 36Gi --- apiVersion: kueue.x-k8s.io/v1beta2 kind: LocalQueue metadata: namespace: "default" name: "user-queue" spec: clusterQueue: "cluster-queue" ``` -------------------------------- ### Apply Kueue Single ClusterQueue Setup Source: https://kueue.sigs.k8s.io/docs/tasks/run/python_jobs Apply the YAML configuration for a single cluster queue setup in Kueue. This should be done after confirming Kueue is ready. ```bash kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/kueue/main/site/static/examples/admin/single-clusterqueue-setup.yaml ``` -------------------------------- ### Define a sample JAXJob Source: https://kueue.sigs.k8s.io/docs/tasks/run/kubeflow/jaxjobs A complete example of a JAXJob configured for Kueue scheduling. ```yaml apiVersion: kubeflow.org/v1 kind: JAXJob metadata: name: jax-simple namespace: default labels: kueue.x-k8s.io/queue-name: user-queue spec: jaxReplicaSpecs: Worker: replicas: 2 restartPolicy: OnFailure template: spec: containers: - name: jax image: docker.io/kubeflow/jaxjob-simple:latest command: - "python3" - "train.py" imagePullPolicy: Always resources: requests: cpu: 1 memory: "200Mi" ``` -------------------------------- ### PodSetAssignment Topology Example 2 Source: https://kueue.sigs.k8s.io/docs/reference/kueue.v1beta2 Demonstrates an equivalent topology assignment to Example 1, but utilizes a universal value for the first level and extracts a common prefix for the second level. This can optimize storage for repetitive values. ```yaml topologyAssignment: levels: - cloud.provider.com/topology-block - cloud.provider.com/topology-rack slices: - domainCount: 2 valuesPerLevel: - universal: block-1 - individual: prefix: rack- roots: [1, 2] podCounts: individual: [4, 2] ``` -------------------------------- ### Sample PaddleJob Configuration Source: https://kueue.sigs.k8s.io/docs/tasks/run/kubeflow/paddlejobs A complete example of a PaddleJob configured for Kueue scheduling. ```yaml apiVersion: kubeflow.org/v1 kind: PaddleJob metadata: name: paddle-simple-cpu namespace: default labels: kueue.x-k8s.io/queue-name: user-queue spec: paddleReplicaSpecs: Worker: replicas: 2 restartPolicy: OnFailure template: spec: containers: - name: paddle image: registry.baidubce.com/paddlepaddle/paddle:2.5.1 command: - python args: - "-m" - paddle.distributed.launch - "run_check" ports: - containerPort: 37777 name: master imagePullPolicy: Always resources: requests: cpu: 1 memory: "256Mi" ``` -------------------------------- ### Build and install from source Source: https://kueue.sigs.k8s.io/docs/installation Clones the repository and uses make to build and deploy Kueue to a custom registry. ```bash git clone https://github.com/kubernetes-sigs/kueue.git cd kueue IMAGE_REGISTRY=registry.example.com/my-user make image-local-push deploy ``` -------------------------------- ### Install MPI Operator Python SDK Source: https://kueue.sigs.k8s.io/docs/tasks/run/python_jobs Clone the MPI Operator repository and install its Python SDK. Ensure you are in the correct directory before running the setup script. ```bash git clone --depth 1 https://github.com/kubeflow/mpi-operator /tmp/mpijob cd /tmp/mpijob/sdk/python/v2beta1 python setup.py install cd - ``` -------------------------------- ### PodSetAssignment Topology Example 1 Source: https://kueue.sigs.k8s.io/docs/reference/kueue.v1beta2 Illustrates a basic topology assignment with two domains and individual root values for each level. This example assigns 4 pods to nodes matching the first set of selectors and 2 pods to nodes matching the second set. ```yaml topologyAssignment: levels: - cloud.provider.com/topology-block - cloud.provider.com/topology-rack slices: - domainCount: 2 valuesPerLevel: - individual: roots: [block-1, block-1] - individual: roots: [rack-1, rack-2] podCounts: individual: [4, 2] ``` -------------------------------- ### Sample Deployment Configuration Source: https://kueue.sigs.k8s.io/docs/tasks/run/deployment A complete example of a Deployment configured for Kueue management. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx kueue.x-k8s.io/queue-name: user-queue spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: registry.k8s.io/nginx-slim:0.27 ports: - containerPort: 80 resources: requests: cpu: "100m" ``` -------------------------------- ### LocalQueue Metrics (alpha) Source: https://kueue.sigs.k8s.io/docs/reference/metrics The following metrics are available only if `LocalQueueMetrics` feature gate is enabled. Refer to the Installation guide for details on configuring feature gates. ```APIDOC ## LocalQueue Metrics (alpha) The following metrics are available only if `LocalQueueMetrics` feature gate is enabled. Check the Change the feature gates configuration section of the Installation for details. ### `kueue_local_queue_admission_checks_wait_time_seconds` - **Type**: Histogram - **Description**: The time from when a workload got the quota reservation until admission, per ’local_queue’. - **Labels**: `name`, `namespace`, `priority_class`, `replica_role` ### `kueue_local_queue_admission_wait_time_seconds` - **Type**: Histogram - **Description**: The time between a workload was created or requeued until admission, per ’local_queue’. - **Labels**: `name`, `namespace`, `priority_class`, `replica_role` ### `kueue_local_queue_admitted_active_workloads` - **Type**: Gauge - **Description**: The number of admitted Workloads that are active, per ’localQueue’. - **Labels**: `name`, `namespace`, `replica_role` ### `kueue_local_queue_admitted_workloads_total` - **Type**: Counter - **Description**: The total number of admitted workloads per ’local_queue’. - **Labels**: `name`, `namespace`, `priority_class`, `replica_role` ### `kueue_local_queue_evicted_workloads_total` - **Type**: Counter - **Description**: The number of evicted workloads per ’local_queue’. - **Labels**: `name`, `namespace`, `reason`, `underlying_cause`, `priority_class`, `replica_role` - **`reason` values**: `Preempted`, `PodsReadyTimeout`, `AdmissionCheck`, `ClusterQueueStopped`, `LocalQueueStopped`, `NodeFailures`, `Deactivated` - **`underlying_cause` values**: ``, `AdmissionCheck`, `MaximumExecutionTimeExceeded`, `RequeuingLimitExceeded` ### `kueue_local_queue_finished_workloads` - **Type**: Gauge - **Description**: The number of finished workloads, per ’local_queue’. - **Labels**: `name`, `namespace`, `replica_role` ### `kueue_local_queue_finished_workloads_total` - **Type**: Counter - **Description**: The total number of finished workloads per ’local_queue’. - **Labels**: `name`, `namespace`, `priority_class`, `replica_role` ### `kueue_local_queue_pending_workloads` - **Type**: Gauge - **Description**: The number of pending workloads, per ’local_queue’ and ‘status’. - **Labels**: `name`, `namespace`, `status`, `replica_role` - **`status` values**: `active`, `inadmissible` ### `kueue_local_queue_quota_reserved_wait_time_seconds` - **Type**: Histogram - **Description**: The time between a workload was created or requeued until it got quota reservation, per ’local_queue’. - **Labels**: `name`, `namespace`, `priority_class`, `replica_role` ### `kueue_local_queue_quota_reserved_workloads_total` - **Type**: Counter - **Description**: The total number of quota reserved workloads per ’local_queue’. - **Labels**: `name`, `namespace`, `priority_class` ``` -------------------------------- ### StatefulSet Example for MultiKueue Source: https://kueue.sigs.k8s.io/docs/tasks/run/multikueue/statefulset This YAML defines a Kubernetes StatefulSet named 'nginx-statefulset' that is configured to be managed by Kueue, indicated by the 'kueue.x-k8s.io/queue-name' label. Ensure Kueue is installed and configured for multi-cluster operation before applying. ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: nginx-statefulset labels: app: nginx kueue.x-k8s.io/queue-name: user-queue spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: registry.k8s.io/nginx-slim:0.27 ports: - containerPort: 80 resources: requests: cpu: "100m" serviceName: "nginx" ``` -------------------------------- ### Apply Example ClusterQueue and LocalQueue Source: https://kueue.sigs.k8s.io/docs/tasks/dev/setup_dev_monitoring Create a ClusterQueue and LocalQueue using the provided YAML configuration to set up basic Kueue resources for testing. ```yaml kubectl apply -f https://kueue.sigs.k8s.io/examples/admin/single-clusterqueue-setup.yaml ``` -------------------------------- ### Get Pending Workloads with Pagination Source: https://kueue.sigs.k8s.io/docs/tasks/manage/monitor_pending_workloads/pending_workloads_on_demand This command retrieves pending workloads from a LocalQueue with specified limits and offsets. Use the `limit` parameter to control the number of results and `offset` to start from a specific position. ```bash kubectl get --raw "/apis/visibility.kueue.x-k8s.io/v1beta2/namespaces/default/localqueues/user-queue/pendingworkloads?limit=1&offset=1" ``` -------------------------------- ### Example RayCluster Configuration Source: https://kueue.sigs.k8s.io/docs/tasks/run/rayclusters This YAML defines a RayCluster with a head node and a worker group. It specifies Ray version, service type, Ray start parameters, container images, ports, lifecycle commands, volume mounts, and resource requests/limits. It also includes configuration for worker scaling. ```yaml apiVersion: ray.io/v1 kind: RayCluster metadata: labels: kueue.x-k8s.io/queue-name: user-queue controller-tools.k8s.io: "1.0" # A unique identifier for the head node and workers of this cluster. name: raycluster-complete spec: rayVersion: '2.9.0' # Ray head pod configuration headGroupSpec: # Kubernetes Service Type. This is an optional field, and the default value is ClusterIP. # Refer to https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types. serviceType: ClusterIP # The `rayStartParams` are used to configure the `ray start` command. # See https://github.com/ray-project/kuberay/blob/master/docs/guidance/rayStartParams.md for the default settings of `rayStartParams` in KubeRay. # See https://docs.ray.io/en/latest/cluster/cli.html#ray-start for all available options in `rayStartParams`. rayStartParams: dashboard-host: '0.0.0.0' # pod template template: metadata: # Custom labels. NOTE: To avoid conflicts with KubeRay operator, do not define custom labels start with `raycluster`. # Refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ labels: {} spec: containers: - name: ray-head image: rayproject/ray:2.9.0 ports: - containerPort: 6379 name: gcs - containerPort: 8265 name: dashboard - containerPort: 10001 name: client lifecycle: preStop: exec: command: ["/bin/sh","-c","ray stop"] volumeMounts: - mountPath: /tmp/ray name: ray-logs # The resource requests and limits in this config are too small for production! # For an example with more realistic resource configuration, see # ray-cluster.autoscaler.large.yaml. # It is better to use a few large Ray pod than many small ones. # For production, it is ideal to size each Ray pod to take up the # entire Kubernetes node on which it is scheduled. resources: limits: cpu: "1" memory: "2G" requests: # For production use-cases, we recommend specifying integer CPU requests and limits. # We also recommend setting requests equal to limits for both CPU and memory. # For this example, we use a 500m CPU request to accommodate resource-constrained local # Kubernetes testing environments such as KinD and minikube. cpu: "1" memory: "2G" volumes: - name: ray-logs emptyDir: {} workerGroupSpecs: # the pod replicas in this group typed worker - replicas: 1 minReplicas: 1 maxReplicas: 10 # logical group name, for this called small-group, also can be functional groupName: small-group # If worker pods need to be added, we can increment the replicas. # If worker pods need to be removed, we decrement the replicas, and populate the workersToDelete list. # The operator will remove pods from the list until the desired number of replicas is satisfied. # If the difference between the current replica count and the desired replicas is greater than the # number of entries in workersToDelete, random worker pods will be deleted. #scaleStrategy: # workersToDelete: # - raycluster-complete-worker-small-group-bdtwh # - raycluster-complete-worker-small-group-hv457 # - raycluster-complete-worker-small-group-k8tj7 # The `rayStartParams` are used to configure the `ray start` command. # See https://github.com/ray-project/kuberay/blob/master/docs/guidance/rayStartParams.md for the default settings of `rayStartParams` in KubeRay. # See https://docs.ray.io/en/latest/cluster/cli.html#ray-start for all available options in `rayStartParams`. rayStartParams: {} #pod template template: spec: containers: - name: ray-worker image: rayproject/ray:2.9.0 lifecycle: preStop: exec: command: ["/bin/sh","-c","ray stop"] # use volumeMounts.Optional. # Refer to https://kubernetes.io/docs/concepts/storage/volumes/ volumeMounts: - mountPath: /tmp/ray name: ray-logs # The resource requests and limits in this config are too small for production! # For an example with more realistic resource configuration, see # ray-cluster.autoscaler.large.yaml. ``` -------------------------------- ### Create Deployment via kubectl Source: https://kueue.sigs.k8s.io/docs/tasks/run/deployment Command to deploy the sample configuration to the cluster. ```bash kubectl create -f https://kueue.sigs.k8s.io/examples/serving-workloads/sample-deployment.yaml ``` -------------------------------- ### Run Sample Jobs and Queue Control Source: https://kueue.sigs.k8s.io/docs/tasks/run/python_jobs Commands to execute sample jobs and the queue control script. ```bash python sample-job.py python sample-job.py python sample-job.py --job-name tacos ``` ```bash python sample-queue-control.py ``` -------------------------------- ### Install Kueue using Python Source: https://kueue.sigs.k8s.io/docs/tasks/run/python_jobs Installs Kueue to an existing cluster by applying its YAML manifests. This script can install from the master branch or a specific release version. ```python #!/usr/bin/env python3 from kubernetes import utils, config, client import tempfile import requests import argparse # install-kueue-queues.py will: # 1. install queue from the latest or a specific version on GitHub # This example will demonstrate installing Kueue and applying a YAML file (local) to install Kueue # Make sure your cluster is running! config.load_kube_config() crd_api = client.CustomObjectsApi() api_client = crd_api.api_client def get_parser(): parser = argparse.ArgumentParser( description="Submit Kueue Job Example", formatter_class=argparse.RawTextHelpFormatter, ) parser.add_argument( "--version", help="Version of Kueue to install (if undefined, will install from master branch)", default=None, ) return parser def main(): """ Install Kueue and the Queue components. This will error if they are already installed. """ parser = get_parser() args, _ = parser.parse_known_args() install_kueue(args.version) def get_install_url(version): """ Get the install version. If a version is specified, use it. Otherwise install from the main branch. """ if version is not None: return f"https://github.com/kubernetes-sigs/kueue/releases/download/v{version}/manifests.yaml" return "https://github.com/kubernetes-sigs/kueue/config/default?ref=main" def install_kueue(version): """ Install Kueue of a particular version. """ print("⭐️ Installing Kueue...") url = get_install_url(version) with tempfile.NamedTemporaryFile(delete=True) as install_yaml: res = requests.get(url) assert res.status_code == 200 install_yaml.write(res.content) utils.create_from_yaml(api_client, install_yaml.name) if __name__ == "__main__": main() ``` -------------------------------- ### Sample Kueue-managed Pod Source: https://kueue.sigs.k8s.io/docs/tasks/run/plain_pods A complete example of a Pod configured for Kueue management and the command to deploy it. ```yaml apiVersion: v1 kind: Pod metadata: generateName: kueue-sleep- labels: kueue.x-k8s.io/queue-name: user-queue spec: containers: - name: sleep image: busybox command: - sleep args: - 3s resources: requests: cpu: 3 restartPolicy: OnFailure ``` ```bash # Create the pod kubectl create -f https://kueue.sigs.k8s.io/examples/pods-kueue/kueue-pod.yaml ``` -------------------------------- ### Stop workload example Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_stop/kueuectl_stop_workload Example command to stop a workload named my-workload. ```bash # Stop the workload kueuectl stop workload my-workload ``` -------------------------------- ### Create ClusterQueue with Cohort, Namespace Selector, and Strategy Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_create/kueuectl_create_clusterqueue Example demonstrating the creation of a ClusterQueue with advanced configurations including cohort, strict FIFO queuing strategy, and namespace selection. ```bash kueuectl create clusterqueue my-cluster-queue \ --cohort cohortname \ --queuing-strategy StrictFIFO \ --namespace-selector fooX=barX,fooY=barY \ --reclaim-within-cohort Any \ --preemption-within-cluster-queue LowerPriority ``` -------------------------------- ### Validate Setup Source: https://kueue.sigs.k8s.io/docs/tasks/dev/setup_multikueue_development_environment Verify that the cluster queue, admission check, and multi-kueue clusters are active. ```bash kubectl --context kind-manager get clusterqueue,admissioncheck,multikueuecluster ``` -------------------------------- ### Stop a LocalQueue example Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_stop/kueuectl_stop_localqueue Example command to stop a specific LocalQueue named my-localqueue. ```bash # Stop the localqueue kueuectl stop localqueue my-localqueue ``` -------------------------------- ### kueuectl create Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_create The create command is used to initialize new resources within the Kueue system. It supports dry-run strategies for testing requests without persistence. ```APIDOC ## CLI Command: kueuectl create ### Description Creates a new resource in the Kueue system. This is a base command that requires a specific resource type (e.g., localqueue, clusterqueue, resourceflavor). ### Options - **--dry-run** (string) - Optional - Must be "none", "server", or "client". If client, prints the object without sending. If server, submits the request without persisting. - **-h, --help** - Optional - Displays help information for the create command. ### Inherited Options - **--as** (string) - Optional - Username to impersonate. - **--as-group** (strings) - Optional - Groups to impersonate. - **--as-uid** (string) - Optional - UID to impersonate. - **--kubeconfig** (string) - Optional - Path to the kubeconfig file. - **-n, --namespace** (string) - Optional - Namespace scope for the request. - **-s, --server** (string) - Optional - Address and port of the Kubernetes API server. ### Example ```bash kueuectl create localqueue my-local-queue -c my-cluster-queue ``` ``` -------------------------------- ### Set up kubectl-kueue command completion script Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/installation Manually create and install a script for kubectl-kueue command completion. This involves creating a shell script and making it executable. ```bash cat <kubectl_complete-kueue #!/usr/bin/env sh # Call the __complete command passing it all arguments kubectl kueue __complete "$@" EOF chmod u+x kubectl_complete-kueue sudo mv kubectl_complete-kueue /usr/local/bin/kubectl_complete-kueue ``` -------------------------------- ### Run Kueue Installation Script Source: https://kueue.sigs.k8s.io/docs/tasks/run/python_jobs Executes the Python script to install Kueue. This can be run without arguments to install the latest version from the master branch, or with a --version flag to specify a particular release. ```bash python install-kueue-queues.py ``` -------------------------------- ### Pod Resource Request Example Source: https://kueue.sigs.k8s.io/docs/tasks/manage/administer_cluster_quotas Example of Pod resource requests that will be processed by the transformation configuration. ```yaml resources: requests: cpu: 1 memory: 100Gi limits: example.com/gpu-type1: 2 example.com/gpu-type2: 1 ``` -------------------------------- ### Configure Kueue Integration Source: https://kueue.sigs.k8s.io/docs/tasks/run/deployment Example configuration for enabling the deployment integration in Kueue v0.15 and earlier. ```yaml apiVersion: config.kueue.x-k8s.io/v1beta2 kind: Configuration integrations: frameworks: - "deployment" ``` -------------------------------- ### View CreatedWorkload event Source: https://kueue.sigs.k8s.io/docs/tasks/troubleshooting/troubleshooting_pods Example output confirming that Kueue has successfully observed all Pods in a group and created the corresponding Workload. ```text ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal CreatedWorkload 14s pod-kueue-controller Created Workload: my-namespace/my-pod-group ``` -------------------------------- ### Build kubectl-kueue from Source Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/installation Compile the kubectl-kueue binary from source code using 'make' and then move it to your system's PATH. Ensure you have the necessary build tools installed. ```bash make kueuectl sudo mv ./bin/kubectl-kueue /usr/local/bin/kubectl-kueue ``` -------------------------------- ### Example Pod List Output Source: https://kueue.sigs.k8s.io/docs/tasks/troubleshooting/troubleshooting_jobs The output displays the current status and readiness of each pod in the job. ```text NAME READY STATUS RESTARTS AGE my-job-0-nxmpx 1/1 Running 0 3m20s my-job-1-vgms2 1/1 Running 0 3m20s my-job-2-74gw7 1/1 Running 0 3m20s my-job-3-d4559 1/1 Running 0 3m20s my-job-4-pg75n 0/1 Pending 0 3m20s ``` -------------------------------- ### Automated MultiKueue TAS Setup Source: https://kueue.sigs.k8s.io/docs/tasks/dev/setup_multikueue_development_environment Execute the provided script to automatically provision a MultiKueue environment with Topology-Aware Scheduling enabled. ```bash git clone https://github.com/kubernetes-sigs/kueue.git cd kueue/examples/multikueue/dev ./setup-kind-multikueue-tas.sh ``` -------------------------------- ### Install and uninstall development version Source: https://kueue.sigs.k8s.io/docs/installation Commands to install or remove the latest development version of Kueue from the main branch. ```bash kubectl apply --server-side -k "github.com/kubernetes-sigs/kueue/config/default?ref=main" ``` ```bash kubectl delete -k "github.com/kubernetes-sigs/kueue/config/default?ref=main" ``` -------------------------------- ### Create Job YAML files Source: https://kueue.sigs.k8s.io/docs/tasks/manage/setup_wait_for_pods_ready Prepare job configuration files by substituting placeholders. Ensure the template file exists before running. ```bash sed 's/_ID_/1/g' job-template.yaml > /tmp/job1.yaml sed 's/_ID_/2/g' job-template.yaml > /tmp/job2.yaml ``` -------------------------------- ### Get LocalQueue Command Source: https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/commands/kueuectl_get/kueuectl_get_localqueue This command retrieves information about local queues. It acts as a pass-through to the kubectl get command. ```APIDOC ## GET /apis/kueue.x-k8s.io/v1beta1/namespaces/{namespace}/localqueues ### Description Retrieves a list of local queues within a specified namespace. ### Method GET ### Endpoint `/apis/kueue.x-k8s.io/v1beta1/namespaces/{namespace}/localqueues` ### Parameters #### Path Parameters - **namespace** (string) - Required - The namespace to list local queues from. #### Query Parameters - **labelSelector** (string) - Optional - A selector to filter results by labels. - **fieldSelector** (string) - Optional - A selector to filter results by fields. ### Response #### Success Response (200) - **items** (array) - A list of local queue objects. - **metadata** (object) - Standard Kubernetes object metadata. - **spec** (object) - Specification of the local queue. - **status** (object) - Current status of the local queue. #### Response Example ```json { "apiVersion": "kueue.x-k8s.io/v1beta1", "kind": "LocalQueueList", "items": [ { "metadata": { "name": "my-local-queue", "namespace": "default" }, "spec": { "clusterCapacity": { "cpu": "1000m" }, "namespaceSelector": {}, "queuingStrategy": { "type": "BestEffort" } }, "status": {} } ] } ``` ## See Also * `kueuectl get` - Display a resource ``` -------------------------------- ### Install Python Dependencies Source: https://kueue.sigs.k8s.io/docs/tasks/run/python_jobs Installs the necessary Python libraries for interacting with Kubernetes and Kueue. It is recommended to use a virtual environment. ```bash python -m venv env source env/bin/activate pip install kubernetes requests ``` -------------------------------- ### Provision Kind Cluster for Kueue Source: https://kueue.sigs.k8s.io/docs/tasks/manage/setup_failure_recovery Download the kind configuration and create a cluster for testing Kueue features. ```bash wget https://raw.githubusercontent.com/kubernetes-sigs/kueue/81cae0608cac3d457b8ecf8a0480e7994c2151c1/hack/kind-cluster.yaml kind create cluster --config hack/kind-cluster.yaml ```