### Install Kueue-Populator with Helm

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/tas.md

Install the Kueue-populator Helm chart, passing a custom topology definition file. The --wait flag ensures the installation completes before proceeding.

```shell
helm install kueue-populator oci://registry.k8s.io/kueue/charts/kueue-populator \
  --version 0.16.1 \
  --namespace kueue-system \
  --create-namespace \
  --wait \
  -f <topology-yaml-file>
```

--------------------------------

### Install Kueue Controller with Helm

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/tas.md

Install the Kueue controller using Helm. Ensure you specify the desired version and namespace.

```shell
$ helm install kueue oci://registry.k8s.io/kueue/charts/kueue --version="0.16.1" --create-namespace --namespace=kueue-system
```

--------------------------------

### Deploy llama.cpp example to kind

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/llamacpp.md

Execute this script to deploy the llama.cpp example to your kind cluster. This process involves building llama.cpp from source and downloading a model.

```shell
./docs/examples/llamacpp/dev/tasks/run-in-kind
```

--------------------------------

### Install LWS Chart from Source

Source: https://github.com/kubernetes-sigs/lws/blob/main/charts/lws/README.md

Installs the LWS chart from a local source directory using Helm v3.0+. Ensure you have cloned the repository and navigated to the charts directory.

```bash
git clone git@github.com:kubernetes-sigs/lws.git
cd charts
helm install lws lws --create-namespace --namespace lws-system
```

--------------------------------

### Download metrics-server Configuration

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/hpa.md

Download the latest metrics-server components configuration file. This is the first step in installing metrics-server.

```shell
wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml -O metrics-server.yaml
```

--------------------------------

### Install LeaderWorkerSet with Volcano Provider via Helm

Source: https://github.com/kubernetes-sigs/lws/blob/main/docs/examples/sample/gang-scheduling/README.md

Use this Helm command to install LeaderWorkerSet with gang scheduling enabled and Volcano as the provider.

```bash
helm install lws oci://ghcr.io/kubernetes-sigs/lws-charts/lws \
    --set gangSchedulingManagement.schedulerProvider=volcano
```

--------------------------------

### Configure Namespace for LeaderWorkerSet Installation

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/installation/_index.md

To install the LeaderWorkerSet controller in a namespace other than `lws-system`, clone the repository, change the `namespace` field in `config/default/kustomization.yaml` to your desired namespace, and then proceed with the installation.

```yaml
namespace: <your-namespace>
```

--------------------------------

### Helm Chart TLS Configuration Example

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/manage/prometheus.md

Example configuration for TLS settings within the Helm chart's values.yaml for Prometheus metrics. Ensure secrets reference cert-manager generated secrets.

```yaml
...
metrics:
  prometheusNamespace: monitoring
# tls configs for serviceMonitor
  serviceMonitor:
    tlsConfig:
      serverName: lws-controller-manager-metrics-service.lws-system.svc
      ca:
        secret:
          name: lws-metrics-server-cert
          key: ca.crt
      cert:
        secret:
          name: lws-metrics-server-cert
          key: tls.crt
      keySecret:
        name: lws-metrics-server-cert
        key: tls.key
```

--------------------------------

### Check SGLang Application Startup

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/sglang.md

Verify that the SGLang application has started successfully by checking its logs for the startup completion message.

```shell
kubectl logs sglang-0 |grep -C 2 -i "Application startup complete"
```

--------------------------------

### Install LeaderWorkerSet using e2e test suite

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/llamacpp.md

Install LeaderWorkerSet by running the end-to-end test suite. Ensure you have a cluster running and set USE_EXISTING_CLUSTER=true and KIND_CLUSTER_NAME to your cluster's name.

```shell
USE_EXISTING_CLUSTER=true KIND_CLUSTER_NAME=kind make test-e2e
```

--------------------------------

### Create a LeaderWorkerSet

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/concepts/_index.md

This example demonstrates the basic structure of a LeaderWorkerSet, defining replicas and worker templates. Ensure the container image and resources are correctly specified.

```yaml
apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: leaderworkerset-sample
spec:
  replicas: 3
  leaderWorkerTemplate:
    size: 4
    workerTemplate:
      spec:
        containers:
        - name: nginx
          image: nginxinc/nginx-unprivileged:1.27
          resources:
            limits:
              cpu: "100m"
            requests:
              cpu: "50m"
          ports:
          - containerPort: 8080
```

--------------------------------

### Two-Role Rollout Example Table

Source: https://github.com/kubernetes-sigs/lws/blob/main/keps/766-DisaggregatedSet/README.md

A step-by-step breakdown of a two-role rollout, illustrating the changes in old and new replicas for decode and prefill roles. This table demonstrates the N-dimensional coordination and capacity maintenance.

```plaintext
┌──────┬────────────┬─────────────┬────────────┬─────────────┬───────┬───────────────────────────────┐
│ STEP │ OLD DECODE │ OLD PREFILL │ NEW DECODE │ NEW PREFILL │ TOTAL │            ACTION             │
├──────┼────────────┼─────────────┼────────────┼─────────────┼───────┼───────────────────────────────┤
│ 0    │ 2          │ 5           │ 0          │ 0           │ 7     │ initial                       │
│ 1    │ 2          │ 5           │ 1          │ 2           │ 10    │ new decode +1, new prefill +2 │
│ 2    │ 2          │ 4           │ 1          │ 2           │ 9     │ old prefill -1                │
│ 3    │ 2          │ 3           │ 1          │ 2           │ 8     │ old prefill -1                │
│ 4    │ 2          │ 3           │ 2          │ 4           │ 11    │ new decode +1, new prefill +2 │
│ 5    │ 1          │ 2           │ 2          │ 4           │ 9     │ old decode -1, old prefill -1 │
│ 6    │ 1          │ 2           │ 2          │ 5           │ 10    │ new prefill +1                │
│ 7    │ 0          │ 0           │ 2          │ 5           │ 7     │ old decode -1, old prefill -2 │
└──────┴────────────┴─────────────┴────────────┴─────────────┴───────┴───────────────────────────────┘
```

--------------------------------

### Install Released Version with Helm (OCI)

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/installation/_index.md

Installs a specific released version of LeaderWorkerSet using Helm from an OCI registry. Ensure CHART_VERSION is set.

```shell
CHART_VERSION=0.9.0
helm install lws oci://registry.k8s.io/lws/charts/lws \
  --version=$CHART_VERSION \
  --namespace lws-system \
  --create-namespace \
  --wait --timeout 300s
```

--------------------------------

### Interact with the llama.cpp service

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/llamacpp.md

After deployment, you can interact with the llama.cpp service using the clichat pod. This example shows how to run the pod and perform a basic chat interaction.

```shell
+ kubectl apply --server-side -f k8s/lws.yaml
leaderworkerset.leaderworkerset.x-k8s.io/llamacpp-llama3-8b-instruct-bartowski-q5km serverside-applied
service/llamacpp serverside-applied
+ kubectl run clichat --image=clichat:latest --rm=true -it --image-pull-policy=IfNotPresent --env=LLM_ENDPOINT=http://llamacpp:80
If you don't see a command prompt, try pressing enter.
What is the capital of France?
        The capital of France is Paris.
```

--------------------------------

### Install Released Version with Helm (Tarball)

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/installation/_index.md

Installs a specific released version of LeaderWorkerSet using Helm from a tarball URL. Ensure VERSION is set.

```shell
VERSION=v0.9.0
helm install lws https://github.com/kubernetes-sigs/lws/releases/download/$VERSION/lws-chart-$VERSION.tgz \
  --namespace lws-system \
  --create-namespace \
  --wait --timeout 300s
```

--------------------------------

### Infinite StatefulSet Creation Loop Example

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/troubleshooting/_index.md

This output illustrates an infinite loop of pod creation within a StatefulSet, often seen in older Kubernetes versions lacking the StatefulSetStartOrdinal feature gate.

```text
vllm-alpha-distributed-serving-0                             0/1     Running             0          10s
vllm-alpha-distributed-serving-0-0                           1/1     Running             0          10s
vllm-alpha-distributed-serving-0-0-0                         0/1     ContainerCreating   0          10s
vllm-alpha-distributed-serving-0-0-0-0                       0/1     ContainerCreating   0          10s
vllm-alpha-distributed-serving-0-0-0-0-0                     0/1     ContainerCreating   0          10s
vllm-alpha-distributed-serving-0-0-0-0-0-0                   0/1     ContainerCreating   0          10s
```

--------------------------------

### Install Released Version with Kubectl

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/installation/_index.md

Installs a specific released version of LeaderWorkerSet using kubectl apply. Ensure the VERSION variable is set correctly.

```shell
VERSION=v0.9.0
kubectl apply --server-side -f https://github.com/kubernetes-sigs/lws/releases/download/$VERSION/manifests.yaml
```

--------------------------------

### Install Latest Development Version

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/installation/_index.md

Installs the latest development version of LeaderWorkerSet from the main branch using kubectl apply with kustomization. The controller will run in the 'lws-system' namespace.

```shell
kubectl apply --server-side -k github.com/kubernetes-sigs/lws/config/default?ref=main
```

--------------------------------

### LeaderWorkerSet Configuration Example

Source: https://github.com/kubernetes-sigs/lws/blob/main/keps/820-distributed-preflight-check/README.md

Configure LeaderWorkerSet to limit group restarts and enable publishing of not-ready addresses for init-phase DNS resolution.

```yaml
apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: serving
spec:
  leaderWorkerTemplate:
    restartPolicy: RecreateGroupOnPodRestart
    maxGroupRestarts: 3
  networkConfig:
    subdomainPolicy: UniquePerReplica
    publishNotReadyAddresses: true
```

--------------------------------

### Check vLLM Model Loading Logs

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/vllm.md

Retrieves logs from a vLLM pod to verify model weights loading, useful for debugging distributed inference setup.

```shell
kubectl logs vllm-0 |grep -i "Loading model weights took"
```

--------------------------------

### Get endpoints for services

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/llamacpp.md

Check the endpoints for your services to verify connectivity. The 'llamacpp' service targets the Leader pod, distributing work to the Workers.

```shell
> kubectl get endpoints
NAME                                         ENDPOINTS                                         AGE
kubernetes                                   192.168.8.3:6443                                  16m
llamacpp                                     10.244.0.90:8080                                  4m
llamacpp-llama3-8b-instruct-bartowski-q5km   10.244.0.90,10.244.0.91,10.244.0.92 + 2 more...   3m59s
```

--------------------------------

### Get pods in the cluster

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/llamacpp.md

Use kubectl to view the status of pods running in your cluster. This helps verify that the llama.cpp components have been deployed successfully.

```shell
> kubectl get pods
NAME                                             READY   STATUS    RESTARTS   AGE
llamacpp-llama3-8b-instruct-bartowski-q5km-0     1/1     Running   0          2m42s
llamacpp-llama3-8b-instruct-bartowski-q5km-0-1   1/1     Running   0          2m42s
llamacpp-llama3-8b-instruct-bartowski-q5km-0-2   1/1     Running   0          2m42s
llamacpp-llama3-8b-instruct-bartowski-q5km-0-3   1/1     Running   0          2m42s
llamacpp-llama3-8b-instruct-bartowski-q5km-0-4   1/1     Running   0          2m42s
```

--------------------------------

### Get or Create Controller Revision (Go)

Source: https://github.com/kubernetes-sigs/lws/blob/main/keps/238-controller-revision/README.md

Retrieves an existing controller revision or creates a new one if it doesn't exist. It handles cases where the leader statefulset might not be present.

```Go
func (r *LeaderWorkerSetReconciler) getOrCreateRevisionIfNonExist(leaderSts, lws) (*appsv1.ControllerRevision) {
	revisionKey := ""
	if leaderSts != nil {
		revisionKey = GetRevisionKey(sts)
	}
	if stsRevision, err := GetRevision(ctx, r.Client, lws, revisionKey); stsRevision != nil || err != nil {
		return stsRevision
	}
	revision := NewRevision(ctx, r.Client, lws, revisionKey)
	return CreateRevision(ctx, r.Client, revision, lws)
}
```

--------------------------------

### Build and Deploy LeaderWorkerSet from Source

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/installation/_index.md

Clone the repository, navigate to the directory, and build the Docker image, push it to a registry, and deploy it to your cluster. Replace `<registry>/<project>` with your image registry and project name.

```sh
git clone https://github.com/kubernetes-sigs/lws.git
cd lws
IMAGE_REGISTRY=<registry>/<project> make image-push deploy
```

--------------------------------

### Create PodGroup If Not Exists in Go

Source: https://github.com/kubernetes-sigs/lws/blob/main/keps/407-gang-scheduling/README.md

Implement a function to create a PodGroup if it doesn't exist, handling different startup policies and setting owner references. Ensure MinMember and MinResources are correctly configured based on the StartupPolicy.

```go
func (p *CustomSchedulerProvider) CreatePodGroupIfNotExists(ctx context.Context, lws *leaderworkerset.LeaderWorkerSet, leaderPod *corev1.Pod) error {
	var pg CustomPodGroup
	pgName := leaderPod.Annotations[CustomGroupNameAnnotationKey]

	if err := p.client.Get(ctx, types.NamespacedName{Name: pgName, Namespace: lws.Namespace}, &pg); err != nil {
		if client.IgnoreNotFound(err) != nil {
			return err
		}
		
		pg = CustomPodGroup{
			ObjectMeta: metav1.ObjectMeta{
				Name:      pgName,
				Namespace: lws.Namespace,
			},
			Spec: CustomPodGroupSpec{
				// Default behavior: all pods must be scheduled together
				MinMember:    *lws.Spec.LeaderWorkerTemplate.Size,
				MinResources: p.calculateMinResources(lws),
			},
		}

		// Handle LeaderReady startup policy
		if lws.Spec.StartupPolicy == leaderworkerset.LeaderReadyStartupPolicy {
			// Allow leader to be scheduled independently
			pg.Spec.MinMember = 1
			// MinResources still represents total replica's resource requirements
			// This ensures that even though only leader is required initially,
			// the scheduler can understand how many resources the replica needs
		}

		// Set optional scheduler-specific configurations
		...

		// Set OwnerReference to leader pod for lifecycle management
		err = ctrl.SetControllerReference(leaderPod, &pg, p.client.Scheme())
		if err != nil {
			return err
		}

		if err = p.client.Create(ctx, &pg); err != nil {
			return err
		}
	}

	return nil
}

```

--------------------------------

### Uninstall Development Version

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/installation/_index.md

Uninstalls LeaderWorkerSet when installed from the development branch using kubectl delete with kustomization.

```shell
kubectl delete -k github.com/kubernetes-sigs/lws/config/default
```

--------------------------------

### Create a kind cluster

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/llamacpp.md

Use this command to create a local Kubernetes cluster with kind.

```shell
kind create cluster
```

--------------------------------

### Wait for LeaderWorkerSet Availability

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/installation/_index.md

Waits for the LeaderWorkerSet controller manager to become available after installation. This command checks the 'available' condition.

```shell
kubectl wait deploy/lws-controller-manager -n lws-system --for=condition=available --timeout=5m
```

--------------------------------

### Enable RecreateGroupAfterStart Policy (v0.9+)

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/concepts/failure-handling/_index.md

Enables a restart policy that recreates the entire pod group only if no pods are pending. This prevents interruptions during large image pulls. Use this on version 0.9 and later.

```yaml
spec:
  leaderWorkerTemplate:
    restartPolicy: RecreateGroupAfterStart
```

--------------------------------

### Apply Kustomize Configuration

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/manage/cert_manager.md

Apply the Kustomize configurations to your cluster to enable cert-manager integration. Ensure internal certificate management is disabled and cert-manager specific configurations are uncommented.

```bash
kubectl apply --server-side -k config/default
```

--------------------------------

### Uninstall Released Version with Helm

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/installation/_index.md

Uninstalls LeaderWorkerSet when installed via Helm. This command targets the 'lws' release in the 'lws-system' namespace.

```shell
helm uninstall lws --namespace lws-system
```

--------------------------------

### LeaderWorkerSet with LeaderReady Startup Policy

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/concepts/_index.md

Configure the startup policy to ensure the worker StatefulSet is created only after the leader Pod is ready. This guarantees leader readiness before worker deployment.

```yaml
apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: leaderworkerset-sample
spec:
  startupPolicy: LeaderReady
  replicas: 3
  leaderWorkerTemplate:
  ...
```

--------------------------------

### Enable RecreateGroupAfterStart Policy (v0.8 Experimental)

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/concepts/failure-handling/_index.md

Enables the experimental RecreateGroupAfterStart feature on version 0.8 using an annotation. This policy recreates the pod group only if no pods are pending, allowing large image pulls to complete.

```yaml
apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: leaderworkerset-sample
  annotations:
    leaderworkerset.sigs.k8s.io/experimental-recreate-group-after-start: true
```

--------------------------------

### Tagging a New Release

Source: https://github.com/kubernetes-sigs/lws/blob/main/RELEASE.md

Use this command to create a signed Git tag for a new release. Replace $VERSION with the actual release version.

```bash
git tag -s $VERSION
```

--------------------------------

### Pushing a New Tag

Source: https://github.com/kubernetes-sigs/lws/blob/main/RELEASE.md

After creating a tag, push it to the repository. Replace $VERSION with the actual release version.

```bash
git push $VERSION
```

--------------------------------

### Get HPA Status

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/hpa.md

Use kubectl to retrieve the status of Horizontal Pod Autoscalers. This command helps verify if the HPA is correctly configured and reporting metrics.

```shell
kubectl get hpa
```

--------------------------------

### Render Static File Shortcode

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/layouts/shortcodes/include.html

Use this shortcode to display static files. It supports rendering plain text or highlighting code files. Specify the file path and optionally the language for highlighting. If the file is not found and 'draft' is set to 'true', a message is displayed; otherwise, a compile error is raised.

```gohtml
{{/* Handle the "file" and "lang" named parameter or two unnamed parameter as the filepath and coding language*/}}{{/* The first parameter should be a relative path to the "static" directory. Ex: "examples/jobs/sample-job.yaml" */}}{{ if .IsNamedParams }} {{ $.Scratch.Set "fparameter" ( .Get "file" ) }} {{ $.Scratch.Set "lang" ( .Get "lang") }} {{ else }} {{ $.Scratch.Set "fparameter" ( .Get 0 ) }} {{ $.Scratch.Set "lang" ( .Get 1 ) }} {{ end }}{{/* If coding language are specified, then the static file is a code file; If not, then the static are non-code file. */}}{{ if eq ($.Scratch.Get "lang") "" }} {{ $.Scratch.Set "iscode" "false"}} {{ else }} {{ $.Scratch.Set "iscode" "true" }} {{ end }}{{/* `fparameter is a relative path of the static file from the `static` directory. For exampl, `/example/simple/exclusive-placement.yaml` */}}{{ $.Scratch.Set "filepath" "/" }} {{ $.Scratch.Add "filepath" "static/" }} {{ $.Scratch.Add "filepath" ($.Scratch.Get "fparameter") }}{{/* If the file exists, read it and highlight it if it's code. Throw a compile error or print an error on the page if the file is not found */}}{{ if fileExists ($.Scratch.Get "filepath") }} {{ $filename := (path.Split ($.Scratch.Get "filepath")).File }} {{ $link := printf "/%s" ($.Scratch.Get "fparameter") | safeURL }} 
[{{ $filename }}]({{ $link }}) {{ if eq ($.Scratch.Get "iscode") "true" }} {{- highlight ($.Scratch.Get "filepath" | readFile | htmlUnescape | safeHTML ) ($.Scratch.Get "lang") "" -}} {{ else }} {{- $.Scratch.Get "filepath" | os.ReadFile | .Page.RenderString | safeHTML -}} {{ end }} 
{{ else if eq (.Get "draft") "true" }} 
**_The file `{{ $.Scratch.Get "filepath" }}` was not found._** 
{{ else }}{{- errorf "Shortcode %q: file %q not found at %s" .Name ($.Scratch.Get "filepath") .Position -}}{{ end }}
```

--------------------------------

### LeaderWorkerSet Implementation Details

Source: https://github.com/kubernetes-sigs/lws/blob/main/keps/135-startup-policy/README.md

Details the implementation strategy for the LeaderWorkerSet, focusing on the 'LeaderReady' startup policy and webhook/controller logic.

```APIDOC
## Implementation Details

### Startup Policy Logic

- **LeaderCreated**: No logic changes are required when `spec.startupPolicy` is set to `LeaderCreated`.
- **LeaderReady**: Requires specific implementation logic to ensure worker StatefulSets are created only after the leader pod is ready.

### LWS Webhook

- The lws webhook is responsible for setting the default value of `spec.startupPolicy` to `LeaderCreated` if not explicitly provided.

### Pod Controller Logic for LeaderReady

When `spec.startupPolicy` is `LeaderReady`, the Pod controller's reconcile logic involves the following steps:

1.  **LeaderWorkerSet Creation**: A new LeaderWorkerSet is created.
2.  **Leader StatefulSet Creation**: The leader StatefulSet is created by the lws controller, but its pod is not yet ready.
3.  **Pod Status Monitoring**: The Pod controller watches for `Update` events for the leader pod. It uses the `IsPodReady(pod *v1.Pod) bool` function (from `k8s.io/kubernetes/pkg/api/v1/pod`) to check the leader pod's status.
    - If `IsPodReady(pod)` returns `false`, no action is taken.
    - If `IsPodReady(pod)` returns `true`, the controller proceeds to construct the worker StatefulSet configuration and create the worker StatefulSet.
4.  **Worker StatefulSet Creation**: Once the leader pods created by the leader StatefulSet are ready, the controller proceeds with creating the worker StatefulSets.
```

--------------------------------

### Get LeaderWorkerSet HPA Pod Selector

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/hpa.md

Retrieve the HPA pod selector from the LeaderWorkerSet status using kubectl. This is useful for understanding how HPA identifies the pods it should manage.

```shell
kubectl get leaderworkerset leaderworkerset-sample -o jsonpath='{.status.hpaPodSelector}'
```

--------------------------------

### Verify LWS Controller Pods

Source: https://github.com/kubernetes-sigs/lws/blob/main/charts/lws/README.md

Checks the status of the LWS controller manager deployment in the 'lws-system' namespace. This command helps verify that the controller pods are running correctly after installation.

```bash
kubectl get deploy -n lws-system
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
lws-system-controller-manager   1/1     1            1           14s
```

--------------------------------

### Enable Gang Scheduling in LeaderWorkerSet ConfigMap

Source: https://github.com/kubernetes-sigs/lws/blob/main/docs/examples/sample/gang-scheduling/README.md

Update the LWS manager ConfigMap to enable gang scheduling and specify Volcano as the provider. This configuration is applied when installing via kubectl.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: lws-manager-config
  namespace: lws-system
data:
  controller_manager_config.yaml: |
    apiVersion: config.lws.x-k8s.io/v1alpha1
    kind: Configuration
    leaderElection:
      leaderElect: true
    internalCertManagement:
      enable: true
    # Add gang scheduling configuration
    gangSchedulingManagement:
      schedulerProvider: volcano
```

--------------------------------

### Define Startup Policy Types for LeaderWorkerSet

Source: https://github.com/kubernetes-sigs/lws/blob/main/keps/135-startup-policy/README.md

Defines the possible startup policies for LeaderWorkerSet, specifying when worker stateful sets should be created relative to the leader stateful set.

```go
type StartupPolicyType string

const (
	// LeaderReady will create worker statefulset after leader statefulset is ready.
	LeaderReadyStartupPolicy StartupPolicyType = "LeaderReady"

	// LeaderCreated will create worker statefulsets immediately while leader pod is created.
	LeaderCreatedStartupPolicy StartupPolicyType = "LeaderCreated"
)
```

--------------------------------

### Define BaseResourceProvider Interface

Source: https://github.com/kubernetes-sigs/lws/blob/main/keps/407-gang-scheduling/README.md

Interface for managing base resources like headless services and resource claims for each replica.

```go
type BaseResourceProvider interface {
    CreateHeadlessService(ctx context.Context, lws *leaderworkerset.LeaderWorkerSet) error

    CreateResourceClaim(ctx context.Context, lws *leaderworkerset.LeaderWorkerSet) error
}
```

--------------------------------

### Enable Volcano RBAC Component in Kustomization

Source: https://github.com/kubernetes-sigs/lws/blob/main/docs/examples/sample/gang-scheduling/README.md

Uncomment the Volcano components section in `config/default/kustomization.yaml` to grant the LWS controller necessary permissions for Volcano PodGroups. This is part of the kubectl installation process.

```yaml
# [VOLCANO] To enable Volcano gang scheduling RBAC, uncomment the following.
components:
- ../components/volcano
```

--------------------------------

### Configure Volcano Scheduler Annotations via LeaderWorkerSet Metadata

Source: https://github.com/kubernetes-sigs/lws/blob/main/docs/examples/sample/gang-scheduling/README.md

Annotations starting with `volcano.sh/` in the LeaderWorkerSet's metadata are inherited by the auto-generated PodGroup, allowing control over Volcano-specific scheduling behaviors.

```yaml
apiVersion: lws.x-k8s.io/v1alpha1
kind: LeaderWorkerSet
metadata:
  name: my-leaderworkerset
  annotations:
    volcano.sh/priority-class: "high"
    volcano.sh/min-available: "2"
spec:
  replicas: 3
  leaderTemplate:
    spec:
      containers:
      - name: leader
        image: "" # Replace with your image
  workerTemplate:
    spec:
      containers:
      - name: worker
        image: "" # Replace with your image
```

--------------------------------

### Deploy vLLM on TPU with LeaderWorkerSet

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/vllm.md

Applies the LeaderWorkerSet configuration for vLLM on TPU. Requires a GKE cluster with TPU slices and your Hugging Face token.

```shell
export HF_TOKEN=<your-hf-token>
curl https://raw.githubusercontent.com/kubernetes-sigs/lws/refs/heads/main/docs/examples/vllm/TPU/lws.yaml -s | envsubst | kubectl apply -f -
```

--------------------------------

### Deploy metrics-server

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/hpa.md

Apply the modified metrics-server configuration to your Kubernetes cluster. This command deploys the metrics-server.

```shell
kubectl apply -f metrics-server.yaml
```

--------------------------------

### Check metrics-server Pod Status

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/hpa.md

Verify that the metrics-server pod is running correctly in the kube-system namespace. This confirms the deployment was successful.

```shell
kubectl get pods -n kube-system | grep metrics
```

--------------------------------

### StatefulSet Name Length Error

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/troubleshooting/_index.md

This error message indicates that a worker pod failed to start due to an invalid label value, stemming from an LWS object name that is too long and exceeds Kubernetes StatefulSet naming constraints.

```text
Pod "<worker-pod-name>" is invalid: metadata.labels: Invalid value: <worker-sts-name>-<10-character-hash>": must be no more than 63 characters
```

--------------------------------

### Deploy vLLM on GPU with LeaderWorkerSet

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/vllm.md

Applies the LeaderWorkerSet configuration for vLLM on GPU. Ensure you have your Hugging Face token set.

```shell
export HF_TOKEN=<your-hf-token>
curl https://raw.githubusercontent.com/kubernetes-sigs/lws/refs/heads/main/docs/examples/vllm/GPU/lws.yaml -s | envsubst | kubectl apply -f -
```

--------------------------------

### Adjust Subgroup Index for TPU Environment Variables

Source: https://github.com/kubernetes-sigs/lws/blob/main/keps/257-Subgroup-leader-only/README.md

This Go code snippet adjusts the subgroup index when injecting TPU environment variables. If the pod is leader-only and not the first subgroup, the index is decremented to maintain correct calculations for start and end indices.

```go
// Take worker 1 as an example with subGroupSize set to 2. It will now have subGroupIndex set to 1, which means that the start will be 3, and end will be 4. 
// Need to shift it down by 1 for the calculation to still work out.
if leaderOnlyType && subGroupIndex != 0 {
    subGroupIndex -= 1
}
start := subGroupSize*subGroupIndex + 1 
end := subGroupSize * (subGroupIndex + 1)
```

--------------------------------

### Define GKE Topology Levels for Kueue

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/tas.md

Configure Kueue's populator with topology levels specific to Google Kubernetes Engine (GKE) to enable topology-aware scheduling.

```shell
kueuePopulator:
  config:
    topology:
      levels:
        - nodeLabel: "cloud.google.com/gce-topology-block"
        - nodeLabel: "cloud.google.com/gce-topology-subblock"
        - nodeLabel: "cloud.google.com/gce-topology-host"
        - nodeLabel: "kubernetes.io/hostname"
    resourceFlavor:
      nodeLabels:
        cloud.google.com/gke-gpu: "true"
```

--------------------------------

### Send Completion Request to vLLM

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/vllm.md

Sends a sample completion request to the vLLM service via the forwarded local port. Adjust model and prompt as needed.

```shell
curl http://localhost:8080/v1/completions \
-H "Content-Type: application/json" \
-d '{
    "model": "meta-llama/Meta-Llama-3.1-405B-Instruct",
    "prompt": "San Francisco is a",
    "max_tokens": 7,
    "temperature": 0
}'
```

--------------------------------

### Serve Model Request with TensorRT-LLM

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/tensorrt-llm.md

Sends a POST request to the TensorRT-LLM service running on localhost:8000 to generate text. The request includes a user prompt and generation parameters like temperature and max_tokens.

```shell
$ USER_PROMPT="I'm new to coding. If you could only recommend one programming language to start with, what would it be and why?"
$ curl -X POST localhost:8000/v2/models/ensemble/generate   -H "Content-Type: application/json"   -d @- <<EOF
{
    "text_input": "<start_of_turn>user\n${USER_PROMPT}<end_of_turn>\n",
    "temperature": 0.9,
    "max_tokens": 128
}
EOF
```

--------------------------------

### Test Pod Metrics Collection

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/hpa.md

Test if metrics-server can collect and display CPU and memory usage for pods. This command should return resource utilization data for pods.

```shell
kubectl top pods
```

--------------------------------

### Troubleshoot metrics-server Logs

Source: https://github.com/kubernetes-sigs/lws/blob/main/site/content/en/docs/examples/hpa.md

Retrieve logs from the metrics-server deployment to diagnose any issues. This is useful for identifying common problems like certificate errors or network issues.

```shell
kubectl logs -n kube-system deployment/metrics-server
```