### Install Kamaji with PostgreSQL datastore and create a TenantControlPlane

Source: https://kamaji.clastix.io/guides/alternative-datastore

Install Kamaji with a PostgreSQL datastore: deploy the PostgreSQL resources, apply the sample Datastore manifest, verify it is not ready until the operator is installed, install the operator with the required Helm values, then create and clean up a TenantControlPlane that uses the datastore. The Datastore stays not ready until the Kamaji chart is installed because no operator is running to reconcile it. The TenantControlPlane manifest sets dataStore to postgresql-gold, which must match the Datastore resource name.

```bash
make -C ./deploy/kine/postgresql/ postgresql NAME=gold

```

```bash
kubectl apply -f ./config/samples/kamaji_v1alpha1_datastore_postgresql_gold.yaml 

```

```bash
kubectl get datastores
NAME                 DRIVER       READY   AGE
postgresql-gold   PostgreSQL              18s

```

```bash
helm install kamaji ./charts/kamaji -n kamaji-system --create-namespace \
  --set kamaji-etcd.deploy=false \
  --set datastore.driver=PostgreSQL \
  --set datastore.endpoints[0]=postgres-gold-rw.postgres-system.svc:5432 \
  --set datastore.basicAuth.usernameSecret.name=postgres-gold-superuser \
  --set datastore.basicAuth.usernameSecret.namespace=postgres-system \
  --set datastore.basicAuth.usernameSecret.keyPath=username \
  --set datastore.basicAuth.passwordSecret.name=postgres-gold-superuser \
  --set datastore.basicAuth.passwordSecret.namespace=postgres-system \
  --set datastore.basicAuth.passwordSecret.keyPath=password \
  --set datastore.tlsConfig.certificateAuthority.certificate.name=postgres-gold-ca \
  --set datastore.tlsConfig.certificateAuthority.certificate.namespace=postgres-system \
  --set datastore.tlsConfig.certificateAuthority.certificate.keyPath=ca.crt \
  --set datastore.tlsConfig.certificateAuthority.privateKey.name=postgres-gold-ca \
  --set datastore.tlsConfig.certificateAuthority.privateKey.namespace=postgres-system \
  --set datastore.tlsConfig.certificateAuthority.privateKey.keyPath=ca.key \
  --set datastore.tlsConfig.clientCertificate.certificate.name=postgres-gold-root-cert \
  --set datastore.tlsConfig.clientCertificate.certificate.namespace=postgres-system \
  --set datastore.tlsConfig.clientCertificate.certificate.keyPath=tls.crt \
  --set datastore.tlsConfig.clientCertificate.privateKey.name=postgres-gold-root-cert \
  --set datastore.tlsConfig.clientCertificate.privateKey.namespace=postgres-system \
  --set datastore.tlsConfig.clientCertificate.privateKey.keyPath=tls.key

```

```bash
NAME                 DRIVER       READY   AGE
postgresql-gold   PostgreSQL   true    4m40s

```

```bash
cat > test-tenant-gold.yaml <<EOF 
apiVersion: kamaji.clastix.io/v1alpha1
kind: TenantControlPlane
metadata:
  name: k8s-133
  labels:
    tenant.clastix.io: k8s-133
spec:
  dataStore: postgresql-gold #this should match the Datastore's resource NAME
  controlPlane:
    deployment:
      replicas: 2
    service:
      serviceType: LoadBalancer
  kubernetes:
    version: "v1.33.0"
    kubelet:
      configurationJSONPatches:
        - op: add
          path: /featureGates
          value:
            KubeletCrashLoopBackOffMax: false
            KubeletEnsureSecretPulledImages: false
        - op: replace
          path: /cgroupDriver
          value: systemd
  networkProfile:
    port: 6443
  addons:
    coreDNS: {}
    kubeProxy: {}
    konnectivity:
      server:
        port: 8132
      agent:
        mode: DaemonSet
EOF

kubectl apply -f test-tenant-gold.yaml

```

```bash
kubectl delete -f test-tenant-gold.yaml
kubectl delete -f ./config/samples/kamaji_v1alpha1_datastore_postgresql_gold.yaml 
make -C ./deploy/kine/postgresql postgresql-destroy

```

--------------------------------

### Watch cert-manager installation progress

Source: https://kamaji.clastix.io/getting-started/kamaji-kind

Lists all pods across all namespaces to monitor the cert-manager installation progress.

```bash
kubectl get pods -Aw

```

--------------------------------

### Install KubeVirt

Source: https://kamaji.clastix.io/cluster-api/external-cluster

Fetch the latest stable KubeVirt version and apply the operator and custom resource manifests. This installs KubeVirt on the cluster.

```bash
export VERSION=$(curl -s "https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt")
kubectl apply -f "https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/kubevirt-operator.yaml"
kubectl apply -f "https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/kubevirt-cr.yaml"
```

--------------------------------

### Install alternative datastore drivers with make

Source: https://kamaji.clastix.io/guides/alternative-datastore

Run these make recipes on the Management Cluster to install the corresponding Datastore driver. Each command targets a different backend: MySQL, PostgreSQL, or NATS.

```shell
$ make -C deploy/kine/mysql mariadb
```

```shell
$ make -C deploy/kine/postgresql postgresql
```

```shell
$ make -C deploy/kine/nats nats
```

--------------------------------

### Install MetalLB with kubectl apply

Source: https://kamaji.clastix.io/getting-started/kamaji-kind

Applies the MetalLB manifest to install MetalLB on the cluster. This command installs all necessary resources.

```bash
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.15.3/config/manifests/metallb-native.yaml

```

--------------------------------

### Install the Tenant Cluster as Helm Release with cluster-api-kamaji-proxmox

Source: https://kamaji.clastix.io/cluster-api/proxmox-infra-provider

Install the Tenant Cluster as a Helm release using the cluster-api-kamaji-proxmox chart. The my-values.yaml file contains the configuration values for the Tenant Cluster.

```bash
helm repo add clastix https://clastix.github.io/cluster-api-kamaji-proxmox
helm repo update
helm install sample clastix/cluster-api-kamaji-proxmox \
    --set cluster.name=sample \
    --namespace default \
    --values my-values.yaml

```

--------------------------------

### Install Proxmox VE infrastructure provider with clusterctl

Source: https://kamaji.clastix.io/cluster-api/proxmox-infra-provider

Initialize the management cluster with the Proxmox VE infrastructure provider.

```bash
clusterctl init --infrastructure proxmox

```

--------------------------------

### Install Kamaji Controller via Helm

Source: https://kamaji.clastix.io/getting-started/kamaji-aws

Install the Kamaji controller using the preferred Helm chart method. Adds the Clastix repository and installs the latest stable release in the kamaji-system namespace.

```bash
helm repo add clastix https://clastix.github.io/charts
helm repo update
helm install kamaji clastix/kamaji -n kamaji-system --create-namespace --version 0.0.0+latest

```

--------------------------------

### Two-Step Kamaji Installation (CRDs then Operator)

Source: https://kamaji.clastix.io/getting-started/kamaji-aws

Two-step installation for separate CRD management (e.g., GitOps or policy requirements). First installs the kamaji-crds chart, then installs the Kamaji operator chart without CRDs.

```bash
# Step 1: Install CRDs
helm install kamaji-crds clastix/kamaji-crds -n kamaji-system --create-namespace --version 0.0.0+latest

# Step 2: Install Kamaji operator
helm install kamaji clastix/kamaji -n kamaji-system --version 0.0.0+latest

```

--------------------------------

### Install etcd DataStore with Helm

Source: https://kamaji.clastix.io/reference/benchmark

Installs etcd as a DataStore using the clastix/kamaji-etcd chart into the kamaji-etcd namespace, enabling the service monitor and datastore. Use this to set up the etcd datastore for Kamaji.

```bash
helm upgrade --install etcd-01 clastix/kamaji-etcd --namespace kamaji-etcd --create-namespace --set "serviceMonitor.enabled=true" --set "datastore.enabled=true"
```

--------------------------------

### Install cert-manager with Helm

Source: https://kamaji.clastix.io/reference/benchmark

Installs cert-manager from the Bitnami chart into the certmanager-system namespace, with node selectors and tolerations for the infra node group and CRD installation enabled. Use this before installing Kamaji to provide certificate management.

```bash
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm upgrade --install cert-manager bitnami/cert-manager --namespace certmanager-system --create-namespace \
    --set cainjector.nodeSelector."eks\.amazonaws\.com/nodegroup"=infra \
    --set "cainjector.tolerations[0].key=pool" \
    --set "cainjector.tolerations[0].operator=Equal" \
    --set "cainjector.tolerations[0].value=infra" \
    --set "cainjector.tolerations[0].effect=NoSchedule" \
    --set "cainjector.tolerations[1].key=pool" \
    --set "cainjector.tolerations[1].operator=Equal" \
    --set "cainjector.tolerations[1].value=infra" \
    --set "cainjector.tolerations[1].effect=NoExecute" \
    --set controller.nodeSelector."eks\.amazonaws\.com/nodegroup"=infra \
    --set "controller.tolerations[0].key=pool" \
    --set "controller.tolerations[0].operator=Equal" \
    --set "controller.tolerations[0].value=infra" \
    --set "controller.tolerations[0].effect=NoExecute" \
    --set "controller.tolerations[1].key=pool" \
    --set "controller.tolerations[1].operator=Equal" \
    --set "controller.tolerations[1].value=infra" \
    --set "controller.tolerations[1].effect=NoSchedule" \
    --set webhook.nodeSelector."eks\.amazonaws\.com/nodegroup"=infra \
    --set "webhook.tolerations[0].key=pool" \
    --set "webhook.tolerations[0].operator=Equal" \
    --set "webhook.tolerations[0].value=infra" \
    --set "webhook.tolerations[0].effect=NoSchedule" \
    --set "webhook.tolerations[1].key=pool" \
    --set "webhook.tolerations[1].operator=Equal" \
    --set "webhook.tolerations[1].value=infra" \
    --set "webhook.tolerations[1].effect=NoExecute" \
    --set "installCRDs=true"
```

--------------------------------

### Install Kamaji with Helm

Source: https://kamaji.clastix.io/getting-started/kamaji-kind

Installs Kamaji using Helm with the specified namespace and version. The 'resources=null' setting disables resource limits.

```bash
helm upgrade --install kamaji clastix/kamaji \
  --namespace kamaji-system \
  --create-namespace \
  --set 'resources=null' \
  --version 0.0.0+latest

```

--------------------------------

### Verify Kamaji installation with kubectl get pods

Source: https://kamaji.clastix.io/getting-started/kamaji-generic

Verifies that Kamaji and its components are running. The expected output shows the kamaji-etcd pods and the kamaji controller pod in Running state.

```bash
kubectl -n kamaji-system get pods

```

```bash
kubectl -n kamaji-system get pods
NAME                         READY   STATUS      RESTARTS      AGE
kamaji-etcd-0                1/1     Running     0             50s
kamaji-etcd-1                1/1     Running     0             60s
kamaji-etcd-2                1/1     Running     0             90s
kamaji-7949578bfb-lj44p      1/1     Running     0             12s

```

--------------------------------

### Install primary Datastore with Helm

Source: https://kamaji.clastix.io/guides/datastore-overrides

Installs the primary Datastore using the kamaji-etcd Helm chart, disabling self-signed certificates and enabling cert-manager with the self-signed ClusterIssuer.

```shell
 helm install etcd-primary clastix/kamaji-etcd -n kamaji-etcd --create-namespace \
   --set selfSignedCertificates.enabled=false \
   --set certManager.enabled=true \
   --set certManager.issuerRef.kind=ClusterIssuer \
   --set certManager.issuerRef.name=self-signed

```

--------------------------------

### Install OpenStack Resource Controller (ORC)

Source: https://kamaji.clastix.io/cluster-api/openstack-infra-provider

Installs ORC v2.0.3 from a GitHub release using kubectl. This must be done before initializing Cluster API providers.

```bash
export ORC_VERSION=v2.0.3
kubectl apply -f "https://github.com/k-orc/openstack-resource-controller/releases/download/${ORC_VERSION}/install.yaml"

```

--------------------------------

### Install Kamaji Console with Helm and required Secret

Source: https://kamaji.clastix.io/guides/console

Create the required Secret in the kamaji-system namespace before installing the console; the installation fails without it. Then add the Clastix Helm repository and install the console with the release name 'console' in the kamaji-system namespace.

```bash
# The secret is required, otherwise the installation will fail
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: kamaji-console
  namespace: kamaji-system
data:
  # Credentials to login into console
  ADMIN_EMAIL: <email>
  ADMIN_PASSWORD: <password>
  # Secret used to sign the browser session
  JWT_SECRET: <jwtSecret>
  # URL where the console is accessible: https://<hostname>/ui
  NEXTAUTH_URL: <nextAuthUrl>
EOF

```

```bash
helm repo add clastix https://clastix.github.io/charts
helm repo update
helm -n kamaji-system install console clastix/kamaji-console
helm status console -n kamaji-system

```

--------------------------------

### Check Tenant Cluster node status after joining

Source: https://kamaji.clastix.io/getting-started/kamaji-azure

Displays the list of nodes in the Tenant Cluster and shows them starting as NotReady until a CNI plugin is installed.

```bash
kubectl --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig get nodes

NAME               STATUS     ROLES    AGE    VERSION
tenant-00-000000   NotReady   <none>   112s   v1.25.0
tenant-00-000002   NotReady   <none>   92s    v1.25.0
tenant-00-000003   NotReady   <none>   71s    v1.25.0
```

--------------------------------

### Install secondary Datastore with Helm

Source: https://kamaji.clastix.io/guides/datastore-overrides

Installs the secondary Datastore, reusing the cert-manager CA created by the etcd-primary release via the nameOverride setting.

```shell
 helm install etcd-secondary clastix/kamaji-etcd -n kamaji-etcd --create-namespace \
   --set selfSignedCertificates.enabled=false \
   --set certManager.enabled=true \
   --set certManager.ca.create=false \
   --set certManager.ca.nameOverride=etcd-primary-kamaji-etcd-ca \
   --set certManager.issuerRef.kind=ClusterIssuer \
   --set certManager.issuerRef.name=self-signed

```

--------------------------------

### Install Kamaji with Helm

Source: https://kamaji.clastix.io/reference/benchmark

Installs Kamaji from the clastix/kamaji chart into the kamaji-system namespace, with node selectors and tolerations for the infra node group and resource constraints disabled. The benchmark runs Kamaji without resource limits to use all available resources.

```bash
helm upgrade --install kamaji clastix/kamaji --namespace kamaji-system --create-namespace \
    --set nodeSelector."eks\.amazonaws\.com/nodegroup"=infra \
    --set "tolerations[0].key=pool" \
    --set "tolerations[0].operator=Equal" \
    --set "tolerations[0].value=infra" \
    --set "tolerations[0].effect=NoExecute" \
    --set "tolerations[1].key=pool" \
    --set "tolerations[1].operator=Equal" \
    --set "tolerations[1].value=infra" \
    --set "tolerations[1].effect=NoSchedule" \
    --set "resources=null"
```

--------------------------------

### Install Tenant Cluster with Helm

Source: https://kamaji.clastix.io/cluster-api/vsphere-infra-provider

Adds the clastix Helm repository, updates it, and installs the cluster-api-kamaji-vsphere chart as a release named 'sample' with the cluster name set to 'sample', in the 'default' namespace, using values from 'my-values.yaml'.

```bash
helm repo add clastix https://clastix.github.io/cluster-api-kamaji-vsphere
helm repo update
helm install sample clastix/cluster-api-kamaji-vsphere \
    --set cluster.name=sample \
    --namespace default \
    --values my-values.yaml

```

--------------------------------

### Install in-cluster IPAM provider with clusterctl

Source: https://kamaji.clastix.io/cluster-api/proxmox-infra-provider

Initialize the management cluster with the in-cluster IPAM provider to assign IP addresses to nodes.

```bash
clusterctl init --ipam in-cluster

```

--------------------------------

### Create Kind cluster

Source: https://kamaji.clastix.io/getting-started/kamaji-kind

Creates a Kind cluster named 'kamaji'. This is the first step in the setup.

```bash
kind create cluster --name kamaji

```

--------------------------------

### Install Calico CNI plugin

Source: https://kamaji.clastix.io/getting-started/kamaji-generic

Install Calico as the CNI plugin for the Tenant Cluster. First download the Calico manifest, optionally customize it, then apply it to the Tenant Cluster using the kubeconfig.

```bash
curl https://raw.githubusercontent.com/projectcalico/calico/v3.24.1/manifests/calico.yaml -O

```

```bash
kubectl --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig apply -f calico.yaml

```

--------------------------------

### Install Kamaji controller (latest edge)

Source: https://kamaji.clastix.io/getting-started/kamaji-generic

Installs the latest edge release of the Kamaji controller using the official Helm chart. As of July 2024, version-pinned stable releases are only available on a subscription basis.

```bash
helm install kamaji clastix/kamaji \
    --version 0.0.0+latest \
    --namespace kamaji-system \
    --create-namespace \
    --set image.tag=latest

```

--------------------------------

### Install ProvisioningRequest CRD

Source: https://kamaji.clastix.io/cluster-api/cluster-autoscaler

Install the ProvisioningRequest custom resource definition using kubectl. The command retrieves the kubeconfig for the capi-quickstart-kubevirt cluster and applies the CRD from the autoscaler repository.

```bash
kubectl kamaji kubeconfig get capi-quickstart-kubevirt > /tmp/capi-quickstart-kubevirt
KUBECONFIG=/tmp/capi-quickstart-kubevirt kubectl apply -f https://raw.githubusercontent.com/kubernetes/autoscaler/refs/tags/cluster-autoscaler-1.34.1/cluster-autoscaler/apis/config/crd/autoscaling.x-k8s.io_provisioningrequests.yaml

```

--------------------------------

### Install kamaji-etcd Helm chart for a new datastore

Source: https://kamaji.clastix.io/guides/datastore-migration

Installs the kamaji-etcd chart to create a managed multi-tenant etcd datastore named 'dedicated' in the 'dedicated' namespace. Requires the clastix Helm repository to be added and updated first. The --set datastore.enabled=true flag enables the datastore creation.

```bash
helm repo add clastix https://clastix.github.io/charts
helm repo update
helm install dedicated clastix/kamaji-etcd -n dedicated --create-namespace --set datastore.enabled=true

```

--------------------------------

### clusterctl init with Kamaji control plane

Source: https://kamaji.clastix.io/cluster-api/control-plane-provider

Initializes the Management Cluster with Cluster API components and the Kamaji Control Plane Provider. Run this command after installing clusterctl, kubectl, and Kamaji in the Management Cluster.

```bash
clusterctl init --control-plane kamaji

```

--------------------------------

### Add Jetstack Helm repo and update

Source: https://kamaji.clastix.io/getting-started/kamaji-kind

Adds the Jetstack Helm repository and updates the Helm repository list. Required before installing cert-manager.

```bash
helm repo add jetstack https://charts.jetstack.io
helm repo update

```

--------------------------------

### Verify worker node readiness after Calico installation

Source: https://kamaji.clastix.io/getting-started/kamaji-azure

Shows that after Calico is applied, all nodes transition to Ready status, confirming they successfully joined the Tenant Cluster.

```bash
kubectl --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig get nodes 

NAME               STATUS   ROLES    AGE     VERSION
tenant-00-000000   Ready    <none>   3m38s   v1.25.0
tenant-00-000002   Ready    <none>   3m18s   v1.25.0
tenant-00-000003   Ready    <none>   2m57s   v1.25.0
```

--------------------------------

### Verify ClusterClass Installation

Source: https://kamaji.clastix.io/cluster-api/cluster-class

Check that Cluster API and Kamaji components are deployed by listing deployments and filtering for capi or kamaji.

```bash
kubectl get deployments -A | grep -E "capi|kamaji"

```

--------------------------------

### Generate and apply cluster manifest with CCM

Source: https://kamaji.clastix.io/cluster-api/vsphere-infra-provider

Generate a cluster manifest with the CCM installed on the Management Cluster using the template capi-kamaji-vsphere-template-ccm.yaml, then apply the generated manifest to create the tenant cluster.

```bash
clusterctl generate cluster $CLUSTER_NAME \
    --from capi-kamaji-vsphere-template-ccm.yaml \
    > capi-kamaji-vsphere-cluster.yaml

```

```bash
kubectl apply -f capi-kamaji-vsphere-cluster.yaml

```

--------------------------------

### Download the Calico manifest

Source: https://kamaji.clastix.io/getting-started/kamaji-azure

Downloads the official Calico manifest v3.24.1 from the projectcalico GitHub repository for later installation on the Tenant Cluster.

```bash
curl https://raw.githubusercontent.com/projectcalico/calico/v3.24.1/manifests/calico.yaml -O
```

--------------------------------

### Verify Kamaji CRDs

Source: https://kamaji.clastix.io/getting-started/kamaji-kind

Checks that Kamaji CRDs are installed by filtering the output of 'kubectl get crds' for 'kamaji'.

```bash
kubectl get crds | grep -i kamaji

```

--------------------------------

### Create EKS cluster configuration and deploy

Source: https://kamaji.clastix.io/getting-started/kamaji-aws

Source environment variables, create an eks-cluster.yaml configuration file, and create the EKS cluster using eksctl.

```bash
source kamaji-aws.env

cat > eks-cluster.yaml <<EOF
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: ${KAMAJI_CLUSTER}
  region: ${KAMAJI_REGION}
  version: ${KAMAJI_CLUSTER_VERSION}
iam:
  withOIDC: true
vpc:
  clusterEndpoints:
    privateAccess: true
    publicAccess: true
managedNodeGroups:
  - name: ${KAMAJI_NODE_NG}
    labels: { role: workers }
    instanceType: ${KAMAJI_NODE_TYPE}
    desiredCapacity: 1
    privateNetworking: true
    availabilityZones: [${KAMAJI_AZ}]
    iam:
      withAddonPolicies:
        certManager: true
        ebs: true
        externalDNS: true
addons:
- name: aws-ebs-csi-driver
EOF

eks create cluster -f eks-cluster.yaml

```

--------------------------------

### Create the Cluster: Switch Context and Apply Cluster Manifest

Source: https://kamaji.clastix.io/cluster-api/external-cluster

Switch to the management cluster context, then apply the Cluster manifest from the provided URL to create the cluster.

```bash
kubectl config use-context kind-management

```

```bash
kubectl apply -f "https://raw.githubusercontent.com/clastix/kamaji/master/config/capi/clusterclass-kubevirt-kamaji-external.yaml"

```

--------------------------------

### Deploy the Prometheus operator with Helm

Source: https://kamaji.clastix.io/reference/benchmark

Adds the Bitnami Helm repository and installs the kube-prometheus chart in the monitoring-system namespace. Configures nodeSelector and tolerations for the operator, Prometheus, Alertmanager, kube-state-metrics, and blackboxExporter to run on the 'infra' node pool.

```bash
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm upgrade --install kube-prometheus bitnami/kube-prometheus --namespace monitoring-system --create-namespace \
    --set operator.nodeSelector."eks\.amazonaws\.com/nodegroup"=infra \
    --set "operator.tolerations[0].key=pool" \
    --set "operator.tolerations[0].operator=Equal" \
    --set "operator.tolerations[0].value=infra" \
    --set "operator.tolerations[0].effect=NoExecute" \
    --set "operator.tolerations[1].key=pool" \
    --set "operator.tolerations[1].operator=Equal" \
    --set "operator.tolerations[1].value=infra" \
    --set "operator.tolerations[1].effect=NoSchedule" \
    --set prometheus.nodeSelector."eks\.amazonaws\.com/nodegroup"=infra \
    --set "prometheus.tolerations[0].key=pool" \
    --set "prometheus.tolerations[0].operator=Equal" \
    --set "prometheus.tolerations[0].value=infra" \
    --set "prometheus.tolerations[0].effect=NoExecute" \
    --set "prometheus.tolerations[1].key=pool" \
    --set "prometheus.tolerations[1].operator=Equal" \
    --set "prometheus.tolerations[1].value=infra" \
    --set "prometheus.tolerations[1].effect=NoSchedule" \
    --set alertmanager.nodeSelector."eks\.amazonaws\.com/nodegroup"=infra \
    --set "alertmanager.tolerations[0].key=pool" \
    --set "alertmanager.tolerations[0].operator=Equal" \
    --set "alertmanager.tolerations[0].value=infra" \
    --set "alertmanager.tolerations[0].effect=NoExecute" \
    --set "alertmanager.tolerations[1].key=pool" \
    --set "alertmanager.tolerations[1].operator=Equal" \
    --set "alertmanager.tolerations[1].value=infra" \
    --set "alertmanager.tolerations[1].effect=NoSchedule" \
    --set kube-state-metrics.nodeSelector."eks\.amazonaws\.com/nodegroup"=infra \
    --set "kube-state-metrics.tolerations[0].key=pool" \
    --set "kube-state-metrics.tolerations[0].operator=Equal" \
    --set "kube-state-metrics.tolerations[0].value=infra" \
    --set "kube-state-metrics.tolerations[0].effect=NoExecute" \
    --set "kube-state-metrics.tolerations[1].key=pool" \
    --set "kube-state-metrics.tolerations[1].operator=Equal" \
    --set "kube-state-metrics.tolerations[1].value=infra" \
    --set "kube-state-metrics.tolerations[1].effect=NoSchedule" \
    --set blackboxExporter.nodeSelector."eks\.amazonaws\.com/nodegroup"=infra \
    --set "blackboxExporter.tolerations[0].key=pool" \
    --set "blackboxExporter.tolerations[0].operator=Equal" \
    --set "blackboxExporter.tolerations[0].value=infra" \
    --set "blackboxExporter.tolerations[0].effect=NoExecute" \
    --set "blackboxExporter.tolerations[1].key=pool" \
    --set "blackboxExporter.tolerations[1].operator=Equal" \
    --set "blackboxExporter.tolerations[1].value=infra" \
    --set "blackboxExporter.tolerations[1].effect=NoSchedule"

```

--------------------------------

### Inspect DataStore dedicated with kubectl get -o yaml

Source: https://kamaji.clastix.io/guides/datastore-migration

Shows the resulting DataStore resource named 'dedicated' after installation. It uses the etcd driver with three endpoints and TLS configuration referencing secrets in the 'dedicated' namespace. The status is empty, indicating the datastore is not yet ready.

```yaml
# kubectl get datastore dedicated -o yaml
apiVersion: kamaji.clastix.io/v1alpha1
kind: DataStore
metadata:
  annotations:
  labels:
  name: dedicated
spec:
  driver: etcd
  endpoints:
  - dedicated-0.dedicated.dedicated.svc.cluster.local:2379
  - dedicated-1.dedicated.dedicated.svc.cluster.local:2379
  - dedicated-2.dedicated.dedicated.svc.cluster.local:2379
  tlsConfig:
    certificateAuthority:
      certificate:
        secretReference:
          keyPath: ca.crt
          name: dedicated-certs
          namespace: dedicated
      privateKey:
        secretReference:
          keyPath: ca.key
          name: dedicated-certs
          namespace: dedicated
    clientCertificate:
      certificate:
        secretReference:
          keyPath: tls.crt
          name: dedicated-root-client-certs
          namespace: dedicated
      privateKey:
        secretReference:
          keyPath: tls.key
          name: dedicated-root-client-certs
          namespace: dedicated
status: {}

```

--------------------------------

### Create KIND clusters for management and external

Source: https://kamaji.clastix.io/cluster-api/external-cluster

Create the management and external KIND clusters, then verify the available contexts.

```bash
kind create cluster --name management

```

```bash
kind create cluster --name external

```

```bash
kubectl config get-contexts

```

--------------------------------

### Enable ClusterClass and Initialize Management Cluster

Source: https://kamaji.clastix.io/cluster-api/cluster-class

Set the CLUSTER_TOPOLOGY environment variable and run clusterctl init with the Kamaji control plane and vSphere infrastructure provider. This installs Cluster API core components with ClusterClass support, Kamaji Control Plane Provider, and the chosen infrastructure provider.

```bash
export CLUSTER_TOPOLOGY=true
clusterctl init --control-plane kamaji --infrastructure vsphere

```

--------------------------------

### Install ExternalDNS via Helm

Source: https://kamaji.clastix.io/getting-started/kamaji-aws

Install ExternalDNS (optional) to dynamically update DNS records from service annotations in EKS. Adds the external-dns Helm repository and installs version 1.15.1 in the external-dns namespace.

```bash
helm repo add external-dns https://kubernetes-sigs.github.io/external-dns/
helm repo update
helm install external-dns external-dns/external-dns \
  --namespace external-dns \
  --create-namespace \
  --version 1.15.1

```

--------------------------------

### Use MySQL as alternative datastore per TenantControlPlane

Source: https://kamaji.clastix.io/guides/alternative-datastore

Install Kamaji without the default datastore, deploy MySQL resources, apply the DataStore and TenantControlPlane manifests, verify the datastore and tenant, then clean up. The TenantControlPlane references the datastore by name in spec.dataStore. The output shows the datastore and tenant statuses.

```bash
helm install kamaji ./charts/kamaji -n kamaji-system --create-namespace --set kamaji-etcd.deploy=false

```

```bash
make -C ./deploy/kine/mysql mariadb NAME=gold

```

```bash
kubectl apply -f ./config/samples/kamaji_v1alpha1_datastore_mysql_gold.yaml

```

```bash
kubectl get datastores

NAME         DRIVER   READY   AGE
mysql-gold   MySQL    true    30s

```

```bash
cat > test-tenant-gold.yaml <<EOF 
apiVersion: kamaji.clastix.io/v1alpha1
kind: TenantControlPlane
metadata:
  name: k8s-133
  labels:
    tenant.clastix.io: k8s-133
spec:
  dataStore: mysql-gold #this should match the Datastore's resource NAME
  controlPlane:
    deployment:
      replicas: 2
    service:
      serviceType: LoadBalancer
  kubernetes:
    version: "v1.33.0"
    kubelet:
      configurationJSONPatches:
        - op: add
          path: /featureGates
          value:
            KubeletCrashLoopBackOffMax: false
            KubeletEnsureSecretPulledImages: false
        - op: replace
          path: /cgroupDriver
          value: systemd
  networkProfile:
    port: 6443
  addons:
    coreDNS: {}
    kubeProxy: {}
    konnectivity:
      server:
        port: 8132
      agent:
        mode: DaemonSet
EOF

kubectl apply -f test-tenant-gold.yaml

```

```bash
kubectl get tcp -A 
NAMESPACE   NAME      VERSION   INSTALLED VERSION   STATUS   CONTROL-PLANE ENDPOINT   KUBECONFIG                 DATASTORE    AGE
default     k8s-133   v1.33.0   v1.33.0             Ready    10.10.10.200:6443        k8s-133-admin-kubeconfig   mysql-gold   43s

```

```bash
kubectl delete -f test-tenant-gold.yaml
kubectl delete -f ./config/samples/kamaji_v1alpha1_datastore_mysql_gold.yaml 
make -C ./deploy/kine/mysql mariadb-destroy NAME=gold

```

--------------------------------

### Clone Kamaji repository and prepare workspace

Source: https://kamaji.clastix.io/getting-started/kamaji-azure

Clone the Kamaji repository and navigate to the deploy directory on the bootstrap machine.

```bash
git clone https://github.com/clastix/kamaji
cd kamaji/deploy

```

--------------------------------

### HelmChartProxy Kamaji installation

Source: https://kamaji.clastix.io/cluster-api/openstack-infra-provider

Defines a HelmChartProxy to install the Kamaji chart from the Clastix repository. It targets clusters with the label addons.cluster.x-k8s.io/kamaji: "true" and sets resources to null in valuesTemplate. The install creates the namespace and the upgrade keeps a max history of 10.

```yaml
apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: kamaji
  namespace: ${CLUSTER_NAMESPACE}
spec:
  clusterSelector:
    matchLabels:
      addons.cluster.x-k8s.io/kamaji: "true"
  repoURL: https://clastix.github.io/charts
  chartName: kamaji
  version: 0.0.0+latest
  namespace: ${CLUSTER_NAMESPACE}
  releaseName: kamaji
  options:
    enableClientCache: false
    timeout: 10m0s
    install:
      createNamespace: true
    upgrade:
      maxHistory: 10
  valuesTemplate: |
    resources: null
```

--------------------------------

### Delete Kamaji webhook configurations

Source: https://kamaji.clastix.io/getting-started/kamaji-generic

Manually removes the validating and mutating webhook configurations installed by Kamaji. Use this if the installation is broken.

```bash
kubectl delete ValidatingWebhookConfiguration kamaji-validating-webhook-configuration
kubectl delete MutatingWebhookConfiguration kamaji-mutating-webhook-configuration

```

--------------------------------

### Set Azure subscription and login

Source: https://kamaji.clastix.io/getting-started/kamaji-azure

Set the Azure subscription and log in using Azure CLI.

```bash
az account set --subscription "MySubscription"
az login

```

--------------------------------

### Initialize Cluster API with Kamaji and KubeVirt

Source: https://kamaji.clastix.io/cluster-api/external-cluster

Switch to the management cluster context, enable ClusterClass support, and initialize Cluster API with the Kamaji control-plane provider and KubeVirt infrastructure provider.

```bash
kubectl config use-context kind-management

```

```bash
export CLUSTER_TOPOLOGY=true
clusterctl init \
  --core cluster-api \
  --bootstrap kubeadm \
  --infrastructure kubevirt \
  --control-plane kamaji

```

--------------------------------

### Create and verify a TenantControlPlane with LoadBalancer

Source: https://kamaji.clastix.io/getting-started/kamaji-generic

Creates a TenantControlPlane manifest with a LoadBalancer service and applies it. After creation, the command checks the resulting resources; the output shows the expected Ready status, three replicas, and the LoadBalancer endpoint.

```bash
cat > ${TENANT_NAMESPACE}-${TENANT_NAME}-tcp.yaml <<EOF
apiVersion: kamaji.clastix.io/v1alpha1
kind: TenantControlPlane
metadata:
  name: ${TENANT_NAME}
  namespace: ${TENANT_NAMESPACE}
  labels:
    tenant.clastix.io: ${TENANT_NAME}
spec:
  dataStore: default
  controlPlane:
    deployment:
      replicas: 3
      additionalMetadata:
        labels:
          tenant.clastix.io: ${TENANT_NAME}
      extraArgs:
        apiServer: []
        controllerManager: []
        scheduler: []
      resources: {}
    service:
      additionalMetadata:
        labels:
          tenant.clastix.io: ${TENANT_NAME}
      serviceType: LoadBalancer
  kubernetes:
    version: ${TENANT_VERSION}
    kubelet:
      cgroupfs: systemd
    admissionControllers:
      - ResourceQuota
      - LimitRanger
  networkProfile:
    port: ${TENANT_PORT}
    certSANs:
    - ${TENANT_NAME}.${TENANT_DOMAIN}
    serviceCidr: ${TENANT_SVC_CIDR}
    podCidr: ${TENANT_POD_CIDR}
    dnsServiceIPs:
    - ${TENANT_DNS_SERVICE}
  addons:
    coreDNS: {}
    kubeProxy: {}
    konnectivity:
      server:
        port: ${TENANT_PROXY_PORT}
EOF

kubectl -n ${TENANT_NAMESPACE} apply -f ${TENANT_NAMESPACE}-${TENANT_NAME}-tcp.yaml

```

```bash
kubectl -n ${TENANT_NAMESPACE} get tcp,deploy,pods,svc

NAME                           VERSION   STATUS   CONTROL-PLANE ENDPOINT   KUBECONFIG                   DATASTORE   AGE
tenantcontrolplane/tenant-00   v1.32.2   Ready    192.168.32.240:6443      tenant-00-admin-kubeconfig   default     2m20s

NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/tenant-00   3/3     3            3           118s

NAME                             READY   STATUS    RESTARTS   AGE
pod/tenant-00-58847c8cdd-7hc4n   4/4     Running   0          82s
pod/tenant-00-58847c8cdd-ft5xt   4/4     Running   0          82s
pod/tenant-00-58847c8cdd-shc7t   4/4     Running   0          82s

NAME                TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                         AGE
service/tenant-00   LoadBalancer   10.32.132.241   192.168.32.240   6443:32152/TCP,8132:32713/TCP   2m20s

```

--------------------------------

### Telemetry payload example: Deleting a TCP

Source: https://kamaji.clastix.io/telemetry

Example JSON payload sent when a TCP is deleted, including the status and TCP version.

```json
// Deleting a TCP
{"status": "Ready", "tcp_version": "v1.27.0", "kamaji_version": "v0.6.1", "kubernetes_version": "v1.27.3"}
```

--------------------------------

### Deploy 100 TenantControlPlanes with envsubst and kubectl

Source: https://kamaji.clastix.io/reference/benchmark

Creates 100 TenantControlPlane resources by substituting environment variables into a YAML template and applying it with kubectl. The YAML defines a minimal control plane with 2 replicas, a ClusterIP service, and no LoadBalancer or Ingress, suitable for resource-consumption benchmarking.

```bash
kubectl create ns benchmark01

for I in {001..100}; do
    DS=etcd-01 NS=benchmark01 I=$I envsubst < kamaji_v1alpha1_tenantcontrolplane.yaml | kubectl apply -f -
done

```

```yaml
apiVersion: kamaji.clastix.io/v1alpha1
kind: TenantControlPlane
metadata:
  name: benchmark$I
  namespace: $NS
  labels:
    tenant.clastix.io: benchmark$I
spec:
  dataStore: $DS
  controlPlane:
    deployment:
      replicas: 2
      resources:
        apiServer:
          requests:
            memory: 320Mi
            cpu: 100m
    service:
      serviceType: ClusterIP
  kubernetes:
    version: "v1.25.4"
    kubelet:
      cgroupfs: cgroupfs
    admissionControllers:
      - ResourceQuota
      - LimitRanger
  networkProfile:
    port: 6443
  addons:
    coreDNS: {}
    kubeProxy: {}

```

--------------------------------

### Create TenantControlPlane with dataStoreOverrides

Source: https://kamaji.clastix.io/guides/datastore-overrides

Creates a TenantControlPlane that uses the primary Datastore for general resources and overrides the /events resource to use the secondary Datastore via the dataStoreOverrides field.

```shell
echo 'apiVersion: kamaji.clastix.io/v1alpha1
kind: TenantControlPlane
metadata:
  name: k8s-133
  labels:
    tenant.clastix.io: k8s-133
spec:
  controlPlane:
    deployment:
      replicas: 2
    service:
      serviceType: LoadBalancer
  kubernetes:
    version: "v1.33.1"
    kubelet:
      cgroupfs: systemd
  dataStore: etcd-primary-kamaji-etcd
  dataStoreOverrides:
    - resource: "/events" # Store events in the secondary ETCD
      dataStore: etcd-secondary-kamaji-etcd
  networkProfile:
    port: 6443
  addons:
    coreDNS: {}
    kubeProxy: {}
    konnectivity:
      server:
        port: 8132
      agent:
        mode: DaemonSet
' | k apply -f -

```

--------------------------------

### HelmChartProxy cert-manager installation

Source: https://kamaji.clastix.io/cluster-api/openstack-infra-provider

Defines a HelmChartProxy to install cert-manager v1.18.3 from the Jetstack chart repository. It targets clusters with the label addons.cluster.x-k8s.io/cert-manager: "true" and enables CRDs via valuesTemplate. The install creates the cert-manager namespace and the upgrade keeps a max history of 10.

```yaml
apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: cert-manager
  namespace: ${CLUSTER_NAMESPACE}
spec:
  version: v1.18.3
  clusterSelector:
    matchLabels:
      addons.cluster.x-k8s.io/cert-manager: "true"
  repoURL: https://charts.jetstack.io
  chartName: cert-manager
  namespace: cert-manager
  releaseName: cert-manager
  options:
    enableClientCache: false
    timeout: 10m0s
    install:
      createNamespace: true
    upgrade:
      maxHistory: 10
  valuesTemplate: |
    crds: 
      enabled: true
```

--------------------------------

### Initialize Cluster API providers

Source: https://kamaji.clastix.io/cluster-api/openstack-infra-provider

Initializes Cluster API core, bootstrap, control-plane, infrastructure, addon, and kamaji providers with clusterctl.

```bash
clusterctl init \
  --core cluster-api \
  --bootstrap kubeadm \
  --control-plane kubeadm \
  --infrastructure openstack \
  --addon helm \
  --control-plane kamaji

```

--------------------------------

### Create namespace and deploy shared templates

Source: https://kamaji.clastix.io/cluster-api/cluster-class

Create a namespace for shared templates, then deploy the shared Kamaji control plane and bootstrap templates into it. These commands are the first steps for cross-provider template reuse.

```bash
kubectl create namespace cluster-templates

```

```bash
kubectl apply -n cluster-templates -f kamaji-controlplane-template.yaml
kubectl apply -n cluster-templates -f kubeadm-config-template.yaml

```

--------------------------------

### Install cert-manager with Helm

Source: https://kamaji.clastix.io/getting-started/kamaji-kind

Installs cert-manager v1.18.3 in the cert-manager namespace with CRDs enabled. Kamaji depends on cert-manager for TLS certificates used by webhooks.

```bash
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.18.3 \
  --set crds.enabled=true

```

--------------------------------

### Run golangci-lint with Make recipe golint

Source: https://kamaji.clastix.io/guides/contribute

Run the Make recipe `golint` to check that changes respect the coding standard. The command executes `golangci-lint` with the project's configuration file `.golangci.yml`. The GitHub Action will also check `golangci-lint` on pull requests.

```bash
# make golint
golangci-lint run -c .golangci.yml

```

--------------------------------

### Telemetry payload example: Creating a TCP

Source: https://kamaji.clastix.io/telemetry

Example JSON payload sent when a Tenant Control Plane (TCP) is created, including the TCP version and Kamaji and Kubernetes versions.

```json
// Creating a TCP
{"tcp_version": "v1.26.0", "kamaji_version": "v0.6.1", "kubernetes_version": "v1.27.3"}
```

--------------------------------

### Telemetry payload example: General status

Source: https://kamaji.clastix.io/telemetry

Example JSON payload sent for general status telemetry, including UUID, counts of running/sleeping/not_ready/upgrading TCPs, and Kamaji and Kubernetes versions.

```json
// General status
{"uuid": "56279633-3131-436b-b8f2-9008a49a2f12", "running": 1, "sleeping": 0, "not_ready": 0, "upgrading": 0, "kamaji_version": "v0.6.1", "kubernetes_version": "v1.27.3"}
```

--------------------------------

### Configure AWS CLI

Source: https://kamaji.clastix.io/getting-started/kamaji-aws

Run 'aws configure' to set up AWS credentials and region.

```bash
aws configure

```

--------------------------------

### Telemetry payload example: Modifying a TCP (version upgrade)

Source: https://kamaji.clastix.io/telemetry

Example JSON payload sent when a TCP is modified, specifically a version upgrade, including new and old TCP versions.

```json
// Modifying a TCP (n.b.: version upgrade)
{"kamaji_version": "v0.6.1", "new_tcp_version": "v1.27.0", "old_tcp_version": "v1.26.0", "kubernetes_version": "v1.27.3"}
```

--------------------------------

### Install the Cluster Autoscaler via Helm

Source: https://kamaji.clastix.io/cluster-api/cluster-autoscaler

Installs the Cluster Autoscaler via Helm in the Management Cluster, in the same namespace as the workload cluster. Uses `cloudProvider=clusterapi`, `clusterAPIMode=kubeconfig-incluster`, and sets autoscaling labels for auto-discovery.

```bash
helm repo add autoscaler https://kubernetes.github.io/autoscaler
helm repo update
helm upgrade --install ${CLUSTER_NAME}-autoscaler autoscaler/cluster-autoscaler \
    --set cloudProvider=clusterapi \
    --set autodiscvovery.namespace=default \
    --set "autoDiscovery.labels[0].autoscaling=enabled" \
    --set clusterAPIKubeconfigSecret=${CLUSTER_NAME}-kubeconfig \
    --set clusterAPIMode=kubeconfig-incluster

```

--------------------------------

### Enable KubeVirt emulation (optional)

Source: https://kamaji.clastix.io/cluster-api/external-cluster

Optional: Enable emulation mode for KubeVirt if virtualization is not supported on the host. This patches the KubeVirt custom resource to set useEmulation to true.

```bash
kubectl -n kubevirt patch kubevirt kubevirt \
  --type=merge \
  --patch '{"spec":{"configuration":{"developerConfiguration":{"useEmulation":true}}}}'
```