### Install and Start iscsi-initiator-utils for Longhorn

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/add-robot-server.md

Install the `iscsi-initiator-utils` package and start the `iscsid` service on Robot Nodes. This is a prerequisite for using Longhorn external storage.

```bash
sudo dnf install -y iscsi-initiator-utils
sudo systemctl start iscsid
```

--------------------------------

### Setup Script Internal Steps

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

Details the internal operations of the setup script, including directory creation, downloading configuration files, setting environment variables, and initializing Packer.

```shell
mkdir /path/to/your/new/folder
cd /path/to/your/new/folder
curl -sL https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/kube.tf.example -o kube.tf
curl -sL https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/packer-template/hcloud-microos-snapshots.pkr.hcl -o hcloud-microos-snapshots.pkr.hcl
export HCLOUD_TOKEN="your_hcloud_token"
packer init hcloud-microos-snapshots.pkr.hcl
packer build hcloud-microos-snapshots.pkr.hcl
hcloud context create <project-name>
```

--------------------------------

### Run Project Setup Script

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

Downloads, makes executable, and runs the setup script to create your project folder and MicroOS snapshot. This is the primary method for initiating a new project.

```shell
tmp_script=$(mktemp) && curl -sSL -o "${tmp_script}" https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/scripts/create.sh && chmod +x "${tmp_script}" && "${tmp_script}" && rm "${tmp_script}"
```

--------------------------------

### Create Project Setup Alias

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

Defines a shell alias 'createkh' to streamline the process of downloading, executing, and cleaning up the project setup script.

```shell
alias createkh='tmp_script=$(mktemp) && curl -sSL -o "${tmp_script}" https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/scripts/create.sh && chmod +x "${tmp_script}" && "${tmp_script}" && rm "${tmp_script}"'
```

--------------------------------

### Run Project Setup Script (Fish Shell)

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

A variant of the setup script execution tailored for the Fish shell, using Fish-specific syntax for temporary file handling and execution.

```fish
set tmp_script (mktemp); curl -sSL -o "{tmp_script}" https://raw.githubusercontent.com/kube-hetzner/terraform-hcloud-kube-hetzner/master/scripts/create.sh; chmod +x "{tmp_script}"; bash "{tmp_script}"; rm "{tmp_script}"
```

--------------------------------

### Robot Network Configuration using nmcli (RHEL/AlmaLinux)

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/add-robot-server.md

Example using `nmcli` to configure a VLAN interface for a Robot node to connect to a vSwitch. Ensure MTU is set to 1400 or less. Adjust subnet, VLAN ID, and main interface to match your setup.

```bash
nmcli connection add type vlan con-name vlan4000 ifname vlan4000 vlan.parent enp6s0 vlan.id 4000

nmcli connection modify vlan4000 802-3-ethernet.mtu 1400
nmcli connection modify vlan4000 ipv4.addresses '10.201.0.2/16'
nmcli connection modify vlan4000 ipv4.gateway '10.201.0.1'
nmcli connection modify vlan4000 ipv4.method manual
nmcli connection modify vlan4000 +ipv4.routes "10.0.0.0/8 10.201.0.1"

nmcli connection down vlan4000
nmcli connection up vlan4000
```

--------------------------------

### Install Development Tools

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

Installs necessary tools like Terraform, Packer, kubectl, and hcloud via package managers. Choose the command appropriate for your operating system.

```shell
brew install hashicorp/tap/terraform hashicorp/tap/packer kubectl hcloud
```

```shell
yay -S terraform packer kubectl hcloud
```

```shell
sudo apt install terraform packer kubectl
```

```shell
sudo dnf install terraform packer kubectl
```

```shell
choco install terraform packer kubernetes-cli hcloud
```

--------------------------------

### Execute Pre-Install Commands on Host OS

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Use 'preinstall_exec' to specify a list of shell commands to run on each node before k3s installation. This is useful for tasks like installing custom CA certificates or prerequisite packages. Ensure commands are idempotent if possible.

```terraform
  # Additional commands to execute on the host OS before the k3s install, for example fetching and installing certs.
  # preinstall_exec = [
  #   "curl https://somewhere.over.the.rainbow/ca.crt > /root/ca.crt",
  #   "trust anchor --store /root/ca.crt", # Command for openSUSE/SLE to add CA to trust store
  # ]
```

--------------------------------

### Specify K3s Installation Version

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Use `install_k3s_version` to pin to an exact k3s release. This takes precedence over `initial_k3s_channel` for the initial installation, ensuring consistency. Refer to k3s GitHub releases for available versions.

```terraform
  # Allows you to specify the k3s version. If defined, supersedes initial_k3s_channel.
  # See https://github.com/k3s-io/k3s/releases for the available versions.
  # install_k3s_version = "v1.30.2+k3s2"
```

--------------------------------

### Specify K3s Installation Channel

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Use `initial_k3s_channel` to select a k3s release channel for initial installation and upgrades. For production, using minor version channels like `"v1.30"` is recommended for stability. Ensure compatibility with Rancher if applicable.

```terraform
  # Allows you to specify either stable, latest, testing or supported minor versions.
  # see https://rancher.com/docs/k3s/latest/en/upgrades/basic/ and https://update.k3s.io/v1-release/channels
  # ⚠️ If you are going to use Rancher addons for instance, it's always a good idea to fix the kube version to one minor version below the latest stable,
  #     e.g. v1.29 instead of the stable v1.30.
  # The default is "v1.30".
  # initial_k3s_channel = "stable"
```

--------------------------------

### Manual OS Upgrade Steps

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

Perform a manual OS upgrade by draining the node, connecting via SSH, starting the transactional update service, and rebooting.

```sh
kubectl drain <node-name>
ssh root@<node-ip>
systemctl start transactional-update.service
reboot
```

--------------------------------

### Kube-Hetzner Module Initialization Notes

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Important notes regarding module initialization. Values like 'location' and 'public_key' have no effect after the initial cluster setup and require manual node reprovisioning for updates.

```terraform
# Note that some values, notably "location" and "public_key" have no effect after initializing the cluster.
# This is to keep Terraform from re-provisioning all nodes at once, which would lose data. If you want to update
# those, you should instead change the value here and manually re-provision each node. Grep for "lifecycle".
```

--------------------------------

### Quick Cluster Status Check

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

Use these commands to quickly check the status of your Kubernetes cluster nodes, network, and load balancer. The `hcloud context create` command is for first-time setup only.

```sh
hcloud context create Kube-hetzner  # First time only
hcloud server list                   # Check nodes
hcloud network describe k3s          # Check network
hcloud loadbalancer describe k3s-traefik  # Check LB
```

--------------------------------

### Use Parameters in a ConfigMap Manifest

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/examples/kustomization_user_deploy/README.md

Reference parameters defined in `kube.tf` within your Kustomize manifests, such as this ConfigMap example using `${my_config_key}`.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: demo-config
  data:
    someConfigKey: ${my_config_key}
```

--------------------------------

### Assign Namespaces to Architectures

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

Assign namespaces to specific architectures (e.g., amd64, arm64) by enabling admission controllers and using node selector annotations in namespace definitions. Includes an example for default tolerations.

```tf
k3s_exec_server_args = "--kube-apiserver-arg enable-admission-plugins=PodTolerationRestriction,PodNodeSelector"
```

```yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/node-selector: kubernetes.io/arch=amd64
  name: this-runs-on-amd64
```

```yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/node-selector: kubernetes.io/arch=arm64
    scheduler.alpha.kubernetes.io/defaultTolerations: '[{ "operator" : "Equal", "effect" : "NoSchedule", "key" : "workload-type", "value" : "machine-learning" }]'
  name: this-runs-on-arm64
```

--------------------------------

### Configure Longhorn Helm Chart Repository

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Specify the `longhorn_repository` to install Longhorn from a different Helm chart repository, such as for Rancher compatibility.

```terraform
longhorn_repository = "https://charts.rancher.io"
```

--------------------------------

### Configure Hetzner vSwitch

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Optionally define `vswitch_id` and `vswitch_subnet_index` to link a Hetzner vSwitch to the private network, which is necessary for hybrid setups with Robot servers. The `vswitch_subnet_index` defaults to 201.

```terraform
  # Hetzner Cloud vSwitch ID. If defined, a subnet will be created in the IP-range defined by vswitch_subnet_index.
  # vswitch_id = 12345

  # Subnet index (0-255) for vSwitch.
  # vswitch_subnet_index = 201
```

--------------------------------

### Enable and Configure Rancher

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Enables Rancher Manager for cluster management. Requires specific node resources and a configured hostname. Optionally set the install channel and bootstrap password.

```terraform
  # You can enable Rancher (installed by Helm behind the scenes) with the following flag, the default is "false".
  # enable_rancher = true
  
  # If using Rancher you can set the Rancher hostname
  # rancher_hostname = "rancher.example.com"
  
  # Rancher install channel
  # rancher_install_channel = "stable"
  
  # Bootstrap password for Rancher (min 48 characters)
  # rancher_bootstrap_password = ""
  
  # Rancher registration manifest URL
  # rancher_registration_manifest_url = ""
```

--------------------------------

### Enable Hetzner CCM Helm Deployment

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Set `hetzner_ccm_use_helm` to `true` to use Helm for installing the Hetzner Cloud Controller Manager. The default is `true`, indicating Helm is the preferred deployment method.

```terraform
  # By default, new installations use Helm to install Hetzner CCM. You can use the legacy deployment method (using `kubectl apply`) by setting `hetzner_ccm_use_helm = false`.
  hetzner_ccm_use_helm = true
```

--------------------------------

### List Kustomization Resources in kustomization.yaml.tpl

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/examples/kustomization_user_deploy/README.md

In the main `kustomization.yaml.tpl` file, list the names of other manifests (without the `.tpl` extension) in the `resources` section to include them in the Kustomize build.

```yaml
resources:
  - demo-config-map.yaml.tpl
  - demo-pod.yaml.tpl
```

--------------------------------

### Create Hetzner Cloud Snapshots with Packer

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

Build Hetzner Cloud machine images using Packer. Requires HCLOUD_TOKEN environment variable to be set.

```bash
export HCLOUD_TOKEN=<your-token>
packer build ./packer-template/hcloud-microos-snapshots.pkr.hcl
```

--------------------------------

### Verify SSH Key is Loaded

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/ssh.md

After adding your SSH key to the agent, use this command to verify that it has been successfully loaded and is accessible.

```bash
ssh-add -l
```

--------------------------------

### Enable Rancher Integration

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Enable Rancher integration for cluster management. Note that Rancher requires significant memory and a configured 'rancher_hostname'. It also installs cert-manager.

```terraform
  # enable_rancher = true
```

--------------------------------

### Custom Post-Install Commands for Deployments

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

Define custom commands to run after initial deployment, useful for CRD-dependent applications like ArgoCD. Ensure CRDs are established before applying manifests.

```hcl
extra_kustomize_deployment_commands = <<-EOT
  kubectl -n argocd wait --for condition=established --timeout=120s crd/appprojects.argoproj.io
  kubectl -n argocd wait --for condition=established --timeout=120s crd/applications.argoproj.io
  kubectl apply -f /var/user_kustomize/argocd-projects.yaml
  kubectl apply -f /var/user_kustomize/argocd-application-argocd.yaml
EOT
```

--------------------------------

### Deprecated Autoscaler Node Labels and Taints

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

These arguments are deprecated and will be removed. They were used to apply labels and taints globally to nodes started by the Cluster Autoscaler. They have no effect if `autoscaler_nodepools` is not set.

```terraform
  # ⚠️ Deprecated, will be removed after a new Cluster Autoscaler version has been released which support the new way of setting labels and taints. See above.
  # Add extra labels on nodes started by the Cluster Autoscaler
  # This argument is not used if autoscaler_nodepools is not set, because the Cluster Autoscaler is installed only if autoscaler_nodepools is set
  # autoscaler_labels = [
  #   "node.kubernetes.io/role=peak-workloads"
  # ]

  # Add extra taints on nodes started by the Cluster Autoscaler
  # This argument is not used if autoscaler_nodepools is not set, because the Cluster Autoscaler is installed only if autoscaler_nodepools is set
  # autoscaler_taints = [
  #   "node.kubernetes.io/role=specific-workloads:NoExecute"
  # ]
```

--------------------------------

### Development Workflow Commands

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

Commands for setting up a local development environment, including running cleanup scripts and building packer templates. Ensure you have forked the project and created a new branch.

```sh
../kube-hetzner/scripts/cleanup.sh
```

```sh
packer build ../kube-hetzner/packer-template/hcloud-microos-snapshots.pkr.hcl
```

--------------------------------

### Pin Calico Version

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Pin the Calico installation to a specific version using `calico_version`. This can help bypass GitHub rate limits or temporary issues when fetching the latest release.

```terraform
  # If you want to disable the k3s kube-proxy, use this flag. The default is "false".
  # Ensure that your CNI is capable of handling all the functionalities typically covered by kube-proxy.
  # disable_kube_proxy = true
```

--------------------------------

### Initialize and Apply Terraform Configuration

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

Initializes the Terraform working directory, validates the configuration, and applies the changes to deploy the Kubernetes cluster. This is the final deployment step.

```shell
cd <your-project-folder>
terraform init --upgrade
terraform validate
terraform apply -auto-approve
```

--------------------------------

### Configure Hetzner Cloud Provider

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Sets up the Hetzner Cloud provider, using either a provided token or a token from local variables. Ensure the `hcloud` provider is correctly configured for authentication.

```terraform
provider "hcloud" {
  token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
}

```

--------------------------------

### Ingress TLS Configuration with Cert-Manager

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

Configure Kubernetes Ingress resources to use Cert-Manager for automatic TLS certificate management. Ensure Cert-Manager is installed and configured with a cluster issuer.

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
    - hosts:
        - "*.example.com"
      secretName: example-com-letsencrypt-tls
  rules:
    - host: "*.example.com"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-service
                port:
                  number: 80
```

--------------------------------

### Fix SELinux Issues with udica

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

Generate targeted SELinux profiles for containers to avoid weakening cluster-wide security. This involves inspecting a container, generating a profile, and installing the SELinux module.

```sh
# Find container
crictl ps

# Generate inspection
crictl inspect <container-id> > container.json

# Create profile
udica -j container.json myapp --full-network-access

# Install module
semodule -i myapp.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}
```

```yaml
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: my-container
          securityContext:
            seLinuxOptions:
              type: myapp.process
```

--------------------------------

### Define Terraform Requirements and Providers

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Specifies the required Terraform version and the necessary providers, including the `hcloud` provider with a minimum version constraint. This block ensures compatibility and correct provider setup.

```terraform
terraform {
  required_version = ">= 1.10.1"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.51.0"
    }
  }
}

```

--------------------------------

### SSH Troubleshooting Commands

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

Commands for SSHing into control plane nodes, viewing k3s logs, checking configuration, and system uptime. Ensure you have the correct private key path.

```sh
ssh root@<control-plane-ip> -i /path/to/private_key -o StrictHostKeyChecking=no

# View k3s logs
journalctl -u k3s          # Control plane
journalctl -u k3s-agent    # Agent nodes

# Check config
cat /etc/rancher/k3s/config.yaml

# Check uptime
last reboot
uptime
```

--------------------------------

### List Terraform State for Kustomization Resources

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/examples/kustomization_user_deploy/README.md

Inspect the Terraform state to verify the kustomization resources that have been applied to the cluster.

```bash
$ terraform state list | grep kustom
  ...
  module.kube-hetzner.terraform_data.kustomization
  module.kube-hetzner.terraform_data.kustomization_user["demo-config-map.yaml.tpl"]
  module.kube-hetzner.terraform_data.kustomization_user["demo-pod.yaml.tpl"]
  module.kube-hetzner.terraform_data.kustomization_user["kustomization.yaml.tpl"]
  ...
```

--------------------------------

### Set Rancher Installation Channel in Terraform

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Choose the Rancher release channel ('stable' or 'latest'). 'stable' is recommended for production environments. This setting affects the Rancher application version, not the Kubernetes version.

```terraform
  # When Rancher is deployed, by default is uses the "latest" channel. But this can be customized.
  # The allowed values are "stable" or "latest".
  # rancher_install_channel = "stable"
```

--------------------------------

### Enable CSI Driver SMB for Hetzner Storage Box

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Enable the Container Storage Interface (CSI) SMB driver to integrate with Hetzner Storage Box. Optionally specify a version for the driver.

```terraform
enable_csi_driver_smb = true
csi_driver_smb_version = "v1.16.0"
```

--------------------------------

### Configure Kubelet Arguments in config.yaml

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Set persistent Kubelet arguments by writing them into the k3s `config.yaml` file. This method is preferred over transient CLI arguments for settings that should persist across restarts and reboots.

```terraform
k3s_global_kubelet_args = [
  "kube-reserved=cpu=100m,ephemeral-storage=1Gi",
  "system-reserved=cpu=memory=200Mi",
  "image-gc-high-threshold=50",
  "image-gc-low-threshold=40",
  "feature-gates=NodeSwap=true"
]
```

```terraform
k3s_control_plane_kubelet_args = [
  "some-control-plane-arg=value"
]
```

```terraform
k3s_agent_kubelet_args = [
  "some-agent-arg=value"
]
```

```terraform
k3s_autoscaler_kubelet_args = [
  "some-autoscaler-arg=value"
]
```

--------------------------------

### Enable k3s Local Storage Controller

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Set `enable_local_storage` to `true` to enable k3s's built-in local path provisioner. Be aware that enabling this alongside the Hetzner CSI driver can result in two default StorageClasses, potentially causing provisioning ambiguity. Consider using the Helm chart for more control or always specifying `storageClassName` in PVCs.

```terraform
  # If you want to enable the k3s built-in local-storage controller set this to "true". Default is "false".
  # Warning: When enabled together with the Hetzner CSI, there will be two default storage classes: "local-path" and "hcloud-volumes"!
  #   Even if patched to remove the "default" label, the local-path storage class will be reset as default on each reboot of
  #   the node where the controller runs.
  #   This is not a problem if you explicitly define which storageclass to use in your PVCs.
  #   Workaround if you don't want two default storage classes: leave this to false and add the local-path-provisioner helm chart
  #   as an extra (https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner#adding-extras).
  # enable_local_storage = false
```

--------------------------------

### Enable iSCSI Daemon for Storage Solutions

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Set `enable_iscsid` to `true` to ensure the iSCSI daemon is installed and running on cluster nodes. This is required for storage solutions like Longhorn if not managed by the module itself.

```terraform
enable_iscsid = true
```

--------------------------------

### Enable Kubernetes CSI Driver for SMB

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Set `enable_csi_driver_smb` to `true` to deploy the Kubernetes CSI driver for SMB. This allows using Hetzner Storage Boxes as persistent storage for Kubernetes workloads.

```terraform
enable_csi_driver_smb = true
```

--------------------------------

### Disable Automatic k3s Upgrades

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Set `automatically_upgrade_k3s` to `false` to disable automatic k3s version upgrades. This requires manual management of k3s updates. Recommended for non-HA setups or when strict control over upgrades is needed.

```terraform
  # If you want to disable the automatic upgrade of k3s, you can set below to "false".
  # Ideally, keep it on, to always have the latest Kubernetes version, but lock the initial_k3s_channel to a kube major version,
  # of your choice, like v1.25 or v1.26. That way you get the best of both worlds without the breaking changes risk.
  # For production use, always use an HA setup with at least 3 control-plane nodes and 2 agents, and keep this on for maximum security.

  # The default is "true" (in HA setup i.e. at least 3 control plane nodes & 2 agents, just keep it enabled since it works flawlessly).
  # automatically_upgrade_k3s = false
```

--------------------------------

### Instantiate kube-hetzner Terraform Module

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Instantiates the kube-hetzner module, configuring the Hetzner Cloud provider and passing the API token. It prioritizes the environment variable `var.hcloud_token` over the local definition.

```terraform
module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
```

--------------------------------

### Configure Longhorn Filesystem Type

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Set the filesystem type for Longhorn-formatted storage. Defaults to 'ext4'. Options include 'ext4' and 'xfs'.

```terraform
  # how many replica volumes should longhorn create (default is 3).
  # longhorn_replica_count = 1
```

--------------------------------

### Configure Rancher Bootstrap Password in Terraform

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Optionally set a bootstrap password for Rancher. If left empty, a password will be generated. This password must be at least 48 characters long and can be used for external Rancher provider setup.

```terraform
  # Finally, you can specify a bootstrap-password for your rancher instance. Minimum 48 characters long!
  # If you leave empty, one will be generated for you.
  # (Can be used by another rancher2 provider to continue setup of rancher outside this module.)
  # rancher_bootstrap_password = ""
```

--------------------------------

### Configure Proxied Private Network for K3s

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

Set up a pre-constructed private network with proxy settings for K3s. This involves defining a network, subnet, server, and server network, then configuring additional K3s environment variables.

```tf
resource "hcloud_network" "k3s_proxied" {
  name     = "k3s-proxied"
  ip_range = "10.0.0.0/8"
}

resource "hcloud_network_subnet" "k3s_proxy" {
  network_id   = hcloud_network.k3s_proxied.id
  type         = "cloud"
  network_zone = "eu-central"
  ip_range     = "10.128.0.0/9"
}

resource "hcloud_server" "your_proxy_server" { ... }

resource "hcloud_server_network" "your_proxy_server" {
  depends_on = [hcloud_server.your_proxy_server]
  server_id  = hcloud_server.your_proxy_server.id
  network_id = hcloud_network.k3s_proxied.id
  ip         = "10.128.0.1"
}

module "kube-hetzner" {
  existing_network_id = [hcloud_network.k3s_proxied.id]  # Note: brackets required!
  network_ipv4_cidr = "10.0.0.0/9"
  additional_k3s_environment = {
    "http_proxy" : "http://10.128.0.1:3128",
    "HTTP_PROXY" : "http://10.128.0.1:3128",
    "HTTPS_PROXY" : "http://10.128.0.1:3128",
    "CONTAINERD_HTTP_PROXY" : "http://10.128.0.1:3128",
    "CONTAINERD_HTTPS_PROXY" : "http://10.128.0.1:3128",
    "NO_PROXY" : "127.0.0.0/8,10.0.0.0/8,",
  }
}
```

--------------------------------

### Default Longhorn Helm Values

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Use `longhorn_values` to override the module's default Helm values for Longhorn. This example shows setting the default data path, filesystem type, replica count, and default StorageClass.

```terraform
  # Merge specific keys without replacing all defaults (recommended for hotfix image tags).
  /*   longhorn_merge_values = <<-EOT
image:
  longhorn:
    manager:
      tag: v1.11.0-hotfix-1
    instanceManager:
      tag: v1.11.0-hotfix-1
  EOT */
```

--------------------------------

### Migrate Nodes with Append Index to False

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

Prevent node replacement during migration from count-based to map-based node definitions by setting `append_index_to_node_name` to `false`. This example shows how to configure labels, server types, and placement groups for individual nodes.

```tf
agent_nodepools = [
  {
    name        = "agent-large",
    server_type = "cx33",
    location    = "nbg1",
    labels      = [],
    taints      = [],
    nodes = {
      "0" : {
        append_index_to_node_name = false,
        labels = ["my.extra.label=special"],
        placement_group = "agent-large-pg-1",
      },
      "1" : {
        append_index_to_node_name = false,
        server_type = "cx43",
        labels = ["my.extra.label=slightlybiggernode"],
        placement_group = "agent-large-pg-2",
      },
    }
  },
]
```

--------------------------------

### Extra Kustomize Deployment Commands

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Specify shell commands to execute after the module's main Kustomize deployment. Useful for post-install actions like waiting for CRDs or applying additional manifests.

```terraform
  # Extra commands to be executed after the `kubectl apply -k` (useful for post-install actions, e.g. wait for CRD, apply additional manifests, etc.).
  # extra_kustomize_deployment_commands=""
```

--------------------------------

### Robot Node IP Route Configuration

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/add-robot-server.md

Verify the IP routing table on a Robot Node. Ensure the default route and private network routes are correctly configured for vSwitch connectivity.

```bash
default via 203.0.113.123 dev enp6s0 proto static onlink 
10.0.0.0/8 via 10.201.0.1 dev enp6s0.4000 proto static onlink 
10.201.0.0/16 dev enp6s0.4000 proto kernel scope link src 10.201.0.2
```

--------------------------------

### Enable Cert-Manager Deployment

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Set `enable_cert_manager = false` to prevent the module from deploying cert-manager via its Helm chart. The default (`true`) installs cert-manager for automated TLS certificate management within Kubernetes. If Rancher is enabled, it might manage its own cert-manager.

```terraform
# You can enable cert-manager (installed by Helm behind the scenes) with the following flag, the default is "true".
# enable_cert_manager = false
```

--------------------------------

### k3s Agent Configuration for Robot Node

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/add-robot-server.md

Configure the k3s agent on a Robot Node by creating `/etc/rancher/k3s/config.yaml`. This includes setting the Flannel interface, enabling bundled binaries, configuring Kubelet arguments, and specifying node labels and taints.

```yaml
flannel-iface: enp6s0  # Set to your main interface (only needed for Flannel CNI)
prefer-bundled-bin: true
kubelet-arg:
  - cloud-provider=external
  - volume-plugin-dir=/var/lib/kubelet/volumeplugins
  - kube-reserved=cpu=50m,memory=300Mi,ephemeral-storage=1Gi
  - system-reserved=cpu=250m,memory=6000Mi  # Optional: reserve some space for system
node-label:
  - k3s_upgrade=true
  - instance.hetzner.cloud/provided-by=robot # To prevent Hetzner CSI pods from being scheduled on robot nodes
node-taint: []
selinux: true
server: https://<API_SERVER_IP>:6443  # Replace with your API server IP
token: <CLUSTER_TOKEN>                # Replace with your cluster token
```

--------------------------------

### Define Multiple Node Pools in Terraform

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Configure different node pools for your Kubernetes cluster, specifying server types, locations, and counts. This example shows standard node pools, a storage-optimized pool, an egress gateway pool, and ARM-based nodes.

```terraform
  agent_nodepools = [
    {
      name        = "agent-small",
      server_type = "cx23",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1
      # swap_size   = "2G"
      # zram_size   = "2G"
      # kubelet_args = ["kube-reserved=cpu=50m,memory=300Mi,ephemeral-storage=1Gi", "system-reserved=cpu=250m,memory=300Mi"]
      # placement_group = "default"
      # backups = true
    },
    {
      name        = "agent-large",
      server_type = "cx33",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1
      # placement_group = "default"
      # backups = true
    },
    {
      name        = "storage",
      server_type = "cx33",
      location    = "nbg1",
      labels      = [
        "node.kubernetes.io/server-usage=storage" # Example label
      ],
      taints      = [], # Could add taints to only allow storage workloads
      count       = 1
      # In the case of using Longhorn, you can use Hetzner volumes instead of using the node's own storage by specifying a value from 10 to 10240 (in GB)
      # It will create one volume per node in the nodepool, and configure Longhorn to use them.
      # Something worth noting is that Volume storage is slower than node storage, which is achieved by not mentioning longhorn_volume_size or setting it to 0.
      # So for something like DBs, you definitely want node storage, for other things like backups, volume storage is fine, and cheaper.
      # longhorn_volume_size = 20 # In GB
      # backups = true
    },
    # Egress nodepool useful to route egress traffic using Hetzner Floating IPs
    # used with Cilium's Egress Gateway feature
    {
      name        = "egress",
      server_type = "cx23",
      location    = "nbg1",
      labels = [
        "node.kubernetes.io/role=egress"
      ],
      taints = [
        "node.kubernetes.io/role=egress:NoSchedule" # Ensures only egress gateway pods run here
      ],
      floating_ip = true # Special attribute for this module
      # Optionally associate a reverse DNS entry with the floating IP(s).
      # floating_ip_rns = "my.domain.com"
      count = 1
    },
    # Arm based nodes
    {
      name        = "agent-arm-small",
      server_type = "cax11", # ARM server type
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    # For fine-grained control over the nodes in a node pool, replace the count variable with a nodes map.
    # In this case, the node-pool variables are defaults which can be overridden on a per-node basis.
    # Each key in the nodes map refers to a single node and must be an integer string ("1", "123", ...).
    {
      name        = "agent-arm-medium",
      server_type = "cax21", # Default server_type for this pool
      location    = "nbg1",  # Default location
      labels      = [],
      taints      = [],
      nodes = { # Overrides 'count' and allows per-node customization
        "1" : { # Node identified as "1" within this pool
          location = "fsn1" # Override location for this specific node
          labels = [
            "testing-labels=a1",
          ]
        },
        "20" : { # Node identified as "20"
          labels = [
            "testing-labels=b1",
          ]
          # server_type could also be overridden here if needed
        }
      }
    },
  ]

```

--------------------------------

### Disable Automatic OS Upgrades for Non-HA Clusters

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

For non-High Availability (HA) clusters, specifically when the number of control-plane nodes is less than 3, it is crucial to disable automatic OS upgrades to prevent potential downtime. This setting directly impacts the stability of the Kubernetes API in single-control-plane setups.

```terraform
  # The default is "true" (in HA setup it works wonderfully well, with automatic roll-back to the previous snapshot in case of an issue).
  # IMPORTANT! For non-HA clusters i.e. when the number of control-plane nodes is < 3, you have to turn it off.
  automatically_upgrade_os = false
```

--------------------------------

### Configure HCCM Settings for Robot Server Integration

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/add-robot-server.md

Enable Robot server integration in HCCM by setting `robot_ccm_enabled` to true and providing Hetzner Webservice User credentials. The `vswitch_id` must also be set.

```yaml
robot_ccm_enabled = true
robot_user = "<your-webservice-user>"
robot_password = "<your-webservice-password>"
vswitch_id = "<your-vswitch-id>"
```

--------------------------------

### Configure Longhorn Filesystem Type

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Set `longhorn_fstype` to choose the filesystem type for Longhorn volumes. Options include `ext4` (default) and `xfs`.

```terraform
longhorn_fstype = "xfs"
```

--------------------------------

### Enable Delete Protection for Resources

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

Configure delete protection for specific resources like floating IPs, load balancers, and volumes to prevent accidental deletion through the Hetzner Console or API.

```tf
enable_delete_protection = {
  floating_ip   = true
  load_balancer = true
  volume        = true
}
```

--------------------------------

### Configure Global Kubelet Arguments for k3s

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Set global kubelet arguments that are passed to the k3s config.yaml, ensuring persistence across reboots. This includes settings for resource reservation, image garbage collection, and feature gates.

```terraform
  # The vars below here passes it to the k3s config.yaml. This way it persist across reboots
  # Make sure you set "feature-gates=NodeSwap=true" if want to use swap_size
  # Note: CloudDualStackNodeIPs was removed in K8s 1.32 (always enabled now)
  # see https://github.com/k3s-io/k3s/issues/8811#issuecomment-1856974516
  # k3s_global_kubelet_args = ["kube-reserved=cpu=100m,ephemeral-storage=1Gi", "system-reserved=cpu=memory=200Mi", "image-gc-high-threshold=50", "image-gc-low-threshold=40"]
  # k3s_control_plane_kubelet_args = []
  # k3s_agent_kubelet_args = []
  # k3s_autoscaler_kubelet_args = []
```

--------------------------------

### Connect to Kube API with Kubectl

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/README.md

Interact with the Kubernetes API using `kubectl`. Specify the kubeconfig file or set it as the default KUBECONFIG environment variable.

```sh
kubectl --kubeconfig clustername_kubeconfig.yaml get nodes
```

```sh
export KUBECONFIG=/<path-to>/clustername_kubeconfig.yaml
```

--------------------------------

### Add Extra Kustomization Parameters

Source: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/blob/master/docs/llms.md

Specifies extra commands to execute after kustomization deployment or additional parameters for kustomization. Useful for post-install tasks like waiting for resources or applying extra manifests.

```terraform
  # Extra commands and parameters for kustomization
  # extra_kustomize_deployment_commands = ["kubectl wait --for=condition=ready pod -l app=myapp -n mynamespace --timeout=300s"]
  # extra_kustomize_parameters = {
  #   myvar = "myvalue"
  # }
```