### Complete hetzner-k3s Configuration Example

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Creating_a_cluster.md

This is a comprehensive example of the hetzner-k3s YAML configuration file, showcasing all available options for cluster setup. Commented lines indicate optional parameters.

```yaml
---
hetzner_token: <your token>
cluster_name: test
kubeconfig_path: "./kubeconfig"
k3s_version: v1.32.0+k3s1

networking:
  ssh:
    port: 22
    use_agent: false # set to true if your key has a passphrase
    use_private_ip: false # set to true to connect to nodes via their private IPs
    public_key_path: "~/.ssh/id_ed25519.pub"
    private_key_path: "~/.ssh/id_ed25519"
  allowed_networks:
    ssh:
      - 0.0.0.0/0
    api: # this will firewall port 6443 on the nodes
      - 0.0.0.0/0
    # OPTIONAL: define extra inbound/outbound firewall rules.
    # Each entry supports the following keys:
    #   description (string, optional)
    #   direction   (in | out, default: in)
    #   protocol    (tcp | udp | icmp | esp | gre, default: tcp)
    #   port        (single port "80", port range "30000-32767", or "any") – only relevant for tcp/udp
    #   source_ips  (array of CIDR blocks) – required when direction is in
    #   destination_ips (array of CIDR blocks) – required when direction is out
    #
    # IMPORTANT: Outbound traffic is allowed by default (implicit allow-all).
    # If you add **any** outbound rule (direction: out), Hetzner Cloud switches
    # the outbound chain to an implicit **deny-all**; only traffic matching your
    # outbound rules will be permitted. Define outbound rules carefully to avoid
    # accidentally blocking required egress (DNS, updates, etc.).
    # NOTE: Hetzner Cloud Firewalls support **max 50 entries per firewall**. The built-
    # in rules (SSH, ICMP, node-port ranges, etc.) use ~10 slots. If the sum of the
    # default rules plus your custom ones exceeds 50, hetzner-k3s will abort with
    # an error.
    # custom_firewall_rules:
    #   - description: "Allow HTTP from any IPv4"
    #     direction: in
    #     protocol: tcp
    #     port: 80
    #     source_ips:
    #       - 0.0.0.0/0
    #   - description: "UDP game servers (outbound)"
    #     direction: out
    #     protocol: udp
    #     port: 60000-60100
    #     destination_ips:
    #       - 203.0.113.0/24
  # node_port_firewall_enabled: true # optional: set false to disable NodePort firewall rules (TCP/UDP)
  # node_port_range: "30000-32767" # optional: NodePort range to open on firewalls (TCP/UDP)
  public_network:
    ipv4: true
    ipv6: true
    # hetzner_ips_query_server_url: https://.. # for large clusters, see https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Recommendations.md
    # use_local_firewall: false # for large clusters, see https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Recommendations.md
  private_network:
    enabled: true
    subnet: 10.0.0.0/16
    existing_network_name: ""
  cni:
    enabled: true
    encryption: false
    mode: flannel
    cilium:
      # Optional: specify a path to a custom values file for Cilium Helm chart
      # When specified, this file will be used instead of the default values
      # helm_values_path: "./cilium-values.yaml"
      # chart_version: "v1.17.2"

  # cluster_cidr: 10.244.0.0/16 # optional: a custom IPv4/IPv6 network CIDR to use for pod IPs
  # service_cidr: 10.43.0.0/16 # optional: a custom IPv4/IPv6 network CIDR to use for service IPs. Warning, if you change this, you should also change cluster_dns!
  # cluster_dns: 10.43.0.10 # optional: IPv4 Cluster IP for coredns service. Needs to be an address from the service_cidr range

datastore:
  mode: etcd # etcd (default) or external
  external_datastore_endpoint: postgres://....
#  etcd:
#    # etcd snapshot configuration (optional)
#    snapshot_retention: 24
#    snapshot_schedule_cron: "0 * * * *"
#
#    # S3 snapshot configuration (optional)
#    s3_enabled: false
#    s3_endpoint: "" # Can also be set with ETCD_S3_ENDPOINT environment variable
#    s3_region: "" # Can also be set with ETCD_S3_REGION environment variable
#    s3_bucket: "" # Can also be set with ETCD_S3_BUCKET environment variable
#    s3_access_key: "" # Can also be set with ETCD_S3_ACCESS_KEY environment variable
#    s3_secret_key: "" # Can also be set with ETCD_S3_SECRET_KEY environment variable
#    s3_folder: ""
#    s3_force_path_style: false

schedule_workloads_on_masters: false # set to true to allow pods to be scheduled on master nodes (useful for small clusters)

# image: rocky-9 # optional: default is ubuntu-24.04
# autoscaling_image: 103908130 # optional, defaults to the `image` setting

```

--------------------------------

### Example: Check System Information

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Run_command.md

This example demonstrates checking system details like kernel version and disk space across all nodes.

```bash
hetzner-k3s run --config my-cluster.yaml --command "uname -a && df -h"
```

--------------------------------

### Install hetzner-k3s via Homebrew

Source: https://context7.com/vitobotta/hetzner-k3s/llms.txt

Recommended installation method for macOS and Linux using Homebrew.

```bash
# macOS / Linux via Homebrew (recommended)
brew install vitobotta/tap/hetzner_k3s
```

--------------------------------

### Install hetzner-k3s Manual Binary (Linux arm64)

Source: https://context7.com/vitobotta/hetzner-k3s/llms.txt

Manual installation for Linux arm64 systems by downloading and placing the binary in the system's PATH.

```bash
# Linux arm64
wget https://github.com/vitobotta/hetzner-k3s/releases/download/v2.4.9/hetzner-k3s-linux-arm64
chmod +x hetzner-k3s-linux-arm64
sudo mv hetzner-k3s-linux-arm64 /usr/local/bin/hetzner-k3s
```

--------------------------------

### Install hetzner-k3s Manual Binary (macOS Apple Silicon)

Source: https://context7.com/vitobotta/hetzner-k3s/llms.txt

Manual installation for macOS Apple Silicon systems by downloading and placing the binary in the system's PATH.

```bash
# macOS Apple Silicon
wget https://github.com/vitobotta/hetzner-k3s/releases/download/v2.4.9/hetzner-k3s-macos-arm64
chmod +x hetzner-k3s-macos-arm64
sudo mv hetzner-k3s-macos-arm64 /usr/local/bin/hetzner-k3s
```

--------------------------------

### Example: Run Maintenance Script on Master Node

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Run_command.md

This example runs a custom maintenance script on a specific master node.

```bash
hetzner-k3s run --config my-cluster.yaml --script maintenance.sh --instance master-1
```

--------------------------------

### Example: Update Packages on Worker Node

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Run_command.md

This example shows how to check for available package updates on a specific worker node.

```bash
hetzner-k3s run --config my-cluster.yaml --command "sudo apt update && sudo apt list --upgradable" --instance worker-1
```

--------------------------------

### Verify hetzner-k3s Installation

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Installation.md

Run this command after installation to confirm that hetzner-k3s is installed correctly and display its version number.

```bash
hetzner-k3s --version
```

--------------------------------

### WordPress with Persistent Storage Example

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Storage.md

This example demonstrates setting up persistent storage for both WordPress and its MySQL database using separate PVCs. Ensure the PVCs are created before the deployments.

```yaml
---
# Persistent Volume Claim for WordPress
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wordpress-pvc
  labels:
    app: wordpress
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
# Persistent Volume Claim for MySQL
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pvc
  labels:
    app: mysql
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
# MySQL Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "rootpassword"
        - name: MYSQL_DATABASE
          value: "wordpress"
        - name: MYSQL_USER
          value: "wordpress"
        - name: MYSQL_PASSWORD
          value: "wordpress"
        ports:
        - containerPort: 3306
        volumeMounts:
        - name: mysql-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-storage
        persistentVolumeClaim:
          claimName: mysql-pvc
---
# WordPress Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
  template:
    metadata:
      labels:
        app: wordpress
    spec:
      containers:
      - name: wordpress
        image: wordpress:latest
        env:
        - name: WORDPRESS_DB_HOST
          value: "mysql"
        - name: WORDPRESS_DB_USER
          value: "wordpress"
        - name: WORDPRESS_DB_PASSWORD
          value: "wordpress"
        - name: WORDPRESS_DB_NAME
          value: "wordpress"
        ports:
        - containerPort: 80
        volumeMounts:
        - name: wordpress-storage
          mountPath: /var/www/html
      volumes:
      - name: wordpress-storage
        persistentVolumeClaim:
          claimName: wordpress-pvc
```

--------------------------------

### Example: Cluster configuration with custom partitioning for storage nodes

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Resizing_root_partition_with_post_create_commands.md

A complete cluster configuration example demonstrating how to set up regular worker nodes with automatic growth and storage worker nodes with manual partitioning.

```yaml
grow_root_partition_automatically: true  # Default for most nodes

masters_pool:
  instance_type: cpx22
  instance_count: 3
  locations: [fsn1, hel1, nbg1]
  # Inherits global: true (automatic growth)

worker_node_pools:
- name: storage-workers
  instance_type: cpx32
  instance_count: 4
  location: fsn1
  grow_root_partition_automatically: false  # Override: manual partitioning
  additional_post_k3s_commands:
  - [ sgdisk, -e, /dev/sda ]
  - [ partprobe ]
  - [ parted, -s, /dev/sda, mkpart, primary, ext4, "50%", "100%" ]
  - [ growpart, /dev/sda, "1" ]
  - [ resize2fs, /dev/sda1 ]

- name: regular-workers
  instance_type: cpx22
  instance_count: 2
  location: hel1
  # Inherits global: true (automatic growth)

# Other cluster settings...
```

--------------------------------

### Usage Example for Local Path Storage

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Storage.md

This example shows how to create a PVC using the 'local-path' storage class and mount it to a Redis deployment for high-performance caching.

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-cache-pvc
spec:
  storageClassName: local-path
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
spec:
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:alpine
        ports:
        - containerPort: 6379
        volumeMounts:
        - name: redis-data
          mountPath: /data
      volumes:
      - name: redis-data
        persistentVolumeClaim:
          claimName: redis-cache-pvc
```

--------------------------------

### Install ingress-nginx with Custom Annotations

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Load_balancers.md

Use this command to install or upgrade the ingress-nginx controller with custom configurations specified in `ingress-nginx-values.yaml`. Ensure the namespace `ingress-nginx` exists or is created.

```bash
helm upgrade --install \
  ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  -f ingress-nginx-values.yaml
```

--------------------------------

### Example: Run SSH Fix Script on All Nodes

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Run_command.md

This example executes a script designed to fix SSH configurations on all nodes in the cluster.

```bash
hetzner-k3s run --config my-cluster.yaml --script fix-ssh.sh
```

--------------------------------

### Start and Access Development Container with Docker Compose

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Contributing_and_support.md

Use this command to build and start the development container. Access the container's shell with the second command.

```bash
docker compose up -d

docker compose exec hetzner-k3s bash
```

--------------------------------

### Install Cert-Manager for LetsEncrypt

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Setting_up_a_cluster.md

Installs the cert-manager Helm chart to manage LetsEncrypt TLS certificates. Ensure the cert-manager namespace is created.

```bash
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install \
--namespace cert-manager \
--create-namespace \
--set crds.enabled=true \
cert-manager jetstack/cert-manager
```

--------------------------------

### Install cert-manager with Helm

Source: https://context7.com/vitobotta/hetzner-k3s/llms.txt

Installs cert-manager using Helm, a Kubernetes package manager. Ensure Helm is installed and configured before running.

```bash
# Install cert-manager via Helm
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true
```

--------------------------------

### Install Sample Application

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Setting_up_a_cluster.md

Applies the 'hello-world.yaml' file to the Kubernetes cluster, deploying the sample application and its associated Ingress resource. This step makes the application accessible via the configured load balancer.

```bash
kubectl apply -f hello-world.yaml
```

--------------------------------

### Command Execution Output Example

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Run_command.md

This output format illustrates how results are displayed per instance after executing a command.

```text
Found 3 instances in the cluster
Command to execute: hostname

Nodes that will be affected:
  - master-1 (192.168.1.100)
  - worker-1 (192.168.1.101)
  - worker-2 (192.168.1.102)

Type 'continue' to execute this command on all nodes: continue

=== Instance: master-1 (192.168.1.100) ===
master-1
Command completed successfully

=== Instance: worker-1 (192.168.1.101) ===
worker-1
Command completed successfully

=== Instance: worker-2 (192.168.1.102) ===
worker-2
Command completed successfully
```

--------------------------------

### Install hetzner-k3s via Homebrew on macOS

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Installation.md

Use this command to install the hetzner-k3s CLI using Homebrew on macOS. Homebrew is also supported on Linux.

```bash
brew install vitobotta/tap/hetzner_k3s
```

--------------------------------

### Install ingress-nginx with Custom Annotations

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Setting_up_a_cluster.md

Installs or upgrades the ingress-nginx controller using Helm. It sets the ingress class as default, applies custom annotations from a file, and creates the necessary namespace.

```bash
helm upgrade --install \
ingress-nginx ingress-nginx/ingress-nginx \
--set controller.ingressClassResource.default=true \
-f ./ingress-nginx-annotations.yaml \
--namespace ingress-nginx \
--create-namespace
```

--------------------------------

### Install hetzner-k3s Binary on Intel macOS

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Installation.md

Download, make executable, and move the hetzner-k3s binary to your PATH for Intel Macs if not using Homebrew.

```bash
wget https://github.com/vitobotta/hetzner-k3s/releases/download/v2.4.9/hetzner-k3s-macos-amd64
chmod +x hetzner-k3s-macos-amd64
sudo mv hetzner-k3s-macos-amd64 /usr/local/bin/hetzner-k3s
```

--------------------------------

### Install hetzner-k3s Binary on amd64 Linux

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Installation.md

Download, make executable, and move the hetzner-k3s binary to your PATH for amd64 Linux systems if not using Homebrew.

```bash
wget https://github.com/vitobotta/hetzner-k3s/releases/download/v2.4.9/hetzner-k3s-linux-amd64
chmod +x hetzner-k3s-linux-amd64
sudo mv hetzner-k3s-linux-amd64 /usr/local/bin/hetzner-k3s
```

--------------------------------

### Download Sample Deployment

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Setting_up_a_cluster.md

Downloads the 'sample-deployment.yaml' file from a GitHub repository using curl. This file contains a sample application deployment that can be used to test the ingress setup.

```bash
curl https://raw.githubusercontent.com/vitobotta/hetzner-k3s/refs/heads/main/sample-deployment.yaml --output hello-world.yaml
```

--------------------------------

### Install Ingress Nginx with Helm

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Setting_up_a_cluster.md

Commands to add the ingress-nginx Helm repository, update chart information, and install or upgrade the ingress-nginx controller. Use the `-f` flag to apply custom annotations defined in a local file.

```bash
# INSTALLATION
# 1. Install Helm: https://helm.sh/docs/intro/install/
# 2. Add ingress-nginx Helm repo: helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
# 3. Update information of available charts: helm repo update
# 4. Install ingress-nginx:
# helm upgrade --install \
# ingress-nginx ingress-nginx/ingress-nginx \
# --set controller.ingressClassResource.default=true \ # Remove this line if you don’t want Nginx to be the default Ingress Controller
# -f ./ingress-nginx-annotations.yaml \
# --namespace ingress-nginx \
# --create-namespace

```

--------------------------------

### Install hetzner-k3s Binary on arm64 Linux

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Installation.md

Download, make executable, and move the hetzner-k3s binary to your PATH for arm64 Linux systems if not using Homebrew.

```bash
wget https://github.com/vitobotta/hetzner-k3s/releases/download/v2.4.9/hetzner-k3s-linux-arm64
chmod +x hetzner-k3s-linux-arm64
sudo mv hetzner-k3s-linux-arm64 /usr/local/bin/hetzner-k3s
```

--------------------------------

### Example hetzner-k3s test results output

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/e2e-tests/README.md

Sample output from the './list-test-results.sh' script, showing test configurations, status (ok/error), and test identifiers.

```text
$ ./list-test-results.sh
config-sshport-image.yaml  IMAGE=alma-8 SSHPORT=222           error  creating   test-c59dc574
config-sshport-image.yaml  IMAGE=alma-8 SSHPORT=22            error  creating   test-15c94339
config-sshport-image.yaml  IMAGE=alma-9 SSHPORT=222           ok     tested ok  test-e9acedda
config-sshport-image.yaml  IMAGE=alma-9 SSHPORT=22            ok     tested ok  test-3a378dbe
config-sshport-image.yaml  IMAGE=centos-stream-8 SSHPORT=222  error  done       test-9063a269
config-sshport-image.yaml  IMAGE=centos-stream-8 SSHPORT=22   error  done       test-0a523221
config-sshport-image.yaml  IMAGE=centos-stream-9 SSHPORT=222  ok     done       test-857926a8
config-sshport-image.yaml  IMAGE=debian-11 SSHPORT=222        ok     done       test-fe655f1c
config-sshport-image.yaml  IMAGE=debian-11 SSHPORT=22         ok     tested ok  test-77bf45fe
...
config-sshport-image.yaml  IMAGE=ubuntu-24.04 SSHPORT=222     ok     tested ok  test-b7c132d6
```

--------------------------------

### Create or Reconcile a Cluster

Source: https://context7.com/vitobotta/hetzner-k3s/llms.txt

Initiates the creation of Hetzner Cloud resources and k3s installation. This command is idempotent and safe to re-run for reconciliation.

```bash
hetzner-k3s create --config cluster.yaml
```

--------------------------------

### Install hetzner-k3s Binary on Apple Silicon macOS

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Installation.md

Download, make executable, and move the hetzner-k3s binary to your PATH for Apple Silicon Macs if not using Homebrew.

```bash
wget https://github.com/vitobotta/hetzner-k3s/releases/download/v2.4.9/hetzner-k3s-macos-arm64
chmod +x hetzner-k3s-macos-arm64
sudo mv hetzner-k3s-macos-arm64 /usr/local/bin/hetzner-k3s
```

--------------------------------

### Helm Install NGINX Ingress Controller

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Load_balancers.md

Commands to add the ingress-nginx Helm repository and update it, preparing for the installation of NGINX Ingress Controller.

```bash
# Add the Helm repository
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
```

--------------------------------

### Example etcd Member List Output

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Masters_in_different_locations.md

An example output from the `etcdctl member list` command, showing the IDs, status, and endpoints of the etcd members in a regional cluster.

```text
285ab4b980c2c8c, started, test-master2-d[REDACTED]af, https://[REDACTED]:2380, https://[REDACTED]:2379, false
aad3fac89b68bfb7, started, test-master1-5e550de0, https://[REDACTED]:2380, https://[REDACTED]:2379, false
c[REDACTED]e25aef34e8, started, test-master3-0ed051a3, https://[REDACTED]:2380, https://[REDACTED]:2379, false
```

--------------------------------

### Get Kubernetes Resources

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Setting_up_a_cluster.md

Use these commands to retrieve information about various Kubernetes resources. Specify the namespace with '-n' or use '-A' for all namespaces.

```bash
kubectl get service [serviceName] -A or -n [nameSpace]
```

```bash
kubectl get ingress [ingressName] -A or -n [nameSpace]
```

```bash
kubectl get pod [podName] -A or -n [nameSpace]
```

```bash
kubectl get all -A
```

```bash
kubectl get events -A
```

```bash
helm ls -A
```

```bash
helm uninstall [name] -n [nameSpace]
```

```bash
kubectl -n ingress-nginx get svc
```

```bash
kubectl describe ingress -A
```

```bash
kubectl describe svc -n ingress-nginx
```

```bash
kubectl delete configmap nginx-config -n ingress-nginx
```

```bash
kubectl rollout restart deployment -n NAMESPACE_OF_YOUR_APP
```

```bash
kubectl get all -A` does not include "ingress", so use `kubectl get ing -A
```

--------------------------------

### Inspect Pod Details

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Troubleshooting.md

Get detailed information about a specific pod or view its logs.

```bash
# Check specific pod details
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace>
```

--------------------------------

### Configure OS Image for Cluster

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Private_clusters_with_public_network_interface_disabled.md

Specify the OS image to be used for the cluster nodes. This example sets both the primary and autoscaling images to `debian-12`.

```yaml
image: debian-12
autoscaling_image: debian-12
```

--------------------------------

### Create hetzner-k3s Cluster Configuration

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/README.md

Example `cluster.yaml` file for configuring a hetzner-k3s cluster. This includes Hetzner API token, cluster name, Kubeconfig path, k3s version, networking settings, and node pool definitions for masters and workers.

```yaml
hetzner_token: <your-token>
cluster_name: my-cluster
kubeconfig_path: "./kubeconfig"
k3s_version: v1.32.0+k3s1

networking:
  ssh:
    port: 22
    use_agent: false
    public_key_path: "~/.ssh/id_ed25519.pub"
    private_key_path: "~/.ssh/id_ed25519"
  allowed_networks:
    ssh:
      - 0.0.0.0/0
    api:
      - 0.0.0.0/0

masters_pool:
  instance_type: cpx22
  instance_count: 3
  locations:
    - fsn1
    - hel1
    - nbg1

worker_node_pools:
- name: workers
  instance_type: cpx32
  instance_count: 3
  location: hel1

```

--------------------------------

### Describe Pod for Resource Issues

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Troubleshooting.md

Get detailed information about a specific pod, including events, to diagnose why it might be stuck in a 'Pending' state due to resource constraints.

```bash
kubectl describe pod <pod-name> -n <namespace>
```

--------------------------------

### Enable Debug Mode for hetzner-k3s

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Troubleshooting.md

Run the hetzner-k3s tool with the DEBUG environment variable set to true to get more verbose output for troubleshooting.

```bash
DEBUG=true hetzner-k3s create --config cluster_config.yaml
```

--------------------------------

### Configure Large Cluster Datastore and Pools

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Recommendations.md

Set up an external datastore for very large clusters and configure multiple node pools with autoscaling for different workloads. This example uses specific instance types and locations for compute and storage node pools.

```yaml
datastore:
  mode: etcd  # or external for very large clusters
  # external_datastore_endpoint: postgres://...

masters_pool:
  instance_type: cpx32
  instance_count: 3
  locations:
    - nbg1
    - hel1
    - fsn1

worker_node_pools:
- name: compute
  instance_type: cpx42
  location: nbg1
  autoscaling:
    enabled: true
    min_instances: 5
    max_instances: 50
- name: storage
  instance_type: cpx52
  location: hel1
  autoscaling:
    enabled: true
    min_instances: 3
    max_instances: 20
```

--------------------------------

### Basic Autoscaling Configuration

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Creating_a_cluster.md

Configure a worker node pool for autoscaling by setting 'enabled' to true and defining 'min_instances' and 'max_instances'. This example uses a 'cpx32' instance type in 'fsn1' location.

```yaml
worker_node_pools:
- name: autoscaled-pool
  instance_type: cpx32
  location: fsn1
  autoscaling:
    enabled: true
    min_instances: 1
    max_instances: 10
```

--------------------------------

### Configure NAT Gateway Network Interfaces

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Private_clusters_with_public_network_interface_disabled.md

These lines should be added to or adjusted in the `/etc/network/interfaces` file on your NAT gateway instance. Ensure the subnet and interface name match your setup. This configuration enables IP forwarding and sets up NAT masquerading for the private network.

```bash
auto enp7s0
iface enp7s0 inet dhcp
    post-up echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/16' -o enp7s0 -j MASQUERADE
```

--------------------------------

### Migrate Hetzner K3s Cluster from v1.x to v2.x

Source: https://context7.com/vitobotta/hetzner-k3s/llms.txt

Step-by-step guide to upgrade Hetzner K3s clusters from v1.x to v2.x. This process involves updating configurations, rotating masters and workers, and verifying etcd health. Requires hetzner-k3s v2.2.4 or higher.

```bash
# Step 1: Create DNS resolv file on all existing nodes
hcloud server list | awk '{print $4}' | tail -n +2 | while read ip; do
  echo "Setting DNS for ${ip}"
  ssh -n root@${ip} "echo nameserver 8.8.8.8 | tee /etc/k8s-resolv.conf"
done

# Step 2: Convert config to v2 format.
# Add legacy_instance_type to ALL node pools (critical):
# masters_pool:
#   instance_type: cpx22          # new supported type
#   legacy_instance_type: cx21    # old/deprecated type currently running
#
# worker_node_pools:
# - name: workers
#   instance_type: cpx32
#   legacy_instance_type: cx31

# Step 3: Run create with the new config
hetzner-k3s create --config cluster-v2.yaml

# Step 4: Rotate masters one at a time
# For each master:
kubectl drain <master-name> --ignore-daemonsets --delete-emptydir-data
kubectl delete node <master-name>
hcloud server delete <master-name>
hetzner-k3s create --config cluster-v2.yaml
# Verify etcd health on the newly created master:
export ETCDCTL_API=3
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
export ETCDCTL_CACERT=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
export ETCDCTL_CERT=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt
export ETCDCTL_KEY=/var/lib/rancher/k3s/server/tls/etcd/server-client.key
etcdctl member list

# Step 5: Rotate worker pools (add 1 temp node, drain/delete/recreate each old node)

# Step 6: Remove legacy_instance_type from config and run create once more
hetzner-k3s create --config cluster-v2.yaml

```

--------------------------------

### Configure Ingress Resource for Sample App

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Setting_up_a_cluster.md

Modifies the 'hello-world.yaml' file to include necessary annotations for Kubernetes Ingress and specifies a hostname that resolves to the load balancer's IP address. Replace 'IP_FROM_STEP_13' with the actual public IP of your Hetzner load balancer.

```yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world
  annotations:                       # <<<--- Add annotation
    kubernetes.io/ingress.class: nginx  # <<<--- Add annotation
spec:
  rules:
  - host: hello-world.IP_FROM_STEP_13.nip.io # <<<--- Replace `IP_FROM_STEP_13` with the IP from step 13.
  ....

```

--------------------------------

### Debian 12 Network Configuration Commands

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Private_clusters_with_public_network_interface_disabled.md

Commands to configure network settings for Debian 12 nodes, including installing necessary packages and setting up network interfaces and DNS resolution for NAT gateway access. These commands are executed before K3s installation.

```bash
additional_pre_k3s_commands:
- apt update
- apt upgrade -y
- apt install ifupdown resolvconf -y
- apt autoremove -y hc-utils
- apt purge -y hc-utils
- echo "auto enp7s0" > /etc/network/interfaces.d/60-private
- echo "iface enp7s0 inet dhcp" >> /etc/network/interfaces.d/60-private
- echo "    post-up ip route add default via 10.0.0.1"  >> /etc/network/interfaces.d/60-private
- echo "[Resolve]" > /etc/systemd/resolved.conf
- echo "DNS=1.1.1.1 1.0.0.1" >> /etc/systemd/resolved.conf
- ifdown enp7s0
- ifup enp7s0
- systemctl start resolvconf
- systemctl enable resolvconf
- echo "nameserver 1.1.1.1" >> /etc/resolvconf/resolv.conf.d/head
- echo "nameserver 1.0.0.1" >> /etc/resolvconf/resolv.conf.d/head
- resolvconf --enable-updates
- resolvconf -u
```

--------------------------------

### Configure LetsEncrypt ClusterIssuer

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Setting_up_a_cluster.md

Defines a ClusterIssuer resource for LetsEncrypt to automate certificate provisioning. Requires cert-manager to be installed.

```yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    email: [REDACTED]
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
    - http01:
        ingress:
          class: nginx
```

--------------------------------

### Update hetzner-k3s via Homebrew

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Installation.md

Use this command to upgrade to the latest version of hetzner-k3s if you installed it via Homebrew.

```bash
brew upgrade vitobotta/tap/hetzner_k3s
```

--------------------------------

### Automate DNS Configuration for Nodes

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Upgrading_a_cluster_from_1x_to_2x.md

Automates the creation of a resolv file on existing nodes to set DNS servers. This is crucial for ensuring proper network resolution during the upgrade process.

```bash
hcloud server list | awk '{print $4}' | tail -n +2 | while read ip;
  do
    echo "Setting DNS for ${ip}"
    ssh -n root@${ip} "echo nameserver 8.8.8.8 | tee /etc/k8s-resolv.conf"
    ssh -n root@${ip} "cat /etc/k8s-resolv.conf"
done

```

--------------------------------

### List Available Images

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Creating_a_cluster.md

Use this command to retrieve a list of available OS images from the Hetzner Cloud API. Ensure your API token is set as an environment variable.

```bash
export API_TOKEN=...

curl -H "Authorization: Bearer $API_TOKEN" 'https://api.hetzner.cloud/v1/images?per_page=100'
```

--------------------------------

### Build a Static Binary for hetzner-k3s

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Contributing_and_support.md

Compile the Crystal project into a statically linked binary. This binary will not depend on external libraries.

```bash
crystal build ./src/hetzner-k3s.cr --static
```

--------------------------------

### Clean Up and Restart Upgrade Controller

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Maintenance.md

Use these commands to clean up failed upgrade jobs and plans, then restart the upgrade controller. This is the first step when an upgrade stalls.

```bash
kubectl -n system-upgrade delete job --all
kubectl -n system-upgrade delete plan --all

kubectl label node --all plan.upgrade.cattle.io/k3s-server- plan.upgrade.cattle.io/k3s-agent-
```

```bash
kubectl -n system-upgrade rollout restart deployment system-upgrade-controller
kubectl -n system-upgrade rollout status deployment system-upgrade-controller
```

--------------------------------

### Disable Kubernetes API Load Balancer

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Private_clusters_with_public_network_interface_disabled.md

Set `create_load_balancer_for_the_kubernetes_api` to `false` to turn off the load balancer for the Kubernetes API in a private cluster setup.

```yaml
create_load_balancer_for_the_kubernetes_api: false
```

--------------------------------

### Run hetzner-k3s Tool Without Building

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Contributing_and_support.md

Execute the Crystal script directly to run the tool, passing configuration and arguments.

```bash
crystal run ./src/hetzner-k3s.cr -- create --config cluster_config.yaml
```

--------------------------------

### Finalize k3s Upgrade with Create Command

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Maintenance.md

After a successful upgrade, run the `hetzner-k3s create` command to update the cluster configuration. This is critical for ensuring future node provisions use the new k3s version and synchronizing cluster state.

```bash
hetzner-k3s create --config cluster_config.yaml
```

--------------------------------

### Configure NAT Gateway Routing

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Setting_up_a_cluster.md

Adds commands to the cluster configuration to set up routing for a NAT server after k3s installation. Ensure the gateway IP is correct.

```yaml
additional_packages:
  - ifupdown
additional_post_k3s_commands:
  - apt update
  - apt upgrade -y
  - apt autoremove -y
  - ip route add default via [REDACTED]  # Replace this with your gateway IP
```

--------------------------------

### Add ingress-nginx Helm Repository

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Setting_up_a_cluster.md

Adds the ingress-nginx Helm chart repository to your local Helm configuration. This command is necessary before updating or installing charts from this repository.

```bash
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
```

--------------------------------

### Create K3s Cluster with Configuration

Source: https://context7.com/vitobotta/hetzner-k3s/llms.txt

Creates a K3s cluster using the specified configuration file. Piping the output to 'tee' saves the creation logs. The `--skip-current-ip-validation` flag can be used in CI environments.

```bash
# Create the cluster (pipe to tee to save logs):
hetzner-k3s create --config cluster.yaml | tee create.log
```

```bash
# Skip current-IP validation (useful in CI pipelines):
hetzner-k3s create --config cluster.yaml --skip-current-ip-validation
```

```bash
# Enable debug output:
DEBUG=true hetzner-k3s create --config cluster.yaml
```

--------------------------------

### Docker Compose for IP Query Server

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Recommendations.md

Sets up a Docker container for the IP query server and Caddy. Replace placeholders with your actual domain and email.

```yaml
version: '3.8'
services:
  ip-query-server:
    build: ./ip-query-server
    ports:
      - "8080:80"
    environment:
      - HETZNER_TOKEN=your_token_here
  caddy:
    image: caddy:2
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
    depends_on:
      - ip-query-server

```

--------------------------------

### Describe PVC for Storage Issues

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Troubleshooting.md

Get detailed information about a Persistent Volume Claim (PVC) to diagnose why it might be stuck in a 'Pending' state or failing to mount.

```bash
kubectl describe pvc <pvc-name> -n <namespace>
```

--------------------------------

### Clean and Restart Upgrade Controller

Source: https://context7.com/vitobotta/hetzner-k3s/llms.txt

Use these commands to clean up and restart the system upgrade controller if an upgrade stalls.

```bash
kubectl -n system-upgrade delete job --all
kubectl -n system-upgrade delete plan --all
kubectl label node --all plan.upgrade.cattle.io/k3s-server- plan.upgrade.cattle.io/k3s-agent-
kubectl -n system-upgrade rollout restart deployment system-upgrade-controller
kubectl -n system-upgrade rollout status deployment system-upgrade-controller
```

--------------------------------

### Recommended Configuration for Small to Medium Clusters

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Recommendations.md

Default configuration suitable for small to medium clusters. Ensure Hetzner token and SSH key paths are correctly set.

```yaml
hetzner_token: <your token>
cluster_name: my-cluster
kubeconfig_path: "./kubeconfig"
k3s_version: v1.32.0+k3s1

networking:
  ssh:
    port: 22
    use_agent: false
    public_key_path: "~/.ssh/id_ed25519.pub"
    private_key_path: "~/.ssh/id_ed25519"
  allowed_networks:
    ssh:
      - 0.0.0.0/0
    api:
      - 10.0.0.0/16  # Restrict to private network
  public_network:
    ipv4: true
    ipv6: true
  private_network:
    enabled: true
    subnet: 10.0.0.0/16
  cni:
    enabled: true
    encryption: false
    mode: flannel

masters_pool:
  instance_type: cpx22
  instance_count: 3  # For HA
  locations:
    - nbg1

worker_node_pools:
- name: workers
  instance_type: cpx32
  instance_count: 3
  location: nbg1
  autoscaling:
    enabled: true
    min_instances: 1
    max_instances: 5

protect_against_deletion: true
create_load_balancer_for_the_kubernetes_api: true

```

--------------------------------

### Check Node Status with kubectl

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Troubleshooting.md

Use kubectl to get detailed information about a specific node or list all nodes with their wide status, useful for diagnosing 'NotReady' states.

```bash
kubectl describe node <node-name>
```

```bash
kubectl get nodes -o wide
```

--------------------------------

### Enable Debug Mode for hetzner-k3s

Source: https://context7.com/vitobotta/hetzner-k3s/llms.txt

Run the hetzner-k3s create command with debug mode enabled by setting the DEBUG environment variable.

```bash
DEBUG=true hetzner-k3s create --config cluster.yaml
```

--------------------------------

### Test Command on Single Instance

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Run_command.md

Use the --instance flag to test commands or scripts on a single node before a cluster-wide rollout.

```bash
--instance
```

--------------------------------

### Kubernetes ClusterIssuer and Ingress for Let's Encrypt

Source: https://context7.com/vitobotta/hetzner-k3s/llms.txt

Defines a ClusterIssuer for Let's Encrypt production certificates and an Ingress resource for automatic TLS provisioning. Requires cert-manager to be installed.

```yaml
# lets-encrypt.yaml — ClusterIssuer for production certificates
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    email: you@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
    - http01:
        ingress:
          class: nginx
---
# hello-world.yaml — Ingress with TLS
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    kubernetes.io/tls-acme: "true"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - yourDomain.com
    secretName: yourdomain-com-tls
  rules:
  - host: yourDomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: hello-world
            port:
              number: 80

```

--------------------------------

### Ubuntu 24.04 Network Configuration Commands

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Private_clusters_with_public_network_interface_disabled.md

Commands to configure network settings for Ubuntu 24.04 nodes to use the NAT gateway for internet access. These commands are executed before K3s installation.

```bash
additional_pre_k3s_commands:
- printf "[Match]\nName=enp7s0\n\n[Network]\nDHCP=yes\nGateway=10.0.0.1\n" > /etc/systemd/network/10-enp7s0.network
- printf "[Resolve]\nDNS=185.12.64.2 185.12.64.1" > /etc/systemd/resolved.conf
- systemctl restart systemd-networkd
- systemctl restart systemd-resolved
```

--------------------------------

### Clean Up Upgrade Resources

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Troubleshooting.md

Use these commands to clean up stuck upgrade resources in the system-upgrade namespace.

```bash
kubectl -n system-upgrade delete job --all
```

```bash
kubectl -n system-upgrade delete plan --all
```

--------------------------------

### Create K3s Cluster with Hetzner

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Creating_a_cluster.md

Initiates the creation of a K3s cluster using the hetzner-k3s tool. It's recommended to pipe the output to a log file for review. The `--skip-current-ip-validation` flag can be used to bypass network validation.

```bash
hetzner-k3s create --config cluster_config.yaml | tee create.log
```

--------------------------------

### Configure Small Production Cluster

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Recommendations.md

Configuration for a small production cluster (5-20 nodes) with HA masters and autoscaling enabled for worker nodes. This setup balances availability and resource utilization.

```yaml
masters_pool:
  instance_type: cpx22
  instance_count: 3  # HA masters
  locations:
    - fsn1
    - hel1
    - nbg1
worker_node_pools:
- name: workers
  instance_type: cpx32
  instance_count: 3
  autoscaling:
    enabled: true
    min_instances: 1
    max_instances: 5
```

--------------------------------

### Verify Etcd Member Status

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Upgrading_a_cluster_from_1x_to_2x.md

Installs the etcd client and verifies the status and synchronization of etcd members within the control plane. This is a critical step after rotating master nodes to ensure data consistency.

```bash
sudo apt-get update
sudo apt-get install etcd-client

export ETCDCTL_API=3
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
export ETCDCTL_CACERT=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
export ETCDCTL_CERT=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt
export ETCDCTL_KEY=/var/lib/rancher/k3s/server/tls/etcd/server-client.key

etcdctl member list

```

--------------------------------

### List Available k3s Versions

Source: https://context7.com/vitobotta/hetzner-k3s/llms.txt

Fetches and displays all k3s versions compatible with hetzner-k3s, which can be specified in the configuration.

```bash
hetzner-k3s releases
# Sample output:
# v1.32.0+k3s1
# v1.31.4+k3s1
# v1.30.8+k3s1
# ...
```

--------------------------------

### Check ingress-nginx Service Status

Source: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Setting_up_a_cluster.md

Verifies the status of the ingress-nginx service, specifically checking for the assignment of an external IP address to the load balancer. This command should be run after the Helm installation to confirm the load balancer is provisioned.

```bash
kubectl get svc -n ingress-nginx
```