### Recommended git commit message for adding installation notes

Source: https://truecharts.org/guides/development/contibuting-example

A concise commit message example: 'Add installation notes to librespeed'. Written in present tense and imperative mood, with a short summary as recommended for simple changes.

```text
Add installation notes to librespeed
```

--------------------------------

### Example node configuration with GPU patch

Source: https://truecharts.org/guides/gpu/nvidia

A complete example showing how to define a node with both the required system extensions and the GPU patch reference.

```yaml

nodes:
    # We would adivce to always stick to a "k8s-something-1" style naming scheme
    - hostname: k8s-control-1
      ipAddress: ${MASTER1IP_IP}
      controlPlane: true
      schematic:
        customization:
          systemExtensions:
            officialExtensions:
              - siderolabs/nonfree-kmod-nvidia-lts
              - siderolabs/nvidia-container-toolkit-lts
      ### Note the placement of the gpu patch reference ###
      patches:
        - "@./patches/nvidia.yaml"

```

--------------------------------

### OpenVPN example with Windscribe provider

Source: https://truecharts.org/guides/addons/vpn-setup

OpenVPN configuration example using Windscribe as the provider. All providers generally need VPN_SERVICE_PROVIDER and VPN_TYPE.

```yaml
addons:
  gluetun:
    enabled: true
    container:
      env:
        FIREWALL: "on"
        FIREWALL_OUTBOUND_SUBNETS: "192.168.178.0/24"
        VPN_SERVICE_PROVIDER: "windscribe"
        VPN_TYPE: "openvpn"
        OPENVPN_USER: "username"
        OPENVPN_PASSWORD: "password"
        SERVER_REGIONS: "Canada East"
        SERVER_CITIES: "Montreal"

```

--------------------------------

### Markdown installation_notes.md example for Librespeed chart

Source: https://truecharts.org/guides/development/contibuting-example

Example Markdown content for the chart's installation_notes.md file. States the chart needs no special steps and reminds users to enable ingress for accessing the web portal via a domain. This entire block is wrapped in an extra pair of triple-backtick fences in the source.

```markdown
```markdown
# Librespeed Installation Notes


## General


This chart does not need any special steps to be installed and can be installed normally, following the general guideliness for TrueCharts charts.
This file exists only as an example.


## Web Portal


Don't forget to enable ingress if you want to access the speed test portal via a domain name!
```
```

--------------------------------

### Install a Helm chart with custom values file

Source: https://truecharts.org/guides/helm

Installs a chart with user-specific settings from a values.yaml file. The values.yaml can be replaced with a path of your choice.

```bash
helm install mychart oci://oci.trueforge.org/truecharts/CHARTNAME -f values.yaml
```

--------------------------------

### Example bootstrap warnings and errors

Source: https://truecharts.org/guides/clustertool/getting-started

These warnings and errors are normal during the bootstrap process while the VM is temporarily unreachable.

```text
WRN Not all required pods are running, waiting before checking again
```

```text
ERR status: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 192.168.0.224:50000
```

--------------------------------

### Install a Helm chart from TrueCharts OCI repository

Source: https://truecharts.org/guides/helm

Installs a chart directly from the TrueCharts OCI repository. Replace CHARTNAME with the chart name and mychart with the desired release name.

```bash
helm install mychart oci://oci.trueforge.org/truecharts/CHARTNAME
```

--------------------------------

### Deploy lvm-disk-watcher HelmRelease

Source: https://truecharts.org/guides/clustertool/csi/topolvm

Example HelmRelease configuration for deploying the lvm-disk-watcher chart.

```yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
    name: lvm-disk-watcher
    namespace: lvm-disk-watcher
spec:
    interval: 5m
    releaseName: lvm-disk-watcher
    chart:
        spec:
            chart: lvm-disk-watcher
            version: 1.1.0
            sourceRef:
                kind: HelmRepository
                name: truecharts
                namespace: flux-system
    install:
        createNamespace: true
        crds: CreateReplace
        remediation:
            retries: 3
    upgrade:
        crds: CreateReplace
        remediation:
            retries: 3
    values: {}
```

--------------------------------

### Configure codeserver addon workload

Source: https://truecharts.org/guides/addons/addon-configuration

Example configuration for the codeserver addon, demonstrating how to define probes, image selectors, resource exclusions, security contexts, and startup arguments.

```yaml
addons:
  codeserver:
    enabled: true
    container:
      enabled: true
      probes:
        liveness:
          enabled: true
          port: 12321
          path: "/"
        readiness:
          enabled: true
          port: 12321
          path: "/"
        startup:
          enabled: true
          port: 12321
          path: "/"
      imageSelector: "codeserverImage"
      resources:
        excludeExtra: true
      securityContext:
        runAsUser: 0
        runAsGroup: 0
        runAsNonRoot: false
        readOnlyRootFilesystem: false
      args:
        - "--port"
        - "12321"
        - "/"
        - --auth
        - none
```

--------------------------------

### Install a Helm chart in a specific namespace

Source: https://truecharts.org/guides/helm

Installs a chart with custom values and deploys it to a specified namespace. Replace NAMESPACE with the desired namespace.

```bash
helm install mychart oci://oci.trueforge.org/truecharts/CHARTNAME -f values.yaml -n NAMESPACE
```

--------------------------------

### Example GPU resource assignment

Source: https://truecharts.org/guides/gpu/intel

Shows how to request one Intel GPU (gpu.intel.com/i915) in a chart's resources limits.

```yaml
resources:
    limits:
      gpu.intel.com/i915: 1
```

--------------------------------

### Wireguard example with Mullvad provider

Source: https://truecharts.org/guides/addons/vpn-setup

Wireguard configuration example using Mullvad as the provider. Pull private key, endpoint port, and Wireguard addresses from a Mullvad wireguard config file. VPN Config File Location is not necessary; leave it blank.

```yaml
addons:
  gluetun:
    enabled: true
    container:
      env:
        FIREWALL: "on"
        FIREWALL_OUTBOUND_SUBNETS: "192.168.178.0/24"
        VPN_TYPE: "wireguard"
        VPN_SERVICE_PROVIDER: "mullvad"
        WIREGUARD_PRIVATE_KEY: "xxxxxxx"
        FIREWALL_VPN_INPUT_PORTS: "59241"
        WIREGUARD_ADDRESSES: "xxx.xxx.xxx.xxx/32"
        SERVER_CITIES: "Torronto"
        VPN_ENDPOINT_PORT: "51820"
        WIREGUARD_PUBLIC_KEY: "xxxxxxxx"
        VPN_ENDPOINT_IP: "xx.xx.xxx.xxx"

```

--------------------------------

### Configure TopoLVM Helm Values

Source: https://truecharts.org/guides/clustertool/csi/topolvm

Example configuration for TopoLVM including device-class, storage classes, and node settings.

```yaml
  values:
    lvmd:
      managed: false
      env:
        - name: LVM_SYSTEM_DIR
          value: /tmp
      deviceClasses:
        - name: thin
          volume-group: topolvm_vg                     # Volume Group name used in LVM_Disk_Watcher
          default: true
          spare-gb: 10
          type: thin
          thin-pool:
            name: topolvm_thin                         # Logical Volume name used in LVM_Disk_Watcher
            overprovision-ratio: 10.0                  # Adjust to your convenience
    storageClasses:
      - name: topolvm-thin-provisioner
        storageClass:
          fsType: xfs
          isDefaultClass: true                         # If don't want to have this as your Default Storage Class in your cluster, set to false.
          volumeBindingMode: WaitForFirstConsumer
          allowVolumeExpansion: true
          additionalParameters:
            "topolvm.io/device-class": "thin"
    node:
      lvmdEmbedded: true


## Optional setting for a single node cluster:
    # controller:
    #   replicaCount: 1
```

--------------------------------

### Check NVIDIA Extensions

Source: https://truecharts.org/guides/gpu/nvidia

List installed extensions to confirm NVIDIA components are present.

```bash
talosctl get extensions
```

```text
192.168.178.9   runtime     ExtensionStatus   3             1         nonfree-kmod-nvidia-lts        535.183.06-v1.8.4
192.168.178.9   runtime     ExtensionStatus   4             1         nvidia-container-toolkit-lts   535.183.06-v1.16.1
```

--------------------------------

### Traefik example values

Source: https://truecharts.org/guides/ingress/traefik

Example values for the Traefik Helm chart, including deployment, service with LoadBalancer and MetalLB annotations, logging, ingressClass, TLS options, providers, and ports. Configure environment variables like ${TRAEFIK_IP} and ${SECRET_PUBLIC_DOMAIN/./-}-tls.

```yaml
# https://artifacthub.io/packages/helm/traefik/traefik?modal=values
deployment:
  enabled: true
  replicas: 2
service:
  enabled: true
  type: LoadBalancer
  annotations:
    metallb.io/ip-allocated-from-pool: main
    metallb.io/loadBalancerIPs: ${TRAEFIK_IP}
  spec:
    externalTrafficPolicy: Local
logs:
  general:
    level: INFO
  access:
    enabled: true
ingressClass:
  enabled: true
  isDefaultClass: true
tlsOptions:
  default:
    minVersion: VersionTLS12
    maxVersion: VersionTLS13
    sniStrict: true
providers:
  kubernetesCRD:
    enabled: true
    allowCrossNamespace: true
    allowExternalNameServices: true
tlsStore:
  default:
    defaultCertificate:
      secretName: "${SECRET_PUBLIC_DOMAIN/./-}-tls"
ports:
  traefik:
    expose:
      default: true
  web:
    redirections:
      port: websecure
  websecure:
    tls:
      enabled: true
      options: "default"

```

--------------------------------

### HelmRelease for nvidia-device-plugin

Source: https://truecharts.org/guides/gpu/nvidia

Example helm-release.yaml configuration for deploying the Nvidia-Device-Plugin via FluxCD.

```yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
    name: nvidia-device-plugin
    namespace: nvidia-device-plugin
spec:
    interval: 5m
    chart:
        spec:
            chart: nvidia-device-plugin
            version:  0.17.1
            sourceRef:
                kind: HelmRepository
                name: home-ops-mirror
                namespace: flux-system
            interval: 5m
    install:
        createNamespace: true
        crds: CreateReplace
        remediation:
            retries: 3
    upgrade:
        crds: CreateReplace
        remediation:
            retries: 3
    values:
      image:
        repository: nvcr.io/nvidia/k8s-device-plugin
        tag: v0.17.0
      runtimeClassName: nvidia
      config:
        map:
          default: |-
            version: v1
            flags:
              migStrategy: none
            sharing:
              timeSlicing:
                renameByDefault: false
                failRequestsGreaterThanOne: false
                resources:
                  - name: nvidia.com/gpu
                    replicas: 5
      gfd:
        enabled: true
```

--------------------------------

### talconfig.yaml with qemu-guest-agent commented out

Source: https://truecharts.org/guides/clustertool/virtual-machines/virtualbox

Forgetool installs the qemu-guest-agent extension by default, which is not compatible with VirtualBox. Comment out the siderolabs/qemu-guest-agent lines in both controlPlane and worker schematic systemExtensions as shown.

```yaml
controlPlane:
    patches:
        - '@./patches/controlplane.yaml'
        - '@./patches/manifests.yaml'
        # - '@./patches/nvidia.yaml'
    schematic:
        customization:
            extraKernelArgs:
                - net.ifnames=0
            systemExtensions:
                officialExtensions:
                    - siderolabs/iscsi-tools
                    # - siderolabs/qemu-guest-agent
                    # Enable where needed
                    # - siderolabs/amd-ucode
                    # - siderolabs/bnx2-bnx2x
                    # - siderolabs/drbd
                    # - siderolabs/gasket-driver
                    # - siderolabs/i915-ucode
                    # - siderolabs/intel-ucode
                    # - siderolabs/thunderbolt
worker:
    patches:
        - '@./patches/worker.yaml'
        # - '@./patches/nvidia.yaml'
    schematic:
        customization:
            systemExtensions:
                officialExtensions:
                    # Enable where needed
                    # - siderolabs/amd-ucode
                    # - siderolabs/bnx2-bnx2x
                    - siderolabs/iscsi-tools
                    # - siderolabs/qemu-guest-agent
                    # - siderolabs/drbd
                    # - siderolabs/gasket-driver
                    # - siderolabs/i915-ucode
                    # - siderolabs/intel-ucode
                    # - siderolabs/thunderbolt

```

--------------------------------

### Descheduler configuration values

Source: https://truecharts.org/guides/multi-node

Example configuration for the Descheduler deployment to handle failed pods and affinity violations.

```yaml
replicas: 2
kind: Deployment
deschedulerPolicyAPIVersion: descheduler/v1alpha2
deschedulerPolicy:
  profiles:
    - name: Default
      pluginConfig:
        - name: DefaultEvictor
          args:
            evictFailedBarePods: true
            evictLocalStoragePods: true
            evictSystemCriticalPods: true
        - name: RemoveFailedPods
          args:
            reasons:
              - ContainerStatusUnknown
              - NodeAffinity
              - NodeShutdown
              - Terminated
              - UnexpectedAdmissionError
            includingInitContainers: true
            excludeOwnerKinds:
              - Job
            minPodLifetimeSeconds: 1800
        - name: RemovePodsViolatingInterPodAntiAffinity
        - name: RemovePodsViolatingNodeAffinity
          args:
            nodeAffinityType:
              - requiredDuringSchedulingIgnoredDuringExecution
        - name: RemovePodsViolatingNodeTaints
        - name: RemovePodsViolatingTopologySpreadConstraint
      plugins:
        balance:
          enabled:
            - RemovePodsViolatingTopologySpreadConstraint
        deschedule:
          enabled:
            - RemoveFailedPods
            - RemovePodsViolatingInterPodAntiAffinity
            - RemovePodsViolatingNodeAffinity
            - RemovePodsViolatingNodeTaints
service:
  enabled: true
serviceMonitor:
  enabled: true
leaderElection:
  enabled: true
```

--------------------------------

### PVC persistence configuration in values.yaml

Source: https://truecharts.org/guides/add-storage

Add this section to your values.yaml to enable PVC storage. Requires a storageClass and Container Storage Interface (CSI) to be set up; some platforms like SCALE and Harvester include this, while others like TalOS require manual setup.

```yaml
persistence:
  config:
    enabled: true
    type: pvc
    mountPath: /mystorage

```

--------------------------------

### Show all Services

Source: https://truecharts.org/guides/cheatsheet

Lists all services across all namespaces.

```bash
kubectl get svc -A
```

--------------------------------

### Bootstrap the first ControlPlane node

Source: https://truecharts.org/guides/clustertool/getting-started

Run this command to apply the configuration to the first node and initiate the cluster bootstrap process.

```bash
clustertool talos apply
```

```bash
clustertool.exe talos apply
```

--------------------------------

### Show all Kustomizations

Source: https://truecharts.org/guides/cheatsheet

Lists all Flux Kustomizations across all namespaces.

```bash
flux get ks -A
```

--------------------------------

### Create a Physical Volume

Source: https://truecharts.org/guides/clustertool/csi/topolvm

Initialize a physical volume on the specified disk device.

```bash
pvcreate /dev/vdb
```

--------------------------------

### Get Values from a running deployment

Source: https://truecharts.org/guides/cheatsheet

Retrieves the Helm values of a specified deployment in a namespace.

```bash
helm get values -n <namespace> <deployment-name>
```

--------------------------------

### Configure CNPG Recovery Mode

Source: https://truecharts.org/guides/backup--restore/cnpg-backup-restore

Setting the mode to recovery to prevent the application from starting with an empty database.

```yaml
cnpg:
  main:
    mode: recovery
    backups:
      enabled: true
      credentials: s3
    recovery:
      credentials: s3
```

--------------------------------

### Basic Gluetun addon configuration with firewall settings

Source: https://truecharts.org/guides/addons/vpn-setup

Base configuration for the Gluetun addon. Set your network in excludedNetworks_IPv4 and enable the killswitch if desired.

```yaml
addons:
  gluetun:
    enabled: true
    container:
      env:
        FIREWALL: "on"
        FIREWALL_OUTBOUND_SUBNETS: "192.168.178.0/24"

```

--------------------------------

### NVIDIA container CLI error output

Source: https://truecharts.org/guides/gpu/nvidia

Example of the error message displayed during GPU-Testing when the patch is not applied correctly.

```text

failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: mount error: failed to add device rules: unable to generate new device filter program from existing programs: unable to create new device filters program: load program: invalid argument: last insn is not an exit or jmp
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

```

--------------------------------

### Test GPU with nvidia-smi

Source: https://truecharts.org/guides/gpu/nvidia

Run a temporary pod to execute nvidia-smi and verify GPU accessibility. PodSecurity warnings can be ignored.

```bash
kubectl run \
  nvidia-test \
  --restart=Never \
  -ti --rm \
  --image nvcr.io/nvidia/cuda:12.1.0-base-ubuntu22.04 \
  --overrides '{"spec": {"runtimeClassName": "nvidia"}}' \
  nvidia-smi
```

```text
Mon Dec 16 12:54:35 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.06             Driver Version: 535.183.06   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------|
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1660 ...    On  | 00000000:00:0A.0 Off |                  N/A |
| 24%   26C    P8               5W / 125W |      1MiB /  6144MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+


+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
pod "nvidia-test" deleted
```

--------------------------------

### Show all Running Pods

Source: https://truecharts.org/guides/cheatsheet

Lists all pods across all namespaces.

```bash
kubectl get pods -A
```

--------------------------------

### Check GPU schedulability with kubectl

Source: https://truecharts.org/guides/gpu/intel

Command to list nodes and their allocatable gpu.intel.com/i915 resource count.

```bash
kubectl get nodes -o=jsonpath="{range .items[*]}{.metadata.name}{'\n'}{' i915: '}{.status.allocatable.gpu\.intel\.com/i915}{'\n'}"
```

--------------------------------

### Show events

Source: https://truecharts.org/guides/cheatsheet

Lists all events across all namespaces.

```bash
kubectl get events -A
```

--------------------------------

### Example HelmRelease object for Sonarr

Source: https://truecharts.org/guides/fluxcd

A HelmRelease manifest for deploying Sonarr with FluxCD. It specifies the chart source, version, and various install/upgrade settings. The values configure ingress with Traefik and cert-manager, using a placeholder ${BASE_DOMAIN} for the host.

```yaml
---
# yaml-language-server: $schema=https://kubernetes-schemas.pages.dev/helm.toolkit.fluxcd.io/helmrelease_v2.json
apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
  name: sonarr
  namespace: media
spec:
  interval: 15m
  chart:
    spec:
      chart: sonarr
      version: 21.5.7
      sourceRef:
        kind: HelmRepository
        name: truecharts
        namespace: flux-system
      interval: 15m
  timeout: 20m
  maxHistory: 3
  driftDetection:
    mode: warn
  install:
    createNamespace: true
    remediation:
      retries: 3
  upgrade:
    cleanupOnFail: true
    remediation:
      retries: 3
  uninstall:
    keepHistory: false
  values:
    ingress:
      main:
        enabled: true
        integrations:
          traefik:
            enabled: true
            middlewares:
              - name: auth
                namespace: traefik
          certManager:
            enabled: true
            certificateIssuer: ks-le-prod
        hosts:
          - host: sonarr.${BASE_DOMAIN}
            paths:
              - path: /
                pathType: Prefix

```

--------------------------------

### External NGINX Ingress Controller Configuration

Source: https://truecharts.org/guides/ingress/nginx

Example values for the external NGINX ingress controller. It uses MetalLB with the IP variable ${NGINX_EXTERNAL_IP}, sets the ingress class to 'external' as non-default, and includes similar performance and security settings. This controller is intended to be forwarded by a router or cloudflare tunnel.

```yaml
controller:
  replicaCount: 2
  service:
    externalTrafficPolicy: Local
    annotations:
      metallb.io/ip-allocated-from-pool: main
      metallb.io/loadBalancerIPs: ${NGINX_EXTERNAL_IP}
  ingressClassByName: true
  watchIngressWithoutClass: false
  ingressClassResource:
    name: external
    default: false
    controllerValue: k8s.io/external
  config:
    allow-snippet-annotations: true
    annotations-risk-level: Critical
    client-body-buffer-size: 100M
    client-body-timeout: 120
    client-header-timeout: 120
    enable-brotli: "true"
    enable-ocsp: "true"
    enable-real-ip: "true"
    force-ssl-redirect: "true"
    hide-headers: Server,X-Powered-By
    hsts-max-age: "31449600"
    keep-alive-requests: 10000
    keep-alive: 120
    proxy-body-size: 0
    proxy-buffer-size: 16k
    proxy-busy-buffers-size: 32k
    ssl-protocols: TLSv1.3 TLSv1.2
    use-forwarded-headers: "true"
  metrics:
    enabled: true
  extraArgs:
    default-ssl-certificate: "clusterissuer/certificate-issuer-general-wildcard"
    publish-status-address: ${NGINX_EXTERNAL_IP}
  terminationGracePeriodSeconds: 120
  publishService:
    enabled: false
  resources:
    requests:
      cpu: 75m
    limits:
      memory: 500Mi
defaultBackend:
  enabled: false

```

--------------------------------

### Show all Flux Resources

Source: https://truecharts.org/guides/cheatsheet

Lists all Flux resources across all namespaces.

```bash
flux get all -A
```

--------------------------------

### Internal NGINX Ingress Controller Configuration

Source: https://truecharts.org/guides/ingress/nginx

Example values for the internal NGINX ingress controller. It uses MetalLB with the IP variable ${NGINX_INTERNAL_IP}, sets the ingress class to 'internal' as default, and includes various performance and security settings. The controller is not routed externally and is intended for internal network access only.

```yaml
controller:
  replicaCount: 2
  service:
    externalTrafficPolicy: Local
    annotations:
      metallb.io/ip-allocated-from-pool: main
      metallb.io/loadBalancerIPs: ${NGINX_INTERNAL_IP}
  ingressClassByName: true
  watchIngressWithoutClass: true
  ingressClassResource:
    name: internal
    default: true
    controllerValue: k8s.io/internal
  config:
    allow-snippet-annotations: true
    annotations-risk-level: Critical
    client-body-buffer-size: 100M
    client-body-timeout: 120
    client-header-timeout: 120
    enable-brotli: "true"
    enable-ocsp: "true"
    enable-real-ip: "true"
    force-ssl-redirect: "true"
    hide-headers: Server,X-Powered-By
    hsts-max-age: "31449600"
    keep-alive-requests: 10000
    keep-alive: 120
    proxy-body-size: 0
    proxy-buffer-size: 16k
    proxy-busy-buffers-size: 32k
    ssl-protocols: TLSv1.3 TLSv1.2
    use-forwarded-headers: "true"
  metrics:
    enabled: true
  extraArgs:
    default-ssl-certificate: "clusterissuer/certificate-issuer-general-wildcard"
    publish-status-address: ${NGINX_INTERNAL_IP}
  terminationGracePeriodSeconds: 120
  publishService:
    enabled: false
  resources:
    requests:
      cpu: 75m
    limits:
      memory: 500Mi
defaultBackend:
  enabled: false

```

--------------------------------

### TrueNAS CRON Job Script - abort stuck replication jobs

Source: https://truecharts.org/guides/clustertool/csi/democratic

Run daily via TrueNAS cron job to clear stuck replication jobs. Uses truenas_api_client to query core.get_jobs for RUNNING replication.run_onetime jobs with empty progress description, aborts those started more than 10 minutes ago, and destroys the associated ZFS dataset. Requires the truenas_api_client module.

```python
from datetime import datetime, timedelta, timezone
from truenas_api_client import Client
import subprocess


def main():
    # Initialize Middleware Client
    middleware = Client()


    try:
        # Get current time as a timezone-aware datetime in UTC
        current_time = datetime.now(timezone.utc)
        ten_minutes_ago = current_time - timedelta(minutes=10)


        filters = [
            ['state', '=', 'RUNNING'],
            ['method', '=', 'replication.run_onetime'],
            ['progress.description', '=', '']
        ]
        jobs = middleware.call('core.get_jobs', filters)


        for job in jobs:
            start_time = job['time_started']


            # Check if the job started more than 10 minutes ago
            if start_time <= ten_minutes_ago:
                job_id = job['id']
                start_time_formatted = start_time.strftime("%d %B %Y %H:%M:%S")
                running_time = current_time - start_time
                running_time_minutes = running_time.total_seconds() // 60.0


                print(f'{job_id}')
                print(f'- Started: {start_time_formatted}')
                print(f'- Runtime: {running_time_minutes}m')


                print(f'- Aborting: {job_id}')


                # Abort the job
                abort_result = middleware.call('core.job_abort', job_id)
                if abort_result is None:
                    print(f"-- Success")
                else:
                    print(f"-- Failed")


                # Parse and format target dataset path
                target_dataset = job['arguments'][0]['target_dataset']
                target_dataset = target_dataset.rsplit('/snapshot-', 1)[0]
                print(f'- Destroying ZFS dataset "{target_dataset}"')


                # Quote dataset path, in case it contains spaces
                destroy_command = f'zfs destroy "{target_dataset}"'


                # Destroy the ZFS dataset
                destroy_result = subprocess.run(destroy_command, capture_output=True, shell=True)
                if destroy_result.returncode == 0:
                    print(f"-- Success")
                else:
                    print(f"-- Failed")
                print("")


    finally:
        # Close the middleware client connection
        middleware.close()


if __name__ == "__main__":
    main()
```

--------------------------------

### Create a Thin Pool

Source: https://truecharts.org/guides/clustertool/csi/topolvm

Create a thin pool named topolvm_thin within the volume group.

```bash
lvcreate -l 100%FREE --chunksize 256 -T -A n -n topolvm_thin topolvm_vg
```

--------------------------------

### Create a Volume Group

Source: https://truecharts.org/guides/clustertool/csi/topolvm

Create a volume group named topolvm_vg using the physical volume.

```bash
vgcreate topolvm_vg /dev/vdb
```

--------------------------------

### Enable CNPG S3 Backups

Source: https://truecharts.org/guides/backup--restore/cnpg-backup-restore

Basic configuration to enable backups and recovery using S3 credentials.

```yaml
cnpg:
  main:
    backups:
      enabled: true
      credentials: s3
    recovery:
      credentials: s3
```

--------------------------------

### Apply configuration to remaining cluster nodes

Source: https://truecharts.org/guides/clustertool/getting-started

Run this command to automate the application of configurations to all nodes in the cluster.

```bash
clustertool talos apply
```

```bash
clustertool talos apply
```

--------------------------------

### Configure Gluetun HTTP Proxy Environment Variables

Source: https://truecharts.org/guides/addons/vpn-setup

Add these environment variables to your Gluetun configuration to enable the HTTP proxy. Ensure you retain your existing VPN settings alongside these additions.

```yaml
addons:
  gluetun:
    enabled: true
    container:
      env:
        FIREWALL: "on"
        FIREWALL_OUTBOUND_SUBNETS: "192.168.178.0/24"
        HTTPPROXY: "on"
        FIREWALL_INPUT_PORTS: 10095,8888
        #optional
        HTTPPROXY_LOG: "on"
        HTTPPROXY_LISTENING_ADDRESS: :8888
```

--------------------------------

### Deploy intel-device-plugin-operator HelmRelease

Source: https://truecharts.org/guides/gpu/intel

HelmRelease configuration for the intel-device-plugins-operator chart, with controllerExtraArgs set to --devices=gpu.

```yaml
---
# yaml-language-server: $schema=https://kubernetes-schemas.pages.dev/helm.toolkit.fluxcd.io/helmrelease_v2.json
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: intel-device-plugin-operator
  namespace: system
spec:
  interval: 30m
  chart:
    spec:
      chart: intel-device-plugins-operator
      version: 0.32.0
      sourceRef:
        kind: HelmRepository
        name: home-ops-mirror
        namespace: flux-system
  install:
    crds: CreateReplace
    remediation:
      retries: 3
  upgrade:
    cleanupOnFail: true
    crds: CreateReplace
    remediation:
      strategy: rollback
      retries: 3
  dependsOn:
    - name: node-feature-discovery
      namespace: kube-system
  values:
    controllerExtraArgs: |
      - --devices=gpu
```

--------------------------------

### Create lvm-disk-watcher Namespace

Source: https://truecharts.org/guides/clustertool/csi/topolvm

Defines the namespace with required labels for pod security and webhook configuration.

```yaml
apiVersion: v1
kind: Namespace
metadata:
    name: lvm-disk-watcher
    labels:
        pod-security.kubernetes.io/enforce: privileged
        topolvm.io/webhook: ignore
```

--------------------------------

### Show all Flux objects which are not ready

Source: https://truecharts.org/guides/cheatsheet

Lists all Flux objects that are not in a ready state.

```bash
flux get all -A --status-selector ready=false
```