### Dependency Management with Renovate Bot Configuration Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Details the configuration and usage of Renovate Bot for automated dependency updates. It specifies enabled managers (kubernetes, kustomize, helm-values), the commit message pattern used for updates, and provides examples of common dependency updates for Helm charts, container images, and Kustomize resources. Renovate automatically creates pull requests for these updates. ```yaml # renovate.json (example snippet) { "packageManagers": [ "kubernetes", "kustomize", "helm-values" ], "commitMessagePrefix": "chore(deps)", "branchPrefix": "renovate/", "assignees": ["renovate-bot"], "//Examples:", "// chore(deps): update helm release argo-cd to v8.5.7", "// chore(deps): update ghcr.io/glitch-soc/mastodon docker tag to v4.4.5", "// chore(deps): update postgres docker tag to v18" } ``` -------------------------------- ### Kubernetes HTTP Routing with Gateway API Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Defines how incoming HTTP traffic is routed to different services within the cluster using Kubernetes Gateway API resources. It specifies listeners, TLS termination, and routing rules based on hostnames and paths. This setup uses shared Gateway resources for efficiency and security. ```yaml # Example Gateway resource configuration apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: external namespace: project-avocado spec: gatewayClassName: cilium listeners: - name: https protocol: HTTPS port: 443 hostname: "*.goingdark.social" tls: mode: Terminate certificateRefs: - name: "wildcard-goingdark-social-tls" - name: http protocol: HTTP port: 80 hostname: "*.goingdark.social" --- # Example HTTPRoute resource configuration apiVersion: gateway.networking.k8s.io/v1beta1 kind: HTTPRoute metadata: name: mastodon-web namespace: project-avocado spec: parentRefs: - name: external namespace: project-avocado hostnames: - "mastodon.goingdark.social" rules: - matches: - path: type: PathPrefix value: "/" backendRefs: - name: mastodon-web port: 3000 --- apiVersion: gateway.networking.k8s.io/v1beta1 kind: HTTPRoute metadata: name: mastodon-streaming namespace: project-avocado spec: parentRefs: - name: external namespace: project-avocado hostnames: - "mastodon.goingdark.social" rules: - matches: - path: type: PathPrefix value: "/api/v1/streaming" backendRefs: - name: mastodon-streaming port: 4000 ``` -------------------------------- ### ExternalSecret for Managing External Secrets in Kubernetes Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Example of an ExternalSecret resource used with the External Secrets Operator to sync secrets from Bitwarden. This allows managing sensitive information securely outside of Kubernetes manifests. ```yaml apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: mastodon-db-secret spec: refreshInterval: "1h" secretStoreRef: name: bitwarden-secret-store key: default target: name: mastodon-db-connection creationPolicy: Owner data: - secretKey: DATABASE_URL remoteRef: key: mastodon_database_url property: value ``` -------------------------------- ### Kubernetes Network Policy for Service Isolation Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Example of a Kubernetes NetworkPolicy using Cilium to enforce network isolation between services. This policy allows ingress traffic from specific pods and egress traffic to certain destinations, enhancing security. ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: mastodon-isolate-web namespace: mastodon spec: podSelector: matchLabels: app: mastodon component: web policyTypes: - Ingress - Egress ingress: - from: - podSelector: matchLabels: app: mastodon component: sidekiq - podSelector: matchLabels: app: mastodon component: streaming ports: - protocol: TCP port: 3000 egress: - to: - podSelector: matchLabels: app: postgresql - podSelector: matchLabels: app: redis ports: - protocol: TCP port: 5432 - protocol: TCP port: 6379 ``` -------------------------------- ### Initialize and Apply OpenTofu Infrastructure Source: https://github.com/goingdark-social/iac/blob/main/README.md Commands to initialize, plan, and apply OpenTofu configurations for provisioning Hetzner Cloud infrastructure. Ensures the Kubernetes cluster, networking, storage, and security groups are set up. ```bash cd opentofu tofu init -upgrade tofu plan # Review planned changes tofu apply # Deploy infrastructure ``` -------------------------------- ### Development Workflow: Infrastructure Changes with Tofu Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Provides commands for managing infrastructure changes using OpenTofu (tofu). It includes formatting code (`tofu fmt`), validating configuration syntax (`tofu validate`), and previewing changes before applying them (`tofu plan`). This workflow ensures code quality and provides a safety net against unintended infrastructure modifications. ```bash cd opentofu to ``` -------------------------------- ### Kustomize Build Command for Testing Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Command to build Kubernetes manifests using Kustomize, including Helm chart rendering. This is used for testing configurations before committing changes. ```bash kustomize build apps/platform/mastodon kustomize build --enable-helm apps/platform/mastodon ``` -------------------------------- ### Bootstrap Kubernetes Applications with ArgoCD Source: https://github.com/goingdark-social/iac/blob/main/README.md Commands to set up cluster access and apply the main ArgoCD application set configuration. This bootstraps the Kubernetes cluster with essential applications like ArgoCD, networking, monitoring, and community services. ```bash # Set up cluster access export TALOSCONFIG=./opentofu/talosconfig export KUBECONFIG=./opentofu/kubeconfig # Deploy all applications via ArgoCD kubectl apply -f kubernetes/application-set.yaml ``` -------------------------------- ### PostgreSQL SSL Configuration for Security Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Illustrates the PostgreSQL configuration for enforcing SSL with certificate verification. This enhances security by ensuring that client connections are authenticated and encrypted. ```sql ALTER SYSTEM SET ssl = on; ALTER SYSTEM SET ssl_cert_file = '/etc/ssl/certs/ssl-cert-snakeoil.pem'; ALTER SYSTEM SET ssl_key_file = '/etc/ssl/private/ssl-cert-snakeoil.key'; ALTER SYSTEM SET ssl_ca_file = '/etc/ssl/certs/ca-certificates.crt'; ALTER SYSTEM SET ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL'; ALTER SYSTEM SET ssl_prefer_server_ciphers = on; ALTER SYSTEM SET pg_hba.conf = 'hostssl all all 0.0.0.0/0 scram-sha-256'; -- For client certificate verification (verify-ca mode) -- ALTER SYSTEM SET ssl_client_cert_file = '/path/to/client.crt'; -- ALTER SYSTEM SET ssl_client_key_file = '/path/to/client.key'; ``` -------------------------------- ### Mastodon Component Orchestration with Kustomize ConfigMaps Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Orchestrates Mastodon components using Kustomize, focusing on ConfigMap generation for environment-specific configurations. It references nested resource and configuration directories and applies strategic patches for priority and topology spread. ```yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../../base - configs/ - resources/ - patches/ configMapGenerator: - name: mastodon-config files: - .env.production.common - .env.production patches: - path: resources/workloads/web.yaml target: kind: Deployment name: mastodon-web patch: spec: template: spec: containers: - name: web resources: requests: cpu: "500m" memory: "512Mi" limits: cpu: "1000m" memory: "1Gi" ``` -------------------------------- ### ArgoCD ApplicationSet Definitions for Infrastructure and Platform Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Defines ArgoCD ApplicationSets for managing infrastructure and platform applications. It specifies sync policies, retry backoffs, and directory structures for synchronization. This configuration ensures automated deployment and healing with server-side apply. ```yaml apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: infrastructure spec: syncPolicy: automated: prune: true selfHeal: true generators: - list: - directory: include: "**/kustomization.yaml" recurse: true path: argocd - directory: include: "**/kustomization.yaml" recurse: true path: base-system - directory: include: "**/kustomization.yaml" recurse: true path: database - directory: include: "**/kustomization.yaml" recurse: true path: crds - directory: include: "**/kustomization.yaml" recurse: true path: default - directory: include: "**/kustomization.yaml" recurse: true path: deployment - directory: include: "**/kustomization.yaml" recurse: true path: kube-system template: metadata: name: "{{.path.basename}}" spec: source: repoURL: "" path: "{{.path.path}}" targetRevision: HEAD destination: server: "" namespace: default syncPolicy: automated: prune: true selfHeal: true syncOptions: - ServerSideApply=true - PruneLast=true - RespectIgnoreDifferences=true retry: backoff: duration: "10s" factor: 2 maxDuration: "3m" --- apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: platform spec: syncPolicy: automated: prune: true selfHeal: true generators: - list: - directory: include: "**/kustomization.yaml" recurse: true path: platform/* template: metadata: name: "{{.path.basename}}" spec: source: repoURL: "" path: "{{.path.path}}" targetRevision: HEAD destination: server: "" namespace: default syncPolicy: automated: prune: true selfHeal: true syncOptions: - ServerSideApply=true - PruneLast=true - RespectIgnoreDifferences=true retry: backoff: duration: "5s" factor: 2 maxDuration: "3m" ``` -------------------------------- ### Development Workflow: Kubernetes Manifest Testing Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Demonstrates methods for testing Kubernetes manifest changes without deploying them to a cluster using Kustomize. It shows how to build manifests directly (`kustomize build apps/platform/mastodon`) and how to test Helm chart integrations (`kustomize build --enable-helm apps/argocd`). The `kubectl diff -k` command is used to validate changes against the live cluster state. ```bash # Test without Helm (fast) kustomize build apps/platform/mastodon # Test with Helm (requires network, 2-10 min) kustomize build --enable-helm apps/argocd # Validate changes kubectl diff -k apps/platform/mastodon ``` -------------------------------- ### ArgoCD Troubleshooting Command for Controller Logs Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Command to retrieve logs from the ArgoCD application controller. This is useful for diagnosing synchronization issues and understanding deployment statuses. ```bash kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller ``` -------------------------------- ### Check ClusterSecretStore Configuration Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Verifies the configuration of a ClusterSecretStore, which defines how ExternalSecrets Operator connects to and retrieves secrets from external secret management systems like Bitwarden. ```bash kubectl describe clustersecretstore bitwarden-backend ``` -------------------------------- ### Kubernetes Diff Command for Validation Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Command to validate Kubernetes configuration changes against the live cluster using Kustomize. This helps in reviewing the differences before applying them. ```bash kubectl diff -k apps/platform/mastodon ``` -------------------------------- ### Mastodon Web Autoscaling Configuration Source: https://github.com/goingdark-social/iac/blob/main/README.md Configuration details for Mastodon web service autoscaling based on queue latency, backlog, and memory targets. Includes sidekiq and streaming worker scaling logic. ```text - The `mastodon-web` service now exposes port 9394 so each pod's `/metrics` endpoint is reachable inside the cluster. - VictoriaMetrics scrapes that endpoint through a `VMServiceScrape` and a Prometheus adapter publishes custom metrics for queue latency, backlog, and request rate. - The web autoscaler scales when p95 queue time stays over 35 ms or backlog rises above three requests, and it keeps an 80 % memory target as a safety net. - Scale ups can add two pods every 30 seconds, while scale downs wait three minutes before stepping back to avoid flapping. - Sidekiq default and federation workers scale on the `sidekiq_queue_latency_seconds` metric (10 seconds for default, 30 seconds for federation) so they grow only when the queues back up. - Streaming workers follow the `mastodon_streaming_connected_clients` metric and add capacity once a pod carries around 200 live connections. ``` -------------------------------- ### Kubernetes HPA Definitions for Mastodon Autoscaling Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Defines Horizontal Pod Autoscaler (HPA) configurations for various Mastodon components, leveraging custom metrics from VictoriaMetrics and Prometheus Adapter. It specifies scaling triggers based on queue latency, memory targets, and connected clients. ```yaml apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: mastodon-web-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: mastodon-web minReplicas: 2 maxReplicas: 10 targetCPUUtilizationPercentage: 80 --- apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: mastodon-sidekiq-default-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: mastodon-sidekiq-default minReplicas: 1 maxReplicas: 5 metrics: - type: Pods pods: metric: name: queue_latency_seconds target: type: AverageValue averageValue: "10s" --- apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: mastodon-streaming-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: mastodon-streaming minReplicas: 1 maxReplicas: 5 metrics: - type: Pods pods: metric: name: connected_clients target: type: AverageValue averageValue: "200" ``` -------------------------------- ### Kubernetes Topology Spread Constraints Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Implements Topology Spread Constraints to ensure pods are distributed as evenly as possible across different nodes (identified by kubernetes.io/hostname). The `maxSkew=1` setting aims for maximum distribution, and `whenUnsatisfiable: ScheduleAnyway` ensures that scheduling continues even if perfect distribution cannot be achieved, preventing pod scheduling failures. These constraints are applied via strategic patches. ```yaml # Example patch file (patches/spread-patches.yaml) demonstrating Topology Spread Constraints - op: add path: /spec/template/spec/topologySpreadConstraints value: - maxSkew: 1 whenUnsatisfiable: ScheduleAnyway topologyKey: kubernetes.io/hostname labelSelector: matchLabels: app: mastodon-web # This patch would be applied to Deployments/StatefulSets to spread pods across nodes. ``` -------------------------------- ### Kubernetes Vertical Pod Autoscaler (VPA) Recommender Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Utilizes Vertical Pod Autoscaler in recommender mode to provide resource usage recommendations for pods. It does not automatically apply these recommendations, thus avoiding disruption. Separate VPA resources are defined for different components like controllers and servers to offer tailored advice on CPU and memory requirements. ```yaml apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: mastodon-controller-vpa spec: targetRef: apiVersion: apps/v1 kind: Deployment name: mastodon-controller updatePolicy: updateMode: "Off" # Or "Initial"/"Recreate" if you want it to manage updates resourcePolicy: containerPolicies: - containerName: "*" minAllowed: cpu: "100m" memory: "128Mi" maxAllowed: cpu: "2" memory: "4Gi" ``` -------------------------------- ### Kubernetes PriorityClass for Critical Workloads Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Defines PriorityClass resources to ensure critical system workloads, such as databases (PostgreSQL, Redis, Elasticsearch), are scheduled before less critical application pods. This is achieved by assigning higher priority values to critical workloads, guaranteeing their availability even under high cluster load. These priorities are applied via strategic patches. ```yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: mastodon-critical spec: value: 1000000 globalDefault: false description: "For stateful and critical system components like databases and message queues." --- apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: mastodon-high spec: value: 500000 globalDefault: false description: "For regular application workloads." --- # Example patch file (patches/priority-patches.yaml) - op: add path: /spec/template/spec/priorityClassName value: mastodon-critical # This patch would be applied to Deployments/StatefulSets of critical components. ``` -------------------------------- ### CloudNative-PG PostgreSQL Custom Resource Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Defines a PostgreSQL cluster using the CloudNative-PG operator (postgresql.acid.zalan.do/v1). This configuration specifies the PostgreSQL version (e.g., Spilo 17), enables logical backups to S3, configures SSL with verify-ca, disables password rotation, and sets PVC retention to retain data even after deletion or scaling. It ensures high availability and data durability for PostgreSQL instances. ```yaml apiVersion: postgresql.acid.zalan.do/v1 kind: Postgresql metadata: name: mastodon-database spec: postgresVersion: 17 instances: 3 volume: size: 100Gi storageClass: hcloud-volumes backups: barman: cronJob: schedule: "0 3 * * *" # Daily at 3 AM epoxide: "daily" # objectStorage: # bucket: "mastodon-pg-backups" # endpoint: "s3.nl-ams.scw.cloud" # prefix: "mastodon-db" users: mastodon: "md sẵn sàng" patroni: dynamicConfiguration: postgresql: parameters: ssl: "on" ssl_cert_file: "/etc/postgresql/server.crt" ssl_key_file: "/etc/postgresql/server.key" ssl_ca_file: "/etc/postgresql/ca.crt" ssl_ciphers: "HIGH:MEDIUM:+3DES:!aNULL" ssl_renegotiation_limit: "512MB" # password_encryption: "scram-sha-256" # # Password rotation is disabled by setting it to false or omitting # enablePasswordRotation: false enableSuperuserClient: false // https://github.com/zalando/postgres-operator/blob/master/manifests/minimal-postgres-manifest.yaml // PVC retention policy: retain on delete/scale pvReclaimPolicy: Retain ``` -------------------------------- ### Check Node Resource Allocation Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Inspects the resource allocation (CPU, memory) on Kubernetes nodes to identify potential scheduling conflicts or resource starvation. Useful for diagnosing pod scheduling failures. ```bash kubectl describe nodes ``` -------------------------------- ### View HPA Status Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Retrieves detailed status and configuration of a Horizontal Pod Autoscaler (HPA) object. This helps in understanding current metrics, target values, and scaling events. Requires specifying the HPA name and its namespace. ```bash kubectl describe hpa [name] -n [namespace] ``` -------------------------------- ### Check ExternalSecrets Operator Logs Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Retrieves logs from the ExternalSecrets Operator (ESO) pods. This is crucial for diagnosing issues related to synchronizing secrets from external sources (like Bitwarden) into Kubernetes. ```bash kubectl logs -n external-secrets -l app.kubernetes.io/name=external-secrets ``` -------------------------------- ### Kubernetes Pod Anti-Affinity Configuration Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Configures Pod Anti-Affinity rules to encourage the Kubernetes scheduler to distribute pod replicas across different nodes. This is implemented as a preferredDuringSchedulingIgnoredDuringExecution constraint with a weight, meaning the scheduler will try to satisfy it but will proceed even if it cannot. This enhances high availability by reducing the impact of a single node failure. ```yaml # Example patch file (patches/spread-patches.yaml) demonstrating Pod Anti-Affinity - op: add path: /spec/template/spec/affinity/podAntiAffinity/preferredDuringSchedulingIgnoredDuringExecution value: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: ["mastodon-web", "mastodon-streaming", "mastodon-api"] topologyKey: "kubernetes.io/hostname" # This patch would be applied to Deployments/StatefulSets to spread pods across nodes. ``` -------------------------------- ### Kubernetes Network Policies Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Applies network segmentation within the Kubernetes cluster using NetworkPolicy resources. It follows a default deny-all approach, with explicit allowlist rules for DNS, ingress from the gateway, and egress to specific services like databases and Redis. Each application manages its own NetworkPolicy for granular control. ```yaml # Example NetworkPolicy to allow DNS ingress apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-dns namespace: default spec: podSelector: {} policyTypes: - Ingress ingress: - from: - namespaceSelector: matchLabels: name: kube-dns ports: - protocol: UDP port: 53 - protocol: TCP port: 53 --- # Example NetworkPolicy to allow ingress from Gateway namespace apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-ingress-from-gateway namespace: my-app spec: podSelector: {} policyTypes: - Ingress ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: gateway-namespace # Replace with actual gateway namespace --- # Example NetworkPolicy to allow egress to database apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-egress-to-db namespace: my-app spec: podSelector: {} policyTypes: - Egress egress: - to: - podSelector: matchLabels: app: database ports: - protocol: TCP port: 5432 ``` -------------------------------- ### Kubernetes Horizontal Pod Autoscaler (HPA) Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Configures Horizontal Pod Autoscaler to automatically scale the number of pods in a deployment based on custom metrics (e.g., request queue duration) and resource utilization (CPU/memory). It includes specific scale-up and scale-down policies with stabilization windows to prevent rapid fluctuations. Metrics are collected via VMServiceScrape and exposed by Prometheus Adapter. ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: mastodon-web-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: mastodon-web minReplicas: 3 maxReplicas: 10 metrics: - type: Pods pods: metric: name: ruby_http_request_queue_duration_seconds_p95 target: type: AverageValue averageValue: "0.5" - type: Resource resource: name: memory target: type: Utilization averageUtilization: 70 behavior: scaleDown: stabilizationWindowSeconds: 180 policies: - type: Pods value: 1 periodSeconds: 60 scaleUp: stabilizationWindowSeconds: 30 policies: - type: Pods value: 2 periodSeconds: 30 ``` -------------------------------- ### Check ExternalSecret Object Status Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Examines the status and events associated with an ExternalSecret custom resource. This helps in understanding why a secret might not be syncing correctly from an external store. ```bash kubectl describe externalsecret [name] -n [namespace] ``` -------------------------------- ### Verify Prometheus Adapter Health Source: https://github.com/goingdark-social/iac/blob/main/CLAUDE.md Checks if the Prometheus adapter is healthy and accessible at the Kubernetes API level. This is a fundamental step to ensure metrics are available for Horizontal Pod Autoscalers (HPA). ```bash kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.