### Provision Infrastructure with Terraform CLI

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/manual-deployment.md

Commands to initialize, plan, and apply Terraform configurations for provisioning the Kubernetes infrastructure on Proxmox. Assumes Terraform is installed and configured.

```bash
cd terraform-provision

# Initialize Terraform
terraform init

# Plan changes
terraform plan -var-file=env/dev/main.tfvars

# Apply changes
terraform apply -var-file=env/dev/main.tfvars
```

--------------------------------

### Configure Kubernetes Cluster with Ansible CLI

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/manual-deployment.md

Steps to configure the provisioned Kubernetes cluster using Ansible. This involves generating an inventory file and running the main playbook. Ensure Ansible and the necessary scripts are available.

```bash
cd ../ansible

# Generate inventory from JSON files
../scripts/generate-all-hosts.sh

# Run the playbook
ansible-playbook -i inventory/hosts.ini site.yaml
```

--------------------------------

### Configure Environment Variables for Proxmox and S3

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/manual-deployment.md

Sets up essential environment variables for connecting to Proxmox and configuring an S3 backend for Terraform state management. Ensure sensitive information like passwords and keys are handled securely.

```bash
# Proxmox credentials
export PM_API_URL="https://proxmox.example.com:8006/api2/json"
export PM_USER="root@pam"
export PM_PASSWORD="your-password"

# S3 backend for Terraform state
export AWS_ACCESS_KEY_ID="your-minio-access-key"
export AWS_SECRET_ACCESS_KEY="your-minio-secret-key"
```

--------------------------------

### Ansible: Main Playbook for RKE2 Deployment

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

The primary Ansible playbook that orchestrates the entire RKE2 Kubernetes cluster deployment and application installation process. It defines the sequence of tasks and roles to be executed on the provisioned infrastructure, ensuring a consistent and automated setup.

```yaml
## Ansible Site Playbook

- name: RKE2 Kubernetes Cluster Deployment
  hosts: all
  become: true
  gather_facts: true

  vars_files:
    - vars/main.yaml

  roles:
    - rke2-bootstrap
    - rke2-config
    - rke2-nodes
    - rke2-apps

  tasks:
    - name: Ensure RKE2 service is enabled and started on control plane nodes
      ansible.builtin.systemd:
        name: k3s
        enabled: true
        state: started
      when: inventory_hostname in groups['rke2_server']

    - name: Wait for RKE2 API to be available
      ansible.builtin.wait_for:
        port: 6443
        host: "{{ hostvars[groups['rke2_server'][0]]['ansible_host'] }}"
        delay: 10
        timeout: 600
      when: inventory_hostname == groups['rke2_server'][0]

    - name: Deploy Kubernetes applications using ArgoCD
      ansible.builtin.command: argocd app create -f manifests/argocd-apps.yaml
      args:
        chdir: "{{ playbook_dir }}/../argocd/"
      when: inventory_hostname == groups['rke2_server'][0]

```

--------------------------------

### Manual Local Deployment: Environment Variable Setup

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

This snippet demonstrates how to set up essential environment variables for manual local deployment. It includes variables for Vault address and token, and the environment name, which are crucial for authenticating with Vault and targeting the correct deployment environment.

```bash
# Set environment variables
export VAULT_ADDR="https://vault.example.com"
export VAULT_TOKEN="your-vault-token"
export ENV_NAME="dev"

```

--------------------------------

### Ansible Playbooks for RKE2 Cluster Deployment

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

Ansible playbooks to automate the deployment of an RKE2 Kubernetes cluster. Includes roles for node preparation, RKE2 server and agent setup, and application deployment. It also demonstrates exporting Kubernetes credentials.

```yaml
- name: Prepare all nodes & Download RKE2
  hosts: rke2
  gather_facts: true
  roles:
    - download_rke2

- name: Bootstrap RKE2 Servers
  hosts: servers
  gather_facts: true
  roles:
    - add_server

- name: Add additional RKE2 agents & longhorn agents
  hosts: agents, longhorn
  gather_facts: true
  roles:
    - add_agent

- name: Deploy applications
  hosts: servers
  gather_facts: true
  run_once: true
  roles:
    - role: apply_kube_vip
    - role: deploy_helm_apps

- name: Export Kubernetes credentials for Vault
  hosts: servers[0]
  gather_facts: false
  tasks:
    - name: Get Kubernetes CA certificate
      ansible.builtin.command:
        cmd: kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}'
      environment:
        KUBECONFIG: /etc/rancher/rke2/rke2.yaml
      register: k8s_ca_cert_b64
      changed_when: false

    - name: Decode and save Kubernetes CA certificate locally
      ansible.builtin.copy:
        content: "{{ k8s_ca_cert_b64.stdout | b64decode }}"
        dest: "{{ playbook_dir }}/k8s-ca.crt"
        mode: "0644"
      delegate_to: localhost
```

--------------------------------

### Define Kubernetes Node Configurations in JSON

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/manual-deployment.md

Specifies the hostnames and IP addresses for Kubernetes nodes. This JSON structure is used by Terraform to provision the cluster nodes. Ensure the IP addresses are correctly assigned within your network.

```json
[
  {
    "hostname": "k8s-server-01",
    "ip": "10.69.0.11"
  },
  {
    "hostname": "k8s-server-02",
    "ip": "10.69.0.12"
  }
]
```

--------------------------------

### Deploy Vault Admin Resources with Terraform

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/vault-setup.md

Initializes and applies Terraform configurations to deploy Vault admin resources, including OIDC backends, environment policies, SSH CAs, Vault roles, and user group memberships. Requires AWS credentials and Vault address to be set as environment variables.

```bash
cd terraform-admin

# Set the same address as a Terraform variable (used for OIDC issuer URL)
export TF_VAR_vault_addr="$VAULT_ADDR"

# Set your S3 backend credentials
export AWS_ACCESS_KEY_ID=$(vault kv get -field=access_key kv/shared/minio)
export AWS_SECRET_ACCESS_KEY=$(vault kv get -field=secret_key kv/shared/minio)

terraform init
terraform apply
```

--------------------------------

### Store Secrets in Vault using CLI

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/vault-setup.md

Stores shared and environment-specific secrets in HashiCorp Vault using the 'vault kv put' command. Requires Vault address and token to be set as environment variables.

```bash
# Set Vault address and authenticate
export VAULT_ADDR="https://your-vault-address"
export VAULT_TOKEN="your-vault-token"

# Shared secrets (used by both dev and prod)
vault kv put kv/shared/minio access_key="..." secret_key="..."
vault kv put kv/shared/proxmox endpoint="..." username="..." password="..."
vault kv put kv/shared/cloudflare api_token="..." domain="..." email="..."

# Dev environment secrets
vault kv put kv/dev/ip vip="10.69.0.10" cidr="24" lb_range="10.69.0.50-10.69.0.100" ingress="10.69.0.50"
vault kv put kv/dev/rke2 token="your-rke2-token"

# Prod environment secrets
vault kv put kv/prod/ip vip="10.69.1.10" cidr="24" lb_range="10.69.1.50-10.69.1.100" ingress="10.69.1.50"
vault kv put kv/prod/rke2 token="your-rke2-token"
```

--------------------------------

### Destroy Kubernetes Infrastructure with Terraform CLI

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/manual-deployment.md

Command to destroy the provisioned Kubernetes infrastructure using Terraform. This is useful for cleaning up resources after testing or when no longer needed. Ensure you are in the correct directory.

```bash
cd terraform-provision
terraform destroy -var-file=env/dev/main.tfvars
```

--------------------------------

### Generate Ansible Inventory and Deploy RKE2 Cluster (Shell)

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

This script first generates an Ansible inventory file and then proceeds to install Ansible roles and deploy the RKE2 cluster using a playbook. It utilizes a previously generated SSH key for authentication and specifies environment-specific variables.

```shell
../scripts/generate-all-hosts.sh ${ENV_NAME}

export OIDC_ISSUER_URL="${VAULT_ADDR}/v1/identity/oidc/provider/${ENV_NAME}"
ansible-galaxy install -r requirements.yml
ansible-playbook -e "env=${ENV_NAME}" --private-key local_key site.yaml
```

--------------------------------

### Configure Vault Users with Terraform Variables

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/vault-setup.md

Defines Vault user configurations, including email, password, and group memberships for different environments, within a `terraform.tfvars` file. This HCL file is used by Terraform to manage Vault user resources.

```bash
cd terraform-admin
cp terraform.tfvars.example terraform.tfvars
```

```hcl
vault_addr = "https://vault.example.com"

users = {
  # Admin user with full access to both dev and prod
  "admin-user" = {
    email    = "admin@example.com"
    password = "CHANGE-ME-SECURE-PASSWORD"
    groups = {
      dev_role  = "admins"
      prod_role = "admins"
    }
  }

  # Developer with dev access only
  "developer" = {
    email    = "developer@example.com"
    password = "CHANGE-ME-SECURE-PASSWORD"
    groups = {
      dev_role  = "developers"
      prod_role = null # No prod access
    }
  }

  # Viewer with read-only access to both environments
  "viewer" = {
    email    = "viewer@example.com"
    password = "CHANGE-ME-SECURE-PASSWORD"
    groups = {
      dev_role  = "viewers"
      prod_role = "viewers"
    }
  }
}
```

--------------------------------

### Authenticate and Access Kubernetes Cluster

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/cluster-access.md

These bash commands set the KUBECONFIG environment variable to use the created OIDC configuration, clear any cached tokens, and then attempt to access the Kubernetes cluster by getting nodes. `kubectl auth whoami` and `kubectl auth can-i` are used to verify authentication and permissions.

```bash
export KUBECONFIG=~/.kube/dev-config.yml

# Clear any cached tokens
rm -rf ~/.kube/cache/oidc-login/

# This will prompt for Vault userpass credentials
kubectl get nodes

# Verify your identity
kubectl auth whoami

# Check permissions
kubectl auth can-i create pods

```

--------------------------------

### Terraform: Generate Cloud-Init User Data for VMs

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

Terraform resource to create cloud-init user data for Proxmox VMs. It configures hostname, timezone, users, updates packages, installs essential utilities, and sets up SSH to trust a provided public key for secure access. This ensures nodes are ready for further configuration.

```hcl
resource "proxmox_virtual_environment_file" "user_data_cloud_config" {
  content_type = "snippets"
  datastore_id = var.vm_datastore_id
  node_name    = var.vm_node_name

  source_raw {
    data = <<-EOF
    #cloud-config
    host-name: ubuntu_cloud_image
    timezone: America/New_York
    users:
      - default
      - name: ubuntu
        groups:
          - sudo
        shell: /bin/bash
        sudo: ALL=(ALL) NOPASSWD:ALL
    package_update: true
    packages:
      - qemu-guest-agent
      - net-tools
      - curl
      - cryptsetup
    write_files:
      - path: /etc/ssh/ca.pub
        content: |
          ${var.proxmox_ssh_public_key}
        permissions: '0644'
    runcmd:
      - systemctl start qemu-guest-agent
      - echo "TrustedUserCAKeys /etc/ssh/ca.pub" >> /etc/ssh/sshd_config
      - systemctl restart sshd
    EOF

    file_name = "${var.env}-user-data-cloud-config.yaml"
  }
}
```

--------------------------------

### Initialize, Plan, and Apply Terraform Infrastructure (Shell)

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

This script navigates to the Terraform configuration directory, initializes the backend with a specific state key, plans the infrastructure changes using environment-specific variables, and applies the plan to provision resources.

```shell
cd terraform-provision
terraform init --backend-config="key=${ENV_NAME}.tfstate"
terraform plan -var-file="env/${ENV_NAME}/main.tfvars" -var="env=${ENV_NAME}"
terraform apply -var-file="env/${ENV_NAME}/main.tfvars" -var="env=${ENV_NAME}"
```

--------------------------------

### GitHub Actions Workflow - Plan and Apply (YAML)

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/github-actions-setup.md

A GitHub Actions workflow file that automates the deployment process. It includes steps for linting, connecting to Tailscale VPN, authenticating with Vault, provisioning infrastructure with Terraform, and configuring the RKE2 cluster with Ansible. The workflow triggers on pull requests for planning and on push to master for applying changes.

```yaml
# This is a conceptual representation of the workflow steps described in the text.
# The actual workflow file would be more detailed.
name: Automated Deployment

on:
  pull_request:
    branches: [ master ]
  push:
    branches: [ master ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Lint Terraform and Ansible
        run: |
          # Add linting commands for Terraform and Ansible here
          echo "Running linting checks..."

      - name: Connect to Tailscale VPN (Optional)
        # This step would involve configuring Tailscale using TS_OAUTH_CLIENT_ID and TS_OAUTH_SECRET
        run: echo "Connecting to Tailscale VPN..."

      - name: Authenticate to Vault
        # This step uses OIDC to authenticate with Vault without hardcoded secrets
        run: echo "Authenticating to Vault..."

      - name: Retrieve Secrets from Vault
        # This step would fetch secrets needed for deployment
        run: echo "Retrieving secrets from Vault..."

      - name: Provision VMs with Terraform
        working-directory: ./terraform-provision
        env:
          ARM_CLIENT_ID: "${{ secrets.ARM_CLIENT_ID }}"
          ARM_CLIENT_SECRET: "${{ secrets.ARM_CLIENT_SECRET }}"
          ARM_SUBSCRIPTION_ID: "${{ secrets.ARM_SUBSCRIPTION_ID }}"
          ARM_TENANT_ID: "${{ secrets.ARM_TENANT_ID }}"
          VAULT_ADDR: "${{ vars.VAULT_ADDR }}"
          ENV_NAME: "${{ vars.ENV_NAME }}"
          DESTROY: "${{ vars.DESTROY }}"
        run: |
          terraform init
          if [ "${{ vars.DESTROY }}" = "true" ]; then
            terraform destroy -auto-approve
          else
            terraform apply -auto-approve
          fi

      - name: Configure RKE2 cluster with Ansible
        # This step would run Ansible playbooks to configure the cluster
        run: echo "Configuring RKE2 cluster with Ansible..."

      - name: Deploy applications via Helm
        # This step would execute the deploy_helm_apps role
        run: echo "Deploying applications with Helm..."

```

--------------------------------

### GitHub Actions Workflow: Terraform Provisioning and Ansible Bootstrap

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

This YAML file defines a GitHub Actions workflow that automates the deployment of Kubernetes on Proxmox. It includes steps for checking out code, connecting to Tailscale, setting up Terraform, fetching secrets from Vault, initializing, planning, and applying Terraform configurations, and then bootstrapping the nodes with Ansible. The workflow triggers on push and pull requests to the master branch, or manually via workflow_dispatch.

```yaml
name: Provision and Bootstrap

on:
  push:
    branches: [master]
  pull_request:
    branches: [master]
  workflow_dispatch:

jobs:
  terraform-plan-apply:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
      pull-requests: write

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Connect to Tailscale
        uses: tailscale/github-action@v2
        with:
          oauth-client-id: ${{ secrets.TS_OAUTH_CLIENT_ID }}
          oauth-secret: ${{ secrets.TS_OAUTH_SECRET }}
          tags: tag:ci

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.6.6

      - name: Set Vault Role
        id: set-role
        run: |
          if [ "${{ github.event_name }}" == "push" ]; then
            echo "role=${{ vars.ENV_NAME }}-github-actions-push-role" >> $GITHUB_OUTPUT
          else
            echo "role=${{ vars.ENV_NAME }}-github-actions-pr-role" >> $GITHUB_OUTPUT
          fi

      - name: Import Terraform Secrets from Vault
        uses: hashicorp/vault-action@v3
        with:
          method: jwt
          url: ${{ vars.VAULT_ADDR }}
          role: ${{ steps.set-role.outputs.role }}
          secrets: |
            kv/shared/data/minio access_key | AWS_ACCESS_KEY_ID ;
            kv/shared/data/minio secret_key | AWS_SECRET_ACCESS_KEY ;
            kv/${{ vars.ENV_NAME }}/data/ssh_ca_public_key public_key | TF_VAR_proxmox_ssh_public_key

      - name: Terraform Init
        run: terraform init --backend-config="key=${{ vars.ENV_NAME }}.tfstate"
        working-directory: ./terraform-provision

      - name: Terraform Plan
        env:
          TF_VAR_env: ${{ vars.ENV_NAME }}
        run: terraform plan -var-file="env/${{ vars.ENV_NAME }}/main.tfvars" -out .planfile
        working-directory: ./terraform-provision

      - name: Terraform Apply
        if: github.ref == 'refs/heads/master' && github.event_name == 'push'
        env:
          TF_VAR_env: ${{ vars.ENV_NAME }}
        run: terraform apply -auto-approve .planfile
        working-directory: ./terraform-provision

  ansible-bootstrap:
    runs-on: ubuntu-latest
    needs: terraform-plan-apply
    if: github.ref == 'refs/heads/master' && github.event_name == 'push' && vars.DESTROY != 'true'

    permissions:
      contents: read
      id-token: write

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Import Secrets from Vault
        uses: hashicorp/vault-action@v3
        id: vault_auth
        with:
          method: jwt
          url: ${{ vars.VAULT_ADDR }}
          role: ${{ steps.set-role.outputs.role }}
          outputToken: true
          secrets: |
            kv/shared/data/cloudflare * | SSL_ ;
            kv/${{vars.ENV_NAME}}/data/ip * | IP_ ;
            kv/${{vars.ENV_NAME}}/data/rke2 * | RKE2_ ;
            kv/${{vars.ENV_NAME}}/data/oidc * | OIDC_ ;

      - name: Generate and Sign SSH Key
        env:
          VAULT_ADDR: ${{ vars.VAULT_ADDR }}
          VAULT_TOKEN: ${{ steps.vault_auth.outputs.vault_token }}
        run: |
          ssh-keygen -t rsa -b 4096 -f ./runner_key -q -N ""
          vault write -field=signed_key ${{vars.ENV_NAME}}-ssh-client-signer/sign/github-runner \
            public_key=@./runner_key.pub \
            valid_principals=ubuntu > runner_key-cert.pub
          chmod 600 runner_key
          chmod 644 runner_key-cert.pub
        working-directory: ./ansible

      - name: Generate Ansible Inventory
        run: ../scripts/generate-all-hosts.sh ${{ vars.ENV_NAME }}
        working-directory: ./ansible

      - name: Install Ansible
        run: |
          sudo apt update
          sudo apt install -y ansible

      - name: Install Ansible collections
        run: ansible-galaxy install -r requirements.yml
        working-directory: ./ansible

      - name: Run playbook
        env:
          OIDC_ISSUER_URL: ${{ vars.VAULT_ADDR }}/v1/identity/oidc/provider/${{ vars.ENV_NAME }}
        run: ansible-playbook -e "env=${{ vars.ENV_NAME }}" --private-key runner_key site.yaml
        working-directory: ./ansible

      - name: Configure Vault Kubernetes Auth
        env:
          VAULT_ADDR: ${{ vars.VAULT_ADDR }}
          VAULT_TOKEN: ${{ steps.vault_auth.outputs.vault_token }}
        run: |
          vault write auth/${ENV_NAME}-kubernetes/config \
            kubernetes_host="https://${{ env.IP_VIP }}:6443"
            kubernetes_ca_cert=@k8s-ca.crt \
            disable_local_ca_jwt=true
        working-directory: ./ansible

```

--------------------------------

### GitHub Actions CI/CD Workflow (YAML)

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

A complete GitHub Actions workflow designed for automated Terraform provisioning and Ansible cluster deployment. This workflow orchestrates the CI/CD pipeline for the project.

```yaml
## GitHub Actions CI/CD Pipeline

Complete GitHub Actions workflow for automated Terraform provisioning and Ansible cluster deployment.

```yaml

```

--------------------------------

### Terraform: Provision Proxmox VMs with Cloud-Init

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

Terraform module for creating Proxmox virtual machines. It utilizes cloud-init for initial VM configuration, specifying resources like CPU, memory, disk, network settings, and cloud-init files. This module is a core component for setting up Kubernetes nodes.

```hcl
module "k8s_server" {
  source = "./modules/vm"

  name         = "dev-server1"
  node_name    = "pve"
  vm_id        = 111
  cpu_cores    = 4
  cpu_type     = "x86-64-v2-AES"
  memory_mb    = 4096
  disk_size_gb = 64

  datastore_id = "local-lvm"
  bridge       = "vmbr1"
  dns_server   = "10.69.0.1"

  ip_address = "10.69.1.111/16"
  ip_gateway = "10.69.0.1"

  disk_file_id      = proxmox_virtual_environment_download_file.ubuntu_cloud_image.id
  user_data_file_id = proxmox_virtual_environment_file.user_data_cloud_config.id
  meta_data_file_id = proxmox_virtual_environment_file.meta_data_cloud_config["dev-server1"].id
}
```

--------------------------------

### Retrieve Proxmox and S3 Credentials from Vault (Shell)

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

This script exports Proxmox and S3 (Minio) credentials as environment variables by fetching them from HashiCorp Vault. It assumes the Vault CLI is configured and accessible. The credentials are used for subsequent infrastructure provisioning and configuration steps.

```shell
export TF_VAR_proxmox_endpoint=$(vault kv get -field=endpoint kv/shared/proxmox)
export TF_VAR_proxmox_username=$(vault kv get -field=username kv/shared/proxmox)
export TF_VAR_proxmox_password=$(vault kv get -field=password kv/shared/proxmox)
export TF_VAR_proxmox_ssh_public_key=$(vault kv get -field=public_key kv/${ENV_NAME}/ssh_ca_public_key)

export AWS_ACCESS_KEY_ID=$(vault kv get -field=access_key kv/shared/minio)
export AWS_SECRET_ACCESS_KEY=$(vault kv get -field=secret_key kv/shared/minio)
```

--------------------------------

### Troubleshoot: Verify ClusterSecretStore Status

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/external-secrets-vault-integration.md

This command retrieves the YAML configuration of a `ClusterSecretStore` named `vault-backend`. By examining the `status.conditions` field in the output, you can determine if the connection to the secret store (in this case, Vault) is healthy and identify any connectivity problems.

```bash
kubectl get clustersecretstore vault-backend -o yaml

```

--------------------------------

### Generate SSH Key and Certificate for Authentication (Shell)

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

This script generates an RSA SSH key pair, requests a signed SSH certificate from Vault using the public key, and sets appropriate permissions for the private key. This certificate is used for secure authentication during Ansible deployments.

```shell
cd ../ansible
ssh-keygen -t rsa -b 4096 -f ./local_key -q -N ""
vault write -field=signed_key ${ENV_NAME}-ssh-client-signer/sign/local-user \
  public_key=@./local_key.pub \
  valid_principals=ubuntu > local_key-cert.pub
chmod 600 local_key
```

--------------------------------

### JSON: Define Environment-Specific Node Configurations

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

A JSON structure defining configuration for individual nodes within an environment. It maps node names to their Proxmox VM ID, the Proxmox node they reside on, their role (e.g., 'servers'), and their IP address. This file serves as input for automation scripts.

```json
{
  "dev-server1": {
    "vm_id": 111,
    "node": "pve",
    "role": "servers",
    "address": "10.69.1.111/16"
  },
  "dev-server2": {
    "vm_id": 112,
    "node": "pve2",
    "role": "servers",
    "address": "10.69.1.112/16"
  },
  "dev-server3": {
    "vm_id": 113,
    "node": "pve3",
    "role": "servers",
    "address": "10.69.1.113/16"
  }
}
```

--------------------------------

### Troubleshoot: Verify ServiceAccount Token

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/external-secrets-vault-integration.md

These commands help verify the ServiceAccount token used by the External Secrets Operator (ESO) pod. First, it shows how to execute commands inside an ESO pod, and then how to display the content of the ServiceAccount token file. This token is crucial for ESO to authenticate with Vault.

```bash
# Exec into ESO pod
kubectl exec -n external-secrets -it <pod-name> -- sh

# Check if SA token exists
cat /var/run/secrets/kubernetes.io/serviceaccount/token

```

--------------------------------

### Vault KV Secret Management Commands (Bash)

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

Provides Vault CLI commands for storing and retrieving secrets, including shared infrastructure secrets (Minio, Proxmox, Cloudflare) and environment-specific secrets (development IP, RKE2).

```bash
# Set Vault address and authenticate
export VAULT_ADDR="https://vault.example.com"
export VAULT_TOKEN="your-vault-token"

# Store shared secrets
vault kv put kv/shared/minio \
  access_key="minio-access-key" \
  secret_key="minio-secret-key"

vault kv put kv/shared/proxmox \
  endpoint="https://proxmox.example.com:8006" \
  username="terraform@pve" \
  password="secure-password"

vault kv put kv/shared/cloudflare \
  api_token="cloudflare-api-token" \
  domain="example.com" \
  email="admin@example.com"

# Store environment-specific secrets
vault kv put kv/dev/ip \
  vip="10.69.0.10" \
  cidr="24" \
  lb_range="10.69.0.50-10.69.0.100" \
  ingress="10.69.0.50"

vault kv put kv/dev/rke2 token="secure-rke2-token"

# Retrieve secrets
vault kv get -field=access_key kv/shared/minio
vault kv get -format=json kv/dev/ip | jq -r '.data.data.vip'
```

--------------------------------

### Shell Script: Generate Ansible Inventory from JSON

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

A bash script to generate an Ansible inventory file from JSON definitions of Kubernetes and Longhorn nodes. It combines node information, extracts unique roles, and formats the output into an INI file suitable for Ansible, mapping hostnames to IP addresses and organizing them into groups.

```bash
#!/bin/bash
# scripts/generate-all-hosts.sh

set -e

ENV=$1
ROOT_DIR="$(git rev-parse --show-toplevel)"
K8S_NODES_PATH="$ROOT_DIR/terraform-provision/env/$ENV/k8s_nodes.json"
LONGHORN_NODES_PATH="$ROOT_DIR/terraform-provision/env/$ENV/longhorn_nodes.json"
HOSTS_INI_PATH="$ROOT_DIR/ansible/inventory/hosts.ini"

# Combine JSON files and generate inventory
COMBINED_JSON=$(jq -s 'add' "$K8S_NODES_PATH" "$LONGHORN_NODES_PATH")
ROLES=$(echo "$COMBINED_JSON" | jq -r 'map(.role) | unique | .[]')

{
  for role in $ROLES;
  do
    group_name=$role
    echo "[$group_name]"
    echo "$COMBINED_JSON" | jq -r --arg role "$role" 
      'to_entries[] | select(.value.role == $role) | "\(.key) \(.value.address)"' | \
      while read -r key address;
      do
        echo "$key ansible_host=${address%/16}"
      done
    echo ""
  done

  echo "[rke2:children]"
  for role in $ROLES;
  do
    echo "$role"
  done
} > "$HOSTS_INI_PATH"

echo "Successfully generated $HOSTS_INI_PATH for environment $ENV"
```

--------------------------------

### Troubleshoot: Check ESO Logs

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/external-secrets-vault-integration.md

This command retrieves the logs for the External Secrets Operator (ESO) pods. It's a common first step in troubleshooting to identify any errors or issues during secret synchronization. The logs are fetched from the `external-secrets` namespace for pods labeled with `app.kubernetes.io/name=external-secrets`.

```bash
kubectl logs -n external-secrets -l app.kubernetes.io/name=external-secrets

```

--------------------------------

### Create Kubeconfig for OIDC Authentication

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/cluster-access.md

This YAML configuration defines a Kubernetes config file that uses `kubectl oidc-login` to obtain OIDC tokens from HashiCorp Vault for authentication. It requires the cluster server URL, OIDC issuer URL, and client ID. Ensure placeholders are replaced with your specific environment details.

```yaml
apiVersion: v1
kind: Config
clusters:
  - cluster:
      server: https://10.69.1.110:6443 # Replace with your VIP
      insecure-skip-tls-verify: true # Only if using self-signed certs
    name: dev-rke2
contexts:
  - context:
      cluster: dev-rke2
      user: vault-oidc
    name: dev-rke2
current-context: dev-rke2
users:
  - name: vault-oidc
    user:
      exec:
        apiVersion: client.authentication.k8s.io/v1beta1
        command: kubectl
        args:
          - oidc-login
          - get-token
          - --oidc-issuer-url=https://vault.example.com/v1/identity/oidc/provider/dev
          - --oidc-client-id=<fetch-from-Vault>
          - --oidc-extra-scope=dev-email
          - --oidc-extra-scope=dev-profile
          - --oidc-extra-scope=dev-groups

```

--------------------------------

### Helm Applications Configuration

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

YAML configuration defining Helm applications to be deployed on the Kubernetes cluster. This includes settings for cert-manager, traefik, longhorn, external-secrets, and argo-cd.

```yaml
# ansible/inventory/group_vars/all/helm.yaml
helm_applications:
  - name: cert-manager
    chart: cert-manager
    version: v1.17.2
    repo: https://charts.jetstack.io
    namespace: cert-manager
    create_namespace: true
    set_values:
      crds.enabled: "true"
      extraArgs[0]: "--dns01-recursive-nameservers-only"
      extraArgs[1]: "--dns01-recursive-nameservers=1.1.1.1:53"
    additional_manifests:
      - cert-manager-issuer

  - name: traefik
    chart: traefik
    repo: https://traefik.github.io/charts
    namespace: traefik
    create_namespace: true
    values_content: |
      service:
        type: LoadBalancer
        spec:
          loadBalancerIP: {{ vip_ingress_ip }}
      tlsStore:
        default:
          defaultCertificate:
            secretName: wildcard-tls
      ingressRoute:
        dashboard:
          enabled: true
          matchRule: Host(`traefik.{{ ssl_local_domain }}`)
          entryPoints: ["websecure"]
      ports:
        web:
          redirections:
            entryPoint:
              to: websecure
              scheme: https
              permanent: true
    additional_manifests:
      - traefik-wildcard-cert

  - name: longhorn
    chart: longhorn
    version: v1.8.1
    repo: https://charts.longhorn.io
    namespace: longhorn-system
    create_namespace: true
    set_values:
      defaultSettings.createDefaultDiskLabeledNodes: "true"
      persistence.reclaimPolicy: "Retain"
    ingress:
      enabled: true
      host: "longhorn.{{ ssl_local_domain }}"
      service_name: longhorn-frontend
      service_port: 80
    additional_manifests:
      - longhorn-iscsi
      - longhorn-nfs

  - name: external-secrets
    chart: external-secrets
    version: v0.19.2
    repo: https://charts.external-secrets.io
    namespace: external-secrets
    create_namespace: true
    values_content: |
      installCRDs: true
      metrics:
        service:
          enabled: true
        serviceMonitor:
          enabled: true
    additional_manifests:
      - external-secrets-rbac
      - external-secrets-cluster-store

  - name: argo-cd
    chart: argo-cd
    version: v8.0.9
    repo: https://argoproj.github.io/argo-helm
    namespace: argo-cd
    create_namespace: true
    values_content: |
      configs:
        params:
          server.insecure: true
    ingress:
      enabled: true
      host: "argo.{{ ssl_local_domain }}"
      service_name: argo-cd-argocd-server
      service_port: 80
```

--------------------------------

### Terraform Variable Configuration for Proxmox VMs (HCL)

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

This HCL code defines environment-specific Terraform variables for provisioning Proxmox virtual machines. It includes settings for general VM parameters, Kubernetes cluster nodes, and Longhorn storage nodes, enabling customization for different environments like 'dev'.

```hcl
# terraform-provision/env/dev/main.tfvars

# General VM settings
vm_node_name    = "pve"
vm_datastore_id = "truenas"
vm_bridge       = "vmbr1"
vm_timezone     = "America/New_York"
vm_ip_gateway   = "10.69.0.1"
dns_server      = "10.69.0.1"

# Kubernetes cluster VM settings
k8s_cpu_cores    = 4
k8s_cpu_type     = "x86-64-v2-AES"
k8s_memory_mb    = 4096
k8s_disk_size_gb = 64
k8s_datastore_id = "local-lvm"

# Longhorn storage VM settings
longhorn_cpu_cores    = 2
longhorn_cpu_type     = "x86-64-v2-AES"
longhorn_memory_mb    = 2048
longhorn_disk_size_gb = 300
longhorn_datastore_id = "local-lvm"
```

--------------------------------

### Configure Vault Kubernetes Authentication (Shell)

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

This command configures Vault to act as an authentication method for a Kubernetes cluster. It specifies the Kubernetes API host, the CA certificate for verifying the API server, and disables local CA JWT verification for enhanced security.

```shell
vault write auth/${ENV_NAME}-kubernetes/config \
  kubernetes_host="https://$(vault kv get -field=vip kv/${ENV_NAME}/ip):6443" \
  kubernetes_ca_cert=@k8s-ca.crt \
  disable_local_ca_jwt=true
```

--------------------------------

### Troubleshoot: Test Vault Authentication

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/external-secrets-vault-integration.md

This script tests the authentication between the External Secrets Operator (ESO) ServiceAccount and HashiCorp Vault. It retrieves the ESO pod's ServiceAccount token and then uses it to attempt a login against a Vault Kubernetes auth method. This helps diagnose issues with Vault authentication configuration.

```bash
# Get SA token from pod
SA_TOKEN=$(kubectl exec -n external-secrets <pod-name> -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)

# Test Vault login
vault write auth/dev-kubernetes/login role=external-secrets jwt=$SA_TOKEN

```

--------------------------------

### Troubleshooting Token Expiration

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/cluster-access.md

This command is used to resolve authentication errors caused by expired OIDC tokens by clearing the cached tokens. It's a common troubleshooting step when facing issues accessing the Kubernetes cluster after the initial login.

```bash
rm -rf ~/.kube/cache/oidc-login/

```

--------------------------------

### Vault OIDC Scope and Provider Configuration (Terraform)

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

Defines a custom OIDC scope for group information and configures an OIDC provider within HashiCorp Vault. It utilizes Terraform to manage Vault resources, enabling Kubernetes integration for authentication.

```terraform
resource "vault_identity_oidc_scope" "groups" {
  name        = "${var.env}-groups"
  description = "Groups scope for ${var.env} Kubernetes"
  template    = <<-EOT
    {
      "groups": {{identity.entity.groups.names}}
    }
  EOT
}

# Create OIDC provider
resource "vault_identity_oidc_provider" "kubernetes" {
  name = var.env

  https_enabled = true
  issuer_host   = replace(var.vault_addr, "https://", "")

  allowed_client_ids = [
    vault_identity_oidc_client.kubernetes.client_id
  ]

  scopes_supported = [
    vault_identity_oidc_scope.email.name,
    vault_identity_oidc_scope.profile.name,
    vault_identity_oidc_scope.groups.name,
  ]
}
```

--------------------------------

### Vault OIDC Client and Secret Storage (Terraform)

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

Configures an OIDC client for Kubernetes integration and stores the client ID in Vault's KV store using Terraform. This allows Kubernetes to authenticate with Vault via OIDC.

```terraform
# Create OIDC client for Kubernetes
resource "vault_identity_oidc_client" "kubernetes" {
  name = "${var.env}-kubernetes"
  key  = vault_identity_oidc_key.kubernetes.name

  redirect_uris = var.redirect_uris

  assignments = [
    vault_identity_oidc_assignment.kubernetes.name
  ]

  id_token_ttl     = 3600
  access_token_ttl = 3600

  client_type = "public"
}

# Store OIDC client_id in Vault KV
resource "vault_generic_secret" "oidc_client_id" {
  path = "kv/${var.env}/oidc"

  data_json = jsonencode({
    client_id = vault_identity_oidc_client.kubernetes.client_id
  })
}
```

--------------------------------

### Terraform Vault OIDC Kubernetes Authentication Module

Source: https://context7.com/phuchoang2603/kubernetes-proxmox/llms.txt

Terraform module to configure Vault as an OIDC provider for Kubernetes authentication. It creates identity groups for RBAC and an OIDC key for signing tokens.

```hcl
# terraform-admin/modules/vault-oidc-kubernetes/main.tf

# Create Vault groups for Kubernetes RBAC
resource "vault_identity_group" "kubernetes_admins" {
  name     = "${var.env}-kubernetes-admins"
  type     = "internal"
  policies = [vault_policy.admin_policy.name]

  metadata = {
    environment = var.env
    purpose     = "Kubernetes cluster administrators"
  }
}

resource "vault_identity_group" "kubernetes_developers" {
  name     = "${var.env}-kubernetes-developers"
  type     = "internal"
  policies = [vault_policy.developer_policy.name]

  metadata = {
    environment = var.env
    purpose     = "Kubernetes application developers"
  }
}

# Create OIDC key for signing tokens
resource "vault_identity_oidc_key" "kubernetes" {
  name               = "${var.env}-kubernetes"
  allowed_client_ids = ["*"]
  rotation_period    = 3600
  verification_ttl   = 3600
}
```

--------------------------------

### Create ExternalSecret Resource

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/external-secrets-vault-integration.md

This YAML defines an ExternalSecret resource to synchronize secrets from HashiCorp Vault into Kubernetes. It specifies the Vault backend and maps a remote secret property to a Kubernetes secret key. The `creationPolicy: Owner` ensures the Kubernetes Secret is managed by this ExternalSecret resource.

```yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: my-app-secrets
  namespace: my-app
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: vault-backend
  target:
    name: my-app-secrets
    creationPolicy: Owner
  data:
    - secretKey: database-password
      remoteRef:
        key: myapp # kv/{env}/data/myapp
        property: db_pass # Field in the secret

```

--------------------------------

### Kubernetes YAML: Configure External Secrets Operator RBAC

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/external-secrets-vault-integration.md

This YAML defines the necessary RBAC resources for the External Secrets Operator to interact with the Kubernetes API. It includes a Namespace, a ServiceAccount for ESO, and a ClusterRoleBinding granting the ServiceAccount 'system:auth-delegator' permissions, allowing it to perform TokenReview API calls required for Vault authentication.

```yaml
# Namespace
apiVersion: v1
kind: Namespace
metadata:
  name: external-secrets
---
# ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets
  namespace: external-secrets
---
# ClusterRoleBinding for TokenReview API access
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-secrets-auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator # Allows TokenReview API calls
subjects:
  - kind: ServiceAccount
    name: external-secrets
    namespace: external-secrets
```

--------------------------------

### Kubernetes YAML: Configure External Secrets Operator ClusterSecretStore

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/external-secrets-vault-integration.md

This YAML defines a ClusterSecretStore resource for External Secrets Operator to connect to a HashiCorp Vault backend. It specifies the Vault server address, the secret path within Vault, and the authentication details using Kubernetes auth with the client JWT mode, referencing the Vault mount path and the ESO ServiceAccount.

```yaml
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "{{ vault_addr }}" # e.g., https://vault.example.com
      path: "kv/{{ env }}" # e.g., kv/dev
      version: "v2"
      auth:
        kubernetes:
          mountPath: "{{ env }}-kubernetes" # e.g., dev-kubernetes
          role: "external-secrets"
          serviceAccountRef:
            name: "external-secrets"
            namespace: "external-secrets"
```

--------------------------------

### Terraform: Configure Vault Kubernetes Auth Backend

Source: https://github.com/phuchoang2603/kubernetes-proxmox/blob/master/docs/external-secrets-vault-integration.md

This Terraform configuration sets up the Vault Kubernetes authentication backend, enables client JWT mode, defines a policy for ESO access to Vault secrets, and creates a role for the ESO ServiceAccount. It requires network access to the Kubernetes API and Vault to be external to the cluster.

```hcl
resource "vault_auth_backend" "kubernetes" {
  type = "kubernetes"
  path = "${var.env}-kubernetes"  # e.g., dev-kubernetes
}

resource "vault_kubernetes_auth_backend_config" "kubernetes" {
  backend              = vault_auth_backend.kubernetes.path
  kubernetes_host      = "https://${VIP}:6443"
  disable_local_ca_jwt = true  # Use client JWT for both auth and TokenReview
  # Note: No kubernetes_ca_cert needed - Vault uses system CA bundle
}

resource "vault_policy" "external_secrets" {
  name   = "${var.env}-external-secrets-policy"
  policy = <<-EOT
    path "kv/${var.env}/data/*" {
      capabilities = ["read", "list"]
    }
    path "kv/${var.env}/metadata/*" {
      capabilities = ["list"]
    }
  EOT
}

resource "vault_kubernetes_auth_backend_role" "external_secrets" {
  backend                          = vault_auth_backend.kubernetes.path
  role_name                        = "external-secrets"
  bound_service_account_names      = ["external-secrets"]
  bound_service_account_namespaces = ["external-secrets"]
  token_policies                   = [vault_policy.external_secrets.name]
  token_ttl                        = 3600   # 1 hour
  token_max_ttl                    = 86400  # 24 hours
}
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.