### Proxmox CCM Troubleshooting Steps

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md

A step-by-step guide to troubleshoot issues with the Proxmox Cloud Controller Manager. It involves scaling down the deployment, increasing log verbosity, and checking kubelet configuration and node object status.

```text
The steps to troubleshoot the Proxmox CCM:
1. scale down the CCM deployment to 1 replica.
2. set log level to `--v=5` in the deployment.
3. check the logs
4. check kubelet flag `--cloud-provider=external`, delete the node resource and restart the kubelet.
5. check the logs
6. wait for 1 minute. If CCM cannot reach the Proxmox API, it will log the error.
7. check tains, labels, and providerID in the `Node` object.
```

--------------------------------

### Troubleshooting Kubelet with External CCM

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md

Explains the interaction between kubelet and an external cloud controller manager. Highlights the importance of the '--cloud-provider=external' flag and the node lifecycle management process.

```text
How `kubelet` works with flag `cloud-provider=external`:

1. kubelet join the cluster and send the `Node` object to the API server.
Node object has values:
    * `node.cloudprovider.kubernetes.io/uninitialized` taint.
    * `alpha.kubernetes.io/provided-node-ip` annotation with the node IP.
    * `nodeInfo` field with system information.
2. CCM detects the new node and sends a request to the Proxmox API to get the VM configuration. Like VMID, hostname, etc.
3. CCM updates the `Node` object with labels, taints and `providerID` field. The `providerID` is immutable and has the format `proxmox://$REGION/$VMID`, it cannot be changed after the first update.
4. CCM removes the `node.cloudprovider.kubernetes.io/uninitialized` taint.

If `kubelet` does not have `cloud-provider=external` flag, kubelet will expect that no external CCM is running and will try to manage the node lifecycle by itself.
This can cause issues with Proxmox CCM.
So, CCM will skip the node and will not update the `Node` object.

If you modify the `kubelet` flags, it's recommended to check all workloads in the cluster.
Please __delete__ the node resource first, and __restart__ the kubelet.
```

--------------------------------

### Proxmox CCM Node Labels and Annotations Example

Source: https://context7.com/sergelogvinov/proxmox-cloud-controller-manager/llms.txt

An example of a Kubernetes Node resource after initialization by the Proxmox CCM. It shows standard Kubernetes topology labels, instance type labels, Proxmox-specific topology labels, and annotations like the Proxmox instance ID.

```yaml
# Example Node resource after CCM initialization
apiVersion: v1
kind: Node
metadata:
  name: worker-1
  labels:
    # Standard Kubernetes topology labels
    topology.kubernetes.io/region: cluster-1
    topology.kubernetes.io/zone: pve-node-1

    # Instance type based on CPU/RAM
    node.kubernetes.io/instance-type: 4VCPU-8GB

    # Proxmox-specific topology labels
    topology.proxmox.sinextra.dev/region: cluster-1
    topology.proxmox.sinextra.dev/zone: pve-node-1

    # HA group labels (when ha_group feature enabled)
    group.topology.proxmox.sinextra.dev/production: ""

  annotations:
    # Proxmox instance ID (for capmox provider)
    proxmox.sinextra.dev/instance-id: "123"

spec:
  # Provider ID format: proxmox://<region>/<vmid>
  providerID: proxmox://cluster-1/123

status:
  addresses:
    - address: 192.168.1.100
      type: InternalIP
    - address: 10.0.0.50
      type: ExternalIP
    - address: worker-1
      type: Hostname
```

--------------------------------

### Helm Values Example for Proxmox CCM Configuration (YAML)

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/charts/proxmox-cloud-controller-manager/README.md

An example of Helm chart values for configuring the Proxmox Cloud Controller Manager. It specifies cluster connection details, including API URLs, tokens, and regions, and enables specific controllers like 'cloud-node' and 'cloud-node-lifecycle'. It also includes node affinity and tolerations to deploy the CCM on control-plane nodes.

```yaml
# proxmox-ccm.yaml

config:
  clusters:
    - url: https://cluster-api-1.exmple.com:8006/api2/json
      insecure: false
      token_id: "kubernetes@pve!csi"
      token_secret: "key"
      region: cluster-1

enabledControllers:
  # Remove `cloud-node` if you use it with Talos CCM
  - cloud-node
  - cloud-node-lifecycle

# Deploy CCM only on control-plane nodes
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
tolerations:
  - key: node-role.kubernetes.io/control-plane
    effect: NoSchedule
```

--------------------------------

### Proxmox Permissions Setup (Shell)

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/charts/proxmox-cloud-controller-manager/README.md

This snippet demonstrates how to create a role, user, and grant necessary permissions within Proxmox for the Cloud Controller Manager. It involves creating a 'CCM' role with specific privileges and assigning it to a 'kubernetes@pve' user, then generating an API token.

```shell
# Create role CCM
pveum role add CCM -privs "VM.Audit Sys.Audit"
# Create user and grant permissions
pveum user add kubernetes@pve
pveum aclmod / -user kubernetes@pve -role CCM
pveum user token add kubernetes@pve ccm -privsep 0
```

--------------------------------

### Automate Proxmox API Token Setup with Terraform

Source: https://context7.com/sergelogvinov/proxmox-cloud-controller-manager/llms.txt

Terraform configuration using the bpg/proxmox provider to automate the creation of Proxmox API tokens, roles, and users required for the CCM. Outputs the token ID and secret.

```hcl
# proxmox-ccm-token.tf
terraform {
  required_providers {
    proxmox = {
      source  = "bpg/proxmox"
      version = "~> 0.40"
    }
  }
}

resource "proxmox_virtual_environment_role" "ccm" {
  role_id = "CCM"
  privileges = [
    "Sys.Audit",
    "VM.Audit",
    "VM.GuestAgent.Audit",
  ]
}

resource "proxmox_virtual_environment_user" "kubernetes" {
  user_id = "kubernetes@pve"
  comment = "Kubernetes CCM Service Account"

  acl {
    path      = "/"
    propagate = true
    role_id   = proxmox_virtual_environment_role.ccm.role_id
  }
}

resource "proxmox_virtual_environment_user_token" "ccm" {
  user_id    = proxmox_virtual_environment_user.kubernetes.user_id
  token_name = "ccm"
  comment    = "Kubernetes CCM API Token"
}

resource "proxmox_virtual_environment_acl" "ccm" {
  token_id  = proxmox_virtual_environment_user_token.ccm.id
  role_id   = proxmox_virtual_environment_role.ccm.role_id
  path      = "/"
  propagate = true
}

output "ccm_token_id" {
  value = proxmox_virtual_environment_user_token.ccm.id
}

output "ccm_token_secret" {
  value     = proxmox_virtual_environment_user_token.ccm.value
  sensitive = true
}

```

--------------------------------

### Proxmox CCM Helm Values for Rancher Daemonset Mode

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md

Example Helm values file for deploying Proxmox CCM in daemonset mode within a Rancher environment. Includes Proxmox cluster configuration and nodeSelector for control-plane nodes.

```yaml
# proxmox-ccm.yaml
config:
  clusters:
    - url: "https://cluster-api-1.exmple.com:8006/api2/json"
      insecure: false
      token_id: "kubernetes@pve!ccm"
      token_secret: "secret"
      region: cluster-1

# Use host resolv.conf to resolve proxmox connection url
useDaemonSet: true

# Set nodeSelector in daemonset mode is required
nodeSelector:
  node-role.kubernetes.io/control-plane: ""
```

--------------------------------

### Deploy Proxmox CCM (Helm - Deployment Mode)

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md

Deploys the Proxmox Cloud Controller Manager using Helm in the standard deployment mode. It installs the CCM into the 'kube-system' namespace.

```shell
helm upgrade -i --namespace=kube-system -f proxmox-ccm.yaml \
    proxmox-cloud-controller-manager \
    oci://ghcr.io/sergelogvinov/charts/proxmox-cloud-controller-manager
```

--------------------------------

### Create Proxmox Role and User for CCM (Terraform)

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md

Defines Proxmox resources using Terraform to create a 'CCM' role with specific privileges, a 'kubernetes@pve' user, and an API token. This approach automates the setup of necessary permissions and credentials for the Proxmox CCM within an infrastructure-as-code workflow.

```hcl
# Plugin: bpg/proxmox

resource "proxmox_virtual_environment_role" "ccm" {
  role_id = "CCM"

  privileges = [
    "Sys.Audit",
    "VM.Audit",
    "VM.GuestAgent.Audit",
  ]
}

resource "proxmox_virtual_environment_user" "kubernetes" {
  acl {
    path      = "/"
    propagate = true
    role_id   = proxmox_virtual_environment_role.ccm.role_id
  }

  comment = "Kubernetes"
  user_id = "kubernetes@pve"
}

resource "proxmox_virtual_environment_user_token" "ccm" {
  comment    = "Kubernetes CCM"
  token_name = "ccm"
  user_id    = proxmox_virtual_environment_user.kubernetes.user_id
}

resource "proxmox_virtual_environment_acl" "ccm" {
  token_id = proxmox_virtual_environment_user_token.ccm.id
  role_id  = proxmox_virtual_environment_role.ccm.role_id

  path      = "/"
  propagate = true
}
```

--------------------------------

### Specify Node IP for Kubelet

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md

Configures the `--node-ip` flag for kubelet, allowing specification of the node's IP address, especially in dual-stack or multi-IP environments. This ensures correct communication within the Kubernetes cluster. The `${IP}` variable can be a single IP or a comma-separated list for dual-stack configurations.

```shell
# ${IP} can be single or comma-separated list of two IPs (dual stack).
kubelet --node-ip=${IP}
```

--------------------------------

### Deploy Proxmox CCM (Helm - Daemonset Mode)

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md

Deploys the Proxmox Cloud Controller Manager using Helm in daemonset mode, intended to run on all control-plane nodes. Requires setting 'useDaemonSet=true' and a 'nodeSelector'.

```shell
helm upgrade -i --namespace=kube-system -f proxmox-ccm.yaml \
    --set useDaemonSet=true \
    proxmox-cloud-controller-manager \
    oci://ghcr.io/sergelogvinov/charts/proxmox-cloud-controller-manager
```

--------------------------------

### Configure Kubelet for Proxmox CCM

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md

Sets the `--cloud-provider=external` argument for kubelet, enabling it to offload cloud-specific node management to the Proxmox CCM. This is a mandatory step for Proxmox CCM to function correctly. Failure to set this flag can lead to conflicts in node lifecycle management.

```shell
kubelet --cloud-provider=external
```

--------------------------------

### Helm Values for Credentials from Separate Secrets (YAML)

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/charts/proxmox-cloud-controller-manager/README.md

This example demonstrates how to configure the Proxmox Cloud Controller Manager to use credentials stored in separate Kubernetes Secrets. It maps token ID and secret files from mounted secrets for multiple Proxmox clusters. It also defines the necessary `extraVolumes` and `extraVolumeMounts` to make these secrets accessible to the CCM pod.

```yaml
# helm-values.yaml
config:
  clusters:
    - url: https://cluster-api-1.exmple.com:8006/api2/json
      insecure: false
      token_id_file: /run/secrets/cluster-1/token_id
      token_secret_file: /run/secrets/cluster-1/token_secret
      region: cluster-1
    - url: https://cluster-api-2.exmple.com:8006/api2/json
      insecure: false
      token_id_file: /run/secrets/cluster-2/token_id
      token_secret_file: /run/secrets/cluster-2/token_secret
      region: cluster-2
extraVolumes:
  - name: credentials-cluster-1
    secret:
      secretName: proxmox-credentials-cluster-1
  - name: credentials-cluster-2
    secret:
      secretName: proxmox-credentials-cluster-2
extraVolumeMounts:
  - name: credentials-cluster-1
    readOnly: true
    mountPath: "/run/secrets/cluster-1"
  - name: credentials-cluster-2
    readOnly: true
    mountPath: "/run/secrets/cluster-2"
```

--------------------------------

### Deploy Proxmox CCM Helm Chart (Shell)

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/charts/proxmox-cloud-controller-manager/README.md

Command to deploy or upgrade the Proxmox Cloud Controller Manager Helm chart. It uses `helm upgrade -i` to install the chart if it doesn't exist or upgrade it otherwise, specifying the namespace as `kube-system` and providing a custom values file `proxmox-ccm.yaml`. The chart is sourced from `ghcr.io/sergelogvinov/charts/proxmox-cloud-controller-manager`.

```shell
helm upgrade -i --namespace=kube-system -f proxmox-ccm.yaml \
    proxmox-cloud-controller-manager oci://ghcr.io/sergelogvinov/charts/proxmox-cloud-controller-manager
```

--------------------------------

### Deploy Proxmox CCM with All Controllers (kubectl)

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md

Deploys the Proxmox Cloud Controller Manager to the Kubernetes cluster using a manifest file from a GitHub URL. This deployment includes both `cloud-node` and `cloud-node-lifecycle` controllers, enabling full management of Proxmox VMs as Kubernetes nodes.

```shell
kubectl apply -f https://raw.githubusercontent.com/sergelogvinov/proxmox-cloud-controller-manager/main/docs/deploy/cloud-controller-manager.yml
```

--------------------------------

### Pod Affinity with Topology Labels for Scheduling

Source: https://context7.com/sergelogvinov/proxmox-cloud-controller-manager/llms.txt

Example Kubernetes Deployment manifest demonstrating how to use Proxmox CCM-applied topology labels for advanced pod scheduling. This configuration utilizes `podAntiAffinity` to spread pods across Proxmox hosts (zones) and `podAffinity` to keep pods within the same Proxmox cluster (region). It also shows how to target specific HA groups using `nodeSelector`.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 3
  template:
    spec:
      affinity:
        # Spread pods across Proxmox hosts (zones)
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: web-app
                topologyKey: topology.kubernetes.io/zone

        # Keep pods in the same Proxmox cluster (region)
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app: web-app
              topologyKey: topology.kubernetes.io/region

      # Target specific HA group
      nodeSelector:
        group.topology.proxmox.sinextra.dev/production: ""

      containers:
        - name: web
          image: nginx:latest
```

--------------------------------

### Deploy Proxmox CCM with Lifecycle Controller Only (kubectl)

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md

Deploys the Proxmox Cloud Controller Manager with only the `cloud-node-lifecycle` controller enabled. This specific deployment is intended for environments like Talos Linux, where only node deletion management from Proxmox is required from the CCM.

```shell
kubectl apply -f https://raw.githubusercontent.com/sergelogvinov/proxmox-cloud-controller-manager/main/docs/deploy/cloud-controller-manager-talos.yml
```

--------------------------------

### Proxmox CCM Configuration File Structure

Source: https://context7.com/sergelogvinov/proxmox-cloud-controller-manager/llms.txt

Defines the structure of the Proxmox CCM configuration file (config.yaml), including provider settings, network modes, and Proxmox cluster connection details. Supports multi-cluster setups and token-based authentication.

```yaml
# config.yaml - Full configuration example
features:
  # Provider type: "default" (proxmox://<region>/<vm-id>) or "capmox" (proxmox://<SystemUUID>)
  provider: default
  # Network mode: "default", "qemu", or "auto"
  network:
    mode: auto
    # Define external IP CIDRs (use ! to exclude)
    external_ip_cidrs: '192.168.0.0/16,2001:db8:85a3::8a2e:370:7334/112,!10.0.0.0/8'
    # IP address sort order
    ip_sort_order: '192.168.0.0/16'
    # Disable IPv6 support
    ipv6_support_disabled: false
  # Use Proxmox HA group as zone label
  ha_group: false
  # Force update topology labels on node migration
  force_update_labels: false

clusters:
  - url: https://proxmox-cluster-1.example.com:8006/api2/json
    insecure: false
    token_id: "kubernetes@pve!ccm"
    token_secret: "12345678-1234-1234-1234-123456789012"
    region: cluster-1

  - url: https://proxmox-cluster-2.example.com:8006/api2/json
    insecure: false
    # Alternative: read credentials from files
    token_id_file: /run/secrets/region-2/token_id
    token_secret_file: /run/secrets/region-2/token_secret
    region: cluster-2

```

--------------------------------

### Specify Provider ID for Kubelet

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md

Sets the `--provider-id` flag for kubelet to specify the Proxmox VM ID. This is necessary when the Cloud Controller Manager cannot automatically determine the VMID, ensuring proper node management by its unique identifier in Proxmox. The `${ID}` format is `proxmox://$REGION/$VMID`.

```shell
# ${ID} has format proxmox://$REGION/$VMID.kubelet --provider-id=${ID}
```

--------------------------------

### Rancher RKE2 Configuration for External CCM

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md

Configuration snippet for Rancher RKE2 to enable external cloud provider integration. It sets the kubelet flag to 'external' and disables the built-in Rancher CCM.

```yaml
machineGlobalConfig:
  # Kubelet predefined value --cloud-provider=external
  cloud-provider-name: external
  # Disable Rancher CCM
  disable-cloud-controller: true
```

--------------------------------

### Create Proxmox Role and User for CCM (CLI)

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md

Creates a custom role 'CCM' with necessary privileges and a 'kubernetes@pve' user, granting the role to this user. It then generates an API token for the user, which will be used by the Proxmox CCM for authentication and authorization with the Proxmox API.

```shell
# Create role CCM
pveum role add CCM -privs "VM.Audit VM.GuestAgent.Audit Sys.Audit"
# Create user and grant permissions
pveum user add kubernetes@pve
pveum aclmod / -user kubernetes@pve -role CCM
pveum user token add kubernetes@pve ccm -privsep 0
```

--------------------------------

### Proxmox CCM Configuration (proxmox-ccm.yaml)

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md

This YAML file configures the Proxmox Cloud Controller Manager, specifying the Proxmox API endpoint, authentication credentials, and region. It's used as a values file for Helm deployments.

```yaml
config:
  clusters:
    - url: "https://cluster-api-1.exmple.com:8006/api2/json"
      insecure: false
      token_id: "kubernetes@pve!ccm"
      token_secret: "secret"
      region: cluster-1
```

--------------------------------

### Override Kubelet Hostname

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md

Configures the `--hostname-override` flag for kubelet to specify the node's hostname. This is useful when the hostname registered in Kubernetes differs from the actual node hostname, ensuring consistent identification within the cluster. The `${NODENAME}` variable represents the node's name.

```shell
# ${NODENAME} is the name of the node.kubelet --hostname-override=${NODENAME}
```

--------------------------------

### Create Proxmox CCM Secret (kubectl)

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md

Creates a Kubernetes secret named `proxmox-cloud-controller-manager` in the `kube-system` namespace using `kubectl`. This secret stores the Proxmox credentials configuration, allowing the CCM to authenticate with the Proxmox API. The `--from-file=config.yaml` flag loads the configuration from a local file.

```shell
kubectl -n kube-system create secret generic proxmox-cloud-controller-manager --from-file=config.yaml
```

--------------------------------

### Proxmox CCM Configuration Example (YAML)

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/config.md

This YAML snippet demonstrates the structure for configuring the Proxmox Cloud Controller Manager. It includes settings for provider type, network mode, IPv6 support, external IP CIDRs, IP sort order, HA group usage, and details for connecting to multiple Proxmox clusters with authentication.

```yaml
features:
  # Provider type
  provider: default|capmox
  # Network mode
  network: default|qemu|auto
  # Enable or disable the IPv6 support
  ipv6_support_disabled: true|false
  # External IP address CIDRs list, comma-separated
  # Use `!` to exclude a CIDR
  external_ip_cidrs: '192.168.0.0/16,2001:db8:85a3::8a2e:370:7334/112,!fd00:1234:5678::/64'
  # IP addresses sort order, comma-separated
  # The IPs that do not match the CIDRs will be kept in the order they
  # were detected.
  ip_sort_order: '192.168.0.0/16,2001:db8:85a3::8a2e:370:7334/112'
  # Enable use of Proxmox HA group as a zone label
  ha_group: true|false

clusters:
  # List of Proxmox clusters
  - url: https://cluster-api-1.exmple.com:8006/api2/json
    # Skip the certificate verification, if needed
    insecure: false
    # Proxmox api token
    token_id: "kubernetes-csi@pve!csi"
    token_secret: "secret"
    # (optional) Proxmox api token from separate file (s. Helm README.md)
    # token_id_file: /run/secrets/region-1/token_id
    # token_secret_file: /run/secrets/region-1/token_secret
    # Region name, which is cluster name
    region: Region-1

  # Add more clusters if needed
  - url: https://cluster-api-2.exmple.com:8006/api2/json
    insecure: false
    token_id: "kubernetes-csi@pve!csi"
    token_secret: "secret"
    region: Region-2

```

--------------------------------

### Proxmox CCM Credentials Configuration

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/install.md

Defines the configuration for the Proxmox Cloud Controller Manager, specifying the Proxmox cluster API endpoint, authentication token ID and secret, and a region name. This YAML file is used to establish the connection and credentials for the CCM to interact with the Proxmox environment.

```yaml
clusters:
  # List of Proxmox clusters, region mast be unique
  - url: https://cluster-api-1.exmple.com:8006/api2/json
    insecure: false
    token_id: "kubernetes@pve!ccm"
    # Token from the previous step
    token_secret: "secret"
    # Region name, can be any string, it will use as for kubernetes topology.kubernetes.io/region label
    region: cluster-1
```

--------------------------------

### Configure Kubelet for External Cloud Provider

Source: https://context7.com/sergelogvinov/proxmox-cloud-controller-manager/llms.txt

Instructions on how to configure the Kubelet on each node to use the external cloud provider. This includes setting the cloud-provider to 'external' and optionally specifying the node IP, provider ID, or overriding the hostname.

```bash
# Required: Set cloud-provider to external
kubelet --cloud-provider=external

# Optional: Specify node IP (single or dual-stack)
kubelet --node-ip=192.168.1.100kubelet --node-ip=192.168.1.100,2001:db8::100

# Optional: Specify provider ID manually (if CCM cannot auto-detect)
# Format: proxmox://<region>/<vmid>
kubelet --provider-id=proxmox://cluster-1/123

# Optional: Override hostname if different from VM name
kubelet --hostname-override=worker-1
```

--------------------------------

### Deploy Proxmox CCM with Helm Chart

Source: https://context7.com/sergelogvinov/proxmox-cloud-controller-manager/llms.txt

Instructions for deploying the Proxmox CCM using its official Helm chart. This includes creating a values file for customization and deploying in either deployment or daemonset mode. It also covers checking the release status.

```bash
# Create values file
cat <<EOF > proxmox-ccm-values.yaml
# CCM configuration
config:
  features:
    provider: default
    network:
      mode: auto
  clusters:
    - url: https://proxmox.example.com:8006/api2/json
      insecure: false
      token_id: "kubernetes@pve!ccm"
      token_secret: "your-token-secret-here"
      region: cluster-1

# Controllers to enable
enabledControllers:
  - cloud-node
  - cloud-node-lifecycle

# Resource limits
resources:
  requests:
    cpu: 10m
    memory: 32Mi
  limits:
    cpu: 100m
    memory: 128Mi

# Run on control-plane nodes
nodeSelector:
  node-role.kubernetes.io/control-plane: ""

# Tolerations for control-plane and uninitialized nodes
tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
    operator: Exists
  - effect: NoSchedule
    key: node.cloudprovider.kubernetes.io/uninitialized
    operator: Exists

# Log verbosity (0-5)
logVerbosityLevel: 2
EOF

# Install CCM (deployment mode)
helm upgrade -i proxmox-cloud-controller-manager \
  --namespace kube-system \
  -f proxmox-ccm-values.yaml \
  oci://ghcr.io/sergelogvinov/charts/proxmox-cloud-controller-manager

# Install CCM (daemonset mode - uses hostNetwork, no CNI required)
helm upgrade -i proxmox-cloud-controller-manager \
  --namespace kube-system \
  -f proxmox-ccm-values.yaml \
  --set useDaemonSet=true \
  oci://ghcr.io/sergelogvinov/charts/proxmox-cloud-controller-manager

# Check release status
helm -n kube-system status proxmox-cloud-controller-manager
```

--------------------------------

### Apply Kubernetes Configuration

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/loadbalancer.md

Command to apply the Kubernetes service and endpoint configuration defined in the `proxmox-service.yaml` file to the cluster.

```bash
kubectl apply -f proxmox-service.yaml

```

--------------------------------

### Create Proxmox API Token using pveum CLI

Source: https://context7.com/sergelogvinov/proxmox-cloud-controller-manager/llms.txt

Command-line instructions to create a dedicated Proxmox API token for the CCM with minimal privileges. This involves creating a role, a user, granting the role to the user, and then generating the token.

```bash
# Create CCM role with audit-only permissions
pveum role add CCM -privs "VM.Audit VM.GuestAgent.Audit Sys.Audit"

# Create user for Kubernetes
pveum user add kubernetes@pve

# Grant CCM role to user
pveum aclmod / -user kubernetes@pve -role CCM

# Create API token (privsep 0 = inherit user privileges)
pveum user token add kubernetes@pve ccm -privsep 0

# Output: Token ID and secret for config.yaml

```

--------------------------------

### Troubleshooting Proxmox CCM Commands

Source: https://context7.com/sergelogvinov/proxmox-cloud-controller-manager/llms.txt

A collection of kubectl commands to help debug issues with the Proxmox Cloud Controller Manager. These commands cover scaling the deployment, increasing log verbosity, checking logs, verifying node status, and inspecting taints and provider IDs.

```bash
# Scale to single replica and increase log verbosity
kubectl -n kube-system scale deployment proxmox-cloud-controller-manager --replicas=1
kubectl -n kube-system set env deployment/proxmox-cloud-controller-manager -- -v=5

# Check CCM logs
kubectl -n kube-system logs -l app.kubernetes.io/name=proxmox-cloud-controller-manager -f

# Verify node status
kubectl get nodes -o wide
kubectl describe node worker-1

# Check for uninitialized taint (should be removed by CCM)
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'

# Verify provider ID is set
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.providerID}{"\n"}{end}'

# Check topology labels
kubectl get nodes --show-labels | grep topology

# Force node re-initialization (delete and let kubelet rejoin)
kubectl delete node worker-1
# Then restart kubelet on the node
```

--------------------------------

### Deploy Proxmox CCM with kubectl

Source: https://context7.com/sergelogvinov/proxmox-cloud-controller-manager/llms.txt

Instructions for deploying the Proxmox CCM using kubectl. This involves creating a Kubernetes secret for Proxmox configuration and applying the CCM deployment manifests.

```bash
# Create namespace secret with Proxmox configuration
cat <<EOF > /tmp/proxmox-config.yaml
clusters:
  - url: https://proxmox.example.com:8006/api2/json
    insecure: false
    token_id: "kubernetes@pve!ccm"
    token_secret: "your-token-secret-here"
    region: cluster-1
EOF

# Upload config as Kubernetes secret
kubectl -n kube-system create secret generic proxmox-cloud-controller-manager \
  --from-file=config.yaml=/tmp/proxmox-config.yaml

# Deploy CCM with cloud-node and cloud-node-lifecycle controllers
kubectl apply -f https://raw.githubusercontent.com/sergelogvinov/proxmox-cloud-controller-manager/main/docs/deploy/cloud-controller-manager.yml

# For Talos Linux (cloud-node-lifecycle only)
kubectl apply -f https://raw.githubusercontent.com/sergelogvinov/proxmox-cloud-controller-manager/main/docs/deploy/cloud-controller-manager-talos.yml

```

--------------------------------

### Update Helm Chart and Documentation

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/release.md

This snippet details the process of updating the Helm chart and documentation for a new release. It involves checking out a specific branch, exporting a tag version, running make commands for Helm and docs, staging changes, and amending the commit.

```shell
git branch -D release-please--branches--main
git checkout release-please--branches--main
export `jq -r '"TAG=v"+.[]' hack/release-please-manifest.json`

make helm-unit docs

git add .
git commit -s --amend
```

--------------------------------

### Enable Prometheus Metrics Scraping

Source: https://context7.com/sergelogvinov/proxmox-cloud-controller-manager/llms.txt

Configure Prometheus to scrape metrics exposed by the Proxmox CCM. This involves setting pod annotations in Helm values to enable scraping, specify the scheme, and define the port. The CCM exposes metrics like API request duration and errors, which can be queried using PromQL.

```yaml
# Enable metrics scraping in Helm values
podAnnotations:
  prometheus.io.scrape: "true"
  prometheus.io.scheme: "https"
  prometheus.io.port: "10258"

---
# Metrics endpoint
# https://localhost:10258/metrics

# Available metrics:
# proxmox_api_request_duration_seconds - Histogram of API call latencies
# proxmox_api_request_errors_total - Counter of API errors

# Example PromQL queries:
# Rate of API requests: rate(proxmox_api_request_duration_seconds_count[5m])
# 95th percentile latency: histogram_quantile(0.95, rate(proxmox_api_request_duration_seconds_bucket[5m]))
# Error rate: rate(proxmox_api_request_errors_total[5m])
```

--------------------------------

### Verify Proxmox CCM Deployment

Source: https://context7.com/sergelogvinov/proxmox-cloud-controller-manager/llms.txt

Commands to check the status and logs of the Proxmox Cloud Controller Manager pods in the kube-system namespace.

```bash
kubectl -n kube-system get pods -l app.kubernetes.io/name=proxmox-cloud-controller-manager
kubectl -n kube-system logs -l app.kubernetes.io/name=proxmox-cloud-controller-manager -f
```

--------------------------------

### Reference Existing Config Secret

Source: https://context7.com/sergelogvinov/proxmox-cloud-controller-manager/llms.txt

Use a pre-existing Kubernetes secret for CCM configuration instead of defining it directly in Helm values. This involves creating a secret manually with the cloud-config.yaml file and then referencing its name and key in the Helm values. The `config` section in Helm values is ignored when `existingConfigSecret` is set.

```yaml
# Create secret manually
kubectl -n kube-system create secret generic my-proxmox-config \
  --from-file=cloud-config.yaml=/path/to/config.yaml

# Reference in Helm values
existingConfigSecret: my-proxmox-config
existingConfigSecretKey: cloud-config.yaml

# config section is ignored when existingConfigSecret is set
config: {}
```

--------------------------------

### Proxmox Cloud Provider Configuration

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/README.md

This YAML configuration defines the connection details for multiple Proxmox clusters. It includes the API URL, authentication credentials (token ID and secret), and a unique region name for each cluster. The `insecure` flag can be set to `true` to bypass SSL certificate verification, though this is not recommended for production environments.

```yaml
clusters:
  - url: https://cluster-api-1.exmple.com:8006/api2/json
    insecure: false
    # Proxox auth token
    token_id: "user!token-id"
    token_secret: "secret"
    # Uniq region name
    region: cluster-1
  - url: https://cluster-api-2.exmple.com:8006/api2/json
    insecure: false
    token_id: "user!token-id"
    token_secret: "secret"
    region: cluster-2
```

--------------------------------

### Configure Proxmox CCM Network Mode

Source: https://context7.com/sergelogvinov/proxmox-cloud-controller-manager/llms.txt

Configuration options for the network mode in the Proxmox CCM. This determines how the CCM discovers and assigns node IP addresses, with options for default, QEMU guest agent, and auto modes, including settings for external IPs and IP ordering.

```yaml
# Default mode - uses kubelet --node-ip flag
features:
  network:
    mode: default

# QEMU mode - retrieves IPs via QEMU guest agent API
features:
  network:
    mode: qemu

# Auto mode - combines kubelet and QEMU agent detection
features:
  network:
    mode: auto
    # Define which IPs are external (use ! to exclude)
    external_ip_cidrs: '192.168.0.0/16,!10.0.0.0/8'
    # Control IP address ordering
    ip_sort_order: '192.168.0.0/16,2001:db8::/32'
    # Disable IPv6 if not needed
    ipv6_support_disabled: true
```

--------------------------------

### Kubernetes Service and Endpoints for Proxmox

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/loadbalancer.md

Defines a headless Kubernetes Service for Proxmox and its corresponding Endpoints. The headless service allows direct access to Proxmox nodes, and the endpoints list the IP addresses of the Proxmox nodes.

```yaml
---
apiVersion: v1
kind: Service
metadata:
  name: proxmox
  namespace: kube-system
spec:
  clusterIP: None
  ports:
    - name: https
      protocol: TCP
      port: 8006
      targetPort: 8006
---
apiVersion: v1
kind: Endpoints
metadata:
  name: proxmox
  namespace: kube-system
subsets:
  - addresses:
      - ip: 192.168.0.1
      - ip: 192.168.0.2
    ports:
      - port: 8006

```

--------------------------------

### Metrics Endpoint

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/metrics.md

The Proxmox CCM exposes metrics on the /metrics endpoint, typically accessible via HTTPS on port 10258. Configuration details for enabling this endpoint are provided.

```APIDOC
## GET /metrics

### Description
Retrieves the exposed metrics from the Proxmox Cloud Controller Manager.

### Method
GET

### Endpoint
`https://<host>:10258/metrics`

### Parameters
#### Query Parameters
None

#### Request Body
None

### Request Example
None

### Response
#### Success Response (200)
- **metrics** (string) - A text-based output of Prometheus metrics.

#### Response Example
```txt
# HELP proxmox_api_request_duration_seconds Proxmox API request duration in seconds.
# TYPE proxmox_api_request_duration_seconds histogram
proxmox_api_request_duration_seconds_bucket{request="getVmInfo",le="0.1"} 13
proxmox_api_request_duration_seconds_bucket{request="getVmInfo",le="0.25"} 172
proxmox_api_request_duration_seconds_bucket{request="getVmInfo",le="0.5"} 199
proxmox_api_request_duration_seconds_bucket{request="getVmInfo",le="1"} 210
proxmox_api_request_duration_seconds_bucket{request="getVmInfo",le="2.5"} 210
proxmox_api_request_duration_seconds_bucket{request="getVmInfo",le="5"} 210
proxmox_api_request_duration_seconds_bucket{request="getVmInfo",le="10"} 210
proxmox_api_request_duration_seconds_bucket{request="getVmInfo",le="30"} 210
proxmox_api_request_duration_seconds_bucket{request="getVmInfo",le="+Inf"} 210
proxmox_api_request_duration_seconds_sum{request="getVmInfo"} 39.698945394000006
proxmox_api_request_duration_seconds_count{request="getVmInfo"} 210
# HELP proxmox_api_request_errors_total Total number of Proxmox API request errors.
# TYPE proxmox_api_request_errors_total counter
proxmox_api_request_errors_total{request="getVmInfo"} 0
```

### Configuration Notes
To expose the metrics endpoint, ensure the Proxmox CCM is started with the `--authorization-always-allow-paths="/metrics"` flag and `--secure-port=10258`. If using the Helm chart, set the following `podAnnotations`:
```yaml
podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/scheme: "https"
  prometheus.io/port: "10258"
```
```

--------------------------------

### Rancher RKE2 Integration with Proxmox CCM

Source: https://context7.com/sergelogvinov/proxmox-cloud-controller-manager/llms.txt

Configuration for deploying the Proxmox CCM in Rancher-managed RKE2 clusters. This involves setting cluster-wide machine configurations to enable the external cloud provider and disable Rancher's built-in CCM, along with Helm values for the Proxmox CCM.

```yaml
# Rancher cluster configuration
machineGlobalConfig:
  # Enable external cloud provider
  cloud-provider-name: external
  # Disable Rancher's built-in CCM
  disable-cloud-controller: true

---
# proxmox-ccm-rancher.yaml - Helm values for Rancher
config:
  clusters:
    - url: https://proxmox.example.com:8006/api2/json
      insecure: false
      token_id: "kubernetes@pve!ccm"
      token_secret: "your-token-secret-here"
      region: cluster-1

# DaemonSet mode uses host resolv.conf for DNS resolution
useDaemonSet: true

# Required: specify control-plane nodes for daemonset
nodeSelector:
  node-role.kubernetes.io/control-plane: ""

# Deploy with Helm
# helm upgrade -i --namespace=kube-system -f proxmox-ccm-rancher.yaml \
#   proxmox-cloud-controller-manager \

```

--------------------------------

### Change Release Version using Git Commit

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/release.md

This snippet demonstrates how to change the release version of the project using Git. It creates an empty commit with specific messages to tag the new release version.

```shell
git commit --allow-empty -m "chore: release 2.0.0" -m "Release-As: 2.0.0"
```

--------------------------------

### Proxmox CCM Helm Chart Values with HAProxy Sidecar

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/loadbalancer.md

Configuration for the Proxmox Cloud Controller Manager (CCM) using a Helm chart, including the HAProxy sidecar for load balancing. It specifies cluster connection details, host aliases for the load balancer, and the HAProxy container configuration.

```yaml
# CCM helm chart values

config:
  clusters:
    - region: cluster
      url: https://proxmox.domain.com:8006/api2/json
      insecure: true
      token_id: kubernetes@pve!ccm
      token_secret: 11111111-1111-1111-1111-111111111111

hostAliases:
  - ip: 127.0.0.1
    hostnames:
      - proxmox.domain.com

initContainers:
  - name: loadbalancer
    restartPolicy: Always
    image: ghcr.io/sergelogvinov/haproxy:2.8.6-alpine3.19
    imagePullPolicy: IfNotPresent
    env:
      - name: SVC
        value: proxmox.kube-system.svc.cluster.local
      - name: PORT
        value: "8006"
    securityContext:
      runAsUser: 99
      runAsGroup: 99
    resources:
      limits:
        cpu: 50m
        memory: 64Mi
      requests:
        cpu: 50m
        memory: 32Mi

```

--------------------------------

### Kubernetes Node Specification with Proxmox Labels

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/README.md

This YAML defines a Kubernetes Node object, showcasing the labels applied by the Proxmox CCM. It includes instance type labels derived from VM resources, topology labels for region and zone based on Proxmox cluster and hypervisor host, and Proxmox-specific labels. The `providerID` field uniquely identifies the Proxmox VM within a specific cluster.

```yaml
apiVersion: v1
kind: Node
metadata:
  labels:
    ...
    # Type generated base on CPU and RAM
    node.kubernetes.io/instance-type: 2VCPU-2GB
    # Proxmox cluster name as in the config
    topology.kubernetes.io/region: cluster-1
    # Proxmox hypervisor host machine name
    topology.kubernetes.io/zone: pve-node-1

    # Proxmox specific labels
    topology.proxmox.sinextra.dev/region: cluster-1
    topology.proxmox.sinextra.dev/zone: pve-node-1
    # HA group labels - the same idea as node-role
    group.topology.proxmox.sinextra.dev/${HAGroup}: ""

  name: worker-1
spec:
  ...
  # providerID - magic string:
  #   cluster-1 - cluster name as in the config
  #   123 - Proxmox VM ID
  providerID: proxmox://cluster-1/123
status:
  addresses:
  - address: 172.16.0.31
    type: InternalIP
  - address: worker-1
    type: Hostname
```

--------------------------------

### Enable QEMU-only Network Mode in Proxmox CCM

Source: https://github.com/sergelogvinov/proxmox-cloud-controller-manager/blob/main/docs/networking.md

This configuration snippet enables the 'qemu' mode for network addressing in Proxmox CCM. In QEMU-only mode, the CCM relies solely on the QEMU guest agent API to retrieve IP addresses for nodes. This mode is useful in environments where guest agent access is guaranteed.

```yaml
features:
  network:
    mode: qemu
```