### Install Kubecost bundle Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=installations-installing-kubecost-plural Installs the specified Kubecost bundle into your workspace. This example shows the AWS configuration. ```bash plural bundle install kubecost kubecost-aws ``` -------------------------------- ### Install Kubecost Primary Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=guide-aws-distro-open-telemetry Install the primary Kubecost instance using Helm. ```bash aws ecr-public get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin public.ecr.aws helm install YOUR_NAMESPACE \ oci://public.ecr.aws/kubecost/cost-analyzer \ -f values-kubecost-primary.yaml -f https://raw.githubusercontent.com/kubecost/cost-analyzer-helm-chart/v2.6/cost-analyzer/values-eks-cost-monitoring.yaml ``` -------------------------------- ### Install Kubecost Agent Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=guide-aws-distro-open-telemetry Install the Kubecost agent for multi-account monitoring. ```bash aws ecr-public get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin public.ecr.aws helm install YOUR_NAMESPACE \ oci://public.ecr.aws/kubecost/cost-analyzer \ -f values-kubecost-agent.yaml -f https://raw.githubusercontent.com/kubecost/cost-analyzer-helm-chart/v2.7/cost-analyzer/values-eks-cost-monitoring.yaml ``` -------------------------------- ### Install and deploy Plural Console Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=installations-installing-kubecost-plural Installs the Plural Console to manage resources and upgrades, followed by the build and deploy commands. ```bash plural bundle install console console-aws plural build plural deploy --commit"deploying the console too" ``` -------------------------------- ### Install Kubecost via Helm Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=installations-installing-kubecost-alibaba Use this command to install the Kubecost cost-analyzer chart into the kubecost namespace. ```bash helm install kubecost/cost-analyzer -n kubecost -f values.yaml ``` -------------------------------- ### Install KSM via Helm Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=overview-kube-state-metrics-ksm-emission Use this command to install the kube-state-metrics chart from the Prometheus Community repository into a dedicated namespace. ```bash helm install kube-state-metrics \ --repo https://prometheus-community.github.io/helm-charts kube-state-metrics \ --namespace kube-state-metrics --create-namespace ``` -------------------------------- ### Install Kubecost via Helm Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=guide-aws-agentless-amp Authenticates with ECR and installs the Kubecost Helm chart using specific configuration files. ```bash aws ecr-public get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin public.ecr.aws helm install $KUBECOST_NAMESPACE -n $KUBECOST_NAMESPACE \ oci://public.ecr.aws/kubecost/cost-analyzer \ -f https://raw.githubusercontent.com/kubecost/cost-analyzer-helm-chart/v2.6/cost-analyzer/values-eks-cost-monitoring.yaml \ -f values-kubecost-primary.yaml ``` ```bash aws ecr-public get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin public.ecr.aws helm install $KUBECOST_NAMESPACE -n $KUBECOST_NAMESPACE \ oci://public.ecr.aws/kubecost/cost-analyzer \ -f https://raw.githubusercontent.com/kubecost/cost-analyzer-helm-chart/v2.6/cost-analyzer/values-eks-cost-monitoring.yaml \ -f values-kubecost-agent.yaml ``` -------------------------------- ### Install Kubecost via Helm Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=configuration-parallel-install-upgrading Commands for installing the primary and parallel Kubecost instances using distinct namespaces and release names. ```bash helm install kubecost --namespace kubecost ... ``` ```bash helm install kubecost2 --namespace kubecost2 ... ``` -------------------------------- ### Example output for describe-addon command Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=installations-amazon-eks-integration The expected JSON response when the add-on is successfully active. ```json { "addon": { "addonName": "kubecost_kubecost", "clusterName": "$YOUR_CLUSTER_NAME", "status": "ACTIVE", "addonVersion": "v1.97.0-eksbuild.1", "health": { "issues": [] }, "addonArn": "arn:aws:eks:$AWS_REGION:xxxxxxxxxxxx:addon/$YOUR_CLUSTER_NAME/kubecost_kubecost/90c23198-cdd3-b295-c410-xxxxxxxxxxxx", "createdAt": "2022-12-01T12:18:26.497000-08:00", "modifiedAt": "2022-11-10T12:53:21.140000-08:00", "tags": {} } } ``` -------------------------------- ### Example output for create-addon command Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=installations-amazon-eks-integration The expected JSON response after initiating the Kubecost add-on creation. ```json { "addon": { "addonName": "kubecost_kubecost", "clusterName": "$YOUR_CLUSTER_NAME", "status": "CREATING", "addonVersion": "v1.97.0-eksbuild.1", "health": { "issues": [] }, "addonArn": "arn:aws:eks:$AWS_REGION:xxxxxxxxxxxx:addon/$YOUR_CLUSTER_NAME/kubecost_kubecost/90c23198-cdd3-b295-c410-xxxxxxxxxxxx", "createdAt": "2022-12-01T12:18:26.497000-08:00", "modifiedAt": "2022-12-01T12:50:52.222000-08:00", "tags": {} } } ``` -------------------------------- ### Install Kubecost with Dedicated Service Account Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=integration-aws-cloud-using-irsaeks-pod-identities Installs Kubecost via Helm using the dedicated service account and cloud integration secret. ```bash helm install kubecost \ --repo https://kubecost.github.io/cost-analyzer/ cost-analyzer \ --namespace kubecost \ --setserviceAccount.name=kubecost-serviceaccount \ --setserviceAccount.create=false\ --setkubecostProductConfigs.cloudIntegrationSecret=cloud-integration ``` -------------------------------- ### Query total costs example Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=apis-query-collection-total-costs Example request and response for retrieving total costs of a collection without group details. ```http GET /model/collection/query/total?id=8a939ba1-ff15-4600-b711-2cb109114914&window=30d ``` ```json { "id": "8a939ba1-ff15-4600-b711-2cb109114914", "name": "Infrastructure", "category": "Team", "window": { "start": "2023-01-01T00:00:00Z", "end": "2023-02-01T00:00:00Z" }, "data": { "cost": { "totalCost": 60.0, "idleCost": 10.0, "domainCosts": { "kubernetesCost": { "totalCost": 40.0, "idleCost": 10.0 }, "cloudCost": { "totalCost": 20.0 } } } } } ``` -------------------------------- ### Install DCGM Exporter Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=configuration-nvidia-gpu-monitoring-configurations Deploy the DCGM Exporter using the specified configuration file. ```bash helm upgrade -i dcgm dcgm-exporter \ --repo https://nvidia.github.io/dcgm-exporter/helm-charts \ -n dcgm-exporter --create-namespace \ -f values-dcgm.yaml ``` -------------------------------- ### Query total costs with groups example Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=apis-query-collection-total-costs Example request and response for retrieving total costs including group-level breakdowns and overlap data by setting groups=true. ```http GET /model/collection/query/total?id=8a939ba1-ff15-4600-b711-2cb109114914&window=30d&groups=true ``` ```json { "id": "8a939ba1-ff15-4600-b711-2cb109114914", "name": "Infrastructure", "category": "Team", "window": { "start": "2023-01-01T00:00:00Z", "end": "2023-02-01T00:00:00Z" }, "data": { "cost": { "totalCost": 60.0, "idleCost": 10.0, "domainCosts": { "kubernetesCost": { "totalCost": 40.0, "idleCost": 10.0 }, "cloudCost": { "totalCost": 20.0 } } }, "overlap": -10.0, "groups": [ { "selectors": { "kubernetes": { "kind": "container", "filterString": "namespace:\"infrastructure\"" } }, "cost": { "totalCost": 30.0, "idleCost": 5.0, "domainCosts": { "kubernetesCost": { "totalCost": 30.0, "idleCost": 5.0 } } } }, { "selectors": { "kubernetes": { "kind": "container", "filterString": "node:\"monitoring\"" } }, "cost": { "totalCost": 20.0, "idleCost": 5.0, "domainCosts": { "kubernetesCost": { "totalCost": 20.0, "idleCost": 5.0 } } } }, { "selectors": { "cloud": { "filterString": "service:\"s3\"+label[team]:\"infrastructure\"" } }, "cost": { "totalCost": 20.0, "domainCosts": { "cloudCost": { "totalCost": 20.0 } } } } ] } } ``` -------------------------------- ### Trend Calculation Output Example Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=apis-cloud-cost-trends-api Example output showing a calculated trend value for a specific project, indicating a percentage change in spending. ```json "trends": { "project-123": { "isInfinite": false, "isNaN": false, "value": -0.1472170691451784 } }, "window": { "start": "2023-11-29T00:00:00Z", "end": "2023-12-06T00:00:00Z" }, "comparisonWindow": { "start": "2023-11-22T00:00:00Z", "end": "2023-11-29T00:00:00Z" } ``` -------------------------------- ### Install Datadog Agent via Helm Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=configuration-integrating-kubecost-datadog Add the Datadog Helm repository and install the agent with the necessary Prometheus scraping flags enabled. ```bash helm repo add datadog https://helm.datadoghq.com helm upgrade -i datadog-agent datadog/datadog \ --setdatadog.site='us5.datadoghq.com'\ --setdatadog.apiKey=$ \ --setdatadog.prometheusScrape.enabled=‘true’ \ --setdatadog.prometheusScrape.serviceEndpoints=‘true’ ``` -------------------------------- ### Create Kubecost Namespace Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=guide-aws-distro-open-telemetry Create a dedicated namespace for the Kubecost installation. ```bash kubectl create ns YOUR_NAMESPACE ``` -------------------------------- ### Example cost metric response Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=guide-amazon-managed-service-prometheus Expected JSON output when verifying cost metrics. ```json { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "__name__": "node_total_hourly_cost", "cluster_id": "qa-eks1", "alpha_eksctl_io_cluster_name": "qa-eks1", "arch": "amd64", "instance": "ip-192-168-47-226.us-east-2.compute.internal", "instance_type": "t3.medium", "job": "kubecost" }, "value": [ 1697630306, "0.04160104542160034" ] } ] } } ``` -------------------------------- ### Initialize Plural configuration Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=installations-installing-kubecost-plural Run this command to start the configuration wizard for your Git repository and cloud provider. ```bash plural init ``` -------------------------------- ### Initialize Namespace and Environment Variables Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=integration-aws-cloud-using-irsaeks-pod-identities Sets up the Kubernetes namespace and required environment variables for cluster configuration. ```bash kubectl create ns kubecostexportCLUSTER_NAME=YOUR_CLUSTERexportAWS_REGION=YOUR_REGION ``` -------------------------------- ### Helm installation error due to missing PodSecurityPolicy Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=troubleshoot-install Example of the error message encountered when installing Kubecost on Kubernetes v1.25+ due to the removal of PodSecurityPolicy. ```bash helm install kubecost kubecost/cost-analyzer Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: [ resource mapping not found for name: "kubecost-grafana" namespace: "" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1" ensure CRDs are installed first, resource mapping not found for name: "kubecost-cost-analyzer-psp" namespace: "" from "": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1" ensure CRDs are installed first ] ``` -------------------------------- ### Retrieve cluster cost events via GET request Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=apis-cost-events-audit-api Example JSON response structure for a successful 200 OK request to the audit events endpoint. ```json { "code": 200, "data": [ { "eventType": "add", "resourceType": "deployment", "timestamp": "2023-02-28T13:58:44.552788548Z", "properties": { "cluster": "cluster-one", "name": "sample-deployment", "namespace": "kubecost" }, "totalRequests": { "cpu": "100m", "memory": "32Mi" }, "resourceCosts": { "costPerCPUCoreHour": 0.031616438356164386, "costPerRAMByteHour": 3.94217363775593e-12, "costPerGPUHour": 0.95 }, "costPrediction": { "totalMonthlyRate": 2.4045625000000004, "cpuMonthlyRate": 2.3080000000000003, "ramMonthlyRate": 0.0965625, "gpuMonthlyRate": 0 } }, ... ] } ``` -------------------------------- ### Upgrade Kubecost Installation Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=federation-kubecost-aggregator Upgrades or installs the Kubecost release using the specified aggregator configuration file. ```bash helm upgrade \ --install"kubecost"\ --repo https://kubecost.github.io/cost-analyzer/ cost-analyzer \ --namespace kubecost \ -f aggregator.yaml ``` -------------------------------- ### Cloud Cost Aggregation Example Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=explorer-cloud-cost-metrics Demonstrates how KubernetesPercent values are calculated when aggregating costs across nodes with different pricing models. ```text Node1 { ListCost: { Cost: 2, KubernetesPercent: 1.0, }, AmortizedNetCost: { Cost: 1, KubernetesPercent: 1.0 }, } Node2 { ListCost: { Cost: 2, KubernetesPercent: 0.0, }, AmortizedNetCost: { Cost: 2, KubernetesPercent: 0.0 }, } agg { ListCost: { Cost: 4, KubernetesPercent: 0.5, }, AmortizedNetCost: { Cost: 3, KubernetesPercent: 0.33 }, } ``` -------------------------------- ### Install Kubecost with Datadog configuration Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=configuration-integrating-kubecost-datadog Uses Helm to deploy Kubecost with a predefined Datadog values file and a specific token. ```bash helm upgrade --install kubecost --namespace kubecost --create-namespace \ --repo https://kubecost.github.io/cost-analyzer/ cost-analyzer \ -f https://raw.githubusercontent.com/kubecost/poc-common-configurations/main/datadog/datadog-values.yaml \ --setkubecostToken="aGVsbUBrdWJlY29zdC5jb20=xm343yadf98" ``` -------------------------------- ### Predict API Response Example Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=apis-spec-cost-prediction-api Example JSON response structure returned by the /speccost endpoint upon a successful request. ```json { "code": 200, "data": [ { "namespace": "kubecost", "controllerKind": "deployment", "controllerName": "kubecost-cost-analyzer", "costBefore": { "totalMonthlyRate": 3.5397661399108418, "cpuMonthlyRate": 2.3273929838395513, "ramMonthlyRate": 1.2123731560712905, "gpuMonthlyRate": 0, "monthlyCPUCoreHours": 73, "monthlyRAMByteHours": 304653271040, "monthlyGPUHours": 0 }, "costAfter": { "totalMonthlyRate": 2.623504800996625, "cpuMonthlyRate": 0.6283961056366789, "ramMonthlyRate": 1.9951086953599462, "gpuMonthlyRate": 0, "monthlyCPUCoreHours": 19.71, "monthlyRAMByteHours": 501344315550, "monthlyGPUHours": 0 }, "costChange": { "totalMonthlyRate": -0.9162613389142167, "cpuMonthlyRate": -1.6989968782028724, "ramMonthlyRate": 0.7827355392886557, "gpuMonthlyRate": 0, "monthlyCPUCoreHours": -53.29, "monthlyRAMByteHours": 196691044510, "monthlyGPUHours": 0 } } ] } ``` -------------------------------- ### Execute GKE service account setup script Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=configuration-cluster-controller Run this command to automatically create the required service account for GKE clusters. ```bash /bin/bash -c "$(curl -fsSL https://github.com/kubecost/cluster-turndown/releases/latest/download/gke-create-service-key.sh)" -- cluster-controller-service-key ``` -------------------------------- ### Install Kubecost with Helm Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=guide-grafana-cloud-integration-kubecost Use this command to install or upgrade the Kubecost Helm release with the necessary Grafana Cloud Prometheus configuration. ```bash helm upgrade -i -n kubecost kubecost kubecost/cost-analyzer \ --namespace kubecost \ --set global.prometheus.fqdn=https:// \ --set global.prometheus.enabled=false \ --set global.prometheus.queryServiceBasicAuthSecretName=dbsecret ``` -------------------------------- ### Clone Configuration Repository Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=guide-aws-distro-open-telemetry Download the necessary configuration files for deploying Kubecost with ADOT. ```bash git clone https://github.com/kubecost/poc-common-configurations.git cd poc-common-configurations/aws-amp/adot ``` -------------------------------- ### Build and deploy Kubecost Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=installations-installing-kubecost-plural Generates the configuration and deploys the infrastructure to your cloud provider. ```bash plural build plural deploy --commit"deploying kubecost" ``` -------------------------------- ### Metadata Connection Error Example Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=troubleshoot-install Example of a timeout error occurring when the Kubecost pod cannot reach the cloud provider metadata server. ```text gcpprovider.go Error loading metadata cluster-name: Get "http://169.254.169.254/computeMetadata/v1/instance/attributes/cluster-name": dial tcp 169.254.169.254:80: i/o timeout ``` -------------------------------- ### Install Kubecost via Flat Manifest Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=installation Deploys Kubecost using a static manifest file. This method is not recommended due to limited flexibility for future upgrades. ```bash kubectl apply -f https://raw.githubusercontent.com/kubecost/cost-analyzer-helm-chart/v2.8/kubecost.yaml ``` -------------------------------- ### Query collection time series costs example Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=apis-query-collection-time-series-costs Example request and response for querying time series costs over a 30-day window. ```http GET /model/collection/query/timeseries?id=8a939ba1-ff15-4600-b711-2cb109114914&window=30d ``` ```json { "id": "8a939ba1-ff15-4600-b711-2cb109114914", "name": "Infrastructure", "category": "Team", "window": { "start": "2023-01-01T00:00:00Z", "end": "2023-02-01T00:00:00Z" }, "data": { "timeseries": [ { "window": { "start": "2023-01-01T00:00:00Z", "end": "2023-01-02T00:00:00Z" }, "cost": { "totalCost": 20.0, "idleCost": 5.0, "domainCosts": { "cloudCost": { "totalCost": 10.0 }, "kubernetesCost": { "totalCost": 10.0, "idleCost": 5.0 } } } }, { "window": { "start": "2023-01-02T00:00:00Z", "end": "2023-01-03T00:00:00Z" }, "cost": { "totalCost": 20.0, "idleCost": 5.0, "domainCosts": { "cloudCost": { "totalCost": 10.0 }, "kubernetesCost": { "totalCost": 10.0, "idleCost": 5.0 } } } }, ... { "window": { "start": "2023-01-31T00:00:00Z", "end": "2023-02-01T00:00:00Z" }, "cost": { "totalCost": 20.0, "idleCost": 5.0, "domainCosts": { "cloudCost": { "totalCost": 10.0 }, "kubernetesCost": { "totalCost": 10.0, "idleCost": 5.0 } } } } ] } } ``` -------------------------------- ### Analyze the CPU profile Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=install-profile-http-endpoint Use the Go pprof tool to visualize the captured CPU profile data. ```bash go tool pprof -http 127.0.0.1:9367 /tmp/profile.cpu.out ``` -------------------------------- ### Install Amazon EBS CSI add-on Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=installations-amazon-eks-integration Installs the Amazon EBS CSI add-on for EKS using the previously created IAM role. ```bash eksctl create addon \ --name aws-ebs-csi-driver \ --cluster $CLUSTER_NAME \ --service-account-role-arn $SERVICE_ACCOUNT_ROLE_ARN \ --force ``` -------------------------------- ### Query category time series costs example Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=apis-query-category-time-series-costs Example request and response for querying time series costs for a category over a 30-day window. ```http GET /model/collections/query/timeseries?category=Team&window=30d ``` ```json { "category": "Team", "window": { "start": "2023-01-01T00:00:00Z", "end": "2023-02-01T00:00:00Z" }, "collections": [ { "id": "8a939ba1-ff15-4600-b711-2cb109114914", "name": "Infrastructure" }, { "id": "5aaaab83-f721-41da-ad90-8dca4f7c0f45", "name": "Core" }, { "id": "ba8bda2f-2cce-4f9d-a9d7-655b3ca52cbd", "name": "Front End" } ], "data": { "timeseries": [ { "window": { "start": "2023-01-01T00:00:00Z", "end": "2023-01-02T00:00:00Z" }, "cost": { "totalCost": 30.0, "idleCost": 5.0, "domainCosts": { "cloudCost": { "totalCost": 10.0 }, "kubernetesCost": { "totalCost": 20.0, "idleCost": 5.0 } } } }, { "window": { "start": "2023-01-02T00:00:00Z", "end": "2023-01-03T00:00:00Z" }, "cost": { "totalCost": 30.0, "idleCost": 5.0, "domainCosts": { "cloudCost": { "totalCost": 10.0 }, "kubernetesCost": { "totalCost": 20.0, "idleCost": 5.0 } } } }, ... { "window": { "start": "2023-01-31T00:00:00Z", "end": "2023-02-01T00:00:00Z" }, "cost": { "totalCost": 30.0, "idleCost": 5.0, "domainCosts": { "cloudCost": { "totalCost": 10.0 }, "kubernetesCost": { "totalCost": 20.0, "idleCost": 5.0 } } } } ] } } ``` -------------------------------- ### Download Kubecost Datadog Dashboard Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=configuration-integrating-kubecost-datadog Use this command to download the example dashboard JSON file from the official Kubecost repository. ```bash wget https://raw.githubusercontent.com/kubecost/poc-common-configurations/main/datadog/Kubecostdashboard.json ``` -------------------------------- ### Download the CPU profile Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=install-profile-http-endpoint Save the generated CPU profile to a local file for further analysis. ```bash curl -G 'localhost:9090/model/httpprofile/cpu' \ -o '/tmp/profile.cpu.out' ``` -------------------------------- ### Update collection example Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=apis-update-existing-collection Example of a PUT request targeting a specific collection ID and the corresponding JSON body containing the updated collection specification. ```http PUT /model/collection?id=8a939ba1-ff15-4600-b711-2cb109114914 ``` ```json { "name": "Infrastructure", "category": "Team", "groups": [ { "selectors": { "kubernetes": { "kind": "container", "filterString": "namespace:\"infrastructure\"" } } }, { "selectors": { "kubernetes": { "kind": "container", "filterString": "label[app]:\"infrastructure\"" } } }, { "selectors": { "cloud": { "filterString": "service:\"AmazonEC2\"+label[team]:\"infrastructure\"" } } } ] } ``` -------------------------------- ### Install Kubecost with Helm Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=guide-google-cloud-managed-service-prometheus Execute the Helm upgrade command to deploy Kubecost using the configured values.yaml file. ```bash helm upgrade -i kubecost cost-analyzer/ \ --namespace kubecost \ --create-namespace \ -f values.yaml ``` -------------------------------- ### Install Kubecost via Helm Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=installation Deploys Kubecost directly using the Helm chart. This method does not generate a product token for tier management or trials. ```bash helm upgrade --install kubecost \ --repo https://kubecost.github.io/cost-analyzer/ cost-analyzer \ --namespace kubecost --create-namespace ``` -------------------------------- ### Install Grafana Agent for Kubecost Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=guide-grafana-cloud-integration-kubecost Configures and installs the Grafana Agent to scrape Kubecost metrics. Replace placeholders with your specific Grafana Cloud credentials and cluster details. ```bash cat <<'EOF' | kind: ConfigMap metadata: name: grafana-agent apiVersion: v1 data: agent.yaml: | metrics: wal_directory: /var/lib/agent/wal global: scrape_interval: 60s external_labels: cluster: configs: - name: integrations remote_write: - url: https:// basic_auth: username: password: scrape_configs: #Need further scrape config update - job_name: kubecost honor_labels: true scrape_interval: 1m scrape_timeout: 10s metrics_path: /metrics scheme: http dns_sd_configs: - names: - kubecost-cost-analyzer.kubecost type: 'A' port: 9003 - job_name: kubecost-networking kubernetes_sd_configs: - role: pod relabel_configs: # Scrape only the the targets matching the following metadata - source_labels: [__meta_kubernetes_pod_label_app] action: keep regex: 'kubecost-network-costs' EOF (export NAMESPACE=kubecost && kubectl apply -n $NAMESPACE -f -) MANIFEST_URL=https://raw.githubusercontent.com/grafana/agent/v0.24.2/production/kubernetes/agent-bare.yaml NAMESPACE=kubecost /bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent/v0.24.2/production/kubernetes/install-bare.sh)" | kubectl apply -f - ``` -------------------------------- ### Create Kubecost Scraper Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=guide-aws-agentless-amp Initialize the AMP scraper for Kubecost using the specified configuration blob. ```bash KUBECOST_SCRAPER_OUTPUT=$(aws amp create-scraper --output json \ --alias kubecost-scraper \ --source eksConfiguration="{clusterArn=$CLUSTER_ARN, securityGroupIds=[$SECURITY_GROUP_IDS],subnetIds=[$SUBNET_IDS]}" \ --scrape-configuration configurationBlob="$(base64 scraper-kubecost-with-networking.yaml|tr -d '\n')" \ --destination ampConfiguration="{workspaceArn=$WORKSPACE_ARN}") echo $KUBECOST_SCRAPER_OUTPUT KUBECOST_SCRAPER_ID=$(echo $KUBECOST_SCRAPER_OUTPUT|jq -r .scraperId) echo $KUBECOST_SCRAPER_ID ``` -------------------------------- ### GET /model/logs/level Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=troubleshoot-install Retrieves the current log level configuration. ```APIDOC ## GET /model/logs/level ### Description Retrieves the current log level currently in use by the system. ### Method GET ### Endpoint /model/logs/level ``` -------------------------------- ### Clone Configuration Repository Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=guide-aws-agentless-amp Download the required configuration files for deploying Kubecost with AWS Agentless AMP. ```bash git clone https://github.com/kubecost/poc-common-configurations.git cd poc-common-configurations/aws/amp-agentless ``` -------------------------------- ### GET /model/savings/turbonomic/suspendContainerPods Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=apis-turbonomic-actions Returns virtual machines that Turbonomic recommends for suspension. ```APIDOC ## GET /model/savings/turbonomic/suspendContainerPods ### Description The Suspend Container Pods API returns virtual machines that Turbonomic recommends for suspension. The list of results returned should align with those in the Turbonomic Actions Center. ### Method GET ### Endpoint http:///model/savings/turbonomic/suspendContainerPods ### Parameters #### Query Parameters - **filter** (string) - Optional - Filter your results by cluster, namespace and/or controller. ### Response #### Success Response (200) - **code** (integer) - Status code - **data** (object) - Response data containing numResults, totalSavings, and actions list #### Response Example { "code": 200, "data": { "numResults": 1, "totalSavings": 9.03, "actions": [ { "action": { "cluster": "standard-cluster-1", "node": "gke-standard-cluster-1-spotpool-b4a02c44-1001", "available": true, "targetId": "11111111111111" }, "currentMonthlyRate": 9.03, "predictedMonthlyRate": 0, "predictedSavings": 9.03 } ] } } ``` -------------------------------- ### Create and configure cross-project service account Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=integrations-gcp-cloud-billing-integration Execute these commands to create the service account and assign the required IAM roles for BigQuery access across projects. ```bash gcloud config set project $KUBECOST_PROJECT_ID gcloud iam service-accounts create $SERVICE_ACCOUNT_NAME \ --display-name "Cross Project CUR" \ --format json gcloud projects add-iam-policy-binding $BIG_QUERY_PROJECT_ID \ --member serviceAccount:$SERVICE_ACCOUNT_NAME@$KUBECOST_PROJECT_ID.iam.gserviceaccount.com \ --role roles/compute.viewer gcloud projects add-iam-policy-binding $BIG_QUERY_PROJECT_ID \ --member serviceAccount:$SERVICE_ACCOUNT_NAME@$KUBECOST_PROJECT_ID.iam.gserviceaccount.com \ --role roles/bigquery.user gcloud projects add-iam-policy-binding $BIG_QUERY_PROJECT_ID \ --member serviceAccount:$SERVICE_ACCOUNT_NAME@$KUBECOST_PROJECT_ID.iam.gserviceaccount.com \ --role roles/bigquery.dataViewer gcloud projects add-iam-policy-binding $BIG_QUERY_PROJECT_ID \ --member serviceAccount:$SERVICE_ACCOUNT_NAME@$KUBECOST_PROJECT_ID.iam.gserviceaccount.com \ --role roles/bigquery.jobUser ``` -------------------------------- ### GET /model/budgets Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=apis-budget-api Retrieves a list of all existing recurring budget rules. ```APIDOC ## GET /model/budgets ### Description Lists all existing recurring budget rules. ### Method GET ### Endpoint http:///model/budgets ### Response #### Success Response (200) - **code** (int) - Status code - **data** (array) - List of existing budget rules ``` -------------------------------- ### Open the database in DuckDB Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=federation-kubecost-aggregator Starts the DuckDB REPL interface using the copied database file. ```bash duckdb kubecost-example.duckdb.read.kubecost.copy ``` -------------------------------- ### Create a product key JSON file Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=configuration-adding-product-key Define the product key in a JSON file to be used for creating a Kubernetes secret. ```json { "key": "" } ``` -------------------------------- ### Example Policy ARN Output Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=integration-aws-cloud-using-irsaeks-pod-identities The ARN generated after creating the IAM policy. ```text arn:aws:iam::SUB_ACCOUNT_222222222:policy/kubecost-access-cur-in-payer-account ``` -------------------------------- ### Create cAdvisor and Node Exporter Scraper Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=guide-aws-agentless-amp Initialize the AMP scraper for cAdvisor and node exporter. ```bash CADVSIOR_SCRAPER_OUTPUT=$(aws amp create-scraper --output json \ --alias cadvisor-scraper \ --source eksConfiguration="{clusterArn=$CLUSTER_ARN, securityGroupIds=[$SECURITY_GROUP_IDS],subnetIds=[$SUBNET_IDS]}" \ --scrape-configuration configurationBlob="$(base64 scraper-cadvisor-node-exporter.yaml|tr -d '\n')" \ --destination ampConfiguration="{workspaceArn=$WORKSPACE_ARN}") echo $CADVSIOR_SCRAPER_OUTPUT CADVSIOR_SCRAPER_ID=$(echo $CADVSIOR_SCRAPER_OUTPUT|jq -r .scraperId) echo $CADVSIOR_SCRAPER_ID ``` -------------------------------- ### Partial API Response Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=apis-spec-cost-prediction-api A truncated example of the JSON response returned by the API. ```json [ { "namespace": "kubecost", "controllerKind": "deployment", "controllerName": "kubecost-cost-analyzer", "costBefore": { "totalMonthlyRate": 3.5397661399108418, "cpuMonthlyRate": 2.3273929838395513, "ramMonthlyRate": 1.2123731560712905, "gpuMonthlyRate": 0, "monthlyCPUCoreHours": 73, ``` -------------------------------- ### Install Kubecost on Amazon EKS Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=installations-amazon-eks-integration Deploys Kubecost using Helm. Replace <$VERSION> with the desired chart version. ```bash helm upgrade -i kubecost \ oci://public.ecr.aws/kubecost/cost-analyzer \ --version <$VERSION> \ --namespace kubecost \ --create-namespace \ -f https://raw.githubusercontent.com/kubecost/cost-analyzer-helm-chart/<$VERSION>/cost-analyzer/values-eks-cost-monitoring.yaml ``` ```bash helm upgrade -i kubecost \ oci://public.ecr.aws/kubecost/cost-analyzer \ --version <$VERSION> \ --namespace kubecost \ --create-namespace \ -f https://raw.githubusercontent.com/kubecost/cost-analyzer-helm-chart/<$VERSION>/cost-analyzer/values-eks-cost-monitoring.yaml \ ``` -------------------------------- ### GET /model/savings/turbonomic/resizeWorkloadControllers Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=apis-turbonomic-actions Returns a list of workloads for which request resizing has been recommended by Turbonomic. ```APIDOC ## GET /model/savings/turbonomic/resizeWorkloadControllers ### Description The Resize Workload Controllers API returns workloads for which request resizing has been recommended by Turbonomic. The list of results returned should align with those in the Turbonomic Actions Center. ### Method GET ### Endpoint http:///model/savings/turbonomic/resizeWorkloadControllers ### Parameters #### Query Parameters - **filter** (string) - Optional - Filter your results by cluster, namespace and/or controller. ### Response #### Success Response (200) - **code** (integer) - Status code - **data** (object) - Response data containing numResults, totalSavings, and actions list #### Response Example { "code": 200, "data": { "numResults": 1, "totalSavings": 2.00, "actions": [ { "action": { "cluster": "standard-cluster-1", "namespace": "kubecost", "controller": "kubecost-cost-analyzer", "replicaCount": 1, "compoundActions": { "cost-model": [ { "target": "VCPURequest", "unit": "mCores", "oldValue": 200, "newValue": 100 } ] }, "available": true, "targetId": "11111111111111" }, "currentMonthlyRate": 4.00, "predictedMonthlyRate": 2.00, "predictedSavings": 2.00 } ] } } ``` -------------------------------- ### GET /model/diagnostic/tableWindowCount Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=apis-aggregator-diagnostic Determines the number of unique WindowStart/WindowEnd pairs in a specified table. ```APIDOC ## GET /model/diagnostic/tableWindowCount ### Description Used to determine the number of unique WindowStart/WindowEnd pairs exist in the table. ### Parameters #### Query Parameters - **table** (string) - Required - The table containing container data to consider. Try `container_1d`. ``` -------------------------------- ### Configure SAML and RBAC via Helm Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=configuration-user-management-saml Example configuration for enabling SAML authentication and mapping identity provider groups to Kubecost roles. ```yaml # EXAMPLE CONFIGURATION # View setup guides below, for full list of Helm configuration values saml: enabled: true secretName: "kubecost-okta" idpMetadataURL: "https://your.idp.subdomain.okta.com/app/exk4h09oysB785123/sso/saml/metadata" appRootURL: "https://kubecost.your.com" authTimeout: 1440 audienceURI: "https://kubecost.your.com" nameIDFormat: "urn:oasis:names:tc:SAML:2.0:attrname-format:basic" rbac: enabled: true groups: - name: admin enabled: true assertionName: "kubecost_group" assertionValues: - "kubecost_admin" - "kubecost_superusers" - name: readonly enabled: true assertionName: "kubecost_group" assertionvalues: - "kubecost_users" customGroups: - assertionName: "kubecost_group" ``` -------------------------------- ### Describe Kubecost add-on status Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=installations-amazon-eks-integration Check the current status of the Kubecost add-on installation. ```bash aws eks describe-addon \ --addon-name kubecost_kubecost \ --cluster-name $YOUR_CLUSTER_NAME \ --region $AWS_REGION ``` -------------------------------- ### Configure Postgres Integration in Helm values Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=configuration-kubecost-postgres-integration Example YAML configuration for enabling and setting up Postgres integration within the Kubecost Helm values file. ```yaml global: integrations: postgres: enabled: true runInterval: "12h" # How frequently to run the integration. databaseHost: "" # REQUIRED. ex: my.postgres.database.azure.com databasePort: "" # REQUIRED. ex: 5432 databaseName: "" # REQUIRED. ex: postgres databaseUser: "" # REQUIRED. ex: myusername databasePassword: "" # REQUIRED. ex: mypassword databaseSecretName: "" # OPTIONAL. Specify your own k8s secret containing the above credentials. Must have key "creds.json". queryConfigs: allocations: - databaseTable: "kubecost_allocation_data" window: "7d" aggregate: "namespace" idle: "true" shareIdle: "true" shareNamespaces: "kubecost,kube-system" shareLabels: "" - databaseTable: "kubecost_allocation_data_by_cluster" window: "10d" aggregate: "cluster" idle: "true" shareIdle: "false" shareNamespaces: "" shareLabels: "" assets: - databaseTable: "kubecost_assets_data" window: "7d" aggregate: "cluster" cloudCosts: - databaseTable: "kubecost_cloudcosts_data" window: "7d" aggregate: "service" # REQUIRED. Aggregator must be enabled and running as a statefulset. kubecostAggregator: deployMethod: "statefulset" kubecostModel: federatedStorageConfigSecret: "federated-store" prometheus: server: global: external_labels: cluster_id: "primary-cluster-1" # REQUIRED. A Kubecost Enterprise license key. kubecostProductConfigs: productKey: enabled: true key: "my-enterprise-key-here" ``` -------------------------------- ### Install DCGM Exporter via Helm Source: https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=configuration-nvidia-gpu-monitoring-configurations Deploy the DCGM Exporter chart using the previously defined values file. ```bash helm upgrade \ -i dcgm dcgm-exporter \ --repo https://nvidia.github.io/dcgm-exporter/helm-charts \ -n dcgm-exporter \ --create-namespace \ -f values-dcgm.yaml ```