### Install Kompat CLI Tool Source: https://github.com/aws/karpenter-provider-aws/blob/main/tools/kompat/README.md Use 'go install' to install the latest version of the Kompat CLI tool. Ensure your Go environment is configured. ```bash go install github.com/aws/karpenter/tools/kompat/cmd/kompat@latest ``` -------------------------------- ### NodePool Configuration Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodepools.md This example demonstrates a NodePool configuration with static replicas, specific requirements for spot instances, and resource limits for CPU, memory, GPU, and the maximum number of nodes. ```yaml apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: default spec: replicas: 10 template: spec: requirements: - key: karpenter.sh/capacity-type operator: In values: ["spot"] limits: cpu: 1000 memory: 1000Gi nvidia.com/gpu: 2 # For static NodePools, only nodes limit is allowed nodes: 20 ``` -------------------------------- ### Install Karpenter Helm Chart for Fargate Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/getting-started/getting-started-with-karpenter/_index.md Installs the Karpenter Helm chart for environments using AWS Fargate. Ensure you have Helm installed and configured. ```bash helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version 1.14.0 --namespace karpenter --create-namespace --set serviceAccount.create=false --set serviceAccount.name=karpenter --set controller.clusterName=my-cluster --set controller.aws.clusterName=my-cluster --set controller.aws.defaultInstanceProfile=karpenter-instance-profile --set controller.aws.region=us-east-1 --set controller.aws.excludeOutboundSecurityGroups=true --set controller.aws.externalDnsName=karpenter.my-cluster.example.com --set controller.aws.instanceProfile=karpenter-instance-profile --set controller.aws.fargateProfile=true --set controller.aws.provisionerSecretRef=karpenter-secret --set controller.aws.s3Bucket=karpenter-state-store-my-cluster --set controller.aws.s3Region=us-east-1 --set controller.aws.signingName=karpenter.k8s.amazonaws.com --set controller.aws.signingRegion=us-east-1 --set controller.aws.tags."karpenter.sh/discovery"=my-cluster --set controller.aws.trustedDomain=example.com --set controller.aws.vpcId=vpc-0123456789abcdef0 --set controller.aws.zone=us-east-1a ``` -------------------------------- ### Install Karpenter Helm Chart for Managed NodeGroups Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/getting-started/getting-started-with-karpenter/_index.md Installs the Karpenter Helm chart for environments using Managed NodeGroups. Ensure you have Helm installed and configured. ```bash helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version 1.14.0 --namespace karpenter --create-namespace --set serviceAccount.create=false --set serviceAccount.name=karpenter --set controller.clusterName=my-cluster --set controller.aws.clusterName=my-cluster --set controller.aws.defaultInstanceProfile=karpenter-instance-profile --set controller.aws.region=us-east-1 --set controller.aws.excludeOutboundSecurityGroups=true --set controller.aws.externalDnsName=karpenter.my-cluster.example.com --set controller.aws.instanceProfile=karpenter-instance-profile --set controller.aws.managedNodeGroups=true --set controller.aws.provisionerSecretRef=karpenter-secret --set controller.aws.s3Bucket=karpenter-state-store-my-cluster --set controller.aws.s3Region=us-east-1 --set controller.aws.signingName=karpenter.k8s.amazonaws.com --set controller.aws.signingRegion=us-east-1 --set controller.aws.tags."karpenter.sh/discovery"=my-cluster --set controller.aws.trustedDomain=example.com --set controller.aws.vpcId=vpc-0123456789abcdef0 --set controller.aws.zone=us-east-1a ``` -------------------------------- ### NodeOverlay Configuration Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeoverlays.md This example demonstrates how to configure a NodeOverlay to modify instance type information. It specifies requirements for instance type, capacity type, and CPU, and then overrides the price and adds extended resources like hugepages and custom GPUs. ```yaml apiVersion: karpenter.sh/v1alpha1 kind: NodeOverlay metadata: name: example-overlay spec: # Optional weight for conflict resolution (higher weight wins) weight: 10 # Requirements determine which instance types this overlay applies to requirements: - key: node.kubernetes.io/instance-type operator: In values: ["m5.large", "m5.xlarge"] - key: karpenter.sh/capacity-type operator: In values: ["spot"] - key: karpenter.k8s.aws/instance-cpu operator: Gte values: ["32"] # Price and priceAdjustment are mutually exclusive # Price override (sets absolute price) price: "5.00" # Price adjustment (modifies existing price) priceAdjustment: "+10%" # or "-0.50" for absolute adjustment # Extended resources to add to matching instance types capacity: hugepages-2Mi: 100Mi hugepages-1Gi: 2Gi custom-device/gpu-slice: 4 ``` -------------------------------- ### Ubuntu NodeClass Example with AL2 AMIFamily Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md This example demonstrates how to configure an EC2NodeClass to use the AL2 AMIFamily for Ubuntu nodes, specifying a root volume and EBS settings. This is a workaround due to dropped Ubuntu support. ```yaml apiVersion: karpenter.k8s.aws/v1 kind: EC2NodeClass spec: amiFamily: AL2 amiSelectorTerms: - id: ami-placeholder blockDeviceMappings: - deviceName: '/dev/sda1' rootVolume: true ebs: encrypted: true volumeType: gp3 volumeSize: 20Gi ``` -------------------------------- ### Capacity Buffer Status Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/capacitybuffers.md This YAML snippet shows an example of the status conditions and other relevant information for a Capacity Buffer, including its readiness, provisioning state, replica count, and referenced pod template. ```yaml status: conditions: - type: ReadyForProvisioning status: "True" reason: Resolved message: "Pod template resolved successfully" - type: Provisioning status: "True" reason: FitsExistingCapacity message: "All 5 virtual pods fit on existing capacity" replicas: 5 podTemplateRef: name: web-buffer-template podTemplateGeneration: 3 provisioningStrategy: "buffer.x-k8s.io/active-capacity" ``` -------------------------------- ### Install Karpenter Helm Chart Source: https://github.com/aws/karpenter-provider-aws/blob/main/charts/karpenter/README.md Installs or upgrades the Karpenter Helm chart using the OCI registry. Ensure the KARPENTER_IAM_ROLE_ARN and CLUSTER_NAME environment variables are set. ```bash helm upgrade --install --namespace karpenter --create-namespace \ karpenter oci://public.ecr.aws/karpenter/karpenter \ --version 1.14.0 \ --set "serviceAccount.annotations.eks.amazonaws.com/role-arn=${KARPENTER_IAM_ROLE_ARN}" \ --set settings.clusterName=${CLUSTER_NAME} \ --set settings.interruptionQueue=${CLUSTER_NAME} \ --wait ``` -------------------------------- ### Example Security Group Selection Logic Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md This example demonstrates selecting security groups based on a combination of tags, names, and IDs. It shows how multiple terms are ORed together, while conditions within a single term are ANDed. ```yaml securityGroupSelectorTerms: # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag # AND the "environment: test" tag OR any security group with the "my-security-group" name # OR any security group with ID "sg-063d7acfb4b06c82c" - tags: karpenter.sh/discovery: "${CLUSTER_NAME}" environment: test - name: my-security-group - id: sg-063d7acfb4b06c82c ``` -------------------------------- ### Pod Disruption Budget Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/troubleshooting.md This example defines a Pod Disruption Budget (PDB) that protects pods with the label `myapp`. Node termination will be blocked if evicting a pod would reduce the number of available pods below 4. ```yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: myapp-pdb spec: minAvailable: 4 selector: matchLabels: app: myapp ``` -------------------------------- ### Custom UserData Script for Kubelet Configuration Source: https://github.com/aws/karpenter-provider-aws/blob/main/designs/custom-user-data-and-amis.md This shell script is executed after the initial Kubelet setup. It allows users to modify Kubelet configurations, such as extra arguments or other Kubelet settings, before the Kubelet fully starts. ```shell #!/bin/bash # Running custom user data script that was in the ProvisionerSpec # At this point, the kubelet hasn't started so as a user I can make mutations # to any of the kubeletConfig / containerRuntime as needed. # To edit kubelet extra-args - /etc/systemd/system/kubelet.service.d/30-kubelet-extra-args.conf # To edit other kubelet args - /etc/systemd/system/kubelet.service.d/ ``` -------------------------------- ### Default NodePool Configuration Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodepools.md This example shows a basic NodePool configuration named 'default'. It specifies labels, annotations, a reference to an EC2NodeClass, taints, startup taints, and node expiration settings. Use this as a starting point for your own NodePool definitions. ```yaml apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: default spec: # Template section that describes how to template out NodeClaim resources that Karpenter will provision # Karpenter will consider this template to be the minimum requirements needed to provision a Node using this NodePool # It will overlay this NodePool with Pods that need to schedule to further constrain the NodeClaims # Karpenter will provision to launch new Nodes for the cluster template: metadata: # Labels are arbitrary key-values that are applied to all nodes labels: billing-team: my-team # Annotations are arbitrary key-values that are applied to all nodes annotations: example.com/owner: "my-team" spec: # References the Cloud Provider's NodeClass resource, see your cloud provider specific documentation nodeClassRef: group: karpenter.k8s.aws # Updated since only a single version will be served kind: EC2NodeClass name: default # Provisioned nodes will have these taints # Taints may prevent pods from scheduling if they are not tolerated by the pod. taints: - key: example.com/special-taint effect: NoSchedule # Provisioned nodes will have these taints, but pods do not need to tolerate these taints to be provisioned by this # NodePool. These taints are expected to be temporary and some other entity (e.g. a DaemonSet) is responsible for # removing the taint after it has finished initializing the node. startupTaints: - key: example.com/another-taint effect: NoSchedule # The amount of time a Node can live on the cluster before being removed # Avoiding long-running Nodes helps to reduce security vulnerabilities as well as to reduce the chance of issues that can plague Nodes with long uptimes such as file fragmentation or memory leaks from system processes # You can choose to disable expiration entirely by setting the string value 'Never' here # Note: changing this value in the nodepool will drift the nodeclaims. expireAfter: 720h | Never # The amount of time that a node can be draining before it's forcibly deleted. A node begins draining when a delete call is made against it, starting # its finalization flow. Pods with TerminationGracePeriodSeconds will be deleted preemptively before this terminationGracePeriod ends to give as much time to cleanup as possible. # If your pod's terminationGracePeriodSeconds is larger than this terminationGracePeriod, Karpenter may forcibly delete the pod # before it has its full terminationGracePeriod to cleanup. # Note: changing this value in the nodepool will drift the nodeclaims. terminationGracePeriod: 48h ``` -------------------------------- ### Configure Architecture-Specific Provisioners Source: https://github.com/aws/karpenter-provider-aws/blob/main/designs/provisioner-priority.md Demonstrates how to set up a default provisioner for amd64 architecture and a fallback provisioner for ARM-64 using provisioner weights. This ensures specific architecture requirements are met without requiring manual nodeSelectors on every workload. ```yaml apiVersion: karpenter.sh/v1alpha5 kind: Provisioner metadata: name: default spec: weight: 100 requirements: - key: kubernetes.io/arch operator: In values: ["amd64"] ``` ```yaml apiVersion: karpenter.sh/v1alpha5 kind: Provisioner metadata: name: arm64 spec: weight: 50 requirements: - key: kubernetes.io/arch operator: Exists ``` -------------------------------- ### Build and Execute Kubereplay Commands Source: https://github.com/aws/karpenter-provider-aws/blob/main/tools/kubereplay/README.md Compile the binary and use the CLI to capture, replay, or generate synthetic test data. Requires EKS audit logging, AWS credentials, and kubectl access. ```bash go build -o kubereplay ./cmd kubereplay capture # capture last hour to replay.json kubereplay replay # replay from replay.json kubereplay replay --speed 24 # replay 24x faster kubereplay demo # generate synthetic test data ``` -------------------------------- ### Karpenter Provisioner Configuration Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/designs/provisioner-priority.md This YAML defines two Karpenter provisioners: 'expensive' which prioritizes 'p3.16xlarge' instances with a high weight, and 'inexpensive' which targets 't3.small' instances. This configuration demonstrates how weights can influence scheduling decisions and potentially lead to higher costs if not managed carefully. ```yaml apiVersion: karpenter.sh/v1alpha5 kind: Provisioner metadata: name: expensive spec: requirements: - key: "node.kubernetes.io/instance-type" operator: In values: ["p3.16xlarge"] weight: 100 ``` ```yaml apiVersion: karpenter.sh/v1alpha5 kind: Provisioner metadata: name: inexpensive spec: requirements: - key: "node.kubernetes.io/instance-type" operator: In values: ["t3.small"] ``` -------------------------------- ### Configure Karpenter Installation Variables Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/getting-started/getting-started-with-karpenter/_index.md This script sets up essential environment variables for Karpenter installation, including cluster name, AWS region, account ID, and versioning. Ensure these are set before proceeding with the installation. ```bash #!/bin/bash # Set the Karpenter version KARPENTER_VERSION="1.14.0" # Set the Kubernetes version K8S_VERSION="1.36" # Set the cluster name CLUSTER_NAME="karpenter-example" # Set the AWS region AWS_DEFAULT_REGION="us-east-1" # Set the AWS account ID AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) # Set the output directory for logs and manifests TEMPOUT=$(mktemp -d) # Set the alias version for Karpenter ALIAS_VERSION="latest" # Enable zonal shift for Karpenter (optional) ENABLE_ZONAL_SHIFT="false" ``` -------------------------------- ### Configure startupTaints for Cilium Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/faq.md Use this example to configure startupTaints in your NodePool to make Karpenter aware of temporary taints, preventing pods from scheduling on nodes that are not yet ready. This is particularly useful for workloads like Cilium. ```yaml apiVersion: karpenter.sh/v1 kind: NodePool ... spec: template: spec: startupTaints: - key: node.cilium.io/agent-not-ready effect: NoSchedule ``` -------------------------------- ### Windows Merged UserData Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md Example of how Karpenter merges UserData for Windows instances, including EKS bootstrap script parameters. ```powershell Write-Host "Running custom user data script" [string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1" & $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=spot,karpenter.sh/nodepool=windows2022" --max-pods=110' -DNSClusterIP '10.0.100.10' ``` -------------------------------- ### Bottlerocket Merged UserData Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md Example of how Karpenter merges UserData for Bottlerocket instances, showing default settings and overridden values. ```toml [settings] [settings.kubernetes] api-server = 'https://cluster' cluster-certificate = 'ca-bundle' cluster-name = 'cluster' [settings.kubernetes.node-labels] 'karpenter.sh/capacity-type' = 'on-demand' 'karpenter.sh/nodepool' = 'default' [settings.kubernetes.node-taints] [settings.kubernetes.eviction-hard] 'memory.available' = '12%%' ``` -------------------------------- ### Example Conflict Resolution with Node Overlays Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeoverlays.md Illustrates how to configure multiple Node Overlays with different weights and settings to demonstrate conflict resolution. Higher weights take precedence, and capacity fields are merged. ```yaml apiVersion: karpenter.sh/v1alpha1 kind: NodeOverlay metadata: name: overlay-a spec: weight: 5 requirements: - key: node.kubernetes.io/instance-type operator: In values: ["m5.large"] priceAdjustment: "-10%" capacity: hugepages-2Mi: 50Mi --- apiVersion: karpenter.sh/v1alpha1 kind: NodeOverlay metadata: name: overlay-b spec: weight: 10 # Higher weight requirements: - key: node.kubernetes.io/instance-type operator: In values: ["m5.large"] priceAdjustment: "-20%" # This overrides overlay-a's adjustment capacity: custom-device/gpu: 1 # This is merged with hugepages-2Mi from overlay-a ``` -------------------------------- ### Windows Passed-in UserData Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md Example of custom UserData for Windows instances using PowerShell. This script is prepended to the Karpenter managed section. ```powershell Write-Host "Running custom user data script" ``` -------------------------------- ### Bottlerocket Passed-in UserData Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md Example of custom UserData for Bottlerocket instances using TOML format. Unknown fields are ignored by Karpenter. ```toml [settings.kubernetes.eviction-hard] "memory.available" = "12%" [settings.kubernetes] "unknown-setting" = "unknown" [settings.kubernetes.node-labels] 'field.controlled.by/karpenter' = 'will-be-overridden' ``` -------------------------------- ### NodeClaim ConsistentStateFound Condition Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/troubleshooting.md An example of the JSON output for a NodeClaim's 'ConsistentStateFound' condition, indicating a 'False' status and 'ConsistencyCheckFailed' reason. ```json { "type": "ConsistentStateFound", "status": "False", "reason": "ConsistencyCheckFailed", "message": "Consistency Check Failed", "lastTransitionTime": "2024-08-19T20:02:16Z" } ``` -------------------------------- ### Example Instance Type Offerings with ODCRs (YAML) Source: https://github.com/aws/karpenter-provider-aws/blob/main/designs/odcr.md This YAML snippet demonstrates how an instance type's offerings are structured when ODCR capacity is available. It includes separate offerings for 'reserved', 'on-demand', and 'spot' capacity types, each with specific requirements including availability zone and price. ```yaml name: c5.large offerings: - price: .... available: .... requirements: - key: karpenter.sh/capacity-type operator: In values: ["reserved"] - key: topology.kubernetes.io/zone operator: In values: ["us-west-2a"] - key: topology.k8s.aws/zone-id operator: In values: ["usw2-az1"] - price: .... available: .... requirements: - key: karpenter.sh/capacity-type operator: In values: ["on-demand"] - key: topology.kubernetes.io/zone operator: In values: ["us-west-2a"] - key: topology.k8s.aws/zone-id operator: In values: ["usw2-az1"] - price: .... available: .... requirements: - key: karpenter.sh/capacity-type operator: In values: ["spot"] - key: topology.kubernetes.io/zone operator: In values: ["us-west-2a"] - key: topology.k8s.aws/zone-id operator: In values: ["usw2-az1"] ``` -------------------------------- ### Node Selector Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/scheduling.md Use nodeSelector to specify scheduling preferences for pods. This example selects nodes in a specific zone and with a 'spot' capacity type. ```yaml nodeSelector: topology.kubernetes.io/zone: us-west-2a karpenter.sh/capacity-type: spot ``` -------------------------------- ### CapacityBuffer Configuration Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/capacitybuffers.md This YAML defines a CapacityBuffer named 'web-app-buffer' in the 'default' namespace. It uses the 'buffer.x-k8s.io/active-capacity' provisioning strategy, references a pod template named 'web-buffer-template', and specifies 5 replicas with CPU and memory limits. ```yaml apiVersion: autoscaling.x-k8s.io/v1alpha1 kind: CapacityBuffer metadata: name: web-app-buffer namespace: default spec: provisioningStrategy: "buffer.x-k8s.io/active-capacity" podTemplateRef: name: web-buffer-template replicas: 5 limits: cpu: "20" memory: "40Gi" ``` -------------------------------- ### NodeOverlay Status Condition Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeoverlays.md This YAML snippet shows an example of a NodeOverlay's status conditions, indicating a configuration conflict that prevents the overlay from being applied. ```yaml status: conditions: - type: ValidationSucceeded status: "False" lastTransitionTime: "2024-07-24T18:30:00Z" reason: "Conflict" message: "conflict with another overlay" ```