### Install Kompat CLI Tool Source: https://github.com/aws/karpenter-provider-aws/blob/main/tools/kompat/README.md Use 'go install' to install the latest version of the Kompat CLI tool. Ensure your Go environment is configured. ```bash go install github.com/aws/karpenter/tools/kompat/cmd/kompat@latest ``` -------------------------------- ### Install Karpenter with Helm (Fargate) Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/getting-started/getting-started-with-karpenter/_index.md Installs Karpenter using the Helm chart for environments utilizing AWS Fargate. ```bash helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version 1.13.0 --namespace karpenter --create-namespace --set "controller.clusterName=" --set "charts.aws/karpenter.clusterName=" --set "fargateProfile.enabled=true" ``` -------------------------------- ### AL2023 MIME UserData Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md This example demonstrates a passed-in UserData in MIME format for AL2023, containing both a shell script and a NodeConfig. ```text MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="//" --// Content-Type: text/x-shellscript; charset="us-ascii" #!/bin/bash echo "Hello, AL2023!" --// Content-Type: application/node.eks.aws apiVersion: node.eks.aws/v1alpha1 kind: NodeConfig spec: kubelet: config: maxPods: 42 --// ``` -------------------------------- ### Install Karpenter with Helm (Managed NodeGroups) Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/getting-started/getting-started-with-karpenter/_index.md Installs Karpenter using the Helm chart for environments with managed node groups. ```bash helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version 1.13.0 --namespace karpenter --create-namespace --set "controller.clusterName=" --set "charts.aws/karpenter.clusterName=" ``` -------------------------------- ### AL2023 Merged NodeConfig Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md This example shows a merged UserData for AL2023, combining a passed-in NodeConfig with Karpenter-generated settings. ```text MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="//" --// Content-Type: application/node.eks.aws apiVersion: node.eks.aws/v1alpha1 kind: NodeConfig spec: kubelet: config: maxPods: 42 --// Content-Type: application/node.eks.aws # Karpenter Generated NodeConfig apiVersion: node.eks.aws/v1alpha1 kind: NodeConfig spec: cluster: apiServerEndpoint: https://test-cluster certificateAuthority: cluster-ca cidr: 10.100.0.0/16 name: test-cluster kubelet: config: clusterDNS: - 10.100.0.10 maxPods: 118 flags: - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default" --//-- ``` -------------------------------- ### AL2023 Merged Bash UserData Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md This example shows a merged UserData for AL2023, combining a passed-in bash script with Karpenter-generated NodeConfig. ```text MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="//" --// Content-Type: text/x-shellscript; charset="us-ascii" #!/bin/bash echo "Hello, AL2023!" --// Content-Type: application/node.eks.aws # Karpenter Generated NodeConfig apiVersion: node.eks.aws/v1alpha1 kind: NodeConfig spec: cluster: apiServerEndpoint: https://test-cluster certificateAuthority: cluster-ca cidr: 10.100.0.0/16 name: test-cluster kubelet: config: clusterDNS: - 10.100.0.10 maxPods: 118 flags: - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default" --//-- ``` -------------------------------- ### AL2023 Merged MIME UserData Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md This example shows a merged UserData for AL2023 in MIME format, combining a passed-in NodeConfig and shell script with Karpenter-generated NodeConfig. ```text MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="//" --// Content-Type: application/node.eks.aws apiVersion: node.eks.aws/v1alpha1 kind: NodeConfig spec: kubelet: config: maxPods: 42 --// Content-Type: text/x-shellscript; charset="us-ascii" #!/bin/bash echo "Hello, AL2023!" --// Content-Type: application/node.eks.aws # Karpenter Generated NodeConfig apiVersion: node.eks.aws/v1alpha1 kind: NodeConfig spec: cluster: apiServerEndpoint: https://test-cluster certificateAuthority: cluster-ca cidr: 10.100.0.0/16 name: test-cluster kubelet: config: clusterDNS: - 10.100.0.10 maxPods: 118 flags: - --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default" --//-- ``` -------------------------------- ### Example Security Group Selection Logic Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md This example demonstrates selecting security groups based on a combination of tags, name, and ID. Multiple terms are ORed together, while conditions within a single term are ANDed. ```yaml securityGroupSelectorTerms: # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag # AND the "environment: test" tag OR any security group with the "my-security-group" name # OR any security group with ID "sg-063d7acfb4b06c82c" - tags: karpenter.sh/discovery: "${CLUSTER_NAME}" environment: test - name: my-security-group - id: sg-063d7acfb4b06c82c ``` -------------------------------- ### Windows Passed-in UserData Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md Example of custom UserData for Windows instances, which must be PowerShell commands. This script is prepended to Karpenter's bootstrap script. ```powershell Write-Host "Running custom user data script" ``` -------------------------------- ### Example Node Launch Failure Log Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/troubleshooting.md This is an example log message indicating a failure to launch a node due to an invalid block device mapping. Such errors often point to misconfigurations in custom launch templates. ```bash 2022-01-19T18:22:23.366Z ERROR controller.provisioning Could not launch node, launching instances, with fleet error(s), InvalidBlockDeviceMapping: Invalid device name /dev/xvda; ... ``` -------------------------------- ### AMI Selector Terms Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md This example demonstrates how to select AMIs using a combination of tags, name, ID, and SSM parameters. Multiple terms are ORed together for flexible AMI selection. ```yaml amiSelectorTerms: # Select on any AMI that has both the `karpenter.sh/discovery: "${CLUSTER_NAME}" # AND `environment: test` tags OR any AMI with the name `my-ami` OR an AMI with # ID `ami-123` OR an AMI with ID matching the value of my-custom-parameter - tags: karpenter.sh/discovery: "${CLUSTER_NAME}" environment: test - name: my-ami - id: ami-123 - ssmParameter: my-custom-parameter # ssm parameter name or ARN # Select EKS optimized AL2023 AMIs with version `v20240807`. This term is mutually # exclusive and can't be specified with other terms. # - alias: al2023@v20240807 ``` -------------------------------- ### Custom UserData Script for Kubelet Configuration Source: https://github.com/aws/karpenter-provider-aws/blob/main/designs/custom-user-data-and-amis.md This shell script is executed after the initial Kubelet setup. It allows users to modify Kubelet configurations, such as extra arguments or other Kubelet settings, before the Kubelet fully starts. ```shell #!/bin/bash # Running custom user data script that was in the ProvisionerSpec # At this point, the kubelet hasn't started so as a user I can make mutations # to any of the kubeletConfig / containerRuntime as needed. # To edit kubelet extra-args - /etc/systemd/system/kubelet.service.d/30-kubelet-extra-args.conf # To edit other kubelet args - /etc/systemd/system/kubelet.service.d/ ``` -------------------------------- ### Example Karpenter Pod Status Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/troubleshooting.md This output shows the status of a Karpenter controller pod, indicating it is running with 2 out of 2 containers ready and has been running for 21 days. This is a typical output from `kubectl get pods -A`. ```bash karpenter karpenter-XXXX 2/2 Running 2 21d ``` -------------------------------- ### Configuring Kubelet via spec.kubelet and UserData Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md This example demonstrates how to configure Kubelet settings, combining `spec.kubelet` for supported fields like `maxPods` and UserData for unsupported fields like `registryPullQPS`. This approach is useful when needing to set advanced Kubelet configurations not directly exposed in the EC2NodeClass. ```yaml apiVersion: karpenter.k8s.aws/v1 kind: EC2NodeClass spec: amiSelectorTerms: - alias: al2023@v20240807 kubelet: maxPods: 42 userData: | apiVersion: node.eks.aws/v1alpha1 kind: NodeConfig spec: kubelet: config: # Configured through UserData since unavailable in `spec.kubelet` registryPullQPS: 10 ``` -------------------------------- ### Default NodePool Configuration Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodepools.md This example shows a basic NodePool configuration named 'default'. It specifies labels, annotations, a reference to an EC2NodeClass, taints, startup taints, and node expiration settings. Use this as a starting point for your own NodePool definitions. ```yaml apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: default spec: # Template section that describes how to template out NodeClaim resources that Karpenter will provision # Karpenter will consider this template to be the minimum requirements needed to provision a Node using this NodePool # It will overlay this NodePool with Pods that need to schedule to further constrain the NodeClaims # Karpenter will provision to launch new Nodes for the cluster template: metadata: # Labels are arbitrary key-values that are applied to all nodes labels: billing-team: my-team # Annotations are arbitrary key-values that are applied to all nodes annotations: example.com/owner: "my-team" spec: # References the Cloud Provider's NodeClass resource, see your cloud provider specific documentation nodeClassRef: group: karpenter.k8s.aws # Updated since only a single version will be served kind: EC2NodeClass name: default # Provisioned nodes will have these taints # Taints may prevent pods from scheduling if they are not tolerated by the pod. taints: - key: example.com/special-taint effect: NoSchedule # Provisioned nodes will have these taints, but pods do not need to tolerate these taints to be provisioned by this # NodePool. These taints are expected to be temporary and some other entity (e.g. a DaemonSet) is responsible for # removing the taint after it has finished initializing the node. startupTaints: - key: example.com/another-taint effect: NoSchedule # The amount of time a Node can live on the cluster before being removed # Avoiding long-running Nodes helps to reduce security vulnerabilities as well as to reduce the chance of issues that can plague Nodes with long uptimes such as file fragmentation or memory leaks from system processes # You can choose to disable expiration entirely by setting the string value 'Never' here # Note: changing this value in the nodepool will drift the nodeclaims. expireAfter: 720h | Never # The amount of time that a node can be draining before it's forcibly deleted. A node begins draining when a delete call is made against it, starting # its finalization flow. Pods with TerminationGracePeriodSeconds will be deleted preemptively before this terminationGracePeriod ends to give as much time to cleanup as possible. # If your pod's terminationGracePeriodSeconds is larger than this terminationGracePeriod, Karpenter may forcibly delete the pod # before it has its full terminationGracePeriod to cleanup. # Note: changing this value in the nodepool will drift the nodeclaims. terminationGracePeriod: 48h ``` -------------------------------- ### Configure Architecture-Specific Provisioners Source: https://github.com/aws/karpenter-provider-aws/blob/main/designs/provisioner-priority.md Demonstrates how to set up a default provisioner for amd64 architecture and a fallback provisioner for ARM-64 using provisioner weights. This ensures specific architecture requirements are met without requiring manual nodeSelectors on every workload. ```yaml apiVersion: karpenter.sh/v1alpha5 kind: Provisioner metadata: name: default spec: weight: 100 requirements: - key: kubernetes.io/arch operator: In values: ["amd64"] ``` ```yaml apiVersion: karpenter.sh/v1alpha5 kind: Provisioner metadata: name: arm64 spec: weight: 50 requirements: - key: kubernetes.io/arch operator: Exists ``` -------------------------------- ### Build and Execute Kubereplay Commands Source: https://github.com/aws/karpenter-provider-aws/blob/main/tools/kubereplay/README.md Compile the binary and use the CLI to capture, replay, or generate synthetic test data. Requires EKS audit logging, AWS credentials, and kubectl access. ```bash go build -o kubereplay ./cmd kubereplay capture # capture last hour to replay.json kubereplay replay # replay from replay.json kubereplay replay --speed 24 # replay 24x faster kubereplay demo # generate synthetic test data ``` -------------------------------- ### AL2023 NodeConfig Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md This is an example of a NodeConfig passed into Karpenter for AL2023. It specifies Kubelet configuration. ```yaml apiVersion: node.eks.aws/v1alpha1 kind: NodeConfig spec: kubelet: config: maxPods: 42 ``` -------------------------------- ### Automate Node Provisioning with Karpenter Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/getting-started/getting-started-with-karpenter/_index.md Execute this script to demonstrate Karpenter's automatic node provisioning capabilities by scaling up a deployment. ```bash kubectl apply -f ./content/en/{VERSION}/getting-started/getting-started-with-karpenter/examples/default/workload.yaml ``` -------------------------------- ### EC2NodeClass Configuration Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md This example demonstrates a comprehensive EC2NodeClass configuration, including Kubelet settings, AMI and subnet/security group selection, IAM roles, and optional parameters like Capacity Reservation and Placement Group selectors. It's crucial to define at least one of `role` or `instanceProfile` for Karpenter to launch nodes. ```yaml apiVersion: karpenter.sh/v1 kind: EC2NodeClass metadata: name: default spec: kubelet: podsPerCore: 2 maxPods: 20 systemReserved: cpu: 100m memory: 100Mi ephemeral-storage: 1Gi kubeReserved: cpu: 200m memory: 100Mi ephemeral-storage: 3Gi evictionHard: memory.available: 5% nodefs.available: 10% nodefs.inodesFree: 10% evictionSoft: memory.available: 500Mi nodefs.available: 15% nodefs.inodesFree: 15% evictionSoftGracePeriod: memory.available: 1m nodefs.available: 1m30s nodefs.inodesFree: 2m evictionMaxPodGracePeriod: 60 imageGCHighThresholdPercent: 85 imageGCLowThresholdPercent: 80 cpuCFSQuota: true clusterDNS: ["10.0.1.100"] # Optional, dictates UserData generation and default block device mappings. # May be ommited when using an `alias` amiSelectorTerm, otherwise required. amiFamily: AL2 # Required, discovers subnets to attach to instances # Each term in the array of subnetSelectorTerms is ORed together # Within a single term, all conditions are ANDed subnetSelectorTerms: # Select on any subnet that has the "karpenter.sh/discovery: ${CLUSTER_NAME}" # AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921" - tags: karpenter.sh/discovery: "${CLUSTER_NAME}" environment: test - id: subnet-09fa4a0a8f233a921 # Required, discovers security groups to attach to instances # Each term in the array of securityGroupSelectorTerms is ORed together # Within a single term, all conditions are ANDed securityGroupSelectorTerms: # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag # AND the "environment: test" tag OR any security group with the "my-security-group" name # OR any security group with ID "sg-063d7acfb4b06c82c" - tags: karpenter.sh/discovery: "${CLUSTER_NAME}" environment: test - name: my-security-group - id: sg-063d7acfb4b06c82c # Optional, IAM role to use for the node identity. # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes role: "KarpenterNodeRole-${CLUSTER_NAME}" # Optional, IAM instance profile to use for the node identity. # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes instanceProfile: "KarpenterNodeInstanceProfile-${CLUSTER_NAME}" # Each term in the array of amiSelectorTerms is ORed together # Within a single term, all conditions are ANDed amiSelectorTerms: # Select on any AMI that has both the `karpenter.sh/discovery: ${CLUSTER_NAME}` # AND `environment: test` tags OR any AMI with the name `my-ami` OR an AMI with # ID `ami-123` OR an AMI with ID matching the value of my-custom-parameter - tags: karpenter.sh/discovery: "${CLUSTER_NAME}" environment: test - name: my-ami - id: ami-123 - ssmParameter: my-custom-parameter # ssm parameter name or ARN # Select EKS optimized AL2023 AMIs with version `v20240703`. This term is mutually # exclusive and can't be specified with other terms. # - alias: al2023@v20240703 # Optional, each term in the array of capacityReservationSelectorTerms is ORed together. capacityReservationSelectorTerms: - tags: karpenter.sh/discovery: ${CLUSTER_NAME} - id: cr-123 - instanceMatchCriteria: open # Optional, the terms are exclusive placementGroupSelector: name: my-pg id: pg-123 # Optional, propagates tags to underlying EC2 resources tags: team: team-a app: team-a-app # Optional, configures IMDS for the instance metadataOptions: httpEndpoint: enabled httpProtocolIPv6: disabled httpPutResponseHopLimit: 1 # This is changed to disable IMDS access from containers not on the host network httpTokens: required # Optional, configures ENI Connection Tracking timeouts. # Omit entirely to use EC2 defaults. Only set fields you want to override. connectionTracking: # Optional, configures timeout (in seconds) for idle TCP connections. tcpEstablishedTimeout: 300 # lower than the EC2 default of 432000 (5 days) # Optional, configures timeout (in seconds) for idle UDP stream flows. udpStreamTimeout: 120 # Optional, configures timeout (in seconds) for idle unidirectional or single-transaction UDP flows. udpTimeout: 45 ``` -------------------------------- ### AL2023 Bash UserData Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md This is an example of a bash script passed into Karpenter for AL2023. It simply echoes a message. ```shell #!/bin/bash echo "Hello, AL2023!" ``` -------------------------------- ### Karpenter Provisioner Configuration Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/designs/provisioner-priority.md This YAML defines two Karpenter provisioners: 'expensive' which prioritizes 'p3.16xlarge' instances with a high weight, and 'inexpensive' which targets 't3.small' instances. This configuration demonstrates how weights can influence scheduling decisions and potentially lead to higher costs if not managed carefully. ```yaml apiVersion: karpenter.sh/v1alpha5 kind: Provisioner metadata: name: expensive spec: requirements: - key: "node.kubernetes.io/instance-type" operator: In values: ["p3.16xlarge"] weight: 100 ``` ```yaml apiVersion: karpenter.sh/v1alpha5 kind: Provisioner metadata: name: inexpensive spec: requirements: - key: "node.kubernetes.io/instance-type" operator: In values: ["t3.small"] ``` -------------------------------- ### Configure startupTaints for Cilium Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/faq.md Use this example to configure startupTaints in your NodePool to make Karpenter aware of temporary taints, preventing pods from scheduling on nodes that are not yet ready. This is particularly useful for workloads like Cilium. ```yaml apiVersion: karpenter.sh/v1 kind: NodePool ... spec: template: spec: startupTaints: - key: node.cilium.io/agent-not-ready effect: NoSchedule ``` -------------------------------- ### Ubuntu NodeClass Example with AL2 AMIFamily Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md Example of an EC2NodeClass configuration for Ubuntu using the AL2 AMIFamily. This is a workaround due to dropped Ubuntu support, requiring manual AMI selection and custom block device mappings. ```yaml apiVersion: karpenter.k8s.aws/v1 kind: EC2NodeClass spec: amiFamily: AL2 amiSelectorTerms: - id: ami-placeholder blockDeviceMappings: - deviceName: '/dev/sda1' rootVolume: true ebs: encrypted: true volumeType: gp3 volumeSize: 20Gi ``` -------------------------------- ### Install Karpenter Helm Chart Source: https://github.com/aws/karpenter-provider-aws/blob/main/charts/karpenter/README.md Installs or upgrades the Karpenter Helm chart. Ensure the KARPENTER_IAM_ROLE_ARN and CLUSTER_NAME environment variables are set. ```bash helm upgrade --install --namespace karpenter --create-namespace \ karpenter oci://public.ecr.aws/karpenter/karpenter \ --version 1.13.0 \ --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=${KARPENTER_IAM_ROLE_ARN}" \ --set settings.clusterName=${CLUSTER_NAME} \ --set settings.interruptionQueue=${CLUSTER_NAME} \ --wait ``` -------------------------------- ### NodePool with MinValues for Spot Instances Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodepools.md This example demonstrates how to configure a NodePool to use spot instances and enforce minimum values for various instance attributes like category, family, and type. It ensures at least 2 instance categories, 5 instance families, and 10 instance types are available. ```yaml spec: template: spec: requirements: - key: kubernetes.io/arch operator: In values: ["amd64"] - key: kubernetes.io/os operator: In values: ["linux"] - key: karpenter.sh/capacity-type operator: In values: ["spot"] - key: karpenter.k8s.aws/instance-category operator: In values: ["c", "m", "r"] minValues: 2 - key: karpenter.k8s.aws/instance-family operator: Exists minValues: 5 - key: node.kubernetes.io/instance-type operator: Exists minValues: 10 - key: karpenter.k8s.aws/instance-generation operator: Gte values: ["3"] ``` -------------------------------- ### AL2023 NodeConfig Example Source: https://github.com/aws/karpenter-provider-aws/blob/main/website/content/en/docs/concepts/nodeclasses.md Example of the NodeConfig generated for the AL2023 AMIFamily. This configuration includes cluster details and Kubelet settings for the node. ```yaml MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="//" --// Content-Type: application/node.eks.aws # Karpenter Generated NodeConfig apiVersion: node.eks.aws/v1alpha1 kind: NodeConfig spec: cluster: name: test-cluster apiServerEndpoint: https://example.com certificateAuthority: ca-bundle cidr: 10.100.0.0/16 kubelet: config: maxPods: 110 flags: - --node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --//-- ```