Amazon EKS Blueprints Addons

https://github.com/aws-ia/terraform-aws-eks-blueprints-addons
Admin
A Terraform module to deploy and manage a comprehensive collection of Kubernetes addons on Amazon...
Tokens:43,858
Snippets:313
Trust Score:8.6
License:Apache-2.0
Update:1 month ago
Show doc for...
Tokens:
Raw
### Example eks_addons configuration with native and third-party add-ons

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/amazon-eks-addons.md

Shows a complete example of the `eks_addons` map, including native EKS add-ons (`aws-ebs-csi-driver`, `coredns`, `vpc-cni`, `kube-proxy`) and third-party add-ons from AWS Marketplace (`kubecost_kubecost`, `teleport_teleport`). Demonstrates setting `most_recent`, `service_account_role_arn`, and `timeouts`.

```hcl
module "eks_blueprints_addons" {
  source = "aws-ia/eks-blueprints-addons/aws"

  # ... truncated for brevity

  eks_addons = {
    # Amazon EKS add-ons
    aws-ebs-csi-driver = {
      most_recent              = true
      service_account_role_arn = module.ebs_csi_driver_irsa.iam_role_arn
    }

    coredns = {
      most_recent = true

      timeouts = {
        create = "25m"
        delete = "10m"
      }
    }

    vpc-cni = {
      most_recent              = true
      service_account_role_arn = module.vpc_cni_irsa.iam_role_arn
    }

    kube-proxy = {}

    # Third party add-ons via AWS Marketplace
    kubecost_kubecost = {
      most_recent = true
    }

    teleport_teleport = {
      most_recent = true
    }
  }
}
```

--------------------------------

### Initialize and apply Terraform configuration

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/tests/complete/README.md

Standard Terraform workflow to initialize, plan, and apply the example configuration. Running this example may create resources that incur monetary charges on your AWS bill.

```bash
terraform init
terraform plan
terraform apply
```

--------------------------------

### Deploy Example GatewayClass

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-gateway-api-controller.md

Deploys an example GatewayClass from the AWS application networking repository, resulting in the creation of the amazon-vpc-lattice GatewayClass.

```sh
$ kubectl apply -f https://raw.githubusercontent.com/aws/aws-application-networking-k8s/main/examples/gatewayclass.yaml
gatewayclass.gateway.networking.k8s.io/amazon-vpc-lattice created
```

--------------------------------

### Get baseline of nodes and CoreDNS pods

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/cluster-proportional-autoscaler.md

Run these commands to capture the current number of nodes and CoreDNS pods before scaling. The output shows 4 nodes and 2 CoreDNS pods.

```bash
kubectl get nodes
NAME                          STATUS   ROLES    AGE   VERSION
ip-10-0-19-243.ec2.internal   Ready    <none>   21h   v1.26.4-eks-0a21954
ip-10-0-25-182.ec2.internal   Ready    <none>   21h   v1.26.4-eks-0a21954
ip-10-0-40-138.ec2.internal   Ready    <none>   21h   v1.26.4-eks-0a21954
ip-10-0-8-136.ec2.internal    Ready    <none>   21h   v1.26.4-eks-0a21954

kubectl get po -n kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-7975d6fb9b-dlkdd   1/1     Running   0          21h
coredns-7975d6fb9b-xqqwp   1/1     Running   0          21h
```

--------------------------------

### List current nodes (Fargate only)

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/karpenter.md

Lists the current nodes. In this example, only Fargate nodes are present because the Karpenter controller is hosted on EKS Fargate.

```sh
kubectl get nodes

# Output should look similar to below
NAME                                                STATUS   ROLES    AGE     VERSION
fargate-ip-10-0-29-25.us-west-2.compute.internal    Ready    <none>   2m56s   v1.26.3-eks-f4dc2c0
fargate-ip-10-0-36-148.us-west-2.compute.internal   Ready    <none>   2m57s   v1.26.3-eks-f4dc2c0
fargate-ip-10-0-42-30.us-west-2.compute.internal    Ready    <none>   2m34s   v1.26.3-eks-f4dc2c0
fargate-ip-10-0-45-112.us-west-2.compute.internal   Ready    <none>   2m33s   v1.26.3-eks-f4dc2c0
```

--------------------------------

### Verify Argo CD pods

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/argocd.md

Command to check that all Argo CD pods are running, with example output.

```sh
$ kubectl get pods -n argocd
NAME                                                        READY   STATUS    RESTARTS   AGE
argo-cd-argocd-application-controller-0                     1/1     Running   0          146m
argo-cd-argocd-applicationset-controller-678d85f77b-rmpcb   1/1     Running   0          146m
argo-cd-argocd-dex-server-7b6c9b5969-zpqnl                  1/1     Running   0          146m
argo-cd-argocd-notifications-controller-6d489b99c9-j6fdw    1/1     Running   0          146m
argo-cd-argocd-redis-59dd95f5b5-8fx74                       1/1     Running   0          146m
argo-cd-argocd-repo-server-7b9bd88c95-mh2fz                 1/1     Running   0          146m
argo-cd-argocd-server-6f9cfdd4d5-8mfpc                      1/1     Running   0          146m
```

--------------------------------

### Verify external-dns pods

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/external-dns.md

Run kubectl get pods to check that external-dns pods are running.

```sh
$ kubectl get pods -n external-dns
NAME                            READY   STATUS    RESTARTS     AGE
external-dns-849b89c675-ffnf6   1/1     Running   1 (2d ago)   2d5h
```

--------------------------------

### Deploy EKS Blueprints Addons module

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/README.md

Example Terraform configuration for deploying the EKS Blueprints Addons module, including EKS managed addons and optional addons. The module references the EKS cluster outputs and requires an OIDC provider ARN. The eks module is shown truncated for brevity.

```hcl
module "eks_blueprints_addons" {
  source = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0" #ensure to update this to the latest/desired version

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  eks_addons = {
    aws-ebs-csi-driver = {
      most_recent = true
    }
    coredns = {
      most_recent = true
    }
    vpc-cni = {
      most_recent = true
    }
    kube-proxy = {
      most_recent = true
    }
  }

  enable_aws_load_balancer_controller    = true
  enable_cluster_proportional_autoscaler = true
  enable_karpenter                       = true
  enable_kube_prometheus_stack           = true
  enable_metrics_server                  = true
  enable_external_dns                    = true
  enable_cert_manager                    = true
  cert_manager_route53_hosted_zone_arns  = ["arn:aws:route53:::hostedzone/XXXXXXXXXXXXX"]

  tags = {
    Environment = "dev"
  }
}

module "eks" {
  source = "terraform-aws-modules/eks/aws"

  cluster_name    = "my-cluster"
  cluster_version = "1.29"

  ... truncated for brevity
}
```

--------------------------------

### Check Certificate and Secret Status

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-private-ca-issuer.md

Checks the certificate status and the associated secret. The certificate should be in Ready state and point to a secret in the same namespace. The output shows a certificate named example with a secret example-clusterissuer.

```sh
kubectl get certificate -o wide
NAME      READY   SECRET                  ISSUER                    STATUS                                          AGE
example   True    example-clusterissuer   tls-with-aws-pca-issuer   Certificate is up to date and has not expired   41m

kubectl get secret example-clusterissuer
NAME                    TYPE                DATA   AGE
example-clusterissuer   kubernetes.io/tls   3      43m

```

--------------------------------

### Get Velero backup locations

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/velero.md

Gets the backup storage locations configured for Velero. The output shows the default location with provider aws, bucket/prefix, phase Available, and access mode ReadWrite.

```bash
velero backup-location get

# Output should look similar to below
NAME      PROVIDER   BUCKET/PREFIX                                 PHASE       LAST VALIDATED                  ACCESS MODE   DEFAULT
default   aws        stateful-20230503175301619800000005/backups   Available   2023-05-04 15:15:00 -0400 EDT   ReadWrite     true
```

--------------------------------

### Verify cluster-autoscaler pods

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/cluster-autoscaler.md

Run `kubectl get pods -n kube-system` to confirm the Cluster Autoscaler pod is running. The example output shows a pod named `cluster-autoscaler-aws-cluster-autoscaler-7ff79bc484-pm8g9` with status `Running`.

```sh
$ kubectl get pods -n kube-system
NAME                                                         READY   STATUS    RESTARTS     AGE
cluster-autoscaler-aws-cluster-autoscaler-7ff79bc484-pm8g9   1/1     Running   1 (2d ago)   2d5h
```

--------------------------------

### Get Ingress to verify ALB address

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-load-balancer-controller.md

Displays the created Ingress, showing the ALB DNS address in the ADDRESS column. This confirms the Ingress was successfully created with the alb class.

```sh
kubectl get ingress
NAME                CLASS   HOSTS   ADDRESS                                                                 PORTS   AGE
example-ingress     alb     *       k8s-example-ingress-7e0d6f03e7-1234567890.us-west-2.elb.amazonaws.com   80      4m9s
```

--------------------------------

### Terraform module configuration with helm_releases variable

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/helm-release.md

Example of using the helm_releases variable in the EKS Blueprints Add-ons module to create Helm releases for add-ons like prometheus-adapter and gpu-operator. The module iterates over the map and passes each configuration to the helm_release resource.

```hcl
module "addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0"

  cluster_name      = "<cluster_name>"
  cluster_endpoint  = "<cluster_endpoint>"
  cluster_version   = "<cluster_version>"
  oidc_provider_arn = "<oidc_provider_arn>"

  # EKS add-ons
  eks_addons = {
    coredns = {}
    vpc-cni = {}
    kube-proxy = {}
  }

  # Blueprints add-ons
  enable_aws_efs_csi_driver                    = true
  enable_aws_cloudwatch_metrics                = true
  enable_cert_manager                          = true
  ...

  # Pass in any number of Helm charts to be created for those that are not natively supported
  helm_releases = {
    prometheus-adapter = {
      description      = "A Helm chart for k8s prometheus adapter"
      namespace        = "prometheus-adapter"
      create_namespace = true
      chart            = "prometheus-adapter"
      chart_version    = "4.2.0"
      repository       = "https://prometheus-community.github.io/helm-charts"
      values = [
        <<-EOT
          replicas: 2
          podDisruptionBudget:
            enabled: true
        EOT
      ]
    }
    gpu-operator = {
      description      = "A Helm chart for NVIDIA GPU operator"
      namespace        = "gpu-operator"
      create_namespace = true
      chart            = "gpu-operator"
      chart_version    = "v23.3.2"
      repository       = "https://nvidia.github.io/gpu-operator"
      values = [
        <<-EOT
          operator:
            defaultRuntime: containerd
        EOT
      ]
    }
  }

  tags = local.tags
}
```

--------------------------------

### Deploy AWS Private CA Issuer with EKS Blueprints Addons

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-private-ca-issuer.md

Deploys the AWS Private CA Issuer add-on via EKS Blueprints Addons. Requires cert-manager to be installed first. The acmca_arn parameter must reference an existing AWS Private CA ARN.

```hcl
module "eks_blueprints_addons" {

  enable_cert_manager         = true
  enable_aws_privateca_issuer = true
  aws_privateca_issuer = {
    acmca_arn        = aws_acmpca_certificate_authority.this.arn
  }
}

```

--------------------------------

### Create sample pause deployment

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/karpenter.md

Creates a sample 'pause' deployment named 'inflate' with zero replicas to demonstrate Karpenter scaling. The deployment requests 1 CPU per pod.

```sh
kubectl apply -f - <<EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 0
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      terminationGracePeriodSeconds: 0
      containers:
        - name: inflate
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
          resources:
            requests:
              cpu: 1
EOF
```

--------------------------------

### Create namespace and run nginx for backup demo

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/velero.md

Creates a new namespace named backupdemo and runs an nginx pod in it. This is used to demonstrate backup and restore.

```bash
kubectl create namespace backupdemo
kubectl run nginx --image=nginx -n backupdemo
```

--------------------------------

### List Velero resources in namespace

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/velero.md

Lists all resources in the velero namespace to verify the Velero deployment. The expected output shows a running pod, a ClusterIP service, a deployment, and a replicaset.

```bash
kubectl get all -n velero

# Output should look similar to below
NAME                         READY   STATUS    RESTARTS   AGE
pod/velero-7b8994d56-z89sl   1/1     Running   0          25h

NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/velero   ClusterIP   172.20.20.118   <none>        8085/TCP   25h

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/velero   1/1     1            1           25h

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/velero-7b8994d56   1         1         1       25h
```

--------------------------------

### Verify BRUPOP resources and APIService

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/bottlerocket.md

Lists all resources in the brupop-bottlerocket-aws namespace and describes the APIService v2.brupop.bottlerocket.aws to verify that the BRUPOP components are running and the API service is available.

```bash
$ kubectl -n brupop-bottlerocket-aws get all

NAME                                                READY   STATUS    RESTARTS      AGE
pod/brupop-agent-5nv6m                              1/1     Running   1 (33h ago)   33h
pod/brupop-agent-h4vw9                              1/1     Running   1 (33h ago)   33h
pod/brupop-agent-sr9ms                              1/1     Running   2 (33h ago)   33h
pod/brupop-apiserver-6ccb74f599-4c9lv               1/1     Running   0             33h
pod/brupop-apiserver-6ccb74f599-h6hg8               1/1     Running   0             33h
pod/brupop-apiserver-6ccb74f599-svw8n               1/1     Running   0             33h
pod/brupop-controller-deployment-58d46595cc-7vxnt   1/1     Running   0             33h

NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/brupop-apiserver           ClusterIP   172.20.153.72   <none>        443/TCP   33h
service/brupop-controller-server   ClusterIP   172.20.7.127    <none>        80/TCP    33h

NAME                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/brupop-agent   3         3         3       3            3           <none>          33h

NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/brupop-apiserver               3/3     3            3           33h
deployment.apps/brupop-controller-deployment   1/1     1            1           33h

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/brupop-apiserver-6ccb74f599               3         3         3       33h
replicaset.apps/brupop-controller-deployment-58d46595cc   1         1         1       33h

$ kubectl describe apiservices.apiregistration.k8s.io v2.brupop.bottlerocket.aws
Name:         v2.brupop.bottlerocket.aws
Namespace:
Labels:       kube-aggregator.kubernetes.io/automanaged=true
Annotations:  <none>
API Version:  apiregistration.k8s.io/v1
Kind:         APIService
Metadata:
  Creation Timestamp:  2024-01-30T16:27:15Z
  Resource Version:    8798
  UID:                 034abe22-7e5f-4040-9b64-8ca9d55a4af6
Spec:
  Group:                   brupop.bottlerocket.aws
  Group Priority Minimum:  1000
  Version:                 v2
  Version Priority:        100
Status:
  Conditions:
    Last Transition Time:  2024-01-30T16:27:15Z
    Message:               Local APIServices are always available
    Reason:                Local
    Status:                True
    Type:                  Available
Events:                    <none>
```

--------------------------------

### Create Velero backup for namespace

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/velero.md

Creates a Velero backup named backup1 that includes the backupdemo namespace. The output confirms the backup request was submitted successfully.

```bash
velero backup create backup1 --include-namespaces backupdemo

# Output should look similar to below
Backup request "backup1" submitted successfully.
Run `velero backup describe backup1` or `velero backup logs backup1` for more details.
```

--------------------------------

### Enable Metrics Server with default settings

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/metrics-server.md

Enables the Metrics Server add-on with default settings.

```hcl
enable_metrics_server = true
```

--------------------------------

### Enable External DNS add-on

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/external-dns.md

Set enable_external_dns to true to deploy ExternalDNS.

```hcl
enable_external_dns = true
```

--------------------------------

### Verify metrics-server pods are running

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/metrics-server.md

Command to verify the metrics-server pods are running in the kube-system namespace. Shows a sample pod with status Running.

```sh
$ kubectl get pods -n kube-system
NAME                                   READY   STATUS    RESTARTS       AGE
metrics-server-6f9cdd486c-njh8b        1/1     Running   1 (2d2h ago)   2d7h
```

--------------------------------

### Enable Argo CD add-on

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/argocd.md

Set enable_argocd to true to deploy Argo CD.

```hcl
enable_argocd = true
```

--------------------------------

### Verify PVC and PV binding

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-fsx-csi-driver.md

Check that the PVC is bound and the PV is created.

```sh
$ kubectl get pvc
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
fsx-claim   Bound    pvc-df385730-72d6-4b0c-8275-cc055a438760   1200Gi     RWX            fsx-sc         7m47s
$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
pvc-df385730-72d6-4b0c-8275-cc055a438760   1200Gi     RWX            Delete           Bound    default/fsx-claim   fsx-sc                  2m13s
```

--------------------------------

### Describe FSx StorageClass

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-fsx-csi-driver.md

Verify the StorageClass configuration with kubectl describe. Note the output shows the applied parameters and annotations.

```sh
$ kubect describe storageclass fsx-sc
Name:            fsx-sc
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"fsx-sc"},"mountOptions":null,"parameters":{"deploymentType":"PERSISTENT_1","perUnitStorageThroughput":"200","securityGroupIds":"sg-q1w2e3r4t5y6u7i8o","subnetId":"subnet-q1w2e3r4t5y6u7i8o"},"provisioner":"fsx.csi.aws.com"}

Provisioner:           fsx.csi.aws.com
Parameters:            deploymentType=PERSISTENT_1,perUnitStorageThroughput=200,securityGroupIds=sg-q1w2e3r4t5y6u7i8o,subnetId=subnet-q1w2e3r4t5y6u7i8o
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>
```

--------------------------------

### Verify Provider Pods Running

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/secrets-store-csi-driver-provider-aws.md

Verifies that the secrets-store-csi-driver and provider-aws pods are running.

```sh
$ kubectl get pods -n kube-system
NAME                                         READY   STATUS    RESTARTS       AGE
secrets-store-csi-driver-9l2z8               3/3     Running   1 (2d5h ago)   2d9h
secrets-store-csi-driver-provider-aws-2qqkk  1/1     Running   1 (2d5h ago)   2d9h
```

--------------------------------

### Enable VPA with default Helm chart config

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/vertical-pod-autoscaler.md

Deploys the Vertical Pod Autoscaler with default Helm chart settings. Requires the Metrics Server add-on to be enabled.

```hcl
  enable_vpa            = true
  enable_metrics_server = true

```

--------------------------------

### Create Kubernetes Ingress with alb class

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-load-balancer-controller.md

Creates a Kubernetes Ingress using the alb IngressClass, pointing to an existing Service named example-svc. The annotations set the scheme to internet-facing and the target type to ip.

```sh
kubectl create ingress example-ingress --class alb --rule="/*=example-svc:80" \
--annotation alb.ingress.kubernetes.io/scheme=internet-facing \
--annotation alb.ingress.kubernetes.io/target-type=ip
```

--------------------------------

### Scale inflate deployment to 5 replicas

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/karpenter.md

Scales the 'inflate' deployment to 5 replicas to trigger Karpenter to provision new nodes. The commented command shows how to view Karpenter controller logs.

```sh
kubectl scale deployment inflate --replicas 5
# To view logs
# kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller
```

--------------------------------

### Verify restored namespace

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/velero.md

Verifies that the backupdemo namespace is restored by listing all resources in it. The expected output shows the nginx pod running.

```bash
kubectl get all -n backupdemo

# Output should look similar to below
NAME        READY   STATUS    RESTARTS   AGE
pod/nginx   1/1     Running   0          21s
```

--------------------------------

### Verify SQS Queue Creation

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-node-termination-handler.md

Verifies that the SQS Queue is created by listing queues.

```sh
$ aws sqs list-queues

{
    "QueueUrls": [
        "https://sqs.us-east-1.amazonaws.com/XXXXXXXXXXXXXX/aws_node_termination_handler20221123072051157700000004"
    ]
}
```

--------------------------------

### Verify Fluent Bit pods are running

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-for-fluentbit.md

Verifies that the aws-for-fluent-bit pods are running in the kube-system namespace.

```sh
$ kubectl -n kube-system get pods -l app.kubernetes.io/name=aws-for-fluent-bit
NAME                       READY   STATUS    RESTARTS   AGE
aws-for-fluent-bit-6lhkj   1/1     Running   0          15m
aws-for-fluent-bit-sbn9b   1/1     Running   0          15m
aws-for-fluent-bit-svhwq   1/1     Running   0          15m
```

--------------------------------

### Restore from Velero backup

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/velero.md

Restores the namespace from the backup1 backup. This command creates a restore from the specified backup.

```bash
velero restore create --from-backup backup1
```

--------------------------------

### List Karpenter pods

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/karpenter.md

Lists the Karpenter pods in the karpenter namespace. The output shows two running pods, confirming the controller is deployed.

```sh
kubectl get pods -n karpenter

# Output should look similar to below
NAME                         READY   STATUS    RESTARTS   AGE
karpenter-6f97df4f77-5nqsk   1/1     Running   0          3m28s
karpenter-6f97df4f77-n7fkf   1/1     Running   0          3m28s
```

--------------------------------

### Enable Fargate FluentBit add-on

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/fargate-fluentbit.md

Enables the Fargate FluentBit add-on and sets the flb_log_cw option to true to route logs to CloudWatch.

```hcl
module "eks_blueprints_addons" {

  enable_fargate_fluentbit = true
  fargate_fluentbit = {
    flb_log_cw = true
  }
}

```

--------------------------------

### Verify Argo Rollouts pods

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/argo-rollouts.md

Verifies that the Argo Rollouts pods are running in the argo-rollouts namespace.

```sh
$ kubectl get pods -n argo-rollouts
NAME                             READY   STATUS    RESTARTS   AGE
argo-rollouts-5db5688849-x89zb   0/1     Running   0          11s
```

--------------------------------

### Create a Certificate resource

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/cert-manager.md

Creates a Certificate resource in the default namespace that requests a certificate from the selfsigned-cluster-issuer. The certificate is configured as a CA with an ECDSA private key of size 256 and stores the resulting secret as example-secret.

```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example
  namespace: default
spec:
  isCA: true
  commonName: example
  secretName: example-secret
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-cluster-issuer
    kind: ClusterIssuer
    group: cert-manager.io
```

--------------------------------

### Create FSx StorageClass

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-fsx-csi-driver.md

Create a StorageClass named fsx-sc using the FSx CSI provisioner. Replace the subnetId and securityGroupIds with your own values.

```sh
$ cat <<EOF | kubectl apply -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fsx-sc
provisioner: fsx.csi.aws.com
parameters:
  subnetId: <YOUR_SUBNET_IDs>
  securityGroupIds: <YOUR_SG_ID>
  perUnitStorageThroughput: "200"
  deploymentType: PERSISTENT_1
mountOptions:
  - flock
EOF
```

--------------------------------

### Configure CoreDNS EKS add-on with affinity and resources

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/amazon-eks-addons.md

Configures the CoreDNS EKS add-on with replica count, tolerations, topology spread constraints, affinity rules, and resource limits. The configuration_values are passed as JSON via jsonencode. Note the source includes a comment indicating truncation for brevity.

```hcl
module "eks_blueprints_addons" {
  source = "aws-ia/eks-blueprints-addons/aws"

  # ... truncated for brevity

  eks_addons = {
     coredns = {
      most_recent = true
      configuration_values = jsonencode({
        replicaCount = 4
        tolerations = [
        {
          key      = "dedicated",
          operator = "Equal",
          effect   = "NoSchedule",
          value    = "orchestration-seb"
        }
        ]

        topologySpreadConstraints = [
          {
            maxSkew = 1
            topologyKey = "topology.kubernetes.io/zone"
            whenUnsatisfiable = "ScheduleAnyway"
            labelSelector = {
              matchLabels = {
                k8s-app: "kube-dns"
              }
            }
          }
        ]

        affinity = {
          nodeAffinity = {
            requiredDuringSchedulingIgnoredDuringExecution = {
              nodeSelectorTerms = [
              {
                matchExpressions = [
                  {
                    key = "kubernetes.io/os"
                    operator = "In"
                    values = ["linux"]
                  },
                  {
                    key = "kubernetes.io/arch"
                    operator = "In"
                    values = ["amd64"]
                  }
                ]
              }]
            }
          }

          podAffinity = {
            requiredDuringSchedulingIgnoredDuringExecution = [{
                labelSelector = {
                  matchExpressions = [
                    {
                      key = "k8s-app"
                      operator = "NotIn"
                      values = ["kube-dns"]
                    }
                  ]
                }
                topologyKey = "kubernetes.io/hostname"
            }
            ]
          }

          podAntiAffinity = {
            preferredDuringSchedulingIgnoredDuringExecution = [{
              podAffinityTerm = {
                labelSelector = {
                  matchExpressions = [
                    {
                      key = "k8s-app"
                      operator = "In"
                      values = ["kube-dns"]
                    }
                  ]
                }
                topologyKey = "kubernetes.io/hostname"
              }
              weight = 100
              }
            ]

            requiredDuringSchedulingIgnoredDuringExecution = [{
                labelSelector = {
                  matchExpressions = [
                    {
                      key = "k8s-app"
                      operator = "In"
                      values = ["kube-dns"]
                    }
                  ]
                }
                topologyKey = "kubernetes.io/hostname"
              }
            ]
          }

        }

        resources = {
          limits = {
            cpu    = "100m"
            memory = "150Mi"
          }
          requests = {
            cpu    = "100m"
            memory = "150Mi"
        }
      })
    }

```

--------------------------------

### Enable Argo Events add-on

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/argo-events.md

Set enable_argo_events to true to deploy Argo Events.

```hcl
enable_argo_events = true
```

--------------------------------

### AWS CLI command to view add-on configuration schema

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/amazon-eks-addons.md

Uses `aws eks describe-addon-configuration` to retrieve the JSON schema for a specific add-on version. The command pipes the output through `jq` to format the JSON schema.

```sh
aws eks describe-addon-configuration \
--addon-name coredns \
--addon-version v1.10.1-eksbuild.2 \
--query 'configurationSchema' \
--output text | jq
```

--------------------------------

### Enable kubelet monitoring with Fluent Bit

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-for-fluentbit.md

Enables the ClusterInsights Use_Kubelet feature by setting kubelet_monitoring to true and providing additional parameters for auto-creating log groups, host network, and DNS policy.

```hcl
  enable_aws_for_fluentbit = true
  aws_for_fluentbit = {
    enable_containerinsights = true
    kubelet_monitoring       = true
    set = [{
        name  = "cloudWatchLogs.autoCreateGroup"
        value = true
      },
      {
        name  = "hostNetwork"
        value = true
      },
      {
        name  = "dnsPolicy"
        value = "ClusterFirstWithHostNet"
      }
    ]
  }
```

--------------------------------

### Check cluster-proportional-autoscaler pod

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/cluster-proportional-autoscaler.md

Verifies the cluster-proportional-autoscaler pod is running in the kube-system namespace. The expected output shows a single pod with status Running.

```bash
kubectl -n kube-system get po -l app.kubernetes.io/instance=cluster-proportional-autoscaler
NAME                                                              READY   STATUS    RESTARTS   AGE
cluster-proportional-autoscaler-kube-dns-autoscaler-d8dc8477xx7   1/1     Running   0          21h
```

--------------------------------

### Enable Kube Prometheus Stack

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/kube-prometheus-stack.md

Enables the Kube Prometheus Stack add-on with default settings.

```hcl
enable_kube_prometheus_stack = true
```

--------------------------------

### Enable Ingress Nginx

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/ingress-nginx.md

Enables the Ingress Nginx Controller add-on in the EKS Blueprints configuration.

```hcl
enable_ingress_nginx = true
```

--------------------------------

### List Pods in aws-privateca-issuer and cert-manager Namespaces

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-private-ca-issuer.md

Lists all pods running in the aws-privateca-issuer and cert-manager namespaces to verify the deployment.

```sh
kubectl get pods -n aws-privateca-issuer
kubectl get pods -n cert-manager

```

--------------------------------

### Terraform apply and destroy targeting addons module

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/helm-release.md

Commands to apply and destroy the addons module, tying the lifecycle of all add-ons to the module.

```sh
terraform apply -target=module.addons

terraform destroy -target=module.addons
```

--------------------------------

### Verify CoreDNS scaled up after adding a node

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/cluster-proportional-autoscaler.md

After increasing the managed node group size from 4 to 5, run these commands to confirm the new node is ready and CoreDNS has scaled to 3 pods.

```bash
NAME                          STATUS   ROLES    AGE   VERSION
ip-10-0-14-120.ec2.internal   Ready    <none>   10m   v1.26.4-eks-0a21954
ip-10-0-19-243.ec2.internal   Ready    <none>   21h   v1.26.4-eks-0a21954
ip-10-0-25-182.ec2.internal   Ready    <none>   21h   v1.26.4-eks-0a21954
ip-10-0-40-138.ec2.internal   Ready    <none>   21h   v1.26.4-eks-0a21954
ip-10-0-8-136.ec2.internal    Ready    <none>   21h   v1.26.4-eks-0a21954

kubectl get po -n kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-7975d6fb9b-dlkdd   1/1     Running   0          21h
coredns-7975d6fb9b-ww64t   1/1     Running   0          10m
coredns-7975d6fb9b-xqqwp   1/1     Running   0          21h
```

--------------------------------

### Enable Cluster Autoscaler add-on

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/cluster-autoscaler.md

Set `enable_cluster_autoscaler` to `true` to deploy the Cluster Autoscaler with default settings.

```hcl
enable_cluster_autoscaler = true
```

--------------------------------

### Customize Metrics Server Helm chart configuration

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/metrics-server.md

Enables Metrics Server and customizes the Helm chart deployment with a chart version, repository, namespace, and custom values from a local values.yaml file.

```hcl
  enable_metrics_server = true

  metrics_server = {
    name          = "metrics-server"
    chart_version = "3.10.0"
    repository    = "https://kubernetes-sigs.github.io/metrics-server/"
    namespace     = "kube-system"
    values        = [templatefile("${path.module}/values.yaml", {})]
  }
```

--------------------------------

### Describe Velero backup status

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/velero.md

Describes the backup1 to check its status. The output shows the backup phase as Completed, included namespace backupdemo, and details such as TTL, start/completion times, and item counts.

```bash
velero backup describe backup1

# Output should look similar to below
Name:         backup1
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.26.2-eks-a59e1f0
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=26+

Phase:  Completed


Namespaces:
  Included:  backupdemo
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2023-05-04 15:16:31 -0400 EDT
Completed:  2023-05-04 15:16:33 -0400 EDT

Expiration:  2023-06-03 15:16:31 -0400 EDT

Total items to be backed up:  9
Items backed up:              9

Velero-Native Snapshots: <none included>
```

--------------------------------

### Delete namespace for restore demo

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/velero.md

Deletes the backupdemo namespace. This is done to demonstrate restoring it from the backup.

```bash
kubectl delete namespace backupdemo
```

--------------------------------

### Create Spot service-linked role

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/karpenter.md

Creates the AWS Spot service-linked role required for Karpenter node templates that use spot instances. The `|| true` ensures the command does not fail if the role already exists.

```sh
aws iam create-service-linked-role --aws-service-name spot.amazonaws.com || true
```

--------------------------------

### Verify aws-cloudwatch-metrics pods are running

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-cloudwatch-metrics.md

Verifies that the aws-cloudwatch-metrics pods are running in the amazon-cloudwatch namespace.

```sh
$ kubectl get pods -n amazon-cloudwatch

NAME                           READY   STATUS    RESTARTS   AGE
aws-cloudwatch-metrics-2dt5h   1/1     Running   0          149m
```

--------------------------------

### Enable cert-manager add-on

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/cert-manager.md

Enables the cert-manager add-on in the EKS Blueprints Addons module by setting enable_cert_manager to true.

```hcl
module "eks_blueprints_addons" {

  enable_cert_manager         = true
}
```

--------------------------------

### Enable Bottlerocket Update Operator with default configuration

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/bottlerocket.md

Deploys the Bottlerocket Update Operator (BRUPOP) with default configuration. The `wait = true` for Cert-Manager is required because BRUPOP needs Cert-Manager CRDs already present in the cluster.

```hcl
module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.13"

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  enable_cert_manager = true
  cert_manager = {
    wait = true
  }
  enable_bottlerocket_update_operator = true
}
```

--------------------------------

### Check cluster-proportional-autoscaler config map

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/cluster-proportional-autoscaler.md

Verifies that the cluster-proportional-autoscaler-kube-dns-autoscaler config map exists in the kube-system namespace. The expected output shows the config map with one data entry.

```bash
kubectl -n kube-system get cm cluster-proportional-autoscaler-kube-dns-autoscaler
NAME                                                  DATA   AGE
cluster-proportional-autoscaler-kube-dns-autoscaler   1      21h
```

--------------------------------

### Create FSx PVC

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-fsx-csi-driver.md

Create a PersistentVolumeClaim named fsx-claim using the fsx-sc StorageClass with ReadWriteMany access and 1200Gi storage.

```sh
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: fsx-claim
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: fsx-sc
  resources:
    requests:
      storage: 1200Gi
EOF
```

--------------------------------

### Check aws-logging ConfigMap

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/fargate-fluentbit.md

Validates that the aws-logging ConfigMap for Fargate FluentBit was created in the aws-observability namespace.

```sh
kubectl -n aws-observability get configmap aws-logging -o yaml
apiVersion: v1
data:
  filters.conf: |
    [FILTER]
      Name parser
      Match *
      Key_Name log
      Parser regex
      Preserve_Key True
      Reserve_Data True
  flb_log_cw: "true"
  output.conf: |
    [OUTPUT]
      Name cloudwatch_logs
      Match *
      region us-west-2
      log_group_name /fargate-serverless/fargate-fluentbit-logs20230509014113352200000006
      log_stream_prefix fargate-logs-
      auto_create_group true
  parsers.conf: |
    [PARSER]
      Name regex
      Format regex
      Regex ^(?<time>[^ ]+) (?<stream>[^ ]+) (?<logtag>[^ ]+) (?<message>.+)$
      Time_Key time
      Time_Format %Y-%m-%dT%H:%M:%S.%L%z
      Time_Keep On
      Decode_Field_As json message
immutable: false
kind: ConfigMap
metadata:
  creationTimestamp: "2023-05-08T21:14:52Z"
  name: aws-logging
  namespace: aws-observability
  resourceVersion: "1795"
  uid: d822bcf5-a441-4996-857e-7fb1357bc07e

```

--------------------------------

### Enable AWS for Fluent Bit add-on

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-for-fluentbit.md

Enables the AWS for Fluent Bit add-on in the EKS Blueprints configuration.

```hcl
enable_aws_for_fluentbit = true
```

--------------------------------

### Enable Velero with S3 backup location

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/velero.md

Enables the Velero add-on and specifies the S3 bucket for backups. The velero_backup_s3_bucket variable sets the bucket name, while the s3_backup_location key in the velero map sets the S3 bucket ARN with an optional prefix.

```hcl
enable_velero           = true
velero_backup_s3_bucket = "<YOUR_BUCKET_NAME>"
velero = {
    s3_backup_location = "<YOUR_S3_BUCKET_ARN>[/prefix]"
  }
```

--------------------------------

### Check node version after BRUPOP update

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/bottlerocket.md

Shows the node version after BRUPOP automatically updates it. The default cron schedule checks for updates every minute, so the node version may change shortly after labeling.

```bash
kubectl get nodes
NAME                                        STATUS                     ROLES    AGE   VERSION
ip-10-0-34-87.us-west-2.compute.internal    Ready                      <none>   34h   v1.28.4-eks-d91a302
```

--------------------------------

### Validate cert-manager pods

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/cert-manager.md

Validates that the cert-manager pods are running in the cert-manager namespace. The expected output shows three pods in Running state.

```sh
kubectl -n cert-manager get pods
NAME                                      READY   STATUS    RESTARTS   AGE
cert-manager-5989bcc87-96qvf              1/1     Running   0          2m49s
cert-manager-cainjector-9b44ddb68-8c7b9   1/1     Running   0          2m49s
cert-manager-webhook-776b65456-k6br4      1/1     Running   0          2m49s
```
Tokens:
Raw