### Deploy Example GatewayClass

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-gateway-api-controller.md

Apply an example GatewayClass resource using kubectl to configure the AWS Gateway API Controller. This creates the `amazon-vpc-lattice` GatewayClass.

```sh
$kubectl apply -f https://raw.githubusercontent.com/aws/aws-application-networking-k8s/main/examples/gatewayclass.yaml
gatewayclass.gateway.networking.k8s.io/amazon-vpc-lattice created
```

--------------------------------

### Example EKS Add-on Deployment

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/amazon-eks-addons.md

Demonstrates the configuration of various Amazon EKS add-ons and third-party add-ons via the AWS Marketplace. Includes settings for CSI drivers, CoreDNS, kube-proxy, and marketplace add-ons like Kubecost and Teleport.

```hcl
module "eks_blueprints_addons" {
  source = "aws-ia/eks-blueprints-addons/aws"

  # ... truncated for brevity

  eks_addons = {
    # Amazon EKS add-ons
    aws-ebs-csi-driver = {
      most_recent              = true
      service_account_role_arn = module.ebs_csi_driver_irsa.iam_role_arn
    }

    coredns = {
      most_recent = true

      timeouts = {
        create = "25m"
        delete = "10m"
      }
    }

    vpc-cni = {
      most_recent              = true
      service_account_role_arn = module.vpc_cni_irsa.iam_role_arn
    }

    kube-proxy = {}

    # Third party add-ons via AWS Marketplace
    kubecost_kubecost = {
      most_recent = true
    }

    teleport_teleport = {
      most_recent = true
    }
  }
}
```

--------------------------------

### Provider Configuration for EKS Addons

Source: https://context7.com/aws-ia/terraform-aws-eks-blueprints-addons/llms.txt

Configure AWS, Kubernetes, and Helm providers to authenticate with the EKS cluster. The AWS CLI must be installed locally for token generation.

```hcl
provider "aws" {
  region = "us-west-2"
}
```

```hcl
# Required for public ECR (used by Karpenter and AWS Gateway API Controller)
provider "aws" {
  region = "us-east-1"
  alias  = "virginia"
}
```

```hcl
provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
  }
}
```

```hcl
provider "helm" {
  kubernetes = {
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)

    exec = {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
    }
  }
}
```

--------------------------------

### Get Baseline Node and CoreDNS Pod Count

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/cluster-proportional-autoscaler.md

Before making configuration changes, record the current number of nodes and CoreDNS pods to establish a baseline for comparison.

```bash
kubectl get nodes
NAME                          STATUS   ROLES    AGE   VERSION
ip-10-0-19-243.ec2.internal   Ready    <none>   21h   v1.26.4-eks-0a21954
ip-10-0-25-182.ec2.internal   Ready    <none>   21h   v1.26.4-eks-0a21954
ip-10-0-40-138.ec2.internal   Ready    <none>   21h   v1.26.4-eks-0a21954
ip-10-0-8-136.ec2.internal    Ready    <none>   21h   v1.26.4-eks-0a21954

kubectl get po -n kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-7975d6fb9b-dlkdd   1/1     Running   0          21h
coredns-7975d6fb9b-xqqwp   1/1     Running   0          21h
```

--------------------------------

### CoreDNS Addon Configuration Schema

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/amazon-eks-addons.md

This is an example of a JSON schema for the CoreDNS add-on configuration. It outlines the available properties such as affinity, replicaCount, and tolerations.

```json
{
  "$ref": "#/definitions/Coredns",
  "$schema": "http://json-schema.org/draft-06/schema#",
  "definitions": {
    "Coredns": {
      "additionalProperties": false,
      "properties": {
        "affinity": {
          "default": {
            "affinity": {
              "nodeAffinity": {
                "requiredDuringSchedulingIgnoredDuringExecution": {
                  "nodeSelectorTerms": [
                    {
                      "matchExpressions": [
                        {
                          "key": "kubernetes.io/os",
                          "operator": "In",
                          "values": [
                            "linux"
                          ]
                        },
                        {
                          "key": "kubernetes.io/arch",
                          "operator": "In",
                          "values": [
                            "amd64",
                            "arm64"
                          ]
                        }
                      ]
                    }
                  ]
                }
              },
              "podAntiAffinity": {
                "preferredDuringSchedulingIgnoredDuringExecution": [
                  {
                    "podAffinityTerm": {
                      "labelSelector": {
                        "matchExpressions": [
                          {
                            "key": "k8s-app",
                            "operator": "In",
                            "values": [
                              "kube-dns"
                            ]
                          }
                        ]
                      },
                      "topologyKey": "kubernetes.io/hostname"
                    },
                    "weight": 100
                  }
                ]
              }
            }
          },
          "description": "Affinity of the coredns pods",
          "type": [
            "object",
            "null"
          ]
        },
        "computeType": {
          "type": "string"
        },
        "corefile": {
          "description": "Entire corefile contents to use with installation",
          "type": "string"
        },
        "nodeSelector": {
          "additionalProperties": {
            "type": "string"
          },
          "type": "object"
        },
        "replicaCount": {
          "type": "integer"
        },
        "resources": {
          "$ref": "#/definitions/Resources"
        },
        "tolerations": {
          "default": [
            {
              "key": "CriticalAddonsOnly",
              "operator": "Exists"
            },
            {
              "key": "node-role.kubernetes.io/master",
              "operator": "NoSchedule"
            }
          ],
          "description": "Tolerations of the coredns pod",
          "items": {
            "type": "object"
          },
          "type": "array"
        },
        "topologySpreadConstraints": {
          "description": "The coredns pod topology spread constraints",
          "type": "array"
        }
      },
      "title": "Coredns",
      "type": "object"
    },
    "Limits": {
      "additionalProperties": false,
      "properties": {
        "cpu": {
          "type": "string"
        },
        "memory": {
          "type": "string"
        }
      },
      "title": "Limits",
      "type": "object"
    },
    "Resources": {
      "additionalProperties": false,
      "properties": {
        "limits": {
          "$ref": "#/definitions/Limits"
        },
        "requests": {
          "$ref": "#/definitions/Limits"
        }
      },
      "title": "Resources",
      "type": "object"
    }
  }
}
```

--------------------------------

### Configure Bottlerocket EC2NodeClass with Karpenter

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/bottlerocket.md

Example of configuring a Karpenter EC2NodeClass for Bottlerocket nodes. The 'bottlerocket.aws/updater-interface-version' label is crucial for BRUPOP.

```yaml
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: bottlerocket-example
spec:
...
  amiFamily: Bottlerocket
  userData:  |
    [settings.kubernetes]
    "kube-api-qps" = 30
    "shutdown-grace-period" = "30s"
    "shutdown-grace-period-for-critical-pods" = "30s"
    [settings.kubernetes.eviction-hard]
    "memory.available" = "20%"
    [settings.kubernetes.node-labels]
     "bottlerocket.aws/updater-interface-version" = "2.0.0"
```

--------------------------------

### Enable Ingress NGINX Controller

Source: https://context7.com/aws-ia/terraform-aws-eks-blueprints-addons/llms.txt

Deploys the NGINX Ingress Controller as the default ingress class. Configure replica count, service type, and load balancer annotations as needed. The example uses a Network Load Balancer (NLB).

```hcl
module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0"

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  enable_ingress_nginx = true

  ingress_nginx = {
    values = [
      <<-EOT
        controller:
          replicaCount: 2
          service:
            type: LoadBalancer
            annotations:
              service.beta.kubernetes.io/aws-load-balancer-type: nlb
      EOT
    ]
  }
}
```

--------------------------------

### Configure Bottlerocket Managed Node Group with Terraform

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/bottlerocket.md

Example of setting up a Bottlerocket managed node group using the Terraform EKS module. Ensure the 'bottlerocket.aws/updater-interface-version' label is set for BRUPOP compatibility.

```hcl
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.21"
...
  eks_managed_node_groups = {
    bottlerocket = {
      platform = "bottlerocket"
      ami_type       = "BOTTLEROCKET_x86_64"
      instance_types = ["m5.large", "m5a.large"]

      iam_role_attach_cni_policy = true

      min_size     = 1
      max_size     = 5
      desired_size = 3

      enable_bootstrap_user_data = true
      bootstrap_extra_args = <<-EOT
            [settings.host-containers.admin]
            enabled = false
            [settings.host-containers.control]
            enabled = true
            [settings.kernel]
            lockdown = "integrity"
            [settings.kubernetes.node-labels]
            "bottlerocket.aws/updater-interface-version" = "2.0.0"
            [settings.kubernetes.node-taints]
            "CriticalAddonsOnly" = "true:NoSchedule"
          EOT
    }
  }
}
```

--------------------------------

### Deploy EKS Addons with Terraform

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/README.md

Use this module to configure and deploy various Kubernetes addons to your EKS cluster. Ensure the `version` is updated to the latest desired version. This example shows enabling several common addons and configuring Cert-Manager with Route53.

```hcl
module "eks_blueprints_addons" {
  source = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0" #ensure to update this to the latest/desired version

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  eks_addons = {
    aws-ebs-csi-driver = {
      most_recent = true
    }
    coredns = {
      most_recent = true
    }
    vpc-cni = {
      most_recent = true
    }
    kube-proxy = {
      most_recent = true
    }
  }

  enable_aws_load_balancer_controller    = true
  enable_cluster_proportional_autoscaler = true
  enable_karpenter                       = true
  enable_kube_prometheus_stack           = true
  enable_metrics_server                  = true
  enable_external_dns                    = true
  enable_cert_manager                    = true
  cert_manager_route53_hosted_zone_arns  = ["arn:aws:route53:::hostedzone/XXXXXXXXXXXXX"]

  tags = {
    Environment = "dev"
  }
}

module "eks" {
  source = "terraform-aws-modules/eks/aws"

  cluster_name    = "my-cluster"
  cluster_version = "1.29"

  ... truncated for brevity
}
```

--------------------------------

### Enable Cluster Proportional Autoscaler for CoreDNS

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/cluster-proportional-autoscaler.md

Configure the Cluster Proportional Autoscaler add-on to manage CoreDNS replicas. This example sets up scaling based on CPU cores and nodes, targeting the CoreDNS deployment.

```hcl
enable_cluster_proportional_autoscaler  = true
cluster_proportional_autoscaler  = {
    values = [
      <<-EOT
        nameOverride: kube-dns-autoscaler

        # Formula for controlling the replicas. Adjust according to your needs
        # replicas = max( ceil( cores * 1/coresPerReplica ) , ceil( nodes * 1/nodesPerReplica ) )
        config:
          linear:
            coresPerReplica: 256
            nodesPerReplica: 16
            min: 1
            max: 100
            preventSinglePointFailure: true
            includeUnschedulableNodes: true

        # Target to scale. In format: deployment/*, replicationcontroller/* or replicaset/* (not case sensitive).
        options:
          target: deployment/coredns # Notice the target as `deployment/coredns`

        serviceAccount:
          create: true
          name: kube-dns-autoscaler

        podSecurityContext:
          seccompProfile:
            type: RuntimeDefault
            supplementalGroups: [65534]
            fsGroup: 65534

        resources:
          limits:
            cpu: 100m
            memory: 128Mi
          requests:
            cpu: 100m
            memory: 128Mi

        tolerations:
          - key: "CriticalAddonsOnly"
            operator: "Exists"
            description: "Cluster Proportional Autoscaler for CoreDNS Service"
      EOT
    ]
}
```

--------------------------------

### Customize VPA Helm Chart Deployment

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/vertical-pod-autoscaler.md

Customize the Helm chart deployment for VPA by specifying chart version, repository, namespace, and custom values. This allows for fine-grained control over the VPA installation.

```hcl
enable_vpa = true
enable_metrics_server = true

vpa = {
  name          = "vpa"
  chart_version = "1.7.5"
  repository    = "https://charts.fairwinds.com/stable"
  namespace     = "vpa"
  values        = [templatefile("${path.module}/values.yaml", {})
]
}
```

--------------------------------

### Create Kubernetes Namespace and Deploy Nginx

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/velero.md

Prepare for backup by creating a new namespace and deploying a sample application (nginx). These resources will be used to demonstrate the backup and restore process.

```bash
kubectl create namespace backupdemo
kubectl run nginx --image=nginx -n backupdemo
```

--------------------------------

### Create Sample Deployment for Scaling Demo

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/karpenter.md

Applies a sample 'pause' deployment with 0 replicas. This is used to demonstrate Karpenter's ability to provision nodes when the replica count is increased.

```yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 0
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      terminationGracePeriodSeconds: 0
      containers:
        - name: inflate
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
          resources:
            requests:
              cpu: 1

```

--------------------------------

### Get ClusterIssuer Status

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/cert-manager.md

Check the status of the `selfsigned-cluster-issuer` to ensure it is ready.

```sh
kubectl get clusterissuers -o wide selfsigned-cluster-issuer
```

--------------------------------

### Terraform Initialization and Application

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/tests/complete/README.md

Standard commands to initialize, plan, and apply Terraform configurations. Ensure you have the necessary AWS credentials configured.

```bash
terraform init
terraform plan
terraform apply
```

--------------------------------

### Get Kubernetes Ingress Status

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-load-balancer-controller.md

Retrieve the status of the created Kubernetes Ingress, including its assigned address, to confirm it's operational.

```sh
kubectl get ingress
NAME                CLASS   HOSTS   ADDRESS                                                                 PORTS   AGE
example-ingress     alb     *       k8s-example-ingress-7e0d6f03e7-1234567890.us-west-2.elb.amazonaws.com   80      4m9s
```

--------------------------------

### Create a Velero Backup of a Namespace

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/velero.md

Initiate a backup of the 'backupdemo' namespace using the Velero CLI. This command creates a backup named 'backup1'.

```bash
velero backup create backup1 --include-namespaces backupdemo
```

--------------------------------

### List Bottlerocket Addon Resources

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/bottlerocket.md

After updating kubeconfig, use this command to list all resources provisioned by the Bottlerocket addon in the specified namespace. This helps verify the deployment.

```bash
$kubectl -n brupop-bottlerocket-aws get all
```

--------------------------------

### Get Velero Backup Locations

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/velero.md

Retrieve information about configured Velero backup locations. This command helps confirm that your S3 bucket is correctly set up as a backup location.

```bash
velero backup-location get
```

--------------------------------

### Enable Bottlerocket Update Operator (BRUPOP)

Source: https://context7.com/aws-ia/terraform-aws-eks-blueprints-addons/llms.txt

Deploy the Bottlerocket Update Operator to manage rolling OS updates for Bottlerocket nodes. The CRD chart is installed automatically.

```hcl
module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0"

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  enable_bottlerocket_update_operator = true

  # CRD chart (bottlerocket_shadow) is installed automatically alongside the operator
  bottlerocket_update_operator = {}
  bottlerocket_shadow          = {}
}
```

--------------------------------

### Complete Production EKS Blueprints Addons Configuration

Source: https://context7.com/aws-ia/terraform-aws-eks-blueprints-addons/llms.txt

This configuration sets up a production-ready EKS cluster with native add-ons, networking, storage, security, autoscaling, observability, GitOps, and backup solutions. Ensure all required module inputs and dependencies are correctly configured.

```hcl
module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0"

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  # Native EKS add-ons
  eks_addons = {
    aws-ebs-csi-driver = {
      most_recent              = true
      service_account_role_arn = module.ebs_csi_driver_irsa.iam_role_arn
    }
    coredns    = { most_recent = true }
    vpc-cni    = { most_recent = true }
    kube-proxy = {}
  }

  # Networking
  enable_aws_load_balancer_controller = true
  aws_load_balancer_controller = {
    set = [{ name = "enableServiceMutatorWebhook", value = "false" }]
  }
  enable_external_dns            = true
  external_dns_route53_zone_arns = ["arn:aws:route53:::hostedzone/*"]
  enable_ingress_nginx           = true

  # Storage
  enable_aws_efs_csi_driver = true

  # Security
  enable_cert_manager = true
  cert_manager        = { wait = true }
  cert_manager_route53_hosted_zone_arns = ["arn:aws:route53:::hostedzone/*"]
  enable_external_secrets              = true
  enable_secrets_store_csi_driver              = true
  enable_secrets_store_csi_driver_provider_aws = true
  enable_gatekeeper                            = true

  # Autoscaling
  enable_karpenter                           = true
  karpenter_enable_instance_profile_creation = true
  karpenter = {
    repository_username = data.aws_ecrpublic_authorization_token.token.user_name
    repository_password = data.aws_ecrpublic_authorization_token.token.password
  }
  enable_metrics_server = true
  enable_vpa            = true

  # Observability
  enable_kube_prometheus_stack  = true
  enable_aws_cloudwatch_metrics = true
  enable_aws_for_fluentbit      = true
  aws_for_fluentbit_cw_log_group = {
    create      = true
    name_prefix = "eks-"
    retention   = 30
  }

  # GitOps
  enable_argocd        = true
  enable_argo_rollouts = true

  # Backup
  enable_velero = true
  velero = {
    s3_backup_location = "${module.velero_backup_s3_bucket.s3_bucket_arn}/backups"
  }

  tags = { Environment = "production" }
}

```

--------------------------------

### Deploying Native EKS Add-ons with Configuration

Source: https://context7.com/aws-ia/terraform-aws-eks-blueprints-addons/llms.txt

Configure and deploy native EKS add-ons using the `eks_addons` map. Supports `most_recent`, pinned versions, custom timeouts, and JSON configuration values.

```hcl
module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0"

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  eks_addons = {
    # Always use the latest available patch
    aws-ebs-csi-driver = {
      most_recent              = true
      service_account_role_arn = module.ebs_csi_driver_irsa.iam_role_arn
    }

    # Pin to a specific version with custom timeouts
    coredns = {
      most_recent = true
      timeouts = {
        create = "25m"
        delete = "10m"
      }
    }

    # Pass JSON configuration values
    vpc-cni = {
      most_recent = true
      configuration_values = jsonencode({
        env = {
          ENABLE_PREFIX_DELEGATION = "true"
          WARM_PREFIX_TARGET       = "1"
        }
      })
    }

    kube-proxy = {}

    # Third-party add-on from AWS Marketplace
    kubecost_kubecost = {
      most_recent = true
    }
  }

  # Global timeout overrides for all EKS add-ons
  eks_addons_timeouts = {
    create = "20m"
    update = "20m"
    delete = "20m"
  }
}

# Verify add-ons are running
# aws eks describe-addon --cluster-name <name> --addon-name aws-ebs-csi-driver
# kubectl get pods -n kube-system | grep ebs-csi
```

--------------------------------

### Customize Velero Helm Chart Deployment

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/velero.md

Configure specific parameters for the Velero Helm chart deployment, including chart version and values. This allows for fine-grained control over the Velero installation.

```hcl
enable_velero           = true

velero = {
  name          = "velero"
  description   = "A Helm chart for velero"
  chart_version = "3.1.6"
  repository    = "https://vmware-tanzu.github.io/helm-charts/"
  namespace     = "velero"
  values        = [templatefile("${path.module}/values.yaml", {})
  ]
}
```

--------------------------------

### List Velero Resources in Kubernetes

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/velero.md

Verify that Velero resources are running in the 'velero' namespace. This command lists all Kubernetes resources within that namespace.

```bash
kubectl get all -n velero
```

--------------------------------

### Describe FSx StorageClass

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-fsx-csi-driver.md

Inspect the details of the created FSx StorageClass, including its provisioner, parameters, and mount options.

```sh
$ kubect describe storageclass fsx-sc
Name:            fsx-sc
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"fsx-sc"},"mountOptions":null,"parameters":{"deploymentType":"PERSISTENT_1","perUnitStorageThroughput":"200","securityGroupIds":"sg-q1w2e3r4t5y6u7i8o","subnetId":"subnet-q1w2e3r4t5y6u7i8o"},"provisioner":"fsx.csi.aws.com"}

Provisioner:           fsx.csi.aws.com
Parameters:            deploymentType=PERSISTENT_1,perUnitStorageThroughput=200,securityGroupIds=sg-q1w2e3r4t5y6u7i8o,subnetId=subnet-q1w2e3r4t5y6u7i8o
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>
```

--------------------------------

### Check Certificate Status and Secret

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/cert-manager.md

Verify that the created `Certificate` is in a `Ready` state and that the corresponding `secret` has been generated in the specified namespace.

```sh
kubectl get certificate -o wide
NAME      READY   SECRET           ISSUER                      STATUS                                          AGE
example   True    example-secret   selfsigned-cluster-issuer   Certificate is up to date and has not expired   44s

kubectl get secret example-secret
NAME             TYPE                DATA   AGE
example-secret   kubernetes.io/tls   3      70s
```

--------------------------------

### Enable and Configure Fargate FluentBit

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/fargate-fluentbit.md

Enable the Fargate FluentBit add-on and set basic CloudWatch logging configuration.

```hcl
module "eks_blueprints_addons" {

  enable_fargate_fluentbit = true
  fargate_fluentbit = {
    flb_log_cw = true
  }
}
```

--------------------------------

### Enable Metrics Server Addon

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/metrics-server.md

Enable the Metrics Server add-on by setting the `enable_metrics_server` variable to `true`.

```hcl
enable_metrics_server = true
```

--------------------------------

### Customize AWS Gateway API Controller Helm Chart

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-gateway-api-controller.md

Customize the deployment of the AWS Gateway API Controller by specifying Helm chart details, repository, namespace, and custom values. This allows for fine-grained control over the installation.

```hcl
enable_aws_gateway_api_controller = true
aws_gateway_api_controller = {
    name                = "aws-gateway-api-controller"
    chart_version       = "v0.0.12"
    repository          = "oci://public.ecr.aws/aws-application-networking-k8s"
    repository_username = data.aws_ecrpublic_authorization_token.token.user_name
    repository_password = data.aws_ecrpublic_authorization_token.token.password
    namespace           = "aws-application-networking-system"
    values              = [templatefile("${path.module}/values.yaml", {})
    set = [{
      name  = "clusterVpcId"
      value = "vpc-12345abcd"
    }]
  }
```

--------------------------------

### Verify Restored Namespace Resources

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/velero.md

Confirm that the 'backupdemo' namespace and its resources, including the nginx pod, have been successfully restored.

```bash
kubectl get all -n backupdemo
```

--------------------------------

### Enable Velero Add-on with S3 Bucket Configuration

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/velero.md

Enable the Velero add-on and specify the S3 bucket for backups. Ensure the S3 bucket name and ARN are correctly provided.

```hcl
enable_velero           = true
velero_backup_s3_bucket = "<YOUR_BUCKET_NAME>"
velero = {
    s3_backup_location = "<YOUR_S3_BUCKET_ARN>[/prefix]"
  }
```

--------------------------------

### Enable OPA Gatekeeper

Source: https://context7.com/aws-ia/terraform-aws-eks-blueprints-addons/llms.txt

Deploys OPA Gatekeeper as an admission webhook to enforce custom policy constraints. Configure replicas, audit interval, and constraint violations limit as required. The example sets 3 replicas and an audit interval of 60 seconds.

```hcl
module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0"

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  enable_gatekeeper = true

  gatekeeper = {
    values = [
      <<-EOT
        replicas: 3
        auditInterval: 60
        constraintViolationsLimit: 20
      EOT
    ]
  }
}
```

--------------------------------

### Enable Kube Prometheus Stack Add-on

Source: https://context7.com/aws-ia/terraform-aws-eks-blueprints-addons/llms.txt

Deploys the Kube Prometheus Stack Helm chart, including Prometheus, Alertmanager, and Grafana. Configuration for Grafana admin password, ingress, and Prometheus storage is provided.

```hcl
module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0"

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  enable_kube_prometheus_stack = true

  kube_prometheus_stack = {
    namespace = "monitoring"
    values = [
      <<-EOT
        grafana:
          adminPassword: prom-operator
          ingress:
            enabled: true
            ingressClassName: alb
            annotations:
              alb.ingress.kubernetes.io/scheme: internet-facing
        prometheus:
          prometheusSpec:
            retention: 15d
            storageSpec:
              volumeClaimTemplate:
                spec:
                  storageClassName: gp3
                  accessModes: ["ReadWriteOnce"]
                  resources:
                    requests:
                      storage: 50Gi
      EOT
    ]
  }
}
```

--------------------------------

### Create Certificate Resource

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/cert-manager.md

Create a `Certificate` resource specifying details like `isCA`, `commonName`, `secretName`, `privateKey` algorithm and size, and the `issuerRef`.

```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example
  namespace: default
spec:
  isCA: true
  commonName: example
  secretName: example-secret
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-cluster-issuer
    kind: ClusterIssuer
    group: cert-manager.io
```

--------------------------------

### Enable Ingress Nginx Add-on

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/ingress-nginx.md

Enable the Ingress Nginx add-on by setting the `enable_ingress_nginx` variable to `true`.

```hcl
enable_ingress_nginx = true
```

--------------------------------

### Deploy Generic Helm Releases

Source: https://context7.com/aws-ia/terraform-aws-eks-blueprints-addons/llms.txt

Configure arbitrary Helm charts using the `helm_releases` map for add-ons not natively supported. All Helm release options are available.

```hcl
module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0"

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  helm_releases = {
    prometheus-adapter = {
      description      = "A Helm chart for k8s prometheus adapter"
      namespace        = "prometheus-adapter"
      create_namespace = true
      chart            = "prometheus-adapter"
      chart_version    = "4.2.0"
      repository       = "https://prometheus-community.github.io/helm-charts"
      values = [
        <<-EOT
          replicas: 2
          podDisruptionBudget:
            enabled: true
        EOT
      ]
    }

    gpu-operator = {
      description      = "A Helm chart for NVIDIA GPU operator"
      namespace        = "gpu-operator"
      create_namespace = true
      chart            = "gpu-operator"
      chart_version    = "v23.9.1"
      repository       = "https://nvidia.github.io/gpu-operator"
      values = [
        <<-EOT
          operator:
            defaultRuntime: containerd
        EOT
      ]
    }
  }
}

# Target only the addons module for isolated apply/destroy
# terraform apply -target=module.eks_blueprints_addons
# terraform destroy -target=module.eks_blueprints_addons
```

--------------------------------

### Enable VPA with Default Helm Chart Config

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/vertical-pod-autoscaler.md

Deploy the Vertical Pod Autoscaler using the default Helm chart configuration. Ensure the Metrics Server add-on is also enabled.

```hcl
enable_vpa            = true
enable_metrics_server = true
```

--------------------------------

### Configure Addon Creation Delay and Dependencies

Source: https://context7.com/aws-ia/terraform-aws-eks-blueprints-addons/llms.txt

Use `create_delay_duration` to introduce a delay before add-ons are created, allowing the EKS control plane to stabilize. Specify `create_delay_dependencies` to explicitly sequence add-on deployments after other resources are ready.

```hcl
module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0"

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  # Wait 30 seconds after cluster endpoint and OIDC are available
  create_delay_duration = "30s"

  # Add explicit dependency on managed node group readiness
  create_delay_dependencies = [
    module.eks.eks_managed_node_groups["initial"].node_group_arn
  ]

  enable_metrics_server = true
}
```

--------------------------------

### Create Kubernetes Ingress

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-load-balancer-controller.md

Create a Kubernetes Ingress resource using the `alb` IngressClass, specifying target service and annotations for scheme and target type.

```sh
kubectl create ingress example-ingress --class alb --rule="/*=example-svc:80" \
--annotation alb.ingress.kubernetes.io/scheme=internet-facing \
--annotation alb.ingress.kubernetes.io/target-type=ip
```

--------------------------------

### Module Declaration - Minimal Bootstrap

Source: https://context7.com/aws-ia/terraform-aws-eks-blueprints-addons/llms.txt

Declare the EKS Blueprints Addons module with essential cluster information. Ensure Terraform version is >= 1.0 and providers are configured correctly.

```hcl
module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0"

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  tags = {
    Environment = "production"
    Team        = "platform"
  }
}
```

--------------------------------

### Enable Kubelet Monitoring for Container Insights

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-for-fluentbit.md

Enable the kubelet monitoring feature for Container Insights, recommended for large clusters, by setting additional parameters.

```hcl
enable_aws_for_fluentbit = true
  aws_for_fluentbit = {
    enable_containerinsights = true
    kubelet_monitoring       = true
    set = [{
        name  = "cloudWatchLogs.autoCreateGroup"
        value = true
      },
      {
        name  = "hostNetwork"
        value = true
      },
      {
        name  = "dnsPolicy"
        value = "ClusterFirstWithHostNet"
      }
    ]
  }
```

--------------------------------

### Enable Argo CD Add-on

Source: https://context7.com/aws-ia/terraform-aws-eks-blueprints-addons/llms.txt

Enables the Argo CD add-on for GitOps continuous delivery. Requires specifying the chart version and namespace. Custom values can be provided via a template file.

```hcl
module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0"

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  # Argo CD for GitOps continuous delivery
  enable_argocd = true
  argocd = {
    chart_version = "5.29.1"
    namespace     = "argocd"
    values        = [templatefile("${path.module}/argocd-values.yaml", {})
    ]
  }

  # Argo Workflows for workflow orchestration
  enable_argo_workflows = true

  # Argo Rollouts for progressive delivery (canary, blue-green)
  enable_argo_rollouts = true

  # Argo Events for event-driven automation
  enable_argo_events = true
}

# Validate Argo CD
# kubectl get pods -n argocd
# argo-cd-argocd-application-controller-0                     1/1  Running  0  146m
# argo-cd-argocd-applicationset-controller-678d85f77b-rmpcb   1/1  Running  0  146m
# argo-cd-argocd-server-6f9cfdd4d5-8mfpc                      1/1  Running  0  146m
```

--------------------------------

### Describe Bottlerocket APIService

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/bottlerocket.md

This command provides detailed information about the APIService for Bottlerocket, including its status and configuration. Useful for debugging API access.

```bash
$kubectl describe apiservices.apiregistration.k8s.io v2.brupop.bottlerocket.aws
```

--------------------------------

### Describe GatewayClass

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/aws-gateway-api-controller.md

Inspect the details of the deployed GatewayClass using `kubectl describe`. This command shows the configuration and status of the `amazon-vpc-lattice` GatewayClass.

```sh
$kubectl describe gatewayclass
Name:         amazon-vpc-lattice
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  gateway.networking.k8s.io/v1beta1
Kind:         GatewayClass
Metadata:
  Creation Timestamp:  2023-06-22T22:33:32Z
  Generation:          1
  Resource Version:    819021
  UID:                 aac59195-8f37-4c23-a2a5-b0f363deda77
Spec:
  Controller Name:  application-networking.k8s.aws/gateway-api-controller
Status:
  Conditions:
    Last Transition Time:  2023-06-22T22:33:32Z
    Message:               Accepted
    Observed Generation:   1
    Reason:                Accepted
    Status:                True
    Type:                  Accepted
Events:                    <none>
```

--------------------------------

### Enable Karpenter Node Autoscaler

Source: https://context7.com/aws-ia/terraform-aws-eks-blueprints-addons/llms.txt

Deploys Karpenter, sets up SQS for interruption events, and provisions IAM roles. Requires ECR Public authentication.

```hcl
data "aws_ecrpublic_authorization_token" "token" {
  provider = aws.virginia
}

module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.0"

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  enable_karpenter = true

  # Allow Karpenter to create the instance profile (v1beta1 behavior)
  karpenter_enable_instance_profile_creation = true

  # Enable native spot termination handling via SQS
  karpenter_enable_spot_termination = true

  karpenter = {
    repository_username = data.aws_ecrpublic_authorization_token.token.user_name
    repository_password = data.aws_ecrpublic_authorization_token.token.password
  }

  karpenter_node = {
    iam_role_additional_policies = {
      AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
    }
  }
}
```

```bash
# Validate Karpenter is running
# aws iam create-service-linked-role --aws-service-name spot.amazonaws.com || true
# aws eks --region us-west-2 update-kubeconfig --name <cluster-name>
# kubectl get pods -n karpenter
# NAME                         READY   STATUS    RESTARTS   AGE
# karpenter-6f97df4f77-5nqsk   1/1     Running   0          3m28s
```

```yaml
# Test autoscaling with a sample deployment
# kubectl apply -f - <<EOF
# apiVersion: apps/v1
# kind: Deployment
# metadata:
#   name: inflate
# spec:
#   replicas: 0
#   selector:
#     matchLabels:
#       app: inflate
#   template:
#     metadata:
#       labels:
#         app: inflate
#     spec:
#       terminationGracePeriodSeconds: 0
#       containers:
#         - name: inflate
#           image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
#           resources:
#             requests:
#               cpu: 1
# EOF
# kubectl scale deployment inflate --replicas 5
# kubectl get nodes  # New EC2 node appears within ~30s
```

```bash
# Access outputs
# module.eks_blueprints_addons.karpenter.iam_role_arn
# module.eks_blueprints_addons.karpenter.node_iam_role_arn
# module.eks_blueprints_addons.karpenter.sqs.queue_url
```

--------------------------------

### Describe EKS Addon Configuration Schema

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/amazon-eks-addons.md

Use this AWS CLI command to retrieve the JSON schema for an EKS add-on's configuration. This schema defines the valid parameters you can supply.

```bash
aws eks describe-addon-configuration \
--addon-name coredns \
--addon-version v1.10.1-eksbuild.2 \
--query 'configurationSchema' \
--output text | jq
```

--------------------------------

### List Karpenter Pods

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/karpenter.md

Verifies that the Karpenter controller pods are running in the 'karpenter' namespace.

```sh
kubectl get pods -n karpenter
```

--------------------------------

### Verify Node and CoreDNS Pod Count After Scaling

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/cluster-proportional-autoscaler.md

After increasing the managed node group size and applying changes, verify that the new node has joined the cluster and that the CoreDNS pods have scaled up accordingly.

```bash
NAME                          STATUS   ROLES    AGE   VERSION
ip-10-0-14-120.ec2.internal   Ready    <none>   10m   v1.26.4-eks-0a21954
ip-10-0-19-243.ec2.internal   Ready    <none>   21h   v1.26.4-eks-0a21954
ip-10-0-25-182.ec2.internal   Ready    <none>   21h   v1.26.4-eks-0a21954
ip-10-0-40-138.ec2.internal   Ready    <none>   21h   v1.26.4-eks-0a21954
ip-10-0-8-136.ec2.internal    Ready    <none>   21h   v1.26.4-eks-0a21954

kubectl get po -n kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-7975d6fb9b-dlkdd   1/1     Running   0          21h
coredns-7975d6fb9b-ww64t   1/1     Running   0          10m
coredns-7975d6fb9b-xqqwp   1/1     Running   0          21h
```

--------------------------------

### Enable Kube Prometheus Stack Addon

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/kube-prometheus-stack.md

Enable the Kube Prometheus Stack addon by setting this variable to true.

```hcl
enable_kube_prometheus_stack = true
```

--------------------------------

### Verify Argo Rollouts Pods

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/argo-rollouts.md

Check the status of Argo Rollouts pods in the specified namespace.

```sh
$ kubectl get pods -n argo-rollouts
NAME                             READY   STATUS    RESTARTS   AGE
argo-rollouts-5db5688849-x89zb   0/1     Running   0          11s
```

--------------------------------

### Verify Cluster Proportional Autoscaler Config Map

Source: https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/main/docs/addons/cluster-proportional-autoscaler.md

Confirm that the configuration map for the Cluster Proportional Autoscaler has been created.

```bash
kubectl -n kube-system get cm cluster-proportional-autoscaler-kube-dns-autoscaler
NAME                                                  DATA   AGE
cluster-proportional-autoscaler-kube-dns-autoscaler   1      21h
```