### Build and install the driver

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/CONTRIBUTING.md

Builds the driver image and installs it into the cluster.

```bash
make cluster/image
make cluster/install
```

--------------------------------

### Delete example resources

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/modify-volume/README.md

Remove all resources created during the example process.

```sh
$ kubectl delete -f manifests 
storageclass.storage.k8s.io "ebs-sc" deleted
persistentvolumeclaim "ebs-claim" deleted
pod "app" deleted
volumeattributesclass.storage.k8s.io "io2-class" deleted
```

--------------------------------

### Deploy example resources

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/modify-volume/README.md

Apply the initial Pod, PersistentVolumeClaim, and StorageClass manifests to the cluster.

```sh
$ kubectl apply -f manifests/pod-with-volume.yaml

storageclass.storage.k8s.io/ebs-sc created
persistentvolumeclaim/ebs-claim created
pod/app created
```

--------------------------------

### Install the driver using Helm

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/CONTRIBUTING.md

Installs the driver using the default release image, optionally using HELM_EXTRA_FLAGS for custom parameters.

```bash
# Use HELM_EXTRA_FLAGS for testing parameters, for example:
# export HELM_EXTRA_FLAGS="--set=controller.newParameter=true"
make cluster/helm
```

--------------------------------

### Create PVC, StorageClass, and Pod Manifests

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/modify-volume.md

Initial setup for a PersistentVolumeClaim, StorageClass, and a Pod that mounts the volume.

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 10Gi
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: centos
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: ebs-claim
```

--------------------------------

### Install driver via Helm

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/windows/README.md

Helm command to install the driver with Windows node support enabled.

```bash
helm upgrade --install aws-ebs-csi-driver --namespace kube-system ./charts/aws-ebs-csi-driver --set node.enableWindows=true --set image.repository=$MY_REGISTRY/aws-ebs-csi-driver --set image.tag=$MY_TAG
```

--------------------------------

### Deploy Driver via Helm

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/install.md

Install the driver using Helm from various supported repositories.

```sh
helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver
helm repo update

helm upgrade --install aws-ebs-csi-driver \
    --namespace kube-system \
    aws-ebs-csi-driver/aws-ebs-csi-driver
```

```sh
helm upgrade --install aws-ebs-csi-driver \
    --namespace kube-system \
    oci://registry.k8s.io/provider-aws/charts/aws-ebs-csi-driver \
    --version 2.62.0
```

```sh
helm upgrade --install aws-ebs-csi-driver \
    --namespace kube-system \
    oci://public.ecr.aws/ebs-csi-driver/charts/aws-ebs-csi-driver \
    --version 2.62.0
```

--------------------------------

### Configure Outpost Snapshot in VolumeSnapshotClass

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/snapshot.md

Example of a VolumeSnapshotClass configured to store snapshots on an AWS Outpost.

```yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-aws-vsc
driver: ebs.csi.aws.com
deletionPolicy: Delete
parameters:
  outpostarn: {arn of your outpost}
```

--------------------------------

### Enable Node-Local Volumes via Helm

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/node-local-volumes.md

Use this command to install or upgrade the driver with the node-local volumes feature enabled.

```bash
helm upgrade --install aws-ebs-csi-driver \
  --namespace kube-system \
  ./charts/aws-ebs-csi-driver \
  --set controller.enableNodeLocalVolumes=true
```

--------------------------------

### Run driver in node mode

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/design.md

Starts only the node service of the CSI driver.

```bash
/bin/aws-ebs-csi-driver node --endpoint=unix://...
```

--------------------------------

### Install and uninstall EBS CSI Driver

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/makefile.md

Commands to manage the lifecycle of the EBS CSI driver installation on the cluster.

```bash
make cluster/image
make cluster/install
```

```bash
make cluster/helm
```

```bash
make cluster/uninstall
```

--------------------------------

### Example output for attachment breakdown

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/hack/cluster-debugging-scripts/README.md

The expected output format after running the attachment breakdown script.

```text
Attachments for ng-f3ecdf71
BlockDeviceMappings  ENIs  Available-Attachment-Slots(Validated-by-aws-ebs-csi-driver)
1                    2     25
```

--------------------------------

### Resulting Snapshot Tags

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/tagging.md

Example output of tags applied to a snapshot in namespace 'ns-prod' with name 'ebs-snapshot'.

```text
key1=value1
key2=<empty string>
snapshotnamespace=ns-prod
snapshotname=ebs-snapshot
snapshotcontentname=<the computed VolumeSnapshotContent name>
key6=true
```

--------------------------------

### Run driver in controller mode

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/design.md

Starts only the controller service, useful for off-cluster controller deployments.

```bash
/bin/aws-ebs-csi-driver controller --extra-volume-tags=foo=bar
```

```bash
AWS_REGION=us-west-1 /bin/aws-ebs-csi-driver controller --extra-volume-tags=foo=bar
```

--------------------------------

### Checkout Dependabot PR

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/updating-dependencies.md

Example command to check out a Dependabot pull request locally for troubleshooting or manual fixes.

```bash
gh pr checkout {pr_number}
```

--------------------------------

### XFS Formatting Error Message

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/faq.md

Example of the error message encountered when XFS features are incompatible with the host kernel.

```text
wrong fs type, bad option, bad superblock on /dev/nvme2n1, missing codepage or helper program, or other error
```

--------------------------------

### Apply configuration via Helm

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/faq.md

Use the configuration file during Helm installation or upgrade of the AWS EBS CSI driver.

```sh
helm upgrade --install aws-ebs-csi-driver \
--namespace kube-system \
--values example-ebs-csi-config-values.yaml
aws-ebs-csi-driver/aws-ebs-csi-driver
```

--------------------------------

### Configure IRSA via Helm

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/install.md

Example Helm values for annotating the controller service account to use an IAM role for service accounts (IRSA).

```yaml
controller:
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::123412341234:role/ebs-csi-role
```

--------------------------------

### Configure Snapshot Lock in Governance Mode

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/snapshot.md

Example of a VolumeSnapshotClass configured for governance mode locking with a 7-day duration.

```yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-aws-vsc-locked
driver: ebs.csi.aws.com
deletionPolicy: Delete
parameters:
  lockMode: "governance"
  lockDuration: "7"
```

--------------------------------

### Deploy Resources

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/static-provisioning/README.md

Apply the manifests to the cluster to create the PV, PVC, and pod.

```bash
$ kubectl apply -f manifests

persistentvolumeclaim/ebs-claim created
pod/app created
persistentvolume/test-pv created
```

--------------------------------

### Deploy Source Application

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/clone/README.md

Deploy the pod and PersistentVolumeClaim to establish the source volume.

```sh
$ kubectl apply -f manifests/app/

persistentvolumeclaim/ebs-claim created
pod/app created
```

--------------------------------

### Deploy EBS resources

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/dynamic-provisioning/README.md

Apply the manifest files to create the StorageClass, PersistentVolumeClaim, and pod.

```sh
$ kubectl apply -f manifests

persistentvolumeclaim/ebs-claim created
pod/app created
storageclass.storage.k8s.io/ebs-sc created
```

--------------------------------

### Deploy Resizing Manifests

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/resizing/README.md

Apply the StorageClass, PersistentVolumeClaim, and pod configuration to the cluster.

```sh
$ kubectl apply -f manifests

persistentvolumeclaim/ebs-claim created
pod/app created
storageclass.storage.k8s.io/resize-sc created
```

--------------------------------

### Deploy Ephemeral Volume Resources

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/ephemeral-volume/README.md

Apply the manifest files to create the pod and storage class.

```sh
$ kubectl apply -f manifests

pod/app created
storageclass.storage.k8s.io/ebs-ephemeral-demo created
```

--------------------------------

### Create a kops test cluster

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/CONTRIBUTING.md

Initializes a kops cluster for testing the driver.

```bash
make cluster/create
```

--------------------------------

### Create Snapshot Resources

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/snapshot/manifests/snapshot-import/README.md

Applies the VolumeSnapshotContent and VolumeSnapshot manifests.

```bash
kubectl apply -f specs/snapshot-import/static-snapshot/
```

--------------------------------

### Build and push driver image

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/windows/README.md

Command to build the driver and push it to a custom image registry.

```bash
TAG=$MY_TAG REGISTRY=$MY_REGISTRY make all-push
```

--------------------------------

### Volume Attachment Warning Symptom

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/faq.md

Example of the warning message generated by the attachdetach-controller when a volume is stuck in a multi-attach state.

```text
Warning  FailedAttachVolume      6m51s              attachdetach-controller  Multi-Attach error for volume "pvc-4a86c32c-fbce-11ea-b7d2-0ad358e4e824" Volume is already exclusively attached to one node and can't be attached to another
```

--------------------------------

### Apply Storage and Snapshot Classes

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/snapshot/manifests/snapshot-import/README.md

Applies the required StorageClass and VolumeSnapshotClass manifests.

```bash
kubectl apply -f specs/classes/
```

--------------------------------

### Create an eksctl cluster with Windows nodes

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/CONTRIBUTING.md

Configures environment variables and creates an eksctl cluster specifically for testing Windows driver implementations.

```bash
export WINDOWS="true"
# Note: CLUSTER_TYPE must be set for all e2e tests and cluster deletion
# Re-export it if necessary (for example, if running tests in a separate terminal tab)
export CLUSTER_TYPE="eksctl"
make cluster/create
```

--------------------------------

### Deploy Raw Block Volume Resources

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/block-volume/README.md

Applies the manifest files to create the pod, storage class, and persistent volume claim.

```sh
$ kubectl apply -f manifests

pod/app created
persistentvolumeclaim/block-claim created
storageclass.storage.k8s.io/ebs-sc created
```

--------------------------------

### Run driver in all mode

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/design.md

Standard mode running both controller and node services within the cluster.

```bash
/bin/aws-ebs-csi-driver --extra-volume-tags=foo=bar
```

```bash
/bin/aws-ebs-csi-driver all --extra-volume-tags=foo=bar
```

--------------------------------

### Configure Snapshot Lock in Compliance Mode

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/snapshot.md

Example of a VolumeSnapshotClass configured for compliance mode locking with an expiration date and cool-off period.

```yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-aws-vsc-compliance
driver: ebs.csi.aws.com
deletionPolicy: Delete
parameters:
  lockMode: "compliance"
  lockExpirationDate: "2030-12-31T23:59:59Z"
  lockCoolOffPeriod: "24"
```

--------------------------------

### Configure Kubelet Graceful Node Shutdown

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/faq.md

User data script to configure systemd and Kubelet settings for graceful node shutdown on EKS-optimized AMIs.

```bash
#!/bin/bash
  echo -e "InhibitDelayMaxSec=45\n" >> /etc/systemd/logind.conf
  systemctl restart systemd-logind
  echo "$(jq ".shutdownGracePeriod=\"45s\"" /etc/kubernetes/kubelet/kubelet-config.json)" > /etc/kubernetes/kubelet/kubelet-config.json
  echo "$(jq ".shutdownGracePeriodCriticalPods=\"15s\"" /etc/kubernetes/kubelet/kubelet-config.json)" > /etc/kubernetes/kubelet/kubelet-config.json
  systemctl restart kubelet
```

--------------------------------

### Create StorageClass

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/clone/README.md

Apply the storage class configuration to the cluster.

```sh
$ kubectl apply -f manifests/classes/

storageclass.storage.k8s.io/ebs-sc created
```

--------------------------------

### Create a cluster with specific configurations

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/makefile.md

Configures cluster creation using environment variables for custom availability zones, cluster types, instance types, and node configurations.

```bash
export AWS_AVAILABILITY_ZONES="us-west-2a"
make cluster/create
```

```bash
export CLUSTER_TYPE="eksctl"
make cluster/create
```

```bash
export INSTANCE_TYPE="m7g.medium"
export AMI_PARAMETER="/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64"
make cluster/create
```

```bash
export WINDOWS="true"
export CLUSTER_TYPE="eksctl"
make cluster/create
```

```bash
export AMI_FAMILY="Bottlerocket"
export CLUSTER_TYPE="eksctl"
make cluster/create
```

```bash
export CLUSTER_TYPE="eksctl"
export OUTPOST_ARN="arn:aws:outposts:us-east-1:123456789012:outpost/op-0f39f7c0af9b166a3"
export OUTPOST_INSTANCE_TYPE=c5.xlarge
make cluster/create
```

--------------------------------

### Run E2E tests

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/windows/README.md

Cordon Linux nodes and run storage tests on Windows nodes.

```bash
kubectl cordon -l kubernetes.io/os=linux
# edit the webhook such that OSLabelSelectorOverride=all, otherwise the webhook
# won't mutate Pods created by the test and they won't run
kubectl edit deployment -n kube-system vpc-admission-webhook
deployment.apps/vpc-admission-webhook edited
ginkgo -nodes=1 -v --focus="External.Storage.*default.fs.*should.store.data" ./tests/e2e-kubernetes/ -- -kubeconfig=$KUBECONFIG -gce-zone=us-west-2a -node-os-distro=windows
```

--------------------------------

### Configure StorageClass with WaitForFirstConsumer

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/outposts.md

Use this configuration to enable topology-aware provisioning without explicitly specifying the Outpost ARN.

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: outpostExampleOne
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
```

--------------------------------

### Configure Dynamic StorageClass Tags

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/tagging.md

Use Go template placeholders for PVC and PV metadata in tag values. Requires the '--extra-create-metadata' flag on the external-provisioner sidecar.

```yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
parameters:
  tagSpecification_1: "pvcnamespace={{ .PVCNamespace }}"
  tagSpecification_2: "pvcname={{ .PVCName }}"
  tagSpecification_3: "pvname={{ .PVName }}"
```

--------------------------------

### Build E2E test images

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/makefile.md

Builds container images for E2E testing, supporting architecture and OS-specific variants.

```bash
make cluster/image
```

```bash
export IMAGE_ARCH="arm64"
make cluster/image
```

```bash
export WINDOWS="true"
make cluster/image
```

--------------------------------

### Run project update

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/updating-dependencies.md

Command to execute project-wide update scripts and verify consistency.

```makefile
make update
```

--------------------------------

### Deploy Node-Local Volume Manifests

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/node-local-volumes/README.md

Apply the necessary Kubernetes manifests to deploy the application using node-local volumes.

```bash
kubectl apply -f manifests/
```

--------------------------------

### Deploy Driver via Kustomize

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/install.md

Apply the driver manifests directly from the stable release branch using kubectl.

```sh
kubectl apply -k "github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/?ref=release-1.62"
```

--------------------------------

### Deploy DaemonSet with Multi-Attach Volume

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/multi-attach.md

Mounts the block device to pods across multiple nodes using volumeDevices.

```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: app-daemon
spec:
  selector:
    matchLabels:
      name: app
  template:
    metadata:
      labels:
        name: app
    spec:
      containers:
      - name: app
        image: busybox
        command: ["/bin/sh", "-c"]
        args: ["tail -f /dev/null"]
        volumeDevices:
        - name: data
          devicePath: /dev/xvda
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: block-claim
```

--------------------------------

### Configure StorageClass with Immediate Binding

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/outposts.md

Use this configuration to explicitly bind volumes to a specific Outpost using the outpost-id topology key.

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: outpostExampleTwo
provisioner: ebs.csi.aws.com
parameters:
  type: gp2  # Specify volume type supported on your outpost
volumeBindingMode: Immediate
allowedTopologies:
- matchLabelExpressions:
  - key: topology.ebs.csi.aws.com/outpost-id
    values:
    -  "arn:aws:outposts:xxxx:xxxxx:outpost/op-xxxxx"  # Replace with your Outpost ARN
```

--------------------------------

### Verify Volume Properties in AWS

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/modify-volume.md

Commands to retrieve the volume handle and inspect its current configuration in AWS.

```bash
$ pv=$(k get -o json pvc ebs-claim | jq -r '.spec | .volumeName')
$ volumename=$(k get -o json pv $pv | jq -r '.spec | .csi | .volumeHandle')
$ aws ec2 describe-volumes --volume-ids $volumename | jq '.Volumes[] | "\(.VolumeType) \(.Iops) \(.Throughput)"'
"gp3 3000 125"
```

--------------------------------

### Deploy Application and PVC

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/snapshot/manifests/snapshot-import/README.md

Deploys the PersistentVolumeClaim and the associated Pod to use the imported snapshot.

```bash
kubectl apply -f specs/snapshot-import/app/
```

--------------------------------

### Describe Persistent Volumes

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/windows/README.md

Validate volume creation and check the volumeHandle.

```bash
kubectl describe pv
```

--------------------------------

### Build local driver binary

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/makefile.md

Compiles the driver binary for the local platform. Supports OS and architecture overrides via environment variables.

```makefile
make
```

```makefile
make bin/aws-ebs-csi-driver
```

```makefile
make bin/aws-ebs-csi-driver.exe
```

```bash
OS=linux ARCH=arm64 make
```

--------------------------------

### Apply StorageClass and VolumeSnapshotClass

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/snapshot/README.md

Initializes the necessary storage classes for snapshot operations.

```sh
$ kubectl apply -f manifests/classes/

volumesnapshotclass.snapshot.storage.k8s.io/csi-aws-vsc created
storageclass.storage.k8s.io/ebs-sc created
```

--------------------------------

### Verify Multi-Attach Deployment

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/multi-attach.md

Checks the status of pods running the DaemonSet.

```bash
$ kubectl get pods -A

NAMESPACE     NAME                                   READY   STATUS    RESTARTS       AGE
default       app-daemon-9hdgw                       1/1     Running   0              18s
default       app-daemon-xm8zr                       1/1     Running   0              18s
```

--------------------------------

### Run EBS CSI Driver scalability tests

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/hack/ebs-scale-test/README.md

Commands to configure parameters, provision the cluster, execute the test, and clean up AWS resources.

```shell
# Set scalability parameters
export CLUSTER_TYPE="karpenter"
export TEST_TYPE="scale-sts"
export REPLICAS="1000"

# Setup an EKS scalability cluster and install EBS CSI Driver. 
./scale-test setup

# Run a scalability test and export results.
./scale-test run

# Cleanup all AWS resources related to scalability cluster. 
./scale-test cleanup
```

--------------------------------

### Update and verify source code

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/CONTRIBUTING.md

Formats source files, regenerates generated files, and runs linters to ensure code quality before submission.

```bash
make update
make verify
```

--------------------------------

### Run E2E tests with go test

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/tests/e2e-kubernetes/README.md

Executes the test suite using the ginkgo test runner with specific focus and skip patterns.

```sh
go test -v -timeout 0 ./... -kubeconfig=$HOME/.kube/config -report-dir=$ARTIFACTS -ginkgo.focus="$FOCUS" -ginkgo.skip="\[Disruptive\]" -gce-zone=us-west-2a
```

--------------------------------

### Run Helm chart tests

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/CONTRIBUTING.md

Executes the Helm chart testing suite.

```bash
make e2e/helm-ct
```

--------------------------------

### Manage EKS add-on configuration

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/faq.md

Apply configuration values when creating or updating the EKS managed add-on.

```sh
ADDON_CONFIG_FILEPATH="./example-addon-config.yaml"

aws eks create-addon \
  --cluster-name "example-cluster" \
  --addon-name "aws-ebs-csi-driver" \
  --service-account-role-arn "arn:aws:iam::123456789012:role/EBSCSIDriverRole" \
  --configuration-values "file://$ADDON_CONFIG_FILEPATH"
```

```sh
ADDON_CONFIG_FILEPATH="./example-addon-config.yaml"

aws eks update-addon \
  --cluster-name "example-cluster" \
  --addon-name "aws-ebs-csi-driver" \
  --configuration-values "file://$ADDON_CONFIG_FILEPATH"
```

--------------------------------

### Verify Data Persistence

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/static-provisioning/README.md

Check that the pod can read data written to the statically provisioned volume.

```bash
$ kubectl exec app -- cat /data/out.txt

Tue Feb 22 20:51:37 UTC 2022
...
```

--------------------------------

### Define PersistentVolume and PersistentVolumeClaim

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/node-local-volumes.md

Create the PV and PVC resources using the local-ebs prefix for the volumeHandle.

```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: node-local-cache-pv
spec:
  capacity:
    storage: 100Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: ebs.csi.aws.com
    volumeHandle: local-ebs://dev/xvdbz
    volumeAttributes:
      ebs.csi.aws.com/fsType: "xfs"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: node-local-cache-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  volumeName: node-local-cache-pv
```

--------------------------------

### Configure sidecar resources and environment variables

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/install.md

Structure for defining resources and environment variables for sidecars in version 2.X.

```yaml
sidecars:
  provisioner:
    env: []
    resources: {}
  attacher:
    env: []
    resources: {}
  snapshotter:
    env: []
    resources: {}
  livenessProbe:
    resources: {}
  resizer:
    env: []
    resources: {}
  nodeDriverRegistrar:
    env: []
    resources: {}
```

--------------------------------

### Enable FIPS mode via Kustomize

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/fips.md

Add these environment variables to the controller and node DaemonSet containers to activate FIPS mode.

```yaml
env:
  - name: GODEBUG
    value: "fips140=on"
  - name: AWS_USE_FIPS_ENDPOINT
    value: "true"
```

--------------------------------

### Apply VolumeAttributesClass

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/modify-volume/README.md

Deploy the VolumeAttributesClass manifest to the cluster.

```sh
$ kubectl apply -f manifests/volumeattributesclass.yaml
```

--------------------------------

### Configure CSI sidecar parameters in YAML

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/faq.md

Define custom scalability arguments for provisioner and resizer sidecars in a configuration file.

```yaml
sidecars:
  provisioner:
    additionalArgs:
    - "--worker-threads=101"
    - "--kube-api-burst=200"
    - "--kube-api-qps=40.0"
    - "--timeout=61s"
  resizer:
    additionalArgs:
    - "--workers=101"
```

--------------------------------

### Cleanup Resources

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/clone/README.md

Remove the cloned application, source application, and storage class resources.

```sh
$ kubectl delete -f manifests/clone/

persistentvolumeclaim "ebs-clone-claim" deleted
pod "clone-app" deleted
```

```sh
$ kubectl delete -f manifests/app/

persistentvolumeclaim "ebs-claim" deleted
pod "app" deleted
```

```sh
$ kubectl delete -f manifests/classes/

storageclass.storage.k8s.io "ebs-sc" deleted
```

--------------------------------

### Build and push official release images

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/makefile.md

Builds and pushes all image variants required for an official release. This target is intended for CI environments only.

```makefile
make all-push
```

--------------------------------

### Verify PV annotations

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/modify-volume.md

Inspect the PersistentVolume to ensure that the required CSI annotations are present and preserved for reconciliation.

```bash
$ k describe pv           
Name:              pvc-6bff47f9-1843-4576-ba15-158c73491e8c
Labels:            <none>
Annotations:       ebs.csi.aws.com/iops: 4000
                   ebs.csi.aws.com/volumeType: io2
                   pv.kubernetes.io/provisioned-by: ebs.csi.aws.com
                   volume.kubernetes.io/provisioner-deletion-secret-name: 
                   volume.kubernetes.io/provisioner-deletion-secret-namespace: 
Finalizers:        [kubernetes.io/pv-protection external-attacher/ebs-csi-aws-com]
StorageClass:      ebs-sc
Status:            Bound
Claim:             default/ebs-claim
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          10Gi
Node Affinity:     
  Required Terms:  
    Term 0:        topology.ebs.csi.aws.com/zone in [us-east-2b]
Message:           
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            ebs.csi.aws.com
    FSType:            ext4
    VolumeHandle:      vol-02cb54d01c685b919
    ReadOnly:          false
    VolumeAttributes:      storage.kubernetes.io/csiProvisionerIdentity=1684161810754-8081-ebs.csi.aws.com
Events:                <none>
```

--------------------------------

### Verify Driver Deployment

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/install.md

Check the status of the driver pods in the kube-system namespace.

```sh
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver
```

--------------------------------

### CreateVolume Sequence Diagram

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/design.md

Flow of the CreateVolume RPC, showing interaction between the CSI Provisioner, EBS Plugin, and AWS API.

```mermaid
sequenceDiagram
    participant s as CSI Provisioner
    participant d as EBS Plugin
    participant e as AWS API

    s->>d: CreateVolume RPC
    activate d
    
    d->>d: Parse Request + Ensure Idempotency

    d->>e: EC2 CreateVolume
    activate e
    e-->>d: Get volumeID <br/> state == 'creating'
    

    d->>e: Poll EC2 DescribeVolumes
    note over e: 1-3 Seconds
    e-->>d: Volume state == 'available'
    deactivate e
    
    d-->>s: Return Volume Response 
    deactivate d
```

--------------------------------

### Build the driver image for E2E tests

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/CONTRIBUTING.md

Builds the driver image required before running automated E2E tests.

```bash
make cluster/image
```

--------------------------------

### Generate test manifest with Go

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/hack/cluster-debugging-scripts/README.md

Generates a YAML manifest for a pod with a specific number of PVCs using the provided template.

```go
go run "generate_example_manifest.go" --node-affinity "some.label:value" --volume-count "22" --test-pod-name "test-pod"
```

--------------------------------

### Generate Changelog with hack/release

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/release.md

Use the hack/release script to generate changelog notes for a specific version and branch. Requires a GitHub user and token with repository permissions.

```bash
python3 hack/release --github-user=$GITHUB_USER --github-token=$GITHUB_TOKEN note --since $PREVIOUS_VERSION --branch $BRANCH
```

--------------------------------

### Define EC2 Block Device Mapping

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/node-local-volumes.md

Configure the EC2 launch template to pre-attach an EBS volume at the required device path.

```json
{
  "BlockDeviceMappings": [{
    "DeviceName": "/dev/xvdbz",
    "Ebs": {
      "VolumeSize": 100,
      "VolumeType": "gp3",
      "DeleteOnTermination": true
    }
  }]
}
```

--------------------------------

### Cleanup resources

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/windows/README.md

Remove test resources and uncordon Linux nodes.

```bash
kubectl delete -f manifests/
kubectl uncordon -l kubernetes.io/os=linux
```

--------------------------------

### Run unit tests

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/makefile.md

Executes all unit tests with race condition detection enabled.

```makefile
make test
```

--------------------------------

### Verify PVC Creation

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/modify-volume.md

Command to check the status of the created PersistentVolumeClaim.

```bash
$ k describe pvc ebs-claim
Name:          ebs-claim
Namespace:     default
StorageClass:  ebs-sc
Status:        Bound
Volume:        pvc-20c5ddec-5913-4b8d-a2fc-bfd0943b966d
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
               volume.kubernetes.io/selected-node: ip-192-168-32-79.ec2.internal
               volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      10Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       app
Events:
  Type     Reason                 Age              From                                                                                     Message
  ----     ------                 ----             ----                                                                                     -------
  Warning  ProvisioningFailed     6s               persistentvolume-controller                                                              storageclass.storage.k8s.io "ebs-sc" not found
  Normal   Provisioning           5s               ebs.csi.aws.com_ebs-csi-controller-b84fbbd88-5z66k_d2cb941a-aee8-41d6-aea5-8f4a40c7c82c  External provisioner is provisioning volume for claim "default/ebs-claim"
  Normal   ExternalProvisioning   4s (x2 over 5s)  persistentvolume-controller                                                              waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
  Normal   ProvisioningSucceeded  1s               ebs.csi.aws.com_ebs-csi-controller-b84fbbd88-5z66k_d2cb941a-aee8-41d6-aea5-8f4a40c7c82c  Successfully provisioned volume pvc-20c5ddec-5913-4b8d-a2fc-bfd0943b966d
```

--------------------------------

### Verify Source Data

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/clone/README.md

Check that the application has successfully written data to the volume.

```sh
$ kubectl exec app -- cat /data/out.txt

Thu Feb 24 04:07:57 UTC 2022
...
```

--------------------------------

### Update Go modules

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/updating-dependencies.md

Command to update all Go modules to their latest versions.

```bash
go get -u ./...
```

--------------------------------

### Validate attachment limits for an instance type

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/hack/cluster-debugging-scripts/README.md

Use these commands to create a test nodegroup, run the attachment breakdown script, and clean up the resources.

```bash
export CLUSTER_NAME="devcluster"
export NODEGROUP_NAME="ng-attachment-limit-test"
export INSTANCE_TYPE="m7g.large"

eksctl create nodegroup -c "$CLUSTER_NAME" --nodes 1 -t "$INSTANCE_TYPE" -n "$NODEGROUP_NAME"

./get-attachment-breakdown "$INSTANCE_TYPE" 

eksctl delete nodegroup -c "$CLUSTER_NAME" -n "$INSTANCE_TYPE"
```

--------------------------------

### ControllerGetCapabilities Response

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/design.md

The static capabilities list returned by the ControllerGetCapabilities RPC.

```text
  rpc:
    - CREATE\_DELETE\_VOLUME
    - PUBLISH\_UNPUBLISH\_VOLUME
    - ...
```

--------------------------------

### Run specific E2E test

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/tests/e2e/README.md

Executes a targeted test case using the Ginkgo test framework.

```bash
ginkgo run --focus='should create multiple PV objects, bind to PVCs and attach all to a single pod'
```

--------------------------------

### Verify Resource Deletion

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/ephemeral-volume/README.md

Confirm that the PVC and PV were automatically removed after pod deletion.

```sh
$ kubectl get pvc app-persistent-storage

Error from server (NotFound): persistentvolumeclaims "app-persistent-storage" not found

$ kubectl get pv <PV_NAME>
   
Error from server (NotFound): persistentvolumes "pvc-9124c6d0-382a-49c5-9494-bcb60f6c0c9c" not found
```

--------------------------------

### Configure Static StorageClass Tags

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/tagging.md

Define static tags for EBS volumes by adding parameters with the 'tagSpecification' prefix to the StorageClass.

```yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
parameters:
  tagSpecification_1: "key1=value1"
  tagSpecification_2: "key2=hello world"
  tagSpecification_3: "key3="
```

--------------------------------

### Configure sidecars in Helm chart v1.X

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/install.md

Legacy structure for defining sidecar images in version 1.X.

```yaml
sidecars:
  provisionerImage:
  attacherImage:
  snapshotterImage:
  livenessProbeImage:
  resizerImage:
  nodeDriverRegistrarImage:
```

--------------------------------

### Verify volume data

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/dynamic-provisioning/README.md

Execute a command inside the pod to verify that data was successfully written to the volume.

```sh
$ kubectl exec app -- cat /data/out.txt

Tue Feb 22 01:24:44 UTC 2022
...
```

--------------------------------

### Identity Service GetPluginCapabilities Response

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/design.md

The static capabilities list returned by the GetPluginCapabilities RPC.

```text
   Capabilities:
     - CONTROLLER_SERVICE
     - ACCESSIBILITY_CONSTRAINTS
     - ...
```

--------------------------------

### Configure VolumeSnapshotClass with Tags

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/tagging.md

Defines a VolumeSnapshotClass with static and interpolated tags. Interpolated tags require the --extra-create-metadata flag.

```yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-aws-vsc
driver: ebs.csi.aws.com
deletionPolicy: Delete
parameters:
  tagSpecification_1: "key1=value1"
  tagSpecification_2: "key2="
  # Interpolated tag
  tagSpecification_3: "snapshotnamespace={{ .VolumeSnapshotNamespace }}"
  tagSpecification_4: "snapshotname={{ .VolumeSnapshotName }}"
  tagSpecification_5: "snapshotcontentname={{ .VolumeSnapshotContentName }}"
  # Interpolated tag w/ function
  tagSpecification_6: 'key6={{ .VolumeSnapshotNamespace | contains "prod" }}'
```

--------------------------------

### Upgrade image dependencies

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/updating-dependencies.md

Automated command to update container image dependencies used in Helm charts and manifests.

```makefile
make upgrade-image-dependencies
```

--------------------------------

### Cleanup Resources

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/snapshot/manifests/snapshot-import/README.md

Removes the application, snapshot, and class resources from the cluster.

```bash
kubectl delete -f specs/snapshot-import/app
kubectl delete -f specs/snapshot-import/static-snapshot
kubectl delete -f specs/classes
```

--------------------------------

### Monitor pod logs

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/modify-volume/README.md

Stream the logs from the running application pod.

```sh
$ kubectl logs -f app
Mon Feb 26 22:28:19 UTC 2024
Mon Feb 26 22:28:24 UTC 2024
Mon Feb 26 22:28:29 UTC 2024
Mon Feb 26 22:28:34 UTC 2024
Mon Feb 26 22:28:39 UTC 2024
...
```

--------------------------------

### Read data from volume

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/examples/kubernetes/windows/README.md

Verify data persistence by reading the file from a new pod.

```powershell
kubectl delete po windows-server-iis-7c5fc8f6c5-t5mk9

kubectl exec -it windows-server-iis-7c5fc8f6c5-j44qv -- powershell

PS C:\> ls data
```

--------------------------------

### Configure KUBECONFIG for manual testing

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/CONTRIBUTING.md

Sets the KUBECONFIG environment variable to allow access to the test cluster.

```bash
eval "$(make cluster/kubeconfig)"
```

--------------------------------

### Tag the release

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/release.md

Commands to tag a release commit locally and push it to the upstream repository.

```bash
git checkout release-0.7
git pull upstream release-0.7
git tag v0.7.0
git push upstream v0.7.0
```

--------------------------------

### Run external E2E tests

Source: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/CONTRIBUTING.md

Executes the external test suite, with an optional variant for Windows-specific behavior.

```bash
# Normal external tests (excluding Windows tests)
make e2e/external
# Additionally, run the windows tests if changing Windows behavior
make e2e/external-windows
```