### Check Go Installation

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/README.md

Verify if Go is installed and accessible on your system.

```bash
go version
```

--------------------------------

### Check Git Installation

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/README.md

Verify if Git is installed and accessible on your system.

```bash
git --version
```

--------------------------------

### Run verify_manifest_lists.sh Locally

Source: https://github.com/kubernetes/kubeadm/blob/main/tests/e2e/manifests/README.md

Execute the verification script directly on your system. Requires Go, curl, and gcc to be installed.

```bash
./verify_manifest_lists.sh
```

--------------------------------

### Build kinder Tool

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/README.md

Build the kinder tool using Go modules. This command installs kinder into your GOPATH/bin directory.

```bash
cd kubeadm/kinder
GO111MODULE=on go install
```

--------------------------------

### Build a node-image with kind

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/prepare-for-tests.md

Build a node image using the 'kind' tool, starting from a specified base image. This is a common way to create images for testing.

```bash
kind build node-image --base-image kindest/base:latest --image kindest/node:vX
```

--------------------------------

### Explore Available Package Versions

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/testing-pre-releases.md

Use these commands to list available versions of Kubernetes packages in your system's repositories. This is useful before installing a specific version.

```bash
# Ubuntu, Debian or HypriotOS
apt-cache madison <package name>
```

```bash
# CentOS, RHEL or Fedora
yum --showduplicates list <package name>
```

--------------------------------

### Example Prow Job Configuration

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/managing-e2e-tests.md

Defines a sample Prow job for kubeadm e2e tests, specifying its name and execution interval.

```yaml
- name: ci-kubernetes-e2e-kubeadm-foo
  interval: 2h
  ...

```

--------------------------------

### Install Local Packages with apt

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/testing-pre-releases.md

Use this command to install local .deb packages for Kubernetes components on Ubuntu or Debian systems. Ensure the command is executed in the same directory as the .deb files.

```bash
sudo apt install path/to/kubectl.deb path/to/kubeadm.deb path/to/kubelet.deb path/to/kubernetes-cni.deb
```

--------------------------------

### Execute kubeadm init with Kinder

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/reference.md

This action initializes a Kubernetes cluster using kubeadm, installs a CNI plugin, and copies the kubeconfig file to the host. It's designed to be developer-friendly with echoed output and documented steps.

```bash
kinder do kubeadm-init
```

--------------------------------

### File Discovery Kubeconfig Example

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/design/design_v1.8.md

This is an example of a kubeconfig file used for file discovery in kubeadm. It specifies the cluster server and certificate authority data.

```yaml
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <really long certificate data>
    server: https://10.138.0.2:6443
  name: ""
contexts: []
current-context: ""
kind: Config
preferences: {}
users: []

```

--------------------------------

### Enable and Start HAProxy and Keepalived Services

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md

Commands to enable and start the HAProxy and Keepalived services using systemd on RedHat-based systems. These services are essential for load balancing and high availability.

```bash
# systemctl enable haproxy --now
# systemctl enable keepalived --now
```

--------------------------------

### Example Test Failure Output

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/test-failures.md

This is an example of a test failure message indicating an issue during cluster provisioning.

```text
error during make -C /workspace/kubernetes-anywhere WAIT_FOR_KUBECONFIG=y deploy: exit status 2
```

--------------------------------

### Build a base node-image

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/prepare-for-tests.md

Build a base image to be used as a foundation for custom node images. This image serves as the starting point for further customization.

```bash
kinder build base-image --image kindest/base:latest
```

--------------------------------

### Install Specific Package Version

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/testing-pre-releases.md

Commands to install a specific version of a Kubernetes package after exploring available versions. Ensure you replace placeholders with actual package and version names.

```bash
# Ubuntu, Debian or HypriotOS
apt-get install <package name>=<version number>
```

```bash
# CentOS, RHEL or Fedora
yum install <package name>-<version number>
```

--------------------------------

### Initialize Control-Plane Node with kubeadm

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/reference.md

Executes `kubeadm init` on a worker node, useful for custom cluster setups. Use `--ignore-preflight-errors=all` to bypass initial checks.

```bash
# execute a command on the kind-control-plane container (the control-plane node)
docker exec kind-worker1 \
      kubeadm init --config=/kind/kubeadm.conf --ignore-preflight-errors=all
```

--------------------------------

### Update-workflows Configuration Example

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/ci/tools/update-workflows/README.md

A sample configuration file for the update-workflows tool. It defines job groups, test-infra job specifications, and kinder workflow specifications, including versioning and templating details.

```yaml
# a job group is an object that contain multiple e2e tests for different version
jobGroups:
- name: foo # name of the job group
  # this means that the feature requires a minimum Kubernetes versions;
  # jobs for a Kubernetes version older than this value are not generated
  minimumKubernetesVersion: v1.19.0
  # contains settings for the test-infra job format
  testInfraJobSpec:
    # file name in test-infra to contain all jobs
    targetFile: kubeadm-kinder-foo.yaml
    # test-infra template
    template: ./templates/testinfra/kubeadm-kinder-foo.yaml
  kinderWorkflowSpec:
    # a file format to be use when writing kinder workflows
    targetFile: foo-{{ .KubernetesVersion }}.yaml
    # path to a kinder template
    template: ./templates/workflows/foo.yaml
    # additional files to copy to the workflow output directory
    additionalFiles:
    - ./templates/workflows/foo-tasks.yaml
  # a list of job objects containing version skew between k8s, kubelet and kubeadm version
  jobs:
  - kubernetesVersion: latest # 'latest' is interpreted as 'latest' version labels and main git branches
    kubeletVersion: -1 # 'kubernetes-version - 1'
    # skip versions 'kubernetes-version' and 'kubernetes-version - 1' (specific for kubelet skew tests)
    skipVersions: [ 0, -1 ]
  - kubernetesVersion: 0 # 'kubernetes-version'
    kubeletVersion: +1 # 'kubernetes-version + 1'
```

--------------------------------

### Example kube-vip ARP Manifest

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md

An example YAML manifest for a kube-vip pod configured for ARP mode, specifying interface, IP address, control plane, services, and leader election settings.

```yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  name: kube-vip
  namespace: kube-system
spec:
  containers:
  - args:
    - manager
    env:
    - name: vip_arp
      value: "true"
    - name: port
      value: "6443"
    - name: vip_interface
      value: ens192
    - name: vip_cidr
      value: "32"
    - name: cp_enable
      value: "true"
    - name: cp_namespace
      value: kube-system
    - name: vip_ddns
      value: "false"
    - name: vip_leaderelection
      value: "true"
    - name: vip_leaseduration
      value: "5"
    - name: vip_renewdeadline
      value: "3"
    - name: vip_retryperiod
      value: "1"
    - name: vip_address
      value: 192.168.0.40
    image: ghcr.io/kube-vip/kube-vip:v0.4.0
    imagePullPolicy: Always
    name: kube-vip
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
        - NET_RAW
        - SYS_TIME
    volumeMounts:
    - mountPath: /etc/kubernetes/admin.conf
      name: kubeconfig
  hostAliases:
  - hostnames:
    - kubernetes
    ip: 127.0.0.1
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/admin.conf
    name: kubeconfig
status: {}
```

--------------------------------

### Create and Initialize Cluster

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/test-XonY.md

Creates a new kinder cluster using a specified node image and initializes the bootstrap control plane. This is the starting point for testing Kubeadm's cluster management capabilities.

```bash
# create a cluster (choose the desired number of control-plane/worker nodes)
kinder create cluster --image kindest/node:vX.on.Y --control-plane-nodes 1 --worker-nodes 0

# initialize the bootstrap control plane
kinder do kubeadm-init

# join secondary control planes and nodes (if any)
kinder do kubeadm-join
```

--------------------------------

### Update Workflows Script Example

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/managing-e2e-tests.md

Illustrates how to update kubeadm test workflows and test-infra jobs using a shell script. Requires setting the TEST_INFRA_KUBEADM_PATH environment variable.

```bash
#!/bin/bash

# ... other setup ...

export TEST_INFRA_KUBEADM_PATH="$(pwd)/../test-infra"

# Update workflows for a new release
# MINOR=... # incremented
# SKEW_SIZE=N+1 # incremented
# ./hack/update-workflows.sh

# Update workflows when oldest release goes out of support
# SKEW_SIZE=N # reset to N
# ./hack/update-workflows.sh

```

--------------------------------

### Replace kubeadm Binary

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/testing-pre-releases.md

If you have a local kubeadm binary, copy it to /usr/bin/kubeadm to override the installed version. Alternatively, use the full path to the local binary when running kubeadm commands.

```bash
cp path/to/local/kubeadm /usr/bin/kubeadm
```

--------------------------------

### Check kubeadm version on all nodes

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/reference.md

Uses `kinder exec` to run the `kubeadm version` command on all nodes within the cluster. This is useful for verifying the installed kubeadm version across your cluster.

```bash
# check the kubeadm version on all the nodes
kinder exec @all -- kubeadm version
```

--------------------------------

### Test Kubeadm Phases with Kinder

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/test-HA.md

Use the `--use-phases` flag with `kubeadm-init` and/or `kubeadm-join` to test specific phases of the kubeadm process, allowing for granular control and debugging.

```bash
# 2. add `--use-phases` flag to `kubeadm-init` and/or `kubeadm-join` to test phases
```

--------------------------------

### Retrieve kubeadm Binary

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/testing-pre-releases.md

Download a pre-compiled kubeadm binary from a release bucket. Replace `{release}` with the desired Kubernetes version. The command also sets execute permissions.

```bash
curl -L https://dl.k8s.io/release/{release}/bin/linux/amd64/kubeadm && chown +x kubeadm
```

--------------------------------

### Retrieve Node Port for Service

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/testing-pre-releases.md

Get the dynamically assigned NodePort for a service. This is necessary to access the service externally.

```bash
NODE_PORT=$(kubectl get svc nginx \
  --output=jsonpath='{range .spec.ports[0]}{.nodePort}')
```

--------------------------------

### Build a node-image variant with init artifacts

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/prepare-for-tests.md

Create a custom node image variant by adding Kubernetes initialization artifacts, such as a specific kubeadm binary, to an existing base image.

```bash
kinder build node-image-variant \
     --base-image kindest/base:vX \
     --image kindest/node:vX-variant \
     --with-init-artifacts $mylocalbinary/kubeadm
```

--------------------------------

### Get Nginx Pod Name

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/testing-pre-releases.md

Retrieves the full name of the first nginx pod created by the deployment. This name is required for port forwarding operations.

```bash
POD_NAME=$(kubectl get pods -l run=nginx -o jsonpath="{.items[0].metadata.name}")
```

--------------------------------

### Build kubeadm Binary

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/testing-pre-releases.md

Build the kubeadm binary for the target platform. This command is run from the Kubernetes source directory.

```bash
cd ${GOPATH}/src/k8s.io/kubernetes

# build kubeadm binary for the target platform used by the machines
make all WHAT=cmd/kubeadm

# To cross build for linux-amd64 from mac
make all WHAT=cmd/kubeadm KUBE_BUILD_PLATFORMS=linux/amd64
```

--------------------------------

### Build Node Image Variant with Init Artifacts

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/reference.md

Builds a node image variant that includes specific Kubernetes init artifacts. This prepares the image for `kinder do kubeadm-init`.

```bash
kinder build node-image-variant \
     --base-image kindest/base:latest \
     --image kindest/node:PR12345 \
     --with-init-artifacts v1.14.0
```

```bash
kinder build node-image-variant \
     --base-image kindest/base:latest \
     --image kindest/node:PR12345 \
     --with-init-artifacts $my-local-packages/v1.12.2/
```

--------------------------------

### Customize node-image by adding pre-loaded images

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/prepare-for-tests.md

Build a node image variant that includes additional container images, pre-loaded into the `/kind/images` folder. This ensures these images are available on nodes at cluster creation.

```bash
kinder build node-image-variant \
     --base-image kindest/node:vX \
     --image kindest/node:vX-variant \
     --with-images $mylocalimages/nginx.tar
```

--------------------------------

### Run kube-vip with Containerd

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md

Alias to pull the kube-vip image and run it using containerd in host network mode.

```bash
alias kube-vip="ctr image pull ghcr.io/kube-vip/kube-vip:$KVVERSION; ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KVVERSION vip /kube-vip"
```

--------------------------------

### Get Busybox Pod Name

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/testing-pre-releases.md

Retrieves the full name of the first busybox pod created by the deployment. This name is then used for executing commands within the pod.

```bash
POD_NAME=$(kubectl get pods -l run=busybox -o jsonpath="{.items[0].metadata.name}")
```

--------------------------------

### Test Automatic Certificate Copying with Kinder

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/test-HA.md

Enable automatic certificate copying for HA setups by adding the `--copy-certs=auto` flag to both `kubeadm-init` and `kubeadm-join` commands.

```bash
# 3. add `--copy-certs=auto` flag both to `kubeadm-init` and `kubeadm-join` to test the automatic copy certs feature
```

--------------------------------

### Create and Initialize kinder Cluster

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/test-upgrades.md

Use this command to create a new kinder cluster with a specified node image and number of control-plane/worker nodes. Followed by initializing the bootstrap control plane.

```bash
# create a cluster (choose the desired number of control-plane/worker nodes)
kinder create cluster --image kindest/node:vX.to.Y --control-plane-nodes 1 --worker-nodes 0

# initialize the bootstrap control plane
kinder do kubeadm-init

# join secondary control planes and nodes (if any)
kinder do kubeadm-join
```

--------------------------------

### HAProxy Configuration for API Server

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md

Configure haproxy to proxy traffic to the control plane nodes. This setup uses TCP mode for the API server frontend.

```cfg
# /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global 
    log stdout format raw local0
    daemon

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 1
    timeout http-request    10s
    timeout queue           20s
    timeout connect         5s
    timeout client          35s
    timeout server          35s
    timeout http-keep-alive 10s
    timeout check           10s

#---------------------------------------------------------------------
# apiserver frontend which proxys to the control plane nodes
#---------------------------------------------------------------------
frontend apiserver
    bind *:${APISERVER_DEST_PORT}
    mode tcp
    option tcplog
    default_backend apiserverbackend

#---------------------------------------------------------------------
```

--------------------------------

### Run Kubeadm Unit Tests

Source: https://github.com/kubernetes/kubeadm/blob/main/CONTRIBUTING.md

Execute unit tests for specific Kubeadm packages. Use GOFLAGS='-v' for verbose output.

```bash
make test WHAT=<package> GOFLAGS="-v"
```

--------------------------------

### Get Latest kube-vip Version from GitHub API

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md

Fetch the latest release version of kube-vip from the GitHub API using curl and jq. Alternatively, set the version manually.

```bash
KVVERSION=$(curl -sL https://api.github.com/repos/kube-vip/kube-vip/releases | jq -r ".[0].name")
```

```bash
export KVVERSION=vx.x.x
```

--------------------------------

### Test External etcd with Kinder

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/test-HA.md

Add the `--external-etcd` flag to `kinder create cluster` to test the integration and functionality of an external etcd cluster with your HA Kubernetes setup.

```bash
# 1. add `--external-etcd` flag to `kinder create cluster` to test usage of external etcd cluster
```

--------------------------------

### Include Generated Kubeadm Command Documentation

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/updating-command-reference.md

When adding a new sub-command, import its generated documentation file into the parent command's Markdown file using the {{< include >}} shortcode. This acts as a placeholder until overwritten by generated files.

```markdown
## kubeadm config view {#cmd-config-newcommand}
{{< include "generated/kubeadm_config_newcommand.md" >}}
```

--------------------------------

### Direct Kubeadm Control with Docker

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/test-HA.md

As an alternative to `kinder do`, use `docker exec` and `docker cp` to directly invoke `kubeadm`, `kubectl`, or other shell commands on cluster nodes for advanced control.

```bash
# As alternative, instead of using kinder pre-defined actions with `kinder do`, it is possible to
# use `docker exec` and `docker cp` to work on nodes invoking directly `kubeadm`, `kubectl` or
# any other shell commands.
```

--------------------------------

### Build Node Image Variant with Pre-loaded Images

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/reference.md

Builds a node image variant that includes pre-loaded container images. These images will be available on all nodes at cluster creation time.

```bash
kinder build node-image-variant \
     --base-image kindest/node:latest \
     --image kindest/node:PR12345 \
     --with-images v1.14.0
```

```bash
kinder build node-image-variant \
     --base-image kindest/node:latest \
     --image kindest/node:PR12345 \
     --with-images $my-local-images/nginx.tar
```

--------------------------------

### Copy Kubeadm Config to Host

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/reference.md

Copies the kubeadm configuration file from the bootstrap control-plane node to the host machine.

```bash
kinder cp @cp1:kind/kubeadm.conf kubeadm.conf
```

--------------------------------

### Run Kubeadm Command Line Integration Tests

Source: https://github.com/kubernetes/kubeadm/blob/main/CONTRIBUTING.md

Execute command-line integration tests for Kubeadm. This command verifies the behavior of Kubeadm commands.

```bash
make test-cmd WHAT=kubeadm
```

--------------------------------

### Log File Paths for Debugging

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/test-failures.md

These are the paths to log files that can be used to debug cluster provisioning failures.

```text
/artifacts/control-plane-node-name/serial-1.log
```

```text
/artifacts/node-name/serial-1.log
```

--------------------------------

### Create Busybox Deployment

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/testing-pre-releases.md

Creates a busybox deployment that runs indefinitely, useful for testing DNS resolution within the cluster. The pod is configured to sleep for an extended period.

```bash
kubectl run busybox --image=busybox --command -- sleep 3600
```

--------------------------------

### Customize node-image for upgrade testing

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/prepare-for-tests.md

Create a node image variant that includes artifacts for a second Kubernetes version, stored in the `/kinder/upgrades` folder. This is specifically for testing Kubernetes upgrade scenarios.

```bash
kinder build node-image-variant \
     --base-image kindest/node:vX \
     --image kindest/node:vX-variant \
     --with-upgrade-artifacts $mylocalbinaries/vY
```

--------------------------------

### Clone kubeadm Repository

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/README.md

Clone the kubeadm repository to access the kinder tool and its source code.

```bash
git clone https://github.com/kubernetes/kubeadm.git
```

--------------------------------

### Build Node Image Variant with Upgrade Artifacts

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/reference.md

Use this command to build a custom node image variant that includes specific upgrade artifacts. Specify the base image, the desired output image tag, and the upgrade artifacts to include. Artifacts can be sourced from a local path or a remote repository.

```bash
kinder build node-image-variant \
     --base-image kindest/node:latest \
     --image kindest/node:PR12345 \
     --with-upgrade-artifacts v1.14.0
```

```bash
kinder build node-image-variant \
     --base-image kindest/node:latest \
     --image kindest/node:PR12345 \
     --with-upgrade-artifacts $my-local-packages/v1.12.2/
```

--------------------------------

### Build Node Image Variant with Custom Kubeadm Binary

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/reference.md

Builds a node image variant with a custom kubeadm binary. This is useful for testing specific kubeadm versions or local builds.

```bash
kinder build node-image-variant \
     --base-image kindest/node:latest \
     --image kindest/node:PR12345 \
     --with-kubeadm v1.13.5
```

```bash
kinder build node-image-variant \
     --base-image kindest/node:latest \
     --image kindest/node:PR12345 \
     --with-kubeadm $working_dir/kubernetes/_output/local/bin/linux/amd64/kubeadm
```

--------------------------------

### Build Node Image Variant for Upgrades

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/test-upgrades.md

Create a variant of a base node image that includes artifacts for a target Kubernetes version. This is essential for testing upgrades.

```bash
kinder build node-image-variant --base-image kindest/node:vX --image kindest/node:vX.to.Y \
    --with-upgrade-artifacts $artifacts/binaries
```

--------------------------------

### Initialize Control Plane with Kubeadm

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/getting-started.md

Initializes the bootstrap control plane for your Kubernetes cluster using kubeadm. This is a core step after cluster creation.

```bash
# initialize the bootstrap control plane
kinder do kubeadm-init
```

--------------------------------

### Create a Kinder Cluster

Source: https://github.com/kubernetes/kubeadm/blob/main/kinder/doc/getting-started.md

Use this command to create a new Kubernetes cluster with a specified number of worker nodes. Ensure you have a compatible node image ready.

```bash
# create a cluster with eventually two worker nodes
kind create cluster --image kindest/node:vX --worker-nodes 2
```

--------------------------------

### Expose Deployment with NodePort Service

Source: https://github.com/kubernetes/kubeadm/blob/main/docs/testing-pre-releases.md

Expose a deployment using a NodePort service. This makes the application accessible externally via a node's IP and a specific port.

```bash
kubectl expose deployment nginx --port 80 --type NodePort
```