### Vagrant Cluster Setup and Access

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/developers/vagrant.md

This example demonstrates the workflow for setting up a Kubespray cluster using Vagrant. It includes steps for creating and activating a Python virtual environment, installing requirements, provisioning the cluster with `vagrant up`, and configuring `KUBECONFIG` to access the cluster.

```shell
# use virtualenv to install all python requirements
VENVDIR=venv
$ virtualenv --python=/usr/bin/python3.7 $VENVDIR
$ source $VENVDIR/bin/activate
$ pip install -r requirements.txt

$ vagrant up

# Access the cluster
$ export INV=.vagrant/provisioners/ansible/inventory
$ export KUBECONFIG=${INV}/artifacts/admin.conf
```

--------------------------------

### Cgroups Hierarchy Example

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operations/cgroups.md

Illustrates the cgroups hierarchy after setup, showing the organization of pods, kube services, and system services.

```bash
/ (Cgroups Root)
├── kubepods.slice
│   ├── ...
│   ├── kubepods-besteffort.slice
│   ├── kubepods-burstable.slice
│   └── ...
├── kube.slice
│   ├── ...
│   ├── {{container_manager}}.service
│   ├── kubelet.service
│   └── ...
├── system.slice
│   └── ...
└── ...
```

--------------------------------

### Start Vagrant Environment

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/README.md

Initiates the Vagrant environment for cluster deployment. Ensure Ansible is installed according to the official guide.

```ShellSession
vagrant up
```

--------------------------------

### Example Ansible Inventory Structure

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operations/integration.md

This example shows how to structure your Ansible inventory file to map your existing host groups to Kubespray's required group names for cluster setup.

```INI
...
#Kubespray groups:
[kube_node:children]
kubenode

[etcd:children]
kubemaster
cubemaster-ha

[kube_control_plane:children]
kubemaster
cubemaster-ha
...
```

--------------------------------

### Install Ansible on openSUSE

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operating_systems/opensuse.md

Use zypper to refresh repositories and install the Ansible package.

```ShellSession
sudo zypper ref
sudo zypper -n install ansible
```

--------------------------------

### Set up Python virtual environment

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/CONTRIBUTING.md

Initializes a virtual environment and installs required development dependencies.

```ShellSession
virtualenv venv
source venv/bin/activate
pip install -r tests/requirements.txt
```

--------------------------------

### Instance List Output

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/getting_started/setting-up-your-first-cluster.md

Example output showing the status and IP addresses of provisioned instances.

```ShellSession
NAME          ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS
controller-0  us-west1-c  e2-standard-2               10.240.0.10  XX.XX.XX.XXX   RUNNING
controller-1  us-west1-c  e2-standard-2               10.240.0.11  XX.XXX.XXX.XX  RUNNING
controller-2  us-west1-c  e2-standard-2               10.240.0.12  XX.XXX.XX.XXX  RUNNING
worker-0      us-west1-c  e2-standard-2               10.240.0.20  XX.XX.XXX.XXX  RUNNING
worker-1      us-west1-c  e2-standard-2               10.240.0.21  XX.XX.XX.XXX   RUNNING
worker-2      us-west1-c  e2-standard-2               10.240.0.22  XX.XXX.XX.XX   RUNNING
```

--------------------------------

### Install Python dependencies on openSUSE

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operating_systems/opensuse.md

Install the required Jinja2 and netaddr Python libraries via zypper.

```ShellSession
sudo zypper -n install python-Jinja2 python-netaddr
```

--------------------------------

### Verify Metrics Server Installation

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/getting_started/setting-up-your-first-cluster.md

Use this command to check if the metrics server addon is installed and functioning correctly by displaying node resource utilization.

```ShellSession
kubectl top nodes
```

--------------------------------

### Install Cloudflare PKI/TLS cfssl Toolkit

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/advanced/cert_manager.md

Install the `golang-cfssl` package on Ubuntu/Debian distributions to use the `cfssl` toolkit for managing PKI and TLS certificates.

```shell
sudo apt-get install -y golang-cfssl
```

--------------------------------

### Install Kubernetes Cluster

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/getting_started/getting-started.md

Execute the cluster playbook to deploy the Kubernetes environment.

```ShellSession
ansible-playbook -i inventory/mycluster/ cluster.yml -b -v \
  --private-key=~/.ssh/private_key
```

--------------------------------

### Containerd Log Format Example

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/upgrades/migrate_docker2containerd.md

Example of the log format produced by containerd for log aggregation configuration.

```text
2020-01-10T18:10:40.01576219Z stdout F application log message...
```

--------------------------------

### Install Ansible Dependencies

Source: https://context7.com/kubernetes-sigs/kubespray/llms.txt

Set up a Python virtual environment and install required dependencies for running Kubespray.

```bash
# Create and activate Python virtual environment
python3 -m venv venv
source venv/bin/activate

# Clone and checkout Kubespray
git clone https://github.com/kubernetes-sigs/kubespray.git
cd kubespray

# Install Ansible and dependencies
pip install -r requirements.txt
```

--------------------------------

### Execute Kubespray Playbook and Cleanup

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/upgrades/migrate_docker2containerd.md

Run the cluster playbook to install containerd and perform post-migration cleanup.

```commandline
ansible-playbook -i inventory/sample/hosts.ini cluster.yml --limit=NODENAME
```

```commandline
rm -fr /var/lib/docker
```

```commandline
crictl ps -a
```

--------------------------------

### Verify and Use Kata RuntimeClass

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/CRI/kata-containers.md

Commands to verify the generated runtime class and an example Pod specification using the kata-qemu runtime.

```shell
$ kubectl get runtimeclass
NAME        HANDLER     AGE
kata-qemu   kata-qemu   3m34s
$ 
$ cat nginx.yaml
apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  runtimeClassName: kata-qemu
  containers:
  - name: nginx
    image: nginx:1.14.2
$ 
$ kubectl apply -f nginx.yaml
```

--------------------------------

### Create and Activate Virtual Environment

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/getting_started/setting-up-your-first-cluster.md

Sets up a new Python virtual environment and activates it for installing Kubespray dependencies.

```ShellSession
python3 -m venv venv
source venv/bin/activate
```

--------------------------------

### Example Playbook for bootstrap_os Role

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operating_systems/bootstrap-os.md

This playbook demonstrates how to apply the bootstrap_os role to all hosts. It's recommended to disable fact gathering as Python might not be present on all hosts initially.

```yaml
- hosts: all
  gather_facts: false  # not all hosts might be able to run modules yet
  roles:
    - kubespray_defaults
    - bootstrap_os

```

--------------------------------

### Install virtctl Binary

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/developers/ci-setup.md

Detects the current KubeVirt version from the cluster and downloads the corresponding virtctl binary for the host architecture.

```bash
#!/bin/bash

VERSION=$(kubectl get kubevirt.kubevirt.io/kubevirt -n kubevirt -o=jsonpath="{.status.observedKubeVirtVersion}")
ARCH=$(uname -s | tr A-Z a-z)-$(uname -m | sed 's/x86_64/amd64/') || windows-amd64.exe
echo ${ARCH}
curl -L -o virtctl https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/virtctl-${VERSION}-${ARCH}
chmod +x virtctl
sudo install virtctl /usr/local/bin
```

--------------------------------

### Configure and run pre-commit hooks

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/CONTRIBUTING.md

Installs pre-commit hooks and executes them against the repository files to ensure code quality.

```ShellSession
pre-commit install
pre-commit run -a  # To run pre-commit hook on all files in the repository, even if they were not modified
```

--------------------------------

### Install Kubespray Dependencies

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/scripts/openstack-cleanup/README.md

Installs the necessary dependencies for the openstack-cleanup tool using pip. It's recommended to run this before using the tool.

```shell
pip install -r requirements.txt
```

--------------------------------

### Install Kubespray Python Packages from Internal PyPi Server (All Packages)

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operations/offline-environment.md

Install all required Python packages for Kubespray from a private PyPi server. Ensure the server URL is correct.

```bash
pip install -i https://pypiserver/pypi -r requirements.txt
```

--------------------------------

### Example Kubeconfig Server Address

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/getting_started/setting-up-your-first-cluster.md

Illustrates the format of the server address within a kubeconfig file, showing an example of an external IP address for the API server.

```yaml
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: XXX
    server: https://35.205.205.80:6443
  name: cluster.local
...
```

--------------------------------

### Select Kata Containers Version

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/CRI/kata-containers.md

Specify a particular release version of Kata Containers to install.

```yaml
kata_containers_version: 2.2.2
```

--------------------------------

### Check cluster status and version

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operations/upgrades.md

Verify current node status and repository tag before starting an upgrade.

```ShellSession
$ kubectl get node
NAME      STATUS    ROLES         AGE       VERSION
apollo    Ready     master,node   1h        v1.10.4
boomer    Ready     master,node   42m       v1.10.4
caprica   Ready     master,node   42m       v1.10.4

$ git describe --tags
v2.6.0

$ git tag
...
v2.6.0
v2.7.0
v2.8.0
v2.8.1
v2.8.2
...

$ git checkout v2.7.0
Previous HEAD position was 8b3ce6e4 bump upgrade tests to v2.5.0 commit (#3087)
HEAD is now at 05dabb7e Fix Bionic networking restart error #3430 (#3431)
```

--------------------------------

### Define inventory structure for control plane nodes

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operations/nodes.md

Example inventory configurations showing the order of control plane and etcd nodes.

```yaml
all:
  hosts:
  children:
    kube_control_plane:
      hosts:
        node-1:
        node-2:
        node-3:
    kube_node:
      hosts:
        node-1:
        node-2:
        node-3:
    etcd:
      hosts:
        node-1:
        node-2:
        node-3:
```

```yaml
all:
  hosts:
  children:
    kube_control_plane:
      hosts:
        node-2:
        node-3:
        node-1:
    kube_node:
      hosts:
        node-2:
        node-3:
        node-1:
    etcd:
      hosts:
        node-2:
        node-3:
        node-1:
```

--------------------------------

### Configure API Server Rate Limiting

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ansible/vars.md

Example configuration for API server rate limiting parameters.

```yaml
    qps: 50
    burst: 100
    cache_size: 2000
  limit_2:
    type: User
    qps: 50
    burst: 100
  ...
```

--------------------------------

### Configure GCP Credentials for CSI Driver

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/CSI/gcp-pd-csi.md

Commands to authenticate with GCP and execute the setup script to generate the required service account credentials.

```ShellSession
# This will open a web page for you to authenticate
gcloud auth login
export PROJECT=nameofmyproject
gcloud config set project $PROJECT

git clone https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver $GOPATH/src/sigs.k8s.io/gcp-compute-persistent-disk-csi-driver

export GCE_PD_SA_NAME=my-gce-pd-csi-sa
export GCE_PD_SA_DIR=/my/safe/credentials/directory

./deploy/setup-project.sh
```

--------------------------------

### Configure EventRateLimit Admission Plugin

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ansible/vars.md

Example configuration for the EventRateLimit admission plugin in kube-apiserver. This structure defines rate limiting rules based on namespaces.

```yaml
kube_apiserver_admission_event_rate_limits:
  limit_1:
    type: Namespace
```

--------------------------------

### Customize OpenStack Instances with Cloud-Init

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/openstack/README.md

Modify the cloud-init template to customize OpenStack instances before provisioning. This example enables novnc console access and root SSH access.

```yaml
#cloud-config
## in some cases novnc console access is required
## it requires ssh password to be set
ssh_pwauth: yes
chpasswd:
  list: |
    root:secret
  expire: False

## in some cases direct root ssh access via ssh key is required
disable_root: false
```

--------------------------------

### Generate Release Notes

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/RELEASE.md

Use this command to generate release notes by specifying the start and end commit SHAs. Ensure the GITHUB_TOKEN and ORG/REPO are set correctly. Set required-author to "" to include all authors.

```shell
export GITHUB_TOKEN=<your-github-token>
export ORG=kubernetes-sigs
export REPO=kubespray
release-notes --start-sha <The start commit-id> --end-sha <The end commit-id> --dependencies=false --output=/tmp/kubespray-release-note --required-author=""
```

--------------------------------

### Create Pod with Additional Network Interface

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/CNI/multus.md

Annotate a pod with `k8s.v1.cni.cncf.io/networks` pointing to the `NetworkAttachmentDefinition` name ('macvlan-conf' in this example) to attach the pod to the specified additional network.

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: samplepod
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
  containers:
  - name: samplepod
    command: ["/bin/bash", "-c", "sleep 2000000000000"]
    image: dougbtv/centos-network
```

--------------------------------

### Force Immediate NTP Time Synchronization

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/advanced/ntp.md

Force an immediate time synchronization using NTP immediately after installation. This is particularly useful on newly installed systems to ensure accurate time from the start.

```ShellSession
ntp_force_sync_immediately: true
```

--------------------------------

### Example HAProxy Configuration for External Load Balancer

Source: https://context7.com/kubernetes-sigs/kubespray/llms.txt

Example HAProxy configuration for an external load balancer providing high availability for the Kubernetes API server. This setup uses TCP mode and round-robin balancing.

```text
# Example HAProxy configuration for external LB
listen kubernetes-apiserver-https
  bind 10.0.0.100:8383
  mode tcp
  option log-health-checks
  timeout client 3h
  timeout server 3h
  server master1 10.240.0.10:6443 check check-ssl verify none inter 10000
  server master2 10.240.0.11:6443 check check-ssl verify none inter 10000
  server master3 10.240.0.12:6443 check check-ssl verify none inter 10000
  balance roundrobin
```

--------------------------------

### Kubespray Inventory Example

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ansible/inventory.md

Defines node assignments to control plane, etcd, and worker nodes. Use 'ansible_host' to specify IP addresses and 'ip' for internal service binding.

```ini
## Configure 'ip' variable to bind kubernetes services on a
## different ip than the default iface
node1 ansible_host=95.54.0.12 ip=10.3.0.1
node2 ansible_host=95.54.0.13 ip=10.3.0.2
node3 ansible_host=95.54.0.14 ip=10.3.0.3
node4 ansible_host=95.54.0.15 ip=10.3.0.4
node5 ansible_host=95.54.0.16 ip=10.3.0.5
node6 ansible_host=95.54.0.17 ip=10.3.0.6

[kube_control_plane]
node1
node2

[etcd]
node1
node2
node3

[kube_node]
node2
node3
node4
node5
node6

```

--------------------------------

### Create Fedora CoreOS Guest with Virt-Install

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operating_systems/fcos.md

Use virt-install to create a Fedora CoreOS virtual machine. Ensure the ignition file is served via HTTP.

```shell
machine_name=myfcos1
ignition_url=http://mywebserver/fcos.ign

fcos_version=34.20210611.3.0
kernel=https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/${fcos_version}/x86_64/fedora-coreos-${fcos_version}-live-kernel-x86_64
initrd=https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/${fcos_version}/x86_64/fedora-coreos-${fcos_version}-live-initramfs.x86_64.img
rootfs=https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/${fcos_version}/x86_64/fedora-coreos-${fcos_version}-live-rootfs.x86_64.img
kernel_args="console=ttyS0 coreos.live.rootfs_url=${rootfs} coreos.inst.install_dev=/dev/sda coreos.inst.stream=stable coreos.inst.ignition_url=${ignition_url}"
sudo virt-install --name ${machine_name} --ram 4048 --graphics=none --vcpus 2 --disk size=20 \
                --network bridge=virbr0 \
                --install kernel=${kernel},initrd=${initrd},kernel_args_overwrite=yes,kernel_args="${kernel_args}"
```

--------------------------------

### Kubernetes Node Status Output

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/getting_started/setting-up-your-first-cluster.md

Example output from `kubectl get nodes`, showing the status, roles, age, and Kubernetes version of each node in the cluster.

```text
NAME           STATUS   ROLES    AGE   VERSION
controller-0   Ready    master   47m   v1.17.9
controller-1   Ready    master   46m   v1.17.9
controller-2   Ready    master   46m   v1.17.9
worker-0       Ready    <none>   45m   v1.17.9
worker-1       Ready    <none>   45m   v1.17.9
worker-2       Ready    <none>   45m   v1.17.9
```

--------------------------------

### Deploy Infrastructure with Terraform

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/vsphere/README.md

Initialize the Terraform provider and apply the configuration to provision the infrastructure.

```bash
terraform init ../../contrib/terraform/vsphere
terraform apply \
    -var-file default.tfvars \
    -state=tfstate-$CLUSTER.tfstate \
    ../../contrib/terraform/vsphere
```

--------------------------------

### Kubespray Inventory with Standalone Route Reflectors

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/CNI/calico.md

Example Kubespray inventory configuration to deploy standalone BGP route reflectors for optimizing Calico's BGP topology at large scale. This setup includes defining `calico_rr` group and `cluster_id`.

```ini
[all]
rr0 ansible_ssh_host=10.210.1.10 ip=10.210.1.10
rr1 ansible_ssh_host=10.210.1.11 ip=10.210.1.11
node2 ansible_ssh_host=10.210.1.12 ip=10.210.1.12
node3 ansible_ssh_host=10.210.1.13 ip=10.210.1.13
node4 ansible_ssh_host=10.210.1.14 ip=10.210.1.14
node5 ansible_ssh_host=10.210.1.15 ip=10.210.1.15

[kube_control_plane]
node2
node3

[etcd]
node2
node3
node4

[kube_node]
node2
node3
node4
node5

[calico_rr]
rr0
rr1

[rack0]
rr0
rr1
node2
node3
node4
node5

[rack0:vars]
cluster_id="1.0.0.1"
calico_rr_id=rr1
calico_group_id=rr1

```

--------------------------------

### Install and Configure Cilium Hubble

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/CNI/cilium.md

Enable Hubble support in Cilium, install Hubble Relay and UI, and generate necessary certificates. After installation, use kubectl port-forward to access the Hubble UI.

```yaml
cilium_enable_hubble: true ## enable support hubble in cilium
cilium_hubble_install: true ## install hubble-relay, hubble-ui
cilium_hubble_tls_generate: true ## install hubble-certgen and generate certificates
```

```shell script
kubectl port-forward -n kube-system svc/hubble-ui 12000:80
```

--------------------------------

### Provision Infrastructure

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/exoscale/README.md

Initialize and apply Terraform configurations to create the cluster infrastructure.

```bash
terraform init ../../contrib/terraform/exoscale
terraform apply -var-file default.tfvars ../../contrib/terraform/exoscale
```

--------------------------------

### Prepare and Deploy Air-Gapped Cluster

Source: https://context7.com/kubernetes-sigs/kubespray/llms.txt

Generate artifact lists for offline environments and execute the deployment playbook.

```bash
# Generate artifact lists for offline preparation
cd contrib/offline
./generate_list.sh

# Output files created:
# - files.list (binaries to download)
# - images.list (container images to mirror)

# Deploy in air-gapped environment
ansible-playbook -i inventory/my_airgap_cluster/hosts.yaml \
  -b cluster.yml
```

--------------------------------

### Run Vagrant Docker Image

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/test-infra/vagrant-docker/README.md

Executes the Vagrant container with libvirt socket mapping and initializes the environment.

```console
$ docker run --net host --rm -it -v /var/run/libvirt/libvirt-sock:/var/run/libvirt/libvirt-sock quay.io/kubespray/vagrant
$ vagrant up
Bringing machine 'k8s-1' up with 'libvirt' provider...
Bringing machine 'k8s-2' up with 'libvirt' provider...
Bringing machine 'k8s-3' up with 'libvirt' provider...
[...]
```

--------------------------------

### Install Ansible in a Python Virtual Environment

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ansible/ansible.md

Installs Ansible and its requirements within a dedicated Python virtual environment for Kubespray. Ensure you are in the Kubespray directory before running.

```ShellSession
VENVDIR=kubespray-venv
KUBESPRAYDIR=kubespray
python3 -m venv $VENVDIR
source $VENVDIR/bin/activate
cd $KUBESPRAYDIR
pip install -r requirements.txt
```

--------------------------------

### Run Kubespray Cluster Upgrade

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operations/upgrades.md

Execute the Ansible playbook to upgrade the Kubernetes cluster. Ensure you have installed necessary dependencies like `pip3 install -r requirements.txt`.

```bash
ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
```

--------------------------------

### Install Specific Kubespray Python Packages from Internal PyPi Server

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operations/offline-environment.md

Install only the missing Python packages from a private PyPi server. Replace `package_you_miss` with the actual package name.

```bash
pip install -i https://pypiserver/pypi package_you_miss
```

--------------------------------

### Initialize Terraform Configuration

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/hetzner/README.md

Initialize the Terraform working directory for the Hetzner provider. Ensure you are in the correct directory before running this command.

```bash
terraform -chdir=./contrib/terraform/hetzner/ init
```

--------------------------------

### Install Kubespray Python Packages via Proxy

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operations/offline-environment.md

Install required Python packages for Kubespray using pip with an HTTP(S) proxy. Replace `proxyserver:port` with your proxy details.

```bash
sudo pip install --proxy=https://[username:password@]proxyserver:port -r requirements.txt
```

--------------------------------

### Select Cilium Version

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/CNI/cilium.md

Specifies the desired version of Cilium to be installed.

```yaml
cilium_version: "1.19.3"
```

--------------------------------

### Deploy Kubernetes with Kubespray

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/hetzner/README.md

Set up the Kubernetes cluster using Kubespray with the generated inventory file. The -b flag enables become (sudo) and -v increases verbosity.

```bash
ansible-playbook -i inventory.ini ../../cluster.yml -b -v
```

--------------------------------

### Configure Kubelet CPU Manager Policy Options

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ansible/vars.md

Define CPU manager policy options as a dictionary. Ensure boolean values are provided as strings.

```yaml
kubelet_cpu_manager_policy_options:
    distribute-cpus-across-numa: "true"
    full-pcpus-only: "true"
```

--------------------------------

### Add Extra Configuration Variables

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/CNI/cilium.md

Injects additional configuration variables into the Cilium setup.

```yaml
cilium_config_extra_vars:
  enable-endpoint-routes: true
```

--------------------------------

### Build and push Kubespray container image

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/RELEASE.md

Use these commands to build the main Kubespray image from the root directory and push it to the registry.

```shell
cd kubespray/
nerdctl build -t quay.io/kubespray/kubespray:vX.Y.Z .
nerdctl push quay.io/kubespray/kubespray:vX.Y.Z
```

--------------------------------

### Install Ansible collections

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ansible/ansible_collection.md

Use the ansible-galaxy command to download the collections defined in your requirements file.

```ShellSession
ansible-galaxy install -r requirements.yml
```

--------------------------------

### Deploy Kubernetes with Ansible

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/gcp/README.md

Run the Ansible playbook using the generated inventory file to configure the Kubernetes cluster.

```bash
ansible-playbook -i contrib/terraform/gcp/inventory.ini cluster.yml -b -v
```

--------------------------------

### Initialize Terraform Configuration

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/openstack/README.md

Run this command from the inventory directory to initialize Terraform and load necessary modules for cluster provisioning.

```ShellSession
cd inventory/$CLUSTER
terraform -chdir="../../contrib/terraform/openstack" init
```

--------------------------------

### Enable CRI-O User Namespaces

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/CRI/cri-o.md

Configure runtime paths and enable user namespace remapping.

```yaml
crio_runtimes:
  - name: runc
    path: /usr/bin/runc
    type: oci
    root: /run/runc
    allowed_annotations:
    - "io.kubernetes.cri-o.userns-mode"

crio_remap_enable: true
```

--------------------------------

### List git tags

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operations/upgrades.md

View available versions for sequential upgrades.

```console
$ git tag
v2.20.0
v2.21.0
v2.22.0
v2.22.1
v2.23.0
v2.23.1
v2.23.2
v2.24.0
...
```

--------------------------------

### Get Kubernetes Nodes

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/getting_started/setting-up-your-first-cluster.md

Uses kubectl to retrieve and display the status of all nodes in the Kubernetes cluster.

```ShellSession
kubectl get nodes
```

--------------------------------

### Configure Kubespray Mirror Sites

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operations/mirror.md

Add these variables to `<your_inventory>/group_vars/k8s_cluster.yml` to redirect image and file downloads to mirror sites. Use these only if you trust the provider.

```yaml
# this should be in <your_inventory>/group_vars/k8s_cluster.yml
gcr_image_repo: "gcr.m.daocloud.io"
kube_image_repo: "k8s.m.daocloud.io"
docker_image_repo: "docker.m.daocloud.io"
quay_image_repo: "quay.m.daocloud.io"
github_image_repo: "ghcr.m.daocloud.io"

files_repo: "https://files.m.daocloud.io"
```

--------------------------------

### Uninstall Docker and Dependencies

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/upgrades/migrate_docker2containerd.md

Remove Docker packages and ensure required dependencies like pigz are installed.

```commandline
apt-get remove -y --allow-change-held-packages containerd.io docker-ce docker-ce-cli docker-ce-rootless-extras
```

```shell
apt-get install pigz
```

--------------------------------

### Provision Infrastructure with Terraform

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/gcp/README.md

Execute the Terraform apply command to provision GCP resources using a specified variable file and state file.

```bash
terraform apply -var-file tfvars.json -state dev-cluster.tfstate -var gcp_project_id=<ID of your GCP project> -var keyfile_location=<location of the json keyfile>
```

--------------------------------

### Prepare Kubespray Inventory

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/upcloud/README.md

Copy the sample inventory and cluster settings to a new directory for your cluster. This prepares the configuration files for Kubespray. Set the ANSIBLE_CONFIG environment variable and navigate to the cluster's inventory directory.

```bash
CLUSTER=my-upcloud-cluster
cp -r inventory/sample inventory/$CLUSTER
cp contrib/terraform/upcloud/cluster-settings.tfvars inventory/$CLUSTER/
export ANSIBLE_CONFIG=ansible.cfg
cd inventory/$CLUSTER
```

--------------------------------

### Enable EPEL Repository

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/CNI/calico.md

Enables the EPEL repository, which is required for installing Wireguard tools on specific Linux distributions.

```yaml
epel_enabled: true
```

--------------------------------

### Deploy GlusterFS

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/openstack/README.md

Run the specific playbook for GlusterFS storage deployment.

```ShellSession
ansible-playbook --become -i inventory/$CLUSTER/hosts ./contrib/network-storage/glusterfs/glusterfs.yml
```

--------------------------------

### Configure Unprivileged Cilium Agent

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/CNI/cilium.md

Set the kube_owner variable to root in the inventory to support unprivileged agent installation.

```yaml
kube_owner: root
```

--------------------------------

### Display Help for OpenStack Cleanup

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/scripts/openstack-cleanup/README.md

Shows the help message for the main script, providing information on available commands and options for the openstack-cleanup tool.

```shell
python main.py --help
```

--------------------------------

### Run Kubespray with Custom Inventory

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operations/offline-environment.md

Execute the Kubespray playbook using a custom inventory file. Ensure your `hosts.yaml` is correctly configured.

```bash
ansible-playbook -i inventory/my_airgap_cluster/hosts.yaml -b cluster.yml
```

--------------------------------

### Configure k8s_nodes via INI

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/openstack/README.md

Examples of defining worker nodes using the k8s_nodes variable in INI format.

```ini
k8s_nodes = {
  "1" = {
    "az" = "sto1"
    "flavor" = "83d8b44a-26a0-4f02-a981-079446926445"
    "floating_ip" = true
  },
  "2" = {
    "az" = "sto2"
    "flavor" = "83d8b44a-26a0-4f02-a981-079446926445"
    "floating_ip" = true
  },
  "3" = {
    "az" = "sto3"
    "flavor" = "83d8b44a-26a0-4f02-a981-079446926445"
    "floating_ip" = true
    "extra_groups" = "calico_rr"
  }
}
```

```ini
number_of_k8s_nodes = 3
flavor_k8s_node = "83d8b44a-26a0-4f02-a981-079446926445"
az_list = ["sto1", "sto2", "sto3"]
```

```ini
k8s_nodes = {
  "ing-1" = {
    "az" = "sto1"
    "flavor" = "83d8b44a-26a0-4f02-a981-079446926445"
    "floating_ip" = true
  },
  "ing-2" = {
    "az" = "sto2"
    "flavor" = "83d8b44a-26a0-4f02-a981-079446926445"
    "floating_ip" = true
  },
  "ing-3" = {
    "az" = "sto3"
    "flavor" = "83d8b44a-26a0-4f02-a981-079446926445"
    "floating_ip" = true
  },
  "big-1" = {
    "az" = "sto1"
    "flavor" = "3f73fc93-ec61-4808-88df-2580d94c1a9b"
    "floating_ip" = false
  },
  "big-2" = {
    "az" = "sto2"
    "flavor" = "3f73fc93-ec61-4808-88df-2580d94c1a9b"
    "floating_ip" = false
  },
  "big-3" = {
    "az" = "sto3"
    "flavor" = "3f73fc93-ec61-4808-88df-2580d94c1a9b"
    "floating_ip" = false
  },
  "small-1" = {
    "az" = "sto1"
    "flavor" = "7a6a998f-ac7f-4fb8-a534-2175b254f75e"
    "floating_ip" = false
  },
  "small-2" = {
    "az" = "sto2"
    "flavor" = "7a6a998f-ac7f-4fb8-a534-2175b254f75e"
    "floating_ip" = false
  },
  "small-3" = {
    "az" = "sto3"
    "flavor" = "7a6a998f-ac7f-4fb8-a534-2175b254f75e"
    "floating_ip" = false
  }
}
```

```ini
k8s_nodes = {
  "key | node name suffix, must be unique" = {
    "az" = string
    "flavor" = string
    "floating_ip" = bool
  },
}
```

--------------------------------

### Initialize Cluster Inventory

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/vsphere/README.md

Copy the sample inventory and default Terraform variables to a new cluster directory.

```bash
CLUSTER=my-vsphere-cluster
cp -r inventory/sample inventory/$CLUSTER
cp contrib/terraform/vsphere/default.tfvars inventory/$CLUSTER/
cd inventory/$CLUSTER
```

--------------------------------

### Initialize Cluster Inventory

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/exoscale/README.md

Copy the sample inventory and default Terraform variables to a new cluster directory.

```bash
CLUSTER=my-exoscale-cluster
cp -r inventory/sample inventory/$CLUSTER
cp contrib/terraform/exoscale/default.tfvars inventory/$CLUSTER/
cd inventory/$CLUSTER
```

--------------------------------

### Configure Kubelet Extra Arguments

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ansible/vars.md

Use a dictionary of key-value pairs to inject configuration parameters directly into the kubelet YAML config file.

```yaml
kubelet_config_extra_args:
  evictionHard:
    memory.available: "100Mi"
  evictionSoftGracePeriod:
    memory.available: "30s"
  evictionSoft:
    memory.available: "300Mi"
```

--------------------------------

### Configure Coscheduling plugin timeout

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/external_storage_provisioners/scheduler_plugins.md

Example configuration for the Coscheduling plugin to set the permit waiting timeout to 10 seconds.

```yaml
scheduler_plugins_plugin_config:
  - name: Coscheduling
    args:
      permitWaitingTimeSeconds: 10 # default is 60
```

--------------------------------

### Prepare container images locally

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ansible/ansible.md

Downloads container images to the Ansible runner node without installing or uploading them to cluster nodes.

```ShellSession
ansible-playbook -i inventory/sample/hosts.ini cluster.yml \
    -e download_run_once=true -e download_localhost=true \
    --tags download --skip-tags upload,upgrade
```

--------------------------------

### Configure Qemu Hypervisor Overhead

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/CRI/kata-containers.md

Define resource overhead values for the Qemu hypervisor to ensure accurate resource accounting.

```yaml
kata_containers_qemu_overhead: true
kata_containers_qemu_overhead_fixed_cpu: 10m
kata_containers_qemu_overhead_fixed_memory: 290Mi
```

--------------------------------

### Initialize and Apply Terraform Configuration

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/upcloud/README.md

Initialize Terraform in the UpCloud provider directory and then apply the cluster configuration using the specified variable file and state file. This command provisions the necessary infrastructure on UpCloud.

```bash
terraform init ../../contrib/terraform/upcloud
terraform apply --var-file cluster-settings.tfvars \
    -state=tfstate-$CLUSTER.tfstate \
     ../../contrib/terraform/upcloud/
```

--------------------------------

### Keep Docker RPM Cache

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/CRI/docker.md

Set to '1' to retain Docker packages after installation, which can speed up subsequent Ansible provisioning runs.

```yaml
docker_rpm_keepcache: 1
```

--------------------------------

### Create OpenStack Inventory Directory

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/openstack/README.md

Copies sample inventory files and links necessary scripts to establish the base for Terraform commands. Ensure `your-cluster-name` is replaced with your actual cluster name.

```ShellSession
CLUSTER=your-cluster-name
cp -LRp contrib/terraform/openstack/sample-inventory inventory/$CLUSTER
cd inventory/$CLUSTER
ln -s ../../contrib/terraform/openstack/hosts
ln -s ../../contrib
```

--------------------------------

### Git Submodule Configuration File

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operations/integration.md

This is an example of the `.gitmodules` file that Git creates when you add a submodule. It records the path and URL of the submodule.

```INI
[submodule "3d/kubespray"]
       path = 3d/kubespray
       url = https://github.com/YOUR_GITHUB/kubespray.git
```

--------------------------------

### Deploy a DaemonSet for Registry Proxy

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/advanced/registry.md

Creates a DaemonSet to expose the registry service on each node via port 5000.

```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-registry-proxy
  namespace: kube-system
  labels:
    k8s-app: kube-registry-proxy
    version: v0.4
spec:
  template:
    metadata:
      labels:
        k8s-app: kube-registry-proxy
        kubernetes.io/name: "kube-registry-proxy"
        version: v0.4
    spec:
      containers:
      - name: kube-registry-proxy
        image: gcr.io/google_containers/kube-registry-proxy:0.4
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        env:
        - name: REGISTRY_HOST
          value: kube-registry.kube-system.svc.cluster.local
        - name: REGISTRY_PORT
          value: "5000"
        ports:
        - name: registry
          containerPort: 80
          hostPort: 5000
```

--------------------------------

### Build and push all images

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/test-infra/image-builder/README.md

Executes the build and push process for all defined images using a Quay robot token.

```bash
cd test-infra/image-builder/
make docker_password=<quay-robot-token>
```

--------------------------------

### Configure Containerd Repository for Ubuntu

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operations/offline-environment.md

Set the base URL, GPG key, and repository key for the Containerd repository on Ubuntu. `ubuntu_repo` should point to your internal repository.

```yaml
containerd_ubuntu_repo_base_url: "{{ ubuntu_repo }}/containerd"
containerd_ubuntu_repo_gpgkey: "{{ ubuntu_repo }}/containerd/gpg"
containerd_ubuntu_repo_repokey: 'YOURREPOKEY'
```

--------------------------------

### Import Kubespray Playbook

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operations/integration.md

Include the main Kubespray cluster playbook (`cluster.yml`) within your own Ansible playbooks to orchestrate the Kubernetes setup.

```YAML
- name: Import kubespray playbook
  ansible.builtin.import_playbook: 3d/kubespray/cluster.yml
```

--------------------------------

### Check Python and Pip Versions

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/README.md

Verify that Python and pip are installed on your system before proceeding with Vagrant provisioning. These are prerequisites for Ansible provisioning tasks.

```ShellSession
python -V && pip -V
```

--------------------------------

### HAProxy External Load Balancer Configuration

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/operations/ha-mode.md

Example configuration for an HAProxy service acting as an external load balancer for the Kubernetes API server.

```raw
listen kubernetes-apiserver-https
  bind <VIP>:8383
  mode tcp
  option log-health-checks
  timeout client 3h
  timeout server 3h
  server master1 <IP1>:6443 check check-ssl verify none inter 10000
  server master2 <IP2>:6443 check check-ssl verify none inter 10000
  balance roundrobin
```

--------------------------------

### Build Vagrant container image

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/RELEASE.md

Execute the build script located in the test-infra directory to create the Vagrant container image.

```shell
cd kubespray/test-infra/vagrant-docker/
./build vX.Y.Z
```

--------------------------------

### Execute Kubespray playbook

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ansible/ansible_collection.md

Run the playbook using ansible-playbook with root privileges and your inventory file.

```ShellSession
ansible-playbook -i INVENTORY --become --become-user=root PLAYBOOK
```

--------------------------------

### Kubectl Get Storage Classes

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/external_storage_provisioners/local_volume_provisioner.md

Verify that StorageClass objects are automatically created for each defined local storage class. These are typically configured with a no-provisioner.

```bash
$ kubectl get storageclasses.storage.k8s.io
NAME            PROVISIONER                    RECLAIMPOLICY
fast-disks      kubernetes.io/no-provisioner   Delete
local-storage   kubernetes.io/no-provisioner   Delete
```

--------------------------------

### Build and push a specific image

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/test-infra/image-builder/README.md

Builds and pushes a single image after adding its definition to the configuration.

```bash
make docker_password=<quay-robot-token>
```

--------------------------------

### Enable NTP Service

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/advanced/ntp.md

Enable the ntpd (or chrony) service and ensure it starts at system boot. This is the basic setting to activate NTP synchronization.

```ShellSession
ntp_enabled: true
```

--------------------------------

### Specify NTP Package for Ubuntu 24.04+

Source: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/advanced/ntp.md

When using Ubuntu 24.04 or similar distributions that include `systemd-timesyncd`, specify `ntpsec` as the NTP package to be installed.

```ShellSession
ntp_package: ntpsec
```