### Install k3s and enable secrets encryption on server start

Source: https://github.com/k3s-io/k3s/wiki/Secrets-Encryption-Test-Plan

Install K3s without starting the service, then manually start the server with the `--secrets-encryption` flag. This allows for a controlled start with encryption enabled.

```shell
$ curl -sfL https://get.k3s.io | INSTALL_K3S_COMMIT=<commithash> INSTALL_K3S_EXEC="server --write-kubeconfig-mode 644" INSTALL_K3S_SKIP_START=true sh -
$ sudo k3s server secrets-encrypt
```

--------------------------------

### Example Full Command List for Cutting a Release

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/expanded/cut_release.md

This extensive example demonstrates the full sequence of commands required to cut a k3s release, from setting up the environment and cloning repositories to rebasing, updating dependencies, and pushing the release tag. It includes notes on platform-specific commands and environment variable setup.

```bash
export SSH_MOUNT_PATH="/var/folders/...krzO/agent.452"
export GLOBAL_GIT_CONFIG_PATH="/Users/mtrachier/.gitconfig"
export OLD_K8S="v1.22.14"
export NEW_K8S="v1.22.15"
export OLD_K8S_CLIENT="v0.22.14"
export NEW_K8S_CLIENT="v0.22.15"
export OLD_K3S_VER="v1.22.14-k3s1" 
export NEW_K3S_VER="v1.22.15-k3s1"
export RELEASE_BRANCH="release-1.22"
export GOPATH="/Users/mtrachier/go"
export GOVERSION="1.16.15"
export GOIMAGE="golang:1.16.15-alpine3.15"
export BUILD_CONTAINER="FROM golang:1.16.15-alpine3.15\n RUN apk add --no-cache bash git make tar gzip curl git coreutils rsync alpine-sdk"

install -d /Users/mtrachier/go/src/github.com/kubernetes
rm -rf /Users/mtrachier/go/src/github.com/kubernetes/kubernetes
git clone --origin upstream https://github.com/kubernetes/kubernetes.git /Users/mtrachier/go/src/github.com/kubernetes/kubernetes
cd /Users/mtrachier/go/src/github.com/kubernetes/kubernetes
git remote add k3s-io https://github.com/k3s-io/kubernetes.git
git fetch --all --tags

# this second fetch should return no more tags pulled, this makes it easier to see pull errors
git fetch --all --tags

# rebase
rm -rf _output
git rebase --onto v1.22.15 v1.22.14 v1.22.14-k3s1~1

# validate go version
echo "GOVERSION is $(yq -e '.dependencies[] | select(.name == "golang: upstream version").version' build/dependencies.yaml)"

# generate build container
echo -e "FROM golang:1.16.15-alpine3.15\n RUN apk add --no-cache bash git make tar gzip curl git coreutils rsync alpine-sdk" | docker build -t golang:1.16.15-alpine3.15-dev - 

# run tag.sh
# note user id is 502, I am not root user
docker run --rm -u 502 \
--mount type=tmpfs,destination=/Users/mtrachier/go/pkg \
-v /Users/mtrachier/go/src:/go/src \
-v /Users/mtrachier/go/.cache:/go/.cache \
-v /Users/mtrachier/.gitconfig:/go/.gitconfig \
-e HOME=/go \
-e GOCACHE=/go/.cache \
-w /go/src/github.com/kubernetes/kubernetes golang:1.16.15-alpine3.15-dev ./tag.sh v1.22.15-k3s1 2>&1 | tee ~/tags-v1.22.15-k3s1.log

# generate and run push.sh, make sure to paste in the tag.sh output below
vim push.sh
chmod +x push.sh
./push.sh

install -d /Users/mtrachier/go/src/github.com/k3s-io
rm -rf /Users/mtrachier/go/src/github.com/k3s-io/k3s
git clone --origin upstream https://github.com/k3s-io/k3s.git /Users/mtrachier/go/src/github.com/k3s-io/k3s
cd /Users/mtrachier/go/src/github.com/k3s-io/k3s

git checkout -B v1.22.15-k3s1 upstream/release-1.22
git clean -xfd


# note that sed has different parameters on MacOS than Linux
# also note that zsh is the default MacOS shell and is not bash/dash (the default Linux shells)
sed -Ei '' "\|github.com/k3s-io/kubernetes| s|v1.22.14-k3s1|v1.22.15-k3s1|" go.mod
git diff
sed -Ei '' "s/k8s.io\/kubernetes v.*$/k8s.io\/kubernetes v1.22.15/" go.mod
git diff
sed -Ei '' "s/v0.22.14/v0.22.15/g" go.mod
git diff
go mod tidy

# make sure go version is updated in all locations
vim .github/workflows/integration.yaml
vim .github/workflows/unitcoverage.yaml
vim Dockerfile.dapper
vim Dockerfile.manifest
vim Dockerfile.test

git commit --all --signoff -m "Update to v1.22.15"
git remote add origin https://github.com/matttrach/k3s-1.git
git push --set-upstream origin v1.22.15-k3s1

# use link to generate pull request, make sure your target is the proper release branch 'release-1.22'
```

--------------------------------

### Example Environment Variable Setup Script

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/expanded/setup_rc.md

This script demonstrates how to set various environment variables required for K3s release processes. It includes settings for git configuration, SSH, Kubernetes and K3s versions, release branches, Go environment, and the build container.

```bash
export SSH_MOUNT_PATH="/var/folders/m7/1d53xcj57d76n1qxv_ykgr040000gp/T//ssh-dmtrX2MOkrzO/agent.45422"

export OLD_K8S="v1.22.13"
export NEW_K8S="v1.22.14"
export OLD_K8S_CLIENT="v0.22.13"
export NEW_K8S_CLIENT="v0.22.14"
export OLD_K3S_VER="v1.22.13-k3s1" 
export NEW_K3S_VER="v1.22.14-k3s1"
export RELEASE_BRANCH="release-1.22"
export GOVERSION="1.16.15"
export GOIMAGE="golang:1.16.15-alpine"
# On Linux
export GLOBAL_GIT_CONFIG_PATH="$HOME/.gitconfig"
export GOPATH="$HOME/go"
# On Mac
export GLOBAL_GIT_CONFIG_PATH="/Users/mtrachier/.gitconfig"
export GOPATH="/Users/mtrachier/go"

export BUILD_CONTAINER="FROM golang:1.16.15-alpine\n RUN apk add --no-cache bash gnupg git make tar gzip curl git coreutils rsync alpine-sdk"
```

--------------------------------

### Push to Origin Example

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/expanded/pr.md

Example command for pushing changes to your origin. Ensure you are pushing to 'origin' and not 'upstream'.

```bash
git push -u origin v1.25.1-k3s1
```

--------------------------------

### Run Install Tests with Vagrant

Source: https://github.com/k3s-io/k3s/blob/main/tests/TESTING.md

Navigate to a specific install test directory and use Vagrant to bring up the virtual machine for testing the installation process.

```shell
cd tests/install/rocky-8
vagrant up
```

--------------------------------

### Install K3s Server

Source: https://github.com/k3s-io/k3s/blob/main/README.md

Installs K3s as a service on a server node. A kubeconfig file is written to /etc/rancher/k3s/k3s.yaml and the service is automatically started or restarted.

```bash
curl -sfL https://get.k3s.io | sh -
```

--------------------------------

### Start K3s server manually

Source: https://github.com/k3s-io/k3s/blob/main/README.md

After downloading the k3s binary, run this command to start the K3s server process in the background.

```bash
sudo k3s server &
```

--------------------------------

### Install Vagrant Plugins

Source: https://github.com/k3s-io/k3s/blob/main/tests/TESTING.md

Install necessary Vagrant plugins for libvirt, SCP, and K3s integration. Ensure the libvirtd service is running if using the libvirt provider.

```shell
vagrant plugin install vagrant-scp vagrant-k3s vagrant-libvirt
```

--------------------------------

### Install goimports

Source: https://github.com/k3s-io/k3s/blob/main/CONTRIBUTING.md

Install the goimports tool, which is required for the code formatting CI check. This command installs the latest version.

```bash
go install golang.org/x/tools/cmd/goimports@latest
```

--------------------------------

### Install K3s with Local Build

Source: https://github.com/k3s-io/k3s/blob/main/CONTRIBUTING.md

Use this command to install K3s using a locally built binary, skipping the download step. Ensure the K3s binary is in your PATH or copied to /usr/local/bin/k3s.

```bash
curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_DOWNLOAD=true sh -
```

--------------------------------

### Install K3s with curl script

Source: https://github.com/k3s-io/k3s/blob/main/README.md

Use this command to automatically download and install K3s. Set K3S_URL and K3S_TOKEN environment variables to join an existing server.

```bash
curl -sfL https://get.k3s.io | K3S_URL=https://myserver:6443 K3S_TOKEN=XXX sh -
```

--------------------------------

### Install Vagrant Plugins

Source: https://github.com/k3s-io/k3s/blob/main/tests/e2e/README.md

Installs essential Vagrant plugins for K3s E2E testing. These plugins facilitate interaction with libvirt and other services.

```bash
vagrant plugin install vagrant-libvirt vagrant-scp vagrant-k3s vagrant-reload
```

--------------------------------

### Start k3s Server with Secrets Encryption Enabled

Source: https://github.com/k3s-io/k3s/wiki/Secrets-Encryption-Test-Plan

Starts a k3s server instance with the secrets encryption feature enabled. This is the initial step for setting up encryption.

```bash
k3s server --secrets-encryption
```

--------------------------------

### Install Kubectl on Linux

Source: https://github.com/k3s-io/k3s/blob/main/tests/e2e/README.md

Downloads and installs the latest stable version of kubectl on Linux systems. Ensures kubectl is executable and placed in the system's PATH.

```bash
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
```

--------------------------------

### Install k3s with secrets encryption enabled

Source: https://github.com/k3s-io/k3s/wiki/Secrets-Encryption-Test-Plan

Install K3s with the `--secrets-encryption` flag enabled. This command ensures that secrets encryption is active from the start.

```shell
$ curl -sfL https://get.k3s.io | INSTALL_K3S_COMMIT=<commithash> INSTALL_K3S_EXEC="server --write-kubeconfig-mode 644 --secrets-encryption" sh -
```

--------------------------------

### Example Pull Request

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/expanded/pr.md

An example of a completed pull request for K3s. This serves as a reference for the expected format and content.

```url
https://github.com/k3s-io/k3s/pull/6164
```

--------------------------------

### Install Libvirt Dependencies on Ubuntu 22.04

Source: https://github.com/k3s-io/k3s/blob/main/tests/e2e/README.md

Installs required packages for libvirt on Ubuntu 22.04. Requires sudo access.

```bash
sudo apt install ruby-libvirt qemu-kvm libvirt-daemon-system libvirt-clients ebtables dnsmasq-base libxslt-dev libxml2-dev libvirt-dev zlib1g-dev ruby-dev libguestfs-tools
```

--------------------------------

### Install Libvirt Dependencies on Ubuntu 20.04

Source: https://github.com/k3s-io/k3s/blob/main/tests/e2e/README.md

Installs necessary packages for libvirt on Ubuntu 20.04. Ensure you have sudo privileges.

```bash
sudo apt install ruby-libvirt qemu libvirt-daemon-system libvirt-clients ebtables dnsmasq-base libxslt-dev libxml2-dev libvirt-dev zlib1g-dev ruby-dev libguestfs-tools
```

--------------------------------

### Run Clusterloader2 Load Test

Source: https://github.com/k3s-io/k3s/blob/main/tests/perf/README.md

After configuring 'tests/perf/tests/load/config.yaml', run 'make test' from the 'tests/perf' directory to start the clusterloader2 load test.

```bash
cd tests/perf
make test
```

--------------------------------

### Install GNU Utilities on macOS

Source: https://github.com/k3s-io/k3s/blob/main/docs/contrib/development.md

Installs essential GNU utilities required for K3s development on macOS. Ensure Homebrew is installed first.

```sh
brew install coreutils findutils gawk gnu-sed gnu-tar grep make
```

--------------------------------

### Example Drone CI Run Link

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/expanded/pr.md

Link to a Drone CI run for debugging. Use this to access detailed logs and identify failures.

```bash
https://drone-pr.k3s.io/k3s-io/k3s/4744
```

--------------------------------

### Example Log Entry - Test Failure

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/expanded/pr.md

Example of a test failure log entry from CI. Look for patterns like '[Fail]' or 'failed.' to identify specific test issues.

```log
[Fail] [sig-network] DNS [It] should support configurable pod DNS nameservers [Conformance]
```

--------------------------------

### Start K3s Server with External DB and Secrets Encryption

Source: https://github.com/k3s-io/k3s/wiki/Secrets-Encryption-Test-Plan

Starts a K3s server node configured to use an external database for its datastore and enables secrets encryption. Replace `<server_ip>` with the actual IP address of your database server.

```bash
k3s server --secrets-encryption --datastore-endpoint "mysql://root:mysql@tcp(<server_ip>:3306)/k3s"
```

--------------------------------

### gsutil Configuration for Log Decryption

Source: https://github.com/k3s-io/k3s/blob/main/contrib/util/DIAGNOSTICS.md

Instructions for setting up gsutil with credentials for accessing log data. Ensure gsutil is installed via pip and configured with appropriate access.

```bash
echo -e 'rancher-dev-file.json\nk3s-diagnostic-logs' | gsutil config -e
```

--------------------------------

### K3s Component Version Verification - version.sh

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/expanded/release_notes.md

Example of how version information for components like Kubernetes is pulled from the k3s repository's version.sh script, which in turn may reference go.mod.

```bash
kubernetes: version.sh pulls from k3s repo go.mod see https://github.com/k3s-io/k3s/blob/v1.23.13-rc2+k3s1/scripts/version.sh#L35
```

--------------------------------

### K3s Component Version Verification - go.mod

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/expanded/release_notes.md

Example of how component versions like Kine and SQLite are sourced from the k3s repository's go.mod file.

```bash
kine: go.mod, see https://github.com/k3s-io/k3s/blob/v1.23.13-rc2+k3s1/go.mod#L93
```

```bash
sqlite: go.mod, see https://github.com/k3s-io/k3s/blob/v1.23.13-rc2+k3s1/go.mod#L97
```

--------------------------------

### Example Log Entry - Filesystem Error

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/expanded/pr.md

Example of a filesystem error log entry from CI. The log level (e.g., 'E0921' for error) and error message provide clues for debugging.

```log
#- Tail: /tmp/bEaiAq/agents/1/logs/system.log
[LATEST-SERVER] E0921 19:16:55.430977      57 cri_stats_provider.go:455] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs
[LATEST-SERVER] I0921 19:16:55.431186      57 proxier.go:667] "Failed to load kernel module with modprobe, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="ip_vs_rr"
```

--------------------------------

### Example LoadBalancer Service

Source: https://github.com/k3s-io/k3s/blob/main/docs/adrs/remove-svclb-daemonset.md

Demonstrates a typical LoadBalancer service configuration for Traefik, showing external IP assignments and port mappings.

```text
NAMESPACE     NAME             TYPE           CLUSTER-IP      EXTERNAL-IP                                                         PORT(S)                      AGE
default       kubernetes       ClusterIP      10.43.0.1       <none>                                                              443/TCP                      56m
kube-system   kube-dns         ClusterIP      10.43.0.10      <none>                                                              53/UDP,53/TCP,9153/TCP       56m
kube-system   metrics-server   ClusterIP      10.43.55.117    <none>                                                              443/TCP                      56m
kube-system   traefik          LoadBalancer   10.43.206.216   10.1.1.13,10.1.1.16,fd56:5da5:a285:eea0::6,fd56:5da5:a285:eea0::8   80:30235/TCP,443:32373/TCP   56m
```

--------------------------------

### Install K3s Worker Node

Source: https://github.com/k3s-io/k3s/blob/main/README.md

To install K3s on worker nodes, you need to provide the K3S_URL and K3S_TOKEN environment variables. These are typically found on the server node.

```bash
curl -sfL https://get.k3s.io | K3S_URL=https://<server_ip>:6443 K3S_TOKEN=<node_token> sh -
```

--------------------------------

### Clone Kubernetes Repository and Setup Remotes

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/kubernetes-upgrade.md

Initial clone of the upstream Kubernetes repository and addition of K3S-specific remotes. Ensure to fetch all remote branches and tags.

```sh
# initial clone
git clone --origin upstream \
  https://github.com/kubernetes/kubernetes.git \
  ${GOPATH}/src/github.com/kubernetes/kubernetes

cd ${GOPATH}/src/github.com/kubernetes/kubernetes

# add the k3s-io remote
git remote add k3s-io https://github.com/k3s-io/kubernetes.git

# fetch all remote branches and tags. If you receive a message saying that
# previous tags will be "clobbered", add --force to the command below.
git fetch --all --tags
```

--------------------------------

### Deprecated Commands Warning (v1.29)

Source: https://github.com/k3s-io/k3s/blob/main/docs/adrs/secrets-encryption-v3.md

Starting from v1.29, the old 'prepare', 'reencrypt', and 'rotate' commands will be deprecated and issue warnings, encouraging the use of 'rotate-keys'.

```bash
k3s secrets-encrypt prepare

```

```bash
k3s secrets-encrypt reencrypt

```

```bash
k3s secrets-encrypt rotate

```

--------------------------------

### K3s Component Version Verification - etcd API

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/expanded/release_notes.md

Example of how the etcd version is determined by referencing the /api/v3 module within the k3s repository's go.mod file.

```bash
etcd: go.mod, use the /api/v3 mod, see https://github.com/k3s-io/k3s/blob/v1.23.13-rc2+k3s1/go.mod#L25
```

--------------------------------

### Vagrant Provisioning for K3s Node Readiness

Source: https://github.com/k3s-io/k3s/blob/main/tests/TESTING.md

Use these commands to provision Vagrant environments and wait for specific K3s components to become ready. This is useful for ensuring a stable testing environment before proceeding.

```bash
vagrant provision --provision-with=k3s-wait-for-node
```

```bash
vagrant provision --provision-with=k3s-wait-for-coredns
```

```bash
vagrant provision --provision-with=k3s-wait-for-local-storage
```

```bash
vagrant provision --provision-with=k3s-wait-for-metrics-server
```

```bash
vagrant provision --provision-with=k3s-wait-for-traefik
```

```bash
vagrant provision --provision-with=k3s-status
```

```bash
vagrant provision --provision-with=k3s-procps
```

--------------------------------

### Build K3s Binary

Source: https://github.com/k3s-io/k3s/blob/main/BUILDING.md

Run 'make' to build the full release binary. The output will be located at ./dist/artifacts/k3s.

```bash
make
```

--------------------------------

### Build K3s Performance Test Environment

Source: https://github.com/k3s-io/k3s/blob/main/tests/perf/README.md

Navigate to the performance testing directory and execute 'make apply' to build the database, server, and agent layers. This command also generates a kubeconfig file.

```bash
cd tests/perf
make apply
```

--------------------------------

### Run All Integration Tests

Source: https://github.com/k3s-io/k3s/blob/main/tests/integration/README.md

Execute all integration tests using the Go test command. Ensure to prefix with 'sudo -E env "PATH=$PATH"' if running as a sudo user.

```bash
go test ./tests/integration/... -run Integration -ginkgo.v -test.v
```

--------------------------------

### K3s Component Version Verification - Containerd Build Script

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/expanded/release_notes.md

Demonstrates how Containerd's version is set via an environment variable in version.sh, influenced by go.mod, and then built using the build script.

```bash
containerd: version.sh sets an env variable based on go.mod, then the build script builds it
  see https://github.com/k3s-io/k3s/blob/v1.23.13-rc2%2Bk3s1/scripts/version.sh#L25
  and https://github.com/k3s-io/k3s/blob/v1.23.13-rc2%2Bk3s1/scripts/build#L36
```

--------------------------------

### Setting Up K3s Git Repository

Source: https://github.com/k3s-io/k3s/blob/main/docs/contrib/git_workflow.md

Commands to fork the K3s repository, clone it locally, add the upstream remote, and configure push settings to prevent direct upstream pushes.

```sh
## Clone fork to local storage
export user="your github profile name"
git clone https://github.com/$user/k3s.git
# or: git clone git@github.com:$user/k3s.git

## Add k3s as upstream to your fork
cd k3s 
git remote add upstream https://github.com/k3s-io/k3s.git
# or: git remote add upstream git@github.com:k3s-io/k3s.git

## Ensure to never push to upstream directly
git remote set-url --push upstream no_push

## Confirm that your remotes make sense:
git remote -v
```

--------------------------------

### Build Docker Image for Go Version

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/kubernetes-upgrade.md

Build a Docker image containing the specific Go version and necessary tools (bash, git, make, etc.) required for building Kubernetes. This ensures a consistent build environment.

```sh
# Kubernetes is specific with the Go version used per release. We use alpine and docker to specify the Go version with which we build the project.
export GOVERSION=$(grep -Po '^\s*\K\S+' .go-version)

export GOIMAGE="golang:${GOVERSION}-alpine"

export BUILD_CONTAINER="FROM ${GOIMAGE}\n \ 
RUN apk add --no-cache \
bash \
git \
make \
tar \
gzip \
curl \
git \
coreutils \
rsync \
alpine-sdk"

echo -e ${BUILD_CONTAINER} | docker build -t ${GOIMAGE}-dev -
```

--------------------------------

### Get Vagrant VM Status

Source: https://github.com/k3s-io/k3s/blob/main/tests/e2e/README.md

Lists the current status of all Vagrant-managed virtual machines for the current test environment.

```bash
vagrant status
```

--------------------------------

### Get K3s Nodes

Source: https://github.com/k3s-io/k3s/blob/main/README.md

Use this command to list all nodes in your K3s cluster. The kubeconfig file is located at /etc/rancher/k3s/k3s.yaml.

```bash
sudo k3s kubectl get nodes
```

--------------------------------

### Create and describe a Kubernetes secret

Source: https://github.com/k3s-io/k3s/wiki/Secrets-Encryption-Test-Plan

This demonstrates how to create a Kubernetes secret and then inspect its details. Note that the `kubectl describe` output will show the size of the data, not the data itself.

```shell
$ kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
secret/secret1 created
$ kubectl describe secret secret1 -n default
Name:         secret1
Namespace:    default
Labels:       &lt;none&gt;
Annotations:  &lt;none&gt;
Type:  Opaque
Data
====
mykey:  6 bytes
```

--------------------------------

### Console Output During Re-encryption

Source: https://github.com/k3s-io/k3s/wiki/Secrets-Encryption-Test-Plan

Indicates that the re-encryption process for secrets has started. This message appears in the console output during the re-encryption stage.

```text
reencryption started
```

--------------------------------

### Check K3s Node Status

Source: https://github.com/k3s-io/k3s/blob/main/README.md

After installation, you can use kubectl to check the status of your K3s nodes. This command requires sudo privileges.

```bash
sudo kubectl get nodes
```

--------------------------------

### K3s Certificate Management CLI Help

Source: https://github.com/k3s-io/k3s/wiki/K3s-Cert-Rotation

Displays the help information for the K3s certificate management command-line interface, outlining available commands and options for certificate operations.

```bash
$ k3s certificate -h
NAME:
   k3s certificate - Certificates management

USAGE:
   k3s certificate command [command options] [arguments...]

COMMANDS:
   rotate  Certificate rotation

OPTIONS:
   --debug                     (logging) Turn on debug logs [$K3S_DEBUG]
   --config FILE, -c FILE      (config) Load configuration from FILE (default: "/etc/rancher/k3s/config.yaml") [$K3S_CONFIG_FILE]
   --log value, -l value       (logging) Log to file
   --alsologtostderr           (logging) Log to standard error as well as file (if set)
   --data-dir value, -d value  (data) Folder to hold state default /var/lib/rancher/k3s or ${HOME}/.rancher/k3s if not root
   --service value, -s value   List of services to rotate certificates for. Options include (admin, api-server, controller-manager, scheduler, k3s-controller, k3s-server, cloud-controller, etcd, auth-proxy, kubelet, kube-proxy)
   --help, -h                  show help

```

--------------------------------

### K3s Error Log Entry

Source: https://github.com/k3s-io/k3s/wiki/Secrets-Encryption-Test-Plan

Example of an error log entry from k3s indicating an issue with secrets encryption handler reencryption.

```log
Dec 10 18:28:15 ip-172-31-32-180 k3s[2146]: time="2021-12-10T18:28:15Z" level=error msg="error syncing 'ip-172-31-32-180': handler reencrypt-controller: invalid hash: 8d0f2a2ad73ce974c4cc39874ebcb3b31d142774aed2c4771e4b4205c94a028f found on node ip-172-31-32-180, requeuing"
```

--------------------------------

### Set Up Release Version Variables

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/kubernetes-upgrade.md

Define environment variables for old and new Kubernetes versions, client versions, and K3S versions to be used in the release process.

```sh
export GLOBAL_GIT_CONFIG_PATH=$(git config --list --show-origin --show-scope --global | awk 'NR==1{ split($2,path,":"); print path[2] }')
export SSH_MOUNT_PATH=$(echo ${SSH_AUTH_SOCK} || echo "${HOME}/.ssh/id_rsa")

# Set up your new/old versions of Kubernetes
export OLD_K8S=<old-k8s-version>
export NEW_K8S=<new-k8s-version>
export OLD_K8S_CLIENT=<old-k8s-client-version>
export NEW_K8S_CLIENT=<new-k8s-client-version>
export OLD_K3S_VER="${OLD_K8S}-k3s1"
export NEW_K3S_VER="${NEW_K8S}-k3s1"
export RELEASE_BRANCH=<k8s-release-branch>
export GOPATH=$(go env GOPATH)

# clean old builds
rm -rf _output
```

--------------------------------

### Run Code Formatting

Source: https://github.com/k3s-io/k3s/blob/main/CONTRIBUTING.md

Execute the make format command to ensure your code adheres to the project's formatting standards. This command utilizes goimports.

```bash
make format
```

--------------------------------

### Create Build Container with Docker

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/expanded/build_container.md

Use Docker to build the custom container image. The Dockerfile content is piped to the 'docker build' command, and the image is tagged using the GOIMAGE variable.

```bash
echo -e $BUILD_CONTAINER | docker build -t ${GOIMAGE}-dev -
```

--------------------------------

### Build K3s Without Linting

Source: https://github.com/k3s-io/k3s/blob/main/BUILDING.md

Build the K3s binaries using 'make' while skipping linting. This is useful if you have uncommitted changes.

```bash
SKIP_VALIDATE=true make
```

--------------------------------

### Define Build Container Dockerfile Content

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/expanded/build_container.md

Set the BUILD_CONTAINER environment variable to the content of a Dockerfile. This Dockerfile specifies the base image and installs necessary build tools and dependencies.

```bash
export BUILD_CONTAINER="FROM ${GOIMAGE}
RUN apk add --no-cache bash gnupg git make tar gzip curl git coreutils rsync alpine-sdk"
```

--------------------------------

### Secrets Encryption Status Output (Finished)

Source: https://github.com/k3s-io/k3s/wiki/Secrets-Encryption-Test-Plan

Example console output indicating that the secrets encryption re-encryption process has finished. It shows the active key and confirms all hashes match.

```text
Encryption Status: Enabled
Current Rotation Stage: reencrypt_finished
Server Encryption Hashes: All hashes match
Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey-2021-12-08T21:34:03Z
```

--------------------------------

### Rotate Server Token (Generate New)

Source: https://github.com/k3s-io/k3s/blob/main/docs/adrs/server-token-rotation.md

Use this command on the first server in an HA setup to rotate the existing token to a newly generated one. This is useful for periodic rotation.

```bash
k3s token rotate -t <OLD_TOKEN>
```

--------------------------------

### Update PATH for GNU Utilities on macOS

Source: https://github.com/k3s-io/k3s/blob/main/docs/contrib/development.md

Prepends the installed GNU utilities to the PATH environment variable in your shell init script. This ensures that the correct GNU versions of commands are used.

```sh
GNUBINS="$(find /usr/local/opt -type d -follow -name gnubin -print)"

for bindir in ${GNUBINS[@]};
  do
  PATH=$bindir:$PATH
done

export PATH
```

--------------------------------

### Generate Release Notes with Docker

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/expanded/release_notes.md

Use this command to generate release notes using the rancher/ecm-distro-tools Docker image. Ensure you have a GitHub token set as an environment variable.

```bash
export GHT=$GITHUB_TOKEN
export PREVIOUS_RELEASE='v1.23.12+k3s1'
export LAST_RELEASE='v1.23.13-rc2+k3s1'
docker run --rm -e GITHUB_TOKEN=$GHT rancher/ecm-distro-tools:latest gen_release_notes -r k3s -m $LAST_RELEASE -p $PREVIOUS_RELEASE
```

--------------------------------

### Secrets Encryption Status Output (Enabled)

Source: https://github.com/k3s-io/k3s/wiki/Secrets-Encryption-Test-Plan

Example console output showing the status of secrets encryption when enabled. It indicates the rotation stage and lists active and inactive encryption keys.

```text
Encryption Status: Enabled
Current Rotation Stage: reencrypt_active
Server Encryption Hashes: All hashes match
Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey-2021-12-08T21:34:03Z
        AES-CBC   aescbckey
```

--------------------------------

### Make Push Script Executable

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/kubernetes-upgrade.md

After generating the release tags, save the outputted git push commands to a file named push.sh and make it executable.

```sh
chmod +x push.sh
```

--------------------------------

### Rotate Server Token (Specify New)

Source: https://github.com/k3s-io/k3s/blob/main/docs/adrs/server-token-rotation.md

Use this command on the first server in an HA setup to rotate the existing token to a specific new token. This is useful for reactive rotation after a leak.

```bash
k3s token rotate -t <OLD_TOKEN> --new-token <NEW_TOKEN>
```

--------------------------------

### GitHub Issue Search for PRs

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/expanded/release_notes.md

Example GitHub issue search query to find pull requests for a specific branch merged after a certain date. Adjust the branch, date, and sorting as needed.

```bash
is:pr base:release-1.23 merged:>2022-09-28 sort:created-asc
```

--------------------------------

### Define Go Base Image

Source: https://github.com/k3s-io/k3s/blob/main/docs/release/expanded/build_container.md

Set the GOIMAGE environment variable to the desired Go container image, including the determined Go version and Alpine variant. This image will serve as the base for the custom build container.

```bash
export GOIMAGE="golang:${GOVERSION}-alpine"
```

--------------------------------

### Example etcd S3 Configuration Secret

Source: https://github.com/k3s-io/k3s/blob/main/docs/adrs/etcd-s3-secret.md

Define a Kubernetes Secret in the 'kube-system' namespace to store etcd S3 snapshot configuration. Fields within this secret correspond to K3s server CLI flags and configuration keys. The secret is checked each time a snapshot operation is performed and is only used for save operations.

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: k3s-etcd-snapshot-s3-config
  namespace: kube-system
stringData:
  etcd-s3-endpoint: ""
  etcd-s3-endpoint-ca: ""
  etcd-s3-endpoint-ca-name: ""
  etcd-s3-skip-ssl-verify: "false"
  etcd-s3-access-key: "AWS_ACCESS_KEY_ID"
  etcd-s3-secret-key: "AWS_SECRET_ACCESS_KEY"
  etcd-s3-bucket: "bucket"
  etcd-s3-folder: "folder"
  etcd-s3-region: "us-east-1"
  etcd-s3-insecure: "false"
  etcd-s3-timeout: "5m"
  etcd-s3-proxy: ""
```