kured

https://kured.dev/docs/
Admin
kured (Kubernetes Reboot Daemon) is a Kubernetes daemonset that performs safe automatic node reboots...
Tokens:8,432
Snippets:128
Trust Score:7.4
Update:2 months ago
Show doc for...
Tokens:
Raw
### Install development tools

Source: https://kured.dev/docs/development

Run this command once after cloning the repository to download all required developer tools defined in mise.toml.

```bash
make install-tools
```

--------------------------------

### Install Kured with default manifests

Source: https://kured.dev/docs/installation

Fetches the latest release tag and applies the combined manifest to the cluster.

```bash
latest=$(curl -s https://api.github.com/repos/kubereboot/kured/releases | jq -r '.[0].tag_name')
kubectl apply -f "https://github.com/kubereboot/kured/releases/download/$latest/kured-$latest-combined.yaml"
```

--------------------------------

### Example taint configuration

Source: https://kured.dev/docs/faq

Example output showing a node taint in JSON and YAML formats.

```json
[
  {
    "effect": "NoExecute",
    "key": "CriticalAddonsOnly",
    "value": "true"
  }
]
```

```yaml
taints:
- effect: NoExecute
  key: CriticalAddonsOnly
  value: "true"
```

--------------------------------

### Customize E2E test execution

Source: https://kured.dev/docs/development

Examples of passing arguments to the E2E test suite for specific test selection or timeout adjustments.

```makefile
# Run only TestE2EWithSignal test for the kubernetes version named "current" (see kind file)
make e2e-test ARGS="-run ^TestE2EWithSignal/current"
# Run all https://github.com/kubereboot/kured/blob/main/tests but make sure to extend the timeout, for slower machines.
make e2e-test ARGS="-timeout 1200s'
```

--------------------------------

### Configure Email Notification URL for Kured

Source: https://kured.dev/docs/configuration

Configure email notifications using an SMTP URL. This example includes username, password, host, port, sender address, and recipient addresses.

```bash
smtp://username:password@host:port/?fromAddress=fromAddress&toAddresses=recipient1[,recipient2,...]
```

--------------------------------

### Configure Slack Notification URL for Kured

Source: https://kured.dev/docs/configuration

Configure Slack notifications by providing a URL. This example shows the format for a new Slack App integration, including bot token, channel, and bot name.

```bash
slack://xoxb:123456789012-1234567890123-4mt0t4l1YL3g1T5L4cK70k3N@<CHANNEL_NAME>?botname=<BOTNAME>
```

--------------------------------

### Alert for Failsafe Reboot Daemon

Source: https://kured.dev/docs/configuration

An example alert configuration to monitor the kured_reboot_required metric. This acts as a failsafe if the reboot daemon cannot automatically reboot nodes.

```prometheus
# Alert if a reboot is required for any machines. Acts as a failsafe for the
# reboot daemon, which will not reboot nodes if there are pending alerts save
```

--------------------------------

### Set Reboot Schedule

Source: https://kured.dev/docs/configuration

Define specific days, start and end times, and the timezone for reboots to occur. This helps in scheduling reboots during predictable hours.

```bash
--reboot-days=mon,tue,wed,thu,fri
--start-time=9am
--end-time=5pm
--time-zone=America/Los_Angeles
```

--------------------------------

### Configure Concurrent Reboots in Kured

Source: https://kured.dev/docs/configuration

Enable concurrent reboots by setting the `--concurrency` argument. This example allows up to three nodes to reboot simultaneously. Use with caution in production.

```bash
--concurrency=3
```

--------------------------------

### Configure Teams Notification URL for Kured

Source: https://kured.dev/docs/configuration

Configure Microsoft Teams notifications by providing a URL. This example includes group, tenant, alternative ID, group owner, and optional host.

```bash
teams://group@tenant/altId/groupOwner?host=organization.webhook.office.com
```

--------------------------------

### Configure Alert Filtering for Kured

Source: https://kured.dev/docs/configuration

When using Kured with alerts, specify `--alert-filter-regexp` to prevent deadlocks. This example filters for the 'RebootRequired' alert.

```bash
--alert-filter-regexp=^RebootRequired$
```

--------------------------------

### Configure Rocketchat Notification URL for Kured

Source: https://kured.dev/docs/configuration

Configure Rocket.Chat notifications by providing a URL. This example shows the general format including optional username, host, token, and recipient.

```bash
rocketchat://[username@]rocketchat-host/token[/channel|@recipient]
```

--------------------------------

### Customize Kured Notification Messages

Source: https://kured.dev/docs/configuration

Customize the text of Kured's notification messages for draining nodes using `--message-template-drain`. This example shows a template for a specific cluster and region.

```bash
--message-template-drain="Draining node %s part of *my-cluster* in region *xyz*"
```

--------------------------------

### Test Kured with Kind (Manual)

Source: https://kured.dev/docs/development

Manual steps to create a Kind cluster and trigger reboot sentinels.

```bash
# create kind cluster
kind create cluster --config .github/kind-cluster-<k8s-version>.yaml

# create reboot required files on pre-defined kind nodes
./https://github.com/kubereboot/kured/blob/main/tests/kind/create-reboot-sentinels.sh

# check if reboot is working fine
./https://github.com/kubereboot/kured/blob/main/tests/kind/follow-coordinated-reboot.sh
```

--------------------------------

### Run basic Go tests

Source: https://kured.dev/docs/development

Executes the basic test suite to verify core behavior.

```makefile
make test
```

--------------------------------

### Run local linting

Source: https://kured.dev/docs/development

Executes golangci-lint to verify code quality.

```makefile
make lint
```

--------------------------------

### Download Kured deployment manifest

Source: https://kured.dev/docs/faq

Commands to fetch the latest Kured release manifest from GitHub.

```bash
# while writing latest is 1.23.0
latest=$(curl -s https://api.github.com/repos/kubereboot/kured/releases | jq -r '.[0].tag_name')
wget "https://github.com/kubereboot/kured/releases/download/$latest/kured-$latest-combined.yaml"
```

--------------------------------

### Test Kured with Minikube

Source: https://kured.dev/docs/development

Commands to deploy and monitor Kured within a Minikube environment.

```bash
# start minikube
minikube start --driver=kvm2 --kubernetes-version <k8s-release>

# build kured image and publish to registry accessible by minikube
make dev-image minikube-publish

# edit kured-ds.yaml to
#   - point to new image
#   - change e.g. period and reboot-days option for immediate results

minikube kubectl -- apply -f kured-rbac.yaml
minikube kubectl -- apply -f kured-ds.yaml
minikube kubectl -- logs daemonset.apps/kured -n kube-system -f

# In separate terminal
minikube ssh
 sudo touch /var/run/reboot-required
minikube logs -f
```

--------------------------------

### Run automated E2E tests with Kind

Source: https://kured.dev/docs/development

Automates the end-to-end testing process using the CI-defined workflow.

```makefile
make e2e-test
```

--------------------------------

### Simulate reboot requirement

Source: https://kured.dev/docs/development

Creates the sentinel file on a node to trigger a reboot process.

```bash
sudo touch /var/run/reboot-required
```

--------------------------------

### Add Node Labels Before and After Reboot

Source: https://kured.dev/docs/configuration

Apply specific node labels before initiating a reboot and after the reboot process is complete. Labels can be comma-delimited or specified multiple times.

```bash
--pre-reboot-node-labels=zalando=notready
--post-reboot-node-labels=zalando=ready
```

--------------------------------

### Provoke Reboot on a Node

Source: https://kured.dev/docs/operation

Use this command to test your kured configuration by simulating a reboot requirement on a node. Ensure you have the necessary permissions.

```bash
sudo touch /var/run/reboot-required

```

--------------------------------

### Configure Kured tolerations

Source: https://kured.dev/docs/faq

Daemonset toleration configurations for Kured, showing default settings and the modification required to tolerate specific taints.

```yaml
tolerations:
- key: node-role.kubernetes.io/control-plane
    effect: NoSchedule
- key: node-role.kubernetes.io/master
    effect: NoSchedule
```

```yaml
tolerations:
- key: CriticalAddonsOnly
    operator: Exists
```

--------------------------------

### Kured Daemonset Usage

Source: https://kured.dev/docs/configuration

This displays the available command-line flags for the kured daemonset. Use these flags to customize kured's behavior regarding alerts, node draining, reboot scheduling, and notifications.

```bash
Kubernetes Reboot Daemon

Usage:
  kured [flags]

Flags:
      --alert-filter-regexp regexp.Regexp   alert names to ignore when checking for active alerts
      --alert-filter-match-only             Only block if the alert-filter-regexp matches active alerts
      --alert-firing-only                   only consider firing alerts when checking for active alerts
      --annotate-nodes                      if set, the annotations 'weave.works/kured-reboot-in-progress' and 'weave.works/kured-most-recent-reboot-needed' will be given to nodes undergoing kured reboots
      --blocking-pod-selector stringArray   label selector identifying pods whose presence should prevent reboots
      --drain-delay duration                delay drain for this duration (default: 0, disabled)
      --drain-grace-period int              time in seconds given to each pod to terminate gracefully, if negative, the default value specified in the pod will be used (default -1)
      --drain-pod-selector string           only drain pods with labels matching the selector (default: '', all pods)
      --drain-timeout duration              timeout after which the drain is aborted (default: 0, infinite time)
      --ds-name string                      name of daemonset on which to place lock (default "kured")
      --ds-namespace string                 namespace containing daemonset on which to place lock (default "kube-system")
      --end-time string                     schedule reboot only before this time of day (default "23:59:59")
      --force-reboot                        force a reboot even if the drain fails or times out
  -h, --help                                help for kured
      --lock-annotation string              annotation in which to record locking node (default "weave.works/kured-node-lock")
      --lock-release-delay duration         delay lock release for this duration (default: 0, disabled)
      --lock-ttl duration                   expire lock annotation after this duration (default: 0, disabled)
      --log-format string                   use text or json log format (default "text")
      --message-template-drain string       message template used to notify about a node being drained (default "Draining node %s")
      --message-template-reboot string      message template used to notify about a node being rebooted (default "Rebooting node %s")
      --message-template-uncordon string    message template used to notify about a node being successfully uncordoned (default "Node %s rebooted & uncordoned successfully!")
      --metrics-host string                 host where metrics will listen (default "")
      --metrics-port int                    port number where metrics will listen (default 8080)
      --node-id string                      node name kured runs on, should be passed down from spec.nodeName via KURED_NODE_ID environment variable
      --notify-url string                   notify URL for reboot notifications (cannot use with --slack-hook-url flags)
      --period duration                     sentinel check period (default 1h0m0s)
      --post-reboot-node-labels strings     labels to add to nodes after uncordoning
      --pre-reboot-node-labels strings      labels to add to nodes before cordoning
      --prefer-no-schedule-taint string     Taint name applied during pending node reboot (to prevent receiving additional pods from other rebooting nodes). Disabled by default. Set e.g. to "weave.works/kured-node-reboot" to enable tainting.
      --prometheus-url string               Prometheus instance to probe for active alerts
      --reboot-command string               command to run when a reboot is required (default "/bin/systemctl reboot")
      --reboot-days strings                 schedule reboot on these days (default [su,mo,tu,we,th,fr,sa])
      --reboot-delay duration               delay reboot for this duration (default: 0, disabled)
      --reboot-method                       method to use for reboots (default command), available: command, signal
      --reboot-signal                       signal to use for reboots (default 39 = SIGRTMIN+5).
      --reboot-sentinel string              path to file whose existence triggers the reboot command (default "/var/run/reboot-required")
      --reboot-sentinel-command string      command for which a zero return code will trigger a reboot command
      --skip-wait-for-delete-timeout int    when seconds is greater than zero, skip waiting for the pods whose deletion timestamp is older than N seconds while draining a node
      --slack-channel string                slack channel for reboot notifications
      --slack-hook-url string               slack hook URL for reboot notifications [deprecated in favor of --notify-url]
      --slack-username string               slack username for reboot notifications (default "kured")

```

--------------------------------

### Build Kured Inside GOPATH

Source: https://kured.dev/docs/building

Use this command to build Kured when your project is located inside your Go workspace, ensuring Go Modules are enabled.

```bash
GO111MODULE=on make
```

--------------------------------

### Build Kured Outside GOPATH

Source: https://kured.dev/docs/building

Use this command to build Kured when your project is located outside of your Go workspace.

```bash
make
```

--------------------------------

### Verify Kured daemonset

Source: https://kured.dev/docs/faq

Command to check the status of the Kured daemonset in the kube-system namespace.

```bash
kubectl -n kube-system get daemonsets/kured
```

--------------------------------

### Inspect node taints

Source: https://kured.dev/docs/faq

Commands to retrieve and inspect taints on cluster nodes using JSON, YAML, or describe output.

```bash
kubectl get nodes ${NODENAME} -o json | jq .spec.taints
```

```bash
kubectl get nodes ${NODENAME} -o yaml | code -
```

```bash
kubectl describe nodes ${NODENAME}
```

--------------------------------

### Update Kind cluster configuration

Source: https://kured.dev/docs/development

Commands to rotate the current Kind cluster configuration to the previous version before updating to a new one.

```bash
cp .github/kind-cluster-current.yaml .github/kind-cluster-previous.yaml
```

--------------------------------

### Configure Reboot Sentinel Command

Source: https://kured.dev/docs/configuration

Use a command to determine if a reboot is required. The command should exit with code 0 if a reboot is needed. This overrides the sentinel file check.

```yaml
configuration:
  rebootSentinelCommand: sh -c "! needs-restarting --reboothint"
```

--------------------------------

### Manually Disable Reboots with Annotation

Source: https://kured.dev/docs/operation

Temporarily stop kured from rebooting nodes by manually acquiring the cluster lock. Remember to release the lock afterwards.

```bash
kubectl -n kube-system annotate ds kured weave.works/kured-node-lock='{"nodeID":"manual"}'

```

--------------------------------

### Block Reboots with Pod Selectors

Source: https://kured.dev/docs/configuration

Prevent reboots on a node if specific pods are scheduled on it, based on label selectors. Multiple selectors can be used with commas for 'AND' logic or specified multiple times for 'OR' logic.

```bash
--blocking-pod-selector=runtime=long,cost=expensive
```

```bash
--blocking-pod-selector=name=temperamental
```

--------------------------------

### Sign-off commit format

Source: https://kured.dev/docs/development

The required format for the Developer Certificate of Origin sign-off in commit messages.

```text
Signed-off-by: Jane Doe <jane.doe@example.com>
```

--------------------------------

### Configure Kured Lock Annotation

Source: https://kured.dev/docs/configuration

Override the default annotation name used by Kured for locking with the `--lock-annotation` argument. The default is generally safe to use.

```bash
--lock-annotation=<annotation-name>
```

--------------------------------

### Update Kubernetes dependencies

Source: https://kured.dev/docs/development

Command to update the kubectl dependency version in the project.

```bash
go get k8s.io/kubectl@v0.{version}
```

--------------------------------

### Sign commits automatically

Source: https://kured.dev/docs/development

Use this flag with the git commit command to automatically append the required sign-off signature.

```bash
git commit -s
```

--------------------------------

### Block Reboots with Prometheus Alerts

Source: https://kured.dev/docs/configuration

Prevent reboots when active alerts are present in Prometheus. Specify the Prometheus URL and optionally filter alerts by regular expression or alert status.

```bash
--prometheus-url=http://prometheus.monitoring.svc.cluster.local
```

```bash
--alert-filter-regexp=^(RebootRequired|AnotherBenignAlert|...$
```

```bash
--alert-firing-only=true
```

```bash
--alert-filter-match-only=true
```

--------------------------------

### Manually Remove Cluster Lock

Source: https://kured.dev/docs/operation

In cases of permanent node failure during a reboot, use this command to remove the cluster lock entirely. The trailing hyphen is crucial for removing the annotation.

```bash
kubectl -n kube-system annotate ds kured weave.works/kured-node-lock-

```

--------------------------------

### Kured Reboot Required Prometheus Metric

Source: https://kured.dev/docs/configuration

Exposes a gauge metric indicating if an OS requires a reboot due to software updates. This metric can be used to create alerts for manual intervention.

```prometheus
# HELP kured_reboot_required OS requires reboot due to software updates.
# TYPE kured_reboot_required gauge
kured_reboot_required{node="ip-xxx-xxx-xxx-xxx.ec2.internal"} 0
```
Tokens:
Raw