### NewRebootChecker Examples Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/internal.md Demonstrates how to use `NewRebootChecker` to create file-based and command-based reboot checkers, as well as an example of an error case. ```go import ( "github.com/kubereboot/kured/internal" log "github.com/sirupsen/logrus" ) // File-based checker (default) checker, err := internal.NewRebootChecker( "", // no custom command "/var/run/reboot-required", ) // Command-based checker (custom priority) checker, err := internal.NewRebootChecker( "systemd-needs-reboot", "/var/run/reboot-required", // ignored if command is set ) // Error case: invalid command syntax checker, err := internal.NewRebootChecker( "echo 'unclosed quote", "", ) // err: error parsing provided sentinel command ``` -------------------------------- ### Taint Constructor Example Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/taints.md Demonstrates how to instantiate a Taint object using the New constructor with a Kubernetes client, node ID, taint name, and effect. This setup is required before performing operations like enabling or disabling the taint. ```go import ( "github.com/kubereboot/kured/pkg/taints" v1 "k8s.io/api/core/v1" "k8s.io/client-go/kubernetes" ) client, _ := kubernetes.NewForConfig(config) taint := taints.New( client, "node-1", "weave.works/kured-node-reboot", v1.TaintEffectPreferNoSchedule, ) ``` -------------------------------- ### Initialize KubernetesBlockingChecker Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/blockers.md Example of how to initialize the KubernetesBlockingChecker. Ensure you have a valid Kubernetes clientset and define the selectors for critical pods. ```go import "k8s.io/client-go/kubernetes" import "github.com/kubereboot/kured/pkg/blockers" clientset, _ := kubernetes.NewForConfig(config) selectors := []string{ "component=critical-app", "tier=system", } checker := blockers.NewKubernetesBlockingChecker(clientset, "node-1", selectors) ``` -------------------------------- ### Create Daily Time Window Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/timewindow.md Example of creating a daily time window from 2am to 4am UTC using the EveryDay constant. ```go // Create a daily window from 2am to 4am window, _ := timewindow.New( timewindow.EveryDay, "02:00", "04:00", "UTC", ) ``` -------------------------------- ### Specific Days and Hours Window Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/timewindow.md Example of creating a time window for specific days (Monday, Wednesday, Friday) and hours. ```go window, _ := timewindow.New( []string{"1", "3", "5"}, // Monday, Wednesday, Friday "02:30", "04:30", "UTC", ) ``` -------------------------------- ### Example Usage of NodeMeta Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/daemonsetlock.md Demonstrates how to create a NodeMeta object using the current node's unschedulable status. This is passed to the Acquire method. ```go nodeMeta := daemonsetlock.NodeMeta{ Unschedulable: node.Spec.Unschedulable, } ``` -------------------------------- ### Release Lock Example Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/daemonsetlock.md Demonstrates how to release a lock held by the current node. It handles potential errors during the release process. ```go err := lock.Release() if err != nil { log.Errorf("Failed to release lock: %v", err) } ``` -------------------------------- ### Every Day All Day Window Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/timewindow.md Example of creating a time window that spans the entire day, every day. ```go window, _ := timewindow.New( timewindow.EveryDay, "0:00", "23:59:59", "UTC", ) ``` -------------------------------- ### Acquire Lock Example Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/daemonsetlock.md Demonstrates how to attempt to acquire a lock in a multi-node scenario. It checks if the lock was successfully acquired or if the maximum number of holders has been reached. ```go nodeMeta := daemonsetlock.NodeMeta{Unschedulable: false} acquired, holders, err := lock.Acquire(nodeMeta) if acquired { log.Infof("Lock acquired. All holders: %s", holders) } else { log.Warnf("Max holders reached: %s", holders) } ``` -------------------------------- ### Kured Configuration for Signal-Based Reboot Source: https://github.com/kubereboot/kured/blob/main/_autodocs/configuration.md Example demonstrating how to configure Kured to use a specific signal for initiating reboots instead of the default method. ```bash kured \ --node-id $(hostname) \ --reboot-method signal \ --reboot-signal 39 \ --period 60m ``` -------------------------------- ### Maintenance Window (Weekends) Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/timewindow.md Example of creating a time window for weekend maintenance, covering almost the entire weekend. ```go window, _ := timewindow.New( []string{"sa", "su"}, "00:00", "23:59:59", "UTC", ) ``` -------------------------------- ### Run Basic Golang Tests Source: https://github.com/kubereboot/kured/blob/main/CONTRIBUTING.md Execute basic Go code tests to get a good idea of the code behavior. ```bash make test ``` -------------------------------- ### Example kured_reboot_required Metric Source: https://github.com/kubereboot/kured/blob/main/_autodocs/metrics.md This is an example of the kured_reboot_required gauge metric, indicating that node-1 requires a reboot. ```text kured_reboot_required{node="node-1"} 1 ``` -------------------------------- ### Initialize Single-Node DaemonSetLock Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/daemonsetlock.md Example demonstrating how to initialize a single-node lock using the New constructor. This is used when only one node should hold the lock at a time. ```go import ( "k8s.io/client-go/kubernetes" "github.com/kubereboot/kured/pkg/daemonsetlock" "time" ) config, _ := rest.InClusterConfig() client, _ := kubernetes.NewForConfig(config) // Single-node lock lock := daemonsetlock.New( client, "node-1", "kube-system", "kured", "weave.works/kured-node-lock", 0, // No TTL 1, // Single lock holder 0, // No release delay ) ``` -------------------------------- ### Log and Process Active Blocking Alerts Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/blockers.md Example of calling ActiveAlerts to get a list of blocking alerts and logging each one. Includes basic error handling for the query. ```go alerts, err := checker.ActiveAlerts() if err != nil { log.Errorf("Failed to query alerts: %v", err) } for _, alert := range alerts { log.Infof("Blocking alert found: %s", alert) } ``` -------------------------------- ### Initialize Multi-Node DaemonSetLock Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/daemonsetlock.md Example showing how to initialize a multi-node lock with a specified TTL, concurrency limit, and release delay. This allows multiple nodes to hold the lock simultaneously up to the concurrency limit. ```go // Multi-node lock (up to 3 concurrent reboots) multiLock := daemonsetlock.New( client, "node-1", "kube-system", "kured", "weave.works/kured-node-lock", 30*time.Minute, // 30-minute lock TTL 3, // Allow 3 concurrent lock holders 5*time.Minute, // Wait 5 minutes before releasing ) ``` -------------------------------- ### Initialize PrometheusBlockingChecker Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/blockers.md Example of initializing the PrometheusBlockingChecker with API configuration, a regular expression for filtering alert names, and boolean flags for firing state and match-only behavior. ```go import "github.com/prometheus/client_golang/api" import "github.com/kubereboot/kured/pkg/blockers" import "regexp" promConfig := api.Config{Address: "http://prometheus:9090"} filter := regexp.MustCompile("unimportant-alert") checker := blockers.NewPrometheusBlockingChecker( promConfig, filter, true, // firingOnly false, // not filterMatchOnly (default exclude matching alerts) ) ``` -------------------------------- ### Enable Taint Example Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/taints.md Shows how to use the Enable method on a Taint object to apply a taint to a node, preventing new pods from being scheduled. This operation modifies the node's spec and logs the change. ```go taint := taints.New(client, "node-1", "maintenance", v1.TaintEffectNoSchedule) taint.Enable() // Node now has taint, no new pods will be scheduled ``` -------------------------------- ### CommandChecker Example Usage Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/checkers.md Demonstrates how to create both unprivileged and privileged CommandChecker instances. The privileged checker uses nsenter to access the host namespace, useful for checking systemd status. ```go import "github.com/kubereboot/kured/pkg/checkers" // Unprivileged command checker, err := checkers.NewCommandChecker( "test -f /var/run/reboot-required", 1, false, ) // Privileged command (enters host namespace) privChecker, err := checkers.NewCommandChecker( "systemctl is-system-running", 1, true, // Enables nsenter to check systemd in host namespace ) if err != nil { log.Fatalf("Failed to create checker: %v", err) } ``` -------------------------------- ### New TimeWindow Constructor Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/timewindow.md Creates a TimeWindow instance from a list of days, start and end times, and a timezone. Supports various formats for days, times, and timezones. ```go func New( days []string, startTime string, endTime string, location string, ) (*TimeWindow, error) ``` ```go import ( "github.com/kubereboot/kured/pkg/timewindow" ) // Weekday business hours in UTC window, err := timewindow.New( []string{"mo", "tu", "we", "th", "fr"}, "09:00", "17:00", "UTC", ) if err != nil { log.Fatalf("Failed to create time window: %v", err) } // Full day, specific timezone window, _ := timewindow.New( []string{"sunday", "saturday"}, "0:00", "23:59:59", "America/New_York", ) // Every day with variable time format window, _ := timewindow.New( timewindow.EveryDay, "2am", "6pm", "Europe/London", ) ``` -------------------------------- ### Kured Configuration for Production with Blocking Conditions Source: https://github.com/kubereboot/kured/blob/main/_autodocs/configuration.md Production configuration example that includes Prometheus integration, alert filtering, blocking pod selection, specific reboot windows, and force reboot options. ```bash kured \ --node-id $(hostname) \ --prometheus-url http://prometheus:9090 \ --alert-filter-regexp "expected-maintenance-alert" \ --blocking-pod-selector "tier=critical" \ --reboot-days mo,we,fr \ --start-time 02:00 \ --end-time 04:00 \ --time-zone America/Chicago \ --force-reboot \ --notify-url "slack://hooks.slack.com/..." \ --lock-ttl 30m ``` -------------------------------- ### Kured Development Configuration Source: https://github.com/kubereboot/kured/blob/main/_autodocs/README.md Minimal setup for testing Kured on a single node. Sets a custom node ID and a reboot sentinel file, with a short check period. ```bash kured \ --node-id test-node \ --reboot-sentinel /tmp/reboot-required \ --period 1m ``` -------------------------------- ### Check for Blocking Pods Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/blockers.md Example of using the IsBlocked method to determine if a reboot should be deferred. If IsBlocked returns true, the reboot is blocked. ```go if checker.IsBlocked() { log.Infof("Reboot blocked by critical pods on this node") // Wait and retry later } ``` -------------------------------- ### Collect Metrics with curl Source: https://github.com/kubereboot/kured/blob/main/_autodocs/metrics.md Example of fetching kured metrics from the default endpoint and filtering for the reboot required metric using curl. ```bash curl http://localhost:8080/metrics | grep kured_reboot_required ``` -------------------------------- ### Handle Reboot Blocked by Alerts Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/blockers.md Example of using the IsBlocked method to conditionally prevent a reboot. Logs a warning if the reboot is blocked. ```go if checker.IsBlocked() { log.Warnf("Reboot blocked by active alerts") // Don't proceed with reboot } ``` -------------------------------- ### Command Line Reboot Method Configuration Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/internal.md Example of configuring kured to use the 'command' reboot method via command-line flags. This shows the necessary arguments for specifying the reboot command. ```bash kured \ --reboot-method command \ --reboot-command "/bin/systemctl reboot" \ ... ``` -------------------------------- ### Business Hours Time Window Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/timewindow.md Example of creating a time window representing typical business hours (9am to 5pm) on weekdays in a specific timezone. ```go window, _ := timewindow.New( []string{"mo", "tu", "we", "th", "fr"}, "09:00", "17:00", "America/Chicago", ) ``` -------------------------------- ### DaemonSetLock with Release Delay Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/daemonsetlock.md Example of initializing DaemonSetLock with a lock release delay. This allows the node to stabilize after a reboot before the lock is released, preventing rapid successive reboots. The delay occurs before the lock is released. ```go lock := daemonsetlock.New( client, nodeID, "kube-system", "kured", "weave.works/kured-node-lock", 0, // No TTL 1, 5*time.Minute, // Wait 5 minutes before releasing lock ) ``` -------------------------------- ### RebootRequired Method Example Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/checkers.md Shows how to use the RebootRequired method to check if a reboot is needed based on a custom command's exit status. If the method returns true, the application proceeds with reboot logic. ```go // Custom check command that exits 0 if reboot needed if checker.RebootRequired() { log.Infof("Custom check indicates reboot is needed") // Proceed with reboot } ``` -------------------------------- ### YAML: Container Security Context and Host PID Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/reboot.md Example container configuration for kured, specifying necessary security contexts and host PID namespace. This is required for the rebooter to function correctly. ```yaml securityContext: privileged: true # OR specific capabilities: capabilities: add: - SYS_ADMIN - SYS_BOOT hostPID: true ``` -------------------------------- ### Customize Kind E2E Test Execution Source: https://github.com/kubereboot/kured/blob/main/CONTRIBUTING.md Examples of passing arguments to `make e2e-test` to control test execution, such as running specific tests or extending timeouts. ```shell # Run only TestE2EWithSignal test for the kubernetes version named "current" (see kind file) make e2e-test ARGS="-run ^TestE2EWithSignal/current" # Run all tests but make sure to extend the timeout, for slower machines. make e2e-test ARGS="-timeout 1200s' ``` -------------------------------- ### DaemonSetLock with TTL Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/daemonsetlock.md Example of initializing DaemonSetLock with a Time-To-Live (TTL) duration. This prevents deadlocks in unreliable clusters by automatically expiring the lock after a set period. The TTL is checked on every Acquire/Holding operation. ```go lock := daemonsetlock.New( client, nodeID, "kube-system", "kured", "weave.works/kured-node-lock", 30*time.Minute, // Lock expires after 30 minutes 1, 0, ) ``` -------------------------------- ### Holding Lock Example Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/daemonsetlock.md Shows how to check if the current node is currently holding a valid lock within the multi-node lock group. It returns whether the node holds the lock and its associated data. ```go holding, lockData, err := lock.Holding() if holding { log.Infof("Node holds lock in multi-lock group") } ``` -------------------------------- ### Consume DelayTick with explicit select Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/delaytick.md This example demonstrates consuming a delaytick channel using a select statement within a goroutine. This pattern allows for more control, including the ability to stop the ticker by closing a separate 'done' channel. ```go ticker := delaytick.New(source, 1*time.Minute) done := make(chan bool) go func() { for { select { case <-done: return case t := <-ticker: log.Infof("Tick at %v", t) } } }() // Stop ticking by closing done channel close(done) ``` -------------------------------- ### Disable Taint Example Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/taints.md Demonstrates using the Disable method to remove a taint from a node, allowing normal pod scheduling to resume. This operation modifies the node's spec and may trigger pod admission reconciliation. ```go taint.Disable() // Node taint removed, normal scheduling can resume ``` -------------------------------- ### Overnight Window Spanning Midnight Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/timewindow.md Example of creating an overnight time window that starts in the evening and ends in the morning of the next day. ```go window, _ := timewindow.New( timewindow.EveryDay, "22:00", // 10pm "06:00", // 6am (next day) "Europe/London", ) ``` -------------------------------- ### Initialize and Use Rebooter and Checker Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/internal.md Demonstrates the typical usage pattern for initializing the Rebooter and RebootChecker using factory functions and integrating them into a control loop. Ensure necessary imports are present. ```Go import ( "github.com/kubereboot/kured/internal" log "github.com/sirupsen/logrus" ) // Parse configuration from flags/environment var rebootMethod string // "command" or "signal" var rebootCommand string // e.g., "/bin/systemctl reboot" var rebootSignal int // e.g., 39 var rebootSentinelCommand string // custom check command var rebootSentinelFile string // e.g., "/var/run/reboot-required" // Create implementations rebooter, err := internal.NewRebooter(rebootMethod, rebootCommand, rebootSignal) if err != nil { log.Fatalf("Failed to initialize rebooter: %v", err) } checker, err := internal.NewRebootChecker(rebootSentinelCommand, rebootSentinelFile) if err != nil { log.Fatalf("Failed to initialize checker: %v", err) } // Use in control loop for { if checker.RebootRequired() { if err := rebooter.Reboot(); err != nil { log.Errorf("Reboot failed: %v", err) } } time.Sleep(period) } ``` -------------------------------- ### Initialize File and Command Checkers Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/checkers.md Demonstrates how to initialize both a default file-based reboot checker and a custom command-based checker. The file checker is suitable for unprivileged use, while the command checker can be configured for privileged execution. ```go import "github.com/kubereboot/kured/pkg/checkers" // File-based checker (default, unprivileged) fileChecker, _ := checkers.NewFileRebootChecker("/var/run/reboot-required") // Command-based checker (custom, privileged) cmdChecker, _ := checkers.NewCommandChecker( "systemd-needs-reboot", 1, true, ) ``` -------------------------------- ### Go: Create and Execute Rebooter Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/reboot.md Demonstrates how to create either a signal-based or command-based rebooter and execute the reboot command within a control loop. Use this when integrating kured into your system's reboot management. ```go import ( "github.com/kubereboot/kured/pkg/reboot" ) // Create rebooter based on configuration var rebooter reboot.Rebooter var err error if useSignal { rebooter, err = reboot.NewSignalRebooter(39) } else { rebooter, err = reboot.NewCommandRebooter("/bin/systemctl reboot") } if err != nil { log.Fatalf("Failed to create rebooter: %v", err) } // Execute reboot in control loop holding, _, _ := lock.Holding() if holding { // Ready to reboot if err := rebooter.Reboot(); err != nil { log.Errorf("Reboot failed: %v", err) // Decide: force? retry? notify? } // Reboot initiated, wait indefinitely for { log.Infof("Waiting for reboot to complete...") time.Sleep(1 * time.Minute) } } ``` -------------------------------- ### Test kured with Minikube Source: https://github.com/kubereboot/kured/blob/main/CONTRIBUTING.md A comprehensive set of commands to set up minikube, build and publish the kured image, apply manifests, and monitor logs for testing. ```cli # start minikube minikube start --driver=kvm2 --kubernetes-version # build kured image and publish to registry accessible by minikube make dev-image minikube-publish # edit kured-ds.yaml to # - point to new image # - change e.g. period and reboot-days option for immediate results minikube kubectl -- apply -f kured-rbac.yaml minikube kubectl -- apply -f kured-ds.yaml minikube kubectl -- logs daemonset.apps/kured -n kube-system -f # In separate terminal minikube ssh sudo touch /var/run/reboot-required minikube logs -f ``` -------------------------------- ### Test kured with Kind (Hard Way) Source: https://github.com/kubereboot/kured/blob/main/CONTRIBUTING.md Commands to create a kind cluster, set up reboot-required files on nodes, and follow the coordinated reboot process. ```cli # create kind cluster kind create cluster --config .github/kind-cluster-.yaml # create reboot required files on pre-defined kind nodes ./tests/kind/create-reboot-sentinels.sh # check if reboot is working fine ./tests/kind/follow-coordinated-reboot.sh ``` -------------------------------- ### Sum of Nodes Requiring Reboot Source: https://github.com/kubereboot/kured/blob/main/_autodocs/metrics.md Use this PromQL query to get the total count of nodes that require a reboot. ```promql sum(kured_reboot_required{job="kured"}) ``` -------------------------------- ### Typical Reboot Coordination Integration Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/timewindow.md Demonstrates how to integrate the timewindow package into a main reboot loop to check if the current time falls within a configured window before proceeding with reboot checks. ```go import ( "time" "github.com/kubereboot/kured/pkg/timewindow" ) // Create window from configuration window, err := timewindow.New( []string{"mo", "tu", "we", "th", "fr"}, "02:00", "06:00", "America/New_York", ) if err != nil { log.Fatalf("Invalid time window: %v", err) } // Check in main reboot loop for { if !window.Contains(time.Now()) { log.Infof("Outside reboot window: %v", window) continue } // Time is within window, check other conditions if checker.RebootRequired() { // Acquire lock, drain, reboot } time.Sleep(period) } ``` -------------------------------- ### Initialize DelayTick with Different Random Sources Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/delaytick.md Shows how to initialize DelayTick with either a time-based seed for non-deterministic behavior or a fixed seed for deterministic testing. Uses `math/rand` for performance. ```go import "math/rand" // Time-based seed (non-deterministic) source := rand.NewSource(time.Now().UnixNano()) ticker := delaytick.New(source, 1*time.Minute) // Fixed seed (deterministic, for testing) source := rand.NewSource(42) ticker := delaytick.New(source, 1*time.Minute) ``` -------------------------------- ### Pod Toleration Configuration Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/taints.md Example of a Pod specification with tolerations to ignore the 'weave.works/kured-node-reboot' taint. This allows the pod to be scheduled on nodes that have this taint applied. ```yaml apiVersion: v1 kind: Pod metadata: name: critical-pod spec: tolerations: - key: "weave.works/kured-node-reboot" operator: "Exists" effect: "PreferNoSchedule" containers: - name: app image: my-app:latest ``` -------------------------------- ### Create and Use Reboot Blockers Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/blockers.md Demonstrates how to instantiate Prometheus and Kubernetes blocking checkers and use them to determine if a reboot is blocked. Ensure Prometheus and Kubernetes clients are properly configured. ```go import ( "github.com/kubereboot/kured/pkg/blockers" "github.com/prometheus/client_golang/api" ) // Create blockers promChecker := blockers.NewPrometheusBlockingChecker( api.Config{Address: "http://prometheus:9090"}, nil, false, false, ) kubeChecker := blockers.NewKubernetesBlockingChecker( client, nodeName, []string{"tier=critical"}, ) blockCheckers := []blockers.RebootBlocker{ promChecker, kubeChecker, } // Check if blocked if blockers.RebootBlocked(blockCheckers...) { log.Warnf("Reboot is blocked, will retry later") continue } // Proceed with reboot log.Infof("No blocking conditions, proceeding with reboot") ``` -------------------------------- ### Automate kured Testing with Kind (Easy Way) Source: https://github.com/kubereboot/kured/blob/main/CONTRIBUTING.md Build the kured:dev image, manifests, and run the 'long' Go end-to-end tests using `make e2e-test`. ```cli # Build kured:dev image, build manifests, and run the "long" go tests make e2e-test ``` -------------------------------- ### Check Lock Holding Status Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/daemonsetlock.md Example of how to check if a lock is currently held and retrieve its data. This is useful for determining if a node has already acquired the lock. ```go holding, lockData, err := lock.Holding() if holding { log.Infof("Node %s acquired lock at %v", lockData.NodeID, lockData.Created) } ``` -------------------------------- ### Internal Rebooter and Checker Initialization Errors Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/internal.md Demonstrates fatal startup errors during the initialization of Rebooter and RebootChecker with invalid configurations. These errors indicate issues with the provided method names or command parsing. ```go rebooter, err := internal.NewRebooter("invalid-method", "", 0) if err != nil { // err contains: "invalid reboot-method configured invalid-method, expected signal or command" } checker, err := internal.NewRebootChecker("bad 'quote", "") if err != nil { // err contains: error parsing provided sentinel command } ``` -------------------------------- ### Kured Multi-Cluster Configuration Source: https://github.com/kubereboot/kured/blob/main/_autodocs/README.md Configuration for multi-cluster setups allowing concurrent reboots with staggering. Specifies concurrency limit, check period, and weekend reboot days. ```bash kured \ --node-id $(hostname) \ --concurrency 3 \ --period 60m \ --reboot-days sa,su ``` -------------------------------- ### Reference Implementation of Kured Source: https://github.com/kubereboot/kured/blob/main/_autodocs/INDEX.md This Go code represents the main entry point and reference implementation for the kured application. It demonstrates how the various components of kured are wired together. ```go package main import ( "context" "flag" "os" "os/signal" "syscall" "time" "github.com/weaveworks/kured/pkg/blockers" "github.com/weaveworks/kured/pkg/checkers" "github.com/weaveworks/kured/pkg/daemonsetlock" "github.com/weaveworks/kured/pkg/logger" "github.com/weaveworks/kured/pkg/reboot" "github.com/weaveworks/kured/pkg/taints" "github.com/weaveworks/kured/pkg/timewindow" ) func main() { ctx, cancel := context.WithCancel(context.Background()) defer cancel() // Flags logLevel := flag.String("log-level", "info", "Log level (debug, info, warn, error)") flag.Parse() // Logger log := logger.New(*logLevel) // Signal handling sigChan := make(chan os.Signal, 1) ssignal.NotifyContext(ctx, sigChan, syscall.SIGINT, syscall.SIGTERM) // Components checker := checkers.NewFileRebootChecker(log) blocker := blockers.NewDaemonSetBlocker(log, daemonsetlock.New(log)) tw := timewindow.New(log) taintsChecker := taints.New(log) // Rebooter rebooter := reboot.New(log, checker, blocker, tw, taintsChecker) // Run go func() { log.Info("Starting kured") if err := rebooter.Run(ctx); err != nil { log.Error(err.Error()) os.Exit(1) } }() // Wait for signal <-ctx.Done() log.Info("Shutting down kured") // Give rebooter a moment to shut down gracefully time.Sleep(5 * time.Second) } ``` -------------------------------- ### DaemonSetLock Integration Pattern Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/daemonsetlock.md Demonstrates the typical usage of DaemonSetLock in a reboot control loop. It shows how to initialize the lock, acquire it before a reboot, and release it afterward. Ensure you have the necessary Kubernetes client configuration. ```go import ( "github.com/kubereboot/kured/pkg/daemonsetlock" "k8s.io/client-go/kubernetes" "k8s.io/client-go/rest" ) // Initialize config, _ := rest.InClusterConfig() client, _ := kubernetes.NewForConfig(config) lock := daemonsetlock.New( client, nodeID, "kube-system", "kured", "weave.works/kured-node-lock", 0, // No TTL 1, // Single lock holder 0, // No release delay ) // Acquire lock before reboot nodeMeta := daemonsetlock.NodeMeta{Unschedulable: node.Spec.Unschedulable} acquired, holder, err := lock.Acquire(nodeMeta) if !acquired { log.Warnf("Another node holds the lock: %s", holder) return } // Drain node and reboot log.Infof("Lock acquired, proceeding with drain and reboot") drain(client, node) reboot() // After reboot, release lock lock.Release() ``` -------------------------------- ### Kured Configuration for Development/Testing Source: https://github.com/kubereboot/kured/blob/main/_autodocs/configuration.md A basic configuration suitable for development or testing environments. It uses a temporary sentinel file and a short reboot period. ```bash kured \ --node-id test-node \ --reboot-sentinel /tmp/reboot-required \ --period 1m \ --concurrency 1 ``` -------------------------------- ### Creating Kubernetes Taints with Different Effects Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/taints.md Demonstrates how to create taints with 'PreferNoSchedule', 'NoSchedule', and 'NoExecute' effects using the taints.New function. Use 'PreferNoSchedule' for soft preferences, 'NoSchedule' to block new pods, and 'NoExecute' to evict existing pods. ```go import v1 "k8s.io/api/core/v1" // Soft preference (pods can still run if needed) taint := taints.New(client, node, "reboot", v1.TaintEffectPreferNoSchedule) // Hard block (no new pods) taint := taints.New(client, node, "reboot", v1.TaintEffectNoSchedule) // Aggressive (evict existing pods) taint := taints.New(client, node, "reboot", v1.TaintEffectNoExecute) ``` -------------------------------- ### Create and Use Delayed Ticker Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/delaytick.md Demonstrates how to create a delayed ticker for a main reboot check loop. The loop body executes after an initial random delay and then periodically at the specified interval. ```go import ( "math/rand" "time" "github.com/kubereboot/kured/pkg/delaytick" ) // Create delayed ticker for main reboot check loop period := 60 * time.Minute // Check every hour source := rand.NewSource(time.Now().UnixNano()) ticker := delaytick.New(source, period) // Consume ticks for range ticker { // This loop body runs after initial random delay, // then periodically every 60 minutes if !window.Contains(time.Now()) { continue } if !checker.RebootRequired() { continue } if blockers.RebootBlocked(blockCheckers...) { continue } // Coordinate reboot with lock, drain, and execute log.Infof("Reboot required, acquiring lock...") } ``` -------------------------------- ### Create and Acquire a DaemonSet Lock Source: https://github.com/kubereboot/kured/blob/main/_autodocs/README.md Demonstrates how to create a new distributed coordination lock using the daemonsetlock package and acquire it. Ensure the Kubernetes client and node ID are properly configured. The lock is automatically released upon function exit via defer. ```go import ( "github.com/kubereboot/kured/pkg/daemonsetlock" "k8s.io/client-go/kubernetes" ) // See daemonsetlock.md for detailed docs lock := daemonsetlock.New( client, nodeID, "kube-system", "kured", "weave.works/kured-node-lock", 0, // no TTL 1, // single lock holder 0, // no release delay ) nodeMeta := daemonsetlock.NodeMeta{Unschedulable: false} acquired, holder, err := lock.Acquire(nodeMeta) if acquired { // Proceed with reboot } deffer lock.Release() ``` -------------------------------- ### Go Error Handling for Reboot Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/reboot.md This Go snippet demonstrates how to handle errors returned by a reboot command. It includes logic for proceeding with a force-reboot or releasing a lock and retrying on failure. ```go if err := rebooter.Reboot(); err != nil { if forceReboot { log.Warnf("Reboot command failed but force-reboot is enabled, proceeding: %v", err) } else { log.Errorf("Reboot command failed: %v", err) // Release lock and retry lock.Release() continue } } ``` -------------------------------- ### Reboot Control Loop with File Checker Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/checkers.md Shows how to integrate the file-based reboot checker into a control loop to detect when a reboot is required. This snippet assumes the checker has already been initialized. ```go // Use in control loop for { if fileChecker.RebootRequired() { log.Infof("Reboot required - sentinel file detected") // Coordinate with lock, check blockers, and reboot } time.Sleep(1 * time.Minute) } ``` -------------------------------- ### NewFileRebootChecker Constructor Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/checkers.md Creates a new FileRebootChecker instance. It takes the path to the sentinel file as input. Currently, no validation is performed on the path. ```go import "github.com/kubereboot/kured/pkg/checkers" checker, _ := checkers.NewFileRebootChecker("/var/run/reboot-required") ``` -------------------------------- ### NewKubernetesBlockingChecker Constructor Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/blockers.md Creates a new KubernetesBlockingChecker instance. Use this to initialize the checker with a Kubernetes client, node name, and pod label selectors. ```go func NewKubernetesBlockingChecker( client *kubernetes.Clientset, nodename string, podSelectors []string, ) *KubernetesBlockingChecker ``` -------------------------------- ### NewCommandRebooter Constructor Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/reboot.md Creates a CommandRebooter instance by parsing a shell command string. It ensures the command is executed in the host's mount namespace via nsenter. ```go func NewCommandRebooter(rebootCommand string) (*CommandRebooter, error) ``` ```go import "github.com/kubereboot/kured/pkg/reboot" // Simple reboot command rebooter, err := reboot.NewCommandRebooter("/bin/systemctl reboot") if err != nil { log.Fatalf("Failed to create rebooter: %v", err) } // Custom reboot command with arguments rebooter, _ := reboot.NewCommandRebooter("/bin/systemctl reboot --force") // Command with complex quoting rebooter, _ := reboot.NewCommandRebooter("shutdown -r +5 'Server reboot in 5 minutes'") ``` -------------------------------- ### Kured Integration for Reboot Coordination with Taints Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/taints.md Shows a typical Kured integration pattern for reboot coordination. It includes creating a taint to prevent pod scheduling, disabling it when a reboot is not needed, and enabling it to discourage new scheduling when reboot is blocked by other factors. ```go import ( "github.com/kubereboot/kured/pkg/taints" v1 "k8s.io/api/core/v1" ) // Create taint for preventing pod scheduling taint := taints.New( client, nodeID, "weave.works/kured-node-reboot", v1.TaintEffectPreferNoSchedule, ) // Remove taint during startup if reboot is not needed if !checker.RebootRequired() { taint.Disable() } // In main reboot loop for { if !window.Contains(time.Now()) { taint.Disable() continue } if !checker.RebootRequired() { taint.Disable() continue } if blockers.RebootBlocked(blockCheckers...) { // Enable taint to discourage new pod scheduling taint.Enable() continue } // Proceed with reboot lock.Acquire(nodeMeta) drain(client, node) rebooter.Reboot() } ``` -------------------------------- ### Create and Use a Delayed Ticker Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/delaytick.md This snippet demonstrates how to create a new delayed ticker using `delaytick.New` and then use it in a loop for periodic tasks. The ticker introduces a random delay before the first tick and then continues at a fixed interval, suitable for coordinating cluster-wide operations like reboots. ```go import ( "math/rand" "time" "github.com/kubereboot/kured/pkg/delaytick" "github.com/kubereboot/kured/pkg/daemonsetlock" "log" ) // Create delayed ticker checkPeriod := 60 * time.Minute source := rand.NewSource(time.Now().UnixNano()) ticker := delaytick.New(source, checkPeriod) // Main reboot coordination loop for range ticker { log.Infof("Starting reboot eligibility check (after random delay)") // Check reboot conditions if !checker.RebootRequired() { log.Infof("Reboot not required") continue } log.Infof("Reboot is required, attempting to acquire lock...") // Try to acquire lock (blocks other nodes) acquired, holder, err := lock.Acquire(nodeMeta) if err != nil { log.Errorf("Lock error: %v", err) continue } if !acquired { log.Warnf("Lock held by %s, will try again next period", holder) continue } log.Infof("Lock acquired! Proceeding with reboot") // Drain node drain(client, node) // Execute reboot rebooter.Reboot() // Block until system reboots for { time.Sleep(1 * time.Minute) } } ``` -------------------------------- ### NewCommandRebooter Constructor Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/reboot.md Creates a new CommandRebooter that executes a specified shell command to trigger a reboot. It automatically wraps the command to run within the host's init namespace. ```APIDOC ### Constructor: NewCommandRebooter Creates a new CommandRebooter from a command string. ```go func NewCommandRebooter(rebootCommand string) (*CommandRebooter, error) ``` **Parameters:** - **rebootCommand** (`string`): Shell command to execute (will be shell-lexed) **Returns:** - `*CommandRebooter` — A configured command-based rebooter instance - `error` — Error if the command string cannot be parsed **Behavior:** - Parses the command string using shell lexer to split into arguments - Automatically wraps command with `/usr/bin/nsenter -m/proc/1/ns/mnt --` to execute in init namespace - This allows rebooting the host from within a container - Command must be executable in the init process's mount namespace **Error Cases:** - Returns error if the command is empty - Returns error if shell lexing fails (invalid quoting, etc.) **Example:** ```go import "github.com/kubereboot/kured/pkg/reboot" // Simple reboot command rebooter, err := reboot.NewCommandRebooter("/bin/systemctl reboot") if err != nil { log.Fatalf("Failed to create rebooter: %v", err) } // Custom reboot command with arguments rebooter, _ := reboot.NewCommandRebooter("/bin/systemctl reboot --force") // Command with complex quoting rebooter, _ := reboot.NewCommandRebooter("shutdown -r +5 'Server reboot in 5 minutes'") ``` ``` -------------------------------- ### Send Reboot Signal Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/reboot.md Executes the reboot by sending the configured signal to the init process (PID 1). This method is intended for Unix-like systems and may fail if permissions are insufficient or the system is not Unix-based. ```go // systemd reboot signal rebooter, _ := reboot.NewSignalRebooter(39) if err := rebooter.Reboot(); err != nil { log.Errorf("Failed to send reboot signal: %v", err) } ``` -------------------------------- ### TimeWindow String Method Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/timewindow.md Returns a human-readable string representation of the time window, including allowed days, time range, and timezone. ```go func (tw *TimeWindow) String() string ``` ```go window, _ := timewindow.New( []string{"mo", "we", "fr"}, "09:00", "17:00", "UTC", ) fmt.Println(window.String()) // Output: "---Mon---WedFri---- between 09:00 and 17:00 UTC" ``` -------------------------------- ### Run Lint Checks Locally Source: https://github.com/kubereboot/kured/blob/main/CONTRIBUTING.md Execute local lint checks using golangci-lint. ```bash make lint ``` -------------------------------- ### NewCommandChecker Constructor Source: https://github.com/kubereboot/kured/blob/main/_autodocs/api-reference/checkers.md Creates a new CommandChecker instance. Use this to set up a custom reboot detection command, specifying the command string, target PID, and whether to run with elevated privileges. ```go func NewCommandChecker( sentinelCommand string, pid int, privileged bool, ) (*CommandChecker, error) ``` -------------------------------- ### Collect Metrics with kubectl port-forward Source: https://github.com/kubereboot/kured/blob/main/_autodocs/metrics.md Demonstrates how to forward the kured port locally and then access the metrics endpoint using curl. This is useful for accessing metrics from pods not directly exposed. ```bash kubectl port-forward -n kube-system daemonset/kured 8080:8080 curl http://localhost:8080/metrics ```