### Runbook Template Example Source: https://github.com/prometheus-operator/runbooks/blob/main/content/docs/add-runbook.md An example of a complete runbook for the NodeFilesystemSpaceFillingUp alert, demonstrating the structure and content expected for new runbooks. It includes sections for Meaning, Impact, Diagnosis, and Mitigation. ```markdown # NodeFilesystemSpaceFillingUp ## Meaning This alert is based on an extrapolation of the space used in a file system. It fires if both the current usage is above a certain threshold _and_ the extrapolation predicts to run out of space in a certain time. This is a warning-level alert if that time is less than 24h. It's a critical alert if that time is less than 4h.
Full context Here is where you can optionally describe some more details about the alert. The "meaning" is the short version for an on-call engineer to quickly read through. The "details" are for learning about the bigger picture or the finer details. > NOTE: The blank lines above and below the text inside this `
` tag are [required to use markdown inside of html tags][1]
## Impact A filesystem running completely full is obviously very bad for any process in need to write to the filesystem. But even before a filesystem runs completely full, performance is usually degrading. ## Diagnosis Study the recent trends of filesystem usage on a dashboard. Sometimes a periodic pattern of writing and cleaning up can trick the linear prediction into a false alert. Use the usual OS tools to investigate what directories are the worst and/or recent offenders. Is this some irregular condition, e.g. a process fails to clean up behind itself, or is this organic growth? ## Mitigation [1]: https://github.github.com/gfm/#html-block ``` -------------------------------- ### Kubernetes Deployment Configuration Example Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/kubernetes/KubeDeploymentReplicasMismatch.md An example of a Kubernetes Deployment YAML configuration, illustrating how replicas and pod templates are defined. This can be useful for understanding the structure referenced in the diagnosis. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-deployment spec: replicas: 3 selector: matchLabels: app: my-app template: metadata: labels: app: my-app spec: containers: - name: my-container image: nginx:latest resources: requests: memory: "64Mi" cpu: "250m" limits: memory: "128Mi" cpu: "500m" ``` -------------------------------- ### Example Alert Details Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/kubernetes/KubeNodeReadinessFlapping.md Illustrates the typical format of alert details for a KubeNodeUnreachable alert, including the affected node. ```txt - alertname = KubeNodeUnreachable ... - node = node1.example.com ... ``` -------------------------------- ### KubeNodeNotReady Alert Details Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/kubernetes/KubeNodeNotReady.md Example output from a KubeNodeNotReady alert, showing alertname and the affected node. ```txt - alertname = KubeNodeNotReady ... - node = node1.example.com ... ``` -------------------------------- ### Check etcdctl Version Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/etcd/etcdInsufficientMembers.md Validates that the 'etcdctl' command is installed and accessible within the etcd container. ```shell etcdctl version ``` -------------------------------- ### Kube Node Unreachable Alert Details Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/kubernetes/KubeNodeUnreachable.md Example output from a KubeNodeUnreachable alert, showing key fields like alertname and the unreachable node. ```txt - alertname = KubeNodeUnreachable ... - node = node1.example.com ... ``` -------------------------------- ### Diagnose Alertmanager Member Inconsistency Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/alertmanager/AlertmanagerMembersInconsistent.md This snippet demonstrates how to diagnose Alertmanager cluster member inconsistencies by comparing Service endpoints with discovered pod IP addresses. It shows `kubectl describe svc` to get service endpoints and `kubectl get pod -o wide` to list pods and their IPs. ```shell $ kubectl describe svc alertmanager-main Name: alertmanager-main Namespace: monitoring ... Endpoints: 10.128.2.3:9095,10.129.2.5:9095,10.131.0.44:9095 $ kubectl get pod -o wide | grep alertmanager-main alertmanager-main-0 5/5 Running 0 11d 10.129.2.6 alertmanager-main-1 5/5 Running 0 2d16h 10.131.0.44 alertmanager-main-2 5/5 Running 0 6d 10.128.2.3 ``` -------------------------------- ### Kubernetes DaemonSet Documentation Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/kubernetes/KubeDaemonSetRolloutStuck.md References for understanding DaemonSet update strategies and troubleshooting stuck rolling updates in Kubernetes. ```APIDOC DaemonSet Update Strategy: https://kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/ DaemonSet Rolling Update Stuck: https://kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/#daemonset-rolling-update-is-stuck ``` ```APIDOC Debugging Pods: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application/#debugging-pods ``` -------------------------------- ### Kubernetes Pod Lifecycle and Debugging Concepts Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/kubernetes/KubePodNotReady.md References to Kubernetes documentation for understanding the Pod lifecycle and general Pod debugging strategies. This includes information on readiness and liveness probes, and common issues that prevent a Pod from becoming ready. ```APIDOC Kubernetes Pod Lifecycle: - pod-lifecycle: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/ Debugging Pods: - debugging-pods: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application/#debugging-pods ``` -------------------------------- ### Monitor Node Filesystem Inode Usage (PromQL) Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/node/NodeFilesystemFilesFillingUp.md This PromQL query helps visualize the free inodes on a specific node's filesystem, allowing for monitoring of inode usage trends. It requires the instance and mountpoint labels from the alert. ```promql node_filesystem_files_free{ instance="", mountpoint="" } ``` -------------------------------- ### Get Alertmanager Pod Events Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/alertmanager/AlertmanagerClusterDown.md Retrieves Kubernetes events related to Alertmanager pods to help diagnose why pods might not be running. ```shell kubectl get events --field-selector involvedObject.kind=Pod | grep alertmanager ``` -------------------------------- ### Debug Node Filesystem Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/node/NodeFilesystemSpaceFillingUp.md These shell commands demonstrate how to debug a Kubernetes node's filesystem. It involves using `kubectl debug` to enter a chroot environment on the node to investigate and clean up disk space. ```shell NODE_NAME= kubectl -n default debug node/$NODE_NAME chroot /host ``` ```shell exit exit ``` -------------------------------- ### Get Events for a Specific Pod Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/alertmanager/AlertmanagerClusterCrashlooping.md Fetches events related to a specific Alertmanager pod to diagnose the root cause of crash looping. ```shell kubectl get events --field-selector involvedObject.name=alertmanager-main-0 ``` -------------------------------- ### Monitor etcd DB Quota Usage with PromQL Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/etcd/etcdBackendQuotaLowSpace.md These PromQL queries help monitor the etcd database's space consumption relative to its quota and calculate the potential space savings after defragmentation. ```promql (etcd_mvcc_db_total_size_in_bytes / etcd_server_quota_backend_bytes) * 100 ``` ```promql (etcd_mvcc_db_total_size_in_bytes - etcd_mvcc_db_total_size_in_use_in_bytes)/1024/1024 ``` -------------------------------- ### Diagnose Slow etcd gRPC Requests Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/etcd/etcdGRPCRequestsSlow.md This query helps to verify the 99th percentile of etcd gRPC request handling times, providing a timeline of when issues may have started. ```promql histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket{job=~".*etcd.*", grpc_type="unary"}[5m])) without(grpc_type)) ``` -------------------------------- ### Debug Node Filesystem Inode Usage (Shell) Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/node/NodeFilesystemFilesFillingUp.md This shell command sequence allows debugging a node's filesystem directly. It uses `oc debug` to access the node and `df -hi` to check inode usage for a specified mount point. Replace placeholders with actual values from the alert. ```shell $ MOUNT_POINT='' $ NODE_NAME='' $ oc debug "node/$NODE_NAME" $ df -hi "/host/$MOUNT_POINT" ``` -------------------------------- ### Kubernetes Pod Debugging Commands Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/kubernetes/KubePodNotReady.md Common kubectl commands to inspect a Pod's status, events, and logs for debugging purposes. These commands help identify issues related to readiness probes, pending states, and container execution. ```bash kubectl -n $NAMESPACE get pod $POD kubectl -n $NAMESPACE describe pod $POD kubectl -n $NAMESPACE logs $POD -c $CONTAINER ``` -------------------------------- ### Shell Command to Get etcd Container Logs Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/etcd/etcdHighNumberOfFailedGRPCRequests.md This shell command retrieves logs from etcd containers within a specified Kubernetes namespace. It uses kubectl to filter logs by app label. ```shell NAMESPACE="kube-etcd" kubectl logs -n $NAMESPACE -lapp=etcd etcd ``` -------------------------------- ### Prometheus Operator Runbook Template Source: https://github.com/prometheus-operator/runbooks/blob/main/archetypes/default.md A template for creating new runbooks within the Prometheus Operator project. It includes frontmatter for title, date, and draft status. ```go --- title: "{{ replace .Name \"-\" \" \" | title }}" date: {{ .Date }} draft: true --- ``` -------------------------------- ### Get Pods on a Specific Node Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/kubernetes/KubeletTooManyPods.md Retrieves all pods across all namespaces that are scheduled on a specified node. This is useful for diagnosing the KubeletTooManyPods alert by identifying the pods consuming capacity on a particular node. ```shell kubectl get pods --all-namespaces --field-selector spec.nodeName= ``` -------------------------------- ### Defragment etcd Database Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/etcd/etcdBackendQuotaLowSpace.md This command is used to defragment the etcd database, optimizing its storage and potentially freeing up space. It should be run in all etcd pods. ```shell etcdctl defrag ``` -------------------------------- ### Diagnose DaemonSet Status Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/kubernetes/KubeDaemonSetRolloutStuck.md Use kubectl to describe the DaemonSet and inspect its status, which is crucial for understanding the rollout process and identifying stuck states. ```bash kubectl -n $NAMESPACE describe daemonset $NAME ``` -------------------------------- ### Check etcd Disk WAL fsync Duration Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/etcd/etcdGRPCRequestsSlow.md This query measures the 99th percentile of etcd disk WAL fsync duration in seconds. Values under 10ms indicate a reasonably fast disk. ```promql histogram_quantile(0.99, sum by (instance, le) (irate(etcd_disk_wal_fsync_duration_seconds_bucket{job="etcd"}[5m]))) ``` -------------------------------- ### Kubernetes StatefulSet API Documentation Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/kubernetes/KubeStatefulSetGenerationMismatch.md References for Kubernetes StatefulSet concepts, including failed deployments and rolling updates. These links provide deeper insights into StatefulSet behavior and management within Kubernetes. ```APIDOC Kubernetes StatefulSet Concepts: Failed Deployment: URL: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#failed-deployment Description: Information on diagnosing and handling failed deployments, applicable to StatefulSets. Rolling Updates: URL: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#rolling-updates Description: Details on how StatefulSets handle rolling updates and rollbacks. Debugging Pods: URL: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application/#debugging-pods Description: General guidance on debugging pods in a Kubernetes cluster. ``` -------------------------------- ### Kubernetes Deployment Diagnosis Commands Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/kubernetes/KubeDeploymentGenerationMismatch.md This snippet provides essential kubectl commands for diagnosing Kubernetes deployment issues, such as checking rollout history, deployment status, and describing deployments. It helps in understanding the state of deployments and identifying potential causes of mismatches or failures. ```bash kubectl -n $NAMESPACE rollout history deployment $NAME ``` ```bash kubectl -n $NAMESPACE describe deployment $NAME ``` -------------------------------- ### PromQL Query for etcd Disk Fsync Duration Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/etcd/etcdHighFsyncDurations.md This PromQL query calculates the 99th percentile of etcd disk fsync duration over a 5-minute interval. It helps diagnose slow disk performance impacting etcd. ```promql histogram_quantile(0.99, sum by (instance, le) (irate(etcd_disk_wal_fsync_duration_seconds_bucket{job="etcd"}[5m]))) ``` -------------------------------- ### Monitor Node Filesystem Free Bytes Source: https://github.com/prometheus-operator/runbooks/blob/main/content/runbooks/node/NodeFilesystemSpaceFillingUp.md This PromQL query allows monitoring the amount of free bytes on a node's filesystem. It's useful for diagnosing filesystem space issues and understanding recent trends. ```promql node_filesystem_free_bytes ```