### Install Robusta Pre-release Version Source: https://github.com/robusta-dev/robusta/blob/master/docs/setup-robusta/upgrade.rst Install a beta version of Robusta using the --devel flag with helm upgrade. ```bash helm upgrade --devel ``` -------------------------------- ### Example generated_values.yaml Source: https://github.com/robusta-dev/robusta/blob/master/docs/setup-robusta/gitops/flux.rst An example of a generated_values.yaml file for Robusta installation. Ensure clusterName and other necessary configurations are present. ```yaml clusterName: my_cluster_name # <- This is the line to be added globalConfig: signing_key: xxxxxx account_id: xxxxxx sinksConfig: - robusta_sink: name: robusta_ui_sink token: xxxxxx enablePrometheusStack: true enablePlatformPlaybooks: true runner: sendAdditionalTelemetry: true ``` -------------------------------- ### Scale Down Deployment Example Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/actions/remediation.rst This example demonstrates scaling down a specific deployment to zero replicas when a pod enters a crash loop. The namespace is dynamically determined by the alert context. ```bash kubectl scale --replicas=0 deployment/payment-processing-worker -n production ``` -------------------------------- ### Install Robusta for Small Clusters with Prometheus Settings Source: https://github.com/robusta-dev/robusta/blob/master/docs/help.rst Install Robusta using Helm, specifying `isSmallCluster=true` and Prometheus-specific configurations for small clusters. ```bash helm install robusta robusta/robusta -f ./generated_values.yaml --set clusterName= --set isSmallCluster=true \ --set kube-prometheus-stack.prometheus.prometheusSpec.retentionSize=9GB \ --set kube-prometheus-stack.prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=10Gi \ --set kube-prometheus-stack.prometheus.prometheusSpec.resources.requests.memory=512Mi ``` -------------------------------- ### Example Response JSON Source: https://github.com/robusta-dev/robusta/blob/master/src/robusta/core/playbooks/internal/CRD_API_DOCUMENTATION.md An example of the JSON structure returned when fetching CR instances, including metadata and selected fields. ```json [ { "name": "prometheus-main", "namespace": "monitoring", "spec.version": "v2.41.0", "spec.replicas": 2, "status.phase": "Running" }, { "name": "prometheus-secondary", "namespace": "monitoring", "spec.version": "v2.40.0", "spec.replicas": 1, "status.phase": "Running" } ] ``` -------------------------------- ### Example pyproject.toml for Playbook Repository Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/actions/develop-actions/playbook-repositories.rst This example pyproject.toml defines the package name, version, and dependencies for a playbook repository. Ensure the package name matches the inner playbook directory. ```bash [tool.poetry] name = "my_playbook_repo" version = "0.0.1" description = "" authors = ["USER NAME "] [tool.poetry.dependencies] # if your playbook requires additional dependencies, add them here #some-dependency = "^1.2.3" [tool.poetry.dev-dependencies] robusta-api = "^0.14" [build-system] requires = ["poetry-core>=1.0.0"] build-backend = "poetry.core.masonry.api" ``` -------------------------------- ### Jira Sink Configuration Example Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/sinks/jira.rst Example configuration for the Jira sink, including basic settings and scope filtering for specific Prometheus alerts. ```yaml sinksConfig: - jira_sink: name: personal_jira_sink url: https://workspace.atlassian.net username: username api_key: api_key dedups: (OPTIONAL) - fingerprint assignee: user_id of the assignee(OPTIONAL) epic: epic_id(OPTIONAL) project_name: project_name priority_mapping: (OPTIONAL) HIGH: "High" LOW: "Low" INFO: "Lowest" scope: include: - identifier: [CPUThrottlingHigh, KubePodCrashLooping] ``` -------------------------------- ### Webex Sink Dynamic Room Routing Example Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/sinks/webex.rst Example demonstrating how to route alerts to different Webex rooms based on namespace labels, with a fallback to a default room. ```yaml sinksConfig: - webex_sink: name: webex_sink bot_access_token: room_id: ``` -------------------------------- ### FileBlock Example Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/actions/develop-actions/findings-api.rst Create a FileBlock to include file contents in the playbook output. Specify the filename and its content. ```python FileBlock("test.txt", "this is the file's contents") ``` -------------------------------- ### Install Robusta in a Custom Namespace Source: https://github.com/robusta-dev/robusta/blob/master/docs/setup-robusta/installation-faq.rst Use this command to install Robusta in a specific namespace, creating it if it doesn't exist. Ensure you replace `` with your actual cluster name. ```bash helm install robusta robusta/robusta -f ./generated_values.yaml -n robusta --create-namespace --set clusterName= ``` -------------------------------- ### Install Multiple Robusta Instances Source: https://github.com/robusta-dev/robusta/blob/master/docs/setup-robusta/installation-faq.rst Install two Robusta instances in different namespaces by using unique Helm release names and specifying distinct cluster names for each. Replace placeholders as needed. ```bash helm install robusta robusta/robusta -f ./generated_values.yaml -n namespace-1 --create-namespace --set clusterName= helm install robusta-2 robusta/robusta -f ./generated_values.yaml -n namespace-2 --create-namespace --set clusterName= ``` -------------------------------- ### Example cURL Request Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/exporting/send-events-api.rst This example demonstrates how to send a sample alert event to the Robusta Send Events API using cURL. Replace placeholders with your actual values. ```bash curl --location --request POST \ 'https://api.robusta.dev/webhooks?type=alert&origin=datadog&account_id=ACCOUNT_ID' \ --header 'Authorization: Bearer API_KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "title": "High error rate", "severity": "high" }' ``` -------------------------------- ### RBAC Configuration Request Body Example Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/exporting/rbac-api.rst An example of the JSON body for setting RBAC configurations, including scopes, groups, and role permission groups. ```json { "account_id": "YOUR_ACCOUNT_ID", "scopes": [ { "name": "production-scope", "type": "namespace", "clusters": { "production-cluster": ["default", "app-namespace"] } }, { "name": "staging-scope", "type": "cluster", "clusters": { "staging-cluster": ["*"] } } ], "groups": [ { "name": "developers", "provider_group_id": "dev-team-id", "type": "namespace", "scopes": ["production-scope"], "permissions": ["APP_VIEW", "POD_LOGS", "METRICS_VIEW"] }, { "name": "devops", "provider_group_id": "devops-team-id", "type": "cluster", "scopes": ["staging-scope"], "permissions": ["NODE_VIEW", "CLUSTER_VIEW", "KRR_SCAN"] } ], "role_permission_groups": [ { "name": "admin-group", "provider_group_id": "admin-team-id", "type": "ADMIN" } ] } ``` -------------------------------- ### Basic Playbook Action Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/actions/develop-actions/writing-playbooks.rst An example of a simple playbook action that retrieves pod information and sends it to Slack. ```python from robusta.api import * @action def my_action(event: PodEvent): # we have full access to the pod on which the alert fired pod = event.get_pod() pod_name = pod.metadata.name pod_logs = pod.get_logs() pod_processes = pod.exec("ps aux") # this is how you send data to slack or other destinations event.add_enrichment([ MarkdownBlock("*Oh no!* An alert occurred on " + pod_name), FileBlock("crashing-pod.log", pod_logs) ]) ``` -------------------------------- ### Copying and Installing Playbooks in a Custom Docker Image Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/defining-playbooks/external-playbook-repositories.rst This snippet demonstrates how to copy custom playbook files into a Docker image and install them into the Python site-packages directory. ```dockerfile COPY my_playbooks/ /etc/robusta/playbooks/my_playbooks RUN python3 -m pip install --no-cache-dir /etc/robusta/playbooks/my_playbooks ``` -------------------------------- ### KRR API Success Response Example Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/resource-recommender.rst Example JSON response for a successful KRR API request, detailing resource recommendations for workloads. ```json { "cluster_id": "my-cluster", "scan_id": "12345-67890-abcde", "scan_date": "2024-01-07T12:00:00Z", "scan_state": "success", "results": [ { "cluster_id": "my-cluster", "namespace": "default", "name": "nginx-deployment", "kind": "Deployment", "container": "nginx", "priority": "MEDIUM", "current_cpu_request": 100, "recommended_cpu_request": 50, "current_cpu_limit": 500, "recommended_cpu_limit": 200, "current_memory_request": 134217728, "recommended_memory_request": 67108864, "current_memory_limit": 536870912, "recommended_memory_limit": 268435456, "pods_count": 3 } ] } ``` -------------------------------- ### Configure Initial Slack Sink Source: https://github.com/robusta-dev/robusta/blob/master/docs/notification-routing/routing-by-type.rst This is an example of an existing Slack sink configuration. Ensure you have at least one sink configured before setting up routing. ```yaml sinksConfig: - slack_sink: name: sink1 slack_channel: general-alerts api_key: secret-key ``` -------------------------------- ### Example Pod YAML Configuration Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/actions/develop-actions/my-first-custom-action.rst This YAML defines a basic Nginx pod with a node selector that can cause scheduling issues if no nodes match the selector. ```yaml apiVersion: v1 kind: Pod metadata: name: nginx labels: env: test spec: containers: - name: nginx image: nginx imagePullPolicy: IfNotPresent nodeSelector: spoiler: alert ``` -------------------------------- ### Send Alerts API Example Request Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/exporting/send-alerts-api.rst This example demonstrates how to send a list of alerts to the Robusta API using curl. Ensure you replace placeholders like ACCOUNT_ID and API-KEY with your actual credentials. ```bash curl --location --request POST 'https://api.robusta.dev/api/alerts' \ --header 'Authorization: Bearer API-KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "account_id": "ACCOUNT_ID", "alerts": [ { "title": "Test Service Down", "description": "The Test Service is not responding.", "source": "monitoring-system", "priority": "high", "aggregation_key": "test-service-issues", "failure": true, "starts_at": "2024-10-07T10:00:00Z", "labels": { "environment": "production" }, "annotations": { "env1": "true" }, "cluster": "prod-cluster-1", "subject_namespace": "prod", "subject_node": "gke-prod-cluster-1-node-1" } ] }' ``` -------------------------------- ### Common Query: Resource Quotas Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/exporting/prometheus-query-api.rst Examples of PromQL queries for resource quotas in OpenShift/Kubernetes. ```json {"promql_query": "kube_resourcequota{type=\"hard\"}"} ``` ```json {"promql_query": "kube_resourcequota{type=\"used\"}"} ``` -------------------------------- ### Basic External Prometheus Setup Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/metric-providers-external.rst Configure Robusta to use a basic external Prometheus and Alertmanager instance. ```yaml globalConfig: prometheus_url: "https://prometheus.company.com:9090" alertmanager_url: "https://alertmanager.company.com:9093" ``` -------------------------------- ### Install Robusta on an Additional Cluster Source: https://github.com/robusta-dev/robusta/blob/master/docs/setup-robusta/multi-cluster.rst Use this command to install Robusta on subsequent clusters, reusing your existing `generated_values.yaml` and setting a unique cluster name. ```bash helm install robusta robusta/robusta -f ./generated_values.yaml --set clusterName= ``` -------------------------------- ### Example Account ID in generated_values.yaml Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/alertmanager-integration/pagerduty-alerting.rst Illustrates the format of the account_id found in the Robusta generated_values.yaml file. ```yaml account_id: caa68d87-XXXX-XXXX-XXXX-a6514ccb11eb ``` -------------------------------- ### Baking Actions into a Custom Image Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/defining-playbooks/external-playbook-repositories.rst Load custom actions by copying them into a custom image and installing them into site-packages. Use a file URL and set pip_install to False. ```yaml playbookRepos: my_extra_playbooks: url: "file:///etc/robusta/playbooks/my_playbooks" pip_install: False ``` -------------------------------- ### Common Query: CPU and Memory Usage Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/exporting/prometheus-query-api.rst Examples of PromQL queries for container CPU and memory usage. ```json {"promql_query": "sum(rate(container_cpu_usage_seconds_total[5m])) by (pod, namespace)"} ``` ```json {"promql_query": "container_memory_usage_bytes{namespace=\"default\"}"} ``` -------------------------------- ### Install Robusta with Existing SCC Source: https://github.com/robusta-dev/robusta/blob/master/docs/setup-robusta/openshift.rst Configure Robusta to use a pre-existing SCC created by an administrator, disabling automatic SCC creation. ```yaml openshift: enabled: true createScc: false sccName: my-admins-scc # reference the existing SCC your admin created ``` -------------------------------- ### Kubectl Describe Action Source: https://github.com/robusta-dev/robusta/blob/master/src/robusta/core/playbooks/internal/CRD_API_DOCUMENTATION.md Use this action to get a detailed description of any Kubernetes resource. It requires the resource kind, name, and optionally the namespace. ```yaml action: kubectl_describe params: kind: deployment name: nginx-deployment namespace: production ``` -------------------------------- ### Example Censored Pod Log (SAME_LENGTH_ASTERISKS) Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/builtin-alert-enrichment.rst Demonstrates how sensitive data in pod logs is replaced with asterisks of the same length after enabling log censoring with the SAME_LENGTH_ASTERISKS style. ```text # Using SAME_LENGTH_ASTERISKS style: 2022-07-28 08:24:45.283 INFO user's uuid: '************************************' 2022-07-28 08:35:00.762 INFO Customer email: **************** 2022-07-28 08:35:01.090 INFO Payment processed with card: ******************* ``` -------------------------------- ### Send Dummy Alert using Robusta CLI Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/alertmanager-integration/_testing_integration.rst Use this command if you have the Robusta CLI installed to send a test alert to Alertmanager. ```bash robusta demo-alert ``` -------------------------------- ### Basic Kafka Sink Configuration Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/sinks/kafka.rst Add this to your generated_values.yaml for basic Kafka sink setup. Ensure Kafka is accessible at the specified URL. ```yaml sinksConfig: - kafka_sink: name: kafka_sink kafka_url: "localhost:9092" topic: "robusta-playbooks" ``` -------------------------------- ### Enable Robusta on OpenShift Source: https://github.com/robusta-dev/robusta/blob/master/docs/setup-robusta/openshift.rst Basic configuration to enable Robusta on OpenShift and set Prometheus/Alertmanager URLs. Obtain host values using `oc get routes`. ```yaml openshift: enabled: true createScc: true globalConfig: prometheus_url: https:// # get the value for by running oc get routes -n openshift-monitoring prometheus-k8s -o jsonpath='{.spec.host}' alertmanager_url: https:// # get the value for by running oc get routes -n openshift-monitoring alertmanager-main -o jsonpath='{.spec.host}' ``` -------------------------------- ### Common Sink Options Example Source: https://github.com/robusta-dev/robusta/blob/master/docs/notification-routing/configuring-sinks.rst Demonstrates common configuration options for a sink, including name, scope, activity, stop, grouping, and default settings, along with sink-specific parameters for Slack. ```yaml sinksConfig: - slack_sink: name: my_sink_name scope: {} activity: {} stop: false grouping: {} default: true # sink-specific parameters - e.g. for Slack, some options are shown below # api_key: xoxb-112... # slack_channel: general-alerts ``` -------------------------------- ### Define a Single Microsoft Teams Sink Source: https://github.com/robusta-dev/robusta/blob/master/docs/notification-routing/configuring-sinks.rst Configure a single Microsoft Teams sink using its webhook URL. This is an example of a basic sink setup. ```yaml sinksConfig: - ms_teams_sink: name: my_teams_sink webhook_url: ``` -------------------------------- ### RBAC Configuration Response Example Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/exporting/rbac-api.rst A successful GET request to the RBAC API returns the current account ID, scopes, groups, and role permission groups. ```json { "account_id": "YOUR_ACCOUNT_ID", "scopes": [ { "name": "production-scope", "type": "namespace", "clusters": { "production-cluster": ["default", "app-namespace"] } } ], "groups": [ { "name": "developers", "provider_group_id": "dev-team-id", "type": "namespace", "scopes": ["production-scope"], "permissions": ["APP_VIEW", "POD_LOGS", "METRICS_VIEW"] } ], "role_permission_groups": [ { "name": "admin-group", "provider_group_id": "admin-team-id", "type": "ADMIN" } ] } ``` -------------------------------- ### Override Trigger in Autogenerated Examples Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/actions/develop-actions/documenting-actions.rst Explicitly set the trigger for autogenerated examples by providing a second parameter to the `robusta-action` directive. This allows you to specify which event should trigger the action in the example. ```rst .. robusta-action:: playbooks.robusta_playbooks.grafana_enrichment.add_deployment_lines_to_grafana on_deployment_update ``` -------------------------------- ### Run Multiple Playbooks Sequentially Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/index.rst Demonstrates how to define multiple playbooks that trigger on the same event (`on_deployment_create`). Actions from these playbooks will execute in the order they are defined. ```yaml # first playbook - triggers: - on_deployment_create: {} actions: - my_first_action: {} # second playbook - triggers: - on_deployment_create: {} actions: - my_second_action: {} ``` -------------------------------- ### Check Prometheus Operator Installation Source: https://github.com/robusta-dev/robusta/blob/master/docs/setup-robusta/alertsui.rst Verify that the Prometheus operator is installed in your cluster by checking for Custom Resource Definitions (CRDs). ```bash kubectl get crd | grep prometheus ``` -------------------------------- ### Get RBAC Configuration Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/exporting/rbac-api.rst Retrieve the current RBAC configuration for your account using a GET request to the RBAC API endpoint. ```bash curl -X GET 'https://api.robusta.dev/api/rbac?account_id=YOUR_ACCOUNT_ID' \ -H 'Authorization: Bearer YOUR_API_KEY' ``` -------------------------------- ### Enable Identical Playbooks with Different Parameters Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/index.rst This configuration shows how to deploy the same playbook logic multiple times, each with distinct parameters (e.g., `name_prefix` for deployments) to target different applications or configurations. ```yaml customPlaybooks: - triggers: - on_deployment_update: name_prefix: MyApp actions: - add_deployment_lines_to_grafana: grafana_api_key: grafana_key_goes_here grafana_dashboard_uid: id_for_dashboard1 grafana_url: http://grafana.namespace.svc - triggers: - on_deployment_update: name_prefix: OtherApp actions: - add_deployment_lines_to_grafana: grafana_api_key: grafana_key_goes_here grafana_dashboard_uid: id_for_dashboard2 grafana_url: http://grafana.namespace.svc ``` -------------------------------- ### Helm Install Fluent Bit Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/logs-triggers/index.rst Command to add the Fluent Bit Helm repository and install the DaemonSet using custom values. ```bash helm repo add fluent https://fluent.github.io/helm-charts && helm repo update helm install metrics-fluent-bit fluent/fluent-bit -f ./fluentbit-values.yaml ``` -------------------------------- ### Send Configuration Changes Request Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/exporting/configuration-changes-api.rst Use this example to send a list of configuration changes to the Robusta API. Ensure to replace placeholders with your actual account ID and API key. ```bash curl --location --request POST 'https://api.robusta.dev/api/config-changes' \ --header 'Authorization: Bearer API-KEY' \ --header 'Content-Type: application/json' \ --data-raw '{ "account_id": "ACCOUNT_ID", "config_changes": [ { "title": "Updated test-service deployment", "old_config": "apiVersion: apps/v1\nkind: Deployment\n....", "new_config": "apiVersion: apps/v1...", "resource_name": "test sercvice", "description": "Changed deployemnt", "source": "test-service", "cluster": "prod-cluster-1", "labels": { "environment": "production" }, "annotations": { "env1": "true" }, "subject_namespace": "prod", "subject_node": "gke-prod-cluster-1-node-1" } ] }' ``` -------------------------------- ### CRD Definition Example Source: https://github.com/robusta-dev/robusta/blob/master/src/robusta/core/playbooks/internal/CRD_API_DOCUMENTATION.md An example of a Custom Resource Definition (CRD) structure, showing API version, kind, scope, and additional printer columns. ```json [ { "apiVersion": "cert-manager.io/v1", "kind": "Certificate", "plural": "certificates", "scope": "Namespaced", "createdAt": "2024-03-15T10:23:45Z", "additionalPrinterColumns": [ { "name": "READY", "type": "string", "jsonPath": ".status.conditions[?(@.type=="Ready")].status" }, { "name": "SECRET", "type": "string", "jsonPath": ".spec.secretName" }, { "name": "ISSUER", "type": "string", "jsonPath": ".spec.issuerRef.name" } ] }, { "apiVersion": "monitoring.coreos.com/v1", "kind": "Prometheus", "plural": "prometheuses", "scope": "Namespaced", "createdAt": "2024-02-10T14:52:18Z", "additionalPrinterColumns": [ { "name": "VERSION", "type": "string", "jsonPath": ".spec.version" }, { "name": "REPLICAS", "type": "integer", "jsonPath": ".spec.replicas" } ] } ] ``` -------------------------------- ### Delete Specific Crashing Pod Example Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/actions/remediation.rst An example of deleting a crashing pod by its kind (Pod), name, and namespace. This is useful for cleaning up problematic resources. ```bash kubectl delete Pod api-worker-1 -n staging ``` -------------------------------- ### Example Request using Duration Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/exporting/prometheus-query-api.rst Example cURL request to query Prometheus data using a specified duration. Replace placeholders with actual values. ```bash curl -X POST "https://api.robusta.dev/api/accounts/ACCOUNT_ID/clusters/CLUSTER_NAME/prometheus/query" \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "promql_query": "up", "duration_minutes": 5, "step": "1m" }' ``` -------------------------------- ### Simulate MySQL Connection Error (10 log lines) Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/logs-triggers/index.rst Send a POST request to the demo pod's API to simulate ten MySQL connection error logs after a 60-second delay. ```bash kubectl run curl --rm -it --image=curlimages/curl --restart=Never -- \ curl -X POST http://postlog1.log-triggers.svc.cluster.local/api/log \ -H "Content-Type: application/json" \ -d '{"content": "mysql connection error", "count": 10}' ``` -------------------------------- ### Action with Parameters Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/actions/develop-actions/writing-playbooks.rst Defines and uses an action that accepts parameters, specifically a bash command to execute on a pod. ```python from robusta.api import * class BashParams(ActionParams): bash_command: str @action def pod_bash_enricher(event: PodEvent, params: BashParams): pod = event.get_pod() if not pod: logging.error(f"cannot run PodBashEnricher on event with no pod: {event}") return block_list: List[BaseBlock] = [] exec_result = pod.exec(params.bash_command) block_list.append(MarkdownBlock(f"Command results for *{params.bash_command}:*")) block_list.append(MarkdownBlock(exec_result)) event.add_enrichment(block_list) ``` -------------------------------- ### Example Holmes AI API Response Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/holmesgpt/holmes-chat-api.rst This is an example of the JSON response received from the Holmes AI API, containing the cluster ID and the analysis in markdown format. ```json { "cluster_id": "my-cluster", "analysis": "## Investigation Summary\n\n### Failing Pods\n\n| Pod | Namespace | Status | Reason |\n|-----|-----------|--------|--------|\n| `frontend-59fbcd7965-drx6w` | production | Pending | ContainerCreating |\n\n**Root Cause**: Missing Secret - The pod is stuck because the secret `frontend-api-keys` does not exist.\n\n### Fix\n\n```bash\nkubectl create secret generic frontend-api-keys --from-literal=API_KEY= -n production\n```" ``` -------------------------------- ### Example Request using Date Range Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/exporting/prometheus-query-api.rst Example cURL request to query Prometheus data within a specific date range. Replace placeholders with actual values. ```bash curl -X POST "https://api.robusta.dev/api/accounts/ACCOUNT_ID/clusters/CLUSTER_NAME/prometheus/query" \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "promql_query": "kube_resourcequota{type=\"hard\"}", "starts_at": "2024-01-27 10:00:00 UTC", "ends_at": "2024-01-27 11:00:00 UTC", "step": "1m" }' ``` -------------------------------- ### Simulate MySQL Connection Error (1 log line) Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/logs-triggers/index.rst Send a POST request to the demo pod's API to simulate a single MySQL connection error log. ```bash kubectl run curl --rm -it --image=curlimages/curl --restart=Never -- \ curl -X POST http://postlog1.log-triggers.svc.cluster.local/api/log \ -H "Content-Type: application/json" \ -d '{"content": "mysql connection error", "count": 1}' ``` -------------------------------- ### Example Alert Statistics Response Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/exporting/alert-statistics-api.rst This is an example of the JSON response format for aggregated alert data. It includes the alert type and its count within the queried time range. ```json [ {"aggregation_key": "KubeJobFailed", "alert_count": 17413}, {"aggregation_key": "KubePodNotReady", "alert_count": 11893}, {"aggregation_key": "KubeDeploymentReplicasMismatch", "alert_count": 2410}, {"aggregation_key": "KubeDeploymentRolloutStuck", "alert_count": 923}, {"aggregation_key": "KubePodCrashLooping", "alert_count": 921}, {"aggregation_key": "KubeContainerWaiting", "alert_count": 752}, {"aggregation_key": "PrometheusRuleFailures", "alert_count": 188}, {"aggregation_key": "KubeMemoryOvercommit", "alert_count": 187}, {"aggregation_key": "PrometheusOperatorRejectedResources", "alert_count": 102}, {"aggregation_key": "KubeletTooManyPods", "alert_count": 94}, {"aggregation_key": "NodeMemoryHighUtilization", "alert_count": 23}, {"aggregation_key": "TargetDown", "alert_count": 19}, {"aggregation_key": "test123", "alert_count": 7}, {"aggregation_key": "KubeAggregatedAPIDown", "alert_count": 4}, {"aggregation_key": "KubeAggregatedAPIErrors", "alert_count": 4}, {"aggregation_key": "KubeMemoryOvercommitTEST2", "alert_count": 1}, {"aggregation_key": "TestAlert", "alert_count": 1}, {"aggregation_key": "TestAlert2", "alert_count": 1}, {"aggregation_key": "dsafd", "alert_count": 1}, {"aggregation_key": "KubeMemoryOvercommitTEST", "alert_count": 1}, {"aggregation_key": "vfd", "alert_count": 1} ] ``` -------------------------------- ### Set Up Namespace-Level Permissions with RBAC API Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/exporting/rbac-api.rst Use this POST request to configure namespace-level permissions for a group. This example sets up permissions for developers on specific clusters and namespaces. ```bash curl -X POST 'https://api.robusta.dev/api/rbac?account_id=YOUR_ACCOUNT_ID' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{ "account_id": "YOUR_ACCOUNT_ID", "scopes": [ { "name": "dev-namespaces", "type": "namespace", "clusters": { "production": ["dev", "staging"], "development": ["*"] } } ], "groups": [ { "name": "developers", "provider_group_id": "github-dev-team", "type": "namespace", "scopes": ["dev-namespaces"], "permissions": ["APP_VIEW", "APP_RESTART", "POD_LOGS", "METRICS_VIEW"] } ] }' ``` -------------------------------- ### Alertmanager Configuration for Other Prometheus Installations Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/alertmanager-integration/_alertmanager-config.rst Add this configuration to your AlertManager configuration for Prometheus installations other than kube-prometheus-stack. Ensure Robusta's route is the first, and set send_resolved to true. ```yaml receivers: - name: 'robusta' webhook_configs: - url: 'http://-runner..svc.cluster.local/api/alerts' send_resolved: true # (3) route: # (1) routes: - receiver: 'robusta' group_by: [ '...' ] group_wait: 1s group_interval: 1s matchers: - severity =~ ".*" repeat_interval: 4h continue: true # (2) ``` -------------------------------- ### Create Flux HelmRepository for Robusta Source: https://github.com/robusta-dev/robusta/blob/master/docs/setup-robusta/gitops/flux.rst Add Robusta's Helm chart as a Flux HelmRepository. Ensure the specified namespace exists in your cluster. ```bash flux create source helm robusta \ --url https://robusta-charts.storage.googleapis.com \ --namespace robusta \ --export > robusta-helm.yaml ``` -------------------------------- ### Basic Robusta UI Sink Configuration Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/sinks/RobustaUI.rst Add the Robusta UI sink to your Helm values, providing the generated token. This is the minimal configuration required. ```yaml sinksConfig: - robusta_sink: name: robusta_ui_sink token: ttl_hours: 4380 ``` -------------------------------- ### Kubernetes Trigger Scope Filtering Example Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/triggers/kubernetes.rst This example demonstrates advanced scope filtering for Kubernetes triggers, including filtering by name, namespace, labels, attributes, and annotations. Use this to precisely control which resource changes trigger playbooks. ```yaml customPlaybooks: - name: "FilteredPodCreation" triggers: - on_pod_create: scope: include: - name: - my-pod.* - other namespace: ns1 labels: - "foo=bar, boo=xx.*" - "foo=xx, boo=xx.*" attributes: - "status.phase=Pending, status.qosClass=BestEffort, metadata.resourceVersion != 123" - "spec.restartPolicy=OnFailure" annotations: - "foo=bar, boo=xx.*" - "foo=xx, boo=xx.*" ``` -------------------------------- ### HTTP Requests Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/actions/networking.rst Perform HTTP GET, POST, and PUT requests. ```APIDOC ## HTTP GET ### Description Performs an HTTP GET request. ### Method GET ### Endpoint playbooks.robusta_playbooks.http_actions.http_get ## HTTP POST ### Description Performs an HTTP POST request. ### Method POST ### Endpoint playbooks.robusta_playbooks.http_actions.http_post ## HTTP PUT ### Description Performs an HTTP PUT request. ### Method PUT ### Endpoint playbooks.robusta_playbooks.http_actions.http_put ``` -------------------------------- ### Simulate Deployment Image Change Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/kubernetes-examples/kubernetes-change-notifications.rst Apply these YAML files to simulate a deployment image change and trigger a Robusta notification. ```bash kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/deployment_image_change/before_image_change.yaml kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/deployment_image_change/after_image_change.yaml ``` -------------------------------- ### Custom Python Action Example Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/actions/index.rst Example of a custom Python action that enriches alerts with pod information. This action runs on specified Prometheus alerts and can access pod details like name, logs, and processes. It demonstrates how to add enrichment data, such as markdown and file blocks, to the event. ```python # this runs on Prometheus alerts you specify in the YAML @action def my_enricher(event: PrometheusKubernetesAlert): # we have full access to the pod on which the alert fired pod = event.get_pod() pod_name = pod.metadata.name pod_logs = pod.get_logs() pod_processes = pod.exec("ps aux") # this is how you send data to slack or other destinations event.add_enrichment([ MarkdownBlock("*Oh no!* An alert occurred on " + pod_name), FileBlock("crashing-pod.log", pod_logs) ]) ``` -------------------------------- ### Create a Test Secret Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/kubernetes-examples/kubernetes-change-notifications.rst Use this bash command to create a sample Kubernetes Secret for testing purposes. ```bash kubectl create secret generic test-secret --from-literal=key1=value1 ``` -------------------------------- ### Create Namespace and Set Context Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/logs-triggers/index.rst This snippet creates a dedicated namespace for the log-triggering demo and sets the current Kubernetes context to use it. ```yaml kubectl create namespace log-triggers kubectl config set-context --current --namespace log-triggers ``` -------------------------------- ### Common Query: Pod Status Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/exporting/prometheus-query-api.rst Examples of PromQL queries for pod status and container restarts. ```json {"promql_query": "kube_pod_status_phase{phase=\"Running\"}"} ``` ```json {"promql_query": "kube_pod_container_status_restarts_total"} ``` -------------------------------- ### Generate Robusta UI Token Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/sinks/RobustaUI.rst Use the 'robusta' CLI to generate an authentication token for the Robusta UI sink. ```bash robusta integrations ui ``` -------------------------------- ### Simulate Alert Firing Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/prometheus-examples/link-alert-enrichment.rst Speed up testing by simulating the KubeContainerCPURequestAlert firing immediately using the Robusta CLI. ```bash robusta demo-alert --alert=KubeContainerCPURequestAlert --labels=label1=test,label2=alert ``` -------------------------------- ### fetch_resource_yaml Source: https://github.com/robusta-dev/robusta/blob/master/src/robusta/core/playbooks/internal/CRD_API_DOCUMENTATION.md Fetches the complete YAML representation of a resource using `kubectl get -o yaml`. ```APIDOC ## fetch_resource_yaml ### Description Fetches the complete YAML representation of a resource using `kubectl get -o yaml`. ### Method POST ### Endpoint /actions/fetch_resource_yaml ### Parameters #### Request Body - **kind** (string) - Required - The resource type - **name** (string) - Required - The resource name - **namespace** (string) - Optional - The namespace ### Request Example ```yaml action: fetch_resource_yaml params: kind: configmap name: app-config namespace: default ``` ### Response #### Success Response - Returns a **FileBlock** containing the complete YAML definition. ``` -------------------------------- ### GET /api/query/alerts Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/exporting/alert-export-api.rst Exports alert history data. Filters can be applied using query parameters. ```APIDOC ## GET /api/query/alerts ### Description Exports alert history data. You can filter the results based on specific criteria using query parameters such as ``alert_name``, ``account_id``, and time range. ### Method GET ### Endpoint https://api.robusta.dev/api/query/alerts ### Parameters #### Query Parameters - **account_id** (string) - Required - The unique account identifier (found in your ``generated_values.yaml`` file). - **start_ts** (string) - Required - Start timestamp for the alert history query (in ISO 8601 format, e.g., ``2024-09-02T04:02:05.032Z``). - **end_ts** (string) - Required - End timestamp for the alert history query (in ISO 8601 format, e.g., ``2024-09-17T05:02:05.032Z``). - **alert_name** (string) - Optional - The name of the alert to filter by (e.g., ``CrashLoopBackoff``). - **namespace** (string) - Optional - The namespace of the alert to filter by (e.g., ``monitoring``). ### Request Example ```bash curl --location 'https://api.robusta.dev/api/query/alerts?alert_name=CrashLoopBackoff&account_id=ACCOUNT_ID&start_ts=2024-09-02T04%3A02%3A05.032Z&end_ts=2024-09-17T05%3A02%3A05.032Z' \ --header 'Authorization: Bearer API-KEY' ``` ### Request Headers - **Authorization** (string) - Bearer token for authentication (e.g., ``Bearer TOKEN_HERE``). The token must have "Read Alerts" permission. ### Response #### Success Response (200) - **alert_name** (string) - Name of the alert (e.g., ``CrashLoopBackoff``). - **title** (string) - A brief description of the alert event. - **source** (string) - Source of the alert (e.g., ``kubernetes_api_server``). - **priority** (string) - Priority level of the alert (e.g., ``HIGH``). - **started_at** (string) - Timestamp when the alert was triggered, in ISO 8601 format. - **resolved_at** (string) - Timestamp when the alert was resolved, or ``null`` if still unresolved. - **cluster** (string) - The cluster where the alert originated. - **namespace** (string) - Namespace where the alert occurred. - **app** (string) - The application that triggered the alert. - **resource_name** (string) - Name of the resource that caused the alert. - **resource_node** (string) - Name of the node where the resource is running. #### Response Example ```json [ { "alert_name": "CrashLoopBackoff", "title": "Crashing pod api-gateway-123abc in namespace prod", "description": null, "source": "kubernetes_api_server", "priority": "HIGH", "started_at": "2024-09-03T04:09:31.342818+00:00", "resolved_at": null, "cluster": "prod-cluster-1", "namespace": "prod", "app": "api-gateway", "kind": null, "resource_name": "api-gateway-123abc", "resource_node": "gke-prod-cluster-1-node-1" } ] ``` ``` -------------------------------- ### Configure Pushover Sink in generated_values.yaml Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/sinks/pushover.rst Add this configuration to your `generated_values.yaml` file to set up the Pushover sink. It includes essential parameters like API token, user key, and optional settings for devices, file sending, HTML formatting, and custom URLs. ```yaml sinksConfig: - pushover_sink: name: pushover_sink token: user: device: # (Optional) If not set, will push to all Pushover enabled devices send_files: true # (Optional - Default is true) Whether to send images and log messages to Pushover send_as_html: true # (Optional - Default is true) Whether you want to send Robustas messages as parsable HTML to Pushover pushover_url: # (Optional) Defaults to https://api.pushover.net/1/messages.json ``` -------------------------------- ### Deploy Demo Alert Rule Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/alertmanager-integration/google-managed-alertmanager.rst Create this Kubernetes Rules resource to deploy a simple alert. This helps verify that alerts are being sent to Robusta correctly. ```yaml apiVersion: monitoring.googleapis.com/v1 kind: Rules metadata: name: test-alert namespace: gmp-public spec: groups: - name: test interval: 30s rules: - alert: TestAlertForGCPAlertmanager expr: vector(1) for: 0m labels: severity: warning annotations: summary: "Test alert to verify Alertmanager webhook" description: "This is a test alert. Delete the Rules resource to stop it." ``` -------------------------------- ### Set Up Cluster-Wide Admin Access with RBAC API Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/exporting/rbac-api.rst Use this POST request to assign predefined roles to SSO provider groups. This example grants 'ADMIN' role to a platform-admins group. ```bash curl -X POST 'https://api.robusta.dev/api/rbac?account_id=YOUR_ACCOUNT_ID' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{ "account_id": "YOUR_ACCOUNT_ID", "role_permission_groups": [ { "name": "platform-admins", "provider_group_id": "github-admin-team", "type": "ADMIN" } ] }' ``` -------------------------------- ### Severity-Based Silencing Example Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/defining-playbooks/silencer-playbooks.rst Silences all alerts with a severity level of LOW. This is useful for filtering out less critical alerts. ```yaml customPlaybooks: - triggers: - on_prometheus_alert: {} actions: - severity_silencer: severity: LOW # silence all LOW severity alerts ``` -------------------------------- ### Test Alerting with OOMKill Demo Source: https://github.com/robusta-dev/robusta/blob/master/docs/configuration/metric-providers-google.rst Apply this demo job to trigger an alert and verify that Robusta is receiving and processing metrics from Google Managed Prometheus. ```bash kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/oomkill/oomkill_job.yaml ``` -------------------------------- ### Update Kubernetes Resource Listing Functions Source: https://github.com/robusta-dev/robusta/blob/master/docs/playbook-reference/actions/develop-actions/migration-guides.rst Update calls to resource-listing functions. For example, 'Deployment.listNamespacedDeployment()' should be changed to 'DeploymentList.listNamespacedDeployment()'. ```python # Old method Deployment.listNamespacedDeployment() # New method DeploymentList.listNamespacedDeployment() ```