### Install Robusta for Small Clusters with Prometheus Source: https://docs.robusta.dev/master/help.html When installing Robusta on a small cluster like Kind or Colima, use the `isSmallCluster=true` property. This example also includes settings for Prometheus retention size, storage, and memory requests. ```bash helm install robusta robusta/robusta -f ./generated_values.yaml --set clusterName= --set isSmallCluster=true \ --set kube-prometheus-stack.prometheus.prometheusSpec.retentionSize=9GB \ --set kube-prometheus-stack.prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=10Gi \ --set kube-prometheus-stack.prometheus.prometheusSpec.resources.requests.memory=512Mi ``` -------------------------------- ### Kubectl Command Execution Example Source: https://docs.robusta.dev/master/playbook-reference/actions/remediation.html Example of a resolved kubectl command and a template for deleting resources. ```bash kubectl scale --replicas=0 deployment/payment-processing-worker -n production ``` ```bash kubectl delete $kind $name -n $namespace ``` ```bash kubectl delete Pod api-worker-1 -n staging ``` -------------------------------- ### Example generated_values.yaml Source: https://docs.robusta.dev/master/setup-robusta/gitops/argocd.html Configuration file required for Robusta installation. Ensure clusterName is set and adjust resource usage settings as needed. ```yaml clusterName: my_cluster_name # <- This is the line to be added isSmallCluster: false # <- Optional. Set this on test clusters to lower Robusta's resource usage. globalConfig: signing_key: xxxxxx account_id: xxxxxx sinksConfig: - slack_sink: name: main_slack_sink slack_channel: robusta-staging-alerts api_key: xxxxxx - robusta_sink: name: robusta_ui_sink token: xxxxxx enablePrometheusStack: true enablePlatformPlaybooks: true runner: sendAdditionalTelemetry: true ``` -------------------------------- ### External Prometheus Provider Examples Source: https://docs.robusta.dev/master/configuration/metric-providers-external.html Configuration examples for specific Prometheus-compatible providers. ```yaml globalConfig: prometheus_url: "https://prometheus.company.com:9090" alertmanager_url: "https://alertmanager.company.com:9093" ``` ```yaml globalConfig: prometheus_url: "https://thanos-query.monitoring.company.com:9090" alertmanager_url: "https://alertmanager.monitoring.company.com:9093" ``` ```yaml globalConfig: prometheus_url: "https://mimir.company.com/prometheus" alertmanager_url: "https://mimir.company.com/alertmanager" ``` -------------------------------- ### Complex RBAC Configuration Source: https://docs.robusta.dev/master/configuration/exporting/rbac-api.html This example demonstrates a complex RBAC setup with multiple scopes (namespace and cluster types) and permission groups. It includes definitions for 'prod-apps' and 'staging-full' scopes, and assigns specific permissions to 'prod-developers' and 'devops-team' groups, along with an 'sre-admins' role. ```bash curl -X POST 'https://api.robusta.dev/api/rbac?account_id=YOUR_ACCOUNT_ID' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{ "account_id": "YOUR_ACCOUNT_ID", "scopes": [ { "name": "prod-apps", "type": "namespace", "clusters": { "prod-cluster": ["app-1", "app-2", "app-3"] } }, { "name": "staging-full", "type": "cluster", "clusters": { "staging-cluster": ["*"] } } ], "groups": [ { "name": "prod-developers", "provider_group_id": "github-prod-dev", "type": "namespace", "scopes": ["prod-apps"], "permissions": [ "APP_VIEW", "APP_RESTART", "POD_LOGS", "METRICS_VIEW", "TIMELINE_VIEW" ] }, { "name": "devops-team", "provider_group_id": "github-devops", "type": "cluster", "scopes": ["staging-full"], "permissions": [ "NODE_VIEW", "NODE_DRAIN", "CLUSTER_VIEW", "KRR_SCAN", "ALERT_CONFIG_VIEW" ] } ], "role_permission_groups": [ { "name": "sre-admins", "provider_group_id": "github-sre", "type": "ADMIN" } ] }' ``` -------------------------------- ### Install Fluent Bit DaemonSet with Helm Source: https://docs.robusta.dev/master/playbook-reference/logs-triggers/index.html Installs the Fluent Bit DaemonSet using Helm, applying the custom values defined in `fluentbit-values.yaml`. ```bash helm repo add fluent https://fluent.github.io/helm-charts && helm repo update helm install metrics-fluent-bit fluent/fluent-bit -f ./fluentbit-values.yaml ``` -------------------------------- ### Install Pre-release Version Source: https://docs.robusta.dev/master/setup-robusta/upgrade.html Installs a beta or pre-release version of Robusta using Helm. This command enables the `--devel` flag for the upgrade process. ```bash helm upgrade --devel ``` -------------------------------- ### Install Robusta on Normal Clusters with Helm Source: https://docs.robusta.dev/master/setup-robusta/installation/standalone-installation.html Standard Helm installation command for Robusta. Replace with your cluster's name. ```bash helm repo add robusta https://robusta-charts.storage.googleapis.com && helm repo update helm install robusta robusta/robusta -f ./generated_values.yaml --set clusterName= ``` -------------------------------- ### Deployment Key Format Source: https://docs.robusta.dev/master/playbook-reference/defining-playbooks/external-playbook-repositories.html Example of the required format for the SSH private key used in the configuration. ```text -----BEGIN OPENSSH PRIVATE KEY----- ewfrcfsfvC1rZXktdjEAAAAABG5vb..... -----END OPENSSH PRIVATE KEY----- ``` -------------------------------- ### Install Robusta on Normal Clusters Source: https://docs.robusta.dev/master/setup-robusta/installation/all-in-one-installation.html Installs Robusta on a standard Kubernetes cluster using Helm. Replace `` with your cluster's name. ```bash helm install robusta robusta/robusta -f ./generated_values.yaml --set clusterName= ``` -------------------------------- ### Configure Kubectl Command Action Source: https://docs.robusta.dev/master/playbook-reference/actions/remediation.html Example configuration for scaling down a deployment upon a pod crash loop. ```yaml customPlaybooks: - name: CrashLoopScaleDown triggers: - on_pod_crash_loop: restart_count: 3 actions: - kubectl_command: description: "Scale Down Deployment" command: kubectl scale --replicas=0 deployment/payment-processing-worker -n $namespace ``` -------------------------------- ### Install Robusta on Local/Test Clusters with Helm Source: https://docs.robusta.dev/master/setup-robusta/installation/standalone-installation.html Helm installation command for Robusta on smaller clusters. Sets `isSmallCluster=true` and adjusts memory requests to conserve resources. ```bash helm repo add robusta https://robusta-charts.storage.googleapis.com && helm repo update helm install robusta robusta/robusta -f ./generated_values.yaml --set clusterName= --set isSmallCluster=true --set holmes.resources.requests.memory=512Mi ``` -------------------------------- ### Install Robusta in a Specific Namespace Source: https://docs.robusta.dev/master/setup-robusta/installation-faq.html Install Robusta in a custom namespace, for example, 'robusta'. This command also sets a cluster name. Ensure the namespace is created if it doesn't exist. ```bash helm install robusta robusta/robusta -f ./generated_values.yaml -n robusta --create-namespace --set clusterName= ``` -------------------------------- ### Example CRD Output from kubectl Source: https://docs.robusta.dev/master/setup-robusta/crds.html This is an example of the output you can expect when running the `kubectl get crd` command to find CRD names and API groups. ```text NAME API_GROUP alertmanagerconfigs monitoring.coreos.com alertmanagers monitoring.coreos.com imagejobs eraser.sh imagelists eraser.sh nodenetworkconfigs acn.azure.com overlayextensionconfigs acn.azure.com ... ``` -------------------------------- ### Configure Slack Sink - Manual Setup Source: https://docs.robusta.dev/master/configuration/sinks/slack.html Add this configuration to your `generated_values.yaml` for manual Slack integration. Ensure you replace placeholders with your actual Slack API key and channel. ```yaml sinksConfig: # slack integration params - slack_sink: name: main_slack_sink api_key: MY SLACK KEY # to avoid putting your key in Helm values, see below slack_channel: MY SLACK CHANNEL max_log_file_limit_kb: # (Default: 1000) The maximum allowed file size for "snippets" (in kilobytes) uploaded to the Slack channel. Larger files can be sent to Slack, but they may not be viewable directly within the Slack. channel_override: DYNAMIC SLACK CHANNEL OVERRIDE (Optional) investigate_link: true/false # optional, if false no investigate links/buttons will be included in Slack messages ``` -------------------------------- ### Develop CLI using pip Source: https://docs.robusta.dev/master/setup-robusta/installation/dev-setup.html Alternative workflow for installing the Robusta CLI into the global Python environment. ```bash pip3 install . ``` -------------------------------- ### Install Robusta on additional clusters Source: https://docs.robusta.dev/master/setup-robusta/multi-cluster.html Use the existing generated_values.yaml file to install Robusta on subsequent clusters, ensuring each has a unique cluster name. ```bash helm install robusta robusta/robusta -f ./generated_values.yaml --set clusterName= ``` -------------------------------- ### Configure Initial Slack Sink Source: https://docs.robusta.dev/master/notification-routing/routing-by-type.html Define a basic Slack sink configuration to serve as the starting point for routing. ```yaml sinksConfig: - slack_sink: name: sink1 slack_channel: general-alerts api_key: secret-key ``` -------------------------------- ### Install Robusta on Local/Test Cluster Source: https://docs.robusta.dev/master/setup-robusta/installation/all-in-one-installation.html Installs Robusta on a local or test cluster with reduced resource requests by setting `isSmallCluster=true`. This command also adjusts Prometheus retention and resource limits. Replace `` with your cluster's name. ```bash helm install robusta robusta/robusta -f ./generated_values.yaml --set clusterName= --set isSmallCluster=true \ --set kube-prometheus-stack.prometheus.prometheusSpec.retentionSize=9GB \ --set kube-prometheus-stack.prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=10Gi \ --set kube-prometheus-stack.prometheus.prometheusSpec.resources.requests.memory=512Mi --set holmes.resources.requests.memory=512Mi ``` -------------------------------- ### Get Kubernetes Warning Events Source: https://docs.robusta.dev/master/playbook-reference/triggers/kubernetes.html Command to retrieve all warning events across all namespaces. ```bash kubectl get events --all-namespaces --field-selector type=Warning ``` -------------------------------- ### Install Robusta on GKE Autopilot with Helm Source: https://docs.robusta.dev/master/setup-robusta/installation/standalone-installation.html Helm installation command for Robusta on GKE Autopilot. Specific settings are disabled due to Autopilot restrictions, but core functionality remains. ```bash helm repo add robusta https://robusta-charts.storage.googleapis.com && helm repo update helm install robusta robusta/robusta -f ./generated_values.yaml \ --set clusterName= \ --set kube-prometheus-stack.coreDns.enabled=false \ --set kube-prometheus-stack.kubeControllerManager.enabled=false \ --set kube-prometheus-stack.kubeDns.enabled=false \ --set kube-prometheus-stack.kubeEtcd.enabled=false \ --set kube-prometheus-stack.kubeProxy.enabled=false \ --set kube-prometheus-stack.kubeScheduler.enabled=false \ --set kube-prometheus-stack.nodeExporter.enabled=false \ --set kube-prometheus-stack.prometheusOperator.kubeletService.enabled=false ``` -------------------------------- ### Add Robusta Helm Repository Source: https://docs.robusta.dev/master/setup-robusta/installation-faq.html Add the Robusta Helm repository and update it to fetch the latest chart information. This is the first step to installing Robusta using Helm. ```bash helm repo add robusta https://robusta-charts.storage.googleapis.com && helm repo update helm show values robusta/robusta ``` -------------------------------- ### Implement Mattermost Sink Parameters Source: https://docs.robusta.dev/master/configuration/sinks/sinks-development.html Define sink parameters using Pydantic for validation. Ensure all required parameters are present for sink setup. ```python class MattermostSinkParams(SinkBaseParams): url: str token: str token_id: str channel: str @validator("url") def set_http_schema(cls, url): parsed_url = urlparse(url) # if netloc is empty string, the url was provided without schema if not parsed_url.netloc: raise AttributeError(f"{url} does not contain the schema!") return url ``` -------------------------------- ### Dynamically Route MS Teams Alerts Source: https://docs.robusta.dev/master/configuration/sinks/ms-teams.html Configuration examples for dynamically setting the webhook URL using annotations and replacement patterns. ```yaml sinksConfig: # MS Teams integration params - ms_teams_sink: name: main_ms_teams_sink webhook_url: teams-incoming-webhook # see instructions below webhook_override: "annotations.ms-team-alerts-sink" ``` ```yaml sinksConfig: - ms_teams_sink: name: main_ms_teams_sink webhook_url: teams-incoming-webhook # see instructions below webhook_override: ${annotations.kubernetes.io/service-name} ``` -------------------------------- ### Quick Start External Prometheus Configuration Source: https://docs.robusta.dev/master/configuration/metric-providers-external.html Basic configuration for connecting Robusta to an external Prometheus and Alertmanager instance. ```yaml globalConfig: prometheus_url: "https://prometheus.example.com:9090" alertmanager_url: "https://alertmanager.example.com:9093" ``` -------------------------------- ### Monitor Deployment Updates with Resource Operation Trigger Source: https://docs.robusta.dev/master/playbook-reference/triggers/kubernetes.html Example playbook configuration to trigger a finding when a deployment resource is updated. ```yaml customPlaybooks: - triggers: - on_kubernetes_resource_operation: resources: ["deployment"] operations: ["update"] actions: - create_finding: title: "Deployment $name on namespace $namespace updated" aggregation_key: "Deployment Update" ``` -------------------------------- ### Bake Custom Actions into a Docker Image Source: https://docs.robusta.dev/master/playbook-reference/defining-playbooks/external-playbook-repositories.html Copy local playbooks into the container image and install them as a Python package during the build process. ```dockerfile COPY my_playbooks/ /etc/robusta/playbooks/my_playbooks RUN python3 -m pip install --no-cache-dir /etc/robusta/playbooks/my_playbooks ``` -------------------------------- ### Configure Common Sink Options Source: https://docs.robusta.dev/master/notification-routing/configuring-sinks.html This example demonstrates common sink options including `scope`, `activity`, `stop`, `grouping`, and `default`. Sink-specific parameters like `api_key` for Slack are also shown. ```yaml sinksConfig: - slack_sink: name: my_sink_name scope: {} activity: {} stop: false grouping: {} default: true # sink-specific parameters - e.g. for Slack, some options are shown below # api_key: xoxb-112... # slack_channel: general-alerts ``` -------------------------------- ### Configure on_deployment_create trigger Source: https://docs.robusta.dev/master/playbook-reference/triggers/kubernetes.html Triggers when a Deployment is created. Use the filtered example to restrict execution by name, namespace, or labels. ```yaml customPlaybooks: - triggers: - on_deployment_create: {} actions: - create_finding: # title: "Deployment $name in namespace $namespace was created" aggregation_key: "Deployment_created" ``` ```yaml customPlaybooks: - triggers: - on_deployment_create: name_prefix: "my-resource" # namespace_prefix: "kube-system" # labels_selector: "label1=value1" # actions: - create_finding: # title: "Deployment $name in namespace $namespace was created" aggregation_key: "Deployment_created" ``` -------------------------------- ### RBAC Configuration Response Source: https://docs.robusta.dev/master/configuration/exporting/rbac-api.html Example JSON response structure for a successful GET request. ```json { "account_id": "YOUR_ACCOUNT_ID", "scopes": [ { "name": "production-scope", "type": "namespace", "clusters": { "production-cluster": ["default", "app-namespace"] } } ], "groups": [ { "name": "developers", "provider_group_id": "dev-team-id", "type": "namespace", "scopes": ["production-scope"], "permissions": ["APP_VIEW", "POD_LOGS", "METRICS_VIEW"] } ], "role_permission_groups": [ { "name": "admin-group", "provider_group_id": "admin-team-id", "type": "ADMIN" } ] } ``` -------------------------------- ### Verify Robusta Pods and Logs Source: https://docs.robusta.dev/master/setup-robusta/installation/all-in-one-installation.html Commands to check if Robusta pods are running correctly and to view their logs. This helps in verifying a successful installation. ```bash kubectl get pods -A | grep robusta ``` ```bash robusta logs ``` -------------------------------- ### Run Multiple Playbooks Sequentially Source: https://docs.robusta.dev/master/playbook-reference/defining-playbooks/index.html Demonstrates how to define multiple playbooks that trigger on the same event (`on_deployment_create`). The actions of these playbooks will execute in the order they are defined. ```yaml # first playbook - triggers: - on_deployment_create: {} actions: - my_first_action: {} # second playbook - triggers: - on_deployment_create: {} actions: - my_second_action: {} ``` -------------------------------- ### Add Helm Repository and Update Source: https://docs.robusta.dev/master/setup-robusta/installation/all-in-one-installation.html Adds the Robusta Helm repository and updates the local Helm chart repository. This is a prerequisite for installing Robusta. ```bash helm repo add robusta https://robusta-charts.storage.googleapis.com && helm repo update ``` -------------------------------- ### Ignore Sink Initialization Errors Source: https://docs.robusta.dev/master/notification-routing/configuring-sinks.html To prevent Robusta from failing to start when sink initialization errors occur, set `continue_on_sink_errors` to `True` within `globalConfig`. ```yaml globalConfig: continue_on_sink_errors: True ``` -------------------------------- ### Override Trigger in Autogenerated Examples Source: https://docs.robusta.dev/master/playbook-reference/actions/develop-actions/documenting-actions.html Explicitly override the default trigger used in autogenerated examples by providing a second parameter to the `robusta-action` directive. This allows specifying a different trigger for the example. ```rst .. robusta-action:: playbooks.robusta_playbooks.grafana_enrichment.add_deployment_lines_to_grafana on_deployment_update ``` -------------------------------- ### Example of Censored Pod Logs (SAME_LENGTH_ASTERISKS) Source: https://docs.robusta.dev/master/setup-robusta/privacy-and-security.html Demonstrates how sensitive data in pod logs is replaced with asterisks of the same length when using the SAME_LENGTH_ASTERISKS regex replacement style. ```text # Original pod log: 2022-07-28 08:24:45.283 INFO user's uuid: '193836d9-9cce-4df9-a454-c2edcf2e80e5' 2022-07-28 08:35:00.762 INFO Customer email: user@example.com 2022-07-28 08:35:01.090 INFO Payment processed with card: 4111-1111-1111-1111 # Using SAME_LENGTH_ASTERISKS style: 2022-07-28 08:24:45.283 INFO user's uuid: '************************************' 2022-07-28 08:35:00.762 INFO Customer email: **************** 2022-07-28 08:35:01.090 INFO Payment processed with card: ******************* ``` -------------------------------- ### Deploy CPU Throttling Demo Source: https://docs.robusta.dev/master/playbook-reference/prometheus-examples/bash-alert-enrichment.html Use this command to deploy a pod that triggers the CPUThrottlingHigh alert for testing purposes. ```bash kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/cpu_throttling/throttling.yaml ``` -------------------------------- ### on_persistentvolume_all_changes Trigger - Simple Example Source: https://docs.robusta.dev/master/playbook-reference/triggers/kubernetes.html Fires when a PersistentVolume is created, updated, or deleted. This is a simple example without filters. ```yaml customPlaybooks: - triggers: - on_persistentvolume_all_changes: {} actions: - create_finding: # title: "PersistentVolume $name in namespace $namespace was all_changes" aggregation_key: "PersistentVolume_all_changes" ``` -------------------------------- ### Enable Identical Playbooks with Different Parameters Source: https://docs.robusta.dev/master/playbook-reference/defining-playbooks/index.html Shows how to enable the same playbook logic multiple times with different parameters, such as `name_prefix` for deployment updates, allowing for distinct configurations for different applications. ```yaml customPlaybooks: - triggers: - on_deployment_update: name_prefix: MyApp actions: - add_deployment_lines_to_grafana: grafana_api_key: grafana_key_goes_here grafana_dashboard_uid: id_for_dashboard1 grafana_url: http://grafana.namespace.svc - triggers: - on_deployment_update: name_prefix: OtherApp actions: - add_deployment_lines_to_grafana: grafana_api_key: grafana_key_goes_here grafana_dashboard_uid: id_for_dashboard2 grafana_url: http://grafana.namespace.svc ``` -------------------------------- ### HTTP GET Action Source: https://docs.robusta.dev/master/playbook-reference/actions/networking.html Perform an HTTP GET request from within the cluster. The response can optionally be sent as a finding. ```APIDOC ## POST /api/actions/http_get ### Description Run an http GET against a url, from within the cluster. Optionally, send the response as a finding. ### Method POST ### Endpoint /api/actions/http_get ### Parameters #### Request Body - **url** (string) - Required - In cluster target url. - **get_response** (boolean) - Optional - (Default: False) Send results to sink. - **params** (dict) - Optional - Dictionary, list of tuples or bytes to send in the query string. ### Request Example ```json { "url": "http://example.com", "get_response": true, "params": { "key": "value" } } ``` ### Response #### Success Response (200) - **status** (string) - The status of the HTTP request. - **body** (string) - The response body of the HTTP request. #### Response Example ```json { "status": "200 OK", "body": "Success" } ``` ``` -------------------------------- ### Clone Robusta source code Source: https://docs.robusta.dev/master/setup-robusta/installation/dev-setup.html Initial step for setting up local development by cloning the repository. ```bash git clone https://github.com/robusta-dev/robusta.git && cd robusta ``` -------------------------------- ### Trigger HTTP GET via CLI Source: https://docs.robusta.dev/master/playbook-reference/actions/networking.html Execute an HTTP GET request manually using the Robusta CLI. ```bash robusta playbooks trigger http_get url=URL ``` -------------------------------- ### on_persistentvolume_delete Trigger - Simple Example Source: https://docs.robusta.dev/master/playbook-reference/triggers/kubernetes.html Fires when a PersistentVolume is deleted. This example shows a basic trigger without specific filters. ```yaml customPlaybooks: - triggers: - on_persistentvolume_delete: {} actions: - create_finding: # title: "PersistentVolume $name in namespace $namespace was deleted" aggregation_key: "PersistentVolume_deleted" ``` -------------------------------- ### on_persistentvolume_update Trigger - Simple Example Source: https://docs.robusta.dev/master/playbook-reference/triggers/kubernetes.html Fires when a PersistentVolume is updated, excluding creations and deletions. This is a basic example without filters. ```yaml customPlaybooks: - triggers: - on_persistentvolume_update: {} actions: - create_finding: # title: "PersistentVolume $name in namespace $namespace was updated" aggregation_key: "PersistentVolume_updated" ``` -------------------------------- ### Command Preparation and Execution Source: https://docs.robusta.dev/master/_modules/robusta/integrations/kubernetes/api_client_utils.html Utilities for normalizing command inputs and executing them via the Kubernetes stream API. ```python def prepare_pod_command(cmd) -> Optional[List[str]]: if type(cmd) == list: return cmd elif type(cmd) == str: return cmd.split(" ") # cmd need to be a list of strings elif cmd is None: return None else: logging.exception(f"cmd {cmd} has unknown type {type(cmd)}") return cmd ``` ```python def exec_commands(name, exec_command, namespace="default", container=None): logging.debug( f"Executing command name: {name} command: {exec_command} namespace: {namespace} container: {container}" ) # verify pod state before connecting pod_status = wait_for_pod_status(name, namespace, RUNNING_STATE, 90, 0.2) # TODO config if pod_status != RUNNING_STATE: msg = f"Not running exec commands. Pod {name} {namespace} is not in running state" logging.error(msg) return msg wsclient = None try: core_v1 = core_v1_api.CoreV1Api() wsclient = stream( core_v1.connect_get_namespaced_pod_exec, name, namespace, ``` -------------------------------- ### on_persistentvolume_create Trigger - Simple Example Source: https://docs.robusta.dev/master/playbook-reference/triggers/kubernetes.html Fires when a PersistentVolume is created. This example shows a basic trigger without specific filters. ```yaml customPlaybooks: - triggers: - on_persistentvolume_create: {} actions: - create_finding: # title: "PersistentVolume $name in namespace $namespace was created" aggregation_key: "PersistentVolume_created" ``` -------------------------------- ### Configure HTTP GET Playbook Source: https://docs.robusta.dev/master/playbook-reference/actions/networking.html Add this configuration to your Helm values.yaml to trigger an HTTP GET request on pod creation. ```yaml customPlaybooks: - actions: - http_get: url: string triggers: - on_pod_create: {} ``` -------------------------------- ### Verify Robusta Action Loading Source: https://docs.robusta.dev/master/playbook-reference/actions/develop-actions/my-first-custom-action.html These commands help verify that your custom action has been loaded correctly by Robusta. `$ robusta logs` shows logs and import status, while `$ robusta playbooks list-dirs` confirms your custom playbook directory is recognized. ```bash $ robusta logs # get robusta logs, see errors, see our playbook loaded ... 2022-08-03 10:53:14.116 INFO importing actions from my_playbook_repo.report_scheduling_failure ... $ robusta playbooks list-dirs # get see if you custom action package was loaded ====================================================================== Listing playbooks directories ====================================================================== ====================================================================== Stored playbooks directories: robusta-pending-pod-playbook ====================================================================== ``` -------------------------------- ### Test Deployment Image Change Source: https://docs.robusta.dev/master/playbook-reference/kubernetes-examples/kubernetes-change-notifications.html Run these commands to simulate a deployment image change and verify the notification. ```bash kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/deployment_image_change/before_image_change.yaml kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/deployment_image_change/after_image_change.yaml ``` -------------------------------- ### Example Robusta metrics output Source: https://docs.robusta.dev/master/setup-robusta/robusta-runner-metrics.html This is an example of the metrics data exposed by the robusta-runner, including counts and timings for discovery, playbooks, and queue events. ```text # HELP discovery_errors_total Number of discovery process failures. # TYPE discovery_errors_total counter discovery_errors_total 0.0 # HELP discovery_errors_created Number of discovery process failures. # TYPE discovery_errors_created gauge discovery_errors_created 1.6925408357619166e+09 # HELP discovery_process_time Total discovery process time (seconds) # TYPE discovery_process_time summary discovery_process_time_count 258.0 discovery_process_time_sum 87.89608378204866 # HELP discovery_process_time_created Total discovery process time (seconds) # TYPE discovery_process_time_created gauge discovery_process_time_created 1.6925408357619708e+09 # HELP playbooks_errors_total Number of playbooks failures. # TYPE playbooks_errors_total counter # HELP playbooks_process_time Total playbooks process time (seconds) # TYPE playbooks_process_time summary playbooks_process_time_count{source=""} 2218.0 playbooks_process_time_sum{source=""} 215.01851963996887 playbooks_process_time_count{source="manual_action"} 22.0 playbooks_process_time_sum{source="manual_action"} 2.7707679271698 # HELP playbooks_process_time_created Total playbooks process time (seconds) # TYPE playbooks_process_time_created gauge playbooks_process_time_created{source=""} 1.692540846287054e+09 playbooks_process_time_created{source="manual_action"} 1.6926052244569693e+09 # HELP queue_event_total Number of queue events status # TYPE queue_event_total counter queue_event_total{queue_name="api_server_queue",status="queued"} 1554.0 queue_event_total{queue_name="api_server_queue",status="processed"} 1554.0 queue_event_total{queue_name="alerts_queue",status="queued"} 169.0 queue_event_total{queue_name="alerts_queue",status="processed"} 169.0 # HELP queue_event_created Number of queue events status # TYPE queue_event_created gauge queue_event_created{queue_name="api_server_queue",status="queued"} 1.6925408454740999e+09 queue_event_created{queue_name="api_server_queue",status="processed"} 1.6925408467937994e+09 queue_event_created{queue_name="alerts_queue",status="queued"} 1.6925409958650925e+09 queue_event_created{queue_name="alerts_queue",status="processed"} 1.6925409967573514e+09 # HELP queue_process_time queue process time (seconds) # TYPE queue_process_time summary queue_process_time_count{queue_name="api_server_queue"} 1554.0 queue_process_time_sum{queue_name="api_server_queue"} 628.65576171875 queue_process_time_count{queue_name="alerts_queue"} 169.0 queue_process_time_sum{queue_name="alerts_queue"} 10279.701471090317 # HELP queue_size Current size of the queue # TYPE queue_size gauge queue_size{queue_name="api_server_queue"} 0.0 queue_size{queue_name="alerts_queue"} 0.0 ... ``` -------------------------------- ### Deploy the postlog demo pod and service Source: https://docs.robusta.dev/master/playbook-reference/logs-triggers/index.html Use this manifest to create a deployment and service for the postlog application in the log-triggers namespace. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: postlog1 namespace: log-triggers spec: replicas: 1 selector: matchLabels: app: postlog1 template: metadata: labels: app: postlog1 spec: containers: - name: postlog1 image: me-west1-docker.pkg.dev/robusta-development/development/postlog:2.0 ports: - containerPort: 8000 resources: requests: memory: "128Mi" cpu: "50m" limits: memory: "256Mi" --- apiVersion: v1 kind: Service metadata: name: postlog1 namespace: log-triggers spec: selector: app: postlog1 ports: - port: 80 targetPort: 8000 type: ClusterIP ``` -------------------------------- ### Apply the manifest to the cluster Source: https://docs.robusta.dev/master/playbook-reference/logs-triggers/index.html Execute this command to apply the postlog configuration to your Kubernetes cluster. ```bash kubectl apply -f postlog.yaml ``` -------------------------------- ### Example Alert Export Response Source: https://docs.robusta.dev/master/configuration/exporting/alert-export-api.html This is an example of the JSON response format returned by the Alert Export API, containing a list of alert objects with detailed information. ```json [ { "alert_name": "CrashLoopBackoff", "title": "Crashing pod api-gateway-123abc in namespace prod", "description": null, "source": "kubernetes_api_server", "priority": "HIGH", "started_at": "2024-09-03T04:09:31.342818+00:00", "resolved_at": null, "cluster": "prod-cluster-1", "namespace": "prod", "app": "api-gateway", "kind": null, "resource_name": "api-gateway-123abc", "resource_node": "gke-prod-cluster-1-node-1" }, { "alert_name": "CrashLoopBackoff", "title": "Crashing pod billing-service-xyz789 in namespace billing", "description": null, "source": "kubernetes_api_server", "priority": "HIGH", "started_at": "2024-09-03T04:09:31.496713+00:00", "resolved_at": null, "cluster": "prod-cluster-2", "namespace": "billing", "app": "billing-service", "kind": null, "resource_name": "billing-service-xyz789", "resource_node": "gke-prod-cluster-2-node-3" } ] ``` -------------------------------- ### Configure Robusta to Load Baked Actions Source: https://docs.robusta.dev/master/playbook-reference/defining-playbooks/external-playbook-repositories.html Point the playbook repository to the local file path where baked actions were installed, disabling automatic pip installation. ```yaml playbookRepos: my_extra_playbooks: url: "file:///etc/robusta/playbooks/my_playbooks" pip_install: False ``` -------------------------------- ### Customize Trigger Parameters in Autogenerated Examples Source: https://docs.robusta.dev/master/playbook-reference/actions/develop-actions/documenting-actions.html Further customize autogenerated trigger examples by adding parameters to the directive using the `:trigger-params:` option. This allows specifying custom parameters for the trigger. ```rst .. robusta-action:: playbooks.robusta_playbooks.bash_enrichments.pod_bash_enricher :trigger-params: {"alert_name": "ExampleLowDiskAlert"} ``` -------------------------------- ### Example pyproject.toml for Playbook Repository Source: https://docs.robusta.dev/master/playbook-reference/actions/develop-actions/playbook-repositories.html This `pyproject.toml` file defines your playbook repository's metadata, dependencies, and build system. Ensure the package name matches your inner playbooks directory name. ```toml [tool.poetry] name = "my_playbook_repo" version = "0.0.1" description = "" authors = ["USER NAME "] [tool.poetry.dependencies] # if your playbook requires additional dependencies, add them here #some-dependency = "^1.2.3" [tool.poetry.dev-dependencies] robusta-api = "^0.14" [build-system] requires = ["poetry-core>=1.0.0"] build-backend = "poetry.core.masonry.api" ``` -------------------------------- ### Configure Webex Sink Source: https://docs.robusta.dev/master/configuration/sinks/webex.html Use this configuration in your `generated-values.yaml` file to enable the Webex sink. Replace placeholders with your actual bot access token and room ID. ```yaml sinksConfig: - webex_sink: name: personal_webex_sink bot_access_token: room_id: ``` -------------------------------- ### Create FileBlock instance Source: https://docs.robusta.dev/master/playbook-reference/actions/develop-actions/findings-api.html Use FileBlock to represent files such as images, logs, or binary data. ```python FileBlock("test.txt", "this is the file's contents") ``` -------------------------------- ### Install Robusta on GKE Autopilot Source: https://docs.robusta.dev/master/setup-robusta/installation/all-in-one-installation.html Installs Robusta on a GKE Autopilot cluster. Some components are disabled due to Autopilot restrictions, but core functionality remains available. Replace `` with your cluster's name. ```bash helm install robusta robusta/robusta -f ./generated_values.yaml \ --set clusterName= \ --set kube-prometheus-stack.coreDns.enabled=false \ --set kube-prometheus-stack.kubeControllerManager.enabled=false \ --set kube-prometheus-stack.kubeDns.enabled=false \ --set kube-prometheus-stack.kubeEtcd.enabled=false \ --set kube-prometheus-stack.kubeProxy.enabled=false \ --set kube-prometheus-stack.kubeScheduler.enabled=false \ --set kube-prometheus-stack.nodeExporter.enabled=false \ --set kube-prometheus-stack.prometheusOperator.kubeletService.enabled=false ``` -------------------------------- ### Create Flux HelmRepository Source: https://docs.robusta.dev/master/setup-robusta/gitops/flux.html Command to register the Robusta Helm chart repository with Flux. ```bash flux create source helm robusta \ --url https://robusta-charts.storage.googleapis.com \ --namespace robusta \ --export > robusta-helm.yaml ``` -------------------------------- ### Install Python Certificates on Mac OS Source: https://docs.robusta.dev/master/help.html Run this command on Mac OS to fix SSL certificate errors by installing necessary certificates for Python packages. Adjust the Python version in the path if needed. ```bash /Applications/Python 3.9/Install Certificates.command ``` -------------------------------- ### Push Local Playbooks Directory Source: https://docs.robusta.dev/master/playbook-reference/actions/develop-actions/loading-custom-actions.html Use the Robusta CLI to load your local playbook repository into the persistent volume. This command copies your repository to `/etc/robusta/playbooks/storage`. ```bash robusta playbooks push ./my-playbooks-project-root ``` -------------------------------- ### Test with Demo Alert Source: https://docs.robusta.dev/master/configuration/metric-providers-google.html Deploy a demo OOMKill job to trigger an alert and verify metrics collection. ```bash kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/oomkill/oomkill_job.yaml ``` -------------------------------- ### Install Robusta with Custom SCC on OpenShift Source: https://docs.robusta.dev/master/setup-robusta/openshift.html Configure Helm values to use an existing SCC created by an administrator when Robusta is installed without full cluster admin permissions. Ensure `createScc` is set to `false` and `sccName` references the custom SCC. ```yaml openshift: enabled: true createScc: false sccName: my-admins-scc # reference the existing SCC your admin created ``` -------------------------------- ### Upgrade Robusta with Helm Source: https://docs.robusta.dev/master/configuration/sinks/DataDog.html Apply the configuration changes by running the helm upgrade command. ```bash helm upgrade robusta robusta/robusta --values=generated_values.yaml ``` -------------------------------- ### Get Investigation URI Source: https://docs.robusta.dev/master/_modules/robusta/core/reporting/base.html Generates a URL for investigating a finding in the Robusta UI. ```python def get_investigate_uri(self, account_id: str, cluster_name: Optional[str] = None): uri_path = self._map_service_to_uri() kind = self.service.resource_type if self.service else None if kind and kind.lower() == "job": params = { "account": account_id, "cluster": f'"{cluster_name}"' if cluster_name else None, "namespace": f'"{self.subject.namespace}"' if self.subject.namespace else None, "name": f'"{self.service.name}"' if self.service else None, } else: params = { "account": account_id, "clusters": f'["{cluster_name}"]' if cluster_name else None, "namespaces": f'["{self.subject.namespace}"]' if self.subject.namespace else None, "kind": kind, "name": self.service.name if self.service else None, "names": f'["{self.aggregation_key}"]' if self.aggregation_key else None, "finding": self.id, } params = {k: v for k, v in params.items() if v is not None} uri_path = f"{uri_path}?{urlencode(params)}" return f"{ROBUSTA_UI_DOMAIN}/{uri_path}" ``` -------------------------------- ### Load Actions from Private Git Repository Source: https://docs.robusta.dev/master/playbook-reference/defining-playbooks/external-playbook-repositories.html Configure a private git repository using SSH and a base64 encoded deployment key. ```yaml playbookRepos: # add the same playbooks as above, this time over ssh my_extra_playbooks: url: "git@github.com:robusta-dev/robusta-chaos.git" key: LS0tLS1CRUdJTiBPUEVOU1NIIFBSSVZBVEUgS0VZLS0tL # branch: some_branch # optional: which git branch to install from # pip_install: True # optional: load this playbook's dependencies (default True) ``` -------------------------------- ### on_replicaset_delete Trigger Example Source: https://docs.robusta.dev/master/playbook-reference/triggers/kubernetes.html Fires when a ReplicaSet is deleted. Use this trigger to be notified when ReplicaSets are removed. ```yaml customPlaybooks: - triggers: - on_replicaset_delete: {} actions: - create_finding: # title: "ReplicaSet $name in namespace $namespace was deleted" aggregation_key: "ReplicaSet_deleted" ``` ```yaml customPlaybooks: - triggers: - on_replicaset_delete: name_prefix: "my-resource" # namespace_prefix: "kube-system" # labels_selector: "label1=value1" # actions: - create_finding: # title: "ReplicaSet $name in namespace $namespace was deleted" aggregation_key: "ReplicaSet_deleted" ``` -------------------------------- ### Develop CLI using Poetry Source: https://docs.robusta.dev/master/setup-robusta/installation/dev-setup.html Standard workflow for developing the Robusta CLI using Poetry dependency management. ```bash poetry install poetry run robusta ``` -------------------------------- ### RBAC Configuration Request Body Source: https://docs.robusta.dev/master/configuration/exporting/rbac-api.html Example JSON payload for setting or updating RBAC configurations. ```json { "account_id": "YOUR_ACCOUNT_ID", "scopes": [ { "name": "production-scope", "type": "namespace", "clusters": { "production-cluster": ["default", "app-namespace"] } }, { "name": "staging-scope", "type": "cluster", "clusters": { "staging-cluster": ["*"] } } ], "groups": [ { "name": "developers", "provider_group_id": "dev-team-id", "type": "namespace", "scopes": ["production-scope"], "permissions": ["APP_VIEW", "POD_LOGS", "METRICS_VIEW"] }, { "name": "devops", "provider_group_id": "devops-team-id", "type": "cluster", "scopes": ["staging-scope"], "permissions": ["NODE_VIEW", "CLUSTER_VIEW", "KRR_SCAN"] } ], "role_permission_groups": [ { "name": "admin-group", "provider_group_id": "admin-team-id", "type": "ADMIN" } ] } ``` -------------------------------- ### Configure Uncordon Action Source: https://docs.robusta.dev/master/playbook-reference/actions/remediation.html Add this configuration to your Helm values.yaml to enable the uncordon action. ```yaml customPlaybooks: - actions: - uncordon: {} triggers: - on_node_create: {} ``` -------------------------------- ### GET /api/rbac Source: https://docs.robusta.dev/master/configuration/exporting/rbac-api.html Retrieve the current RBAC configuration for your account. Requires RBAC: READ permission. ```APIDOC ## GET /api/rbac ### Description Retrieve the current RBAC configuration for your account. ### Method GET ### Endpoint `/api/rbac?account_id=YOUR_ACCOUNT_ID` ### Query Parameters - **account_id** (string) - Required - Your Robusta Account ID. ### Authentication Requires API key authentication. Include your API key in the request headers: `Authorization: Bearer YOUR_API_KEY` ### Permissions Requires **RBAC: READ** permission. ### Response #### Success Response (200 OK) - **account_id** (string) - The account ID. - **scopes** (array) - A list of defined scopes. - **name** (string) - The name of the scope. - **type** (string) - The type of the scope (e.g., "namespace", "cluster"). - **clusters** (object) - A mapping of cluster names to a list of namespaces within that cluster. - **groups** (array) - A list of defined permission groups. - **name** (string) - The name of the group. - **provider_group_id** (string) - The ID of the group from the identity provider. - **type** (string) - The type of the group (e.g., "namespace", "cluster"). - **scopes** (array) - A list of scope names associated with the group. - **permissions** (array) - A list of permissions granted to the group. - **role_permission_groups** (array) - A list of role-based permission groups. - **name** (string) - The name of the role permission group. - **provider_group_id** (string) - The ID of the group from the identity provider. - **type** (string) - The type of the role (e.g., "ADMIN"). ### Response Example (200 OK) ```json { "account_id": "YOUR_ACCOUNT_ID", "scopes": [ { "name": "production-scope", "type": "namespace", "clusters": { "production-cluster": ["default", "app-namespace"] } } ], "groups": [ { "name": "developers", "provider_group_id": "dev-team-id", "type": "namespace", "scopes": ["production-scope"], "permissions": ["APP_VIEW", "POD_LOGS", "METRICS_VIEW"] } ], "role_permission_groups": [ { "name": "admin-group", "provider_group_id": "admin-team-id", "type": "ADMIN" } ] } ``` ``` -------------------------------- ### Configure Disk Benchmark Action Source: https://docs.robusta.dev/master/playbook-reference/actions/miscellaneous.html Add this to your Helm values.yaml to configure the disk_benchmark action. It requires a storage class name and can be triggered on pod creation. ```yaml customPlaybooks: - actions: - disk_benchmark: storage_class_name: string triggers: - on_pod_create: {} ``` -------------------------------- ### Pydantic Model Initialization in Robusta Source: https://docs.robusta.dev/master/_modules/robusta/core/reporting/blocks.html Demonstrates the use of Pydantic models with custom __init__ methods for enhanced convenience and validation in Robusta's reporting blocks. This approach is chosen over dataclasses for stricter type validation. ```python # TODO: add a KubernetesBlock for rendering Kubernetes object in a standard way # Notes on how we define all the classes below: # 1. We use pydantic and not dataclasses so that field types are validated # 2. We add __init__ methods ourselves for convenience. Without our own __init__ method, something like ``` -------------------------------- ### on_replicaset_create Trigger Example Source: https://docs.robusta.dev/master/playbook-reference/triggers/kubernetes.html Fires when a ReplicaSet is created. Use this trigger to react to new ReplicaSet deployments. ```yaml customPlaybooks: - triggers: - on_replicaset_create: {} actions: - create_finding: # title: "ReplicaSet $name in namespace $namespace was created" aggregation_key: "ReplicaSet_created" ``` ```yaml customPlaybooks: - triggers: - on_replicaset_create: name_prefix: "my-resource" # namespace_prefix: "kube-system" # labels_selector: "label1=value1" # actions: - create_finding: # title: "ReplicaSet $name in namespace $namespace was created" aggregation_key: "ReplicaSet_created" ``` -------------------------------- ### Configure Basic Kafka Sink Source: https://docs.robusta.dev/master/configuration/sinks/kafka.html Add this configuration to your `generated_values.yaml` to set up a basic Kafka sink. Ensure your Kafka URL and topic name are correctly specified. ```yaml sinksConfig: - kafka_sink: name: kafka_sink kafka_url: "localhost:9092" topic: "robusta-playbooks" ``` -------------------------------- ### Update Kubernetes Listing Functions from Deployment to DeploymentList Source: https://docs.robusta.dev/master/playbook-reference/actions/develop-actions/migration-guides.html Resource-listing functions have moved. For example, 'Deployment.listNamespacedDeployment' is now 'DeploymentList.listNamespacedDeployment'. ```python # Old method Deployment.listNamespacedDeployment() # New method DeploymentList.listNamespacedDeployment() ``` -------------------------------- ### Load Actions from Public Git Repository Source: https://docs.robusta.dev/master/playbook-reference/defining-playbooks/external-playbook-repositories.html Configure a public git repository using HTTPS. ```yaml playbookRepos: # add playbooks from https://github.com/robusta-dev/robusta-chaos my_extra_playbooks: url: "https://github.com/robusta-dev/robusta-chaos.git" # branch: some_branch # optional: which git branch to install from # pip_install: True # optional: load this playbook's dependencies (default True) ```