### Run Sloth CLI Examples Source: https://github.com/slok/sloth/blob/main/CONTRIBUTING.md Execute Sloth's command-line interface with example specifications for generation and validation. ```bash go run ./cmd/sloth generate -i ./examples/getting-started.yml ``` ```bash go run ./cmd/sloth/ validate -i ./examples/ -p ./examples/plugins/ -e _gen ``` -------------------------------- ### Run Sloth Controller in Fake Mode Source: https://github.com/slok/sloth/blob/main/CONTRIBUTING.md Start the Sloth controller in a simulated environment without a Kubernetes cluster connection using fake memory-based clients. ```bash go run ./cmd/sloth/ controller --mode=fake --debug ``` -------------------------------- ### Install Sloth with Helm Source: https://context7.com/slok/sloth/llms.txt Deploy Sloth to your Kubernetes cluster using Helm. You can install it to watch all namespaces or customize the deployment with a values file. ```bash # Install with Helm (watches all namespaces) helm install sloth ./deploy/kubernetes/helm/sloth \ --namespace monitoring \ --create-namespace # Customize deployment helm install sloth ./deploy/kubernetes/helm/sloth \ --namespace monitoring \ -f custom-values.yaml ``` -------------------------------- ### Run Sloth Kubernetes Controller In-Cluster Source: https://context7.com/slok/sloth/llms.txt Start the Sloth Kubernetes controller to watch `PrometheusServiceLevel` CRs and reconcile them into `PrometheusRule` CRDs. This command assumes in-cluster configuration. ```bash sloth kubernetes-controller ``` -------------------------------- ### Add Plugin via Command Line Argument Source: https://github.com/slok/sloth/blob/main/internal/plugin/slo/contrib/remove_labels_v1/README.md Example of how to add the remove_labels_v1 plugin as a command-line argument to the 'sloth generate' command. This configuration includes preserving the 'namespace' label. ```shell sloth generate --plugins-path=/plugins --slo-plugins='{"id": "sloth.dev/contrib/remove_labels/v1", "priority": 100, "config": {"preserveLabels": ["namespace"]}}' ... ``` -------------------------------- ### Example PrometheusServiceLevel CRD Source: https://context7.com/slok/sloth/llms.txt This is an example of a PrometheusServiceLevel Custom Resource Definition (CRD) that you can apply to your Kubernetes cluster. The Sloth controller will reconcile this CR and create a PrometheusRule. ```yaml apiVersion: sloth.slok.dev/v1 kind: PrometheusServiceLevel metadata: name: sloth-slo-my-api namespace: monitoring spec: service: "my-api" labels: owner: "platform-team" slos: - name: "requests-availability" objective: 99.9 sli: events: errorQuery: sum(rate(http_requests_total{job="my-api",status=~"5.."}[{{.window}}])) totalQuery: sum(rate(http_requests_total{job="my-api"}[{{.window}}])) alerting: name: MyAPIHighErrorRate pageAlert: labels: severity: critical ticketAlert: labels: severity: warning ``` -------------------------------- ### Get GroupResource from Resource Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Converts an unqualified resource string into a qualified GroupResource. ```go func Resource(resource string) schema.GroupResource ``` -------------------------------- ### Example SLO Specification Source: https://github.com/slok/sloth/blob/main/README.md This YAML defines an SLO for request availability, specifying an objective of 99.9%. It includes Prometheus queries for errors and total requests, and custom alert configurations for page and ticket alerts. ```yaml version: "prometheus/v1" service: "myservice" labels: owner: "myteam" repo: "myorg/myservice" tier: "2" slos: # We allow failing (5xx and 429) 1 request every 1000 requests (99.9%). - name: "requests-availability" objective: 99.9 description: "Common SLO based on availability for HTTP request responses." labels: category: availability sli: events: error_query: sum(rate(http_request_duration_seconds_count{job="myservice",code=~"(5..|429)"}[{{.window}}])) total_query: sum(rate(http_request_duration_seconds_count{job="myservice"}[{{.window}}])) alerting: name: "MyServiceHighErrorRate" labels: category: "availability" annotations: # Overwrite default Sloth SLO alert summmary on ticket and page alerts. summary: "High error rate on 'myservice' requests responses" page_alert: labels: severity: "pageteam" routing_key: "myteam" ticket_alert: labels: severity: "slack" slack_channel: "#alerts-myteam" ``` -------------------------------- ### Apply PrometheusServiceLevel CR and Get PrometheusRules Source: https://context7.com/slok/sloth/llms.txt Apply your PrometheusServiceLevel CRD to the cluster using kubectl. The Sloth controller will then automatically create the corresponding PrometheusRule in the same namespace. You can verify this by listing the PrometheusRules. ```bash kubectl apply -f my-api-slo.yaml # The controller reconciles the CR and creates a PrometheusRule in the same namespace kubectl get prometheusrules -n monitoring ``` -------------------------------- ### Generated PrometheusRule Output Structure Source: https://context7.com/slok/sloth/llms.txt This is an example of the generated PrometheusRule output for a single SLO. It includes recording rules for SLI error ratios across various time windows. ```yaml # Abbreviated output for one SLO (myservice / requests-availability / 99.9%) groups: # 1. SLI error ratio recording rules (8 time windows) - name: sloth-slo-sli-recordings-myservice-requests-availability rules: - record: slo:sli_error:ratio_rate5m expr: | (sum(rate(http_request_duration_seconds_count{job="myservice",code=~"(5..|429)"}[5m]))) / (sum(rate(http_request_duration_seconds_count{job="myservice"}[5m]))) labels: sloth_id: myservice-requests-availability sloth_service: myservice sloth_slo: requests-availability sloth_window: 5m # ... repeated for 30m, 1h, 2h, 6h, 1d, 3d, 30d ``` -------------------------------- ### Create Kind Kubernetes Cluster Source: https://github.com/slok/sloth/blob/main/CONTRIBUTING.md Set up a local Kubernetes cluster using Kind for integration testing. This involves creating the cluster and exporting its kubeconfig. ```bash kind create cluster --name sloth ``` ```bash kind get kubeconfig --name sloth > /tmp/kind-sloth.kubeconfig ``` -------------------------------- ### Run Unit Tests Source: https://github.com/slok/sloth/blob/main/CONTRIBUTING.md Execute the project's unit tests to verify individual components. ```bash make test ``` -------------------------------- ### Get GroupVersionKind from Kind Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Converts an unqualified kind string into a qualified GroupVersionKind. ```go func VersionKind(kind string) schema.GroupVersionKind ``` -------------------------------- ### Import v1 Alert Windows Package Source: https://github.com/slok/sloth/blob/main/pkg/prometheus/alertwindows/v1/README.md Import the v1 package for Prometheus alert windows. ```go import "github.com/slok/sloth/pkg/prometheus/alertwindows/v1" ``` -------------------------------- ### SLI Plugin for HTTP Availability (Go) Source: https://context7.com/slok/sloth/llms.txt This Go plugin generates a PromQL query for HTTP availability. It requires 'job' as an option and optionally accepts a 'filter' for additional label filtering. Ensure SLIPluginVersion and SLIPluginID constants are exported. ```go // examples/plugins/getting-started/availability/plugin.go package availability import ( "bytes" "context" "fmt" "text/template" ) const ( SLIPluginVersion = "prometheus/v1" SLIPluginID = "getting_started_availability" ) var queryTpl = template.Must(template.New("").Parse( ` sum(rate(http_request_duration_seconds_count{ {{.filter}}job=\"{{.job}}\",code=~"(5..|429)" }[{{"{{.window}}"}}})) / sum(rate(http_request_duration_seconds_count{ {{.filter}}job=\"{{.job}}\" }[{{"{{.window}}"}}})) `)) // SLIPlugin returns an error-ratio PromQL expression for HTTP availability. func SLIPlugin(ctx context.Context, meta, labels, options map[string]string) (string, error) { job, ok := options["job"] if !ok { return "", fmt.Errorf("job options is required") } filter := options["filter"] // optional extra label filter e.g. 'env="prod",' var b bytes.Buffer err := queryTpl.Execute(&b, map[string]string{"job": job, "filter": filter}) if err != nil { return "", fmt.Errorf("could not execute template: %w", err) } return b.String(), nil } ``` ```yaml # Reference the plugin in a spec slos: - name: "requests-availability" objective: 99.9 sli: plugin: id: "getting_started_availability" options: job: "myservice" filter: 'env="prod",' alerting: name: MyServiceHighErrorRate page_alert: labels: severity: critical ``` ```bash # Load plugin from local directory sloth generate -i ./slos/myservice.yml -p ./examples/plugins/getting-started/ ``` -------------------------------- ### Get GroupKind from Kind Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Converts an unqualified kind string into a qualified GroupKind. ```go func Kind(kind string) schema.GroupKind ``` -------------------------------- ### Implement Custom SLO Plugin Source: https://context7.com/slok/sloth/llms.txt This Go code implements the `pluginslov1.Plugin` interface to add an 'env' label to SLI recording rules. It requires a configuration with an 'env' field. ```go package myplugin import ( "context" "encoding/json" "fmt" pluginslov1 "github.com/slok/sloth/pkg/prometheus/plugin/slo/v1" ) const ( PluginVersion = "prometheus/slo/v1" PluginID = "myorg/add_env_label/v1" ) type Config struct { Env string `json:"env"` } // NewPlugin is the required factory function name. func NewPlugin(configData json.RawMessage, _ pluginslov1.AppUtils) (pluginslov1.Plugin, error) { var cfg Config if err := json.Unmarshal(configData, &cfg); err != nil { return nil, fmt.Errorf("invalid config: %w", err) } if cfg.Env == "" { return nil, fmt.Errorf("env is required") } return &myPlugin{env: cfg.Env}, nil } type myPlugin struct{ env string } func (p *myPlugin) ProcessSLO(ctx context.Context, req *pluginslov1.Request, res *pluginslov1.Result) error { // Add an env label to every SLI recording rule for i := range res.SLORules.SLIErrorRecRules.Rules { if res.SLORules.SLIErrorRecRules.Rules[i].Labels == nil { res.SLORules.SLIErrorRecRules.Rules[i].Labels = map[string]string{} } res.SLORules.SLIErrorRecRules.Rules[i].Labels["env"] = p.env } return nil } ``` -------------------------------- ### Import Sloth Kubernetes API v1 Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Import the Sloth Kubernetes API v1 package to use its types and functions. ```go import "github.com/slok/sloth/pkg/kubernetes/api/sloth/v1" ``` -------------------------------- ### QuickSlowWindow Struct Definition Source: https://github.com/slok/sloth/blob/main/pkg/prometheus/alertwindows/v1/README.md Defines quick and slow alerting windows. ```go type QuickSlowWindow struct { // Quick represents the windows for the quick alerting trigger. Quick Window `yaml:"quick"` // Slow represents the windows for the slow alerting trigger. Slow Window `yaml:"slow"` } ``` -------------------------------- ### Run Sloth Controller with Local Cluster Source: https://github.com/slok/sloth/blob/main/CONTRIBUTING.md Connect the Sloth controller to a local Kubernetes cluster using provided credentials. ```bash go run ./cmd/sloth/ controller --kube-local ``` -------------------------------- ### Initialize Scheme Builder and AddToScheme Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Initializes a scheme builder and provides a global function to register the API group and version to a scheme. ```go var ( // SchemeBuilder initializes a scheme builder. SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes) // AddToScheme is a global function that registers this API group & version to a scheme. AddToScheme = SchemeBuilder.AddToScheme ) ``` -------------------------------- ### Window Struct Definition Source: https://github.com/slok/sloth/blob/main/pkg/prometheus/alertwindows/v1/README.md Defines the parameters for an alerting window. ```go type Window struct { // ErrorBudgetPercent is the max error budget consumption allowed in the window. ErrorBudgetPercent float64 `yaml:"errorBudgetPercent"` // Shortwindow is the window that will stop the alerts when a huge amount of // error budget has been consumed but the error has already gone. ShortWindow prometheusmodel.Duration `yaml:"shortWindow"` // Longwindow is the window used to get the error budget for all the window. LongWindow prometheusmodel.Duration `yaml:"longWindow"` } ``` -------------------------------- ### Example Prometheus v1 SLO Spec Source: https://github.com/slok/sloth/blob/main/pkg/prometheus/api/v1/README.md This YAML defines two SLOs for a Kubernetes API server: one for request availability and another for request latency. It includes detailed SLI queries and alerting configurations for both page and ticket alerts. ```yaml version: "prometheus/v1" service: "k8s-apiserver" labels: cluster: "valhalla" component: "kubernetes" slos: - name: "requests-availability" objective: 99.9 description: "Common SLO based on availability for Kubernetes apiserver HTTP request responses." sli: events: error_query: sum(rate(apiserver_request_total{code=~"(5..|429)"}[{{.window}}])) total_query: sum(rate(apiserver_request_total[{{.window}}])) alerting: name: K8sApiserverAvailabilityAlert labels: category: "availability" annotations: runbook: "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorshigh" page_alert: labels: severity: critical ticket_alert: labels: severity: warning - name: "requests-latency" objective: 99 description: "Common SLO based on latency for Kubernetes apiserver HTTP request responses." sli: events: error_query: | ( sum(rate(apiserver_request_duration_seconds_count{verb!="WATCH"}[{{.window}}])) - sum(rate(apiserver_request_duration_seconds_bucket{le="0.4",verb!="WATCH"}[{{.window}}])) ) total_query: sum(rate(apiserver_request_duration_seconds_count{verb!="WATCH"}[{{.window}}])) alerting: name: K8sApiserverLatencyAlert labels: category: "latency" annotations: runbook: "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapilatencyhigh" page_alert: labels: severity: critical ticket_alert: labels: disable: true ``` -------------------------------- ### Run Kubernetes Integration Tests Source: https://github.com/slok/sloth/blob/main/CONTRIBUTING.md Execute integration tests for Sloth's Kubernetes controller functionality. Requires the Sloth binary and Kubernetes cluster configuration to be set. ```bash make ci-integration-k8s ``` -------------------------------- ### Default Plugin Usage Source: https://github.com/slok/sloth/blob/main/internal/plugin/slo/core/sli_rules_v1/README.md This snippet shows the default configuration where the SLI Rules V1 plugin is automatically loaded. ```yaml chain: - id: "sloth.dev/core/sli_rules/v1" ``` -------------------------------- ### Configure Sloth Kubernetes Controller with Custom Settings Source: https://context7.com/slok/sloth/llms.txt Customize the plugin directory (`-p`), resync interval (`--resync-interval`), and worker count (`--workers`) for the Sloth Kubernetes controller. ```bash sloth kubernetes-controller -p ./plugins/ --resync-interval 10m --workers 10 ``` -------------------------------- ### Include No-op Plugin in Chain Source: https://github.com/slok/sloth/blob/main/internal/plugin/slo/core/noop_v1/README.md Demonstrates how to include the no-op plugin in a plugin chain configuration. ```yaml chain: - id: "sloth.dev/core/noop/v1" ``` -------------------------------- ### Run CLI Integration Tests Source: https://github.com/slok/sloth/blob/main/CONTRIBUTING.md Execute integration tests for the Sloth command-line interface. ```bash make ci-integration-cli ``` -------------------------------- ### Set Sloth Integration Binary Path Source: https://github.com/slok/sloth/blob/main/CONTRIBUTING.md Export the path to the Sloth binary for integration tests. Ensure the binary is built first (e.g., using `make build`). ```bash export SLOTH_INTEGRATION_BINARY=${PWD}/bin/sloth-linux-amd64 ``` -------------------------------- ### Generate SLO Rules with Global SLO Plugins Source: https://context7.com/slok/sloth/llms.txt Apply SLO plugins globally via the CLI using the `--slo-plugins` flag with a JSON string specifying plugin ID and configuration. ```bash sloth generate -i ./slos/ --slo-plugins '{"id":"sloth.dev/contrib/info_labels/v1","config":{"labels":{"env":"prod"}}}' ``` -------------------------------- ### AlertWindows Kind Constant Source: https://github.com/slok/sloth/blob/main/pkg/prometheus/alertwindows/v1/README.md Defines the resource kind for AlertWindows. ```go const Kind = "AlertWindows" ``` -------------------------------- ### Create PrometheusSLOGenerator with Custom Config Source: https://context7.com/slok/sloth/llms.txt Instantiate a PrometheusSLOGenerator with optional configurations like custom plugin filesystems or extra labels. This is useful for integrating SLO generation into existing Go applications. ```go import ( "context" "os" "github.com/slok/sloth/pkg/lib" ) func main() { ctx := context.Background() gen, err := lib.NewPrometheusSLOGenerator(lib.PrometheusSLOGeneratorConfig{ // Optional: provide custom plugin filesystems // PluginsFS: []fs.FS{os.DirFS("./plugins/")}, // Optional: provide custom alert window catalog // WindowsFS: os.DirFS("./windows/"), ExtraLabels: map[string]string{"source": "my-automation"}, // CallerAgent: lib.CallerAgentAPI (default when unset) }) if err != nil { panic(err) } sloSpec := []byte(` version: "prometheus/v1" service: "myservice" labels: owner: "myteam" slos: - name: "requests-availability" objective: 99.9 sli: events: error_query: sum(rate(http_request_duration_seconds_count{job="myservice",code=~"(5..|429)"}[{{.window}}])) total_query: sum(rate(http_request_duration_seconds_count{job="myservice"}[{{.window}}])) alerting: name: MyServiceHighErrorRate page_alert: labels: severity: critical `) // GenerateFromRaw auto-detects prometheus/v1, k8s CRD, or OpenSLO format result, err := gen.GenerateFromRaw(ctx, sloSpec) if err != nil { panic(err) } // Write as standard Prometheus rules YAML err = gen.WriteResultAsPrometheusStd(ctx, *result, os.Stdout) if err != nil { panic(err) } } ``` -------------------------------- ### AlertWindows API Version Constant Source: https://github.com/slok/sloth/blob/main/pkg/prometheus/alertwindows/v1/README.md Defines the API version for AlertWindows resources. ```go const APIVersion = "sloth.slok.dev/v1" ``` -------------------------------- ### PageWindow Struct Definition Source: https://github.com/slok/sloth/blob/main/pkg/prometheus/alertwindows/v1/README.md Represents the configuration for page alerting. ```go type PageWindow struct { QuickSlowWindow `yaml:",inline"` } ``` -------------------------------- ### Configure Sloth Kubernetes Controller with Custom Addresses Source: https://context7.com/slok/sloth/llms.txt Set custom addresses for metrics and hot-reloading using `--metrics-listen-addr` and `--hot-reload-addr`, along with the hot-reload path via `--hot-reload-path`. ```bash sloth kubernetes-controller \ --metrics-listen-addr :9090 \ --hot-reload-addr :9091 \ --hot-reload-path /-/reload ``` -------------------------------- ### Define SLOPlugin for Plugin Chain Configuration Source: https://github.com/slok/sloth/blob/main/pkg/prometheus/api/v1/README.md Configure SLOPlugin to specify a plugin by its ID, provide configuration via raw JSON, and set its priority within the plugin chain. Lower numbers indicate higher priority. ```go type SLOPlugin struct { // ID is the ID of the plugin to load . ID string `json:"id"` // Config is the configuration of the plugin creation. Config json.RawMessage `json:"config,omitempty"` // Priority is the priority of the plugin in the chain. The lower the number // the higher the priority. The first plugin will be the one with the lowest // priority. // The default plugins loaded by Sloth use `0` priority. If you want to // execute plugins before the default ones, you can use negative priority. // It is recommended to use round gaps of numbers like 10, 100, 1000, -200, -1000... Priority int `json:"priority,omitempty"` } ``` -------------------------------- ### Generate SLO Rules with Custom SLO Period Windows Source: https://context7.com/slok/sloth/llms.txt Use a custom catalog of SLO period windows by providing a path with `--slo-period-windows-path` and set a default period with `--default-slo-period`. ```bash sloth generate -i ./slos/myservice.yml --slo-period-windows-path ./windows/ --default-slo-period 7d ``` -------------------------------- ### SLI DeepCopyInto Method Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated DeepCopyInto method for SLI. Use this to copy the SLI object into another SLI. ```go func (in *SLI) DeepCopyInto(out *SLI) ``` -------------------------------- ### Run Sloth Kubernetes Controller with Label Selector Source: https://context7.com/slok/sloth/llms.txt Filter the `PrometheusServiceLevel` resources the controller watches by using a label selector with the `--label-selector` flag. ```bash sloth kubernetes-controller --label-selector "team=platform" ``` -------------------------------- ### SLI DeepCopy Method Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated DeepCopy method for SLI. Use this to create a deep copy of the SLI object. ```go func (in *SLI) DeepCopy() *SLI ``` -------------------------------- ### Run Sloth Kubernetes Controller Locally Source: https://context7.com/slok/sloth/llms.txt Run the Sloth Kubernetes controller locally against a cluster using `--kube-local` and specify the Kubernetes context with `--kube-context`. ```bash sloth kubernetes-controller --kube-local --kube-context my-context ``` -------------------------------- ### Set Kubernetes Integration Test Environment Variables Source: https://github.com/slok/sloth/blob/main/CONTRIBUTING.md Configure environment variables for Kubernetes integration tests, specifying the Sloth binary path and the Kubernetes cluster configuration. ```bash export SLOTH_INTEGRATION_BINARY=${PWD}/bin/sloth-linux-amd64 ``` ```bash export SLOTH_INTEGRATION_KUBE_CONFIG=/tmp/kind-sloth.kubeconfig ``` -------------------------------- ### PrometheusServiceLevelList DeepCopyInto Go Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated deepcopy function for PrometheusServiceLevelList. Copies the receiver into the provided out parameter. Ensure the in parameter is non-nil. ```go func (in *PrometheusServiceLevelList) DeepCopyInto(out *PrometheusServiceLevelList) ``` -------------------------------- ### Generate SLO Rules with Custom Plugin Directory Source: https://context7.com/slok/sloth/llms.txt Specify a custom directory for SLI/SLO plugins using the `-p` flag when generating rules. ```bash sloth generate -i ./slos/myservice.yml -p ./plugins/ ``` -------------------------------- ### Alerting Configuration for SLOs Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Wraps all alert configurations for a given SLO, including a name, default labels and annotations, and specific configurations for page (critical) and ticket (warning) alerts. Use this to define the alerting behavior for an SLO. ```go type Alerting struct { // Name is the name used by the alerts generated for this SLO. // +optional Name string `json:"name,omitempty" // Labels are the Prometheus labels that will have all the alerts generated by this SLO. // +optional Labels map[string]string `json:"labels,omitempty" // Annotations are the Prometheus annotations that will have all the alerts generated by // this SLO. // +optional Annotations map[string]string `json:"annotations,omitempty" // Page alert refers to the critical alert (check multiwindow-multiburn alerts). PageAlert Alert `json:"pageAlert,omitempty" // TicketAlert alert refers to the warning alert (check multiwindow-multiburn alerts). TicketAlert Alert `json:"ticketAlert,omitempty" } ``` -------------------------------- ### SLOPlugin DeepCopyInto Method Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated deepcopy function for SLOPlugin. It copies the receiver into the provided out parameter. The in parameter must be non-nil. ```go func (in *SLOPlugin) DeepCopyInto(out *SLOPlugin) ``` -------------------------------- ### DeepCopyInto for Alerting Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated function to deep copy an Alerting object into another. Ensure the destination object is non-nil. ```go func (in *Alerting) DeepCopyInto(out *Alerting) ``` -------------------------------- ### Use Built-in Contrib SLO Plugin for Info Labels Source: https://context7.com/slok/sloth/llms.txt This YAML demonstrates using the built-in `sloth.dev/contrib/info_labels/v1` plugin to add extra labels to the `sloth_slo_info` metadata metric. Configure the desired labels within the plugin's config. ```yaml version: "prometheus/v1" service: "myservice" labels: owner: "myteam" tier: "2" slos: # sloth.dev/contrib/info_labels/v1 — adds extra labels to the sloth_slo_info metadata metric - name: "requests-availability-with-info-labels" objective: 99.9 plugins: chain: - id: "sloth.dev/contrib/info_labels/v1" config: labels: team: platform cost_center: "12345" sli: events: error_query: sum(rate(http_request_duration_seconds_count{job="myservice",code=~"(5..|429)"}[{{.window}}])) total_query: sum(rate(http_request_duration_seconds_count{job="myservice"}[{{.window}}])) alerting: name: MyServiceHighErrorRate page_alert: labels: severity: critical ``` -------------------------------- ### AlertWindows Struct Definition Source: https://github.com/slok/sloth/blob/main/pkg/prometheus/alertwindows/v1/README.md Represents the root type of the Alerting window configuration. ```go type AlertWindows struct { Kind string `yaml:"kind"` APIVersion string `yaml:"apiVersion"` Spec Spec `yaml:"spec"` } ``` -------------------------------- ### DeepCopy for Alerting Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated function to create a deep copy of an Alerting object. ```go func (in *Alerting) DeepCopy() *Alerting ``` -------------------------------- ### SLIPlugin DeepCopyInto Method Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated DeepCopyInto method for the SLIPlugin type. ```go func (in *SLIPlugin) DeepCopyInto(out *SLIPlugin) ``` -------------------------------- ### PrometheusServiceLevelStatus DeepCopy Method Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated DeepCopy method for PrometheusServiceLevelStatus. Use this to create a deep copy of the object. ```go func (in *PrometheusServiceLevelStatus) DeepCopy() *PrometheusServiceLevelStatus ``` -------------------------------- ### TicketWindow Struct Definition Source: https://github.com/slok/sloth/blob/main/pkg/prometheus/alertwindows/v1/README.md Represents the configuration for ticket alerting. ```go type TicketWindow struct { QuickSlowWindow `yaml:",inline"` } ``` -------------------------------- ### Run Sloth Kubernetes Controller in Fake Mode Source: https://context7.com/slok/sloth/llms.txt Run the Sloth Kubernetes controller in fake mode using `--mode fake`. This mode does not require a Kubernetes cluster. ```bash sloth kubernetes-controller --mode fake ``` -------------------------------- ### DeepCopyInto for PrometheusServiceLevel Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated function to deep copy a PrometheusServiceLevel object into another. Ensure the destination object is non-nil. ```go func (in *PrometheusServiceLevel) DeepCopyInto(out *PrometheusServiceLevel) ``` -------------------------------- ### PrometheusServiceLevelStatus DeepCopyInto Method Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated DeepCopyInto method for PrometheusServiceLevelStatus. Use this to copy the object into another PrometheusServiceLevelStatus. ```go func (in *PrometheusServiceLevelStatus) DeepCopyInto(out *PrometheusServiceLevelStatus) ``` -------------------------------- ### SLO DeepCopyInto Method Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated DeepCopyInto method for the SLO type. ```go func (in *SLO) DeepCopyInto(out *SLO) ``` -------------------------------- ### SLOPlugin DeepCopy Method Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated deepcopy function for SLOPlugin. It copies the receiver and creates a new SLOPlugin instance. ```go func (in *SLOPlugin) DeepCopy() *SLOPlugin ``` -------------------------------- ### Define SLIPlugin for SLI Calculation Source: https://github.com/slok/sloth/blob/main/pkg/prometheus/api/v1/README.md Configure SLIPlugin to use a specific SLI plugin by its ID and provide custom options for its execution. ```go type SLIPlugin struct { // Name is the name of the plugin that needs to load. ID string `json:"id"` // Options are the options used for the plugin. Options map[string]string `json:"options"` } ``` -------------------------------- ### DeepCopyInto for Alert Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated function to deep copy an Alert object into another. Ensure the destination object is non-nil. ```go func (in *Alert) DeepCopyInto(out *Alert) ``` -------------------------------- ### DeepCopy for PrometheusServiceLevel Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated function to create a deep copy of a PrometheusServiceLevel object. ```go func (in *PrometheusServiceLevel) DeepCopy() *PrometheusServiceLevel ``` -------------------------------- ### Generate SLO Rules from Directory to Directory Source: https://context7.com/slok/sloth/llms.txt Process all SLO spec files within a directory and mirror the input directory structure in the output directory for the generated rules. ```bash sloth generate -i ./slos/ -o ./rules/ ``` -------------------------------- ### Generate SLO Rules with Custom Alert Windows Source: https://context7.com/slok/sloth/llms.txt Specify a custom directory for alert window definitions using --slo-period-windows-path and set a default SLO period with --default-slo-period. This allows for non-standard SLO periods like 7 days. ```yaml # examples/windows/7d.yaml apiVersion: sloth.slok.dev/v1 kind: AlertWindows spec: sloPeriod: 7d page: quick: errorBudgetPercent: 8 shortWindow: 5m longWindow: 1h slow: errorBudgetPercent: 12.5 shortWindow: 30m longWindow: 6h ticket: quick: errorBudgetPercent: 20 shortWindow: 2h longWindow: 1d slow: errorBudgetPercent: 42 shortWindow: 6h longWindow: 3d ``` ```bash # Use custom windows directory and set 7d as the default period sloth generate -i ./slos/ -o ./rules/ \ --slo-period-windows-path ./examples/windows/ \ --default-slo-period 7d ``` -------------------------------- ### PrometheusServiceLevelSpec DeepCopyInto Go Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated deepcopy function for PrometheusServiceLevelSpec. Copies the receiver into the provided out parameter. Ensure the in parameter is non-nil. ```go func (in *PrometheusServiceLevelSpec) DeepCopyInto(out *PrometheusServiceLevelSpec) ``` -------------------------------- ### SLIEvents DeepCopy Method Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated DeepCopy method for SLIEvents. Use this to create a deep copy of the SLIEvents object. ```go func (in *SLIEvents) DeepCopy() *SLIEvents ``` -------------------------------- ### Explicitly Include Alert Rules V1 Plugin Source: https://github.com/slok/sloth/blob/main/internal/plugin/slo/core/alert_rules_v1/README.md Demonstrates how to explicitly include the alert_rules_v1 plugin in the Sloth plugin chain configuration. ```yaml chain: - id: "sloth.dev/core/alert_rules/v1" ``` -------------------------------- ### SLOPlugins DeepCopyInto Method Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated deepcopy function for SLOPlugins. It copies the receiver into the provided out parameter. The in parameter must be non-nil. ```go func (in *SLOPlugins) DeepCopyInto(out *SLOPlugins) ``` -------------------------------- ### Validate SLO Manifests with Custom Plugin Path Source: https://context7.com/slok/sloth/llms.txt Provide a custom plugin directory path using `-p` when validating SLO manifests, which is necessary if specs reference custom SLI plugins. ```bash sloth validate -i ./slos/ -p ./plugins/ ``` -------------------------------- ### Run Project Checks Source: https://github.com/slok/sloth/blob/main/CONTRIBUTING.md Execute automated checks to ensure code quality and adherence to project standards. ```bash make check ``` -------------------------------- ### SLIEvents DeepCopyInto Method Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated DeepCopyInto method for SLIEvents. Use this to copy the SLIEvents object into another SLIEvents. ```go func (in *SLIEvents) DeepCopyInto(out *SLIEvents) ``` -------------------------------- ### PrometheusServiceLevelList DeepCopyObject Go Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated deepcopy function for PrometheusServiceLevelList. Creates and returns a new runtime.Object by copying the receiver. ```go func (in *PrometheusServiceLevelList) DeepCopyObject() runtime.Object ``` -------------------------------- ### PrometheusServiceLevel DeepCopyObject Go Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated deepcopy function for PrometheusServiceLevel. Creates and returns a new runtime.Object by copying the receiver. ```go func (in *PrometheusServiceLevel) DeepCopyObject() runtime.Object ``` -------------------------------- ### SLIRaw DeepCopyInto Method Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated DeepCopyInto method for the SLIRaw type. ```go func (in *SLIRaw) DeepCopyInto(out *SLIRaw) ``` -------------------------------- ### SLO Plugin Chain Configuration (Basic) Source: https://github.com/slok/sloth/blob/main/internal/plugin/slo/contrib/remove_labels_v1/README.md Basic configuration for the remove_labels_v1 plugin within an SLO plugin chain. This enables the plugin with a specified priority. ```yaml chain: - id: "sloth.dev/contrib/remove_labels/v1" priority: 100 ``` -------------------------------- ### Add Custom Labels with Custom Info Metric Name Source: https://github.com/slok/sloth/blob/main/internal/plugin/slo/contrib/info_labels_v1/README.md Configure the plugin to use a custom metric name (e.g., `🦥_info`) for the info metric while also adding custom labels. This provides more flexibility in metric naming. ```yaml chain: - id: "sloth.dev/contrib/info_labels/v1" config: metricName: 🦥_info labels: label_k_1: label_v_2 label_k_3: label_v_4 ``` -------------------------------- ### SLO DeepCopy Method Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated DeepCopy method for the SLO type. ```go func (in *SLO) DeepCopy() *SLO ``` -------------------------------- ### Validate SLO Manifests with File Filtering Source: https://context7.com/slok/sloth/llms.txt Filter the files to be validated using include (`-n`) and exclude (`-e`) regular expressions. ```bash sloth validate -i ./slos/ -n '.*\.yml$' -e '.*_test\.yml$' ``` -------------------------------- ### DeepCopy for Alert Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated function to create a deep copy of an Alert object. ```go func (in *Alert) DeepCopy() *Alert ``` -------------------------------- ### PrometheusServiceLevelList DeepCopy Go Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated deepcopy function for PrometheusServiceLevelList. Creates a new PrometheusServiceLevelList by copying the receiver. ```go func (in *PrometheusServiceLevelList) DeepCopy() *PrometheusServiceLevelList ``` -------------------------------- ### PrometheusServiceLevelSpec DeepCopy Go Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated deepcopy function for PrometheusServiceLevelSpec. Creates a new PrometheusServiceLevelSpec by copying the receiver. ```go func (in *PrometheusServiceLevelSpec) DeepCopy() *PrometheusServiceLevelSpec ``` -------------------------------- ### SLIPlugin DeepCopy Method Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated DeepCopy method for the SLIPlugin type. ```go func (in *SLIPlugin) DeepCopy() *SLIPlugin ``` -------------------------------- ### SLOPlugins DeepCopy Method Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Autogenerated deepcopy function for SLOPlugins. It copies the receiver and creates a new SLOPlugins instance. ```go func (in *SLOPlugins) DeepCopy() *SLOPlugins ``` -------------------------------- ### Log All Details with Debug Plugin as Last Plugin Source: https://github.com/slok/sloth/blob/main/internal/plugin/slo/core/debug_v1/README.md Configure the debug plugin to log the custom message, the received result, and the request struct. Set a high priority to ensure it runs last in the chain. Sloth must be in debug mode for output. ```yaml chain: - id: "sloth.dev/core/debug/v1" priority: 9999999 config: {msg: "Last plugin", result: true, request: true} ``` -------------------------------- ### Generate SLOs from OpenSLO spec Source: https://context7.com/slok/sloth/llms.txt Use the Sloth CLI to generate standard Prometheus rules from OpenSLO-format specifications. This format uses `ratioMetrics` with `good`/`total` Prometheus queries. ```bash sloth generate -i ./examples/openslo-getting-started.yml ``` -------------------------------- ### Generate SLO Rules with Disabled Default SLO Plugins Source: https://context7.com/slok/sloth/llms.txt Disable all built-in SLO plugins using `--disable-default-slo-plugins` and specify a fully custom plugin chain with `--slo-plugins`. ```bash sloth generate -i ./slos/ --disable-default-slo-plugins --slo-plugins '{"id":"my-custom-plugin/v1"}' ``` -------------------------------- ### Generate SLO Rules with File Inclusion/Exclusion Regex Source: https://context7.com/slok/sloth/llms.txt Filter files for generation in directory mode using include (`-n`) and exclude (`-e`) regular expressions. ```bash sloth generate -i ./slos/ -o ./rules/ -n '.*prod.*' -e '.*test.*' ``` -------------------------------- ### Querying SLO Metrics with Removed Labels Source: https://github.com/slok/sloth/blob/main/internal/plugin/slo/contrib/remove_labels_v1/README.md Demonstrates how to query SLO metrics when custom labels have been removed by the plugin. Use the 'and' operator to join the SLO metric with the sloth_slo_info metric to filter by the removed label. ```promql slo:period_error_budget_remaining:ratio{team="observability"} ``` ```promql slo:period_error_budget_remaining:ratio and on (sloth_id) sloth_slo_info{team="observability"} ``` ```promql slo:period_error_budget_remaining:ratio * on (sloth_id) group_left(team) sloth_slo_info ``` -------------------------------- ### Run Sloth Kubernetes Controller in Dry-Run Mode Source: https://context7.com/slok/sloth/llms.txt Operate the Sloth Kubernetes controller in dry-run mode using `--mode dry-run`. This uses real Kubernetes clients but skips write operations. ```bash sloth kubernetes-controller --mode dry-run ``` -------------------------------- ### Generate SLO Rules from Multiple YAML Files Source: https://context7.com/slok/sloth/llms.txt Use the 'sloth generate' command with the -i flag to specify input YAML files containing SLO definitions. The -o flag specifies the output directory for the generated rules. ```yaml version: "prometheus/v1" service: "myservice" labels: owner: "myteam" slos: - name: "requests-availability" objective: 99.9 sli: events: error_query: sum(rate(http_request_duration_seconds_count{job="myservice",code=~"(5..|429)"}[{{.window}}])) total_query: sum(rate(http_request_duration_seconds_count{job="myservice"}[{{.window}}])) alerting: name: MyServiceHighErrorRate page_alert: labels: severity: critical --- version: "prometheus/v1" service: "myservice2" labels: owner: "myteam2" slos: - name: "requests-availability" objective: 99.99 sli: events: error_query: sum(rate(http_request_duration_seconds_count{job="myservice2",code=~"(5..|429)"}[{{.window}}])) total_query: sum(rate(http_request_duration_seconds_count{job="myservice2"}[{{.window}}])) alerting: name: MyService2HighErrorRate page_alert: labels: severity: critical ``` ```bash sloth generate -i ./examples/multifile.yml -o ./rules/multifile.yml ``` -------------------------------- ### Simple Message Log with Debug Plugin Source: https://github.com/slok/sloth/blob/main/internal/plugin/slo/core/debug_v1/README.md Configure the debug plugin to log a custom message. Ensure the Sloth runner is in debug mode to see this output. ```yaml chain: - id: "sloth.dev/core/debug/v1" config: msg: "Hello world" ``` -------------------------------- ### SLIEvents Go Struct Source: https://github.com/slok/sloth/blob/main/pkg/kubernetes/api/sloth/v1/README.md Defines an SLI based on event counts. Use this when your SLO can be calculated as a ratio of bad events to total events. Requires Prometheus queries with the `{{.window}}` template variable. ```go type SLIEvents struct { // ErrorQuery is a Prometheus query that will get the number/count of events // that we consider that are bad for the SLO (e.g "http 5xx", "latency > 250ms"...). // Requires the usage of `{{.window}}` template variable. ErrorQuery string `json:"errorQuery"` // TotalQuery is a Prometheus query that will get the total number/count of events // for the SLO (e.g "all http requests"...). // Requires the usage of `{{.window}}` template variable. TotalQuery string `json:"totalQuery"` } ``` -------------------------------- ### Alerting Type Source: https://github.com/slok/sloth/blob/main/pkg/prometheus/api/v1/README.md Wraps all the configuration required by the SLO alerts, including names, labels, annotations, and page/ticket alerts. ```APIDOC ## type Alerting Alerting wraps all the configuration required by the SLO alerts. ```go type Alerting struct { // Name is the name used by the alerts generated for this SLO. Name string `json:"name" validate:"required"` // Labels are the Prometheus labels that will have all the alerts generated by this SLO. Labels map[string]string `json:"labels,omitempty"` // Annotations are the Prometheus annotations that will have all the alerts generated by // this SLO. Annotations map[string]string `json:"annotations,omitempty"` // Page alert refers to the critical alert (check multiwindow-multiburn alerts). PageAlert Alert `json:"page_alert,omitempty"` // TicketAlert alert refers to the warning alert (check multiwindow-multiburn alerts). TicketAlert Alert `json:"ticket_alert,omitempty"` } ``` ``` -------------------------------- ### Run Sloth Controller in Dry-Run Mode Source: https://github.com/slok/sloth/blob/main/CONTRIBUTING.md Operate the Sloth controller in read-only mode, useful for testing configurations without making changes. Can be combined with local cluster connection. ```bash go run ./cmd/sloth/ controller --mode=dry-run ``` ```bash go run ./cmd/sloth/ controller --kube-local --mode=dry-run ``` -------------------------------- ### Generate SLO Rules from Single File to Output File Source: https://context7.com/slok/sloth/llms.txt Specify an output file path using the `-o` flag to save the generated Prometheus rules from a single SLO spec file. ```bash sloth generate -i ./slos/myservice.yml -o ./rules/myservice.yml ``` -------------------------------- ### Configure App Plugin for Victoria Metrics Validation via Env Vars Source: https://github.com/slok/sloth/blob/main/internal/plugin/slo/contrib/validate_victoria_metrics_v1/README.md This script demonstrates how to configure the Victoria Metrics validator using environment variables. It disables default plugins and sets the custom validator and default generator plugins for Sloth. ```bash #! /bin/bash export SLOTH_DISABLE_DEFAULT_SLO_PLUGINS=true export SLOTH_SLO_PLUGINS='{"id": "sloth.dev/contrib/validate_victoria_metrics/v1"} {"id": "sloth.dev/core/sli_rules/v1"} {"id": "sloth.dev/core/metadata_rules/v1"} {"id": "sloth.dev/core/alert_rules/v1"}' sloth generate -i ./examples/victoria-metrics.yml ``` -------------------------------- ### Validate SLO Manifests Ignoring Duplicates Source: https://context7.com/slok/sloth/llms.txt Allow duplicate SLO IDs (service+name pair) across different files during validation by using the `--ignore-slo-duplicates` flag. ```bash sloth validate -i ./slos/ --ignore-slo-duplicates ``` -------------------------------- ### Define SLOPlugins for Plugin Management Source: https://github.com/slok/sloth/blob/main/pkg/prometheus/api/v1/README.md SLOPlugins manages the list of plugins used in SLO generation. The `OverridePrevious` flag determines if preceding plugin lists are replaced. The `Chain` field holds the actual list of SLOPlugin configurations. ```go type SLOPlugins struct { // OverridePrevious will override the previous SLO plugins declared. // Depending on where is this SLO plugins block declared will override: // - If declared at SLO group level: Overrides the default plugins. // - If declared at SLO level: Overrides the default + SLO group plugins. // The declaration order is default plugins -> SLO Group plugins -> SLO plugins. OverridePrevious bool `json:"overridePrevious,omitempty"` // chain ths the list of plugin chain to add to the SLO generation. Chain []SLOPlugin `json:"chain"` } ``` -------------------------------- ### Set Default Interval via CLI Source: https://github.com/slok/sloth/blob/main/internal/plugin/slo/contrib/rule_intervals_v1/README.md Apply a default evaluation interval to all generated rules and SLOs using the Sloth CLI. ```bash sloth generate \ -i ./examples/getting-started.yml \ -s '{"id": "sloth.dev/contrib/rule_intervals/v1","config":{"interval": {"default":"2m"}}}' ``` -------------------------------- ### Define Spec Type for SLO Declarations Source: https://github.com/slok/sloth/blob/main/pkg/prometheus/api/v1/README.md Defines the root structure for SLO declarations. Includes version, service, labels, SLO plugins, and the list of SLOs. ```go type Spec struct { // Version is the version of the spec. Version string `json:"version"` // Service is the application of the SLOs. Service string `json:"service"` // Labels are the Prometheus labels that will have all the recording // and alerting rules generated for the service SLOs. Labels map[string]string `json:"labels,omitempty"` // SLOPlugins will be added to the SLO generation plugin chain of all SLOs. SLOPlugins SLOPlugins `json:"slo_plugins,omitempty"` // SLOs are the SLOs of the service. SLOs []SLO `json:"slos,omitempty"` } ```