Sloth Prometheus SLO Generator

https://sloth.dev
Admin
Sloth Prometheus SLO generator helps to stop using complex specs and processes to create Prometheus...
Tokens:135,256
Snippets:966
Trust Score:7.2
Update:3 months ago
Show doc for...
Tokens:
Raw
### Run Sloth Controller and Deploy SLOs

Source: https://sloth.dev/usage/kubernetes?q=

Deploy the Sloth controller to the 'monitoring' namespace and apply example SLOs. This demonstrates the basic setup for managing SLOs in Kubernetes.

```bash
# Run sloth controller.
$ kubectl create ns monitoring
$ kubectl apply -f ./deploy/kubernetes/raw/sloth.yaml

# Deploy some SLOs.
$ kubectl apply -f ./examples/k8s-getting-started.yml

```

--------------------------------

### Configure Availability Plugin

Source: https://sloth.dev/sli-plugins/slok-go-http-metrics-availability?q=

Example configuration for the availability plugin, specifying the plugin ID and required filter options.

```yaml
sli:
  plugin:
    id: "sloth-common/slok-go-http-metrics/availability"
    options:
      filter: exported_service="alertmanager-api",service="alertgram"

```

--------------------------------

### Run the Sloth Built-in UI

Source: https://sloth.dev/introduction/ui

Start the Sloth server to make the built-in UI accessible. The UI is available at http://localhost:8080 by default.

```bash
sloth server

```

--------------------------------

### Plugin Declarations at SLO Group and SLO Levels

Source: https://sloth.dev/usage/slo-plugins

Example demonstrating plugin declarations within an SLO group ('chain') and within specific SLOs.

```yaml
version: "prometheus/v1"
service: "myservice"
#...
slo_plugins:
  chain:
    - id: "sloth.dev/core/debug/v1"
      priority: 1000
      config:
        msg: "Plugin 3"

slos:
  - name: "slo1"
    plugins:
      chain:
        - id: "sloth.dev/core/debug/v1"
          config:
            msg: "Plugin 2"
        - id: "sloth.dev/core/debug/v1"
          priority: -999999
          config:
            msg: "Plugin 1"
    #...
    - name: "slo2"
    plugins:
      chain:
        - id: "sloth.dev/core/debug/v1"
          config:
            msg: "Plugin 1"
#...

```

--------------------------------

### CoreDNS Availability: Basic Usage

Source: https://sloth.dev/sli-plugins/coredns-availability

Example of using the CoreDNS availability plugin without any specific filters or custom rcode regex.

```yaml
sli:
  plugin:
    id: "sloth-common/coredns/availability"
```

--------------------------------

### CoreDNS Latency Plugin Configuration (Without Filter)

Source: https://sloth.dev/sli-plugins/coredns-latency

Configure the CoreDNS latency plugin with a required bucket option. This example shows the basic setup without any filtering.

```yaml
sli:
  plugin:
    id: "sloth-common/coredns/latency"
    options:
      bucket: "0.25"

```

--------------------------------

### Noop Plugin Configuration

Source: https://sloth.dev/sli-plugins/noop?q=

Configure the Noop plugin by specifying its ID. This is a basic setup for using the plugin as a placeholder.

```yaml
#...
 sli:
  plugin:
  id: "sloth-common/noop"


```

--------------------------------

### Noop Plugin Usage Example

Source: https://sloth.dev/sli-plugins/noop

This snippet shows how to configure and use the Noop plugin within a Sli configuration. It specifies the plugin ID for the Noop functionality.

```yaml
#...
sli:
  plugin:
  id: "sloth-common/noop"

```

--------------------------------

### Install Sloth CRDs and Manifests using Kustomize

Source: https://sloth.dev/introduction/install

Install Sloth Custom Resource Definitions (CRDs) and core manifests using Kustomize. This is part of the Kubernetes installation process.

```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- https://raw.githubusercontent.com/slok/sloth/v0.15.0/pkg/kubernetes/gen/crd/sloth.slok.dev_prometheusservicelevels.yaml
- https://raw.githubusercontent.com/slok/sloth/v0.15.0/deploy/kubernetes/raw/sloth-with-common-plugins.yaml
```

--------------------------------

### SLO Plugin Configuration Example

Source: https://sloth.dev/examples/default/slo-plugin

This snippet shows a complete SLO configuration, including service details, SLO definitions with multiple plugins, and alerting configurations. It demonstrates how to define error and total queries for the SLI.

```yaml
version: "prometheus/v1"
service: "myservice"
labels:
  owner: "myteam"
  repo: "myorg/myservice"
  tier: "2"
slo_plugins:
  chain:
    - id: "sloth.dev/core/debug/v1"
      priority: 9999999
      config: {msg: "Plugin 99"}
    - id: "sloth.dev/core/debug/v1"
      priority: -999999
      config: {msg: "Plugin 0"}

slos:
  # We allow failing (5xx and 429) 1 request every 1000 requests (99.9%).
  - name: "requests-availability"
    objective: 99.9
    description: "Common SLO based on availability for HTTP request responses."
    plugins:
      chain:
        - id: "sloth.dev/core/debug/v1"
          priority: 1050
          config: {msg: "Plugin 5"}
        - id: "sloth.dev/core/debug/v1"
          priority: -1000
          config: {msg: "Plugin 1"}
        - id: "sloth.dev/core/debug/v1"
          priority: 1000
          config: {msg: "Plugin 4"}
        - id: "sloth.dev/core/debug/v1"
          priority: -200
          config: {msg: "Plugin 2"}
        - id: "sloth.dev/core/debug/v1"
          config: {msg: "Plugin 3"}

    sli:
      events:
        error_query: sum(rate(http_request_duration_seconds_count{job="myservice",code=~"(5..|429)"}[{{.window}}]))
        total_query: sum(rate(http_request_duration_seconds_count{job="myservice"}[{{.window}}]))
    alerting:
      name: MyServiceHighErrorRate
      labels:
        category: "availability"
      annotations:
        # Overwrite default Sloth SLO alert summmary on ticket and page alerts.
        summary: "High error rate on 'myservice' requests responses"
      page_alert:
        labels:
          severity: pageteam
          routing_key: myteam
      ticket_alert:
        labels:
          severity: "slack"
          slack_channel: "#alerts-myteam"

```

--------------------------------

### No-op Plugin in Chain

Source: https://sloth.dev/slo-plugins/core/noop_v1?q=

This example shows how to include the no-op plugin in a plugin chain. It is useful for testing the plugin chain mechanism.

```yaml
chain:
  - id: "sloth.dev/core/noop/v1"

```

--------------------------------

### Add Sloth Helm Repository and Template

Source: https://sloth.dev/introduction/install

Add the Sloth Helm repository and generate Helm templates. This is an alternative method for installing Sloth on Kubernetes.

```bash
helm repo add sloth https://slok.github.io/sloth
helm repo update
helm template sloth/sloth --include-crds
```

--------------------------------

### SLO Spec using a Plugin

Source: https://sloth.dev/usage/sli-plugins?q=

An example SLO specification demonstrating how to reference a custom SLI plugin. The plugin is identified by its ID, and custom options can be passed to it.

```yaml
version: "prometheus/v1"
service: "myservice"
slos:
  - name: "some-slo"
    objective: 99.9
    sli:
      plugin:
        id: "test_plugin"
        options:
          opt1: "something"
          opt2: "something"
    alerting:
#...

```

--------------------------------

### Load and Use Remote SLO Plugins in Go

Source: https://sloth.dev/usage/go-library

This example demonstrates how to load and use remote SLO plugins within a Go application. It downloads a plugin from a URL and loads it into an in-memory filesystem for the SLO generator.

```go
package main

import (
    "context"
    "fmt"
    "io"
    "io/fs"
    "net/http"
    "os"
    "testing/fstest"

    sloth "github.com/slok/sloth/pkg/lib"
)

const (
    remotePluginURL = "https://gist.githubusercontent.com/slok/9408924d844049b675a8da9185422b40/raw/e36b7dd881f3465cb568db3edf5afb0d38b5427f/sloth-slo-plugin.go"
)

const sloSpec = `---
version: "prometheus/v1"
service: "myservice"
slo_plugins:
  chain:
  - id: "sloth-examples/add_description_to_info/v1"
slos:
  - name: "requests-availability"
    objective: 99.9
    description: "Common SLO based on availability for HTTP request responses."
    sli:
      events:
        error_query: sum(rate(http_request_duration_seconds_count{job="myservice",code=~"(5..|429)"}[{{.window}}]))
        total_query: sum(rate(http_request_duration_seconds_count{job="myservice"}[{{.window}}]))
    alerting:
      page_alert:
        disable: true
      ticket_alert:
        disable: true
`

func main() {
    ctx := context.Background()

    // Download the remote plugin.
    resp, err := http.Get(remotePluginURL)
    if err != nil {
        panic(err)
    }
    if resp.StatusCode != http.StatusOK {
        panic("failed to download remote plugin")
    }

    pluginB, err := io.ReadAll(resp.Body)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()

    // Load the downloaded plugin in an in-memory fs.
    f := fstest.MapFS{}
    f["remote-example/plugin.go"] = &fstest.MapFile{Data: pluginB}

    gen, err := sloth.NewPrometheusSLOGenerator(sloth.PrometheusSLOGeneratorConfig{PluginsFS: []fs.FS{f}})
    if err != nil {
        panic(fmt.Errorf("could not create SLO generator: %w", err))
    }

    // Generate SLOs.
    result, err := gen.GenerateFromRaw(ctx, []byte(sloSpec))
    if err != nil {
        panic(fmt.Sprintf("could not generate SLOs: %v", err))
    }
    err = gen.WriteResultAsPrometheusStd(ctx, *result, os.Stdout)
    if err != nil {
        panic(fmt.Sprintf("could not write result: %v", err))
    }
}

```

--------------------------------

### CoreDNS Latency Plugin Configuration (Without Filter)

Source: https://sloth.dev/sli-plugins/coredns-latency?q=

Example configuration for the CoreDNS latency plugin when no Prometheus filter is applied. Requires a 'bucket' option.

```yaml
sli:
  plugin:
    id: "sloth-common/coredns/latency"
    options:
      bucket: "0.25"
```

--------------------------------

### SLO Specification Example

Source: https://sloth.dev/introduction

Define SLOs for a service, including error event queries, total event queries, and alerting configurations. This example specifies availability for HTTP request responses.

```yaml
version: "prometheus/v1"
service: "myservice"
labels:
  owner: "myteam"
  repo: "myorg/myservice"
  tier: "2"
slos:
  # We allow failing (5xx and 429) 1 request every 1000 requests (99.9%).
  - name: "requests-availability"
    objective: 99.9
    description: "Common SLO based on availability for HTTP request responses."
    sli:
      events:
        error_query: sum(rate(http_request_duration_seconds_count{job="myservice",code=~"(5..|429)"}[{{.window}}]))
        total_query: sum(rate(http_request_duration_seconds_count{job="myservice"}[{{.window}}]))
    alerting:
      name: MyServiceHighErrorRate
      labels:
        category: "availability"
      annotations:
        # Overwrite default Sloth SLO alert summmary on ticket and page alerts.
        summary: "High error rate on 'myservice' requests responses"
      page_alert:
        labels:
          severity: pageteam
          routing_key: myteam
      ticket_alert:
        labels:
          severity: "slack"
          slack_channel: "#alerts-myteam"


```

--------------------------------

### SLO Metric Recording Example

Source: https://sloth.dev/examples/kubernetes/getting-started

Defines how to record SLO and SLI error rates over different time windows (3 days and 30 days). This is useful for tracking historical performance against SLOs.

```yaml
record: slo:sli_error:ratio_rate3d
    - expr: |
        sum_over_time(slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability", sloth_service="myservice", sloth_slo="requests-availability"}[30d])
        / ignoring (sloth_window)
        count_over_time(slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability", sloth_service="myservice", sloth_slo="requests-availability"}[30d])
      labels:
        cmd: examplesgen.sh
        owner: myteam
        repo: myorg/myservice
        sloth_id: myservice-requests-availability
        sloth_service: myservice
        sloth_slo: requests-availability
        sloth_window: 30d
        tier: "2"
      record: slo:sli_error:ratio_rate30d
```

--------------------------------

### Traefik V2 Latency Plugin Configuration (With Filter)

Source: https://sloth.dev/sli-plugins/traefik-v2-latency

Configure the Traefik V2 latency plugin with a Prometheus filter to narrow down the metrics. This example filters requests by the GET method.

```yaml
sli:
  plugin:
    id: "sloth-common/traefik/v2/latency"
    options:
      service_name_regex: "^default-slok-sloth$"
      bucket: "0.3"
      filter: method="GET"
```

--------------------------------

### Example Prometheus SLI Plugin

Source: https://sloth.dev/usage/sli-plugins?q=

A basic Go plugin for Prometheus v1 SLI. It defines required global constants and the main plugin function that returns a Prometheus query string. Options passed to the plugin are available as a map.

```go
package testplugin

import "context"

const (
  SLIPluginVersion = "prometheus/v1"
  SLIPluginID      = "test_plugin"
)

func SLIPlugin(ctx context.Context, meta, labels, options map[string]string) (string, error) {
  return "rate(my_raw_error_ratio_query{}[{{.window}}])", nil
}

```

--------------------------------

### CoreDNS Latency Plugin Configuration (With Filter)

Source: https://sloth.dev/sli-plugins/coredns-latency?q=

Example configuration for the CoreDNS latency plugin with a Prometheus filter. Includes 'bucket' and 'filter' options for metric selection.

```yaml
sli:
  plugin:
    id: "sloth-common/coredns/latency"
    options:
      bucket: "0.032"
      filter: job="kube-dns",server="dns://:53",service="kube-dns", zone="."
```

--------------------------------

### SLO Info Recording Example

Source: https://sloth.dev/examples/kubernetes/getting-started

Records general information about the SLO, including its objective, service, and version. This is useful for cataloging and querying SLO details.

```yaml
record: sloth_slo_info
    - expr: vector(1)
      labels:
        cmd: examplesgen.sh
        owner: myteam
        repo: myorg/myservice
        sloth_id: myservice-requests-availability
        sloth_mode: cli-gen-k8s
        sloth_objective: "99.9"
        sloth_service: myservice
        sloth_slo: requests-availability
        sloth_spec: sloth.slok.dev/v1
        sloth_version: dev
        tier: "2"
      record: sloth_slo_info
```

--------------------------------

### Example ConfigMap Transformer Plugin

Source: https://sloth.dev/usage/k8s-transformer-plugins

A simplified Go plugin that transforms Prometheus rules into a Kubernetes ConfigMap. It collects all rules, converts them to YAML, and embeds them within the ConfigMap's data.

```go
package plugin

import (
    "context"
    "github.com/slok/sloth/pkg/common/model"
    k8sutils "github.com/slok/sloth/pkg/common/utils/k8s"
    plugink8stransformv1 "github.com/slok/sloth/pkg/prometheus/plugin/k8stransform/v1"
    "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
)

const (
    PluginVersion = "prometheus/k8stransform/v1"
    PluginID      = "mycompany.com/prometheus-configmap/v1"
)

func NewPlugin() (plugink8stransformv1.Plugin, error) {
    return plugin{}, nil
}

type plugin struct{}

func (p plugin) TransformK8sObjects(
    ctx context.Context,
    kmeta model.K8sMeta,
    sloResult model.PromSLOGroupResult,
) (*plugink8stransformv1.K8sObjects, error) {
    u := &unstructured.Unstructured{}
    u.SetAPIVersion("v1")
    u.SetKind("ConfigMap")
    u.SetNamespace(kmeta.Namespace)
    u.SetName(kmeta.Name)
    u.SetLabels(kmeta.Labels)

    // Collect all rules
    groups := []model.PromRuleGroup{}
    for _, slo := range sloResult.SLOResults {
        groups = append(groups, 
            slo.PrometheusRules.SLIErrorRecRules,
            slo.PrometheusRules.MetadataRecRules,
            slo.PrometheusRules.AlertRules,
        )
    }

    // Convert to YAML
    rulesYAML, err := k8sutils.UnstructuredToYAMLString(
        map[string]any{"groups": groups},
    )
    if err != nil {
        return nil, err
    }

    u.Object["data"] = map[string]any{
        "prometheus-rules.yaml": rulesYAML,
    }

    return &plugink8stransformv1.K8sObjects{
        Items: []*unstructured.Unstructured{u},
    }, nil
}

```

--------------------------------

### Expose SLO Generation via REST API with Go

Source: https://sloth.dev/usage/go-library

This example shows how to expose the Sloth SLO generation functionality as a REST HTTP API. It requires the Go library and standard HTTP packages. Check with `curl -XPOST http://127.0.0.1:8080/sloth/generate -d "$(cat ./examples/getting-started.yml)" `.

```go
package main

import (
    "fmt"
    "io"
    "net/http"

    sloth "github.com/slok/sloth/pkg/lib"
)

// This example shows a basic usage of sloth library by exposing sloth SLO generation functionality as a rest HTTP API.
// Check with `curl -XPOST http://127.0.0.1:8080/sloth/generate -d "$(cat ./examples/getting-started.yml)" `.
func main() {
    gen, err := sloth.NewPrometheusSLOGenerator(sloth.PrometheusSLOGeneratorConfig{})
    if err != nil {
        panic(fmt.Errorf("could not create SLO generator: %w", err))
    }

    mux := http.NewServeMux()
    mux.HandleFunc("POST /sloth/generate", func(w http.ResponseWriter, r *http.Request) {
        // Get body from request.
        body, err := io.ReadAll(r.Body)
        if err != nil {
            http.Error(w, "failed to read request body", http.StatusBadRequest)
            return
        }
        defer r.Body.Close()

        // Generate SLOs.
        result, err := gen.GenerateFromRaw(r.Context(), body)
        if err != nil {
            http.Error(w, fmt.Sprintf("could not generate SLOs: %v", err), http.StatusInternalServerError)
            return
        }
        w.WriteHeader(http.StatusOK)
        err = gen.WriteResultAsPrometheusStd(r.Context(), *result, w)
        if err != nil {
            http.Error(w, fmt.Sprintf("could not write result: %v", err), http.StatusInternalServerError)
            return
        }
    })

    httpServer := &http.Server{Addr: ":8080", Handler: mux}

    fmt.Println("Starting server at :8080")

    err = httpServer.ListenAndServe()
    if err != nil {
        panic(err)
    }
}

```

--------------------------------

### SLO Metadata Recording Example

Source: https://sloth.dev/examples/kubernetes/getting-started

Records metadata for SLOs, including the objective ratio, error budget ratio, and time period in days. This helps in understanding the SLO's parameters.

```yaml
record: slo:objective:ratio
    - expr: vector(0.9990000000000001)
      labels:
        cmd: examplesgen.sh
        owner: myteam
        repo: myorg/myservice
        sloth_id: myservice-requests-availability
        sloth_service: myservice
        sloth_slo: requests-availability
        tier: "2"
      record: slo:objective:ratio
    - expr: vector(1-0.9990000000000001)
      labels:
        cmd: examplesgen.sh
        owner: myteam
        repo: myorg/myservice
        sloth_id: myservice-requests-availability
        sloth_service: myservice
        sloth_slo: requests-availability
        tier: "2"
      record: slo:error_budget:ratio
    - expr: vector(30)
      labels:
        cmd: examplesgen.sh
        owner: myteam
        repo: myorg/myservice
        sloth_id: myservice-requests-availability
        sloth_service: myservice
        sloth_slo: requests-availability
        tier: "2"
      record: slo:time_period:days
```

--------------------------------

### CoreDNS Availability: Custom Rcode with Filters

Source: https://sloth.dev/sli-plugins/coredns-availability

Example of configuring the CoreDNS availability plugin with both a custom Prometheus filter and a custom regex for error rcodes.

```yaml
sli:
  plugin:
    id: "sloth-common/coredns/availability"
    options:
      filter: job="kube-dns",server="dns://:53",service="kube-dns", zone="."
      custom_rcode_regex: (NXDOMAIN|SERVFAIL|FORMERR)
```

--------------------------------

### Kooper Availability Plugin With Filter

Source: https://sloth.dev/sli-plugins/kubernetes-kooper-availability

Example configuration for the Kooper availability plugin with a Prometheus extra label filter. This allows for more granular monitoring by specifying labels like 'job' and 'env'.

```yaml
sli:
  plugin:
    id: "sloth-common/kubernetes/kooper/availability"
    options:
      controller: "sloth"
      filter: "job=\"svc1\",env=\"prod\""
```

--------------------------------

### Traefik V1 Availability Plugin Without Filter

Source: https://sloth.dev/sli-plugins/traefik-v1-availability?q=

Example configuration for the Traefik V1 availability plugin when no specific filters are applied. Requires a regex to match Traefik backends.

```yaml
sli:
  plugin:
    id: "sloth-common/traefik/v1/availability"
    options:
      backend_regex: "^github.com/slok/sloth/?$"

```

--------------------------------

### Kubernetes SLO CRD with Plugins

Source: https://sloth.dev/examples/kubernetes/slo-plugin?q=

Defines a PrometheusServiceLevel CRD with multiple debug plugins in the SLO chain. This example showcases the structure for integrating plugins into SLO definitions.

```yaml
apiVersion: sloth.slok.dev/v1
kind: PrometheusServiceLevel
metadata:
  name: sloth-slo-my-service-with-slo-plugins
  namespace: monitoring
spec:
  service: "myservice"
  labels:
    owner: "myteam"
    repo: "myorg/myservice"
    tier: "2"
  sloPlugins:
    chain:
      - id: "sloth.dev/core/debug/v1"
        priority: 9999999
        config: {msg: "Plugin 99"}
      - id: "sloth.dev/core/debug/v1"
        priority: -999999
        config: {msg: "Plugin 0"}
  slos:
    - name: "requests-availability"
      objective: 99.9
      description: "Common SLO based on availability for HTTP request responses."
      plugins:
        chain:
          - id: "sloth.dev/core/debug/v1"
            priority: 1050
            config: {msg: "Plugin 5"}
          - id: "sloth.dev/core/debug/v1"
            priority: -1000
            config: {msg: "Plugin 1"}
          - id: "sloth.dev/core/debug/v1"
            priority: 1000
            config: {msg: "Plugin 4"}
          - id: "sloth.dev/core/debug/v1"
            priority: -200
            config: {msg: "Plugin 2"}
          - id: "sloth.dev/core/debug/v1"
            config: {msg: "Plugin 3"}
      sli:
        events:
          errorQuery: sum(rate(http_request_duration_seconds_count{job="myservice",code=~"(5..|429)"}[{{.window}}]))
          totalQuery: sum(rate(http_request_duration_seconds_count{job="myservice"}[{{.window}}]))
      alerting:
        name: MyServiceHighErrorRate
        labels:
          category: "availability"
        annotations:
          summary: "High error rate on 'myservice' requests responses"
        pageAlert:
          labels:
            severity: pageteam
            routing_key: myteam
        ticketAlert:
          labels:
            severity: "slack"
            slack_channel: "#alerts-myteam"

```

--------------------------------

### SLO Alert Configuration Example

Source: https://sloth.dev/examples/default/slo-plugin

This configuration defines an alert for a high error rate on a service's requests. It uses Prometheus query language to monitor SLO compliance over different time windows. Configure 'sloth_id', 'sloth_service', and 'sloth_slo' to match your service.

```yaml
  - alert: MyServiceHighErrorRate
    expr: |
      (
          max(slo:sli_error:ratio_rate2h{sloth_id="myservice-requests-availability", sloth_service="myservice", sloth_slo="requests-availability"} > (3 * 0.0009999999999999432)) without (sloth_window)
          and
          max(slo:sli_error:ratio_rate1d{sloth_id="myservice-requests-availability", sloth_service="myservice", sloth_slo="requests-availability"} > (3 * 0.0009999999999999432)) without (sloth_window)
      )
      or
      (
          max(slo:sli_error:ratio_rate6h{sloth_id="myservice-requests-availability", sloth_service="myservice", sloth_slo="requests-availability"} > (1 * 0.0009999999999999432)) without (sloth_window)
          and
          max(slo:sli_error:ratio_rate3d{sloth_id="myservice-requests-availability", sloth_service="myservice", sloth_slo="requests-availability"} > (1 * 0.0009999999999999432)) without (sloth_window)
      )
    labels:
      category: availability
      severity: slack
      slack_channel: '#alerts-myteam'
      sloth_severity: ticket
    annotations:
      summary: High error rate on 'myservice' requests responses
      title: (ticket) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget
        burn rate is too fast.
```

--------------------------------

### Traefik V1 Availability Plugin With Filter

Source: https://sloth.dev/sli-plugins/traefik-v1-availability?q=

Example configuration for the Traefik V1 availability plugin using a Prometheus filter to narrow down metrics. Requires a regex to match Traefik backends and a filter string.

```yaml
sli:
  plugin:
    id: "sloth-common/traefik/v1/availability"
    options:
      backend_regex: "^github.com/slok/sloth/?$"
      filter: method="GET"

```

--------------------------------

### Istio V1 Latency Plugin Configuration (Without Filter)

Source: https://sloth.dev/sli-plugins/istio-v1-latency?q=

Configure the Istio V1 latency plugin with required namespace, service, and bucket options. This basic setup measures latency without specific request filtering.

```yaml
sli:
  plugin:
    id: "sloth-common/istio/v1/latency"
    options:
      namespace: "default"
      service: "test"
      bucket: "300"

```

--------------------------------

### One-Liner Plugin Configuration

Source: https://sloth.dev/usage/slo-plugins

Demonstrates using YAML's one-liner syntax for plugin configurations, useful for brevity.

```yaml
# One liner on the config.
- id: "sloth.dev/core/debug/v1"
  config: {msg: "Plugin 2"}
# One liner on all the plugin.
- {id: "sloth.dev/core/debug/v1", priority: -999999, config: {msg: "Plugin 1"}}

```

--------------------------------

### Example SLO Alert Rule

Source: https://sloth.dev/examples/default/getting-started

This example shows a Prometheus alert rule for an SLO's error rate. It checks both 6-hour and 3-day error rates against a threshold.

```prometheus
max(slo:sli_error:ratio_rate6h{sloth_id="myservice-requests-availability", sloth_service="myservice", sloth_slo="requests-availability"} > (1 * 0.0009999999999999432)) without (sloth_window)
          and
          max(slo:sli_error:ratio_rate3d{sloth_id="myservice-requests-availability", sloth_service="myservice", sloth_slo="requests-availability"} > (1 * 0.0009999999999999432)) without (sloth_window)
      )
    labels:
      category: availability
      severity: slack
      slack_channel: '#alerts-myteam'
      sloth_severity: ticket
    annotations:
      summary: High error rate on 'myservice' requests responses
      title: (ticket) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget
        burn rate is too fast.
```

--------------------------------

### Load SLO Plugins from Paths

Source: https://sloth.dev/usage/slo-plugins

Use the '--plugins-path' or '-p' flag to specify directories for Sloth to recursively search for plugins. This flag can be repeated to include multiple paths.

```bash
sloth kubernetes-controller  -p ./plugins/ -p ../ext-plugins
INFO[0000] Plugins loaded                                sli-plugins=15 slo-plugins=24 version=dev window=30d

```

--------------------------------

### Prometheus SLO Time Period Recording

Source: https://sloth.dev/examples/kubernetes/multifile

Records the SLO time period in days. This example sets the period to 30 days.

```PromQL
owner: myteam
repo: myorg/myservice
sloth_id: myservice-requests-availability
sloth_service: myservice
sloth_slo: requests-availability
tier: "2"
record: slo:time_period:days
expr: vector(30)

```

--------------------------------

### Define SLO Time Period in Days

Source: https://sloth.dev/examples/default/raw-sli?q=

Specifies the time period for SLO calculations in days. This example sets it to 30 days.

```promql
vector(30)
```

--------------------------------

### Import Testing Utilities for K8s Transformer Plugins

Source: https://sloth.dev/usage/k8s-transformer-plugins?q=

Import the testing utilities provided by Sloth for simulating real plugin execution environments and catching errors early during development.

```go
import "github.com/slok/sloth/pkg/prometheus/plugin/k8stransform/v1/testing"
```

--------------------------------

### Build Sloth from Source Code

Source: https://sloth.dev/introduction/install

Clone the Sloth repository and build the binary from source. This method ensures you have the latest version and allows for custom builds.

```bash
git clone git@github.com:slok/sloth.git && \
cd ./sloth && \
make build && \
ls -la ./bin
```

--------------------------------

### Import Testing Utility for K8s Transformer Plugins

Source: https://sloth.dev/usage/k8s-transformer-plugins

Import the testing utility package provided by Sloth for Kubernetes transformer plugins. This utility helps simulate the real plugin execution environment and catch errors early during testing.

```go
import "github.com/slok/sloth/pkg/prometheus/plugin/k8stransform/v1/testing"

```

--------------------------------

### Go SLI Plugin Implementation

Source: https://sloth.dev/examples/default/sli-plugin?q=

This Go function implements a SLI plugin that generates a Prometheus query. It requires 'job' as an option and validates 'owner' and 'tier' labels. It also sanitizes a Prometheus filter string.

```go
package availability

import (
    "bytes"
    "context"
    "fmt"
    "regexp"
    "strings"
    "text/template"
)

const (
    SLIPluginVersion = "prometheus/v1"
    SLIPluginID      = "getting_started_availability"
)

var queryTpl = template.Must(template.New("").Parse(`
sum(rate(http_request_duration_seconds_count{ {{.filter}}job=\"{{.job}}\",code=~\"(5..|429)\" }[{{"{{.window}}"}}]))) 
/
sum(rate(http_request_duration_seconds_count{ {{.filter}}job=\"{{.job}}\" }[{{"{{.window}}"}}]))`))

var filterRegex = regexp.MustCompile(`([^="]+="[^",]+",)+`)

// SLIPlugin is the getting started plugin example.
//
// It will return an Sloth error ratio raw query that returns the error ratio of HTTP requests based
// on the HTTP response status code, taking 5xx and 429 as error events.
func SLIPlugin(ctx context.Context, meta, labels, options map[string]string) (string, error) {
    // Get job.
    job, ok := options["job"]
    if !ok {
        return "", fmt.Errorf("job options is required")
    }

    // Validate labels.
    err := validateLabels(labels, "owner", "tier")
    if err != nil {
        return "", fmt.Errorf("invalid labels: %w", err)
    }

    // Sanitize filter.
    filter := options["filter"]
    if filter != "" {
        filter = strings.Trim(filter, "{}")
        filter = strings.Trim(filter, ",")
        filter = filter + ","
        match := filterRegex.MatchString(filter)
        if !match {
            return "", fmt.Errorf("invalid prometheus filter: %s", filter)
        }
    }

    // Create query.
    var b bytes.Buffer
    data := map[string]string{
        "job":    job,
        "filter": filter,
    }
    err = queryTpl.Execute(&b, data)
    if err != nil {
        return "", fmt.Errorf("could not execute template: %w", err)
    }

    return b.String(), nil
}

// validateLabels will check the labels exist.
func validateLabels(labels map[string]string, requiredKeys ...string) error {
    for _, k := range requiredKeys {
        v, ok := labels[k]
        if !ok || (ok && v == "") {
            return fmt.Errorf("%q label is required", k)
        }
    }

    return nil
}

```

--------------------------------

### Get Sloth SLO CRs

Source: https://sloth.dev/usage/kubernetes?q=

Retrieve the Sloth Service Level Objects (SLOs) from the 'monitoring' namespace. This command shows the status and age of deployed SLOs.

```bash
$ kubectl -n monitoring get slos

```

--------------------------------

### Optimize SLI Error Total Window Logic with Plugin

Source: https://sloth.dev/faq

This Go plugin optimizes SLI error total window performance by inferring metrics from user-defined SLIs and creating new rule groups with specific intervals. It's designed for SLI events types and requires Prometheus metrics.

```go
package plugin

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "html/template"
    "time"

    prommodel "github.com/prometheus/common/model"
    "github.com/prometheus/prometheus/model/rulefmt"
    "github.com/prometheus/prometheus/promql/parser"

    "github.com/slok/sloth/pkg/common/conventions"
    "github.com/slok/sloth/pkg/common/model"
    promutils "github.com/slok/sloth/pkg/common/utils/prometheus"
    pluginslov1 "github.com/slok/sloth/pkg/prometheus/plugin/slo/v1"
)

const (
    PluginVersion = "prometheus/slo/v1"
    PluginID      = "github.com/slok/sloth/issues/618"
)

const (
    metricNameBadDefault           = "slo:bad_events:count%s"
    metricNameTotalDefault         = "slo:total_events:count%s"
    sliErrorTotalWindowQueryMetric = `sum_over_time(%[1]s%[4]s[%[3]s]) / ignoring (sloth_window) sum_over_time(%[2]s%[4]s[%[3]s])`
)

type Config struct {
    AggregationWindow prommodel.Duration `json:"aggregation_window,omitempty"`
}

func NewPlugin(configData json.RawMessage, _ pluginslov1.AppUtils) (pluginslov1.Plugin, error) {
    config := Config{}
    err := json.Unmarshal(configData, &config)
    if err != nil {
        return nil, fmt.Errorf("invalid config: %w", err)
    }

    if config.AggregationWindow == 0 {
        config.AggregationWindow = prommodel.Duration(5 * time.Minute)
    }

    return plugin{config: config}, nil
}

type plugin struct {
    config Config
}

func (p plugin) ProcessSLO(ctx context.Context, request *pluginslov1.Request, result *pluginslov1.Result) error {
    if request.SLO.SLI.Events == nil {
        return fmt.Errorf("only supported with SLI events type")
    }

    errorMetricCounter, err := getCounterMetricFromSLIQuery(request.SLO.SLI.Events.ErrorQuery)
    if err != nil {
        return fmt.Errorf("could not get counter metric from error events SLI query: %w", err)
    }

    totalMetricCounter, err := getCounterMetricFromSLIQuery(request.SLO.SLI.Events.TotalQuery)
    if err != nil {
        return fmt.Errorf("could not get counter metric from total events SLI query: %w", err)
    }

    // Add precomputed buckets into a new rule group.
    badRec := rulefmt.Rule{
        Record: fmt.Sprintf(metricNameBadDefault, p.config.AggregationWindow),
        Expr:   fmt.Sprintf("increase(%s[%s])", errorMetricCounter, p.config.AggregationWindow),
        Labels: result.SLORules.SLIErrorRecRules.Rules[0].Labels,
    }
    totalRec := rulefmt.Rule{
        Record: fmt.Sprintf(metricNameTotalDefault, p.config.AggregationWindow),
        Expr:   fmt.Sprintf("increase(%s[%s])", totalMetricCounter, p.config.AggregationWindow),
        Labels: result.SLORules.SLIErrorRecRules.Rules[0].Labels,
    }
    result.SLORules.ExtraRules = append(result.SLORules.ExtraRules, model.PromRuleGroup{
        Name:     "sloth-slo-events-precomputed-" + request.SLO.ID,
        Interval: time.Duration(p.config.AggregationWindow),
        Rules:    []rulefmt.Rule{badRec, totalRec},
    })

    // Replace long window query to use the precomputed ones.
    sliLongWindowRecord := conventions.GetSLIErrorMetric(request.SLO.TimeWindow)
    sloLabels := promutils.LabelsToPromFilter(conventions.GetSLOIDPromLabels(request.SLO))
    for i, r := range result.SLORules.SLIErrorRecRules.Rules {
        if r.Record == sliLongWindowRecord {
            result.SLORules.SLIErrorRecRules.Rules[i].Expr = fmt.Sprintf(sliErrorTotalWindowQueryMetric, badRec.Record, totalRec.Record, promutils.TimeDurationToPromStr(request.SLO.TimeWindow), sloLabels)
            break
        }
    }

    return nil
}

func getCounterMetricFromSLIQuery(query string) (string, error) {
    tmpl, err := template.New("").Parse(query)
    if err != nil {
        return "", fmt.Errorf("could not parse events SLI query: %w", err)
    }
    var b bytes.Buffer
    err = tmpl.Execute(&b, map[string]any{"window": "5m"})
    if err != nil {
        return "", fmt.Errorf("could not execute events SLI query template: %w", err)
    }
    
```

```go
return b.String(), nil
}
```

--------------------------------

### CoreDNS Availability: Default Rcode with Custom Filter

Source: https://sloth.dev/sli-plugins/coredns-availability

Example of configuring the CoreDNS availability plugin with a custom Prometheus filter but using the default rcode error matching.

```yaml
sli:
  plugin:
    id: "sloth-common/coredns/availability"
    options:
      filter: job="kube-dns",server="dns://:53",service="kube-dns", zone="."
```

--------------------------------

### Define a Sloth Kubernetes SLO

Source: https://sloth.dev/examples/kubernetes/getting-started

This example defines a PrometheusServiceLevel CRD for a 'myservice' with an SLO for request availability. It specifies error and total query metrics, and alerting configurations.

```yaml
# This example shows the same example as getting-started.yml but using Sloth Kubernetes CRD.
# It will generate the Prometheus rules in a Kubernetes prometheus-operator PrometheusRules CRD.
#
# `sloth generate -i ./examples/k8s-getting-started.yml`
#
apiVersion: sloth.slok.dev/v1
kind: PrometheusServiceLevel
metadata:
  name: sloth-slo-my-service
  namespace: monitoring
spec:
  service: "myservice"
  labels:
    owner: "myteam"
    repo: "myorg/myservice"
    tier: "2"
  slos:
    - name: "requests-availability"
      objective: 99.9
      description: "Common SLO based on availability for HTTP request responses."
      sli:
        events:
          errorQuery: sum(rate(http_request_duration_seconds_count{job="myservice",code=~"(5..|429)"}[{{.window}}]))
          totalQuery: sum(rate(http_request_duration_seconds_count{job="myservice"}[{{.window}}]))
      alerting:
        name: MyServiceHighErrorRate
        labels:
          category: "availability"
        annotations:
          summary: "High error rate on 'myservice' requests responses"
        pageAlert:
          labels:
            severity: pageteam
            routing_key: myteam
        ticketAlert:
          labels:
            severity: "slack"
            slack_channel: "#alerts-myteam"

```

--------------------------------

### Get Prometheus Rules CRs

Source: https://sloth.dev/usage/kubernetes?q=

Retrieve the PrometheusRule Custom Resources (CRs) generated by Sloth from the 'monitoring' namespace. This verifies that Sloth has created the necessary Prometheus rules.

```bash
$ kubectl -n monitoring get prometheusrules

```

--------------------------------

### CLI Mode for Raw Prometheus Rules

Source: https://sloth.dev/faq

Use the CLI's `generate` mode when you need raw Prometheus rules and do not have Kubernetes.

```bash
sloth generate --output rules.yaml
```

--------------------------------

### CoreDNS Availability: Custom Rcode without Filters

Source: https://sloth.dev/sli-plugins/coredns-availability

Example of using the CoreDNS availability plugin with a custom regex to define error rcodes, without applying any specific filters.

```yaml
sli:
  plugin:
    id: "sloth-common/coredns/availability"
    options:
      custom_rcode_regex: (NXDOMAIN|SERVFAIL|FORMERR)
```

--------------------------------

### Default Usage (Auto-loaded)

Source: https://sloth.dev/slo-plugins/core/sli_rules_v1?q=

This snippet shows the default configuration where the plugin is automatically loaded. No specific configuration is needed.

```yaml
chain:
  - id: "sloth.dev/core/sli_rules/v1"
```

--------------------------------

### Load Custom K8s Transformer Plugin

Source: https://sloth.dev/usage/k8s-transformer-plugins

Use the --plugins-path flag to load custom transformer plugins not built into Sloth. Sloth will automatically discover plugins in the specified paths.

```bash
sloth generate -p ./plugins/ --k8s-transform-plugin-id="mycompany.com/custom-plugin/v1"

```

--------------------------------

### Kooper Availability Plugin Without Filter

Source: https://sloth.dev/sli-plugins/kubernetes-kooper-availability

Example configuration for the Kooper availability plugin when no Prometheus label filter is applied. Ensure the 'controller' option is set to the name of the controller being measured.

```yaml
sli:
  plugin:
    id: "sloth-common/kubernetes/kooper/availability"
    options:
      controller: "sloth"
```

--------------------------------

### SLO Alerting Rule Example

Source: https://sloth.dev/examples/kubernetes/getting-started

Defines an alert that triggers when the SLO error rate exceeds a certain threshold for both 5-minute and 1-hour windows. This helps in proactively identifying potential SLO breaches.

```yaml
alert: MyServiceHighErrorRate
      annotations:
        summary: High error rate on 'myservice' requests responses
        title: (page) {{$labels.sloth_service}} {{$labels.sloth_slo}} SLO error budget
          burn rate is too fast.
      expr: |
        (
            max(slo:sli_error:ratio_rate5m{sloth_id="myservice-requests-availability", sloth_service="myservice", sloth_slo="requests-availability"} > (14.4 * 0.0009999999999999432)) without (sloth_window)
            and
            max(slo:sli_error:ratio_rate1h{sloth_id="myservice-requests-availability", sloth_service="myservice", sloth_slo="requests-availability"} > (14.4 * 0.0009999999999999432)) without (sloth_window)
        )
        or
        (

```

--------------------------------

### Load Custom K8s Transformer Plugin

Source: https://sloth.dev/usage/k8s-transformer-plugins?q=

Use the `--plugins-path` flag to load custom K8s transformer plugins not built into Sloth. Sloth will automatically discover plugins in the specified paths.

```bash
sloth generate -p ./plugins/ --k8s-transform-plugin-id="mycompany.com/custom-plugin/v1"
```
Tokens:
Raw