### Complete Deployment Examples

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/FILE_MANIFEST.txt

Offers 3 complete deployment examples for Grafana alerting, covering Docker, Kubernetes, and Terraform setups.

```dockerfile
# Example Docker deployment (details not provided in manifest)
```

```yaml
# Example Kubernetes deployment (details not provided in manifest)
```

```terraform
# Example Terraform deployment (details not provided in manifest)
```

--------------------------------

### Install Grafana on Minikube

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/DEPLOYMENT_GUIDE.md

Steps to start Minikube, create a namespace, deploy Grafana using a YAML file, and verify the deployment on Minikube.

```bash
# Start minikube
minikube start

# Create namespace
kubectl create namespace my-grafana

# Deploy
cd kubernetes
kubectl apply -f grafana.yaml --namespace=my-grafana

# Verify resources
kubectl get all --namespace=my-grafana
```

--------------------------------

### Start Minikube Cluster

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/kubernetes/README.md

Initiates a local Kubernetes cluster using Minikube. Ensure Minikube is installed and configured.

```bash
minikube start
```

--------------------------------

### Dashboard Examples

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/FILE_MANIFEST.txt

Provides examples for creating Grafana dashboards, including JSON definitions and pattern-based examples.

```json
# Example dashboard JSON (details not provided in manifest)
```

```text
# Example dashboard pattern (details not provided in manifest)
```

--------------------------------

### Run Grafana Instance with Docker Compose

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/config-files/README.md

Starts a Grafana instance and a local SMTP container for email testing. Ensure Docker Compose is installed.

```bash
docker compose up -d
```

--------------------------------

### Start Grafana Services with Docker Compose

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/DEPLOYMENT_GUIDE.md

Use this command to start Grafana and other services defined in your docker-compose.yaml file. Ensure you are in the 'config-files' directory.

```bash
cd config-files

# Start services
docker compose up -d

# Verify services running
docker compose ps
```

--------------------------------

### Example Deployment Architecture

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/PROJECT_OVERVIEW.md

Illustrates the dependency graph for Grafana alerting resources, from datasources to mute timings.

```text
Datasource (TestData)
    ↓
Dashboard Panel (references datasource)
    ↓
Alert Rule Group (queries datasource, links to dashboard)
    ↓
Contact Point (receives notifications)
    ↓
Notification Policy Tree (routes alerts to contact point)
    ↓
Message Templates (customize notification content)
    ↓
Mute Timing (silence alerts on weekends)
```

--------------------------------

### Copy Terraform Variables Example

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/terraform/README.md

Create a custom variables file by copying the example. This file will store your specific configurations, such as the notification email address.

```bash
cp terraform.tfvars.example terraform.tfvars
```

--------------------------------

### Example Test Data Datasource Configuration

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/api-reference/YAML_CONFIG_FILES.md

An example of how to configure a test data datasource in Grafana.

```yaml
datasources:
  - name: testdata
    type: testdata
    access: proxy
    uid: testdata_uid
    orgId: 1
    isDefault: true
```

--------------------------------

### Example File Provider for Dashboard Provisioning

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/api-reference/YAML_CONFIG_FILES.md

Configures Grafana to load dashboards from a specified directory using the file provider.

```yaml
apiVersion: 1
providers:
  - name: 'Default'
    type: file
    folder: 'TestFolder'
    disableDeletion: false
    updateIntervalSeconds: 10
    editable: true
    options:
      path: /etc/grafana/provisioning/dashboards/definitions
```

--------------------------------

### Terraform Project Structure Example

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/api-reference/TERRAFORM_RESOURCES.md

Organize your Terraform configuration files by resource type for better maintainability and clarity.

```hcl
terraform-infra/
├── main.tf                 # Provider configuration
├── variables.tf            # Input variables
├── datasources.tf          # grafana_data_source resources
├── folders.tf              # grafana_folder resources
├── dashboards.tf           # grafana_dashboard resources
├── alert_rules.tf          # grafana_rule_group resources
├── alert_resources.tf      # Contact points, templates, policies, mute timings
└── dashboards/definitions/ # JSON dashboard files
```

--------------------------------

### Complete Alert Rule Example

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/api-reference/YAML_CONFIG_FILES.md

A comprehensive example of an alert rule definition in YAML, including datasource queries, reduce and threshold expressions, and various configuration options like interval, labels, and states.

```yaml
groups:
  - orgId: 1
    name: group_1m
    folder: TestFolder
    interval: 1m
    rules:
      - uid: random_walk_threshold_uid
        title: random_walk_threshold
        condition: C
        data:
          - refId: A
            relativeTimeRange: { from: 300, to: 0 }
            datasourceUid: testdata_uid
            model:
              datasource: { type: grafana-testdata-datasource, uid: testdata_uid }
              scenarioId: random_walk
              refId: A
          - refId: B
            datasourceUid: __expr__
            model:
              type: reduce
              expression: A
              reducer: mean
              refId: B
          - refId: C
            datasourceUid: __expr__
            model:
              type: threshold
              expression: B
              conditions:
                - evaluator: { type: gt, params: [0] }
                  operator: { type: and }
                  query: { params: [C] }
                  reducer: { type: last }
                  type: query
              refId: C
        noDataState: NoData
        execErrState: Error
        for: 0s
        labels: { monitor: testdata }
        isPaused: false
```

--------------------------------

### Copy Terraform Variables Template

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/DEPLOYMENT_GUIDE.md

Create a local variables file by copying the example template.

```bash
cp terraform.tfvars.example terraform.tfvars
```

--------------------------------

### Duration Format Example

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/SCHEMA_DEFINITIONS.md

Specifies the format for duration strings, using Go's duration format.

```plaintext
String in Go duration format: [0-9]+[smhd]

Examples:
- 30s — 30 seconds
- 5m — 5 minutes
- 1h — 1 hour
- 7d — 7 days
```

--------------------------------

### Example Email Contact Point Configuration

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/api-reference/YAML_CONFIG_FILES.md

Defines an email contact point with custom subject and message templates. Ensure 'your_email@company.org' is replaced with a valid recipient address.

```yaml
contactPoints:
  - orgId: 1
    name: My Contact Email Point
    receivers:
      - uid: my_contact_email_point_uid
        type: email
        settings:
          addresses: [your_email@company.org]
          subject: '{{ template "custom_email.subject" . }}'
          message: '{{ template "custom_email.message" . }}'
          singleEmail: false
        disableResolveMessage: false
```

--------------------------------

### NotificationPolicy Example with Routing

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/api-reference/YAML_CONFIG_FILES.md

Example of a root notification policy that routes alerts based on labels to specific contact points and applies mute timings.

```yaml
policies:
  - orgId: 1
    receiver: My Contact Email Point
    group_by: [grafana_folder, alertname]
    routes:
      - receiver: My Contact Email Point
        object_matchers:
          - [monitor, =, testdata]
        mute_time_intervals: [no_weekends]
```

--------------------------------

### MuteTime Example: Silence Alerts on Weekends

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/api-reference/YAML_CONFIG_FILES.md

Example of a mute time configuration to silence alerts specifically on Saturdays and Sundays.

```yaml
muteTimes:
  - orgId: 1
    name: no_weekends
    time_intervals:
      - weekdays: [saturday, sunday]
```

--------------------------------

### TimeRange Object Example

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/SCHEMA_DEFINITIONS.md

Example of a TimeRange object, specifying time in seconds relative to the current time.

```json
{
  "from": 300,
  "to": 0
}
```

--------------------------------

### Install Grafana on Cloud Kubernetes

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/DEPLOYMENT_GUIDE.md

Instructions for deploying Grafana on cloud Kubernetes services like AWS EKS, GKE, or AKS, including namespace creation and obtaining the external IP.

```bash
# Create namespace
kubectl create namespace my-grafana

# Deploy
kubectl apply -f grafana.yaml --namespace=my-grafana

# Get LoadBalancer external IP
kubectl get service grafana --namespace=my-grafana
# External IP appears under EXTERNAL-IP column
# Access at: http://<EXTERNAL_IP>:3000
```

--------------------------------

### Create Contact Point

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/SCHEMA_DEFINITIONS.md

Example of how to create a new contact point for alerts using the Grafana API.

```APIDOC
## POST /api/v1/provisioning/contact-points

### Description
Creates a new contact point for receiving alerts.

### Method
POST

### Endpoint
/api/v1/provisioning/contact-points

### Request Body
- **name** (string) - Required - The name of the contact point.
- **type** (string) - Required - The type of the contact point (e.g., "email").
- **settings** (object) - Required - Configuration settings specific to the contact point type.
  - **addresses** (array of strings) - Required - Email addresses to send alerts to.
  - **subject** (string) - Required - The subject line for alert emails.

### Request Example
```json
{
  "name": "my-email",
  "type": "email",
  "settings": {
    "addresses": ["alerts@example.com"],
    "subject": "Alert: {{ .GroupLabels.alertname }}"
  }
}
```

### Response
#### Success Response (200)
(No specific response schema provided in source)

#### Response Example
(No example provided in source)
```

--------------------------------

### Contact Point Patterns

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/FILE_MANIFEST.txt

Demonstrates 4 patterns for configuring contact points in Grafana, with examples in both Terraform and YAML.

```terraform
# Example contact point pattern 1 (Terraform - details not provided in manifest)
```

```yaml
# Example contact point pattern 1 (YAML - details not provided in manifest)
```

```terraform
# Example contact point pattern 2 (Terraform - details not provided in manifest)
```

```yaml
# Example contact point pattern 2 (YAML - details not provided in manifest)
```

```terraform
# Example contact point pattern 3 (Terraform - details not provided in manifest)
```

```yaml
# Example contact point pattern 3 (YAML - details not provided in manifest)
```

```terraform
# Example contact point pattern 4 (Terraform - details not provided in manifest)
```

```yaml
# Example contact point pattern 4 (YAML - details not provided in manifest)
```

--------------------------------

### Load Dashboard from File with Variable Substitution

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/api-reference/TERRAFORM_RESOURCES.md

This example demonstrates how to load a dashboard definition from a JSON file and substitute variables, such as datasource UIDs, using Terraform's `templatefile` function. It then creates the dashboard within a specified Grafana folder.

```hcl
locals {
  custom_dashboard = templatefile("../dashboards/definitions/random-walk-dashboard.json", {
    datasource_uid = grafana_data_source.testdata_datasource.uid
  })
}

resource "grafana_dashboard" "custom_dashboard" {
  folder      = grafana_folder.test_folder.id
  config_json = local.custom_dashboard
}
```

--------------------------------

### Docker Compose Management Commands

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/CONFIGURATION_REFERENCE.md

Common commands for managing Docker Compose services, including starting, stopping, viewing logs, and removing services and volumes.

```bash
# Start services in background
docker compose up -d

# View running services
docker compose ps

# View logs
docker compose logs grafana
docker compose logs smtp

# Stop services
docker compose down

# Remove volumes (careful—deletes data!)
docker compose down -v
```

--------------------------------

### Example Custom Email Message Template

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/api-reference/YAML_CONFIG_FILES.md

Defines a custom email message template using Go templating. This template provides a basic custom message body.

```yaml
templates:
  - orgId: 1
    name: custom_email.message
    template: |
      {{ define "custom_email.message" }}
        Lorem ipsum - Custom alert!
      {{ end }}
```

--------------------------------

### Notification Policy Patterns

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/FILE_MANIFEST.txt

Showcases 4 patterns for setting up notification policies in Grafana. These examples illustrate how to route alerts to different contact points.

```yaml
# Example notification policy pattern 1 (details not provided in manifest)
```

```yaml
# Example notification policy pattern 2 (details not provided in manifest)
```

```yaml
# Example notification policy pattern 3 (details not provided in manifest)
```

```yaml
# Example notification policy pattern 4 (details not provided in manifest)
```

--------------------------------

### Threshold Alert Rule (YAML)

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/EXAMPLES_AND_PATTERNS.md

Configure a threshold alert to fire when a metric exceeds a specified value. This example checks for high memory usage.

```yaml
groups:
  - orgId: 1
    name: monitoring
    folder: Alerts
    interval: 1m
    rules:
      - uid: memory_threshold
        title: High Memory Usage
        condition: C
        data:
          # Query A: Fetch memory metric from Prometheus
          - refId: A
            relativeTimeRange: { from: 300, to: 0 }
            datasourceUid: prometheus_uid
            model: |
              {
                "expr": "node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100",
                "refId": "A"
              }
          # Query B: Reduce to scalar (take last value)
          - refId: B
            datasourceUid: __expr__
            model: |
              {
                "type": "reduce",
                "expression": "A",
                "reducer": "last",
                "refId": "B"
              }
          # Query C: Check if > 80
          - refId: C
            datasourceUid: __expr__
            model: |
              {
                "type": "threshold",
                "expression": "B",
                "conditions": [{
                  "evaluator": { "type": "gt", "params": [80] },
                  "operator": { "type": "and" },
                  "query": { "params": ["C"] },
                  "reducer": { "type": "last" },
                  "type": "query"
                }],
                "refId": "C"
              }
        noDataState: NoData
        execErrState: Error
        for: 5m
        annotations:
          description: Memory usage exceeds 80% on {{ $labels.instance }}
          runbook_url: https://wiki.internal/runbooks/memory
        labels:
          severity: warning
          team: platform
        isPaused: false

```

--------------------------------

### Example Custom Email Subject Template

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/api-reference/YAML_CONFIG_FILES.md

Defines a custom email subject template using Go templating. This template displays the count of firing and resolved alerts.

```yaml
templates:
  - orgId: 1
    name: custom_email.subject
    template: |
      {{ define "custom_email.subject" }}
      {{ len .Alerts.Firing }} firing alert(s), {{ len .Alerts.Resolved }} resolved alert(s)
      {{ end }}
```

--------------------------------

### Verify Datasource UID in Grafana Provisioning Files

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/DEPLOYMENT_GUIDE.md

Use this command to recursively search for datasource UIDs within Grafana provisioning configuration files. Ensure consistency across your setup.

```bash
grep -r "uid:" config-files/grafana/provisioning/
```

--------------------------------

### Multi-Channel Routing for Critical and Non-Critical Alerts

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/EXAMPLES_AND_PATTERNS.md

Implement routing logic to send alerts to different contact points based on severity or other conditions. This example routes critical alerts to PagerDuty and non-critical alerts to email.

```hcl
# Critical alerts → PagerDuty
resource "grafana_contact_point" "pagerduty" {
  name = "PagerDuty Critical"
  
  pagerduty {
    integration_key = var.pagerduty_key
    severity        = "{{ .Alerts.Firing | len | gt 0 | ternary \"critical\" \"warning\" }}"
  }
}

# Non-critical → Email
resource "grafana_contact_point" "email" {
  name = "Email Non-Critical"
  
  email {
    addresses = ["ops@company.org"]
  }
}
```

--------------------------------

### Initialize Terraform Working Directory

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/terraform/README.md

Run this command in your Terraform configuration directory to download the necessary provider plugins and set up the backend.

```bash
terraform init
```

--------------------------------

### Terraform Example: Alert Rule Group with Data Query and Threshold Expression

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/api-reference/TERRAFORM_RESOURCES.md

This example demonstrates how to create a Grafana alert rule group using Terraform. It includes a data query to fetch time-series data, a reduction query to get a scalar value, and a threshold query to define the alert condition. The `jsonencode()` function is used to serialize the query models into JSON strings as required by the `model` argument.

```hcl
resource "grafana_rule_group" "alert_rule_group1" {
  name              = "group_1m"
  disable_provenance = true
  folder_uid        = grafana_folder.test_folder.uid
  interval_seconds  = 60

  rule {
    name      = "random_walk_threshold"
    condition = "C"

    # Query A: Fetch data from test datasource
    data {
      ref_id = "A"
      relative_time_range {
        from = 300
        to   = 0
      }
      datasource_uid = grafana_data_source.testdata_datasource.uid
      model = jsonencode({
        datasource = {
          type = "grafana-testdata-datasource"
          uid  = grafana_data_source.testdata_datasource.uid
        }
        refId      = "A"
        scenarioId = "random_walk"
      })
    }

    # Query B: Reduce time series to scalar (mean)
    data {
      ref_id         = "B"
      datasource_uid = "__expr__"
      model = jsonencode({
        type       = "reduce"
        expression = "A"
        reducer    = "mean"
        refId      = "B"
      })
    }

    # Query C: Apply threshold condition
    data {
      ref_id         = "C"
      datasource_uid = "__expr__"
      model = jsonencode({
        type       = "threshold"
        expression = "B"
        refId      = "C"
        conditions = [
          {
            evaluator = { type = "gt", params = [0] }
            operator  = { type = "and" }
            query     = { params = ["C"] }
            reducer   = { type = "last" }
            type      = "query"
          }
        ]
      })
    }

    no_data_state  = "NoData"
    exec_err_state = "Error"
    annotations = {
      __dashboardUid__ = grafana_dashboard.custom_dashboard.uid
      __panelId__      = "1"
    }
    labels = {
      monitor = "testdata"
    }
    is_paused = false
  }
}

```

--------------------------------

### Create Provisioning Directory Structure

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/api-reference/YAML_CONFIG_FILES.md

Use this command to create the necessary directory structure for Grafana provisioning.

```bash
mkdir -p grafana/provisioning/{alerting,dashboards/definitions,datasources}
```

--------------------------------

### Apply Terraform Configuration

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/DEPLOYMENT_GUIDE.md

Create the defined resources in Grafana. Confirm the action by typing 'yes' when prompted.

```bash
terraform apply
```

--------------------------------

### Verify Provisioning Files in Pod

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/kubernetes/README.md

Executes a command inside the Grafana pod to list the contents of the '/etc/grafana/provisioning' directory, confirming that the configuration files have been copied successfully.

```bash
kubectl exec -n my-grafana <pod_name> -- ls /etc/grafana/provisioning
```

--------------------------------

### Timestamp Format Example

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/SCHEMA_DEFINITIONS.md

Defines the required format for timestamps, adhering to RFC 3339.

```plaintext
RFC 3339 format: 2024-01-15T10:30:00Z
```

--------------------------------

### Navigate to Terraform Configuration Directory

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/terraform/README.md

Change to the directory containing the Terraform configuration files before initializing or applying.

```bash
cd terraform-infra
```

--------------------------------

### Deploy Grafana Resources

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/kubernetes/README.md

Applies the Kubernetes manifest file 'grafana.yaml' to deploy Grafana within the 'my-grafana' namespace.

```bash
kubectl apply -f grafana.yaml --namespace=my-grafana
```

--------------------------------

### Navigate to Terraform Directory

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/DEPLOYMENT_GUIDE.md

Change the current directory to the Terraform configuration files.

```bash
cd terraform/terraform-infra
```

--------------------------------

### Create Alert Rule Group

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/SCHEMA_DEFINITIONS.md

Example of how to create a new alert rule group using the Grafana Ruler API.

```APIDOC
## POST /api/ruler/grafana/rules/{group_name}

### Description
Creates a new alert rule group with specified rules.

### Method
POST

### Endpoint
/api/ruler/grafana/rules/{group_name}

### Parameters
#### Path Parameters
- **group_name** (string) - Required - The name of the alert rule group.

### Request Body
- **name** (string) - Required - The name of the rule group.
- **interval** (string) - Required - The evaluation interval for the rules (e.g., "60s").
- **rules** (array of objects) - Required - A list of alert rules.
  - **uid** (string) - Required - Unique identifier for the rule.
  - **title** (string) - Required - The title of the alert rule.
  - **condition** (string) - Required - The condition that triggers the alert.
  - **data** (array of objects) - Required - Data source queries for the rule.
    - **refId** (string) - Required - Reference ID for the query.
    - **datasourceUid** (string) - Required - UID of the datasource.
    - **model** (object) - Required - Query model details.

### Request Example
```json
{
  "name": "my-rules",
  "interval": "60s",
  "rules": [
    {
      "uid": "rule1",
      "title": "CPU Alert",
      "condition": "C",
      "data": [
        {
          "refId": "A",
          "datasourceUid": "prometheus-uid",
          "model": {...}
        }
      ]
    }
  ]
}
```

### Response
#### Success Response (200)
(No specific response schema provided in source)

#### Response Example
(No example provided in source)
```

--------------------------------

### Grafana Alert APIs

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/SUMMARY.txt

Reference for Grafana Alerting APIs, including GET, POST, and DELETE operations for managing alerts.

```APIDOC
## Grafana Alert APIs

### Description
Provides access to Grafana's alerting functionalities, allowing for the retrieval, creation, and deletion of alert-related resources.

### Method
GET, POST, DELETE

### Endpoint
/api/alerting/*

### Parameters
#### Path Parameters
None explicitly defined in source.

#### Query Parameters
None explicitly defined in source.

#### Request Body
None explicitly defined for GET. Specific fields for POST/DELETE are not detailed in the source.

### Request Example
```json
{
  "example": "POST /api/alerting/rules - Example request body not provided in source."
}
```

### Response
#### Success Response (200)
- **field1** (type) - Description
  Response structure for alert APIs is not detailed in the source.

#### Response Example
```json
{
  "example": "200 OK - Example response body not provided in source."
}
```
```

--------------------------------

### Threshold Alert Rule (Terraform)

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/EXAMPLES_AND_PATTERNS.md

Configure a threshold alert using Terraform. This example mirrors the YAML configuration for high memory usage.

```hcl
resource "grafana_rule_group" "memory_alerts" {
  name             = "monitoring"
  folder_uid       = grafana_folder.alerts.uid
  interval_seconds = 60

  rule {
    name      = "High Memory Usage"
    condition = "C"

    data {
      ref_id = "A"
      relative_time_range {
        from = 300
        to   = 0
      }
      datasource_uid = grafana_data_source.prometheus.uid
      model = jsonencode({
        expr  = "node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100"
        refId = "A"
      })
    }

    data {
      ref_id         = "B"
      datasource_uid = "__expr__"
      model = jsonencode({
        type       = "reduce"
        expression = "A"
        reducer    = "last"
        refId      = "B"
      })
    }

    data {
      ref_id         = "C"
      datasource_uid = "__expr__"
      model = jsonencode({
        type       = "threshold"
        expression = "B"
        conditions = [{
          evaluator = { type = "gt", params = [80] }
          operator  = { type = "and" }
          query     = { params = ["C"] }
          reducer   = { type = "last" }
          type      = "query"
        }]
        refId = "C"
      })
    }

    for             = "5m"
    no_data_state   = "NoData"
    exec_err_state  = "Error"
    
    annotations = {
      description = "Memory usage exceeds 80% on {{ $labels.instance }}"
      runbook_url = "https://wiki.internal/runbooks/memory"
    }

    labels = {
      severity = "warning"
      team     = "platform"
    }

    is_paused = false
  }
}

```

--------------------------------

### Backup Terraform State File

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/DEPLOYMENT_GUIDE.md

Create a backup of the local Terraform state file.

```bash
cp terraform.tfstate terraform.tfstate.backup
```

--------------------------------

### Copy Provisioning Files to Grafana Pod

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/kubernetes/README.md

Copies alerting, dashboards, and datasources configuration directories from the local filesystem into the Grafana pod's provisioning directory.

```bash
kubectl cp ../config-files/grafana/provisioning/alerting my-grafana/<pod_name>:/etc/grafana/provisioning/

kubectl cp ../config-files/grafana/provisioning/dashboards my-grafana/<pod_name>:/etc/grafana/provisioning/

kubectl cp ../config-files/grafana/provisioning/datasources my-grafana/<pod_name>:/etc/grafana/provisioning/
```

--------------------------------

### Grafana Kubernetes StatefulSet Configuration

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/DEPLOYMENT_GUIDE.md

Example Kubernetes StatefulSet definition for a multi-node Grafana deployment. This ensures stable network identities and ordered deployment.

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: grafana
spec:
  serviceName: grafana
  replicas: 3
  # ... pod spec ...
```

--------------------------------

### View Available Storage Classes in Kubernetes

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/CONFIGURATION_REFERENCE.md

Command to list available StorageClasses in a Kubernetes cluster, showing the different storage backends that can be used for PersistentVolumeClaims.

```bash
kubectl get storageclass
# Shows available storage backends (EBS, GCE, NFS, etc.)
```

--------------------------------

### Alert Rule Patterns

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/FILE_MANIFEST.txt

Provides 5 distinct patterns for defining alert rules in Grafana. These examples cover various alerting scenarios and configurations.

```yaml
# Example alert rule pattern 1 (details not provided in manifest)
```

```yaml
# Example alert rule pattern 2 (details not provided in manifest)
```

```yaml
# Example alert rule pattern 3 (details not provided in manifest)
```

```yaml
# Example alert rule pattern 4 (details not provided in manifest)
```

```yaml
# Example alert rule pattern 5 (details not provided in manifest)
```

--------------------------------

### Configure Grafana Provider with Environment Variables

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/CONFIGURATION_REFERENCE.md

Configures the Grafana provider using environment variables for URL and authentication. Recommended for production environments.

```bash
export GRAFANA_URL="https://grafana.example.com"
export GRAFANA_AUTH="username:password"
terraform apply
```

--------------------------------

### Copy Provisioning Configuration to Grafana Pod

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/DEPLOYMENT_GUIDE.md

Copies Grafana provisioning files (alerting, dashboards, datasources) to the running Grafana pod in the specified namespace.

```bash
# Get pod name
POD_NAME=$(kubectl get pods --namespace=my-grafana \
  -l app=grafana -o jsonpath='{.items[0].metadata.name}')

# Copy alerting configuration
kubectl cp ../config-files/grafana/provisioning/alerting \
  my-grafana/$POD_NAME:/etc/grafana/provisioning/ \
  --namespace=my-grafana

# Copy dashboards
kubectl cp ../config-files/grafana/provisioning/dashboards \
  my-grafana/$POD_NAME:/etc/grafana/provisioning/ \
  --namespace=my-grafana

# Copy datasources
kubectl cp ../config-files/grafana/provisioning/datasources \
  my-grafana/$POD_NAME:/etc/grafana/provisioning/ \
  --namespace=my-grafana

# Verify copy
kubectl exec -n my-grafana $POD_NAME -- ls /etc/grafana/provisioning
```

--------------------------------

### Configure Remote Terraform State Backend

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/DEPLOYMENT_GUIDE.md

Set up a remote backend (e.g., S3) for storing Terraform state, which is recommended for team collaboration and state safety.

```bash
cat > backend.tf <<EOF
terraform {
  backend "s3" {
    bucket = "my-terraform-state"
    key    = "grafana/terraform.tfstate"
    region = "us-east-1"
  }
}
EOF
```

--------------------------------

### Grafana Provisioning Directory Layout

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/CONFIGURATION_REFERENCE.md

Illustrates the standard directory structure for Grafana provisioning files, including alerting, dashboards, and datasources.

```tree
grafana/provisioning/
├── alerting/
│   ├── alert_resources.yaml      # Contact points, policies, templates, mute timings
│   └── alert_rules.yaml          # Alert rule definitions
├── dashboards/
│   ├── dashboard.yaml            # Dashboard provider configuration
│   └── definitions/
│       └── *.json                # Dashboard JSON files
└── datasources/
    └── datasource.yaml           # Data source definitions
```

--------------------------------

### Enable Grafana Port-Forwarding

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/kubernetes/README.md

Establishes a port-forwarding connection to access the Grafana service on port 3000 from your local machine.

```bash
kubectl port-forward service/grafana 3000:3000 --namespace=my-grafana
```

--------------------------------

### Hierarchical Notification Routing

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/EXAMPLES_AND_PATTERNS.md

Implement multi-level notification routing, starting from team, then service, and finally severity. This allows for granular control over alert delivery in complex environments.

```hcl
resource "grafana_notification_policy" "hierarchical" {
  contact_point = grafana_contact_point.default.name
  group_by      = ["team"]

  # Team level
  policy {
    contact_point = grafana_contact_point.team_default.name
    matcher {
      label = "team"
      match = "="
      value = "infrastructure"
    }
    group_by = ["service"]

    # Service level (nested)
    policy {
      contact_point = grafana_contact_point.database_team.name
      matcher {
        label = "service"
        match = "="
        value = "database"
      }
      group_by = ["severity"]

      # Severity level (nested)
      policy {
        contact_point = grafana_contact_point.pagerduty.name
        matcher {
          label = "severity"
          match = "="
          value = "critical"
        }
        group_wait = "0s"  # Immediate notification
      }
    }
  }
}
```

--------------------------------

### Kubernetes Namespace Creation and Resource Application

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/CONFIGURATION_REFERENCE.md

Commands to create a Kubernetes namespace and apply Grafana resources to it. This provides resource isolation and simplifies cleanup.

```bash
kubectl create namespace my-grafana
kubectl apply -f grafana.yaml --namespace=my-grafana
```

--------------------------------

### Verify Grafana Services Status

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/DEPLOYMENT_GUIDE.md

This command displays the status of running services managed by Docker Compose. It helps confirm that Grafana and any other related services have started successfully.

```bash
docker compose ps
```

--------------------------------

### SMTP Service Definition for Docker Compose

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/CONFIGURATION_REFERENCE.md

Defines a lightweight SMTP server for local email testing within a Docker Compose setup. The port is mapped to localhost only.

```yaml
smtp:
  image: ixdotai/smtp:v0.5.2@sha256:...
  ports:
    - 127.0.0.1:25:25
```

--------------------------------

### Create Contact Point (API)

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/SCHEMA_DEFINITIONS.md

Use this snippet to create a new email contact point for alerts. Ensure you have a valid API token and the correct endpoint.

```bash
curl -X POST http://localhost:3000/api/v1/provisioning/contact-points \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-email",
    "type": "email",
    "settings": {
      "addresses": ["alerts@example.com"],
      "subject": "Alert: {{ .GroupLabels.alertname }}"
    }
  }'
```

--------------------------------

### Query Look-back Window Optimization

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/EXAMPLES_AND_PATTERNS.md

Optimize query performance by using a shorter look-back window. For example, use '[5m]' for rate calculations instead of a long '[24h]' window.

```text
# Bad: Long look-back window for simple condition
expr: node_cpu{job="app"}[24h]

# Good: Shorter look-back window
expr: rate(node_cpu{job="app"}[5m])
```

--------------------------------

### Check Grafana Provisioning Logs

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/CONFIGURATION_REFERENCE.md

Retrieve container logs to monitor the Grafana provisioning process, filtering for relevant keywords like 'provisioning', 'alert', or 'datasource'.

```bash
docker logs <container_id> | grep -i "provisioning\|alert\|datasource"
```

--------------------------------

### Comparison Alert Expression

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/EXAMPLES_AND_PATTERNS.md

Define a comparison alert rule using an expression query. This example detects traffic spikes by comparing current traffic to yesterday's average.

```json
{
  "type": "math",
  "expression": "A / B > 2",
  "refId": "C"
}

```

--------------------------------

### Create Grafana Namespace

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/kubernetes/README.md

Creates a dedicated Kubernetes namespace named 'my-grafana' for deploying Grafana resources.

```bash
kubectl create namespace my-grafana
```

--------------------------------

### Show Terraform Resource Details

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/DEPLOYMENT_GUIDE.md

Display detailed information about a specific Terraform-managed resource.

```bash
terraform state show grafana_data_source.testdata_datasource
```

--------------------------------

### View Alert Rules via API

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/EXAMPLES_AND_PATTERNS.md

Use this curl command to view all configured alert rules from the Grafana API. Ensure jq is installed for pretty-printing the JSON output.

```bash
# View alert rules
curl -s http://localhost:3000/api/ruler/grafana/rules | jq .
```

--------------------------------

### Mount Provisioning Directory in Docker

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/api-reference/YAML_CONFIG_FILES.md

Mount the local provisioning directory into the Grafana Docker container to apply configurations.

```bash
docker run -v $(pwd)/grafana/provisioning:/etc/grafana/provisioning grafana/grafana
```

--------------------------------

### Plan Terraform Deployment

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/DEPLOYMENT_GUIDE.md

Review the planned infrastructure changes before applying them. This command shows what resources Terraform will create, modify, or destroy.

```bash
terraform plan
```

--------------------------------

### Time-Based Silencing for Business Hours

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/EXAMPLES_AND_PATTERNS.md

Define mute timings for specific intervals, such as business hours, to control when notifications are sent. This example silences 'info' severity alerts outside of business hours.

```hcl
resource "grafana_mute_timing" "business_hours" {
  name = "business_hours"
  
  intervals {
    times {
      start = "09:00"
      end   = "17:00"
    }
    weekdays = ["monday", "tuesday", "wednesday", "thursday", "friday"]
  }
}

resource "grafana_notification_policy" "working_hours_only" {
  contact_point = grafana_contact_point.email.name
  group_by      = ["alertname"]

  policy {
    contact_point = grafana_contact_point.email.name
    mute_timings  = [grafana_mute_timing.business_hours.name]
    matcher {
      label = "severity"
      match = "="
      value = "info"
    }
  }
}
```

--------------------------------

### Check Grafana Provisioning Logs

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/DEPLOYMENT_GUIDE.md

Monitor Grafana's logs to confirm that datasources, dashboards, and alert rules are being provisioned. This is useful for debugging provisioning issues.

```bash
# Check logs for provisioning messages
docker compose logs grafana | grep -i provisioning
```

--------------------------------

### Create Kubernetes Namespace

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/api-reference/KUBERNETES_MANIFESTS.md

Creates a new namespace for Grafana deployment. Use this before applying any Grafana-related resources.

```bash
kubectl create namespace my-grafana
```

--------------------------------

### Webhook with Custom JSON Payload

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/EXAMPLES_AND_PATTERNS.md

Send alerts to a generic webhook endpoint with a custom JSON payload. This pattern is useful for integrating with external systems that consume structured data. The example shows the expected JSON structure received by the webhook.

```hcl
resource "grafana_contact_point" "webhook_generic" {
  name = "Generic Webhook"
  
  webhook {
    url = "https://api.internal/webhooks/grafana"
    
    # Optional headers
    # headers = { "Authorization" = "Bearer ${var.webhook_token}" }
  }
}
```

```json
{
  "alerts": [
    {
      "status": "firing",
      "labels": { "alertname": "HighCPU" },
      "annotations": { "description": "CPU exceeds 80%" },
      "startsAt": "2024-01-15T10:30:00Z"
    }
  ],
  "groupLabels": { "alertname": "HighCPU" },
  "commonLabels": { "team": "platform" },
  "commonAnnotations": { "runbook_url": "https://wiki.internal/..." }
}
```

--------------------------------

### Apply Terraform Configuration

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/terraform/README.md

Execute this command to create or update the Grafana resources defined in your Terraform configuration files. Review the plan before applying.

```bash
terraform apply
```

--------------------------------

### Create Alert Rule Group (API)

Source: https://github.com/grafana/provisioning-alerting-examples/blob/main/_autodocs/SCHEMA_DEFINITIONS.md

This snippet demonstrates how to create a new alert rule group. It includes a sample rule with a condition and data source configuration.

```bash
curl -X POST http://localhost:3000/api/ruler/grafana/rules/group_1 \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-rules",
    "interval": "60s",
    "rules": [
      {
        "uid": "rule1",
        "title": "CPU Alert",
        "condition": "C",
        "data": [
          {
            "refId": "A",
            "datasourceUid": "prometheus-uid",
            "model": {...}
          }
        ]
      }
    ]'
  }'
```