### Start Demo with Docker Compose

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/README.md

Run the demo environment using Docker Compose. Ensure Docker is installed. This command starts Prometheus, Grafana, node exporter, and an OTEL demo instance.

```bash
cd demo
make start
```

--------------------------------

### Copy Rules to Prometheus Directory

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/deployment.md

Copy the adaptive and robust strategy rule files to the Prometheus rules directory. Ensure example files are also copied if needed. Create the examples subdirectory if it doesn't exist.

```bash
cp adaptive.yml /etc/prometheus/rules/
cp robust.yml /etc/prometheus/rules/
mkdir -p /etc/prometheus/rules/examples
cp examples/*.yml /etc/prometheus/rules/examples/
```

--------------------------------

### Start Docker Compose Services

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/deployment.md

Command to start the services defined in the Docker Compose file in detached mode.

```bash
docker-compose up -d
```

--------------------------------

### PromQL Label Matching Examples

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/promql-reference.md

Demonstrates exact, regex, and negative label matching for filtering time series. Useful for selecting metrics based on specific attributes.

```promql
# Exact match
{anomaly_name="request_rate", anomaly_strategy="adaptive"}
```

```promql
# Regex match
{job=~"node_exporter|custom_exporter"}
```

```promql
# Negative match
{anomaly_select=""}  # Only selects series where label is absent or empty
{anomaly_name!=""}   # Selects series where label is present and not empty
```

--------------------------------

### Alert Instance Grouping Example

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/alerting.md

Illustrates how unique label combinations create separate alert instances. Each instance has its own evaluation timer.

```promql
anomaly_name="request_rate", instance="server1" → separate alert
anomaly_name="request_rate", instance="server2" → separate alert
anomaly_name="cpu_usage", instance="server1"   → separate alert
```

--------------------------------

### Example Anomaly Strategy Tags (YAML)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/README.md

Examples of choosing suitable anomaly detection strategies based on metric characteristics. 'adaptive' is for stable metrics, 'robust' for spiky or non-normal distributions.

```yaml
# Stable, normally distributed metrics
anomaly_strategy: "adaptive"
```

```yaml
# Spiky metrics or non-normal distributions
anomaly_strategy: "robust"
```

--------------------------------

### Faster Alerting Configuration (YAML)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/README.md

This example shows how to configure the `AnomalyDetected` alert to fire faster, in this case, after only 2 minutes instead of the default 5 minutes.

```yaml
alert: AnomalyDetected
for: 2m
expr: ...

```

--------------------------------

### File Structure of PromQL Anomaly Detection Framework

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/overview.md

Illustrates the directory layout for rules, examples, and demo components of the framework.

```plaintext
rules/
├── adaptive.yml          # Adaptive strategy recording and alerting rules
├── robust.yml            # Robust strategy recording and alerting rules
├── examples/
│   ├── node_exporter.yml # CPU and memory resource metrics examples
│   └── otel_demo.yml     # OpenTelemetry span metrics examples
└── README.md             # Rules documentation

demo/
├── docker-compose.yml    # Demo environment specification
├── Makefile              # Build and start targets
└── src/
    ├── prometheus/       # Prometheus configuration
    ├── grafana/          # Grafana dashboards and datasources
    ├── otelcollector/    # OpenTelemetry Collector configuration
    └── flagd/            # Feature flag configuration
```

--------------------------------

### Example Anomaly Name Tag (YAML)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/README.md

Example of setting a descriptive and stable anomaly name tag. Consistent naming is crucial for managing and understanding anomaly detection results.

```yaml
anomaly_name: "http_request_latency_p95"  # Good
anomaly_name: "http_req_lat_p95"          # Unclear abbreviations
```

--------------------------------

### Test Shorter Offset Queries

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

When testing, start with shorter offsets like 1 hour instead of 1 week to quickly verify functionality before relying on longer historical data.

```promql
avg_over_time(metric[1h] offset 1h)
```

--------------------------------

### Calculate Percentage Utilization and Baseline

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/promql-reference.md

Perform percentage calculations by dividing a value by the total and multiplying by 100, then subtracting a baseline percentage. This is common in CPU and memory utilization examples.

```promql
# Percentage calculation
(value / total * 100) - baseline_percent
```

--------------------------------

### PromQL Arithmetic Operators

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/promql-reference.md

Provides examples of basic arithmetic operations in PromQL, including addition, subtraction, multiplication, and division. Used for calculating derived metrics.

```promql
# Addition
avg_1h + (stddev_st * multiplier)
```

```promql
# Subtraction
avg_1h - (stddev_st * multiplier)
```

```promql
# Multiplication
avg * 0.5
```

```promql
# Division
busy_cpus / total_cpus * 100
```

--------------------------------

### PromQL Joining with `on()` and `group_left`

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/promql-reference.md

Control how label sets are matched when performing operations between instant vectors using `on()` and `group_left`/`group_right`. This example demonstrates a many-to-one match where a scalar is broadcast to multiple time series.

```promql
stddev_1h > avg_1h * on() group_left anomaly:adaptive:threshold_by_covar
```

--------------------------------

### Query anomaly:level

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/metrics-and-labels.md

Example of how to query the reference level for anomaly detection, which is compared against the anomaly bands.

```promql
anomaly:level{anomaly_name="cpu_usage"}
```

--------------------------------

### Prometheus Configuration for Anomaly Detection Rules

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/examples.md

Example Prometheus configuration snippet to include custom anomaly detection rule files.

```yaml
rule_files:
  - /etc/prometheus/rules/adaptive.yml
  - /etc/prometheus/rules/examples/node_exporter.yml
```

--------------------------------

### Label Inheritance Example

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/metrics-and-labels.md

Illustrates how labels are inherited and modified from a source metric to an anomaly detection output. Shows preserved instance and grouping labels, along with added anomaly-specific labels.

```promql
http_request_duration_seconds{instance="server1", job="api", service="checkout", le="0.5"}
```

```promql
http_request_duration_seconds{instance="server1", job="api", service="checkout", le="0.5", anomaly_name="checkout_latency", anomaly_type="latency", anomaly_strategy="adaptive"}
```

```promql
anomaly:upper_band{instance="server1", job="api", service="checkout", anomaly_name="checkout_latency", anomaly_type="latency", anomaly_strategy="adaptive"}
```

--------------------------------

### PromQL Offset Modifier Examples

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/promql-reference.md

Use the offset modifier to shift the current time backward, enabling the analysis of past data for detecting seasonal patterns. Examples show weekly and daily offsets.

```promql
# 1 week ago (for weekly seasonality)
avg_over_time(metric[1h] offset 167h30m)
```

```promql
# 1 day ago (for daily seasonality)
avg_over_time(metric[1h] offset 23h45m)
```

--------------------------------

### PromQL last_over_time Function Example

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/promql-reference.md

Returns the most recent sample within a specified time window. Useful for obtaining stable band values that update periodically.

```promql
# Get most recent value within 2 minutes
last_over_time(anomaly:adaptive:avg_1h[2m])
```

```promql
# Get most recent seasonality band within 10 minutes
last_over_time(anomaly:adaptive:lower_band_lt[10m])
```

--------------------------------

### Per-Type Alerting with Different Severity (YAML)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/README.md

This example demonstrates how to create different alert rules based on the `anomaly_type` label, assigning critical severity for 'requests' and warning severity for 'resource' anomalies, with varying durations.

```yaml
alert: AnomalyDetected:Critical
for: 2m
expr: |
  (... alert condition ...)
  and
  anomaly_type="requests"
labels:
  severity: critical

alert: AnomalyDetected:Warning
for: 10m
expr: |
  (... alert condition ...)
  and
  anomaly_type="resource"
labels:
  severity: warning
```

--------------------------------

### Example Anomaly Type Tags (YAML)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/README.md

Examples of classifying metrics by operational category using anomaly type tags. This aids in organizing and filtering anomaly data.

```yaml
anomaly_type: "requests"   # Request rates
anomaly_type: "latency"    # Response times
anomaly_type: "errors"     # Error rates
anomaly_type: "resource"   # CPU, memory, disk
```

--------------------------------

### Query anomaly:upper_band

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/metrics-and-labels.md

Example of how to query the upper bound of the anomaly band for a specific metric.

```promql
anomaly:upper_band{anomaly_name="request_rate"}
```

--------------------------------

### PromQL Aggregation Modifiers Example

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/promql-reference.md

Controls which labels are preserved during aggregation. Use `without` to remove labels and `by` to keep specific labels.

```promql
# Remove prediction_type label in final band calculation
min without(prediction_type) (...)
```

```promql
# Keep only instance and job labels
sum by (instance, job) (...)
```

--------------------------------

### Query anomaly:lower_band

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/metrics-and-labels.md

Example of how to query the lower bound of the anomaly band for a specific metric.

```promql
anomaly:lower_band{anomaly_name="error_rate"}
```

--------------------------------

### Clone PromQL Anomaly Detection Framework

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/deployment.md

Clone the repository to obtain the necessary rule files. Navigate to the rules directory after cloning.

```bash
git clone https://github.com/grafana/promql-anomaly-detection.git
cd promql-anomaly-detection/rules
```

--------------------------------

### Configure Prometheus Storage Retention

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

Set the Prometheus startup flag to increase the storage retention time. Ensure this value is greater than or equal to the required data age for offset queries.

```bash
# Prometheus startup flag
--storage.tsdb.retention.time=15d  # Or higher
```

--------------------------------

### Fetch Latest Version

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/deployment.md

Execute this command to pull the latest changes and updates for the PromQL anomaly detection framework from the main branch of its GitHub repository.

```bash
git pull origin main
```

--------------------------------

### Configure Alert Duration

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/deployment.md

Example of setting the 'for' duration in an alert rule. Increase this value to reduce false positives by requiring the condition to persist longer.

```yaml
for: 10m
```

--------------------------------

### PromQL unless Operator Example

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/promql-reference.md

Filters a left vector to exclude time series present in the right vector. Useful for excluding sparse metrics from a selection.

```promql
# Select requests, excluding sparse metrics
{anomaly_name!="", anomaly_type="requests"} > 0
unless avg_over_time({...}[1h]) < on() group_left sparse_threshold
```

--------------------------------

### Check Data Sufficiency with Sample Count (PromQL)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

Query the number of samples for a metric over the past 24 hours. If the count is below 1000, historical data may be insufficient for reliable baseline calculations.

```promql
# Count samples in past 24 hours
count(increase(anomaly:level[1d]))
```

--------------------------------

### Copy Rule Files to Prometheus

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/README.md

Copies the adaptive and robust rule files to the Prometheus rules directory. Ensure the path /etc/prometheus/rules/ exists and is writable.

```bash
cp rules/{adaptive,robust}.yml /etc/prometheus/rules/
```

--------------------------------

### Check if Input Metrics Exist (PromQL)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

Verify if the target metric is being scraped by Prometheus. If no results are returned, check Prometheus targets and scrape configuration.

```promql
# Replace "my_metric" with your actual metric name
my_metric
```

--------------------------------

### Rule Group Interval Configuration

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/deployment.md

Configure specific rule groups to override the global evaluation interval. This example sets a 5-minute interval for a rule group named 'AnomalyAdaptiveLongTerm'.

```yaml
- name: AnomalyAdaptiveLongTerm
  interval: 5m  # Runs every 5 minutes
  rules:
    # ...

```

--------------------------------

### Check if Labels are Applied (PromQL)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

Verify if the necessary labels are being applied to the metric. If no results are returned, check your recording rule or relabeling configuration.

```promql
my_metric{anomaly_name!=""}
```

--------------------------------

### Docker Compose for Prometheus and Alertmanager

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/deployment.md

Use this Docker Compose file to set up Prometheus and Alertmanager services. Ensure Prometheus configuration and rules are mounted correctly.

```yaml
version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./rules:/etc/prometheus/rules
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'

  alertmanager:
    image: prom/alertmanager:latest
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    ports:
      - "9093:9093"
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'

volumes:
  prometheus_data:
```

--------------------------------

### Check Band Value Generation in PromQL

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/alerting.md

Verify that upper and lower bands are being generated for anomaly detection. If no results appear, ensure metric selection is working and rule evaluation intervals are appropriate.

```promql
anomaly:upper_band{anomaly_name="my_metric"}
```

```promql
anomaly:lower_band{anomaly_name="my_metric"}
```

--------------------------------

### PromQL Time Window Modifier Examples

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/promql-reference.md

Time window modifiers, specified as `metric[duration]`, are used to define the time range for aggregation functions. Different durations are used for short-term baselines, recent value sampling, seasonal band updates, and long-term smoothing.

```promql
metric[1h]
```

```promql
metric[2m]
```

```promql
metric[10m]
```

```promql
metric[1d]
```

```promql
metric[26h]
```

--------------------------------

### View CPU Utilization with Bands

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/examples.md

PromQL query to visualize CPU utilization along with its anomaly detection bands.

```promql
# View CPU utilization with bands
{
  anomaly:resource:cpu,
  anomaly:upper_band{anomaly_name="node_cpu"},
  anomaly:lower_band{anomaly_name="node_cpu"}
}
```

--------------------------------

### Kubernetes StatefulSet for Prometheus

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/deployment.md

Kubernetes StatefulSet configuration for deploying Prometheus. Includes volume mounts for configuration, rules, and persistent storage.

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: prometheus
spec:
  serviceName: prometheus
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus
        image: prom/prometheus:latest
        ports:
        - containerPort: 9090
        volumeMounts:
        - name: prometheus-config
          mountPath: /etc/prometheus
        - name: prometheus-rules
          mountPath: /etc/prometheus/rules
        - name: prometheus-storage
          mountPath: /prometheus
      volumes:
      - name: prometheus-config
        configMap:
          name: prometheus-config
      - name: prometheus-rules
        configMap:
          name: prometheus-rules
      - name: prometheus-storage
        emptyDir: {}
```

--------------------------------

### PromQL Comparison Operators

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/promql-reference.md

Illustrates the use of comparison operators to filter metrics based on threshold values. Essential for setting alert conditions or data validation.

```promql
# Greater than
metric > 0
```

```promql
# Less than
metric < threshold
```

```promql
# Equal
metric == 100
```

```promql
# Not equal
metric != 0
```

```promql
# Greater than or equal
metric >= minimum_value
```

```promql
# Less than or equal
metric <= maximum_value
```

--------------------------------

### PromQL Filtering with 'unless'

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/promql-reference.md

Shows how to use the 'unless' operator in conjunction with comparison operators to filter out metrics that meet certain conditions. Useful for excluding specific data points.

```promql
# Filter sparse metrics
unless avg_over_time(...[1h]) < on() group_left anomaly:adaptive:sparse_threshold
```

--------------------------------

### Observe Anomaly Detection Bands (PromQL)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/examples.md

After an hour, query these PromQL expressions to observe the upper and lower anomaly bands generated by the adaptive strategy for response time. This indicates that anomaly detection is active.

```promql
anomaly:upper_band{anomaly_name="myapp_response_time_p99"}
anomaly:lower_band{anomaly_name="myapp_response_time_p99"}

```

--------------------------------

### Check Label Values (PromQL)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

Inspect all label combinations to ensure `anomaly_name` is not empty and `anomaly_strategy` is correctly spelled. This helps in diagnosing issues with label configuration.

```promql
# View all label combinations
group by (anomaly_name, anomaly_strategy) (my_metric{anomaly_name!=""})
```

--------------------------------

### Calculate 1-Hour Average Reference Level

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/adaptive-strategy.md

Calculates the mean of the selected metric over the past hour. This serves as the mid-line reference for band calculations, balancing responsiveness and smoothness.

```promql
avg_over_time(anomaly:adaptive:select[1h])

```

--------------------------------

### Correct YAML Multi-line String Syntax

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

Demonstrates the correct way to define a multi-line expression in YAML using the `|-` literal block scalar, which preserves newlines but chomps the final newline.

```yaml
expr: |-
  metric > 0
```

--------------------------------

### Check for New Versions

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/deployment.md

Use this command to view the commit history and check for new versions of the PromQL anomaly detection framework from its GitHub repository.

```bash
git log --oneline https://github.com/grafana/promql-anomaly-detection.git
```

--------------------------------

### Visualize Bands with Metric (PromQL)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/README.md

Display the original metric alongside its upper and lower anomaly bands. Use this to visually inspect how the metric relates to the detected anomaly boundaries.

```promql
# Original metric alongside bands
{metric_name}
```

```promql
# Upper and lower bounds
anomaly:upper_band{anomaly_name="..."}
anomaly:lower_band{anomaly_name="..."}
```

--------------------------------

### Verify Offset Query Functionality

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

Test offset queries directly to ensure historical data is available and accessible. If this query returns no data, historical data might be missing or not retained.

```promql
# Test offset query directly
anomaly:adaptive:select offset 167h30m
```

--------------------------------

### Check Metric Selection in PromQL

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/alerting.md

Use these PromQL queries to verify if metrics are being selected by the anomaly detection system. If no results are returned, check label values and strategy names.

```promql
# Adaptive strategy
anomaly:adaptive:select{anomaly_name="my_metric"}
```

```promql
# Robust strategy
anomaly:robust:select{anomaly_name="my_metric"}
```

--------------------------------

### Prometheus Configuration for Rule Files (YAML)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/examples.md

Update your Prometheus configuration to include the custom recording rule file. Ensure the path to your custom rules file is correctly specified.

```yaml
rule_files:
  - /etc/prometheus/rules/adaptive.yml
  - /etc/prometheus/rules/robust.yml
  - /etc/prometheus/rules/custom/myapp.yml

```

--------------------------------

### Per-Type Alerting Thresholds for Requests and Resources

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/alerting.md

This snippet defines two distinct alert rules for different metric types: 'requests' and 'resource'. The 'requests' alert uses a shorter 'for' duration (2m) for faster detection of critical issues, while the 'resource' alert uses a longer 'for' duration (10m) to minimize false positives for less critical metrics. This allows for tailored sensitivity based on the metric's importance.

```yaml
alert: AnomalyDetected:Request
for: 2m  # Fast alerts for request rates
expr: >
  (last_over_time(anomaly:level{anomaly_type="requests"}[2m])
   < last_over_time(anomaly:lower_band{anomaly_type="requests", anomaly_strategy="adaptive"}[2m])
   or
   last_over_time(anomaly:level{anomaly_type="requests"}[2m])
   > last_over_time(anomaly:upper_band{anomaly_type="requests", anomaly_strategy="adaptive"}[2m]))
labels:
  severity: critical
  metric_type: requests

---

alert: AnomalyDetected:Resource
for: 10m  # Slower alerts for resource metrics
expr: >
  (last_over_time(anomaly:level{anomaly_type="resource"}[2m])
   < last_over_time(anomaly:lower_band{anomaly_type="resource", anomaly_strategy="robust"}[2m])
   or
   last_over_time(anomaly:level{anomaly_type="resource"}[2m])
   > last_over_time(anomaly:upper_band{anomaly_type="resource", anomaly_strategy="robust"}[2m]))
labels:
  severity: warning
  metric_type: resource
```

--------------------------------

### Configure Prometheus Rule Files

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/README.md

Add this configuration to your Prometheus config file to include the anomaly detection rules. Ensure the path to the rules folder is correct.

```yaml
rule_files:
- /etc/prometheus/rules/*.yml
```

--------------------------------

### Set Permissions for Prometheus Rules

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/deployment.md

Ensure Prometheus has read permissions for the rule files and ownership is set correctly.

```bash
chmod 644 /etc/prometheus/rules/*.yml
chown prometheus:prometheus /etc/prometheus/rules/*.yml
```

--------------------------------

### Check Rule Syntax with Promtool

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/deployment.md

Validate the syntax of your Prometheus rule files using the promtool command-line utility. This helps catch errors before deployment.

```bash
promtool check rules /path/to/rules.yml
```

--------------------------------

### Check Sparse Metric Threshold (PromQL)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

This PromQL query checks if the hourly average of a metric is below a threshold, which is relevant for `anomaly_type="requests"` when the metric is sparse. If below the threshold, consider increasing the `sparse_threshold` constant or classifying the metric differently.

```promql
avg_over_time(my_metric[1h]) >= 0.0833  # 5 observations per 60 seconds
```

--------------------------------

### Verify Intermediate Metrics for Adaptive Anomaly Detection

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

Check if essential intermediate metrics for the adaptive anomaly detection method are being generated. These metrics should return values after approximately 5 minutes, with `stddev_st` potentially taking longer (up to 26 hours).

```promql
# Should return values after 5 min
count(anomaly:adaptive:avg_1h)
```

```promql
# Should return values after 5 min with high pass filter
count(anomaly:adaptive:stddev_1h:filtered)
```

```promql
# Should return values after 26h (until then may be absent)
count(anomaly:adaptive:stddev_st)
```

--------------------------------

### Intermediate Metrics - Robust Strategy (PromQL)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/README.md

These metrics provide insights into the intermediate calculations for the robust anomaly detection strategy, including median, median absolute deviation, and seasonal bands.

```promql
anomaly:robust:select                # Selected metrics
anomaly:robust:median_lt             # 24-hour median
anomaly:robust:mad_lt                # Median absolute deviation
anomaly:robust:upper_seasonality_band   # Seasonal upper band
anomaly:robust:lower_seasonality_band   # Seasonal lower band
```

--------------------------------

### Debug Anomaly Selection (PromQL)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/README.md

Check which metrics are being selected by specific anomaly detection strategies. Use this to verify that the correct metrics are being fed into the anomaly detection process.

```promql
# Metrics selected by adaptive strategy
count(anomaly:adaptive:select) by (anomaly_name)
```

```promql
# Metrics selected by robust strategy
count(anomaly:robust:select) by (anomaly_name)
```

--------------------------------

### Check `last_over_time` for Sparse Metrics

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

If intermediate metrics exist but anomaly bands are not generated, check if `last_over_time` is returning values for the relevant metric. If it returns no results, the metrics might be too sparse, requiring an increased lookback window in the band query.

```promql
last_over_time(anomaly:adaptive:avg_1h[2m])
```

--------------------------------

### Intermediate Metrics - Adaptive Strategy (PromQL)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/README.md

These metrics are useful for debugging and visualizing the intermediate steps of the adaptive anomaly detection strategy. They include selected metrics, smoothed averages, and filtered standard deviations.

```promql
anomaly:adaptive:select              # Selected metrics
anomaly:adaptive:avg_1h              # 1-hour mean
anomaly:adaptive:stddev_st           # 26-hour smoothed variability
anomaly:adaptive:stddev_1h:filtered  # High-pass filtered 1-hour stddev
anomaly:adaptive:upper_band_lt       # Long-term upper band
anomaly:adaptive:lower_band_lt       # Long-term lower band
```

--------------------------------

### Calculate Request and Error Rates

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/promql-reference.md

Use the `rate()` function to calculate the per-second increase of counters. This is typically used to prepare metrics for anomaly detection rules.

```promql
# Request rate (requests per second)
rate(requests_total[5m])
```

```promql
# Error rate
rate(errors_total[5m])
```

--------------------------------

### Monitor Anomaly Detection Progress with PromQL

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/deployment.md

Use these PromQL queries to check if metrics are being selected, if short-term bands are generated, and if long-term bands are available, indicating the anomaly detection system is operational.

```promql
# Check if metrics are selected
count(anomaly:adaptive:select)
count(anomaly:robust:select)

# Check if short-term bands exist
count(anomaly:adaptive:avg_1h)

# Check if long-term bands exist
count(anomaly:adaptive:lower_band_lt)
```

--------------------------------

### PromQL Anomaly Detection File Structure

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/README.md

This is the directory structure for the PromQL Anomaly Detection project. It details the purpose of each markdown file within the project.

```text
output/
├── README.md                 # This file
├── overview.md               # Architecture and concepts
├── deployment.md             # Installation and setup
├── examples.md               # Real-world usage examples
├── adaptive-strategy.md      # Adaptive algorithm reference
├── robust-strategy.md        # Robust algorithm reference
├── metrics-and-labels.md     # Metrics and label reference
├── alerting.md               # Alert configuration guide
├── promql-reference.md       # PromQL functions used
└── troubleshooting.md        # Common issues and solutions
```

--------------------------------

### Calculate Daily Seasonal Median

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/robust-strategy.md

Calculates the median of a 1-hour window from 23 hours and 45 minutes prior. Use this to establish a typical value for the current time on the previous day.

```promql
quantile_over_time(0.5, anomaly:robust:select[1h] offset 23h45m)
```

--------------------------------

### Verify Custom Metrics Generation (PromQL)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/examples.md

Query these PromQL expressions to verify that your custom metrics are being generated by Prometheus after applying the recording rules. Results should appear after a short delay.

```promql
# Should return results after 1 min
myapp:response_time:p99
myapp:error_rate
myapp:db_query_latency:p95

```

--------------------------------

### Reload or Restart Prometheus

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/deployment.md

Apply configuration changes by either performing a hot reload of the Prometheus configuration using a HUP signal or by restarting the Prometheus service.

```bash
# Hot reload
kill -HUP $(pidof prometheus)
# or
curl -X POST http://localhost:9090/-/reload

# Full restart
systemctl restart prometheus
# or
docker restart prometheus
```

--------------------------------

### Update Prometheus Configuration with Rule Files

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/deployment.md

Edit the Prometheus configuration file (`prometheus.yml`) to include the paths to the anomaly detection rule files. The `evaluation_interval` should be at least 30 seconds.

```yaml
global:
  scrape_interval: 15s
  evaluation_interval: 30s  # Minimum 30s; can be higher

rule_files:
  - /etc/prometheus/rules/adaptive.yml
  - /etc/prometheus/rules/robust.yml
  # Include examples if desired:
  - /etc/prometheus/rules/examples/node_exporter.yml
  - /etc/prometheus/rules/examples/otel_demo.yml

# Remaining configuration...
scrape_configs:
  # Your scrape jobs...
```

--------------------------------

### Debug Band Stability (PromQL)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/README.md

Analyze the stability and width of anomaly bands over time. This helps in understanding the sensitivity and consistency of the anomaly detection.

```promql
# Band width over time
anomaly:upper_band - anomaly:lower_band
```

```promql
# Band as percentage of median
(anomaly:upper_band - anomaly:lower_band) / anomaly:robust:median_lt * 100
```

--------------------------------

### Primary Output Metrics (PromQL)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/README.md

These metrics represent the upper and lower anomaly thresholds and the reference level being monitored. They are available for both adaptive and robust anomaly detection strategies.

```promql
# Upper anomaly threshold
anomaly:upper_band{anomaly_name="...", anomaly_type="...", anomaly_strategy="..."}

# Lower anomaly threshold
anomaly:lower_band{anomaly_name="...", anomaly_type="...", anomaly_strategy="..."}

# Reference level being monitored
anomaly:level{anomaly_name="...", anomaly_type="...", anomaly_strategy="..."}
```

--------------------------------

### Check Prometheus Version

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

Verify that your Prometheus version is 2.15 or higher, which is required for certain PromQL functions like `label_replace` and `quantile_over_time` used in the anomaly detection framework.

```bash
curl http://localhost:9090/api/v1/version
# Look for version >= 2.15.0
```

--------------------------------

### Verify Intermediate Metrics for Robust Anomaly Detection

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

Check if essential intermediate metrics for the robust anomaly detection method are being generated. These metrics should return values after approximately 5 minutes.

```promql
count(anomaly:robust:median_lt)
```

```promql
count(anomaly:robust:mad_lt)
```

--------------------------------

### Calculate Upper Band with Adaptive Strategy

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/adaptive-strategy.md

Calculates the upper band by taking the maximum of short-term, margin, and long-term predictions. Use this to define the upper limit for anomaly detection.

```promql
max without(prediction_type)
(
  label_replace(
      last_over_time(anomaly:adaptive:avg_1h[2m]) + last_over_time(anomaly:adaptive:stddev_st[2m]) * on() group_left anomaly:adaptive:stddev_multiplier,
      "prediction_type", "short_term", "", ""
  )
  or
  label_replace(
      last_over_time(anomaly:adaptive:avg_1h[2m]) + last_over_time(anomaly:adaptive:avg_1h[2m]) * on() group_left anomaly:adaptive:margin_multiplier,
      "prediction_type", "margin", "", ""
  )
  or
  label_replace(
      last_over_time(anomaly:adaptive:upper_band_lt[10m]),
      "prediction_type", "long_term", "", ""
  )
)
```

--------------------------------

### Adaptive Strategy Debug Queries

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/metrics-and-labels.md

Provides PromQL queries to inspect intermediate metrics generated by the adaptive anomaly detection strategy. Useful for understanding selected values, mean, variability, and filtered standard deviation.

```promql
# View raw selected values
anomaly:adaptive:select{anomaly_name="request_rate"}
```

```promql
# View mean and variability
anomaly:adaptive:avg_1h{anomaly_name="request_rate"}
anomaly:adaptive:stddev_st{anomaly_name="request_rate"}
```

```promql
# Visualize high-pass filtered stddev
anomaly:adaptive:stddev_1h:filtered{anomaly_name="request_rate"}
```

--------------------------------

### Check Metric Cardinality with PromQL

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

Use this PromQL query to check the cardinality of a metric. If the count is 0, it indicates that long-term bands are not being generated.

```promql
# Should be consistent with input cardinality
count(anomaly:adaptive:lower_band_lt)
```

--------------------------------

### Check Prometheus Version

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

Retrieve the current Prometheus version to ensure compatibility with specific PromQL functions. Upgrade Prometheus if it's older than version 2.15.0.

```bash
curl http://localhost:9090/api/v1/version
```

--------------------------------

### Define Adaptive and Robust Recording Rules

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/examples.md

Create separate recording rules for the same metric using different anomaly detection strategies. This allows for comparison and validation of which strategy performs best for your specific use case.

```yaml
- record: myapp:latency:adaptive
  expr: histogram_quantile(0.95, sum(...))
  labels:
    anomaly_name: "myapp_latency"
    anomaly_type: "latency"
    anomaly_strategy: "adaptive"

- record: myapp:latency:robust
  expr: histogram_quantile(0.95, sum(...))
  labels:
    anomaly_name: "myapp_latency_robust"
    anomaly_type: "latency"
    anomaly_strategy: "robust"
```

--------------------------------

### Visualize Anomaly Bands in Grafana

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/deployment.md

Configure Grafana panels to display original metrics alongside their calculated anomaly upper and lower bands. Use line charts with fill transparency for clear visualization.

```promql
{original_metric_name}

# Upper and lower bands
anomaly:upper_band{anomaly_name="my_metric"}
anomaly:lower_band{anomaly_name="my_metric"}
```

--------------------------------

### Prometheus Configuration for OTEL Anomaly Rules

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/examples.md

This snippet shows how to include the OpenTelemetry demo anomaly detection rule file in your Prometheus configuration.

```yaml
rule_files:
  - /etc/prometheus/rules/adaptive.yml
  - /etc/prometheus/rules/examples/otel_demo.yml
```

--------------------------------

### Find Current Anomalies (PromQL)

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/README.md

Identify metrics that are currently outside their expected anomaly bands. This helps in detecting and alerting on unusual behavior.

```promql
# Metrics above upper band
anomaly:level > anomaly:upper_band
```

```promql
# Metrics below lower band
anomaly:level < anomaly:lower_band
```

```promql
# Either direction
anomaly:level < anomaly:lower_band or anomaly:level > anomaly:upper_band
```

--------------------------------

### Configure Alertmanager Static Targets

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

Ensure the Alertmanager address is correctly configured in Prometheus's `prometheus.yml` file. This static configuration tells Prometheus where to send alerts.

```yaml
# In prometheus.yml
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']
```

--------------------------------

### Select Metrics for Adaptive Anomaly Detection

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/adaptive-strategy.md

Selects metrics tagged for adaptive anomaly detection. For request-rate metrics, it applies sparse filtering to exclude metrics with a low 1-hour average. Other metric types are included without sparsity checks. Returns values greater than 0.

```promql
( 
  {anomaly_name!="", anomaly_type="requests", anomaly_select="", anomaly_strategy=~"^$|adaptive"} > 0
  unless avg_over_time({anomaly_name!="", anomaly_type="requests", anomaly_select="", anomaly_strategy=~"^$|adaptive"}[1h]) < on() group_left anomaly:adaptive:sparse_threshold
)
OR
{anomaly_name!="", anomaly_type="latency", anomaly_select="", anomaly_strategy=~"^$|adaptive"} > 0
OR
{anomaly_name!="", anomaly_type="errors", anomaly_select="", anomaly_strategy=~"^$|adaptive"} > 0
OR
{anomaly_name!="", anomaly_type="resource", anomaly_select="", anomaly_strategy=~"^$|adaptive"} > 0
OR
{anomaly_name!="", anomaly_select="", anomaly_strategy=~"^$|adaptive"} > 0

```

--------------------------------

### Monitor Prometheus Process Metrics

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

Utilize PromQL to monitor the resident memory usage and CPU utilization of the Prometheus process itself. This helps in identifying resource bottlenecks.

```promql
# Memory usage
process_resident_memory_bytes
```

```promql
# CPU usage
rate(process_cpu_seconds_total[5m])
```

--------------------------------

### Calculate Weekly Seasonal Median

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/robust-strategy.md

Calculates the median of a 1-hour window from 167 hours and 45 minutes prior. Use this to establish a typical value for the current time on a previous week.

```promql
quantile_over_time(0.5, anomaly:robust:select[1h] offset 167h45m)
```

--------------------------------

### Kubernetes ConfigMap for Prometheus Rules

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/deployment.md

Defines a Kubernetes ConfigMap to store Prometheus rule files. Supports multiple rule files like adaptive.yml and robust.yml.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-rules
  namespace: monitoring
data:
  adaptive.yml: |
    groups:
      - name: AnomalyAdaptiveShortTerm
        # ... rule content ...
  robust.yml: |
    groups:
      - name: AnomalyRobust
        # ... rule content ...
```

--------------------------------

### Calculate Band Width

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

Use these PromQL queries to calculate the width of anomaly bands for adaptive and robust strategies. Compare this to the metric's typical variation.

```promql
# Adaptive
anomaly:upper_band - anomaly:lower_band
```

```promql
# Robust
anomaly:upper_band - anomaly:lower_band
```

--------------------------------

### Robust Strategy Debug Queries

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/metrics-and-labels.md

Offers PromQL queries for debugging the robust anomaly detection strategy. These queries help in examining raw selected values, median, Median Absolute Deviation (MAD), and seasonal band components.

```promql
# View raw selected values
anomaly:robust:select{anomaly_name="cpu_usage"}
```

```promql
# View median and MAD
anomaly:robust:median_lt{anomaly_name="cpu_usage"}
anomaly:robust:mad_lt{anomaly_name="cpu_usage"}
```

```promql
# Visualize seasonal components
anomaly:robust:upper_seasonality_band{anomaly_name="cpu_usage"}
anomaly:robust:lower_seasonality_band{anomaly_name="cpu_usage"}
```

--------------------------------

### Include Custom Rules in Prometheus Configuration

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/deployment.md

Add the path to your custom rule file (`custom.yml`) to the `rule_files` section in your `prometheus.yml`.

```yaml
rule_files:
  - /etc/prometheus/rules/adaptive.yml
  - /etc/prometheus/rules/robust.yml
  - /etc/prometheus/rules/custom.yml
```

--------------------------------

### Check Rule File Syntax with Promtool

Source: https://github.com/grafana/promql-anomaly-detection/blob/main/_autodocs/troubleshooting.md

Validate the syntax of your Prometheus rule files using the `promtool check rules` command. This is crucial if rule groups appear empty in the Prometheus UI.

```bash
promtool check rules /path/to/rule.yml
```