### Install OpenEBS Monitoring Mixin

Source: https://context7.com/openebs/monitoring/llms.txt

Installs prerequisites, clones the repository, installs dependencies, and sets up output directories for generating dashboards and rules.

```bash
# Install prerequisites
go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb
sudo apt-get install jsonnet

# Clone repository and install dependencies
git clone https://github.com/openebs/monitoring
cd monitoring/jsonnet/openebs-mixin
jb install

# Update build.sh with output paths
cat > build.sh << 'EOF'
#!/bin/bash
dashboardsDirPath='./output/dashboards'
rulesDirPath='./output/rules'
mkdir -p $dashboardsDirPath $rulesDirPath
EOF

# Generate dashboards and rules
make generate

# Output files are created in:
# - ./output/dashboards/*.json (Grafana dashboards)
# - ./output/rules/*.json (Prometheus alert rules)
```

--------------------------------

### Install Helm Chart from Umbrella Folder

Source: https://github.com/openebs/monitoring/blob/develop/docs/extends-stack-with-umbrella-chart.md

Install the chart by running this command in the umbrella folder. Replace `[RELEASE_NAME]` and `[NAMESPACE]` with your desired values.

```console
helm install [RELEASE_NAME] . -n  [NAMESPACE] --create-namespace
```

```console
helm install monitoring . -n monitoring --create-namespace
```

--------------------------------

### Install Monitoring Chart with Helm (One Line)

Source: https://github.com/openebs/monitoring/blob/develop/docs/guide.md

Install the monitoring chart with Helm in a single command, setting key parameters via --set.

```bash
helm install [RELEASE_NAME] monitoring/monitoring -n  [PROMETHEUS-STACK-NAMESPACE]  --set kube-prometheus-stack.install=false,kube-prometheus-stack.grafana.sidecar.dashboards.label=[UPDATED_LABEL]
```

--------------------------------

### Install jsonnet and jb

Source: https://github.com/openebs/monitoring/blob/develop/jsonnet/openebs-mixin/README.md

Install the jsonnet and jb tools required for managing the OpenEBS mixin.

```bash
$ go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb
$ sudo apt-get install jsonnet
```

--------------------------------

### Install Grafana Plugins

Source: https://github.com/openebs/monitoring/blob/develop/docs/guide.md

Exec into the Grafana container and use the grafana-cli to install necessary dashboard plugins.

```bash
grafana-cli plugins install grafana-polystat-panel 
grafana-cli plugins install snuids-trafficlights-panel
```

--------------------------------

### Install Monitoring Chart with Helm

Source: https://github.com/openebs/monitoring/blob/develop/docs/guide.md

Install the monitoring chart using Helm, providing a values.yaml file for configuration.

```bash
helm install -f values.yaml [RELEASE_NAME] monitoring/monitoring
```

--------------------------------

### Install Helm Chart from Repository

Source: https://github.com/openebs/monitoring/blob/develop/docs/extends-stack-with-umbrella-chart.md

Install the chart from a Helm chart repository using this command. Replace `[RELEASE_NAME]`, `[REPOSITORY]`, `[CHART_NAME]`, and `[NAMESPACE]` accordingly.

```console
helm install [RELEASE_NAME] [REPOSITORY]/[CHART_NAME] -n  [NAMESPACE]  --create-namespace
```

```console
helm install monitoring chartmuseum/monitoring-stack -n monitoring --create-namespace
```

--------------------------------

### Clone and Install Mixin Dependencies

Source: https://github.com/openebs/monitoring/blob/develop/jsonnet/openebs-mixin/README.md

Clone the OpenEBS monitoring repository and install its dependencies using jb.

```bash
$ git clone https://github.com/openebs/monitoring
$ cd monitoring/jsonnet/openebs-mixin
$ jb install
```

--------------------------------

### Install OpenEBS Monitoring Helm Chart

Source: https://github.com/openebs/monitoring/blob/develop/README.md

Installs the OpenEBS monitoring Helm chart into the 'openebs' namespace. The namespace will be created if it does not exist.

```bash
helm install monitoring monitoring/monitoring --namespace openebs --create-namespace
```

--------------------------------

### Install OpenEBS Monitoring Addon

Source: https://context7.com/openebs/monitoring/llms.txt

Install only the OpenEBS monitoring dashboards and alerts to an existing Prometheus stack. Ensure Grafana plugins are installed and Grafana is restarted before applying the Helm chart.

```bash
kubectl exec -it <grafana-pod> -n <prometheus-namespace> -- \
  grafana-cli plugins install grafana-polystat-panel
kubectl exec -it <grafana-pod> -n <prometheus-namespace> -- \
  grafana-cli plugins install snuids-trafficlights-panel

kubectl rollout restart deployment grafana -n <prometheus-namespace>

helm install monitoring monitoring/monitoring \
  -n prometheus-operator \
  --set kube-prometheus-stack.install=false \
  --set kube-prometheus-stack.grafana.sidecar.dashboards.label=grafana_dashboard

kubectl get servicemonitors -n prometheus-operator -l release="monitoring"
# NAME                            AGE
# monitoring-openebs-mayastor     33m
# monitoring-openebs-lvmlocalpv   33m

kubectl get cm -n prometheus-operator -l release="monitoring"
# NAME                                     DATA   AGE
# monitoring-openebs-diskpool              1      37m
# monitoring-openebs-volume                1      37m
```

--------------------------------

### PromQL Query Examples for LVM Storage

Source: https://context7.com/openebs/monitoring/llms.txt

Example PromQL queries demonstrating how to monitor LVM storage utilization, thin pool usage, total capacity, and identify volume groups with missing physical volumes.

```promql
# Example PromQL Queries
# Volume group utilization percentage
((lvm_vg_total_size_bytes - lvm_vg_free_size_bytes) / lvm_vg_total_size_bytes) * 100

# Thin pool usage percentage by pool name
lvm_lv_used_percent{segtype="thin-pool"}

# Total storage capacity across all volume groups
sum(lvm_vg_total_size_bytes)

# Volume groups with missing physical volumes
lvm_vg_missing_pv_count > 0
```

--------------------------------

### Install OpenEBS Monitoring Stack with Helm

Source: https://context7.com/openebs/monitoring/llms.txt

Installs the complete OpenEBS monitoring stack, including Prometheus, Grafana, and Alertmanager. Use a custom values file for specific configurations.

```bash
# Add the OpenEBS monitoring Helm repository
helm repo add monitoring https://openebs.github.io/monitoring/
helm repo update

# View available charts
helm search repo monitoring

# Install the complete monitoring stack with default values
helm install monitoring monitoring/monitoring --namespace openebs --create-namespace

# Install with custom values file
helm install monitoring monitoring/monitoring \
  --namespace openebs \
  --create-namespace \
  -f custom-values.yaml

# Verify installation - check pods are running
kubectl get pods -n openebs | grep -E "(prometheus|grafana|alertmanager)"

# Access Grafana (default credentials: admin/admin)
kubectl get svc -n openebs | grep grafana
# Open browser at http://<NodeIP>:32515

# Port-forward if NodePort is not accessible
kubectl port-forward --namespace openebs svc/monitoring-grafana 3000:80
# Access at http://127.0.0.1:3000
```

--------------------------------

### values.yaml with Default Monitoring Install

Source: https://github.com/openebs/monitoring/blob/develop/docs/extends-stack-with-umbrella-chart.md

This values.yaml file enables the OpenEBS monitoring stack using its default configurations by setting the 'monitoring.install' condition to true.

```yaml
monitoring:
  install: true
```

--------------------------------

### Install Openebs-Mixin

Source: https://github.com/openebs/monitoring/blob/develop/docs/openebs-mixin-user-guide.md

Create an openebs-mixin object to generate Prometheus rules and Grafana dashboards. Add the grafanaDashboards object to the dashboards field.

```jsonnet
	local openebsMixin=(
	    import 'kube-prometheus/lib/mixin.libsonnet')({
	    name: 'openebsMixin',
		mixin: (
		import './openebs-mixin/mixin.libsonnet'),
	});
	
```

```jsonnet
	. . .
	values+:: {
      grafana+: {
        // To install openebs dashboards
        dashboards+: openebsMixin.grafanaDashboards,
      },
	}
	. . .
	
```

```jsonnet
	. . .
	// To generate prometheusRule object 
	{ 'openebs-addons/openebs-mixin-prometheus-rules': openebsMixin.prometheusRules } 
	. . .
	
```

--------------------------------

### Configure OpenEBS Monitoring Helm Values

Source: https://context7.com/openebs/monitoring/llms.txt

Example of a complete Helm values file for customizing the OpenEBS monitoring stack, including Prometheus, Alertmanager, Grafana, and local PV provisioner settings.

```yaml
# values.yaml - Complete configuration example
nameOverride: ""
namespaceOverride: ""
fullnameOverride: ""

# Kube-Prometheus-Stack Configuration
kube-prometheus-stack:
  install: true

  # Prometheus Configuration
  prometheus:
    service:
      type: NodePort
      nodePort: 32514
    prometheusSpec:
      serviceMonitorSelectorNilUsesHelmValues: false
      podMonitorSelectorNilUsesHelmValues: false
      ruleSelectorNilUsesHelmValues: false
      replicas: 1
      storageSpec:
        volumeClaimTemplate:
          metadata:
            name: openebs-prometheus-pv
          spec:
            storageClassName: openebs-hostpath
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 10Gi

  # Alertmanager Configuration
  alertmanager:
    enabled: true
    service:
      type: NodePort
      nodePort: 30903
    config:
      global:
        resolve_timeout: 5m
      route:
        group_by: ["alertname", "job", "volName"]
        group_wait: 30s
        group_interval: 5m
        repeat_interval: 4h
        receiver: "null"
        routes:
          - matchers:
              - product="openebs"
            receiver: "null"
      receivers:
        - name: "null"

  # Grafana Configuration
  grafana:
    enabled: true
    adminPassword: admin
    service:
      type: NodePort
      nodePort: 32515
    grafana.ini:
      panels:
        disable_sanitize_html: true
    plugins:
      - grafana-polystat-panel
      - snuids-trafficlights-panel
    persistence:
      enabled: true
      storageClassName: openebs-hostpath
      accessModes:
        - ReadWriteOnce
      size: 1Gi

# OpenEBS Storage Provisioner
localpv-provisioner:
  enabled: true
  hostpathClass:
    name: openebs-hostpath
    enabled: true

# Node Problem Detector (optional)
node-problem-detector:
  install: false
  metrics:
    serviceMonitor:
      enabled: true

```

--------------------------------

### Add Monitoring Helm Repository

Source: https://github.com/openebs/monitoring/blob/develop/README.md

Adds the OpenEBS monitoring Helm repository to your local Helm configuration. Run this command before installing the monitoring chart.

```bash
helm repo add monitoring https://openebs.github.io/monitoring/
helm repo update
```

--------------------------------

### Helm Chart Configuration for Monitoring Stack

Source: https://context7.com/openebs/monitoring/llms.txt

Configure the installation of Prometheus, Alertmanager, and Grafana within the Helm chart. This includes enabling ingress, setting up storage, and defining host paths and domains.

```yaml
monitoring:
  install: true

  kube-prometheus-stack:
    # Prometheus with ingress
    prometheus:
      ingress:
        enabled: true
        annotations:
          nginx.ingress.kubernetes.io/ssl-redirect: "false"
        paths:
          - /prometheus(/|$)(.*)
        hosts:
          - monitoring.example.com
      prometheusSpec:
        routePrefix: /prometheus
        storageSpec:
          volumeClaimTemplate:
            spec:
              storageClassName: fast-storage
              accessModes: ["ReadWriteOnce"]
              resources:
                requests:
                  storage: 50Gi

    # Alertmanager with storage
    alertmanager:
      ingress:
        enabled: true
        annotations:
          nginx.ingress.kubernetes.io/ssl-redirect: "false"
          nginx.ingress.kubernetes.io/rewrite-target: /$2
        paths:
          - /alertmanager(/|$)(.*)
        hosts:
          - monitoring.example.com
      alertmanagerSpec:
        storage:
          volumeClaimTemplate:
            spec:
              storageClassName: fast-storage
              accessModes: ["ReadWriteOnce"]
              resources:
                requests:
                  storage: 10Gi

    # Grafana with ingress
    grafana:
      grafana.ini:
        server:
          root_url: http://monitoring.example.com/grafana
      persistence:
        enabled: true
        storageClassName: fast-storage
        size: 10Gi
      ingress:
        enabled: true
        annotations:
          nginx.ingress.kubernetes.io/ssl-redirect: "false"
          nginx.ingress.kubernetes.io/rewrite-target: /$2
        path: /grafana(/|$)(.*)
        hosts:
          - monitoring.example.com

  # Enable Node Problem Detector
  node-problem-detector:
    install: true
```

--------------------------------

### Chart.yaml for OpenEBS Monitoring Stack

Source: https://github.com/openebs/monitoring/blob/develop/docs/extends-stack-with-umbrella-chart.md

Define your umbrella chart's metadata and declare the OpenEBS monitoring stack as a dependency. The 'condition' property allows conditional installation.

```yaml
apiVersion: v2
name: monitoring-stack
description: Umbrella Chart for OpenEBS monitoring stack
icon: https://avatars.githubusercontent.com/u/20769039?s=200&v=4
type: application
version: 0.1.1
appVersion: 1.0.0
dependencies:
  - name: monitoring
    version: 0.4.8
    repository: "https://openebs.github.io/monitoring/"
    condition: monitoring.install
```

--------------------------------

### Prometheus Alert Rule Example

Source: https://github.com/openebs/monitoring/blob/develop/docs/extends-stack-with-umbrella-chart.md

An example of a Prometheus alert rule defined in JSON format. This specific rule, 'InstanceDown', triggers when an instance has been down for more than 10 seconds, with severity set to 'page'.

```json
{
  "groups": [
    {
      "name": "Instances",
      "rules": [
        {
          "alert": "InstanceDown",
          "expr": "up == 0",
          "for": "10s",
          "labels": {
            "severity": "page"
          },
          "annotations": {
            "description": "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minute.",
            "summary": "Instance {{ $labels.instance }} down"
          }
        }
      ]
    }
  ]
}
```

--------------------------------

### Verify ServiceMonitors

Source: https://github.com/openebs/monitoring/blob/develop/docs/guide.md

Use kubectl to get ServiceMonitors in the specified namespace, filtered by the release label.

```bash
kubectl get servicemonitors -n prometheus-operator -l  release="monitoring"
NAME                              AGE
monitoring-openebs-mayastor   33m
```

--------------------------------

### Get Node Information

Source: https://github.com/openebs/monitoring/blob/develop/README.md

Retrieves wide information about your Kubernetes nodes, including their public IP addresses. This is useful for accessing Grafana externally.

```bash
kubectl get nodes -o wide
```

--------------------------------

### List Services in Namespace

Source: https://github.com/openebs/monitoring/blob/develop/deploy/charts/templates/NOTES.txt

Lists all services within the specified Kubernetes namespace.

```bash
kubectl get svc -n {{ .Release.Namespace }}
```

--------------------------------

### Run Helm Chart Generation Script

Source: https://github.com/openebs/monitoring/blob/develop/contribute/release.md

Navigate to the `jsonnet/openebs-mixin` directory and execute `make generate` to create the necessary JSON files for dashboards and alert rules.

```bash
$ cd jsonnet/openebs-mixin
$ make generate
```

--------------------------------

### Helm Commands for Deploying Monitoring

Source: https://context7.com/openebs/monitoring/llms.txt

Commands to deploy, build dependencies, and upgrade the OpenEBS monitoring Helm chart. Ensure you are in the umbrella-chart directory before executing.

```bash
# Deploy umbrella chart
cd umbrella-chart

# Download dependencies
helm dep build

# Install the umbrella chart
helm install monitoring . -n monitoring --create-namespace

# Upgrade with new values
helm upgrade monitoring . -n monitoring -f custom-values.yaml

# Access applications via ingress
# http://monitoring.example.com/prometheus
# http://monitoring.example.com/grafana
# http://monitoring.example.com/alertmanager
```

--------------------------------

### Build Helm Dependencies

Source: https://github.com/openebs/monitoring/blob/develop/docs/extends-stack-with-umbrella-chart.md

Run this command in the umbrella folder to download chart dependencies. Use `helm dep up` if Chart.yaml has been modified.

```console
helm dep build
```

```console
helm dep up
```

--------------------------------

### Get Grafana Service IP

Source: https://github.com/openebs/monitoring/blob/develop/README.md

Retrieves the Cluster IP address for the Grafana service within the specified namespace. Replace [NAMESPACE] with your actual namespace.

```bash
kubectl get svc -n [NAMESPACE] | grep -i grafana
```

--------------------------------

### Build OpenEBS Mixin

Source: https://github.com/openebs/monitoring/blob/develop/jsonnet/openebs-mixin/README.md

Build the OpenEBS mixin after updating the dashboard and rules directory paths in build.sh.

```bash
$ make generate
```

--------------------------------

### Update Folders for Dashboard and Rule Generation

Source: https://github.com/openebs/monitoring/blob/develop/contribute/release.md

Modify the `dashboardsFolder` and `rulesFolder` arrays in `build.sh` to include new service names for which dashboards and alert rules will be generated.

```bash
. . .
dashboardsFolder=(npd lvmLocalPV zfsLocalPV mayastor)
. . .

generateRules(){
	. . .
	rulesFolder=(npd lvmLocalPV zfsLocalPV mayastor)
    . . .
}
```

--------------------------------

### Configure New Dashboard Inclusion

Source: https://github.com/openebs/monitoring/blob/develop/contribute/dashboards.md

Add this configuration to `config.libsonnet` to enable the inclusion of a new dashboard. Set `newDashboard` to `true` to import the dashboard JSON as a `grafanaDashboards` object.

```jsonnet
. . .
// dashboards configuration. If set, then dashboards are included in grafanaDashboards object.
dashboards: {
	newDashboard: true # to import dashboard as grafanaDashboards object
},
. . .

```

--------------------------------

### Import and Include New Alerts in prometheus-rules.libsonnet

Source: https://github.com/openebs/monitoring/blob/develop/contribute/alerts.md

Import the newly defined alert rules and merge them into the main 'prometheusRules' object in the prometheus-rules.libsonnet file.

```libsonnet
. . .
	local newAlertRules = (import './new-alert-rules.libsonnet');

	prometheusRules+::
		. . .
		+ newAlertRules(prometheusRules._config).prometheusRules.newAlertRules,
	. . .
	
```

--------------------------------

### Verify Dashboard ConfigMaps

Source: https://github.com/openebs/monitoring/blob/develop/docs/guide.md

Use kubectl to list ConfigMaps in the specified namespace, filtered by the release label.

```bash
kubectl get cm -n prometheus-operator -l release="monitoring"
NAME                                                           DATA   AGE
monitoring-openebs-diskpool                          	 1      37m
```

--------------------------------

### Configure Monitoring Addon in values.yaml

Source: https://github.com/openebs/monitoring/blob/develop/contribute/release.md

Update the `values.yaml` file to enable the OpenEBS monitoring addon and its components (dashboards, alert rules) for a specific service, such as Mayastor.

```yaml
. . .
openebsMonitoringAddon:
  mayastor:
    enabled: true
    dashboards:
      enabled: true
    alertRules:
      enabled: true
. . .

```

--------------------------------

### CSI Volume Metrics (kubelet)

Source: https://context7.com/openebs/monitoring/llms.txt

Metrics exposed by kubelet for CSI volumes, including capacity, available space, used space, and inode information.

```promql
# CSI Volume Metrics (exposed by kubelet)
kubelet_volume_stats_capacity_bytes    # Total capacity of the volume
kubelet_volume_stats_available_bytes   # Available space in the volume
kubelet_volume_stats_used_bytes        # Used space in the volume
kubelet_volume_stats_inodes            # Total inodes in the volume
kubelet_volume_stats_inodes_free       # Free inodes in the volume
kubelet_volume_stats_inodes_used       # Used inodes in the volume
```

--------------------------------

### OpenEBS Monitoring Directory Structure

Source: https://github.com/openebs/monitoring/blob/develop/contribute/design.md

This snippet displays the directory structure for the main OpenEBS monitoring project, highlighting key configuration and addon files.

```directory
jsonnet
├── addons # monitoring addon
│   ├── podMonitors.libsonnet # podMonitor to be created for different cas types
│   ├── prometheusRules.libsonnet # create prometheusRule object which includes alert rules defined in openebs-mixin
│   └── serviceMonitors.libsonnet # serviceMonitor to be created for different cas types
|
├── config.libsonnet # configuration to be used for openebs monitoing
|
├── jsonnetfile.json # direct dependencies
├── jsonnetfile.lock.json # all dependencies with exact version
|
├── lib # contains code that is used in different places
│   └── utils.libsonnet # code which is used to create monitoring addons
|
├── main.jsonnet # entrypoint of code(openEBS Monitoring)
├── Makefile
|
└── openebs-mixin # contains grafanaDashboards and prometheusRules

```

--------------------------------

### values.yaml with Ingress and Storage Enabled

Source: https://github.com/openebs/monitoring/blob/develop/docs/extends-stack-with-umbrella-chart.md

Configure ingress for accessing monitoring UIs and specify a storage class for Prometheus persistent data. Ensure the 'sc-metrics' storage class exists in your cluster.

```yaml
monitoring:
  install: true

  ## Configuration for kube-prometheus-stack subchart
  kube-prometheus-stack:
# PROMETHEUS SECTION
    prometheus:
      ingress:
        enabled: true
        annotations:
          nginx.ingress.kubernetes.io/ssl-redirect: "false"
        paths:
          - /prometheus(/|$)(.*)
        hosts:
          - dev.dynamic.cluster109.local

      prometheusSpec:
        ## Prefix used to register routes, overriding externalUrl route.
        ## Useful for proxies that rewrite URLs.
        ##
        routePrefix: /prometheus

        ## Prometheus StorageSpec for persistent data
        ## ref: https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/storage.md
        ##
        storageSpec:
          volumeClaimTemplate:
            metadata:
              name: prom
            spec:
              storageClassName: sc-metrics
              accessModes: [ "ReadWriteOnce" ]
              resources:
                requests:
                  storage: 8Gi
```

--------------------------------

### Port-Forward to Grafana Pod

Source: https://github.com/openebs/monitoring/blob/develop/README.md

Establishes a port-forwarding connection from your local machine to the Grafana pod. This allows access to Grafana via localhost if a public IP is not available. Replace [NAMESPACE], [grafana-pod-name], [grafana-foward-port], and [grafana-cluster-port] with appropriate values.

```bash
kubectl port-forward --namespace [NAMESPACE] pods/[grafana-pod-name] [grafana-foward-port]:[grafana-cluster-port]
```

--------------------------------

### Check Grafana Pod Status

Source: https://github.com/openebs/monitoring/blob/develop/README.md

Verifies that the Grafana pod is running within the specified namespace. Replace [NAMESPACE] with your actual namespace.

```bash
kubectl get pods -n [NAMESPACE] | grep -i grafana
```

--------------------------------

### Import and Include New Alerts in rules.libsonnet

Source: https://github.com/openebs/monitoring/blob/develop/contribute/alerts.md

Import the new alert rules and append their groups to the 'prometheusRules' object within the rules.libsonnet file.

```libsonnet
. . .
	local newAlertRules = (import './new-alert-rules.libsonnet');

	prometheusRules+::
		. . .
		+ newAlertRules(prometheusRules._config).prometheusRules.newAlertRules.groups,
	. . .
	
```

--------------------------------

### Add ServiceMonitor for Mayastor

Source: https://github.com/openebs/monitoring/blob/develop/contribute/service-monitor.md

Configure a ServiceMonitor for the Mayastor CAS type by adding an entry to the `openebsMonitoringAddon` configuration in `config.libsonnet`. Ensure the `serviceMonitor` function is called with appropriate parameters for port, path, and selector name.

```jsonnet
. . .
_config+:: {
. . .
// Configuration for different cas types.
openebsMonitoringAddon: {
mayastor: {
serviceMonitor: serviceMonitor(<PORTt>, <PATH>, <SELECTOR_NAME>) {
enabled: true,
endpoints+: {
. . .
}
}
}
}
. . .

```

--------------------------------

### Configure Alert Rules Inclusion in Libsonnet

Source: https://github.com/openebs/monitoring/blob/develop/contribute/alerts.md

Enable the inclusion of new alert rules by setting the 'newAlertRules' parameter to true in the alertRules configuration object.

```libsonnet
. . . 
	// // AlertRules configuration. If set, then alert rules are included in the prometheusRules object.
	alertRules: {
		newAlertRules: true # to include alert rules in prometheusRules object
	},
	. . . 
	
```

--------------------------------

### Helm Chart Helper Templates

Source: https://github.com/openebs/monitoring/blob/develop/docs/extends-stack-with-umbrella-chart.md

These are standard Helm chart helper templates used for generating names, namespaces, and labels within the monitoring chart. They ensure consistent naming conventions and facilitate dynamic configuration.

```yaml
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "monitoring.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 50 | trimSuffix "-" }}
{{- end }}

{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "monitoring.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 26 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 26 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 26 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Allow the release namespace to be overridden for multi-namespace deployments in combined charts
*/}}
{{- define "monitoring.namespace" -}}
  {{- if .Values.namespaceOverride -}}
    {{- .Values.namespaceOverride -}}
  {{- else -}}
    {{- .Release.Namespace -}}
  {{- end -}}
{{- end -}}


{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "monitoring.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Common labels
*/}}
{{- define "monitoring.labels" -}}
helm.sh/chart: {{ include "monitoring.chart" . }}
{{ include "monitoring.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

{{/*
Selector labels
*/}}
{{- define "monitoring.selectorLabels" -}}
app.kubernetes.io/name: {{ include "monitoring.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
release: {{ $.Release.Name | quote }}
{{- end }}

{{/*
Template to support multiple levels of sub-charts

Call a template from the context of a subchart.

Usage:
  {{ include "call-nested" (list . "<subchart_name>" "<subchart_template_name>") }}
*/}}
{{- define "call-nested" }}
{{- $dot := index . 0 }}
{{- $subchart := index . 1 | splitList "." }}
{{- $template := index . 2 }}
{{- $values := $dot.Values }}
{{- range $subchart }}
{{- $values = index $values . }}
{{- end }}
{{- include $template (dict "Chart" (dict "Name" (last $subchart)) "Values" $values "Release" $dot.Release "Capabilities" $dot.Capabilities) }}
{{- end }}
```

--------------------------------

### Access Grafana Dashboard via NodePort

Source: https://github.com/openebs/monitoring/blob/develop/deploy/charts/templates/NOTES.txt

Provides instructions to access Grafana using a NodePort service. It outputs a URL with the node's public IP and the assigned NodePort. A note advises using a public IP. It also includes commands for port-forwarding to access Grafana on localhost.

```bash
export NODE_PORT={{ index $.Values "kube-prometheus-stack" "grafana" "service" "nodePort" }}
export NODE_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "call-nested" (list . "kube-prometheus-stack.grafana" "grafana.name") }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].spec.nodeName}")
export NODE_IP=$(kubectl get node $NODE_NAME -o jsonpath='{.status.addresses[?(@.type=="ExternalIP")].address}')
echo http://$NODE_IP:$NODE_PORT
NOTE: The above IP should be a public IP
echo 
export SVC_NAME=$(kubectl -n {{ .Release.Namespace }} get svc -l "app.kubernetes.io/name={{ include "call-nested" (list . "kube-prometheus-stack.grafana" "grafana.name") }},app.kubernetes.io/instance={{ .Release.Name }}" -o name)
kubectl --namespace {{ .Release.Namespace }} port-forward $SVC_NAME 8080:80
echo "Visit http://127.0.0.1:8080"
```

--------------------------------

### Node Problem Detector Metrics

Source: https://context7.com/openebs/monitoring/llms.txt

PromQL queries for monitoring node-level problems detected by Node Problem Detector. These metrics track kernel, filesystem, and Docker issues.

```promql
# Kernel Monitor Metrics
problem_counter{reason="OOMKilling"}           # Out of memory kills
problem_counter{reason="TaskHung"}             # Tasks blocked for extended periods
problem_counter{reason="UnregisterNetDevice"}  # Network device issues
problem_counter{reason="KernelOops"}           # Kernel oops errors
problem_counter{reason="Ext4Error"}            # EXT4 filesystem errors
problem_counter{reason="Ext4Warning"}          # EXT4 filesystem warnings
problem_counter{reason="IOError"}              # Buffer I/O errors
problem_counter{reason="MemoryReadError"}      # Memory read errors
problem_counter{reason="AUFSUmountHung"}       # AUFS unmount hung
problem_counter{reason="DockerHung"}           # Docker process hung
problem_counter{reason="FilesystemIsReadOnly"} # Filesystem remounted read-only

# Docker Monitor Metrics
problem_counter{reason="CorruptDockerImage"}           # Corrupt Docker images
problem_counter{reason="CorruptDockerOverlay2"}        # Corrupt overlay2 storage
problem_counter{reason="DockerContainerStartupFailure"} # Container startup failures

# Example queries
# Total OOM kills across all nodes
sum(problem_counter{reason="OOMKilling"})

# Nodes with filesystem errors
problem_counter{reason=~"Ext4Error|IOError"} > 0

# Docker-related problems by node
sum by (instance) (problem_counter{reason=~".*Docker.*"})
```

--------------------------------

### PrometheusRule Resource Creation

Source: https://github.com/openebs/monitoring/blob/develop/docs/extends-stack-with-umbrella-chart.md

This Helm template generates PrometheusRule custom resources for Prometheus alerting. It iterates through JSON files in the 'rules' directory, creating a PrometheusRule object for each, and includes common labels and namespace.

```yaml
{{- $files := .Files.Glob "rules/**.json" }}
  {{- if $files }}
  {{- range $fileName, $fileContents := $files }}
  {{- $ruleName := regexReplaceAll "(^.*/)(.*)\.json$" $fileName "${2}" }}
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: {{ lower (printf "%s-%s" (include "monitoring.name" $) $ruleName) | trunc 63 | trimSuffix "-" }}
namespace: {{ template "monitoring.namespace" $ }}
labels:
  app: {{ template "monitoring.name" $ }}
  {{ include "monitoring.labels" $ | indent 4 }}
spec:
  {{ $.Files.Get $fileName | indent 2 }}
---
{{- end }}
  {{- end }}
```

--------------------------------

### Update Jsonnet Dependencies

Source: https://github.com/openebs/monitoring/blob/develop/contribute/release.md

Run `jb update` in the `jsonnet` directory to update external dependencies used by the Jsonnet files, such as `kube-prometheus`.

```bash
cd jsonnet
jb update
```

--------------------------------

### Generate Manifests for kube-prometheus Stack with OpenEBS Addons

Source: https://github.com/openebs/monitoring/blob/develop/docs/openebs-mixin-user-guide.md

This Jsonnet code generates Kubernetes manifests for the kube-prometheus stack, including OpenEBS-specific ServiceMonitors, PodMonitors, and alert rules. It assumes `config.libsonnet` is imported and configured.

```jsonnet
. . .
local config = import './config.libsonnet';
local openebsMixin = config.openebsMixin; 

local kp =
    config.kp {
		values+:: {
	    	grafana+:: {
		    	dashboards+: openebsMixin.grafanaDashboards,
		},
};

// kube-prometheus stack
{	'setup/0namespace-namespace': kp.kubePrometheus.namespace }
{
	['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
	for name in std.filter((function(name) name != 'serviceMonitor' && name != 'prometheusRule'), std.objectFields(kp.prometheusOperator))	
} +
// serviceMonitor and prometheusRule are separated so that they can be created after the CRDs are ready
{	'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor }
{	'prometheus-operator-prometheusRule': kp.prometheusOperator.prometheusRule }
{	'kube-prometheus-prometheusRule': kp.kubePrometheus.prometheusRule }
{	['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) }
{	['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }
{	['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) }
{	['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) }
{	['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) }


// servicemonitor, podmonitor and prometheusrules for different casTypes of openebs
{	['openebs-addons/openebs-servicemonitor-' + casType]: serviceMonitor.serviceMonitors[casType] for casType in std.objectFields(serviceMonitor.serviceMonitors) }
{	['openebs-addons/openebs-podmonitor-' + casType]: podMonitors.podMonitors[casType] for casType in std.objectFields(podMonitors.podMonitors) }
// to enable alert rules
{	['openebs-addons/openebs-rule-' + name]: rules.prometheusRules[name] for name in std.objectFields(rules.prometheusRules) }

```

--------------------------------

### Access Grafana Dashboard via ClusterIP

Source: https://github.com/openebs/monitoring/blob/develop/deploy/charts/templates/NOTES.txt

Provides instructions to access Grafana using a ClusterIP service by port-forwarding to a specific local port. It outputs the local URL to access Grafana.

```bash
export PORT= {{ index $.Values "kube-prometheus-stack" "grafana" "service" "port" }}
export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "call-nested" (list . "kube-prometheus-stack.grafana" "grafana.name") }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 32515:$PORT
echo "Visit http://127.0.0.1:32515"
```

--------------------------------

### Access Grafana Dashboard via Ingress

Source: https://github.com/openebs/monitoring/blob/develop/deploy/charts/templates/NOTES.txt

Constructs the Grafana URL when Ingress is enabled. Ensure TLS is configured if specified.

```go-template
{{- if (index $.Values "kube-prometheus-stack" "install") }}
{{- if (index $.Values "kube-prometheus-stack" "grafana" "ingress" "enabled") }}
{{- range $host := (index $.Values "kube-prometheus-stack" "grafana" "ingress" "hosts") }}
http{{ if (index $.Values "kube-prometheus-stack" "grafana" "ingress" "tls") }}s{{ end }}://{{ $host }}{{ regexReplaceAll "\\(.*\\)" (index $.Values "kube-prometheus-stack" "grafana" "ingress" "path") "" }}
{{- end }}
{{- end }}
{{- end }}
```

--------------------------------

### Import New Dashboard JSON

Source: https://github.com/openebs/monitoring/blob/develop/contribute/dashboards.md

Import the new dashboard JSON file into the `grafanaDashboards` object within `dashboards.libsonnet`. This uses a conditional import based on the configuration set in `config.libsonnet`.

```jsonnet
. . .
grafanaDashboards+:: {
	. . .
	[if dashboard._config.dashboards.newDashboard then 'new-dashboard.json']: import './new-dashboard.json',
},
. . .

```

--------------------------------

### Openebs-mixin Directory Structure

Source: https://github.com/openebs/monitoring/blob/develop/contribute/design.md

This snippet outlines the directory structure for the openebs-mixin component, which contains dashboards and alert rules for OpenEBS.

```directory
openebs-mixin # openebs-mixin for grafanaDashboards and prometheusRules.
    ├── build.sh
    ├── config.libsonnet # configuration that is used while writing dashboards and alert rules
    ├── dashboards
    │   ├── openebs/
    │   	├── . . .
    │   ├── dashboards.libsonnet # to import all the dashboard json through grafanaDashboards object.
    |
    ├── jsonnetfile.json # direct dependencies
    |
    ├── lib
    │   ├── dashboards.jsonnet # to generate dashboards for helm charts
    │   └── rules.jsonnet # to generate alert rules for helm charts
    |
    ├── Makefile
    |
    ├── mixin.libsonnet # openebs-mixin that imports grafanaDashboards and prometheusRules
    |
    └── rules # contains alert rules for different cas types
        ├── openebs/
        	├── . . .
        ├── rules.libsonnet # used in openebs monitoring
        └── prometheus-rules.libsonnet # to create prometheusRules object

```

--------------------------------

### LVM Logical Volume Metrics

Source: https://context7.com/openebs/monitoring/llms.txt

Metrics related to LVM logical volumes, including size, usage, snapshot status, health, permissions, and thin pool behavior.

```promql
# Logical Volume Metrics
lvm_lv_total_size_bytes                # LVM LV total size in bytes
lvm_lv_used_percent                    # LVM LV used size in percentage (1-100)
lvm_lv_snap_percent                    # LVM LV snap used size in percentage (1-100)
lvm_lv_health_status                   # LV health: [-1: undefined], [0: ""], [1: partial], [2: refresh needed], [3: mismatches exist]
lvm_lv_permission                      # LV permissions: [-1: undefined], [0: unknown], [1: writeable], [2: read-only], [3: read-only-override]
lvm_lv_mda_total_size_bytes            # LVM LV metadata size in bytes
lvm_lv_mda_used_percent                # LVM LV metadata used size in percentage
lvm_lv_raid_sync_action                # RAID sync action: [-1: undefined], [0: idle], [1: frozen], [2: resync], [3: recover], [4: check], [5: repair]
lvm_lv_when_full                       # Thin pool behavior when full: [-1: undefined], [0: error], [1: queue]
```

--------------------------------

### Check OpenEBS Monitoring Pod Status

Source: https://github.com/openebs/monitoring/blob/develop/deploy/charts/templates/NOTES.txt

Use this command to check the status of OpenEBS monitoring pods in the specified namespace.

```bash
kubectl get pods -n {{ .Release.Namespace }} -o wide
```

--------------------------------

### List of CSI Volume Metrics

Source: https://github.com/openebs/monitoring/blob/develop/docs/metrics-csi.md

These are the raw metric names for CSI volume statistics exposed by kubelet. They provide insights into volume availability, capacity, and inode usage.

```plaintext
kubelet_volume_stats_available_bytes
```

```plaintext
kubelet_volume_stats_capacity_bytes
```

```plaintext
kubelet_volume_stats_inodes
```

```plaintext
kubelet_volume_stats_inodes_free
```

```plaintext
kubelet_volume_stats_inodes_used
```

```plaintext
kubelet_volume_stats_used_bytes
```

--------------------------------

### Integrate Mixin with Existing Prometheus Stack

Source: https://context7.com/openebs/monitoring/llms.txt

Adds OpenEBS dashboards and alert rules to an existing kube-prometheus stack by importing the openebs-mixin and configuring Grafana values.

```jsonnet
// Import openebs-mixin into existing prometheus stack
local openebsMixin = (import 'kube-prometheus/lib/mixin.libsonnet')({
  name: 'openebsMixin',
  mixin: (import './openebs-mixin/mixin.libsonnet') {
    _config+:: {
      dashboards+: {
        mayastor: true,
        lvmLocalPV: true,
        zfsLocalPV: true,
      },
      alertRules+: {
        lvmLocalPV: true,
        volume: true,
      },
    },
  },
});

// Add dashboards to Grafana values
local kp = (import 'kube-prometheus/main.libsonnet') + {
  values+:: {
    grafana+: {
      // Include OpenEBS dashboards
      dashboards+: openebsMixin.grafanaDashboards,
    },
  },
};

// Generate all manifests including OpenEBS prometheus rules
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +
// Add OpenEBS prometheus rules as separate manifest
{ 'openebs-addons/openebs-mixin-prometheus-rules': openebsMixin.prometheusRules }
```

--------------------------------

### LVM Volume Group Metrics

Source: https://context7.com/openebs/monitoring/llms.txt

Metrics for LVM volume groups, covering total and free size, counts of LVs and PVs, snapshot information, and allocation policies.

```promql
# Volume Group Metrics
lvm_vg_total_size_bytes                # LVM VG total size in bytes
lvm_vg_free_size_bytes                 # LVM VG free size in bytes
lvm_vg_lv_count                        # Number of LVs in VG
lvm_vg_pv_count                        # Number of PVs in VG
lvm_vg_missing_pv_count                # Number of PVs in VG which are missing
lvm_vg_snap_count                      # Number of snapshots in VG
lvm_vg_max_lv_count                    # Maximum number of LVs allowed in VG (0 if unlimited)
lvm_vg_max_pv_count                    # Maximum number of PVs allowed in VG (0 if unlimited)
lvm_vg_mda_count                       # Number of metadata areas on this VG
lvm_vg_mda_used_count                  # Number of metadata areas in use on this VG
lvm_vg_mda_total_size_bytes            # Size of smallest metadata area for this VG in bytes
lvm_vg_mda_free_size_bytes             # Free metadata area space for this VG in bytes
lvm_vg_allocation_policy               # VG allocation policy: [-1: undefined], [0: normal], [1: contiguous], [2: cling], [3: anywhere], [4: inherited]
lvm_vg_permission                      # VG permissions: [-1: undefined], [0: writeable], [1: read-only]
```

--------------------------------

### Automate DCO Signing with Git Alias

Source: https://github.com/openebs/monitoring/blob/develop/CONTRIBUTING.md

Configure a Git alias to automatically sign your commits with the DCO. This simplifies the commit process by using `git ci` instead of `git commit -s`.

```git
git config --global alias.ci 'commit -s'
```

--------------------------------

### OpenEBS Mixin Configuration

Source: https://github.com/openebs/monitoring/blob/develop/jsonnet/openebs-mixin/README.md

Configure the OpenEBS mixin by setting casTypes, dashboards, and alertRules. This configuration determines which dashboards and alert rules are generated.

```jsonnet
{
  // openebs-mixin configurations
  _config+:: {
    // Configuration to set which cas types is installed. Based on this, dashboards and alert rules configuration will be set.
    casTypes: {
      mayastor: true,
      lvmLocalPV: true,
      zfsLocalPV: true,
    },
    // dashboards configuration. If set, then dashboards json will be generated.
    dashboards: {
      mayastor: $._config.casTypes.mayastor
      lvmLocalPV: $._config.casTypes.lvmLocalPV,
      zfsLocalPV: $._config.casTypes.zfsLocalPV,
      npd: true,
    },
    // AlertRules configuration. If set, then rules json will be generated.
    alertRules: {
      lvmLocalPV: $._config.casTypes.lvmLocalPV,
      volume: $._config.casTypes.lvmLocalPV,
      npd: true,
    },
  },
}
```

--------------------------------

### Umbrella Chart Folder Structure

Source: https://github.com/openebs/monitoring/blob/develop/docs/extends-stack-with-umbrella-chart.md

This is the basic folder structure for an umbrella chart. It includes a Chart.yaml for metadata and a values.yaml for configuration.

```yaml
📦umbrella
┣ 📜Chart.yaml
┗ 📜values.yaml
```

--------------------------------

### Docker Monitor Metrics

Source: https://github.com/openebs/monitoring/blob/develop/docs/metrics-node-problem-detector.md

Metrics generated by the docker-monitor component, useful for identifying issues related to Docker images and containers. These metrics help in debugging containerization problems.

```text
problem_counter{reason="CorruptDockerImage"} # system logs with pattern- "Error trying v2 registry: failed to register layer: rename /var/lib/docker/image/(.+) /var/lib/docker/image/(.+): directory not empty._"
```

```text
problem_counter{reason="CorruptDockerOverlay2"} # system logs with pattern- "returned error: readlink /var/lib/docker/overlay2.: invalid argument.*"
```

```text
problem_counter{reason="DockerContainerStartupFailure"} # system logs with pattern- "OCI runtime start failed: container process is already dead: unknown"
```