### Start minikube with specific configuration

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/README.md

This command starts a minikube instance with specific configurations for container runtime, Kubernetes version, memory allocation, and kubelet/scheduler settings. It ensures a suitable environment for testing the kube-prometheus stack. The command first deletes any existing minikube instance before starting a new one.

```shell
minikube delete && minikube start --container-runtime=containerd --kubernetes-version=v1.33.1 --memory=6g --bootstrapper=kubeadm --extra-config=kubelet.authentication-token-webhook=true --extra-config=kubelet.authorization-mode=Webhook --extra-config=scheduler.bind-address=0.0.0.0 --extra-config=controller-manager.bind-address=0.0.0.0
```

--------------------------------

### Install kube-prometheus Library with jsonnet-bundler

Source: https://context7.com/prometheus-operator/kube-prometheus/llms.txt

This snippet details the steps to initialize a new project, install the kube-prometheus jsonnet library using jsonnet-bundler, and download necessary example files and build tools. It sets up the project structure for configuration and manifest generation.

```bash
# Create project directory structure
mkdir my-kube-prometheus && cd my-kube-prometheus

# Initialize jsonnet-bundler (creates jsonnetfile.json)
jb init

# Install kube-prometheus library from main branch
jb install github.com/prometheus-operator/kube-prometheus/jsonnet/kube-prometheus@main

# Download example files
wget https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/example.jsonnet -O example.jsonnet
wget https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/build.sh -O build.sh
chmod +x build.sh

# Install required build tools
go install github.com/google/go-jsonnet/cmd/jsonnet@latest
go install github.com/brancz/gojsontoyaml@latest

# Expected directory structure:
# my-kube-prometheus/
# ├── jsonnetfile.json       (dependency manifest)
# ├── jsonnetfile.lock.json  (locked versions)
# ├── vendor/                (installed libraries)
# ├── example.jsonnet        (configuration file)
# └── build.sh               (build script)
```

--------------------------------

### Install Kube-Prometheus with jsonnet-bundler

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizing.md

Installs the kube-prometheus library as a dependency in a new project using jsonnet-bundler. This involves initializing jsonnet-bundler, installing the specific library version, and downloading example configuration files.

```shell
#!/usr/bin/env bash

mkdir my-kube-prometheus; cd my-kube-prometheus
jb init  # Creates the initial/empty `jsonnetfile.json`
# Install the kube-prometheus dependency
jb install github.com/prometheus-operator/kube-prometheus/jsonnet/kube-prometheus@main # Creates `vendor/` & `jsonnetfile.lock.json`, and fills in `jsonnetfile.json`

wget https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/example.jsonnet -O example.jsonnet
wget https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/build.sh -O build.sh
chmod +x build.sh

```

--------------------------------

### Example of Importing and Using an Addon in Jsonnet

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/migration-guide.md

Demonstrates how to import the main kube-prometheus library and then merge it with an addon, such as 'all-namespaces.libsonnet'. This pattern is used to enhance the base deployment with additional features provided by addons.

```jsonnet
local kp = (import 'kube-prometheus/main.libsonnet') + (import 'kube-prometheus/addons/all-namespaces.libsonnet')
```

--------------------------------

### Kube-Prometheus Example Configuration (jsonnet)

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizing.md

An example jsonnet file demonstrating how to configure and import the kube-prometheus library. It includes options for enabling various addons and setting common values like the namespace.

```jsonnet
local kp =
  (import 'kube-prometheus/main.libsonnet') +
  // Uncomment the following imports to enable its patches
  // (import 'kube-prometheus/addons/anti-affinity.libsonnet') +
  // (import 'kube-prometheus/addons/managed-cluster.libsonnet') +
  // (import 'kube-prometheus/addons/node-ports.libsonnet') +
  // (import 'kube-prometheus/addons/static-etcd.libsonnet') +
  // (import 'kube-prometheus/addons/custom-metrics.libsonnet') +
  // (import 'kube-prometheus/addons/external-metrics.libsonnet') +
  // (import 'kube-prometheus/addons/pyrra.libsonnet') +
  {
    values+:: {
      common+: {
        namespace: 'monitoring',
      },
    },
  };

{ 'setup/0namespace-namespace': kp.kubePrometheus.namespace } + { ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.filter((function(name) name != 'serviceMonitor' && name != 'prometheusRule'), std.objectFields(kp.prometheusOperator)) } + // { 'setup/pyrra-slo-CustomResourceDefinition': kp.pyrra.crd } + // serviceMonitor and prometheusRule are separated so that they can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } + { 'prometheus-operator-prometheusRule': kp.prometheusOperator.prometheusRule } + { 'kube-prometheus-prometheusRule': kp.kubePrometheus.prometheusRule } + { ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } + { ['blackbox-exporter-' + name]: kp.blackboxExporter[name] for name in std.objectFields(kp.blackboxExporter) } + { ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } + // { ['pyrra-' + name]: kp.pyrra[name] for name in std.objectFields(kp.pyrra) if name != 'crd' } + { ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } + { ['kubernetes-' + name]: kp.kubernetesControlPlane[name] for name in std.objectFields(kp.kubernetesControlPlane) }
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } + { ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } + { ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter)
}

```

--------------------------------

### Install Kube-Prometheus Stack using kubectl

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizing.md

This shell command sequence applies the generated Kubernetes manifests for the kube-prometheus stack. It first applies setup configurations and CRDs, then applies the main manifests. It highlights the use of server-side apply (`--server-side`) for compatibility with Kubernetes 1.22+ and suggests using `kubectl create` for older versions. A consolidated command is also provided.

```shell
# Update the namespace and CRDs, and then wait for them to be available before creating the remaining resources
$ kubectl apply --server-side -f manifests/setup
$ kubectl apply -f manifests/

# Alternatively, the resources in both folders can be applied with a single command
# `kubectl apply --server-side -Rf manifests`, but it may be necessary to run the command multiple times for all components to be created successfully.
```

--------------------------------

### Commit Message Format Example (Shell)

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/CONTRIBUTING.md

An example demonstrating the conventional format for commit messages, including subsystem, a brief description of the change, and the reasoning behind it. This format helps in understanding the changes made.

```shell
scripts: add the test-cluster command

this uses tmux to setup a test cluster that you can easily kill and
start for debugging.

Fixes #38
```

--------------------------------

### Install Latest 'jb' Binary using Go

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/update.md

Installs the latest version of the 'jb' (jsonnet-bundler) binary using the Go toolchain. Ensure Go is installed and configured in your PATH.

```shell
go install -a github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest
```

--------------------------------

### Install Grafana

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/kube-prometheus-on-kubeadm.md

Applies the Grafana deployment manifests. This installs the Grafana visualization tool within the monitoring namespace.

```bash
kubectl --namespace="$NAMESPACE" apply -f manifests/grafana

```

--------------------------------

### Full Example: Including Etcd Mixin in Kube-Prometheus

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/developing-prometheus-rules-and-grafana-dashboards.md

This comprehensive example shows how to include the etcd mixin within a kube-prometheus setup. It defines the etcd mixin, configures it, and then integrates its Prometheus rules and Grafana dashboards into the main kube-prometheus configuration. This includes setting the namespace and rendering various components like Prometheus Operator, Alertmanager, and Grafana.

```jsonnet
local addMixin = (import 'kube-prometheus/lib/mixin.libsonnet');
local etcdMixin = addMixin({
  name: 'etcd',
  mixin: (import 'github.com/etcd-io/etcd/contrib/mixin/mixin.libsonnet') + {
    _config+: {},  // mixin configuration object
  },
});

local kp = (import 'kube-prometheus/main.libsonnet') + 
           {
             values+:: {
               common+: {
                 namespace: 'monitoring',
               },
               grafana+: {
                 // Adding new dashboard to grafana. This will modify grafana configMap with dashboards
                 dashboards+: etcdMixin.grafanaDashboards,
               },
             },
           };

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +
// Rendering prometheusRules object. This is an object compatible with prometheus-operator CRD definition for prometheusRule
{ 'external-mixins/etcd-mixin-prometheus-rules': etcdMixin.prometheusRules }
```

--------------------------------

### Component Configuration Defaults in Jsonnet

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/migration-guide.md

Shows an example of how default parameters for a component, like `node_exporter`, are defined using a `defaults` map at the top of the library file. This API allows users to override or set required parameters.

```jsonnet
// Defaults for node_exporter component
local defaults = {
  // ... default parameters ...
};

```

--------------------------------

### Clone Kube-Prometheus Repository

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/kube-prometheus-on-kubeadm.md

Clones the kube-prometheus repository from GitHub to your local machine. This is the first step to access the necessary manifests for installation.

```bash
git clone https://github.com/prometheus-operator/kube-prometheus
cd kube-prometheus/

```

--------------------------------

### Install Node Exporter and Kube-State-Metrics

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/kube-prometheus-on-kubeadm.md

Applies the manifests for node-exporter and kube-state-metrics. These components collect cluster and node-level metrics, respectively.

```bash
kubectl --namespace="$NAMESPACE" apply -f manifests/node-exporter
kubectl --namespace="$NAMESPACE" apply -f manifests/kube-state-metrics

```

--------------------------------

### Prometheus etcd Job Configuration Example

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/monitoring-external-etcd.md

This snippet shows an example configuration for an etcd job within the Prometheus UI. It specifies scrape intervals and timeouts, essential for monitoring etcd metrics. Ensure your Prometheus instance is configured to scrape these metrics.

```yaml
- job_name: monitoring/etcd-k8s/0
  scrape_interval: 30s
  scrape_timeout: 10s
  ...
```

--------------------------------

### Jsonnet: Import and Configuration Structure Comparison (kube-prometheus)

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/migration-example/readme.md

Compares the import statements and configuration structure in release-0.3 and release-0.8 of kube-prometheus jsonnet files. It shows how libraries are imported and how configuration values like namespace are defined, with release-0.8 utilizing a `values` block.

```jsonnet
local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-kubeadm.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-static-etcd.libsonnet') +

  {
    _config+:: {
      // Override namespace
      namespace: 'monitoring',
  
  
  
  
   
   
   

```

```jsonnet
local kp =
  (import 'kube-prometheus/main.libsonnet') +
  // kubeadm now achieved by setting platform value - see 9 lines below
  (import 'kube-prometheus/addons/static-etcd.libsonnet') +
  (import 'kube-prometheus/addons/podsecuritypolicies.libsonnet') +
  {
    values+:: {
      common+: {
        namespace: 'monitoring',
      },

      // Add kubeadm platform-specific items,
      // including kube-contoller-manager and kube-scheduler discovery
      kubePrometheus+: {
        platform: 'kubeadm',
      },

```

--------------------------------

### Deploy kube-prometheus Stack using kubectl

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/README.md

This command sequence applies the necessary Kubernetes manifests to set up the monitoring stack. It first creates the namespace and CustomResourceDefinitions (CRDs), waits for CRDs to be established, and then applies the remaining resources. Server-side apply, available since Kubernetes 1.22, is used for efficient resource management.

```shell
kubectl apply --server-side -f manifests/setup
kubectl wait \
    --for condition=Established \
    --all CustomResourceDefinition \
    --namespace=monitoring
kubectl apply -f manifests/
```

--------------------------------

### Install Alertmanager

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/kube-prometheus-on-kubeadm.md

Applies the Alertmanager manifests to the monitoring namespace. This deploys the Alertmanager component for handling alerts.

```bash
kubectl --namespace="$NAMESPACE" apply -f manifests/alertmanager

```

--------------------------------

### Configure kube-state-metrics Resource Allocation (Jsonnet)

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/troubleshooting.md

This snippet shows how to configure resource allocation for kube-state-metrics using Jsonnet. It allows setting base CPU and memory, as well as per-node CPU and memory. These settings are managed by the addon-resizer.

```jsonnet
    kubeStateMetrics+:: {
      baseCPU: '100m',
      cpuPerNode: '2m',
      baseMemory: '150Mi',
      memoryPerNode: '30Mi',
    }
```

--------------------------------

### Deploy Kube-Prometheus on Minikube with NodePort Services

Source: https://context7.com/prometheus-operator/kube-prometheus/llms.txt

This section details setting up kube-prometheus on a local minikube instance. It includes commands to start minikube with specific configurations, disable conflicting addons, and export the minikube IP. The subsequent Jsonnet snippet configures the deployment for NodePort access.

```bash
# Start minikube with required configurations
minikube delete && minikube start \
  --container-runtime=containerd \
  --kubernetes-version=v1.33.1 \
  --memory=6g \
  --bootstrapper=kubeadm \
  --extra-config=kubelet.authentication-token-webhook=true \
  --extra-config=kubelet.authorization-mode=Webhook \
  --extra-config=scheduler.bind-address=0.0.0.0 \
  --extra-config=controller-manager.bind-address=0.0.0.0

# Disable metrics-server (conflicts with prometheus-adapter)
minikube addons disable metrics-server

# Get minikube IP for external URLs
export MINIKUBE_IP=$(minikube ip)
echo "Minikube IP: $MINIKUBE_IP"
```

--------------------------------

### Override Component Namespaces and Resource Names

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/components-name-namespace-overrides.md

This example demonstrates overriding namespaces and resource names for individual components like Prometheus and Alertmanager. It also shows how to configure other components and includes setup configurations.

```jsonnet
local kp = (import 'kube-prometheus/main.libsonnet') +
           {
             values+:: {
               common+: {
                 namespace: 'monitoring',
               },

               prometheus+: {
                 namespace: 'foo',
                 name: 'bar',
               },

               alertmanager+: {
                 namespace: 'bar',
                 name: 'foo',
               },
             },
           };

{ 'setup/0namespace-namespace': kp.kubePrometheus.namespace } +
// Add the restricted psp to setup
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor' && name != 'prometheusRule'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor and prometheusRule are separated so that they can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ 'prometheus-operator-prometheusRule': kp.prometheusOperator.prometheusRule } +
{ 'kube-prometheus-prometheusRule': kp.kubePrometheus.prometheusRule } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['blackbox-exporter-' + name]: kp.blackboxExporter[name] for name in std.objectFields(kp.blackboxExporter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['kubernetes-' + name]: kp.kubernetesControlPlane[name] for name in std.objectFields(kp.kubernetesControlPlane) }
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) }
```

--------------------------------

### Install gojsontoyaml for Rule Conversion

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/developing-prometheus-rules-and-grafana-dashboards.md

Installs the gojsontoyaml tool, which is used to convert YAML Prometheus rules to JSON format for easier integration.

```bash
go get -u -v github.com/brancz/gojsontoyaml
```

--------------------------------

### Deploy Kube Prometheus Stack with Kubectl

Source: https://context7.com/prometheus-operator/kube-prometheus/llms.txt

These commands apply the setup manifests, wait for CRDs to be established, deploy the monitoring stack components, and verify the deployment by listing pods in the monitoring namespace. Finally, it provides a command to tear down the stack.

```bash
kubectl apply --server-side -f manifests/setup
kubectl wait \
    --for condition=Established \
    --all CustomResourceDefinition \
    --namespace=monitoring

kubectl apply -f manifests/

kubectl get pods -n monitoring

# Expected output:
# NAME                                   READY   STATUS    RESTARTS   AGE
# prometheus-operator-xxx                2/2     Running   0          2m
# prometheus-k8s-0                       2/2     Running   0          1m
# prometheus-k8s-1                       2/2     Running   0          1m
# alertmanager-main-0                    2/2     Running   0          1m
# grafana-xxx                            1/1     Running   0          1m
# node-exporter-xxx                      2/2     Running   0          1m
# kube-state-metrics-xxx                 3/3     Running   0          1m

# Teardown stack
kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup
```

--------------------------------

### Teardown kube-prometheus Stack using kubectl

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/README.md

This command removes the monitoring stack resources previously deployed. It deletes resources from both the `manifests/` and `manifests/setup` directories, ignoring any resources that may have already been deleted.

```shell
kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup
```

--------------------------------

### Configure kube-prometheus with Custom Rules and Dashboards using Jsonnet

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/developing-prometheus-rules-and-grafana-dashboards.md

This example demonstrates how to configure kube-prometheus using Jsonnet to include custom Prometheus rules and Grafana dashboards. It involves importing the main kube-prometheus library and conditionally including various addons. The configuration allows for namespace definition and specific component setup, including serviceMonitors and prometheusRules.

```jsonnet
local kp =
  (import 'kube-prometheus/main.libsonnet') +
  // Uncomment the following imports to enable its patches
  // (import 'kube-prometheus/addons/anti-affinity.libsonnet') +
  // (import 'kube-prometheus/addons/managed-cluster.libsonnet') +
  // (import 'kube-prometheus/addons/node-ports.libsonnet') +
  // (import 'kube-prometheus/addons/static-etcd.libsonnet') +
  // (import 'kube-prometheus/addons/custom-metrics.libsonnet') +
  // (import 'kube-prometheus/addons/external-metrics.libsonnet') +
  // (import 'kube-prometheus/addons/pyrra.libsonnet') +
  {
    values+:: {
      common+: {
        namespace: 'monitoring',
      },
    },
  };

{ 'setup/0namespace-namespace': kp.kubePrometheus.namespace } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor' && name != 'prometheusRule'), std.objectFields(kp.prometheusOperator))
} +
// { 'setup/pyrra-slo-CustomResourceDefinition': kp.pyrra.crd } +
// serviceMonitor and prometheusRule are separated so that they can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ 'prometheus-operator-prometheusRule': kp.prometheusOperator.prometheusRule } +
{ 'kube-prometheus-prometheusRule': kp.kubePrometheus.prometheusRule } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['blackbox-exporter-' + name]: kp.blackboxExporter[name] for name in std.objectFields(kp.blackboxExporter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +
// { ['pyrra-' + name]: kp.pyrra[name] for name in std.objectFields(kp.pyrra) if name != 'crd' } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['kubernetes-' + name]: kp.kubernetesControlPlane[name] for name in std.objectFields(kp.kubernetesControlPlane) }
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) }
```

--------------------------------

### Jsonnet: Ingress Configuration Conversion (kube-prometheus)

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/migration-example/readme.md

Demonstrates the conversion of Ingress resource definition in jsonnet for kube-prometheus. It contrasts the older, more programmatic approach in release-0.3 with the more declarative Kubernetes object definition in release-0.8, including metadata and spec details.

```jsonnet
    // Add additional ingresses
    // See https://github.com/coreos/kube-prometheus/...
    //           tree/master/examples/ingress.jsonnet
    ingress+:: {
      alertmanager:
        ingress.new() +


        ingress.mixin.metadata.withName('alertmanager') +
        ingress.mixin.metadata.withNamespace($._config.namespace) +
        ingress.mixin.metadata.withAnnotations({
          'kubernetes.io/ingress.class': 'nginx-api',
        }) +

        ingress.mixin.spec.withRules(
          ingressRule.new() +
          ingressRule.withHost(alert_manager_host) +
          ingressRule.mixin.http.withPaths(
            ingressRuleHttpPath.new() +




            ingressRuleHttpPath.mixin.backend
                               .withServiceName('alertmanager-operated') +
            ingressRuleHttpPath.mixin.backend.withServicePort(9093)
          ),
        ) +
        // Note we do not need a TLS secretName here as we are going to use the
        // nginx-ingress default secret which is a wildcard
        // secretName would need to be in the same namespace at this time,
        // see https://github.com/kubernetes/ingress-nginx/issues/2371
        ingress.mixin.spec.withTls(
          ingressTls.new() +
          ingressTls.withHosts(alert_manager_host)
        ),
  
  

```

```jsonnet
    // Add additional ingresses
    // See https://github.com/prometheus-operator/kube-prometheus/...
    //           blob/main/examples/ingress.jsonnet
    ingress+:: {
      alertmanager: {
        apiVersion: 'networking.k8s.io/v1',
        kind: 'Ingress',
        metadata: {
          name: 'alertmanager',
          namespace: $.values.common.namespace,
          annotations: {
            'kubernetes.io/ingress.class': 'nginx-api',
          },
        },
        spec: {
          rules: [{
            host: alert_manager_host,
            http: {
              paths: [{
                path: '/',
                pathType: 'Prefix',
                backend: {
                  service: {
                    name: 'alertmanager-operated',
                    port: {
                      number: 9093,
                    },
                  },
                },
              }],
            },
          }],
          tls: [{

            hosts: [alert_manager_host],
          }],
        },
      },

```

--------------------------------

### Compile Kube-Prometheus Manifests with jsonnet

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizing.md

Generates Kubernetes manifests by compiling a specified jsonnet file. This process requires the `gojsontoyaml` and `jsonnet` tools to be installed beforehand.

```shell
#!/usr/bin/env bash

# This script uses arg $1 (name of *.jsonnet file to use) to generate the manifests/*.yaml files.

set -e
set -x
# only exit with zero if all commands of the pipeline exit successfully
set -o pipefail

# Make sure to use project tooling
PATH="$(pwd)/tmp/bin:${PATH}"

# Make sure to start with a clean 'manifests' dir
rm -rf manifests
mkdir -p manifests/setup

```

--------------------------------

### Accessing Component-Specific Mixins in Jsonnet

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/migration-guide.md

Illustrates how to access monitoring mixins like `prometheusRules`, `prometheusAlerts`, and `grafanaDashboards` per component using the `mixin` object. This approach replaces the previous global accessibility.

```jsonnet
$.alertmanager.mixin.prometheusAlerts
```

--------------------------------

### Create Probe Resource with YAML

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/blackbox-exporter.md

This YAML defines a Kubernetes `Probe` resource for the monitoring stack. It specifies the interval for checks, the module to use (e.g., http_2xx), the blackbox-exporter endpoint, and a list of static targets to probe. This resource is typically created after the necessary CRDs and Prometheus Operator are installed.

```yaml
kind: Probe
apiVersion: monitoring.coreos.com/v1
metadata:
  name: example-com-website
  namespace: monitoring
spec:
  interval: 60s
  module: http_2xx
  prober:
    url: blackbox-exporter.monitoring.svc.cluster.local:19115
  targets:
    staticConfig:
      static:
      - http://example.com
      - https://example.com
```

--------------------------------

### Jsonnet: Prometheus Rules Import Conversion (kube-prometheus)

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/migration-example/readme.md

Shows the difference in how Prometheus rules are imported in jsonnet configurations for kube-prometheus. Release-0.3 uses a direct import of a json file, while release-0.8's snippet is incomplete but suggests a similar import mechanism within a `groups` block.

```jsonnet
    // Additional prometheus rules
    // See https://github.com/coreos/kube-prometheus/docs/...
    //           developing-prometheus-rules-and-grafana-dashboards.md
    //
    // cat my-prometheus-rules.yaml |
    //   gojsontoyaml -yamltojson |
    //   jq . > my-prometheus-rules.json
    prometheusRules+:: {







      groups+: import 'my-prometheus-rules.json',


    },
  };
  
  
  
  

```

```jsonnet
    // Additional prometheus rules
    // See https://github.com/prometheus-operator/kube-prometheus/blob/main/...

```

--------------------------------

### Generate Prometheus Resources Dynamically

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/migration-example/readme.md

This code snippet shows how to dynamically generate Prometheus resources based on an object's fields. It iterates through the fields of `kp.prometheusMe` and creates corresponding resources, prepending 'prometheus-my-' to each name. This is often used in templating systems like Jsonnet to manage multiple Prometheus configurations.

```jsonnet
+ { ["prometheus-my-" + name]: kp.prometheusMe[name] for name in std.objectFields(kp.prometheusMe) }
```

--------------------------------

### Disable metrics-server addon in minikube

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/README.md

This command disables the `metrics-server` addon in minikube. The kube-prometheus stack includes its own resource metrics API server, making the minikube addon redundant and potentially conflicting.

```shell
minikube addons disable metrics-server
```

--------------------------------

### Apply Weave Net monitoring configurations with kubectl

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/weave-net-support.md

This section provides the kubectl commands necessary to apply the generated Weave Net monitoring configurations to a Kubernetes cluster. It includes commands for creating services, ServiceMonitors, Prometheus rules, and Grafana dashboards, ensuring all components for comprehensive Weave Net observability are deployed.

```bash
kubectl create -f prometheus-serviceWeaveNet.yaml
kubectl create -f prometheus-serviceMonitorWeaveNet.yaml
kubectl apply -f  prometheus-rules.yaml
kubectl apply -f grafana-dashboardDefinitions.yaml
kubectl apply -f grafana-deployment.yaml
```

--------------------------------

### Alertmanager Configuration Example in Jsonnet

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizing.md

This Jsonnet code snippet illustrates the configuration of the Alertmanager component within the kube-prometheus stack. It shows how Alertmanager inherits default values for its namespace, version, and image from the common configuration (`$.values.common`). It also demonstrates using the `mixin+` syntax for merging additional configurations, such as `ruleLabels`.

```jsonnet
    alertmanager: {
      name: 'main',
      # Use the namespace specified under values.common by default.
      namespace: $.values.common.namespace,
      version: $.values.common.versions.alertmanager,
      image: $.values.common.images.alertmanager,
      mixin+: { ruleLabels: $.values.common.ruleLabels },
    }
```

--------------------------------

### Convert Prometheus Rules YAML to JSON

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/migration-example/readme.md

This command-line snippet demonstrates how to convert Prometheus rule definitions from YAML format to JSON. It uses `gojsontoyaml` to perform the YAML to JSON conversion and `jq` to format the output. This is useful for preparing rule files for import into Kubernetes manifests.

```shell
cat my-prometheus-rules.yaml | \
  gojsontoyaml -yamltojson | \
  jq . > my-prometheus-rules.json
```

--------------------------------

### Update kube-prometheus using 'jb'

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/update.md

Synchronizes the local kube-prometheus project with its upstream repository using the 'jb update' command. This command requires the 'jb' binary to be installed.

```shell
jb update
```

--------------------------------

### Check Volume Expansion Availability (Bash)

Source: https://github.com/prometheus-operator/kube-prometheus/wiki/KubePersistentVolumeFillingUp

This command checks if volume expansion is enabled for a Persistent Volume Claim (PVC) by inspecting its associated StorageClass. It requires kubectl and jq to be installed and assumes you have the correct namespace and PVC name.

```bash
$ kubectl get storageclass `kubectl -n <my-namespace> get pvc <my-pvc> -ojson | jq -r '.spec.storageClassName'`
NAME                 PROVISIONER            RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard (default)   kubernetes.io/gce-pd   Delete          Immediate           true                   28d
```

--------------------------------

### Query Prometheus Metrics via kubectl (Bash)

Source: https://context7.com/prometheus-operator/kube-prometheus/llms.txt

This bash script provides commands to access the Prometheus API through kubectl port-forwarding. It includes examples for querying active targets, specific metrics (like 'up' and CPU usage), time-range queries, checking configuration, and listing available metrics. It also shows how to kill the port-forward process.

```bash
# Port-forward to Prometheus service
kubectl port-forward -n monitoring svc/prometheus-k8s 9090:9090 &

# Query current Prometheus targets
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health, instance: .labels.instance}'

# Query specific metrics
curl -s 'http://localhost:9090/api/v1/query?query=up' | jq '.data.result[] | {metric: .metric.job, value: .value[1]}'

# Query node CPU usage
curl -s 'http://localhost:9090/api/v1/query?query=node_cpu_seconds_total' | jq '.data.result[0]'

# Query with time range
curl -s 'http://localhost:9090/api/v1/query_range?query=rate(node_cpu_seconds_total[5m])&start=2025-10-16T00:00:00Z&end=2025-10-16T01:00:00Z&step=60s' | jq

# Check Prometheus configuration
curl -s http://localhost:9090/api/v1/status/config | jq '.data.yaml' | head -50

# List all available metrics
curl -s http://localhost:9090/api/v1/label/__name__/values | jq '.data' | head -20

# Kill port-forward
pkill -f "port-forward.*prometheus-k8s"
```

--------------------------------

### Define PrometheusRule Kubernetes Resource

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/migration-example/readme.md

This snippet defines a Kubernetes `PrometheusRule` custom resource. It specifies the API version, kind, metadata (name, namespace, labels), and the rule groups imported from a JSON file. This is a core component for deploying Prometheus alerting rules to a cluster managed by Prometheus Operator.

```json
{
  prometheusMe: {
    rules: {
      apiVersion: 'monitoring.coreos.com/v1',
      kind: 'PrometheusRule',
      metadata: {
        name: 'my-prometheus-rule',
        namespace: $.values.common.namespace,
        labels: {
          'app.kubernetes.io/name': 'kube-prometheus',
          'app.kubernetes.io/part-of': 'kube-prometheus',
          prometheus: 'k8s',
          role: 'alert-rules',
        },
      },
      spec: {
        groups: import 'my-prometheus-rules.json',
      },
    },
  },
}
```

--------------------------------

### Configure Mixin Parameters in Jsonnet

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/developing-prometheus-rules-and-grafana-dashboards.md

This example shows how to configure a mixin by merging a configuration object into the '_config+' field of the imported mixin. This allows customization of the mixin's behavior, such as setting a 'myMixinSelector' or an 'interval'. The configuration is applied when defining the mixin object using 'addMixin'.

```jsonnet
local myMixin = addMixin({
  name: 'myMixin',
  mixin: (import 'my-mixin/mixin.libsonnet') + {
    _config+:: {
      myMixinSelector: 'my-selector',
      interval: '30d', // example
    },
  },
});
```

--------------------------------

### Configure Weave Net monitoring with kube-prometheus (Jsonnet)

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/weave-net-support.md

This Jsonnet code snippet configures kube-prometheus to monitor Weave Net. It imports necessary libraries, sets the namespace, and customizes Prometheus rules for alerts like 'WeaveNetFastDPFlowsLow' and 'WeaveNetIPAMUnreachable' by adjusting their expression thresholds. This allows for environment-specific tuning of alerts.

```jsonnet
local kp = (import 'kube-prometheus/main.libsonnet') + (import 'kube-prometheus/addons/weave-net/weave-net.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
  },
  kubernetesControlPlane+: {
    prometheusRuleWeaveNet+: {
      spec+: {
        groups: std.map(
          function(group)
            if group.name == 'weave-net' then
              group {
                rules: std.map(
                  function(rule)
                    if rule.alert == 'WeaveNetFastDPFlowsLow' then
                      rule {
                        expr: 'sum(weave_flows) < 20000',
                      }
                    else if rule.alert == 'WeaveNetIPAMUnreachable' then
                      rule {
                        expr: 'weave_ipam_unreachable_percentage > 25',
                      }
                    else
                      rule
                  ,
                  group.rules
                ),
              }
            else
              group,
          super.groups
        ),
      },
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }
```

--------------------------------

### Add Custom Prometheus Alerting Rules with Jsonnet

Source: https://context7.com/prometheus-operator/kube-prometheus/llms.txt

This Jsonnet code defines custom Prometheus alerting and recording rules. It imports the base kube-prometheus library, sets the namespace, and defines a PrometheusRule resource with example alerts for high error rates and application downtime, as well as a recording rule for request rates.

```jsonnet
local kp = (import 'kube-prometheus/main.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
  },
  // Add custom application monitoring
  exampleApplication: {
    prometheusRuleExample: {
      apiVersion: 'monitoring.coreos.com/v1',
      kind: 'PrometheusRule',
      metadata: {
        name: 'my-application-rules',
        namespace: $.values.common.namespace,
        labels: {
          app: 'my-application',
          role: 'alert-rules',
        },
      },
      spec: {
        groups: [
          {
            name: 'my-application.rules',
            interval: '30s',
            rules: [
              {
                alert: 'HighErrorRate',
                expr: 'rate(http_requests_total{job="my-app",status=~"5.."}[5m]) > 0.05',
                for: '5m',
                labels: {
                  severity: 'warning',
                  team: 'backend',
                },
                annotations: {
                  summary: 'High HTTP error rate detected',
                  description: 'Error rate is {{ $value | humanizePercentage }} for {{ $labels.instance }}',
                },
              },
              {
                alert: 'ApplicationDown',
                expr: 'up{job="my-app"} == 0',
                for: '2m',
                labels: {
                  severity: 'critical',
                  team: 'backend',
                },
                annotations: {
                  summary: 'Application instance is down',
                  description: 'Instance {{ $labels.instance }} has been down for more than 2 minutes',
                },
              },
            ],
          },
          {
            name: 'my-application.recording',
            interval: '30s',
            rules: [
              {
                record: 'job:http_requests:rate5m',
                expr: 'sum by (job) (rate(http_requests_total[5m]))',
              },
            ],
          },
        ],
      },
    },
  },
};

// Include custom rules in manifest generation
{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['example-application-' + name]: kp.exampleApplication[name] for name in std.objectFields(kp.exampleApplication) }
```

--------------------------------

### Create Monitoring Namespace

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/kube-prometheus-on-kubeadm.md

Sets an environment variable for the namespace and creates it using kubectl. This namespace will host all the monitoring components.

```bash
export NAMESPACE='monitoring'
kubectl create namespace "$NAMESPACE"

```

--------------------------------

### Deploy Prometheus Application and Roles

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/kube-prometheus-on-kubeadm.md

Applies the Prometheus application manifests, followed by its roles and role-bindings. This sets up the Prometheus monitoring instance.

```bash
find manifests/prometheus -type f ! -name prometheus-k8s-roles.yaml ! -name prometheus-k8s-role-bindings.yaml -exec kubectl --namespace "$NAMESPACE" apply -f {} \;
kubectl apply -f manifests/prometheus/prometheus-k8s-roles.yaml
kubectl apply -f manifests/prometheus/prometheus-k8s-role-bindings.yaml

```

--------------------------------

### Update Kube-Prometheus dependency

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizing.md

Updates the installed kube-prometheus dependency to the latest available version using the jsonnet-bundler update command.

```shell
jb update

```

--------------------------------

### Deploy Kube-Prometheus Manifests and Access UIs on Minikube

Source: https://context7.com/prometheus-operator/kube-prometheus/llms.txt

This section provides the bash commands to build and deploy the configured kube-prometheus stack on minikube. It includes applying manifests, waiting for CustomResourceDefinitions, and accessing the Prometheus, Alertmanager, and Grafana UIs via NodePorts. It also shows how to retrieve the actual NodePorts assigned to the services.

```bash
# Build and deploy
./build.sh minikube.jsonnet
kubectl apply --server-side -f manifests/setup
kubectl wait --for condition=Established --all CustomResourceDefinition --namespace=monitoring
kubectl apply -f manifests/

# Access UIs (replace with actual minikube IP)
# Prometheus: http://192.168.99.100:30900
# Alertmanager: http://192.168.99.100:30903
# Grafana: http://192.168.99.100:30902

# Get actual NodePorts
kubectl get svc -n monitoring
```

--------------------------------

### Deploy Prometheus Operator Components

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/kube-prometheus-on-kubeadm.md

Applies the Prometheus Operator manifests to the specified namespace. This command deploys the core components required for managing Prometheus instances.

```bash
kubectl --namespace="$NAMESPACE" apply -f manifests/prometheus-operator

```

--------------------------------

### Clean Build Environment

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/RELEASE.md

Command to clean the build environment. This is typically the first step before updating dependencies or generating manifests.

```bash
make clean
```

--------------------------------

### Configure Windows HostProcess Exporter (Libsonnet)

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/windows.md

This Libsonnet code configures the windows-hostprocess addon for kube-prometheus. It specifies the image and version for windows_exporter and deploys it as a hostProcess pod. This requires the cluster to use containerd runtime and Kubernetes version v1.22 or later. It imports kube-prometheus and the windows-hostprocess addon, then customizes the windowsExporter image and version.

```libsonnet
local kp = (import 'kube-prometheus/main.libsonnet') +
  (import 'kube-prometheus/addons/windows-hostprocess.libsonnet') +
  {
    values+:: {
      windowsExporter+:: {
        image: "ghcr.io/prometheus-community/windows-exporter",
        version: "0.21.0",
      },
    },
  };

{ ['windows-exporter-' + name]: kp.windowsExporter[name] for name in std.objectFields(kp.windowsExporter) }
```

--------------------------------

### Generate Kubernetes Objects including Ingress using Jsonnet

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/exposing-prometheus-alertmanager-grafana-ingress.md

This jsonnet code demonstrates how to generate a Kubernetes List object containing various components of the Prometheus Operator, including namespaces, Prometheus Operator itself, node-exporter, kube-state-metrics, Alertmanager, Prometheus, Grafana, and Ingress resources. This is useful for deploying the entire monitoring stack.

```jsonnet
{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +
{ ['ingress-' + name]: kp.ingress[name] for name in std.objectFields(kp.ingress) }

```

--------------------------------

### Configure Windows Scrape Config (Libsonnet)

Source: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/windows.md

This Libsonnet code configures the Windows addon for kube-prometheus when using Docker runtime. It sets up a static scrape configuration to target windows_exporter instances running on specific node IPs and ports. This addon does not deploy windows_exporter itself, relying on external deployment and configuration. The primary input is the `static_configs` which lists the targets to scrape.

```libsonnet
local kp = (import 'kube-prometheus/main.libsonnet') +
  (import 'kube-prometheus/addons/windows.libsonnet') +
  {
    values+:: {
      windowsScrapeConfig+:: {
          static_configs: {
              targets: ["10.240.0.65:5000", "10.240.0.63:5000"],
          },
      },
    },
  };

```