Prometheus

https://prometheus.io/docs
Admin
Prometheus is an open-source systems monitoring and alerting toolkit. It collects and stores metrics...
Tokens:462,132
Snippets:2,912
Trust Score:9.9
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Install Prometheus Go client libraries with go get

Source: https://prometheus.io/docs/guides/go-application

Install the prometheus, promauto, and promhttp libraries required for instrumenting Go applications. Run these commands in your Go module directory.

```bash
go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promauto
go get github.com/prometheus/client_golang/prometheus/promhttp
```

--------------------------------

### OM 2.0 combined example with labels, timestamps, st@, and exemplar

Source: https://prometheus.io/docs/guides/open_metrics_2_0_migration

Combined example showing counters and a histogram together with labels, timestamps, Start Timestamps, and an exemplar, demonstrating the full field ordering in OM 2.0.

```text
# TYPE http_requests_total counter
http_requests_total{method="GET",code="200"} 1027 1710000000.123 st@1000000000.000 # {trace_id="abc123",span_id="def456"} 1.0 1709999999.456
http_requests_total{method="POST",code="201"} 53 1710000000.123 st@1000000000.000
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds{method="GET"} {count:1027,sum:172.5,bucket:[0.1:800,0.5:950,+Inf:1027]} 1710000000.123 st@1000000000.000
# EOF

```

--------------------------------

### Start Prometheus with a configuration file

Source: https://prometheus.io/docs/introduction/first_steps

Run this command from the directory containing the Prometheus binary to start the server with the specified configuration file. After starting, Prometheus will be available at http://localhost:9090 and will begin collecting metrics about itself after about 30 seconds.

```bash
./prometheus --config.file=prometheus.yml

```

--------------------------------

### Start Node Exporter sample targets

Source: https://prometheus.io/docs/prometheus/latest/getting_started

Extract the Node Exporter archive and start three example targets on ports 8080, 8081, and 8082, each in a separate terminal. The targets will be available at http://localhost:8080/metrics, http://localhost:8081/metrics, and http://localhost:8082/metrics.

```bash
tar -xzvf node_exporter-*.*.tar.gz
cd node_exporter-*.*

# Start 3 example targets in separate terminals:
./node_exporter --web.listen-address 127.0.0.1:8080
./node_exporter --web.listen-address 127.0.0.1:8081
./node_exporter --web.listen-address 127.0.0.1:8082
```

--------------------------------

### Prometheus startup success log line

Source: https://prometheus.io/docs/guides/file-sd

This is an example log line that appears when Prometheus has started successfully.

```text
level=info ts=2018-08-13T20:39:24.905651509Z caller=main.go:500 msg="Server is ready to receive web requests."
```

--------------------------------

### Start Prometheus with config file

Source: https://prometheus.io/docs/guides/node-exporter

Starts Prometheus with the configuration file created earlier.

```bash
./prometheus --config.file=./prometheus.yml
```

--------------------------------

### Start Prometheus with --config.file

Source: https://prometheus.io/docs/prometheus/latest/getting_started

Starts Prometheus with the specified configuration file. By default, Prometheus stores its database in ./data (flag --storage.tsdb.path). After starting, Prometheus should be accessible at localhost:9090 and its metrics endpoint at localhost:9090/metrics.

```bash
# Start Prometheus.
# By default, Prometheus stores its database in ./data (flag --storage.tsdb.path).
./prometheus --config.file=prometheus.yml
```

--------------------------------

### Get all targets with curl

Source: https://prometheus.io/docs/prometheus/latest/querying/api

Example curl command to get all targets and its JSON response showing active and dropped targets.

```bash
curl http://localhost:9090/api/v1/targets

```

```json
{
  "status": "success",
  "data": {
    "activeTargets": [
      {
        "discoveredLabels": {
          "__address__": "127.0.0.1:9090",
          "__metrics_path__": "/metrics",
          "__scheme__": "http",
          "job": "prometheus"
        },
        "labels": {
          "instance": "127.0.0.1:9090",
          "job": "prometheus"
        },
        "scrapePool": "prometheus",
        "scrapeUrl": "http://127.0.0.1:9090/metrics",
        "globalUrl": "http://example-prometheus:9090/metrics",
        "lastError": "",
        "lastScrape": "2017-01-17T15:07:44.723715405+01:00",
        "lastScrapeDuration": 0.050688943,
        "health": "up",
        "scrapeInterval": "1m",
        "scrapeTimeout": "10s"
      }
    ],
    "droppedTargets": [
      {
        "discoveredLabels": {
          "__address__": "127.0.0.1:9100",
          "__always_scrape_classic_histograms__": "false",
          "__convert_classic_histograms_to_nhcb__": "false",
          "__metrics_path__": "/metrics",
          "__scheme__": "http",
          "__scrape_interval__": "1m",
          "__scrape_native_histograms__": "false",
          "__scrape_timeout__": "10s",
          "job": "node"
        },
        "scrapePool": "node"
      }
    ]
  }
}

```

--------------------------------

### Start Docker Compose stack

Source: https://prometheus.io/docs/guides/cadvisor

Runs the Docker Compose installation defined in the docker-compose.yml file. This command starts all three containers (prometheus, cadvisor, redis).

```bash
docker-compose up
```

--------------------------------

### Start Prometheus with --web.config.file

Source: https://prometheus.io/docs/guides/tls-encryption

Starts Prometheus with the TLS web configuration. The --web.external-url flag is optional.

```bash
prometheus \
  --config.file=/path/to/prometheus.yml \
  --web.config.file=/path/to/web-config.yml \
  --web.external-url=https://example.com/
```

--------------------------------

### Get scrape pool configuration with curl

Source: https://prometheus.io/docs/prometheus/latest/querying/api

Example curl command to get the effective configuration for the 'prometheus' scrape pool and its JSON response. Authentication credentials and other secrets are redacted.

```bash
curl -G http://localhost:9090/api/v1/scrape_pools/config \
  --data-urlencode 'scrapePool=prometheus'

```

```json
{
  "status": "success",
  "data": {
    "yaml": "job_name: prometheus\nscrape_interval: 15s\nscrape_timeout: 10s\nmetrics_path: /metrics\nscheme: http\n"
  }
}

```

--------------------------------

### Launch Prometheus with web.config.file

Source: https://prometheus.io/docs/guides/basic-auth

Starts Prometheus with the web configuration file that contains basic auth credentials.

```bash
$ prometheus --web.config.file=web.yml

```

--------------------------------

### Start Prometheus

Source: https://prometheus.io/docs/guides/file-sd

Run this command to start Prometheus. A successful startup is indicated by a log line like 'Server is ready to receive web requests.'

```bash
./prometheus
```

--------------------------------

### Run the Go application

Source: https://prometheus.io/docs/guides/go-application

Start the Go application that exposes metrics. Run this command in the directory containing main.go.

```bash
go run main.go
```

--------------------------------

### Enable UTF-8 strict mode

Source: https://prometheus.io/docs/alerting/latest/configuration

Command to start Alertmanager in UTF-8 strict mode, which disables support for classic matchers. This mode should be enabled for new installations and for existing installations once all warnings of incompatible matchers have been resolved.

```bash
alertmanager --config.file=config.yml --enable-feature="utf8-strict-mode"
```

--------------------------------

### Run the instrumented server

Source: https://prometheus.io/docs/tutorials/instrumenting_http_server_in_go

Commands to initialize a Go module, fetch dependencies, and run the instrumented server. After running, hit /ping a few times and then /metrics to see the counter.

```bash
go mod init prom_example
go mod tidy
go run server.go

```

--------------------------------

### Example curl request for instant query

Source: https://prometheus.io/docs/prometheus/latest/querying/api

Evaluates the expression 'up' at the time 2015-07-01T20:10:51.781Z using the GET endpoint.

```bash
curl 'http://localhost:9090/api/v1/query?query=up&time=2015-07-01T20:10:51.781Z'

```

--------------------------------

### Run prometheus --help

Source: https://prometheus.io/docs/introduction/first_steps

Run the Prometheus binary with the --help flag to see its usage and options.

```bash
./prometheus --help
usage: prometheus [<flags>]

The Prometheus monitoring server

. . .

```

--------------------------------

### UTF-8 strict mode startup failure

Source: https://prometheus.io/docs/alerting/latest/configuration

Example error log when Alertmanager fails to start in UTF-8 strict mode because the configuration still contains incompatible matchers. Alertmanager will not start until all warnings of incompatible matchers have been resolved.

```log
ts=2024-02-11T10:00:00Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=config.yml err="end of input: expected label value"
```

--------------------------------

### Get label names

Source: https://prometheus.io/docs/prometheus/latest/querying/api

Returns a list of label names. Supports optional start, end, match[], and limit parameters.

```http
GET /api/v1/labels
POST /api/v1/labels
```

--------------------------------

### Add example.com to /etc/hosts

Source: https://prometheus.io/docs/guides/tls-encryption

Adds an entry to /etc/hosts to route example.com to localhost for local TLS testing.

```text
127.0.0.1     example.com
```

--------------------------------

### Example output from /metrics endpoint

Source: https://prometheus.io/docs/guides/node-exporter

Example output from the /metrics endpoint, showing Go runtime metrics and the format of the exposition.

```text
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.8996e-05
go_gc_duration_seconds{quantile="0.25"} 4.5926e-05
go_gc_duration_seconds{quantile="0.5"} 5.846e-05
# etc.
```

--------------------------------

### Configure federation scrape_configs with match[] and honor_labels

Source: https://prometheus.io/docs/prometheus/latest/federation

Use this scrape_configs to federate metrics from source Prometheus servers. It scrapes the /federate endpoint with honor_labels: true and passes match[] parameters to select series with job="prometheus" or metric names starting with job:. The example targets source-prometheus-{1,2,3}:9090.

```yaml
scrape_configs:
  - job_name: 'federate'
    scrape_interval: 15s

    honor_labels: true
    metrics_path: '/federate'

    params:
      'match[]':
        - '{job="prometheus"}'
        - '{__name__=~"job:.*"}'

    static_configs:
      - targets:
        - 'source-prometheus-1:9090'
        - 'source-prometheus-2:9090'
        - 'source-prometheus-3:9090'
```

--------------------------------

### Native-Only Histogram CompositeValue

Source: https://prometheus.io/docs/guides/open_metrics_2_0_migration

Shows a native-only histogram CompositeValue with fields count, sum, schema, zero_threshold, zero_count, positive_spans, and positive_buckets. The example has two spans: the first starts at bucket index 0 with 3 consecutive buckets, then skips 2 empty buckets, then 2 more consecutive buckets. The total bucket count is 3 + 2 = 5, matching the five values in positive_buckets.

```prometheus
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds {count:59,sum:120.0,schema:3,zero_threshold:1e-4,zero_count:2,positive_spans:[0:3,2:2],positive_buckets:[10,15,12,8,12]} 1710000000 st@1000000000

```

--------------------------------

### Build and run the Go server

Source: https://prometheus.io/docs/tutorials/instrumenting_http_server_in_go

Commands to compile and run the simple Go HTTP server. After running, open http://localhost:8090/ping in a browser to see 'pong'.

```bash
go build server.go
./server

```
Tokens:
Raw