Grafana Loki

https://grafana.com/docs/loki/latest/
Admin
Grafana Loki is a multi-tenant log aggregation system designed for storing and querying logs, often...
Tokens:843,527
Snippets:5,834
Trust Score:10
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Clone and start the mail-house example

Source: https://grafana.com/docs/loki/latest/query/logcli/logcli-tutorial

Clone the Alloy Scenario repository and start the mail-house example with Docker Compose. This exposes Loki at http://localhost:3100 and Grafana at http://localhost:3000.

```bash
git clone https://github.com/grafana/alloy-scenarios.git
```

```bash
docker compose -f alloy-scenarios/mail-house/docker-compose.yml up -d
```

--------------------------------

### Install loki-canary Tanka setup with jsonnet-bundler

Source: https://grafana.com/docs/loki/latest/operations/loki-canary

Installs the Tanka setup for loki-canary from the production folder using jsonnet-bundler.

```shell
jb install github.com/grafana/loki/production/ksonnet/loki-canary
```

--------------------------------

### Output of docker compose up -d

Source: https://grafana.com/docs/loki/latest/get-started/quick-start/quick-start

Example output from docker compose up -d showing the created network and started containers.

```console
✔ Network evaluate-loki_loki          Created      0.1s
✔ Container evaluate-loki-minio-1     Started      0.6s
✔ Container evaluate-loki-flog-1      Started      0.6s
✔ Container evaluate-loki-backend-1   Started      0.8s
✔ Container evaluate-loki-write-1     Started      0.8s
✔ Container evaluate-loki-read-1      Started      0.8s
✔ Container evaluate-loki-gateway-1   Started      1.1s
✔ Container evaluate-loki-grafana-1   Started      1.4s
✔ Container evaluate-loki-alloy-1     Started      1.4s
```

--------------------------------

### Clone and navigate to tutorial repository

Source: https://grafana.com/docs/loki/latest/send-data/k8s-monitoring-helm

Clone the tutorial repository and change to the k8s/logs directory. The rest of the tutorial assumes you are in this directory.

```bash
git clone https://github.com/grafana/alloy-scenarios.git
```

```bash
cd alloy-scenarios/k8s/logs
```

--------------------------------

### Get rule groups by namespace example response

Source: https://grafana.com/docs/loki/latest/api

Example response for the Get rule groups by namespace endpoint. Shows the YAML structure of a single rule group.

```yaml
name: <string>
interval: <duration;optional>
rules:
  - alert: <string>
    expr: <string>
    for: <duration>
    annotations:
      <annotation_name>: <string>
    labels:
      <label_name>: <string>

```

--------------------------------

### Set up environment with git clone and Docker Compose

Source: https://grafana.com/docs/loki/latest/send-data/otel/otel-collector-getting-started

These commands set up the environment: clone the demo repository, start the observability stack, and check container status.

```bash
git clone -b microservice-otel-collector  https://github.com/grafana/loki-fundamentals.git
```

```bash
docker compose -f loki-fundamentals/docker-compose.yml up -d
```

```bash
docker ps -a
```

--------------------------------

### Install xk6-loki extension and build k6

Source: https://grafana.com/docs/loki/latest/send-data/k6

These commands install the xk6 extension bundler, clone the xk6-loki repository, and build a custom k6 binary with the Loki extension. Run them in order to set up the load testing tool.

```bash
go install go.k6.io/xk6/cmd/xk6@latest
```

```bash
git clone https://github.com/grafana/xk6-loki
cd xk6-loki
```

```bash
make k6
```

--------------------------------

### List installed Docker plugins

Source: https://grafana.com/docs/loki/latest/send-data/docker-driver

Checks installed Docker plugins. Plugins that have started successfully are listed as enabled.

```bash
docker plugin ls
```

--------------------------------

### Install httpx library

Source: https://grafana.com/docs/loki/latest/reference/python-client-examples

Install the httpx library if an async-capable client is required.

```bash
pip install httpx
```

--------------------------------

### Example runtime configuration file with overrides and multi_kv_config

Source: https://grafana.com/docs/loki/latest/configuration

Example runtime configuration file showing per-tenant overrides and multi KV store settings. The `overrides` section sets limits for tenant1 and tenant2, and `multi_kv_config` configures the multi KV store with mirroring disabled and consul as primary.

```yaml
overrides:
  tenant1:
    ingestion_rate_mb: 10
    max_streams_per_user: 100000
    max_chunks_per_query: 100000
  tenant2:
    max_streams_per_user: 1000000
    max_chunks_per_query: 1000000

multi_kv_config:
    mirror-enabled: false
    primary: consul
```

--------------------------------

### Query Loki logs with httpx (GET /loki/api/v1/query_range)

Source: https://grafana.com/docs/loki/latest/reference/python-client-examples

Uses the httpx library to call the Loki query_range endpoint. The function converts start and end datetimes to nanosecond timestamps, sets direction to backward, and raises an exception on HTTP errors. The example then prints each stream's labels and log lines.

```python
import httpx
from datetime import datetime, timedelta


def query_range(
    url: str,
    query: str,
    start: datetime,
    end: datetime,
    limit: int = 1000,
    headers: dict[str, str] | None = None,
    auth: tuple[str, str] | None = None,
    verify: bool | str = True,  # False to skip TLS, or path to CA bundle; httpx also accepts ssl.SSLContext
) -> list:
    """Query Loki for log entries within a time range."""
    resp = httpx.get(
        f"{url}/loki/api/v1/query_range",
        params={
            "query": query,
            "start": str(int(start.timestamp() * 1e9)),  # nanoseconds
            "end": str(int(end.timestamp() * 1e9)),
            "limit": limit,
            "direction": "backward",
        },
        headers=headers,
        auth=auth,
        verify=verify,
    )
    resp.raise_for_status()
    return resp.json()["data"]["result"]


results = query_range(
    url="http://localhost:3100",
    query='{job="varlogs"} |= "error"',
    start=datetime.now() - timedelta(hours=1),
    end=datetime.now(),
)

for stream in results:
    print(f"Labels: {stream['stream']}")
    for ts, line in stream["values"]:
        print(f"  {datetime.fromtimestamp(int(ts) / 1e9)}: {line}")
```

--------------------------------

### Start the demo environment with docker compose

Source: https://grafana.com/docs/loki/latest/get-started/quick-start/quick-start

Starts the demo environment in detached mode. The output shows the created network and started containers.

```bash
docker compose up -d
```

--------------------------------

### Query Loki logs with requests (GET /loki/api/v1/query_range)

Source: https://grafana.com/docs/loki/latest/reference/python-client-examples

Uses the requests library to call the Loki query_range endpoint. The function converts start and end datetimes to nanosecond timestamps, sets direction to backward, and raises an exception on HTTP errors. The example then prints each stream's labels and log lines.

```python
import requests
from datetime import datetime, timedelta


def query_range(
    url: str,
    query: str,
    start: datetime,
    end: datetime,
    limit: int = 1000,
    headers: dict[str, str] | None = None,
    auth: tuple[str, str] | None = None,
    verify: bool | str = True,  # False to skip TLS, or path to CA bundle
) -> list:
    """Query Loki for log entries within a time range."""
    resp = requests.get(
        f"{url}/loki/api/v1/query_range",
        params={
            "query": query,
            "start": str(int(start.timestamp() * 1e9)),  # nanoseconds
            "end": str(int(end.timestamp() * 1e9)),
            "limit": limit,
            "direction": "backward",
        },
        headers=headers,
        auth=auth,
        verify=verify,
    )
    resp.raise_for_status()
    return resp.json()["data"]["result"]


results = query_range(
    url="http://localhost:3100",
    query='{job="varlogs"} |= "error"',
    start=datetime.now() - timedelta(hours=1),
    end=datetime.now(),
)

for stream in results:
    print(f"Labels: {stream['stream']}")
    for ts, line in stream["values"]:
        print(f"  {datetime.fromtimestamp(int(ts) / 1e9)}: {line}")
```

--------------------------------

### Start Loki on Linux

Source: https://grafana.com/docs/loki/latest/setup/install/local

Starts Loki on Linux using the downloaded configuration file.

```bash
./loki-linux-amd64 -config.file=loki-local-config.yaml
```

--------------------------------

### curl with valid start and end timestamps

Source: https://grafana.com/docs/loki/latest/operations/troubleshooting/troubleshoot-operations

Example curl command that sets `start` and `end` times to ensure the end time is after the start time, resolving the 'end timestamp must not be before or equal to start time' error.

```bash
curl "http://loki:3100/loki/api/v1/query_range?\
query={job=\"app\"}&\
start=2024-01-01T00:00:00Z&\
end=2024-01-02T00:00:00Z"
```

--------------------------------

### Start Loki on Windows

Source: https://grafana.com/docs/loki/latest/setup/install/local

Starts Loki on Windows using the downloaded configuration file.

```bash
.\loki-windows-amd64.exe --config.file=loki-local-config.yaml
```

--------------------------------

### Get GKE cluster credentials example

Source: https://grafana.com/docs/loki/latest/setup/install/helm/deployment-guides/gcp

Example of the gcloud container clusters get-credentials command with variables filled in. Authenticates to the cluster named loki-gcp in the europe-west4 region.

```bash
gcloud container clusters get-credentials loki-gcp \
  --region=europe-west4
```

--------------------------------

### Create Kubernetes Service Account example

Source: https://grafana.com/docs/loki/latest/setup/install/helm/deployment-guides/gcp

Example of creating a Kubernetes Service Account named loki-gcp-ksa in the loki namespace.

```bash
kubectl create serviceaccount loki-gcp-ksa \
  --namespace loki
```

--------------------------------

### GET /loki/api/v1/query_range - Example 1

Source: https://grafana.com/docs/loki/latest/api

This example demonstrates how to query the Loki query_range endpoint using a LogQL query that aggregates log rates by level. The response is a matrix of time-series values.

```APIDOC
## [GET] /loki/api/v1/query_range

### Description
This endpoint performs a LogQL query over a range of time. It can return either a matrix of aggregated values or a stream of log lines depending on the query.

### Method
GET

### Endpoint
/loki/api/v1/query_range

### Parameters
#### Query Parameters
- **query** (string) - Required - The LogQL query to execute.
- **step** (string) - Optional - The query resolution step width in seconds. Used for range queries.

### Request Example
```bash
curl -G -s  "http://localhost:3100/loki/api/v1/query_range" \
  --data-urlencode 'query=sum(rate({job="varlogs"}[10m])) by (level)' \
  --data-urlencode 'step=300' | jq
```

### Response
#### Success Response (200)
- **status** (string) - The status of the query, e.g., "success".
- **data.resultType** (string) - The type of result, either "matrix" or "streams".
- **data.result** (array) - The query result. For matrix results, each item contains a metric and values. For stream results, each item contains a stream and values.
- **data.stats** (object) - Statistics about the query execution.

#### Response Example
```json
{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
        "metric": {
          "level": "info"
        },
        "values": [
          [1588889221, "137.95"],
          [1588889221, "467.115"],
          [1588889221, "658.8516666666667"]
        ]
      }
    ],
    "stats": {}
  }
}
```
```

--------------------------------

### GET /loki/api/v1/query_range - Example 2

Source: https://grafana.com/docs/loki/latest/api

This example demonstrates how to query the Loki query_range endpoint using a LogQL query that selects all log streams for a given job. The response is a stream of log lines.

```APIDOC
## [GET] /loki/api/v1/query_range

### Description
This endpoint performs a LogQL query over a range of time. It can return either a matrix of aggregated values or a stream of log lines depending on the query.

### Method
GET

### Endpoint
/loki/api/v1/query_range

### Parameters
#### Query Parameters
- **query** (string) - Required - The LogQL query to execute.

### Request Example
```bash
curl -G -s "http://localhost:3100/loki/api/v1/query_range" \
  --data-urlencode 'query={job="varlogs"}' | jq
```

### Response
#### Success Response (200)
- **status** (string) - The status of the query, e.g., "success".
- **data.resultType** (string) - The type of result, either "matrix" or "streams".
- **data.result** (array) - The query result. For stream results, each item contains a stream and values.
- **data.stats** (object) - Statistics about the query execution.

#### Response Example
```json
{
  "status": "success",
  "data": {
    "resultType": "streams",
    "result": [
      {
        "stream": {
          "filename": "/var/log/myproject.log",
          "job": "varlogs",
          "level": "info"
        },
        "values": [
          ["1569266497240578000", "foo"],
          ["1569266492548155000", "bar"]
        ]
      }
    ],
    "stats": {}
  }
}
```
```

--------------------------------

### GET /loki/api/v1/index/stats

Source: https://grafana.com/docs/loki/latest/api

The GET /loki/api/v1/index/stats endpoint queries the index for the number of streams, chunks, entries, and bytes that a query resolves to. It accepts query, start, and end parameters. The response is a JSON object with the counts.

```APIDOC
## GET /loki/api/v1/index/stats

### Description
The `/loki/api/v1/index/stats` endpoint can be used to query the index for the number of `streams`, `chunks`, `entries`, and `bytes` that a query resolves to.

### Method
GET

### Endpoint
/loki/api/v1/index/stats

### Parameters
#### Query Parameters
- **query** (string) - Required - The LogQL matchers to check (for example, `{job="foo", env!="dev"}`).
- **start** (integer) - Required - Start timestamp in nanosecond Unix epoch.
- **end** (integer) - Required - End timestamp in nanosecond Unix epoch.

### Request Example
GET /loki/api/v1/index/stats?query={job="foo", env!="dev"}&start=1609459200000000000&end=1609545600000000000

### Response
#### Success Response (200)
- **streams** (integer) - The number of streams that the query resolves to.
- **chunks** (integer) - The number of chunks that the query resolves to.
- **entries** (integer) - The number of entries that the query resolves to.
- **bytes** (integer) - The number of bytes that the query resolves to.

#### Response Example
{
  "streams": 100,
  "chunks": 1000,
  "entries": 5000,
  "bytes": 100000
}

### Notes
- This endpoint does not include data from the ingesters.
- It is a probabilistic technique.
- Streams/chunks which span multiple period configurations may be counted twice.
- It is generally more helpful for larger queries and can be used for better understanding the throughput requirements and data topology for a list of matchers over a period of time.
```

--------------------------------

### Install Fluent Bit Helm chart with values.yaml

Source: https://grafana.com/docs/loki/latest/send-data/fluentbit/community-plugin

Adds the Fluent Helm chart repository, updates it, and installs the fluent-bit chart using the provided values.yaml file.

```bash
helm repo add fluent https://fluent.github.io/helm-charts
helm repo update
helm install fluent-bit fluent/fluent-bit -f values.yaml
```

--------------------------------

### GET /loki/api/v1/query_range

Source: https://grafana.com/docs/loki/latest/reference/api

Documentation for the Loki query_range API endpoint, including parameters, response format, and examples.

```APIDOC
## GET /loki/api/v1/query_range

### Description
Queries logs or metrics over a range of time. This endpoint is used for both log and metric type LogQL queries.

### Method
GET

### Endpoint
/loki/api/v1/query_range

### Parameters
#### Query Parameters
- **query** (string) - Required - The LogQL query to perform.
- **limit** (integer) - Optional - The max number of entries to return. Defaults to 100. Only applies to query types which produce a stream (log lines) response.
- **start** (string) - Optional - The start time for the query as a nanosecond Unix epoch or another supported format. Defaults to one hour ago. Loki returns results with timestamp greater or equal to this value.
- **end** (string) - Optional - The end time for the query as a nanosecond Unix epoch or another supported format. Defaults to now. Loki returns results with timestamp lower than this value.
- **since** (string) - Optional - A duration used to calculate start relative to end. If end is in the future, start is calculated as this duration before now. Any value specified for start supersedes this parameter.
- **step** (string) - Optional - Query resolution step width in duration format or float number of seconds. Defaults to a dynamic value based on start and end. Only applies to query types which produce a matrix response.
- **interval** (string) - Optional - Only return entries at (or greater than) the specified interval. Only applies to queries which produce a stream response.
- **direction** (string) - Optional - Determines the sort order of logs. Supported values are forward or backward. Defaults to backward.

### Response
#### Success Response (200)
- **status** (string) - The status of the query, typically "success".
- **data** (object) - The query result data.
  - **resultType** (string) - The type of result: "matrix" or "streams".
  - **result** (array) - The result values, either matrix values or stream values.
  - **stats** (object) - Statistics about the query.

#### Response Example
```json
{
  "status": "success",
  "data": {
    "resultType": "matrix" | "streams",
    "result": [<matrix value>] | [<stream value>],
    "stats" : [<statistics>]
  }
}
```

For matrix results, each value is:
```json
{
  "metric": {
    <label key-value pairs>
  },
  "values": [
    [
      <number: second unix epoch>,
      <string: value>
    ],
    ...
  ]
}
```

For stream results, each value is:
```json
{
  "stream": {
    <label key-value pairs>
  },
  "values": [
    [
      <string: nanosecond unix epoch>,
      <string: log line>
    ],
    ...
  ]
}
```

Parquet can be requested as a response format by setting the Accept header to `application/vnd.apache.parquet`. The schema for streams is: timestamp (TIMESTAMP WITH TIME ZONE), labels (MAP(VARCHAR, VARCHAR)), line (VARCHAR). For metrics: timestamp (TIMESTAMP WITH TIME ZONE), labels (MAP(VARCHAR, VARCHAR)), value (DOUBLE).
```

--------------------------------

### Example metric output with job label

Source: https://grafana.com/docs/loki/latest/community/lids/0005-loki-mixin-configuration-improvements

Example output of the scraping configuration, showing a metric with job="kubernetes-pods" and other labels. This illustrates why the hardcoded selectors in the Loki mixin dashboards are incompatible with such setups.

```none
up{job="kubernetes-pods", ...}
```

--------------------------------

### Start the observability stack

Source: https://grafana.com/docs/loki/latest/send-data/alloy/examples/alloy-kafka-logs

Use Docker Compose to spin up the required services including Alloy, Loki, and Kafka.

```bash
docker compose -f loki-fundamentals/docker-compose.yml up -d
```

--------------------------------

### Initialize Tanka application and create Loki environment

Source: https://grafana.com/docs/loki/latest/setup/install/tanka

Run these commands in your config repo to create a Tanka application and add an environment for Loki. Replace <application name> with your app folder name and <Kubernetes API server> with the API server URL (e.g., https://localhost:6443). Requires Tanka v0.31.0 or newer.

```bash
mkdir <application name>
cd <application name>
tk init
tk env add environments/loki --namespace=loki --server=<Kubernetes API server>
```

--------------------------------

### Create GCS buckets with example values

Source: https://grafana.com/docs/loki/latest/setup/install/helm/deployment-guides/gcp

Example command with all variables filled in, creating buckets named loki-gcp-chunks and loki-gcp-ruler in the europe-west4 region. Use this as a reference for the expected command structure.

```bash
gcloud storage buckets create gs://loki-gcp-chunks gs://loki-gcp-ruler \
  --location=europe-west4 \
  --default-storage-class=STANDARD \
  --public-access-prevention \
  --uniform-bucket-level-access \
  --soft-delete-duration=7d
```

--------------------------------

### GET /loki/api/v1/query

Source: https://grafana.com/docs/loki/latest/reference/loki-http-api

Query Loki for log data using LogQL. This example demonstrates a basic query with a rate aggregation and grouping by level.

```APIDOC
## [GET] /loki/api/v1/query

### Description
Query Loki for log data using LogQL. This example demonstrates a basic query with a rate aggregation and grouping by level.

### Method
GET

### Endpoint
/loki/api/v1/query

### Parameters
#### Query Parameters
- **query** (string) - Required - The LogQL query to execute.

### Request Example
```bash
curl -G -s  "http://localhost:3100/loki/api/v1/query" \
  --data-urlencode 'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq
```

### Response
#### Success Response (200)
- **status** (string) - The status of the query, e.g., "success".
- **data** (object) - The query result data.
  - **resultType** (string) - The type of result, e.g., "vector".
  - **result** (array) - The result entries.
    - **metric** (object) - The metric labels.
    - **value** (array) - The timestamp and value.
  - **stats** (object) - Query statistics.

#### Response Example
```json
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {},
        "value": [
          1588889221,
          "1267.1266666666666"
        ]
      },
      {
        "metric": {
          "level": "warn"
        },
        "value": [
          1588889221,
          "37.77166666666667"
        ]
      },
      {
        "metric": {
          "level": "info"
        },
        "value": [
          1588889221,
          "37.69"
        ]
      }
    ],
    "stats": {
      ...
    }
  }
}
```

### Authentication
- **X-Scope-OrgID** (header) - Optional - Identifies the tenant for multi-tenant clusters. For multiple tenants, separate names with a pipe (`|`).
- **Basic Auth** - Optional - For Grafana Enterprise Logs, use tenant names as user and an access policy token as password. For Grafana Cloud, use the provided User and an access policy token.

### Request Examples with Authentication

#### Multi-Tenancy Header (Single Tenant)
```bash
curl -H 'X-Scope-OrgID:Tenant1' \
  -G -s "http://localhost:3100/loki/api/v1/query" \
  --data-urlencode 'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq
```

#### Multi-Tenancy Header (Multiple Tenants)
```bash
curl -H 'X-Scope-OrgID:Tenant1|Tenant2|Tenant3' \
  -G -s "http://localhost:3100/loki/api/v1/query" \
  --data-urlencode 'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq
```

#### Basic Authentication (Grafana Enterprise Logs)
```bash
curl -u "Tenant1|Tenant2|Tenant3:$API_TOKEN" \
  -G -s "http://localhost:3100/loki/api/v1/query" \
  --data-urlencode 'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq
```

#### Basic Authentication (Grafana Cloud)
```bash
curl -u "User:$API_TOKEN" \
  -G -s "<URL_PROVIDED_IN_LOKI_DATA_SOURCE_SETTINGS>/loki/api/v1/query" \
  --data-urlencode 'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq
```
```

--------------------------------

### Install xk6 extension bundler

Source: https://grafana.com/docs/loki/latest/send-data/k6

Install the xk6 extension bundler to build a custom k6 binary with the xk6-loki extension.

```APIDOC
## Install xk6 extension bundler

### Description
Install the xk6 extension bundler to build a custom k6 binary with the xk6-loki extension.

### Command
```bash
go install go.k6.io/xk6/cmd/xk6@latest
```
```
Tokens:
Raw