### Set up Grafana Mimir Jsonnet Example Project Source: https://grafana.com/docs/mimir/latest/set-up/jsonnet/deploy Initializes a Tanka project, installs the Grafana Mimir Jsonnet library, copies an example manifest file, and generates the initial YAML manifests. This script sets up a basic environment for deploying Mimir. ```sh #!/usr/bin/env bash # SPDX-License-Identifier: AGPL-3.0-only set -e # Initialise the Tanka. mkdir jsonnet-example && cd jsonnet-example tk init --k8s=1.29 # Install Mimir jsonnet. jb install github.com/grafana/mimir/operations/mimir@main # Use the provided example. cp vendor/mimir/mimir-manifests.jsonnet.example environments/default/main.jsonnet # Generate the YAML manifests. export PAGER=cat tk show environments/default ``` -------------------------------- ### Start Grafana Mimir and Dependencies with Docker Compose Source: https://grafana.com/docs/mimir/latest/get-started/play-with-grafana-mimir This command initiates the local Grafana Mimir setup using Docker Compose. It starts Grafana Mimir instances, Minio for S3 storage, Prometheus for metrics, Grafana for visualization, and a load balancer. The command requires Docker and Docker Compose to be installed and running. ```bash docker compose up -d ``` -------------------------------- ### Run Grafana Mimir using Docker Source: https://grafana.com/docs/mimir/latest/get-started Starts a Grafana Mimir instance using Docker. It creates a Docker network, runs the Mimir container, maps the port, mounts a configuration file, and specifies the configuration file path. Ensure the 'demo.yaml' file exists in the current directory. ```bash docker network create grafanet docker run \ --rm \ --name mimir \ --network grafanet \ --publish 9009:9009 \ --volume "$(pwd)"/demo.yaml:/etc/mimir/demo.yaml grafana/mimir:latest \ --config.file=/etc/mimir/demo.yaml ``` -------------------------------- ### Run Grafana Mimir using local binary Source: https://grafana.com/docs/mimir/latest/get-started Executes the Grafana Mimir binary directly from the local file system. This method requires the Mimir binary to be compiled or downloaded and placed in the current directory, along with the configuration file 'demo.yaml'. ```bash ./mimir --config.file=./demo.yaml ``` -------------------------------- ### Ingester Migration Steps: Consul to etcd (CLI and Runtime) Source: https://grafana.com/docs/mimir/latest/configure/configure-hash-rings This example outlines the steps to migrate ingesters from Consul to etcd using Grafana Mimir. It involves configuring the multi KV store via CLI flags for initial setup and then using runtime configuration to switch the primary and disable mirroring, ensuring a smooth transition. ```bash # Step 1: Initial configuration with multi-backend and mirroring enabled -ingester.ring.store=multi -ingester.ring.multi.primary=consul -ingester.ring.multi.secondary=etcd -ingester.ring.multi.mirror-enabled=true # Configure specific settings for Consul and etcd as needed. -ingester.ring.consul.* -ingester.ring.etcd.* # Step 3: Update runtime configuration to switch primary # primary: etcd in multi_kv_config # Step 5: Update runtime configuration to disable mirroring # mirror_enabled: false in multi_kv_config # Step 7: Final configuration to use etcd directly -ingester.ring.store=etcd # Remove multi and Consul configurations as they are no longer required. ``` -------------------------------- ### Run Grafana server using Docker Source: https://grafana.com/docs/mimir/latest/get-started Launches a Grafana server instance using Docker. This command creates a Grafana container, maps the port, and connects it to the 'grafanet' Docker network, enabling it to access other services on that network like Grafana Mimir. ```bash docker run --rm --name=grafana --network=grafanet -p 3000:3000 grafana/grafana ``` -------------------------------- ### Download Grafana Mimir Binary for Linux AMD64 Source: https://grafana.com/docs/mimir/latest/get-started This command downloads the Grafana Mimir binary for Linux AMD64 architecture using curl and makes it executable. This is an alternative to using Docker for running Mimir. ```Bash curl -fLo mimir https://github.com/grafana/mimir/releases/latest/download/mimir-linux-amd64 chmod +x mimir ``` -------------------------------- ### Alertmanager Configuration Example (YAML) Source: https://grafana.com/docs/mimir/latest/manage/tools/mimirtool An example of an Alertmanager configuration file in YAML format, defining routing, receivers, and templates. ```yaml route: receiver: "example_receiver" group_by: ["example_groupby"] receivers: - name: "example_receiver" templates: - example_template.tpl ``` -------------------------------- ### Install Tanka and jsonnet-bundler Source: https://grafana.com/docs/mimir/latest/set-up/jsonnet/deploy Installs the Tanka CLI and jsonnet-bundler tools, which are essential for managing Jsonnet configurations and generating Kubernetes manifests. This command assumes Go is installed locally. ```console # make sure to be outside of GOPATH or a go.mod project go install github.com/grafana/tanka/cmd/tk@latest go install github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest ``` -------------------------------- ### Download Grafana Mimir using Docker Source: https://grafana.com/docs/mimir/latest/get-started This command pulls the latest Grafana Mimir image from Docker Hub. It's a prerequisite for running Mimir with Docker. ```Bash docker pull grafana/mimir:latest ``` -------------------------------- ### Alertmanager Template Example (Go Template) Source: https://grafana.com/docs/mimir/latest/manage/tools/mimirtool An example of an Alertmanager template file written in Go template syntax, defining a custom message format for alerts. ```gotemplate {{ define "alert_customer_env_message" }} [{{ .CommonLabels.alertname }} | {{ .CommonLabels.customer }} | {{ .CommonLabels.environment }}] {{ end }} ``` -------------------------------- ### Install Grafana Mimir with Custom Configuration (Puppet) Source: https://grafana.com/docs/mimir/latest/set-up/puppet This Puppet code demonstrates how to customize the Grafana Mimir deployment by passing a configuration hash to the 'mimir' class. This example sets the target to 'querier'. ```puppet class {'mimir': config_hash => { target => 'querier', }, } ``` -------------------------------- ### Install Grafana Mimir Libraries using jsonnet-bundler Source: https://grafana.com/docs/mimir/latest/set-up/migrate/migrate-from-cortex Installs the updated Grafana Mimir libraries using the jsonnet-bundler. This ensures that the latest versions of the Mimir operations and mixin libraries are available for use in your Jsonnet projects. Ensure you have `jsonnet-bundler` installed before running these commands. ```bash jb install github.com/grafana/mimir/operations/mimir@main jb install github.com/grafana/mimir/operations/mimir-mixin@main ``` -------------------------------- ### Download and Run Grafana Mimir Continuous Test via Docker Source: https://grafana.com/docs/mimir/latest/manage/tools/mimir-continuous-test This snippet shows how to pull the latest Grafana Mimir Docker image and run the `mimir-continuous-test` tool. This is a convenient way to execute smoke tests without local installation. It requires Docker to be installed and running. ```Bash docker pull "grafana/mimir:latest" docker run --rm -ti grafana/mimir -target=continuous-test ``` -------------------------------- ### Download and Install Mimirtool for Linux AMD64 Source: https://grafana.com/docs/mimir/latest/manage/tools/mimirtool This command downloads the Mimirtool binary for Linux with an AMD64 architecture using curl and makes it executable. It's an alternative installation method for environments without Homebrew. ```bash curl -fLo mimirtool https://github.com/grafana/mimir/releases/latest/download/mimirtool-linux-amd64 chmod +x mimirtool ``` -------------------------------- ### Configure Grafana Alloy for remote write to Mimir Source: https://grafana.com/docs/mimir/latest/get-started Demonstrates how to configure Grafana Alloy to send metrics to Grafana Mimir using the `prometheus.remote_write` component. Includes examples for basic remote write setup and a complete configuration for scraping self-metrics. ```hcl prometheus.remote_write "LABEL" { endpoint { url = http://localhost:9009/api/v1/push ... } ... } ``` ```hcl prometheus.exporter.self "self_metrics" { } prometheus.scrape "self_scrape" { targets = prometheus.exporter.self.self_metrics.targets forward_to = [prometheus.remote_write.mimir.receiver] } prometheus.remote_write "mimir" { endpoint { url = "http://localhost:9009/api/v1/push" } } ``` -------------------------------- ### Configure Prometheus to write to Grafana Mimir Source: https://grafana.com/docs/mimir/latest/get-started Sets up Prometheus to remotely write metrics to a Grafana Mimir instance. This involves adding a `remote_write` section to the Prometheus configuration file, specifying the Mimir URL. An example is provided for a Prometheus server scraping itself. ```yaml remote_write: - url: http://localhost:9009/api/v1/push ``` ```yaml remote_write: - url: http://localhost:9009/api/v1/push scrape_configs: - job_name: prometheus honor_labels: true static_configs: - targets: ["localhost:9090"] ``` -------------------------------- ### Install Mimirtool using Homebrew on macOS Source: https://grafana.com/docs/mimir/latest/manage/tools/mimirtool This command installs the Mimirtool using the Homebrew package manager on macOS. It's a simple and direct way to get the tool set up for users on this platform. ```bash brew install mimirtool ``` -------------------------------- ### Download and Run Grafana Mimir Continuous Test via Local Binary (Linux AMD64) Source: https://grafana.com/docs/mimir/latest/manage/tools/mimir-continuous-test This snippet demonstrates how to download the Grafana Mimir binary for Linux AMD64 architecture using `curl`, make it executable with `chmod`, and then run the `mimir-continuous-test` target. This method is suitable for users who prefer not to use Docker. ```Bash curl -Lo mimir https://github.com/grafana/mimir/releases/latest/download/mimir-linux-amd64 chmod +x mimir mimir -target=continuous-test ``` -------------------------------- ### Prepare Instance for Ring Downscale (Classic Architecture) Source: https://grafana.com/docs/mimir/latest/manage/run-production-environment/scaling-out Places an ingester instance into read-only mode, preparing it for a scaled-down operation in the classic architecture. ```APIDOC ## POST /ingester/prepare-instance-ring-downscale ### Description Places an ingester instance into read-only mode, which is the first step in scaling down ingesters in the classic architecture. This allows existing blocks to become queryable before the ingester is terminated. ### Method POST ### Endpoint /ingester/prepare-instance-ring-downscale ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example None ### Response #### Success Response (200) Indicates that the ingester instance has been successfully placed into read-only mode. #### Response Example None ``` -------------------------------- ### Initialize Terraform for Deployment Source: https://grafana.com/docs/mimir/latest/manage/monitor-grafana-mimir/installing-dashboards-and-alerts Initializes the Terraform working directory. This downloads the necessary provider plugins required for your Terraform configuration. ```bash terraform init ``` -------------------------------- ### Enable HA Tracker and Configure Memberlist via YAML Source: https://grafana.com/docs/mimir/latest/configure/configure-high-availability-deduplication Enable the HA tracker globally for all tenants and configure its key-value store using a YAML configuration file. This example uses `memberlist` for the KV store and specifies join members. ```yaml limits: accept_ha_samples: true distributor: ha_tracker: enable_ha_tracker: true kvstore: store: memberlist memberlist: join_members: - - ``` -------------------------------- ### Clone Grafana Mimir Repository and Navigate Source: https://grafana.com/docs/mimir/latest/get-started/play-with-grafana-mimir This snippet demonstrates how to clone the Grafana Mimir repository using Git and then navigate into the specific tutorial directory. It requires Git to be installed on the host machine. ```bash git clone https://github.com/grafana/mimir.git cd mimir cd docs/sources/mimir/get-started/play-with-grafana-mimir/ ``` -------------------------------- ### Configure Multi KV Store Runtime Settings (YAML) Source: https://grafana.com/docs/mimir/latest/configure/configure-hash-rings This snippet shows how to configure the multi KV store using the runtime configuration file. It allows overriding the primary backend and enabling/disabling mirroring. These settings apply to all components using a multi KV store and take precedence over CLI flags. ```yaml multi_kv_config: # The runtime configuration only allows to override the primary backend and whether mirroring is enabled. primary: consul mirror_enabled: true ``` -------------------------------- ### Grafana Mimir Monolithic Configuration (demo.yaml) Source: https://grafana.com/docs/mimir/latest/get-started This YAML configuration is used to run Grafana Mimir in a monolithic mode with local filesystem storage. It's intended for demonstration purposes and should not be used in production environments. It configures storage, compactor, distributor, ingester, ruler, server, and store gateway settings. ```YAML # Do not use this configuration in production. # It is for demonstration purposes only. multitenancy_enabled: false blocks_storage: backend: filesystem bucket_store: sync_dir: /tmp/mimir/tsdb-sync filesystem: dir: /tmp/mimir/data/tsdb tsdb: dir: /tmp/mimir/tsdb compactor: data_dir: /tmp/mimir/compactor sharding_ring: kvstore: store: memberlist distributor: ring: instance_addr: 127.0.0.1 kvstore: store: memberlist ingester: ring: instance_addr: 127.0.0.1 kvstore: store: memberlist replication_factor: 1 ruler_storage: backend: filesystem filesystem: dir: /tmp/mimir/rules server: http_listen_port: 9009 log_level: error store_gateway: sharding_ring: replication_factor: 1 ``` -------------------------------- ### Prepare Ingester for Downscale (Ingest Storage Architecture) Source: https://grafana.com/docs/mimir/latest/manage/run-production-environment/scaling-out Manually prepare ingesters for downscaling by preparing their partitions. This is an alternative to using the rollout-operator. ```APIDOC ## POST /ingester/prepare-partition-downscale ### Description Prepares ingesters for downscaling by moving their partitions from `ACTIVE` to `INACTIVE` state in the ingest storage architecture. ### Method POST ### Endpoint /ingester/prepare-partition-downscale ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example None ### Response #### Success Response (200) Indicates that the ingester has been prepared for downscaling. #### Response Example None ``` -------------------------------- ### Get Label Values Endpoint Source: https://grafana.com/docs/mimir/latest/references/http-api This endpoint, GET /api/v1/label/{name}/values, allows fetching distinct values for a specified label name. Caching behavior is similar to the get label names endpoint. ```http GET /api/v1/label/{name}/values ``` -------------------------------- ### Get Metric Metadata Endpoint Source: https://grafana.com/docs/mimir/latest/references/http-api The GET /api/v1/metadata endpoint provides Prometheus-compatible metric metadata. This endpoint requires authentication. ```http GET /api/v1/metadata ``` -------------------------------- ### Grafana Mimir Configuration Parameters - Basic Source: https://grafana.com/docs/mimir/latest/configure/configuration-parameters Lists basic configuration parameters for Grafana Mimir. These parameters control fundamental aspects of the Mimir instance, such as component targeting and multitenancy. ```yaml # Comma-separated list of components to include in the instantiated process. # CLI flag: -target [target: | default = "all"] # When set to true, incoming HTTP requests must specify tenant ID in HTTP # X-Scope-OrgId header. When set to false, tenant ID from -auth.no-auth-tenant # is used instead. # CLI flag: -auth.multitenancy-enabled [multitenancy_enabled: | default = true] # (advanced) Tenant ID to use when multitenancy is disabled. # CLI flag: -auth.no-auth-tenant [no_auth_tenant: | default = "anonymous"] # (advanced) How long to wait between SIGTERM and shutdown. After receiving # SIGTERM, Mimir will report not-ready status via /ready endpoint. # CLI flag: -shutdown-delay [shutdown_delay: | default = 0s] ``` -------------------------------- ### Grafana Mimir Query-Tee Docker Installation Source: https://grafana.com/docs/mimir/latest/manage/tools/query-tee Download and run the Grafana Mimir query-tee using Docker. ```APIDOC ## Grafana Mimir Query-Tee Docker Installation ### Description Download and run the Grafana Mimir query-tee using Docker. ### Method N/A (Docker command) ### Endpoint N/A ### Parameters None ### Request Example ```bash docker pull "grafana/query-tee:latest" ``` ### Response N/A (Docker pull command output) ``` -------------------------------- ### Install Grafana Mimir Mixin using Jsonnet Source: https://grafana.com/docs/mimir/latest/manage/monitor-grafana-mimir/installing-dashboards-and-alerts Installs the Grafana Mimir mixin into your Jsonnet project. This allows you to leverage pre-defined configurations and resources for Mimir. ```bash jb install github.com/grafana/mimir/operations/mimir-mixin@main ``` -------------------------------- ### Prepare Ingester Partition for Downscale (Mimir API) Source: https://grafana.com/docs/mimir/latest/references/http-api This API prepares an ingester's partition for downscaling by setting it to the INACTIVE state. A GET call returns the timestamp of the state change, a POST call transitions to INACTIVE, and a DELETE call reverts it to ACTIVE. This endpoint fails if ingest-storage is not configured. ```http GET,POST,DELETE /ingester/prepare-partition-downscale ``` -------------------------------- ### Grafana Mimir Configuration Parameters - Experimental Source: https://grafana.com/docs/mimir/latest/configure/configuration-parameters Lists experimental configuration parameters for Grafana Mimir. These parameters are subject to change and are provided for early testing and feedback. ```yaml # (experimental) Maximum number of groups allowed per user by which specified # distributor and ingester metrics can be further separated. # CLI flag: -max-separate-metrics-groups-per-user [max_separate_metrics_groups_per_user: | default = 1000] ``` -------------------------------- ### Get Label Names Endpoint Source: https://grafana.com/docs/mimir/latest/references/http-api This section describes the API endpoint for retrieving label names. It supports both GET and POST methods and is located at the /api/v1/labels path. Caching is possible if query-frontend.cache-results is enabled. ```http GET,POST /api/v1/labels ``` -------------------------------- ### Initialize Jsonnet for Grafana Mimir Mixin Source: https://grafana.com/docs/mimir/latest/manage/monitor-grafana-mimir/installing-dashboards-and-alerts Initializes a Jsonnet environment to manage infrastructure as code for Grafana Mimir. This step is crucial before installing any mixins. ```bash jb init ``` -------------------------------- ### Example Store Gateway Error Log Source: https://grafana.com/docs/mimir/latest/manage/mimir-runbooks This is an example log message from a Mimir store-gateway indicating a problem with object storage, specifically a broken segment data during a multipart upload. ```log create index header reader: write index header: new index reader: get TOC from object storage of 01H9QMTQRE2MT8XVDWP6RWAMC6/index: Multipart upload has broken segment data. ```