Grafana Tempo

https://grafana.com/docs/tempo/latest
Admin
Grafana Tempo is an open-source, high-scale distributed tracing backend designed to search traces,...
Tokens:182,954
Snippets:2,460
Trust Score:10
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install MinIO from Source

Source: https://grafana.com/docs/tempo/latest/configuration/hosted-storage/s3

Install the MinIO server from source using Go. This is necessary as pre-compiled binaries are no longer published. Ensure Go 1.24 or later is installed.

```bash
go install github.com/minio/minio@latest
```

--------------------------------

### Start SeaweedFS in Mini Mode

Source: https://grafana.com/docs/tempo/latest/configuration/hosted-storage/s3

Use this command to start a single-node SeaweedFS setup with an S3 gateway for local evaluation. Ensure the data directory exists and has correct ownership.

```bash
sudo mkdir -p /data/seaweedfs
sudo chown -R $USER:$USER /data/seaweedfs
weed mini -dir=/data/seaweedfs
```

--------------------------------

### Install Tempo Jsonnet library

Source: https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/deploy/kubernetes/tanka

Install the Tempo microservices library and its dependencies.

```bash
jb install github.com/grafana/tempo/operations/jsonnet/microservices@main
```

--------------------------------

### Migrate Tenant Example

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

Example of migrating tenant data using specific configuration files for source and destination backends.

```bash
tempo-cli migrate tenant --source-config-file source.yaml --config-file dest.yaml my-tenant my-other-tenant
```

--------------------------------

### Navigate to Docker Compose Example

Source: https://grafana.com/docs/tempo/latest/docker-example

Change directory into the single-binary Docker Compose example within the cloned Tempo repository.

```bash
cd tempo/example/docker-compose/single-binary
```

--------------------------------

### Example: Query Trace Summary

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

A basic example of how to query for a trace summary using a specific trace ID and tenant.

```bash
tempo-cli query trace-summary f1cfe82a8eef933b single-tenant
```

--------------------------------

### Tempo CLI Query Search Example (Relative Time)

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

Example of searching traces using relative time values for the start and end of the search range. This command also specifies backend and bucket configurations.

```bash
tempo-cli query search http.method GET now-1h now single-tenant --backend=gcs --bucket=tempo-trace-data
```

--------------------------------

### Migrate Tempo Config Example (Monolithic Mode)

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

Example of migrating a Tempo 2.x configuration file to monolithic mode. The migrated configuration is printed to stdout, and warnings are sent to stderr. Review the output carefully as it's a starting point.

```bash
tempo-cli migrate config old-config.yaml > new-config.yaml
```

--------------------------------

### Run Tempo in Monolithic Mode

Source: https://grafana.com/docs/tempo/latest/reference-tempo-architecture/deployment-modes

This command starts Tempo with all components compiled into a single binary, which is the default monolithic deployment mode. It's useful for getting started or development environments.

```bash
tempo -target=all
```

--------------------------------

### Example Cache Configuration

Source: https://grafana.com/docs/tempo/latest/configuration

A complete YAML example demonstrating how to configure both Memcached and Redis roles within the cache block.

```yaml
cache:
  background:
    writeback_goroutines: 5
  caches:
    - roles:
        - parquet-footer
      memcached:
        host: memcached-instance
    - roles:
        - bloom
      redis:
        endpoint: redis-instance
```

--------------------------------

### Start Tempo with a Configuration File

Source: https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/command-line-flags

Use this command to start Tempo using a specified configuration file. This is the standard way to manage Tempo's settings.

```bash
tempo --config.file=/etc/tempo/config.yaml
```

--------------------------------

### Example Usage Metrics Output

Source: https://grafana.com/docs/tempo/latest/configuration/usage-tracker

This example shows the format of the usage metrics exposed by the distributor, typically in Prometheus exposition format.

```none
curl http://localhost:3200/usage_metrics
# HELP tempo_usage_tracker_bytes_received_total bytes total received with these attributes
# TYPE tempo_usage_tracker_bytes_received_total counter
tempo_usage_tracker_bytes_received_total{service_name="article-service",tenant="single-tenant",tracker="cost-attribution"} 7327
tempo_usage_tracker_bytes_received_total{service_name="auth-service",tenant="single-tenant",tracker="cost-attribution"} 8938
tempo_usage_tracker_bytes_received_total{service_name="billing-service",tenant="single-tenant",tracker="cost-attribution"} 2401
tempo_usage_tracker_bytes_received_total{service_name="cart-service",tenant="single-tenant",tracker="cost-attribution"} 4116
tempo_usage_tracker_bytes_received_total{service_name="postgres",tenant="single-tenant",tracker="cost-attribution"} 3571
tempo_usage_tracker_bytes_received_total{service_name="shop-backend",tenant="single-tenant",tracker="cost-attribution"} 17619
```

--------------------------------

### Create installation directory

Source: https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/deploy/kubernetes/tanka

Initialize a new directory for the Tempo deployment project.

```bash
mkdir tempo
cd tempo
```

--------------------------------

### Example: Query Trace ID

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

A basic example of how to query for a specific trace ID and tenant.

```bash
tempo-cli query trace-id f1cfe82a8eef933b single-tenant
```

--------------------------------

### Tempo CLI Example Range Query for Request Rates

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

Example of a range query to get request rates by service. It specifies the host-port, a TraceQL query, and a time range of the last hour.

```bash
tempo-cli query api metrics localhost:3200 "{} | rate() by (resource.service.name)" now-1h now
```

--------------------------------

### Tempo CLI: Search Tags Example (Basic)

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

A basic example of searching for tags using the Tempo CLI, specifying the host and port.

```bash
tempo-cli query api search-tags localhost:3200
```

--------------------------------

### Tempo CLI Query Search Example (Absolute Time)

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

Example of searching traces using absolute RFC3339 formatted timestamps for the start and end of the search range. This command also specifies backend and bucket configurations.

```bash
tempo-cli query search http.method GET 2024-01-01T00:00:00Z 2024-01-01T00:05:00Z single-tenant --backend=gcs --bucket=tempo-trace-data
```

--------------------------------

### Install Memcached library

Source: https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/deploy/kubernetes/tanka

Install the Memcached library required for Tempo operations.

```bash
jb install github.com/grafana/jsonnet-libs/memcached@master
```

--------------------------------

### Deploy MinIO Storage Backend

Source: https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/deploy/kubernetes/operator/quickstart

Apply the MinIO deployment manifest to set up MinIO as a storage backend for Tempo. This is used for the quick start guide.

```shell
kubectl apply -f https://raw.githubusercontent.com/grafana/tempo-operator/main/minio.yaml
```

--------------------------------

### Tempo CLI: Search Tag Values Example (Service Names)

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

Example to find all service names by searching tag values using the Tempo CLI.

```bash
tempo-cli query api search-tag-values localhost:3200 resource.service.name
```

--------------------------------

### Tempo Live Store Configuration Example

Source: https://grafana.com/docs/tempo/latest/configuration/manifest

Configuration for the live store component, including ring settings, metrics, WAL, and block management. This example shows settings for Consul and etcd as key-value stores.

```yaml
tls_min_version: ""
username: ""
password: ""
multi:
    primary: ""
    secondary: ""
    mirror_enabled: false
    mirror_timeout: 2s
heartbeat_period: 5s
heartbeat_timeout: 1m0s
wait_stability_min_duration: 1m0s
wait_stability_max_duration: 5m0s
instance_id: hostname
instance_interface_names:
    - eth0
    - en0
instance_port: 0
instance_addr: ""
enable_inet6: false
wait_active_instance_timeout: 10m0s
finish_on_shutdown_timeout: 30s
live_store:
    ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 5s
        heartbeat_timeout: 1m0s
        instance_id: hostname
        instance_interface_names:
            - eth0
        instance_addr: ""
        instance_port: 0
        enable_inet6: false
        instance_zone: ""
    partition_ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        min_partition_owners_count: 1
        min_partition_owners_duration: 10s
        delete_inactive_partition_after: 13h0m0s
    metrics:
        time_overlap_cutoff: 0.2
    commit_interval: 5s
    wal:
        path: /var/tempo/live-store/traces
        ingestion_time_range_slack: 2m0s
    query_block_concurrency: 10
    complete_block_timeout: 20m0s
    complete_block_concurrency: 2
    block_reclaim_grace: 2m0s
    shutdown_marker_dir: /var/tempo/live-store/shutdown-marker
    flush_check_period: 5s
    flush_op_timeout: 5m0s
    max_trace_live: 30s
    max_trace_idle: 5s
    max_live_traces_bytes: 250000000
    max_block_duration: 30s
    max_block_bytes: 52428800
    readiness_target_lag: 0s
    readiness_max_wait: 30m0s
    fail_on_high_lag: true
    remove_owner_on_shutdown: true
```

--------------------------------

### Generate Attribute Index with Example Path

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

Example of generating an attribute index for a specified path.

```bash
tempo-cli gen attr-index ./path/to/block
```

--------------------------------

### Migrate Per-Tenant Overrides Example

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

Example demonstrating how to migrate a per-tenant overrides file and specify an output destination for the migrated content. Be aware of potential data loss due to `omitempty` tags and the need to manually restore secret values.

```bash
tempo-cli migrate overrides-per-tenant overrides.yaml -d migrated-overrides.yaml
```

--------------------------------

### Install k8s-libsonnet

Source: https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/deploy/kubernetes/tanka

Install the Kubernetes library for Jsonnet matching your cluster version.

```bash
mkdir -p lib
export K8S_VERSION=1.32
jb install github.com/jsonnet-libs/k8s-libsonnet/${K8S_VERSION}@main
cat <<EOF > lib/k.libsonnet
import 'github.com/jsonnet-libs/k8s-libsonnet/${K8S_VERSION}/main.libsonnet'
EOF
```

--------------------------------

### Example HTTP Headers Configuration

Source: https://grafana.com/docs/tempo/latest/configuration

A practical example demonstrating how to configure masked secrets, plaintext values, and file-based header inputs within a remote write block.

```yaml
metrics_generator:
  storage:
    remote_write:
      - url: https://prometheus.example.com/api/v1/write
        http_headers:
          # Masked as <secret> in the displayed configuration.
          X-Custom-Token:
            secrets:
              - <SECRET_VALUE>
          # Stored and displayed in plaintext.
          X-Custom-Header:
            values:
              - custom-value
          # Read from a file on disk.
          X-Token-From-File:
            files:
              - /etc/tempo/custom-token
```

--------------------------------

### Install Tempo Package on Linux

Source: https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/deploy/locally/linux

Installs the downloaded Tempo .deb package using `dpkg`. This command requires administrator privileges and should be run with `sudo`.

```bash
sudo dpkg -i tempo_<TEMPO_VERSION_NUMBER>_linux_amd64.deb
```

--------------------------------

### Migrate Tempo Config Example (Microservices Mode)

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

Example of migrating a Tempo 2.x configuration file to microservices mode, specifying Kafka broker address and topic. Ensure legacy overrides are migrated first and manually adjust per-tenant overrides if necessary.

```bash
tempo-cli migrate config --kafka-address=kafka:9092 --kafka-topic=tempo-traces old-config.yaml > new-config.yaml
```

--------------------------------

### Tempo CLI: Search Tags Example (Absolute Time)

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

Example of searching for tags with an absolute time range using the Tempo CLI.

```bash
tempo-cli query api search-tags localhost:3200 2024-01-01T00:00:00Z 2024-01-02T00:00:00Z
```

--------------------------------

### Tempo CLI: Search Tags Example (Relative Time)

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

Example of searching for tags with a relative time range using the Tempo CLI.

```bash
tempo-cli query api search-tags localhost:3200 now-1h now
```

--------------------------------

### Tempo CLI Parquet Convert v3 to v4 Example

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

Example demonstrating the conversion of a vParquet3 block to vParquet4, specifying an input file, output path, and dedicated columns.

```bash
tempo-cli parquet convert-3to4 ./block-in ./out span.db.statement span.db.name
```

--------------------------------

### Start MinIO Server

Source: https://grafana.com/docs/tempo/latest/configuration/hosted-storage/s3

Start the MinIO server, specifying the data directory. The server runs in the foreground and uses default credentials 'minioadmin' for access and secret keys. The console is available on port 9001.

```bash
sudo mkdir -p /data/minio
sudo chown -R $USER:$USER /data/minio
minio server /data/minio --console-address ':9001'
```

--------------------------------

### Start Tempo with a Specific Target Module

Source: https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/command-line-flags

This command starts Tempo and targets a specific module (e.g., distributor) for configuration or operation, using a configuration file.

```bash
tempo --target=distributor --config.file=/etc/tempo/config.yaml
```

--------------------------------

### Example Docker Compose Services Output

Source: https://grafana.com/docs/tempo/latest/docker-example

An example of the expected output from 'docker compose ps', showing the status and ports of running services.

```console
docker compose ps
NAME                            COMMAND                  SERVICE             STATUS              PORTS
single-binary-alloy-1           "/bin/alloy run /etc…"   alloy               running             0.0.0.0:4319->4317/tcp, 0.0.0.0:12345->12345/tcp
single-binary-grafana-1         "/run.sh"                grafana             running             0.0.0.0:3000->3000/tcp
single-binary-k6-tracing-1      "/k6-tracing run /ex…"   k6-tracing          running
single-binary-prometheus-1      "/bin/prometheus --c…"   prometheus          running             0.0.0.0:9090->9090/tcp
single-binary-redpanda-1        "/entrypoint.sh redp…"   redpanda            running             0.0.0.0:9092->9092/tcp
single-binary-redpanda-console-1 "/bin/sh -c 'echo \"$…" redpanda-console    running             0.0.0.0:8080->8080/tcp
single-binary-tempo-1           "/tempo -target=all …"   tempo               running             0.0.0.0:3200->3200/tcp, 0.0.0.0:4317-4318->4317-4318/tcp
single-binary-vulture-1         "/tempo-vulture -pro…"   vulture             running
```

--------------------------------

### Start Tempo in Monolithic Mode with Multitenancy and Graceful Shutdown

Source: https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/command-line-flags

Starts Tempo in monolithic mode, enables multitenancy, and configures a graceful shutdown delay. This allows in-flight requests to complete during rolling updates.

```bash
tempo --config.file=/etc/tempo/config.yaml \
  --multitenancy.enabled \
  --shutdown-delay=30s
```

--------------------------------

### Tempo CLI: Search Tag Values Example (Filtered)

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

Example of searching for tag values with a TraceQL query filter to find service names that have errors.

```bash
tempo-cli query api search-tag-values --query '{status = error}' localhost:3200 resource.service.name
```

--------------------------------

### Tempo CLI Parquet Convert v4 to v5 Example

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

Example demonstrating the conversion of a vParquet4 block to vParquet5, specifying dedicated columns with type prefixes for integer and blob encoding.

```bash
tempo-cli parquet convert-4to5 ./block-in ./block-out "span.http.method" "int/span.http.status_code" "blob/span.db.statement"
```

--------------------------------

### Configure Storage Class for Live Store

Source: https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/deploy/kubernetes/tanka

Add this configuration to the _config section if the live-stores fail to start due to storage class issues.

```jsonnet
  live_store+: {
    pvc_storage_class: 'standard',
  },
```

--------------------------------

### Enable KEDA autoscaling for Tempo components

Source: https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/deploy/kubernetes/tanka

Configure KEDA scalers for distributor, metrics-generator, backend-worker, and block-builder components. Ensure the KEDA operator and CRDs are installed in your cluster before applying.

```jsonnet
_config+:: {
  distributor+: {
    keda: {
      enabled: true,
      min_replicas: 2,
      max_replicas: 200,
      target_cpu: '330m',
    },
  },
  metrics_generator+: {
    keda: {
      enabled: true,
      min_replicas: 1,
      max_replicas: 200,
      target_cpu: '500m',
    },
  },
  backend_worker+: {
    keda: {
      enabled: true,
      min_replicas: 3,
      max_replicas: 200,
      prometheus_address: 'http://prometheus-operated.monitoring.svc.cluster.local:9090',
      threshold: 200,
    },
  },
  block_builder+: {
    keda: {
      enabled: true,
      min_replicas: 1,
      max_replicas: 200,
      partitions_per_instance: 1,
      pod_selector: 'name=live-store-zone-a',
    },
  },
},
```

--------------------------------

### Retrieve Overrides with Etag

Source: https://grafana.com/docs/tempo/latest/operations/manage-advanced-systems/user-configurable-overrides

Example of a GET request showing the Etag header, which represents the current version of the overrides.

```shell
$ curl -v http://localhost:3100/api/overrides
... 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Etag: 1697726795401423
< Date: Wed, 07 Feb 2024 17:49:04 GMT
< Content-Length: 118
...
```

--------------------------------

### Build and Run Local Tempo CLI Binary

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

Steps to build a local Tempo CLI binary using 'make' and then run it. Requires a working Go installation.

```bash
make tempo-cli
./bin/$(go env GOOS)/tempo-cli-$(go env GOARCH) [arguments...]
```

--------------------------------

### Tempo CLI Example Instant Query for Error Rates

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

Example of an instant query to get current error rates. It uses the --instant flag and specifies a TraceQL query for errors within the last hour.

```bash
tempo-cli query api metrics --instant localhost:3200 "{status = error} | rate()" now-1h now
```

--------------------------------

### Initialize Tanka environment

Source: https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/deploy/kubernetes/tanka

Initialize a new Tanka project and configure the environment for Tempo.

```bash
tk init --k8s=false
tk env add environments/tempo
tk env set environments/tempo \
 --namespace=tempo \
 --server-from-context=$(kubectl config current-context)
```

--------------------------------

### Count Spans by Status Code Over Time

Source: https://grafana.com/docs/tempo/latest/metrics-from-traces/metrics-queries/functions

This example counts the number of spans with the name 'GET /:endpoint' and aggregates the counts by HTTP status code. It demonstrates how to use `count_over_time` with the `by()` operator.

```traceql
{ span:name = "GET /:endpoint" } | count_over_time() by (span.http.status_code)
```

--------------------------------

### Build and Run Tempo Vulture from Source

Source: https://grafana.com/docs/tempo/latest/operations/tempo-vulture

Build Tempo Vulture from source using 'make' and then run the executable. Replace '[arguments...]' with desired configuration options.

```bash
make tempo-vulture
./tempo-vulture [arguments...]
```

--------------------------------

### Start rclone S3 Server

Source: https://grafana.com/docs/tempo/latest/configuration/hosted-storage/s3

Launch an rclone S3 server that serves a local directory. This is an experimental feature and requires rclone to be installed. It runs on port 8080 with specified authentication keys.

```bash
sudo mkdir -p /data/rclone-s3
sudo chown -R $USER:$USER /data/rclone-s3
rclone serve s3 /data/rclone-s3 --auth-key tempokey,temposecret --addr :8080
```

--------------------------------

### Compare Spans with Time Window

Source: https://grafana.com/docs/tempo/latest/metrics-from-traces/metrics-queries/functions

This example shows how to use the `compare` function with a specific time window defined by start and end Unix nanoseconds. It filters for spans with `resource.service.name="a"` and `span.http.path="/myapi"`, then compares them against spans with `span:status=error` within the specified time range.

```traceql
{ resource.service.name="a" && span.http.path="/myapi" } | compare({span:status=error}, 10, 1678886400000000000, 1678886460000000000)
```

--------------------------------

### Tempo CLI Help Commands

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

Examples of how to display help for the Tempo CLI and its commands or subcommands.

```bash
tempo-cli -h
tempo-cli command [subcommand] -h
```

--------------------------------

### Install Tempo Operator

Source: https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/deploy/kubernetes/operator/quickstart

Apply the Tempo Operator manifest to your Kubernetes cluster. Ensure cert-manager is installed beforehand.

```shell
kubectl apply -f https://github.com/grafana/tempo-operator/releases/latest/download/tempo-operator.yaml
```

--------------------------------

### Migrate Overrides Config Example

Source: https://grafana.com/docs/tempo/latest/operations/tempo_cli

Example of migrating overrides configuration, specifying an output file for the migrated section.

```bash
tempo-cli migrate overrides-config config.yaml -d migrated-overrides.yaml
```

--------------------------------

### Tempo Configuration with S3 Backend

Source: https://grafana.com/docs/tempo/latest/configuration/hosted-storage/s3

This is a comprehensive Tempo configuration example using an S3-compatible backend. It includes settings for the distributor, backend scheduler, backend worker, and metrics generator. Ensure you replace placeholder values like <S3_ENDPOINT>, <S3_ACCESS_KEY>, and <S3_SECRET_KEY> with your specific object store credentials.

```yaml
stream_over_http_enabled: true

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"
        http:
          endpoint: "0.0.0.0:4318"

backend_scheduler:
  provider:
    compaction:
      compaction:
        block_retention: 1h

backend_worker:
  compaction:
    block_retention: 1h

metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: linux-monolithic
  storage:
    path: /tmp/tempo/generator/wal
    remote_write:
      - url: http://<PROMETHEUS_URL>/api/v1/write
        send_exemplars: true

storage:
  trace:
    backend: s3
    s3:
      endpoint: <S3_ENDPOINT>
      bucket: tempo
      access_key: <S3_ACCESS_KEY>
      secret_key: <S3_SECRET_KEY>
      insecure: true
  wal:
    path: /var/tempo/wal

overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics]

usage_report:
  reporting_enabled: false

```
Tokens:
Raw