Red Hat OpenShift Distributed Tracing Platform

https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9
Admin
Red Hat OpenShift Distributed Tracing Platform provides distributed tracing capabilities using the...
Tokens:20,054
Snippets:258
Trust Score:10
Update:4 months ago
Show doc for...
Tokens:
Raw
### TempoStack Custom Resource Example

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/configuring_distributed_tracing/distr-tracing-tempo-configuring

This is an example of a TempoStack custom resource definition. It shows how to set the deployment size, storage, and enable the query frontend.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: example
  namespace: tracing-system
spec:
  size: 1x.pico
  storageSize: 1Gi
  storage:
    secret:
      name: object-storage-secret
      type: s3
  template:
    queryFrontend:
      jaegerQuery:
        enabled: true
```

--------------------------------

### Create Project for TempoStack Instance

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Create a permitted project for your TempoStack instance. Project names cannot start with 'openshift-'.

```bash
$ oc apply -f - << EOF
apiVersion: project.openshift.io/v1
kind: Project
metadata:
  name: <permitted_project_of_tempostack_instance>
EOF
```

--------------------------------

### Create Permitted Project for TempoMonolithic Instance

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Run this command to create a project for your TempoMonolithic instance. Project names starting with 'openshift-' are not allowed.

```bash
$ oc apply -f - << EOF
apiVersion: project.openshift.io/v1
kind: Project
metadata:
  name: <permitted_project_of_tempomonolithic_instance>
EOF
```

--------------------------------

### Get Tempo Route Details

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Retrieve the route details for accessing the Tempo console.

```bash
oc get route
```

--------------------------------

### Example TempoStack Custom Resource

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/configuring_distributed_tracing/distr-tracing-tempo-configuring

This is a sample TempoStack CR defining the architecture and settings for distributed tracing resources. Modify parameters to customize your implementation.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: <name>
spec:
  storage: {}
  resources: {}
  replicationFactor: 1
  retention:
    global:
      traces: 48h
    perTenant: {}
  template:
      distributor: {}
      ingester: {}
      compactor: {}
      querier: {}
      queryFrontend: {}
      gateway: {}
  limits:
    global:
      ingestion: {}
      query: {}
  observability:
    grafana: {}
    metrics: {}
    tracing: {}
  search: {}
  managementState: managed

```

--------------------------------

### Configure Jaeger Query Ingress and Monitor Tab

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/configuring_distributed_tracing/distr-tracing-tempo-configuring

This example demonstrates how to enable the Jaeger Query frontend, configure its ingress route, and set up the monitor tab with a Prometheus endpoint within a TempoStack custom resource.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: simplest
spec:
  storage:
    secret:
      name: minio
      type: s3
  storageSize: 200M
  resources:
    total:
      limits:
        memory: 2Gi
        cpu: 2000m
  template:
    queryFrontend:
      jaegerQuery:
        enabled: true
        ingress:
          route:
            termination: edge
          type: route
```

--------------------------------

### Define Tenant in TempoStack CR

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

This example shows how to define a tenant within a TempoStack custom resource by adding `tenantName` and `tenantId` parameters. This is part of configuring multi-tenancy and permissions.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: redmetrics
spec:
```

--------------------------------

### Example TempoMonolithic Custom Resource (CR)

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

This YAML defines a TempoMonolithic custom resource for deployment. It allows customization of storage, Jaeger UI, resource limits, and multitenancy settings.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoMonolithic
metadata:
  name: <metadata_name>
  namespace: <permitted_project_of_tempomonolithic_instance>
spec:
  storage:
    traces:
      backend: <supported_storage_type>
      size: <value>Gi
      s3:
        secret: <secret_name>
    tls:
      enabled: true
      caName: <ca_certificate_configmap_name>
  jaegerui:
    enabled: true
    route:
      enabled: true
  resources:
    total:
      limits:
        memory: <value>Gi
        cpu: <value>m
  multitenancy:
    enabled: true
    mode: openshift
    authentication:
      - tenantName: dev
        tenantId: "1610b0c3-c509-4592-a256-a1871353dbfa"
      - tenantName: prod
        tenantId: "1610b0c3-c509-4592-a256-a1871353dbfb"
```

--------------------------------

### Configure Tempo Operator with Subscription CR

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/configuring_distributed_tracing/distr-tracing-tempo-configuring

Use this Subscription CR example to override default Tempo Operator settings with environment variables. These settings take precedence over ConfigMap configurations.

```yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: tempo-product
  namespace: openshift-tempo-operator
spec:
  channel: stable
  name: tempo-product
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  config:
    env:
    - name: FEATURE_GATES
      value: "openshift.route,openshift.servingCertsService"
    - name: TLS_PROFILE
      value: "Modern"
```

--------------------------------

### Create IBM Cloud Object Storage Service Key

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Use this command to create a service key for connecting to an IBM Cloud Object Storage bucket. Ensure you have the IBM Cloud CLI installed and configured.

```bash
$ ibmcloud resource service-key-create <ibm_bucket_name> Writer \
  --instance-name <ibm_bucket_name> --parameters '{"HMAC":true}'
```

--------------------------------

### Configure OpenTelemetry Collector for Span Metrics

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/configuring_distributed_tracing/distr-tracing-tempo-configuring

This example configures the OpenTelemetry Collector to enable the span metrics connector, derive metrics from traces, and export them in Prometheus format. It also sets up OTLP receivers and Prometheus exporters.

```yaml
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel
spec:
  mode: deployment
  observability:
    metrics:
      enableMetrics: true
  config: |
    connectors:
      spanmetrics:
        metrics_flush_interval: 15s

    receivers:
      otlp:
        protocols:
          grpc:
          http:

    exporters:
      prometheus:
        endpoint: 0.0.0.0:8889
        add_metric_suffixes: false
        resource_to_telemetry_conversion:
          enabled: true

      otlp/traces:
        auth:
          authenticator: bearertokenauth
        endpoint: tempo-redmetrics-gateway.mynamespace.svc.cluster.local:8090
        headers:
          X-Scope-OrgID: dev
        tls:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
          insecure: false

    extensions:
      bearertokenauth:
        filename: /var/run/secrets/kubernetes.io/serviceaccount/token

    service:
      extensions:
      - bearertokenauth
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [otlp/traces, spanmetrics]
        metrics:
          receivers: [spanmetrics]
          exporters: [prometheus]

```

--------------------------------

### Get TempoStack Instance Name

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/removing_distributed_tracing/distr-tracing-tempo-removing

Retrieve the name of the TempoStack instance by listing deployments in the specified project. This is a prerequisite for deleting the instance.

```bash
$ oc get deployments -n <project_of_tempostack_instance>
```

--------------------------------

### TempoMonolithic Storage Configuration for Azure

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Example of a TempoMonolithic custom resource configuring Azure as the storage backend for traces. Ensure the secret is correctly referenced.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoMonolithic
metadata:
  name: <name>
  namespace: <namespace>
spec:
# ...
  storage:
    traces:
      backend: azure
      azure:
        secret: <secret_name>
# ...
```

--------------------------------

### Get Jaeger UI Route Details

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Query the route details for the Jaeger UI to access it in a web browser.

```bash
$ oc get route
```

--------------------------------

### Check Tempo Operator CSV Status

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Check the status of the ClusterServiceVersion (CSV) for the Tempo Operator using `oc get csv`. This command helps verify the installation status in the specified namespace.

```bash
$ oc get csv -n openshift-tempo-operator
```

--------------------------------

### TempoStack Custom Resource with Component Resource Overrides

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/configuring_distributed_tracing/distr-tracing-tempo-configuring

Use this example to override default resource requests for specific components like the ingester. This is useful when a component requires more or fewer resources than the default size profile provides.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: example
  namespace: tracing-system
spec:
  size: 1x.small
  storageSize: 1Gi
  storage:
    secret:
      name: object-storage-secret
      type: s3
  template:
    ingester:
      resources:
        requests:
          cpu: 4
          memory: 20Gi
    querier:
      replicas: 4
    queryFrontend:
      jaegerQuery:
        enabled: true

```

--------------------------------

### TempoMonolithic Custom Resource for S3 Storage

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

This example shows how to configure the storage for Tempo using an S3 backend. Ensure you replace `<name>`, `<namespace>`, and `<secret_name>` with your specific values. The `credentialMode: token-cco` is used when the Cloud Credential Operator (CCO) is enabled.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoMonolithic
metadata:
  name: <name>
  namespace: <namespace>
spec:
# ...
  storage:
    traces:
      backend: s3
      s3:
        secret: <secret_name>
        credentialMode: token-cco
# ...
```

--------------------------------

### TempoStack Custom Resource for Azure Storage Configuration

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

An example of a TempoStack custom resource that configures distributed tracing to use Azure Blob Storage. This resource specifies the name and namespace for the Tempo instance and links to the object storage configuration.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: <name>
  namespace: <namespace>
spec:
```

--------------------------------

### ClusterRole for Tempo Trace Reader

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Defines RBAC rules to grant read ('get') permissions for traces within specified tenants. Use this to allow specific users or groups to view trace data.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tempostack-traces-reader
rules:
  - apiGroups:
      - 'tempo.grafana.com'
    resources:
      - dev
      - prod
    resourceNames:
      - traces
    verbs:
      - 'get'
```

--------------------------------

### Collect Diagnostic Data with oc adm must-gather

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/troubleshooting_distributed_tracing/distr-tracing-tempo-troubleshooting

Use this command to gather diagnostic data for TempoStack or TempoMonolithic resources. Specify the operator namespace where the Operator is installed.

```bash
$ oc adm must-gather --image=ghcr.io/grafana/tempo-operator/must-gather -- \
/usr/bin/must-gather --operator-namespace <operator_namespace>
```

--------------------------------

### Configure OpenTelemetryCollector Custom Resource

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Configures the OpenTelemetryCollector custom resource with the bearertokenauth extension and specifies tenant IDs in OTLP exporter headers. This example shows both OTLP gRPC and OTLP HTTP exporters.

```yaml
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: cluster-collector
  namespace: <project_of_tempostack_instance>
spec:
  mode: deployment
  serviceAccount: otel-collector
  config: |
      extensions:
        bearertokenauth:
          filename: "/var/run/secrets/kubernetes.io/serviceaccount/token"
      exporters:
        otlp/dev:
          endpoint: sample-gateway.tempo.svc.cluster.local:8090
          tls:
            insecure: false
            ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt"
          auth:
            authenticator: bearertokenauth
          headers:
            X-Scope-OrgID: "dev"
        otlphttp/dev:
          endpoint: https://sample-gateway.<project_of_tempostack_instance>.svc.cluster.local:8080/api/traces/v1/dev
          tls:
            insecure: false
            ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt"
          auth:
            authenticator: bearertokenauth
          headers:
            X-Scope-OrgID: "dev"
      service:
        extensions: [bearertokenauth]
        pipelines:
          traces:
            exporters: [otlp/dev]
```

--------------------------------

### Example TempoStack CR for AWS S3 and MinIO with Two Tenants

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

This TempoStack custom resource (CR) configures a TempoStack deployment for AWS S3 and MinIO storage, enabling a gateway for authentication and authorization, and exposing Jaeger UI for data visualization. It also defines two tenants, 'dev' and 'prod', with unique tenant IDs.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: simplest
  namespace: <permitted_project_of_tempostack_instance>
spec:
  storage:
    secret:
      name: <secret_name>
      type: <secret_provider>
  storageSize: <value>Gi
  resources:
    total:
      limits:
        memory: 2Gi
        cpu: 2000m
  tenants:
    mode: openshift
    authentication:
      - tenantName: dev
        tenantId: "1610b0c3-c509-4592-a256-a1871353dbfa"
      - tenantName: prod
        tenantId: "1610b0c3-c509-4592-a256-a1871353dbfb"
  template:
    gateway:
      enabled: true
    queryFrontend:
      jaegerQuery:
        enabled: true
```

--------------------------------

### Get OpenID Connect (OIDC) Issuer URL

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Retrieve the OpenID Connect (OIDC) issuer URL for your cluster, which is required for creating federated identity credentials. This command uses `oc` and `jq` to extract the URL from the cluster authentication configuration.

```bash
$ oc get authentication cluster -o json | jq -r .spec.serviceAccountIssuer
```

--------------------------------

### Create Project for Tempo Operator

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Create a dedicated project for the Tempo Operator using `oc apply`. This command defines the project's metadata and labels, including cluster monitoring.

```bash
$ oc apply -f - << EOF
apiVersion: project.openshift.io/v1
kind: Project
metadata:
  labels:
    kubernetes.io/metadata.name: openshift-tempo-operator
    openshift.io/cluster-monitoring: "true"
  name: openshift-tempo-operator
EOF
```

--------------------------------

### Create Subscription for Tempo Operator

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Create a subscription for the Tempo Operator using `oc apply`. This configures the operator's channel, approval strategy, name, source, and source namespace.

```bash
$ oc apply -f - << EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: tempo-product
  namespace: openshift-tempo-operator
spec:
  channel: stable
  installPlanApproval: Automatic
  name: tempo-product
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF
```

--------------------------------

### Create GCP Service Account

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Creates a new service account on Google Cloud Platform with a specified display name and project. It outputs the email of the created service account.

```bash
SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts create <iam_service_account_name> \
    --display-name="Tempo Account" \
    --project <project_id>  \
    --format='value(email)' \
    --quiet)
```

--------------------------------

### Get OpenShift Service Account Issuer

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Retrieve the OIDC provider's service account issuer URL from your OpenShift cluster configuration. This value is used in the IAM trust policy.

```bash
$ oc get authentication cluster -o json | jq -r '.spec.serviceAccountIssuer' | sed 's~http[s]*://~~g'
```

--------------------------------

### Create Federated Identity Credential for Tempo Components

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

This command establishes a federated identity credential, allowing distributed tracing service accounts to authenticate as an Azure managed identity without secrets. This is for all components except the Query Frontend. Replace placeholders with your credential name, identity name, resource group, OIDC provider URL, service account subject, and audience.

```bash
$ az identity federated-credential create \
  --name <credential_name> \
  --identity-name <identity_name> \
  --resource-group <resource_group> \
  --issuer <oidc_provider> \
  --subject <tempo_service_account_subject> \
  --audiences <audience>
```

--------------------------------

### Enable TempoStack Metrics

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/configuring_distributed_tracing/distr-tracing-tempo-configuring

Set `spec.observability.metrics.createServiceMonitors` to `true` to enable metrics for a TempoStack instance. This is useful for monitoring the health and performance of your TempoStack deployment.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: <name>
spec:
  observability:
    metrics:
      createServiceMonitors: true
```

--------------------------------

### Verify TempoMonolithic Pod Status

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Check if the pod for the TempoMonolithic instance is running.

```bash
$ oc get pods
```

--------------------------------

### Example Secret for Amazon S3 and MinIO Storage

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

This YAML defines a Kubernetes Secret for configuring object storage access for TempoStack. It includes endpoint, bucket name, and credentials for services like Amazon S3 or MinIO.

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: minio-test
stringData:
  endpoint: http://minio.minio.svc:9000
  bucket: tempo
  access_key_id: tempo
  access_key_secret: <secret>
type: Opaque
```

--------------------------------

### Enable TempoStack Alerts

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/configuring_distributed_tracing/distr-tracing-tempo-configuring

Set `spec.observability.metrics.createPrometheusRules` to `true` to enable alerts for a TempoStack instance. This ensures you are notified of potential issues with your TempoStack deployment.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: <name>
spec:
  observability:
    metrics:
      createPrometheusRules: true
```

--------------------------------

### Verify TempoMonolithic Status

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Verify that all components of the TempoMonolithic instance are running and ready by checking its status and conditions.

```bash
$ oc get tempomonolithic.tempo.grafana.com <metadata_name_of_tempomonolithic_cr> -o yaml
```

--------------------------------

### Verify TempoStack Pods Status

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Confirm that all pods associated with the TempoStack deployment are in a running state.

```bash
oc get pods
```

--------------------------------

### Create ClusterRole and ClusterRoleBinding for Tempo RBAC

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/configuring_distributed_tracing/distr-tracing-tempo-configuring

Define a ClusterRole and ClusterRoleBinding to grant authenticated users permissions to access specific tenants for Tempo trace data. This is part of setting up query RBAC.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tempo-dev-read
rules:
- apiGroups: [tempo.grafana.com]
  resources: [dev]
  resourceNames: [traces]
  verbs: [get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tempo-dev-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tempo-dev-read
subjects:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: system:authenticated
```

--------------------------------

### Get Azure Managed Identity Client ID

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Fetch the client ID of the Azure managed identity you created. This ID is often required for configuring applications or services to use the managed identity. Replace placeholders with your identity name and resource group.

```bash
CLIENT_ID=$(az identity show \
  --name <identity_name> \
  --resource-group <resource_group> \
  --query clientId \
  -o tsv)
```

--------------------------------

### Apply TempoStack Custom Resource

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Apply the customized TempoStack Custom Resource (CR) to your OpenShift cluster.

```bash
$ oc apply -f - << EOF
<tempostack_cr>
EOF
```

--------------------------------

### Create Operator Group for Tempo Operator

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Create an OperatorGroup for the Tempo Operator using `oc apply`. This defines the group's name and namespace, specifying the `Default` upgrade strategy.

```bash
$ oc apply -f - << EOF
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: openshift-tempo-operator
  namespace: openshift-tempo-operator
spec:
  upgradeStrategy: Default
EOF
```

--------------------------------

### Create GCP Workload Identity Pool Credential Configuration

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Generates a credential configuration file for a GCP Workload Identity Pool, using a service account and a token from a specified path.

```bash
$ gcloud iam workload-identity-pools create-cred-config \
    "projects/<project_number>/locations/global/workloadIdentityPools/<pool_id>/providers/<provider_id>" \
    --service-account="$SERVICE_ACCOUNT_EMAIL" \
    --credential-source-file=/var/run/secrets/storage/serviceaccount/token \
    --credential-source-type=text \
    --output-file=<output_file_path>
```

--------------------------------

### Create ClusterRoleBinding for OpenTelemetry Collector

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Binds the OpenTelemetry Collector service account to the ClusterRole that grants write permissions for traces. This allows the collector to write trace data.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tempostack-traces
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tempostack-traces-write
subjects:
  - kind: ServiceAccount
    name: otel-collector
    namespace: otel
```

--------------------------------

### Configure Receiver TLS with User-Provided Certificate

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/configuring_distributed_tracing/distr-tracing-tempo-configuring

Enables TLS for Tempo receivers using a user-provided certificate stored in a secret. Requires a tls.key and tls.crt in the specified secret.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind:  TempoStack
# ...
spec:
# ...
  template:
    distributor:
      tls:
        enabled: true
        certName: <tls_secret>
        caName: <ca_name>
# ...
```

--------------------------------

### Add Role to User for Project Access

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/configuring_distributed_tracing/distr-tracing-tempo-configuring

Use this command to grant a user the 'view' role for a specific project, allowing them to read attributes. This is a step in configuring query RBAC.

```bash
$ oc adm policy add-role-to-user view <username> -n <project>
```

--------------------------------

### Enable Multitenancy and Query RBAC in TempoStack

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/configuring_distributed_tracing/distr-tracing-tempo-configuring

Configure the TempoStack custom resource to enable multitenancy and query RBAC. Ensure the gateway is enabled and Jaeger query is disabled for this configuration.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: simplest
  namespace: chainsaw-multitenancy
spec:
  storage:
    secret:
      name: minio
      type: s3
  storageSize: 1Gi
  resources:
    total:
      limits:
        memory: 2Gi
        cpu: 2000m
  tenants:
    mode: openshift
    authentication:
      - tenantName: dev
        tenantId: "1610b0c3-c509-4592-a256-a1871353dbfb"
  template:
    gateway:
      enabled: true
      rbac:
        enabled: true
    queryFrontend:
      jaegerQuery:
        enabled: false
```

--------------------------------

### Create Object Storage Secret

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Create a secret for your object storage bucket within the TempoStack instance's project.

```bash
$ oc apply -f - << EOF
<object_storage_secret>
EOF
```

--------------------------------

### Create Federated Identity Credential for Query Frontend

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

This command creates a federated identity credential specifically for the Query Frontend component of distributed tracing. It allows the frontend service account to authenticate as an Azure managed identity. Replace placeholders with your credential name, identity name, resource group, cluster issuer URL, service account subject, and audience.

```bash
$ az identity federated-credential create \
  --name <credential_name>-frontend \
  --identity-name <identity_name> \
  --resource-group <resource_group> \
  --issuer <cluster_issuer> \
  --subject <tempo_service_account_query_frontend_subject> \
  --audiences <audience> | jq
```

--------------------------------

### Configure Receiver TLS with OpenShift Service Serving Certificates

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/configuring_distributed_tracing/distr-tracing-tempo-configuring

This snippet shows how to enable TLS for Tempo receivers using service serving certificates automatically generated by OpenShift. Mutual TLS authentication is not supported with this method.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind:  TempoMonolithic
# ...
  spec:
# ...
  ingestion:
    otlp:
      grpc:
        tls:
          enabled: true
      http:
        tls:
          enabled: true
# ...
```

--------------------------------

### Configure GCS Storage in TempoMonolithic CR

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

This snippet shows how to configure Google Cloud Storage (GCS) as the backend for trace storage within a TempoMonolithic custom resource.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoMonolithic
metadata:
  name: <name>
  namespace: <namespace>
spec:
# ...
  storage:
    traces:
      backend: gcs
      gcs:
        secret: <secret_name>
```

--------------------------------

### Apply Customized TempoMonolithic CR

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Apply the customized TempoMonolithic Custom Resource (CR) definition to your OpenShift cluster.

```bash
$ oc apply -f - << EOF
<tempomonolithic_cr>
EOF
```

--------------------------------

### Create Service Account for OpenTelemetry Collector

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Creates a Kubernetes ServiceAccount for the OpenTelemetry Collector. Ensure the namespace matches your OpenTelemetry Collector instance.

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-collector
  namespace: <project_of_opentelemetry_collector_instance>
```

--------------------------------

### Configure Receiver TLS with OpenShift Service Serving Certificates

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/configuring_distributed_tracing/distr-tracing-tempo-configuring

Enables TLS for Tempo receivers using OpenShift's service-serving certificates. Mutual TLS authentication is not supported with this configuration.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind:  TempoStack
# ...
spec:
# ...
  template:
    distributor:
      tls:
        enabled: true
# ...
```

--------------------------------

### Create AWS IAM Role for S3 Access

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Create an AWS IAM role named 'tempo-s3-access' and attach the previously defined trust policy document. This command outputs the ARN of the created role.

```bash
$ aws iam create-role \
      --role-name "tempo-s3-access" \
      --assume-role-policy-document "file:///tmp/trust.json" \
      --query Role.Arn \
      --output text
```

--------------------------------

### Configure IBM COS Storage in TempoStack CR

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

This snippet demonstrates how to configure IBM Cloud Object Storage (COS) as the storage backend for a TempoStack custom resource, referencing the previously created secret.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
# ...
spec:
# ...
  storage:
    secret:
      name: <ibm_cos_secret>
      type: s3
```

--------------------------------

### Verify TempoStack Component Status

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Check the status and conditions of TempoStack components to ensure they are running and ready.

```bash
oc get tempostacks.tempo.grafana.com simplest -o yaml
```

--------------------------------

### Bind GCP Roles to Service Account

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

Binds the 'roles/storage.objectAdmin' role to a service account at the project level, granting it administrative access to storage objects.

```bash
$ gcloud projects add-iam-policy-binding <project_id> \
    --member "serviceAccount:$SERVICE_ACCOUNT_EMAIL" \
    --role "roles/storage.objectAdmin"
```

--------------------------------

### Verify Cluster Administrator Login

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/removing_distributed_tracing/distr-tracing-tempo-removing

Verify your login to the OpenShift cluster as a cluster administrator. Ensure your OpenShift CLI (oc) version is up to date.

```bash
$ oc whoami
```

--------------------------------

### Configure Receiver TLS with User-Provided Certificate

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/configuring_distributed_tracing/distr-tracing-tempo-configuring

Use this snippet to enable TLS for Tempo receivers by specifying a secret containing your TLS certificate and key. This is applicable when not using the Tempo Gateway.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind:  TempoMonolithic
# ...
  spec:
# ...
  ingestion:
    otlp:
      grpc:
        tls:
          enabled: true
          certName: <tls_secret>
          caName: <ca_name>
# ...
```

--------------------------------

### Create OpenShift Secret for IBM COS Credentials

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/installing_distributed_tracing/distr-tracing-tempo-installing

This command creates a generic secret in OpenShift containing the necessary credentials to access an IBM Cloud Object Storage bucket.

```bash
$ oc -n <namespace> create secret generic <ibm_cos_secret> \
  --from-literal=bucket="<ibm_bucket_name>" \
  --from-literal=endpoint="<ibm_bucket_endpoint>" \
  --from-literal=access_key_id="<ibm_bucket_access_key>" \
  --from-literal=access_key_secret="<ibm_bucket_secret_key>"
```

--------------------------------

### Enable Monitor Tab in TempoStack

Source: https://docs.redhat.com/en/documentation/red_hat_openshift_distributed_tracing_platform/3.9/html/configuring_distributed_tracing/distr-tracing-tempo-configuring

Enables the monitoring tab in the Jaeger console and sets the Prometheus endpoint for querying data from a user-defined monitoring stack.

```yaml
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: redmetrics
spec:
  storage:
    secret:
      name: minio-test
      type: s3
  storageSize: 1Gi
  tenants:
    mode: openshift
    authentication:
      - tenantName: dev
        tenantId: "1610b0c3-c509-4592-a256-a1871353dbfa"
  template:
    gateway:
      enabled: true
    queryFrontend:
      jaegerQuery:
        monitorTab:
          enabled: true
          prometheusEndpoint: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
          redMetricsNamespace: ""
```
Tokens:
Raw