OpenTelemetry at Grafana Labs

https://grafana.com/docs/opentelemetry/
Admin
OpenTelemetry at Grafana Labs provides tools and guidance for instrumenting applications, building...
Tokens:56,637
Snippets:274
Trust Score:10
Update:1 month ago
Show doc for...
Tokens:
Raw
### Run the example REST service

Source: https://grafana.com/docs/opentelemetry/docker-lgtm

Commands to run the example REST service on Unix/Linux, Windows PowerShell, and via mise. The example app implements a rolldice service.

```sh
# Unix/Linux
./run-example.sh

# Windows (PowerShell)
./run-example

# Using mise (Unix/Linux)
mise run example
```

--------------------------------

### Install OpenTelemetry PHP SDK via Composer

Source: https://grafana.com/docs/opentelemetry/instrument/php

Run this command in your project folder to install the OpenTelemetry SDK, OTLP exporter, and Guzzle 7 adapter.

```sh
composer require \
  open-telemetry/sdk \
  open-telemetry/exporter-otlp \
  php-http/guzzle7-adapter
```

--------------------------------

### Install OpenTelemetry SDK and bootstrap instrumentation

Source: https://grafana.com/docs/opentelemetry/instrument/python

Run these commands in your project folder to install the OpenTelemetry distribution with OTLP support and automatically instrument your application. Requires a Python 3 development environment and an existing Python application.

```sh
pip install "opentelemetry-distro[otlp]"
opentelemetry-bootstrap -a install
```

--------------------------------

### Install OpenTelemetry Go SDK packages

Source: https://grafana.com/docs/opentelemetry/instrument/go

Run this command in your project folder to install the required OpenTelemetry Go packages. Requires a Go 1.22+ development environment and an existing Go application.

```sh
go get "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" \
  "go.opentelemetry.io/contrib/instrumentation/runtime" \
  "go.opentelemetry.io/otel" \
  "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp" \
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace" \
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" \
  "go.opentelemetry.io/otel/sdk" \
  "go.opentelemetry.io/otel/sdk/metric"
```

--------------------------------

### Example logs output for ListeningOnAddress

Source: https://grafana.com/docs/opentelemetry/instrument/grafana-dotnet

Example logs output from the instrumented application. It shows a LogRecord with severity Information and body 'Now listening on: {address}', indicating the application is listening on http://localhost:5117.

```log
LogRecord.Timestamp:               2024-06-05T02:14:47.9338272Z
LogRecord.CategoryName:            Microsoft.Hosting.Lifetime
LogRecord.Severity:                Info
LogRecord.SeverityText:            Information
LogRecord.Body:                    Now listening on: {address}
LogRecord.Attributes (Key:Value):
    address: http://localhost:5117
    OriginalFormat (a.k.a Body): Now listening on: {address}
LogRecord.EventId:                 14
LogRecord.EventName:               ListeningOnAddress
```

--------------------------------

### Install OpenTelemetry PHP extensions via PECL

Source: https://grafana.com/docs/opentelemetry/instrument/php

Install the opentelemetry and protobuf extensions using PECL.

```sh
pecl install opentelemetry
pecl install protobuf
```

--------------------------------

### Install Grafana.OpenTelemetry and OpenTelemetry.Exporter.Console

Source: https://grafana.com/docs/opentelemetry/instrument/grafana-dotnet

Install the Grafana OpenTelemetry distribution for .NET and the console exporter for local testing. Run these commands in your project folder.

```sh
dotnet add package Grafana.OpenTelemetry
```

```sh
dotnet add package OpenTelemetry.Exporter.Console
```

--------------------------------

### Advanced manual setup with Alloy OTLP pipeline

Source: https://grafana.com/docs/opentelemetry/collector/grafana-alloy

Complete Alloy configuration for advanced manual setup. It receives OTLP data, adds resource detection, drops unneeded resource attributes, copies deployment.environment and service.version to metric datapoints, batches, and exports to Grafana Cloud using basic auth. Requires environment variables GRAFANA_CLOUD_OTLP_ENDPOINT, GRAFANA_CLOUD_INSTANCE_ID, and GRAFANA_CLOUD_API_KEY.

```alloy
otelcol.receiver.otlp "default" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.receiver.otlp/

	// configures the default grpc endpoint "0.0.0.0:4317"
	grpc { }
	// configures the default http/protobuf endpoint "0.0.0.0:4318"
	http { }

	output {
		metrics = [otelcol.processor.resourcedetection.default.input]
		logs    = [otelcol.processor.resourcedetection.default.input]
		traces  = [otelcol.processor.resourcedetection.default.input]
	}
}

otelcol.processor.resourcedetection "default" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.resourcedetection/
	detectors = ["env", "system"] // add "gcp", "ec2", "ecs", "elastic_beanstalk", "eks", "lambda", "azure", "aks", "consul", "heroku"  if you want to use cloud resource detection

	system {
		hostname_sources = ["os"]

		resource_attributes {
			host.id   { enabled = true }
			host.name { enabled = true }
		}
	}

	output {
		metrics = [otelcol.processor.transform.drop_unneeded_resource_attributes.input]
		logs    = [otelcol.processor.transform.drop_unneeded_resource_attributes.input]
		traces  = [otelcol.processor.transform.drop_unneeded_resource_attributes.input]
	}
}

otelcol.processor.transform "drop_unneeded_resource_attributes" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.transform/
	error_mode = "ignore"

	trace_statements {
		context    = "resource"
		statements = [
			"delete_key(attributes, \"k8s.pod.start_time\")",
			"delete_key(attributes, \"os.description\")",
			"delete_key(attributes, \"os.type\")",
			"delete_key(attributes, \"process.command_args\")",
			"delete_key(attributes, \"process.executable.path\")",
			"delete_key(attributes, \"process.pid\")",
			"delete_key(attributes, \"process.runtime.description\")",
			"delete_key(attributes, \"process.runtime.name\")",
			"delete_key(attributes, \"process.runtime.version\")",
		]
	}

	metric_statements {
		context    = "resource"
		statements = [
			"delete_key(attributes, \"k8s.pod.start_time\")",
			"delete_key(attributes, \"os.description\")",
			"delete_key(attributes, \"os.type\")",
			"delete_key(attributes, \"process.command_args\")",
			"delete_key(attributes, \"process.executable.path\")",
			"delete_key(attributes, \"process.pid\")",
			"delete_key(attributes, \"process.runtime.description\")",
			"delete_key(attributes, \"process.runtime.name\")",
			"delete_key(attributes, \"process.runtime.version\")",
		]
	}

	log_statements {
		context    = "resource"
		statements = [
			"delete_key(attributes, \"k8s.pod.start_time\")",
			"delete_key(attributes, \"os.description\")",
			"delete_key(attributes, \"os.type\")",
			"delete_key(attributes, \"process.command_args\")",
			"delete_key(attributes, \"process.executable.path\")",
			"delete_key(attributes, \"process.pid\")",
			"delete_key(attributes, \"process.runtime.description\")",
			"delete_key(attributes, \"process.runtime.name\")",
			"delete_key(attributes, \"process.runtime.version\")",
		]
	}

	output {
		metrics = [otelcol.processor.transform.add_resource_attributes_as_metric_attributes.input]
		logs    = [otelcol.processor.batch.default.input]
		traces  = [
			otelcol.processor.batch.default.input,
		]
	}
}

otelcol.processor.transform "add_resource_attributes_as_metric_attributes" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.transform/
	error_mode = "ignore"

	metric_statements {
		context    = "datapoint"
		statements = [
			"set(attributes[\"deployment.environment\"], resource.attributes[\"deployment.environment\"])",
			"set(attributes[\"service.version\"], resource.attributes[\"service.version\"])",
		]
	}

	output {
		metrics = [otelcol.processor.batch.default.input]
	}
}

otelcol.processor.batch "default" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.batch/
	output {
		metrics = [otelcol.exporter.otlphttp.grafana_cloud.input]
		logs    = [otelcol.exporter.otlphttp.grafana_cloud.input]
		traces  = [otelcol.exporter.otlphttp.grafana_cloud.input]
	}
}

otelcol.exporter.otlphttp "grafana_cloud" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.exporter.otlphttp/
	client {
		endpoint = env("GRAFANA_CLOUD_OTLP_ENDPOINT")
		auth     = otelcol.auth.basic.grafana_cloud.handler
	}
}

otelcol.auth.basic "grafana_cloud" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.auth.basic/
	username = env("GRAFANA_CLOUD_INSTANCE_ID")
	password = env("GRAFANA_CLOUD_API_KEY")
}
```

--------------------------------

### Bootstrap OpenTelemetry SDK in Go (otel.go)

Source: https://grafana.com/docs/opentelemetry/instrument/go

Initializes the OpenTelemetry SDK with OTLP trace and metric exporters, sets up propagators, and starts runtime instrumentation. Must be called before any telemetry is generated; ensure shutdown is called for cleanup.

```go
package main

import (
	"context"
	"errors"
	"go.opentelemetry.io/contrib/instrumentation/runtime"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/metric"
	"go.opentelemetry.io/otel/sdk/trace"
	"log"
	"time"
)

// setupOTelSDK bootstraps the OpenTelemetry pipeline.
// If it does not return an error, make sure to call shutdown for proper cleanup.
func setupOTelSDK(ctx context.Context) (shutdown func(context.Context) error, err error) {
	var shutdownFuncs []func(context.Context) error

	// shutdown calls cleanup functions registered via shutdownFuncs.
	// The errors from the calls are joined.
	// Each registered cleanup will be invoked once.
	shutdown = func(ctx context.Context) error {
		var err error
		for _, fn := range shutdownFuncs {
			err = errors.Join(err, fn(ctx))
		}
		shutdownFuncs = nil
		return err
	}

	// handleErr calls shutdown for cleanup and makes sure that all errors are returned.
	handleErr := func(inErr error) {
		err = errors.Join(inErr, shutdown(ctx))
	}

	prop := propagation.NewCompositeTextMapPropagator(
		propagation.TraceContext{},
		propagation.Baggage{},
	)
	otel.SetTextMapPropagator(prop)

	traceExporter, err := otlptrace.New(ctx, otlptracehttp.NewClient())
	if err != nil {
		return nil, err
	}

	tracerProvider := trace.NewTracerProvider(trace.WithBatcher(traceExporter))
	if err != nil {
		handleErr(err)
		return
	}
	shutdownFuncs = append(shutdownFuncs, tracerProvider.Shutdown)
	otel.SetTracerProvider(tracerProvider)

	metricExporter, err := otlpmetrichttp.New(ctx)
	if err != nil {
		return nil, err
	}

	meterProvider := metric.NewMeterProvider(metric.WithReader(metric.NewPeriodicReader(metricExporter)))
	if err != nil {
		handleErr(err)
		return
	}
	shutdownFuncs = append(shutdownFuncs, meterProvider.Shutdown)
	otel.SetMeterProvider(meterProvider)

	err = runtime.Start(runtime.WithMinimumReadMemStatsInterval(time.Second))
	if err != nil {
		log.Fatal(err)
	}

	return
}
```

--------------------------------

### Example metrics output for process.cpu.count

Source: https://grafana.com/docs/opentelemetry/instrument/grafana-dotnet

Example metrics output from the instrumented application. It shows a process.cpu.count metric with a value of 12, indicating the number of processors available.

```log
Metric Name: process.cpu.count, The number of processors (CPU cores) available to the current process., Unit: {processors}, Meter: OpenTelemetry.Instrumentation.Process/0.5.0.3
(2024-06-05T02:14:47.6851243Z, 2024-06-05T02:14:57.7092810Z] LongSumNonMonotonic
Value: 12
```

--------------------------------

### Install OpenTelemetry SDK packages

Source: https://grafana.com/docs/opentelemetry/instrument/node

Run these npm commands in your project folder to install the OpenTelemetry API and auto-instrumentation packages. Requires a Node.js development environment with NPM or Yarn.

```sh
npm install --save @opentelemetry/api
npm install --save @opentelemetry/auto-instrumentations-node
```

--------------------------------

### Install PSR-3 auto-instrumentation

Source: https://grafana.com/docs/opentelemetry/instrument/php

Install the package for automatic log collection via PSR-3 loggers.

```sh
composer require open-telemetry/opentelemetry-auto-psr3
```

--------------------------------

### Run the Docker image

Source: https://grafana.com/docs/opentelemetry/docker-lgtm

Commands to start the docker-otel-lgtm image on Unix/Linux, Windows PowerShell, and via mise. The image is intended for development, demo, and testing environments.

```sh
# Unix/Linux
./run-lgtm.sh

# Windows (PowerShell)
./run-lgtm

# Using mise (Unix/Linux)
mise run lgtm
```

--------------------------------

### Example JSON log output with OpenTelemetry attributes

Source: https://grafana.com/docs/opentelemetry/collector/opentelemetry-collector/java-json-logs

Example JSON log output from the Spring Boot application after deploying on Kubernetes and running `kubectl logs my_pod_name | jq`. Shows the enriched log with OpenTelemetry trace and resource attributes in the `mdc` field.

```json
{
  "timestamp": 1727346005788,
  "level": "INFO",
  "threadName": "http-nio-8080-exec-5",
  "loggerName": "com.grafana.example.RollController",
  "context": {
    "name": "default",
    "birthdate": 1727345887787,
    "properties": {
    }
  },
  "mdc": {
    "trace_id": "97a39974ba2dfc75e4d31dc2730ee4",
    "trace_flags": "01",
    "service.name": "dice",
    "service.instance.id": "0fb18318-06c0-4893-9a8b-353c00b45227",
    "service.version": "1.1",
    "span_id": "08d6d83d645e3ad2",
    "service.namespace": "shop",
    "deployment.environment": "staging"
  },
  "formattedMessage": "Anonymous player is rolling the dice: 6",
  "throwable": null
}
```

--------------------------------

### Run PHP application with OpenTelemetry environment variables

Source: https://grafana.com/docs/opentelemetry/instrument/php

Use this command to start a PHP application with OpenTelemetry instrumentation. Set the environment variables OTEL_RESOURCE_ATTRIBUTES, OTEL_EXPORTER_OTLP_ENDPOINT, and OTEL_EXPORTER_OTLP_PROTOCOL to configure service name, namespace, environment, and OTLP endpoint. Replace placeholders like <name>, <namespace>, <environment>, and <endpoint> with your specific values. Some backends, such as Grafana Cloud, may require additional authentication headers via OTEL_EXPORTER_OTLP_HEADERS.

```sh
OTEL_RESOURCE_ATTRIBUTES="service.name=<name>,service.namespace=<namespace>,deployment.environment=<environment>" \
OTEL_EXPORTER_OTLP_ENDPOINT=<endpoint> \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
php -S localhost:8080
```

--------------------------------

### Generate traffic

Source: https://grafana.com/docs/opentelemetry/docker-lgtm

Commands to generate traffic for the example app on Unix/Linux, Windows PowerShell, and via mise. Use OTel Checker to verify instrumentation.

```sh
# Unix/Linux
./generate-traffic.sh

# Windows (PowerShell)
./generate-traffic

# Using mise (Unix/Linux)
mise run generate-traffic
```

--------------------------------

### Run JVM app with grafana-opentelemetry-java.jar

Source: https://grafana.com/docs/opentelemetry/instrument/grafana-java

Start a JVM application with the Grafana OpenTelemetry Java agent. Set the service name, namespace, and deployment environment via OTEL_RESOURCE_ATTRIBUTES, and configure the OTLP endpoint and protocol. Replace variables in angle brackets with your values; some backends like Grafana Cloud also require OTEL_EXPORTER_OTLP_HEADERS.

```sh
OTEL_RESOURCE_ATTRIBUTES="service.name=<name>,service.namespace=<namespace>,deployment.environment=<environment>" \
OTEL_EXPORTER_OTLP_ENDPOINT=<endpoint> \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
java -javaagent:<path>/grafana-opentelemetry-java.jar -jar <my-app>.jar
```

--------------------------------

### Log message example

Source: https://grafana.com/docs/opentelemetry/instrument/php

Example of a log message that will be automatically sent to Grafana Cloud via OTLP.

```php
/** @var \Psr\Log\LoggerInterface $logger */
$logger->info('This message will be sent to Grafana Cloud via OTLP');
```

--------------------------------

### Build the Docker image from scratch

Source: https://grafana.com/docs/opentelemetry/docker-lgtm

Commands to build the docker-otel-lgtm image from the docker/ directory, with a mise alternative.

```sh
cd docker/
docker build . -t grafana/otel-lgtm

# Using mise
mise build-lgtm
```

--------------------------------

### Instrument HTTP server with otelhttp in Go (main.go)

Source: https://grafana.com/docs/opentelemetry/instrument/go

Sets up the OpenTelemetry SDK and instruments an HTTP server using the otelhttp library. Handles graceful shutdown on SIGINT and adds HTTP instrumentation with route tagging.

```go
package main

import (
	"context"
	"errors"
	"log"
	"net"
	"net/http"
	"os"
	"os/signal"
	"time"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func main() {
	if err := run(); err != nil {
		log.Fatalln(err)
	}
}

func run() (err error) {
	// Handle SIGINT (CTRL+C) gracefully.
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
	defer stop()

	// Set up OpenTelemetry.
	otelShutdown, err := setupOTelSDK(ctx)
	if err != nil {
		return
	}
	// Handle shutdown properly so nothing leaks.
	defer func() {
		err = errors.Join(err, otelShutdown(context.Background()))
	}()

	// Start HTTP server.
	srv := &http.Server{
		Addr:         ":8080",
		BaseContext:  func(_ net.Listener) context.Context { return ctx },
		ReadTimeout:  time.Second,
		WriteTimeout: 10 * time.Second,
		Handler:      newHTTPHandler(),
	}
	srvErr := make(chan error, 1)
	go func() {
		srvErr <- srv.ListenAndServe()
	}()

	// Wait for interruption.
	select {
	case err = <-srvErr:
		// Error when starting HTTP server.
		return
	case <-ctx.Done():
		// Wait for first CTRL+C.
		// Stop receiving signal notifications as soon as possible.
		stop()
	}

	// When Shutdown is called, ListenAndServe immediately returns ErrServerClosed.
	err = srv.Shutdown(context.Background())
	return
}

func newHTTPHandler() http.Handler {
	mux := http.NewServeMux()

	// handleFunc is a replacement for mux.HandleFunc
	// which enriches the handler's HTTP instrumentation with the pattern as the http.route.
	handleFunc := func(pattern string, handlerFunc func(http.ResponseWriter, *http.Request)) {
		// Configure the "http.route" for the HTTP instrumentation.
		handler := otelhttp.WithRouteTag(pattern, http.HandlerFunc(handlerFunc))
		mux.Handle(pattern, handler)
	}

	// Register handlers.
	handleFunc("/rolldice", rolldice)

	// Add HTTP instrumentation for the whole server.
	handler := otelhttp.NewHandler(mux, "/")
	return handler
}
```

--------------------------------

### Unstructured log example

Source: https://grafana.com/docs/opentelemetry/collector/opentelemetry-collector/kubernetes-logs

Example of an unstructured log line that includes resource attributes and span_id appended at the end. This format preserves human readability but requires regular expression parsing, which can be fragile for multi-line messages.

```none
2024-09-17T11:29:54  INFO [nio-8080-exec-1] c.e.OrderController  : Order completed - service.name=order-processor, service.instance.id=i-123456, span_id=1d5f8ca3f9366fac...
```

--------------------------------

### Structured JSON log example

Source: https://grafana.com/docs/opentelemetry/collector/opentelemetry-collector/kubernetes-logs

Example of a structured JSON log line that includes resource attributes and span_id. This format is easier to parse reliably with native JSON parsers, but the verbosity can reduce human readability.

```json
{"timestamp": "2024-09-17T11:29:54", "level": "INFO", "body":"Order completed", "logger": "c.e.OrderController", "service_name": "order-processor", "service_instance_id": "i-123456", "span_id":"1d5f8ca3f9366fac"...}
```

--------------------------------

### Run Go application with OpenTelemetry environment variables

Source: https://grafana.com/docs/opentelemetry/instrument/go

Run a Go application with OpenTelemetry instrumentation using environment variables. Set OTEL_EXPORTER_OTLP_INSECURE to "true" to use HTTP instead of HTTPS, and configure service attributes and endpoint. Replace <name>, <namespace>, <environment>, and <endpoint> with your values.

```sh
sh ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```sh
# to use http instead of https
# needed because of https://github.com/open-telemetry/opentelemetry-go/issues/4834
OTEL_EXPORTER_OTLP_INSECURE="true" \
OTEL_RESOURCE_ATTRIBUTES="service.name=<name>,service.namespace=<namespace>,deployment.environment=<environment>" \
OTEL_EXPORTER_OTLP_ENDPOINT=<endpoint> \
go run .
```
```

--------------------------------

### Console metric output from LoggingMetricExporter

Source: https://grafana.com/docs/opentelemetry/instrument/grafana-java

Example console output from the LoggingMetricExporter when exporting metrics to the console. To find metrics, look for lines that include `LoggingMetricExporter`.

```log
INFO io.opentelemetry.exporter.logging.LoggingMetricExporter - metric: ImmutableMetricData{resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.21.0, attributes={container.id="048b9982e0b98cdc5579334bb1decc157ed1ebc23f391ebe306898898ec32fa4", host.arch="amd64", host.name="048b9982e0b9", os.description="Linux 6.2.0-39-generic", os.type="linux", process.command_line="/usr/lib/jvm/jdk-8u312-bellsoft-x86_64/jre/bin/java -javaagent:/opentelemetry-javaagent.jar -Dgrafana.otel.use-tested-instrumentations=true io.opentelemetry.smoketest.springboot.SpringbootApplication", process.executable.path="/usr/lib/jvm/jdk-8u312-bellsoft-x86_64/jre/bin/java", process.pid=1, process.runtime.description="BellSoft OpenJDK 64-Bit Server VM 25.312-b07", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="1.8.0_312-b07", service.instance.id="8231ca95-e9aa-474a-bd98-88349a9942ad", service.name="unknown_service:java", telemetry.auto.version="1.32.0", telemetry.distro.name="grafana-opentelemetry-java", telemetry.distro.version="0.32.0-beta.1", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.32.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.opentelemetry.runtime-telemetry-java8, version=1.32.0-alpha, schemaUrl=null, attributes={}}, name=jvm.cpu.count, description=Number of processors available to the Java virtual machine., unit={cpu}, type=LONG_SUM, data=ImmutableSumData{points=[ImmutableLongPointData{startEpochNanos=1704964347622000000, epochNanos=1704964351627000000, attributes={}, value=12, exemplars=[]}], monotonic=false, aggregationTemporality=CUMULATIVE}}
```
Tokens:
Raw