### Run the example REST service

Source: https://grafana.com/docs/opentelemetry/docker-lgtm

Commands to run the example REST service on Unix/Linux, Windows PowerShell, and via mise. The example app implements a rolldice service.

```sh
# Unix/Linux
./run-example.sh

# Windows (PowerShell)
./run-example

# Using mise (Unix/Linux)
mise run example
```

--------------------------------

### Install OpenTelemetry PHP SDK via Composer

Source: https://grafana.com/docs/opentelemetry/instrument/php

Run this command in your project folder to install the OpenTelemetry SDK, OTLP exporter, and Guzzle 7 adapter.

```sh
composer require \
  open-telemetry/sdk \
  open-telemetry/exporter-otlp \
  php-http/guzzle7-adapter
```

--------------------------------

### Install OpenTelemetry SDK and bootstrap instrumentation

Source: https://grafana.com/docs/opentelemetry/instrument/python

Run these commands in your project folder to install the OpenTelemetry distribution with OTLP support and automatically instrument your application. Requires a Python 3 development environment and an existing Python application.

```sh
pip install "opentelemetry-distro[otlp]"
opentelemetry-bootstrap -a install
```

--------------------------------

### Install OpenTelemetry Go SDK packages

Source: https://grafana.com/docs/opentelemetry/instrument/go

Run this command in your project folder to install the required OpenTelemetry Go packages. Requires a Go 1.22+ development environment and an existing Go application.

```sh
go get "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" \
  "go.opentelemetry.io/contrib/instrumentation/runtime" \
  "go.opentelemetry.io/otel" \
  "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp" \
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace" \
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" \
  "go.opentelemetry.io/otel/sdk" \
  "go.opentelemetry.io/otel/sdk/metric"
```

--------------------------------

### Example logs output for ListeningOnAddress

Source: https://grafana.com/docs/opentelemetry/instrument/grafana-dotnet

Example logs output from the instrumented application. It shows a LogRecord with severity Information and body 'Now listening on: {address}', indicating the application is listening on http://localhost:5117.

```log
LogRecord.Timestamp:               2024-06-05T02:14:47.9338272Z
LogRecord.CategoryName:            Microsoft.Hosting.Lifetime
LogRecord.Severity:                Info
LogRecord.SeverityText:            Information
LogRecord.Body:                    Now listening on: {address}
LogRecord.Attributes (Key:Value):
    address: http://localhost:5117
    OriginalFormat (a.k.a Body): Now listening on: {address}
LogRecord.EventId:                 14
LogRecord.EventName:               ListeningOnAddress
```

--------------------------------

### Install OpenTelemetry PHP extensions via PECL

Source: https://grafana.com/docs/opentelemetry/instrument/php

Install the opentelemetry and protobuf extensions using PECL.

```sh
pecl install opentelemetry
pecl install protobuf
```

--------------------------------

### Install Grafana.OpenTelemetry and OpenTelemetry.Exporter.Console

Source: https://grafana.com/docs/opentelemetry/instrument/grafana-dotnet

Install the Grafana OpenTelemetry distribution for .NET and the console exporter for local testing. Run these commands in your project folder.

```sh
dotnet add package Grafana.OpenTelemetry
```

```sh
dotnet add package OpenTelemetry.Exporter.Console
```

--------------------------------

### Advanced manual setup with Alloy OTLP pipeline

Source: https://grafana.com/docs/opentelemetry/collector/grafana-alloy

Complete Alloy configuration for advanced manual setup. It receives OTLP data, adds resource detection, drops unneeded resource attributes, copies deployment.environment and service.version to metric datapoints, batches, and exports to Grafana Cloud using basic auth. Requires environment variables GRAFANA_CLOUD_OTLP_ENDPOINT, GRAFANA_CLOUD_INSTANCE_ID, and GRAFANA_CLOUD_API_KEY.

```alloy
otelcol.receiver.otlp "default" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.receiver.otlp/

	// configures the default grpc endpoint "0.0.0.0:4317"
	grpc { }
	// configures the default http/protobuf endpoint "0.0.0.0:4318"
	http { }

	output {
		metrics = [otelcol.processor.resourcedetection.default.input]
		logs    = [otelcol.processor.resourcedetection.default.input]
		traces  = [otelcol.processor.resourcedetection.default.input]
	}
}

otelcol.processor.resourcedetection "default" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.resourcedetection/
	detectors = ["env", "system"] // add "gcp", "ec2", "ecs", "elastic_beanstalk", "eks", "lambda", "azure", "aks", "consul", "heroku"  if you want to use cloud resource detection

	system {
		hostname_sources = ["os"]

		resource_attributes {
			host.id   { enabled = true }
			host.name { enabled = true }
		}
	}

	output {
		metrics = [otelcol.processor.transform.drop_unneeded_resource_attributes.input]
		logs    = [otelcol.processor.transform.drop_unneeded_resource_attributes.input]
		traces  = [otelcol.processor.transform.drop_unneeded_resource_attributes.input]
	}
}

otelcol.processor.transform "drop_unneeded_resource_attributes" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.transform/
	error_mode = "ignore"

	trace_statements {
		context    = "resource"
		statements = [
			"delete_key(attributes, \"k8s.pod.start_time\")",
			"delete_key(attributes, \"os.description\")",
			"delete_key(attributes, \"os.type\")",
			"delete_key(attributes, \"process.command_args\")",
			"delete_key(attributes, \"process.executable.path\")",
			"delete_key(attributes, \"process.pid\")",
			"delete_key(attributes, \"process.runtime.description\")",
			"delete_key(attributes, \"process.runtime.name\")",
			"delete_key(attributes, \"process.runtime.version\")",
		]
	}

	metric_statements {
		context    = "resource"
		statements = [
			"delete_key(attributes, \"k8s.pod.start_time\")",
			"delete_key(attributes, \"os.description\")",
			"delete_key(attributes, \"os.type\")",
			"delete_key(attributes, \"process.command_args\")",
			"delete_key(attributes, \"process.executable.path\")",
			"delete_key(attributes, \"process.pid\")",
			"delete_key(attributes, \"process.runtime.description\")",
			"delete_key(attributes, \"process.runtime.name\")",
			"delete_key(attributes, \"process.runtime.version\")",
		]
	}

	log_statements {
		context    = "resource"
		statements = [
			"delete_key(attributes, \"k8s.pod.start_time\")",
			"delete_key(attributes, \"os.description\")",
			"delete_key(attributes, \"os.type\")",
			"delete_key(attributes, \"process.command_args\")",
			"delete_key(attributes, \"process.executable.path\")",
			"delete_key(attributes, \"process.pid\")",
			"delete_key(attributes, \"process.runtime.description\")",
			"delete_key(attributes, \"process.runtime.name\")",
			"delete_key(attributes, \"process.runtime.version\")",
		]
	}

	output {
		metrics = [otelcol.processor.transform.add_resource_attributes_as_metric_attributes.input]
		logs    = [otelcol.processor.batch.default.input]
		traces  = [
			otelcol.processor.batch.default.input,
		]
	}
}

otelcol.processor.transform "add_resource_attributes_as_metric_attributes" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.transform/
	error_mode = "ignore"

	metric_statements {
		context    = "datapoint"
		statements = [
			"set(attributes[\"deployment.environment\"], resource.attributes[\"deployment.environment\"])",
			"set(attributes[\"service.version\"], resource.attributes[\"service.version\"])",
		]
	}

	output {
		metrics = [otelcol.processor.batch.default.input]
	}
}

otelcol.processor.batch "default" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.batch/
	output {
		metrics = [otelcol.exporter.otlphttp.grafana_cloud.input]
		logs    = [otelcol.exporter.otlphttp.grafana_cloud.input]
		traces  = [otelcol.exporter.otlphttp.grafana_cloud.input]
	}
}

otelcol.exporter.otlphttp "grafana_cloud" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.exporter.otlphttp/
	client {
		endpoint = env("GRAFANA_CLOUD_OTLP_ENDPOINT")
		auth     = otelcol.auth.basic.grafana_cloud.handler
	}
}

otelcol.auth.basic "grafana_cloud" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.auth.basic/
	username = env("GRAFANA_CLOUD_INSTANCE_ID")
	password = env("GRAFANA_CLOUD_API_KEY")
}
```

--------------------------------

### Bootstrap OpenTelemetry SDK in Go (otel.go)

Source: https://grafana.com/docs/opentelemetry/instrument/go

Initializes the OpenTelemetry SDK with OTLP trace and metric exporters, sets up propagators, and starts runtime instrumentation. Must be called before any telemetry is generated; ensure shutdown is called for cleanup.

```go
package main

import (
	"context"
	"errors"
	"go.opentelemetry.io/contrib/instrumentation/runtime"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/metric"
	"go.opentelemetry.io/otel/sdk/trace"
	"log"
	"time"
)

// setupOTelSDK bootstraps the OpenTelemetry pipeline.
// If it does not return an error, make sure to call shutdown for proper cleanup.
func setupOTelSDK(ctx context.Context) (shutdown func(context.Context) error, err error) {
	var shutdownFuncs []func(context.Context) error

	// shutdown calls cleanup functions registered via shutdownFuncs.
	// The errors from the calls are joined.
	// Each registered cleanup will be invoked once.
	shutdown = func(ctx context.Context) error {
		var err error
		for _, fn := range shutdownFuncs {
			err = errors.Join(err, fn(ctx))
		}
		shutdownFuncs = nil
		return err
	}

	// handleErr calls shutdown for cleanup and makes sure that all errors are returned.
	handleErr := func(inErr error) {
		err = errors.Join(inErr, shutdown(ctx))
	}

	prop := propagation.NewCompositeTextMapPropagator(
		propagation.TraceContext{},
		propagation.Baggage{},
	)
	otel.SetTextMapPropagator(prop)

	traceExporter, err := otlptrace.New(ctx, otlptracehttp.NewClient())
	if err != nil {
		return nil, err
	}

	tracerProvider := trace.NewTracerProvider(trace.WithBatcher(traceExporter))
	if err != nil {
		handleErr(err)
		return
	}
	shutdownFuncs = append(shutdownFuncs, tracerProvider.Shutdown)
	otel.SetTracerProvider(tracerProvider)

	metricExporter, err := otlpmetrichttp.New(ctx)
	if err != nil {
		return nil, err
	}

	meterProvider := metric.NewMeterProvider(metric.WithReader(metric.NewPeriodicReader(metricExporter)))
	if err != nil {
		handleErr(err)
		return
	}
	shutdownFuncs = append(shutdownFuncs, meterProvider.Shutdown)
	otel.SetMeterProvider(meterProvider)

	err = runtime.Start(runtime.WithMinimumReadMemStatsInterval(time.Second))
	if err != nil {
		log.Fatal(err)
	}

	return
}
```

--------------------------------

### Example metrics output for process.cpu.count

Source: https://grafana.com/docs/opentelemetry/instrument/grafana-dotnet

Example metrics output from the instrumented application. It shows a process.cpu.count metric with a value of 12, indicating the number of processors available.

```log
Metric Name: process.cpu.count, The number of processors (CPU cores) available to the current process., Unit: {processors}, Meter: OpenTelemetry.Instrumentation.Process/0.5.0.3
(2024-06-05T02:14:47.6851243Z, 2024-06-05T02:14:57.7092810Z] LongSumNonMonotonic
Value: 12
```

--------------------------------

### Install OpenTelemetry SDK packages

Source: https://grafana.com/docs/opentelemetry/instrument/node

Run these npm commands in your project folder to install the OpenTelemetry API and auto-instrumentation packages. Requires a Node.js development environment with NPM or Yarn.

```sh
npm install --save @opentelemetry/api
npm install --save @opentelemetry/auto-instrumentations-node
```

--------------------------------

### Install PSR-3 auto-instrumentation

Source: https://grafana.com/docs/opentelemetry/instrument/php

Install the package for automatic log collection via PSR-3 loggers.

```sh
composer require open-telemetry/opentelemetry-auto-psr3
```

--------------------------------

### Log message example

Source: https://grafana.com/docs/opentelemetry/instrument/php

Example of a log message that will be automatically sent to Grafana Cloud via OTLP.

```php
/** @var \Psr\Log\LoggerInterface $logger */
$logger->info('This message will be sent to Grafana Cloud via OTLP');
```

--------------------------------

### Run the Docker image

Source: https://grafana.com/docs/opentelemetry/docker-lgtm

Commands to start the docker-otel-lgtm image on Unix/Linux, Windows PowerShell, and via mise. The image is intended for development, demo, and testing environments.

```sh
# Unix/Linux
./run-lgtm.sh

# Windows (PowerShell)
./run-lgtm

# Using mise (Unix/Linux)
mise run lgtm
```

--------------------------------

### Example JSON log output with OpenTelemetry attributes

Source: https://grafana.com/docs/opentelemetry/collector/opentelemetry-collector/java-json-logs

Example JSON log output from the Spring Boot application after deploying on Kubernetes and running `kubectl logs my_pod_name | jq`. Shows the enriched log with OpenTelemetry trace and resource attributes in the `mdc` field.

```json
{
  "timestamp": 1727346005788,
  "level": "INFO",
  "threadName": "http-nio-8080-exec-5",
  "loggerName": "com.grafana.example.RollController",
  "context": {
    "name": "default",
    "birthdate": 1727345887787,
    "properties": {
    }
  },
  "mdc": {
    "trace_id": "97a39974ba2dfc75e4d31dc2730ee4",
    "trace_flags": "01",
    "service.name": "dice",
    "service.instance.id": "0fb18318-06c0-4893-9a8b-353c00b45227",
    "service.version": "1.1",
    "span_id": "08d6d83d645e3ad2",
    "service.namespace": "shop",
    "deployment.environment": "staging"
  },
  "formattedMessage": "Anonymous player is rolling the dice: 6",
  "throwable": null
}
```

--------------------------------

### Run PHP application with OpenTelemetry environment variables

Source: https://grafana.com/docs/opentelemetry/instrument/php

Use this command to start a PHP application with OpenTelemetry instrumentation. Set the environment variables OTEL_RESOURCE_ATTRIBUTES, OTEL_EXPORTER_OTLP_ENDPOINT, and OTEL_EXPORTER_OTLP_PROTOCOL to configure service name, namespace, environment, and OTLP endpoint. Replace placeholders like <name>, <namespace>, <environment>, and <endpoint> with your specific values. Some backends, such as Grafana Cloud, may require additional authentication headers via OTEL_EXPORTER_OTLP_HEADERS.

```sh
OTEL_RESOURCE_ATTRIBUTES="service.name=<name>,service.namespace=<namespace>,deployment.environment=<environment>" \
OTEL_EXPORTER_OTLP_ENDPOINT=<endpoint> \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
php -S localhost:8080
```

--------------------------------

### Generate traffic

Source: https://grafana.com/docs/opentelemetry/docker-lgtm

Commands to generate traffic for the example app on Unix/Linux, Windows PowerShell, and via mise. Use OTel Checker to verify instrumentation.

```sh
# Unix/Linux
./generate-traffic.sh

# Windows (PowerShell)
./generate-traffic

# Using mise (Unix/Linux)
mise run generate-traffic
```

--------------------------------

### Run JVM app with grafana-opentelemetry-java.jar

Source: https://grafana.com/docs/opentelemetry/instrument/grafana-java

Start a JVM application with the Grafana OpenTelemetry Java agent. Set the service name, namespace, and deployment environment via OTEL_RESOURCE_ATTRIBUTES, and configure the OTLP endpoint and protocol. Replace variables in angle brackets with your values; some backends like Grafana Cloud also require OTEL_EXPORTER_OTLP_HEADERS.

```sh
OTEL_RESOURCE_ATTRIBUTES="service.name=<name>,service.namespace=<namespace>,deployment.environment=<environment>" \
OTEL_EXPORTER_OTLP_ENDPOINT=<endpoint> \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
java -javaagent:<path>/grafana-opentelemetry-java.jar -jar <my-app>.jar
```

--------------------------------

### Unstructured log example

Source: https://grafana.com/docs/opentelemetry/collector/opentelemetry-collector/kubernetes-logs

Example of an unstructured log line that includes resource attributes and span_id appended at the end. This format preserves human readability but requires regular expression parsing, which can be fragile for multi-line messages.

```none
2024-09-17T11:29:54  INFO [nio-8080-exec-1] c.e.OrderController  : Order completed - service.name=order-processor, service.instance.id=i-123456, span_id=1d5f8ca3f9366fac...
```

--------------------------------

### OpenTelemetry SDK: Set multi-zone metrics endpoint via environment variables

Source: https://grafana.com/docs/opentelemetry/ingest/multi-zone

Set OTEL_EXPORTER_OTLP_METRICS_ENDPOINT to the multi-zone endpoint for metrics, while keeping the regular endpoint for other signals. The example uses environment variables and includes authentication headers.

```diff
OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-gateway-prod-eu-north-0.grafana.net/otlp"
+OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="https://otlp-gateway-prod-eu-north-0-multi-zone.grafana.net/otlp/v1/metrics"
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(echo -n $GRAFANA_INSTANCE_ID:$TOKEN | base64 -w 0)"
```

--------------------------------

### Console metric output from LoggingMetricExporter

Source: https://grafana.com/docs/opentelemetry/instrument/grafana-java

Example console output from the LoggingMetricExporter when exporting metrics to the console. To find metrics, look for lines that include `LoggingMetricExporter`.

```log
INFO io.opentelemetry.exporter.logging.LoggingMetricExporter - metric: ImmutableMetricData{resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.21.0, attributes={container.id="048b9982e0b98cdc5579334bb1decc157ed1ebc23f391ebe306898898ec32fa4", host.arch="amd64", host.name="048b9982e0b9", os.description="Linux 6.2.0-39-generic", os.type="linux", process.command_line="/usr/lib/jvm/jdk-8u312-bellsoft-x86_64/jre/bin/java -javaagent:/opentelemetry-javaagent.jar -Dgrafana.otel.use-tested-instrumentations=true io.opentelemetry.smoketest.springboot.SpringbootApplication", process.executable.path="/usr/lib/jvm/jdk-8u312-bellsoft-x86_64/jre/bin/java", process.pid=1, process.runtime.description="BellSoft OpenJDK 64-Bit Server VM 25.312-b07", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="1.8.0_312-b07", service.instance.id="8231ca95-e9aa-474a-bd98-88349a9942ad", service.name="unknown_service:java", telemetry.auto.version="1.32.0", telemetry.distro.name="grafana-opentelemetry-java", telemetry.distro.version="0.32.0-beta.1", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.32.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.opentelemetry.runtime-telemetry-java8, version=1.32.0-alpha, schemaUrl=null, attributes={}}, name=jvm.cpu.count, description=Number of processors available to the Java virtual machine., unit={cpu}, type=LONG_SUM, data=ImmutableSumData{points=[ImmutableLongPointData{startEpochNanos=1704964347622000000, epochNanos=1704964351627000000, attributes={}, value=12, exemplars=[]}], monotonic=false, aggregationTemporality=CUMULATIVE}}
```

--------------------------------

### Structured JSON log example

Source: https://grafana.com/docs/opentelemetry/collector/opentelemetry-collector/kubernetes-logs

Example of a structured JSON log line that includes resource attributes and span_id. This format is easier to parse reliably with native JSON parsers, but the verbosity can reduce human readability.

```json
{"timestamp": "2024-09-17T11:29:54", "level": "INFO", "body":"Order completed", "logger": "c.e.OrderController", "service_name": "order-processor", "service_instance_id": "i-123456", "span_id":"1d5f8ca3f9366fac"...}
```

--------------------------------

### Console trace output from LoggingSpanExporter

Source: https://grafana.com/docs/opentelemetry/instrument/grafana-java

Example console output from the LoggingSpanExporter when exporting traces to the console. Look for lines that include `LoggingSpanExporter` to find trace spans.

```log
INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'WebController.withSpan' : 2337335133908c9ce6e0dfc7bda74d7c 8bfef4eaac83e8cb INTERNAL [tracer: io.opentelemetry.opentelemetry-extension-annotations-1.0:1.32.0-alpha] AttributesMap{data={thread.id=32, code.namespace=io.opentelemetry.smoketest.springboot.controller.WebController, code.function=withSpan, thread.name=http-nio-8080-exec-1}, capacity=128, totalAddedValues=4}
```

--------------------------------

### Build the Docker image from scratch

Source: https://grafana.com/docs/opentelemetry/docker-lgtm

Commands to build the docker-otel-lgtm image from the docker/ directory, with a mise alternative.

```sh
cd docker/
docker build . -t grafana/otel-lgtm

# Using mise
mise build-lgtm
```

--------------------------------

### Instrument HTTP server with otelhttp in Go (main.go)

Source: https://grafana.com/docs/opentelemetry/instrument/go

Sets up the OpenTelemetry SDK and instruments an HTTP server using the otelhttp library. Handles graceful shutdown on SIGINT and adds HTTP instrumentation with route tagging.

```go
package main

import (
	"context"
	"errors"
	"log"
	"net"
	"net/http"
	"os"
	"os/signal"
	"time"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func main() {
	if err := run(); err != nil {
		log.Fatalln(err)
	}
}

func run() (err error) {
	// Handle SIGINT (CTRL+C) gracefully.
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
	defer stop()

	// Set up OpenTelemetry.
	otelShutdown, err := setupOTelSDK(ctx)
	if err != nil {
		return
	}
	// Handle shutdown properly so nothing leaks.
	defer func() {
		err = errors.Join(err, otelShutdown(context.Background()))
	}()

	// Start HTTP server.
	srv := &http.Server{
		Addr:         ":8080",
		BaseContext:  func(_ net.Listener) context.Context { return ctx },
		ReadTimeout:  time.Second,
		WriteTimeout: 10 * time.Second,
		Handler:      newHTTPHandler(),
	}
	srvErr := make(chan error, 1)
	go func() {
		srvErr <- srv.ListenAndServe()
	}()

	// Wait for interruption.
	select {
	case err = <-srvErr:
		// Error when starting HTTP server.
		return
	case <-ctx.Done():
		// Wait for first CTRL+C.
		// Stop receiving signal notifications as soon as possible.
		stop()
	}

	// When Shutdown is called, ListenAndServe immediately returns ErrServerClosed.
	err = srv.Shutdown(context.Background())
	return
}

func newHTTPHandler() http.Handler {
	mux := http.NewServeMux()

	// handleFunc is a replacement for mux.HandleFunc
	// which enriches the handler's HTTP instrumentation with the pattern as the http.route.
	handleFunc := func(pattern string, handlerFunc func(http.ResponseWriter, *http.Request)) {
		// Configure the "http.route" for the HTTP instrumentation.
		handler := otelhttp.WithRouteTag(pattern, http.HandlerFunc(handlerFunc))
		mux.Handle(pattern, handler)
	}

	// Register handlers.
	handleFunc("/rolldice", rolldice)

	// Add HTTP instrumentation for the whole server.
	handler := otelhttp.NewHandler(mux, "/")
	return handler
}
```

--------------------------------

### Run Go application with OpenTelemetry environment variables

Source: https://grafana.com/docs/opentelemetry/instrument/go

Run a Go application with OpenTelemetry instrumentation using environment variables. Set OTEL_EXPORTER_OTLP_INSECURE to "true" to use HTTP instead of HTTPS, and configure service attributes and endpoint. Replace <name>, <namespace>, <environment>, and <endpoint> with your values.

```sh
sh ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```sh
# to use http instead of https
# needed because of https://github.com/open-telemetry/opentelemetry-go/issues/4834
OTEL_EXPORTER_OTLP_INSECURE="true" \
OTEL_RESOURCE_ATTRIBUTES="service.name=<name>,service.namespace=<namespace>,deployment.environment=<environment>" \
OTEL_EXPORTER_OTLP_ENDPOINT=<endpoint> \
go run .
```
```

--------------------------------

### Run Python app with OpenTelemetry Distro

Source: https://grafana.com/docs/opentelemetry/instrument/python

Run your Python application with the OpenTelemetry Distro. Set the service name, namespace, and deployment environment via OTEL_RESOURCE_ATTRIBUTES, the OTLP endpoint via OTEL_EXPORTER_OTLP_ENDPOINT, and the protocol to http/protobuf. Replace variables in angle brackets with your values; some backends like Grafana Cloud also require OTEL_EXPORTER_OTLP_HEADERS.

```sh
OTEL_RESOURCE_ATTRIBUTES="service.name=<name>,service.namespace=<namespace>,deployment.environment=<environment>" \
OTEL_EXPORTER_OTLP_ENDPOINT=<endpoint> \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
opentelemetry-instrument <your_run_command>
```

--------------------------------

### .NET Framework: UseGrafana() in WebApiApplication

Source: https://grafana.com/docs/opentelemetry/instrument/grafana-dotnet

For .NET Framework applications. Configures tracing and metrics in Application_Start and disposes them in Application_End. Does not include console exporters, so it is suitable for production.

```netfx
// .NET Framework
using Grafana.OpenTelemetry;
using OpenTelemetry;
using OpenTelemetry.Metrics;
using OpenTelemetry.Trace;

namespace DemoApplication
{
    public class WebApiApplication : System.Web.HttpApplication
    {
        private TracerProvider tracerProvider;
        private MeterProvider meterProvider;

        protected void Application_Start()
        {
            // MVC, Web API, etc. configuration

            tracerProvider = Sdk.CreateTracerProviderBuilder()
                .UseGrafana()
                .Build();
            meterProvider = Sdk.CreateMeterProviderBuilder()
                .UseGrafana()
                .Build();
        }

        protected void Application_End()
        {
            tracerProvider?.Dispose();
            meterProvider?.Dispose();
        }
    }
}
```

--------------------------------

### Run .NET app with OpenTelemetry environment variables

Source: https://grafana.com/docs/opentelemetry/instrument/grafana-dotnet

Run the .NET application with OpenTelemetry environment variables. Replace <name>, <namespace>, <environment>, and <endpoint> with your values. Some backends, such as Grafana Cloud, also require setting authentication headers in the OTEL_EXPORTER_OTLP_HEADERS environment variable.

```sh
OTEL_RESOURCE_ATTRIBUTES="service.name=<name>,service.namespace=<namespace>,deployment.environment=<environment>" \
OTEL_EXPORTER_OTLP_ENDPOINT=<endpoint> \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
dotnet run
```

--------------------------------

### Set OTEL_INSTRUMENTATION_COMMON_MDC_RESOURCE_ATTRIBUTES for Logback

Source: https://grafana.com/docs/opentelemetry/collector/opentelemetry-collector/java-json-logs

Sets the OpenTelemetry Java Agent to include the specified resource attributes in the MDC (Mapped Diagnostic Context) for Logback logs. Use this environment variable before starting the Spring Boot application to enrich logs with service and deployment context.

```shell
export OTEL_INSTRUMENTATION_COMMON_MDC_RESOURCE_ATTRIBUTES=service.namespace,service.name,service.instance.id,service.version,deployment.environment
```

--------------------------------

### Verify Beyla instrumentation with curl and expected metrics output

Source: https://grafana.com/docs/opentelemetry/instrument/beyla

Run this command to verify Beyla instrumentation by fetching metrics from the exposed Prometheus endpoint. The output shows the expected metrics, including beyla_build_info and HTTP request body size histograms. If using the Grafana Helm chart, first port-forward the Beyla pod with `kubectl -n beyla port-forward <BEYLA_POD_NAME>`.

```sh
curl http://localhost:9090/metrics
```

```log
# HELP beyla_build_info A metric with a constant '1' value labeled by version, revision, branch, goversion from which Beyla was built, the goos and goarch for the build, and thelanguage of the reported services
# TYPE beyla_build_info gauge
beyla_build_info{goarch="arm64",goos="linux",goversion="go1.22.4",revision="b672ca01",target_lang="generic",version="b672ca01"} 1
beyla_build_info{goarch="arm64",goos="linux",goversion="go1.22.4",revision="b672ca01",target_lang="go",version="b672ca01"} 1
# HELP http_client_request_body_size_bytes size, in bytes, of the HTTP request body as sent from the client side
# TYPE http_client_request_body_size_bytes histogram
http_client_request_body_size_bytes_bucket{http_request_method="GET",http_response_status_code="200",http_route="/**",service_name="coredns",service_namespace="",target_instance="",le="0"} 2
http_client_request_body_size_bytes_bucket{http_request_method="GET",http_response_status_code="200",http_route="/**",service_name="coredns",service_namespace="",target_instance="",le="32"} 2
```

--------------------------------

### Run Node.js application with auto-instrumentation and OTLP exporters

Source: https://grafana.com/docs/opentelemetry/instrument/node

Run your application with environment variables to enable auto-instrumentation and export OTLP data. Replace variables in <...> with your service name, namespace, environment, endpoint, and application file. Some backends like Grafana Cloud require setting OTEL_EXPORTER_OTLP_HEADERS for authentication.

```sh
OTEL_TRACES_EXPORTER="otlp" \
OTEL_METRICS_EXPORTER="otlp" \
OTEL_LOGS_EXPORTER="otlp" \
OTEL_NODE_RESOURCE_DETECTORS="env,host,os" \
OTEL_RESOURCE_ATTRIBUTES="service.name=<name>,service.namespace=<namespace>,deployment.environment=<environment>" \
OTEL_EXPORTER_OTLP_ENDPOINT=<endpoint> \
NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register" \
node <my-app>.js
```

--------------------------------

### Scale Alloy tail sampling with span metrics and service graphs

Source: https://grafana.com/docs/opentelemetry/collector/sampling/scale

Complete Alloy configuration for scaling tail sampling with span metrics and service graphs. It receives OTLP traces, applies tail sampling (latency policy >5000ms), generates span metrics and service graphs, batches, and exports to Grafana Cloud using basic auth. Requires environment variables GRAFANA_CLOUD_OTLP_ENDPOINT, GRAFANA_CLOUD_INSTANCE_ID, and GRAFANA_CLOUD_API_KEY.

```alloy
otelcol.receiver.otlp "default" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.receiver.otlp/

	// configures the default grpc endpoint "0.0.0.0:4317"
	grpc { }

	output {
		traces = [
			otelcol.processor.tail_sampling.default.input,
			otelcol.connector.servicegraph.default.input,
			otelcol.connector.spanmetrics.default.input,
		]
	}
}

otelcol.processor.transform "add_source" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.transform/
	error_mode = "ignore"

	metric_statements {
		context    = "scope"
		statements = [
			"set(resource.attributes[\"source\"], \"spanmetrics\") where name == \"spanmetricsconnector\"",
		]
	}

	output {
		metrics = [otelcol.processor.filter.drop_unneeded_span_metrics.input]
	}
}

otelcol.connector.spanmetrics "default" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.connector.spanmetrics/

	dimension {
		name = "service.namespace"
	}

	dimension {
		name = "service.version"
	}

	dimension {
		name = "deployment.environment"
	}

	dimension {
		name = "k8s.cluster.name"
	}

	dimension {
		name = "k8s.namespace.name"
	}

	dimension {
		name = "cloud.region"
	}

	dimension {
		name = "cloud.availability_zone"
	}

	histogram {
		explicit {
			buckets = ["0s", "0.005s", "0.01s", "0.025s", "0.05s", "0.075s", "0.1s", "0.25s", "0.5s", "0.75s", "1s", "2.5s", "5s", "7.5s", "10s"]
		}
		unit = "s"
	}

	namespace = "traces.spanmetrics"

	output {
		metrics = [otelcol.processor.transform.add_source.input]
	}
}

otelcol.processor.transform "use_grafana_metric_names" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.transform/
	error_mode = "ignore"

	metric_statements {
		context    = "metric"
		statements = [
			"set(name, \"traces.spanmetrics.latency\") where name == \"traces.spanmetrics.duration\"",
			"set(name, \"traces.spanmetrics.calls.total\") where name == \"traces.spanmetrics.calls\"",
		]
	}

	output {
		metrics = [otelcol.processor.batch.default.input]
	}
}

otelcol.processor.filter "drop_unneeded_span_metrics" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.filter/
	error_mode = "ignore"

	metrics {
		datapoint = [
			"IsMatch(metric.name, \"traces.spanmetrics.calls|traces.spanmetrics.duration\") and IsMatch(attributes[\"span.kind\"], \"SPAN_KIND_INTERNAL\")",
		]
	}

	output {
		metrics = [otelcol.processor.transform.use_grafana_metric_names.input]
	}
}

otelcol.connector.servicegraph "default" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.connector.servicegraph/
	dimensions = [
		"service.namespace",
		"service.version",
		"deployment.environment",
		"k8s.cluster.name",
		"k8s.namespace.name",
		"cloud.region",
		"cloud.availability_zone",
	]
	latency_histogram_buckets = ["0s", "0.005s", "0.01s", "0.025s", "0.05s", "0.075s", "0.1s", "0.25s", "0.5s", "0.75s", "1s", "2.5s", "5s", "7.5s", "10s"]

	store {
		ttl = "2s"
	}

	output {
		metrics = [otelcol.processor.batch.default.input]
	}
}

otelcol.processor.batch "default" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.batch/
	output {
		metrics = [otelcol.exporter.otlphttp.grafana_cloud.input]
		traces  = [otelcol.exporter.otlphttp.grafana_cloud.input]
	}
}

otelcol.processor.tail_sampling "default" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.tail_sampling/
	// Examples: keep all traces that take more than 5000 ms
	policy {
		name    = "all_traces_above_5000ms"
		type    = "latency"
		latency = {
			threshold_ms = 5000,
		}
	}

	output {
		traces = [otelcol.processor.batch.default.input]
	}
}

otelcol.exporter.otlphttp "grafana_cloud" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.exporter.otlphttp/
	client {
		endpoint = env("GRAFANA_CLOUD_OTLP_ENDPOINT")
		auth     = otelcol.auth.basic.grafana_cloud.handler
	}
}

otelcol.auth.basic "grafana_cloud" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.auth.basic/
	username = env("GRAFANA_CLOUD_INSTANCE_ID")
	password = env("GRAFANA_CLOUD_API_KEY")
}
```

--------------------------------

### Add PHP extensions to php.ini

Source: https://grafana.com/docs/opentelemetry/instrument/php

Add the protobuf and opentelemetry extensions to your php.ini file.

```ini
extension=protobuf.so

[opentelemetry]
extension=opentelemetry.so
```

--------------------------------

### Run lgtm in Kubernetes

Source: https://grafana.com/docs/opentelemetry/docker-lgtm

Commands to create Kubernetes resources and port-forward the lgtm service, plus mise alternatives. Port forwarding exposes ports 3000, 4317, and 4318.

```sh
# create k8s resources
kubectl apply -f k8s/lgtm.yaml

# port forwarding
kubectl port-forward service/lgtm 3000:3000 4317:4317 4318:4318

# Using mise
mise k8s-apply
mise k8s-port-forward
```

--------------------------------

### Configure otelcol.exporter.otlphttp with basic auth

Source: https://grafana.com/docs/opentelemetry/collector/sampling/tail

Configure the otelcol.exporter.otlphttp component to send telemetry to Grafana Cloud, using the otelcol.auth.basic component for authentication. The endpoint, username, and password are read from environment variables.

```alloy
otelcol.exporter.otlphttp "grafana_cloud" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.exporter.otlphttp/
	client {
		endpoint = env("GRAFANA_CLOUD_OTLP_ENDPOINT")
		auth     = otelcol.auth.basic.grafana_cloud.handler
	}
}

otelcol.auth.basic "grafana_cloud" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.auth.basic/
	username = env("GRAFANA_CLOUD_INSTANCE_ID")
	password = env("GRAFANA_CLOUD_API_KEY")
}
```

--------------------------------

### Scale Alloy tail sampling configuration

Source: https://grafana.com/docs/opentelemetry/collector/sampling/scale

Complete Alloy configuration for tail sampling with latency policy, span metrics, service graph, batching, and OTLP export to Grafana Cloud. Requires environment variables GRAFANA_CLOUD_OTLP_ENDPOINT, GRAFANA_CLOUD_INSTANCE_ID, and GRAFANA_CLOUD_API_KEY. The tail_sampling policy keeps traces with latency above 5000 ms.

```alloy
otelcol.receiver.otlp "default" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.receiver.otlp/

	// configures the default grpc endpoint "0.0.0.0:4317"
	grpc { }

	output {
		traces = [
			otelcol.processor.tail_sampling.default.input,
			otelcol.connector.servicegraph.default.input,
			otelcol.connector.spanmetrics.default.input,
		]
	}
}

otelcol.processor.transform "add_source" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.transform/
	error_mode = "ignore"

	metric_statements {
		context    = "scope"
		statements = [
			"set(resource.attributes[\"source\"], \"spanmetrics\") where name == \"spanmetricsconnector\"",
		]
	}

	output {
		metrics = [otelcol.processor.filter.drop_unneeded_span_metrics.input]
	}
}

otelcol.connector.spanmetrics "default" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.connector.spanmetrics/

	dimension {
		name = "service.namespace"
	}

	dimension {
		name = "service.version"
	}

	dimension {
		name = "deployment.environment"
	}

	dimension {
		name = "k8s.cluster.name"
	}

	dimension {
		name = "k8s.namespace.name"
	}

	dimension {
		name = "cloud.region"
	}

	dimension {
		name = "cloud.availability_zone"
	}

	histogram {
		explicit {
			buckets = ["0s", "0.005s", "0.01s", "0.025s", "0.05s", "0.075s", "0.1s", "0.25s", "0.5s", "0.75s", "1s", "2.5s", "5s", "7.5s", "10s"]
		}
		unit = "s"
	}

	output {
		metrics = [otelcol.processor.transform.add_source.input]
	}
}

otelcol.processor.filter "drop_unneeded_span_metrics" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.filter/
	error_mode = "ignore"

	metrics {
		datapoint = [
			"IsMatch(metric.name, \"calls|duration\") and IsMatch(attributes[\"span.kind\"], \"SPAN_KIND_INTERNAL\")",
		]
	}

	output {
		metrics = [otelcol.processor.batch.default.input]
	}
}

otelcol.connector.servicegraph "default" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.connector.servicegraph/
	dimensions = [
		"service.namespace",
		"service.version",
		"deployment.environment",
		"k8s.cluster.name",
		"k8s.namespace.name",
		"cloud.region",
		"cloud.availability_zone",
	]
	latency_histogram_buckets = ["0s", "0.005s", "0.01s", "0.025s", "0.05s", "0.075s", "0.1s", "0.25s", "0.5s", "0.75s", "1s", "2.5s", "5s", "7.5s", "10s"]

	store {
		ttl = "2s"
	}

	output {
		metrics = [otelcol.processor.batch.default.input]
	}
}

otelcol.processor.batch "default" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.batch/
	output {
		metrics = [otelcol.exporter.otlphttp.grafana_cloud.input]
		traces  = [otelcol.exporter.otlphttp.grafana_cloud.input]
	}
}

otelcol.processor.tail_sampling "default" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.tail_sampling/
	// Examples: keep all traces that take more than 5000 ms
	policy {
		name    = "all_traces_above_5000ms"
		type    = "latency"
		latency = {
			threshold_ms = 5000,
		}
	}

	output {
		traces = [otelcol.processor.batch.default.input]
	}
}

otelcol.exporter.otlphttp "grafana_cloud" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.exporter.otlphttp/
	client {
		endpoint = env("GRAFANA_CLOUD_OTLP_ENDPOINT")
		auth     = otelcol.auth.basic.grafana_cloud.handler
	}
}

otelcol.auth.basic "grafana_cloud" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.auth.basic/
	username = env("GRAFANA_CLOUD_INSTANCE_ID")
	password = env("GRAFANA_CLOUD_API_KEY")
}
```

--------------------------------

### Include Composer autoloader

Source: https://grafana.com/docs/opentelemetry/instrument/php

Add this line to your application entry point if Composer autoloading is not enabled.

```php
require __DIR__ . '/vendor/autoload.php';
```

--------------------------------

### .NET 6+ console: UseGrafana() with AddConsoleExporter()

Source: https://grafana.com/docs/opentelemetry/instrument/grafana-dotnet

For .NET 6+ console applications. Uses the Grafana.OpenTelemetry package to configure tracing, metrics, and logging with the UseGrafana() extension. The AddConsoleExporter() calls are for local testing only and must be removed for production.

```netconsole
// .NET 6+ console
using OpenTelemetry;
using OpenTelemetry.Metrics;
using OpenTelemetry.Trace;
using Grafana.OpenTelemetry;

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .UseGrafana()
    .AddConsoleExporter()
    .Build();
using var meterProvider = Sdk.CreateMeterProviderBuilder()
    .UseGrafana()
    .AddConsoleExporter()
    .Build();
using var loggerFactory = LoggerFactory.Create(builder =>
{
    builder.AddOpenTelemetry(logging =>
    {
        logging.UseGrafana()
            .AddConsoleExporter();
    });
});
```

--------------------------------

### Configure service graph connector with otelcol.connector.servicegraph

Source: https://grafana.com/docs/opentelemetry/collector/sampling/tail

Configures the service graph connector to generate metrics for service relationships. Sets dimensions for service.namespace, service.version, deployment.environment, and others. Uses latency_histogram_buckets with values from 0.005s to 10s, and a store TTL of 2s. Outputs metrics to the batch processor.

```hcl
otelcol.connector.servicegraph "default" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.connector.servicegraph/
	dimensions = [
		"service.namespace",
		"service.version",
		"deployment.environment",
		"k8s.cluster.name",
		"k8s.namespace.name",
		"cloud.region",
		"cloud.availability_zone",
	]
	latency_histogram_buckets = ["0.005s", "0.01s", "0.025s", "0.05s", "0.075s", "0.1s", "0.25s", "0.5s", "0.75s", "1s", "2.5s", "5s", "7.5s", "10s"]

	store {
		ttl = "2s"
	}

	output {
		metrics = [otelcol.processor.batch.default.input]
	}
}
```

--------------------------------

### Set HTTP OTLP endpoint environment variables

Source: https://grafana.com/docs/opentelemetry/collector/grafana-alloy-kubernetes

Use this shell snippet to configure your instrumentation to send telemetry data to Alloy over HTTP. Replace <HTTP_ENDPOINT> with the HTTP endpoint listed in the Configure Application Instrumentation section of the Cluster Configuration tab.

```shell
export OTEL_EXPORTER_OTLP_ENDPOINT=<HTTP_ENDPOINT>
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

```