OpenTelemetry eBPF Instrumentation

https://opentelemetry.io/docs/zero-code/obi/
Admin
OpenTelemetry eBPF Instrumentation is an auto-instrumentation tool that uses eBPF to automatically...
Tokens:43,462
Snippets:561
Trust Score:10
Update:5 months ago
Show doc for...
Tokens:
Raw
### Run Docker Compose

Source: https://opentelemetry.io/docs/zero-code/obi/setup/docker

Starts the services defined in the Docker Compose file. Use this to replicate the Docker CLI example setup.

```bash
docker compose -f compose-example.yml up
```

--------------------------------

### Install OBI Binaries to System PATH

Source: https://opentelemetry.io/docs/zero-code/obi/setup/standalone

Install the OBI binary to a directory included in your system's PATH, such as /usr/local/bin, to make it accessible from any directory. Verify the installation by checking the OBI version.

```bash
# Move binaries to a directory in your PATH
sudo cp obi /usr/local/bin/

# Verify installation
obi --version

```

--------------------------------

### Standalone OBI Configuration Example

Source: https://opentelemetry.io/docs/zero-code/obi/configure/collector-receiver

Example configuration for standalone OBI, showing trace export endpoint and open port.

```yaml
# obi-config.yaml
otel_traces_export:
  endpoint: http://backend:4318

open_port: 8080
```

--------------------------------

### Example YAML Configuration Snippet

Source: https://opentelemetry.io/docs/zero-code/obi/configure/options

Illustrates sample global configuration properties in YAML format for OBI.

```yaml
trace_printer: json
shutdown_timeout: 30s
channel_buffer_len: 33
```

--------------------------------

### Example of URL Paths

Source: https://opentelemetry.io/docs/zero-code/obi/configure/routes-decorator

These are example URL paths that demonstrate a structure where the heuristic route decorator can be effective.

```text
document/d/CfMkAGbE_aivhFydEpaRafPuGWbmHfG/edit (no numbers in the ID)
document/d/C2fMkAGb3E_aivhFyd5EpaRafP123uGWbmHfG/edit
```

--------------------------------

### Example OBI Log Output

Source: https://opentelemetry.io/docs/zero-code/obi/cilium-compatibility

Example output from OBI pod logs, demonstrating detected HTTP requests with trace-context propagation details.

```text
2025-01-17 21:42:18.11794218 (5.045099ms[5.045099ms]) HTTPClient 200 GET /tracemetoo [10.244.1.92 as go-service.default:37450]->[10.96.214.17 as python-service.default:8080] size:0B svc=[default/go-service go] traceparent=[00-14f07e11b5e57f14fd2da0541f0ddc2f-319fb03373427a41[cfa6d5d448e40b00]-01]
2025-01-17 21:42:18.11794218 (5.284521ms[5.164701ms]) HTTP 200 GET /gotracemetoo [10.244.2.144 as nodejs-service.default:57814]->[10.244.1.92 as go-service.default:8080] size:0B svc=[default/go-service go] traceparent=[00-14f07e11b5e57f14fd2da0541f0ddc2f-cfa6d5d448e40b00[cce1e6b5e932b89a]-01]
2025-01-17 21:42:18.11794218 (1.934744ms[1.934744ms]) HTTP 403 GET /users [10.244.2.32 as ruby-service.default:46876]->[10.244.2.176 as ruby-service.default:3000] size:222B svc=[default/ruby-service ruby] traceparent=[00-14f07e11b5e57f14fd2da0541f0ddc2f-57d77d99e9665c54[3d97d26b0051112b]-01]
2025-01-17 21:42:18.11794218 (2.116628ms[2.116628ms]) HTTPClient 403 GET /users [10.244.2.32 as ruby-service.default:46876]->[10.96.69.89 as ruby-service.default:3000] size:256B svc=[default/ruby-service ruby] traceparent=[00-14f07e11b5e57f14fd2da0541f0ddc2f-ff48ab147cc92f93[2770ac4619aa0042]-01]
2025-01-17 21:42:18.11794218 (4.281525ms[4.281525ms]) HTTP 200 GET /tracemetoo [10.244.1.92 as go-service.default:37450]->[10.244.2.32 as ruby-service.default:8080] size:178B svc=[default/ruby-service ruby] traceparent=[00-14f07e11b5e57f14fd2da0541f0ddc2f-2770ac4619aa0042[319fb03373427a41]-01]
2025-01-17 21:42:18.11794218 (5.391191ms[5.391191ms]) HTTPClient 200 GET /gotracemetoo [10.244.2.144 as nodejs-service.default:57814]->[10.96.134.167 as go-service.default:8080] size:256B svc=[default/nodejs-service nodejs] traceparent=[00-14f07e11b5e57f14fd2da0541f0ddc2f-202ee68205e4ef3b[9408610968fa20f8]-01]
2025-01-17 21:42:18.11794218 (6.939027ms[6.939027ms]) HTTP 200 GET /traceme [127.0.0.1 as 127.0.0.1:44720]->[127.0.0.1 as 127.0.0.1.default:3000] size:86B svc=[default/nodejs-service nodejs] traceparent=[00-14f07e11b5e57f14fd2da0541f0ddc2f-9408610968fa20f8[0000000000000000]-01]
```

--------------------------------

### Example OBI Trace Output

Source: https://opentelemetry.io/docs/zero-code/obi/setup/docker

Sample output demonstrating OBI successfully capturing and printing trace requests to stdout.

```text
time=2023-05-22T14:03:42.402Z level=INFO msg="creating instrumentation pipeline"
time=2023-05-22T14:03:42.526Z level=INFO msg="Starting main node"
2023-05-22 14:03:53.5222353 (19.066625ms[942.583µs]) 200 GET / [172.17.0.1]->[localhost:18443] size:0B
2023-05-22 14:03:53.5222353 (355.792µs[321.75µs]) 200 GET /static/style.css [172.17.0.1]->[localhost:18443] size:0B
2023-05-22 14:03:53.5222353 (170.958µs[142.916µs]) 200 GET /static/img.png [172.17.0.1]->[localhost:18443] size:0B
2023-05-22 14:13:47.52221347 (7.243667ms[295.292µs]) 200 GET /entry/201710281345_instructions.md [172.17.0.1]->[localhost:18443] size:0B
2023-05-22 14:13:47.52221347 (115µs[75.625µs]) 200 GET /static/style.css [172.17.0.1]->[localhost:18443] size:0B
```

--------------------------------

### Run Go Blog Service with Docker

Source: https://opentelemetry.io/docs/zero-code/obi/setup/docker

Starts a containerized Go blog service for testing OBI. Ensure the VERSION environment variable is set.

```bash
export VERSION=v0.8.0
docker run -p 18443:8443 --name goblog mariomac/goblog:dev
```

--------------------------------

### SQL++ Request Body Example

Source: https://opentelemetry.io/docs/zero-code/obi/configure/export-data

Example of a SQL++ query sent as a JSON body to the /query/service endpoint.

```json
{
  "statement": "SELECT * FROM `bucket`.`scope`.`collection` WHERE id = $1",
  "query_context": "default:`bucket`.`scope`"
}
```

--------------------------------

### Discover Pods by Namespace and Labels

Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery

This example discovers all Pods in the `frontend` namespace that have a label `instrument` with a value matching the glob `obi`.

```yaml
discovery:
  instrument:
    - k8s_namespace: frontend
      k8s_pod_labels:
        instrument: obi
```

--------------------------------

### Install OBI Helm Chart

Source: https://opentelemetry.io/docs/zero-code/obi/setup/kubernetes-helm

Deploy a OBI DaemonSet with default configuration in a specified namespace. The `--create-namespace` flag ensures the namespace exists.

```bash
helm install obi -n obi --create-namespace open-telemetry/opentelemetry-ebpf-instrumentation
```

--------------------------------

### Discover Pods by Namespace and Annotations

Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery

This example discovers all Pods in the `backend` namespace that have an annotation `obi.instrument` with a value matching the glob `true`.

```yaml
discovery:
  instrument:
    - k8s_namespace: backend
      k8s_pod_annotations:
        obi.instrument: 'true'
```

--------------------------------

### OBI Configuration YAML Example

Source: https://opentelemetry.io/docs/zero-code/obi/configure/example

Use this YAML configuration to set up OBI. It includes options for service discovery, logging level, eBPF context propagation, and trace/metric export endpoints.

```yaml
discovery:
  instrument:
    - open_ports: 8443
log_level: DEBUG

ebpf:
  context_propagation: all

otel_traces_export:
  endpoint: http://localhost:4318

prometheus_export:
  port: 8999
  path: /metrics

```

--------------------------------

### JSON Log Format Before OBI Enrichment

Source: https://opentelemetry.io/docs/zero-code/obi/trace-log-correlation

Example of a JSON log entry before trace context is injected by OBI.

```json
{ "level": "info", "message": "Request processed", "duration_ms": 125 }

```

--------------------------------

### Cosign Verification Output Example

Source: https://opentelemetry.io/docs/zero-code/obi/setup/docker

An example of successful Cosign verification output, indicating that the claims were validated and the image digest is confirmed.

```json
Verification for index.docker.io/otel/ebpf-instrument:main --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates

[{"critical":{"identity":{"docker-reference":"index.docker.io/otel/ebpf-instrument:main"},"image":{"docker-manifest-digest":"sha256:55426a2bbb8003573a961697888aa770a1f5f67fcda2276dc2187d1faf7181fe"},"type":"https://sigstore.dev/cosign/sign/v1"},"optional":{}}]

```

--------------------------------

### Example Network Metric Output

Source: https://opentelemetry.io/docs/zero-code/obi/network/quickstart

This is an example of the debug output you might see in OBI logs, indicating a captured network flow with IP addresses and interface information.

```text
network_flow: obi.ip=172.18.0.2 iface= direction=255 src.address=10.244.0.4 dst.address=10.96.0.1
```

--------------------------------

### Enable Prometheus Exporter with HTTP/SQL Instrumentation

Source: https://opentelemetry.io/docs/zero-code/obi/configure/export-data

Example of setting the `instrumentations` option to enable collection of HTTP/HTTPS/HTTP2 and SQL application metrics.

```yaml
instrumentations: ['http', 'sql']
```

--------------------------------

### OBI Debug Trace Span Example

Source: https://opentelemetry.io/docs/zero-code/obi/troubleshooting

Example output of raw trace spans logged to the console when the debug traces exporter is enabled. This format is human-readable and matches OTel Collector's detailed verbosity.

```text
Traces	{"resource spans": 1, "spans": 1}
ResourceSpans #0
Resource SchemaURL:
Resource attributes:
     -> service.name: Str(flagd)
     -> telemetry.sdk.language: Str(go)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.distro.name: Str(opentelemetry-ebpf-instrumentation)
     -> telemetry.sdk.version: Str(main)
     -> host.name: Str(flagd-5cccb4c4f5-sfkcm)
     -> os.type: Str(linux)
     -> service.namespace: Str(opentelemetry-demo)
     -> k8s.owner.name: Str(flagd)
     -> k8s.kind: Str(Deployment)
     -> k8s.replicaset.name: Str(flagd-5cccb4c4f5)
     -> k8s.pod.name: Str(flagd-5cccb4c4f5-sfkcm)
     -> k8s.container.name: Str(flagd)
     -> k8s.deployment.name: Str(flagd)
     -> service.version: Str(2.0.2)
     -> k8s.namespace.name: Str(default)
     -> otel.library.name: Str(go.opentelemetry.io/obi)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope
Span #0
    Trace ID       : 63a2723a58e0033170e58b1ff27ef03d
    Parent ID      :
    ID             : fab47609b60cc4e0
    Name           : /opentelemetry.proto.collector.metrics.v1.MetricsService/Export
    Kind           : Client
    Start time     : 2025-11-28 16:10:35.4241749 +0000 UTC
    End time       : 2025-11-28 16:10:35.42555658 +0000 UTC
    Status code    : Unset
    Status message :
Attributes:
     -> rpc.method: Str(/opentelemetry.proto.collector.metrics.v1.MetricsService/Export)
     -> rpc.system: Str(grpc)
     -> rpc.grpc.status_code: Int(0)
     -> server.address: Str(otel-collector.default)
     -> peer.service: Str(otel-collector.default)
     -> server.port: Int(4317)

```

--------------------------------

### Docker Compose File for OBI Setup

Source: https://opentelemetry.io/docs/zero-code/obi/setup/docker

Defines services for a Go blog application and the OBI autoinstrumenter using Docker Compose. Ensure the image tag for autoinstrumenter matches your desired version.

```yaml
version: '3.8'

services:
  # Service to instrument. Change it to any
  # other container that you want to instrument.
  goblog:
    image: mariomac/goblog:dev
    ports:
      # Exposes port 18843, forwarding it to container port 8443
      - '18443:8443'

  autoinstrumenter:
    image: otel/ebpf-instrument:main
    pid: 'host'
    privileged: true
    environment:
      OTEL_EBPF_TRACE_PRINTER: text
      OTEL_EBPF_OPEN_PORT: 8443
```

--------------------------------

### Dockerfile for OBI Collector

Source: https://opentelemetry.io/docs/zero-code/obi/configure/collector-receiver

Create a Dockerfile to package the custom collector binary for deployment. This includes installing necessary tools and copying the collector binary.

```dockerfile
FROM alpine:latest

# Install required tools
RUN apk --no-cache add ca-certificates

# Copy the custom collector binary built with OCB
COPY dist/otelcol-obi /otelcol-obi

# Make it executable
RUN chmod +x /otelcol-obi

ENTRYPOINT ["/otelcol-obi"]
```

--------------------------------

### Service Name Template Example

Source: https://opentelemetry.io/docs/zero-code/obi/configure/metrics-traces-attributes

Use Go templates to dynamically set service names based on Kubernetes metadata like namespace, labels, and container name. Ensure to trim whitespace.

```go
{{- .Meta.Namespace }}/{{ index .Meta.Labels "app.kubernetes.io/name" }}/{{ index .Meta.Labels "app.kubernetes.io/component" -}}{{ if .ContainerName }}/{{ .ContainerName -}}{{ end -}}
```

```go
{{- .Meta.Namespace }}/{{ index .Meta.Labels "app.kubernetes.io/name" }}/{{ index .Meta.Labels "app.kubernetes.io/component" -}}
```

--------------------------------

### Pessimistic Cardinality Calculation

Source: https://opentelemetry.io/docs/zero-code/obi/cardinality

This example shows a pessimistic calculation for cardinality, multiplying all potential factors. Use this as an upper bound when estimating cardinality.

```shell
#Instances * #MetricNames * #HistoBuckets * #Operations * #Endpoints * #ReturnCodes =
= 2 * 5 * 177/3 * 37/3 =2771
```

--------------------------------

### Service Name Example: Before Enhanced Lookup

Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery

Illustrates how service names might appear as IP addresses without enhanced lookup enabled.

```yaml
service.name: "10.0.1.42"
peer.service: "10.0.2.15"
```

--------------------------------

### Prometheus Metric with CIDR Attributes

Source: https://opentelemetry.io/docs/zero-code/obi/network/quickstart

This example shows the equivalent Prometheus metric format for `obi_network_flow_bytes` when CIDR breakdown is configured, demonstrating aggregation by source and destination CIDR names.

```text
obi_network_flow_bytes:
  src_cidr="cluster-internal"
  dst_cidr="private"

```

--------------------------------

### Service Name Example: After Enhanced Lookup

Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery

Shows how service names are resolved to meaningful identifiers like 'frontend' and 'backend-api' with enhanced lookup enabled.

```yaml
service.name: "frontend"
peer.service: "backend-api"
```

--------------------------------

### Prometheus Metric with Allowed Attributes

Source: https://opentelemetry.io/docs/zero-code/obi/network/quickstart

This example shows the equivalent Prometheus metric format for `obi.network.flow.bytes` when specific Kubernetes owner attributes are allowed, demonstrating aggregation by source and destination owner type.

```text
obi.network.flow.bytes:
  k8s_src_owner_name="frontend"
  k8s_src_owner_type="deployment"
  k8s_dst_owner_name="backend"
  k8s_dst_owner_type="deployment"

```

--------------------------------

### Configure OTLP Export Endpoint

Source: https://opentelemetry.io/docs/zero-code/obi/configure/export-modes

Specify where traces and metrics are sent. This example configures OBI to export to a local OTel Collector running on port 4318.

```yaml
otel_metrics_export:
  endpoint: http://localhost:4318
otel_traces_export:
  endpoint: http://localhost:4318

```

--------------------------------

### Custom OBI Configuration with Helm

Source: https://opentelemetry.io/docs/zero-code/obi/setup/kubernetes-helm

Override default OBI configurations by creating a YAML file and passing it to the Helm install or upgrade command using the -f flag.

```yaml
config:
  data:
    # Contents of the actual OBI configuration file
    discovery:
      instrument:
        - k8s_namespace: demo
        - k8s_namespace: blog
    routes:
      unmatched: heuristic

```

```bash
helm install obi open-telemetry/opentelemetry-ebpf-instrumentation -f helm-obi.yml
```

```bash
helm upgrade obi open-telemetry/opentelemetry-ebpf-instrumentation -f helm-obi.yml
```

--------------------------------

### OpenTelemetry Collector Configuration with OBI Receiver

Source: https://opentelemetry.io/docs/zero-code/obi/configure/collector-receiver

Example configuration for the OpenTelemetry Collector to enable the OBI receiver. This snippet is intended to be part of a larger collector configuration file.

```yaml

```

--------------------------------

### Instrument Executable with Open Ports

Source: https://opentelemetry.io/docs/zero-code/obi/configure/export-modes

Specify the executable to instrument by adding the `open_ports` property to the YAML document. This example instruments a service running on port 443.

```yaml
discovery:
  instrument:
    - open_ports: 443

```

--------------------------------

### SQL++ Form Encoded Request Example

Source: https://opentelemetry.io/docs/zero-code/obi/configure/export-data

Example of a SQL++ query sent as form-encoded data to the /query/service endpoint.

```urlencoded
statement=SELECT+*+FROM+users&query_context=default:`travel-sample`.`inventory`
```

--------------------------------

### Exclude Specific Namespaces from Instrumentation

Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery

This example demonstrates how to instrument all namespaces except for `development` and `staging`.

```yaml
discovery:
  instrument:
    - k8s_namespace: '*' # Instrument all namespaces
  exclude_instrument:
    - k8s_namespace: development # Except development namespace
    - k8s_namespace: staging # And staging namespace
```

--------------------------------

### Run OBI with Configuration File

Source: https://opentelemetry.io/docs/zero-code/obi/configure/export-modes

Execute OBI with a specified configuration file using the `-config` flag or by setting the `OTEL_EBPF_CONFIG_PATH` environment variable.

```bash
obi -config instrument-config.yml

```

```bash
OTEL_EBPF_CONFIG_PATH=instrument-config.yml obi

```

--------------------------------

### JSON Log Format After OBI Enrichment

Source: https://opentelemetry.io/docs/zero-code/obi/trace-log-correlation

Example of a JSON log entry after OBI has injected 'trace_id' and 'span_id' fields.

```json
{
  "level": "info",
  "message": "Request processed",
  "duration_ms": 125,
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7"
}

```

--------------------------------

### Log Merged Configuration in OBI

Source: https://opentelemetry.io/docs/zero-code/obi/troubleshooting

Instruct OBI to log its final merged configuration at startup using 'yaml' for readability or 'json' for log shippers.

```bash
# Log configuration in YAML format
export OTEL_EBPF_LOG_CONFIG=yaml

# Log configuration in JSON format
export OTEL_EBPF_LOG_CONFIG=json
```

--------------------------------

### Prometheus Metric with CIDR-based Labels

Source: https://opentelemetry.io/docs/zero-code/obi/network

An example of a Prometheus metric labeled with source and destination CIDR names, as configured in OBI.

```prometheus
obi_network_flow_bytes:
  src_cidr="cluster-internal"
  dst_cidr="private"
```

--------------------------------

### Build and Push Docker Image

Source: https://opentelemetry.io/docs/zero-code/obi/configure/collector-receiver

Build the Docker image for the OBI collector and push it to a container registry. Replace 'my-registry/otelcol-obi:v0.6.0' with your registry and tag.

```bash
docker build -t my-registry/otelcol-obi:v0.6.0 .
docker push my-registry/otelcol-obi:v0.6.0
```

--------------------------------

### Heuristic Route Decorator Output

Source: https://opentelemetry.io/docs/zero-code/obi/configure/routes-decorator

This shows the reduced, low-cardinality route achieved by the heuristic decorator for the preceding example URLs.

```text
document/d/*/edit
```

--------------------------------

### Run OBI for Application Observability with Trace Context Propagation

Source: https://opentelemetry.io/docs/zero-code/obi/security

Runs OBI with environment variables to enable application observability and trace context propagation. This requires a broad set of capabilities, including `CAP_BPF`, `CAP_NET_ADMIN`, and `CAP_SYS_PTRACE`.

```bash
OTEL_EBPF_CONTEXT_PROPAGATION=all OTEL_EBPF_OPEN_PORT=8080 OTEL_EBPF_TRACE_PRINTER=text bin/obi
```

--------------------------------

### Run OBI with sudo

Source: https://opentelemetry.io/docs/zero-code/obi/setup/standalone

Use this command to run OBI with elevated privileges, specifying the path to your configuration file.

```bash
sudo obi --config=<path to config file>
```

--------------------------------

### Exclude Specific Executables from Instrumentation

Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery

This example excludes Prometheus and Grafana executables from instrumentation when they are running on ports 80, 443, or 8080.

```yaml
discovery:
  rules:
    - match:
        open_ports: 80,443,8080
      exclude:
        exe_path: '*prometheus*'
        exe_path: '*grafana*'
```

--------------------------------

### Run OBI for Network Metrics via Socket Filter

Source: https://opentelemetry.io/docs/zero-code/obi/security

Execute OBI with specific environment variables to enable network metrics collection via socket filters. Requires `CAP_BPF` and `CAP_NET_RAW` capabilities to be set on the binary.

```bash
OTEL_EBPF_NETWORK_METRICS=1 OTEL_EBPF_NETWORK_PRINT_FLOWS=1 bin/obi
```

--------------------------------

### Configure OBI Internal Metrics Exporter

Source: https://opentelemetry.io/docs/zero-code/obi/troubleshooting

Enable OBI internal metrics by configuring the exporter. Options include 'none' (default), 'prometheus' for an HTTP server, or 'otlp' for OTLP export.

```bash
# Export internal metrics in Prometheus format
export OTEL_EBPF_INTERNAL_METRICS_EXPORTER=prometheus

# Export internal metrics via OTLP
export OTEL_EBPF_INTERNAL_METRICS_EXPORTER=otlp
```

--------------------------------

### Deploy OBI as a DaemonSet

Source: https://opentelemetry.io/docs/zero-code/obi/setup/kubernetes

Configure the DaemonSet with `hostPID: true` to access host processes. Set `OTEL_EBPF_AUTO_TARGET_EXE` to select executables by name, useful when ports are internal or dynamic. Enable Kubernetes metadata decoration with `OTEL_EBPF_KUBE_METADATA_ENABLE`.

```yaml
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: obi
  labels:
    app: obi
spec:
  selector:
    matchLabels:
      app: obi
  template:
    metadata:
      labels:
        app: obi
    spec:
      hostPID: true # Required to access the processes on the host
      serviceAccountName: obi # required if you want kubernetes metadata decoration
      containers:
        - name: autoinstrument
          image: otel/ebpf-instrument:main
          securityContext:
            privileged: true
          env:
            # Select the executable by its name instead of OTEL_EBPF_OPEN_PORT
            - name: OTEL_EBPF_AUTO_TARGET_EXE
              value: '*/goblog'
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: 'http://otelcol:4318'
              # required if you want kubernetes metadata decoration
            - name: OTEL_EBPF_KUBE_METADATA_ENABLE
              value: 'true'

```

--------------------------------

### Download and Verify OBI Binaries

Source: https://opentelemetry.io/docs/zero-code/obi/setup/standalone

Download the OBI archive and checksums for your architecture, then verify the integrity of the downloaded archive using sha256sum. Ensure the VERSION and ARCH variables are set correctly.

```bash
# Set your desired version (find latest at
# https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/releases)
VERSION=0.6.0

# Determine your architecture
# For Intel/AMD 64-bit: amd64
# For ARM 64-bit: arm64
ARCH=amd64

# Download the archive for your architecture
wget https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/releases/download/v${VERSION}/obi-v${VERSION}-linux-${ARCH}.tar.gz

# Download checksums
wget https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/releases/download/v${VERSION}/SHA256SUMS

# Verify the archive
sha256sum -c SHA256SUMS --ignore-missing

# Extract the archive
tar -xzf obi-v${VERSION}-linux-${ARCH}.tar.gz

```

--------------------------------

### Prometheus Metric with Kubernetes Owner Attributes

Source: https://opentelemetry.io/docs/zero-code/obi/network

An example of how a Prometheus metric might appear after aggregation by Kubernetes owner attributes, as configured in OBI.

```prometheus
obi_network_flow_bytes:
  k8s_src_owner_name="frontend"
  k8s_src_owner_type="deployment"
  k8s_dst_owner_name="backend"
  k8s_dst_owner_type="deployment"
```

--------------------------------

### Create OBI Namespace

Source: https://opentelemetry.io/docs/zero-code/obi/cilium-compatibility

Creates the Kubernetes namespace where OBI components will be deployed.

```bash
kubectl create namespace obi
```

--------------------------------

### Enable Network Metrics in OBI

Source: https://opentelemetry.io/docs/zero-code/obi/network/quickstart

Set this environment variable to true to enable the collection of network metrics.

```bash
export OTEL_EBPF_NETWORK_METRICS=true
```

--------------------------------

### Run OBI from extracted directory

Source: https://opentelemetry.io/docs/zero-code/obi/setup/standalone

Execute this command if OBI is not installed in your system's PATH, running it directly from the extracted directory with elevated privileges.

```bash
sudo ./obi --config=<path to config file>
```

--------------------------------

### Run OBI for Application Observability

Source: https://opentelemetry.io/docs/zero-code/obi/security

Executes OBI with environment variables to enable application observability, such as tracing and port monitoring. Requires specific capabilities like `CAP_BPF` and `CAP_SYS_PTRACE`.

```bash
OTEL_EBPF_OPEN_PORT=8080 OTEL_EBPF_TRACE_PRINTER=text bin/obi
```

--------------------------------

### Enable SQL++ Instrumentation

Source: https://opentelemetry.io/docs/zero-code/obi/configure/export-data

Enable SQL++ instrumentation and configure the BPF buffer size for capturing request/response bodies. A larger buffer is needed for full body capture.

```bash
export OTEL_EBPF_HTTP_SQLPP_ENABLED=true
export OTEL_EBPF_BPF_BUFFER_SIZE_HTTP=2048  # Larger than default; needed to capture request/response bodies
```

--------------------------------

### OBI Unprivileged DaemonSet Configuration

Source: https://opentelemetry.io/docs/zero-code/obi/setup/kubernetes

This configuration demonstrates how to deploy OBI as a DaemonSet with reduced privileges. It replaces `privileged:true` with specific Linux capabilities required for OBI's functionality, such as `BPF`, `SYS_PTRACE`, `NET_RAW`, `CHECKPOINT_RESTORE`, `DAC_READ_SEARCH`, and `PERFMON`. Ensure `hostPID: true` is set for process discovery.

```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: obi
  namespace: obi-demo
  labels:
    k8s-app: obi
spec:
  selector:
    matchLabels:
      k8s-app: obi
  template:
    metadata:
      labels:
        k8s-app: obi
    spec:
      serviceAccount: obi
      hostPID: true           # <-- Important. Required in Daemonset mode so OBI can discover all monitored processes
      containers:
      - name: obi
        terminationMessagePolicy: FallbackToLogsOnError
        image: otel/ebpf-instrument:main
        env:
          - name: OTEL_EBPF_TRACE_PRINTER
            value: "text"
          - name: OTEL_EBPF_KUBE_METADATA_ENABLE
            value: "autodetect"
          - name: KUBE_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          ...
        securityContext:
          runAsUser: 0
          readOnlyRootFilesystem: true
          capabilities:
            add:
              - BPF                 # <-- Important. Required for most eBPF probes to function correctly.
              - SYS_PTRACE          # <-- Important. Allows OBI to access the container namespaces and inspect executables.
              - NET_RAW             # <-- Important. Allows OBI to use socket filters for http requests.
              - CHECKPOINT_RESTORE  # <-- Important. Allows OBI to open ELF files.
              - DAC_READ_SEARCH     # <-- Important. Allows OBI to open ELF files.
              - PERFMON             # <-- Important. Allows OBI to load BPF programs.
              #- SYS_RESOURCE       # <-- pre 5.11 only. Allows OBI to increase the amount of locked memory.
              #- SYS_ADMIN          # <-- Required for Go application trace context propagation, or if kernel.perf_event_paranoid >= 3 on Debian distributions.
            drop:
              - ALL
        volumeMounts:
        - name: var-run-obi
          mountPath: /var/run/obi
        - name: cgroup
          mountPath: /sys/fs/cgroup
      tolerations:
      - effect: NoSchedule
        operator: Exists
      - effect: NoExecute
        operator: Exists
      volumes:
      - name: var-run-obi
        emptyDir: {}
      - name: cgroup
        hostPath:
          path: /sys/fs/cgroup
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: some-service
  namespace: obi-demo
  ...
---
```

--------------------------------

### Combine Numeric and String Filters for Network Metrics

Source: https://opentelemetry.io/docs/zero-code/obi/configure/filter-metrics-traces

Example combining a string matcher (`not_match`) with a numeric matcher (`less_than`) to filter network metrics.

```yaml
filter:
  network:
    transport:
      not_match: UDP
    dst_port:
      less_than: 1024
```

--------------------------------

### Inspect OBI SBOMs with jq and Grype

Source: https://opentelemetry.io/docs/zero-code/obi/setup/standalone

Use `jq` to parse and display component information from the SBOMs, or use `grype` to scan the SBOM for vulnerabilities. These commands help in reviewing dependencies and licenses without executing the binaries.

```bash
# List component names and versions from the archive SBOM
jq '.components[] | {name, version}' obi-v${VERSION}-linux-${ARCH}.cyclonedx.json

# Scan the SBOM with Grype
grype sbom:obi-v${VERSION}-linux-${ARCH}.cyclonedx.json

# Inspect the Java agent dependency graph
jq '.components[] | {name, version}' obi-java-agent-v${VERSION}.cyclonedx.json

```

--------------------------------

### Enable Instrumentation for Default-Excluded Namespace

Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery

To instrument a namespace excluded by default, remove it from `default_exclude_instrument` and add it to the `instrument` section. This example enables instrumentation for the 'monitoring' namespace.

```yaml
discovery:
  # Include the monitoring namespace in instrumentation
  instrument:
    - k8s_namespace: monitoring

  # Override default exclusions to remove monitoring namespace
  # This list keeps other default exclusions but removes monitoring
  default_exclude_instrument:
    - exe_path: '{*/obi,obi,*otelcol,*otelcol-contrib,*otelcol-contrib[!/]*}'
    - k8s_namespace: kube-system
    - k8s_namespace: kube-node-lease
    - k8s_namespace: local-path-storage
    - k8s_namespace: cert-manager
    # monitoring namespace removed from this list
    - k8s_namespace: gke-connect
    - k8s_namespace: gke-gmp-system
    - k8s_namespace: gke-managed-cim
    - k8s_namespace: gke-managed-filestorecsi
    - k8s_namespace: gke-managed-metrics-server
    - k8s_namespace: gke-managed-system
    - k8s_namespace: gke-system
    - k8s_namespace: gke-managed-volumepopulator
    - k8s_namespace: gatekeeper-system

```

--------------------------------

### Exclude Services by Pod Labels

Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery

This example excludes services in the `production` namespace that have the label `skip-instrumentation` set to `true`. You can use any custom label key and value.

```yaml
discovery:
  rules:
    - match:
        k8s_namespace: production
      exclude:
        k8s_pod_labels:
          skip-instrumentation: 'true'
```

--------------------------------

### Enable Kubernetes Metadata Decoration

Source: https://opentelemetry.io/docs/zero-code/obi/network/quickstart

Set this environment variable to true to decorate network metrics with Kubernetes metadata, which is required for network metrics.

```bash
export OTEL_EBPF_KUBE_METADATA_ENABLE=true
```

--------------------------------

### Target Processes by Executable Path

Source: https://opentelemetry.io/docs/zero-code/obi/configure/options

Selects processes to instrument by matching the full executable path using Glob.

```bash
OTEL_EBPF_AUTO_TARGET_EXE=/opt/app/server
```

--------------------------------

### Filter Network Metrics by Transport and Destination Port

Source: https://opentelemetry.io/docs/zero-code/obi/configure/filter-metrics-traces

Example of filtering network metrics to include only connections to destination port 53, excluding UDP protocol.

```yaml
filter:
  network:
    transport:
      not_match: UDP
    dst_port:
      match: '53'
```

--------------------------------

### Run Collector with sudo

Source: https://opentelemetry.io/docs/zero-code/obi/configure/collector-receiver

Run the OpenTelemetry Collector with the OBI receiver using sudo. OBI requires elevated privileges for eBPF instrumentation.

```bash
sudo ./otelcol-obi --config collector-config.yaml
```

--------------------------------

### Override Kubernetes Service Name and Namespace Labels

Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery

Override Kubernetes labels used for service name and namespace. This example specifies fallback names if primary labels are not found.

```yaml
attributes:
  kubernetes:
    resource_labels:
      service.name:
        # gets service name from the first existing Pod label
        - override-svc-name
        - app.kubernetes.io/name
      service.namespace:
        # gets service namespace from the first existing Pod label
        - override-svc-ns
        - app.kubernetes.io/part-of

```

--------------------------------

### Set Capabilities for Application Observability with Trace Context Propagation

Source: https://opentelemetry.io/docs/zero-code/obi/security

Sets a comprehensive set of Linux capabilities for OBI to enable advanced application observability, including trace context propagation. This requires capabilities such as `CAP_BPF`, `CAP_NET_ADMIN`, and `CAP_SYS_PTRACE`.

```bash
sudo setcap cap_bpf,cap_dac_read_search,cap_perfmon,cap_net_raw,cap_sys_ptrace,cap_net_admin+ep ./bin/obi
```

--------------------------------

### Configure CIDR Breakdown for Network Metrics

Source: https://opentelemetry.io/docs/zero-code/obi/network/quickstart

This YAML configuration defines CIDR ranges and their corresponding names to break down network metrics. It includes examples for cluster-internal, private, and container-internal networks.

```yaml
network:
  cidrs:
    - cidr: 10.0.0.0/8
      name: 'cluster-internal'
    - cidr: 192.168.0.0/16
      name: 'private'
    - cidr: 172.16.0.0/12
      name: 'container-internal'

```

--------------------------------

### Verify Service Discovery Logs

Source: https://opentelemetry.io/docs/zero-code/obi/configure/collector-receiver

Search collector logs for 'discovered service' to confirm OBI is finding applications.

```bash
grep "discovered service" collector.log
```

--------------------------------

### Node.js --heapsnapshot-signal Flag

Source: https://opentelemetry.io/docs/zero-code/obi/troubleshooting

Node.js flags like --heapsnapshot-signal can register their own signal handling, potentially interfering with OBI. This example shows the flag that uses SIGUSR1 for heap snapshots.

```bash
node --heapsnapshot-signal=SIGUSR1
```

--------------------------------

### Configure Instance ID Decoration

Source: https://opentelemetry.io/docs/zero-code/obi/configure/metrics-traces-attributes

Customize how the instance ID is composed for metrics and traces. Setting 'dns: false' uses the local hostname, while overriding it with 'OTEL_EBPF_HOSTNAME' takes precedence.

```yaml
attributes:
  instance_id:
    dns: false

```

--------------------------------

### Handle SIGUSR1 Signal in Node.js

Source: https://opentelemetry.io/docs/zero-code/obi/troubleshooting

Custom SIGUSR1 handlers in Node.js applications can conflict with OBI's signal handling, leading to crashes. This example shows a custom handler that exits the process.

```javascript
process.on('SIGUSR1', () => {
  process.exit(0);
});
```

--------------------------------

### Configure Network CIDRs via Environment Variable

Source: https://opentelemetry.io/docs/zero-code/obi/network/config

Set network CIDRs using the `OTEL_EBPF_NETWORK_CIDRS` environment variable as a comma-separated list of CIDR strings.

```shell
OTEL_EBPF_NETWORK_CIDRS=10.0.0.0/8,192.168.0.0/16
```

--------------------------------

### Basic Configuration for Trace Export and Log Enrichment

Source: https://opentelemetry.io/docs/zero-code/obi/trace-log-correlation

Configure trace export endpoint and select services for instrumentation and log enrichment. Ensure service selectors align between discovery and enrichment.

```yaml
otel_traces_export:
  endpoint: http://otel-collector:4318/v1/traces

discovery:
  instrument:
    - open_ports: '8380'

ebpf:
  log_enricher:
    services:
      - service:
          - open_ports: '8380'

```

--------------------------------

### Configure Allowed Attributes for Network Metric Aggregation

Source: https://opentelemetry.io/docs/zero-code/obi/network

Specify attributes to aggregate network metrics by, reducing the number of metrics sent to the backend. This example aggregates by Kubernetes owner name and type.

```yaml
network:
  allowed_attributes:
    - k8s.src.owner.name
    - k8s.dst.owner.name
    - k8s.src.owner.type
    - k8s.dst.owner.type
```

--------------------------------

### Instrument by Command-line Arguments

Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery

Selects processes by matching a glob pattern against their full command-line arguments. This can be combined with other selectors.

```yaml
discovery:
  instrument:
    - cmd_args: '*--profile=prod*'
```

--------------------------------

### Verify OBI capabilities

Source: https://opentelemetry.io/docs/zero-code/obi/configure/collector-receiver

Check if the capabilities have been successfully set on the OBI binary.

```bash
getcap ./otelcol-obi
```

--------------------------------

### Filter Application Metrics by Server Port (Numeric Filter)

Source: https://opentelemetry.io/docs/zero-code/obi/configure/filter-metrics-traces

Example using a numeric filter to include application spans where the server port is greater than or equal to 8000. Requires OBI v0.6.0 or later.

```yaml
filter:
  application:
    server.port:
      greater_equals: 8000
```

--------------------------------

### Configure Per-Application Metrics Export Features

Source: https://opentelemetry.io/docs/zero-code/obi/configure/export-data

Override global metrics export features for specific applications or ports. This example shows how to configure different metric groups for services based on open ports, name, or both.

```yaml
discovery:
  instrument:
    - open_ports: 3030,3040
      metrics:
        features:
          - 'application'
          - 'application_span'
          - 'application_service_graph'
    - name: pyserver
      open_ports: 7773
      metrics:
        features:
          - 'application'
    - name: apache
      open_ports: 8080
    - name: nginx
      open_ports: 8085
    - name: tomcat
      open_ports: 8090
```

--------------------------------

### Set Capabilities for Application Observability

Source: https://opentelemetry.io/docs/zero-code/obi/security

Sets Linux capabilities for OBI to enable application observability features. This includes `CAP_BPF`, `CAP_DAC_READ_SEARCH`, `CAP_CHECKPOINT_RESTORE`, `CAP_PERFMON`, `CAP_NET_RAW`, and `CAP_SYS_PTRACE`.

```bash
sudo setcap cap_bpf,cap_dac_read_search,cap_perfmon,cap_net_raw,cap_sys_ptrace+ep ./bin/obi
```
Tokens:
Raw