### Run Docker Compose Source: https://opentelemetry.io/docs/zero-code/obi/setup/docker Starts the services defined in the Docker Compose file. Use this to replicate the Docker CLI example setup. ```bash docker compose -f compose-example.yml up ``` -------------------------------- ### Install OBI Binaries to System PATH Source: https://opentelemetry.io/docs/zero-code/obi/setup/standalone Install the OBI binary to a directory included in your system's PATH, such as /usr/local/bin, to make it accessible from any directory. Verify the installation by checking the OBI version. ```bash # Move binaries to a directory in your PATH sudo cp obi /usr/local/bin/ # Verify installation obi --version ``` -------------------------------- ### Standalone OBI Configuration Example Source: https://opentelemetry.io/docs/zero-code/obi/configure/collector-receiver Example configuration for standalone OBI, showing trace export endpoint and open port. ```yaml # obi-config.yaml otel_traces_export: endpoint: http://backend:4318 open_port: 8080 ``` -------------------------------- ### Example YAML Configuration Snippet Source: https://opentelemetry.io/docs/zero-code/obi/configure/options Illustrates sample global configuration properties in YAML format for OBI. ```yaml trace_printer: json shutdown_timeout: 30s channel_buffer_len: 33 ``` -------------------------------- ### Example of URL Paths Source: https://opentelemetry.io/docs/zero-code/obi/configure/routes-decorator These are example URL paths that demonstrate a structure where the heuristic route decorator can be effective. ```text document/d/CfMkAGbE_aivhFydEpaRafPuGWbmHfG/edit (no numbers in the ID) document/d/C2fMkAGb3E_aivhFyd5EpaRafP123uGWbmHfG/edit ``` -------------------------------- ### Example OBI Log Output Source: https://opentelemetry.io/docs/zero-code/obi/cilium-compatibility Example output from OBI pod logs, demonstrating detected HTTP requests with trace-context propagation details. ```text 2025-01-17 21:42:18.11794218 (5.045099ms[5.045099ms]) HTTPClient 200 GET /tracemetoo [10.244.1.92 as go-service.default:37450]->[10.96.214.17 as python-service.default:8080] size:0B svc=[default/go-service go] traceparent=[00-14f07e11b5e57f14fd2da0541f0ddc2f-319fb03373427a41[cfa6d5d448e40b00]-01] 2025-01-17 21:42:18.11794218 (5.284521ms[5.164701ms]) HTTP 200 GET /gotracemetoo [10.244.2.144 as nodejs-service.default:57814]->[10.244.1.92 as go-service.default:8080] size:0B svc=[default/go-service go] traceparent=[00-14f07e11b5e57f14fd2da0541f0ddc2f-cfa6d5d448e40b00[cce1e6b5e932b89a]-01] 2025-01-17 21:42:18.11794218 (1.934744ms[1.934744ms]) HTTP 403 GET /users [10.244.2.32 as ruby-service.default:46876]->[10.244.2.176 as ruby-service.default:3000] size:222B svc=[default/ruby-service ruby] traceparent=[00-14f07e11b5e57f14fd2da0541f0ddc2f-57d77d99e9665c54[3d97d26b0051112b]-01] 2025-01-17 21:42:18.11794218 (2.116628ms[2.116628ms]) HTTPClient 403 GET /users [10.244.2.32 as ruby-service.default:46876]->[10.96.69.89 as ruby-service.default:3000] size:256B svc=[default/ruby-service ruby] traceparent=[00-14f07e11b5e57f14fd2da0541f0ddc2f-ff48ab147cc92f93[2770ac4619aa0042]-01] 2025-01-17 21:42:18.11794218 (4.281525ms[4.281525ms]) HTTP 200 GET /tracemetoo [10.244.1.92 as go-service.default:37450]->[10.244.2.32 as ruby-service.default:8080] size:178B svc=[default/ruby-service ruby] traceparent=[00-14f07e11b5e57f14fd2da0541f0ddc2f-2770ac4619aa0042[319fb03373427a41]-01] 2025-01-17 21:42:18.11794218 (5.391191ms[5.391191ms]) HTTPClient 200 GET /gotracemetoo [10.244.2.144 as nodejs-service.default:57814]->[10.96.134.167 as go-service.default:8080] size:256B svc=[default/nodejs-service nodejs] traceparent=[00-14f07e11b5e57f14fd2da0541f0ddc2f-202ee68205e4ef3b[9408610968fa20f8]-01] 2025-01-17 21:42:18.11794218 (6.939027ms[6.939027ms]) HTTP 200 GET /traceme [127.0.0.1 as 127.0.0.1:44720]->[127.0.0.1 as 127.0.0.1.default:3000] size:86B svc=[default/nodejs-service nodejs] traceparent=[00-14f07e11b5e57f14fd2da0541f0ddc2f-9408610968fa20f8[0000000000000000]-01] ``` -------------------------------- ### Example OBI Trace Output Source: https://opentelemetry.io/docs/zero-code/obi/setup/docker Sample output demonstrating OBI successfully capturing and printing trace requests to stdout. ```text time=2023-05-22T14:03:42.402Z level=INFO msg="creating instrumentation pipeline" time=2023-05-22T14:03:42.526Z level=INFO msg="Starting main node" 2023-05-22 14:03:53.5222353 (19.066625ms[942.583µs]) 200 GET / [172.17.0.1]->[localhost:18443] size:0B 2023-05-22 14:03:53.5222353 (355.792µs[321.75µs]) 200 GET /static/style.css [172.17.0.1]->[localhost:18443] size:0B 2023-05-22 14:03:53.5222353 (170.958µs[142.916µs]) 200 GET /static/img.png [172.17.0.1]->[localhost:18443] size:0B 2023-05-22 14:13:47.52221347 (7.243667ms[295.292µs]) 200 GET /entry/201710281345_instructions.md [172.17.0.1]->[localhost:18443] size:0B 2023-05-22 14:13:47.52221347 (115µs[75.625µs]) 200 GET /static/style.css [172.17.0.1]->[localhost:18443] size:0B ``` -------------------------------- ### Run Go Blog Service with Docker Source: https://opentelemetry.io/docs/zero-code/obi/setup/docker Starts a containerized Go blog service for testing OBI. Ensure the VERSION environment variable is set. ```bash export VERSION=v0.8.0 docker run -p 18443:8443 --name goblog mariomac/goblog:dev ``` -------------------------------- ### SQL++ Request Body Example Source: https://opentelemetry.io/docs/zero-code/obi/configure/export-data Example of a SQL++ query sent as a JSON body to the /query/service endpoint. ```json { "statement": "SELECT * FROM `bucket`.`scope`.`collection` WHERE id = $1", "query_context": "default:`bucket`.`scope`" } ``` -------------------------------- ### Discover Pods by Namespace and Labels Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery This example discovers all Pods in the `frontend` namespace that have a label `instrument` with a value matching the glob `obi`. ```yaml discovery: instrument: - k8s_namespace: frontend k8s_pod_labels: instrument: obi ``` -------------------------------- ### Install OBI Helm Chart Source: https://opentelemetry.io/docs/zero-code/obi/setup/kubernetes-helm Deploy a OBI DaemonSet with default configuration in a specified namespace. The `--create-namespace` flag ensures the namespace exists. ```bash helm install obi -n obi --create-namespace open-telemetry/opentelemetry-ebpf-instrumentation ``` -------------------------------- ### Discover Pods by Namespace and Annotations Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery This example discovers all Pods in the `backend` namespace that have an annotation `obi.instrument` with a value matching the glob `true`. ```yaml discovery: instrument: - k8s_namespace: backend k8s_pod_annotations: obi.instrument: 'true' ``` -------------------------------- ### OBI Configuration YAML Example Source: https://opentelemetry.io/docs/zero-code/obi/configure/example Use this YAML configuration to set up OBI. It includes options for service discovery, logging level, eBPF context propagation, and trace/metric export endpoints. ```yaml discovery: instrument: - open_ports: 8443 log_level: DEBUG ebpf: context_propagation: all otel_traces_export: endpoint: http://localhost:4318 prometheus_export: port: 8999 path: /metrics ``` -------------------------------- ### JSON Log Format Before OBI Enrichment Source: https://opentelemetry.io/docs/zero-code/obi/trace-log-correlation Example of a JSON log entry before trace context is injected by OBI. ```json { "level": "info", "message": "Request processed", "duration_ms": 125 } ``` -------------------------------- ### Cosign Verification Output Example Source: https://opentelemetry.io/docs/zero-code/obi/setup/docker An example of successful Cosign verification output, indicating that the claims were validated and the image digest is confirmed. ```json Verification for index.docker.io/otel/ebpf-instrument:main -- The following checks were performed on each of these signatures: - The cosign claims were validated - Existence of the claims in the transparency log was verified offline - The code-signing certificate was verified using trusted certificate authority certificates [{"critical":{"identity":{"docker-reference":"index.docker.io/otel/ebpf-instrument:main"},"image":{"docker-manifest-digest":"sha256:55426a2bbb8003573a961697888aa770a1f5f67fcda2276dc2187d1faf7181fe"},"type":"https://sigstore.dev/cosign/sign/v1"},"optional":{}}] ``` -------------------------------- ### Example Network Metric Output Source: https://opentelemetry.io/docs/zero-code/obi/network/quickstart This is an example of the debug output you might see in OBI logs, indicating a captured network flow with IP addresses and interface information. ```text network_flow: obi.ip=172.18.0.2 iface= direction=255 src.address=10.244.0.4 dst.address=10.96.0.1 ``` -------------------------------- ### Enable Prometheus Exporter with HTTP/SQL Instrumentation Source: https://opentelemetry.io/docs/zero-code/obi/configure/export-data Example of setting the `instrumentations` option to enable collection of HTTP/HTTPS/HTTP2 and SQL application metrics. ```yaml instrumentations: ['http', 'sql'] ``` -------------------------------- ### OBI Debug Trace Span Example Source: https://opentelemetry.io/docs/zero-code/obi/troubleshooting Example output of raw trace spans logged to the console when the debug traces exporter is enabled. This format is human-readable and matches OTel Collector's detailed verbosity. ```text Traces {"resource spans": 1, "spans": 1} ResourceSpans #0 Resource SchemaURL: Resource attributes: -> service.name: Str(flagd) -> telemetry.sdk.language: Str(go) -> telemetry.sdk.name: Str(opentelemetry) -> telemetry.distro.name: Str(opentelemetry-ebpf-instrumentation) -> telemetry.sdk.version: Str(main) -> host.name: Str(flagd-5cccb4c4f5-sfkcm) -> os.type: Str(linux) -> service.namespace: Str(opentelemetry-demo) -> k8s.owner.name: Str(flagd) -> k8s.kind: Str(Deployment) -> k8s.replicaset.name: Str(flagd-5cccb4c4f5) -> k8s.pod.name: Str(flagd-5cccb4c4f5-sfkcm) -> k8s.container.name: Str(flagd) -> k8s.deployment.name: Str(flagd) -> service.version: Str(2.0.2) -> k8s.namespace.name: Str(default) -> otel.library.name: Str(go.opentelemetry.io/obi) ScopeSpans #0 ScopeSpans SchemaURL: InstrumentationScope Span #0 Trace ID : 63a2723a58e0033170e58b1ff27ef03d Parent ID : ID : fab47609b60cc4e0 Name : /opentelemetry.proto.collector.metrics.v1.MetricsService/Export Kind : Client Start time : 2025-11-28 16:10:35.4241749 +0000 UTC End time : 2025-11-28 16:10:35.42555658 +0000 UTC Status code : Unset Status message : Attributes: -> rpc.method: Str(/opentelemetry.proto.collector.metrics.v1.MetricsService/Export) -> rpc.system: Str(grpc) -> rpc.grpc.status_code: Int(0) -> server.address: Str(otel-collector.default) -> peer.service: Str(otel-collector.default) -> server.port: Int(4317) ``` -------------------------------- ### Docker Compose File for OBI Setup Source: https://opentelemetry.io/docs/zero-code/obi/setup/docker Defines services for a Go blog application and the OBI autoinstrumenter using Docker Compose. Ensure the image tag for autoinstrumenter matches your desired version. ```yaml version: '3.8' services: # Service to instrument. Change it to any # other container that you want to instrument. goblog: image: mariomac/goblog:dev ports: # Exposes port 18843, forwarding it to container port 8443 - '18443:8443' autoinstrumenter: image: otel/ebpf-instrument:main pid: 'host' privileged: true environment: OTEL_EBPF_TRACE_PRINTER: text OTEL_EBPF_OPEN_PORT: 8443 ``` -------------------------------- ### Dockerfile for OBI Collector Source: https://opentelemetry.io/docs/zero-code/obi/configure/collector-receiver Create a Dockerfile to package the custom collector binary for deployment. This includes installing necessary tools and copying the collector binary. ```dockerfile FROM alpine:latest # Install required tools RUN apk --no-cache add ca-certificates # Copy the custom collector binary built with OCB COPY dist/otelcol-obi /otelcol-obi # Make it executable RUN chmod +x /otelcol-obi ENTRYPOINT ["/otelcol-obi"] ``` -------------------------------- ### Service Name Template Example Source: https://opentelemetry.io/docs/zero-code/obi/configure/metrics-traces-attributes Use Go templates to dynamically set service names based on Kubernetes metadata like namespace, labels, and container name. Ensure to trim whitespace. ```go {{- .Meta.Namespace }}/{{ index .Meta.Labels "app.kubernetes.io/name" }}/{{ index .Meta.Labels "app.kubernetes.io/component" -}}{{ if .ContainerName }}/{{ .ContainerName -}}{{ end -}} ``` ```go {{- .Meta.Namespace }}/{{ index .Meta.Labels "app.kubernetes.io/name" }}/{{ index .Meta.Labels "app.kubernetes.io/component" -}} ``` -------------------------------- ### Pessimistic Cardinality Calculation Source: https://opentelemetry.io/docs/zero-code/obi/cardinality This example shows a pessimistic calculation for cardinality, multiplying all potential factors. Use this as an upper bound when estimating cardinality. ```shell #Instances * #MetricNames * #HistoBuckets * #Operations * #Endpoints * #ReturnCodes = = 2 * 5 * 177/3 * 37/3 =2771 ``` -------------------------------- ### Service Name Example: Before Enhanced Lookup Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery Illustrates how service names might appear as IP addresses without enhanced lookup enabled. ```yaml service.name: "10.0.1.42" peer.service: "10.0.2.15" ``` -------------------------------- ### Prometheus Metric with CIDR Attributes Source: https://opentelemetry.io/docs/zero-code/obi/network/quickstart This example shows the equivalent Prometheus metric format for `obi_network_flow_bytes` when CIDR breakdown is configured, demonstrating aggregation by source and destination CIDR names. ```text obi_network_flow_bytes: src_cidr="cluster-internal" dst_cidr="private" ``` -------------------------------- ### Service Name Example: After Enhanced Lookup Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery Shows how service names are resolved to meaningful identifiers like 'frontend' and 'backend-api' with enhanced lookup enabled. ```yaml service.name: "frontend" peer.service: "backend-api" ``` -------------------------------- ### Prometheus Metric with Allowed Attributes Source: https://opentelemetry.io/docs/zero-code/obi/network/quickstart This example shows the equivalent Prometheus metric format for `obi.network.flow.bytes` when specific Kubernetes owner attributes are allowed, demonstrating aggregation by source and destination owner type. ```text obi.network.flow.bytes: k8s_src_owner_name="frontend" k8s_src_owner_type="deployment" k8s_dst_owner_name="backend" k8s_dst_owner_type="deployment" ``` -------------------------------- ### Configure OTLP Export Endpoint Source: https://opentelemetry.io/docs/zero-code/obi/configure/export-modes Specify where traces and metrics are sent. This example configures OBI to export to a local OTel Collector running on port 4318. ```yaml otel_metrics_export: endpoint: http://localhost:4318 otel_traces_export: endpoint: http://localhost:4318 ``` -------------------------------- ### Custom OBI Configuration with Helm Source: https://opentelemetry.io/docs/zero-code/obi/setup/kubernetes-helm Override default OBI configurations by creating a YAML file and passing it to the Helm install or upgrade command using the -f flag. ```yaml config: data: # Contents of the actual OBI configuration file discovery: instrument: - k8s_namespace: demo - k8s_namespace: blog routes: unmatched: heuristic ``` ```bash helm install obi open-telemetry/opentelemetry-ebpf-instrumentation -f helm-obi.yml ``` ```bash helm upgrade obi open-telemetry/opentelemetry-ebpf-instrumentation -f helm-obi.yml ``` -------------------------------- ### OpenTelemetry Collector Configuration with OBI Receiver Source: https://opentelemetry.io/docs/zero-code/obi/configure/collector-receiver Example configuration for the OpenTelemetry Collector to enable the OBI receiver. This snippet is intended to be part of a larger collector configuration file. ```yaml ``` -------------------------------- ### Instrument Executable with Open Ports Source: https://opentelemetry.io/docs/zero-code/obi/configure/export-modes Specify the executable to instrument by adding the `open_ports` property to the YAML document. This example instruments a service running on port 443. ```yaml discovery: instrument: - open_ports: 443 ``` -------------------------------- ### SQL++ Form Encoded Request Example Source: https://opentelemetry.io/docs/zero-code/obi/configure/export-data Example of a SQL++ query sent as form-encoded data to the /query/service endpoint. ```urlencoded statement=SELECT+*+FROM+users&query_context=default:`travel-sample`.`inventory` ``` -------------------------------- ### Exclude Specific Namespaces from Instrumentation Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery This example demonstrates how to instrument all namespaces except for `development` and `staging`. ```yaml discovery: instrument: - k8s_namespace: '*' # Instrument all namespaces exclude_instrument: - k8s_namespace: development # Except development namespace - k8s_namespace: staging # And staging namespace ``` -------------------------------- ### Run OBI with Configuration File Source: https://opentelemetry.io/docs/zero-code/obi/configure/export-modes Execute OBI with a specified configuration file using the `-config` flag or by setting the `OTEL_EBPF_CONFIG_PATH` environment variable. ```bash obi -config instrument-config.yml ``` ```bash OTEL_EBPF_CONFIG_PATH=instrument-config.yml obi ``` -------------------------------- ### JSON Log Format After OBI Enrichment Source: https://opentelemetry.io/docs/zero-code/obi/trace-log-correlation Example of a JSON log entry after OBI has injected 'trace_id' and 'span_id' fields. ```json { "level": "info", "message": "Request processed", "duration_ms": 125, "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7" } ``` -------------------------------- ### Log Merged Configuration in OBI Source: https://opentelemetry.io/docs/zero-code/obi/troubleshooting Instruct OBI to log its final merged configuration at startup using 'yaml' for readability or 'json' for log shippers. ```bash # Log configuration in YAML format export OTEL_EBPF_LOG_CONFIG=yaml # Log configuration in JSON format export OTEL_EBPF_LOG_CONFIG=json ``` -------------------------------- ### Prometheus Metric with CIDR-based Labels Source: https://opentelemetry.io/docs/zero-code/obi/network An example of a Prometheus metric labeled with source and destination CIDR names, as configured in OBI. ```prometheus obi_network_flow_bytes: src_cidr="cluster-internal" dst_cidr="private" ``` -------------------------------- ### Build and Push Docker Image Source: https://opentelemetry.io/docs/zero-code/obi/configure/collector-receiver Build the Docker image for the OBI collector and push it to a container registry. Replace 'my-registry/otelcol-obi:v0.6.0' with your registry and tag. ```bash docker build -t my-registry/otelcol-obi:v0.6.0 . docker push my-registry/otelcol-obi:v0.6.0 ``` -------------------------------- ### Heuristic Route Decorator Output Source: https://opentelemetry.io/docs/zero-code/obi/configure/routes-decorator This shows the reduced, low-cardinality route achieved by the heuristic decorator for the preceding example URLs. ```text document/d/*/edit ``` -------------------------------- ### Run OBI for Application Observability with Trace Context Propagation Source: https://opentelemetry.io/docs/zero-code/obi/security Runs OBI with environment variables to enable application observability and trace context propagation. This requires a broad set of capabilities, including `CAP_BPF`, `CAP_NET_ADMIN`, and `CAP_SYS_PTRACE`. ```bash OTEL_EBPF_CONTEXT_PROPAGATION=all OTEL_EBPF_OPEN_PORT=8080 OTEL_EBPF_TRACE_PRINTER=text bin/obi ``` -------------------------------- ### Run OBI with sudo Source: https://opentelemetry.io/docs/zero-code/obi/setup/standalone Use this command to run OBI with elevated privileges, specifying the path to your configuration file. ```bash sudo obi --config= ``` -------------------------------- ### Exclude Specific Executables from Instrumentation Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery This example excludes Prometheus and Grafana executables from instrumentation when they are running on ports 80, 443, or 8080. ```yaml discovery: rules: - match: open_ports: 80,443,8080 exclude: exe_path: '*prometheus*' exe_path: '*grafana*' ``` -------------------------------- ### Run OBI for Network Metrics via Socket Filter Source: https://opentelemetry.io/docs/zero-code/obi/security Execute OBI with specific environment variables to enable network metrics collection via socket filters. Requires `CAP_BPF` and `CAP_NET_RAW` capabilities to be set on the binary. ```bash OTEL_EBPF_NETWORK_METRICS=1 OTEL_EBPF_NETWORK_PRINT_FLOWS=1 bin/obi ``` -------------------------------- ### Configure OBI Internal Metrics Exporter Source: https://opentelemetry.io/docs/zero-code/obi/troubleshooting Enable OBI internal metrics by configuring the exporter. Options include 'none' (default), 'prometheus' for an HTTP server, or 'otlp' for OTLP export. ```bash # Export internal metrics in Prometheus format export OTEL_EBPF_INTERNAL_METRICS_EXPORTER=prometheus # Export internal metrics via OTLP export OTEL_EBPF_INTERNAL_METRICS_EXPORTER=otlp ``` -------------------------------- ### Deploy OBI as a DaemonSet Source: https://opentelemetry.io/docs/zero-code/obi/setup/kubernetes Configure the DaemonSet with `hostPID: true` to access host processes. Set `OTEL_EBPF_AUTO_TARGET_EXE` to select executables by name, useful when ports are internal or dynamic. Enable Kubernetes metadata decoration with `OTEL_EBPF_KUBE_METADATA_ENABLE`. ```yaml --- apiVersion: apps/v1 kind: DaemonSet metadata: name: obi labels: app: obi spec: selector: matchLabels: app: obi template: metadata: labels: app: obi spec: hostPID: true # Required to access the processes on the host serviceAccountName: obi # required if you want kubernetes metadata decoration containers: - name: autoinstrument image: otel/ebpf-instrument:main securityContext: privileged: true env: # Select the executable by its name instead of OTEL_EBPF_OPEN_PORT - name: OTEL_EBPF_AUTO_TARGET_EXE value: '*/goblog' - name: OTEL_EXPORTER_OTLP_ENDPOINT value: 'http://otelcol:4318' # required if you want kubernetes metadata decoration - name: OTEL_EBPF_KUBE_METADATA_ENABLE value: 'true' ``` -------------------------------- ### Download and Verify OBI Binaries Source: https://opentelemetry.io/docs/zero-code/obi/setup/standalone Download the OBI archive and checksums for your architecture, then verify the integrity of the downloaded archive using sha256sum. Ensure the VERSION and ARCH variables are set correctly. ```bash # Set your desired version (find latest at # https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/releases) VERSION=0.6.0 # Determine your architecture # For Intel/AMD 64-bit: amd64 # For ARM 64-bit: arm64 ARCH=amd64 # Download the archive for your architecture wget https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/releases/download/v${VERSION}/obi-v${VERSION}-linux-${ARCH}.tar.gz # Download checksums wget https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/releases/download/v${VERSION}/SHA256SUMS # Verify the archive sha256sum -c SHA256SUMS --ignore-missing # Extract the archive tar -xzf obi-v${VERSION}-linux-${ARCH}.tar.gz ``` -------------------------------- ### Prometheus Metric with Kubernetes Owner Attributes Source: https://opentelemetry.io/docs/zero-code/obi/network An example of how a Prometheus metric might appear after aggregation by Kubernetes owner attributes, as configured in OBI. ```prometheus obi_network_flow_bytes: k8s_src_owner_name="frontend" k8s_src_owner_type="deployment" k8s_dst_owner_name="backend" k8s_dst_owner_type="deployment" ``` -------------------------------- ### Create OBI Namespace Source: https://opentelemetry.io/docs/zero-code/obi/cilium-compatibility Creates the Kubernetes namespace where OBI components will be deployed. ```bash kubectl create namespace obi ``` -------------------------------- ### Enable Network Metrics in OBI Source: https://opentelemetry.io/docs/zero-code/obi/network/quickstart Set this environment variable to true to enable the collection of network metrics. ```bash export OTEL_EBPF_NETWORK_METRICS=true ``` -------------------------------- ### Run OBI from extracted directory Source: https://opentelemetry.io/docs/zero-code/obi/setup/standalone Execute this command if OBI is not installed in your system's PATH, running it directly from the extracted directory with elevated privileges. ```bash sudo ./obi --config= ``` -------------------------------- ### Run OBI for Application Observability Source: https://opentelemetry.io/docs/zero-code/obi/security Executes OBI with environment variables to enable application observability, such as tracing and port monitoring. Requires specific capabilities like `CAP_BPF` and `CAP_SYS_PTRACE`. ```bash OTEL_EBPF_OPEN_PORT=8080 OTEL_EBPF_TRACE_PRINTER=text bin/obi ``` -------------------------------- ### Enable SQL++ Instrumentation Source: https://opentelemetry.io/docs/zero-code/obi/configure/export-data Enable SQL++ instrumentation and configure the BPF buffer size for capturing request/response bodies. A larger buffer is needed for full body capture. ```bash export OTEL_EBPF_HTTP_SQLPP_ENABLED=true export OTEL_EBPF_BPF_BUFFER_SIZE_HTTP=2048 # Larger than default; needed to capture request/response bodies ``` -------------------------------- ### OBI Unprivileged DaemonSet Configuration Source: https://opentelemetry.io/docs/zero-code/obi/setup/kubernetes This configuration demonstrates how to deploy OBI as a DaemonSet with reduced privileges. It replaces `privileged:true` with specific Linux capabilities required for OBI's functionality, such as `BPF`, `SYS_PTRACE`, `NET_RAW`, `CHECKPOINT_RESTORE`, `DAC_READ_SEARCH`, and `PERFMON`. Ensure `hostPID: true` is set for process discovery. ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: obi namespace: obi-demo labels: k8s-app: obi spec: selector: matchLabels: k8s-app: obi template: metadata: labels: k8s-app: obi spec: serviceAccount: obi hostPID: true # <-- Important. Required in Daemonset mode so OBI can discover all monitored processes containers: - name: obi terminationMessagePolicy: FallbackToLogsOnError image: otel/ebpf-instrument:main env: - name: OTEL_EBPF_TRACE_PRINTER value: "text" - name: OTEL_EBPF_KUBE_METADATA_ENABLE value: "autodetect" - name: KUBE_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ... securityContext: runAsUser: 0 readOnlyRootFilesystem: true capabilities: add: - BPF # <-- Important. Required for most eBPF probes to function correctly. - SYS_PTRACE # <-- Important. Allows OBI to access the container namespaces and inspect executables. - NET_RAW # <-- Important. Allows OBI to use socket filters for http requests. - CHECKPOINT_RESTORE # <-- Important. Allows OBI to open ELF files. - DAC_READ_SEARCH # <-- Important. Allows OBI to open ELF files. - PERFMON # <-- Important. Allows OBI to load BPF programs. #- SYS_RESOURCE # <-- pre 5.11 only. Allows OBI to increase the amount of locked memory. #- SYS_ADMIN # <-- Required for Go application trace context propagation, or if kernel.perf_event_paranoid >= 3 on Debian distributions. drop: - ALL volumeMounts: - name: var-run-obi mountPath: /var/run/obi - name: cgroup mountPath: /sys/fs/cgroup tolerations: - effect: NoSchedule operator: Exists - effect: NoExecute operator: Exists volumes: - name: var-run-obi emptyDir: {} - name: cgroup hostPath: path: /sys/fs/cgroup --- apiVersion: apps/v1 kind: Deployment metadata: name: some-service namespace: obi-demo ... --- ``` -------------------------------- ### Combine Numeric and String Filters for Network Metrics Source: https://opentelemetry.io/docs/zero-code/obi/configure/filter-metrics-traces Example combining a string matcher (`not_match`) with a numeric matcher (`less_than`) to filter network metrics. ```yaml filter: network: transport: not_match: UDP dst_port: less_than: 1024 ``` -------------------------------- ### Inspect OBI SBOMs with jq and Grype Source: https://opentelemetry.io/docs/zero-code/obi/setup/standalone Use `jq` to parse and display component information from the SBOMs, or use `grype` to scan the SBOM for vulnerabilities. These commands help in reviewing dependencies and licenses without executing the binaries. ```bash # List component names and versions from the archive SBOM jq '.components[] | {name, version}' obi-v${VERSION}-linux-${ARCH}.cyclonedx.json # Scan the SBOM with Grype grype sbom:obi-v${VERSION}-linux-${ARCH}.cyclonedx.json # Inspect the Java agent dependency graph jq '.components[] | {name, version}' obi-java-agent-v${VERSION}.cyclonedx.json ``` -------------------------------- ### Enable Instrumentation for Default-Excluded Namespace Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery To instrument a namespace excluded by default, remove it from `default_exclude_instrument` and add it to the `instrument` section. This example enables instrumentation for the 'monitoring' namespace. ```yaml discovery: # Include the monitoring namespace in instrumentation instrument: - k8s_namespace: monitoring # Override default exclusions to remove monitoring namespace # This list keeps other default exclusions but removes monitoring default_exclude_instrument: - exe_path: '{*/obi,obi,*otelcol,*otelcol-contrib,*otelcol-contrib[!/]*}' - k8s_namespace: kube-system - k8s_namespace: kube-node-lease - k8s_namespace: local-path-storage - k8s_namespace: cert-manager # monitoring namespace removed from this list - k8s_namespace: gke-connect - k8s_namespace: gke-gmp-system - k8s_namespace: gke-managed-cim - k8s_namespace: gke-managed-filestorecsi - k8s_namespace: gke-managed-metrics-server - k8s_namespace: gke-managed-system - k8s_namespace: gke-system - k8s_namespace: gke-managed-volumepopulator - k8s_namespace: gatekeeper-system ``` -------------------------------- ### Exclude Services by Pod Labels Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery This example excludes services in the `production` namespace that have the label `skip-instrumentation` set to `true`. You can use any custom label key and value. ```yaml discovery: rules: - match: k8s_namespace: production exclude: k8s_pod_labels: skip-instrumentation: 'true' ``` -------------------------------- ### Enable Kubernetes Metadata Decoration Source: https://opentelemetry.io/docs/zero-code/obi/network/quickstart Set this environment variable to true to decorate network metrics with Kubernetes metadata, which is required for network metrics. ```bash export OTEL_EBPF_KUBE_METADATA_ENABLE=true ``` -------------------------------- ### Target Processes by Executable Path Source: https://opentelemetry.io/docs/zero-code/obi/configure/options Selects processes to instrument by matching the full executable path using Glob. ```bash OTEL_EBPF_AUTO_TARGET_EXE=/opt/app/server ``` -------------------------------- ### Filter Network Metrics by Transport and Destination Port Source: https://opentelemetry.io/docs/zero-code/obi/configure/filter-metrics-traces Example of filtering network metrics to include only connections to destination port 53, excluding UDP protocol. ```yaml filter: network: transport: not_match: UDP dst_port: match: '53' ``` -------------------------------- ### Run Collector with sudo Source: https://opentelemetry.io/docs/zero-code/obi/configure/collector-receiver Run the OpenTelemetry Collector with the OBI receiver using sudo. OBI requires elevated privileges for eBPF instrumentation. ```bash sudo ./otelcol-obi --config collector-config.yaml ``` -------------------------------- ### Override Kubernetes Service Name and Namespace Labels Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery Override Kubernetes labels used for service name and namespace. This example specifies fallback names if primary labels are not found. ```yaml attributes: kubernetes: resource_labels: service.name: # gets service name from the first existing Pod label - override-svc-name - app.kubernetes.io/name service.namespace: # gets service namespace from the first existing Pod label - override-svc-ns - app.kubernetes.io/part-of ``` -------------------------------- ### Set Capabilities for Application Observability with Trace Context Propagation Source: https://opentelemetry.io/docs/zero-code/obi/security Sets a comprehensive set of Linux capabilities for OBI to enable advanced application observability, including trace context propagation. This requires capabilities such as `CAP_BPF`, `CAP_NET_ADMIN`, and `CAP_SYS_PTRACE`. ```bash sudo setcap cap_bpf,cap_dac_read_search,cap_perfmon,cap_net_raw,cap_sys_ptrace,cap_net_admin+ep ./bin/obi ``` -------------------------------- ### Configure CIDR Breakdown for Network Metrics Source: https://opentelemetry.io/docs/zero-code/obi/network/quickstart This YAML configuration defines CIDR ranges and their corresponding names to break down network metrics. It includes examples for cluster-internal, private, and container-internal networks. ```yaml network: cidrs: - cidr: 10.0.0.0/8 name: 'cluster-internal' - cidr: 192.168.0.0/16 name: 'private' - cidr: 172.16.0.0/12 name: 'container-internal' ``` -------------------------------- ### Verify Service Discovery Logs Source: https://opentelemetry.io/docs/zero-code/obi/configure/collector-receiver Search collector logs for 'discovered service' to confirm OBI is finding applications. ```bash grep "discovered service" collector.log ``` -------------------------------- ### Node.js --heapsnapshot-signal Flag Source: https://opentelemetry.io/docs/zero-code/obi/troubleshooting Node.js flags like --heapsnapshot-signal can register their own signal handling, potentially interfering with OBI. This example shows the flag that uses SIGUSR1 for heap snapshots. ```bash node --heapsnapshot-signal=SIGUSR1 ``` -------------------------------- ### Configure Instance ID Decoration Source: https://opentelemetry.io/docs/zero-code/obi/configure/metrics-traces-attributes Customize how the instance ID is composed for metrics and traces. Setting 'dns: false' uses the local hostname, while overriding it with 'OTEL_EBPF_HOSTNAME' takes precedence. ```yaml attributes: instance_id: dns: false ``` -------------------------------- ### Handle SIGUSR1 Signal in Node.js Source: https://opentelemetry.io/docs/zero-code/obi/troubleshooting Custom SIGUSR1 handlers in Node.js applications can conflict with OBI's signal handling, leading to crashes. This example shows a custom handler that exits the process. ```javascript process.on('SIGUSR1', () => { process.exit(0); }); ``` -------------------------------- ### Configure Network CIDRs via Environment Variable Source: https://opentelemetry.io/docs/zero-code/obi/network/config Set network CIDRs using the `OTEL_EBPF_NETWORK_CIDRS` environment variable as a comma-separated list of CIDR strings. ```shell OTEL_EBPF_NETWORK_CIDRS=10.0.0.0/8,192.168.0.0/16 ``` -------------------------------- ### Basic Configuration for Trace Export and Log Enrichment Source: https://opentelemetry.io/docs/zero-code/obi/trace-log-correlation Configure trace export endpoint and select services for instrumentation and log enrichment. Ensure service selectors align between discovery and enrichment. ```yaml otel_traces_export: endpoint: http://otel-collector:4318/v1/traces discovery: instrument: - open_ports: '8380' ebpf: log_enricher: services: - service: - open_ports: '8380' ``` -------------------------------- ### Configure Allowed Attributes for Network Metric Aggregation Source: https://opentelemetry.io/docs/zero-code/obi/network Specify attributes to aggregate network metrics by, reducing the number of metrics sent to the backend. This example aggregates by Kubernetes owner name and type. ```yaml network: allowed_attributes: - k8s.src.owner.name - k8s.dst.owner.name - k8s.src.owner.type - k8s.dst.owner.type ``` -------------------------------- ### Instrument by Command-line Arguments Source: https://opentelemetry.io/docs/zero-code/obi/configure/service-discovery Selects processes by matching a glob pattern against their full command-line arguments. This can be combined with other selectors. ```yaml discovery: instrument: - cmd_args: '*--profile=prod*' ``` -------------------------------- ### Verify OBI capabilities Source: https://opentelemetry.io/docs/zero-code/obi/configure/collector-receiver Check if the capabilities have been successfully set on the OBI binary. ```bash getcap ./otelcol-obi ``` -------------------------------- ### Filter Application Metrics by Server Port (Numeric Filter) Source: https://opentelemetry.io/docs/zero-code/obi/configure/filter-metrics-traces Example using a numeric filter to include application spans where the server port is greater than or equal to 8000. Requires OBI v0.6.0 or later. ```yaml filter: application: server.port: greater_equals: 8000 ``` -------------------------------- ### Configure Per-Application Metrics Export Features Source: https://opentelemetry.io/docs/zero-code/obi/configure/export-data Override global metrics export features for specific applications or ports. This example shows how to configure different metric groups for services based on open ports, name, or both. ```yaml discovery: instrument: - open_ports: 3030,3040 metrics: features: - 'application' - 'application_span' - 'application_service_graph' - name: pyserver open_ports: 7773 metrics: features: - 'application' - name: apache open_ports: 8080 - name: nginx open_ports: 8085 - name: tomcat open_ports: 8090 ``` -------------------------------- ### Set Capabilities for Application Observability Source: https://opentelemetry.io/docs/zero-code/obi/security Sets Linux capabilities for OBI to enable application observability features. This includes `CAP_BPF`, `CAP_DAC_READ_SEARCH`, `CAP_CHECKPOINT_RESTORE`, `CAP_PERFMON`, `CAP_NET_RAW`, and `CAP_SYS_PTRACE`. ```bash sudo setcap cap_bpf,cap_dac_read_search,cap_perfmon,cap_net_raw,cap_sys_ptrace+ep ./bin/obi ```