OpenSearch Observability

https://observability.opensearch.org/docs/
Admin
OpenSearch Observability is a comprehensive monitoring and analytics platform providing AI agent...
Tokens:165,641
Snippets:2,170
Trust Score:10
Update:1 month ago
Show doc for...
Tokens:
Raw
### Get All Logs Example

Source: https://observability.opensearch.org/docs/ppl/commands

A basic example of the `search` command to retrieve all logs from a specified index pattern.

```ppl
source = logs-otel-v1*
```

--------------------------------

### Install via Docker Compose

Source: https://observability.opensearch.org/docs/get-started/installation

Clones the repository and starts the stack using Docker Compose.

```bash
git clone https://github.com/opensearch-project/observability-stack.git
cd observability-stack
docker compose up -d
```

--------------------------------

### Install Gin Instrumentation

Source: https://observability.opensearch.org/docs/send-data/applications/go

Install the OpenTelemetry Gin instrumentation package using 'go get'.

```bash
go get go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin

```

--------------------------------

### Install gRPC Instrumentation

Source: https://observability.opensearch.org/docs/send-data/applications/go

Use 'go get' to install the OpenTelemetry gRPC instrumentation package.

```bash
go get go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc

```

--------------------------------

### Install the SDK

Source: https://observability.opensearch.org/docs/ai-observability/getting-started

Install the core SDK and optional provider-specific dependencies for auto-instrumentation.

```bash
pip install opensearch-genai-observability-sdk-py
```

```bash
pip install "opensearch-genai-observability-sdk-py[openai]"   # or [anthropic], [bedrock], etc.
```

--------------------------------

### Install via Interactive Script

Source: https://observability.opensearch.org/docs/get-started/installation

Executes the installation script to launch the TUI for component selection and configuration.

```bash
curl -fsSL https://raw.githubusercontent.com/opensearch-project/observability-stack/main/install.sh | bash
```

--------------------------------

### Install OpenSearch SDK

Source: https://observability.opensearch.org/docs/agent-evals/evaluation

Install the required SDK package with the opensearch extra.

```bash
pip install "opensearch-genai-observability-sdk-py[opensearch]"
```

--------------------------------

### Install Base SDK

Source: https://observability.opensearch.org/docs/send-data/ai-agents/python

Install the core Python SDK for OpenSearch GenAI Observability.

```bash
pip install opensearch-genai-observability-sdk-py
```

--------------------------------

### Install and start Agent Health

Source: https://observability.opensearch.org/docs/agent-health/getting-started

Run the application using npx for immediate execution or install it globally for frequent use.

```bash
npx @opensearch-project/agent-health@latest
```

```bash
npm install -g @opensearch-project/agent-health
agent-health
```

--------------------------------

### Install Python Auto-Instrumentation

Source: https://observability.opensearch.org/docs/send-data/opentelemetry/auto-instrumentation

Install the necessary distribution and exporter packages, then use the bootstrap tool to detect and install instrumentation for existing libraries.

```bash
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
```

--------------------------------

### Minimal OpenTelemetry Gem Installation

Source: https://observability.opensearch.org/docs/send-data/applications/ruby

For a minimal setup, install only the required OpenTelemetry SDK, OTLP exporter, and specific instrumentation gems needed for your application.

```ruby
gem "opentelemetry-sdk"
gem "opentelemetry-exporter-otlp"
gem "opentelemetry-instrumentation-rack"
gem "opentelemetry-instrumentation-rails"
```

--------------------------------

### Call SDK Setup in main()

Source: https://observability.opensearch.org/docs/send-data/applications/go

Demonstrates how to call the telemetry setup function and defer the shutdown process in the main function of a Go application.

```go
func main() {
  ctx := context.Background()
  shutdown, err := telemetry.Setup(ctx, "my-go-service")
  if err != nil {
    log.Fatal(err)
  }
  defer shutdown()

  // ... start application
}
```

--------------------------------

### Install SDK with Provider Support

Source: https://observability.opensearch.org/docs/send-data/ai-agents/python

Install the Python SDK with optional support for specific LLM providers or OpenTelemetry instrumentors.

```bash
pip install "opensearch-genai-observability-sdk-py[openai]"
```

```bash
pip install "opensearch-genai-observability-sdk-py[anthropic]"
```

```bash
pip install "opensearch-genai-observability-sdk-py[bedrock]"
```

```bash
pip install "opensearch-genai-observability-sdk-py[google]"
```

```bash
pip install "opensearch-genai-observability-sdk-py[langchain]"
```

```bash
pip install "opensearch-genai-observability-sdk-py[llamaindex]"
```

```bash
pip install "opensearch-genai-observability-sdk-py[otel-instrumentors]"
```

```bash
pip install "opensearch-genai-observability-sdk-py[opensearch]"
```

```bash
pip install "opensearch-genai-observability-sdk-py[all]"
```

--------------------------------

### Install Auto-instrumentation for Node.js

Source: https://observability.opensearch.org/docs/send-data/applications/nodejs

Install the package for automatic instrumentation of common Node.js libraries. This simplifies the setup process.

```bash
npm install @opentelemetry/auto-instrumentations-node
```

--------------------------------

### Table Command Example

Source: https://observability.opensearch.org/docs/ppl/commands

Example usage of the `table` command to specify fields to display.

```ppl
| table time body severityText
| head 20
```

--------------------------------

### Query observability data with natural language

Source: https://observability.opensearch.org/docs/claude-code

Examples of natural language queries to ask the observability stack after installation.

```text
Show me the slowest traces from the last hour
```

```text
What's the error rate across services?
```

```text
Check if the observability stack is healthy
```

```text
Calculate the error budget for a 99.9% availability SLO
```

--------------------------------

### Install Go Auto-Instrumentation Agent

Source: https://observability.opensearch.org/docs/send-data/opentelemetry/auto-instrumentation

Install the Go auto-instrumentation agent using `go install`. This agent uses eBPF to instrument applications at the kernel level without modifying the binary.

```go
go install go.opentelemetry.io/auto/cmd/instrumentation@latest
```

--------------------------------

### Install Logstash with OpenSearch Output Plugin

Source: https://observability.opensearch.org/docs/send-data/infrastructure/logstash

Instructions for downloading and installing Logstash with the OpenSearch output plugin, or installing the plugin into an existing Logstash instance.

```APIDOC
## Install Logstash

Download and install Logstash:
```bash
wget https://artifacts.opensearch.org/logstash/logstash-oss-with-opensearch-output-plugin-8.9.0-linux-x64.tar.gz
tar -xzf logstash-oss-with-opensearch-output-plugin-8.9.0-linux-x64.tar.gz
cd logstash-8.9.0
```

Or install the plugin into an existing Logstash installation:
```bash
bin/logstash-plugin install logstash-output-opensearch
```
```

--------------------------------

### Install Observability Plugin

Source: https://observability.opensearch.org/docs/claude-code

Installs the observability plugin into the Claude Code environment.

```bash
/plugin install observability@observability
```

--------------------------------

### Install net/http OTel Middleware

Source: https://observability.opensearch.org/docs/send-data/applications/go

Installs the OpenTelemetry instrumentation middleware for Go's net/http package.

```bash
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
```

--------------------------------

### Install Core SDK and OTLP Exporter for Node.js

Source: https://observability.opensearch.org/docs/send-data/applications/nodejs

Install the necessary OpenTelemetry SDK components and OTLP exporters for gRPC. Ensure Node.js 16+ is used.

```bash
npm install @opentelemetry/sdk-node \
  @opentelemetry/api \
  @opentelemetry/exporter-trace-otlp-grpc \
  @opentelemetry/exporter-metrics-otlp-grpc \
  @opentelemetry/exporter-logs-otlp-grpc \
  @opentelemetry/resources \
  @opentelemetry/semantic-conventions
```

--------------------------------

### Install OpenTelemetry Dependencies

Source: https://observability.opensearch.org/docs/send-data/applications/ruby

Run `bundle install` after updating your Gemfile to install the necessary OpenTelemetry gems.

```bash
bundle install
```

--------------------------------

### Install LangGraph with LangChain SDK

Source: https://observability.opensearch.org/docs/send-data/ai-agents/integrations

Install the necessary packages for LangGraph and the GenAI Observability SDK with LangChain auto-instrumentation.

```bash
pip install "opensearch-genai-observability-sdk-py[langchain]" langgraph langchain-openai
```

--------------------------------

### Create and Use Shared Docker Network

Source: https://observability.opensearch.org/docs/send-data/infrastructure/docker

If your application runs in a separate Compose file, create an external Docker network named 'observability' and connect both projects to it. This example shows the commands to create the network and start the stacks.

```bash
# Create a shared network
docker network create observability

# Start the Observability Stack
docker compose -f observability-stack/docker-compose.yml up -d

# Start your application
docker compose -f my-app/docker-compose.yml up -d

```

--------------------------------

### Download and Install Logstash

Source: https://observability.opensearch.org/docs/send-data/infrastructure/logstash

Use these commands to download and install Logstash with the OpenSearch output plugin. Ensure Java 11 or later is installed.

```bash
wget https://artifacts.opensearch.org/logstash/logstash-oss-with-opensearch-output-plugin-8.9.0-linux-x64.tar.gz
tar -xzf logstash-oss-with-opensearch-output-plugin-8.9.0-linux-x64.tar.gz
cd logstash-8.9.0
```

--------------------------------

### Install Amazon Bedrock with OpenSearch Observability

Source: https://observability.opensearch.org/docs/send-data/ai-agents/integrations

Install the OpenSearch GenAI Observability SDK with the 'bedrock' extra for auto-instrumentation of direct Bedrock calls.

```bash
pip install "opensearch-genai-observability-sdk-py[bedrock]"
```

--------------------------------

### Install RAGAS and Observability SDK

Source: https://observability.opensearch.org/docs/agent-evals/evaluation-integrations

Required packages for running RAGAS evaluations and logging them to OpenSearch.

```bash
pip install opensearch-genai-observability-sdk-py ragas datasets
```

--------------------------------

### Install via Helm

Source: https://observability.opensearch.org/docs/get-started/installation

Deploys the stack to a Kubernetes cluster using Helm.

```bash
git clone https://github.com/opensearch-project/observability-stack.git
cd observability-stack
helm install obs charts/observability-stack
```

--------------------------------

### Install OpenSearch Fluentd Plugin

Source: https://observability.opensearch.org/docs/send-data/infrastructure/fluentd

Install the necessary Fluentd plugin to enable output to OpenSearch. This command should be run within the Fluentd environment or container.

```bash
fluent-gem install fluent-plugin-opensearch
```

--------------------------------

### Start observability stack

Source: https://observability.opensearch.org/docs/claude-code

Command to launch the required OpenSearch and Prometheus services locally.

```bash
docker compose up -d opensearch prometheus
```

--------------------------------

### Install MLflow Integration Dependencies

Source: https://observability.opensearch.org/docs/agent-evals/evaluation-integrations

Install the required SDK and MLflow packages.

```bash
pip install opensearch-genai-observability-sdk-py mlflow
```

--------------------------------

### Install .NET Auto-Instrumentation Script

Source: https://observability.opensearch.org/docs/send-data/opentelemetry/auto-instrumentation

Download and execute the .NET auto-instrumentation script for Linux/macOS. This script installs the necessary agent for automatic instrumentation.

```bash
curl -L -o otel-dotnet.sh \
  https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/releases/latest/download/otel-dotnet-auto-install.sh
sh otel-dotnet.sh
```

--------------------------------

### Install OpenAI Agents SDK with OpenSearch Observability

Source: https://observability.opensearch.org/docs/send-data/ai-agents/integrations

Install the OpenAI Agents SDK and the OpenSearch GenAI Observability SDK with the 'openai' extra for auto-instrumentation.

```bash
pip install "opensearch-genai-observability-sdk-py[openai]" openai-agents
```

--------------------------------

### Install Fluent Bit via Helm

Source: https://observability.opensearch.org/docs/send-data/infrastructure/fluentd

Commands to add the Fluent repository and install Fluent Bit into a Kubernetes cluster.

```bash
helm repo add fluent https://fluent.github.io/helm-charts
helm repo update


helm install fluent-bit fluent/fluent-bit \
  -n observability \
  --create-namespace \
  -f fluent-bit-values.yaml
```

--------------------------------

### Install Observability Stack via Helm

Source: https://observability.opensearch.org/docs/deploy/kubernetes

Clone the repository and deploy the stack using the provided Helm chart.

```bash
git clone https://github.com/opensearch-project/observability-stack.git
cd observability-stack


helm install obs charts/observability-stack
```

--------------------------------

### Start Agent Health Web Server

Source: https://observability.opensearch.org/docs/agent-health/cli

Starts the web server. This is the default action when no subcommand is specified. Configure the server port and environment file path.

```bash
agent-health [serve] [options]
```

```bash
agent-health --port 8080 --env-file prod.env
```

```bash
agent-health serve -p 8080 --no-browser
```

--------------------------------

### Install Strands Agents with Bedrock SDK

Source: https://observability.opensearch.org/docs/send-data/ai-agents/integrations

Install the necessary packages for Strands Agents and the GenAI Observability SDK with Bedrock support.

```bash
pip install "opensearch-genai-observability-sdk-py[bedrock]" strands-agents
```

--------------------------------

### Register with Self-hosted Data Prepper

Source: https://observability.opensearch.org/docs/send-data/ai-agents/python

Example of registering the SDK when using a self-hosted OpenSearch Data Prepper instance.

```python
# Self-hosted with Data Prepper
register(service_name="my-agent")

```

--------------------------------

### Install Strands Agents with OpenTelemetry SDK

Source: https://observability.opensearch.org/docs/send-data/ai-agents/integrations

Install the necessary packages for Strands Agents and the GenAI Observability SDK, compatible with OpenTelemetry.

```bash
pip install opensearch-genai-observability-sdk-py strands-agents
```

--------------------------------

### Install DeepEval and Observability SDK

Source: https://observability.opensearch.org/docs/agent-evals/evaluation-integrations

Required packages for integrating DeepEval with the OpenSearch observability SDK.

```bash
pip install opensearch-genai-observability-sdk-py deepeval
```

--------------------------------

### Install OpenTelemetry Go Dependencies

Source: https://observability.opensearch.org/docs/send-data/applications/go

Installs necessary OpenTelemetry SDK and exporter packages for Go applications. Ensure Go modules are enabled.

```bash
go get go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/sdk \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc \
  go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc \
  go.opentelemetry.io/otel/sdk/metric \
  go.opentelemetry.io/otel/semconv/v1.26.0
```

--------------------------------

### Select Specific Fields

Source: https://observability.opensearch.org/docs/ppl/commands

Example of the `fields` command to select a specific set of fields to display.

```ppl
| fields time, body, severityText, `resource.attributes.service.name`
| head 20
```

--------------------------------

### URL Synchronization Example

Source: https://observability.opensearch.org/docs/dashboards/variables/managing-variables

This example shows how variable values are synchronized to the dashboard URL using the `variableValues` query parameter. This allows for sharing dashboard states and preserving selections.

```url
?variableValues=(service:(api),region:(us-east,us-west))
```

--------------------------------

### Register with OTel Collector on Localhost

Source: https://observability.opensearch.org/docs/send-data/ai-agents/python

Example of registering the SDK to send traces to an OpenTelemetry Collector running on localhost.

```python
# OTel Collector on localhost
register(endpoint="http://localhost:4318/v1/traces", service_name="my-agent")

```

--------------------------------

### Run plugin test suite

Source: https://observability.opensearch.org/docs/claude-code

Commands to install dependencies and execute property or integration tests for the plugin.

```bash
cd claude-code-observability-plugin/tests
pip install -r requirements.txt


# Property tests (no running stack needed)
pytest test_properties.py -v


# Integration tests (requires running stack)
pytest -v
```

--------------------------------

### Initialize project configuration with init

Source: https://observability.opensearch.org/docs/agent-health/cli

Creates necessary configuration files like agent-health.config.ts and .env.example.

```bash
agent-health init [options]
```

```bash
agent-health init
agent-health init --force --with-examples
```

--------------------------------

### Register with gRPC Endpoint

Source: https://observability.opensearch.org/docs/send-data/ai-agents/python

Example of registering the SDK to use the gRPC protocol for sending traces.

```python
# gRPC
register(endpoint="grpc://localhost:4317", service_name="my-agent")

```

--------------------------------

### Override Helm Configuration Values

Source: https://observability.opensearch.org/docs/deploy/kubernetes

Customize deployment settings such as passwords and example enablement during installation.

```bash
helm install obs charts/observability-stack \
  --set opensearchPassword="YourSecurePassword123!" \
  --set examples.enabled=false
```

--------------------------------

### Setup OpenTelemetry SDK in Go

Source: https://observability.opensearch.org/docs/send-data/applications/go

Initializes trace and metric providers with OTLP gRPC exporters. Requires a running OTel Collector. Returns a shutdown function to gracefully stop providers.

```go
package telemetry


import (
  "context"
  "time"

  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
  "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
  "go.opentelemetry.io/otel/sdk/metric"
  "go.opentelemetry.io/otel/sdk/resource"
  sdktrace "go.opentelemetry.io/otel/sdk/trace"
  semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)


func Setup(ctx context.Context, serviceName string) (func(), error) {
  res, err := resource.New(ctx,
    resource.WithAttributes(
      semconv.ServiceName(serviceName),
      semconv.ServiceVersion("1.0.0"),
      semconv.DeploymentEnvironment("development"),
    ),
  )
  if err != nil {
    return nil, err
  }

  // Traces
  traceExporter, err := otlptracegrpc.New(ctx,
    otlptracegrpc.WithEndpoint("localhost:4317"),
    otlptracegrpc.WithInsecure(),
  )
  if err != nil {
    return nil, err
  }

  tp := sdktrace.NewTracerProvider(
    sdktrace.WithResource(res),
    sdktrace.WithBatcher(traceExporter),
  )
  otel.SetTracerProvider(tp)

  // Metrics
  metricExporter, err := otlpmetricgrpc.New(ctx,
    otlpmetricgrpc.WithEndpoint("localhost:4317"),
    otlpmetricgrpc.WithInsecure(),
  )
  if err != nil {
    return nil, err
  }

  mp := metric.NewMeterProvider(
    metric.WithResource(res),
    metric.WithReader(metric.NewPeriodicReader(metricExporter,
      metric.WithInterval(2*time.Second),
    )),
  )
  otel.SetMeterProvider(mp)

  // Return shutdown function
  shutdown := func() {
    tp.Shutdown(ctx)
    mp.Shutdown(ctx)
  }
  return shutdown, nil
}
```

--------------------------------

### Run Node.js Applications with Instrumentation

Source: https://observability.opensearch.org/docs/send-data/opentelemetry/auto-instrumentation

Load the instrumentation module using the --require flag before starting the application.

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_SERVICE_NAME="frontend-service"


node --require @opentelemetry/auto-instrumentations-node/register app.js
```

--------------------------------

### Quick Start: Instrumenting an AI Agent

Source: https://observability.opensearch.org/docs/send-data/ai-agents/python

Demonstrates basic usage of `register` and `observe` to instrument an AI agent with tool execution and LLM invocation.

```python
from opensearch_genai_observability_sdk_py import register, observe, Op, enrich


register(endpoint="http://localhost:4318/v1/traces", service_name="my-agent")


@observe(op=Op.EXECUTE_TOOL)
def get_weather(city: str) -> dict:
    return {"city": city, "temp": 22, "condition": "sunny"}


@observe(op=Op.INVOKE_AGENT)
def assistant(query: str) -> str:
    enrich(model="gpt-4o", provider="openai")
    data = get_weather("Paris")
    return f"{data['condition']}, {data['temp']}C"


result = assistant("What's the weather?")

```

--------------------------------

### PPL Query for Trace Spans

Source: https://observability.opensearch.org/docs/get-started/core-concepts

Use Piped Processing Language (PPL) to query trace span data. This example filters spans by status code and sorts them by start time.

```ppl
source = otel-v1-apm-span-* | where status.code = 2 | sort - startTime
```

--------------------------------

### Create Spans for User Interactions

Source: https://observability.opensearch.org/docs/send-data/applications/browser

Manually creates spans for user interactions like button clicks. This example demonstrates starting a span, setting attributes, and handling promise resolution or rejection to end the span and record exceptions.

```typescript
import { trace } from "@opentelemetry/api";


const tracer = trace.getTracer("my-frontend");


function handleCheckout(cartId) {
  const span = tracer.startSpan("checkout.click", {
    attributes: {
      "cart.id": cartId,
      "cart.item_count": getCartItemCount(),
    },
  });


  submitOrder(cartId)
    .then((result) => {
      span.setAttribute("order.id", result.orderId);
      span.end();
    })
    .catch((err) => {
      span.recordException(err);
      span.setStatus({ code: 2, message: err.message });
      span.end();
    });
}

```

--------------------------------

### Install OpenSearch Output Plugin

Source: https://observability.opensearch.org/docs/send-data/infrastructure/logstash

Install the OpenSearch output plugin into an existing Logstash installation using this command.

```bash
bin/logstash-plugin install logstash-output-opensearch
```

--------------------------------

### Evaluate RAG Pipeline and Log to OpenSearch

Source: https://observability.opensearch.org/docs/agent-evals/evaluation-integrations

Example demonstrating how to register the service, run RAGAS metrics, and log the results to OpenSearch via the Benchmark class.

```python
from opensearch_genai_observability_sdk_py import register, Benchmark
from ragas import evaluate as ragas_evaluate
from ragas.metrics import faithfulness, answer_correctness, context_precision
from datasets import Dataset


register(service_name="ragas-eval")


# Prepare your RAG evaluation dataset
data = {
    "question": ["What is OpenSearch?", "How does RAG work?"],
    "answer": [
        "OpenSearch is an open-source search and analytics engine.",
        "RAG retrieves documents and generates answers from them.",
    ],
    "contexts": [
        ["OpenSearch is a community-driven, open-source search and analytics suite."],
        ["RAG combines information retrieval with text generation."],
    ],
    "ground_truth": [
        "OpenSearch is a search and analytics engine.",
        "Retrieval-augmented generation retrieves then generates.",
    ],
}
dataset = Dataset.from_dict(data)


# Run RAGAS evaluation
ragas_result = ragas_evaluate(
    dataset=dataset,
    metrics=[faithfulness, answer_correctness, context_precision],
)


# Upload results to OpenSearch via Benchmark
df = ragas_result.to_pandas()
with Benchmark("ragas_eval", metadata={"framework": "ragas"}) as b:
    for i, row in df.iterrows():
        b.log(
            input=data["question"][i],
            output=data["answer"][i],
            expected=data["ground_truth"][i],
            scores={col: row[col] for col in df.columns if col not in data},
            case_name=data["question"][i][:50],
        )
```

--------------------------------

### Integrate DeepEval with Benchmark

Source: https://observability.opensearch.org/docs/agent-evals/evaluation-integrations

Example showing how to register the service, define test cases, and log DeepEval metrics to the observability stack using the Benchmark context manager.

```python
from opensearch_genai_observability_sdk_py import register, Benchmark
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase


register(service_name="deepeval-eval")


# Define your test cases
test_cases = [
    {
        "input": "What is OpenSearch?",
        "output": "OpenSearch is an open-source search and analytics engine.",
        "expected": "OpenSearch is a search and analytics engine.",
        "context": ["OpenSearch is an open-source search and analytics suite."],
    },
    {
        "input": "How does RAG work?",
        "output": "RAG retrieves relevant documents and uses them to generate answers.",
        "expected": "Retrieval-augmented generation combines retrieval with generation.",
        "context": ["RAG first retrieves documents, then feeds them to an LLM."],
    },
]


# Run DeepEval metrics and upload results
relevancy = AnswerRelevancyMetric(model="gpt-4o")
faithfulness = FaithfulnessMetric(model="gpt-4o")


with Benchmark("deepeval_run", metadata={"framework": "deepeval"}) as b:
    for case in test_cases:
        tc = LLMTestCase(
            input=case["input"],
            actual_output=case["output"],
            expected_output=case["expected"],
            retrieval_context=case["context"],
        )
        relevancy.measure(tc)
        faithfulness.measure(tc)
        b.log(
            input=case["input"],
            output=case["output"],
            expected=case["expected"],
            scores={
                "answer_relevancy": relevancy.score,
                "faithfulness": faithfulness.score,
            },
            case_name=case["input"][:50],
        )
```

--------------------------------

### Install Browser OpenTelemetry Dependencies

Source: https://observability.opensearch.org/docs/send-data/applications/browser

Installs the necessary OpenTelemetry packages for browser instrumentation. Ensure you have npm, yarn, or pnpm installed.

```bash
npm install @opentelemetry/api \
  @opentelemetry/sdk-trace-web \
  @opentelemetry/sdk-trace-base \
  @opentelemetry/resources \
  @opentelemetry/semantic-conventions \
  @opentelemetry/exporter-trace-otlp-http \
  @opentelemetry/context-zone \
  @opentelemetry/instrumentation-document-load \
  @opentelemetry/instrumentation-fetch \
  @opentelemetry/instrumentation-xml-http-request
```

--------------------------------

### Install Agent Health CLI

Source: https://observability.opensearch.org/docs/agent-health/cli

Install the Agent Health CLI globally using npm or run commands without installation using npx.

```bash
npm install -g @opensearch-project/agent-health   # Global install
```

```bash
npx @opensearch-project/agent-health <command>    # No install required
```

--------------------------------

### Join Examples

Source: https://observability.opensearch.org/docs/ppl/commands/join

Illustrates practical use cases of the join command, including joining two indexes and joining with a subsearch.

```APIDOC
## Examples

### Join two indexes

Correlate log entries with trace spans using `traceId` to see which spans produced each log line:
```
source = logs-otel-v1*
| inner join left=l right=r ON l.traceId = r.traceId otel-v1-apm-span-*
```

### Join with a subsearch

Join logs with a filtered subset of trace spans - only slow spans above a latency threshold:
```
source = logs-otel-v1* as l
| where severityText = 'ERROR'
| left join ON l.traceId = r.traceId [
    source = otel-v1-apm-span-*
    | where durationInNanos > 5000000000
    | sort - durationInNanos
    | head 100
  ] as r
```
```

--------------------------------

### Initialize OpenSearchTraceRetriever

Source: https://observability.opensearch.org/docs/agent-evals/evaluation

Instantiate the retriever with host, authentication, and TLS verification settings.

```python
from opensearch_genai_observability_sdk_py import OpenSearchTraceRetriever


retriever = OpenSearchTraceRetriever(
    host="https://localhost:9200",
    auth=("admin", "admin"),
    verify_certs=False,
)
```

--------------------------------

### Install pytest Integration Dependencies

Source: https://observability.opensearch.org/docs/agent-evals/evaluation-integrations

Install the required SDK and pytest packages.

```bash
pip install opensearch-genai-observability-sdk-py pytest
```

--------------------------------

### Start Agent Health with demo data

Source: https://observability.opensearch.org/docs/agent-health

Launches the Agent Health server with pre-loaded sample data for exploration at http://localhost:4001.

```bash
# Start Agent Health with demo data (no configuration needed)
npx @opensearch-project/agent-health@latest
```

--------------------------------

### Rename Field Example

Source: https://observability.opensearch.org/docs/ppl/commands

Example of the `rename` command to change the name of a field.

```ppl
| rename `resource.attributes.service.name` as service
| head 20
```

--------------------------------

### Install OpenTelemetry Dependencies

Source: https://observability.opensearch.org/docs/get-started/quickstart/first-traces

Installs the required Python packages for OpenTelemetry API, SDK, and OTLP exporter.

```bash
pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp
```

--------------------------------

### Enable OpenTelemetry Demo

Source: https://observability.opensearch.org/docs/get-started/installation

Uncomment this line in the .env file to include the microservices e-commerce demo.

```bash
INCLUDE_COMPOSE_OTEL_DEMO=docker-compose.otel-demo.yml
```

--------------------------------

### Search with Inline Filter Example

Source: https://observability.opensearch.org/docs/ppl/commands

Example of using the `search` command with an inline boolean expression to filter results.

```ppl
source = logs-otel-v1* severityText = 'ERROR'
```

--------------------------------

### Run Go Application and Attach eBPF Agent

Source: https://observability.opensearch.org/docs/send-data/opentelemetry/auto-instrumentation

Configure environment variables, start your Go application in the background, and then attach the eBPF instrumentation agent using its PID. Requires elevated privileges.

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_SERVICE_NAME="shipping-service"

./shipping-service &
APP_PID=$!

sudo instrumentation -pid $APP_PID
```

--------------------------------

### Install Deployment Collector via Helm

Source: https://observability.opensearch.org/docs/send-data/infrastructure/kubernetes

Command to install the OpenTelemetry Collector Deployment using the gateway configuration file.

```bash
helm install otel-gateway open-telemetry/opentelemetry-collector \
  -f values-deployment.yaml \
  -n observability
```

--------------------------------

### Run Benchmark in Quick Mode

Source: https://observability.opensearch.org/docs/agent-health/evaluations/experiments

Use this command to automatically create and run a benchmark using all stored test cases.

```bash
npx @opensearch-project/agent-health benchmark
```

--------------------------------

### Install DaemonSet Collector via Helm

Source: https://observability.opensearch.org/docs/send-data/infrastructure/kubernetes

Command to install the OpenTelemetry Collector DaemonSet using the previously defined configuration file.

```bash
helm install otel-daemonset open-telemetry/opentelemetry-collector \
  -f values-daemonset.yaml \
  -n observability \
  --create-namespace
```

--------------------------------

### Configure and Run .NET Application

Source: https://observability.opensearch.org/docs/send-data/opentelemetry/auto-instrumentation

Set environment variables for endpoint and service name, source the instrumentation script, and then run your .NET application. This enables automatic instrumentation for supported libraries.

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_SERVICE_NAME="inventory-service"

. $HOME/.otel-dotnet-auto/instrument.sh

dotnet run
```

--------------------------------

### Auto-instrumentation

Source: https://observability.opensearch.org/docs/send-data/ai-agents/python

Discover and activate installed instrumentor packages for various providers and frameworks. No code changes are needed after installing the appropriate extra.

```APIDOC
## Auto-instrumentation

### Description
Discovers and activates installed instrumentor packages. Install the extra for your provider - no code changes needed.

### Providers and Frameworks

| Provider / framework | Extra |
|---|---|
| OpenAI, OpenAI Agents | `[openai]` |
| Anthropic | `[anthropic]` |
| Amazon Bedrock | `[bedrock]` |
| Google Generative AI + Vertex AI | `[google]` |
| LangChain | `[langchain]` |
| LlamaIndex | `[llamaindex]` |
| All providers + frameworks | `[otel-instrumentors]` |

`[otel-instrumentors]` includes all of the above plus Cohere, Mistral, Groq, Ollama, Together, Replicate, Writer, Voyage AI, Aleph Alpha, SageMaker, watsonx, Haystack, CrewAI, Agno, MCP, Transformers, ChromaDB, Pinecone, Qdrant, Weaviate, Milvus, LanceDB, and Marqo.

### Usage
```python
register(auto_instrument=False)  # to disable
```
```

--------------------------------

### Register Auto-instrumentation

Source: https://observability.opensearch.org/docs/send-data/ai-agents/python

Call `register()` to discover and activate installed instrumentor packages. No code changes are needed after installing the provider-specific extra.

```python
register(auto_instrument=False)  # to disable
```

--------------------------------

### List Available Resources

Source: https://observability.opensearch.org/docs/agent-health/cli

List available resources such as agents, connectors, models, test cases, and benchmarks. Output can be formatted as a table or JSON.

```bash
agent-health list <resource> [-o table|json]
```

```bash
agent-health list agents
```

```bash
agent-health list tc -o json
```

```bash
agent-health list bench
```

--------------------------------

### Install CrewAI with OpenSearch Observability

Source: https://observability.opensearch.org/docs/send-data/ai-agents/integrations

Install the necessary packages for CrewAI integration with OpenSearch GenAI Observability SDK, includingotel-instrumentors for auto-instrumentation.

```bash
pip install "opensearch-genai-observability-sdk-py[otel-instrumentors]" crewai
```

--------------------------------

### Configure Data Prepper for Trace Analytics and Sampling

Source: https://observability.opensearch.org/docs/send-data/opentelemetry/sampling

A complete Data Prepper configuration example that includes the otel_apm_service_map processor followed by a tail_sampler for trace analytics.

```yaml

otel-trace-pipeline:
  source:
    otel_trace_source:
      ssl: false
  processor:
    - otel_apm_service_map:
        window_duration: 180
    - aggregate:
        identification_keys: ["traceId"]
        action:
          tail_sampler:
            percent: 20
            wait_period: "10s"
            condition: '/status == "ERROR" or /durationInNanos > 5000000000'
  sink:
    - opensearch:
        hosts: ["https://opensearch:9200"]
        index_type: trace-analytics-raw


service-map-pipeline:
  source:
    pipeline:
      name: "otel-trace-pipeline"
  processor:
    - service_map:
  sink:
    - opensearch:
        hosts: ["https://opensearch:9200"]
        index_type: trace-analytics-service-map

```

--------------------------------

### Install .NET OpenTelemetry Dependencies

Source: https://observability.opensearch.org/docs/send-data/applications/dotnet

Installs the necessary OpenTelemetry packages for .NET applications. Ensure you have .NET 6.0+ and NuGet package management.

```bash
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
```

--------------------------------

### Initialize OpenTelemetry Node.js SDK

Source: https://observability.opensearch.org/docs/send-data/applications/nodejs

Set up the OpenTelemetry Node.js SDK with resource attributes, exporters for traces, metrics, and logs, and auto-instrumentations. The SDK is started and configured to shut down gracefully on SIGTERM.

```javascript
const { NodeSDK } = require("@opentelemetry/sdk-node");
const {
  OTLPTraceExporter,
} = require("@opentelemetry/exporter-trace-otlp-grpc");
const {
  OTLPMetricExporter,
} = require("@opentelemetry/exporter-metrics-otlp-grpc");
const {
  OTLPLogExporter,
} = require("@opentelemetry/exporter-logs-otlp-grpc");
const {
  PeriodicExportingMetricReader,
} = require("@opentelemetry/sdk-metrics");
const { Resource } = require("@opentelemetry/resources");
const {
  ATTR_SERVICE_NAME,
  ATTR_SERVICE_VERSION,
} = require("@opentelemetry/semantic-conventions");
const {
  getNodeAutoInstrumentations,
} = require("@opentelemetry/auto-instrumentations-node");


const resource = new Resource({
  [ATTR_SERVICE_NAME]: "my-node-service",
  [ATTR_SERVICE_VERSION]: "1.0.0",
  "deployment.environment": "development",
});


const sdk = new NodeSDK({
  resource,
  traceExporter: new OTLPTraceExporter({
    url: "grpc://localhost:4317",
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: "grpc://localhost:4317",
    }),
    exportIntervalMillis: 2000,
  }),
  logRecordExporter: new OTLPLogExporter({
    url: "grpc://localhost:4317",
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});


sdk.start();


process.on("SIGTERM", () => {
  sdk.shutdown().then(() => process.exit(0));
});

```

--------------------------------

### Upload pre-computed results with Benchmark

Source: https://observability.opensearch.org/docs/agent-evals/evaluation

Use the Benchmark class to log evaluation results as OTel spans. This example demonstrates both context manager and manual lifecycle management.

```python
from opensearch_genai_observability_sdk_py import register, Benchmark


register(service_name="eval-upload")


with Benchmark("ragas_eval_v2", metadata={"framework": "ragas"}) as b:
    b.log(
        input="What is OpenSearch?",
        output="OpenSearch is an open-source search engine.",
        expected="search and analytics engine",
        scores={"faithfulness": 0.92, "relevance": 0.88},
        case_name="opensearch_definition",
    )
    b.log(
        input="How does RAG work?",
        output="RAG retrieves documents then generates answers.",
        scores={"faithfulness": 0.95, "relevance": 0.91},
        case_name="rag_explanation",
    )
# summary printed on close
```

```python
b = Benchmark(name="my-eval")
b.log(input="q1", output="a1", scores={"accuracy": 1.0})
summary = b.close()
```

--------------------------------

### Install OpenSearch Agent Health CLI

Source: https://observability.opensearch.org/docs/agent-health/configuration

Use this command to install the agent health CLI tool. This tool is used for validating your agent configuration.

```bash
npx @opensearch-project/agent-health doctor
```

--------------------------------

### Deploy Observability Stack via CLI

Source: https://observability.opensearch.org/docs/deploy/aws

Use the CLI installer to deploy the stack with either managed or serverless backends.

```bash
npx @opensearch-project/observability-stack
```

```bash
npx @opensearch-project/observability-stack --serverless
```

--------------------------------

### Eval Command Extended Examples

Source: https://observability.opensearch.org/docs/ppl/commands/eval

Provides advanced examples of using the `eval` command for complex data transformations, such as categorizing log severity and building composite identifiers.

```APIDOC
### Extended Examples

#### OTel: Categorize log severity into alert levels
Derive an `alert_level` field from the numeric severity of log events, useful for routing alerts or filtering dashboards.
```
| eval alert_level = case(
    severityNumber >= 21, 'CRITICAL',
    severityNumber >= 17, 'ERROR',
    severityNumber >= 13, 'WARN',
    severityNumber >= 9,  'INFO'
    else 'DEBUG')
| stats count() as cnt by alert_level, `resource.attributes.service.name`
```

#### OTel: Build a composite service identifier
Combine the service name and severity into a single field for downstream grouping or display.
```
| eval service_status = `resource.attributes.service.name` + ' [' + severityText + ']'
| head 20
```
```

--------------------------------

### Configure OpenTelemetry SDK

Source: https://observability.opensearch.org/docs/send-data/applications/python

Function to initialize TracerProvider, MeterProvider, and LoggerProvider with OTLP exporters.

```python
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter


def setup_telemetry(service_name: str = "my-service"):
    resource = Resource.create({
        "service.name": service_name,
        "service.version": "1.0.0",
        "deployment.environment": "development",
    })


    # Traces
    tracer_provider = TracerProvider(resource=resource)
    tracer_provider.add_span_processor(
        BatchSpanProcessor(
            OTLPSpanExporter(endpoint="localhost:4317", insecure=True)
        )
    )
    trace.set_tracer_provider(tracer_provider)


    # Metrics
    metric_reader = PeriodicExportingMetricReader(
        OTLPMetricExporter(endpoint="localhost:4317", insecure=True),
        export_interval_millis=2000,
    )
    meter_provider = MeterProvider(
        resource=resource, metric_readers=[metric_reader]
    )
    metrics.set_meter_provider(meter_provider)


    # Logs
    logger_provider = LoggerProvider(resource=resource)
    logger_provider.add_log_record_processor(
        BatchLogRecordProcessor(
            OTLPLogExporter(endpoint="localhost:4317", insecure=True)
        )
    )


    return tracer_provider, meter_provider, logger_provider
```
Tokens:
Raw