Observe

https://docs.observeinc.com/docs/
Admin
Observe is a comprehensive observability platform that collects, analyzes, and visualizes data from...
Tokens:367,583
Snippets:4,879
Trust Score:9.4
Update:2 months ago
Show doc for...
Tokens:
Raw
### Install and start the Observe Agent

Source: https://docs.observeinc.com/docs/macos

Downloads the binary and starts the launch daemon. Requires administrator privileges.

```bash
curl -OL "https://github.com/observeinc/observe-agent/releases/latest/download/install_mac.sh" && \
bash install_mac.sh --token=<YOUR_INGEST_TOKEN> --observe_url=<YOUR_OBSERVE_COLLECTION_ENDPOINT> --logs_enabled=true --metrics_enabled=true
```

--------------------------------

### Start Application with Instrumentation

Source: https://docs.observeinc.com/docs/send-nodejs-application-data-to-observe

Use the --require or --import flags to load the instrumentation setup file at runtime.

```bash
node --require ./instrumentation.cjs app.js
```

```bash
node --experimental-loader=@opentelemetry/instrumentation/hook.mjs --import ./instrumentation.mjs app.js
```

--------------------------------

### Install and start Telegraf on CentOS

Source: https://docs.observeinc.com/docs/install-the-mysql-app

Commands to configure the repository, install Telegraf, and start the service on CentOS.

```bash
cat <<EOF | sudo tee /etc/yum.repos.d/influxdb.repo
[influxdb]
name = InfluxDB Repository - RHEL \$releasever
baseurl = https://repos.influxdata.com/rhel/\$releasever/\$basearch/stable
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key
EOF
sudo yum install telegraf
```

```bash
sudo service telegraf start
```

--------------------------------

### Install and Verify Observe CLI

Source: https://docs.observeinc.com/docs/use-ai-to-install-the-observe-agent-on-kubernetes

Commands to install the Observe CLI tool and verify the installation version.

```text
curl -fsSL https://raw.githubusercontent.com/observeinc/cli/main/install.sh | bash
```

```text
observe --version
```

--------------------------------

### Install Observe CLI

Source: https://docs.observeinc.com/docs/use-ai-to-install-the-observe-agent-on-a-host

Command to install the Observe CLI tool on your system.

```text
curl -fsSL https://raw.githubusercontent.com/observeinc/cli/main/install.sh | bash
```

--------------------------------

### Enable and start the Observe Agent

Source: https://docs.observeinc.com/docs/linux-1

Use systemctl to enable and immediately start the agent service.

```bash
sudo systemctl enable --now observe-agent
```

--------------------------------

### Configure Prisma Instrumentation Setup

Source: https://docs.observeinc.com/docs/send-nodejs-application-data-to-observe

Create a setup file to initialize the OpenTelemetry SDK with Prisma instrumentation.

```javascript
const { NodeSDK } = require("@opentelemetry/sdk-node");
const {
  getNodeAutoInstrumentations,
} = require("@opentelemetry/auto-instrumentations-node");
const { PrismaInstrumentation } = require("@prisma/instrumentation");

const sdk = new NodeSDK({
  instrumentations: [
    getNodeAutoInstrumentations(),
    new PrismaInstrumentation(),
  ],
});

sdk.start();
```

```typescript
import { NodeSDK } from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { PrismaInstrumentation } from "@prisma/instrumentation";

const sdk = new NodeSDK({
  instrumentations: [
    getNodeAutoInstrumentations(),
    new PrismaInstrumentation(),
  ],
});

sdk.start();
```

--------------------------------

### Install Observe Skills

Source: https://docs.observeinc.com/docs/use-ai-to-install-the-observe-agent-on-kubernetes

Commands to install specific or all available Observe skills using npx.

```text
# Install all necessary skills
npx skills add observeinc/skills -s debug-k8s-collection deploy-k8s-explorer setup-k8s-backend setup-k8s-collection

# Or, install all skills in the skills repo, without prompts
npx skills add observeinc/skills --all

# Or choose skills interactively
npx skills add observeinc/skills
```

--------------------------------

### Install OpenTelemetry dependencies

Source: https://docs.observeinc.com/docs/send-python-application-data-to-observe

Install the necessary OpenTelemetry libraries and bootstrap instrumentation.

```bash
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
```

--------------------------------

### Install OpenTelemetry Instrumentation on Linux

Source: https://docs.observeinc.com/docs/send-net-application-data-to-observe

Downloads and executes the installation script to set up instrumentation for the current shell session.

```bash
# Download the bash script
curl -sSfL https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/releases/latest/download/otel-dotnet-auto-install.sh -O

# Install core files
sh ./otel-dotnet-auto-install.sh

# Enable execution for the instrumentation script
chmod +x $HOME/.otel-dotnet-auto/instrument.sh

# Setup the instrumentation for the current shell session
. $HOME/.otel-dotnet-auto/instrument.sh
```

--------------------------------

### Verify CLI Installation

Source: https://docs.observeinc.com/docs/use-ai-to-install-the-observe-agent-on-a-host

Command to verify that the Observe CLI is correctly installed.

```text
observe --version
```

--------------------------------

### Install Traceloop SDK

Source: https://docs.observeinc.com/docs/use-python-instrumentation-for-llm-observability

Install the required dependency via pip.

```bash
pip install traceloop-sdk
```

--------------------------------

### Install Observe Agent via Bash Script

Source: https://docs.observeinc.com/docs/linux-1

Download and execute the official installation script for systems without yum or apt.

```bash
curl -OL "https://github.com/observeinc/observe-agent/releases/latest/download/install_linux.sh"
bash install_linux.sh --token=<YOUR_INGEST_TOKEN> --observe_url=<YOUR OBSERVE COLLECTION ENDPOINT> --logs_enabled=true --metrics_enabled=true
```

--------------------------------

### Install Observe Agent on Ubuntu/Debian

Source: https://docs.observeinc.com/docs/linux-1

Add the Observe debian repository and install the agent package.

```bash
echo 'deb [trusted=yes] https://repo.observeinc.com/apt/ /' | sudo tee /etc/apt/sources.list.d/observeinc.list
sudo apt update
sudo apt install observe-agent
```

--------------------------------

### Install Prisma Instrumentation

Source: https://docs.observeinc.com/docs/send-nodejs-application-data-to-observe

Install the required Prisma instrumentation package alongside existing auto-instrumentations.

```bash
npm install @prisma/instrumentation
```

--------------------------------

### Install the Observe for Snowflake App via SQL

Source: https://docs.observeinc.com/docs/install-the-observe-for-snowflake-app

Execute these commands in a Snowflake worksheet to install the application from the marketplace listing.

```sql
--------------------------------------------------------------------------------
-- [ manual ]
-- Step 1: Install the APP
--         By default, the app is called 'OBSERVE_FOR_SNOWFLAKE'
-- You must run it as ACCOUNTADMIN
--------------------------------------------------------------------------------
USE ROLE ACCOUNTADMIN;

CREATE APPLICATION OBSERVE_FOR_SNOWFLAKE FROM LISTING GZTYZY3AR0U;
```

--------------------------------

### Verify Observe Agent installation

Source: https://docs.observeinc.com/docs/macos

Checks the installed version of the agent to confirm successful installation.

```text
observe-agent version
```

--------------------------------

### Install Fastify OTel Plugin

Source: https://docs.observeinc.com/docs/send-nodejs-application-data-to-observe

Command to install the Fastify OTel plugin for enhanced request-handler spans.

```bash
npm install @fastify/otel
```

--------------------------------

### Pod deployment output

Source: https://docs.observeinc.com/docs/node-affinity-taints-and-tolerations

Example output showing pods running on the expected node.

```text
$kubectl get pods -o wide -n observe
NAME                                             READY   STATUS    RESTARTS   AGE   IP           NODE                                       NOMINATED NODE   READINESS GATES
observe-agent-cluster-events-7bd9ccddcf-m6svd    1/1     Running   0          54s   10.248.8.6   gke-cluster-1-default-pool-020b00c1-9s2c   <none>           <none>
observe-agent-cluster-metrics-7fc5987bcb-v95rl   1/1     Running   0          54s   10.248.8.5   gke-cluster-1-default-pool-020b00c1-9s2c   <none>           <none>
observe-agent-monitor-b9f8c59c-9gpx2             1/1     Running   0          53s   10.248.8.8   gke-cluster-1-default-pool-020b00c1-9s2c   <none>           <none>
observe-agent-node-logs-metrics-agent-j59kh      1/1     Running   0          54s   10.248.8.7   gke-cluster-1-default-pool-020b00c1-9s2c   <none>           <none>
```

--------------------------------

### Install OpenTelemetry Instrumentation on Windows

Source: https://docs.observeinc.com/docs/send-net-application-data-to-observe

Downloads the PowerShell module and installs the core instrumentation files.

```powershell
    # Download the module

    $module_url = "[https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/releases/latest/download/OpenTelemetry.DotNet.Auto.psm1](https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/releases/latest/download/OpenTelemetry.DotNet.Auto.psm1)"
    $download_path = Join-Path $env:temp "OpenTelemetry.DotNet.Auto.psm1"
    Invoke-WebRequest -Uri $module_url -OutFile $download_path -UseBasicParsing

    # Import the module to use its functions

    Import-Module $download_path

    # Install core files (online vs offline method)

    Install-OpenTelemetryCore
```

--------------------------------

### bin_start_time()

Source: https://docs.observeinc.com/docs/function-bin_start_time

Returns the start time of the current bin (inclusive). When used with timechart or align, it refers to the bin start; when used with statsby, it returns the query window start time.

```APIDOC
## bin_start_time()

### Description
Returns the start time of the current bin (inclusive). This is a scalar function typically used with time binning verbs like `timechart` or `align`, or with `statsby` to return the query window start time.

### Signature
`bin_start_time() -> timestamp`

### Examples

#### Calculate difference between first data point and bin start
`timechart 10m, first_to_bin_start:min(timestamp - bin_start_time()), group_by(key)`

#### Calculate minimum gap between timestamp and query window start
`statsby least_recent:min(timestamp - bin_start_time()), group_by(key)`
```

--------------------------------

### Install Observe Agent via PowerShell script

Source: https://docs.observeinc.com/docs/ansible-on-windows

This Ansible task file handles downloading the agent ZIP and installation script, executing the installer, and managing service state or reboots.

```yaml
############################################################
# roles/agent/tasks/install_with_ps1.yaml  (ZIP workflow)
############################################################
- name: Ensure staging directory exists
  ansible.windows.win_file:
    path: C:\Temp\Observe
    state: directory

# 1. Download the latest Observe Agent ZIP
- name: Download latest Observe Agent ZIP
  ansible.windows.win_get_url:
    url: https://github.com/observeinc/observe-agent/releases/latest/download/observe-agent_Windows_x86_64.zip
    dest: C:\Temp\Observe\observe-agent.zip
    force: yes

# 2. Download install.ps1
- name: Download install.ps1
  ansible.windows.win_get_url:
    url: https://github.com/observeinc/observe-agent/releases/latest/download/install.ps1
    dest: C:\Temp\Observe\install.ps1
    force: yes

# 3. Run the script (enable TLS 1.2 first)
- name: Run install.ps1 with token, endpoint, and zip_dir
  ansible.windows.win_shell: |
    [Net.ServicePointManager]::SecurityProtocol = 'Tls, Tls11, Tls12, Ssl3'
    C:\Temp\Observe\install.ps1 `
      -observe_token "{{ YOUR_INGEST_TOKEN }}" `
      -observe_collection_endpoint "{{ YOUR_OBSERVE_COLLECTION_ENDPOINT }}" `
      -zip_dir "C:\Temp\Observe\observe-agent.zip"
  args:
    executable: powershell.exe
  register: ps_install

# 4. Reboot only when the script returns 3010
- name: Reboot if install.ps1 returned 3010
  ansible.windows.win_reboot:
  when: ps_install.rc == 3010

# 5. Make sure the service is up (covers non-reboot cases)
- name: Ensure ObserveAgent service is running
  ansible.windows.win_service:
    name: ObserveAgent
    start_mode: auto
    state: started
```

--------------------------------

### Install Observe Agent on Red Hat

Source: https://docs.observeinc.com/docs/linux-1

Add the Observe YUM repository and install the agent package.

```bash
echo '[fury]
name=Gemfury Private Repo
baseurl=https://yum.fury.io/observeinc/
enabled=1
gpgcheck=0' | sudo tee /etc/yum.repos.d/fury.repo

sudo yum install observe-agent
```

--------------------------------

### Example policy ARN output

Source: https://docs.observeinc.com/docs/install-on-amazon-ecs-fargate-sidecar-pattern

Example of the ARN returned after creating the IAM policy.

```bash
arn:aws:iam::<ACCOUNT_ID>:policy/MyConfigAccessPolicy
```

--------------------------------

### Install Observe Agent

Source: https://docs.observeinc.com/docs/deploy-in-a-custom-namespace

Install the Observe Agent using Helm, specifying namespace overrides for all internal components.

```bash
helm install observe-agent observe/agent -n observability \
  --set cluster.namespaceOverride.value="observability" \
  --set cluster-events.namespaceOverride="observability" \
  --set cluster-metrics.namespaceOverride="observability" \
  --set prometheus-scraper.namespaceOverride="observability" \
  --set node-logs-metrics.namespaceOverride="observability" \
  --set monitor.namespaceOverride="observability" \
  --set forwarder.namespaceOverride="observability" \
  --set observe.collectionEndpoint.value="https://<your-collector>.collect.observe-staging.com/" \
  --set cluster.name="k8s-cluster" \
  --set node.enabled="true" \
  --set application.prometheusScrape.enabled="false" \
  --set node.forwarder.enabled="true" \
  --set node.forwarder.metrics.outputFormat="otel"
```

--------------------------------

### Run Ansible Playbook for Windows Agent Installation

Source: https://docs.observeinc.com/docs/ansible-on-windows

Executes the installation playbook against a specified inventory file and displays the task execution status.

```bash
$ ansible-playbook install-observe-agent-windows.yaml -i /etc/ansible/hosts/windows.aws_ec2.ini

PLAY [Install or upgrade Observe Agent on Windows fleet] **********************************************************************************************************************************************************

TASK [agent : Ensure staging directory exists] ********************************************************************************************************************************************************************
ok: [ansible-managed-node-2]
ok: [ansible-managed-node-1]

TASK [agent : Download latest Observe Agent ZIP] ******************************************************************************************************************************************************************
ok: [ansible-managed-node-2]
ok: [ansible-managed-node-1]

TASK [agent : Download install.ps1] *******************************************************************************************************************************************************************************
ok: [ansible-managed-node-2]
ok: [ansible-managed-node-1]

TASK [agent : Run install.ps1 with token, endpoint, and zip_dir] **************************************************************************************************************************************************
changed: [ansible-managed-node-2]
changed: [ansible-managed-node-1]

TASK [agent : Reboot if install.ps1 returned 3010] ****************************************************************************************************************************************************************
skipping: [ansible-managed-node-2]
skipping: [ansible-managed-node-1]

TASK [agent : Ensure ObserveAgent service is running] *************************************************************************************************************************************************************
changed: [ansible-managed-node-2]
changed: [ansible-managed-node-1]

PLAY RECAP ********************************************************************************************************************************************************************************************************
ansible-managed-node-1     : ok=5    changed=2    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   
ansible-managed-node-2     : ok=5    changed=2    unreachable=0    failed=0    skipped=1 
```

--------------------------------

### Install Traceloop SDK

Source: https://docs.observeinc.com/docs/use-nodejs-server-instrumentation-for-llm-observability

Add the Traceloop Node.js server SDK as a project dependency.

```bash
npm install --save @traceloop/node-server-sdk
```

--------------------------------

### Install Observe AI Skills

Source: https://docs.observeinc.com/docs/using-ai-skills

Commands to add specific OpenTelemetry-related skills to your environment.

```bash
npx skills add observeinc/skills -s opentelemetry-auto-instrumentation
npx skills add observeinc/skills -s opentelemetry-validation
npx skills add observeinc/skills -s opentelemetry-manual-instrumentation
```

--------------------------------

### Verify operator installation

Source: https://docs.observeinc.com/docs/auto-instrumentation-with-opentelemetry-operator-in-kubernetes

Checks the status of pods in the observe namespace.

```bash
kubectl get pods -n observe
```

--------------------------------

### Install Observe Agent via PowerShell

Source: https://docs.observeinc.com/docs/windows

Run this script in an administrator PowerShell terminal to download and install the agent with your specific ingest token and collection endpoint.

```text
[Net.ServicePointManager]::SecurityProtocol = "Tls, Tls11, Tls12, Ssl3"; Invoke-WebRequest -UseBasicParsing "https://github.com/observeinc/observe-agent/releases/latest/download/install.ps1" -outfile .\install.ps1; .\install.ps1 -observe_token "<YOUR_INGEST_TOKEN>" -observe_collection_endpoint "<YOUR_OBSERVE_COLLECTION_ENDPOINT>"
```

--------------------------------

### Example multiline log records

Source: https://docs.observeinc.com/docs/configure-the-observe-agent-on-linux-windows-and-macos

Sample log output containing multiline entries starting with brackets.

```log
[2025-06-18T18:52:24.089689Z] INFO: User login successful. User ID: 92065
[2025-06-18T18:52:24.561449Z] INFO: User login successful. User ID: 65749
[2025-06-18T18:52:25.309899Z] DEBUG: Starting backup process.
Directory: /data/backup
Estimated files: 556
[2025-06-18T18:52:26.226822Z] DEBUG: Starting backup process.
Directory: /data/backup
Estimated files: 446
[2025-06-18T18:52:27.538841Z] ERROR: Failed to load configuration file.
File path: /etc/app/config.yaml
Cause: FileNotFoundError
Stack trace:
    File "/app/main.py", line 23, in load_config
    config = open(config_path, 'r')
FileNotFoundError: [Errno 2] No such file or directory: '/etc/app/config.yaml'
[2025-06-18T18:52:29.004533Z] ERROR: Failed to load configuration file.
File path: /etc/app/config.yaml
Cause: FileNotFoundError
Stack trace:
    File "/app/main.py", line 23, in load_config
    config = open(config_path, 'r')
FileNotFoundError: [Errno 2] No such file or directory: '/etc/app/config.yaml'
[2025-06-18T18:52:30.190177Z] INFO: User login successful. User ID: 30051
```

--------------------------------

### Example tokenized output

Source: https://docs.observeinc.com/docs/unified-search-syntax

Resulting array of keywords after processing a sample log string.

```json
['I20240912', '18', '04', '10', '088056', '23943', 'core', 'src', 'handler', 'cpp', '216', 'msg', 'ingest', 'error', 'customer', 'id', '1234567890', 'path', 'v2', 'otel', 'v1', 'metrics', 'request', 'id', 'c2cd80b32a04d384cbdae5e8c4cc80c5', 'ingest', 'error', 'decode', 'error', 'opentelemetry', 'sum', 'metric', 'has', 'no', 'data', 'points']
```

--------------------------------

### Minimal Fluent Bit Configuration for Observe

Source: https://docs.observeinc.com/docs/fluent-bit

A complete configuration example using the dummy input plugin to send data to an Observe HTTP endpoint.

```text
[SERVICE]
    flush        5
    daemon       Off
    log_level    info

# Uncomment the below section if using AWS EC2
#[FILTER]
#    Name aws
#    Match *
#    imds_version v1
#    az true
#    ec2_instance_id true
#    ec2_instance_type true
#    account_id true
#    hostname true
#    vpc_id true

[INPUT]
    name  dummy
    dummy {"message":"Hello World!"}
    rate  1

[OUTPUT]
    name         http
    match        *
    host         ${OBSERVE_CUSTOMER}.collect.observeinc.com
    port         443
    tls          on

    uri          /v1/http/fluentbit

    format       msgpack
    header       Authorization     Bearer ${OBSERVE_TOKEN}
    header       X-Observe-Decoder fluent
    compress     gzip
```

--------------------------------

### Deploy Observe GCP Collection via Terraform

Source: https://docs.observeinc.com/docs/install

Use these modules to install the GCP collection stack. Choose the folder module for multi-project environments or the project module for single-project setups.

```hcl
provider "google" {
  project = "YOUR_PROJECT_ID"
  region  = "YOUR_DEFAULT_REGION"
}

module "observe_gcp_collection" {
  source   = "observeinc/collection/google"
  name     = "dev"
  resource = "folders/YOUR_FOlDER_ID"
  project_id  = "YOUR_PROJECT_ID" # this is scoping project set in step 8.
}

output "subscription" {
  description = "The Pub/Sub subscription created by this module."
  value       = module.observe_gcp_collection.subscription
}

output "service_account_private_key" {
  description = "A service account key sent to the pollers for Pub/Sub and Cloud Monitoring"
  value       = base64decode(module.observe_gcp_collection.service_account_key.private_key)
  sensitive   = true
}
```

```hcl
provider "google" {
  project = "YOUR_PROJECT_ID"
  region  = "YOUR_DEFAULT_REGION"
}

module "observe_gcp_collection" {
  source   = "observeinc/collection/google"
  name     = "dev"
  resource = "projects/YOUR_PROJECT_ID"
  project_id  = "YOUR_PROJECT_ID"
}

output "subscription" {
  description = "The Pub/Sub subscription created by this module."
  value       = module.observe_gcp_collection.subscription
}

output "service_account_private_key" {
  description = "A service account key sent to the pollers for Pub/Sub and Cloud Monitoring"
  value       = base64decode(module.observe_gcp_collection.service_account_key.private_key)
  sensitive   = true
}
```

--------------------------------

### slice_array(array: array, start: const int64, [end: const int64]?) -> array

Source: https://docs.observeinc.com/docs/function-slice_array

Returns a subset of an array starting at the 'start' index and ending before the 'end' index. If 'end' is omitted, it returns the array from 'start' to the end.

```APIDOC
## slice_array(array, start, [end])

### Description
Returns a possibly smaller array, starting at the given `start` index, and including items up to but not including the `end` index. Indices start at 0; negative indices count from the end of the array.

### Parameters
- **array** (array) - The input array to slice.
- **start** (const int64) - The starting index (inclusive).
- **end** (const int64) - Optional. The ending index (exclusive).

### Usage
- `slice_array(ary, 1, 2)` returns elements from index 1 up to 2.
- `slice_array(ary, -2)` returns elements from the second-to-last index to the end.

### Notes
- The `start` and `end` values must be constant at compile time.
- If `end` <= `start`, or indices are out of bounds, an empty array is returned.
```

--------------------------------

### make_array_range(start: int64, end: int64) -> array of int64

Source: https://docs.observeinc.com/docs/function-make_array_range

Generates an array of consecutive integers starting at 'start' (inclusive) and ending before 'end' (exclusive).

```APIDOC
## make_array_range(start: int64, end: int64)

### Description
Builds an array of consecutive integers starting at `start` (inclusive) and ending before `end` (exclusive), stepping by 1. If `end` is less than or equal to `start`, the result is an empty array. The function is capped at a maximum of 1,000,000 elements.

### Parameters
- **start** (int64) - Required - The starting integer (inclusive).
- **end** (int64) - Required - The ending integer (exclusive).

### Returns
- **array of int64** - An array containing the generated sequence.

### Examples
```
make_col idx:make_array_range(0, 5)
```
Produces `[0,1,2,3,4]`.

```
make_col idx:make_array_range(10, 20)
```
Produces `[10,11,12,13,14,15,16,17,18,19]`.
```

--------------------------------

### Describe Kubernetes pod

Source: https://docs.observeinc.com/docs/collect-annotations-and-labels

Example output from a kubectl command showing pod labels and annotations.

```bash
$ kubectl describe pod my-otel-demo-adservice-86f8b6d9c6-l4hvh

Name:             my-otel-demo-adservice-86f8b6d9c6-l4hvh
Namespace:        default
Priority:         0
Service Account:  my-otel-demo
Node:             gke-cluster-5-default-pool-d1b464b0-4v86/10.128.0.15
Start Time:       Wed, 29 Jan 2025 03:08:33 +0000
Labels:           app.kubernetes.io/component=adservice
                  app.kubernetes.io/instance=my-otel-demo
                  app.kubernetes.io/name=my-otel-demo-adservice
                  opentelemetry.io/name=my-otel-demo-adservice
                  pod-template-hash=86f8b6d9c6
Annotations:      example.com/maintainer: teamA
...
```

--------------------------------

### Execute Ansible Playbook for Red Hat

Source: https://docs.observeinc.com/docs/ansible-on-linux

Example command to run the Ansible playbook for Red Hat systems and the expected output.

```text
$ ansible-playbook --private-key observe-agent-test.pem /etc/ansible/roles/agent/tasks/package.yaml

PLAY [Install Observe Agent] ************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************
ok: [54.202.68.91]
ok: [35.87.95.246]

TASK [add observe-agent yum source] *****************************************************************************************************************
ok: [54.202.68.91]
ok: [35.87.95.246]

TASK [install observe-agent package] ****************************************************************************************************************
changed: [54.202.68.91]
changed: [35.87.95.246]

PLAY RECAP ******************************************************************************************************************************************
35.87.95.246               : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
54.202.68.91               : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
```

--------------------------------

### substring(value: string, start: int64, [length: int64]?) -> string

Source: https://docs.observeinc.com/docs/function-substring

Returns a contiguous slice of a string. Non-negative start is a zero-based offset from the left, while negative start indexes from the right.

```APIDOC
## substring(value: string, start: int64, [length: int64]?) -> string

### Description
Returns a contiguous slice of a string. Non-negative `start` is a zero-based offset from the left edge of the string. Negative `start` follows SQL `SUBSTRING` rules for indexing from the right.

### Parameters
- **value** (string) - Required - The input string to slice.
- **start** (int64) - Required - The zero-based starting index.
- **length** (int64) - Optional - The number of characters to include in the slice.

### Returns
- **string** - The resulting substring or null if input is null.

### Examples
```
make_col rest:substring(message, 1)
make_col prefix:substring(trace_id, 0, 8)
```
```

--------------------------------

### Create OpenShift project

Source: https://docs.observeinc.com/docs/install-on-red-hat-openshift

Initializes a new project namespace for the Observe Agent.

```bash
oc new-project observe
```

--------------------------------

### Run Metricbeat with Configuration

Source: https://docs.observeinc.com/docs/elastic-beats

Execute Metricbeat using the specified configuration file to begin streaming metrics.

```console
$ metricbeat -e -c example.yaml
...
INFO    [publisher]     pipeline/module.go:113  Beat name: docs-example
INFO    instance/beat.go:468    metricbeat start running.
INFO    [monitoring]    log/log.go:117  Starting metrics logging every 30s
INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(elasticsearch(https://{OBSERVE_CUSTOMER}.collect.observeinc.com/v1/elastic))
INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
INFO    [publisher]     pipeline/retry.go:223     done
INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.0.0
INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(elasticsearch(https://{OBSERVE_CUSTOMER}.collect.observeinc.com/v1/elastic)) established
```

--------------------------------

### Install coreutils on macOS

Source: https://docs.observeinc.com/docs/send-net-application-data-to-observe

Installs the required coreutils dependency via Homebrew.

```bash
brew install coreutils
```

--------------------------------

### Install OpenTelemetry dependencies

Source: https://docs.observeinc.com/docs/send-ruby-application-data-to-observe

Use bundle to add the necessary SDK, instrumentation, and OTLP exporter gems to your project.

```bash
bundle add opentelemetry-sdk \
opentelemetry-instrumentation-all \
opentelemetry-exporter-otlp
```

--------------------------------

### Install Telegraf on Linux

Source: https://docs.observeinc.com/docs/install-the-mysql-app

Commands to install Telegraf on various Linux distributions.

```bash
sudo tee /etc/yum.repos.d/influxdb.repo > /dev/null <<EOT
[influxdb]
name = InfluxDB Repository - RHEL
baseurl = https://repos.influxdata.com/rhel/7/x86_64/stable/
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key
EOT
sudo yum install telegraf -y
```

```bash
wget -qO- https://repos.influxdata.com/influxdata-archive_compat.key | sudo apt-key add -
source /etc/os-release
if ! grep -Fq "deb https://repos.influxdata.com/${ID} ${CODENAME} stable" /etc/apt/sources.list.d/influxdb.list
then
  echo "deb https://repos.influxdata.com/${ID} ${CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
fi
sudo apt-get update
sudo apt-get install -y telegraf
sudo apt-get install -y ntp
```

```bash
wget -qO- https://repos.influxdata.com/influxdata-archive_compat.key | sudo apt-key add -
source /etc/os-release
if ! grep -Fq "deb https://repos.influxdata.com/${ID} ${CODENAME} stable" /etc/apt/sources.list.d/influxdb.list
then
  echo "deb https://repos.influxdata.com/${ID} ${CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
fi
sudo apt-get update
sudo apt-get install -y telegraf
sudo apt-get install -y ntp
```

```bash
cat <<EOF | sudo tee /etc/yum.repos.d/influxdb.repo
[influxdb]
name = InfluxDB Repository - RHEL \$releasever
baseurl = https://repos.influxdata.com/rhel/\$releasever/\$basearch/stable
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key
EOF
sudo yum install telegraf
```

--------------------------------

### Install OpenTelemetry operator

Source: https://docs.observeinc.com/docs/deploy-to-a-serverless-kubernetes-cluster

Install the OpenTelemetry operator Helm chart into the observe namespace.

```bash
helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
--set "manager.collectorImage.repository=ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s" \
--set admissionWebhooks.certManager.enabled=false \
--set admissionWebhooks.autoGenerateCert.enabled=true \
--namespace observe
```

--------------------------------

### Execute Ansible Playbook for Ubuntu/Debian

Source: https://docs.observeinc.com/docs/ansible-on-linux

Example command to run the Ansible playbook for Ubuntu/Debian systems and the expected output.

```text
$ ansible-playbook -i /etc/ansible/hosts/my.aws_ec2.ini --private-key observe-agent-test.pem /etc/ansible/roles/agent/tasks/package.yaml

PLAY [Observe Agent Getting Started Playbook] ******************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************
ok: [ansible-managed-node-1]
ok: [ansible-managed-node-2]

TASK [add observe-agent apt source] ****************************************************************************************************************************************************************
changed: [ansible-managed-node-1]
changed: [ansible-managed-node-2]
```

--------------------------------

### View Azure Account Details

Source: https://docs.observeinc.com/docs/install-and-configure-the-microsoft-azure-app

Example output showing the structure of the account information returned after login.

```text
    [
      {
        "cloudName": "AzureCloud",
        "homeTenantId": "########-####-####-####-############",
        "id": "########-####-####-####-############",
        "isDefault": true,
        "managedByTenants": [],
        "name": "Acme Inc",
        "state": "Enabled",
        "tenantId": "########-####-####-####-############",
        "user": {
          "name": "joe@somecompany.com",
          "type": "user"
        }
      }
    ]
```

--------------------------------

### Enable instrumentation in Deployment

Source: https://docs.observeinc.com/docs/auto-instrumentation-with-opentelemetry-operator-in-kubernetes

Example of a Deployment manifest with annotations to enable Go auto-instrumentation.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
      annotations:
        instrumentation.opentelemetry.io/inject-go: "observe/observe-auto-instrumentation"
        instrumentation.opentelemetry.io/otel-go-auto-target-exe: "/usr/local/bin/sample-app"
```

--------------------------------

### Find substring index with start offset

Source: https://docs.observeinc.com/docs/function-position

Searches for a substring starting from a specified character index.

```Opal
make_col colon_after: position(log_line, ":", 5)
```

--------------------------------

### Configure Storage Bucket

Source: https://docs.observeinc.com/docs/install

Create a storage bucket and assign necessary access permissions to the service account.

```text
gsutil mb -l US gs://${PROJECT_ID}-observe
gsutil iam ch serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com:objectCreator gs://${PROJECT_ID}-observe
gsutil iam ch serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com:objectViewer gs://${PROJECT_ID}-observe
```

--------------------------------

### Configure Tail Input Plugin

Source: https://docs.observeinc.com/docs/fluent-bit

Example configuration for reading logs from files using the tail plugin.

```text
[INPUT]
    name        tail
    path        /var/log/my/service/*.log
    path_key    filename

    # Additional options for logs with very long lines
    buffer_max_size  2MB
    skip_long_lines  on
    mem_buf_limit 10MB

    # Ignores files older than ignore_older.
    ignore_older "2d"
```

--------------------------------

### Install Fluent Bit on macOS

Source: https://docs.observeinc.com/docs/fluent-bit

Command to install Fluent Bit using the Homebrew package manager.

```console
$ brew install fluent-bit
```

--------------------------------

### Init Container Command

Source: https://docs.observeinc.com/docs/install-on-amazon-ecs-fargate-sidecar-pattern

Shell script for the init container to install dependencies, prepare directories, and download the agent configuration.

```shell
set -euo pipefail; 
apk add --no-cache aws-cli; 
mkdir -p /mnt/shared-config/; 
mkdir -p /mnt/shared-logs/logs; 
aws s3 cp s3://my-config-bucket/observe-agent/observe-agent.yaml /mnt/shared-config/observe-agent.yaml || true; 
chmod 0644 /mnt/shared-config/observe-agent.yaml || true; 
chmod -R 0777 /mnt/shared-logs /mnt/shared-config || true; 
echo "Init Completed Successfully"; 
exit 0
```

--------------------------------

### Configure pod resource values

Source: https://docs.observeinc.com/docs/tune-service-resource-requests-and-limits

Example configuration for setting specific CPU and memory requests and limits in the Helm chart.

```yaml
resources:
  requests:
    cpu: 250m
    memory: 375Mi
  limits:
    memory: 375Mi
```

--------------------------------

### Configure OTLP endpoint via environment variables

Source: https://docs.observeinc.com/docs/deploy-to-multiple-clusters-using-rancher

Example configuration for the OpenTelemetry Astronomy Shop Demo app using an environment variable override.

```yaml
default:
  envOverrides:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: 'http://observe-agent-forwarder.observe.svc.cluster.local:4318'
```

--------------------------------

### Define Windows agent installation playbook

Source: https://docs.observeinc.com/docs/ansible-on-windows

A playbook to target Windows hosts and execute the agent installation role.

```yaml
---
# install-observe-agent-windows.yaml
#
# Launch from the control node:
#
#   ansible-playbook install-observe-agent-windows.yaml \
#     -i /etc/ansible/hosts/windows.aws_ec2.ini \
#     --ask-vault-pass            # only if token / password are vaulted
#
# What this playbook does
# ─────────────────────────────────────────────────────────
# • Targets every host in the [windows] group (defined in
#   windows.aws_ec2.ini).
# • Loads variables from /etc/ansible/hosts/group_vars/windows.yaml
#   – YOUR_OBSERVE_COLLECTION_ENDPOINT
#   – YOUR_INGEST_TOKEN
# • Executes the “agent” role, whose sole task-file
#   (install_with_ps1.yaml) does:
#     1. Download the latest install.ps1 from GitHub
#     2. Run the script with token & endpoint
#        – If the agent is missing, it installs it.
#        – If the agent is older, it upgrades in place.
#     3. Reboot only when the script returns exit-code 3010
#
# Idempotent: re-running the playbook simply ensures the
# agent is at the newest version and running.

- name: Install or upgrade Observe Agent on Windows fleet
  hosts: windows
  gather_facts: no      # WinRM fact-gathering not required here

  roles:
    - agent              # looks under /etc/ansible/roles/agent/
```

--------------------------------

### Slice an array with start and end indices

Source: https://docs.observeinc.com/docs/function-slice_array

Extracts elements starting from index 1 up to, but not including, index 2.

```Opal
make_col arr:slice_array(make_array('a','b','c','d'), 1, 2)
```

--------------------------------

### Initialize Traceloop SDK

Source: https://docs.observeinc.com/docs/use-nodejs-server-instrumentation-for-llm-observability

Initialize the SDK with your service name.

```javascript
traceloop.initialize(  
    appName: "<YOUR_SERVICE_NAME>",
  );
```

--------------------------------

### Ceil function usage examples

Source: https://docs.observeinc.com/docs/function-ceil

Examples demonstrating rounding up to the next integer and rounding up to a specific decimal precision.

```Opal
make_col slots:ceil(@.utilization)
```

```Opal
make_col billed:ceil(@.usage_hours, 2)
```

--------------------------------

### Install Telegraf on Ubuntu 20.04 and Debian 11

Source: https://docs.observeinc.com/docs/install-the-mysql-app

Commands to add the InfluxDB repository and install Telegraf on Debian-based systems.

```bash
wget -qO- https://repos.influxdata.com/influxdata-archive_compat.key | sudo apt-key add -
source /etc/os-release
if ! grep -Fq "deb https://repos.influxdata.com/${ID} ${CODENAME} stable" /etc/apt/sources.list.d/influxdb.list
then
  echo "deb https://repos.influxdata.com/${ID} ${CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
fi
sudo apt-get update
sudo apt-get install -y telegraf
sudo apt-get install -y ntp
```

--------------------------------

### Install OpenTelemetry NuGet Packages

Source: https://docs.observeinc.com/docs/send-net-application-data-to-observe

Run these commands to add the core OpenTelemetry instrumentation packages to your .NET project.

```bash
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package OpenTelemetry.Instrumentation.Runtime
dotnet add package OpenTelemetry.Instrumentation.Process
```

--------------------------------

### Implement Metric Interface

Source: https://docs.observeinc.com/docs/verb-interface

Examples of implementing the 'metric' interface by either matching column names or explicitly binding them.

```opal
make_col metric:string(FIELDS.metric_name), value:float64(FIELDS.sample_value)
interface "metric"
```

```opal
interface "metric", metric:metricNameColumn, value:metricValueColumn
```

--------------------------------

### Install Observe Agent on Ubuntu/Debian

Source: https://docs.observeinc.com/docs/ansible-on-linux

Ansible task configuration to add the Observe apt repository and install the agent package.

```yaml
- name : Install Observe Agent
  hosts: all
  become: yes
  tasks:
    - name: add observe-agent apt source
      ansible.builtin.copy:
        dest: /etc/apt/sources.list.d/observeinc.list
        mode: 0644
        content: |
          deb [trusted=yes] https://repo.observeinc.com/apt/ /
    - name: install observe-agent package
      ansible.builtin.apt:
        install_recommends: no
        name: observe-agent
        state: latest
        update_cache: yes   
```

--------------------------------

### Implement Log Interface

Source: https://docs.observeinc.com/docs/verb-interface

Example of implementing the 'log' interface by extracting a log message field.

```opal
make_col log:string(FIELDS.log_msg)
interface "log"
```

--------------------------------

### Install cert-manager via Helm

Source: https://docs.observeinc.com/docs/auto-instrumentation-with-opentelemetry-operator-in-kubernetes

Adds the Jetstack repository and installs cert-manager with CRDs enabled to manage TLS certificates.

```bash
# Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io --force-update

# Install cert-manager with CRDs
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.17.0 \
  --set crds.enabled=true
```

--------------------------------

### Validate Observe Agent Installation

Source: https://docs.observeinc.com/docs/windows

Use these commands to check if the ObserveAgent service is running and to verify the agent status from the installation directory.

```text
Get-Service ObserveAgent
Set-Location "${Env:Programfiles}\Observe\observe-agent"
./observe-agent status
```

--------------------------------

### Legacy multiline log records example

Source: https://docs.observeinc.com/docs/configure-the-observe-agent-on-linux-windows-and-macos

Sample log output for older agent versions requiring manual pattern matching.

```log
[2024-11-09T05:47:56.722293Z] INFO: User login successful. User ID: 12345
[2024-11-09T06:00:23.982100Z] WARNING: API request failed.
Error: ConnectionTimeout
Endpoint: /api/data/fetch
Retrying in 10 seconds...
[2024-11-09T06:01:13.742852Z] DEBUG: Starting backup process.
Directory: /data/backup
Estimated files: 543
[2024-11-09T06:02:45.123812Z] ERROR: Failed to load configuration file.
File path: /etc/app/config.yaml
Cause: FileNotFoundError
Stack trace:
    File "/app/main.py", line 23, in load_config
    config = open(config_path, 'r')
FileNotFoundError: [Errno 2] No such file or directory: '/etc/app/config.yaml'
```

--------------------------------

### Configure Windows Inventory File

Source: https://docs.observeinc.com/docs/ansible-on-windows

Example inventory configuration for Windows hosts using WinRM over HTTPS.

```text
; --------------------------
; WINDOWS INVENTORY EXAMPLE
; --------------------------

[win_proxy_servers]
ansible-managed-node-2 ansible_host=172.31.34.10

[win_load_balancer]
ansible-managed-node-1 ansible_host=172.31.35.121

[windows:children]
win_proxy_servers
win_load_balancer

[windows:vars]
; use WinRM instead of ssh
ansible_connection=winrm
; 5986 = HTTPS listener (recommended)
ansible_port=5986
ansible_winrm_scheme=https
; escape backslash in INI
ansible_user=Administrator
; store in Ansible Vault in real life 
ansible_password=<YOUR_PASSWORD>
; or ntlm / credssp / basic   
ansible_winrm_transport=ntlm
; OK for self-signed lab certs
ansible_winrm_server_cert_validation=ignore
```

--------------------------------

### Install OpenTelemetry libraries via Composer

Source: https://docs.observeinc.com/docs/send-php-application-data-to-observe

Use these commands to install the necessary OpenTelemetry SDK and auto-instrumentation packages for your PHP application.

```bash
composer require \
    open-telemetry/sdk \
    open-telemetry/exporter-otlp \
    open-telemetry/opentelemetry-auto-slim \
    open-telemetry/opentelemetry-auto-psr18
```

```bash
composer require \
    open-telemetry/sdk \
    open-telemetry/exporter-otlp \
    open-telemetry/opentelemetry-auto-wordpress
```

--------------------------------

### Run Docker Compose

Source: https://docs.observeinc.com/docs/install-the-observe-agent-as-a-docker-image-1

Start the Observe Agent container using the specified Docker Compose file.

```bash
docker compose -f compose-debian-host.yaml -p observe-agent up
```

--------------------------------

### Install Telegraf on Amazon Linux 2 and 2023

Source: https://docs.observeinc.com/docs/install-the-mysql-app

Commands to add the InfluxDB repository and install Telegraf on Amazon Linux systems.

```bash
sudo tee /etc/yum.repos.d/influxdb.repo > /dev/null <<EOT
[influxdb]
name = InfluxDB Repository - RHEL
baseurl = https://repos.influxdata.com/rhel/7/x86_64/stable/
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key
EOT
sudo yum install telegraf -y
```
Tokens:
Raw