# Elastic Integrations

Elastic Integrations is a repository containing sources for Elastic Packages that define how to observe specific products and services with the Elastic Stack. Each integration package includes configuration for the Elastic Agent, Elasticsearch ingest pipelines, field mappings, Kibana dashboards, and comprehensive documentation to enable end-to-end data collection and visualization.

The integrations follow a well-defined package specification and are built using the `elastic-package` CLI tool. Packages are published to the Package Registry and served to Kibana's Fleet UI, where users can discover, install, and configure them to collect logs, metrics, and other observability data from hundreds of different sources including cloud providers, databases, web servers, security tools, and custom applications.

## Package Manifest (manifest.yml)

The package manifest defines integration metadata including name, version, description, categories, Kibana version constraints, subscription requirements, policy templates, and owner information. This is the primary configuration file that controls how the integration appears and behaves in Fleet.

```yaml
# packages/myapp/manifest.yml
format_version: 3.4.0
name: myapp
title: "My Application"
version: 1.0.0
description: Collect logs and metrics from My Application with Elastic Agent.
type: integration
categories:
  - observability
  - web
conditions:
  kibana:
    version: "^8.13.0 || ^9.0.0"
  elastic:
    subscription: basic
icons:
  - src: /img/logo.svg
    title: My Application logo
    size: 32x32
    type: image/svg+xml
screenshots:
  - src: /img/dashboard-overview.png
    title: Overview Dashboard
    size: 1920x1080
    type: image/png
policy_templates:
  - name: myapp
    title: My Application logs and metrics
    description: Collect logs and metrics from My Application instances
    inputs:
      - type: logfile
        title: Collect logs from My Application
        description: Collecting application access and error logs
      - type: myapp/metrics
        vars:
          - name: hosts
            type: text
            title: Hosts
            multi: true
            required: true
            show_user: true
            default:
              - http://127.0.0.1:8080
        title: Collect metrics from My Application
        description: Collecting application performance metrics
owner:
  github: elastic/your-team
  type: elastic
```

## Create New Package

Use the `elastic-package create package` command to bootstrap a new integration package with all required directory structure and configuration files.

```bash
# Create a new integration package
elastic-package create package

# Interactive prompts:
# Package type: integration
# Package name: sample
# Version: 0.0.1
# License: Elastic-2.0
# Package title: My sample package
# Description: My first integrations package to collect logs
# Categories: custom
# Kibana version constraint: ^8.12.2
# Required Elastic subscription: basic
# Github owner: <your github name>
# Owner type: elastic

# Navigate to package directory
cd sample

# Validate package structure
elastic-package lint

# Format package files
elastic-package format

# Build the package
elastic-package build

# Or run all checks at once
elastic-package check
```

## Data Stream Manifest

Data streams define how specific types of data (logs, metrics) are collected, processed, and stored. Each data stream has its own manifest, ingest pipeline, field definitions, and agent configuration.

```yaml
# packages/myapp/data_stream/access/manifest.yml
title: My Application access logs
type: logs
streams:
  - input: logfile
    vars:
      - name: paths
        type: text
        title: Paths
        multi: true
        required: true
        show_user: true
        default:
          - /var/log/myapp/access.log*
      - name: tags
        type: text
        title: Tags
        multi: true
        required: true
        show_user: false
        default:
          - myapp-access
      - name: preserve_original_event
        required: true
        show_user: true
        title: Preserve original event
        description: Preserves a raw copy of the original event, added to the field `event.original`
        type: bool
        multi: false
        default: false
      - name: processors
        type: yaml
        title: Processors
        multi: false
        required: false
        show_user: false
        description: >
          Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata.
      - name: ignore_older
        type: text
        title: Ignore events older than
        default: 72h
        required: false
        show_user: false
        description: >-
          If this option is specified, events that are older than the specified amount of time are ignored.
    title: My Application access logs
    description: Collect My Application access logs
```

## Create Data Stream

Use the `elastic-package create data-stream` command to add new data streams to your integration package.

```bash
# Create a new data stream
elastic-package create data-stream

# Interactive prompts:
# Data stream name: log
# Data stream title: My log lines
# Type: logs

# This creates the following structure:
# data_stream/
#   log/
#     manifest.yml
#     fields/
#       base-fields.yml
#       fields.yml
#     elasticsearch/
#       ingest_pipeline/
#         default.yml
#     agent/
#       stream/
#         stream.yml.hbs

# Import ECS mappings by creating _dev/build/build.yml
cat > _dev/build/build.yml << 'EOF'
dependencies:
  ecs:
    reference: git@v8.6.0
    import_mappings: true
EOF

# Validate and reinstall
elastic-package check
elastic-package install
```

## Ingest Pipeline Definition

Ingest pipelines transform incoming data before indexing using processors like grok, dissect, date parsing, and field manipulation.

```yaml
# packages/myapp/data_stream/access/elasticsearch/ingest_pipeline/default.yml
description: Pipeline for parsing My Application access logs
processors:
  - set:
      field: event.ingested
      value: '{{_ingest.timestamp}}'
  - set:
      field: ecs.version
      value: '8.11.0'
  - rename:
      field: message
      target_field: event.original
      ignore_missing: true
      if: 'ctx.event?.original == null'
  # Parse log line using dissect for fixed format
  - dissect:
      field: event.original
      pattern: "%{@timestamp} %{log.level} %{message}"
  # Or use grok for complex patterns
  - grok:
      field: event.original
      patterns:
        - '%{IPORHOST:source.ip} - %{USER:user.name} \[%{HTTPDATE:timestamp}\] "%{WORD:http.request.method} %{DATA:url.path} HTTP/%{NUMBER:http.version}" %{NUMBER:http.response.status_code:long} %{NUMBER:http.response.body.bytes:long}'
      ignore_missing: true
  # Parse timestamps
  - date:
      field: timestamp
      target_field: '@timestamp'
      formats:
        - dd/MMM/yyyy:HH:mm:ss Z
        - ISO8601
        - UNIX_MS
      on_failure:
        - append:
            field: error.message
            value: '{{ _ingest.on_failure_message }}'
  # Extract JSON fields
  - json:
      field: message
      target_field: parsed
      add_to_root: true
      on_failure:
        - append:
            field: error.message
            value: "Failed to parse JSON: {{{ _ingest.on_failure_message }}}"
  # Set event categorization
  - set:
      field: event.kind
      value: event
  - append:
      field: event.category
      value: web
  - set:
      field: event.outcome
      value: success
      if: "ctx?.http?.response?.status_code != null && ctx.http.response.status_code < 400"
  - set:
      field: event.outcome
      value: failure
      if: "ctx?.http?.response?.status_code != null && ctx.http.response.status_code >= 400"
  # GeoIP enrichment
  - geoip:
      field: source.ip
      target_field: source.geo
      ignore_missing: true
  # User agent parsing
  - user_agent:
      field: user_agent.original
      ignore_missing: true
on_failure:
  - set:
      field: error.message
      value: '{{ _ingest.on_failure_message }}'
```

## Field Mappings Definition

Field mappings define the data types and properties for each field in your data stream. Use ECS fields when possible for consistency.

```yaml
# packages/myapp/data_stream/access/fields/fields.yml
- name: myapp
  type: group
  fields:
    - name: access
      type: group
      fields:
        - name: ssl.protocol
          type: keyword
          description: |
            SSL protocol version.
        - name: response_time
          type: long
          description: |
            Response time in milliseconds.
        - name: remote_ip_list
          type: ip
          description: |
            List of remote IP addresses.

# packages/myapp/data_stream/access/fields/ecs.yml
- external: ecs
  name: '@timestamp'
- external: ecs
  name: message
- external: ecs
  name: source.ip
- external: ecs
  name: source.geo
- external: ecs
  name: http.request.method
- external: ecs
  name: http.response.status_code
- external: ecs
  name: url.path
- external: ecs
  name: user_agent.original
- external: ecs
  name: event.kind
- external: ecs
  name: event.category
- external: ecs
  name: event.outcome

# packages/myapp/data_stream/access/fields/base-fields.yml
- name: data_stream.type
  type: constant_keyword
  description: Data stream type.
- name: data_stream.dataset
  type: constant_keyword
  description: Data stream dataset.
- name: data_stream.namespace
  type: constant_keyword
  description: Data stream namespace.
- name: event.module
  type: constant_keyword
  description: Event module.
  value: myapp
- name: event.dataset
  type: constant_keyword
  description: Event dataset.
  value: myapp.access
```

## Spin Up Elastic Stack

Use `elastic-package stack` to start a local Elastic Stack environment for development and testing.

```bash
# Start full Elastic Stack (Elasticsearch, Kibana, Package Registry)
elastic-package stack up --version=8.12.2 -v

# Access:
# - Kibana: https://localhost:5601 (user: elastic, password: changeme)
# - Elasticsearch: https://localhost:9200
# - Package Registry: http://localhost:8080

# Update to latest stack version
elastic-package stack update --version=8.12.2 -v

# Start only specific services
elastic-package stack up -d --services=elasticsearch

# Check stack status
elastic-package stack status

# View service logs
elastic-package stack dump

# Stop the stack
elastic-package stack down
```

## Install Package to Kibana

Install your integration package to Kibana for testing and validation.

```bash
# Build and install package (Kibana 8.7.0+)
elastic-package install

# Install specific zip file
elastic-package install --zip /path/to/myapp-1.0.0.zip -v

# Install to remote Kibana instance
export ELASTIC_PACKAGE_KIBANA_HOST="https://my-kibana.example.com:5601"
export ELASTIC_PACKAGE_ELASTICSEARCH_USERNAME="elastic"
export ELASTIC_PACKAGE_ELASTICSEARCH_PASSWORD="changeme"
export ELASTIC_PACKAGE_CA_CERT=""
elastic-package install -v

# For Kibana < 8.7.0, build first then restart package registry
elastic-package build -v
elastic-package stack up -v -d --services package-registry
elastic-package install -v

# Uninstall package
elastic-package uninstall
```

## Pipeline Testing

Pipeline tests validate your ingest pipelines by processing test data and comparing output against expected results.

```bash
# Start only Elasticsearch for pipeline tests
elastic-package stack up -d --services=elasticsearch

# Run pipeline tests for current package
elastic-package test pipeline

# Run tests for specific data streams
elastic-package test pipeline --data-streams access,error

# Generate expected results (first time or after pipeline changes)
elastic-package test pipeline --generate

# Run with verbose output
elastic-package test pipeline -v --report-format human
```

```json
// packages/myapp/data_stream/access/_dev/test/pipeline/test-access.json
{
  "events": [
    {
      "@timestamp": "2024-01-15T10:30:00.000Z",
      "message": "127.0.0.1 - admin [15/Jan/2024:10:30:00 +0000] \"GET /api/users HTTP/1.1\" 200 1234"
    },
    {
      "@timestamp": "2024-01-15T10:31:00.000Z",
      "message": "192.168.1.100 - - [15/Jan/2024:10:31:00 +0000] \"POST /api/login HTTP/1.1\" 401 89"
    }
  ]
}
```

```yaml
# packages/myapp/data_stream/access/_dev/test/pipeline/test-access.json-config.yml
fields:
  "@timestamp": "2024-01-15T10:30:00.000Z"
  ecs.version: "8.11.0"
  event.dataset: "myapp.access"

dynamic_fields:
  source.ip: "^\\d+\\.\\d+\\.\\d+\\.\\d+$"

numeric_keyword_fields:
  - http.response.status_code
```

```json
// packages/myapp/data_stream/access/_dev/test/pipeline/test-access.json-expected.json
{
  "expected": [
    {
      "@timestamp": "2024-01-15T10:30:00.000Z",
      "event": {
        "category": ["web"],
        "dataset": "myapp.access",
        "outcome": "success"
      },
      "http": {
        "request": { "method": "GET" },
        "response": { "status_code": 200, "body": { "bytes": 1234 } },
        "version": "1.1"
      },
      "source": { "ip": "127.0.0.1" },
      "url": { "path": "/api/users" },
      "user": { "name": "admin" }
    }
  ]
}
```

## System Testing

System tests validate end-to-end data flow from your service through Elastic Agent to Elasticsearch.

```bash
# Start full Elastic Stack
elastic-package stack up -d

# Run system tests
elastic-package test system

# Run for specific data streams
elastic-package test system --data-streams access

# Generate sample events during testing
elastic-package test system --generate

# Keep services running for debugging
elastic-package test system --defer-cleanup 10m

# Run with verbose output
elastic-package test system -v --report-format human
```

```yaml
# packages/myapp/_dev/deploy/docker/docker-compose.yml
version: '2.3'
services:
  myapp:
    image: myapp:latest
    ports:
      - "8080"
    volumes:
      - ${SERVICE_LOGS_DIR}:/var/log/myapp
    environment:
      - LOG_LEVEL=info
```

```yaml
# packages/myapp/data_stream/access/_dev/test/system/test-default-config.yml
vars: ~
input: logfile
data_stream:
  vars:
    paths:
      - "{{SERVICE_LOGS_DIR}}/access.log*"
    exclude_files: [".gz$"]
```

## Simulate Ingest Pipeline API

Test ingest pipelines directly using the Elasticsearch Simulate Pipeline API in Kibana Dev Tools.

```bash
# Test pipeline via Elasticsearch API
POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "Test pipeline",
    "processors": [
      {
        "dissect": {
          "field": "message",
          "pattern": "%{@timestamp} %{log.level} %{message}"
        }
      },
      {
        "date": {
          "field": "@timestamp",
          "formats": ["yyyy-MM-dd'T'HH:mm:ss.SSSZ", "ISO8601"]
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "logs-myapp.access-default",
      "_id": "1",
      "_source": {
        "message": "2024-01-15T10:30:00.000+0000 INFO User logged in successfully"
      }
    }
  ]
}

# Expected response:
{
  "docs": [
    {
      "doc": {
        "_index": "logs-myapp.access-default",
        "_id": "1",
        "_source": {
          "@timestamp": "2024-01-15T10:30:00.000Z",
          "message": "User logged in successfully",
          "log": { "level": "INFO" }
        },
        "_ingest": {
          "timestamp": "2024-01-15T10:30:00.000Z"
        }
      }
    }
  ]
}
```

## Index Documents and Query Data

Manually index test documents and query data to verify integration setup.

```bash
# Index a test document
POST logs-myapp.access-default/_doc
{
  "message": "2024-01-15T10:30:00.000+0000 INFO Request completed",
  "@timestamp": "2024-01-15T10:30:00.000Z"
}

# Response:
{
  "_index": ".ds-logs-myapp.access-default-2024.01.15-000001",
  "_id": "abc123",
  "_version": 1,
  "result": "created",
  "_shards": { "total": 2, "successful": 1, "failed": 0 }
}

# Search indexed documents
GET logs-myapp.access-default/_search
{
  "query": {
    "match_all": {}
  },
  "size": 10
}

# Check data stream info
GET _data_stream/logs-myapp.access-default

# View index mappings
GET logs-myapp.access-default/_mapping

# Check pipeline stats
GET _nodes/stats/ingest
```

## Export Kibana Dashboards

Create dashboards in Kibana and export them to your integration package.

```bash
# Start service stack for dashboard development
elastic-package service

# Build and install package
elastic-package build
elastic-package install

# Create dashboards in Kibana UI at https://localhost:5601
# Use Lens for visualizations when possible

# Export dashboards and dependencies to package
elastic-package export

# Exported files go to:
# packages/myapp/kibana/
#   dashboard/
#     myapp-overview.json
#   visualization/
#     myapp-requests-chart.json
#   search/
#     myapp-access-logs.json
```

```text
# Dashboard naming convention
[Metrics MYAPP] Overview
[Logs MYAPP] Access Logs

# Visualization naming (without package prefix)
Requests per Second
Error Rate Gauge
Response Time Distribution

# Add navigation between dashboards using Markdown
[Overview](#/dashboard/myapp-overview) | **[Access Logs](#/dashboard/myapp-access-logs)** | [Errors](#/dashboard/myapp-errors)
```

## Standalone Elastic Agent Configuration

Configure a standalone Elastic Agent to collect data using your integration.

```yaml
# elastic-agent.yml
inputs:
  - type: logfile
    streams:
      - data_stream:
        dataset: myapp.access
        paths:
          - /var/log/myapp/access.log*
        tags:
          - myapp-access
        processors:
          - add_fields:
              target: ''
              fields:
                ecs.version: '8.11.0'

  - type: myapp/metrics
    streams:
      - data_stream:
          dataset: myapp.metrics
        hosts:
          - http://localhost:8080/metrics
        period: 30s
        username: monitor
        password: "${MYAPP_PASSWORD}"

outputs:
  default:
    type: elasticsearch
    hosts: ["https://localhost:9200"]
    username: "elastic"
    password: "changeme"
    ssl.verification_mode: none

# Or use API key authentication
outputs:
  default:
    type: elasticsearch
    hosts: ["https://localhost:9200"]
    api_key: "base64_encoded_api_key"
```

```bash
# Run standalone Elastic Agent
sudo ./elastic-agent -e

# Enroll agent with Fleet
./elastic-agent enroll https://kibana:5601 <enrollment-token>
./elastic-agent run
```

## Package Version and Changelog Management

Update package version and maintain changelog for releases.

```yaml
# packages/myapp/manifest.yml
format_version: 3.4.0
name: myapp
title: "My Application"
version: 1.2.0  # Update version following semver
# ...
```

```yaml
# packages/myapp/changelog.yml
- version: "1.2.0"
  changes:
    - description: Added support for JSON log format
      type: enhancement
      link: https://github.com/elastic/integrations/pull/12345
    - description: Fixed timestamp parsing for ISO8601 format
      type: bugfix
      link: https://github.com/elastic/integrations/pull/12346
- version: "1.1.0"
  changes:
    - description: Added error log data stream
      type: enhancement
      link: https://github.com/elastic/integrations/pull/12340
- version: "1.0.0"
  changes:
    - description: Initial release
      type: enhancement
      link: https://github.com/elastic/integrations/pull/12300
```

```bash
# Update changelog automatically
elastic-package changelog add --description "Added new feature" --type enhancement --link https://github.com/elastic/integrations/pull/12347

# Version types:
# - Patch (x.y.Z): Backward-compatible bug fixes
# - Minor (x.Y.z): Backward-compatible new features
# - Major (X.y.z): Breaking changes

# Validate package before release
elastic-package check
```

## Summary

Elastic Integrations provides a comprehensive framework for building data collection packages that work seamlessly with the Elastic Stack. The primary use cases include creating integrations for cloud services (AWS, Azure, GCP), databases (MySQL, PostgreSQL, MongoDB), web servers (Nginx, Apache), security tools, and custom applications. Developers use the `elastic-package` CLI to scaffold packages, define data streams with ingest pipelines and field mappings, create Kibana dashboards, and run comprehensive tests before publishing.

The integration patterns follow a consistent structure: a package manifest defines metadata and policy templates, data streams specify how different types of data are collected, ingest pipelines transform raw data into structured documents following ECS conventions, and field mappings ensure proper indexing in Elasticsearch. Testing is crucial, with pipeline tests validating data transformation logic and system tests verifying end-to-end data flow. Once built and tested, integrations are published to the Package Registry where Fleet users can easily discover and deploy them to monitor their infrastructure and applications.