### Minimal pino-elasticsearch Setup Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/usage-patterns.md This is the most basic configuration to get pino-elasticsearch running. Ensure pino and pino-elasticsearch are installed. ```javascript const pino = require('pino') const pinoElasticsearch = require('pino-elasticsearch') const stream = pinoElasticsearch({ node: 'http://localhost:9200' }) const logger = pino(stream) logger.info('Hello, Elasticsearch!') ``` -------------------------------- ### Basic Pino-Elasticsearch Setup Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/api-reference-main.md Configure pino-elasticsearch to send logs to a local Elasticsearch instance. Ensure pino and pino-elasticsearch are installed. ```javascript const pino = require('pino') const pinoElasticsearch = require('pino-elasticsearch') const streamToElastic = pinoElasticsearch({ index: 'my-logs', node: 'http://localhost:9200' }) const logger = pino(streamToElastic) logger.info('Hello, Elasticsearch!') ``` -------------------------------- ### Basic Express.js Setup with Pino-Elasticsearch Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/integration-guide.md Integrate pino-elasticsearch into a basic Express application. This setup logs incoming requests and server start messages to Elasticsearch. ```javascript // express-app.js const express = require('express') const pino = require('pino') const pinoElasticsearch = require('pino-elasticsearch') const app = express() // Create Elasticsearch stream const elasticStream = pinoElasticsearch({ node: 'http://localhost:9200', index: 'express-logs-%{DATE}' }) // Create logger const logger = pino({ level: 'info' }, elasticStream) // Middleware to attach logger to request app.use((req, res, next) => { req.logger = logger next() }) // Log requests app.use((req, res, next) => { logger.info({ method: req.method, path: req.path, ip: req.ip }, 'Incoming request') next() }) // Routes app.get('/api/users/:id', (req, res) => { const userId = req.params.id req.logger.info({ userId }, 'Fetching user') res.json({ id: userId, name: 'John Doe' }) }) app.listen(3000, () => { logger.info('Server started on port 3000') }) ``` -------------------------------- ### Install pino-elasticsearch CLI Source: https://github.com/pinojs/pino-elasticsearch/blob/master/README.md Install the pino-elasticsearch command-line interface globally using npm. ```bash npm install pino-elasticsearch -g ``` -------------------------------- ### Install pino-elasticsearch Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/README.md Install the pino-elasticsearch package using npm. ```bash npm install pino-elasticsearch ``` -------------------------------- ### Display Version Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md Prints the installed version of the pino-elasticsearch CLI. ```bash pino-elasticsearch --version ``` -------------------------------- ### Configure Pino-Elasticsearch Stream Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/types.md Example of how to configure and create a pino-elasticsearch stream with various options. This setup specifies the Elasticsearch node, index name, buffer sizes, flush intervals, Elasticsearch version, and authentication credentials. ```javascript const options: Options = { node: 'http://localhost:9200', index: 'my-logs', flushBytes: 5000, flushInterval: 60000, esVersion: 8, auth: { username: 'user', password: 'pass' } } const stream = pinoElasticsearch(options) ``` -------------------------------- ### Basic ECS Logging to Elasticsearch Source: https://github.com/pinojs/pino-elasticsearch/blob/master/README.md Configure pino-elasticsearch with ECS formatting for structured logging. Requires installing `@elastic/ecs-pino-format`. ```js const pino = require('pino') const ecsFormat = require('@elastic/ecs-pino-format')() const pinoElastic = require('pino-elasticsearch') const streamToElastic = pinoElastic({ index: 'an-index', node: 'http://localhost:9200', esVersion: 7, flushBytes: 1000 }) const logger = pino({ level: 'info', ...ecsFormat() }, streamToElastic) logger.info('hello world') // ... ``` -------------------------------- ### Set Up Production Logging Pipeline with CLI Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/usage-patterns.md Use a bash script to pipe application logs to pino-elasticsearch for production. This example configures node, index, API key, and flush settings, ensuring required environment variables are set. ```bash #!/bin/bash # production-logs.sh ES_NODE=${ELASTICSEARCH_NODE:-https://elasticsearch.example.com:9200} ES_INDEX=${ELASTICSEARCH_INDEX:-app-logs-%{DATE}} ES_API_KEY=${ELASTICSEARCH_API_KEY:?} node app.js | pino-elasticsearch \ --node "$ES_NODE" \ --index "$ES_INDEX" \ --api-key "$ES_API_KEY" \ --flush-bytes 50000 \ --flush-interval 60000 ``` -------------------------------- ### Pipe Application Logs to pino-elasticsearch with Environment Variables Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md This example shows how to pipe the output of a Node.js application (app.js) to the elasticsearch-pipe.sh script, using environment variables to configure the Elasticsearch connection details. ```bash ES_NODE=https://elastic.example.com:9200 \ ES_INDEX=prod-logs \ ES_API_KEY=myapikey \ node app.js | ./elasticsearch-pipe.sh ``` -------------------------------- ### Pino-Elasticsearch Module Quick Start Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/00_START_HERE.txt This snippet shows how to initialize the pino-elasticsearch module for use within a Node.js application. It requires the 'pino' and 'pino-elasticsearch' modules and configures the Elasticsearch node. ```javascript const pino = require('pino') const elastic = require('pino-elasticsearch') const logger = pino(elastic({ node: 'http://localhost:9200' })) ``` -------------------------------- ### Logging Event Statistics Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/types.md Example of how to listen for the 'insert' event on a pino-elasticsearch stream and log statistics such as the number of indexed documents, time taken, and bytes processed. ```javascript stream.on('insert', (stats) => { console.log(`Indexed ${stats.successful} documents`) console.log(`Time: ${stats.time}ms`) console.log(`Bytes: ${stats.bytes}`) }) ``` -------------------------------- ### Pino-Elasticsearch CLI Quick Start Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/00_START_HERE.txt This snippet demonstrates how to use the pino-elasticsearch CLI tool to pipe log data from standard input to Elasticsearch. It requires logs in ndjson format and specifies the Elasticsearch node. ```bash cat logs.ndjson | pino-elasticsearch --node http://localhost:9200 ``` -------------------------------- ### Listening to DestinationStream Events Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/types.md Example of how to attach event listeners to the DestinationStream to handle batch insertion statistics and errors. ```javascript const stream = pinoElasticsearch({ /* ... */ }) stream.on('insert', (stats) => { console.log('Batch inserted:', stats) }) stream.on('insertError', (error) => { console.log('Failed document:', error.document) }) ``` -------------------------------- ### Import Historical Logs into Elasticsearch Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/usage-patterns.md Use the 'cat' command to pipe log data from a file into pino-elasticsearch for archival or analysis. This example sets a large flush batch size for efficient import. ```bash # Import historical logs cat historical-logs.ndjson | pino-elasticsearch \ --node http://localhost:9200 \ --index imported-logs-%{DATE} \ --flush-bytes 1000000 # Large batches for speed ``` -------------------------------- ### Docker Compose Setup for Pino-Elasticsearch Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/integration-guide.md This Docker Compose configuration sets up an application service, Elasticsearch, and Kibana. The application service is configured to send logs to Elasticsearch using environment variables for the node and index name. Ensure Elasticsearch is accessible via the specified node URL. ```yaml # docker-compose.yml version: '3.8' services: app: build: . ports: - "3000:3000" environment: - ELASTICSEARCH_NODE=http://elasticsearch:9200 - ELASTICSEARCH_INDEX=app-logs-%{DATE} depends_on: - elasticsearch elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:8.3.0 environment: - discovery.type=single-node - xpack.security.enabled=false - "ES_JAVA_OPTS=-Xms512m -Xmx512m" ports: - "9200:9200" volumes: - elasticsearch-data:/usr/share/elasticsearch/data kibana: image: docker.elastic.co/kibana/kibana:8.3.0 ports: - "5601:5601" environment: - ELASTICSEARCH_HOSTS=http://elasticsearch:9200 volumes: elasticsearch-data: ``` -------------------------------- ### Configure Flush by Size Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/configuration.md Set the maximum buffer size in bytes before logs are automatically flushed to Elasticsearch. This example sends logs every 5KB. ```javascript { flushBytes: 5000 // Send every 5KB } ``` -------------------------------- ### Mock Elasticsearch Client for Testing Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/usage-patterns.md Utilize a mock Elasticsearch client when testing Pino-Elasticsearch configurations. This allows you to simulate Elasticsearch behavior without a live connection, ensuring your logging setup is correct. ```javascript const pinoElasticsearch = require('pino-elasticsearch') const pino = require('pino') // Mock client for testing class MockESClient { constructor(config) { this.config = config } get diagnostic() { return { on: () => {} } } get connectionPool() { return { resurrect: () => {} } } get helpers() { return { async bulk(opts) { // Simulate processing for await (const chunk of opts.datasource) { // Process each chunk } return { successful: 1, failed: 0 } } } } } const stream = pinoElasticsearch( { index: 'test-logs', node: 'http://localhost:9200' }, { Client: MockESClient } ) const logger = pino(stream) logger.info('Test log') ``` -------------------------------- ### Stream Logs to Elasticsearch Source: https://github.com/pinojs/pino-elasticsearch/blob/master/usage.txt Pipe logs from standard input to pino-elasticsearch, specifying the Elasticsearch node URL. This is the basic command to start sending logs. ```bash cat log | pino-elasticsearch --node http://localhost:9200 ``` -------------------------------- ### Configure Flush by Time Interval Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/configuration.md Set the interval in milliseconds for flushing logs to Elasticsearch, regardless of buffer size. This example sends logs every 60 seconds. ```javascript { flushInterval: 60000 // Send every 60 seconds } ``` -------------------------------- ### Integrate Pino-Elasticsearch with MongoDB Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/integration-guide.md Set up Pino logging to send messages to Elasticsearch, specifically logging MongoDB connection events. This example demonstrates connecting to a MongoDB database and logging connection status. ```javascript const mongoose = require('mongoose') const pino = require('pino') const pinoElasticsearch = require('pino-elasticsearch') const logger = pino(pinoElasticsearch({ node: 'http://localhost:9200', index: 'app-logs' })) mongoose.connect('mongodb://localhost/mydb', { serverSelectionTimeoutMS: 5000 }) mongoose.connection.on('connected', () => { logger.info('MongoDB connected') }) mongoose.connection.on('error', (err) => { logger.error({ err }, 'MongoDB connection error') }) ``` -------------------------------- ### Read Configuration from JavaScript File Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md Load all configuration options from a specified JavaScript file. ```bash pino-elasticsearch -r ./es-config.js ``` ```javascript module.exports = { node: 'https://elasticsearch.example.com:9200', index: 'my-logs-%{DATE}', flushBytes: 5000, flushInterval: 60000, esVersion: 8, auth: { apiKey: 'VnVhQ0JHd0JDSkR1cDI4QjJLVzpSMkR6T29FMlZfR3pqMjBkeGVfWnA=' } } ``` -------------------------------- ### Using a Configuration File Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md Specify pino-elasticsearch options by referencing a JSON configuration file. This simplifies complex command lines. ```bash cat logs.ndjson | pino-elasticsearch --read-config ./elasticsearch-config.json ``` -------------------------------- ### Read Configuration from JSON File Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md Load all configuration options from a specified JSON file. ```bash pino-elasticsearch --read-config config.json ``` ```json { "node": "https://elasticsearch.example.com:9200", "index": "my-logs-%{DATE}", "flushBytes": 5000, "flushInterval": 60000, "esVersion": 8, "username": "elastic", "password": "changeme" } ``` -------------------------------- ### Check Pino version Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/integration-guide.md Verify your installed Pino version to ensure compatibility. pino-elasticsearch works with Pino v8.0.0 and later. ```bash npm list pino ``` -------------------------------- ### Basic CLI Usage Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/configuration.md Pipe logs to pino-elasticsearch with a specified Elasticsearch node and index name. ```bash cat logs.ndjson | pino-elasticsearch --node http://localhost:9200 --index my-logs ``` -------------------------------- ### Pino-Elasticsearch Documentation Map Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/README.md Navigate the pino-elasticsearch documentation. This map outlines the structure for API, configuration, CLI, and integration guides. ```text START HERE ↓ [REFERENCE_INDEX.md] ← Quick navigation and best practices ↓ ├─→ Want API docs? → [API Reference - Main Function](api-reference-main.md) ├─→ Want to configure? → [Configuration Reference](configuration.md) ├─→ Want type info? → [Types and Interfaces](types.md) ├─→ Having errors? → [Errors and Events](errors.md) ├─→ Using CLI? → [CLI Reference](cli-reference.md) ├─→ Need recipes? → [Common Usage Patterns](usage-patterns.md) ├─→ Integrating? → [Integration Guide](integration-guide.md) └─→ Understanding internals? → [Architecture and Design](architecture.md) ``` -------------------------------- ### JSON Configuration File Format Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/configuration.md When using the `--read-config` flag, provide configuration options in a JSON file. Only specific keys are recognized. ```json { "node": "http://localhost:9200", "index": "my-logs-%{DATE}", "flushBytes": 5000, "flushInterval": 60000, "esVersion": 8 } ``` -------------------------------- ### CLI Usage with Authentication Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/configuration.md Configure pino-elasticsearch with username and password for basic authentication. ```bash cat logs.ndjson | pino-elasticsearch \ --node https://elasticsearch.example.com:9200 \ --username elastic \ --password changeme ``` -------------------------------- ### Continuous Monitoring with Log Reloading Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md Set up continuous monitoring by teeing logs to a file and then piping filtered logs to pino-elasticsearch. This allows for real-time analysis and error alerting. ```bash # Terminal 1: Run application node app.js | tee app-logs.ndjson | pino-elasticsearch --read-config config.json # Terminal 2: Monitor for errors tail -f app-logs.ndjson | jq 'select(.level >= 40)' | \ pino-elasticsearch --index error-logs --node http://localhost:9200 ``` -------------------------------- ### Distributed Logging with Pino-Elasticsearch Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/usage-patterns.md Configure Pino-Elasticsearch to send logs from a service to an Elasticsearch cluster. This setup includes base log information like hostname and service name. ```javascript const hostname = require('os').hostname() const serviceName = 'my-service' const stream = pinoElasticsearch({ node: 'http://elasticsearch:9200', index: 'app-logs-%{DATE}' }) const logger = pino({ base: { hostname, service: serviceName, version: '1.0.0' } }, stream) logger.info('Request received') // Logs include: hostname, service, version ``` -------------------------------- ### JavaScript Configuration File Format Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/configuration.md Alternatively, configuration can be provided in a JavaScript file using `module.exports`. Only specific keys are recognized. ```javascript module.exports = { node: 'http://localhost:9200', index: 'my-logs-%{DATE}', flushBytes: 5000, flushInterval: 60000, esVersion: 8 } ``` -------------------------------- ### Display Help Message Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md Shows the help message for the pino-elasticsearch CLI, listing all available commands and flags. ```bash pino-elasticsearch --help ``` -------------------------------- ### Express.js with Request ID Tracking Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/integration-guide.md Enhance Express logging with unique request IDs for better traceability. This setup includes middleware for generating IDs and logging request completion. ```javascript const express = require('express') const { v4: uuid } = require('uuid') const pino = require('pino') const pinoElasticsearch = require('pino-elasticsearch') const elasticStream = pinoElasticsearch({ node: 'http://localhost:9200', index: 'app-logs-%{DATE}' }) const logger = pino(elasticStream) const app = express() // Request ID middleware app.use((req, res, next) => { const requestId = uuid() req.logger = logger.child({ requestId }) res.setHeader('X-Request-ID', requestId) next() }) // Logging middleware app.use((req, res, next) => { req.logger.info({ method: req.method, path: req.path, query: req.query }, 'Request received') res.on('finish', () => { req.logger.info({ statusCode: res.statusCode, duration: Date.now() - req.startTime }, 'Request completed') }) next() }) app.get('/api/data', (req, res) => { req.logger.debug('Processing request') res.json({ success: true }) }) app.listen(3000) ``` -------------------------------- ### Integrate Pino-Elasticsearch with PostgreSQL Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/integration-guide.md Configure Pino logging to send database-related events from a PostgreSQL connection pool to Elasticsearch. This example shows how to log connection and error events from a pg.Pool. ```javascript const pg = require('pg') const pino = require('pino') const pinoElasticsearch = require('pino-elasticsearch') const logger = pino(pinoElasticsearch({ node: 'http://localhost:9200', index: 'db-logs' })) const pool = new pg.Pool({ host: 'localhost', database: 'mydb' }) pool.on('connect', () => { logger.info('Database pool connected') }) pool.on('error', (err) => { logger.error({ err }, 'Database pool error') }) ``` -------------------------------- ### Log Creation and Initial Serialization Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/architecture.md Create a logger instance with the Elasticsearch stream and log a message. Pino serializes the log to JSON and writes it to the stream. ```javascript const logger = pino(elasticsearchStream) logger.info({ userId: 123 }, 'User logged in') // Pino serializes to JSON and writes to stream ``` -------------------------------- ### CLI Usage with Elastic Cloud Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/configuration.md Connect to Elastic Cloud using a deployment name and API key. ```bash cat logs.ndjson | pino-elasticsearch \ --cloud deployment-name:base64String \ --api-key base64EncodedKey ``` -------------------------------- ### Stream Live Application Logs to Elasticsearch Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md Pipe application logs directly to pino-elasticsearch for real-time indexing. Ensure your Elasticsearch node is accessible. ```bash node app.js | pino-elasticsearch --node http://elasticsearch:9200 --index app-logs ``` -------------------------------- ### CLI Usage with Config File Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/configuration.md Specify a configuration file for pino-elasticsearch settings. ```bash cat logs.ndjson | pino-elasticsearch --read-config config.json ``` -------------------------------- ### Implement Sampling for High Frequency Events Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/integration-guide.md Use a custom sampling function to control the ingestion of high-frequency events, such as debug logs, to reduce Elasticsearch load. This example samples 10% of debug logs. ```javascript const pino = require('pino') const pinoElasticsearch = require('pino-elasticsearch') // Sample 10% of debug logs const shouldSendToES = (level) => { if (level === 10) { // DEBUG return Math.random() < 0.1 } return true } const elasticStream = pinoElasticsearch({ node: 'http://localhost:9200', index: 'app-logs' }) const logger = pino(elasticStream) // Custom sampling const sampleLogger = { trace: (obj, msg) => shouldSendToES(10) && logger.trace(obj, msg), debug: (obj, msg) => shouldSendToES(20) && logger.debug(obj, msg), info: (obj, msg) => logger.info(obj, msg), error: (obj, msg) => logger.error(obj, msg) } ``` -------------------------------- ### Basic CLI Command Syntax Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md The general syntax for using the pino-elasticsearch command. It reads from stdin and streams to Elasticsearch. ```bash pino-elasticsearch [flags] < logs.ndjson ``` -------------------------------- ### Koa Integration with Pino-Elasticsearch Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/integration-guide.md Integrates pino-elasticsearch into a Koa application. Sets up a logger middleware to log request start and completion details, including status code and duration. Requires Koa, koa-router, pino, and pino-elasticsearch. ```javascript const Koa = require('koa') const Router = require('koa-router') const pino = require('pino') const pinoElasticsearch = require('pino-elasticsearch') const elasticStream = pinoElasticsearch({ node: 'http://localhost:9200', index: 'koa-logs' }) const logger = pino(elasticStream) const app = new Koa() const router = new Router() // Logger middleware app.use(async (ctx, next) => { ctx.logger = logger.child({ path: ctx.path }) ctx.logger.info('Request started') const start = Date.now() await next() ctx.logger.info({ statusCode: ctx.status, duration: Date.now() - start }, 'Request completed') }) router.get('/api/users', async (ctx) => { ctx.logger.info('Fetching users') ctx.body = [{ id: 1, name: 'User 1' }] }) app.use(router.routes()) app.listen(3000) ``` -------------------------------- ### CLI Authentication with Username and Password Flags Source: https://github.com/pinojs/pino-elasticsearch/blob/master/README.md Connect to Elasticsearch using the CLI with basic authentication by providing username and password via separate flags. ```sh cat log | pino-elasticsearch --node https://localhost:9200 -u user -p pwd ``` -------------------------------- ### Configure pino-elasticsearch with Environment Variables Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md This script demonstrates how to use environment variables to dynamically configure the Elasticsearch node, index, and API key for the pino-elasticsearch CLI. It provides default values if variables are not set. ```bash #!/bin/bash # elasticsearch-pipe.sh ES_NODE="${ES_NODE:-http://localhost:9200}" ES_INDEX="${ES_INDEX:-app-logs}" ES_API_KEY="${ES_API_KEY:-}" if [ -z "$ES_API_KEY" ]; then pino-elasticsearch --node "$ES_NODE" --index "$ES_INDEX" else pino-elasticsearch --node "$ES_NODE" --index "$ES_INDEX" --api-key "$ES_API_KEY" fi ``` -------------------------------- ### Create New Index for Field Type Mismatch Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/errors.md To resolve field type mismatches (e.g., 'duration' as number vs. string), create a new index with the correct mapping. This example shows creating a new index daily. ```javascript // Problem: 'duration' field is number in new logs but string in index // Solution: Create new index with correct mapping const stream = pinoElasticsearch({ index: `logs-app-${new Date().toISOString().substring(0, 10)}`, // Fresh index per day node: 'http://localhost:9200' }) ``` -------------------------------- ### Unit Test with Mock Elasticsearch Client Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/integration-guide.md Demonstrates how to unit test pino-elasticsearch by mocking the Elasticsearch client. This allows for isolated testing of the logger's behavior without requiring a live Elasticsearch instance. ```javascript const pinoElasticsearch = require('pino-elasticsearch') const pino = require('pino') class MockESClient { constructor() { this.bulk = jest.fn() } get diagnostic() { return { on: jest.fn() } } get connectionPool() { return { resurrect: jest.fn() } } get helpers() { return { bulk: async (opts) => { for await (const _ of opts.datasource) { // Mock processing } return { successful: 1 } } } } } describe('Logging', () => { it('logs messages', async () => { const mockClient = new MockESClient() const stream = pinoElasticsearch( { index: 'test-logs' }, { Client: mockClient } ) const logger = pino(stream) logger.info('Test message') await new Promise(resolve => stream.on('insert', resolve)) }) }) ``` -------------------------------- ### Comprehensive Error Handling for Elasticsearch Stream Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/usage-patterns.md Implement detailed error handling for the pino-elasticsearch stream, including unknown lines, insert failures, connection errors, and monitoring successful inserts. This setup includes a fallback to save failed logs to a local file. ```javascript const stream = pinoElasticsearch({ node: 'http://localhost:9200', index: 'app-logs', flushBytes: 5000, flushInterval: 30000 }) // Handle unparseable lines stream.on('unknown', (line, error) => { console.error(`[unknown] ${error}: ${line}`) }) // Handle Elasticsearch insert failures stream.on('insertError', (error) => { console.error(`[insertError] Failed to index document:`, error.document) console.error(`[insertError] Error:`, error.message) // Fallback: save to local file fs.appendFileSync('failed-logs.json', JSON.stringify(error.document) + '\n') }) // Handle Elasticsearch connection errors stream.on('error', (error) => { console.error(`[error] Elasticsearch error:`, error.message) }) // Monitor successful inserts stream.on('insert', (stats) => { console.log(`[insert] Successfully indexed ${stats.successful} documents`) if (stats.failed > 0) { console.warn(`[insert] Failed: ${stats.failed} documents`) } }) const logger = pino(stream) logger.info('With comprehensive error handling') ``` -------------------------------- ### Connect to Local Elasticsearch Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md Streams logs from a file to a local Elasticsearch instance using default settings. ```bash cat logs.ndjson | pino-elasticsearch ``` -------------------------------- ### Pino-Elasticsearch Bulk Handler Initialization Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/architecture.md Initializes the bulk handler for sending logs to Elasticsearch. This involves parsing version-specific options, resolving index naming strategies, and setting up bulk operations. ```javascript function initializeBulkHandler (opts, client, splitter) { // 1. Parse version-specific options // 2. Resolve index naming strategy // 3. Create bulk operation // 4. Attach event listeners } ``` -------------------------------- ### CLI Usage with API Key Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/configuration.md Use an API key for authentication with pino-elasticsearch. ```bash cat logs.ndjson | pino-elasticsearch \ --node https://elasticsearch.example.com:9200 \ --api-key base64EncodedKey ``` -------------------------------- ### Console + Elasticsearch Logging (Transport) Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/usage-patterns.md Use Pino's transport method to log to both the console and Elasticsearch simultaneously. This is the recommended approach for multi-destination logging. ```javascript const pino = require('pino') const pinoElasticsearch = require('pino-elasticsearch') const logger = pino(pino.transport({ targets: [ { target: 'pino/file', options: { destination: process.stdout.fd } }, { target: 'pino-elasticsearch', options: { node: 'http://localhost:9200', index: 'app-logs' } } ] })) logger.info('Logged to both console and Elasticsearch') ``` -------------------------------- ### CLI Authentication with API Key Flag Source: https://github.com/pinojs/pino-elasticsearch/blob/master/README.md Connect to Elasticsearch using the CLI with API key authentication by providing the API key via the `--api-key` flag. ```sh cat log | pino-elasticsearch --node https://localhost:9200 --api-key=base64EncodedKey ``` -------------------------------- ### Configure Pino-Elasticsearch Directly from Environment Variables Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/usage-patterns.md Set Elasticsearch connection and logging parameters directly using environment variables. Fallback values are provided for common settings like node and index. ```javascript const stream = pinoElasticsearch({ node: process.env.ELASTICSEARCH_NODE || 'http://localhost:9200', index: process.env.ELASTICSEARCH_INDEX || 'app-logs', flushBytes: parseInt(process.env.ES_FLUSH_BYTES || '1000'), flushInterval: parseInt(process.env.ES_FLUSH_INTERVAL || '30000'), auth: process.env.ES_API_KEY ? { apiKey: process.env.ES_API_KEY } : undefined }) const logger = pino(stream) ``` -------------------------------- ### File + Elasticsearch Logging (Multistream) Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/usage-patterns.md Configure pino-elasticsearch with a file stream using the multistream method. This allows logging to both a local file and Elasticsearch. ```javascript const pino = require('pino') const pinoElasticsearch = require('pino-elasticsearch') const fs = require('fs') const elasticStream = pinoElasticsearch({ node: 'http://localhost:9200', index: 'app-logs' }) const fileStream = fs.createWriteStream('app.log', { flags: 'a' }) const logger = pino(pino.multistream([ { level: 'info', stream: fileStream }, { level: 'info', stream: elasticStream } ])) logger.info('Logged to file and Elasticsearch') ``` -------------------------------- ### Pino with Multiple Streams (Console and Elasticsearch) Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/api-reference-main.md Configure Pino to simultaneously log to the console and Elasticsearch using pino transports. ```javascript const pino = require('pino') const pinoElasticsearch = require('pino-elasticsearch') const streamToElastic = pinoElasticsearch({ index: 'my-logs', node: 'http://localhost:9200' }) const logger = pino(pino.transport({ targets: [ { target: 'pino/file', // stdout }, { target: 'pino-elasticsearch', options: { index: 'my-logs', node: 'http://localhost:9200' } } ] })) ``` -------------------------------- ### Elastic Cloud Deployment Configuration Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md Connect to an Elastic Cloud deployment using its deployment name and a base64 encoded API key. ```bash cat logs.ndjson | pino-elasticsearch \ --cloud deployment-name:dXMtY2VudHJhbDEuZ2NwLmNsb3VkLmVzLmlvJHBsYXRmb3JtLWdlbmVyYWwkYWJjZGVmZ2hpag== \ --api-key base64EncodedKey \ --index cloud-logs ``` -------------------------------- ### Import Pino-Elasticsearch Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/REFERENCE_INDEX.md How to import the pino-elasticsearch module in a Node.js project. ```javascript const pinoElasticsearch = require('pino-elasticsearch') ``` -------------------------------- ### Initialize pino-elasticsearch Stream and Listen for Events Source: https://github.com/pinojs/pino-elasticsearch/blob/master/README.md This snippet shows how to initialize the pino-elasticsearch stream and attach an event listener for debugging. Use this to capture any event emitted by the stream handler. ```javascript const pinoElastic = require('pino-elasticsearch'); const streamToElastic = pinoElastic({ index: 'an-index', node: 'http://localhost:9200', esVersion: 7, flushBytes: 1000 }) streamToElastic.on('', (error) => console.log(event)); ``` -------------------------------- ### Connect to Elastic Cloud Deployment Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md Uses an Elastic Cloud deployment ID and API key to connect to a cloud-hosted Elasticsearch cluster. Use this instead of the --node flag for cloud deployments. ```bash pino-elasticsearch \ --cloud my-deployment:dXMtY2VudHJhbDEuZ2NwLmNsb3VkLmVzLmlvJHBsYXRmb3JtLWdlbmVyYWwkYWJjZGVmZ2hpag== \ --api-key base64EncodedApiKey ``` -------------------------------- ### Configure Elasticsearch Client Options Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/architecture.md Defines the options for configuring the Elasticsearch client, including node details, authentication, cloud settings, TLS, and custom connection classes. Instantiate the client with these options. ```javascript const clientOpts = { node: opts.node, // Single or multiple nodes auth: opts.auth, // Basic, API key, or bearer cloud: opts.cloud, // Elasticsearch Cloud tls: { rejectUnauthorized, ... }, caFingerprint: opts.caFingerprint, Connection: opts.Connection, // Custom connection class ConnectionPool: opts.ConnectionPool // Custom pool class } const client = new Client(clientOpts) ``` -------------------------------- ### CLI Authentication with Basic Auth in URL Source: https://github.com/pinojs/pino-elasticsearch/blob/master/README.md Connect to Elasticsearch using the CLI with basic authentication by including username and password in the node URL. ```sh cat log | pino-elasticsearch --node https://user:pwd@localhost:9200 ``` -------------------------------- ### Pino Elasticsearch CLI Usage Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/REFERENCE_INDEX.md Instructions and available flags for using the pino-elasticsearch command-line interface to process log files. ```APIDOC ## CLI Usage ```bash pino-elasticsearch [options] ``` ### Key Flags - `--help`: Display help information for the CLI. - `--version`: Show the installed version of pino-elasticsearch. - `--node URL`: Specify the Elasticsearch URL. Can be provided multiple times for multiple nodes. - `--index NAME`: Set the index name. Supports date formatting like `% {DATE}`. - `--username USER`: Basic authentication username. - `--password PASS`: Basic authentication password. - `--api-key KEY`: API key for authentication. - `--cloud-id ID`: Elasticsearch Cloud deployment ID. - `--flush-bytes SIZE`: Set the buffer size in bytes for bulk operations. - `--flush-interval MS`: Set the flush interval in milliseconds. - `--opType TYPE`: Set the bulk operation type (`'create'` or `'index'`). - `--read-config FILE`: Load configuration options from a specified JSON or JavaScript file. ``` -------------------------------- ### Low-Latency Logging Configuration Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/usage-patterns.md Configure pino-elasticsearch for real-time monitoring by using smaller buffer sizes (`flushBytes`) and shorter flush intervals (`flushInterval`). This ensures logs are sent to Elasticsearch more frequently. ```javascript const stream = pinoElasticsearch({ node: 'http://localhost:9200', index: 'app-logs', flushBytes: 1000, // Small batches flushInterval: 5000, // Flush frequently esVersion: 8 }) ``` -------------------------------- ### CLI Authentication with Cloud ID and API Key Source: https://github.com/pinojs/pino-elasticsearch/blob/master/README.md Connect to Elastic Cloud using the CLI with a cloud ID and API key. ```sh cat log | pino-elasticsearch --cloud=name:bG9jYWxob3N0JGFiY2QkZWZnaA== --api-key=base64EncodedKey ``` -------------------------------- ### Stream Development Logs to Elasticsearch via CLI Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/usage-patterns.md Pipe application output to pino-elasticsearch for real-time logging during development. Debug tracing can be enabled using the --trace-level option. ```bash # Development with debug output npm run dev | pino-elasticsearch \ --node http://localhost:9200 \ --index dev-logs \ --trace-level debug ``` -------------------------------- ### Basic Authentication with Username and Password Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md Use username and password for basic authentication with Elasticsearch. Ensure both flags are provided. ```bash pino-elasticsearch \ --node https://elasticsearch.example.com:9200 \ --username elastic \ --password changeme ``` -------------------------------- ### Configure pino-elasticsearch for Multiple Elasticsearch Nodes Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md This command demonstrates how to configure pino-elasticsearch to use connection pooling across multiple Elasticsearch nodes by passing an array of node addresses. ```bash pino-elasticsearch --node http://es1:9200 --node http://es2:9200 --node http://es3:9200 ``` -------------------------------- ### Migrate removed 'bulk-size' to 'flushBytes' Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/errors.md The 'bulk-size' option has been removed and is replaced by 'flushBytes'. Using 'bulk-size' will emit a warning and the option will be ignored. ```javascript // Before (removed) pinoElasticsearch({ 'bulk-size': 1000 }) // After (use flushBytes) pinoElasticsearch({ flushBytes: 1000 }) ``` -------------------------------- ### Basic Authentication Configuration Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/configuration.md Configure basic authentication using a username and password. This method is suitable for simple authentication scenarios. ```javascript { auth: { username: 'elastic', password: 'changeme' } } ``` -------------------------------- ### Dynamic Index Name Configuration with Function Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/configuration.md Provide a synchronous function to dynamically generate index names based on the log timestamp. The function must not throw exceptions and must return a string. ```javascript index: (logTime) => { // logTime = '2026-07-02T10:30:45.123Z' const yearMonth = logTime.substring(0, 7) // '2026-07' return `logs-${yearMonth}` } The function: - Must be synchronous - Must not throw exceptions - Must return a string ``` -------------------------------- ### Configure Pino-Elasticsearch with Environment-Based Settings Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/usage-patterns.md Use a configuration object to define settings for different environments. The active configuration is selected based on the NODE_ENV environment variable. ```javascript const config = { development: { node: 'http://localhost:9200', index: 'dev-logs', flushInterval: 5000 }, production: { node: process.env.ES_NODE, index: 'prod-logs-%{DATE}', flushInterval: 60000, auth: { apiKey: process.env.ES_API_KEY } } } const env = process.env.NODE_ENV || 'development' const esConfig = config[env] const stream = pinoElasticsearch(esConfig) const logger = pino(stream) ``` -------------------------------- ### Pino-Elasticsearch with Custom Elasticsearch Client Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/api-reference-main.md Integrate pino-elasticsearch with a custom Elasticsearch client implementation. Ensure the custom connection class is defined. ```javascript const { Client } = require('@elastic/elasticsearch') class CustomConnection { // Custom connection implementation } const streamToElastic = pinoElasticsearch({ index: 'my-logs', node: 'http://localhost:9200', Connection: CustomConnection }) ``` -------------------------------- ### Development Logging to Elasticsearch Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/REFERENCE_INDEX.md Pipe logs from a Node.js application to a local Elasticsearch instance for development. ```bash node app.js | pino-elasticsearch --node http://localhost:9200 --index dev-logs ``` -------------------------------- ### Basic Usage of pino-elasticsearch Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/README.md Configure and use pino-elasticsearch as a Pino stream for logging to Elasticsearch. Ensure Elasticsearch is running at the specified node. ```javascript const pino = require('pino') const pinoElasticsearch = require('pino-elasticsearch') const stream = pinoElasticsearch({ node: 'http://localhost:9200', index: 'my-logs' }) const logger = pino(stream) logger.info('Hello, Elasticsearch!') ``` -------------------------------- ### Production with Secure Connection and API Key Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md Configure pino-elasticsearch for production with HTTPS, an API key, and optimized flush settings for high-volume data. ```bash cat logs.ndjson | pino-elasticsearch \ --node https://elasticsearch.example.com:9200 \ --index prod-logs-%{DATE} \ --api-key $ES_API_KEY \ --flush-bytes 10000 \ --flush-interval 60000 ``` -------------------------------- ### Pino Elasticsearch Configuration Options Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/REFERENCE_INDEX.md Details the available configuration options for customizing the behavior of the pino-elasticsearch stream. ```APIDOC ## Core Configuration Options | Option | Type | Default | Purpose | |---|---|---|---| | `node` | string \| string[] | — | Elasticsearch URL(s) for the connection. | | `index` | string \| function | `'pino'` | The name of the Elasticsearch index. Can be a static string, a template string (e.g., with date formatting), or a function. | | `flushBytes` | number | `1000` | The maximum number of bytes to buffer before sending a bulk request to Elasticsearch. | | `flushInterval` | number | `30000` | The maximum time in milliseconds to wait before sending a bulk request, even if `flushBytes` is not reached. | | `esVersion` | number | `7` | The major version of the Elasticsearch cluster. Affects API compatibility. | | `auth` | object | — | Authentication credentials for Elasticsearch. Supports username/password, API key, or bearer token. | | `opType` | `'create'` \| `'index'` | — | Specifies the operation type for bulk requests. Use `'create'` to prevent overwriting existing documents, or `'index'` for standard indexing. Useful for data streams. ``` -------------------------------- ### Indexing to Datastreams Source: https://github.com/pinojs/pino-elasticsearch/blob/master/README.md Configure pino-elasticsearch for indexing to Elasticsearch datastreams by setting the `opType` to 'create'. ```js const pino = require('pino') const pinoElastic = require('pino-elasticsearch') const streamToElastic = pinoElastic({ index: "type-dataset-namespace", node: 'http://localhost:9200', opType: 'create' }) // ... ``` -------------------------------- ### Optimize Batch Size for High Throughput Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/integration-guide.md Configure `flushBytes` and `flushInterval` for high-throughput logging scenarios (over 1000 logs/sec) to manage batching efficiently. ```javascript // For high-throughput logging (>1000 logs/sec) const elasticStream = pinoElasticsearch({ node: 'http://elasticsearch:9200', index: 'high-volume-logs', flushBytes: 100000, // 100KB batches flushInterval: 60000, // Every 60 seconds esVersion: 8 }) ``` -------------------------------- ### Pino-Elasticsearch with Event Handlers Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/api-reference-main.md Configure pino-elasticsearch with event handlers for unknown lines, insert errors, successful inserts, and general client errors. Set flush options for batching. ```javascript const streamToElastic = pinoElasticsearch({ index: 'my-logs', node: 'http://localhost:9200', flushBytes: 1000, flushInterval: 30000 }) streamToElastic.on('unknown', (line, error) => { console.error('Parse error:', error) }) streamToElastic.on('insertError', (error) => { console.error('Insert failed for document:', error.document) }) streamToElastic.on('insert', (stats) => { console.log(`Inserted batch with ${stats.successful} documents`) }) streamToElastic.on('error', (error) => { console.error('Client error:', error) }) const logger = pino(streamToElastic) logger.info('Test') ``` -------------------------------- ### Use Pino Pretty with Split Streams for Elasticsearch Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md Configure Pino to send logs to both standard output (pretty-printed) and Elasticsearch using multistream. ```javascript // app.js const pino = require('pino') const multistream = pino.multistream([ { stream: process.stdout }, // Pretty print to console { stream: require('pino-elasticsearch')({ index: 'app-logs', node: 'http://localhost:9200' }) } ]) const logger = pino({}, multistream) logger.info('Visible in console and Elasticsearch') ``` -------------------------------- ### Token-Based Authentication with API Key Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md Use a Base64-encoded API key for token-based authentication. This method takes precedence over username/password. ```bash pino-elasticsearch \ --node https://elasticsearch.example.com:9200 \ --api-key VnVhQ0JHd0JDSkR1cDI4QjJLVzpSMkR6T29FMlZfR3pqMjBKeGVfWnA= ``` -------------------------------- ### Import pino-elasticsearch Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/api-reference-main.md Import the pino-elasticsearch module using either ES module or CommonJS syntax. ```typescript import pinoElasticsearch from 'pino-elasticsearch' // or const pinoElasticsearch = require('pino-elasticsearch') ``` -------------------------------- ### CLI Usage with Data Streams Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/configuration.md Use pino-elasticsearch to send logs to an Elasticsearch data stream with a create operation type. ```bash cat logs.ndjson | pino-elasticsearch \ --node http://localhost:9200 \ --index logs-app-production \ --opType create ``` -------------------------------- ### Development Environment with Debug Output Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md Use debug trace level for detailed output during development. This helps in troubleshooting log processing. ```bash cat logs.ndjson | pino-elasticsearch \ --node http://localhost:9200 \ --index dev-logs \ --trace-level debug ``` -------------------------------- ### Bulk Import Logs from a File Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md Import historical logs from an NDJSON file into Elasticsearch. Adjust flush settings for optimal performance. ```bash cat historical-logs.ndjson | pino-elasticsearch \ --node http://localhost:9200 \ --index historical-%{DATE} \ --flush-bytes 50000 ``` -------------------------------- ### Pino-Elasticsearch CLI Usage Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/REFERENCE_INDEX.md Basic command-line usage for pino-elasticsearch, piping NDJSON logs to Elasticsearch. ```bash pino-elasticsearch [options] < logs.ndjson ``` -------------------------------- ### Tune pino-elasticsearch Batch Size for Lower Volume Source: https://github.com/pinojs/pino-elasticsearch/blob/master/_autodocs/cli-reference.md Use these settings for lower-volume logging to improve throughput by using larger batch sizes and less frequent flushes. ```bash # Lower volume (larger batches, better throughput) pino-elasticsearch \ --flush-bytes 1000000 \ --flush-interval 60000 ``` -------------------------------- ### Usage: Multiple Streams with pino.transport Source: https://github.com/pinojs/pino-elasticsearch/blob/master/README.md Configure pino to output logs to both the console and Elasticsearch using pino.transport and pino-elasticsearch. ```javascript const pino = require('pino'); const pinoOptions = {}; return pino(pinoOptions, pino.transport({ targets: [ { target: 'pino/file', // This will log to stdout }, { target: 'pino-elasticsearch', options: { index: 'an-index', node: 'http://localhost:9200', esVersion: 7, flushBytes: 1000 } } ] })); ```