### Install and start fluentd-ui via gem

Source: https://docs.fluentd.org/deployment/fluentd-ui

Commands to install the fluentd-ui gem and start the service, including expected output.

```text
$ gem install -V fluentd-ui
$ fluentd-ui start
Puma 2.9.2 starting...
* Min threads: 0, max threads: 16
* Environment: production
* Listening on tcp://0.0.0.0:9292
```

--------------------------------

### Install and Start Elasticsearch

Source: https://docs.fluentd.org/how-to-guides/free-alternative-to-splunk-by-fluentd

Download, extract, and start the Elasticsearch service.

```text
$ curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.17.1-linux-x86_64.tar.gz
$ tar -xf elasticsearch-8.17.1-linux-x86_64.tar.gz
$ cd elasticsearch-8.17.1
```

```text
$ ./bin/elasticsearch
```

--------------------------------

### Verify Fluentd Installation

Source: https://docs.fluentd.org/installation/install-by-gem

Run these commands to set up a configuration directory, start the Fluentd daemon, and send a test message to verify functionality.

```text
$ fluentd --setup ./fluent
$ fluentd -c ./fluent/fluent.conf -vv &
$ echo '{"json":"message"}' | fluent-cat debug.test
```

--------------------------------

### Install and Start Kibana

Source: https://docs.fluentd.org/how-to-guides/free-alternative-to-splunk-by-fluentd

Download, extract, and start the Kibana service.

```text
$ curl -O https://artifacts.elastic.co/downloads/kibana/kibana-8.17.1-linux-x86_64.tar.gz
$ tar -xf kibana-8.17.1-linux-x86_64.tar.gz
$ cd kibana-8.17.1-linux-x86_64
```

```text
$ ./bin/kibana
```

--------------------------------

### Implementing metrics in a Fluentd Input plugin

Source: https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-metrics

This example demonstrates how to load the metrics helper, create a metrics instance in the configure method, increment it during start, and retrieve the value in the statistics method.

```ruby
require 'fluent/plugin/input'

module Fluent::Plugin
  class ExampleInput < Input
    Fluent::Plugin.register_input('example', self)

    # 1. Load metrics helper
    helpers :metrics

    def configure(conf)
      super

      # 2. Create parser plugin instance
      @metrics = metrics_create(namespace: "fluentd", subsystem: "input", name: "example", help_text: "Example metrics")
    end

    def start
      super

      # 3. Increase metrics value
      @metrics.inc

    end

    def statistics
      stats = super

      # 4. Retrieve metrics value
      stats = {
        'input' => stats["input"].merge({ 'example' => @metrics.get })
      }
      stats
    end
end
```

--------------------------------

### Create an HTTP server in a Fluentd plugin

Source: https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-http_server

This example demonstrates how to load the http_server helper and start an HTTP server within a plugin class.

```ruby
require 'fluent/plugin/input'

module Fluent::Plugin
  class ExampleInput < Input
    Fluent::Plugin.register_output('example', self)

    # 1. Load http_server helper
    helpers :http_server

    config_param :bind, :string
    config_param :port, :integer

    def start
      super

      # 2. Create and start HTTP server
      create_http_server(:example_http_server, addr: @bind, port: @port, logger: log) do |serv|
        # Define endpoint `/hello` with GET method
        serv.get('/hello') { [200, { 'Content-Type' => 'text/plain' }, 'hello!'] }
      end
    end
  end
end
```

--------------------------------

### Start Fluentd Instance

Source: https://docs.fluentd.org/container-deployment/docker-logging-driver

Starts a Fluentd container using the demo.conf configuration file.

```bash
$ docker run -it -p 24224:24224 -v $(pwd)/demo.conf:/fluentd/etc/demo.conf -e FLUENTD_CONF=demo.conf fluent/fluentd:edge-debian
```

--------------------------------

### Install fluent-plugin-opensearch via fluent-gem

Source: https://docs.fluentd.org/output/opensearch

Use this command to install the plugin if you are using a standard Fluentd installation.

```text
$ fluent-gem install fluent-plugin-opensearch
```

--------------------------------

### Implementing Storage Helper in a Fluentd Input Plugin

Source: https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-storage

This example demonstrates how to load the storage helper, initialize it in the configure method, and use get, put, and update methods to manage plugin state.

```ruby
require 'fluent/plugin/input'

module Fluent::Plugin
  class ExampleInput < Input
    Fluent::Plugin.register_input('awesome_example', self)

    # 1. Load storage helper
    helpers :storage, :thread

    DEFAULT_STORAGE_TYPE = 'local'

    # Omit `shutdown` and other plugin APIs

    def initialize
      super

      @storage = nil
    end

    def configure(conf)
      super

      # 2. Create storage with unique name
      config = conf.elements(name: 'storage').first
      @storage = storage_create(usage: 'awesome_index', conf: config, default_type: DEFAULT_STORAGE_TYPE)
    end

    def start
      super

      # 3. Call storage plugin helpers get/put methods
      @storage.put(:awesome_index, 0) unless @storage.get(:awesome_index)
      thread_create(:awesome_input_runner, &method(:run))
    end

    def run
      while thread_current_running?
        current_time = Time.now.to_i
        break unless (thread_current_running? && Time.now.to_i <= current_time)
        router.emit('awesome', Fluent::Engine.now, generate)
        sleep 0.1
      end
    end

    def generate
      # 4. Update storage plugin helper's storing value
      @storage.update(:awesome_index) { |v| v + 1 }
    end
  end
end
```

--------------------------------

### Setup Gem configuration

Source: https://docs.fluentd.org/configuration/config-file

Commands to initialize and edit the configuration file for Ruby Gem installations.

```text
$ sudo fluentd --setup /etc/fluent
$ sudo vi /etc/fluent/fluent.conf
```

--------------------------------

### CSV Configuration Example

Source: https://docs.fluentd.org/formatter/csv

A basic configuration example for the CSV formatter.

```text
<format>
  @type csv
  fields host,method
</format>
```

--------------------------------

### Configure map plugin example

Source: https://docs.fluentd.org/configuration/config-file

Example configuration for the map plugin.

```text
<match tag>
  @type map
  map '[["code." + tag, time, { "code" => record["code"].to_i}], ["time." + tag, time, { "time" => record["time"].to_i}]]'
  multi true
</match>
```

--------------------------------

### Build and Install Dependencies

Source: https://docs.fluentd.org/installation/install-from-source

Use Bundler to install the necessary gem dependencies for the project.

```text
$ bundle install
Fetching gem metadata from https://rubygems.org/.........
...
Your bundle is complete!
Use `bundle show [gemname]` to see where a bundled gem is installed.
```

--------------------------------

### Install fluent-package 5 on Ubuntu Focal

Source: https://docs.fluentd.org/installation/install-fluent-package/install-by-deb-fluent-package-v5

Automated installation script for Ubuntu Focal.

```bash
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-ubuntu-focal-fluent-package5.sh | sh
```

--------------------------------

### Installing fluent-logger-python

Source: https://docs.fluentd.org/language-bindings/python

Install the required Python library using pip.

```text
$ pip install fluent-logger
```

--------------------------------

### Install fluent-package 5 on Ubuntu Noble

Source: https://docs.fluentd.org/installation/install-fluent-package/install-by-deb-fluent-package-v5

Automated installation script for Ubuntu Noble.

```bash
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-ubuntu-noble-fluent-package5.sh | sh
```

--------------------------------

### Invalid configuration example

Source: https://docs.fluentd.org/output/elasticsearch

Example of an invalid configuration where placeholders are missing the required %{} syntax.

```text
user demo+
password @secret
```

--------------------------------

### File buffer output example

Source: https://docs.fluentd.org/buffer/file

Example of the generated buffer chunk files on disk.

```text
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log.meta
```

--------------------------------

### Install fluent-package 5 on Ubuntu Jammy

Source: https://docs.fluentd.org/installation/install-fluent-package/install-by-deb-fluent-package-v5

Automated installation script for Ubuntu Jammy.

```bash
curl -fsSL https://fluentd.cdn.cncf.io/sh/install-ubuntu-jammy-fluent-package5.sh | sh
```

--------------------------------

### Install dependencies via bundle

Source: https://docs.fluentd.org/deployment/linux-capability

Execute the bundle command to install the added gem.

```text
$ bundle
```

--------------------------------

### Start Fluentd Service

Source: https://docs.fluentd.org/how-to-guides/logs-to-sematext

Use systemctl to start the Fluentd service after completing the configuration.

```text
$ sudo systemctl start fluentd
```

--------------------------------

### Implement a server using the server helper

Source: https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-server

This example demonstrates how to load the server helper, create a server instance, and process incoming data within a Fluentd input plugin.

```ruby
require 'fluent/plugin/input'

module Fluent::Plugin
  class ExampleInput < Input
    Fluent::Plugin.register_input('example', self)

    # 1. Load server helper
    helpers :server

    # Omit `configure`, `shutdown` and other plugin APIs

    def start
      # 2. Create server
      server_create(:title, @port) do |data|
        #3. Process data
      end
    end
  end
end
```

--------------------------------

### Start InfluxDB Service

Source: https://docs.fluentd.org/how-to-guides/syslog-influxdb

Command to start the InfluxDB service after installation.

```text
$ sudo systemctl start influxdb
```

--------------------------------

### Configure in_forward and out_file plugins

Source: https://docs.fluentd.org/installation/post-installation-guide

Example configuration for setting up an input source using the forward plugin and an output endpoint using the file plugin.

```text
<source>
  @type forward
  port 9999
</source>
<match app.**>
  @type file
  path /var/log/app/data.log
  compress gzip
</match>
```

--------------------------------

### Example Configuration for in_exec

Source: https://docs.fluentd.org/input/exec

A basic configuration example showing how to define the command, parsing keys, and extraction settings for the exec plugin.

```text
<source>
  @type exec
  command cmd arg arg
  <parse>
    keys k1,k2,k3
  </parse>
  <extract>
    tag_key k1
    time_key k2
    time_format %Y-%m-%d %H:%M:%S
  </extract>
  run_interval 10s
</source>
```

--------------------------------

### Example out_file Configuration

Source: https://docs.fluentd.org/output/file

A sample configuration for the out_file plugin using gzip compression and a daily timekey buffer.

```text
<match pattern>
  @type file
  path /var/log/fluent/myapp
  compress gzip
  <buffer>
    timekey 1d
    timekey_use_utc true
    timekey_wait 10m
  </buffer>
</match>
```

--------------------------------

### Run Test Driver and feed events

Source: https://docs.fluentd.org/plugin-development/plugin-test-code

Examples of running the Test Driver to feed events, including variations for starting and shutting down.

```ruby
# Run Test Driver and feed an event (output)
d = create_driver
d.run do
  d.feed(time, record)
end

# Emit multiple events (output)
d = create_driver

d.run(default_tag: 'test', expect_emits: 1, timeout: 10, start: true,  shutdown: false) { d.feed(time, { "k1" => 1 })}

d.run(default_tag: 'test', expect_emits: 1, timeout: 10, start: false, shutdown: false) { d.feed(time, { "k1" => 2 })}

d.run(default_tag: 'test', expect_emits: 1, timeout: 10, start: true,  shutdown: true ) { d.feed(time, { "k1" => 3 })}
```

--------------------------------

### Buffer Path Examples

Source: https://docs.fluentd.org/buffer/file_single

Examples showing the structure of buffer file paths for different chunk keys.

```text
# Example with <buffer tag> and tag is test.log
/path/to/buffer/fsb.test.log.b513b61c9791029c2513b61c9791029c2.buf

# Example with <buffer key> and record is {"key":"hello"}
/path/to/buffer/fsb.hello.b513b61c9791029c2513b61c9791029c2.buf
```

--------------------------------

### Native extension build failure log

Source: https://docs.fluentd.org/deployment/plugin-management

Example of a build error log occurring when required development packages like make are missing during plugin installation.

```text
Building native extensions.  This could take a while...
ERROR:  Error installing fluent-plugin-twitter:
        ERROR: Failed to build gem native extension.

    /opt/td-agent/embedded/bin/ruby extconf.rb

checking for rb_str_scrub()... yes
creating Makefile

make "DESTDIR = " clean
sh: 1: make: not found

make "DESTDIR = "
sh: 1: make: not found

make failed, exit code 127

Gem files will remain installed in /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/string-scrub-0.0.3 for inspection.
Results logged to /opt/td-agent/embedded/lib/ruby/gems/2.1.0/extensions/x86_64-linux/2.1.0/string-scrub-0.0.3/gem_make.out
```

--------------------------------

### Placeholder configuration example

Source: https://docs.fluentd.org/output/file

Example of a path configuration using placeholders that are replaced during buffer flush.

```text
path /path/to/file.${tag}.%Y%m%d
```

--------------------------------

### Run Test Driver and feed events

Source: https://docs.fluentd.org/plugin-development/plugin-test-code

Demonstrates how to use the run method to feed events into a plugin driver, including examples with different start and shutdown configurations.

```ruby
# Run Test Driver and feed an event (owner plugin)
d = create_driver
d.run do
  d.feed(time, record)
end

# Emit multiple events (owner plugin)
d = create_driver

d.run(default_tag: 'test', expect_emits: 1, timeout: 10, start: true,  shutdown: false) { d.feed(time, { "k1" => 1 })}

d.run(default_tag: 'test', expect_emits: 1, timeout: 10, start: false, shutdown: false) { d.feed(time, { "k1" => 2 })}

d.run(default_tag: 'test', expect_emits: 1, timeout: 10, start: true,  shutdown: true ) { d.feed(time, { "k1" => 3 })}
```

--------------------------------

### Path traversal validation error example

Source: https://docs.fluentd.org/configuration/buffer-section

Shows how Fluentd raises a Fluent::UnrecoverableError when a placeholder resolves to a path containing parent directory references or starting with a separator.

```text
# tag: "../etc/cron.d" or "/etc/passwd"
<match **>
  @type file
  path /data/${tag}/access.log  #=> Fluent::UnrecoverableError
  <buffer tag>
    # ...
  </buffer>
</match>
```

--------------------------------

### Example Configuration for out_forward

Source: https://docs.fluentd.org/output/forward

A sample configuration showing the match pattern, server definitions for load balancing, and a secondary file output for failed events.

```text
<match pattern>
  @type forward
  send_timeout 60s
  recover_wait 10s
  hard_timeout 60s

  <server>
    name myserver1
    host 192.168.1.3
    port 24224
    weight 60
  </server>
  <server>
    name myserver2
    host 192.168.1.4
    port 24224
    weight 60
  </server>
  ...

  <secondary>
    @type file
    path /var/log/fluent/forward-failed
  </secondary>
</match>
```

--------------------------------

### Configure local storage with conf.arg

Source: https://docs.fluentd.org/storage/local

Demonstrates using conf.arg to specify the storage path and setting the root directory in the system configuration.

```text
<storage awesome_path>
  @type local
</storage>

<system>
  root_dir tmp
</system>
```

--------------------------------

### Install Plugins

Source: https://docs.fluentd.org/installation/install-calyptia-fluentd/install-by-msi-calyptia-fluentd

Use the calyptia-fluentd-gem command as Administrator in the Calyptia-fluentd Command Prompt to install plugins.

```text
C:\opt\calyptia-fluentd> calyptia-fluentd-gem install fluent-plugin-xyz --version=1.2.3
```

--------------------------------

### Define a custom Input plugin with Fluent::Plugin::Base

Source: https://docs.fluentd.org/plugin-development/api-plugin-base

Example of a custom Input plugin class demonstrating registration, configuration parameters, and lifecycle methods like configure, start, and shutdown.

```ruby
require 'fluent/plugin/input' # This may be input, filter, output, parser, formatter, storage or buffer.

module Fluent::Plugin
  class MyExampleInput < Input
    # Plugins should be registered by calling `Fluent::Plugin.register_TYPE` method with name and `self`.
    # The first argument String is to identify the plugin in the configuration file.
    # The second argument is class of the plugin itself, `self` in most cases.
    Fluent::Plugin.register_input('my_example', self)

    desc 'The port number'
    # `config_param` Defines a parameter. You can refer the following parameter via @port instance variable.
    # Without `:default`, a parameter is required.
    config_param :port, :integer

    config_section :user, param_name: :users, multi: true, required: false do
      desc 'Username for authentication'
      config_param :username, :string
      desc 'Password for authentication'
      config_param :password, :string, secret: true
    end

    def configure(conf)
      super

      # If the configuration is invalid, raise `Fluent::ConfigError`.
      if @port <= 1024
        raise Fluent::ConfigError, "port number is too small: #{@port}"
      end

      @users.each do |user|
        if user.password.length < 5
          raise Fluent::ConfigError, "password is too short for user '#{user.username}'"
        end
      end
    end

    def start
      super
      # ...
    end

    def shutdown
      # ...
      super
    end
  end
end
```

--------------------------------

### Implement Parser Helper in Fluentd Input Plugin

Source: https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-parser

Example showing how to load the parser helper, create a parser instance in the configure phase, and use it to parse raw data in the start phase.

```ruby
require 'fluent/plugin/input'

module Fluent::Plugin
  class ExampleInput < Input
    Fluent::Plugin.register_input('example', self)

    # 1. Load parser helper
    helpers :parser

    # Omit `shutdown` and other plugin APIs

    def configure(conf)
      super

      # 2. Create parser plugin instance
      @parser = parser_create
    end

    def start
      super

      # Use parser helper in combination usually with other plugin helpers
      timer_execute(:example_timer, 10) do
        read_raw_data do |text|
          # 3. Call `@parser.parse(text)` to parse raw data
          @parser.parse(text) do |time, record|
            router.emit(tag, time, record)
          end
        end
      end
    end
  end
end
```

--------------------------------

### stdout Configuration Example

Source: https://docs.fluentd.org/output/stdout

Basic configuration for the stdout output plugin using a match pattern.

```text
<match pattern>
  @type stdout
</match>
```

--------------------------------

### Full regexp parsing example

Source: https://docs.fluentd.org/parser/regexp

A complete configuration example showing named captures, time parsing, and type conversion, followed by the input event and the resulting parsed output.

```text
<parse>
  @type regexp
  expression /^\[(?<logtime>[^\]]*)\] (?<name>[^ ]*) (?<title>[^ ]*) (?<id>\d*)$/
  time_key logtime
  time_format %Y-%m-%d %H:%M:%S %z
  types id:integer
</parse>
```

```text
[2013-02-28 12:00:00 +0900] alice engineer 1
```

```text
time:
1362020400 (2013-02-28 12:00:00 +0900)

record:
{
  "name" : "alice",
  "title": "engineer",
  "id"   : 1
}
```

--------------------------------

### Install fluent-plugin-prometheus for td-agent

Source: https://docs.fluentd.org/monitoring-fluentd/monitoring-prometheus

Install the required plugin specifically for td-agent installations.

```text
$ sudo td-agent-gem install fluent-plugin-prometheus
```

--------------------------------

### Configure in_sample and in_dummy Input Plugins

Source: https://docs.fluentd.org/input/sample

Configuration examples for the current in_sample plugin and the legacy in_dummy plugin used in Fluentd v1.11.1 or earlier.

```text
<source>
  @type sample
  sample {"hello":"world"}
  tag sample
</source>

# If you use fluentd v1.11.1 or earlier, use following configuration
<source>
  @type dummy
  dummy {"hello":"world"}
  tag dummy
</source>
```

--------------------------------

### Install fluent-plugin-elasticsearch

Source: https://docs.fluentd.org/output/elasticsearch

Command to install the Elasticsearch output plugin for Fluentd installations without td-agent.

```text
$ fluent-gem install fluent-plugin-elasticsearch
```

--------------------------------

### Install Ruby Development Package

Source: https://docs.fluentd.org/how-to-guides/raspberrypi-cloud-data-logger

Install the necessary development package for Ruby on Raspbian to support Fluentd installation.

```text
$ sudo aptitude install ruby-dev
```

--------------------------------

### Start td-agent-ui

Source: https://docs.fluentd.org/deployment/fluentd-ui

Command to start the UI for td-agent, including expected output showing the Puma server startup.

```text
$ sudo /usr/sbin/td-agent-ui start
Puma 2.9.2 starting...
* Min threads: 0, max threads: 16
* Environment: production
* Listening on tcp://0.0.0.0:9292
```

--------------------------------

### Install td-agent 4 on Red Hat or CentOS

Source: https://docs.fluentd.org/installation/install-by-rpm-td-agent-v4

Download and execute the installation script to register the repository and install td-agent.

```bash
curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent4.sh | sh
```

--------------------------------

### server_create usage examples for UDP, TCP, and TLS

Source: https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-server

Examples demonstrating how to use server_create for different protocols, with and without connection objects.

```ruby
# UDP (w/o socket)
server_create(:title, @port, proto: :udp, max_bytes: 2048) do |data|
  # data is received data
end

# UDP (w/ socket)
server_create(:title, @port, proto: :udp, max_bytes: 2048) do |data, sock|
  # data is received data
  # sock is UDPSocket
end

# TCP (w/o connection)
server_create(:title, @port) do |data|
  # data is received data
end

# TCP (w/ connection)
server_create(:title, @port) do |data, conn|
  # data is received data
  # conn is Fluent::PluginHelper::Server::TCPCallbackSocket
end

# TLS (w/o connection)
server_create(:title, @port, proto: :tls) do |data|
  # data is received data
end

# TLS (w/ connection)
server_create(:title, @port, proto: :tls) do |data, conn|
  # data is received data
  # conn is Fluent::PluginHelper::Server::TLSCallbackSocket
end
```

--------------------------------

### Install calyptia-fluentd on Red Hat / CentOS

Source: https://docs.fluentd.org/installation/install-calyptia-fluentd/install-by-rpm-calyptia-fluentd

Download and execute the installation script to register the RPM repository and install the package.

```bash
# calyptia-fluentd 1
$ curl -L https://calyptia-fluentd.s3.us-east-2.amazonaws.com/calyptia-fluentd-1-redhat.sh | sh
```

--------------------------------

### Launch and verify fluentd service

Source: https://docs.fluentd.org/installation/install-fluent-package/install-by-dmg-fluent-package

Use launchctl to start the fluentd daemon and check the logs for successful initialization.

```text
$ sudo launchctl load /Library/LaunchDaemons/fluentd.plist
$ less /var/log/fluent/fluentd.log
2023-08-01 16:55:03 -0700 [info]: starting fluentd-1.16.2
2023-08-01 16:55:03 -0700 [info]: reading config file path="/etc/fluent/fluentd.conf"
```

--------------------------------

### Install calyptia-fluentd on Debian Buster

Source: https://docs.fluentd.org/installation/install-calyptia-fluentd/install-by-deb-calyptia-fluentd

Executes the installation script for Debian Buster to register the apt repository and install the package.

```bash
# calyptia-fluentd 1
curl -fsSL https://calyptia-fluentd.s3.us-east-2.amazonaws.com/calyptia-fluentd-1-debian-buster.sh | sh
```

--------------------------------

### Install calyptia-fluentd on Ubuntu Xenial

Source: https://docs.fluentd.org/installation/install-calyptia-fluentd/install-by-deb-calyptia-fluentd

Executes the installation script for Ubuntu Xenial to register the apt repository and install the package.

```bash
# calyptia-fluentd 1
curl -fsSL https://calyptia-fluentd.s3.us-east-2.amazonaws.com/calyptia-fluentd-1-ubuntu-xenial.sh | sh
```

--------------------------------

### Install calyptia-fluentd on Ubuntu Bionic

Source: https://docs.fluentd.org/installation/install-calyptia-fluentd/install-by-deb-calyptia-fluentd

Executes the installation script for Ubuntu Bionic to register the apt repository and install the package.

```bash
# calyptia-fluentd 1
curl -fsSL https://calyptia-fluentd.s3.us-east-2.amazonaws.com/calyptia-fluentd-1-ubuntu-bionic.sh | sh
```

--------------------------------

### Install calyptia-fluentd on Ubuntu Focal

Source: https://docs.fluentd.org/installation/install-calyptia-fluentd/install-by-deb-calyptia-fluentd

Executes the installation script for Ubuntu Focal to register the apt repository and install the package.

```bash
# calyptia-fluentd 1
curl -fsSL https://calyptia-fluentd.s3.us-east-2.amazonaws.com/calyptia-fluentd-1-ubuntu-focal.sh | sh
```

--------------------------------

### Buffer Retry Configuration Example

Source: https://docs.fluentd.org/buffer

A complete configuration example demonstrating various parameters for controlling retry behaviors in the <system> and <buffer> sections.

```text
<system>
  root_dir /var/log/fluentd         # For handling unrecoverable chunks
</system>

<buffer>
  retry_wait 1                      # The wait interval for the first retry.
  retry_exponential_backoff_base 2  # Increase the wait time by a factor of N.
  retry_type exponential_backoff    # Set 'periodic' for constant intervals.

  # retry_max_interval 1h           # Cap the wait interval. (see above)
  retry_randomize true              # Apply randomization. (see above)
  retry_timeout 72h                 # Maximum duration before giving up.

  # retry_max_times 17              # Maximum retry count before giving up.
  retry_forever false               # Set 'true' for infinite retry loops.
  retry_secondary_threshold 0.8     # See the "Secondary Output" section in
</buffer>                           # 'Output Plugins' > 'Overview'.
```

--------------------------------

### Start plugin with timer_execute

Source: https://docs.fluentd.org/plugin-development/api-plugin-base

The start method is called when Fluentd starts. Use plugin helpers like timer_execute after calling super.

```ruby
def start
  super

  timer_execute(:my_example_timer, 30) do
    # Code that will be executed every 30 seconds
  end
end
```

--------------------------------

### Example S3 Configuration

Source: https://docs.fluentd.org/output/s3

A sample configuration for the out_s3 plugin, including AWS credentials, bucket details, and buffer settings.

```text
<match pattern>
  @type s3

  aws_key_id YOUR_AWS_KEY_ID
  aws_sec_key YOUR_AWS_SECRET_KEY
  s3_bucket YOUR_S3_BUCKET_NAME
  s3_region ap-northeast-1
  path logs/
  # if you want to use ${tag} or %Y/%m/%d/ like syntax in path / s3_object_key_format,
  # need to specify tag for ${tag} and time for %Y/%m/%d in <buffer> argument.
  <buffer tag,time>
    @type file
    path /var/log/fluent/s3
    timekey 3600 # 1 hour partition
    timekey_wait 10m
    timekey_use_utc true # use utc
    chunk_limit_size 256m
  </buffer>
</match>
```

--------------------------------

### Install fluent-plugin-rewrite-tag-filter

Source: https://docs.fluentd.org/output/rewrite_tag_filter

Command to install the plugin for Fluentd gem users.

```text
$ fluent-gem install fluent-plugin-rewrite-tag-filter
```

--------------------------------

### Install fluent-plugin-prometheus gem

Source: https://docs.fluentd.org/monitoring-fluentd/monitoring-prometheus

Install the required plugin for Fluentd monitoring.

```text
$ fluent-gem install fluent-plugin-prometheus
```

--------------------------------

### Configure Service Discovery

Source: https://docs.fluentd.org/output/forward

Example of using a service discovery plugin to manage destination servers dynamically instead of using a fixed list.

```text
<match pattern>
  @type forward

  <service_discovery>
    @type file
    path /path/to/servers.yaml
  </service_discovery>
</match>
```

--------------------------------

### Using the Parser Plugin Helper

Source: https://docs.fluentd.org/plugin-development/api-plugin-parser

Demonstrates how to include the parser helper, initialize a parser instance, and process data within a plugin.

```ruby
# in class definition
helpers :parser

# in #configure
@parser = parser_create(type: 'json')

# in input loop or #filter or ...
@parser.parse do |time, record|
  # ...
end
```

--------------------------------

### Installing Dependencies

Source: https://docs.fluentd.org/language-bindings/nodejs

Command to install local dependencies using npm.

```text
$ npm install
```

--------------------------------

### Configure static service discovery with out_forward

Source: https://docs.fluentd.org/service_discovery/static

An example configuration showing how to define a static server list within the service_discovery block of an out_forward source.

```text
<source>
  @type forward

  <service_discovery>
    @type static
    <server>
      host 127.0.0.1
    </server>
  </service_discovery>
</source>
```

--------------------------------

### Install fluent-plugin-windows-eventlog

Source: https://docs.fluentd.org/input/windows_eventlog

Command to install the plugin for Fluentd gem users.

```text
$ fluent-gem install fluent-plugin-windows-eventlog
```

--------------------------------

### Rule matching configuration example

Source: https://docs.fluentd.org/input/tail

This configuration demonstrates two rules where Rule2 has more constraints than Rule1. A file matching both will be assigned to Rule2 due to its higher priority.

```text
  <rule> ## Rule1
    match {
      namespace: /monitoring/
    }
    limit 100
  </rule>

  <rule> ## Rule2
    match {
      namespace: /monitoring/,
      podname: /logger/,
    }
    limit 2000
  </rule>
```

--------------------------------

### Create and write to a socket using the socket helper

Source: https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-socket

This example demonstrates loading the socket helper, creating a TCP socket with configured host and port, and writing JSON-encoded records to it. The socket must be manually closed in an ensure block to prevent resource leaks.

```ruby
require 'fluent/plugin/output'

module Fluent::Plugin
  class ExampleOutput < Output
    Fluent::Plugin.register_output('example', self)

    # 1. Load socket helper
    helpers :socket

    config_param :host, :string
    config_param :port, :integer

    # Omit `configure`, `shutdown` and other plugin APIs

    def try_write(chunk)
      # 2. Create socket
      socket = socket_create(:tcp, @host, @port)
      chunk.each do |time, record|
        # 3. Write data to socket
        socket.write(record.to_json)
      end
    ensure
      # 4. Close socket
      socket.close if socket
    end
  end
end
```

--------------------------------

### Install fluent-plugin-webhdfs

Source: https://docs.fluentd.org/output/webhdfs

Command to install the webhdfs plugin for Fluentd gem users.

```text
$ fluent-gem install fluent-plugin-webhdfs
```

--------------------------------

### Install fluent-plugin-mongo

Source: https://docs.fluentd.org/output/mongo

Command to install the MongoDB output plugin gem for Fluentd.

```text
$ sudo fluent-gem install fluent-plugin-mongo
```

--------------------------------

### Launch and verify calyptia-fluentd

Source: https://docs.fluentd.org/installation/install-calyptia-fluentd/install-by-dmg-calyptia-fluentd

Use launchctl to start the service and check the log file to confirm it is running correctly.

```text
$ sudo launchctl load /Library/LaunchDaemons/calyptia-fluentd.plist
$ less /var/log/calyptia-fluentd/calyptia-fluentd.log
2021-05-31 14:29:38 +0900 [info]: starting fluentd-1.12.3 pid=72608 ruby="3.0.1"
2021-05-31 14:29:38 +0900 [info]: spawn command to main:  cmdline=["/opt/calyptia-fluentd/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/calyptia-fluentd/usr/sbin/calyptia-fluentd", "--log", "/var/log/calyptia-fluentd/calyptia-fluentd.log", "--use-v1-config", "--under-supervisor"]
```

--------------------------------

### Install fluent-plugin-kafka

Source: https://docs.fluentd.org/output/kafka

Command to install the Kafka plugin for Fluentd gem users.

```text
$ fluent-gem install fluent-plugin-kafka
```

--------------------------------

### Running the Application

Source: https://docs.fluentd.org/language-bindings/nodejs

Command to start the Node.js application.

```text
$ node index.js
```