Fly Log Shipper

https://github.com/superfly/fly-log-shipper
Admin
Fly Log Shipper is a utility for shipping logs from Fly.io applications to other providers using...
Tokens:26,492
Snippets:448
Trust Score:9.4
Update:2 months ago
Show doc for...
Tokens:
Raw
### S3 Key Prefix Example

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Example of the resulting S3 object path structure.

```text
s3://my-bucket/myapp/2024-07-24/logs-chunk-001.json.gz
```

--------------------------------

### Deploy Application

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Initial deployment command to start the Vector process and subscribe to NATS.

```bash
fly deploy
```

--------------------------------

### Example log_json input and output

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/transforms.md

Demonstrates the transformation of a raw JSON string message into a structured event.

```json
{
  "message": "{\"level\": \"info\", \"text\": \"app started\"}",
  "timestamp": "2024-07-24T15:30:45.123Z"
}
```

```json
{
  "level": "info",
  "text": "app started"
}
```

--------------------------------

### Scale Instances with Queue Group

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/nats-source.md

Example of scaling multiple instances to the same queue group for NATS-managed load balancing.

```bash
# Instance 1
fly scale count 1 --name log-shipper-ha-1
fly secrets set QUEUE=log-shipper-group

# Instance 2
fly scale count 1 --name log-shipper-ha-2
fly secrets set QUEUE=log-shipper-group

# NATS ensures only one instance processes each message
```

--------------------------------

### Configure an HTTP-based sink

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sinks-overview.md

Example configuration for an HTTP sink using bearer authentication and gzip compression.

```toml
[sinks.example]
  type = "http"
  inputs = ["log_json"]
  uri = "${ENDPOINT_URL}"
  method = "post"  # Usually POST
  encoding.codec = "json"
  auth.strategy = "bearer"  # Or "basic"
  auth.token = "${API_KEY}"
  compression = "gzip"
```

--------------------------------

### Example Slack Alert Message

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Displays the expected format of the alert message received in Slack.

```text
New alert from *myapp*
AppId: 228e4c1f928d3001
Message:
ERROR: database connection failed
```

--------------------------------

### NATS Subject Example

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/nats-source.md

A concrete example of a NATS subject string following the hierarchical schema.

```text
logs.myapp.iad.228e4c1f928d3001
```

--------------------------------

### GET /metrics

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/vector-internals.md

Retrieves service metrics in Prometheus text format.

```APIDOC
## GET /metrics

### Description
Returns performance and operational metrics for the Vector service in Prometheus text format.

### Method
GET

### Endpoint
http://[::1]:9598/metrics
```

--------------------------------

### Validate Vector TOML Configuration

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/file-structure.md

Use these commands to verify TOML syntax and sink configurations. Requires a local Vector installation version 0.29.1 or compatible.

```bash
# Validate TOML syntax
vector validate --config-toml vector-configs/vector.toml

# Check specific sink
vector validate --config-toml vector-configs/sinks/datadog.toml

# Test with environment variables
export DATADOG_API_KEY=test-key
vector validate --config-toml vector-configs/sinks/datadog.toml
```

--------------------------------

### Set Custom Network Prefix

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/nats-source.md

Configures the NETWORK environment variable for non-default 6PN network setups, such as WireGuard tunnels.

```bash
fly secrets set NETWORK=fdaa:1:2:3
```

--------------------------------

### NDJSON Log Format

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Example of the newline-delimited JSON format used for stored logs.

```json
{"level":"info","message":"app started"}
{"level":"error","message":"failed to connect"}
```

--------------------------------

### GET /health

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/vector-internals.md

Retrieves the current health status of the Vector service.

```APIDOC
## GET /health

### Description
Returns the health status of the Vector service.

### Method
GET

### Endpoint
http://[::1]:8686/health

### Response
#### Success Response (200)
- **status** (string) - The health status of the service, typically 'ok'.

#### Response Example
{
  "status": "ok"
}
```

--------------------------------

### HTTP Sink JSON Request Format

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Example of the JSON payload structure sent to the HTTP endpoint.

```json
{
  "level": "info",
  "message": "application started",
  "timestamp": "2024-07-24T15:30:45Z"
}
```

--------------------------------

### Vector Startup Command

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/vector-internals.md

Executes Vector with the main configuration file and an additional directory for sink configurations.

```bash
vector -c /etc/vector/vector.toml -C /etc/vector/sinks
```

--------------------------------

### Recreate application from configuration

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Commands to recreate an app using existing configuration and secrets.

```bash
fly launch --copy-config
```

```bash
fly secrets set
```

```bash
fly status
```

--------------------------------

### Deploy log-shipper via CLI

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Initializes the application, sets required secrets for NATS and providers, and triggers the deployment.

```bash
# 1. Create new log-shipper app
fly launch --image flyio/log-shipper:latest --no-public-ips

# 2. Configure NATS source (required)
fly secrets set ACCESS_TOKEN=$(fly tokens create readonly personal)

# 3. Configure desired provider(s) — see Configuration Reference
fly secrets set DATADOG_API_KEY=xxx

# 4. Deploy
fly deploy
```

--------------------------------

### Deployment Commands

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Commands for building and deploying custom configurations.

```bash
fly deploy --build-only
```

```bash
# Create/edit vector-configs/custom.toml
fly deploy --build-only  # Rebuild image with new configs
```

--------------------------------

### Set Up Multiple Instances (HA)

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/README.md

Scale the application to multiple instances and configure a queue group to prevent duplicate message processing.

```bash
# 1. Scale to multiple instances
fly scale count 3

# 2. Configure queue group to avoid duplicate messages
fly secrets set QUEUE=my-log-shipper

# 3. Redeploy
fly deploy
```

--------------------------------

### Capture raw logs and configuration

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/troubleshooting.md

Inspect current metrics, configuration files, and sink definitions.

```bash
fly ssh console

# Dump metrics to see activity
curl http://localhost:9598/metrics | grep "_total"

# Check Vector config
cat /etc/vector/vector.toml

# View loaded sinks
cat /etc/vector/sinks/*.toml | head -50
```

--------------------------------

### Scaling and Monitoring

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Commands for scaling instances and monitoring log volume metrics.

```bash
fly scale count 2  # 2x monthly cost
fly scale count 1  # 1x monthly cost (single point of failure)
```

```bash
fly ssh console
curl http://localhost:9598/metrics | grep "source_nats_messages"
```

--------------------------------

### Visualize Build and Deployment Workflow

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/file-structure.md

A visual representation of the CI/CD pipeline from Git push to container runtime execution.

```text
Git Push
    ↓
GitHub Actions (release.yaml)
    ↓
Build Docker Image
    ├─ FROM timberio/vector:0.29.1-debian
    ├─ COPY vector-configs/ /etc/vector/
    ├─ COPY start-fly-log-transporter.sh /
    └─ CMD ["bash", "start-fly-log-transporter.sh"]
    ↓
Push to Registry
    ├─ flyio/log-shipper:latest
    └─ flyio/log-shipper:v<version>
    ↓
Developer Deploy
    ├─ fly launch --image flyio/log-shipper:latest
    ├─ fly secrets set (NATS + provider config)
    └─ fly deploy
    ↓
Container Runtime
    ├─ start-fly-log-transporter.sh
    │  └─ Template-expand /etc/vector/sinks/*.toml
    │  └─ Execute: vector -c vector.toml -C sinks/
    └─ Vector Process Running
       ├─ NATS Source: Consume logs
       ├─ Transforms: Parse, filter, remap
       └─ Sinks: Forward to providers (1-25)
```

--------------------------------

### Define core Vector configuration

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/file-structure.md

Sets up the API, NATS source, JSON parsing transform, and Prometheus/blackhole sinks.

```toml
[api]
  enabled = true
  address = "[::]:8686"

[sources.fly_log_metrics]
  type = "internal_metrics"

[sources.nats]
  type = "nats"
  url = "nats://[${NETWORK-fdaa}::3]:4223"
  subject = "${SUBJECT-logs.>}"
  # ... other nats config

[transforms.log_json]
  type = "remap"
  inputs = ["nats"]
  source = '''
  . = parse_json!(.message)
  '''

[sinks.fly_log_metrics_prometheus]
  type = "prometheus_exporter"
  inputs = ["fly_log_metrics"]
  address = "[::]:9598"
  default_namespace = "fly-logs"

[sinks.blackhole]
  type = "blackhole"
  inputs = ["log_json"]
  print_interval_secs = 100000
```

--------------------------------

### Startup script for environment variable expansion

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/file-structure.md

Expands environment variables in TOML files using shell templates before launching the Vector process.

```bash
#!/bin/bash
set -euo pipefail

template() { eval $'cat <<_EOF\n'"$(awk '1;END{print"_EOF"}')"; }
sponge() { cat <<<"$(cat)" >"$1"; }
filter() { for i in "$@"; do template <"$i" | sponge "$i" || rm "$i"; done; }
filter /etc/vector/sinks/*.toml 2>&-
echo 'Configured sinks:'
find /etc/vector/sinks -type f -exec basename -s '.toml' {} \;

exec vector -c /etc/vector/vector.toml -C /etc/vector/sinks
```

--------------------------------

### Prepare Logs for OpsVerse

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/transforms.md

Identical transform logic used by OpsVerse sinks.

```toml
[transforms.loki_json]
  type = "remap"
  inputs = ["log_json"]
  source = '''
  .level = .log.level
  if starts_with(.message, "{") ?? false {
    json = object!(parse_json!(.message))
    del(.message)
    . |= json
  }
  '''
```

--------------------------------

### Launch Fly Logger App

Source: https://github.com/superfly/fly-log-shipper/blob/main/README.md

Command to initialize the log-shipper application using the official Docker image.

```bash
fly launch --image flyio/log-shipper:latest --no-public-ips
```

--------------------------------

### Define Dockerfile for Vector container

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/file-structure.md

Configures the base image, copies necessary configuration files, and sets the entrypoint to the startup script.

```dockerfile
FROM timberio/vector:0.29.1-debian
COPY vector-configs /etc/vector/
COPY ./start-fly-log-transporter.sh .
CMD ["bash", "start-fly-log-transporter.sh"]
ENTRYPOINT []
```

--------------------------------

### Launch NATS client app

Source: https://github.com/superfly/fly-log-shipper/blob/main/README.md

Deploys a NATS client container using the official nightly server image.

```bash
fly launch --image="synadia/nats-server:nightly" --name="nats-client"
```

--------------------------------

### Scale Application

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Scale instances and enable queue groups for high-volume log processing.

```bash
# Scale to multiple instances
fly scale count 3

# Enable queue group to avoid duplicates
fly secrets set QUEUE=log-shipper-ha

# Redeploy
fly deploy
```

--------------------------------

### Custom Dockerfile Configuration

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Use this Dockerfile to extend the base image with additional tools or custom startup scripts.

```dockerfile
FROM flyio/log-shipper:latest

# Add custom scripts, tools, etc.
RUN apt-get update && apt-get install -y \
    curl \
    jq

# Override start script if needed
COPY custom-start.sh /
CMD ["bash", "/custom-start.sh"]
```

--------------------------------

### Reset configuration

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/troubleshooting.md

Clear existing secrets and re-initialize with minimal required configuration.

```bash
# Save current secrets
fly secrets list > backup.txt

# Unset all secrets
fly secrets unset $(fly secrets list | awk '{print $1}')

# Set only required ones
fly secrets set ACCESS_TOKEN=$(fly tokens create readonly personal)

# Redeploy with minimal config
fly deploy
```

--------------------------------

### Configure Datadog

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/README.md

Set the required Datadog API credentials and redeploy the application to enable log shipping to Datadog.

```bash
# 1. Get API key from Datadog dashboard
# 2. Set secrets
fly secrets set DATADOG_API_KEY=xyz
fly secrets set DATADOG_SITE=datadoghq.com  # or datadoghq.eu

# 3. Redeploy
fly deploy
```

--------------------------------

### Configure NATS context

Source: https://github.com/superfly/fly-log-shipper/blob/main/README.md

Sets up the NATS CLI context with the required server address, organization slug, and personal access token.

```bash
nats context add nats --server [fdaa::3]:4223 --description "NATS Demo" --select \
  --user <YOUR FLY ORG SLUG> \
  --password <YOUR PAT>
```

--------------------------------

### Minimal fly.toml configuration

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Defines the base application settings, including the image, internal port, and metrics endpoint.

```toml
app = "my-log-shipper"

[build]
  image = "flyio/log-shipper:latest"

[[services]]
  protocol = "tcp"
  internal_port = 8686
  processes = ["app"]

[metrics]
  port = 9598
  path = "/metrics"

[env]
  RUST_LOG = "info"

[[statics]]
  guest_path = "/app"
  url_prefix = "/static"
```

--------------------------------

### Scale and rollback deployment

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Commands to scale the application for high availability or roll back to a previous release.

```bash
fly scale count 3
fly secrets set QUEUE=my-log-shipper
fly deploy
```

```bash
fly releases
fly releases rollback RELEASE_ID
```

--------------------------------

### Project Directory Layout

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/file-structure.md

Visual representation of the repository structure, highlighting configuration files and provider-specific sink definitions.

```text
fly-log-shipper/
├── .git/                          # Git repository metadata
├── .github/
│   └── workflows/
│       └── release.yaml           # GitHub Actions CI/CD pipeline
├── .dockerignore                  # Files to exclude from Docker build
├── .gitignore                     # Files to exclude from git
├── Dockerfile                     # Container image definition
├── fly.toml                       # Fly.io deployment configuration
├── README.md                      # Project documentation
├── start-fly-log-transporter.sh   # Container startup script
└── vector-configs/
    ├── vector.toml                # Vector core configuration
    └── sinks/                     # Provider-specific sink configs
        ├── appsignal.toml
        ├── aws_cloudwatch.toml
        ├── aws_s3.toml
        ├── axiom.toml
        ├── baselime.toml
        ├── better-stack.toml
        ├── datadog.toml
        ├── erasearch.toml
        ├── highlight.toml
        ├── honeybadger.toml
        ├── honeycomb.toml
        ├── http.toml
        ├── humio.toml
        ├── hyperdx.toml
        ├── logdna.toml
        ├── logflare.toml
        ├── loki.toml
        ├── new_relic.toml
        ├── openobserve.toml
        ├── opsverse.toml
        ├── papertrail.toml
        ├── sematext.toml
        ├── signoz.toml
        ├── slack.toml
        ├── uptrace.toml
        └── erasearch.toml
```

--------------------------------

### Troubleshoot template variable errors

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/troubleshooting.md

Verify and set missing environment variables when encountering template errors.

```text
template error: DATADOG_API_KEY not set
```

```bash
# Check which variables are set
fly secrets list

# Set the missing variable
fly secrets set DATADOG_API_KEY=xxx

# Redeploy
fly deploy
```

--------------------------------

### Configure Highlight Sink

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Defines the HTTP sink configuration for sending logs to Highlight.io.

```toml
[sinks.highlight]
  type = "http"
  inputs = ["log_json"]
  encoding.codec = "json"
  framing.method = "newline_delimited"
  compression = "gzip"
  uri = "https://pub.highlight.io/v1/logs/json"
  headers.x-highlight-project = "${HIGHLIGHT_PROJECT_ID}"
```

--------------------------------

### Configure Alert Rules

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Instructions for adding alert rules via the Fly Dashboard.

```bash
# Via Fly Dashboard
# Settings → Alerts → Add Alert Rule
```

--------------------------------

### Redeploy and monitor application

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Commands to redeploy the application and monitor logs for issues.

```bash
fly deploy
```

```bash
fly logs
```

--------------------------------

### Handle deployment lease conflicts

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/troubleshooting.md

Check current deployment status and force a retry if necessary.

```bash
# Wait for current deployment to finish
fly status

# If stuck, cancel and retry
fly deploy --now
```

--------------------------------

### Verify Deployment Status

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Commands to check application health, logs, and metrics.

```bash
# Check app status
fly status

# View logs
fly logs

# Check NATS connectivity
fly logs | grep "nats"

# Verify sinks are active
fly logs | grep "Configured sinks"

# Check metrics endpoint
fly ssh console
curl http://localhost:9598/metrics
```

--------------------------------

### Configure SigNoz Sink

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

TOML configuration for the SigNoz HTTP sink.

```toml
[sinks.signoz]
  type = "http"
  inputs = ["log_json"]
  uri = "${SIGNOZ_INGESTION_URL:-https://ingest.us.signoz.cloud/logs/vector}"

[sinks.signoz.encoding]
  codec = "json"

[sinks.signoz.request.headers]
  signoz-ingestion-key = "${SIGNOZ_INGESTION_KEY}"
```

--------------------------------

### Configure Sink Compression

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sinks-overview.md

Sets the compression algorithm for the sink to optimize bandwidth usage.

```toml
compression = "gzip"
```

--------------------------------

### Fix missing metrics directory

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/troubleshooting.md

Manually create the required metrics directory or restart the service.

```bash
# Create missing directory
fly ssh console
sudo mkdir -p /var/lib/vector

# Or restart to auto-create
fly restart
```

--------------------------------

### Configure Sematext Sink

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

TOML configuration for the Sematext logs sink.

```toml
[sinks.sematext]
  type = "sematext_logs"
  inputs = ["log_json"]
  region = "${SEMATEXT_REGION}"
  token = "${SEMATEXT_TOKEN}"
```

--------------------------------

### Configure Better Stack Sink

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Sets up a remap transform for timestamp compatibility and configures the HTTP sink for Better Stack.

```toml
[transforms.remap_better_stack_timestamp]
  type = "remap"
  inputs = ["log_json"]
  source = '''
  .dt = del(.timestamp)
  '''

[sinks.better_stack]
  type = "http"
  inputs = ["remap_better_stack_timestamp"]
  uri = "https://${BETTER_STACK_INGESTING_HOST:-in.logs.betterstack.com}"
  encoding.codec = "json"
  auth.strategy = "bearer"
  auth.token = "${BETTER_STACK_SOURCE_TOKEN:-$LOGTAIL_TOKEN}"
```

```bash
fly secrets set BETTER_STACK_SOURCE_TOKEN=xxx
```

--------------------------------

### Diagnose Vector configuration syntax errors

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/troubleshooting.md

Inspect recent logs to identify syntax errors in Vector configuration files.

```bash
fly logs | head -50
# Look for error line number and message
```

```bash
cat vector-configs/sinks/*.toml | head -20
```

--------------------------------

### Set Uptrace Environment Variables

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Use the fly CLI to set the required environment variables for the Uptrace sink.

```bash
fly secrets set UPTRACE_API_KEY=xxx
fly secrets set UPTRACE_PROJECT=123
fly secrets set UPTRACE_SINK_INPUT=log_json
fly secrets set UPTRACE_SINK_ENCODING=json
```

--------------------------------

### Test NATS subscription

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/troubleshooting.md

Verify NATS connectivity and log subscription within the container.

```bash
fly ssh console

# Test NATS connectivity
nats context add test \
  --server nats://[fdaa:0:0::3]:4223 \
  --user ORG_SLUG \
  --password TOKEN

nats sub "logs.>"
```

--------------------------------

### Configure custom sink directory

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/troubleshooting.md

Ensure custom sink configuration files are placed in the correct directory and reloaded.

```bash
# Files must be in /etc/vector/sinks/ directory
# Ensure file ends with .toml extension
# Restart vector to reload

fly deploy
```

--------------------------------

### Configure Axiom Sink

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Defines the Axiom sink configuration in TOML and sets the required environment variables via the Fly CLI.

```toml
[sinks.axiom]
  type = "axiom"
  inputs = ["log_json"]
  token = "${AXIOM_TOKEN}"
  dataset = "${AXIOM_DATASET}"
```

```bash
fly secrets set AXIOM_TOKEN=xxx
fly secrets set AXIOM_DATASET=logs
```

--------------------------------

### Configure S3 Provider Secrets

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Set AWS credentials, bucket name, and region for S3 log storage.

```bash
fly secrets set AWS_ACCESS_KEY_ID=<key>
fly secrets set AWS_SECRET_ACCESS_KEY=<secret>
fly secrets set AWS_BUCKET=logs-bucket
fly secrets set AWS_REGION=us-east-1
```

--------------------------------

### Configure a generic sink

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sinks-overview.md

Standard TOML configuration pattern for defining a sink in Fly Log Shipper.

```toml
[sinks.<sink_name>]
  type = "<vector_sink_type>"
  inputs = ["<input_source>"]
  # Provider-specific fields
  api_key = "${API_KEY_ENV_VAR}"
  # Optional configuration
  encoding.codec = "json"
  compression = "gzip"
```

--------------------------------

### Configure Honeycomb Sink

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Defines the Honeycomb sink configuration in TOML format.

```toml
[sinks.honeycomb]
  type = "honeycomb"
  inputs = ["log_json"]
  api_key = "${HONEYCOMB_API_KEY}"
  dataset = "${HONEYCOMB_DATASET}"
```

--------------------------------

### Set Honeycomb Environment Variables

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Commands to set the required secrets for the Honeycomb sink.

```bash
fly secrets set HONEYCOMB_API_KEY=xxx
fly secrets set HONEYCOMB_DATASET=logs
```

--------------------------------

### Configure Loki Provider Secrets

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Set URL and authentication credentials for Loki log forwarding.

```bash
fly secrets set LOKI_URL=https://loki.example.com
fly secrets set LOKI_USERNAME=<username>
fly secrets set LOKI_PASSWORD=<password>
```

--------------------------------

### Deploy and Configure Fly Log Shipper

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/README.md

Commands to launch the log-shipper container, set NATS access tokens, and configure provider API keys via Fly secrets.

```bash
# Deploy
fly launch --image flyio/log-shipper:latest --no-public-ips

# Configure NATS
fly secrets set ACCESS_TOKEN=$(fly tokens create readonly personal)

# Configure provider(s)
fly secrets set DATADOG_API_KEY=xxx

# Deploy
fly deploy
```

--------------------------------

### Update log-shipper image configuration

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Update the build image in fly.toml to the latest version.

```toml
build.image = "flyio/log-shipper:latest"  # Or specify version
```

--------------------------------

### Setting Secrets via Fly CLI

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/configuration.md

Configures necessary environment secrets for NATS and Datadog providers before deployment.

```bash
# NATS source configuration
fly secrets set ACCESS_TOKEN=$(fly tokens create readonly personal)
fly secrets set ORG=my-org

# Provider example: Datadog
fly secrets set DATADOG_API_KEY=your-api-key
fly secrets set DATADOG_SITE=datadoghq.com

# Deployment
fly deploy
```

--------------------------------

### Configure Datadog Provider Secrets

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Set API key and site region for Datadog log forwarding.

```bash
fly secrets set DATADOG_API_KEY=<your-api-key>
fly secrets set DATADOG_SITE=datadoghq.com  # or datadoghq.eu
```

--------------------------------

### Configure Slack Alert Keywords

Source: https://github.com/superfly/fly-log-shipper/blob/main/README.md

Set the keywords that trigger log alerts in Slack using the fly CLI.

```bash
fly secrets set SLACK_ALERT_KEYWORDS="[r'SIGTERM', r'reboot']"
```

--------------------------------

### Access and inspect container

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Commands to access the container via SSH and inspect metrics or connectivity.

```bash
fly logs
```

```bash
fly ssh console
```

```bash
fly ssh console
curl http://localhost:9598/metrics | grep "nats_messages"
```

```bash
fly ssh console
curl -H "Authorization: Bearer TOKEN" https://sink-endpoint.example.com
```

--------------------------------

### Configure Optional Sink Variable

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sinks-overview.md

Uses shell-style parameter expansion to provide a default value for a configuration setting.

```toml
site = "${DATADOG_SITE:-datadoghq.com}"
```

--------------------------------

### Configure Fly.io deployment

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/file-structure.md

Specifies the application name and Prometheus metrics endpoint configuration.

```toml
app = "fly-log-shipper-example"

[metrics]
  port = 9598
  path = "/metrics"
```

--------------------------------

### Configure Optional NATS Settings

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Customize NATS subjects, queue groups for high availability, or network settings.

```bash
# Custom NATS subject (default: logs.>)
fly secrets set SUBJECT="logs.myapp.>"

# Queue group for HA (optional)
fly secrets set QUEUE="log-shipper-ha"

# Custom network (default: fdaa:0:0)
fly secrets set NETWORK="fdaa:1:2:3"
```

--------------------------------

### Rebuild container image

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/troubleshooting.md

Force a fresh build of the container image to resolve potential corruption issues.

```bash
# Force rebuild
fly deploy --build-only --no-cache

# Redeploy
fly deploy
```

--------------------------------

### Troubleshoot health check failures

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/troubleshooting.md

Verify port configuration and adjust health check grace periods.

```bash
# Check internal_port matches Vector API port (8686)
cat fly.toml | grep "internal_port"

# Should show:
# internal_port = 8686

# Extend health check grace period if slow startup
fly scale vm performance-2x

# Redeploy
fly deploy
```

--------------------------------

### Manage secrets

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Commands to list and update application secrets.

```bash
fly secrets list
```

```bash
fly secrets set VAR=value
fly deploy
```

--------------------------------

### Define provider-specific sink configuration pattern

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/file-structure.md

Template for creating individual sink files that support environment variable injection and optional transforms.

```toml
[sinks.<provider_name>]
  type = "<vector_sink_type>"
  inputs = ["<transform_or_source>"]
  <provider_field_1> = "${ENV_VAR_1}"
  <provider_field_2> = "${ENV_VAR_2}"
  # Optional transforms specific to provider
[transforms.<optional_transform>]
  type = "remap|filter|throttle"
  inputs = ["log_json"]
  # Transform-specific VRL or config
```

--------------------------------

### Configure Vector debug via fly.toml

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/vector-internals.md

Persist debug logging configuration by adding the environment variable to the fly.toml file.

```toml
[env]
  RUST_LOG = "debug"
```

--------------------------------

### Set Logflare Secrets

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Commands to configure the required API key and source token for Logflare.

```bash
fly secrets set LOGFLARE_API_KEY=xxx
fly secrets set LOGFLARE_SOURCE_TOKEN=uuid-here
```

--------------------------------

### Inspect Vector Logs

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sinks-overview.md

View logs from the Vector process to troubleshoot sink issues.

```bash
fly logs
```

--------------------------------

### Vector TOML Configuration Structure

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/vector-internals.md

Standard structure for Vector configuration files defining sources, transforms, and sinks.

```toml
[api]
  # Vector control plane API

[sources.<name>]
  # Input sources (logs, metrics, etc.)

[transforms.<name>]
  # Processing transforms

[sinks.<name>]
  # Output destinations
```

--------------------------------

### S3 Sink Configuration

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Configure an S3 sink for cost-effective long-term log archival.

```toml
[sinks.s3_archive]
  type = "aws_s3"
  inputs = ["log_json"]
  bucket = "archive-bucket"
  compression = "gzip"
  # ... other settings
```

--------------------------------

### Check Application Health

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Use the Fly CLI to verify the current health status of the application.

```bash
# Check app health
fly status

# Output shows health check status
```

--------------------------------

### Configure Queue Group

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Set a queue group to enable load-balancing and prevent duplicate log processing.

```bash
# Set queue group to load-balance
fly secrets set QUEUE=my-log-shipper

# Redeploy all instances
fly deploy
```

--------------------------------

### Set Alert Keywords via CLI

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Sets the regex patterns used to trigger Slack alerts.

```bash
fly secrets set SLACK_ALERT_KEYWORDS="[r'SIGTERM', r'crash', r'failed.*connect']"
```

--------------------------------

### Set SigNoz Secrets

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

CLI commands to set environment variables for SigNoz cloud and self-hosted instances.

```bash
fly secrets set SIGNOZ_INGESTION_URL=https://ingest.us.signoz.cloud/logs/vector
fly secrets set SIGNOZ_INGESTION_KEY=xxx
```

```bash
fly secrets set SIGNOZ_INGESTION_URL=https://localhost:3000/logs/vector
# SIGNOZ_INGESTION_KEY not needed
```

--------------------------------

### Roll back deployment

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/troubleshooting.md

Revert to a previous stable release if a new deployment causes issues.

```bash
# View recent releases
fly releases

# Rollback to previous version
fly releases rollback <RELEASE_ID>

# Monitor for recovery
fly logs
fly status
```

--------------------------------

### Monitor Vector Performance Metrics

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/vector-internals.md

Use these metrics to track event throughput and identify potential data loss due to buffer overflows.

```text
fly_logs_sink_<name>_events_total / time
fly_logs_sink_<name>_dropped_events_total
```

--------------------------------

### Visualize log flow architecture

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/overview.md

Diagram representing the data pipeline from Fly.io apps through NATS and Vector to external providers.

```text
[Fly.io Apps]
      ↓
  [NATS Broker] (nats://[${NETWORK}::3]:4223)
      ↓
[Vector NATS Source] (subject: ${SUBJECT}, queue: ${QUEUE})
      ↓
[Log JSON Transform] (parses JSON from message field)
      ↓
[Sink-Specific Transforms] (optional remapping, filtering, parsing)
      ↓
[25+ External Providers] (HTTP, S3, Elasticsearch, etc.)
```

--------------------------------

### Check Sink Health via API

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sinks-overview.md

Use the Vector API to verify the health status of configured sinks.

```bash
fly ssh console
curl http://localhost:8686/health
```

--------------------------------

### Configure OpsVerse Loki Sink

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Defines a remap transform and a Loki-compatible sink for OpsVerse log ingestion.

```toml
[transforms.loki_json]
  type = "remap"
  inputs = ["log_json"]
  source = '''
  .level = .log.level
  if starts_with(.message, "{") ?? false {
    json = object!(parse_json!(.message))
    del(.message)
    . |= json
  }
  '''

[sinks.opsverse]
  type = "loki"
  inputs = ["loki_json"]
  endpoint = "${OPSVERSE_LOGS_ENDPOINT}"
  compression = "gzip"
  auth.strategy = "basic"
  auth.user = "${OPSVERSE_USERNAME}"
  auth.password = "${OPSVERSE_PASSWORD}"
  encoding.codec = "logfmt"
  out_of_order_action = "accept"
  
  labels.event_provider = "{{event.provider}}"
  labels.fly_region = "{{fly.region}}"
  labels.fly_app_name = "{{fly.app.name}}"
  labels.fly_app_instance = "{{fly.app.instance}}"
  labels.host = "{{host}}"
  labels.level = "{{level}}"
```

--------------------------------

### Set OpenObserve Secrets

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Use the fly CLI to set the required environment variables for OpenObserve authentication.

```bash
fly secrets set OPENOBSERVE_URI=https://openobserve.example.com
fly secrets set OPENOBSERVE_USER=user
fly secrets set OPENOBSERVE_PASSWORD=password
```

--------------------------------

### Configure Slack Sink in TOML

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Defines the transformation pipeline and HTTP sink configuration for sending alerts to Slack.

```toml
[transforms.log_filter]
  type = "filter"
  inputs = ["log_json"]
  condition = '''
  match_any!(.message, ${SLACK_ALERT_KEYWORDS})
  '''

[transforms.log_throttle]
  type = "throttle"
  inputs = ["log_filter"]
  threshold = 1
  window_secs = 1

[transforms.log_parse]
  type = "remap"
  inputs = ["log_throttle"]
  source = '''
  .blocks = [{"type": "section", "text": {"type": "mrkdwn", "text":"New alert from *" + .fly.app.name + "*"}},{ "type":"section", "text":{"type": "mrkdwn", "text": "*AppId:* " + .fly.app.instance + "\n*Message:*\n" + "\`\`\`" + .message + "\`\`\`"} }] ?? {}
  del(.message)
  del(.event)
  del(.fly)
  del(.host)
  del(.log)
  del(.timestamp)
  . = {"message": encode_json(.)}
  '''

[sinks.log_http]
  type = "http"
  inputs = ["log_parse"]
  method = "post"
  encoding.codec = "text"
  uri = "${SLACK_WEBHOOK_URL}"
```

--------------------------------

### Configure Slack Sink Secrets

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Sets the required webhook URL and alert keywords as environment secrets.

```bash
fly secrets set SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
fly secrets set SLACK_ALERT_KEYWORDS="[r'error', r'FATAL']"
```

--------------------------------

### Emergency troubleshooting commands

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Commands used for diagnosing and resolving common issues like crashes, connection loss, or sink failures.

```bash
fly status
```

```bash
fly logs -n 500
```

```bash
fly restart
```

```bash
fly releases rollback LAST_GOOD_RELEASE
```

```bash
fly ssh console && ping [fdaa:0:0::3]
```

```bash
fly tokens list
```

```bash
fly secrets set ACCESS_TOKEN=$(fly tokens create readonly personal)
```

```bash
fly deploy
```

```bash
fly secrets list
```

```bash
fly ssh console && curl ENDPOINT
```

```bash
fly logs | grep "sink"
```

```bash
fly ssh console && curl http://localhost:9598/metrics | grep health
```

--------------------------------

### Configure OpenObserve Sink

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Define the OpenObserve sink using the HTTP type with basic authentication and JSON encoding.

```toml
[sinks.openobserve]
  type = "http"
  inputs = ["log_json"]
  uri = "${OPENOBSERVE_URI}"
  method = "post"
  auth.strategy = "basic"
  auth.user = "${OPENOBSERVE_USER}"
  auth.password = "${OPENOBSERVE_PASSWORD}"
  compression = "gzip"
  encoding.codec = "json"
  encoding.timestamp_format = "rfc3339"
  healthcheck.enabled = false
```

--------------------------------

### Diagnose NATS Connection Issues

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/troubleshooting.md

Commands to verify token validity, organization slugs, and IPv6 connectivity.

```bash
# Check token is valid
fly tokens list

# Verify organization slug
fly orgs list

# Test IPv6 connectivity
fly ssh console
ping [fdaa:0:0::3]
```

--------------------------------

### Retrieve and Search Logs

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Commands for viewing real-time logs, historical logs, or filtering for specific errors.

```bash
# Real-time logs
fly logs

# Last N logs
fly logs -n 200

# Search logs
fly logs | grep "error"
```

--------------------------------

### Configure S3-Compatible Service Secrets

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Commands to set secrets for S3-compatible services like Cloudflare R2 or Wasabi.

```bash
fly secrets set S3_ENDPOINT=https://abc123.r2.cloudflarestorage.com
fly secrets set AWS_ACCESS_KEY_ID=xxx  # R2 or Wasabi credentials
fly secrets set AWS_SECRET_ACCESS_KEY=xxx
fly secrets set AWS_BUCKET=bucket-name
fly secrets set AWS_REGION=auto
```

--------------------------------

### Diagnose high CPU usage

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/troubleshooting.md

Check transform and processing metrics to identify performance bottlenecks.

```bash
fly ssh console
curl http://localhost:9598/metrics | grep "transform\|processing"
```

```bash
fly scale vm performance-2x
```

--------------------------------

### Resolve NATS Connection Issues

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/troubleshooting.md

Commands to recreate tokens and redeploy the service.

```bash
# Recreate token
fly secrets set ACCESS_TOKEN=$(fly tokens create readonly personal)

# Verify ORG is correct
fly secrets set ORG=my-org

# Redeploy
fly deploy
```

--------------------------------

### Manage disk space usage

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/troubleshooting.md

Clear log files and reduce verbosity to recover disk space.

```bash
# Check disk usage
fly ssh console
df -h

# Clear logs
rm /var/log/*.log

# Reduce log verbosity
fly secrets set RUST_LOG=warn
fly deploy
```

--------------------------------

### Set Datadog API Key

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sinks-overview.md

Sets the required environment variable to activate the Datadog sink.

```bash
fly secrets set DATADOG_API_KEY=abc123def456
```

--------------------------------

### Rollback Releases

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

View release history and revert to a previous version.

```bash
# View recent releases
fly releases

# Rollback to specific release
fly releases rollback <release_id>
```

--------------------------------

### Set Sematext Secrets

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

CLI commands to set required environment variables for Sematext.

```bash
fly secrets set SEMATEXT_REGION=us
fly secrets set SEMATEXT_TOKEN=xxx
```

--------------------------------

### Diagnose Sink Connectivity

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/troubleshooting.md

Commands to test endpoint connectivity and authorization headers.

```bash
fly ssh console

# Test endpoint connectivity
curl https://api.datadoghq.com

# Test with credentials
curl -H "Authorization: Bearer TOKEN" https://endpoint.example.com
```

--------------------------------

### Manage Secrets

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Commands to list, update, or remove application secrets.

```bash
fly secrets list

# Output:
# NAME                  DIGEST
# ACCESS_TOKEN          abc123...
# DATADOG_API_KEY       def456...
```

```bash
fly secrets set VAR=new_value
```

```bash
fly secrets unset VAR
```

--------------------------------

### Set OpsVerse Secrets

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Configures the required environment variables for OpsVerse authentication and endpoint connectivity.

```bash
fly secrets set OPSVERSE_LOGS_ENDPOINT=https://logs.opsverse.io
fly secrets set OPSVERSE_USERNAME=user
fly secrets set OPSVERSE_PASSWORD=password
```

--------------------------------

### Configure Humio Sink

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Defines the Humio sink configuration in TOML and sets the required API token via Fly secrets.

```toml
[sinks.humio]
  type = "humio_logs"
  inputs = ["log_json"]
  compression = "gzip"
  host_key = "fly.app.instance"
  token = "${HUMIO_TOKEN}"
  encoding.codec = "json"
  ${HUMIO_ENDPOINT+endpoint = "$HUMIO_ENDPOINT"}
```

```bash
fly secrets set HUMIO_TOKEN=xxx
```

--------------------------------

### Resolve Sink Health Check Failures

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/troubleshooting.md

Commands to update credentials and redeploy.

```bash
# Verify and update credentials
fly secrets set DATADOG_API_KEY=<new-key>

# For self-hosted/on-prem endpoints, disable health checks in sink config
# (Requires modifying vector-configs/sinks/)

# Redeploy
fly deploy
```

--------------------------------

### Configure a Filter Transform

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/transforms.md

Sets up a filter transform to drop or pass events based on a VRL condition.

```toml
[transforms.<name>]
  type = "filter"
  inputs = ["<source>"]
  condition = '''
  match_any!(.message, [r'ERROR', r'FATAL'])
  '''
```

--------------------------------

### Prepare Logs for Loki

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/transforms.md

Maps the log level and conditionally parses JSON messages for Loki sinks.

```toml
[transforms.loki_json]
  type = "remap"
  inputs = ["log_json"]
  source = '''
  .level = .log.level
  if starts_with(.message, "{") ?? false {
    # parse json messages
    json = object!(parse_json!(.message))
    del(.message)
    . |= json
  }
  '''
```

--------------------------------

### Configure Fly Secrets

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Commands to set required environment variables for the S3 sink.

```bash
fly secrets set AWS_ACCESS_KEY_ID=xxx
fly secrets set AWS_SECRET_ACCESS_KEY=xxx
fly secrets set AWS_BUCKET=my-logs-bucket
fly secrets set AWS_REGION=us-east-1
```

--------------------------------

### Set Datadog Secrets

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Commands to configure Datadog API credentials via Fly secrets.

```bash
fly secrets set DATADOG_API_KEY=xxx
fly secrets set DATADOG_SITE=datadoghq.com
```

--------------------------------

### Configure Fly Services

Source: https://github.com/superfly/fly-log-shipper/blob/main/README.md

Configuration snippet for fly.toml to set the internal port for Vector health checks.

```toml
[[services]]
  http_checks = []
  internal_port = 8686
```

--------------------------------

### Set HTTP Sink Environment Variables

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Commands to set the required secrets for the HTTP sink.

```bash
fly secrets set HTTP_URL=https://logs.example.com/ingest
fly secrets set HTTP_TOKEN=my-secret-token
```

--------------------------------

### Configure AppSignal Sink

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Defines the AppSignal sink configuration using a TOML block.

```toml
[sinks.appsignal]
  type = "appsignal"
  inputs = ["log_json"]
  push_api_key = "${APPSIGNAL_PUSH_API_KEY}"
```

--------------------------------

### Verify NATS Connectivity

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Use ping to verify IPv6 connectivity within the container.

```bash
fly ssh console
ping [fdaa:0:0::3]  # Should respond
```

--------------------------------

### Configure NATS Source Secrets

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Set the required organization slug and read-only access token for NATS connectivity.

```bash
# Get organization slug
fly orgs list

# Create read-only token
ACCESS_TOKEN=$(fly tokens create readonly personal)

# Set secrets
fly secrets set ACCESS_TOKEN=$ACCESS_TOKEN
fly secrets set ORG=<your-org-slug>
```

--------------------------------

### Monitor Buffer Metrics

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sinks-overview.md

Prometheus metrics for tracking buffer events and dropped logs.

```text
fly_logs_sink_<sink_name>_buffer_events
fly_logs_sink_<sink_name>_dropped_events_total
```

--------------------------------

### Monitor Metrics

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/README.md

Access Prometheus metrics via SSH to verify system performance and sink status.

```bash
# View Prometheus metrics
fly ssh console
curl http://localhost:9598/metrics

# Or check specific metric
curl http://localhost:9598/metrics | grep "sink_datadog"
```

--------------------------------

### Configure Baselime Sink

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Configures the HTTP sink for Baselime and sets the necessary authentication and dataset secrets.

```toml
[sinks.baselime]
  type = "http"
  inputs = ["log_json"]
  uri = "https://events.baselime.io/v1/${BASELIME_DATASET:-flyio}"
  encoding.codec = "json"
  auth.strategy = "bearer"
  auth.token = "${BASELIME_API_KEY}"
```

```bash
fly secrets set BASELIME_API_KEY=xxx
fly secrets set BASELIME_DATASET=flyio
```

--------------------------------

### Configure Internal Metrics Source

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/configuration.md

Defines the source for Vector's internal metrics.

```toml
[sources.fly_log_metrics]
  type = "internal_metrics"
```

--------------------------------

### Vector API Configuration

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/vector-internals.md

Enables the Vector control plane API on the specified IPv6 address.

```toml
[api]
  enabled = true
  address = "[::]:8686"
```

--------------------------------

### Configure Logflare Sink

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Defines a remap transform and an HTTP sink to send logs to Logflare.

```toml
[transforms.remap_logflare_loglevel]
  type = "remap"
  inputs = ["log_json"]
  source = '''
  .level = .log.level
  '''

[sinks.logflare]
  type = "http"
  inputs = ["remap_logflare_loglevel"]
  uri = "https://api.logflare.app/logs/vector?source=${LOGFLARE_SOURCE_TOKEN}"
  request.headers = {"X-API-Key" = "${LOGFLARE_API_KEY}"}
  encoding.codec = "json"
  compression = "none"
```

--------------------------------

### Configure Datadog Sink

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Defines the Datadog sink configuration in TOML, including API key and site settings.

```toml
[sinks.datadog]
  type = "datadog_logs"
  inputs = ["log_json"]
  default_api_key = "${DATADOG_API_KEY}"
  site = "${DATADOG_SITE:-datadoghq.com}"
  compression = "gzip"
  healthcheck.enabled = true
```

--------------------------------

### Format Slack Block Kit

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/transforms.md

Formats logs into Slack Block Kit JSON structure and encodes the result as a string.

```toml
[transforms.log_parse]
  type = "remap"
  inputs = ["log_throttle"]
  source = '''
  .blocks = [
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "New alert from *" + .fly.app.name + "*"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "*AppId:* " + .fly.app.instance + "\n*Message:*\n" + "```" + .message + "```"
      }
    }
  ] ?? {}
  del(.message)
  del(.event)
  del(.fly)
  del(.host)
  del(.log)
  del(.timestamp)
  . = {"message": encode_json(.)}
  '''
```

--------------------------------

### Diagnose Missing Logs

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/deployment-guide.md

Check sink metrics and logs to identify why logs are not appearing in the destination.

```bash
# Check sink metrics
fly ssh console
curl http://localhost:9598/metrics | grep sink_<provider>_events

# Check logs for errors
fly logs | grep -E "error|failed|sink"
```

--------------------------------

### Configure LogDNA Sink

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Defines the LogDNA sink configuration and sets the ingestion key via Fly secrets.

```toml
[sinks.logdna]
  type = "logdna"
  inputs = ["log_json"]
  api_key = "${LOGDNA_API_KEY}"
  hostname = "{{fly.app.instance}}"
```

```bash
fly secrets set LOGDNA_API_KEY=xxx
```

--------------------------------

### Configure Loki Sink

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Defines a remap transform and a Loki sink with specific labels for log aggregation.

```toml
[transforms.loki_json]
  type = "remap"
  inputs = ["log_json"]
  source = '''
  .level = .log.level
  if starts_with(.message, "{") ?? false {
    json = object!(parse_json!(.message))
    del(.message)
    . |= json
  }
  '''

[sinks.loki]
  type = "loki"
  inputs = ["loki_json"]
  endpoint = "${LOKI_URL}"
  compression = "gzip"
  auth.strategy = "basic"
  auth.user = "${LOKI_USERNAME}"
  auth.password = "${LOKI_PASSWORD}"
  encoding.codec = "json"
  
  labels.event_provider = "{{event.provider}}"
  labels.fly_region = "{{fly.region}}"
  labels.fly_app_name = "{{fly.app.name}}"
  labels.fly_app_instance = "{{fly.app.instance}}"
  labels.host = "{{host}}"
  labels.level = "{{level}}"
```

--------------------------------

### Set Elasticsearch Authorization

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sink-reference.md

Commands to configure Bearer token or Basic authentication for Elasticsearch.

```bash
fly secrets set ERASEARCH_AUTH="Bearer YOUR_API_KEY"
```

```bash
# Encode username:password as base64
echo -n "user:password" | base64
# abc123defg==

fly secrets set ERASEARCH_AUTH="Basic abc123defg=="
```

--------------------------------

### Monitor Sink Performance Metrics

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/sinks-overview.md

Metrics for tracking total events, failed events, and processing errors per sink.

```text
fly_logs_sink_<name>_events_total
fly_logs_sink_<name>_events_failed_total
fly_logs_sink_<name>_processing_errors_total
```

--------------------------------

### Access NATS Metrics via CLI

Source: https://github.com/superfly/fly-log-shipper/blob/main/_autodocs/nats-source.md

Access the metrics endpoint from within a Fly.io machine console.

```bash
fly ssh console
curl http://localhost:9598/metrics
```
Tokens:
Raw