### Enable Profiling Mode

Source: https://github.com/snowplow/snowbridge/wiki/Profiling-with-pprof

Run the binary with the -p flag to start the profiling web-server on port 8080.

```bash
./stream-replicator -p
```

--------------------------------

### Setup Retry Configuration

Source: https://context7.com/snowplow/snowbridge/llms.txt

Configure retry behavior for setup errors. Specify the initial `delay_ms` and `max_attempts` for retries.

```hcl
retry {
  setup {
    # Initial delay for setup errors (default: 20000ms)
    delay_ms = 30000

    # Maximum retry attempts (default: 5)
    max_attempts = 3
  }
}
```

--------------------------------

### HCL Configuration Example

Source: https://github.com/snowplow/snowbridge/wiki/Home

Full HCL configuration block defining sources, transformations, targets, and observability settings.

```hcl
source {
  use "kinesis" {
    # Kinesis stream name to read from (required)
    stream_name       = "my-stream"

    # AWS region of Kinesis stream (required)
    region            = "us-west-1"

    # App name for Stream Replicator (required)
    app_name          = "StreamReplicatorProd1"

    # Optional ARN to use on source stream (default: "")
    role_arn          = "arn:aws:iam::123456789012:role/myrole"

    # Timestamp for the kinesis shard iterator to begin processing.
    # Format YYYY-MM-DD HH:MM:SS.MS (miliseconds optional)
    # (default: TRIM_HORIZON)
    start_timestamp   = "2020-01-01 10:00:00"

    # Number of events to process concurrently (default: 50)
    concurrent_writes = 15
  }
}

transform {
  use "spEnrichedFilter" {
    # keep only page views
    atomic_field = "event_name"
    regex = "^page_view$"
  }
}

transform {
  use "js" {
    # changes app_id to "1"
    source_b64 = "ZnVuY3Rpb24gbWFpbih4KSB7CiAgICB2YXIganNvbk9iaiA9IEpTT04ucGFyc2UoeC5EYXRhKTsKICAgIGpzb25PYmpbImFwcF9pZCJdID0gIjEiOwogICAgcmV0dXJuIHsKICAgICAgICBEYXRhOiBKU09OLnN0cmluZ2lmeShqc29uT2JqKQogICAgfTsKfQ=="
  }
}

target {
  use "pubsub" {
    # ID of the GCP Project
    project_id = "acme-project"

    # Name of the topic to send data into
    topic_name = "some-acme-topic"
  }
}

// log level configuration (default: "info")
log_level = "info"

sentry {
  # The DSN to send Sentry alerts to
  dsn   = "https://acme.com/1"

  # Whether to put Sentry into debug mode (default: false)
  debug = true

  # Escaped JSON string with tags to send to Sentry (default: "{}")
  tags  = "{\"aKey\":\"aValue\"}"
}

stats_receiver {
  use "statsd" {
    # StatsD server address
    address = "127.0.0.1:8125"

    # StatsD metric prefix (default: "snowplow.stream-replicator")
    prefix  = "snowplow.stream-replicator"

    # Escaped JSON string with tags to send to StatsD (default: "{}")
    tags    = "{\"aKey\": \"aValue\"}"
  }

  # Time (seconds) the observer waits for new results (default: 1)
  timeout_sec = 2

  # Aggregation time window (seconds) for metrics being collected (default: 15)
  buffer_sec  = 20
}

enable_telemetry = true
user_provided_id = "hello-this-is-us"
```

--------------------------------

### HTTP Target Go Template

Source: https://context7.com/snowplow/snowbridge/llms.txt

Example of using Go templates to format HTTP request bodies. This template includes environment variables and iterates over data, applying `prettyPrint`.

```go-template
{
  "secret_key": "{{ env "SECRET_KEY" }}",
  "data": [
    {{ range $i, $data := . }} {{if $i}},{{end}} {{ prettyPrint .foo }} {{ end }}
  ]
}
```

--------------------------------

### Implement JavaScript Transformation Logic

Source: https://context7.com/snowplow/snowbridge/llms.txt

Examples of custom JavaScript functions for filtering, modifying data, and setting HTTP headers.

```javascript
// Example: Filter and transform Snowplow data
function main(input) {
    var spData = input.Data;

    // Filter out non-web events
    if (spData["platform"] != "web") {
        return { FilterOut: true };
    }

    // Set user ID
    if ("user_id" in spData) {
        spData["uid"] = spData["user_id"];
    } else {
        spData["uid"] = spData["domain_userid"];
    }

    return {
        Data: spData,
        PartitionKey: spData["app_id"]
    };
}
```

```javascript
// Example: Transform generic JSON data
function main(input) {
    var jsonObj = JSON.parse(input.Data);

    // Filter based on condition
    if (jsonObj.batmobileCount < 1) {
        return { FilterOut: true };
    }

    // Modify data
    jsonObj.name = "Bruce Wayne";

    return {
        Data: jsonObj,
        PartitionKey: jsonObj.id
    };
}
```

```javascript
// Example: Set custom HTTP headers
function main(x) {
    var httpHeaders = {
        foo: 'bar'
    };
    x.HTTPHeaders = httpHeaders;
    return x;
}
```

--------------------------------

### Configure Kinesis Source with Environment Variables

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Sources

Set these environment variables to configure the Kinesis source. This includes stream name, region, application name, and optional role ARN, start timestamp, and concurrent writes.

```bash
export SOURCE_NAME="kinesis"                                    \
SOURCE_KINESIS_STREAM_NAME="my-stream"                          \
SOURCE_KINESIS_REGION="us-west-1"                               \
SOURCE_KINESIS_APP_NAME="StreamReplicatorProd1"                 \
SOURCE_KINESIS_ROLE_ARN="arn:aws:iam::123456789012:role/myrole" \
SOURCE_KINESIS_START_TSTAMP="2020-01-01 10:00:00"               \
SOURCE_CONCURRENT_WRITES=15
```

--------------------------------

### Snowbridge Environment Variable Configuration

Source: https://context7.com/snowplow/snowbridge/llms.txt

Configure Snowbridge using environment variables. Examples cover source (Kinesis), target (Pub/Sub), transformations (Base64 encoded HCL), logging, monitoring (Sentry, StatsD), and telemetry settings.

```bash
# Source configuration
export SOURCE_NAME="kinesis"
export SOURCE_KINESIS_STREAM_NAME="my-stream"
export SOURCE_KINESIS_REGION="us-west-1"
export SOURCE_KINESIS_APP_NAME="SnowbridgeProd1"
export SOURCE_KINESIS_ROLE_ARN="arn:aws:iam::123456789012:role/myrole"
export SOURCE_KINESIS_START_TSTAMP="2020-01-01 10:00:00"
export SOURCE_CONCURRENT_WRITES=15

# Target configuration
export TARGET_NAME="pubsub"
export TARGET_PUBSUB_PROJECT_ID="acme-project"
export TARGET_PUBSUB_TOPIC_NAME="some-acme-topic"

# Transformations (base64 encoded HCL)
export TRANSFORM_CONFIG_B64="dHJhbnNmb3JtIHsKICB1c2UgIn..."

# Logging
export LOG_LEVEL="info"

# Monitoring
export SENTRY_DSN="https://acme.com/1"
export SENTRY_DEBUG=true
export SENTRY_TAGS="{\"aKey\":\"aValue\"}"

export STATS_RECEIVER_NAME="statsd"
export STATS_RECEIVER_STATSD_ADDRESS="127.0.0.1:8125"
export STATS_RECEIVER_STATSD_PREFIX="snowplow.snowbridge"
export STATS_RECEIVER_TIMEOUT_SEC=2
export STATS_RECEIVER_BUFFER_SEC=20

# Telemetry
export DISABLE_TELEMETRY=false
export USER_PROVIDED_ID="user@acme.com"
```

--------------------------------

### Configure HTTP Source

Source: https://context7.com/snowplow/snowbridge/llms.txt

Sets up an experimental HTTP server to receive incoming data batches.

```hcl
source {
  use "http" {
    # HTTP server bind address (required)
    url = "localhost:8080"

    # Receiver endpoint path (default: /)
    path = "/receiver"

    # Maximum events per request (default: 50)
    request_batch_limit = 15
  }
}
```

--------------------------------

### Configure Stream Replicator with Environment Variables

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Logging,-Observability-and-Sentry

Set logging level, Sentry DSN, debug mode, and tags using environment variables. Configure StatsD address, prefix, timeout, and buffer size similarly.

```bash
# logging
export LOG_LEVEL="debug"

# reporting and stats
export SENTRY_DSN="https://acme.com/1"    \
SENTRY_DEBUG=true                         \
SENTRY_TAGS="{\"aKey\":\"aValue\"}"

export STATS_RECEIVER_NAME="statsd"                       \
STATS_RECEIVER_STATSD_ADDRESS="127.0.0.1:8125"            \
STATS_RECEIVER_STATSD_PREFIX="snowplow.stream-replicator" \
STATS_RECEIVER_TIMEOUT_SEC=2                              \
STATS_RECEIVER_BUFFER_SEC=20
```

--------------------------------

### Analyze Heap Profile with pprof

Source: https://github.com/snowplow/snowbridge/wiki/Profiling-with-pprof

Use curl to download the heap profile and the go tool pprof to inspect the data.

```bash
host$ curl http://localhost:8080/debug/pprof/heap > heap.out
host$ go tool pprof heap.out
Type: inuse_space
Time: Jan 8, 2021 at 9:42am (CET)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 8197.61kB, 100% of 8197.61kB total
Showing top 10 nodes out of 45
      flat  flat%   sum%        cum   cum%
 3072.56kB 37.48% 37.48%  4608.63kB 56.22%  github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil.XMLToStruct
 1536.07kB 18.74% 56.22%  1536.07kB 18.74%  github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil.(*XMLNode).findNamespaces
 1024.16kB 12.49% 68.71%  1024.16kB 12.49%  github.com/aws/aws-sdk-go/aws/endpoints.init
  514.63kB  6.28% 74.99%   514.63kB  6.28%  bytes.makeSlice
     514kB  6.27% 81.26%      514kB  6.27%  bufio.NewReaderSize
  512.10kB  6.25% 87.51%   512.10kB  6.25%  fmt.glob..func1
  512.07kB  6.25% 93.75%   512.07kB  6.25%  net/url.parse
  512.02kB  6.25%   100%   512.02kB  6.25%  crypto/tls.(*Conn).makeClientHello
         0     0%   100%   514.63kB  6.28%  bytes.(*Buffer).Write
         0     0%   100%   514.63kB  6.28%  bytes.(*Buffer).grow
(pprof)
```

--------------------------------

### Configure Kinesis Source with HCL

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Sources

Use this HCL block to configure the Kinesis source. Specify the stream name, AWS region, application name, and optionally a role ARN, start timestamp, and concurrent writes.

```hcl
source {
  use "kinesis" {
    # Kinesis stream name to read from (required)
    stream_name       = "my-stream"

    # AWS region of Kinesis stream (required)
    region            = "us-west-1"

    # App name for Stream Replicator (required)
    app_name          = "StreamReplicatorProd1"

    # Optional ARN to use on source stream (default: "")
    role_arn          = "arn:aws:iam::123456789012:role/myrole"

    # Timestamp for the kinesis shard iterator to begin processing.
    # Format YYYY-MM-DD HH:MM:SS.MS (miliseconds optional)
    # (default: TRIM_HORIZON)
    start_timestamp   = "2020-01-01 10:00:00"

    # Number of events to process concurrently (default: 50)
    concurrent_writes = 15
  }
}
```

--------------------------------

### Configure Pub/Sub Target with Environment Variables

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Targets

Set these environment variables to configure the Pub/Sub target, including the target name, GCP project ID, and topic name.

```bash
export TARGET_NAME="pubsub"                 \
TARGET_PUBSUB_PROJECT_ID="acme-project"     \
TARGET_PUBSUB_TOPIC_NAME="some-acme-topic"
```

--------------------------------

### Basic JS and Lua Script Configuration

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Transformations-and-Filters

Demonstrates the basic structure for configuring JavaScript and Lua scripts using base64 encoded source code.

```hcl
transform {
  use "js" {
    source_b64 = "ZnVuY3Rpb24gbWFpbihpbnB1dCkgewogICAgcmV0dXJuIHsgRGF0YTogIkhlbGxvIFdvcmxkIiB9Cn0="
  }
}

transform {
  use "lua" {
    source_b64 = "ZnVuY3Rpb24gbWFpbihpbnB1dCkKICByZXR1cm4gaW5wdXQKZW5k"
  }
}
```

--------------------------------

### Run linter

Source: https://github.com/snowplow/snowbridge/blob/master/README.md

Executes golangci-lint to verify code standards.

```bash
make lint
```

--------------------------------

### Configure Telemetry via Environment Variables

Source: https://github.com/snowplow/snowbridge/wiki/Telemetry

Set telemetry configuration options using environment variables. Use `export` to set them for the current session.

```bash
export DISABLE_TELEMETRY=false         \
USER_PROVIDED_ID="elmer.fudd@acme.com"
```

--------------------------------

### Configure Apache Kafka Source

Source: https://context7.com/snowplow/snowbridge/llms.txt

Sets up a Kafka consumer with support for SASL authentication and TLS encryption.

```hcl
source {
  use "kafka" {
    # Kafka broker connection string (required)
    brokers = "my-kafka-connection-string"

    # Kafka topic name (required)
    topic_name = "snowplow-enriched-good"

    # Kafka consumer group name (required)
    consumer_name = "snowplow-stream-replicator"

    # Offset configuration: -1 = new messages, -2 = oldest available
    offsets_initial = -2

    # Kafka assignor strategy
    assignor = "sticky"

    # The Kafka version
    target_version = "2.7.0"

    # SASL authentication
    enable_sasl = true
    sasl_username = "mySaslUsername"
    sasl_password = env.SASL_PASSWORD
    sasl_algorithm = "sha256"  # "plaintext", "sha512", or "sha256"
    sasl_version = 1           # 0 or 1

    # TLS configuration
    enable_tls = true
    cert_file = "myLocalhost.crt"
    key_file = "myLocalhost.key"
    ca_file = "myRootCA.crt"
    skip_verify_tls = false
  }
}
```

--------------------------------

### Run Stream Replicator Binary

Source: https://github.com/snowplow/snowbridge/wiki/Runtime:-CLI-(Standalone)

After downloading or compiling the stream replicator, run the executable. This is a long-running process that can be stopped with Ctrl+C or on error. Ensure environment variables or a configuration file are set.

```bash
./stream-replicator
```

--------------------------------

### Run integration tests

Source: https://github.com/snowplow/snowbridge/blob/master/README.md

Manages the lifecycle of Docker containers for integration testing.

```makefile
make integration-up
```

```makefile
make integration-test
```

```makefile
make integration-down
```

--------------------------------

### Advanced Lua Script Configuration

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Transformations-and-Filters

Illustrates advanced configuration for Lua scripts, including timeout, sandbox mode, and Snowplow mode.

```hcl
# lua configuration
transform {
  use "lua" {
    source_b64 = "ZnVuY3Rpb24gbWFpbihpbnB1dCkKICByZXR1cm4gaW5wdXQKZW5k"
    timeout_sec = 20
    sandbox     = true
    snowplow_mode = false
  }
}
```

--------------------------------

### Configure transformations via environment variables

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Transformations-and-Filters

Provide a base64-encoded HCL configuration string to the TRANSFORM_CONFIG_B64 environment variable.

```bash
export TRANSFORM_CONFIG_B64="dHJhbnNmb3JtIHsKICB1c2UgInNwRW5yaWNoZWRGaWx0ZXIiIHsKICAgICMga2VlcCBvbmx5IHBhZ2Ugdmlld3MKICAgIGF0b21pY19maWVsZCA9ICJldmVudF9uYW1lIgogICAgcmVnZXggPSAiXnBhZ2VfdmlldyQiCiAgfQp9Cgp0cmFuc2Zvcm0gewogIHVzZSAianMiIHsKICAgICMgY2hhbmdlcyBhcHBfaWQgdG8gIjEiCiAgICBzb3VyY2VfYjY0ID0gIlpuVnVZM1JwYjI0Z2JXRnBiaWg0S1NCN0NpQWdJQ0IyWVhJZ2FuTnZiazlpYWlBOUlFcFRUMDR1Y0dGeWMyVW9lQzVFWVhSaEtUc0tJQ0FnSUdwemIyNVBZbXBiSW1Gd2NGOXBaQ0pkSUQwZ0lqRWlPd29nSUNBZ2NtVjBkWEp1SUhzS0lDQWdJQ0FnSUNCRVlYUmhPaUJLVTA5T0xuTjBjbWx1WjJsbWVTaHFjMjl1VDJKcUtRb2dJQ0FnZlRzS2ZRPT0iCiAgfQp9"
```

--------------------------------

### Configure HCL File Path

Source: https://github.com/snowplow/snowbridge/wiki/Home

Set the environment variable to point to the HCL configuration file.

```bash
export STREAM_REPLICATOR_CONFIG_FILE="/Users/example_user/conf.hcl"
```

--------------------------------

### Configure Reporting and Stats Environment Variables

Source: https://github.com/snowplow/snowbridge/wiki/Home

Set these environment variables to enable Sentry error tracking, StatsD metrics collection, and general telemetry reporting.

```bash
export SENTRY_DSN="https://acme.com/1"    \
SENTRY_DEBUG=true                         \
SENTRY_TAGS="{\"aKey\":\"aValue\"}"

export STATS_RECEIVER_NAME="statsd"                       \
STATS_RECEIVER_STATSD_ADDRESS="127.0.0.1:8125"            \
STATS_RECEIVER_STATSD_PREFIX="snowplow.stream-replicator" \
STATS_RECEIVER_TIMEOUT_SEC=2                              \
STATS_RECEIVER_BUFFER_SEC=20

export DISABLE_TELEMETRY=false         \
USER_PROVIDED_ID="elmer.fudd@acme.com"
```

--------------------------------

### Run end-to-end tests

Source: https://github.com/snowplow/snowbridge/blob/master/README.md

Builds the project and manages Docker containers for full end-to-end validation.

```makefile
make all
```

```makefile
make e2e-up
```

```makefile
make e2e-test
```

```makefile
make e2e-down
```

--------------------------------

### Run unit tests

Source: https://github.com/snowplow/snowbridge/blob/master/README.md

Executes unit tests while skipping those that require Docker resources.

```bash
go test ./... -short
```

--------------------------------

### Configure failure target using environment variables

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Failure-Targets-&-Formats

Set failure target configuration using environment variables with the FAILURE_ prefix.

```bash
export FAILURE_TARGET_NAME="pubsub"                 \
FAILURE_TARGET_PUBSUB_PROJECT_ID="acme-project"     \
FAILURE_TARGET_PUBSUB_TOPIC_NAME="some-acme-topic
```

--------------------------------

### Environment Variable Configuration

Source: https://github.com/snowplow/snowbridge/wiki/Home

Alternative configuration method using environment variables when no HCL file is provided.

```bash
# source
export SOURCE_NAME="kinesis"                                    \
SOURCE_KINESIS_STREAM_NAME="my-stream"                          \
SOURCE_KINESIS_REGION="us-west-1"                               \
SOURCE_KINESIS_APP_NAME="StreamReplicatorProd1"                 \
SOURCE_KINESIS_ROLE_ARN="arn:aws:iam::123456789012:role/myrole" \
SOURCE_KINESIS_START_TSTAMP="2020-01-01 10:00:00"               \
SOURCE_CONCURRENT_WRITES=15

# transformations
export TRANSFORM_CONFIG_B64="dHJhbnNmb3JtIHsKICB1c2UgInNwRW5yaWNoZWRGaWx0ZXIiIHsKICAgICMga2VlcCBvbmx5IHBhZ2Ugdmlld3MKICAgIGF0b21pY19maWVsZCA9ICJldmVudF9uYW1lIgogICAgcmVnZXggPSAiXnBhZ2VfdmlldyQiCiAgfQp9Cgp0cmFuc2Zvcm0gewogIHVzZSAianMiIHsKICAgICMgY2hhbmdlcyBhcHBfaWQgdG8gIjEiCiAgICBzb3VyY2VfYjY0ID0gIlpuVnVZM1JwYjI0Z2JXRnBiaWg0S1NCN0NpQWdJQ0IyWVhJZ2FuTnZiazlpYWlBOUlFcFRUMDR1Y0dGeWMyVW9lQzVFWVhSaEtUc0tJQ0FnSUdwemIyNVBZbXBiSW1Gd2NGOXBaQ0pkSUQwZ0lqRWlPd29nSUNBZ2NtVjBkWEp1SUhzS0lDQWdJQ0FnSUNCRVlYUmhPaUJLVTA5T0xuTjBjbWx1WjJsbWVTaHFjMjl1VDJKcUtRb2dJQ0FnZlRzS2ZRPT0iCiAgfQp9"

# target
export TARGET_NAME="pubsub"                 \
TARGET_PUBSUB_PROJECT_ID="acme-project"     \
TARGET_PUBSUB_TOPIC_NAME="some-acme-topic"

# logging
export LOG_LEVEL="debug"
```

--------------------------------

### Metrics Configuration

Source: https://context7.com/snowplow/snowbridge/llms.txt

Enable optional metrics collection, including E2E latency tracking and kinsumer memory metrics. Set `enable_e2e_latency` and `enable_kinsumer_memory_metrics` to `true`.

```hcl
metrics {
  # E2E latency: collector timestamp to target write timestamp
  enable_e2e_latency = true

  # Kinsumer memory metrics (records in memory count and bytes)
  enable_kinsumer_memory_metrics = true
}
```

--------------------------------

### Configure PubSub Target

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Targets

Defines the configuration structure for the PubSub target. Options can be set using HCL or environment variables.

```go
type PubSubTargetConfig struct {
	ProjectID string `hcl:"project_id" env:"TARGET_PUBSUB_PROJECT_ID"`
	TopicName string `hcl:"topic_name" env:"TARGET_PUBSUB_TOPIC_NAME"`
}
```

--------------------------------

### PubSub Configuration

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Targets

Configuration parameters for the PubSub target.

```APIDOC
## PubSub Configuration

### Request Body
- **project_id** (string) - Required - TARGET_PUBSUB_PROJECT_ID
- **topic_name** (string) - Required - TARGET_PUBSUB_TOPIC_NAME
```

--------------------------------

### Configure Telemetry via File

Source: https://github.com/snowplow/snowbridge/wiki/Telemetry

Set telemetry configuration options in the HCL configuration file. Ensure `disable_telemetry` is false to enable.

```hcl
disable_telemetry = false
user_provided_id = "elmer.fudd@acme.com"
```

--------------------------------

### stdin Source Configuration

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Sources

Configuration parameters for the stdin data source.

```APIDOC
## stdin Configuration

### Description
Configures the stdin source for debugging purposes.

### Parameters
#### Configuration Fields
- **concurrent_writes** (int) - Optional - Number of events to process concurrently (default: 50)
```

--------------------------------

### Configure Google Pub/Sub Target

Source: https://context7.com/snowplow/snowbridge/llms.txt

Use this configuration to write data to Google Cloud Pub/Sub topics. Requires project ID, topic name, and path to service account credentials.

```hcl
target {
  use "pubsub" {
    batching {
      max_batch_messages     = 20
      max_batch_bytes        = 10000000
      max_message_bytes      = 10000000
      max_concurrent_batches = 2
      flush_period_millis    = 200
    }

    # GCP Project ID (required)
    project_id = "acme-project"

    # Topic name (required)
    topic_name = "some-acme-topic"

    # Path to service account JSON credentials
    credentials_path = "/path/to/service-account.json"
  }
}
```

--------------------------------

### PubSub Source Configuration Struct

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Sources

Defines the structure for PubSub source configuration, mapping HCL and environment variable names to Go struct fields. Requires ProjectID and SubscriptionID.

```go
type configuration struct {
	ProjectID        string `hcl:"project_id" env:"SOURCE_PUBSUB_PROJECT_ID"`
	SubscriptionID   string `hcl:"subscription_id" env:"SOURCE_PUBSUB_SUBSCRIPTION_ID"`
	ConcurrentWrites int    `hcl:"concurrent_writes,optional" env:"SOURCE_CONCURRENT_WRITES"`
}
```

--------------------------------

### Accept Snowbridge License

Source: https://context7.com/snowplow/snowbridge/llms.txt

Set `accept = true` within the `license` block in your configuration to enable Snowbridge. This is a prerequisite for running the tool.

```hcl
license {
  accept = true
}
```

--------------------------------

### Configure HTTP Target

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Targets

Defines the configuration structure for the HTTP target. Options can be set using HCL or environment variables.

```go
type HTTPTargetConfig struct {
	HTTPURL                 string `hcl:"url" env:"TARGET_HTTP_URL"`
	ByteLimit               int    `hcl:"byte_limit,optional" env:"TARGET_HTTP_BYTE_LIMIT"`
	RequestTimeoutInSeconds int    `hcl:"request_timeout_in_seconds,optional" env:"TARGET_HTTP_TIMEOUT_IN_SECONDS"`
	ContentType             string `hcl:"content_type,optional" env:"TARGET_HTTP_CONTENT_TYPE"`
	Headers                 string `hcl:"headers,optional" env:"TARGET_HTTP_HEADERS" `
	BasicAuthUsername       string `hcl:"basic_auth_username,optional" env:"TARGET_HTTP_BASICAUTH_USERNAME"`
	BasicAuthPassword       string `hcl:"basic_auth_password,optional" env:"TARGET_HTTP_BASICAUTH_PASSWORD"`
	CertFile                string `hcl:"cert_file,optional" env:"TARGET_HTTP_TLS_CERT_FILE"`
	KeyFile                 string `hcl:"key_file,optional" env:"TARGET_HTTP_TLS_KEY_FILE"`
	CaFile                  string `hcl:"ca_file,optional" env:"TARGET_HTTP_TLS_CA_FILE"`
	SkipVerifyTLS           bool   `hcl:"skip_verify_tls,optional" env:"TARGET_HTTP_TLS_SKIP_VERIFY_TLS"` // false
}
```

--------------------------------

### Configure Stream Replicator with HCL

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Logging,-Observability-and-Sentry

Use HCL to set logging levels, Sentry DSN and debug mode, and StatsD server address and prefix. Ensure tags are provided as escaped JSON strings.

```hcl
// log level configuration (default: "info")
log_level = "info"

sentry {
  # The DSN to send Sentry alerts to
  dsn   = "https://acme.com/1"

  # Whether to put Sentry into debug mode (default: false)
  debug = true

  # Escaped JSON string with tags to send to Sentry (default: "{}")
  tags  = "{\"aKey\":\"aValue\"}"
}

stats_receiver {
  use "statsd" {
    # StatsD server address
    address = "127.0.0.1:8125"

    # StatsD metric prefix (default: "snowplow.stream-replicator")
    prefix  = "snowplow.stream-replicator"

    # Escaped JSON string with tags to send to StatsD (default: "{}")
    tags    = "{\"aKey\": \"aValue\"}"
  }

  # Time (seconds) the observer waits for new results (default: 1)
  timeout_sec = 2

  # Aggregation time window (seconds) for metrics being collected (default: 15)
  buffer_sec  = 20
}
```

--------------------------------

### PubSub Source Configuration

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Sources

Configuration parameters for the GCP PubSub data source.

```APIDOC
## PubSub Configuration

### Description
Configures the GCP PubSub topic source for data replication.

### Parameters
#### Configuration Fields
- **project_id** (string) - Required - GCP Project ID
- **subscription_id** (string) - Required - PubSub Subscription ID
- **concurrent_writes** (int) - Optional - Number of events to process concurrently (default: 50)
```

--------------------------------

### Configure Kafka Target

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Targets

Defines the configuration structure for the Kafka target. Options can be set using HCL or environment variables.

```go
type KafkaConfig struct {
	Brokers        string `hcl:"brokers" env:"TARGET_KAFKA_BROKERS"`
	TopicName      string `hcl:"topic_name" env:"TARGET_KAFKA_TOPIC_NAME"`
	TargetVersion  string `hcl:"target_version,optional" env:"TARGET_KAFKA_TARGET_VERSION"`
	MaxRetries     int    `hcl:"max_retries,optional" env:"TARGET_KAFKA_MAX_RETRIES"`
	ByteLimit      int    `hcl:"byte_limit,optional" env:"TARGET_KAFKA_BYTE_LIMIT"`
	Compress       bool   `hcl:"compress,optional" env:"TARGET_KAFKA_COMPRESS"`
	WaitForAll     bool   `hcl:"wait_for_all,optional" env:"TARGET_KAFKA_WAIT_FOR_ALL"`
	Idempotent     bool   `hcl:"idempotent,optional" env:"TARGET_KAFKA_IDEMPOTENT"`
	EnableSASL     bool   `hcl:"enable_sasl,optional" env:"TARGET_KAFKA_ENABLE_SASL"`
	SASLUsername   string `hcl:"sasl_username,optional" env:"TARGET_KAFKA_SASL_USERNAME" `
	SASLPassword   string `hcl:"sasl_password,optional" env:"TARGET_KAFKA_SASL_PASSWORD"`
	SASLAlgorithm  string `hcl:"sasl_algorithm,optional" env:"TARGET_KAFKA_SASL_ALGORITHM"`
	CertFile       string `hcl:"cert_file,optional" env:"TARGET_KAFKA_TLS_CERT_FILE"`
	KeyFile        string `hcl:"key_file,optional" env:"TARGET_KAFKA_TLS_KEY_FILE"`
	CaFile         string `hcl:"ca_file,optional" env:"TARGET_KAFKA_TLS_CA_FILE"`
	SkipVerifyTLS  bool   `hcl:"skip_verify_tls,optional" env:"TARGET_KAFKA_TLS_SKIP_VERIFY_TLS"`
	ForceSync      bool   `hcl:"force_sync_producer,optional" env:"TARGET_KAFKA_FORCE_SYNC_PRODUCER"`
	FlushFrequency int    `hcl:"flush_frequency,optional" env:"TARGET_KAFKA_FLUSH_FREQUENCY"`
	FlushMessages  int    `hcl:"flush_messages,optional" env:"TARGET_KAFKA_FLUSH_MESSAGES"`
	FlushBytes     int    `hcl:"flush_bytes,optional" env:"TARGET_KAFKA_FLUSH_BYTES"`
}
```

--------------------------------

### Configure HTTP Target with Batching and Auth

Source: https://context7.com/snowplow/snowbridge/llms.txt

Use this configuration to send data to HTTP endpoints. Supports batching, OAuth2, templating, and custom response rules. Ensure environment variables are set for sensitive credentials.

```hcl
target {
  use "http" {
    batching {
      max_batch_messages     = 20
      max_batch_bytes        = 10000000
      max_message_bytes      = 10000000
      max_concurrent_batches = 2
      flush_period_millis    = 200
    }

    # URL endpoint (required)
    url = "https://acme.com/api/events"

    # Request timeout (default: 5000ms)
    request_timeout_in_millis = 2000

    # Content type (default: "application/json")
    content_type = "application/json"

    # Custom headers
    headers = {
      Accept-Language = "en-US"
    }

    # Basic auth
    basic_auth_username = "myUsername"
    basic_auth_password = env.MY_AUTH_PASSWORD

    # TLS configuration
    enable_tls = true
    cert_file = "myLocalhost.crt"
    key_file = "myLocalhost.key"
    ca_file = "myRootCA.crt"
    skip_verify_tls = false

    # OAuth2 authentication
    oauth2_client_id = env.CLIENT_ID
    oauth2_client_secret = env.CLIENT_SECRET
    oauth2_refresh_token = env.REFRESH_TOKEN
    oauth2_token_url = "https://oauth2.googleapis.com/token"

    # Request template file
    template_file = "myTemplate.file"

    # Dynamic headers (events sent individually, not batched)
    dynamic_headers = true

    # Response handling rules
    response_rules {
      rule {
        type = "invalid"
        http_codes = [400]
        body = "Invalid value for 'purchase' field"
      }
      rule {
        type = "throttle"
        http_codes = [429]
      }
      rule {
        type = "setup"
        http_codes = [401, 403]
      }
      rule {
        type = "fatal"
        http_codes = [413]
      }
    }
  }
}
```

--------------------------------

### Configure Google Pub/Sub Source

Source: https://context7.com/snowplow/snowbridge/llms.txt

Configures a subscription to a Google Cloud Pub/Sub topic with specific message handling and connection pool limits.

```hcl
source {
  use "pubsub" {
    # GCP Project ID (required)
    project_id = "project-id"

    # Subscription ID (required)
    subscription_id = "subscription-id"

    # Maximum unprocessed messages (default: 1000)
    max_outstanding_messages = 2000

    # Maximum size of unprocessed messages in bytes (default: 1e9)
    max_outstanding_bytes = 2e9

    # Minimum ack extension period when message received
    min_extension_period_seconds = 10

    # Number of streaming pull connections
    streaming_pull_goroutines = 1

    # GRPC connection pool size
    grpc_connection_pool_size = 4
  }
}
```

--------------------------------

### stdin Source Configuration Struct

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Sources

Defines the structure for stdin source configuration, mapping HCL and environment variable names to Go struct fields. Only supports optional ConcurrentWrites.

```go
type configuration struct {
	ConcurrentWrites int `hcl:"concurrent_writes,optional" env:"SOURCE_CONCURRENT_WRITES"`
}
```

--------------------------------

### Configure EventHubs Target

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Targets

Defines the configuration structure for the EventHubs target. Options can be set using HCL or environment variables.

```go
type EventHubConfig struct {
	EventHubNamespace       string `hcl:"namespace" env:"TARGET_EVENTHUB_NAMESPACE"`
	EventHubName            string `hcl:"name" env:"TARGET_EVENTHUB_NAME"`
	MaxAutoRetries          int    `hcl:"max_auto_retries,optional" env:"TARGET_EVENTHUB_MAX_AUTO_RETRY"`
	MessageByteLimit        int    `hcl:"message_byte_limit,optional" env:"TARGET_EVENTHUB_MESSAGE_BYTE_LIMIT"`
	ChunkByteLimit          int    `hcl:"chunk_byte_limit,optional" env:"TARGET_EVENTHUB_CHUNK_BYTE_LIMIT"`
	ChunkMessageLimit       int    `hcl:"chunk_message_limit,optional" env:"TARGET_EVENTHUB_CHUNK_MESSAGE_LIMIT"`
	ContextTimeoutInSeconds int    `hcl:"context_timeout_in_seconds,optional" env:"TARGET_EVENTHUB_CONTEXT_TIMEOUT_SECONDS"`
	BatchByteLimit          int    `hcl:"batch_byte_limit,optional" env:"TARGET_EVENTHUB_BATCH_BYTE_LIMIT"`
	SetEHPartitionKey       bool   `hcl:"set_eh_partition_key,optional" env:"TARGET_EVENTHUB_SET_EH_PK"`
}
```

--------------------------------

### SQS Configuration

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Targets

Configuration parameters for the SQS target.

```APIDOC
## SQS Configuration

### Request Body
- **queue_name** (string) - Required - TARGET_SQS_QUEUE_NAME
- **region** (string) - Required - TARGET_SQS_REGION
- **role_arn** (string) - Optional - TARGET_SQS_ROLE_ARN
```

--------------------------------

### EventHubs Configuration

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Targets

Configuration parameters for the EventHubs target.

```APIDOC
## EventHubs Configuration

### Request Body
- **namespace** (string) - Required - TARGET_EVENTHUB_NAMESPACE
- **name** (string) - Required - TARGET_EVENTHUB_NAME
- **max_auto_retries** (int) - Optional - TARGET_EVENTHUB_MAX_AUTO_RETRY
- **message_byte_limit** (int) - Optional - TARGET_EVENTHUB_MESSAGE_BYTE_LIMIT
- **chunk_byte_limit** (int) - Optional - TARGET_EVENTHUB_CHUNK_BYTE_LIMIT
- **chunk_message_limit** (int) - Optional - TARGET_EVENTHUB_CHUNK_MESSAGE_LIMIT
- **context_timeout_in_seconds** (int) - Optional - TARGET_EVENTHUB_CONTEXT_TIMEOUT_SECONDS
- **batch_byte_limit** (int) - Optional - TARGET_EVENTHUB_BATCH_BYTE_LIMIT
- **set_eh_partition_key** (bool) - Optional - TARGET_EVENTHUB_SET_EH_PK
```

--------------------------------

### Kafka Configuration

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Targets

Configuration parameters for the Kafka target.

```APIDOC
## Kafka Configuration

### Request Body
- **brokers** (string) - Required - TARGET_KAFKA_BROKERS
- **topic_name** (string) - Required - TARGET_KAFKA_TOPIC_NAME
- **target_version** (string) - Optional - TARGET_KAFKA_TARGET_VERSION
- **max_retries** (int) - Optional - TARGET_KAFKA_MAX_RETRIES
- **byte_limit** (int) - Optional - TARGET_KAFKA_BYTE_LIMIT
- **compress** (bool) - Optional - TARGET_KAFKA_COMPRESS
- **wait_for_all** (bool) - Optional - TARGET_KAFKA_WAIT_FOR_ALL
- **idempotent** (bool) - Optional - TARGET_KAFKA_IDEMPOTENT
- **enable_sasl** (bool) - Optional - TARGET_KAFKA_ENABLE_SASL
- **sasl_username** (string) - Optional - TARGET_KAFKA_SASL_USERNAME
- **sasl_password** (string) - Optional - TARGET_KAFKA_SASL_PASSWORD
- **sasl_algorithm** (string) - Optional - TARGET_KAFKA_SASL_ALGORITHM
- **cert_file** (string) - Optional - TARGET_KAFKA_TLS_CERT_FILE
- **key_file** (string) - Optional - TARGET_KAFKA_TLS_KEY_FILE
- **ca_file** (string) - Optional - TARGET_KAFKA_TLS_CA_FILE
- **skip_verify_tls** (bool) - Optional - TARGET_KAFKA_TLS_SKIP_VERIFY_TLS
- **force_sync_producer** (bool) - Optional - TARGET_KAFKA_FORCE_SYNC_PRODUCER
- **flush_frequency** (int) - Optional - TARGET_KAFKA_FLUSH_FREQUENCY
- **flush_messages** (int) - Optional - TARGET_KAFKA_FLUSH_MESSAGES
- **flush_bytes** (int) - Optional - TARGET_KAFKA_FLUSH_BYTES
```

--------------------------------

### Run Snowbridge CLI with HCL Config

Source: https://context7.com/snowplow/snowbridge/llms.txt

Run the Snowbridge CLI by setting the `STREAM_REPLICATOR_CONFIG_FILE` environment variable to the path of your HCL configuration file. Docker execution is also demonstrated.

```bash
# Set configuration file path
export STREAM_REPLICATOR_CONFIG_FILE="/path/to/config.hcl"

# Run Snowbridge
./snowbridge

# Or with Docker
docker run -v /path/to/config.hcl:/config.hcl \
  -e STREAM_REPLICATOR_CONFIG_FILE=/config.hcl \
  snowplow/snowbridge:latest
```

--------------------------------

### Configure Kafka Target with SASL and TLS

Source: https://context7.com/snowplow/snowbridge/llms.txt

Use this configuration to write data to Apache Kafka topics. Supports SASL, TLS, compression, and exactly-once semantics. Ensure environment variables are set for sensitive credentials.

```hcl
target {
  use "kafka" {
    batching {
      max_batch_messages     = 20
      max_batch_bytes        = 10000000
      max_message_bytes      = 10000000
      max_concurrent_batches = 2
      flush_period_millis    = 200
    }

    # Kafka broker connection string (required)
    brokers = "my-kafka-connection-string"

    # Kafka topic name (required)
    topic_name = "snowplow-enriched-good"

    # Kafka version
    target_version = "2.7.0"

    # Max retries (default: 5)
    max_retries = 11

    # Enable compression (default: false)
    compress = true

    # Wait for min.insync.replicas to ACK (default: false)
    wait_for_all = true

    # Exactly-once writes (default: false)
    idempotent = true

    # SASL authentication
    enable_sasl = true
    sasl_username = "mySaslUsername"
    sasl_password = env.SASL_PASSWORD
    sasl_algorithm = "sha256"
    sasl_version = 1

    # TLS configuration
    enable_tls = true
    cert_file = "myLocalhost.crt"
    key_file = "myLocalhost.key"
    ca_file = "myRootCA.crt"
    skip_verify_tls = false

    # Force synchronous producer (default: false)
    force_sync_producer = true
  }
}
```

--------------------------------

### HTTP Target Configuration

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Targets

Configuration parameters for the HTTP target.

```APIDOC
## HTTP Target Configuration

### Request Body
- **url** (string) - Required - TARGET_HTTP_URL
- **byte_limit** (int) - Optional - TARGET_HTTP_BYTE_LIMIT
- **request_timeout_in_seconds** (int) - Optional - TARGET_HTTP_TIMEOUT_IN_SECONDS
- **content_type** (string) - Optional - TARGET_HTTP_CONTENT_TYPE
- **headers** (string) - Optional - TARGET_HTTP_HEADERS
- **basic_auth_username** (string) - Optional - TARGET_HTTP_BASICAUTH_USERNAME
- **basic_auth_password** (string) - Optional - TARGET_HTTP_BASICAUTH_PASSWORD
- **cert_file** (string) - Optional - TARGET_HTTP_TLS_CERT_FILE
- **key_file** (string) - Optional - TARGET_HTTP_TLS_KEY_FILE
- **ca_file** (string) - Optional - TARGET_HTTP_TLS_CA_FILE
- **skip_verify_tls** (bool) - Optional - TARGET_HTTP_TLS_SKIP_VERIFY_TLS
```

--------------------------------

### Configure JavaScript Transformation

Source: https://context7.com/snowplow/snowbridge/llms.txt

Sets up custom JavaScript execution for data processing, supporting both external files and inline scripts.

```hcl
transform {
  use "js" {
    # Path to JavaScript file
    script_path = "/path/to/script.js"

    # Or inline script
    script = "function main(input) { return input }"

    # Execution timeout in seconds
    timeout_sec = 20

    # Enable Snowplow TSV to JSON conversion
    snowplow_mode = true

    # Enable JSON mode for generic JSON data
    json_mode = true

    # Remove null/empty keys from output
    remove_nulls = true

    # Salt for hash function
    hash_salt_secret = env.SHA1_SALT
  }
}
```

--------------------------------

### Configure AWS SQS Source

Source: https://context7.com/snowplow/snowbridge/llms.txt

Defines an SQS queue source, supporting custom endpoints for local development environments like LocalStack.

```hcl
source {
  use "sqs" {
    # SQS queue name (required)
    queue_name = "mySqsQueue"

    # AWS region of SQS queue (required)
    region = "us-west-1"

    # Role ARN to use on source queue
    role_arn = "arn:aws:iam::123456789012:role/myrole"

    # Custom endpoint for local testing (e.g., localstack)
    custom_aws_endpoint = "http://localhost:4566"
  }
}
```

--------------------------------

### Configure SQS Target

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Targets

Defines the configuration structure for the SQS target. Options can be set using HCL or environment variables.

```go
type SQSTargetConfig struct {
	QueueName string `hcl:"queue_name" env:"TARGET_SQS_QUEUE_NAME"`
	Region    string `hcl:"region" env:"TARGET_SQS_REGION"`
	RoleARN   string `hcl:"role_arn,optional" env:"TARGET_SQS_ROLE_ARN"`
}
```

--------------------------------

### SQS Source Configuration

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Sources

Configuration parameters for the Amazon SQS data source.

```APIDOC
## SQS Configuration

### Description
Configures the Amazon SQS queue source for data replication.

### Parameters
#### Configuration Fields
- **queue_name** (string) - Required - SQS Queue name
- **region** (string) - Required - AWS region of SQS queue
- **role_arn** (string) - Optional - ARN to use on source queue
- **concurrent_writes** (int) - Optional - Number of events to process concurrently (default: 50)
```

--------------------------------

### Configure Partition Key Setting

Source: https://context7.com/snowplow/snowbridge/llms.txt

Sets the partition key using a specified Snowplow atomic field.

```hcl
transform {
  use "spEnrichedSetPk" {
    # Atomic field to use as partition key
    atomic_field = "app_id"
  }
}
```

--------------------------------

### Pull Stream Replicator Docker Images

Source: https://github.com/snowplow/snowbridge/wiki/Runtime:-CLI-(Standalone)

Use these commands to pull the pre-compiled Docker images for the stream replicator. Choose the AWS or GCP version based on your deployment environment.

```bash
docker pull snowplow/stream-replicator-aws:1.0.0
```

```bash
docker pull snowplow/stream-replicator-gcp:1.0.0
```

--------------------------------

### Lua Scripting Interface - Basic

Source: https://github.com/snowplow/snowbridge/wiki/Config:-Transformations-and-Filters

Define a main function in Lua that accepts an input table and returns it. This serves as the basic structure for custom transformations.

```lua
function main(input)
  return input
end

```