### Salesforce Connected App Setup Output

Source: https://github.com/redpanda-data/connect/blob/main/internal/impl/salesforce/README.md

Example output from the setup script, showing a deep-link to the Connected App's Setup page in Salesforce.

```bash
Connected App: https://<org>.salesforce.com/lightning/setup/ConnectedApplication/page?address=%2F<id>
```

--------------------------------

### Quickstart Benchmark Commands

Source: https://github.com/redpanda-data/connect/blob/main/internal/impl/aws/s3/bench/write/redpanda-connect/README.md

Use these commands to quickly start, run, and stop the Redpanda Connect S3 sink benchmark. Ensure Docker and Taskfile runner are installed.

```bash
task up
task bench:run COUNT=3000000
task down
```

--------------------------------

### Setup Kafka Connector Infrastructure

Source: https://github.com/redpanda-data/connect/blob/main/internal/impl/postgresql/bench/README.md

Navigates to the Kafka connector directory and starts the Docker environment for Kafka and PostgreSQL.

```bash
cd kafka-connector
task up
```

--------------------------------

### Minimal Select Example (Cassandra/Scylla)

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/inputs/cassandra.adoc

Example configuration for selecting all data from a table in Cassandra or Scylla. Assumes a specific network setup.

```yaml
input:
  cassandra:
    addresses:
      - 172.17.0.2
    query:
      'SELECT * FROM learn_cassandra.users_by_country'
```

--------------------------------

### Example Start Timestamp Configuration

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/inputs/gcp_spanner_cdc.adoc

Example of how to configure the `start_timestamp` field to specify an inclusive start time for reading from the change stream.

```yaml
start_timestamp: "2022-01-01T00:00:00Z"
```

--------------------------------

### Timeout Configuration Examples

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/outputs/gcp_cloud_storage.adoc

Examples demonstrating how to configure the upload timeout.

```yaml
timeout: 1s

timeout: 500ms
```

--------------------------------

### Install Redpanda Connect Prerequisites

Source: https://github.com/redpanda-data/connect/blob/main/CLAUDE.md

Installs necessary tools like brew, redpanda, python3, jq, and the rpk connect command. Ensure these are available before proceeding with installation.

```bash
brew install redpanda-data/tap/redpanda python3 jq
rpk connect install
```

--------------------------------

### Example Configuration File

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/configuration/pages/unit_testing.adoc

This is an example of a Redpanda Connect configuration file with input, pipeline, and output sections.

```yaml
input:
  kafka:
    addresses: [ TODO ]
    topics: [ foo, bar ]
    consumer_group: foogroup

pipeline:
  processors:
  - mapping: '"%vend".format(content().uppercase().string())'

output:
  aws_s3:
    bucket: TODO
    path: '${! meta("kafka_topic") }/${! json("message.id") }.json'
```

--------------------------------

### Batching Configuration Examples

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/inputs/aws_kinesis.adoc

Examples demonstrating different batching configurations for byte size, count, and period.

```yaml
batching:
  byte_size: 5000
  count: 0
  period: 1s
```

```yaml
batching:
  count: 10
  period: 1s
```

```yaml
batching:
  check: this.contains("END BATCH")
  count: 0
  period: 1m
```

--------------------------------

### Qdrant gRPC Host Examples

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/processors/qdrant.adoc

Examples demonstrating how to specify the gRPC host for the Qdrant server.

```yaml
# Examples

grpc_host: localhost:6334

grpc_host: xyz-example.eu-central.aws.cloud.qdrant.io:6334
```

--------------------------------

### Quick Start Tasks

Source: https://github.com/redpanda-data/connect/blob/main/resources/docker/redpanda/README.md

Commands to quickly set up and start the Redpanda cluster and access the console.

```bash
task setup    # Download docker-compose.yml
task up       # Start the cluster
task console  # Open Redpanda Console in browser
```

--------------------------------

### Filesystem Path Example

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/outputs/azure_data_lake_gen2.adoc

Example of configuring the filesystem path using timestamp interpolation.

```yml
filesystem: messages-${!timestamp("2006")}
```

--------------------------------

### CosmosDB Endpoint Configuration Example

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/inputs/azure_cosmosdb.adoc

Example showing how to configure the CosmosDB endpoint.

```yaml
endpoint: https://localhost:8081
```

--------------------------------

### CosmosDB Container Configuration Example

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/inputs/azure_cosmosdb.adoc

Example showing how to configure the CosmosDB container name.

```yaml
container: testcontainer
```

--------------------------------

### Password Configuration Examples

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/caches/redis.adoc

Examples for configuring passwords, including plain text and environment variable substitution.

```yaml
# Examples

password: foo

password: ${KEY_PASSWORD}
```

--------------------------------

### OAuth2 Configuration Example

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/inputs/otlp_grpc.adoc

Example for enabling and configuring OAuth2 authentication.

```yaml
oauth2:
  enabled: true
  client_key: "your_client_key"
  client_secret: "your_client_secret"
  token_url: "http://your.token.provider/oauth2/token"
  scopes:
    - "read"
    - "write"
  endpoint_params:
    audience: ["your_audience"]
```

--------------------------------

### Kafka Partition Configuration Example

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/outputs/ockam_kafka.adoc

Example of how to set the Kafka partition using interpolation based on metadata.

```yaml
partition: ${! meta("partition") }
```

--------------------------------

### CosmosDB Database Configuration Example

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/inputs/azure_cosmosdb.adoc

Example showing how to configure the CosmosDB database name.

```yaml
database: testdb
```

--------------------------------

### Install PostgreSQL Client

Source: https://github.com/redpanda-data/connect/blob/main/internal/impl/postgresql/bench/README.md

Installs the PostgreSQL client on macOS using Homebrew or on Ubuntu/Debian using apt.

```bash
# macOS
brew install postgresql

# Ubuntu/Debian
sudo apt install postgresql-client
```

--------------------------------

### Install Redpanda Connect with External Dependencies

Source: https://github.com/redpanda-data/connect/blob/main/README.md

Use this command to install Redpanda Connect when it requires external C libraries like `zmq4`. Ensure you have the necessary system libraries installed.

```shell
# With go
go install -tags "x_benthos_extra" github.com/redpanda-data/connect/v4/cmd/redpanda-connect@latest

# With task
TAGS=x_benthos_extra task build:all
```

--------------------------------

### Start MySQL and Kafka Containers

Source: https://github.com/redpanda-data/connect/blob/main/internal/impl/mysql/bench/mysql-read/rpcn/README.md

Starts the necessary Docker containers for MySQL and Kafka and waits for them to be ready.

```bash
task up
```

--------------------------------

### Batching Processors Examples

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/outputs/gcp_cloud_storage.adoc

Examples of using processors to archive batches in different formats.

```yaml
processors:
  - archive:
      format: concatenate

processors:
  - archive:
      format: lines

processors:
  - archive:
      format: json_array
```

--------------------------------

### Batching Period Examples

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/inputs/postgres_cdc.adoc

Examples of configuring the batching period using different time units.

```yaml
# Examples

period: 1s

period: 1m

period: 500ms
```

--------------------------------

### Location Configuration Example

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/processors/gcp_vertex_ai_chat.adoc

Example showing how to configure the 'location' field for the Vertex AI processor.

```yaml
location: us-central1
```

--------------------------------

### SQL Cache Configuration Example

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/caches/sql.adoc

This example shows the configuration fields for the SQL cache component, including required and optional parameters like driver, DSN, table, key/value columns, and an example init_statement.

```yaml
sql:
  driver: ""
  dsn: "clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60"
  table: "foo"
  key_column: "foo"
  value_column: "bar"
  set_suffix: "ON DUPLICATE KEY UPDATE bar=VALUES(bar)"
  init_files: []
  init_statement: |
    CREATE TABLE IF NOT EXISTS some_table (
      foo varchar(50) not null,
      bar integer,
      baz varchar(50),
      primary key (foo)
    ) WITHOUT ROWID;
  conn_max_idle_time: ""
  conn_max_life_time: ""
  conn_max_idle: 2
  conn_max_open: 0
```

--------------------------------

### HDFS Hosts Configuration Example

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/inputs/hdfs.adoc

Example of how to configure the 'hosts' field for the HDFS input connector.

```yaml
hosts: localhost:9000
```

--------------------------------

### OpenAI Speech Processor Model Examples

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/processors/openai_speech.adoc

Examples demonstrating how to specify the OpenAI model for text-to-speech.

```yaml
model: tts-1
```

```yaml
model: tts-1-hd
```

--------------------------------

### Avro Schema Path Example

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/processors/avro.adoc

Examples demonstrating how to specify the path to an Avro schema file or URL.

```yaml
schema_path: file:///path/to/spec.avsc
```

```yaml
schema_path: http://localhost:8081/path/to/spec/versions/1
```

--------------------------------

### Command Name Examples

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/processors/command.adoc

Examples of how to specify the command name, including direct names and interpolation.

```yaml
name: bash
```

```yaml
name: go
```

```yaml
name: ${! @command }
```

--------------------------------

### Start Infrastructure

Source: https://github.com/redpanda-data/connect/blob/main/internal/impl/iceberg/bench/README.md

Starts the necessary infrastructure for the benchmark, including MinIO and the Iceberg REST catalog. Use this to set up the environment.

```bash
task infra:up     # start MinIO + Iceberg REST catalog
```

--------------------------------

### Quickstart Benchmark Commands

Source: https://github.com/redpanda-data/connect/blob/main/internal/impl/aws/s3/bench/write/kafka-connect/README.md

Use these commands to quickly bring up the environment, run a benchmark, and tear it down.

```bash
task up
task bench:run COUNT=10000000
task down
```

--------------------------------

### Couchbase Connection String Example

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/caches/couchbase.adoc

Example of a Couchbase connection string. Ensure the URL format is correct for your Couchbase setup.

```yaml
url: couchbase://localhost:11210
```

--------------------------------

### Start Redpanda Services

Source: https://github.com/redpanda-data/connect/blob/main/resources/docker/redpanda_benchmarking/README.md

Starts Redpanda, Grafana, and other necessary services using Docker Compose. Ensure Docker is installed and running.

```shell
docker-compose up -d
```

--------------------------------

### AWS CloudWatch Logs Input: Log Stream Prefix Example

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/inputs/aws_cloudwatch_logs.adoc

This example demonstrates using the `log_stream_prefix` field to filter log streams, consuming only those that start with the specified prefix.

```yaml
log_stream_prefix: prod-
```

--------------------------------

### Start Kafka, MySQL, and Kafka Connect

Source: https://github.com/redpanda-data/connect/blob/main/internal/impl/mysql/bench/mysql-write/jdbc-sink/README.md

Starts the necessary services for the benchmark and waits for them to be ready. Ensure Docker is running before execution.

```bash
task up   # starts Kafka, MySQL, and Kafka Connect with JDBC plugin — waits until ready
```

--------------------------------

### Include Metadata by Content Prefix

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/inputs/http_client.adoc

Example of including metadata keys that start with 'content-'.

```yaml
include_prefixes:
  - content-
```

--------------------------------

### Include Metadata by Specific Prefix

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/inputs/http_client.adoc

A more specific example of including metadata keys that start with 'kafka_'.

```yaml
include_prefixes:
  - kafka_
```

--------------------------------

### AMQP 1.0 URLs Example

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/inputs/amqp_1.adoc

Demonstrates how to configure the list of URLs for connecting to an AMQP 1.0 server.

```yaml
urls:
  - amqp://guest:guest@127.0.0.1:5672/
```

```yaml
urls:
  - amqp://127.0.0.1:5672/,amqp://127.0.0.2:5672/
```

```yaml
urls:
  - amqp://127.0.0.1:5672/
  - amqp://127.0.0.2:5672/
```

--------------------------------

### Example: Get Next Element from Sorted Set and Update Score

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/processors/redis_script.adoc

This example demonstrates using a LUA script with the redis_script processor to retrieve the next element from a Redis sorted set and update its score with a nanosecond timestamp.

```yaml
pipeline:
  processors:
    - redis_script:
        url: TODO
        script: |
          local value = redis.call("ZRANGE", KEYS[1], '0', '0')

          if next(elements) == nil then
            return ''
          end

          redis.call("ZADD", "XX", KEYS[1], ARGV[1], value)

          return value
        keys_mapping: 'root = [ meta("key") ]'
        args_mapping: 'root = [ timestamp_unix_nano() ]'
```

--------------------------------

### Get Metadata Value in JavaScript

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/processors/javascript.adoc

This JavaScript example demonstrates how to retrieve the value of a specific metadata key from the message using `benthos.v0_msg_get_meta()`.

```javascript
let key = benthos.v0_msg_get_meta("kafka_key");
```

--------------------------------

### HTTP Verb Examples

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/outputs/http_client.adoc

Demonstrates the usage of different HTTP verbs for the client output. The verb can be set to POST, GET, or DELETE.

```yaml
verb: POST

verb: GET

verb: DELETE
```

--------------------------------

### AWS CloudWatch Logs Configuration Example

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/inputs/aws_cloudwatch_logs.adoc

This snippet shows a basic configuration for AWS CloudWatch Logs, including start time, poll interval, and data limit.

```yaml
start_time: "2024-01-01T00:00:00Z"

start_time: now
```

--------------------------------

### Get Message Content as Structured Data in JavaScript

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/processors/javascript.adoc

This JavaScript example demonstrates how to obtain the root of the message as a structured value using `benthos.v0_msg_as_structured()`. It assumes the message is valid JSON or already expanded.

```javascript
let foo = benthos.v0_msg_as_structured().foo;
```

--------------------------------

### Salesforce CDC Topics Examples

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/inputs/salesforce_cdc.adoc

Define the Pub/Sub topics to subscribe to. This can include sObject names, explicit CDC channels, the CDC firehose, or Platform Event topics. Each topic gets an independent subscription.

```yaml
topics:
  - Account
  - Contact
```

```yaml
topics:
  - /data/ChangeEvents
```

```yaml
topics:
  - Account
  - /event/Order__e
```

```yaml
topics:
  - Opportunity
  - MyCustom__c
  - /event/Sync_Requested__e
```

--------------------------------

### Start PostgreSQL and Create Tables

Source: https://github.com/redpanda-data/connect/blob/main/internal/impl/postgresql/bench/README.md

Starts a PostgreSQL 16 instance with logical replication enabled and creates necessary tables.

```bash
task postgres:up      # start PostgreSQL 16 with wal_level=logical
task psql:create      # create tables
```

--------------------------------

### Start MySQL and Create Tables

Source: https://github.com/redpanda-data/connect/blob/main/internal/impl/mysql/bench/README.md

Starts MySQL 8.0 with binlog enabled and creates the necessary tables (users, products, cart) for the benchmark. Ensure Docker is running before executing these commands.

```bash
task up            # start MySQL 8.0 with binlog enabled (waits until ready)
task mysql:create  # create tables (users, products, cart)
```

--------------------------------

### Retry Processor with Archived Batch

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/processors/retry.adoc

Example demonstrating how to use the retry processor with an archive and unarchive processor to handle batch-aware operations within retries. This setup allows for processing across an entire batch and repeating it in case of errors.

```yaml
pipeline:
  processors:
    - archive:
        format: json_array
    - retry:
        processors:
          - http:
              url: example.com/nope
              verb: POST
    - unarchive:
        format: json_array
```

--------------------------------

### Define Window Offset

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/buffers/system_window.adoc

Offsets the start of each window from the default alignment to the UTC clock's zeroth hour and minute. The offset cannot be equal to or larger than the window size or slide. Examples show an offset of -6 hours and 30 minutes.

```yaml
offset: -6h
```

```yaml
offset: 30m
```

--------------------------------

### SQL Initialization Files

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/outputs/sql_raw.adoc

Example of specifying a list of file paths containing SQL statements to be executed upon initial connection. Glob patterns are supported for file matching.

```yaml
init_files:
  - /path/to/init/schema.sql
  - /path/to/other/*.sql
```

--------------------------------

### Use GPT-4o to Call a Tool for Weather Information

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/processors/openai_chat_completion.adoc

This example asks GPT-4o to respond with the weather by invoking an HTTP processor to get the forecast. It defines a 'GetWeather' tool with city as a parameter. The API key can be set via an environment variable.

```yaml
input:
  generate:
    count: 1
    mapping: |
      root = "What is the weather like in Chicago?"
pipeline:
  processors:
    - openai_chat_completion:
        model: gpt-4o
        api_key: "${OPENAI_API_KEY}"
        prompt: "${!content().string()}"
        tools:
          - name: GetWeather
            description: "Retrieve the weather for a specific city"
            parameters:
              required: ["city"]
              properties:
                city:
                  type: string
                  description: the city to look up the weather for
            processors:
              - http:
                  verb: GET
                  url: 'https://wttr.in/${!this.city}?T'
                  headers:
                    User-Agent: curl/8.11.1 # Returns a text string from the weather website
output:
  stdout: {}
```

--------------------------------

### MQTT URL Configuration Example

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/inputs/mqtt.adoc

This example demonstrates how to configure the 'urls' field for the MQTT input connector, specifying the protocol, host, and port for connecting to an MQTT broker.

```yaml
urls:
  - tcp://localhost:1883
```

--------------------------------

### Azure Blob Storage Input with Targets Input

Source: https://github.com/redpanda-data/connect/blob/main/docs/modules/components/pages/inputs/azure_blob_storage.adoc

Example of configuring the Azure Blob Storage input to use a 'targets_input' for streaming new files. This setup uses MQTT and processors to map incoming Event Grid events to blob names.

```yaml
targets_input:
  mqtt:
    topics:
      - some-topic
    urls:
      - example.westeurope-1.ts.eventgrid.azure.net:8883
  processors:
    - unarchive:
        format: json_array
    - mapping: |- 
        if this.eventType == "Microsoft.Storage.BlobCreated" {
          root.name = this.data.url.parse_url().path.trim_prefix("/foocontainer/")
        } else {
          root = deleted()
        }
```

--------------------------------

### Go Benchmark Example with Metrics and Allocations

Source: https://github.com/redpanda-data/connect/blob/main/docs/benchmarking.md

Use standard Go testing.B benchmarks for unit-level tests. Report custom metrics like items/sec and enable allocation tracking with b.ReportAllocs().

```go
func BenchmarkConvert(b *testing.B) {
    // setup...
    b.ReportAllocs()
    for b.Loop() {
        // operation under test
    }
    b.ReportMetric(float64(itemCount)/b.Elapsed().Seconds(), "items/sec")
}
```

--------------------------------

### Install Redpanda Connect Plugin

Source: https://github.com/redpanda-data/connect/blob/main/CLAUDE.md

Provides commands for installing the Redpanda Connect plugin, supporting both local development and marketplace installation. Restart Claude Code after installation.

```bash
/plugin marketplace add /path/to/connect   # local dev
```

```bash
/plugin install redpanda-connect
```