### Example URL Configuration Source: https://docs.redpanda.com/connect/components/inputs/zmq4.md Example of defining a connection URL for the zmq4 input. ```yaml # Examples urls: - tcp://localhost:5555 ``` -------------------------------- ### Model Configuration Examples Source: https://docs.redpanda.com/connect/components/processors/gcp_vertex_ai_embeddings.md Examples of specifying different Vertex AI models. ```yaml # Examples: model: text-embedding-004 # --- model: text-multilingual-embedding-002 ``` -------------------------------- ### Verify Installation Source: https://docs.redpanda.com/connect/install/prebuilt-binary.md Check that the binary is correctly installed by querying its version. ```bash redpanda-connect --version ``` -------------------------------- ### Configure token_encoding Examples Source: https://docs.redpanda.com/connect/components/processors/text_chunker.md Examples of valid token_encoding values. ```yaml # Examples: token_encoding: cl100k_base # --- token_encoding: r50k_base ``` -------------------------------- ### Model Configuration Example Source: https://docs.redpanda.com/connect/components/processors/cohere_rerank.md Example of specifying the Cohere model. ```yaml # Examples: model: rerank-v3.5 ``` -------------------------------- ### Install and verify the binary Source: https://docs.redpanda.com/connect/install/build-from-source.md Move the compiled binary to a system path and check the installed version. ```bash sudo mv redpanda-connect /usr/local/bin/ ``` ```bash redpanda-connect --version ``` -------------------------------- ### Define Redpanda Connect Metrics Configuration Source: https://docs.redpanda.com/connect/components/metrics/about.md Example configuration showing input, pipeline, output, and Prometheus metrics setup. ```yaml input: label: foo http_server: {} pipeline: processors: - mapping: | root.message = this root.meta.link_count = this.links.length() root.user.age = this.user.age.number() output: label: bar stdout: {} metrics: prometheus: {} ``` -------------------------------- ### Image Size Examples Source: https://docs.redpanda.com/connect/components/processors/openai_image_generation.md Examples of specifying the dimensions of the generated image. ```yaml # Examples: size: 1024x1024 # --- size: 512x512 # --- size: 1792x1024 # --- size: 1024x1792 ``` -------------------------------- ### Image Style Examples Source: https://docs.redpanda.com/connect/components/processors/openai_image_generation.md Examples of setting the image style, supported only for dall-e-3. ```yaml # Examples: style: vivid # --- style: natural ``` -------------------------------- ### Configure Window Size Source: https://docs.redpanda.com/connect/components/buffers/system_window.md Examples for setting the duration of each window. ```yml size: 30s size: 10m ``` -------------------------------- ### SFTP Codec Configuration Examples Source: https://docs.redpanda.com/connect/components/outputs/sftp.md Examples showing different codec configurations for writing message data to the output stream. ```yaml codec: lines ``` ```yaml codec: delim: ``` ```yaml codec: delim:foobar ``` -------------------------------- ### Configure Tables List Source: https://docs.redpanda.com/connect/components/inputs/postgres_cdc.md Example configuration for the tables parameter. ```yaml # Examples: tables: - my_table_1 - "MyCaseSensitiveTableNeedingQuotes" ``` -------------------------------- ### Model Selection Examples Source: https://docs.redpanda.com/connect/components/processors/openai_image_generation.md Examples of specifying the OpenAI model to use. ```yaml # Examples: model: dall-e-3 # --- model: dall-e-2 ``` -------------------------------- ### Table Name Example Source: https://docs.redpanda.com/connect/components/inputs/azure_table_storage.md Example of specifying the target table name. ```yaml # Examples: table_name: Foo ``` -------------------------------- ### Parquet file path configuration examples Source: https://docs.redpanda.com/connect/components/inputs/parquet.md Examples of specifying file paths, including single files and glob patterns. ```yaml # Examples: paths: /tmp/foo.parquet # --- paths: /tmp/bar/*.parquet # --- paths: /tmp/data/**/*.parquet ``` -------------------------------- ### Address Field Examples Source: https://docs.redpanda.com/connect/components/inputs/socket.md Examples of specifying Unix socket paths and TCP addresses. ```yaml address: /tmp/benthos.sock ``` ```yaml address: 127.0.0.1:6000 ``` -------------------------------- ### Install Redpanda Connect via Helm Source: https://docs.redpanda.com/connect/get-started/quickstarts/helm-chart.md Commands to add the Redpanda repository and install the chart into a specific namespace. ```bash helm repo add redpanda https://charts.redpanda.com (1) helm repo update (2) helm install redpanda-connect redpanda/connect --namespace --create-namespace (3) ``` -------------------------------- ### Filesystem Interpolation Example Source: https://docs.redpanda.com/connect/components/outputs/azure_data_lake_gen2.md Example of using interpolation functions to dynamically name the target filesystem. ```yaml # Examples: filesystem: messages-${!timestamp("2006")} ``` -------------------------------- ### Avro schema path configuration examples Source: https://docs.redpanda.com/connect/components/processors/avro.md Examples of specifying schema locations using local file paths or HTTP URLs. ```yaml # Examples: schema_path: file:///path/to/spec.avsc # --- schema_path: http://localhost:8081/path/to/spec/versions/1 ``` -------------------------------- ### Retry Initial Interval Examples Source: https://docs.redpanda.com/connect/components/caches/memcached.md Examples of setting the initial retry interval. ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` -------------------------------- ### Echo Slackbot Example Source: https://docs.redpanda.com/connect/components/inputs/slack.md A complete pipeline example that echoes messages from other users back to the same channel. ```yaml input: slack: app_token: "${APP_TOKEN:xapp-demo}" bot_token: "${BOT_TOKEN:xoxb-demo}" pipeline: processors: - mutation: | # ignore hidden or non message events if this.event.type != "message" || (this.event.hidden | false) { root = deleted() } # Don't respond to our own messages if this.authorizations.any(auth -> auth.user_id == this.event.user) { root = deleted() } output: slack_post: bot_token: "${BOT_TOKEN:xoxb-demo}" channel_id: "${!this.event.channel}" thread_ts: "${!this.event.ts}" text: "ECHO: ${!this.event.text}" ``` -------------------------------- ### Queue Name Configuration Examples Source: https://docs.redpanda.com/connect/components/inputs/azure_queue_storage.md Examples showing static queue naming and dynamic naming using Bloblang interpolation. ```yaml # Examples: queue_name: foo_queue # --- queue_name: ${! env("MESSAGE_TYPE").lowercase() } ``` -------------------------------- ### Mapping Example Input Source: https://docs.redpanda.com/connect/components/processors/jq.md Example JSON input document for the mapping processor demonstration. ```json { "locations": [ {"name": "Seattle", "state": "WA"}, {"name": "New York", "state": "NY"}, {"name": "Bellevue", "state": "WA"}, {"name": "Olympia", "state": "WA"} ] } ``` -------------------------------- ### Image Quality Examples Source: https://docs.redpanda.com/connect/components/processors/openai_image_generation.md Examples of setting the image quality, supported only for dall-e-3 models. ```yaml # Examples: quality: standard # --- quality: hd ``` -------------------------------- ### Model Configuration Examples Source: https://docs.redpanda.com/connect/components/processors/cohere_embeddings.md Examples of different Cohere model identifiers that can be used in the configuration. ```yaml # Examples: model: embed-english-v3.0 # --- model: embed-english-light-v3.0 # --- model: embed-multilingual-v3.0 # --- model: embed-multilingual-light-v3.0 ``` -------------------------------- ### Install latest rpk binary on Linux Source: https://docs.redpanda.com/connect/get-started/quickstarts/rpk.md Downloads and installs the latest rpk binary to ~/.local/bin for amd64 or arm64 architectures. ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ ``` -------------------------------- ### Enable Stream Snapshot Source: https://docs.redpanda.com/connect/components/inputs/postgres_cdc.md Example configuration for the stream_snapshot parameter. ```yaml # Examples: stream_snapshot: true ``` -------------------------------- ### Configure URLs Source: https://docs.redpanda.com/connect/components/outputs/zmq4.md Example configuration for the urls field. ```yaml # Examples urls: - tcp://localhost:5556 ``` -------------------------------- ### Configure columns Source: https://docs.redpanda.com/connect/components/inputs/sql_select.md Examples of specifying columns to select from the database table. ```yaml # Examples: columns: - "*" ``` ```yaml columns: - foo - bar - baz ``` -------------------------------- ### Configure Initialization Files Source: https://docs.redpanda.com/connect/components/caches/sql.md Examples of specifying SQL initialization files using glob patterns or explicit file lists. ```yaml init_files: - ./init/*.sql ``` ```yaml init_files: - ./foo.sql - ./bar.sql ``` -------------------------------- ### Initialize Plugin Environment Source: https://docs.redpanda.com/connect/plugins/about.md Create a project directory and initialize the environment using uv. ```shell mkdir fizzbuzz_plugin cd fizzbuzz_plugin # Initialize with link:https://docs.astral.sh/uv/[uv] (or use pip/conda) uv init . uv add redpanda_connect ``` -------------------------------- ### Usage Examples for Open Telemetry Collector Source: https://docs.redpanda.com/connect/components/metrics/open_telemetry_collector.md Common deployment patterns including local sidecar setups, production TLS configurations, and adding custom metric tags. ```yaml metrics: open_telemetry_collector: service: my-service-name grpc: - address: localhost:4317 ``` ```yaml metrics: open_telemetry_collector: service: my-service-name grpc: - address: otel-collector.example.com:4317 secure: true ``` ```yaml metrics: open_telemetry_collector: service: my-service-name grpc: - address: localhost:4317 tags: environment: production cluster: kafka-01 ``` -------------------------------- ### Example Configuration File Source: https://docs.redpanda.com/connect/configuration/unit_testing.md A sample configuration file containing input, pipeline processors, and output definitions. ```yaml input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup pipeline: processors: - mapping: '"%vend".format(content().uppercase().string())' output: aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' ``` -------------------------------- ### Model Field Example Source: https://docs.redpanda.com/connect/components/processors/openai_transcription.md Example of specifying the OpenAI model. ```yaml # Examples: model: whisper-1 ``` -------------------------------- ### Create Helper File Source: https://docs.redpanda.com/connect/configuration/using_cue.md Command to initialize the helper configuration file. ```shell touch benthos/helpers.cue ``` -------------------------------- ### table example Source: https://docs.redpanda.com/connect/components/processors/gcp_bigquery_select.md Example of a fully-qualified BigQuery table name. ```yaml table: bigquery-public-data.samples.shakespeare ``` -------------------------------- ### Create Topic and Produce Sample Data Source: https://docs.redpanda.com/connect/cookbooks/rag.md Initialize the articles topic and inject a sample JSON document for processing. ```bash rpk topic create articles echo '{ "type": "article", "article": { "id": "123foo", "title": "Dogs Stop Barking", "content": "The world was shocked this morning to find that all dogs have stopped barking." } }' | rpk topic produce articles -f '%v' ``` -------------------------------- ### args_mapping example Source: https://docs.redpanda.com/connect/components/processors/gcp_bigquery_select.md Example of a Bloblang mapping for query arguments. ```yaml args_mapping: root = [ "article", now().ts_format("2006-01-02") ] ``` -------------------------------- ### Get Hostname Source: https://docs.redpanda.com/connect/guides/bloblang/functions.md Returns the machine hostname for instance identification. ```bloblang root.processed_by = hostname() ``` -------------------------------- ### Generate configuration examples Source: https://docs.redpanda.com/connect/configuration/about.md Use the create subcommand to generate boilerplate configuration files for specific component pipelines. ```bash rpk connect create websocket/mapping/kafka ``` -------------------------------- ### CSV File Example Source: https://docs.redpanda.com/connect/components/inputs/csv.md Example of a CSV file structure. ```csv foo,bar,baz first foo,first bar,first baz second foo,second bar,second baz ``` -------------------------------- ### Example configuration structure Source: https://docs.redpanda.com/connect/configuration/about.md A sample YAML configuration showing the structure for an AMQP input and stdout output. ```yaml input: amqp_0_9: urls: [ amqp://guest:guest@localhost:5672/ ] consumer_tag: benthos-consumer queue: benthos-queue prefetch_count: 10 prefetch_size: 0 output: stdout: {} ``` -------------------------------- ### JSON Message Example Source: https://docs.redpanda.com/connect/components/outputs/azure_table_storage.md Example of a JSON message structure to be stored. ```json { "foo": 55, "bar": { "baz": "a", "bez": "b" }, "diz": ["a", "b"] } ``` -------------------------------- ### range Source: https://docs.redpanda.com/connect/guides/bloblang/functions.md Creates an array of integers from start (inclusive) to stop (exclusive) with an optional step. ```APIDOC ## range ### Description Creates an array of integers from start (inclusive) to stop (exclusive) with an optional step. Use this to generate sequences for iteration, indexing, or creating numbered lists. ### Parameters - **start** (integer) - The start value. - **stop** (integer) - The stop value. - **step** (integer) - The step value. ### Example root.a = range(0, 10) root.b = range(start: 0, stop: this.max, step: 2) ``` -------------------------------- ### List available components Source: https://docs.redpanda.com/connect/configuration/about.md Use the list subcommand to view all available components for configuration. ```bash rpk connect list ``` -------------------------------- ### Language Field Examples Source: https://docs.redpanda.com/connect/components/processors/openai_transcription.md Examples of ISO-639-1 language codes for the language field. ```yaml # Examples: language: en # --- language: fr # --- language: de # --- language: zh ``` -------------------------------- ### Download Example Processor Source: https://docs.redpanda.com/connect/plugins/about.md Fetch the sample Python processor script from the official repository. ```shell curl -o fizzbuzz_processor.py \ https://raw.githubusercontent.com/redpanda-data/connect/main/public/plugin/python/examples/fizzbuzz_processor.py ``` -------------------------------- ### Configure input_headers include_prefixes Source: https://docs.redpanda.com/connect/components/processors/redpanda_data_transform.md Examples for defining metadata key prefixes to match. ```yaml # Examples: include_prefixes: - foo_ - bar_ ``` ```yaml include_prefixes: - kafka_ ``` ```yaml include_prefixes: - content- ``` -------------------------------- ### Get batch size and index Source: https://docs.redpanda.com/connect/guides/bloblang/functions.md Use batch_size to determine the total number of messages in a batch and batch_index to identify the current message position. ```bloblang root.total = batch_size() ``` ```bloblang root.is_last = batch_index() == batch_size() - 1 ``` -------------------------------- ### TTL Duration Examples Source: https://docs.redpanda.com/connect/components/outputs/cache.md Examples of duration strings for setting item TTLs. ```yaml # Examples: ttl: 60s # --- ttl: 5m # --- ttl: 36h ``` -------------------------------- ### Get current timestamp with now() Source: https://docs.redpanda.com/connect/guides/bloblang/functions.md Returns the current timestamp as an RFC 3339 string. Use ts_format to customize the output. ```bloblang root.received_at = now() ``` ```bloblang root.received_at = now().ts_format("Mon Jan 2 15:04:05 -0700 MST 2006", "UTC") ``` -------------------------------- ### Generate Snowflake IDs in Bloblang Source: https://docs.redpanda.com/connect/guides/bloblang/functions.md Examples of generating unique, time-ordered Snowflake IDs with and without a custom node ID. ```bloblang root.id = snowflake_id() root.payload = this ``` ```bloblang root.id = snowflake_id(42) root.data = this ``` -------------------------------- ### Initialize CUE Module Source: https://docs.redpanda.com/connect/configuration/using_cue.md Commands to create a new directory and initialize a CUE module. ```shell mkdir hello-cue cd hello-cue cue mod init example.com/hello-cue touch config.cue ``` -------------------------------- ### Get current Unix timestamp in microseconds Source: https://docs.redpanda.com/connect/guides/bloblang/functions.md Returns the current Unix timestamp in microseconds. Use for high-precision timing measurements. ```bloblang root.received_at = timestamp_unix_micro() ``` ```bloblang root.processing_duration_us = timestamp_unix_micro() - this.start_time_us ``` -------------------------------- ### Subprocess Command Examples Source: https://docs.redpanda.com/connect/components/inputs/subprocess.md Examples of valid command names for the subprocess input. ```yaml # Examples: name: cat # --- name: sed # --- name: awk ``` -------------------------------- ### count(name) Source: https://docs.redpanda.com/connect/guides/bloblang/functions.md A counter starting at 1 which increments after each time it is called. ```APIDOC ## count(name) ### Description Deprecated: The count function is a counter starting at 1 which increments after each time it is called. Count takes an argument which is an identifier for the counter, allowing you to specify multiple unique counters in your configuration. ### Parameters - **name** (string) - Required - An identifier for the counter. ### Examples ```bloblang root.id = count("bloblang_function_example") ``` ``` -------------------------------- ### Couchbase Connection URL Example Source: https://docs.redpanda.com/connect/components/caches/couchbase.md Example of a Couchbase connection string format. ```yaml # Examples: url: couchbase://localhost:11210 ``` -------------------------------- ### Configure Cloud Platform Source: https://docs.redpanda.com/connect/components/outputs/snowflake_put.md Examples of setting the cloud platform when using an Account Locator. ```yaml cloud: aws ``` ```yaml cloud: gcp ``` ```yaml cloud: azure ``` -------------------------------- ### Pipeline Configuration Example Source: https://docs.redpanda.com/connect/components/processors/try.md A basic pipeline configuration demonstrating how the try processor skips subsequent steps if a child processor fails. ```yaml pipeline: processors: - resource: foo - try: - resource: bar - resource: baz - resource: buz ``` -------------------------------- ### Get current Unix timestamp in nanoseconds Source: https://docs.redpanda.com/connect/guides/bloblang/functions.md Returns the current Unix timestamp in nanoseconds. Provides the highest precision for unique identifiers or seeds. ```bloblang root.received_at = timestamp_unix_nano() ``` ```bloblang root.random_value = random_int(timestamp_unix_nano()) ``` -------------------------------- ### Run pipeline with sample data Source: https://docs.redpanda.com/connect/get-started/quickstarts/rpk.md Pipe sample JSON documents into the running pipeline. ```bash echo '{"id":"1","names":["celine","dion"]} {"id":"2","names":["chad","robert","kroeger"]}' | rpk connect run connect.yaml ``` ```bash echo '{"id":"1","names":["celine","dion"]} {"id":"2","names":["chad","robert","kroeger"]}' | docker run --rm -i -v $(pwd)/connect.yaml:/connect.yaml docker.redpanda.com/redpandadata/connect run ``` -------------------------------- ### Check Field Examples Source: https://docs.redpanda.com/connect/components/processors/while.md Examples of Bloblang queries used to determine loop continuation. ```yaml # Examples: check: errored() # --- check: this.urls.unprocessed.length() > 0 ``` -------------------------------- ### Get current Unix timestamp in seconds Source: https://docs.redpanda.com/connect/guides/bloblang/functions.md Returns the current Unix timestamp in seconds. Useful for numeric timestamps or random seeds. ```bloblang root.received_at = timestamp_unix() ``` ```bloblang root.id = "%v-%v".format(timestamp_unix(), batch_index()) ``` -------------------------------- ### Get current Unix timestamp in milliseconds Source: https://docs.redpanda.com/connect/guides/bloblang/functions.md Returns the current Unix timestamp in milliseconds. Suitable for web APIs and JavaScript-compatible systems. ```bloblang root.received_at = timestamp_unix_milli() ``` ```bloblang meta processing_time_ms = timestamp_unix_milli() ``` -------------------------------- ### Apply license via rpk CLI Source: https://docs.redpanda.com/connect/get-started/licensing.md Run the Redpanda Connect service with a license string provided as a command-line flag. ```bash rpk connect run --redpanda-license ./.yaml ``` -------------------------------- ### Minimal Jira Processor Configuration Source: https://docs.redpanda.com/connect/components/processors/jira.md Basic setup requiring only the base URL, username, and API token. ```yaml pipeline: processors: - jira: base_url: "https://your-domain.atlassian.net" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" ``` -------------------------------- ### Query Field Examples Source: https://docs.redpanda.com/connect/components/inputs/timeplus.md Examples of SQL queries used within the query field. ```yaml # Examples: query: select * from iot # --- query: select count(*) from table(iot) ``` -------------------------------- ### Generate a configuration file Source: https://docs.redpanda.com/connect/get-started/quickstarts/rpk.md Create a default Redpanda Connect configuration file. ```bash rpk connect create > connect.yaml ``` ```bash docker run --rm docker.redpanda.com/redpandadata/connect create > ./connect.yaml ``` -------------------------------- ### Configure codec options Source: https://docs.redpanda.com/connect/components/outputs/file.md Examples of different codec configurations for file output. ```yaml # Examples: codec: lines # --- codec: delim: # --- codec: delim:foobar ``` -------------------------------- ### Bloblang Check Examples Source: https://docs.redpanda.com/connect/components/inputs/read_until.md Examples of Bloblang queries used to determine when the input should close. ```yaml # Examples: check: this.type == "foo" # --- check: count("messages") >= 100 ``` -------------------------------- ### Configure stdin input Source: https://docs.redpanda.com/connect/components/inputs/stdin.md Basic and advanced configuration examples for the stdin input component. ```yaml inputs: label: "" stdin: scanner: lines: {} auto_replay_nacks: true ``` ```yaml inputs: label: "" stdin: scanner: lines: {} auto_replay_nacks: true ``` -------------------------------- ### SQS Envelope Path Example Source: https://docs.redpanda.com/connect/components/inputs/aws_s3.md Example of setting the envelope path for SQS messages. ```yaml # Examples: envelope_path: Message ``` -------------------------------- ### Template configuration usage Source: https://docs.redpanda.com/connect/configuration/templating.md Example configuration file utilizing the defined template. ```yaml input: aws_sqs_list: urls: - https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue1 - https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue2 pipeline: processors: - mapping: | root.id = uuid_v4() root.foo = this.inner.foo root.body = this.outer ``` -------------------------------- ### SQS Delay Period Examples Source: https://docs.redpanda.com/connect/components/inputs/aws_s3.md Examples of configuring the delay period for SQS-triggered downloads. ```yaml # Examples: delay_period: 10s # --- delay_period: 5m ``` -------------------------------- ### Select Expression Example Source: https://docs.redpanda.com/connect/components/inputs/azure_table_storage.md Example of an OData select expression to limit returned columns. ```yaml # Examples: select: PartitionKey,RowKey,Foo,Bar,Timestamp ``` -------------------------------- ### Filter Expression Example Source: https://docs.redpanda.com/connect/components/inputs/azure_table_storage.md Example of an OData filter expression for querying specific rows. ```yaml # Examples: filter: PartitionKey eq 'foo' and RowKey gt '1000' ``` -------------------------------- ### Basic Configuration Source: https://docs.redpanda.com/connect/components/processors/openai_image_generation.md Minimal configuration required to initialize the openai_image_generation processor. ```yaml processors: label: "" openai_image_generation: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) ``` -------------------------------- ### Start Docker Containers Source: https://docs.redpanda.com/connect/cookbooks/redpanda_migrator.md Command to launch the defined services using Docker Compose. ```bash docker compose -f docker-compose.yml up --force-recreate -V ```