### Connector Check Command Examples

Source: https://seatunnel.apache.org/docs/2.3.10/command/connector-check

Provides examples of how to use the connector check command to list plugins and get option rules.

```bash
# List all supported connectors(sources and sinks) and transforms  
bin/seatunnel-connector.sh -l  

```

```bash
# List all supported sinks  
bin/seatunnel-connector.sh -l -pt sink  

```

```bash
# Get option rule of the connector or transform by the name  
bin/seatunnel-connector.sh -o Paimon  

```

```bash
# Get option rule of paimon sink  
bin/seatunnel-connector.sh -o Paimon -pt sink  

```

--------------------------------

### Start Kubernetes Minikube Cluster

Source: https://seatunnel.apache.org/docs/2.3.10/start-v2/kubernetes/helm/#install

Starts a local Kubernetes cluster using Minikube with a specified Kubernetes version. Ensure Minikube is installed and configured.

```bash
minikube start --kubernetes-version=v1.23.3  

```

--------------------------------

### Simple Sls Source Configuration

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/Sls

This example demonstrates a basic configuration for the Sls source connector. It reads data from a specified logstore and project, printing it to the console. Ensure you have the necessary SeaTunnel installation and deployment steps completed before running.

```hocon
# Defining the runtime environment
env {
  parallelism = 2
  job.mode = "STREAMING"
  checkpoint.interval = 30000
}

source {
  Sls {
    endpoint = "cn-hangzhou-intranet.log.aliyuncs.com"
    project = "project1"
    logstore = "logstore1"
    access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx"
    access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    schema = {
      fields = {
            id = "int"
            name = "string"
            description = "string"
            weight = "string"
      }
    }
  }
}

sink {
  Console {
  }
}

```

--------------------------------

### Install SeaTunnel Connectors

Source: https://seatunnel.apache.org/docs/2.3.10/start-v2/locally/quick-start-seatunnel-engine

Execute the installation script to download and install the configured connectors.

```bash
# Install the connector.  
sh bin/install-plugin.sh  

```

--------------------------------

### Binary File Synchronization Example

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/CosFile

When synchronizing files in binary format (e.g., compressed packages, images), ensure both the source and sink connectors use the 'binary' format. This example illustrates the basic setup for such synchronization.

```plaintext
connector = "CosFile"
path = "Cos://your-bucket/your-prefix"
file_format_type = "binary"
# ... other CosFile options
```

--------------------------------

### Install SeaTunnel with Default Settings using Helm

Source: https://seatunnel.apache.org/docs/2.3.10/start-v2/kubernetes/helm/#install

Installs Apache SeaTunnel using Helm with default configurations. This involves pulling the Helm chart, extracting it, and performing the installation.

```bash
# Choose the corresponding version yourself  
export VERSION=2.3.10  
helm pull oci://registry-1.docker.io/apache/seatunnel-helm --version ${VERSION}  
tar -xvf seatunnel-helm-${VERSION}.tgz  
cd seatunnel-helm  
helm install seatunnel .  

```

--------------------------------

### Simple Sls Sink Configuration

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/sink/Sls

This example demonstrates how to configure the Sls sink connector to write data to an Aliyun Sls logstore. Ensure you have the necessary Aliyun credentials and have installed SeaTunnel.

```hocon
env {
  parallelism = 2
  job.mode = "STREAMING"
  checkpoint.interval = 30000
}

source {
  FakeSource {
    row.num = 10
    map.size = 10
    array.size = 10
    bytes.length = 10
    string.length = 10
    schema = {
      fields = {
            id = "int"
            name = "string"
            description = "string"
            weight = "string"
      }
    }
  }
}

sink {
  Sls {
    endpoint = "cn-hangzhou-intranet.log.aliyuncs.com"
    project = "project1"
    logstore = "logstore1"
    access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx"
    access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  }
}

```

--------------------------------

### Configure Required Example Application Plugins

Source: https://seatunnel.apache.org/docs/2.3.10/seatunnel-engine/download-seatunnel

Specify the connector plugins required for example applications to function correctly, such as connector-fake and connector-console.

```properties
--seatunnel-connectors--  
connector-fake  
connector-console  
--end--  
```

--------------------------------

### RocketMQ Source Options Example

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/RocketMQ

Example configuration for the RocketMQ source connector, specifying topics, name server address, and consumer group.

```properties
topics = "tpc1,tpc2"
name.srv.addr = "localhost:9876"
consumer.group = "SeaTunnel-Consumer-Group"
```

--------------------------------

### Install Connector Plugins

Source: https://seatunnel.apache.org/docs/2.3.10/seatunnel-engine/download-seatunnel

Install all default connector plugins using the provided shell script. This is required for versions 2.2.0-beta and later.

```bash
sh bin/install-plugin.sh  
```

--------------------------------

### IoTDB Source Configuration Example

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/IoTDB

Example configuration for the IoTDB source connector. Ensure the jdbc driver jar is in the appropriate plugins directory.

```json
{
  "node_urls": "localhost:6667",
  "username": "root",
  "password": "iotdb",
  "sql": "SELECT * FROM root.ln.wf01.wt01 LIMIT 100",
  "schema": {
    "measurement_ids": [
      "sensor_1",
      "sensor_2"
    ],
    "timestamp_precision": "ms"
  },
  "fetch_size": 1000,
  "lower_bound": 1609459200000,
  "upper_bound": 1609545600000,
  "num_partitions": 3
}
```

--------------------------------

### Initial Docker Compose Setup for SeaTunnel Cluster

Source: https://seatunnel.apache.org/docs/2.3.10/start-v2/docker

This docker-compose.yaml file defines the services for a SeaTunnel cluster with a master and two worker nodes. It configures networking and entry points for each container. Use 'docker-compose up -d' to start the cluster.

```yaml
version: '3.8'  
  
services:  
  master:  
    image: apache/seatunnel  
    container_name: seatunnel_master  
    environment:  
      - ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4      
    entrypoint: >  
      /bin/sh -c "  
      /opt/seatunnel/bin/seatunnel-cluster.sh -r master  
      "      
    ports:  
      - "5801:5801"    
    networks:  
      seatunnel_network:  
        ipv4_address: 172.16.0.2  
  
  worker1:  
    image: apache/seatunnel  
    container_name: seatunnel_worker_1  
    environment:  
      - ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4  
    entrypoint: >  
      /bin/sh -c "  
      /opt/seatunnel/bin/seatunnel-cluster.sh -r worker  
      "   
    depends_on:  
      - master  
    networks:  
      seatunnel_network:  
        ipv4_address: 172.16.0.3  
  
  worker2:  
    image: apache/seatunnel  
    container_name: seatunnel_worker_2  
    environment:  
      - ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4  
    entrypoint: >  
      /bin/sh -c "  
      /opt/seatunnel/bin/seatunnel-cluster.sh -r worker  
      "   
    depends_on:  
      - master  
    networks:  
      seatunnel_network:  
        ipv4_address: 172.16.0.4  
  
networks:  
  seatunnel_network:  
    driver: bridge  
    ipam:  
      config:  
        - subnet: 172.16.0.0/24  
  

```

--------------------------------

### Example SeaTunnel Engine Local Run Output

Source: https://seatunnel.apache.org/docs/2.3.10/contribution/setup

This is an example output from running the SeaTunnel engine local example. It displays job statistics after successful execution.

```text
2024-08-10 11:45:32,839 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand -   
***********************************************  
           Job Statistic Information  
***********************************************  
Start Time                : 2024-08-10 11:45:30  
End Time                  : 2024-08-10 11:45:32  
Total Time(s)             :                   2  
Total Read Count          :                   5  
Total Write Count         :                   5  
Total Failed Count        :                   0  
***********************************************  

```

--------------------------------

### SeaTunnel Job Statistic Information Example

Source: https://seatunnel.apache.org/docs/2.3.10/start-v2/locally/quick-start-seatunnel-engine

Example output showing the job statistics after a successful SeaTunnel batch job execution.

```text
***********************************************  
           Job Statistic Information  
***********************************************  
Start Time                : 2024-08-13 10:21:49  
End Time                  : 2024-08-13 10:21:53  
Total Time(s)             :                   4  
Total Read Count          :                1000  
Total Write Count         :                1000  
Total Failed Count        :                   0  
***********************************************  

```

--------------------------------

### Example UDF Implementation

Source: https://seatunnel.apache.org/docs/2.3.10/transform-v2/sql-udf

A sample Java class implementing the ZetaUDF interface. This example defines a UDF named 'EXAMPLE' that prefixes input strings.

```java
@AutoService(ZetaUDF.class)
public class ExampleUDF implements ZetaUDF {
    @Override
    public String functionName() {
        return "EXAMPLE";
    }

    @Override
    public SeaTunnelDataType<?> resultType(List<SeaTunnelDataType<?>> argsType) {
        return BasicType.STRING_TYPE;
    }

    @Override
    public Object evaluate(List<Object> args) {
        String arg = (String) args.get(0);
        if (arg == null) return null;
        return "UDF: " + arg;
    }
}

```

--------------------------------

### Example Cluster Members Configuration

Source: https://seatunnel.apache.org/docs/2.3.10/start-v2/kubernetes

Example of how to list cluster members within the hazelcast-client ConfigMap.

```yaml
- seatunnel-0.seatunnel.default.svc.cluster.local:5801  
- seatunnel-1.seatunnel.default.svc.cluster.local:5801  
- seatunnel-2.seatunnel.default.svc.cluster.local:5801  
```

--------------------------------

### HOCON Configuration Example

Source: https://seatunnel.apache.org/docs/2.3.10/concept/config

A comprehensive HOCON configuration example demonstrating the structure for env, source, transform, and sink modules. This example includes specific configurations for FakeSource and Clickhouse, along with a Filter transform.

```hocon
env {
  job.mode = "BATCH"
}

source {
  FakeSource {
    plugin_output = "fake"
    row.num = 100
    schema = {
      fields {
        name = "string"
        age = "int"
        card = "int"
      }
    }
  }
}

transform {
  Filter {
    plugin_input = "fake"
    plugin_output = "fake1"
    fields = [name, card]
  }
}

sink {
  Clickhouse {
    host = "clickhouse:8123"
    database = "default"
    table = "seatunnel_console"
    fields = ["name", "card"]
    username = "default"
    password = ""
    plugin_input = "fake1"
  }
}

```

--------------------------------

### Install Cert-Manager for Kubernetes

Source: https://seatunnel.apache.org/docs/2.3.10/start-v2/kubernetes

Install the cert-manager on your Kubernetes cluster to enable the webhook component. This is a one-time setup per cluster.

```bash
kubectl create -f https://github.com/jetstack/cert-manager/releases/download/v1.8.2/cert-manager.yaml  

```

--------------------------------

### Configure Specific Connector Plugin

Source: https://seatunnel.apache.org/docs/2.3.10/seatunnel-engine/download-seatunnel

Specify individual connector plugins to install by modifying the plugin_config file. This example shows how to install only the connector-console plugin.

```properties
--seatunnel-connectors--  
connector-console  
--end--  
```

--------------------------------

### Build Seatunnel Docker Image from Source (Step-by-Step)

Source: https://seatunnel.apache.org/docs/2.3.10/start-v2/docker

Build the Seatunnel Docker image step-by-step. This involves building the binary package first, then building the Docker image using the Dockerfile.

```bash
# Build binary package from source code  
sh ./mvnw clean package -DskipTests -Dskip.spotless=true  

# Build docker image  
cd seatunnel-dist  
docker build -f src/main/docker/Dockerfile --build-arg VERSION=2.3.10 -t apache/seatunnel:2.3.10 .  

# If you build from dev branch, you should add SNAPSHOT suffix to the version  
docker build -f src/main/docker/Dockerfile --build-arg VERSION=2.3.10-SNAPSHOT -t apache/seatunnel:2.3.10-SNAPSHOT .  

# Check the docker image  
docker images | grep apache/seatunnel  

```

--------------------------------

### HdfsFile Connector: Match files starting with abc and specific fourth character

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/HdfsFile

Example of using file_filter_pattern to match files starting with 'abc' where the fourth character is 'h' or 'g'.

```regex
/data/seatunnel/20241007/abc[h,g].*  

```

--------------------------------

### Prometheus Range Query Example

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/Prometheus

Example configuration for reading data from Prometheus using a range query. This includes specifying start time, end time, and step interval for the query.

```hocon
source {
  Prometheus {
    plugin_output = "http"
    url = "http://mockserver:1080"
    query = "up"
    query_type = "Range"
    content_field = "$.data.result.*"
    format = "json"
    start = "2024-07-22T20:10:30.781Z"
    end = "2024-07-22T20:11:00.781Z"
    step = "15s"
    schema = {
        fields {
            metric = "map<string, string>"
            value = double
            time = long
            }
        }
    }
  }

```

--------------------------------

### Configuring Multiple Pipelines with Source and Transform

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source-common-options

This example demonstrates configuring a 'FakeSource' to produce data with a defined schema, then transforming it using 'Sql' before writing to two different 'Console' sinks. Ensure `plugin_input` matches the `plugin_output` of the preceding stage.

```java
env {
  job.mode = "BATCH"
}

source {
  FakeSource {
    plugin_output = "fake"
    row.num = 100
    schema = {
      fields {
        id = "int"
        name = "string"
        age = "int"
        c_timestamp = "timestamp"
        c_date = "date"
        c_map = "map<string, string>"
        c_array = "array<int>"
        c_decimal = "decimal(30, 8)"
        c_row = {
          c_row = {
            c_int = int
          }
        }
      }
    }
  }
}

transform {
  Sql {
    plugin_input = "fake"
    plugin_output = "fake1"
    # the query table name must same as field 'plugin_input'
    query = "select id, regexp_replace(name, '.+', 'b') as name, age+1 as age, pi() as pi, c_timestamp, c_date, c_map, c_array, c_decimal, c_row from dual"
  }
  # The SQL transform support base function and criteria operation
  # But the complex SQL unsupported yet, include: multi source table/rows JOIN and AGGREGATE operation and the like
}

sink {
  Console {
    plugin_input = "fake1"
  }
   Console {
    plugin_input = "fake"
  }
}

```

--------------------------------

### HdfsFile Connector: Match all files starting with abc

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/HdfsFile

Example of using file_filter_pattern to match all files that begin with 'abc'.

```regex
/data/seatunnel/20241002/abc.*  

```

--------------------------------

### Example: Matching files starting with 'abc'

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/CosFile

This regular expression matches all files that begin with 'abc' in the specified path.

```regex
/data/seatunnel/20241002/abc.*
```

--------------------------------

### RocketMQ Source with Specific Start Offsets

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/RocketMQ

Demonstrates how to configure the RocketMQ source to start consuming from specific offsets for given topics and partitions.

```properties
start.mode = "CONSUME_FROM_SPECIFIC_OFFSETS"
start.mode.offsets = {
  topic1-0 = 70
  topic1-1 = 10
  topic1-2 = 10
}
```

--------------------------------

### Example: Matching CSV files in specific subfolders

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/CosFile

This regular expression matches all .csv files located in third-level directories that start with '202410'.

```regex
/data/seatunnel/202410\d*/.*\.csv
```

--------------------------------

### Parsing Partition from File Path

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/FtpFile

This example demonstrates how to enable partition parsing from the file path. When enabled, partition keys and values are added as fields to each record. Do not define partition fields in the schema.

```text
name| age
---
tyrantlucifer| 26
```

--------------------------------

### Run SeaTunnel with Spark 3.x.x

Source: https://seatunnel.apache.org/docs/2.3.10/start-v2/locally/quick-start-spark

Command to start a SeaTunnel application using Spark 3.x.x in local mode. Ensure you are in the SeaTunnel installation directory.

```bash
cd "apache-seatunnel-${version}"
./bin/start-seatunnel-spark-3-connector-v2.sh \
--master local[4] \
--deploy-mode client \
--config ./config/v2.streaming.conf.template  
```

--------------------------------

### Run SeaTunnel with Spark 2.4.x

Source: https://seatunnel.apache.org/docs/2.3.10/start-v2/locally/quick-start-spark

Command to start a SeaTunnel application using Spark 2.4.x in local mode. Ensure you are in the SeaTunnel installation directory.

```bash
cd "apache-seatunnel-${version}"
./bin/start-seatunnel-spark-2-connector-v2.sh \
--master local[4] \
--deploy-mode client \
--config ./config/v2.streaming.conf.template  
```

--------------------------------

### Run SeaTunnel with Flink (1.15.x - 1.18.x)

Source: https://seatunnel.apache.org/docs/2.3.10/start-v2/locally/quick-start-flink

Command to start a SeaTunnel application using Flink versions between 1.15.x and 1.18.x. Ensure you are in the SeaTunnel installation directory.

```bash
cd "apache-seatunnel-${version}"
./bin/start-seatunnel-flink-15-connector-v2.sh --config ./config/v2.streaming.conf.template
```

--------------------------------

### Run SeaTunnel with Flink (1.12.x - 1.14.x)

Source: https://seatunnel.apache.org/docs/2.3.10/start-v2/locally/quick-start-flink

Command to start a SeaTunnel application using Flink versions between 1.12.x and 1.14.x. Ensure you are in the SeaTunnel installation directory.

```bash
cd "apache-seatunnel-${version}"
./bin/start-seatunnel-flink-13-connector-v2.sh --config ./config/v2.streaming.conf.template
```

--------------------------------

### Pulsar Source Configuration Example

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/Pulsar

Example of how to configure the Pulsar source connector with essential parameters like topic, subscription name, service URL, and admin URL.

```hocon
source {
  Pulsar {
    topic = "example"
    subscription.name = "seatunnel"
    client.service-url = "pulsar://localhost:6650"
    admin.service-url = "http://my-broker.example.com:8080"
    plugin_output = "test"
  }
}

```

--------------------------------

### Example: Matching 'abc' files with specific fourth character

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/CosFile

This regular expression filters files starting with 'abc' where the fourth character is either 'h' or 'g'.

```regex
/data/seatunnel/20241007/abc[h,g].*
```

--------------------------------

### HdfsFile Connector: File Structure Example

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/HdfsFile

Illustrates the expected directory and file structure for data processed by the HdfsFile connector.

```text
/data/seatunnel/20241001/report.txt  
/data/seatunnel/20241007/abch202410.csv  
/data/seatunnel/20241002/abcg202410.csv  
/data/seatunnel/20241005/old_data.csv  
/data/seatunnel/20241012/logo.png  


```

--------------------------------

### Snowflake Source Configuration Example

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/Snowflake

This example demonstrates a basic configuration for the Snowflake source connector, querying all fields from a table in single parallel execution.

```toml
{
  "source": {
    "name": "snowflake",
    "kind": "snowflake",
    "url": "jdbc:snowflake://<account_name>.snowflakecomputing.com",
    "driver": "net.snowflake.client.jdbc.SnowflakeDriver",
    "user": "<user>",
    "password": "<password>",
    "query": "select * from "test.database.table" limit 16"
  },
  "sink": [
    {
      "name": "console",
      "kind": "console"
    }
  ]
}
```

--------------------------------

### Simple Paimon Source Example

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/Paimon

Basic configuration for using the Paimon connector as a data source. Specify warehouse, database, and table.

```java
source {
  Paimon {
    warehouse = "/tmp/paimon"
    database = "default"
    table = "st_test"
  }
}

```

--------------------------------

### Http Sink with Dynamic URL (Mysql-CDC Source)

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/sink/Http

Example of an Http Sink configured to use dynamic placeholders in the URL, sourced from a Mysql-CDC stream. This setup is suitable for streaming jobs.

```hocon
env {
  parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 5000
}

source {
  Mysql-CDC {
    base-url = "jdbc:mysql://127.0.0.1:3306/seatunnel"
    username = "root"
    password = "******"
      
    table-names = ["seatunnel.role","seatunnel.user","galileo.Bucket"]
  }
}

transform {
}

sink {
  Http {
    ...
    url = "http://localhost/test/${database_name}_test/${table_name}_test"
  }
}

```

--------------------------------

### HdfsFile Connector: Match files in specific dated folders ending with .csv

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/HdfsFile

Example of using file_filter_pattern to match .csv files within third-level folders that start with '202410' followed by digits.

```regex
/data/seatunnel/202410\d*/.*\.csv  

```

--------------------------------

### JSON Configuration Example

Source: https://seatunnel.apache.org/docs/2.3.10/concept/config

An example of a JSON configuration file for SeaTunnel, specifying job mode, source, transform, and sink plugins. Ensure the file name ends with .json.

```json
{
  "env": {
    "job.mode": "batch"
  },
  "source": [
    {
      "plugin_name": "FakeSource",
      "plugin_output": "fake",
      "row.num": 100,
      "schema": {
        "fields": {
          "name": "string",
          "age": "int",
          "card": "int"
        }
      }
    }
  ],
  "transform": [
    {
      "plugin_name": "Filter",
      "plugin_input": "fake",
      "plugin_output": "fake1",
      "fields": ["name", "card"]
    }
  ],
  "sink": [
    {
      "plugin_name": "Clickhouse",
      "host": "clickhouse:8123",
      "database": "default",
      "table": "seatunnel_console",
      "fields": ["name", "card"],
      "username": "default",
      "password": "",
      "plugin_input": "fake1"
    }
  ]
}

```

--------------------------------

### Job Configuration with Replace Transform

Source: https://seatunnel.apache.org/docs/2.3.10/transform-v2/replace

A complete job configuration example using the Replace transform. This setup reads from a FakeSource, replaces all characters in the 'name' field with 'b', and outputs to the Console sink.

```hcl
env {
  job.mode = "BATCH"
}

source {
  FakeSource {
    plugin_output = "fake"
    row.num = 100
    schema = {
      fields {
        id = "int"
        name = "string"
      }
    }
  }
}

transform {
  Replace {
    plugin_input = "fake"
    plugin_output = "fake1"
    replace_field = "name"
    pattern = ".+"
    replacement = "b"
    is_regex = true
  }
}

sink {
  Console {
    plugin_input = "fake1"
  }
}
```

--------------------------------

### Example: Run Spark 3 Batch Job

Source: https://seatunnel.apache.org/docs/2.3.10/command/usage

Example of running a Spark 3 SeaTunnel batch job using a configuration file and specifying local deploy mode.

```bash
bin/start-seatunnel-spark-3-connector-v2.sh --config config/v2.batch.config.template -m local -e client  
```

--------------------------------

### Example: Run Spark 2 Batch Job

Source: https://seatunnel.apache.org/docs/2.3.10/command/usage

Example of running a Spark 2 SeaTunnel batch job using a configuration file and specifying local deploy mode.

```bash
bin/start-seatunnel-spark-2-connector-v2.sh --config config/v2.batch.config.template -m local -e client  
```

--------------------------------

### JSON File Format with Schema

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/FtpFile

When using JSON format, a schema must be provided to guide the connector on how to parse the data into rows. This example shows a JSON object and the corresponding schema definition.

```json
{"code": 200, "data": "get success", "success": true}
```

```json
schema {
    fields {
        code = int
        data = string
        success = boolean
    }
}
```

--------------------------------

### Read Data from Kafka with Avro Format and Print to Console

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/formats/avro

Example of reading data from Kafka with Avro format and printing it to the console. This configuration uses 'earliest' start mode and skips format errors.

```hocon
env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  Kafka {
    bootstrap.servers = "kafkaCluster:9092"
    topic = "test_avro_topic"
    plugin_output = "kafka_table"
    start_mode = "earliest"
    format = avro
    format_error_handle_way = skip
    schema = {
      fields {
        id = bigint
        c_map = "map<string, smallint>"
        c_array = "array<tinyint>"
        c_string = string
        c_boolean = boolean
        c_tinyint = tinyint
        c_smallint = smallint
        c_int = int
        c_bigint = bigint
        c_float = float
        c_double = double
        c_decimal = "decimal(2, 1)"
        c_bytes = bytes
        c_date = date
        c_timestamp = timestamp
      }
    }
  }
}

sink {
  Console {
    plugin_input = "kafka_table"
  }
}

```

--------------------------------

### Example: Run Flink 15 Batch Job

Source: https://seatunnel.apache.org/docs/2.3.10/command/usage

Example of running a Flink 15 SeaTunnel batch job using a configuration file.

```bash
bin/start-seatunnel-flink-15-connector-v2.sh --config config/v2.batch.config.template  
```

--------------------------------

### Enable Cache with Local HDFS Storage

Source: https://seatunnel.apache.org/docs/2.3.10/seatunnel-engine/checkpoint-storage

Configure Seatunnel to use HDFS for checkpoint storage with caching enabled. This example uses `file:///` for the default file system, suitable for local HDFS setups.

```yaml
seatunnel:
  engine:
    checkpoint:
      interval: 6000
      timeout: 7000
      storage:
        type: hdfs
        max-retained: 3
        plugin-config:
          storage.type: hdfs
          disable.cache: false
          fs.defaultFS: file:///

```

--------------------------------

### Kudu Multiple Table Source and Assert Sink Example

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/Kudu

This example shows how to configure the Kudu source to read from multiple tables ('kudu_source_table_1', 'kudu_source_table_2') in streaming mode and use an Assert sink to verify the table names.

```yaml
env {
  # You can set engine configuration here
  parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 5000
}

source {
  # This is a example source plugin **only for test and demonstrate the feature source plugin**
  kudu{
   kudu_masters = "kudu-master:7051"
   table_list = [
   {
    table_name = "kudu_source_table_1"
   },{
    table_name = "kudu_source_table_2"
   }
   ]
   plugin_output = "kudu"
}
}

transform {
}

sink {
  Assert {
    rules {
      table-names = ["kudu_source_table_1", "kudu_source_table_2"]
    }
  }
}
```

--------------------------------

### InfluxDB Sink with Multi-Table Support

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/sink/InfluxDB

Example of configuring the InfluxDB sink to support multiple tables, dynamically setting the measurement name based on the upstream table. This configuration is part of a larger streaming job setup.

```hocon
env {
  parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 5000
}

source {
  Mysql-CDC {
    base-url = "jdbc:mysql://127.0.0.1:3306/seatunnel"
    username = "root"
    password = "******"
      
    table-names = ["seatunnel.role","seatunnel.user","galileo.Bucket"]
  }
}

transform {
}

sink {
  InfluxDB {
    url = "http://influxdb-host:8086"
    database = "test"
    measurement = "${table_name}_test"
  }
}

```

--------------------------------

### Prometheus Sink Configuration Example

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/sink/Prometheus

This example demonstrates a basic configuration for the Prometheus sink connector in Apache SeaTunnel. It includes setting up a FakeSource to generate sample data and configuring the Prometheus sink with essential parameters like URL, key label, key value, key timestamp, and batch size.

```hocon
env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  FakeSource {
    schema = {
      fields {
        c_map = "map<string, string>"
        c_double = double
        c_timestamp = timestamp
      }
    }
    plugin_output = "fake"
    rows = [
       {
         kind = INSERT
         fields = [{"__name__": "test1"},  1.23, "2024-08-15T17:00:00"]
       },
       {
         kind = INSERT
         fields = [{"__name__": "test2"},  1.23, "2024-08-15T17:00:00"]
       }
    ]
  }
}


sink {
  Prometheus {
    url = "http://prometheus:9090/api/v1/write"
    key_label = "c_map"
    key_value = "c_double"
    key_timestamp = "c_timestamp"
    batch_size = 1
  }
}

```

--------------------------------

### InfluxDB Partitioning Example

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/InfluxDB

Illustrates how the split_column range is divided into partitions based on lower_bound, upper_bound, and partition_num. This helps in parallel data processing.

```text
split the $split_column range into $partition_num parts
     if partition_num is 1, use the whole `split_column` range
     if partition_num < (upper_bound - lower_bound), use (upper_bound - lower_bound) partitions
       
     eg: lower_bound = 1, upper_bound = 10, partition_num = 2
     sql = "select * from test where age > 0 and age < 10"
       
     split result
  
     split 1: select * from test where ($split_column >= 1 and $split_column < 6)  and (  age > 0 and age < 10 )
       
     split 2: select * from test where ($split_column >= 6 and $split_column < 11) and (  age > 0 and age < 10 )
```

--------------------------------

### Scaling SeaTunnel Cluster with Docker Compose

Source: https://seatunnel.apache.org/docs/2.3.10/start-v2/docker

This docker-compose.yaml extends the initial setup by adding a 'worker3' service to scale the cluster. Ensure the IP address in ST_DOCKER_MEMBER_LIST and ipv4_address are unique and unused. Run 'docker-compose up -d' to start the new node without restarting the existing cluster.

```yaml
version: '3.8'  
  
services:  
  master:  
    image: apache/seatunnel  
    container_name: seatunnel_master  
    environment:  
      - ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4      
    entrypoint: >  
      /bin/sh -c "  
      /opt/seatunnel/bin/seatunnel-cluster.sh -r master  
      "      
    ports:  
      - "5801:5801"    
    networks:  
      seatunnel_network:  
        ipv4_address: 172.16.0.2  
  
  worker1:  
    image: apache/seatunnel  
    container_name: seatunnel_worker_1  
    environment:  
      - ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4  
    entrypoint: >  
      /bin/sh -c "  
      /opt/seatunnel/bin/seatunnel-cluster.sh -r worker  
      "   
    depends_on:  
      - master  
    networks:  
      seatunnel_network:  
        ipv4_address: 172.16.0.3  
  
  worker2:  
    image: apache/seatunnel  
    container_name: seatunnel_worker_2  
    environment:  
      - ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4  
    entrypoint: >  
      /bin/sh -c "  
      /opt/seatunnel/bin/seatunnel-cluster.sh -r worker  
      "   
    depends_on:  
      - master  
    networks:  
      seatunnel_network:  
        ipv4_address: 172.16.0.4  
  
  worker3:  
    image: apache/seatunnel  
    container_name: seatunnel_worker_3  
    environment:  
      - ST_DOCKER_MEMBER_LIST=172.16.0.2,172.16.0.3,172.16.0.4,172.16.0.5 # add ip to here  
    entrypoint: >  
      /bin/sh -c "  
      /opt/seatunnel/bin/seatunnel-cluster.sh -r worker  
      "   
    depends_on:  
      - master  
    networks:  
      seatunnel_network:  
        ipv4_address: 172.16.0.5        # use a not used ip  
  
networks:  
  seatunnel_network:  
    driver: bridge  
    ipam:  
      config:  
        - subnet: 172.16.0.0/24  
  

```

--------------------------------

### StarRocks request_tablet_size Example Explanation

Source: https://seatunnel.apache.org/docs/2.3.10/connector-v2/source/StarRocks

Illustrates how the 'request_tablet_size' parameter controls partition generation by limiting the number of StarRocks Tablets per partition. Smaller values increase parallelism but also load on StarRocks.

```text
the tablet distribution of StarRocks table in cluster as follower

be_node_1 tablet[1, 2, 3, 4, 5]
be_node_2 tablet[6, 7, 8, 9, 10]
be_node_3 tablet[11, 12, 13, 14, 15]

1.If not set request_tablet_size, there will no limit on the number of tablets in a single partition. The partitions will be generated as follows    

partition[0] read data of tablet[1, 2, 3, 4, 5] from be_node_1   
partition[1] read data of tablet[6, 7, 8, 9, 10] from be_node_2   
partition[2] read data of tablet[11, 12, 13, 14, 15] from be_node_3   

2.if set request_tablet_size=3, the limit on the number of tablets in a single partition is 3. The partitions will be generated as follows  

partition[0] read data of tablet[1, 2, 3] from be_node_1   
partition[1] read data of tablet[4, 5] from be_node_1   
partition[2] read data of tablet[6, 7, 8] from be_node_2   
partition[3] read data of tablet[9, 10] from be_node_2   
partition[4] read data of tablet[11, 12, 13] from be_node_3   
partition[5] read data of tablet[14, 15] from be_node_3   
```