### Install Missing Dependencies Source: https://github.com/dtstack/chunjun/wiki/Home Run these scripts to manually install driver packages located in the jars directory. ```bash ## windows ./$CHUNJUN_HOME/bin/install_jars.bat ## unix ./$CHUNJUN_HOME/bin/install_jars.sh ``` -------------------------------- ### Install javac-shaded Dependency Manually Source: https://github.com/dtstack/chunjun/wiki/Home Command to install the missing javac-shaded artifact locally after downloading the JAR file. ```shell mvn install:install-file -DgroupId=com.google.errorprone -DartifactId=javac-shaded -Dversion=9+181-r4173-1 -Dpackaging=jar -Dfile=./jars/javac-shaded-9+181-r4173-1.jar ``` -------------------------------- ### Start Flink Standalone Cluster Source: https://github.com/dtstack/chunjun/wiki/Home Initialize the Flink Standalone environment before submitting tasks. ```shell sh $FLINK_HOME/bin/start-cluster.sh ``` -------------------------------- ### Start Yarn Session environment Source: https://github.com/dtstack/chunjun/wiki/Home Initialize the yarn-session environment, requiring HADOOP_HOME and FLINK_HOME to be set. ```shell cd $FLINK_HOME/bin ./yarn-session -t $CHUNJUN_HOME -d ``` -------------------------------- ### Spotless Plugin Error Log Source: https://github.com/dtstack/chunjun/wiki/Home Example error message encountered during the compilation of the ChunJun-core module. ```java [ERROR]Failed to execute goal com.diffplug.spotless:spotless-maven-plugin:2.4.2:check(spotless-check)on project flinkx-core: Execution spotless-check of goal com.diffplug.spotless:spotless-maven-plugin:2.4.2:check failed:Unable to resolve dependencies: Failed to collect dependencies at com.google.googlejavaformat:google-java-format:jar:1.7->com.google.errorprone:javac-shaded:jar:9+181-r4173-1: Failed to read artifact descriptor for com.google.errorprone:javac-shaded:jar:9+181-r4173-1:Could not transfer artifact com.google.errorprone:javac-shaded:pom:9+181-r4173-1 from/to aliyunmaven(https://maven.aliyun.com/repository/public): Access denied to:https://maven.aliyun.com/repository/public/com/google/errorprone/javac-shaded/9+181-r4173-1/javac-shaded-9+181-r4173-1.pom -> [Help 1] ``` -------------------------------- ### Troubleshoot spotless-maven-plugin dependency error Source: https://github.com/dtstack/chunjun/blob/master/README.md This error occurs when the spotless-maven-plugin fails to resolve dependencies. Manually install the missing artifact locally to resolve the issue. ```java [ERROR]Failed to execute goal com.diffplug.spotless:spotless-maven-plugin:2.4.2:check(spotless-check)on project chunjun-core: Execution spotless-check of goal com.diffplug.spotless:spotless-maven-plugin:2.4.2:check failed:Unable to resolve dependencies: Failed to collect dependencies at com.google.googlejavaformat:google-java-format:jar:1.7->com.google.errorprone:javac-shaded:jar:9+181-r4173-1: Failed to read artifact descriptor for com.google.errorprone:javac-shaded:jar:9+181-r4173-1:Could not transfer artifact com.google.errorprone:javac-shaded:pom:9+181-r4173-1 from/to aliyunmaven(https://maven.aliyun.com/repository/public): Access denied to:https://maven.aliyun.com/repository/public/com/google/errorprone/javac-shaded/9+181-r4173-1/javac-shaded-9+181-r4173-1.pom -> [Help 1] ``` ```shell mvn install:install-file -DgroupId=com.google.errorprone -DartifactId=javac-shaded -Dversion=9+181-r4173-1 -Dpackaging=jar -Dfile=./jars/javac-shaded-9+181-r4173-1.jar ``` -------------------------------- ### Build the project Source: https://github.com/dtstack/chunjun/blob/master/README.md Execute these commands within the project directory to compile the source code. ```shell ./mvnw clean package ``` ```shell sh build/build.sh ``` -------------------------------- ### Build ChunJun Project Source: https://github.com/dtstack/chunjun/wiki/Home Commands to compile the project using Maven or the provided build script. ```shell ./mvnw clean package -DskipTests ``` ```shell sh build/build.sh ``` -------------------------------- ### Clone ChunJun Repository Source: https://github.com/dtstack/chunjun/wiki/Home Use this command to download the source code from the official GitHub repository. ```shell git clone https://github.com/DTStack/chunjun.git ``` -------------------------------- ### Create Flink Table with Protobuf-x Format Source: https://github.com/dtstack/chunjun/blob/master/docs/docs_en/ChunJun Extend Data Format/protobuf-x.md DDL configuration for reading Protobuf data using the kafka-x connector and protobuf-x format, mapping Protobuf fields to SQL types. ```sql CREATE TABLE reader ( GroupInfo MAP ,Messages ARRAY< ROW< TAGNAME VARCHAR , TagValue ROW< `Value` ROW< ValueCase INTEGER, ValueBool BOOlEAN, ArrayBool ROW<`Values` ARRAY>, ValueInt32 INTEGER, ArrayInt32 ROW<`Values` ARRAY>, ValueUint32 INTEGER, ArrayUint32 ROW<`Values` ARRAY>, ValueInt64 BIGINT, ArrayInt64 ROW<`Values` ARRAY>, ValueUint64 BIGINT, ArrayUint64 ROW<`Values` ARRAY>, ValueFloat FLOAT, ArrayFloat ROW<`Values` ARRAY>, ValueDouble DOUBLE, ArrayDouble ROW<`Values` ARRAY>, ValueString STRING, ArrayString ROW<`Values` ARRAY>, ValueBytes BINARY, ValueTimestamp BIGINT >, boolx BOOLEAN, Value2 ROW< ValueCase INTEGER , Value2Value BOOLEAN, ArrayBool2 ROW<`Values` ARRAY> >, booly BOOLEAN >, UaDataType INTEGER, Quality BOOLEAN, `Timestamp` BIGINT, TagInfos MAP, ExValues MAP > > -- , `topic` STRING METADATA VIRTUAL -- from Kafka connector -- , `leader-epoch` int METADATA VIRTUAL -- from Kafka connector -- , `offset` BIGINT METADATA VIRTUAL -- from Kafka connector -- , ts TIMESTAMP(3) METADATA FROM 'timestamp' -- from Kafka connector -- , `timestamp-type` STRING METADATA VIRTUAL -- from Kafka connector -- , partition_id BIGINT METADATA FROM 'partition' VIRTUAL -- from Kafka connector ) WITH ( 'connector' = 'kafka-x' ,'topic' = 'liuliu_proto_source' ,'properties.bootstrap.servers' = 'flink01:9092' ,'properties.group.id' = 'luna_g' ,'scan.startup.mode' = 'earliest-offset' ,'format' = 'protobuf-x' ,'protobuf-x.message-class-name' = 'ZPMC.Message.MessageGroupOuterClass$MessageGroup' ,'scan.parallelism' = '1' ); ``` -------------------------------- ### Execute task in Local mode Source: https://github.com/dtstack/chunjun/wiki/Home Run a task in a local JVM process without requiring Flink or Hadoop environments. ```shell sh bin/chunjun-local.sh -job $SCRIPT_PATH ``` -------------------------------- ### Deploy ChunJun Jars to Flink Source: https://github.com/dtstack/chunjun/blob/master/README.md Copies the distribution jars to the Flink library directory. This must be performed on all machines in the Flink cluster. ```shell cp -r chunjun-dist $FLINK_HOME/lib ``` -------------------------------- ### Fix Line Ending Issues Source: https://github.com/dtstack/chunjun/blob/master/README.md Use this command if packaging on Windows and running scripts on Linux to resolve carriage return issues. ```shell sed -i "s/\r//g" bin/*.sh ``` -------------------------------- ### Submit task in Yarn Per-Job mode Source: https://github.com/dtstack/chunjun/wiki/Home Submit a job to Yarn in Per-Job mode, requiring HADOOP_HOME and FLINK_HOME configuration. ```shell sh ./bin/chunjun-yarn-perjob.sh -job chunjun-examples/json/stream/stream.json ``` -------------------------------- ### Submit task in Yarn Session mode Source: https://github.com/dtstack/chunjun/wiki/Home Submit a job to an existing Yarn Session using the application ID. ```shell sh ./bin/chunjun-yarn-session.sh -job chunjun-examples/json/stream/stream.json -confProp {"yarn.application.id":"SESSION_APPLICATION_ID"} ``` -------------------------------- ### Submit task in Standalone mode Source: https://github.com/dtstack/chunjun/wiki/Home Submit a job to a running Flink Standalone cluster. ```shell sh bin/chunjun-standalone.sh -job chunjun-examples/json/stream/stream.json ``` -------------------------------- ### Define Protobuf Message Structure Source: https://github.com/dtstack/chunjun/blob/master/docs/docs_en/ChunJun Extend Data Format/protobuf-x.md A complex Protobuf definition using oneof, repeated, and map types to represent structured data. ```protobuf syntax = "proto3"; package ZPMC.Message; message RepeatedBool {repeated bool Values = 1;} message RepeatedMyBool {repeated bool Values = 1;} message RepeatedInt32 {repeated int32 Values = 1;} message RepeatedUint32 {repeated uint32 Values = 1;} message RepeatedInt64 {repeated int64 Values = 1;} message RepeatedUint64 {repeated uint64 Values = 1;} message RepeatedFloat {repeated float Values = 1;} message RepeatedDouble {repeated double Values = 1;} message RepeatedString {repeated string Values = 1;} message Variant{ oneof Value{ bool ValueBool = 1; RepeatedBool ArrayBool = 2; int32 ValueInt32 = 3; RepeatedInt32 ArrayInt32 = 4; uint32 ValueUint32 = 5; RepeatedUint32 ArrayUint32 = 6; int64 ValueInt64 = 7; RepeatedInt64 ArrayInt64 = 8; uint64 ValueUint64 = 9; RepeatedUint64 ArrayUint64 = 10; float ValueFloat = 11; RepeatedFloat ArrayFloat = 12; double ValueDouble = 13; RepeatedDouble arrayDouble=14; string ValueString = 15; RepeatedString ArrayString = 16; bytes ValueBytes = 17; int64 ValueTimestamp = 18; }; bool boolx = 19; oneof Value2{ bool ValueBool2 = 20; RepeatedBool ArrayBool2 = 21; } bool booly=22; } message MessageItem{ string TagName = 1; //默认optional Variant TagValue = 2; int32 UaDataType = 3; bool Quality = 4; int64 Timestamp = 5; mapTagInfos = 6; mapExValues = 7; } message MessageGroup{ map GroupInfo = 1; repeated MessageItem Messages = 2; } ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.